Cherry pick a PR (pull request) from github

Sometime you might want to test pull requests (from github) in local machine  by cherry picking it. This usually happens before it get merged in the upstream repo and released by the project.

I searched the internet but  did not get good reference about how to do it. After little bit of trial and error I came up with below steps.

Cherry picking a pull request:

For example you want to cherry pick https://github.com/fgrehm/vagrant-cachier/pull/164

Cherry picking a commit:

Using Imagefactory to build Vagrant images

Fedora Koji buildsystem and CentOS Community build system i.e. cbs  uses  imagefactory at the back-end of Koji to build Vagrant images.  I have used it as through cbs/koji but wanted to give it a try as I am looking for  easier methods to build adb-atomic-developer-bundle . Specially for developers who don’t have access to Fedora or CentOS build system.

Imagefactory needs a kvm/libvirt hypervisor to build images and it converts them for other providers e.g. Virtualbox or VMware Fusion

Setup:

I have used my laptop (which runs Fedora 23) for this. As I have plan to hack imagefactory and I did not want to damage my laptop’s kvm setup.  So I have used nested virtualization for this. Which means I have a CentOS 7 VM which can run virtual machines.

All below steps are done on a CentOS 7 VM which has a kvm setup in place.

Installation:

Imagefactory is available in Fedora and EPEL repo. But I wanted to try/test the latest code, so I generated RPMs from latest code and then installed the RPMs.

$yum install  rpmdevtools epel-release
$git clone https://github.com/redhat-imaging/imagefactory.git
$cd imagefactory
$make rpm
$cd imagefactory_plugins
$make rpm
$cd ~/rpmbuild/RPMS/noarch/
$sudo yum localinstall ./*

Building Vagrant Images:

For building Vagrant box I have used Ian’s example git repo. He is the maintainer and one of the primary developer for imagefactory.

Below commands are copied from imagefactory-examples git repo.

$ git clone https://github.com/imcleod/imagefactory-examples.git
$ cd imagefactory-examples/vagrant/

Once you are in the “imagefactory-examples/vagrant/” directory, you can see the required files are already there for running commands to generate image for Fedora 22. So we can start running required commands.

For getting a working Vagrant box we need to run three commands (as mentioned below) to create appropriate OVA image. Each command will give a UUID for the intermediate image file name and we need to use the UUID in the next command.

$ sudo imagefactory --debug base_image \
  --file-parameter install_script ./f22-vagrant.ks \
  --parameter offline_icicle true \
  ./f22-minimal-40g.tdl
Output:
xxxxxxxxxxxxxxxxxxxxxxxxxxx
============ Final Image Details ============
UUID: 109cb45f-bbd2-4a27-ba5f-42e2d368be32
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Image build completed SUCCESSFULLY!
$ sudo imagefactory --debug target_image --id 109cb45f-bbd2-4a27-ba5f-42e2d368be32  rhevm

Output:
============ Final Image Details ============
UUID: ce0dce5f-a1d1-4c1a-8e9b-fc56e022a1bc
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Image build completed SUCCESSFULLY!
$ sudo imagefactory --debug target_image --parameter rhevm_ova_format vagrant-libvirt --id ce0dce5f-a1d1-4c1a-8e9b-fc56e022a1bc ova

Output:
============ Final Image Details ============
UUID: 36fcb589-06b8-447b-85bf-ed4715bd2a93
Type: target_image
Image filename: /var/lib/imagefactory/storage/36fcb589-06b8-447b-85bf-ed4715bd2a93.body
Image build completed SUCCESSFULLY!

The last step will generate the F22 image for libvirt provider. You can rename it as f22.libvirt.box (usually Vagrant images have .box extension) and start using it.

$ cp /var/lib/imagefactory/storage/36fcb589-06b8-447b-85bf-ed4715bd2a93.body ./f22.libvirt.box

[1] http://imgfac.org/
[2] https://github.com/redhat-imaging/imagefactory
[3] https://lalatendumohanty.wordpress.com/2015/11/01/kvm-nested-virtualization-in-fedora-23/
[4] https://lalatendumohanty.wordpress.com/2015/05/28/installing-vagrant-in-centos7/

KVM Nested Virtualization In Fedora 23

Nested virtualization allows you to run a virtual machine (VM) inside another VM [1]. Both Intel and AMD supports nested virtualization.

This is very helpful when you are experimenting with the hypervisor related technologies. Example: I will be able to run KVM and Virtualbox both on my laptop but in different VMs. Also I will be able to run local installation of imagefactory to build Vagrant images in a VM  as imagefactory need a hypervisor to run the build . The best part is, I can experiment with all of these inside different VMs without damaging my primary workstation’s hypervisor.

The below steps are done on a Fedora 23 running a Lenovo Thinkpad with Intel chipset.

Step 1: Make sure Intel virtualization (VT) is enabled for the host machine.

$ cat /proc/cpuinfo | grep vmx

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts

The output should contain vmx else Intel virtualization (VT) is not enabled on the machine. You should first fix the setting in the BIOS.

Step 2: Install KVM on the F23 host.

$ dnf install @virtualization

Nested virtualization should be disabled bydefault

$ cat /sys/module/kvm_intel/parameters/nested
 N

Step 3: Enable nested virtualization.  Run below commands as root

  • Temporarily remove the kvm kernel module
      $ sudo rmmod kvm-intel
  • Add the following directive to /etc/modprobe.d/dist.conf
    $ sudo sh -c "echo 'options kvm-intel nested=y' >> /etc/modprobe.d/dist.conf"
  • Insert the kvm module back in the kernel
     $ sudo modprobe kvm-intel

There is alternative way to do the same i.e. pass  kvm-intel.nested=1 on kernel commandline [3]

Step 4: Reboot and verify that nested virtualization is enabled

  • Check that nested virt is enabled
$ sudo cat /sys/module/kvm_intel/parameters/nested
 Y

Step 5: Install the beefy VM. (Lets call it parent VM)

  • I used CentOS 7 minimal ISO i.e. CentOS-7-x86_64-Minimal-1503-01.iso to install the VM through Virtual Machine Manger.
  • Parent VM configuration : 50GB disk, 4GB RAM and 4 vCPUs

Step 6: Enable the VM to use nested virt

  • Go to -> Virtual Machine Manger GUI -> CPU properties -> select “Copy host CPU configuration”

There is also another option i.e. host-passthrough [1] . It is supposed to be more stable then “Copy host CPU configuration” but I have not tried that yet.

Step 7:  Check that Intel virtualization (VT) is enabled in the VM

$ cat /proc/cpuinfo | grep vmx

Step 8: Install KVM inside the VM  [4]

$ yum install qemu-kvm qemu-img
$ yum install libvirt libvirt-python python-virtinst

$ systemctl enable libvirtd
$ systemctl start libvirtd
$ systemctl status libvirtd

Step 9:  Install the child VM inside the parent VM

  • I used Virtual Machine Manger to connect to the parent VM and then install the child VM.
  • Used the same CentOS 7 minimal ISO i.e. CentOS-7-x86_64-Minimal-1503-01.iso to install  the child VM.

[1] https://fedoraproject.org/wiki/How_to_enable_nested_virtualization_in_KVM

[2] http://docs.openstack.org/developer/devstack/guides/devstack-with-nested-kvm.html

[3] http://kashyapc.com/2012/01/14/nested-virtualization-with-kvm-intel/

[4] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/chap-Virtualization_Host_Configuration_and_Guest_Installation_Guide-Guest_Installation.html

vagrant-cachier in Fedora 23 with KVM Libvirt

Vagrant cachier is a very useful plugin for Vagrant users.  It helps to reduce time and  the amount of packages get downloaded from internet between each “vagrant destroy”.

For example, you are using a CentOS 7 image in Vagrant setup and want it to update with the latest packages every time you start working in the guest then the usual work flow is “vagrant up” -> “vagrant ssh” > “sudo yum update -y” -> “Do your stuff” -> “vagrant destroy” .  But the amount of packages get downloaded during yum update and the time consumed for it is somehow undesirable .

vagrant-cachier  keeps the downloaded packages in the file system of the host machine and uses this for the guest as cache. The yum update in the guest gets the packages from the cache  and the time and internet usage is drastically reduced.  Which is really cool!

I tried to install vagrant-cachier on my Fedora 23 laptop with KVM and libvirt and got in to below issue.

Issue:

[root@dhcp35-203 ~]# vagrant plugin install vagrant-cachier
Installing the 'vagrant-cachier' plugin. This can take a few minutes...
Bundler, the underlying system Vagrant uses to install plugins,
reported an error. The error is shown below. These errors are usually
caused by misconfigured plugin installations or transient network
issues. The error from Bundler is:

An error occurred while installing ruby-libvirt (0.5.2), and Bundler cannot continue.
Make sure that `gem install ruby-libvirt -v '0.5.2'` succeeds before bundling.

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

/usr/bin/ruby -r ./siteconf20151027-20676-13hfub7.rb extconf.rb
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
extconf.rb:73:in `<main>': libvirt library not found in default locations (RuntimeError)

extconf failed, exit code 1

Gem files will remain installed in /root/.vagrant.d/gems/gems/ruby-libvirt-0.5.2 for inspection.
Results logged to /root/.vagrant.d/gems/extensions/x86_64-linux/ruby-libvirt-0.5.2/gem_make.out

After installing “libvirt-devel” package the issue got resolved.

[root@dhcp35-203 ~]# dnf install libvirt-devel

[root@dhcp35-203 ~]# vagrant plugin install vagrant-cachier
Installing the 'vagrant-cachier' plugin. This can take a few minutes...
Installed the plugin 'vagrant-cachier (1.2.1)'!

However the vagrant up command again failed.

$ vagrant init centos/7

Then we need to modify the vagrantfile as vagrant-cachier by-default uses NFS to mount the host filesystem in to the guest.

$ cat Vagrantfile
Vagrant.configure(2) do |config|
  config.vm.box = "centos/7"
  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :box

    config.cache.synced_folder_opts = {
      type: :nfs,
      mount_options: ['rw', 'vers=3', 'tcp', 'nolock']
    }
  end

end

Next step was

$ vagrant up
xxxxxxxxxxxxxxxxxxxx
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

mount -o 'rw,vers=3,tcp,nolock' 192.168.121.1:'/home/lmohanty/.vagrant.d/cache/fedora/23-cloud-base' /tmp/vagrant-cache

Stdout from the command:

Stderr from the command:

mount.nfs: Connection timed out

After little troubleshooting it turned out to be a firewall i.e. iptable issue. iptable was blocking the nfs service of host for the operation. As a temporary workaround I removed all the iptable rules from the host.

$ iptables -F

After that “vagrant up” worked fine and I can see the changes vagrant-cachier did in the guest to make the caching work.

Here are the things done by vagrant-cachier for the caching to work.

  • Mounts the ~/.vagrant.d/cache/<guest-name> from host  in the guest on /tmp/vagrant-cache/
  • In Guest
    • It enables the yum caching i.e. sed -i ‘s/keepcache=0/keepcache=1/g’ /etc/yum.conf
    • It creates a symlink of /tmp/vagrant-cache/yum to /var/cache/yum
vagrant@localhost ~]$ ls -l /var/cache
total 8
drwx------. 2 root root 4096 Nov 15 00:08 ldconfig
drwxr-xr-x. 2 root root 4096 Jun  9  2014 man
lrwxrwxrwx. 1 root root   22 Nov 15 00:06 yum -> /tmp/vagrant-cache/yum

vagrant-cachier works fine with CentOS7 guests. However I found an issue with Fedora 23 guests as the default package manager is dnf instead of yum. I have filed an issue with vagrant-cachier and also working on a fix.

Inviting you to FUDCon APAC 2015

Greetings,

We take great pleasure in inviting you to FUDCon APAC 2015, to be held at MIT COE, Pune from 26 to 28 June 2015.

What is FUDCon?

FUDCon is the Fedora User’s and Developer’s Conference, a major free software event held in various regions around the world, annually one per region.
The 2015 APAC edition of FUDCon will be held in Pune.

At FUDCon, you get to-

  • Attend sessions on technology introductions and deep dives.
  • Participate in hands-on workshops and sessions where like-minded people get together to discuss a project/technology.
  • Be a part of Hack-fests where contributors work on specific initiatives, ranging from packaging, writing features to UI mock-ups.
  • ‘Un-Conference’ at Bar Camps, where people interested in a wide range of issues/topics can come together to share and learn.
    The speakers (that could be you) will pitch the topics they are interested in discussing in the morning of the event.
    Anyone eager to listen in or participate can attend the session at the allotted time slot.
    The most exciting part here is that anybody can stand up and talk in front of the large audience and share their views on the topic of their interest.

Why FUDCon?

FUDCon has always been a high quality event where Fedora contributors from different parts of the world converge together to bounce ideas off each other.

Fedora has the best people, professionals and developers working for it and they all collaborate on events like FUDCon. We have delegates flying in from far places to come and talk to our audience about Fedora, the future of Fedora (which is the upstream of a premier Enterprise Operating system).
Thus, FUDCon would be a great opportunity for you to engage with some of the best minds in the industry, update yourselves on the latest and the cutting edge technological innovations and experience the excitement and the thrill of the open source environment.

For more information, visit our website – fudcon.in

Looking forward to seeing you there.

For any assistance contact: Rupali Talwatkar at rtalwatk@redhat.com

Reference : https://fedoraproject.org/wiki/Fudcon/Invitation_Letter_College

Linux Container Track at FUDCon, Pune 2015

FUDCon is the Fedora Users and Developers Conference, a major free software event. It is an annual event per region (e.g. APAC). This year FUDCon APAC is  happening at Pune, India From 26th June to 28th June 2015.  For details check out the FUDCon India home page.

FUDCon is always free to attend for anyone in the world. If you are looking for open source and free software contributors/developers, learn latest/new technologies, learn how to contribute to Fedora and other upstream projects, want to hangout and hack on interesting stuff , it is one of the finest place to be present.

This year we are having a Linux container track in FUDCon for full day. The idea of a separate track came because of the more number of talks/workshops proposals submitted related to containers and high interest from community about learning Linux containers e.g. Docker related technologies.

To start with, we have a talk about “Running Project Atomic and Docker on Fedora” in the main track by me and Aditya on Saturday, 27th June 2015. And the container track is on next day i.e. Sunday 28th June 2015.

Here are the list of talks and workshops planned for the container track.

Note : The topics of the container track has changed from the time I had published the blog.  So the I have updated the above list.

For the workshops, you need to carry your laptop. We are planning provide preconfigured virtual machine images for KVM and Virtual box. Also planning to distribute the images using use usb sticks/thumb drives. So if you have usb/thumb drive, please carry them with you as it will help you/others to quickly set-up their environments.

So if you are interested to learn about Linux containers or hack around Linux containers, Docker, come join us on container track in FUDCon 2015, Pune.

Installing Vagrant in CentOS7

Recently I started working on creating a Vagrant box[1]  for Atomic Developer Bundle project. For testing the Vagrant I wanted to use CentOS 7 with the KVM + Vagrant.

However yum install for Vagrant packages failed in CentOS as Vagrant packages are not available in CentOS7. However these are available through CentOS software collection SIG

Thanks to SCL(softwarecollections.org) community for making the Vagrant packages available.

So here are the steps that will install Vagrant with libvirt provider in CentOS7 machine.

$ sudo yum -y install centos-release-scl
$ sudo yum -y install sclo-vagrant1 qemu-kvm
$ sudo scl enable sclo-vagrant1 bash

# Start libvirtd
$ sudo systemctl start libvirtd

# Permanently enable libvirtd
$ sudo systemctl enable libvirtd

Refer documentation [2] of Vagrant project for more details.

[1] https://github.com/projectatomic/adb-atomic-developer-bundle

[2] https://www.vagrantup.com/

[2] https://docs.vagrantup.com/v2/

Comments/Suggestions are welcome!