Run Fedora Apps Under CentOS

Update: I posted my spec files for building HandBrake under CentoS, for anyone that is interested.

Motivation

I run CentOS on my home systems. I used to play with all the different Linux distributions, but having kids plus working full time (did I mention kids?) leaves me with little free time. We use CentOS at work, so it's easier and faster to be able to share my knowledge between work and play. Plus, CentOS is supposed to be a true "enterprise grade" operating system, which implies stability.

However, that stability comes at a price, in terms of the software experience:

  • the software selection tends to be somewhat limited; and
  • generally, the software isn't as new, favoring maturity over bleeding-edge features

The first point becomes less of an issue when extra repositories are added to the base system. Popular repositories include:

The above is by no means a complete list, just a few with which I am familiar. This CentOS wiki page lists many more repos, complete with a brief description of each.

Sometimes, an extra repository isn't enough to fix both of the CentOS software limitations mentioned above. Furthermore, the more obscure and/or less-popular the repository, the more likely it will eventually cease to be maintained, or at least suffer long lapses between updates.

Another problem that is introduced by adding extra repos to a CentOS system is the inevitable "RPM hell" or "dependency hell". The idea here is that repo A wants to install version X of a package, but repo B wants to install version Y, yet both versions cannot co-exist on the system. The problem is compounded when both repos need their respective version to satisfy a dependency.

MythTV

I originally ran into this problem trying to run MythTV on CentOS. MythTV itself has several dozen dependencies that are satisfied by more multiple external repos. Inevitably I'd go to do a system update, then run into some dependency issue or other package conflict. The result was frustration and wasted time.

Initially I tried to solve this problem with virtual machines. I ran into some problems with that approach that are beyond the scope of this document. Even ignoring my particular problems, the bigger issue was that a virtual machine is massive overkill for this situation. What's really needed is a container, which has recently became somewhat of a buzzword. The idea is to run whatever programs you want under the host operating system (i.e. no virtualized instance of another kernel), but keep all the application data, including program files and libraries, walled-off from the rest of the system.

The original "container" on Linux was chroot, which is precisely what I used under CentOS 6. When I upgraded my system to CentOS 7, I found chroot had been deprecated in favor of systemd-nspawn, which has been described as "chroot on steroids". Like many changes that are forced upon us in life, I was initially annoyed and reluctant. But once I understood the concept and worked through a few examples and tutorials, I came to agree with the "on steroids" characterization.

systemd-nspawn

This LWN.net article, Creating containers with systemd-nspawn, by Jake Edge (November 7, 2013), does an excellent job of introducing systemd-nspawn. No need for me to rehash it here.

In short, the idea is very simple: create a directory on your filesystem that has a complete operating system tree. Then, use systemd-nspawn to "boot" into that system. The beauty here is that you now have two (or more!) completely independent system trees: your nspawn (chroot) tree, and your main system tree.

What I do with this setup is to keep my main host system as clean and pristine as possible. I do install more than the base CentOS repo, but I limit myself to just epel, and try to be very judicious about what I install from it. If I want something that requires additional repos, I create an nspawn container for it. This way, I can upgrade my main host system with minimized risk of dependency hell and other conflicts that arise from having too many repos and/or a glut of packages installed.

The only real downside I can see to creating these nspawn containers is extra disk space usage. But we're not talking about ridiculous amounts of storage: 1.8 GB for my MythTV container, and 3.4 GB for my HandBrake container. That's nothing these days, and storage is cheap anyway.

On further thought, there may be a RAM penalty as well. I believe Linux is generally smart enough to only load shared libraries once, and different programs will share the one loaded instance. However, this may break down in an nspawn container. But I'm not sure about this, it requires more research/investigation. Either way, I'm sure it uses less memory than a virtual machine!

The more I think about it, the more I love the idea of being able to spin up a container and do whatever I want in it without worrying about "polluting" my main system:

  • Have you ever wanted to install a package just to try it out, but found it pulled in too many dependencies? Systemd-nspawn to the rescue: spin up a container, install the package, try it out and see if it's worthwhile. If so, install it on your main system. If not, blow it away and don't worry about cleaning up the base system.
  • How about maintaining a clean build environment, such as for rpmbuild, where you want to make sure your builds will work on a base, "clean" system? Nothing worse than trying to share a build with someone only to find out you have some wacky package or version fulfilling a dependency in your build.

Now, what I recently discovered: you're not limited to just spinning up containers of the host OS...

HandBrake

The MythTV problem was solved very simply with a CentOS container, as there are MythTV RPM packages available for CentOS.

But I wanted to use another piece of software on my Linux server, namely, HandBrake. HandBrake is similar to MythTV in that it is a sophisticated program that is built on top of many other pieces of software (i.e. it has a lot of dependencies). So while it's not hard to build, it's not easy to package into a clean RPM. However, a quick web search for "handbrake rpm centos" led me to the helpful page on negativo17.

End of story, right? Well, no. The HandBrake RPM for CentOS is for an older version. So I first spent the better part of a day re-building all the HandBrake dependencies under CentOS 7. And there are a lot of dependencies, mostly introduced by the gtk3 requirement of the latest version of HandBrake.

Fortunately, systemd-nspawn allowed me to create an "island" on which I could install the countless "-devel" packages needed to build all of HandBrake's requirements. I could build and install RPMs quickly without worrying about what kind of mayhem or breakage I might be causing to the host system. Everything was neatly contained within the nspawn instance, politely isolated from my main system.

While the above exercise left me with a working copy of the latest HandBrake version, it was not a perfect solution: it basically turned me into a maintainer of the HandBrake and underlying dependency packages. My goal is to be a simple end-user and actually get "real work" done.

Newer versions of Fedora ship with gtk3, and the good folks at Negativo17 have an RPM of the latest version of HandBrake for Fedora... so what if I could spin up an nspawn Fedora container instead of a CentOS container? Well, it looks like I can, and the HOWTO below illustrates the process.

I'm not sure exactly how robust it is to use nspawn to spin up a "foreign" operating system. My understanding is that the net result is basically all the Fedora programs running under an CentOS kernel. Presumably, some of the Fedora system expects an actual Fedora kernel; it's conceivable that an incompatible system call could be made that would affect the stability of the server. But at least in the case of HandBrake, I so far haven't witnessed any instability with this approach.

HOWTO

Native CentOS Containers

The Drewskiwooskie Blogarooskie blog has a nice article titled "Centos 7 and systemd-nspawn" which gives an overview of creating CentOS 7 nspawn containers. I will borrow from that and include more information for running a Fedora container.

  1. Create a basic centos 7 bootstrap:

    # export ROOT_DIR=/nspawn/container_root
    # yum -y --releasever=7 --nogpg --installroot=${ROOT_DIR} install systemd passwd yum fedora-release vim-minimal
    
  2. Reset the password in the container:

    # systemd-nspawn -D ${ROOT_DIR}
    # passwd
    
  3. Boot your container:

    # systemd-nspawn -D ${ROOT_DIR} -b
    
  4. Start container at boot time. Create a file called /etc/systemd/system/container_name.service, where container_name is what you want to call this container. In case it's not obvious, you will have one of these files for each nspawn container you want to have on your system. The contents of the file should look something like this:

    [Unit]
    Description=Container mythtv_chroot
    Documentation=man:systemd-nspawn(1)
    
    [Service]
    ExecStart=/usr/bin/systemd-nspawn --directory=/nspawn/container_root/ --boot
    KillMode=process
    
    [Install]
    WantedBy=multi-user.target
    
  5. Start ssh within container to access easily:
    1. From within the container, edit the /etc/ssh/sshd_config file. Change the port on which the container's sshd instance will run (i.e. anything but 22), for example:

      # ...
      # Port 2201
      # ...
      
    2. Again, from within the container, set sshd to run at boot time:

      # systemctl enable sshd
      

Fedora Containers

The only real difference from the steps above for a "native" CentOS container is step 1. You need to create a Fedora bootstrap, rather than a CentOS bootstrap. Here is what I did, although I am sure there is an easier way. A web search of "Fedora bootstrap" yields lots of results that look promising, and are likely more elegant.

  1. Download an ISO image of a Fedora "Live" distribution. The filename I ended up with is Fedora-Live-Workstation-x86_64-23-10.iso. Here is one link where you can download the file, but please check the Fedora mirrors page for a site close to you.
  1. Perform a loopback mount of the ISO you just downloaded:

    # mkdir /mnt/live_iso
    # mount -o loop /path/to/Fedora-Live-Workstation-x86_64-23-10.iso /mnt/live_iso/
    
  2. Mount the SquashFS filesystem contained in the Fedora Live ISO:

    # mkdir /mnt/squashfs
    # mount /mnt/live_iso/LiveOS/squashfs.img /mnt/squashfs/
    
  3. Mount the actual bootstrap filesystem contained within the SquashFS filesystem:

    # mkdir /mnt/fedora_bootstrap
    # mount /mnt/squashfs/LiveOS/ext3fs.img /mnt/fedora_bootstrap/
    
  4. After doing all the above, the output of the "mount" command should look something like the following. Note all of these mounts are "ro" (read-only), so in this state they are not actually usable to us:

    # mount
    ...
    /path/to/Fedora-Live-Workstation-x86_64-23-10.iso on /mnt/live_iso type iso9660 (ro,relatime)
    /mnt/live_iso/LiveOS/squashfs.img on /mnt/squashfs type squashfs (ro,relatime)
    /mnt/squashfs/LiveOS/ext3fs.img on /mnt/fedora_bootstrap type ext4 (ro,relatime,data=ordered)
    
  5. Now you can copy the contents of /mnt/fedora_bootstrap to your local filesystem, so that you can actually modify it:

    # rsync -av /mnt/fedora_bootstrap/ /nspawn/fedora/
    # export ROOT_DIR=/nspawn/fedora_bootstrap
    

    With the ROOT_DIR environment variable set, you can go back to step 2 above under "Native CentOS Containers".

  6. Clean up all the no-longer needed ISO mounts:

    # umount /mnt/fedora_bootstrap/
    # umount /mnt/squashfs/
    # umount /mnt/live_iso/
    # rmdir /mnt/fedora_bootstrap/ /mnt/squashfs/ /mnt/live_iso/