This article describes how to set up a system for fast testing and package builds on a single machine using btrfs snapshots and chroot. There are tools around that perform similar tasks; even if you choose one of them, I hope this is still useful in demonstrating what those tools do under the covers.
I've used this approach to test Puppet setups in LXC scheduled as a Jenkins job. These days, I'm using it to build packages for our monitoring agent with fpm (https://github.com/jordansissel/fpm). My team has to support an agent build on many Ubuntu LTS releases, 32-bit (old EC2 m1.small / c1.medium instances) and 64-bit architectures. The agent is written in Ruby - rather than trying to support mangled or ancient rubies, we're bundling our own copy of Ruby 1.9.3 with all the gems in the same package. We started with a single binary built on the lowest common denominator platform, but it wasn't ideal. When building across 6 installations ([hardy, lucid, precise] x [i386, x86_64]), running VM's for builds isn't really efficient. Using chroots and snapshots makes building the package consistently on one box a snap.
Requirements
- a modern 64-bit Linux installation that has a build of debootstrap
- around 20GB of disk space
- btrfs-progs
System Configuration
If you're sitting on Mac OSX or Windows, you should fire up VirtualBox (or VMWare / Parallels / etc.) and install your favorite modern Linux distribution. Since I'm using debootstrap, using a Debian variant makes things a much easier. I've had luck building / installing debootstrap on other distros, but that's a lot of work that's out of scope for this article.
Once Linux is installed, make sure you can sudo to root and have a chroot command available.
Install Packages
Debian / Ubuntu
If you're using the newest version you want to build for, you can simply install deboostrap. Otherwise, you'll probably want to install a newer package than comes with the distribution. When I went to build Ubuntu 12.04 images from 10.04, I had to compromise and pull the precise script from the debootstrap tarball.
sudo apt-get install debootstrap
This worked fine on my Ubuntu Lucid VM, YMMV:
wget http://archive.ubuntu.com/ubuntu/pool/main/d/debootstrap/debootstrap_1.0.40~precise1_all.deb
sudo dpkg -i debootstrap_1.0.40~precise1_all.deb
Gentoo / Funtoo
sudo emerge -av dev-util/debootstrap
CentOS (and other RPM distros, adjust to taste):
sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
sudo yum install debootstrap # from EPEL
Disk Configuration
In this section, I'll describe 3 methods for setting up a volume to hold your btrfs filesystem.
If your root disk is already using btrfs and you're comfortable experimenting on it, you can skip head to "create top-level subvolumes."
loopback
This is the lowest common denominator and also consumes the least space. I would not use this in production if avoidable, but it's fine for messing around.
sudo modprobe loop
device=$(losetup --find)
sudo truncate -s 20G snapfs.img
sudo losetup $device snapfs.img
LVM
Assuming you only have one volume group and maybe don't know what it was named (e.g. your distro created it). Adjust to taste if you're comfortable with LVM. This is what I'm using on my primary workstation for now.
vg=$(sudo vgdisplay |awk '/VG Name/{print $3}')
sudo lvcreate -L 20G -n lv_snapfs $vg
device="/dev/$vg/lv_snapfs"
Whole disks / Partitions
When I'm developing inside a VM, I often create an additional virtual disk to do things like this. Most of the GUI-ridden hypervisors will walk you through this. On a production system, it would be awful nice to use a fast RAID. In EC2, this would apply well to any of the non-root drive options like ephemeral disks or EBS volumes. The simplest, though not totally correct approach, is to mkfs the whole drive. This is how I've configured my VM on my machine at home.
drive="/dev/sdc"
device="${drive}1"
sudo sgdisk -Z $drive # wipe any existing partition tables
sudo sgdisk -og $drive # create a new GPT label
sudo sgdisk -n 1:2048:0 -c 1:snapfs -t 1:8300 $drive # create one partition using the whole drive
Create & Mount btrfs Filesystem
# where $drive is set by one of the commands above
sudo mkfs.btrfs -L snapfs $drive
sudo mkdir /snapfs
sudo mount $drive /snapfs
Create Top-level Subvolumes
These don't necessarily have to be subvolumes, but it seems like the right thing to do.
sudo btrfs subvolume create /snapfs/roots
sudo btrfs subvolume create /snapfs/snapshots
Create OS Roots
Now create your OS roots using debootstrap. febootstrap usage is probably similar but I haven't tried it. We're only going to do this once and use snapshots to effectively cache the OS root with fast copies. If you're messing around with this a lot, you might want to use a simple caching proxy to reduce bandwidth and speed up further downloads of the same packages.
sudo btrfs subvolume create /snapfs/roots/ubuntu-precise-amd64
sudo debootstrap --variant=buildd --arch x86_64 precise /snapfs/roots/ubuntu-precise-amd64 http://archive.ubuntu.com/ubuntu/
Update
The sources.list installed by debootstrap only contains main. Before moving on, it's best to update it. Sometimes I'll take a snapshot before this step, but I feel that outdated images are not that useful. If your configuration managemen tools know how to configure roots other than /, you probably want to use whatever apt recipes you have for that instead.
cat > /snapfs/roots/ubuntu-precise-amd64 <<EOF
deb http://archive.ubuntu.com/ubuntu/ precise main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ precise-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu/ precise-security main restricted universe multiverse
EOF
sudo chroot /snapfs/roots/ubuntu-precise-amd64 apt-get update
sudo chroot /snapfs/roots/ubuntu-precise-amd64 apt-get dist-upgrade
# needed for simple scripting
sudo chroot /snapfs/roots/ubuntu-precise-amd64 apt-get install apt-utils lsb-release curl wget
Take a Snapshot
Right after creating the root, I usually take a snapshot in case I make a mistake later on. Snapshots aren't really different from subvolumes in treatment, so they're trivial to snapshot again back to /snapfs/roots/ubuntu-precise-amd64 if it gets messed up.
sudo btrfs subvolume snapshot /snapfs/roots/ubuntu-precise-amd64 /snapfs/snapshots/ubuntu-precise-amd64-debootstrap
Add Packages to Taste
Now is when you install your configuration management system and let it take over for a bit. It's also a good time to consider switching to nslite's nschroot or LXC to containerize its run so it can't affect as much of your host system.
Puppet
This is minimal. You might want to follow their instructions and import the puppet key. Obviously you'll want to import your puppet config, etc. as necessary.
echo "deb http://apt.puppetlabs.com/ubuntu precise main" > /snapfs/roots/ubuntu-precise-amd64/etc/apt/sources.list.d/puppetlabs.list
sudo chroot /snapfs/roots/ubuntu-precise-amd64 apt-get install puppet-common puppet facter openssl
Building Ruby Packages
Debian and Ubuntu modify their Ruby packages and make them pretty difficult to use in some ways. They're also usually quite a ways behind current resulting in wide-spread use of tools like RVM and rbenv. Both of those tools tend to promote the practice of building machine code on edge machines, which some of us neckbeards think is a pretty bad idea. My preferred workaround is to build Ruby into packages that install into /opt/ruby/$version and are accessed by setting PATH, rbenv, or shebang modification.
I prefer to set up most of the build environment in the "root" to save time and bandwidth on each build.
sudo chroot /snapfs/roots/ubuntu-precise-amd64 apt-get install build-essential gpgv libssl-dev zlib1g-dev git-core uuid-dev libreadline-dev pkg-config libffi-dev
cd /snapfs/roots/ubuntu-precise-amd64/tmp
git clone https://github.com/sstephenson/ruby-build.git
cd ruby-build
sudo bash install.sh
Then, each time I build Ruby, I create a snapshot of the root so I can destroy it and any pollution afterwards.
sudo chroot /snapfs/roots/ubuntu-precise-amd64 /usr/local/bin/ruby-build 1.9.3-p194 /opt/ruby/1.9.3-p194
Building 32-bit Binaries From 64-bit Systems
util-linux has a utility called "setarch" that's really handy for building 32-bit binaries. True cross-build is a bit trickier, whereas setarch and its linux32 alias are quite easy and reliable. Simply prefix your commands with "linux32" and the environment will be mangled enough to trick most scripts into thinking they're running on i386 systems. The Ruby configure script builds a cleaner RbConfig with this setup; other packages (like openssl) get totally confused without it.
sudo btrfs subvolume create /snapfs/roots/ubuntu-precise-i386
sudo debootstrap --variant=buildd --arch i386 precise /snapfs/roots/ubuntu-precise-i386 http://archive.ubuntu.com/ubuntu/
# repeate further setup as above ... but prefix chroot as below if you like (it's not really necessary until you get to the builds)
sudo linux32 --uname-2.6 chroot /snapfs/roots/ubuntu-precise-i386 /usr/local/bin/ruby-build 1.9.3-p194 /opt/ruby/1.9.3-p194
nschroot
nschroot is a tiny utility I created to get most of the benefits of containers without all of LXC's statefulness and complexity. It's simple enough to build, just get the code from github (https://github.com/tobert/nslite) and run make. Then simply use "nschroot" instead of plain chroot and the new processes will run in a namespace.
sudo nschroot /snapfs/roots/ubuntu-precise-amd64 /bin/bash
> mount none /proc -t proc
> ps -ef
Many applications don't need proc or sys so you don't need to mount them. Another simple trick is to write a shell script into the root and use it as init, just like in initramfs (a.k.a. initrd).
cat > init.sh << EOF
#!/bin/bash
mount none /proc -t proc
mount none /sys -t sysfs
mount none /tmp -t tmpfs
# run some stuff
puppet --verbose --debug /etc/puppet/manifests/site.pp
# kill everything but $$, be really careful testing this!
for pid in $(ps -eo pid)
do
if [ $pid -ne $$ -a -d /proc/$pid ] ; then
kill -9 $pid
fi
done
umount /proc
umount /sys
umount /tmp
EOF
sudo cp init.sh /snapfs/roots/ubuntu-precise-amd64/init.sh
sudo nschroot /snapfs/roots/ubuntu-precise-amd64 /bin/bash /init.sh
Testing With LXC
LXC is a set of tools on top of facilities that are standard in modern Linux kernels. Cgroups provides a solid resource limiting and tracking system, namespaces provide containers with private pids, ipc, and the like. It also hooks up capabilities to decrease security exposure. Many distributions require modification to run fully in LXC. There's also the lxc-exec option to run a single process in the container, but that requires lxc-exec inside the container.
I haven't tested this bit in a while. Here's most of what was in my old scripts.
now=$(date +%s)
snapshot="/snapfs/snapshots/ubuntu-precise-amd64-test-${now}"
sudo btrfs subvolume snapshot /snapfs/roots/ubuntu-precise-amd64 $snapshot
sudo chroot $snapshot apt-get install lxc
cat > test-$now.conf << EOF
lxc.utsname = puppet-test-${now}
lxc.rootfs = $snapshot
lxc.cgroup.cpuset.cpus = 0,1
lxc.cgroup.cpu.shares = 100
lxc.mount.entry=none proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=none dev/shm tmpfs defaults 0 0
lxc.mount.entry=none dev/pts devpts defaults 0 0
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c 1:3 rw # null
lxc.cgroup.devices.allow = c 1:5 rw # zero
lxc.cgroup.devices.allow = c 1:8 rwm # urandom
lxc.cgroup.devices.allow = c 1:9 rwm # random
lxc.cgroup.devices.allow = c 4:0 rwm # tty0
lxc.cgroup.devices.allow = c 4:1 rwm # tty1
lxc.cgroup.devices.allow = c 5:0 rwm # tty
lxc.cgroup.devices.allow = c 5:1 rwm # console
lxc.cgroup.devices.allow = c 5:2 rwm # pts/ptmx
lxc.cgroup.devices.allow = c 136:* rwm # pts/*
lxc.cap.drop = sys_boot sys_module sys_time
EOF
# install puppet / chef / cfengine / whatever
# if serverless, push your config code in
# if using a server, you probably want to copy pre-cached ssl/cfengine keys in to avoid
# spamming the server with ephemeral test keys
# e.g. puppet masterless
sudo lxc-execute -f test-$now.conf -n puppet-test-$now -- puppet --verbose --debug /etc/puppet/manifests/site.pp
Other Approaches
I started out using all this stuff with Funtoo root & child filesystems. With a little rpath and other linker tinkering you can use Funtoo/Gentoo to build packages with totally self-contained libc and everything.
There are a number of other ways to do this. It looks like Debian uses dchroot / schroot to do almost exactly the same thing in their build system. This is the low-level way to do it and does not require much more than the standard OS tools + btrfsprogs once the initial setup is complete.
With a little adjustment, a similar procedure will work with ZFS (http://zfsonlinux.org/) or even LVM snapshots (but those can be slow). I've even used git branches to do the same thing, but they get slow quite quickly with the amount of data required.
At some point, I might give multistrap (http://wiki.debian.org/Multistrap) a try.
I have experimented with using unionfs. This works great for testing in chroots and LXC on normal filesystems. Another advantage is that you can inspect the COW area to see the files and content that changed relative to the source image. Accounting and managing mounts is more tedious, but for some people it may be a superior base to build on.