It s great to see more and more packages in Debian and Ubuntu getting an autopkgtest. We now have some 660, and soon we ll
get another ~ 4000 from Perl and Ruby packages. Both
Debian s and
Ubuntu s autopkgtest runner machines are currently static manually maintained machines which ache under their load. They just don t scale, and at least Ubuntu s runners need quite a lot of handholding.
This needs to stop. To quote Tim The Tool Man Taylor: We need more power!. This is a perfect scenario to be put into a cloud with ephemeral VMs to run tests in. They scale, there is no privacy problem, and maintenance of the hosts then becomes Somebody Else s Problem.
I recently brushed up autopkgtest s ssh runner and the
Nova setup script. Previous versions didn t support revert yet, tests that leaked processes caused eternal hangs due to the way ssh works, and image building wasn t yet supported well. autopkgtest 3.5.5 now gets along with all that and has a dozen other fixes. So let me introduce the Binford 6100 variable horsepower DEP-8 engine python-coated cloud test runner!
While you can run
adt-run from your home machine, it s probably better to do it from an autopkgtest controller cloud instance as well. Testing frequently requires copying files and built package trees between testbeds and controller, which can be quite slow from home and causes timeouts. The requirements on the controller node are quite low you either need the autopkgtest 3.5.5 package installed (possibly a backport to Debian Wheezy or Ubuntu 12.04 LTS), or run it from git (
$checkout_dir/run-from-checkout), and other than that you only need
python-novaclient and the usual
$OS_* OpenStack environment variables. This controller can also stay running all the time and easily drive dozens of tests in parallel as all the real testing action is happening in the ephemeral testbed VMs.
The most important preparation step to do for testing in the cloud is quite similar to testing in local VMs with
adt-virt-qemu: You need to have suitable VM images. They should be generated every day so that the tests don t have to spend 15 minutes on dist-upgrading and rebooting, and they should be minimized. They should also be as similar as possible to local VM images that you get with
vmdebootstrap or
adt-buildvm-ubuntu-cloud, so that test failures can easily be reproduced by developers on their local machines.
To address this, I refactored the entire knowledge how to turn a pristine default vmdebootstrap or cloud image into an autopkgtest environment into a single
/usr/share/autopkgtest/adt-setup-vm script.
adt-buildvm-ubuntu-cloud now uses this, you shold use it with
vmdebootstrap --customize (see
adt-virt-qemu(1) for details), and it s also easy to run for building custom cloud images: Essentially, you pick a suitable pristine image,
nova boot an instance from it, run
adt-setup-vm through ssh, then turn this into a new adt specific "daily" image with
nova image-create. I wrote a little script
create-nova-adt-image.sh to demonstrate and automate this, the only parameter that it gets is the name of the pristine image to base on. This was tested on Canonical's Bootstack cloud, so it might need some adjustments on other clouds.
Thus something like this should be run daily (pick the base images from
nova image-list):
$ ./create-nova-adt-image.sh ubuntu-utopic-14.10-beta2-amd64-server-20140923-disk1.img
$ ./create-nova-adt-image.sh ubuntu-utopic-14.10-beta2-i386-server-20140923-disk1.img
This will generate
adt-utopic-i386 and
adt-utopic-amd64.
Now I picked 34 packages that have the "most demanding" tests, in terms of package size (libreoffice), kernel requirements (udisks2, network manager), reboot requirement (systemd), lots of brittle tests (glib2.0, mysql-5.5), or needing Xvfb (shotwell):
$ cat pkglist
apport
apt
aptdaemon
apache2
autopilot-gtk
autopkgtest
binutils
chromium-browser
cups
dbus
gem2deb
glib-networking
glib2.0
gvfs
kcalc
keystone
libnih
libreoffice
lintian
lxc
mysql-5.5
network-manager
nut
ofono-phonesim
php5
postgresql-9.4
python3.4
sbuild
shotwell
systemd-shim
ubiquity
ubuntu-drivers-common
udisks2
upstart
Now I created a shell wrapper around
adt-run to work with the
parallel tool and to keep the invocation in a single place:
$ cat adt-run-nova
#!/bin/sh -e
adt-run "$1" -U -o "/tmp/adt-$1" --- ssh -s nova -- \
--flavor m1.small --image adt-utopic-i386 \
--net-id 415a0839-eb05-4e7a-907c-413c657f4bf5
Please see
/usr/share/autopkgtest/ssh-setup/nova for details of the arguments.
--image is the image name we built above,
--flavor should use a suitable memory/disk size from
nova flavor-list and
--net-id is an "always need this constant to select a non-default network" option that is specific to Canonical Bootstack.
Finally, let' run the packages from above with using ten VMs in parallel:
parallel -j 10 ./adt-run-nova -- $(< pkglist)
After a few iterations of bug fixing there are now only two failures left which are due to flaky tests, the infrastructure now seems to hold up fairly well.
Meanwhile, Vincent Ladeuil is working full steam to integrate this new stuff into
the next-gen Ubuntu CI engine, so that we can soon deploy and run all this fully automatically in production.
Happy testing!
In more 
I've wanted to ride to Paris for a few months now but was put off by the hassle of taking a bicycle on the Eurostar, as well having a somewhat philosophical and aesthetic objection to taking a bike on a train in the first place. After all, if one already is possession of a mode of transport...
My itinerary was straightforward:
Three months ago version 2.0 of 
I've just recently built the large bulk of VMs that we use for first semester
teaching. This year that was 112. We use the same general approach for these
as our others: get a generic base image up and running, with just enough
configuration complete so a puppet client starts up; get it talking to our
master; let puppet take it from there.
There are pragmatic balances between how much we do in the kickstart versus how
much we do in puppet, but also when we build a new VM from scratch versus when
we clone an existing image, and how specialisation we do in the clone image.
Unfortunately this year we ended up in a situation where our clone image
wouldn't talk to our puppet master out of the box, due to some changes we'd
made to our master set up since the clone image was prepared. We didn't really
have enough time to re-clone the entire set of VMs from a fixed base image, and
instead needed to fix them whilst up. However we couldn't rely on puppet to do
that, since they wouldn't talk to the puppet master.
We needed to manually reset the puppet client state per VM and then
re-establish a trust relationship with the correct master (which is not the
default master hostname in our environment anymore). Luckily, we deploy a local
account with a known passphrase via the kickstart, which also has sudo access,
as an interim measure before puppet strips it back out again and sets up proper
LDAP and Kerberos authentication. So we can at least get into the boxes. However
logging into 112 VMs by hand is not a particularly pleasant task.
In the past I might have tried to achieve this using something like
Before our recent trip to Poland I took the time to create my own e-book, containing the names/addresses of people to whom we wanted to send postcards.
Authoring ebooks is simple, and this was a useful use. (Ordinarily I'd have my contacts on my phone, but I deliberately left it at home ..)
I did mean to copy and paste some notes from wikipedia about transport, tourist destinations, etc, into a brief guide. But I forgot.
In other news 
I just installed, for 
It has been more than three years since last release of 
The last 
After 10 years I finally got around to re-decorating my website. One reason was ICFP, where just too many people told me that I don t look like on my old website any more (which is 
As preannounced 
