Antonio Terceiro: An introduction to the Debian Continuous Integration project
Debian is a big system. At the time of writing, looking at my local package list caches tells me that the unstable suite contains
21306 source packages, and 42867 binary packages on amd64. Between these 42867
binary packages, there is an unthinkable number of inter-package dependencies.
For example the dependency graph of the ruby packages
contains other 20-something packages.
A new version of any of these packages can potentially break some functionality in the
There is documentation available, including a FAQ
for package maintainers with further information about how the system works, how to declare test suites in their packages and
how to reproduce test runs locally. Also available is development information
about debci itself, to those inclined to help improve the system.
This is just the beginning.
A new version of any of these packages can potentially break some functionality in the ruby package.
And that dependency graph is very small. Looking at the dependency graph for,
say, the rails package will make your eyes bleed. I tried it here, and
GraphViz needed a PNG image with 7653 10003 pixels to draw it. It ain t pretty.
Installing rails on a clean Debian system will pull in another 109 packages as
part of the dependency chain. Again, as new versions of those packages are
uploaded the archive, there is a probability that a backwards-incompatible
change, or even a bug fix which was being worked around, might make some
funcionality in rails stop working. Even if that probability is low for each
package in the dependency chain, with enough packages the probability of any
of them causing problems for rails is quite high.
And still the rails dependency chain is not that big. libreoffice will pull
in another 264 packages. gnome will pull in 1311 dependencies, and kde-full
1320 (!).
With a system this big, problems will arrive, and that s a fact of life. As
developers, what we can do is try to spot these problems as early as possible,
and fixing them in time to make a solid release with the high quality Debian is
known for.
While automated testing is not the proverbial Silver Bullet of Software
Engineering, it is an effective way of finding regressions.
Back in 2006, Ian Jackson started the development of
autopkgtest as a tool to test Debian
packages in their installed form (as opposed to testing packages using their
source tree).
In 2011, the autopkgtest test suite format was proposed as a standard for the
Debian project, in what we now know as the
DEP-8 specification.
Since then, some maintainers such as myself started experimenting with DEP-8
tests in their packages. There was an expectation in the air that someday,
someone would run those tests for the entire archive, and that would be a
precious source of QA information.
Durign the holiday break last year, I decided to give it a shot. I initially
called the codebase dep8. Later I renamed it to debci, since it could
potentially also run other other types of test suites in the future. Since
early January, ci.debian.net run an instance of debci
for the Debian Project.
The Debian continuous Integration will trigger tests at most 4 times a day, 3
hours after each dinstall run. It will update a local APT cache and look for
packages that declare a DEP-8 test suite. Each package with a test suite will
then have its test suite executed if there was any change in its dependency chain
since the last test run. Existing test results are published at
ci.debian.net every hour, and at the end of each batch
a global status is updated.
Maintainers can subscribe to a per package Atom feed to keep up with their
package test results. People interested in the overall status can subscribe to
a global Atom feed of events.
Since the introduction of Debian CI in mid-January 2014, we have seen an amazing increase in
the number of packages with test suites. We had little less than 200 packages
with test suites back then, against around 350 now (early June 2014). The ratio
of packages passing passing their test suite has also improved a lot, going
from less than 50% to more than 75%.
There is documentation available, including a FAQ
for package maintainers with further information about how the system works, how to declare test suites in their packages and
how to reproduce test runs locally. Also available is development information
about debci itself, to those inclined to help improve the system.
This is just the beginning. debci is under a good rate of development, and
you can expect to see a constant flux of improvements. In special, I would like
to mention a few people who are giving amazing contributions to the project:
- Martin Pitt has been working on improving debci to support parallel and distributed workers. Being the current autopkgtest maintainer, Martin also already got some bug fixes and fixes into autopkgtest motivated by debci use cases.
- Brandon Fairchild is a GSOC student working on improving the debci web interface to provide more useful information, display information for multiple suites and architectures, plus making the UI work even without Javascript enabled.
- Lucas Kanashiro is another GSOC student, who is working on investigating patterns among packages that fail their test suites, so that we can figure out how we can fix them, or if there are classes of failures that are actually caused by problems in the debci infrastructure.



Last week was the
The summit has been great. It was my third OpenStack design summit, and the
first one not being a PTL, meaning it was a largely more relaxed summit for
me!
On Monday, we started by a 2.5 hours meeting with Ceilometer core developers
and contributors about the Gnocchi experimental project that I've started a
few weeks ago. It was a great and productive afternoon, and allowed me to
introduce and cover this topic extensively, something that would not have
been possible in the allocated session we had later in the week.
Ceilometer had his design sessions running mainly during Wednesday. We noted
a lot of things and commented during the sessions in our
The 

Allowing computer suspend
Major desktop environments such as Xfce or KDE have a built-in
computer suspend feature, but when you use a lighter alternative, things
are a bit more complicated, because basically: only root can suspend
the computer. Possible solutions include:
2013 is nearly all finished up and so I thought I'd spend a little time
writing up what was noteable in the last twelve months. When I did so I
found an unfinished draft from the year before. It would be a shame for
it to go to waste, so here it is.
2012 was an interesting year in many respects with personal
For ages I've been advocating for the dissolution of the tech-ctte,
one of the biggest bugs in the Constitution. However, for the longest
time I was unable to suggest a suitable replacement; there will always
be people whose characters are so flawed that they will appeal to
authority as a matter of first recourse, and those people need some
sort of forum within which to waste time.
Then
At one particular customer, increasing storage needs require that we add
a large file server which should be as fast as possible. In that
context, I'm currently evaluating using Debian GNU/kFreeBSD ("Debian,"
from now on) on ZFS. Not being hampered by a large degree of knowledge
on the subject, here's a blog post with things I've learned so far.
For context, the system on which we'll be installing will have 64 gigs
of RAM, about two dozen SAS disks connected to a RAID controller of
which I'm reasonably sure (though not 100% so) that it supports JBOD
mode, and a PCIe-attached solid state storage device of several
terabytes. If it turns out the RAID controller doesn't support JBOD
mode, I'll probably create as many RAID0 devices as possible and hand
those over to ZFS as "hard disks", instead; this wouldn't be an ideal
solution, but I don't have much of a choice of hard disk attachment,
unfortunately.
That being said, the system on which I'm testing right now is the latest
incantation of samba, an HP ProLiant ML115 server. A system with far
more modest hardware specifications, it has 2G of RAM (which isn't
really enough for ZFS), a single 160G SATA hard disk, and a dual-core
AMD processor of about half a decade ago. And oh yes, no SSD. So for
evaluating ZFS performance it's not ideal, but for evaluating Debian and
ZFS features, it will do.
Anyway, what I've found so far on samba:


This is the fourth post in a series of posts (
I've somehow managed to break firefox: