Debsources (
http://sources.debian.net) is
still under active development. We recently had a Gnome Outreachy
intern, Jingjie Jiang, and we're about to work with 2 GSoC students,
Cl ment Schreiner and Orestis Ioannou.
I will present here the GitHub mirror we've set up, in order to allow
external pull requests to be submitted, and to use the continous
integration service provided by Travis-CI.
GitHub and Travis-CI
Debsources' source code is hosted on
Debian's git servers,
and from there is mirrored to
GitHub. Every time a commit is
pushed (to
master or other branches) or a pull request is open, the
test suite will be automatically run on Travis-CI, and the result
(tests pass or don't) is displayed on GitHub. This allows us to
quickly filter external contributions (when they are submitted on
GitHub), and be sure everything works with our setup, before reviewing
work.
Travis-CI runs the tests on OpenVZ containers. The complete
infrastructure was a bit challenging to setup, but as we now have a
Docker recipe to quicly begin to hack on Debsources, most of the work
could be done using the
Dockerfile instructions.
In average, a run on Travis-CI (which includes git cloning the code
and test data, setup the server, and run the tests suite) takes 7
minutes, which is an ok amount of time to wait for before submitting a
pull request, in my opinion.
Bugs discovered in the process
Setting up this continuous integration infrastructure made me discover
a few bugs.
Python magic does black magic
Debsources runs fine on Debian (not surprisingly), but I got tricked
by black magic when I tried to run it on Ubuntu (which is the OS run
in Travis-CI's containers).
We use the
magic library to guess the type of files we're dealing
with, for instance when we need to decide between rendering a file
(for text files) or downloading it (for binary files).
Here comes the tricky part: the Python bindings for libmagic are not
the same in Debian and Pypi. Debsources uses Debian package
python-magic, which is not in Ubuntu 12.04. Moreover, there's no
Python egg for it on Pypi, which has however another package (called
magic) which provides a different API.
I solved this with a dirty hack, using the fact python-magic lies in a
single file:
mkdir /tmp/python-magic && wget https://raw.githubusercontent.com/file/file/master/python/magic.py -O /tmp/python-magic/magic.py && export PYTHONPATH=/tmp/python-magic/:$PYTHONPATH
It simply downloads the library, saves it in a temporary folder and
includes it in the Python path. Let's see for how long it works before
everything breaks!
Size of a directory
One test in the suite was ensuring the information returned by
ls -l
on a directory and stored in the DB was the right information. Inode
metadata was tested, such as name, permissions, type, or size.
Interestingly enough, the size of a directory was tested, and expected
to be 4096 bytes. The size of a directory actually depends on the
filesystem in use, and on the number of files this directory
contains. We often see 4096 because it's the size of a not-too-big
directory on ext4.
Travis-CI doesn't use ext4:
$ df -T
Filesystem Type 1K-blocks Used Available Use%
Mounted on
/vz/private/209140041 simfs 125829120 103460612 22368508 83% /
none devtmpfs 1572864 8 1572856 1% /dev
none tmpfs 314576 56 314520 1% /run
none tmpfs 5120 4 5116 1%
/run/lock
none tmpfs 1572864 0 1572864 0%
/run/shm
/dev/null tmpfs 786432 171584 614848 22%
/var/ramfs
Simfs is a container filesystem for OpenVZ, on which directories have
different sizes than on ext4:
$ ls -al /
total 0
drwxr-xr-x 23 root root 480 Feb 4 18:08 .
drwxr-xr-x 23 root root 480 Feb 4 18:08 ..
drwxr-xr-x 2 root root 2480 Feb 4 18:20 bin
drwxr-xr-x 2 root root 40 Apr 19 2012 boot
drwxr-xr-x 5 root root 660 Apr 30 13:56 dev
drwxr-xr-x 99 root root 3560 Apr 30 13:56 etc
-rw-r--r-- 1 root root 0 Feb 4 17:56 fastboot
drwxr-xr-x 3 root root 80 Feb 4 17:57 home
[...]
Directory sizes are not even powers of 2.
Hence I changed the test to not check directory sizes. Hopefully this
will help to make Debsources work on more filesystems!
An empty file is hiding
Last but not least, because this
bug is
still open in the wild. A file, which appears to be empty, is not
taken into account by Debsources' updater. This file is
sources/non-free/m/make-doc-non-dfsg/4.0-2/.pc/applied-patches
. It
is present in the filesystem in the container, is not the only empty
file over there, but still doesn't appear in the database, and make
fail the test which counts files.
The test has been commented out (booooooh), so that we still can use
Travis-CI's platform for our GSoC students, before it's fixed.
Conclusion
Making Debsources run automatically on a different platform as the one
we usually use permitted us to spot bugs, write dirty hacks, and
expand the filesystems it's supposed to run on.
Now, let's hope the continuous integration will help our GSoC
students, and let's wish them good luck!