Author: ceridwen
The 0.1 alpha release of reprotest has been accepted into Debian
unstable and is available for install at
packages.debian.org or
through apt.
I've been working on redesigning reprotest so that it runs commands
through autopkgtest's adt_testbed interface. For the most part, I
needed to replace explicit calls to Python standard library functions
for copying files and directories with calls to
adt_testbed.Testbed.command() with
copyup and
copydown, and to
use
Testbed.execute() and
Testbed.check_exec() to run commands
instead of
subprocess.
To test reprotest on the actual containers requires having containers
constructed for this purpose. autopkgtest has a test that builds a
minimal chroot. I considered doing something like this approach or
using BusyBox. However, I have a Python script that mocks a build
process, which requires having Python available in the container, and
while I looked into
busybox-python and
MicroPython to keep the
footprint small, I decided that for now this would take too much work
and decided to go straight to the autopkgtest recommendations for
building containers,
mk-sbuild
and
vmdebootstrap. (I also
ended up discovering a
bug
in debootstrap.) This means that to get the tests run requires some
manual setup at the moment. In the long run, I'd like to improve
that, but it's not an immediate priority. While working on adding
tests for the other containers supported by autopkgtest, I also
converted to
py.test so that I
could use
fixtures and
parametrization to run
the Cartesian product of each variation with each container.
With tests written, I started trying to verify that my new code
worked. One problem I encountered while trying to debug was that I
wasn't getting full error output. In
VirtSubproc.check_exec(),
execute_timeout() acts something like a
Popen() call:
(status, out, err) = execute_timeout(None, timeout, real_argv,
stdout=stdout, stderr=subprocess.PIPE)
if status:
bomb("%s%s failed (exit status %d)" %
((downp and "(down) " or ""), argv, status))
if err:
bomb("%s unexpectedly produced stderr output %s'" %
(argv, err))
The problem with this is that if the call returns a non-zero exit
code, which is typical for program failures, stderr doesn't get
included in the error message.
I changed the first if-block to:
if status:
bomb("%s%s failed (exit status %d)\n%s" %
((downp and "(down) " or ""), argv, status, err))
Another example is that autopkgtest calls
schroot with the
--quiet
flag, which in one case was making
schroot fail without any output
due to a misconfiguration. I'm still trying to find and eliminate
more places where errors are silenced.
autopkgtest was designed to be installed with Debian's packaging
system, which handles arbitrary files and directory layouts.
Unfortunately, setuptools is
completely different in a way that
doesn't work well with autopkgtest's design. (I'm sure this is partly
because setuptools has to support all the different major OSes that
run Python, including Windows.) As I discussed
last
week,
autopkgtest has Python scripts in
virt/ that are executed by
subprocess calls in
adt_testbed. Because these scripts import
files from
lib/, there needs to be an
__init__.py in
virt/ to
make it into a package and a
sys.path hack in each script to allow
it to find modules in
lib/. Unfortunately, setuptools
will not
install this structure. First, setuptools will not install any file
without a
.py extension into a package. Theoretically, this is
fixable, the files in
virt/ are Python scripts so I could rename
them. (Theoretically, there's supposed to be some workaround
involving
MANIFEST.in or
package_data in
setup.py, but I have
yet to find any documentation or explanation giving a method for
installing non-Python files inside a Python package.) Second,
however, setuptools does not preserve the executable bit when
installing package files. The obvious workaround, changing the
subprocess calls so that they invoke
python virt/foo.py rather
than
virt/foo.py requires changing all the internal calls in the
autopkgtest code, which I'm loathe to do for fear of breaking it.
(It's not clear to me I can easily find all of the calls, for
starters.)
There are about three solutions to this I see at the moment, all of
them difficult. The first involves using either the
scripts keyword
or
console_scripts entry point in
setup.py, as explained
here.
The
scripts keyword is supposed to preserve the executable bit
according to
this StackExchange
question,
but I haven't verified this myself, and like everything to do with
setuptools I don't trust anything anyone says about it without testing
it myself. It also has the disadvantage of dumping them all into the
common scripts directory. Using
console_scripts involves rewriting
all of them to have an executable function I can refer to in
setup.py. I worry that this would be both fragile and break
existing expectations in the rest of the autopkgtest code, but it
might be the best solution. The third solution involves refactoring
of all the autopkgtest code to import the code in the scripts rather
than running it through
subprocess calls. I'm reluctant to do this
because I think it's almost certain to break things that will require
significant work to fix.
Getting setuptools to install the autopkgtest code correctly is one
blocker for the next release. Another is that autopkgtest's handling
of errors during the build process involves closing the
adt_testbed.Testbed so it won't take further commands.
Unfortunately, this handling runs
before any cleanup code I write to
run outside it, which means that at the moment errors during the build
will result in things like disorderfs being left mounted.
The last release blocker is that
adt_testbed doesn't have any way to
set a working directory when running commands. For instance, the
virt/schroot script always calls
schroot with
--directory=/. I
thought about trying to use absolute paths, but decided this was
unintuitive and impractical. For the user, this would mean that
instead of running something simple like
make in the correct
directory, they would have to run
make
--file=/absolute/path/to/Makefile or something similar, making all
paths absolute. I worry that some build scripts wouldn't handle this
correctly, either: for instance, running
python setup.py from a
different directory can have different effects because Python's path
is initialized to contain the current directory. Changing this is
going to require going deeper into the autopkgtest code than I'd
hoped.
I intend to try to resolve these three issues over the next week and
then prepare the next release, though how much progress I make depends
on how thorny they turn out to be.
.
Debian LTS
This was my twenty-fourth month that I did some work for the 
This is a story about counter-intuitive probabilities and how a small bit of doubt turned out to be very justified.
It begins with the game 
My monthly report covers a large part of what I have been doing in the free software world. I write it for 
Here is my monthly update covering a large part of what I have been doing in the free software world (
The past few days I've been working on 
This summer I m living in a flat five minutes walk from Bucheon
station, near Seoul. Today there is a threat of rain and it s very
humid, which tends to make one feel that time has stopped: it s as if
everyone and everything is waiting for the rain to fall before getting
on with their lives. There are two other reasons why one might think
that time has stopped. There is a household goods shop outside the
station that has a poster up which says last day of business , but of
course it says this every day. A few weeks ago it said last three
days of business instead, but they must have decided that was
starting to look implausible or something. They do various things to
look like they re struggling to get rid of their wares. The other day
they just piled everying up in a huge pile on the street outside the
shop. They have a guy with a megaphone shouting all day about how
cheap everything is in an urgent tone.
The other reason to think time has stopped is that today s coffee in
Starbucks is always the same coffee. On the little blackboard that
all Starbucks branches have they have written: now brewing: hot: iced
coffee blend. iced: iced coffee blend. Every time I order a cup of
today s coffee I have to wait five minutes while they actually brew it
because it seems like no-one else is ordering it. And it tastes
exactly the same as yesterday s coffee.
I had planned to do some work on NBD while here at debcamp. Here's a progress report:
Since quite a long time 
It s rare to be in a position to be able to publicly announce you re looking for a new job, but as the opportunity is currently available to me I feel I should take advantage of it. That s especially true given the fact I ll be at