Over my time working with Debian packages, I've always been concerned that
I have been missing catchable mistakes by not running all the static checking
tools I could run. As a result, I've been interested in writing some code that
automates this process, a place where I can push a package and come back a few
hours later to check on the results. This is great, since it provides a slightly
less scary interface to new packagers, and helps avoid thinking they've
just been told off by a Developer.
I've spent the time to actually write this code, and I've called it
debuild.me. The code itself is in its fourth
iteration, and is built up from a few core components. The client / server code
(
lucy and
ethel) are quite interconnected, but
firehose works great
on it's own, and is a single, unified (and sane!) spec that is easy
to hack with (or even on!). Hopefully, this means that our wrappers will be
usable outside of debuild.me, which is a win for everyone.
Backend Design
The backend (
lucy) was the first part
I wanted to design. I made the decision (very early on) that everything was
going to be 100% Python 3.3+. This lets me use some of the (frankly, sweet)
tools in the stdlib. Since I've written this type of thing before
(i've tried to write this tool
many,
many,
many,
many times before), so I had a rough
sense of how I wanted to design the backend. Past iterations had suffered from
and overly complex server half, so I decided to go ultra minimal with the
design of debuild.me.
<aside class="left">
You can find the code for the server (lucy) on
my GitHub
</aside>
The backend watches a directory (using a simple
inotify script) and processes
.changes files as they come in. If the package is a source package, a set of
jobs are triggered (such as
lintian,
build and
desktop-file-validate),
as well as a different set for binary packages (such as
lintian,
piuparts
and
adequite). Only people may upload source packages (without any debs) and
only builders can upload binary packages (without source).
The client and server talk using
XML-RPC with BASIC HTTP
auth. I'm going to (eventually) SSL secure the transport layer, but for now,
this will work as a proof of concept.
Since I tend to like to keep my codebase simple and straightforward, I've used
MongoDB as Lucy's DB. This lets me move between
documents in Mongo to Python objects without any trouble. In addition, I
evaluated some of the queue code out there (ZMQ, etc), and they all seemed
like overkill for my problem, and had a hard time keeping track of jobs that
(must never!) get lost. As a result, I wrote my own (very simple) job queue
in Mongo, which has no sense of scheduling (at all), but can do it's job (and
do it well).
Jobs describe what's to be built with a link to the
package document
that the job relates to, and it's
arch and
suite (don't worry about the
rest just yet). Jobs get assigned via natural sort on it's
UUID based
_id,
and assigned to the first builder that can process it's
arch /
suite.
Source packages are considered
arch:all /
suite:unstable (so they always
get the most up-to-date linters on any arch that comes along).
Lucy also allows for uploads to be given an
X-Lucy-Group tag to manage which
set of packages they're a part of. This comes in handy for doing partial
archive rebuilds, or eventually using it to manage what jobs should be run
on which uploads. This will allow me to run much more time-consuming tools
for packages I want to review versus rebuilding to ensure packages don't
FTBFS or aren't adequite.
Client Design
The buildd client (
ethel) talks with
lucy
via
XML-RPC to get assigned new jobs, release old jobs, close finished jobs,
and upload package report data. When the
etheld requests a new job, it also
passes along what
suites it knows of, which
arches it can build, as well
as what
types it can run (stuff like
lintian,
build or
cppcheck.) Lucy
then assigns the builder to that job (so that we don't allocate the same job
twice), and what time it was assigned at.
<aside class="right">
You can find the code for the client (ethel) on
my GitHub
</aside>
Ethel then takes the result of the job (in the form of a
firehose.model tree)
and transmits it over the line back to the Lucy server as a
report (which also
contains information on if the build failed or not), at which point
lucy hands back a location (on the lucy host) that the daemon can write the log
to.
If the job was a binary build, the
etheld process will
dput the package to
the server, with a special
X-Lucy-Job tag to signal which job that build
relates to, so that future lint runs can fetch the
deb files that the build
produced.
Tooling
Ethel runs a set of static checkers on the source code, which are basically
fancy wrappers around the tools we all know and love (like
lintian,
desktop-file-validate,
or
piuparts) which output Firehose in place of
home-grown stdout. This allows us to programmatically deal with the output
of these tools in a normal and consistent way.
<aside class="left">
You can read more about Firehose over in the Firehose
README.rst
</aside>
Some of the more complex runners are made of 3 parts - a
runner,
wrapper
and
command. The server invokes the
command routine, which invokes the
runner (the command just provides a unified interface to all the runners),
who's output gets parsed by the
wrapper to turn it into a Firehose model
tree.
The goal here is that tons of very quick-running tools get run over a
distributed network, and machine-readable reports get filed in a central
location to aid in reviewing packages.
Ricky
In addition to the actual code to run builds, I've worked on a few tools to
aid with using debuild.me for my DD related life. I have some uncommon
use-cases that are nice to support. One such use-case is the ability to rebuild
packages from the archive (unmodified) to check that they rebuild OK against
the target. This is handy for things like
arch:all packages that get
uploaded (since they never get rebuilt on the buildd machines, and FTBFSs are
sadly common) or packages that have had a
Build-Dependency change on them.
Ricky is able to create a
.dsc url to your friendly local mirror, and fetch
that exact version of the package. Ricky can then also use the
.dsc (in a
monumental hack) to forge a
package_version_source.changes file, and sign
it with an autobuild key and upload it to the debuild.me instance. Since it
can also modify the
.changes's target distribution, you can also use this to
test if a package will build on
stable or
testing, unmodified.
Fred
Fred is a wrapper around Ricky, to help with fetching packages that may not
exist yet. Fred also contains an email scraper to read off such lists as
debian-devel-changes, and
add an entry to fetch that upload when it becomes available on the local
mirror, pass it to
ricky, and allow debuild.me to rebuild new packages
that match a set of criteria.
I'm currently playing around with the idea of rebuilding all incoming
Python packages to ensure they don't FTBFS in a clean chroot.
Loofah
Loofah is also another wrapper around Ricky, but for use manually. Loofah
is able to sync down the apt
Sources list, and place it in Mongo for fast
queries. This than allows me to manually run rebuilds on any Source package
that fits a set of critera (written in the form of a Mongo query), which get
pulled and uploaded by
Ricky.
An example script to rebuild any packages that
Build-Depend on
python3-all-dev in Debian
unstable /
main would look like:
<aside class="right">
You can find more queries in the Loofah
examples
</aside>
[
"version": "unstable", "suite": "main" ,
"Build-Depends": "python3-all-dev"
]
Or, a script to rebuild any package that depends on CDBS:
[
,
"$or": [ "Build-Depends": "cdbs" ,
"Build-Depends-Indep": "cdbs" ]
]
You can use anything that exists in the
Sources.gz file to query off of (
including
Maintainer!)
Future Work
The future work on debuild.me will be centered around making it easier for
buildd nodes to be added to the network, with more and more automation in that
process (likely in the form of debs). I also want to add better control over
the jobs, so that packages I upload only go to my personal servers.
I'd also very much like to get better EC2 / Virtualization support integrated
into the network, so that the buildd count grows with the queue size. This
is a slightly hard problem that I'm keen to fix.
I'm also considering moving the log parsing code
out of the workers, so that
the parsing code can be fixed without upgrading all the workers. This would also
drop the
Firehose dep on the client code, which would be nice.
Migration from a debuild.me build into a local
reprepro repo is something
that would be fairly easy to do as well, likely to be done remotely via
the
XML-RPC interface, which calls a couple of
reprepro commands (such as
includedsc and
includedeb) and publishes it to the user's repo. This is
a nice use of the debs that get built, and could also allow debuild.me to be
used like a PPA system, but this allows the user to
not migrate packages
that may contain
piuparts issues.
Normally Skype isn't mentioned by name on this blog, but today I'm making an exception.
When a new scandal broke out in the Australian Army today, one of the first questions asked was how does it compare to the Skype Scandal which had everybody talking (and blushing) in 2011. A concise military reply direct from the General:
"I'd say it's worse than the Skype matter"
(the original Skype Scandal cadets are going on trial in six weeks)
Jedi Council brings cultural change in the Army: from Skype to email
Since the original scandal, there have been plenty of PR-orchestrated headlines about cultural change in the Army. Sadly, it seems that several ranking officers believed that was a cue to simply start using email instead of live feeds
Operating under the operational code name Jedi Council, they've apparently been swapping annotated images of female colleagues and members of the public service.
A more detailed account summarises it:
"They were calling themselves the Jedi Council but they were doing the exact opposite of what Jedis are supposed to do"
While the Skype scandal involved cadets behaving much like college kids elsewhere, this latest scandal involves the officer corp, the most senior officer implicated is allegedly a Lieutenant Colonel. Senior ranking officers like this have given many years of dedicated service to the country. They are widely respected and the vast majority of officers in these positions are unlikely to ever be implicated in such a scandal.
Officers are in leadership roles, it should be no surprise if they have top secret security clearances.
After the revelation that Australia has a direct and high-volume data sharing arrangement with the US Government (and in turn Facebook, Google, etc), there are all kinds of questions about whether a couple of rogue officers have access to pursue their private Jedi Council project using tools like PRISM and Boundless Informant. It's already been confirmed that they were keeping their own Jedi database of their victim's names, phone numbers, home addresses and various clothing measurements and if they are already emailing these photos about on Defence computers, does it seem improbable that they would be using PRISM or other sources to cross-reference their data?
Behavior described as systematic and predatory
It can't be ruled out that some defence computer systems may have direct access to NSA copies of images from those invasive airport body scanners - like these pictures that were not supposed to exist. Just like the British undercover cops stealing the identities of dead children and fully engaging with impressionable young women from environment campaign groups, there are a long list of possible abuses when the security state is over-empowered.
While this latest Army example involves private pursuits, it is not hard to imagine many other permutations: perhaps a couple of rogue Government employees making unauthorised use of PRISM data for a commercial objective and gaining the upper hand on private sector competitors who have no such data feed.
100% trust?
In every organisation there will always be a few people bending the rules. It's human nature, if it wasn't that way, we'd all be robots (or drones). The Army and the Catholic Church tend to be hammered a lot more in the public eye when these transgressions are discovered although statistically their crime rates are no higher than average. The bottom line is that no organisation is perfect and people have to take some responisbility to object to the excessive growth of the security state and keep our data to ourselves as you just never know who you can trust.
Migrating a Debian installation between architectures has always been difficult. The recommended way to
crossgrade an i386 Debian to amd64 Debian was to reinstall the system, move over data and configuration.
For the more brave, in-place crossgrades usually involved chroots, rescue CDs, a lot of ar p tar xf - data.tar.gz and
luck.
I have never been brave when it comes to system administration, have done a lot of architecture migrations with
reinstallation, and have always taken the opportunity to clear out the contamination that accumulates itself when a
system is running for a long time. I would even recommend doing this to most people even now. However, I have a few very
ugly systems in place that are still on i386 because I didn t dare going the reinstallation path.
Doing in-place crossgrades has become a lot easier since wheezy s release, since once now can have both i386 and
amd64 libraries installed in parallel, which allows to replace foo:i386 with foo:amd64 without influencing the other
parts of the system. The process is still full of pitfalls:
An upcoming release of 
So I'd previously talked about migrating machines. From having one virtual machine running "mail" + "web" + "stuff" I've now got three hosts:
The left column, Urgent items, typically must be executed by a certain date (e.g. buying a gift before a birthday or installing a new SSL certificate before the old one expires). A security update or Acrobat reader update does not have this same characteristic. Under this model:
Completing the DRI3 Extension
This week marks a pretty significant milestone for the glorious
This is my monthly summary of my free software related activities. If you re among the people who 
Yesterday, the first milestone of the Havana developement branch of
Ceilometer has been released and is now available for testing and download.
This means the first quarter of the OpenStack Havana development has
passed!
New features
Ten blueprints have been implemented as you can see on the
A few days ago we the 