You might not know who Peter is because he s not very visible on Debian mailing lists. He s very active however and in particular on IRC. He was an admin of the OFTC IRC network at the time Debian switched from Freenode to OFTC. Nowadays he s a member of the Debian System Administration team who runs all the debian.org servers. If you went to a Debconf you probably met him since he s always looking for new signatures of his GPG key. He owns the best connected key in the PGP web of trust. He also wrote caff a popular GPG key signing tool. Raphael: Who are you? Peter: I m Peter Palfrader, also known as weasel. I m in my early 30s, born and raised in Innsbruck, Austria and am now living and working in Salzburg, Austria. In my copious free time, other than help running Debian s servers I also help maintaining the Tor project s infrastructure. Away from the computer I enjoy reading fiction (mostly English language Science Fiction and Fantasy), playing board games and going to the movies. Weather permitting, I also occasionally do some cycling. Raphael: How did you start contributing to Debian? Peter: I installed my first Debian the week slink came out. That was Debian 2.1 for the youngsters, in early 1999. The one thing I immediately liked about slink was that Debian s pppd supported RAS authentication which my university s dial-up system required. No way I d go back to SuSE 5.3 when I had working Internet with my Debian box. During that year I started getting involved in the German language Debian channel on IRCnet which got me in contact with some DDs. Christian Kurz (<shorty>) was working on Debian QA at the time and he asked my help in writing a couple of scripts. Some of that work, debcheck, still produces parts of the qa.d.o website, tho the relevance of that nowadays is probably negligible. While trying to learn more Perl earlier, I had written a program to produce syntax highlighted HTML for code snippets in various languages. I didn t really know what I was doing but it kinda worked, and probably still does since I still get mail from users every now and then. I figured that it would be really nice if people could just get my software together with Debian. According to code2html s Debian changelog the initial release of the package was done on a weekday at 2:30 in the morning early in 2000, and if my memory serves me correctly, shorty uploaded it shortly afterwards. I started packaging a couple of other piece of software and in the same year I sent my mail to the debian account managers to register my intent to become a DD. No new developers where being accepted at that time since the DAMs wanted to overhaul the entire process so I wasn t surprised to not get any immediate reply. Of course what the silence also meant was that the mail had been lost, but I only learned of that later when I took all my courage to ask DAM about the status of application a couple months later. Once that was sorted out I was assigned an AM, did the usual dance, and got my account late in November 2000. Raphael: Four years ago, the Debian System Administration team was a real bottleneck for the project and personal conflicts made it almost impossible to find solutions. You were eager to help and at some point you got dropped as a new member in that team. Can you share your story and how you managed the transition in the difficult climate at that time? Peter: Ah, that was quite the surprise for an awful lot of people, me included. Branden Robinson, who was our DPL for the 2005-2006 term, tried to get some new blood added to DSA who were at the time quite divided. He briefly talked to me on IRC some time in summer 2005, telling me I had come recommended for a role on the sysadmin team . In the course of these 15 minutes he outlined some of the issues he thought a new member of DSA would face and asked me if I thought I could help. My reply was cautiously positive, saying that I didn t want to step on anybody s toes but maybe I could be of some assistance. And that was the first and last of it, until some fine November day two years later I got an email from Phil Hands saying I ve just added you to the adm group, and added you to the debian-admin@d.o alias. and welcome on board . *blink* What!? My teammates at the time were James Troup (elmo), Phil Hands (fil), Martin Joey Schulze and Ryan Murray (neuro). The old team, while apparently not on good terms with one another, was however still around to do heavy lifting when required. I still remember when on my first or second day on the team two disks failed in the raid5 of ftp-master.debian.org aka ries. Neuro did the reinstall once new disks had arrived at Brown University. I m sure I d have been way out of my league had this job fallen to me. Fortunately my teammates were all willing and able to help me find whatever pieces of information existed that might help me learn how debian.org does its stuff. Unfortunately a lot of it only existed in various heads, or when lucky, in one of the huge mbox archives of the debian-admin alias or list. Anyway, soon I was able to get my hands dirty with upgrading from sarge to etch, which had been released about half a year earlier. Raphael: I know the DSA team has accomplished a lot over the last few years. Can you share some interesting figures? Peter: Indeed we have accomplished a lot. In my opinion the most important of these accomplishment is that we re actually once again a team nowadays. A team where people talk to one another and where nobody should be a SPoF. Since this year s debconf we are six people in the admin team: Tollef Fog Heen (Mithrandir) and Faidon Liambotis (paravoid) joined the existing members: Luca Filipozzi, Stephen Gran, Martin Zobel-Helas, and myself. Growing a core team, especially one where membership comes with uid0 on all machines, is not easy and that s why I m very glad we managed to actually do this step. I also think the infrastructure and our workflows have matured well over the last four years. We now have essential monitoring as a matter of course: Nagios not only checks whether all daemons that should be running are in fact running, but it also monitors hardware health of disks, fans, etc. where possible. We are alerted of outstanding security updates that need to be installed and of changes made to our systems that weren t then explicitly acked by one of us. We have set up a centralized configuration system, puppet, for some of our configuration that is the same, or at least similar, on all our machines. Most, if not all, pieces of software, scripts and helpers that we use on debian.org infrastructure is in publicly accessible git repositories. We have good communication with other teams in Debian that need our support, like the ftp folks or the buildd people. As for figures, I don t think there s anything spectacular. As of the time of our BoF at this year s DebConf, we take care of approximately 135 systems, about 100 of them being real iron, the other virtual machines (KVM). They are hosted at over 30 different locations, tho we are trying to cut down on that number, but that s a long and difficult process. We don t really collect a lot of other figures like web hits on www.debian.org or downloads from the ftp archive. The web team might do the former and the latter is pretty much impossible due to the distributed nature of our mirrors, as you well know. Raphael: The DSA team has a policy of eating its own dog food, i.e. you re trying to rely only on what s available in Debian. How does that work out and what are the remaining gaps? Peter: Mostly Debian, the OS, just meets our needs. Sure, the update frequency is a bit high, we probably wouldn t mind a longer release cycle. But on the other hand most software is recent enough. And when it s not, that s easy to fix with backports. If they aren t on backports.debian.org already, we ll just put them there (or ask somebody else to prepare a backport for us) and so everybody else benefits from that work too. Some things we need just don t, and probably won t, exist in Debian. These are mainly proprietary hardware health checks like HP s tools for their servers, or various vendors programs to query their raid controller. HP actually makes packages for their stuff which is very nice, but other things we just put into /usr/local, or if we really need it on a number of machines, package ourselves. The push to cripple our installers and kernels by removing firmware was quite annoying, since it made installing from the official media next to impossible in some cases. Support for working around these limitations has improved with squeeze so that s probably ok now. One of the other problems is that especially on embedded platforms most of the buildd work happens on some variation of development boards, usually due to increased memory and hard disk requirements than the intended market audience. This often implies that the kernel shipped with Debian won t be usable on our own debian.org machines. This makes keeping up with security and other kernel fixes way more error prone and time intensive. We keep annoying the right people in Debian to add kernel flavors that actually boot on our machines, and things are getting better, so maybe in the future this will no longer be a problem. Raphael: If you could spend all your time on Debian, what would you work on? Peter: One of the things that I think is a bit annoying for admins that maintain machines all over the globe is mirror selection. I shouldn t have to care where my packages come from, apt-get should just fetch them from a mirror, any mirror, that is close by, fast and recent. I don t need to know which one it was. We have deployed geodns for security.debian.org a while ago, and it seems to work quite well for the coarse granularity we desired for that setup, but geodns is an ugly hack (I think it is a layer violation), it might not scale to hundreds or thousands of mirrors, and it doesn t play well with DNSSEC. What I d really like to see is Debian support apt s mirror method that I think (and I apologize if I m wronging somebody) Michael Vogt implemented recently. The basic idea is that you simply add deb mirror://mirror.debian.org/ or something like that to your sources.list, and apt goes and asks that server for a list of mirrors it should use right now. The client code exists, but I don t know how well tested it is. What is missing is the server part. One that gives clients a mirror, or list of mirrors, that are close to them, current, and carry their architecture. It s probably not a huge amount of work, but at the same time it s also not entirely trivial. If I had more time on my hands this is something that I d try to do. Hopefully somebody will pick it up. Raphael: What motivates you to continue to contribute year after year? Peter: It s fun, mostly. Sure, there are things that need to be done regularly that are boring or become so after a while, but as a sysadmin you tend to do things once or twice and then seek to automate it. DSA s users, i.e. DDs, constantly want to play with new services or approaches to make Debian better and often they need our support or help in their endeavors. So that s a constant flow of interesting challenges. Another reason is that Debian is simply where some of my friends are. Working on Debian with them is interacting with friends. I not only use Debian at debian.org. I use it at work, I use it on my own machines, on the servers of the Tor project. When I was with OFTC Debian is what we put on our machines. Being a part of Debian is one way to ensure what Debian releases is actually usable to me, professionally and with other projects. Raphael: Is there someone in Debian that you admire for their contributions? Peter: That s a hard one. There are certainly people who I respect greatly for their technical or other contributions to Debian, but I don t want to single anybody out in particular. I think we all, everyone who ever contributed to Debian with code, support or a bug report, can be very proud of what we are producing one of the best operating systems out there.

---

Thank you to Peter for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

No comment Liked this article? Click here. My blog is Flattr-enabled.

If you were building a digital container to store your personal data, what would it look like? Personal data being information associated with you: your contacts, your photos, the web pages you ve visited, the places you ve been, the messages you ve sent and received, and so on. In short, your stuff. Here s my personal wish list of technical requirements:

• It has to be made of free software, of course
• It must keep my data secure, while allowing me to share it when and how I want to
• It needs to handle a range of different data types natively, and be extensible to new types, from photos to real-time sensor data
• It should be able to collect my data from many different places where it is being created and stored
• It should have a rich API, so that I can create applications which access my data
• If I want to, I should be able to host it myself, on my own hardware, without compromising my ability to access and share it

Of course, this isn t merely an academic exercise, as my new day job at Singly is about building exactly this type of system. With a technical team including Jeremie Miller of Jabber and XMPP fame, our goal is to develop a personal data platform which meets these criteria and more. There s a lot of work to do, but today, you can check out the code and run a locker of your own, which can sync in data from Facebook, Twitter, Google, Foursquare, Github and dozens of other services. It s a bit of a bear to set up, particularly if you don t already have API keys for these services, but that s fairly normal at this early stage of development. If you try it, or have thoughts about what we re doing, please let me know in the comments.

The new release switches to GDBus and adds an application icon. Together with the recently released Modem Manager 0.5 it's now able to fetch the balance information via USSD on Huawei and ZTE modems as well. This blog is flattr enabled.

The Border Gateway Protocol is what network operators use to exchange the various routes forming the Internet. It's a pretty critical piece of infrastructure. Most providers are probably using proprietary solutions using dedicated hardware, but free software on comodity hardware can also be an option for some networks. In Debian, we already have Quagga and Bird that both implement BGP on top of Linux for quite a while. Back in time, FOSDEM feb. 2006, Ging is being demonstrated. Immediately my mind races with something like: waow, we could port OpenBGPD on that! OpenBGPD is another BGP routing daemon. This new effort was started in 2003 by Henning Brauer for OpenBSD. To sum up its goal, it was meant to be secure, reliable, lean, easy to configure, fast and memory efficient. This is quite the opposite of Quagga codebase, which is inherited from an older project, Zebra which was mostly documented in Japanese. Another issue I have seen hapenning with Quagga on Linux is that its blocking implementation and the time that the kernel needs to insert/remove routes made the daemon unresponsive enough to created cascading failures. The IPv4 Internet is nowdays made of more than 350 000 prefixes, so when a network peer fails, the router needs to process those changes while continuing to answer other peers. Otherwise, they will kill the session meaning yet another 350 000 prefixes to process on both sides. And again when the session is restored. Leading to more downtime, flaps, and failures. OpenBGPD was ported to FreeBSD. The port is maintained by Hiroki Sato. Starting with this port and using libbsd made the creation of a package for Debian GNU/kFreeBSD pretty doable while at DebCamp11. It probably still has some pretty rough edges and the manpage needs to be reviewed (and some unsupported features trimmed, maybe), but the package should be usable. So OpenBGPD is now in Debian, and it could be one of the killer features of a port that some describe as a toy OS. Unfortunately I did not yet have access to proper network equipment to test the package on real production scenario (with 2 or 3 full BGP views). If you have a lab (or feel adventurous) and have access to such gears, please have a try!

About 24 hours ago I rebooted the system that runs the secondary DNS for my zone and a few other zones. I d upgraded a few things and the system had been running for almost 200 days without a reboot so it was time for it. Unfortunately it didn t come back up. Even more unfortunately the other DNS server for my zone is ns.sws.net.au which is also the only other server for the sws.net.au zone. Normally this will work because the servers for the net.au zone have a glue record containing the server IP address. So when asked for the NS records for the sws.net.au domain the reply will include the IP address of ns.sws.net.au. The unfortunate part was that the IP address was the old IP address from before the sws.net.au servers changed to a new IP address range, I wonder whether this was due to the recovery process after the Distribute IT hack [1], as forgetting to change a glue record is not something that I or the other guy who runs that network would forget. But it is possible that we both stuffed up. The DNS secondary was an IBM P3-1GHz desktop system with two IDE disks in a RAID-1 array. It s been quite reliable, it s been running in the same hardware configuration for about four years now with only one disk replacement. It turned out that the cooling fan in the front of the case had seized up due to a lot of dirt and the BIOS wouldn t let the system boot in that state. Also one of the disks was reporting serious SMART problems and needed to be replaced poor cooling tends to cause disk errors. It seems that Compaq systems are good at informing the user of SMART problems, two different Compaq desktop systems (one from before the HP buyout and one from after) made very forceful recommendations that I replace the disk, it s a pity that the BIOS doesn t allow a normal boot process after the warning as following the recommendation to backup the data is difficult when the system won t boot. I have a temporary server running now, but my plan is to install a P3-866 system and use a 5400rpm disk to replace the 7200rpm that s currently in the second position in the RAID array. I ve done some tests on power use and an old P3 system uses a lot less than most new systems [2]. Power use directly maps to heat dissipation and a full size desktop system with big fans that dissipates less than 50W is more likely to survive a poorly cooled room in summer. Laptops dissipate less heat but as their vents are smaller (thus less effective at the best of times and more likely to get blocked) this doesn t provide a great benefit. Also my past experience of laptops as servers is that they don t want to boot up when the lid is closed and getting RAID-1 and multiple ethernet ports on a laptop is difficult. Finally I am going to create a third DNS server for the sws.net.au domain. While it is more pain to run extra servers, for some zones it s just worth it.

• [1] http://www.zdnet.com.au/distributeit-data-unrecoverable-post-hack-339317149.htm
• [2] http://doc.coker.com.au/environment/computer-power-use/

As our GSoC students are approaching quite well, we've already numerous features included in current master, which are worth mentioning. You can try all bellow mentioned features on our demo server. Grid editing provides really simple way of editing data, by simply clicking on it while browsing table. Just try it on some table. Performance suggestions are part of completely reworked server status and can give you valuable information how to tune your server. You can see it on demo server. Another nice addition to server status page is live monitor, which charts basic server information in real time. Again, try it on demo server (you need to press start for now, this will probably go away soon). Other big, thought not that fancy change is complete support for routines. This includes stored procedures/functions, triggers and events. For example see routines in sakilla database. Of course there is much more to mention, like improving of AJAX in lot of places, GIS support. On the other side, we've also recently decided to remove support for Microsoft Excel import and export as used PHPExcel library seems to suffer some licensing issues, which prevented phpMyAdmin from being included in Fedora. You can still use CSV or OpenDocument formats though.

Filed under: English Phpmyadmin 0 comments Flattr this!

I like so much the git feature branch workflow, that in the early days of development on Python with Mercurial I created some named branches; well, that is something you should not do.

In Mercurial, the changeset contains the branch name, so you cannot develop on a separated (named) branch and then merge on default and hope that branch goes away, because it will stay.

What do I do now? Python Mercurial repository is quite big (around 200Megs) so I wanted to avoid to re-check it out. Thanks to the help of the folks on #mercurial (on freenode IRC network) I found my solution: strip the branch!

Please note that strip is dangerous. Use it only as last resort, and mind you can lose data with it. That said, it's a very powerful tool :) My main aim was to remove completely those named branches, leave no traces, and lose the changes I made on them. Another important aspect is that I didn't merged those branches on default.

So, how to get rid of a named branch:

$ hg strip "branch($ BRANCHNAME )"

and re-iterate for all the branches you have, that's it. Now, to be completely sure they were removed and no spurious changes are in the repository, you can:

$ hg pull -u
$ hg outgoing

and if it says "no changes found" you're sure that those branches are really gone.

This is a direct followup on my blog posting Finding packages for deinstallation on the commandline with aptitude. In the meanwhile on more alias for finding obosolete packages made it into my zsh configuration. It s an alias to find installed libraries, -data, -common and other usually only automatically installed packages, which are not marked as being installed automatically nevertheless:

alias aptitude-review-unmarkauto-libraries='aptitude -o "Aptitude::Pkg-Display-Limit=( ^lib !-dev$ !-dbg$ !-utils$ !-tools$ !-bin$ !-doc$ !^libreoffice   -data$   -common$   -base$ !^r-base ) !~M"'

And yes, this pattern is slightly larger than those from the previous posting, so here s the used filter in a little bit more readable way:

(
  ^lib
    !-dev$
    !-dbg$
    !-utils$
    !-tools$
    !-bin$
    !-doc$
    !^libreoffice   
  -data$   
  -common$   
  -base$
    !^r-base
)
!~M

It matches all non-automatically installed packages whose name starts with lib , but is neither a debug symbols package, a development header package, a documentation package, a package containing supplied commands, nor a libreoffice package. Additionally it matches all non-automatically installed packages ending in -data, -common, or -base, but excludes r-base packages. Of course you can then mark any erroneously unmarked library by pressing M (Shift-m). If you press g for Go afterwards and wonder why nothing to remove shows up, be reminded that the filter limit is active in this view, too. So press l for Limit and then Ctrl-u to erase the current filter limit of this view and press enter to set the new (now empty) filter, et voil Hope this is of help for some others, too.

Documentation! That is what the next version of Lintian will have. In 2.5.1, Lintian
had support for adding options in the lintianrc. Unfortunately, the
user had to guess the syntax, which makes the feature less useful.
With the next release, the manpage and the example config file[1]
will contain documentation and examples. The User Manual[2] also saw some improvements. Particularly, it will
mention that lintian can process a .changes file and that it does
cross-package checks. Of course, it also documents the format of the
new Vendor Profiles. Though even with these improvements, the current
documentation still leaves much to be wanted. The last unpack script (unpack-srcpkg-l1) was finally removed and
was replaced by a smarter version of index collection. This means
that Lintian can now schedule index in parallel with a few other
collections (namely unpacked, which is the current bottleneck). The speed improvement is probably not noticeable, but it was also a
good time to upgrade our index collection to properly handle
multi-tarball source packages. Previously Lintian only extracted the
index from the base tarball. The fields collection also saw some attention. Previously it
extracted fields from the control file (in the .deb, etc.) into a
directory. Each field became a file and the value of the field would
be written in the field. Today, the collection basically looks like
this:

[...]
if (-d 'fields') 
    system('rm', '-rf', 'fields') == 0
        or die 'Could not remove old fields directory';
 
[...]

It has been deprecated the minute Lintian::Collect API saw a little
upgrade to fetch these fields itself. This change should remove some
unnecessary file I/O in a normal run[3]. If you have a friend with his/her own little home-grown Lintian check
(or collection)[4], they will have to update it to A) not depend on
fields, B) use the Lintian::Collect API for accessing fields and C)
depend on index for source packages if it uses the index information.
For the latter, Lintian::Collect has grown support for source package
index. Lintian has also seen a lot of decrufting in the 2.5.2 development
cycle. The new test suite now features skeletons ( standard base
environments ) and template support for all package test suites[5]. This means it takes a lot less of writing (or copy/waste) to generate
a simple specialized test in the low-level test suites. As a part
of cleaning up the test-suites, I got to run:

$ git rm t/ source,debs /*/copyright

without breaking a single test. I have to admit that commit 8584996
is among my favorite ones in the 2.5.2 development cycle. Finally, if you are one of the few that have ever used the
--packages-file option, then I am pleased to inform you that it will
be deprecated in 2.5.2 and replaced by --packages-from-file. The
old option had some specialized format, whereas the new option parses
the file (or stdin) as one package per line . The latter being far
more "find -name '*.deb' "-friendly. In case you are wondering why --packages-file is deprecated, here is
the packages-file parser from lintian 2.5.1:

    while (my $line = <$pkgin>)  
	chomp($line);
	my (undef, undef, undef, $file) = split(/\s+/, $line, 4);
	$pool->add_file($file);
     

- [1] Installed as /etc/lintianrc [2] Installed as /usr/share/doc/lintian/lintian. html/,txt.gz [3] Namely, 2 opens + 2 closes + 1 read and one write for each field
in every package (the fields were cached, once they were read). Not
to mention a directory listing and having to remove the files again
later. [4] Of course, you do not have such a thing Because we all know
that no one (except this not too close friend of yours) messes with
Internal APIs of other projects [5] The exception being that the changes suite does not have a
standard base environment, since it would be completely unused.

Wondering what is going on with the port of Debian to the Minix 3 microkernal? Here's an update. dpkg Ported Well, the big news as I mentioned in my last post is that dpkg has been successfully ported. I haven't sent the patches to the maintainers just yet as they need review and clean up but I will do that ASAP. The architecture name is defined as minix_i386. (There is no k because I am using the Minix libc as well as the kernel.) I am currently targetting squeeze. A Maze of Twisty Dependencies All Different Armed with my new dpkg, I have begun building packages which has been an interesting experience. The Debian packaging systems greatest asset dependencies is also its most frustrating during initial bootstrapping. Take debhelper for instance; nowadays most packages depend on it and I personally wouldn't think of creating a .deb without it but it build-depends e.g. po4a which build-depends on...debhelper. Also some programs which are assumed to be present on linux such as ldconfig are simply not available on Minix. So currently theres a lot of dpkg --force-depends and even --force-all going on. Of course this is only a temporary problem until I get a full set of base packages built. The equivs package is coming in handy too. In lieu of working .debs, I have been using Minix native packages to fill in the gaps. They use NetBSD pkgsrc which is a new experience for me. I can't say I'm that impressed but they work and, being uptodate, are a good source of patches. Works in Progress The two big packages I need to work on next are perl 5.10.1 and gcc 4.4.5. I haven't touched the latter yet. Perl is failing in its second-stage bootstrap for reasons I haven't gotten to the bottom of just yet. I suspect it has more to do with deficiencies in my build environment than a problem in perl itself. libc Deficiencies As I mentioned last time, the situation has improved tremendously since Minix switched over to using the NetBSD libc but I am still running into problems with unimplemented APIs. Take getpwent(3) and getpwuid(3) for instance. Being able to map between user ids and names is kind of important you know? Even worse instead of being completely absent some of these exist as stubs. So they compile and link without warning but always return ENOFILE and so on. I was surprised to find that there is no standardized API for getting information on mounted filesystems. (Which e.g. df(1) needs.) Linux uses getmnt(3) similar but incompatible with System V. The BSDs use getmntinfo so probably this is what Minix will implement. GNU programs are good about portability but a lot of Free software is unfortunately Linux-centric. Hopefully the Debian FreeBSD port already has patches for these cases. There are several annoyances like this but I suppose I shouldn't complain too much. I am using an unreleased version of Minix which is under heavy development and I must say the upstream developers have been good about responding to my concerns. Looking forward: fakeroot, grub2, ext2 fakeroot is a very handy development tool that frees a developer from having to create .debs as root. Alas it won't work under Minix as it depends on LD_PRELOAD and Minix doesn't even support shared libraries. fakeroot-ng sounds a bit more promising as it uses the ptrace(2) system call which is implemented in the Minix kernel though I don't know how well. Minix is now multiboot compliant so it is possible to use grub2 instead of the Minix boot loader. Definitely something I'm going to look at soon. Minix now supports ext2 (though not ext3 or ext4) in addition to its native file system. For compatability this should become the default in the Debian port. While the kernel support is part of the standard Minix system, the userland support (e2fsprogs) is not for some reason. So I'll have to see about porting that. Can I Help the Porting Effort? Sorry not yet. Though if you are determined I'll try and help you get set up. Hopefully in a month or less I will be able to distribute a pre-alpha version so I suggest waiting for that. I am definitely going to be looking for outside help.

The construction, deployment, and use of software involves a number of roles. I do not have an exhaustive list, but I have a couple in my list that seem to be surprising to some developers. In the traditional Unix (mainframe) model, there are three roles:

• developers actually create the software; they include architects, coders, documenters, testers, maintenance programmers, etc; their deliverable is a release tarball
• sysadmins install the software on a host, and install new versions when appropriate; often, the sysadmin is expected to carefully read README files, and construct or tweak configuration files accoriding to the specific needs of their users; their deliverable is some files in /usr/local and /etc, plus perhaps an announcement e-mail to an internal mailing list
• users use the software; they have no deliverable in this context

In my personal reality distortion bubble, this model is badly broken, and has been since the middle of the 1990s. Here's an updated model:

• users control their own machines, and are not experts on the software, or on using computers at all, and requiring them to be would be to limit their freedom in important ways; the big change is that users now have much more control of their computing environment than before
• sysadmins mainly work for companies or other organizations; they need not be experts on what their users do, and in fact often aren't, but they know how to handle systems that are by nature too complicated or hard for the layman to handle, because of performance, reliability, security, or other such needs
• distro developers create a system consisting of thousands or tens of thousands of components (mostly provided by upstream developers), and integrate and adjust things so that they work together; they also monitor upstream projects and other news sources for updates, and provide those
• upstream developers actually create the software; they include architects, coders, documenters, testers, maintenance programmers, etc;
  their deliverable is a release tarball (no change from earlier)

A common misconception is that distro developers merely compile software and shove the binaries into glorified tarballs. That is, of course, part of the job, but it's also the part that most distro's tools completely automate. There's a lot more to do, though. For example, just keeping track of known security problems among thirty thousand packages, and making or finding fixes for them, and testing the fixes, and releasing updates for them, takes up quite a bit of effort. Another thing distro developers have to spend effort on is dealing with the multitude of build systems upstream projects use. The freedom to experiment with different solutions is one of the cornerstones of free software development. However, when it comes to build systems, many of them are not serious experiments, and instead are quick hacks thrown together to imperfectly solve problems that sometimes should not exist at all. This highlights one role prominently still missing: build engineer. Their task is to solve the problem of taking the upstream source code, dealing with build-time configuration issues, actually building the software, and then installing it in whatever location the person installing the software wants it installed. Further, their task is to provide the software to do this in a sensible manner. Build engineers already exist, of course. For example, the developers of the autotools suite are such. Every upstream project also has someone who sets up autotools for their project, or some other build tools, or writes them from scratch for that project. However, what's lacking, perhaps, is recognition that build engineering is an important part of software development. Mostly, build systems are set up (or, worse, written) by upstream developers to fit their narrow view of how things should work, a view anchored in the old, broken model of roles. This attitude matters, because for the most part, build systems do not seem to be written to the same quality standards as the rest of the upstream software. For example, when an upstream developer writes a Makefile, and does not consider distro development, they'll often make it hard to differentiate between install-time location and run-time location of various components of the software. Thus, it might be hard for the Debian packager to get the software installed into debian/tmp rather than /usr/local, because the build system hardcodes debian/tmp/usr/lib/foo/plugins as the location of the plugins, since that's where the software gets installed, even if, in the Debian scenario, it's not where it gets run from. Luckily for much software these problems have been solved or avoided. Indeed, as crufty as autotools is, it's one of the things that has kept upstreams sane about their build systems, by providing such good tools that it's more work to write a new, bad one than to use autotools. We currently have roving bands of translators, who move from project to project and translate them into various natural languages. I would like to see a future where there are people dedicated to build engineering, who go from project to project and set up or fix their build problems. A good build system, once set up, should require a minimum of work to keep working. I'd also like to see much improvement in the build software. As good as autotools has been to free software, I'm sure there's a lot of room for improvement. For example, autotools requires setting up a bunch of boilerplate files in each project, which seems like something that could be avoided by relying on specific conventions. However, those improvements are a separate topic. I might go into them some day, if I decide to start developing build tools myself, but until then, I'll let those who do that decide how they want to do it. PS. There are lots more missing roles in my model of free software development, of course. For example, security engineers (find and fix problems, but also provide ways to prevent problems, or limit their scope), internal politicians (keep the project's human wheels from squeaking too much), diplomats (keep different projects from clashing too much), release engineers, etc. Someone should make an exhaustive list.

I got a call from Mum today to inform me that my grandmother had passed away last night, Brisbane time, at 94 years of age. If you want to get all genealogically correct, she's technically my step-maternal grandmother, but she's the grandmother I spent the most time with, growing up, so I'm not going to split hairs. It's come as a bit of a shock, because despite having had progressively worse dementia for probably over a decade, she's been in incredibly good physical condition. I think it was in her 70's that she had a heart valve replaced, and I for one had been expecting that to be the thing to wear out and take her down slowly. I last saw her in the flesh in January, when the above photo was taken, and more recently a few weeks ago on Skype. She was recently moved from an aged-care hostel in Toowoomba to a nursing home in Brisbane, as her dementia had progressed beyond the point that she could be cared for in the previous facility. Mum had visited her earlier in the day yesterday, and she was fine, but apparently the staff found her unconscious at around 5pm when they went to get her for dinner. She was taken to the Royal Brisbane and Women's Hospital where it was determined she had suffered from a stroke, and she passed away later that evening at around 11:30pm. So all in all, she had a very good innings, and of all the ways to leave this world, this was one of the quicker and more painless ways to do it. That's the problem with a quick exit - it's usually completely unexpected. It's looking like the funeral will be on Friday, and I've got a flight tentatively booked to leave on Wednesday and get in on Friday morning and return again on Monday. At the time I arranged this, I'd completely forgotten that I was flying to Calgary on Monday night to babysit the son of friends, while they go to the US Consulate for a visa appointment. Fortunately this travel doesn't conflict with that travel, it just means I'll be away from work for longer, and spending a lot of time inside pressurized tubes. I'm fairly confident with all of the short international trips, I'm going to look like a drug mule or something, and someone's going to thoroughly cavity search me.

FreedomBox activities at Debconf11 I spent the last two weeks of July 2011 in Banja Luka. The occasion was the annual Debian developer's conference, Debconf11 and preceding work week known as Debcamp. This was my tenth successive year attending Debconf, and I had a very productive and pleasant time! The facilities were good, the local team was friendly, enthusiastic, and very helpful, and in addition to giving three talks and hosting a couple panel discussions, I managed to put a burst of energy into work on FreedomBox. Several other developers working on FreedomBox were also present, and a good number of Sheeva and Dream plugs were evident in the hacklabs sporting new FreedomBox stickers. Working together in the same place for several days, we made good progress on several projects, and also had some great discussions about what we want to do going forward. image building tools For some time, I've been working towards a light-weight tool set to build FreedomBox software images. Shortly before Debconf started, I chose the name 'freedom-maker' for this tool and shared a link to a readable copy of my git repository with other developers I expected to work with in Banja Luka. With input from Bert Agaz and Jonas Smedegaard during Debcamp, freedom-maker went from almost useful to actually useful. It still deserves work to be more useful to others, but I have now pushed a copy of the git repository to git.debian.org so that we can take advantage of the tools supported there to enable others to more easily contribute to the code. Very soon, Bert plans to add support to freedom-maker for using Lars Wirzenius' vmdebootstrap to build x86 images suitable for testing in a virtualized environment. At the same time, we plan to refactor the existing code slightly to enable lists of desired packages for the various image flavors we expect to produce independently of the configuration for each specific image building tool. Jonas continued in parallel to work on his alternate packaging toolset boxer. It offers some potentially interesting features for the future, and we may eventually merge some or all of it into freedom-maker, but for now it remains a separate utility. uAP user space tools Several weeks ago, we received from Marvell the source code to two user space programs that are necessary for configuring and monitoring the binary firmware provided for the uAP wireless chip used in the DreamPlug. Early during my stay in Banja Luka, I packaged these for Debian as uaputl and uapevent, and I am pleased to note that they were quickly accepted into the archive and are now present in Debian mirrors. u-boot Another bit of code received very shortly before Debconf started was the source for the version of u-boot shipped by Globalscale in the DreamPlug units we're working with. During Debcamp, Clint Adams passed a copy of this source to Jason Cooper, who was already trying to add support for the DreamPlug to upstream u-boot, but had stalled due to a lack of information. Jason has now merged his own work with the sources we got from the manufacturer, and is making good progress towards merging DreamPlug support into upstream u-boot. Once that happens, we should be able to flash our Sheeva and Dream plug devices with a u-boot image built from the source in the Debian u-boot package, in the process enabling things that matter to us like the ability to boot from an ext2 partition, and hopefully the ability to execute command scripts from that partition instead of having to hard-code kernel filenames in flash. This will allow us to support the ongoing effort in Debian to move away from the need for kernel symlinks. DreamPlug kernels With respect to kernels, another work stream at Debconf primarily involving H ctor Or n and Nick Bane was to analyze the current state of the patches from Marvell and Globalscale used to support the DreamPlug against both upstream and current Debian kernel sources. To my surprise and our collective pleasure, the remaining patch set required against current upstream kernels is much smaller than we previously believed! There are still several patches critical to us that are not merged upstream, but the work remaining to be able to build images for our devices from mainline and Debian kernel source trees now seems like something we might be able to complete before Debian's next stable release. One of our discoveries during the u-boot and kernel work during Debconf was that Globalscale did not obtain a new machine id for the DreamPlug, but instead re-used the one for the GuruPlug series, despite there being some differences in the hardware that require at least one additional driver. After much discussion, we plan to continue using the existing machine id instead of requesting another, particularly because the ARM kernel community has apparently stopped issuing new ids for the moment. We will add a new kernel config option for the DreamPlug, however, and are likely to build distinct Sheeva and Dream kernel packages that do not require initrd for use in FreedomBox images, even if doing so is not strictly necessary. This will allow us to optimize both the in-memory footprint and boot times for our devices. software configuration Another area of investigation in Banja Luka was technology for package configuration. Mirsal Ennaime performed various tests using debconf and Config::Model, with some results reflected in this commit relating to configuring the bitcoind daemon in the bitcoin package for Debian. identity and trust management While we did not actually do any FreedomBox specific work on the trust management layer we know is necessary, after several rounds of conversation, I am now more convinced than ever that the right path forward is to base our trust relationships on OpenPGP keys using GnuPG and Monkeysphere as starting software elements. Our thinking to date is captured on an Identity Management page in the wiki. communication services Another thing that became fairly clear to me during discussions at Debconf is that in the near term, planning to build communication services around XMPP is the approach most likely to give good results. Investigating the software choices available to build an interesting XMPP infrastructure is now a high priority for me. Jonas has done some work towards configuring and integrating ejabberd or Prosody, I've started studying yate as a possible call manager and VoIP server choice with XMPP/jingle support, and we await with great interest a release from the Buddycloud developers to evaluate as a possible basis for deploying social network services. Some of these software choices will lead us to use Apache as our web services base technology because of the need for features that only it supports well among daemons that are Free Software. Jonas completed packaging GNU Sip Witch for Debian, and it is now available in the mirror network. Tzafrir Cohen and Jonas did some initial testing on its use. documentation A number of new wiki pages were written (or at least started) in order to sum up ideas, design various aspects of FreedomBox, and reflect discussions that happened during DebConf11. A lot of work is needed to complete these pages though, as well as others to capture more of the current state of the project. press coverage Finally, while in Banja Luka I got some great press coverage for FreedomBox! On Sunday the 24th, I was interviewed by the main television network serving the Republika Srpska. This led to a couple of minutes of coverage near the top of the national news program that night, immediately following the lead story about the President and several ministers appearing at Debian Day that morning to help open the conference. This interview was later re-used in another TV program that summarized Debconf11. On the morning of Thursday the 28th, I was part of a small group that spent more than an hour meeting with the Minister of Science and Technology in his office, and the relationship between Debian and our work on FreedomBox was one of the items of discussion in that meeting and the associated press conference. I'm told this resulted in more press coverage, but if true I have not seen it yet. summary On Friday afternoon the 29th, I gave a talk in the main Debconf program containing a FreedomBox Progress Report . In it, I talked about the structure of the FreedomBox Foundation, progress the foundation has made, and the work that was still then underway in Banja Luka. It was streamed live over the internet, and replays are available online. The reaction from Debian developers present was very positive, which was good to learn since by that time my energy level was quite low after the nearly two weeks of intense technical and social interaction that is Debconf! All in all, we got lots of work done on FreedomBox in Banja Luka, enough that I think at least the next few steps along the road towards an eventual "1.0" release of a reference implementation are now much clearer than they were two weeks ago!

I don't want to keep my GPG private keys on my laptop, so I've been keeping them on an encrypted USB drive instead, and loading them onto the laptop only when needed. That's cumbersome, when making Debian packages, and needing to sign the packages frequently. So I'm going to start using subkeys instead. However, I couldn't find good documentation for that, so after figuring things out, I wrote something that will hopefully be useful for others. It's at http://wiki.debian.org/subkeys for your fixing pleasure.

Hello, this is the fourth report for the project. First, I d like to
apologize for being late, I took a short trip over the weekend and
missed the deadline. This time I ve good news for my project. As
expected I ve made the necesary changes to have Eucalyptus support ARM
images. The approach was extending and exposing an arch field that
will be used by libvirt s XML domain file. I m working currently on
having the involved functions working with the new extension and I
expect to have this done over the middle of the next week. What was done so far

- Extended the ncInstance_t struct on util/data.h adding the archId
parameter. This struct keeps the information regarding a node (nc),
say RAM, Kernel, etc. Currently it has no way to set an arch in order
to be used later. typdef struct ncInstance_t
...
char reservationId[CHAR_BUFFER_SIZE];
char userId[CHAR_BUFFER_SIZE];
char archId[CHAR_BUFFER_SIZE]; // arch field added
int retries;
...

The Eucalyptus team was OK with this approach that I ve proposed few
weeks ago. For all my changes the approach is maintain compatibility
with existing features.
We still have to define if we are setting a default value, say amd64,
or keep it unset until the user sets one. For now we will be supporting
amd64 and arm as valid fields. - Modified the allocate_intance() function to support the archId
field. This involved adding a new parameter and detecting whether its
present and setting de value accordingly. This function is later used
by the doRunInstance() function in node/handlers_kvm.c - Modified the involved functions that are called by the KVM handlers
and result in passing parameters to the XML file generator for being
used by libvirt s and KVM and have the image running under
Eucalyptus. The new parameters and field are not required and only
being used if they are set with a value. The archId parameter has been
added as the last one, so existing function calls can work. This could
be rearranged later, but will involve a more extensive revision with
Eucalyptus developers. - I ve set a git local repo in order to keep track of the changes to
Eucalyptus. Since this part is a sort of a Eucalyptus branch I took
this approach. Steffen and I are discussing whether to release this as
a pkg-eucalyptus branch (as dpatch, using existing packaging). Either
way I m putting the diff files form my git repo over the wiki and will
put them on my site too. I plan to use git-svn in order to send them
later to the pkg-eucalyptus repo once we are settled with this. - Updated the project s wiki[1] page with information regarding all that s
involved on my work, so it can be re-played or resumed later. 1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog What I plan to to over the next weeks
-
- Finish the modifications to the relevant functions. Expected dealine:
next week.
- Do integration tests:
Write a test program, there s an example under node/ that could be
helpful.
Generate the XML domain definition from Eucalyptus functions. The
full cycle: handler_kvm -> allocate_instance -> get_instance_xml ->
have the XML file properly generated -> qemu-kvm runs the instance
using the proper emulator (arm in our case).
- Test runnning the image I ve worked on the project s begining.
- Fixing bits and cooperate with the pkg-eucalyptus team on the
packaging bits.
- Talk to Eucalyptus so they adopt our patches.
- Rock and roll :) What would be nice to have after the project ends
-
- The Debian packaging *needs* work. I ve already discussed this on the
pkg-eucalyptus mailing list. I ve also committed myself to help on
that side even after the SoC ends.
- Talk with upstream and Ubuntu to coordinate the packaging efforts and
modifications to the software.
- Get adopters to use our work :) By now, few weeks about to finish the SoC, I can say that it s been a
good experience and as I stated I m planning to keep maintaing/working
on the software after that. I ve already joined the pkg-eucalyptus team
and plan to contribute actively. I m sad I didn t managed to get Debconf
this year, time was a constraint again, hopefully we can meet in 2012! I
guess that all by now! Best regards,
Rud

DebConf 11 -- Banja Luka, Bosnia and Herzegovina I'm writing this on my way back from my fifth DebConf, waiting in the Frankfurt Airport for my flight to Munich then another wait for a flight back to Los Angeles. DebConf is the annual convention of Debian Developers and other interested in how Debian Linux (or Debian BSD) works, and want to help make it better. Debian is an international volunteer project, and Debconf is held each year in a different city, normally not on the same Continent two years in a row. Only one so far has been held in the United States, last year Debconf 10 was in New York City. (Mexico and Canada have also hosted Debconfs.) DebConf is a week long conference, with one day designated as "Debian Day" with more introductory level talks, frequently in the local language, and local people are invited to learn about Debian. The rest of DebConf is held in English. This year had over 350 attendees from a couple of dozen countries. Unlike most computer conferences, DebConf is free to attend. (Professional and Corporate memberships are available for those who would like to financially support DebConf.) If you apply early enough, you can request help paying for food, lodging, and travel as well, but such support is limited and you have to justify it. This is all make possible by our generous sponsors, who include some major corporations who use Debian. This year DebConf was held in Banja Luka, Bosnia and Herzegovina. The Bosnian bid won partially based on the generous support of the Bosnian government, not only financial and letting us use one of their buildings for the conference, but also help with getting visas and other such aid. (Visas are not required from the US or most of Europe, but they are from some other countries.) I arrived on Saturday about 24 hours after I left home, starting with too little sleep. I actually managed to eat dinner before passing out in my hotel bed. The hotel room I had in Hotel Vidovic (accent on the c) was nicer than any I have stayed at in Europe (although I always chose by price), and the conference price was less than half what a similar room goes for at a US hotel. There was a nice breakfast buffet included in the hotel cost. (Hot omelets and sausages as well as the usual cold cuts, cheese, fruit, juice, coffee, tea, cereal, yogurt, etc.) The in-room ethernet connection worked all the time except one morning, although it was not particularly fast. The hotel is relatively new, the Lift (elevator) had a date of 2005 on it and I suspect that is when the building was constructed. Sunday was Debian Day, and I went to a couple of sessions before deciding to take a nap in the afternoon. My nap wound up being from 3pm to after 11pm, sleeping through dinner. I was awake for an hour or so checking my email, and then slept till 4am. That was the morning I made it to breakfast before 7am. This was the first time I ate breakfast at normal breakfast hours for a full week for many years. After breakfast Monday, I spent several hours at the front desk helping give out badges, conference bags, and t-shirts. Lunches and dinners were in the Hotel Bosna, across the street from venue. For lunches we were served soup, salad, a plate of food, and desert. For dinners there was a buffet. The food served at Bosna was fairly good, but repetitive from day to day. (I've had much worse hotel food in the US for much more money.) Tuesday was more or less a repeat of Monday, with me opening the front desk by myself and not nearly as many new arrivals. The sessions varied of course, and I played "Debconf Experimental 5-card Mao". (Mao is a card game, popular in Cambridge and at DebConf.) Wednesday was the Day Trip. One day at DebConf is dedicated to socializing while seeing some of the local culture of the hosting city/country. This year there was an option of rafting (with some white water) or visiting a historical monastery and hiking to a waterfall with several grain mills powered by diverted water. This is the first time I had seen one actually grinding grain. (It's mainly a historical tourist attraction, but I don't know what happens to the flower made.) We then met at the rafting location for a BBQ lunch. Apparently there was some miscommunicaiton and the rafting place was not prepared for 70 or 80 people who wished to go rafting, so some did not get to raft until after lunch, and I'm not sure all that wanted to did. Dinner was back at Bosna. Thursday was another normal conference day, with the new wrinkle of distributing food tickets since Bosna was not willing/able to follow our instructions on who should be allowed to eat at Debconf expense. Those who wished to dine with the other Debconf attendees could purchase the tickets from the front desk, although this was not well publicised and few people took us up on it. The special "Conference Dinner" was Thursday night. This was held at a restaurant on the 14th floor of a building a few blocks away, and was a fancier buffet. There didn't seem to be much difference between the first and second course, and they ran out of desert before many people got any. This one was not as memorable as the one in Mexico, but that one was for the wrong reasons, including an unplanned indoor waterfall. After the dinner I played Mao again, this time till 2am. Friday was the only day I did not arrive at the venue in time for the first session (10am). Saturday I opened the front desk by myself again. In the afternoon was the "Debbugs Skills Exchange" that was requested by several people, so Collin Watson, Ian Jackson and I gave some information about it and Don Armstrong (who has done most of the recent coding on debbugs) participated via voip. (Debbugs is software for the Debian Bug Tracking System, often called the BTS.) Collin and Ian are emeritus members of the BTS team, and not active in it, while I am handling the spam filtering and haven't done much with the rest of the BTS. This was in the small "Meeting Room" (rather than the Auditorium or the round room) so we don't have to worry about video archives of it being available. (All the sessions is the two main rooms were streamed live on the internet, and will be edited and archived on http://meetings-archive.debian.net/pub/debian-meetings/2011/debconf11/.) We created a mini intro to the BTS document at that session that should be cleaned up and published. Less than two hours after that talk, the conference was over and teardown was started. As usual, the teardown and cleanup of what we had set up over several days was done is several hours, with many people helping. Sunday was leaving day. I was one of the people on the 11am charter bus to Zagreb DebConf arranged. The two such charter busses were an excellent idea, since the normal busses and trains would not have been able to manage that many extra passengers. This actually took longer than the planned 4 hours due to the long lines at the border crossing on Sunday afternoon. Both Bosnia and Croatia check passports both coming and leaving, so it is checked on both sides of the river that is the border between the countries. The check is pretty cursory unless you are from a country that needs a visa. I then had a plane flight to Frankfurt, and am currently in the middle of my 11 hour layover here. After that I fly to Munich early in the morning, have an 8 hour layover, and fly to Los Angeles. This schedule was in order to get flights at something resembling reasonable cost when I did it. From what I saw of Banja Luka, it's a nice place to visit, although most of the people do not speak English. Vegetarians may have a hard time finding meals, and sometimes they try to serve fish to them. (Assuming vegetarian means "no meat".) Transportation to Banja Luka is a bit hard, there are only six or so flights per week to the local airport, and one of the airlines is not on the international ticket sites. Most DebConf attendees flew to Zagreb then took busses, trains, vans, or taxis to Banja Luka (a 2.5-5 hour ride depending on border lines, road construction, etc.). The weather was rainy off and on throughout Debconf, but the day of the Day trip the rain held off until most of us were back and we had a lovely day. Temperatures were warm but not overly hot, and I never needed my jacket. There are many large alarm clocks in Banja Luka, (such as the one pictured here, across the street from the venue) but there doesn't seem to be a way to change the time setting or put them on snooze.

Boy, the last three months have been very eventful, developmentally. First off, about 2 weeks after she turned 1, Zoe started walking in earnest. Within about 2 weeks of that there was no stopping her. It's funny to think she's only been walking for under 3 months and yet she's very stable on her feet. Secondly, she developed a mild egg allergy. Sarah was giving her some scrambled eggs (one egg's worth) for lunch one day, and she mixed in some cream cheese, and Zoe absolutely scoffed it all down, so she scrambled another egg and she scoffed that down too. She then broke out in some hives around her mouth and Sarah noticed that she was coughing a bit and a little wheezy when she put her down for her afternoon nap. Upon subsequent consultation with her paediatrician and a blood test, the allergy was confirmed and we now have to avoid eggs and Zoe was prescribed an Epi Pen Jr. We went and saw an allergist as a followup, and he did some skin tests that showed eggs were the only thing she's allergic to. He said her allergy was very mild in the grand scheme of things, and we're going to do another skin test when she's 2 to see if she's grown out of it. Eggs in baked goods are generally okay, as the offending protein is denatured at high temperatures. She's more allergic to the egg whites (1.56, which is in the "moderate" range) than the egg yolks (0.55, which is in the "low" range). Egg allergies are apparently something that one does eventually out grow, so that's a good thing. We'd also been keeping her off gluten and dairy because of a strong correlation between them and rather disgusting bowel motions. We've successfully reintroduced gluten with no adverse effects, and we've just started her back on cow's milk. We of course had our epic trip to the Grand Canyon, which Zoe handled absolutely marvelously. That was closely followed by a weekend down in LA, which was a bit more of a disaster as Sarah and I both came down with a 24 hour stomach bug that Zoe had had during the week. There's nothing quite like languishing in a non-childproofed hotel room, feeling like death, with a full of beans toddler busy running around jamming her fingers in drawers. Fortunately we were able to use a babysitter for a few hours respite, but it did rather completely mess up our plans for the weekend. Zoe did handle the drive to and from LA very well though. Zoe's been doing Saturday morning swim classes at the YMCA, which I've taken her to. I don't think she's enjoyed them as much as the indoor ones she did at the Sutton Swim School, where she went with Sarah previously, and I definitely don't feel like the classes are as structured. Unfortunately we just finished an 8 week session, and the next one filled up before we got a chance to book, so we're hoping to get back into something at the Sutton Swim School once an opening comes up. I might do something ad hoc with her on Saturdays at the Rinconada Pool in Palo Alto in the meantime. Zoe has also started going to regular daycare on Thursdays, so Sarah gets a day to focus on her University study. It's a home daycare that isn't too far away, and our friend's daughter goes there too, so there's someone she knows. The first few times Zoe got pretty upset when she was dropped off, but she barely makes a whimper now when Sarah drops her off. She's also usually down to one nap a day now. She has four teeth, top and bottom, with one molar definitely through and another three starting to make an appearance. I just bought an iBert safe-T-seat for my bike, because I've been itching to get out on the bike with Zoe on the weekends when I'm looking after her so Sarah can get some more study done. The weather is currently nice, and I've just learned about a bike trail that cuts across the bay in front of Moffett Field, so I'm keen to check that out. Apparently there's geese out there. I did my first decent ride with her last Sunday to the Mountain View Farmer's Markets and back. Zoe seemed to be okay with it. The helmet we got her drew a lot of comments from people at the markets. She's very much in the "point and grunt" stage at the moment. What's really cool is she'll nod or shake her head in response to yes/no questions. It's so cool when you can ask her a question and she solemnly nods her head in response. Often in the morning Sarah takes her to the park around the back of our home, and so now Zoe's gotten into that habit, so she'll scratch around at the front door and they head over there. The other day, Sarah opened the front door and she took off in the direction of the park, so Sarah just followed her to see where she went. Sure enough, she made her way over to the park. Tonight, Zoe fetched me my shoes and then dragged me over by the hand to the front door to demand that I take her out for a walk. She's also really getting into books now. She actively seeks out books, and has a bit of a flick through them, or brings one over and climbs into our laps, or backs up and sits in our laps if we're sitting on the floor, to have us read them to her. She's now strong enough to hold her bottle herself at bedtime, so I sit with her in my lap in the glider and read a story to her while she has her bedtime bottle. It's indescribably wonderful. She seems to understand quite a large array of words. We can go "Where's your <insert object here> and she'll look around the living room and generally find the item we're talking about. She also seems to understand questions about breakfast, lunch, going to the park, going to the bathroom, having a bath, etc as she seems to respond appropriately. In the speech department, I haven't really counted the words, but she's not saying a huge number of words. This morning, apparently she yelled "MAMA! MAMA! PARK!". "Park" being a new word for her. We bought an annual membership to the Children's Discovery Museum in San Jose, and we've probably taken her there half a dozen times or so since getting the membership. She particularly loves the water area. We finally got around to getting our back yard landscaped and made generally fit for a small child to run around in. Sarah's planted a butterfly garden in one corner, so hopefully once that's established we'll have some fun things for Zoe to watch from the window. I'm looking forward to planting some bulbs in the autumn. I expect the next 3 months will be more of the same. We've got some more overseas travel coming up, but a lot of relatives will be around, so hopefully we'll get a bit of downtime to ourselves.

Now that the debconf videos are available (big THANKS to the video team), I was able to watch the talk about packages in git at Debconf11 and wanted to share some insights:

• Feature branches
  I've looked a long time into topgit, and like many other gave up. I think there is one big problem: It is not compatible with git.
  Git tracks the history of commits. Everything that is in the history is assumed to be already applied. So if you merge something with a change applied and reverted with something with a change applied, the logical conclusion for git is to not apply it (or rather also apply the revert).
  This means that if you have multiple feature branches on top of each other, you cannot easily remove a branch from under the other. You can also not easily switch the order of patches. There are many ideas floating around how to work around this, but I personally would not wait until some manages to finish topgit or some replacement of it with that functionality.
• Reverting to an older state or branching off
  The other problem with many feature branches is that while due to aggressive merging you can fast-forward to a newer state, reverting to an older state of all the branches is relatively complicated.
  With the "rebased patch branch merged into you debian branch history" approach of git-dpm on the other hand, branching of a state is extremely simple:
  git checkout -b squeeze reprepro-debian-4.2.0-2
  for example is the only thing that was needed to branch of a branch for stable updates. Reverting to a previous state with git reset --hard is not much more complicated (at most you need to remove the upstream and patched branches, if they are needed they will be recreated anyway).

In the past week, I ran a lot of benchmarks using a Spirent Avalanche which is an appliance providing performance testing of network related products (a load balancer, a router, a web server, ). The reporting module does not provide a lot of flexibility and the plots are not the most beautiful ones. Fortunately, the results are also exported as CSV. matplotlib is a python plotting library which produces great figures without making things hard when they should be easy. It is a good replacement for GNU Plot and you don t need a lot of Python knowledge to use it. The documentation includes a fine user s guide that should get you started in less than 20 minutes.

Quick introduction You can use matplotlib from IPython to experiment:

$ ipython -pylab
Python 2.6.7 (r267:88850, Jul 10 2011, 08:11:54) 
Type "copyright", "credits" or "license" for more information.
  Welcome to pylab, a matplotlib-based Python environment.
  For more information, type 'help(pylab)'.
> In [1]: plot([1,1,4,5,10,11])

When you are ready, you can build a simple Python script:

#!/usr/bin/env python
from matplotlib.pylab import *
plot([1,1,4,5,10,11])
savefig("my-plot.pdf")

Grabbing results from CSV Data are contained into a file named realtime.csv. It contains the raw data as well as some general information about the benchmark (test name, description, paramaters ). We need to skip those. Fortunately, csv2rec() allows to load data from a CSV file into a record array and skip the first rows if necessary.

from matplotlib.pylab import *
import sys, os
import gzip
skip = 0
for line in gzip.open(sys.argv[1]):
    if line.startswith("Seconds Elapsed,"):
        break
    skip = skip + 1
ava = csv2rec(gzip.open(sys.argv[1]),skiprows=skip)

Now, the Elapsed seconds column can be accessed with ava['elapsed_seconds'].

General structure I need 4 plots:

• successful, unsuccesful and attempted transactions per second,
• minimum, average and maximum response time per page,
• Avalanche CPU usage,
• incoming and outgoing bandwidth.

The most important plot is the first one. The second one is less important and the last ones are here only to check we did not hit some bottleneck during the benchmark. We want to produce a page like this: matplotlib allows to plot subfigures. We create 4 subfigures sharing the same X axis (which the number of seconds elapsed since the beginning of the benchmark). We save the result to PDF.

# Create the figure (A4 format)
figure(num=None, figsize=(8.27, 11.69), dpi=100)
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
# [...]
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
# [...]
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
# [...]
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
# [...]
# Save to PDF
savefig("%s.pdf" % TITLE)

Bandwidth plot Let s start with the easiest plot: bandwidth usage.

# Plot 4: Bandwidth
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
plot(ava['seconds_elapsed'], ava['incoming_traffic_kbps']/1000.,
     'b-', label='Incoming traffic')
plot(ava['seconds_elapsed'], -ava['outgoing_traffic_kbps']/1000.,
     'r-', label='Outgoing traffic')
grid(True, which="both", linestyle="dotted")
ylabel("Mbps", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Here, in the suplot positioned in (3,1), we plot the number of seconds elapsed versus the incoming traffic with a blue line (b-). We also plot the seconds elapsed versus the outgoing traffic with a red line (r-). Here is the result: Most functions of matplotlib are exposed as a method of the object they refer to and as a global function. In the latest case, the function is applied to the latest created figure or plotting area. For example, plot() is called a function and therefore refer to the plotting area ax4. We could have written ax4.plot() instead.

CPU plot The average CPU utilization data available in the CSV file needs to be normalized. We assume the Avalanche to be mostly idle on start. The plotting part is pretty similar to our previous case.

# CPU
max = np.max(ava['average_cpu_utilization'])
order = 10**np.floor(np.log10(max))
max = np.ceil(max/order)*order
cpu = (max - ava['average_cpu_utilization'])*100/max
# Plot 3: CPU
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
plot(ava['seconds_elapsed'],
     cpu,
     'r-', label="Avalanche CPU")
grid(True, which="both", linestyle="dotted")
ylabel("Avalanche CPU%", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Response time We would like to plot response time. We have three corresponding metrics: minimum response time, average response time, maximum response time. Because those metrics start from 0 ms up to several seconds, a logarithmic scale is used:

# Plot 2: response time
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
plot(ava['seconds_elapsed'], ava['minimum_response_time_per_page_msec'],
     'b-', label="Minimum response time")
plot(ava['seconds_elapsed'], ava['maximum_response_time_per_page_msec'],
     'r-', label="Maximum response time")
plot(ava['seconds_elapsed'], ava['average_response_time_per_page_msec'],
     'g-', linewidth=2, label="Average response time")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=8))
grid(True, which="major", linestyle="dotted")
yscale("log")
ylabel("Response time (msec)", fontsize=9)
xticks(fontsize=9)
yticks(fontsize=9)

This is also the first graphic with a legend.

Transactions per second The most important plot is the number of transactions per second.

# Plot 1: TPS
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
plot(ava['seconds_elapsed'], ava['desired_load_transactionssec'],
     '-', color='0.7', label="Desired Load")
plot(ava['seconds_elapsed'], ava['successful_transactionssecond'],
     'g:', label="Successful")
plot(ava['seconds_elapsed'], smooth(ava['successful_transactionssecond']),
     'g-', linewidth=2)
plot(ava['seconds_elapsed'], ava['attempted_transactionssecond'],
     'b-', label="Attempted")
plot(ava['seconds_elapsed'], ava['aborted_transactionssecond'],
     'k-', label="Aborted")
plot(ava['seconds_elapsed'][:-1], ava['unsuccessful_transactionssecond'][:-1],
     'r-', label="Unsuccessful")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=10))
grid(True, which="both", linestyle="dotted")
ylabel("Transactions/s")

The number of successful transactions is plotted twice: when the benchmarked equipment becomes overloaded, we get a lot of noise in this metric and it can be difficult to read. Therefore, we plot a smoothed version with the help of NumPy:

import numpy as np
def smooth(x, win=4):
    s = np.r_[x[win-1:0:-1],x,x[-1:-win:-1]]
    w = np.ones(win, 'd')
    y = np.convolve(w/w.sum(),s,mode='valid')
    return y[(win-1)/2:-(win-1)/2]

np.r_() is just here to extend our data by the size of the window. np.ones() build a weight vector of the size of the window. If the window is 4, we get [0.25, 0.25, 0.25, 0.25]. We use this vector to apply a convolution to the original data. Here is the result: The original data is a green dotted line while the smoothed one is a green thick line. What about the three annotations? matplotlib allows to put annotations on a figure. Here is how this is done:

# Noticeable points
count = 0
def highlight(index, reason):
    global count
    if index and index > 0:
        x,y = (ava['seconds_elapsed'][index],
               smooth(ava['successful_transactionssecond'])[index])
        plot([x], [y], 'ko')
        annotate('%d TPS\n(%s)' % (y,reason), xy=(x,y),
                 xytext=(20, -(count+4.7)*22), textcoords='axes points',
                 arrowprops=dict(arrowstyle="-",
                 connectionstyle="angle,angleA=0,angleB=80,rad=10"),
                 horizontalalignment='left',
                 verticalalignment='bottom',
                 fontsize=8)
        count = count + 1
highlight(np.argmax(smooth(ava['successful_transactionssecond'])), "Max TPS")
highlight(np.argmax(cpu > 99), "CPU>99%")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 500), ">500ms")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 100), ">100ms")

np.argmax() returns the index of the first maximum value. The trick here is that when I write ava['average_response_time_per_page_msec'] > 100, I get an array with 1 when the value is more than 100 and 0 otherwise. Therefore, np.argmax() will return the first index where the value is superior to 100 ms. The highlight() function will add a point (plot([x], [y], 'ko')) on the smoothed successful transactions par second plot and add an annotation with some fancy arrow. Look at this benchmark of nginx as SSL termination for a complete output of this script.

NEW YORK, Jul 29, 2011 The crowd of thousands gathered here to attend PreventaConf 2011 were abuzz over the unveiling of the culmination of 3 years of aimless arsing about^ww^w concentrated development which has led to the porting of dpkg to the MINIX 3 operating system. Women fainted, men wept openly, medical personnel were kept busy dealing with the many instances of heart seizure, multisensory hallucination and spontaneous combustion which accompanied this momentous event. Much remains to be done.

• start-stop-daemon needs to be ported. It is not built for now.
• A basic set of essential packages needs to be created. Currently all that stuff lives in /usr/local or /usr/pkg. This has already caused problems when e.g. dpkg-architecture assumed perl was in /usr/bin.
• fakeroot needs to be ported so you don't have to build .debs as root which is currently the case.
• Other support programs such as update-rc.d and ldconfig etc. are needed. Currently .debs have to be installed with --force-all to get around their lack. (ldconfig atleast could just be a noop as Minix doesn't actually support shared libraries yet.)
• There are still some API deficiencies. lchown(2) doesn't exist, getpwent(3) doesn't seem to work reliably. etc. Could this cause problems? We will have to investigate and provide workarounds if necessary. No doubt other problems will crop up as work progresses.

Still this is a major milestone of which the Debian Minix developers are justifiably proud.