We just released LLVM & Clang 3.4 (already available in Debian Jessie and Ubuntu Trusty). The clang new release introduces some more warning and error detections. The point of the blog post is not to detail the changes (LLVM / Clang) but more to take a step back on the LLVM and Clang toolchains, usages and dissemination. For the last couple years, I have been presenting our work on the Debian side to make Clang part of the Debian infrastructure. I add the chance to speak at many events (Debconf 12 & 13, FOSDEM 2013, Linux Plumbers 2013, Distro recipes, etc). I even had Linus Torvalds attending to one of these talks (and proposing new approach). I had many feedback from the Debian/Ubuntu communities but also from other communities.
More and more projects are using Clang as part of their workflows, in term of development, production and QA. For example, some of them are running continuous integration using clang instead of gcc, others are running scan-build (Wireshark or LLVM itself) or Address Sanitizer (asan) (Firefox for example), etc.
More and more actors are moving from gcc to clang for their developments (Sony with their PS4, Chrome under Mac OS X, Apple, the Linux Kernel with the LLVMLinux project, FreeBSD, etc).
I stopped counting the number of times when people answered me "Yeh, I am now using Clang" when I said I am involved upstream. In parallel, more and more projects are using LLVM as a backend or computing engine and, in the meantime, getting more and more attraction (not because they are based on LLVM but just because they are great projects). For example, Mesa (llvmpipe), Julia, Rust, Emscripten or Native Client (NaCL) are excellent examples of this. The two last are excellent proofs that LLVM based technologies have an impact which goes further than geek-tools and will be used on a daily basis by millions of users. I also saw an interesting and growing number of new projects using libclang and libTooling. My feeling is that the lack of plugin capabilities in gcc for a long time and, now, its current complexity limited the innovation in term of compilation capabilities, analysis and fun tools. The features and the quality of this library are enabling new developments and I am sure we will see more and more excellent tools based on this library during 2014.
For example, the LLVM toolchain ships Clang modernize and clang format and some cool projects like include-what-you-use, DXR, OCLint or creduce are emerging. Thanks to this great competition, gcc is also improving (ASAN, JIT, better warnings, etc), remaining still relevant as before. No doubt that 2014 will be a great year for compilers.

Coverity is a nice tool to find problems in C, C++ and Java code using static source code analysis. It can detect a lot of different problems, and is very useful to find memory and locking bugs in the error handling part of the source. The company behind it provide check of free software projects as a community service, and many hundred free software projects are already checked. A few days ago I decided to have a closer look at the Coverity system, and discovered that the gnash and ipmitool projects I am involved with was already registered. But these are fairly big, and I would also like to have a small and easy project to check, and decided to request checking of the chrpath project. It was added to the checker and discovered seven potential defects. Six of these were real, mostly resource "leak" when the program detected an error. Nothing serious, as the resources would be released a fraction of a second later when the program exited because of the error, but it is nice to do it right in case the source of the program some time in the future end up in a library. Having fixed all defects and added a mailing list for the chrpath developers, I decided it was time to publish a new release. These are the release notes: New in 0.16 released 2014-01-14:

• Fixed all minor bugs discovered by Coverity.
• Updated config.sub and config.guess from the GNU project.
• Mention new project mailing list in the documentation.

You can download the new version 0.16 from alioth. Please let us know via the Alioth project if something is wrong with the new release. The test suite did not discover any old errors, so if you find a new one, please also include a test suite check.

Liberation vs expectations I feel like I found a key struggle: liberation vs expectations. Many ideas liberate me, like poliamory, atheism, anarchism, nonviolent communication, sexual liberation, discordianism, free software, nonjudgemental environments, E-prime, sillyness, love, pataphysics, dada, constructivism, rolling naked in the mud, fantasy, bdsm exploration, pansexuality, this very idea that i'm trying to describe, crying, laughing, mindfulness, what have you. Sometimes however they become an end rather than a mean, they give me expectations and judgements, and take some other freedom away. I feel like I found where to draw a line: It's the point inbetween freeing myself of my limits and making myself some new ones. It's one of those epiphany moments. Weeeeeee!

Hey Michael Stapelberg, thanks for creating i3 and making me believe again that people have a clue and things can be done just right (specially when it comes to something easily done crappy like window managers). Thanks for just getting it all right, and not only from the software point of view, but also for the community, the style around it, etc. Kudos.

Abstract I recently moved our primary nameserver from orff.debian.org, which is an aging blade in Greece, to a VM on one of our ganeti clusters. In the process, I rediscovered a lot about our DNS infrastructure. In this post, I will describe the many sources of information and how they all come together.

Introduction The Domain Name System is the hierarchical database and query protocol that is in use on the Internet today to map hostnames to IP addresses, to map the reverse thereof, to lookup relevant servers for certain services such as mail, and a gazillion other things. Management and authority in the DNS is split into different zones, subtrees of the global tree of domain names. Debian currently has a bit over a score of zones. The two most prominents clearly are debian.org and debian.net. The rest is made up of debian domains in various other top level domains and reverse zones, which are utilized in IP address to hostname mappings.

Types and sources of information The data we put into DNS comes from a wide range of different systems:

• Classical zonefiles maintained in git. This represents the core of our domain data. It maps services like blends.debian.org to static.debian.org or specifies the servers responsible for accepting mail to @debian.org addresses. It also is where all the ftp.CC.debian.org entries are kept and maintained together with the mirror team.
• Information about debian.org hosts, such as master, is maintained in Debian's userdir LDAP, queryable using LDAP¹.
  • This includes first and foremost the host's IP addresses (v4 and v6).
  • Additionally, we store the server responsible for receiving a host's mail in LDAP (mXRecord LDAP attribute, DNS MX record type).
  • LDAP also has some specs on computers, which we put into each host's HINFO record, mainly because we can and we are old-school.
  • Last but not least, LDAP also has each host's public ssh key, which we extract into SSHFP records for DNS.
• LDAP also has per-user information. Users of debian infrastructure can attach limited DNS elements as dnsZoneEntry attributes to their user².
• The auto-dns system (more on that below).
• Our puppet also is a source of DNS information. Currently it generates only the TLSA records that enable clients to securely authenticate certificates used for mail and HTTPS, similar to how SSHFP works for authenticating ssh host keys.

Debian's auto-dns and geo setup We try to provide the best service we can. As such, our goal is that, for instance, user access to www or bugs should always work. These services are, thus, provided by more than one machine on the Internet. However, HTTP did not specify a requirement for clients to re-try a different server if one of those in a set is unavailable. This means for us that when a host goes down, it needs to be removed from the corresponding DNS entry. Ideally, the world wouldn't have to wait for one of us to notice and react before they can have their service in a working manner. Our solution for this is our auto-dns setup. We maintain a list of hosts that are providing a service. We monitor them closely. Whenever a server goes away or comes back we automatically rebuild the zone that contains the element. This setup also lets us reboot servers cleanly since one of the things we monitor is "is there a shutdown running", we can, simply by issuing a shutdown -r 30 kernel-update, de-rotate the machine in question from DNS. Once the host is back it'll automatically get re-added to the round-robin zone entry. The auto-dns system produces two kinds of output:

• In service-mode it generates a file with just the address records for a specific service. This snippet is then included in its zone using a standard bind $INCLUDE directive. Services that work like this include bugs and static (service definition for static).
• In zone-mode, auto-dns produces zonefiles. For each service it produces a set of zonefiles, one for each out of a set of different geographic regions. These individual zonefiles are then transferred using rsync to our GEO-IP enabled nameservers. This enables us to give users a list of security mirrors closer to them and thus hopefully faster for them.

Tying it all together Figure 1: The Debian DNS Rube Goldberg Machine. Once all the individual pieces of source information have been collected, the dns-update and write_zonefile scripts from our dns-helpers repository take over the job of building complete zonefiles and a bind configuration snippet. Bind then loads the zones and notifies its secondaries. For geozones, the zonefiles are already produced by auto-dns' build-zones and those are pulled from the geo nameservers via rsync over ssh, after an ssh trigger.

and also DNSSEC All of our zones are signed using DNSSEC. We have a script in dns-helpers that produces, for all zones, a set of rolling signing keys. For the normal zones, bind 9.9 takes care of signing them in-process before serving the zones to its secondaries. For our geo-zones we sign them in the classical dnssec-signzone way before shipping them. The secure delegation status (DS set in parent matches DNSKEY in child) is monitored by a set of nagios tests, from both [dsa-nagios] and dns-helpers. Of these, manage-dnssec-keys has a dual job: not only will it warn us if an expiring key is still in the DSset, it can also prevent it from getting expired by issuing timly updates of the keys metadata.

Relevant Git repositories

• domains
• auto-dns
• mini-nag
• dns-helpers
• puppet
• nagioschecks

-- Peter Palfrader

---

1. ldapsearch -h db.debian.org -x -ZZ -b dc=debian,dc=org -LLL 'host=master'
2. ldapsearch -h db.debian.org -x -ZZ -b dc=debian,dc=org -LLL 'dnsZoneEntry=*' dnsZoneEntry

I have written a number of scripts to create Debian foreign architecture (mostly armel and armhf) rootfs images for SD cards or NAND flashing. I started with putting Debian on my Openmoko gta01 and gta02 and continued with devices like the qi nanonote, a marvel kirkwood based device, the Always Innovating Touchbook (close to the Beagleboard), the Notion Ink Adam and most recently the Golden Delicious gta04. Once it has been manufactured, I will surely also get my hands dirty with the Neo900 whose creators are currently looking for potential donors/customers to increase the size of the first batch and get the price per unit further down. Creating a Debian rootfs disk image for all these devices basically follows the same steps:

1. create an disk image file, partition it, format the partitions and mount the / partition into a directory
2. use debootstrap or multistrap to extract a selection of armel or armhf packages into the directory
3. copy over /usr/bin/qemu-arm-static for qemu user mode emulation
4. chroot into the directory to execute package maintainer scripts with dpkg --configure -a
5. copy the disk image onto the sd card

It was not long until I started wondering why I had to run all of the above steps with superuser privileges even though everything except the final step (which I will not cover here) was in principle nothing else than writing some magic bytes to files I had write access to (the disk image file) in some more or less fancy ways. So I tried using fakeroot+fakechroot and after some initial troubles I managed to build a foreign architecture rootfs without needing root priviliges for steps two, three and four. I wrote about my solution which still included some workarounds in another article here. These workarounds were soon not needed anymore as upstream fixed the outstanding issues. As a result I wrote the polystrap tool which combines multistrap, fakeroot, fakechroot and qemu user mode emulation. Recently I managed to integrate proot support in a separate branch of polystrap. Last year I got the LEGO ev3 robot for christmas and since it runs Linux I also wanted to put Debian on it by following the instructions given by the ev3dev project. Even though ev3dev calls itself a "distribution" it only deviates from pure Debian by its kernel, some configuration options and its initial package selection. Otherwise it's vanilla Debian. The project also supplies some multistrap based scripts which create the rootfs and then partition and populate an SD card. All of this is of course done as the superuser. While the creation of the file/directory structure of the foreign Debian armel rootfs can by now easily be done without superuser priviliges by running multistrap under fakeroot/fakechroot/proot, creating the SD card image still seems to be a bit more tricky. While it is no problem to write a partition table to a regular file, it turned out to be tricky to mount these partition because tools like kpartx and losetup require superuser permissions. Tools like mkfs.ext3 and fuse-ext2 which otherwise would be able to work on a regular file without superuser privileges do not seem to allow to specify the required offsets that the partitions have within the disk image. With fuseloop there exists a tool which allows to "loop-mount" parts of a file in userspace to a new file and thus allows tools like mkfs.ext3 and fuse-ext2 to work as they normally do. But fuseloop is not packaged for Debian yet and thus also not in the current Debian stable. An obvious workaround would be to create and fill each partition in a separate file and concatenate them together. But why do I have to write my data twice just because I do not want to become the superuser? Even worse: because parted refuses to write a partition table to a file which is too small to hold the specified partitions, one spends twice the disk space of the final image: the image with the partition table plus the image with the main partition's content. So lets summarize: a bootable foreign architecture SD card disk image is nothing else than a regular file representing the contents of the SD card as a block device. This disk image is created in my home directory and given enough free disk space there is nothing stopping me from writing any possible permutation of bits to that file. Obviously I'm interested in a permutation representing a valid partition table and file systems with sensible content. Why do I need superuser privileges to generate such a sensible permutation of bits? Gladly it seems that the (at least in my opinion) hardest part of faking chroot and executing foreign architecture package maintainer scripts is already possible without superuser privileges by using fakeroot and fakechroot or proot together with qemu user mode emulation. But then there is still the blocker of creating the disk image itself through some user mode loop mounting of a filesystem occupying a virtual "partition" in the disk image. Why has all this only become available so very recently and still requires a number of workarounds to fully work in userspace? There exists a surprising amount of scripts which wrap debootstrap/multistrap. Most of them require superuser privileges. Does everybody just accept that they have to put a sudo in front of every invocation and hope for the best? While this might be okay for well tested code like debootstrap and multistrap the countless wrapper scripts might accidentally (be it a bug in the code or a typo in the given command line arguments) write to your primary hard disk instead of your SD card. Such behavior can easily be mitigated by not executing any such script with superuser privileges in the first place. Operations like loop mounting affect the whole system. Why do I have to touch anything outside of my home directory (/dev/loop in this case) to populate a file in it with some meaningful bits? Virtualization is no option because every virtualization solution again requires root privileges. One might argue that a number of solutions just require some initial setup by root to then later be used by a regular user (for example /etc/fstab configuration or the schroot approach). But then again: why do I have to write anything outside of my home directory (even if it is only once) to be able to write something meaningful to a file in it? The latter approach also does not work if one cannot become root in the first place or is limited by a virtualized environment. Imagine you are trying to build a Debian rootfs on a machine where you just have a regular user account. Or a situation I was recently in: I had a virtual server which denied me operations like loop mounting. Given all these downsides, why is it still so common to just assume that one is able and willing to use sudo and be done with it in most cases? I really wonder why technologies like fakeroot and fakechroot have only been developed this late. Has this problem not been around since the earliest days of Linux/Unix? Am I missing something and rambling around for nothing? Is this idea a lost cause or something that is worth spending time and energy on to extend and fix the required tools?

Packt Publishing sent me Krzysztof Niki ski's Instant Haml to do a short blog review... Thing is, and those who read me know, I cannot write short reviews ;-) So lets see how this goes. Packt's Instant series First, as for the format this book follows and the series it fits in: They are very short books. I was the tech reviewer for their Instant Debian Build a Web Server , and when I opened it for the first time, I thought I had received only the first chapter. This series' motto is Short Fast Focused so, yes, the books are ~50 pages long, with ample margins and many big screenshots. I told the editors I was quite unsure about the value this kind of publications add, as they are very similar to online tutorials, but given they have so many books in this series (printed and electronic), there must be a market for them. They surely know their business better than me! There is one other detail regarding the publisher I'll stick in this section, although it is not specific to their Instant series: I do most of my reading in my Kindle. Packt seems to have chosen to specify a very small font for their texts. Besides the two books mentioned, I also have their Object Oriented JavaScript book. For all of the other books I read, I use the smallest font setting. Packt books in mobi format (what they present as Amazon format in their web site) are basically illegible without bumping up the font size through the roof. As I understand it, publishers should use industry-wide default sizes, and let the reader adjust their reading experience. Yes, books of technical nature (such as Packt's) often benefit from having more displayed information than what an ebook reader normally shows... But that's for me to decide! And no, this is not a Kindle-specific glitch: I also get a similar experience when opening it from Calibre. And as for the book... I am a long-time Haml convert/fan; I packaged its Ruby implementation for Debian (and should update it soon :-/ as new versions have appeared), the corresponding Emacs Lisp mode, and I use it for basically all of my static HTML sites and Ruby-based Web systems. Haml is a great templating language, meant to make dynamic HTML generation leaner and easier. It is a clean and elegant technology, and can clearly sell itself. And the project webpage has very good documentation, ranging from a short tutorial and a full API documentation for people to contribute to the project. The first part of this book, of course, covers the "why is Haml so great?" part, presenting the basic equivalences with HTML. In my opinion, the book could be structured a bit differently and would generate a stronger following. Why? Because, even given that Haml is greatly benefited from its Ruby integration and has many hooks to be tightly integrated in a Rails workflow, it can also be used by people who don't want to do Ruby. I have some static websites that are just a bunch of Haml files, compiled by a very simple Makefile and calling the haml command from the shell. I think there would be some value in presenting this invocation before learning how to call it from the Gems environment (i.e. bundle install). Using Haml as a standalone tool is only presented almost as an appendix ("Using HAML outside of Rails" section). Even more so if the author presents four different static-site solutions (probably more non-programmer-friendly than my trusty Makefile). Oh, and this little section uses the (old? obsolete?) uppercase HAML notation for the project's name instead of the now-standard Haml. Dirty screenshots About the screenshots presented in this first section (and I refer mainly to the scaffolding examples), they are meant to show how easy things are, but have an important stylistic mistake: Terminals with transparency might look cool (I hate using them, but then again I'm a very boring person when it comes to my computer habits ;-) ), but transparencies have no place in a book (be it printed or e-book), as the images appear very uncrisp. In the Kindle, they just seemed dirty Now that I'm browsing the PDF from my desktop, I see that the screenshot in "Step 1" has a photo of Earth from space behind the terminal, and even worse, in "Step 2" it is shown over a Web page detailing the installation! These elements are just distracting, and the text would be much cleaerer if it were presented typeset just as a regular code fragment. Opening many of those screenshots in the Kindle was plain useless. Editor integration It is perfectly clear from this text that the author strongly prefers Sublime Text, sometimes uses TextMate, and has just heard reports that Haml has some support in Vim, Rubymine, Emacs and Coda. Of course, devoting a full page (as it is for the first case) for each editor would be out of proportion for this size of book... I just felt the imbalance too big, specially given that most editors will sport approximately the same feature set. Text redundancy If the Instant series is about making short books, I would expect them to be loaded with easy to follow content (as they are), but free of repetition. I felt, however, a large chunk of step 1 of the "Quick start" chapter to be repeated with the the first section of the "Top 6 features" chapter. Yes, the second one adds some depth, but very little. Still, in the later parts of this "Top 6 features" chapter, a more thorough explanation could be used for the Filters, Multiline attributes/code/strings and helpers and extensions sections. It seems the author was on a hurry trying to get it finished, as it is barely explained Explaining a bit more deeply would surely benefit readers. Wrapping up Reading my review, I'm mostly talking about negative issues. IMO, the book is not excellent, but it is clearly not bad. Specially given it fits very well the format for which it was designed. There are many items that could be fixed, and I hope some of them can be fixed (i.e. the screenshots) before the book goes to print If I understand Packt's page correctly, it is currently only available as ebook, contrary to most of the Instant series' titles.

Ok, so I'm back in Mexico! This year, the best fare I found for travelling to spend the Winter^WSummer season with Regina's family had an oddity: I usually have a layover at either Santiago de Chile or Lima (Per ) of between 45 minutes and 2 hours, clearly less than enough to do anything. But this time, I had a massive 10 hours layover in Santiago. And spending 10 hours in an airport is far from fun. Specially when you have a good group of friends in town! I visited Chile in 2004 for Encuentro Linux (still before the time I had a digital camera: Those photos are all taken by Martin Michlmayr), and I have stayed in touch with a group of systems administrators since then. So, I mailed the list, and we managed to get eight people to have lunch together. In the order we appear in the photo:

• Victor Hugo dos Santos
• Mauricio Troncoso
• lvaro Herrera
• Ricardo Lemus
• Marcelo Riquelme
• Carlos Sep lveda
• Gunnar Wolf
• Pablo Silva

Some of them, even living in the same city, had never met in person before So, of course, we had a table reserved at the restaurant to the name of Dennis Ritchie. And having had nice, fun, sometimes-technical talks... Well, a tiny bit of his spirit was there. Of course, we can only trust he was there, as no Ouija boards were used and no null pointers were dereferenced (just to make sure not to disturb him). Victor Hugo and lvaro took me for a short Santiago city trip before lunch, we had a very nice time. Thanks! :-)

Dr Suelette Dreyfus gave an interesting LCA keynote speech on Monday (it s online now for people who aren t attending LCA [1]). One of the interesting points she made was regarding the greater support for privacy protection in Germany, this is apparently due to so many German citizens having read their own Stasi files. The section of her talk about the technology that is being used against us today was very concerning. I wonder whether we should plan to move away from using any hardware or closed source software from the US, China, and probably most countries other than Germany. We really need to consider these issues at election time. I have previously blogged some rough ideas about having organisations such as Linux Australia poll parties to determine how well they represent the interests of citizens who use Linux [2]. I think that such things are even more important now. Steven Levy wrote an interesting summary of the situation for Wired [3]. At the end of her talk Suelette suggested that Aspies might be more likely to be whistle-blowers due to being unable to recognise the social signals about such things (IE managers say that they won t punish people for speaking out but most people recognise that to be lies). It s a plausible theory but I m worried that managers might decide to avoid hiring Aspies because of this. I wonder how many managers plan to have illegal activity as an option. But I guess that having criminals refuse to hire me wouldn t be such a bad thing.

• [1] http://mirror.linux.org.au/pub/linux.conf.au/2014/
• [2] http://etbe.coker.com.au/2013/12/23/political-advocacy-clubs/
• [3] http://tinyurl.com/lz4h4e6

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (147.56 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. The Debian Administrator s Handbook I spent a good chunk of December on the book. First finalizing the English version and getting it out (BTW, just for the launch, there s a 10% discount on the paperback that lasts only until January 9th!). Then working on updating the French translation. Eyrolles will publish a new edition of the French book based on this translation. Expect some further news about this during January! Debian France I contributed to many discussions within Debian France. Starting with a complaint that most events are organized in Paris, I proposed to map the location of Debian France members. We added new fields in the membership management page so that members can add their GPS coordinates and Fr d ric Decou made some experiments with Openstreetmap. Someone else (Kiriarat) volunteered to write the required glue code. A manual map is currently maintained on the website. In the discussions about the setup of the Debian France shop, I suggested to update our logo with a nicer looking one. We got a few suggestions and after further discussions with Alexandre Delano and Sylvestre Ledru, we organized a small contest to entice designers to submit a logo proposal to us (the winner earns a set of Debian goodies). We got 46 proposals (see my favorite on the right)! The board is currently pre-selecting the logos and setting up the final vote for our members. The winner shall be announced at the end of the upcoming mini-debconf in Paris. I also continued the work to finalize new bylaws and new internal rules. They shall be adopted during the next general assembly which will happen during the mini-debconf. Misc Debian Work WordPress maintenance. I mentored Pablo Vasquez to do his first small contribution to the WordPress packaging. I really appreciate this but he s not yet ready to assume maintenance of a big package like WordPress on his own. I got multiple other offers of help and pinged them all while filing #733726 to coordinate the work on the new upstream version. But I got no reply Handing over packages to new maintainers is hard Init system discussion. The technical committee has the hard task of picking the default init system that will replace the traditional System V init (see #727708). I followed this huge discussion closely and contributed a bit where I add something meaningful to say. Final decision is expected sometimes in January. FWIW, I share entirely Russ Allbery s point of view in those discussions. I have been running systemd on some of my computers for a few months already. Fixing lxc in stable. The lxc package in stable has a non-working debian template. I really dislike documenting that things are broken so instead of doing that in the Debian Administrator s Handbook, I opted to do something about it. I prepared a non maintainer upload for stable (see #680469 for the problem and #732358 for the stable update request). Misc stuff. I sponsored a tcpdf upload. I filed an enhancement request on Publican to have it keep processing instructions present in translations. I uploaded new versions of publican-librement and debian-handbook. I filed #732678 against git-buildpackage because it failed to properly call lintian when given the -A dpkg-buildpackage argument. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Following the release of Debian Wheezy, I have (finally) updated my set of Debian QEMU images for both Squeeze (6.0.8) and Wheezy (7.3). The following images are now available:

• amd64 (Squeeze and Wheezy)
• armel (Squeeze and Wheezy)
• armhf (Wheezy)
• i386 (Squeeze and Wheezy)
• mips (Squeeze and Wheezy)
• mipsel (Squeeze and Wheezy)
• powerpc (Squeeze and Wheezy)

Each of these directories contains a GPG signed README.txt file with the md5sums all files, detailed instructions how to run these images and especially the minimum QEMU version to use. The requirements to run the default desktop environment have increased a lot between Squeeze and Wheezy. An accelerated graphics card is now needed to be able to use Gnome (unless you use the fallback mode), which is not something provided by QEMU (actually there is now a QXL para-virtualized video card, but the driver is only in unstable). In addition GDM now needs more than 128MiB to start, while this is the default amount of memory provided for virtual machines. I have therefore decided to switch the default desktop environment on the Wheezy images to Xfce and the display manager to LightDM. Both Gnome and GDM are still installed on the images, and the original default can easily be restored using the following commands:

• update-alternatives --auto x-session-manager
• echo /usr/sbin/gdm3 > /etc/X11/default-display-manager

Beside this the new images only contain minor changes. The filesystems have been tweaked to not run fsck after a certain amount of days, and locales-all and openssh-server are now installed in all images. For MIPS and MIPSEL, 64-bit kernels are now also installed and provided, so that it is possible to choose between a 32-bit or a 64-bit kernel (see the README.txt for more details). There is no Debian Wheezy SPARC image available, as QEMU does not fully support SPARC64 yet (it is actually possible to run it, but then the VM crashes often), and Debian Wheezy now only supports 64-bit kernels. I will also invest time to build an S390X image, but so far I haven t been successful on that. The following images are still available at the same location, though they haven t been updated:

• sparc (Etch)
• SH4 (Sid from a few years ago)

The Ceilometer project supports various database backend that can be used as storage. Among them are MongoDB, SQLite MySQL, PostgreSQL, HBase, DB2 All Ceilometer's code is unit tested, but when dealing with external storage services, one cannot be sure that the code is really working. You could be inserting data with an incorrect SQL statement, or in the wrong table. Only having the real database storage running and being used can tell you that. Over the months, we developed integration testing on top of our unit testing to validate that our storage drivers are able to deal with real world databases. That is not really different from generic integration testing. Integration testing is about plugging all the pieces of your software all together and running. In what I call "database integration testing", the pieces will be both your software and the database system that you are going to rely on. The only difference here is that one of the module is not coming from the application itself but is an external project. The type of database that you use (RDBMS, NoSQL ) does not matter. Taking a step back, what I will describe here could also apply to a lot of other different software modules, even something that would not be a database sytem at all. Writing tests for integration Presumably, your Python application has unit tests. In order to test against a database back-end, you need to write a few specific classes of tests that will use the database subsystem for real. For example:
This code will try to fetch the database URL to use from an environment variable, and then will rely on SQLAlchemy to create a database connection.
You can then add as many tests as you want using the connection stored in self.engine. If no test database URL is, the tests will be skipped; however that decision is up to you. You may want to have these tests always run and fail if they can't be run. In the setUp() method, you may also need to do more work, like create a database and delete a database.
This will make sure that the database you need is clean and ready to be used to testing. Launching modules, a.k.a. databases The main problem we encountered when building integration testing with databases, is to find a way to start them. Most users are used to start them system-wide with some sort of init script, but when running sandboxed tests, that is not really a good option. Browsing the documentation of each storage allowed us to find a way to start them in foreground and control them "interactively" via a shell script. The following is a script that you can use to run Python tests using nose and is heavily inspired by the one we wrote for Ceilometer.
If you use tox to automatize your test run, you can use this scripts (I call it run-test.sh) in your tox.ini file.
Most databases are able to be run in some sort of standalone mode where you can connect to them using a either a Unix domain socket, or a fixed port. Here are the snippet used in Ceilometer to run with MongoDB and MySQL:

The mechanism is always the same. We create a fifo with mkfifo, and then run the database daemon with the output redirected to that fifo. We then read from it until we find a line stating the the database is ready to be used. At that point, we can continue and start running the tests. You have to read continuously from the fifo, otherwise the process writing to it will block. We redirect the output to /dev/null, but you could also redirect it to a different log file, or not at all. One step further: using parallelism and scenarios The described approach is quite simple, as it only support one database type. When using an abstraction layer, such as SQLAlchemy, it would be a good idea to run all these tests against different RDBMS, such as MySQL and PostgreSQL for example. The snippet above allows to run both RDBMS in parallel, but the classic approach of unit tests does not allow that. Using one scenario for each database backend would be a great idea. To that end, you can use the testscenarios library.

To speed up tests run, you could also run the test in parallel. It can be intesting as you'll be able to spread the workload among a lot of different CPUs. However, note that it can require a different database for each test or a locking mechanism to be in place. It's likely that your tests won't be able to work altogether at the same time on only one database. (Both usage of scenarios and parallelism in testing will be covered in The Hacker's Guide to Python, in case you wonder.)

In 1994/1995, I spent probably the most unique and memorable year of my life with the people of several Zionist youth movements in Israel, working in a kibutz, learning in Jerusalem... In 1995, after most of my group returned to Mexico, I stuck with the newly arrived South American group of Hashomer Hatzair (plus some affiliates). We were 18-19. And 18-19 years later, here we are: Fabi n (from Buenos Aires) and Fabiana (from Montevideo), living in Buenos Aires, with two kids. Regina (from Paran ) and me (from Mexico), living in Mexico, no kids yet. Two couples. Three countries. Four cities. And a lovely evening. And the hopes to stay in touch, to meet every year when we come to visit Argentina.

Recently I wrote about docker, after a brief diversion into using runit for service management, I then wrote about it some more. I'm currently setting up a new PXE-boot environment which uses docker for serving DHCP and TFTPD, which is my first "real" usage of any note. It is fun, although I now discover I'm not alone in using docker for this purpose. Otherwise life is good, and my blog-spam detection service recently broke through the 11 million-rejected-comment barrier. The Wordpress Plugin is seeing a fair amount of use, which is encouraging - but more reviews would be nice ;) I could write about work, I've not done that since changing job, but I'm waiting for something disruptive to happen first.. ObQuote: Dune. (film)

The primary addition in this release is the Autoconf macros and header wrapper that I developed for lbcd to add systemd support. These are inspired by the code in systemd's daemon(7) man page but adjusted for my normal software conventions. This is just preliminary and doesn't yet stub out all of the functions, only the ones I've needed so far. Also in this release, I fixed a bug with process_start_fakeroot, and hence with remctld_start_fakeroot, introduced in the last release when I rewrote the TAP process management add-on. You can get the latest release from the rra-c-util distribution page.

I read a lot this year - I'll write more about that and reflections on goodreads in another post - but most of the things I read weren't published in 2013. (I should also write a bit about my thoughts on e-readers). However, it seems I have enough to write about 2013's novels to make a round-up post worthwhile, so here we go. This year, crime author Robert Galbraith published his first novel The Cuckoo's Calling. I'd never have heard of it if Galbraith was not outed as an alias for Joanne "JK" Rowling. Clues that Rowling was working on a detective story exist as early as a Guardian preview article in 2012 for her last novel, The Casual Vacancy. Further hints, for me, that this was no first-time author were the taglines from Ian Rankin and Val McDermid on the cover, writers of a calibre I'd be surprised a new author could attract. However I don't know whether they were on the pre-unveiling cover or not. Rowling was upset be outed, having enjoyed the freedom to write without the baggage of expectation that she is subject to. I hope she's pleased: prior to her unmasking the novel was warmly received by the (admittedly relatively small) number of people who read it. And a very good novel it is too. It starts with a genre clich of a grizzled, meloncholy detective, Mr. Cormoran Strike, in an upstairs office with a neon light flickering through the window, but fleshes the story out both forwards - a client, a mysterious death - and backwards - how did Mr. Strike end up in that upstairs office - living out of it, no less? As is traditional for the genre there's a very clever twist. What I really enjoyed about Cormoran Strike was Galbraith/Rowling moving quickly from Chandler-esque everyman to a well fleshed-out, complex protagonist, intertwining the development of the character with the unfolding of the wider plot. I'm looking forward to the sequel, expected in 2014. A second surprise favourite this year was Lauren Beukes' time-tripping crime story The Shining Girls. A monsterous murder of women somehow finds a room in Chicago that lets him travel through time (or perhaps the room finds him). He uses this facility to stalk and murder a set of Shining Girls: women who, for one reason or another, literally 'shine' in his perception of them. One such woman survives his first attack and decides to try and find out who attacked her, and why. The crimes are described in a brutal fashion which - from a distance - resemble the sometimes glorified violence for which crime fiction is sometimes criticised, but the focus of the story is very much on the victims: they are fully fleshed out characters and each death is felt by the reader as a genuine tragedy. I discovered Beukes when her earlier novel Zoo City was included in a Humble eBook bundle. On reading The Shining Girls I felt that the novel deserved to be more widely known than I would expect it to be trapped in the ghetto of genre fiction, so I was pleased to discover that the very mainstream Richard and Judy Book Club discovered it. In established author news, Terry Pratchett, having adopted speech recognition for writing (to combat his debilitating Alzheimer's) has seemingly managed to accelerate his rate of production and squeezed out at least two this year: The Long War with Stephen Baxter is the sequel to 2012's The Long Earth which I very much enjoyed, but it really felt like "difficult second novel" to me. Hopefully there'll be a third. Raising Steam, the 40th Discworld novel, was an enjoyable romp around the concept of steam trains, featuring the relatively new Moist von Lipwig who has managed to become one of my favourite Discworld characters. I can't think of much more to say about the novel, really. It's a Discworld novel, probably not the best introduction to the series for a new reader, but will give a reader familiar with the franchise everything they expect, and possibly no more. Iain Banks sadly died this year, shortly after the publication of his last novel, The Quarry. It's sat on my hardback shelf for the time being. I couldn't bring myself to read it in 2013. I did read his last SF offering from the year prior, The Hydrogen Sonata. Sadly, yet coincidentally, both of these books examine the nature of living and dying, The Quarry in particular from the point of view of a terminal cancer sufferer. I have a small backlog of unread Banks fiction which I want to take my time over with. Finally, whilst not really a book, I thoroughly enjoyed the BBC's 2013 adaptation of Neil Gaiman's Neverwhere. Natalie Dormer wrote a piece on the making of the drama which should serve as a good introduction. At the time of writing, most of the programmes have disappeared from iPlayer, but I would be surprised if this wasn't released commercially at some point.

So, got a beep this morning from our work monitoring system. One of our customers domain names is hosted with livedns.co.uk (which, as far as I can tell, is part of the Fasthosts franchise)... It appears that Fasthosts have managed to entirely break their DNS:

brettp@laptop:~$ host www.fasthosts.com
;; connection timed out; no servers could be reached
brettp@laptop:~$ whois fasthosts.com   grep -i "Name Server"
   Name Server: NS1.FASTHOSTS.NET.UK
   Name Server: NS2.FASTHOSTS.NET.UK
Name Server: NS1.FASTHOSTS.NET.UK
Name Server: NS2.FASTHOSTS.NET.UK
brettp@laptop:~$ whois fasthosts.net.uk   grep -A 2 "Name servers:"
    Name servers:
        ns1.fasthosts.net.uk      213.171.192.252
        ns2.fasthosts.net.uk      213.171.193.248
brettp@laptop:~$  host -t ns fasthosts.net.uk 213.171.192.252
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns fasthosts.net.uk 213.171.193.248
;; connection timed out; no servers could be reached
brettp@laptop:~$

So, that's fasthosts core nameservers not responding, good start! They also provide livedns.co.uk, so lets have a look at that:

brettp@laptop:~$ whois livedns.co.uk   grep -A 3 "Name servers:"
    Name servers:
        ns1.livedns.co.uk         213.171.192.250
        ns2.livedns.co.uk         213.171.193.250
        ns3.livedns.co.uk         213.171.192.254
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.192.250
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.193.250
;; connection timed out; no servers could be reached
brettp@laptop:~$ host -t ns ns1.livedns.co.uk 213.171.192.254
;; connection timed out; no servers could be reached

So, erm, apparently that's all their DNS servers "Not entirely functioning correctly"! That's quite impressive!