After having dealt with the removal of obsolete conffiles, I ll now explain what you should do when a configuration file managed by dpkg must be renamed. The problem Let s suppose that version 1.2 of the software stopped providing /etc/foo.conf. Instead it provides /etc/bar.conf because the configuration file got renamed. If you do nothing special, the new conffile will be installed with the default configuration, and the old one will stay around. Any customization made by the administrator are lost in the process (in fact they are not lost, they are still in foo.conf but they are unused). Of course, you could do mv /etc/foo.conf /etc/bar.conf in the pre-installation script. But that s not satisfactory: it will generate a spurious conffile prompt that the end-user will not understand. The solution In the preinst script, you have to verify if the old conffile has been modified by the administrator. If yes, you want to keep the file around. Otherwise you know you will be able to ditch it once the upgrade is over, and you rename it to /etc/foo.conf.dpkg-remove to remember this fact. In the postinst script, you remove /etc/foo.conf.dpkg-remove. If the old conffile (/etc/foo.conf) still exists, it s because it was modified by the administrator. You make a backup of the new conffile in /etc/bar.conf.dpkg-dist and rename the old one into /etc/bar.conf. In the postrm, when called to abort an upgrade, you move /etc/foo.conf.dpkg-remove back to its original name. In practice, use dpkg-maintscript-helper dpkg-maintscript-helper can automate all those tasks. You just have to put the following snippet in the maintainer scripts (postinst, postrm, preinst):

if dpkg-maintscript-helper supports mv_conffile 2>/dev/null; then
    dpkg-maintscript-helper mv_conffile /etc/foo.conf /etc/bar.conf 1.1-3 -- "$@"
fi

In this example, I assumed that version 1.1-3 was the last version of the package that contained /etc/foo.conf (i.e. the last version released before 1.2-1 was packaged). You can avoid the preliminary test if you pre-depend on dpkg (>= 1.15.7.2) or if enough time has passed to assume that everybody has a newer version anyway. You can learn all the details in dpkg-maintscript-helper s manual page. Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email (just check Send me blog updates ). No comment Support my work

A little while back, as part of a discussion on debian-devel, I spent some time bad-mouthing pkg-config, and then later realized (with the help of an IRC conversation) that I was doing so unfairly. So as penance here is some documentation for how to add a pkg-config configuration file to a typical Automake project. This information is also in my build system coding style and will be maintained there (so if you see this some time after I wrote it, check that page for more up-to-date information). pkg-config is a tool to provide the flags required to compile or link against a shared library. It's a replacement for complex Autoconf probes and for all of the incompatible and occasionally buggy *-config scripts. To install a pkg-config configuration file, provide a template pkg-config configuration in a file named after the library but with a .pc.in extension. Here is an example:

    prefix=@prefix@
    exec_prefix=@exec_prefix@
    includedir=@includedir@
    libdir=@libdir@
    Name: <library>
    Description: <short description>
    URL: <upstream url>
    Version: @PACKAGE_VERSION@
    Cflags: -I$ includedir 
    Libs: -L$ libdir  -l<library>
    Libs.private: @KRB5_LDFLAGS@ @KRB5_LIBS@

Replace <library> with the official name of the library (which isn't necessarily the library name with lib removed). This is only informational, may contain spaces, and isn't used to locate the file. Its filename is used for that. Replace <short description> with a short description of the purpose of the library. Often the short description of the package from the start of the README file works well here. Replace <upstream url> with the URL of the package's distribution page. Normally, all Cflags needs to contain is what's indicated above, which lets the compiler find the header files if installed in a non-standard location. For Libs, note that pkg-config distinguishes between the shared libraries that must be linked against on a system with proper transitive dependency support and the libraries that must be linked for a static build or on a system with poor support for shared libraries. For a well-designed shared library, one should rarely need anything in Libs other than what's listed above. Libs.private, on the other hand, needs all the LDFLAGS and LIBS of everything the library itself links with external to this package. In this example, the shared library is linked with the Kerberos libraries and therefore needs the Kerberos LDFLAGS and LIBS; modify as needed for your example. If the dependencies of the shared library also use pkg-config, just list the other pkg-config file names in Requires or Requires.private instead of using Libs or Libs.private for them. Finally, add the Automake code to build and install the pkg-config configuration file. It should look like this:

    pkgconfigdir = $(libdir)/pkgconfig
    nodist_pkgconfig_DATA = lib/libfoo.pc
    lib/libfoo.pc: $(srcdir)/lib/libfoo.pc.in
            sed -e 's![@]prefix[@]!$(prefix)!g' \
                -e 's![@]exec_prefix[@]!$(exec_prefix)!g' \
                -e 's![@]includedir[@]!$(includedir)!g' \
                -e 's![@]libdir[@]!$(libdir)!g' \
                -e 's![@]PACKAGE_VERSION[@]!$(PACKAGE_VERSION)!g' \
                -e 's![@]KRB5_LDFLAGS[@]!$(KRB5_LDFLAGS)!g' \
                -e 's![@]KRB5_LIBS[@]!$(KRB5_LIBS)!g' \
                $(srcdir)/lib/libfoo.pc.in > $@

Replace libfoo with the actual name of the library and change its path in the source package as needed. Note the pattern above and use it for any Autoconf substitution variables that you need to replace in the pkg-config template. The above example adds KRB5_LDFLAGS and KRB5_LIBS to the variables; you may have other ones you need, particularly to construct Libs.private. The above installs the pkg-config configuration file under $(libdir)/pkgconfig. $(datadir)/pkgconfig is appropriate instead if one is certain that the flags required to link with this library will never vary by architecture, but that's not the case for most libraries due to Libs.private. Note that the Automake documentation recommends using comma as the sed delimiter for doing this sort of trick, but LDFLAGS frequently contains a comma since it's the separator for parts of a linker-specific option. I use an exclamation point instead.

Today, I prepared a patch to dpkg to add support using Lzip compression. A patched dpkg of squeeze for amd64 and i386 can be fetched from here. An example package using Lzip compression, lzip itself, is here (for this example, the package is build as 3.0 native on purpose). Hopefully the patch will be merged so we can make use of lzip compressed binary and source packages in Debian.

The recent CVE-2010-2961 mountall vulnerability got a nice write-up by xorl today. I ve seen a few public exploits for it, but those that I ve seen, including the one in xorl s post, miss a rather important point: udev events can be triggered by regular users without any hardware fiddling. While the bug that kept udev from running inotify correctly on the /dev/.udev/rules.d directory during initial boot kept this vulnerability exposure pretty well minimized, the fact that udev events can be triggered at will made it pretty bad too. If udev had already been restarted, an attacker didn t have to wait at all, nor have physical access to the system. While it is generally understood that udev events are related to hardware, it s important to keep in mind that it also sends events on module loads, and module loads can happen on demand from unprivileged users. For example, say you want to send an X.25 packet, when you call socket(AF_X25, SOCK_STREAM), the kernel will go load net-pf-9, which modules.alias lists as the x25 module. And once loaded, udev sends a module event. (Which, by the way, should serve as a reminder to people to block module loading if you can.) So, as I mentioned, here s yet another exploit for the mountall vulnerability: mountall-CVE-2010-2961.py. It writes to the vulnerable udev rule file and then attempts to trigger udev immediately by walking a list of possible socket() AF_* types.

This essay is a summary of my presentation at the workshop Inlaws and Outlaws, held on August 19-20, 2010 in Split, Croatia. The workshop brought together advocates of piracy with participants in the free culture and free software movements.

In Why Software Should Not Have Owners, Richard Stallman explains that, if a friend asks you for a piece of software and the license of the software bars you from sharing, you will have to choose between being a bad friend or violating the license of the software. Stallman suggests that users will have to choose between the lesser of two evils and will choose to violate the license. He emphasizes that it's unfair to ask a user to make such a choice. Over the past few years, pirate parties have grown across much of the developed world. Of course, piracy remains the primary means of distributing media across most of the rest. Advocates of access to information have gathered and organized under the "pirate" banner, representing the choice of sharing with friends over compliance with license terms. The free/libre and open source software (FLOSS) and free culture movements seem to have a confused and conflicted reaction to all this. On one hand, major proponents of several pirate parties are FLOSS and free culture stalwarts and several pirate parties have made FLOSS advocacy a component of their political platforms. Pirate Parties' clear opposition to software patents and DRM resonates with both FLOSS and free culture communities. On the other hand, FLOSS leaders, including Stallman, have warned us about "pirate" anti-copyright policies. Free culture leaders, like Lawrence Lessig, have repeatedly and vociferously denounced piracy, treated even the intimation of an association with piracy as an affront, and systematically distanced themselves and their work from piracy. Should FLOSS and free culture advocates embrace pirates as comrades in arms or condemn them? Must we choose between being either with the pirates or against them? Our communities seem to have no clearly and consistently articulated consensus. I believe that, unintuitively, if you take a strong principled position in favor of information freedom and distinguish between principles and tactics, a more nuanced "middle ground" response to piracy is possible. In light of a principled belief that users should be able to share information, we can conclude there is nothing ethically wrong with piracy. Licenses have the power of the law but they are protected by unjust "intellectual property" laws. That said, principles are not the only reason activists choose to do things. Many political stunts are bad ideas not because they are wrong, but because they won't work and have negative side effects. Tactics matter too. Even though there might not be anything ethically wrong with piracy from the perspective of free software or free culture, it might still be a bad idea. There are at least three such tactical reasons that might motivate free software and culture to not support piracy or participate in pro-piracy movements and politics. First, a systematic disrespect for copyright undermines respect for all licenses which have been of a huge tactical benefit to free software and a increasingly important factor in the success of free culture. Copyleft licenses like the GPL or CC BY-SA have power only because copyright does. As Stallman has suggested, anti-copyright actions are anti-copyleft. That needn't be an argument against attempts to limit copyright. Indeed, I think we must limit and reduce copyright. But we must tread carefully. In the current copyright climate, I believe that copyleft offers a net advantage. Why should others respect our licenses if we don't respect theirs? Looking at the long term, we must weigh the benefits of promoting the systematic violation of proprietary licenses with the benefits of adherence to free software and free culture. Second, piracy is fundamentally reactionary. Part of its resonance as a political symbol comes from the fact that the piracy represents a way that consumers of media can fight back against a set of companies which have attacked them -- with lawsuits, DRM systems, and demonization in propaganda -- for sharing in ways that most consumers think are natural and socially positive. But piracy focuses on reaction rather the fundamental importance of sharing that drives it. As a result, most pirates do not support, or are even familiar with, a principled approach to access to information. As a result, many piracy advocates who speak out against DRM on DVDs will be as happy to use NetFlix to stream DRMed movies for $5 a month as they were to download for free. The best rallying cries do not always translate into be the most robust movements. Third, through its focus on a reaction, a dialog about piracy avoids engagement with the tough questions of what we will replace the current broken copyright system with. A principled position suggests that it is our ethical prerogative to create alternative models. The free software movement has succeeded because it created such a prerogative and then, slowly over time, provided examples of workable alternatives. A principled position on free software did not require that one provide working new systems immediately, but it makes the development of creative, sustainable approaches a priority. Attacking the system without even trying to speak about alternative modes of production is unsustainable. Free software and free culture call for a revolution. Piracy only calls for a riot. Piracy, in these three senses, can be seen as tactically unwise, without necessarily being unethical. By taking a principled position, one can go build on, and go beyond, RMS's comment. On free culture and free software's terms, we can suggest that piracy is not ethically wrong, but that it is an unwise way to try to promote sharing. Without being hypocritical, we can say: "I don't think piracy is unethical. But I also do not support it."

Fresh up on arxiv.org is a paper by Kevin Heng, Kristen Menou and Peter Phillipps: Atmospheric circulation of tidally-locked exoplanets: a suite of benchmark tests for dynamical solvers. Kristen Menou was at exoclimes 2010 where one of the background topics was how to compare and validate our various models in exoplanetary climates. Earth-based climate modellers have test cases such as Held-Suarez which are used to compare the models, but to date no equivalents are used for exoplanetary models. This work fills that gap for Tidally-locked atmospheres (similar to Zarmina for example, and some 'Hot Jupiters'. I'm particularly interested in the tidally-locked Earth case; it's relevant for my own PhD work. Incidentally, for a good summary of the different ways we test climate models, see Steve Easterbrook's recent blog entry; he's been doing a tour of major modelling centres, learning how they do things. Worth a look. Tags exoplanets, climate, benchmarks

Web Video and Global Innovation Chris Anderson (the curator of TED) gave an insightful TED talk about Web Video and Global Innovation [1]. Probably most people who have used the Internet seriously have an intuitive knowledge of the basic points of this talk, Chris had the insight to package it together in a clear manner. He describes how the printing press decreased the importance of verbal communication skills and services such as Youtube have caused a resurgence in the popularity and importance of speeches. He has some interesting theories on how this can be leveraged to improve education and society. Lectures for Developers vs Users Now how can we use these principles to advance the development of Free Software? It seems to me that a good lecture about Free Software achieve will achieve some of the following goals:

1. Promoting projects to new developers.
2. Teaching developers some new aspects of software development related to the system.
3. Promoting projects to new users.
4. Teaching users (and prospective users) how to use the software.

The talks aimed at developers need to be given by technical experts, but talks aimed at users don t need to be given by experts on the technology and someone who has less knowledge of the software but better public speaking skills could probably do a better job when speaking to users. Would it do some good to encourage people to join Free Software projects for the purpose of teaching users? It seems that there are already some people doing such work, but there seems little evidence of people being actively recruited for such work which is a stark contrast to the effort that is sometimes put in to recruiting developers. One problem in regard to separating the user-training and developer-training parts of Free Software advocacy and education is that most conferences seem to appeal to developers and the more Geeky users. Talks for such conferences tend to be given by developers but the audience is a mix of developers and users. Would it be better to have streams in conferences for developers and users with different requirements for getting a talk accepted for each stream? Publishing Videos It has become a standard feature of Free Software related conferences to release videos of all the talks so anyone anywhere in the world can watch them, but it seems that this isn t used as much as we would like. The incidence of Free Software developers citing TED talks in blog posts appears to exceed the incidence of them citing lectures by their peers, while TED talks are world leading in terms of presentation quality the talks by peers are more relevant to the typical Free Software developer who blogs. This seems to be an indication that there is a problem in getting the videos of talks to the audience. Would it help this to make it a standard feature to allow comments (and comments that are rated by other readers) on every video? Would having a central repository (or multiple repositories) of links to Free Software related talks help? Would it help to have a service such as Youtube or Blip.tv used as a separate repository for such talks? Instead of having each conference just use it s own servers if multiple conferences uploaded talks to Youtube (or one of it s competitors) then users could search for relevant talks (including conference content and videos made by individuals not associated with conferences). What about video replies ? What if after each conference there was an RSS feed of links to videos that had one video featured per day in a similar manner to the way TED dribbles the talks out. If you publish 40 videos of 45 minute lectures in one week you can be sure that almost no-one will watch them all and very few people will watch even half of them. But if you had an RSS feed that gave a summary of one talk per day for 6 weeks then maybe many people would watch half of them. Defining Success Chris cites as an example of the success of online video the competition by amateur dancers to create videos of their work and the way that this was used in selecting dancers for The LXD (Legion of eXtraordinary Dancers) [2]. I think that we need a similar culture in our community. Apart from people who give lectures at conferences and some of the larger user group meetings there are very few people giving public video talks related to Free Software. There is also a great lack of instructional videos. This is something that anyone could start doing at home, the basic video mixing that you need can be done with ffmpeg (it s not very good for that purpose, but for short videos it s probably adequate) and Istanbul is good for making videos of X sessions. If we had hundreds of Free Software users making videos of what they were doing then I m sure that the quality would increase rapidly. I expect that some people who made such videos would find themselves invited to speak at major conferences even if they hadn t previously considered themself capable of doing so (the major conferences can be a bit intimidating). How do we Start? Publishing videos requires some significant bandwidth, a cheap VPS has a bandwidth quota of 200GB per month, if short videos are used with an average size of 30MB (which seems about typical for Youtube videos) then that allows more than 6000 video views per month which is OK but as my blog averages about 2000 visits per day (according to Webalizer) it seems that 6000 views per month isn t enough for any serious vlogging. Not to mention the fact that videos in higher resolution or a sudden spike in popularity can drive the usage a lot higher. It seems that a site like Youtube or blip.tv is necessary, which one is best? There are lots of things that can be changed along the way, but a hosting service is difficult to change when people link to it. Conclusion I don t claim to have many answers to these questions. I m planning to start vlogging soon so I will probably learn along the way. I would appreciate any suggestions. Also if anyone has a long suggestion then a blog post will be best (I ll link to any posts that reference this one). If anyone has a long suggestion that is worthy of a blog post but they don t have a blog then I would be happy to post it on my blog.

• [1] http://www.ted.com/talks/chris_anderson_how_web_video_powers_global_innovation.html
• [2] http://thelxd.com/

A conffile is a configuration file managed by dpkg, I m sure you remember the introductory article about conffiles. When your package stops providing a conffile, the file stays on disk and it s recorded as obsolete by the package manager. It s only removed during purge. If you want the file to go away, you have to remove it yourself within your package s configuration scripts. You will now learn how to do this right. When is that needed? dpkg errs on the side of safety by not removing the file until purge but in most cases it s best to remove it sooner so as to not confuse the user. In some cases, it s even required because keeping the file could break the software (for example if the file is in a .d configuration directory, and if it contains directives that are either no longer supported by the new version or in conflict with other new configuration files). What s complicated in rm ? So you want to remove the conffile. Adding an rm command in debian/postinst sounds easy. Except it s not the right thing to do. The conffile might contain customizations made by the administrator and you don t want to wipe those. Instead you want to keep the file around so that he can get his changes back and do whatever is required with those. The correct action is thus to move the file away in the prerm, to ensure it doesn t disturb the new version. At the same time, you need to verify whether the conffile has been modified by the administrator and remember it for later. In the postinst, you need to remove the file if it s unmodified, or keep it under a different name that doesn t interfere with the software. In many cases adding a simple .dpkg-bak suffix is enough. For instance, run-parts ignore files that contain a dot, and many other software are configured to only include files with a certain extension say *.conf. In the postrm, you have to remove the obsolete conffiles that were kept due to local changes and you should also restore the original conffile in case the upgrade obsoleting the conffile is aborted. Automating everything with dpkg-maintscript-helper Phewww that s a lot of things to do for a seemingly simple task. Fortunately everything can be automated with dpkg-maintscript-helper. Let s assume you want to remove /etc/foo/conf.d/bar because it s obsolete and you re going to prepare a new version 1.2-1 with the appropriate code to remove the file on upgrade. You just have to put this snippet in the 3 relevant scripts (preinst, postinst, postrm):

if dpkg-maintscript-helper supports rm_conffile 2>/dev/null; then
    dpkg-maintscript-helper rm_conffile /etc/foo/conf.d/bar 1.2-1 -- "$@"
fi

You can avoid the preliminary test if you pre-depend on dpkg (>= 1.15.7.2) or if enough time has passed to assume that everybody has a newer version anyway. You can learn all the details in dpkg-maintscript-helper s manual page. I hope you found this article helpful. You can follow me on Identi.ca, Twitter and Facebook. 5 comments Support my work

Roland Mas writes an interesting article about decentralized naming, in which he says:

Monkeysphere aims at adding a web of trust to the SSL certificates system, but the CA chain problem seems to persist (although I must admit I'm not up to speed with the actual details).

Since i'm one of the Monkeysphere developers, i figure i should respond! Let me clarify that Monkeysphere doesn't just work in places where X.509 (the SSL certificate system) works. It works in other places too (like SSH connections). And I don't think that the CA chain problem that remains in Monkeysphere is anything like the dangerous mess that common X.509 usage has given us. I do think that at some level, people need to think about who is introducing them to other people -- visual or human-comprehensible representations of public key material are notoriously difficult to make unspoofable. On the subject of distributed naming: OpenPGP already allows distributed naming: every participant in the WoT is allowed to assert that any given key maps to any given identity. Duplicates and disagreements can exist just fine. How an entity decides to certify another entity's ID without a consensus global namespace is a tough one, though. If i've always been known as "John Smith" to my friends, and someone else has also been known as "John Smith" to his friends, our friends aren't actually disagreeing or in conflict -- it's just that neither of us has a unique name. The trouble comes when someone new wants to find "John Smith" -- which of us should they treat as the "correct" one? I think the right answer probably has to do with who they're actually looking for, which has to do with why they're looking for someone named "John Smith". If they're looking for John Smith because the word on the street is that John Smith is a good knitter and they need a pair of socks, they can just examine what information we each publish about ourselves, and decide on a sock-by-sock basis which of us best suits their needs. But if they're looking for "John Smith" because their cousin said "hey, i know this guy John Smith. I think you would like to argue politics over a beer with him", then what matters is the introduction. And OpenPGP handles that just fine -- if their cousin has only ever met a single John Smith, that's the right one. If their cousin has met several John Smiths, then the searcher would do well to ask their cousin some variant of "hey, do you mean John Smith or John Smith ", or even "do you mean the John Smith who Molly has met, or the one who Charles has met?" (assuming that Molly and Charles have each only certified one John Smith in common with the cousin, and not the same one as each other), or to get a real-time introduction to a particular John Smith, where his specific key is somehow recordable by the searcher for future conversations (or beer drinking). This is what we do in the real world anyway. We currently lack good UIs for doing this over the network, but the certification infrastructure is in place already. What we're lacking in infrastructure, though, is a way to have a distributed addressing. Roland's proposal was to publish addresses corresponding to cryptographic identities within some DNS zone, or in freenet or gnutella. Another approach (piggybacking on existing infrastructure) would be to include IP address information in the OpenPGP self-certification, so the holder of the name could claim exactly their own IP address. This could be distributed through the keyserver network, just like other updates are today, and it could be done simply and immediately with a well-defined OpenPGP notation. I'd be happy to talk to interested people about how to specify such a notation, and what possible corner cases we might run into. Drop a note here, or mail the Monkeysphere mailing list or hop onto #monkeysphere on irc.oftc.netTags: authentication, distributed naming, identity, monkeysphere

Review: Thinking in Java, by Bruce Eckel

Publisher:	Prentice Hall
Copyright:	2006
Printing:	October 2009
ISBN:	0-13-187248-6
Format:	Trade paperback
Pages:	1461

Thinking in Java has gone through multiple editions. This review is of the 4th edition, which is targetted at Java 5 (with some minor updates for Java 6, released as it was going to press). Prior to reading this book, I'd never written code from scratch in Java and had never learned anything about Java in a structured fashion. My CS degrees come before the era of ubiquitous Java; the most I'd ever done with the language was look over and make some minor fixes to code other people had written, using general programming knowledge. I choose this book as my introduction because I was looking for a ground-up introduction with good reviews and was avoiding books that contained a printed version of the Java library documentation. I can get that myself on-line. I kept a set of notes on my impressions of the Java language itself separately and will not get into that here. This is, instead, a review of Thinking in Java as a book and as a training tool for someone who wants to pick up the language. First off, this is a remarkably well-titled book, much more so than the typical programming language book. Eckel is not just teaching Java as one programming language among many. He tries hard to teach the reader to think in Java, with a lot of attention to idiomatic use of the language and natural ways to use its expressiveness. This is exactly what I was looking for I don't want to just write C or Perl in Java syntax and I was very satisfied with the result. The book is also very heavy on source code examples; this is much of the reason why it's 1,461 pages. Eckel illustrates everything with Java code, generally complete functioning short programs. Even exploratory musings and experiments are implemented as Java code. I don't think there's any significant thought in this book that isn't shown in both text and in explained Java code. Eckel has also clearly put significant effort into choosing good examples and designing his sample programs. I have some past (mild) experience with object-oriented design, and a lot of experience with higher-level system design, and I was satisfied and occasionally intrigued by the object structures and design tips that Eckel provides. He uses design pattern language heavily, but not in a confusing way, and he takes some time to explain typical design patterns and illustrate them repeatedly. This is a solid introduction to good object-oriented program design as well as an introduction to Java. One oddity of this book, though, is that Eckel introduces the language from the ground up, starting with syntax and basic types, and is very careful not to use any concepts before he introduces them. The result is a very thorough introduction to the Java language, but it means that it takes 900 pages (!) before Eckel introduces I/O and the possibility of writing a useful program based on just the information in the book. This is for entirely sound conceptual reasons: I/O in Java relies on lots of concepts, like exceptions and generics, that require extensive introduction and discussion. However, it does mean that through well more than half of a monumental tome the reader is learning the language through rather artificial programs. I have extensive prior programming background and a high tolerance for more abstract examples (I have a master's degree in software theory), so I could deal with this, but even with that background I was starting to feel like my head was getting too full and too crowded with concepts without being able to make them concrete. It probably would have helped to work through some of the exercises, but I dislike exercises that aren't grounded in some problem I personally want to solve. If you're someone who only learns languages through practical application, this approach to teaching the language may drive you nuts, and you may want to look for another book that compromises conceptual progression to get you started faster. Related to this, Thinking in Java is not about the standard library. It's about the language, and the vast majority of this book is focused directly on the language level: inheritance, information hiding, generics, exceptions, introspection, concurrency, annotations, serialization, and so forth. The only parts of the standard library that Eckel covers in depth are the container classes, I/O, and a somewhat odd and misplaced final chapter on Swing. There is nothing about databases, essentially nothing about web applications or containers, no network programming, and very little in the way of practical, applied problems. As you might guess from the combination of that limitation of scope and the length of this book, that means it is extremely comprehensive; I doubt there are many corners of the basic language semantics, including around generics, that Eckel leaves unexplored. This is exactly what I wanted (I know how to read library documentation myself and all language core libraries start looking mostly the same after you've programmed in five or six languages), but it's very different from the typical approach of a programming language book. Don't be surprised. This biggest problem with Thinking in Java from my perspective is that it defers many interesting topics to Eckel's web pages and to free companion books and material that is supposedly found there, to the point of advertising that on the cover. This is a good way to allow the reader to explore extra topics that are of specific interest while skipping others, provided that one actually delivers on that promise. However, Eckel's web site is dire. The first time I looked at it earlier this year I was unable to find most of the supplemental material the book promises is there, and now www.mindview.net doesn't resolve in DNS. I've found copies of some of that material hosted elsewhere in Google searches, but this is a rather frustrating experience if one was relying on that supplemental material to fill in the gaps. And the book is clearly written expecting people to go look at the web materials; Eckel refers to them probably a hundred times over the course of the book. Less of a serious problem, and more of a quirk, is the closing chapter on Swing. This chapter really doesn't fit much with the book. I suppose Eckel felt obligated to put in at least one chapter on some concrete applied topic (I personally would have preferred databases be that topic, but oh well), and it does serve a useful purpose in showing how concurrency, inner classes, inheritance, and introspection all work together in a significant Java library. But it's not presented as just an example, and it fits in oddly with the rest of the book. It also has an inexplicable middle section about Flash programming. I ended up mostly skimming this. But this isn't much of a flaw in the book; if one ignores 100 pages out of 1,450, that's not much of a loss. Eckel comes from a C++ background (he was a founding member of the C++ standardization committee), which shows up most prominantly in the generics chapter. It's a solid, thorough introduction to Java generics, but it also has an extensive exploration of all the problems with the Java erasure model and many, many examples of the things that one would find intuitive from C++ that don't work the same way in Java. If you come from a C++ background, this is quite possibly useful; if, like me, you don't care about C++, it's at best mildly interesting and gets a touch tedious. There are a few other places where Eckel pokes surprisingly sharp sticks at the Java designers, but taken in context I think this is a quirk of the book's conversational style (which is otherwise a feature). Eckel's love of Python is also somewhat entertaining (particularly the long advertisement for Python in the introductory chapters of a book on Java), but has little impact on the book. I think the largest drawback of it is that all the examples in this book are tested using a Python framework, which isn't so much a problem with them as a lost opportunity. They're annotated with metadata used by Eckel's Python test framework, and in a Java book I would have been much more interested in a good Java test framework built on top of something like JUnit. Oh well. Overall, this is a very solid book which, despite being enormous, does not feel pointlessly padded. It's thorough and exhaustive on its topic, but there's a point to nearly everything in here. It was also exactly the Java book I wanted: an introduction to the language, built on a firm foundation of object-oriented program design, with discussion and pointers on how to think idiomatically in the language and use it to its fullest potential, and without mindlessly reproducing the standard library documentation. If that's also what you're looking for in a Java book, I recommend it highly. Just know going in what sort of book you're getting, since its pacing and order of presentation may frustrate some learning styles. Rating: 8 out of 10

I ve been keeping up with my amateur radio activities lately. The big news is that I passed the amateur extra exam at a hamfest in Wichita on Saturday. That s the highest level of amateur radio licensing available, and only about 17% of hams have it. I passed it with 100% correct, as I did with my other two exams as well. Pretty happy with that. I m also getting ready to get a radio installed in my car. My FT-857D, which I ve been using in the house, is really a mobile. I picked up antennas and cables for that at the Wichita hamfest. My brother, who is great with cars, will probably help me do that. I m excited about that. I also think I can get radio going on my bicycle. I need a half-wave vertical antenna to mount on it. Really the only challenge is how to attach that to the bicycle, as there isn t exactly a thriving antenna mount industry for bicyclists.

Many people know Ultimate Debian Database already, and do use of it to create useful QA tools, or other Debian related activities. There is another database which is widely used by all of us, every day, and runs behind the scenes without being noticed. It is called projectb, the huge database of the Debian archive. projectb is the place where dak stores all the information it needs to run the archive (including but not limited to source packages, related binaries, suite definitions and associations, overrides, and so on), and it tells dak to hit you with a REJECT if you try to upload a package twice, or if you do something The Wrong Way. OK OK, it eventually allows your packages to be accepted too, sometimes it also behaves good projectb is hosted on ftp-master.debian.org. Most fellow Debian Developers cannot login over there because access is restricted to FTP and Release people only, but can enjoy its clone on merkel.debian.org ^[1]. Please note that clone is synced on a regular basis, but it could be a little bit not up-to-date with the very latest changes. Just run the following: psql -d projectb -p 5433 Now that you have access to it, you could want look at its structure to start looking at it. I just provide a couple of trivial examples of what you can do with it, your imagination is the limit ^[2] This query displays which dpkg version known suites ship:
SELECT s.source, max(s.version) AS version, su.suite_name AS suite
FROM source s
JOIN src_associations sa ON sa.source = s.id
JOIN suite su ON su.id = sa.suite
WHERE s.source = 'dpkg'
GROUP BY s.source, su.suite_name
ORDER BY version; This query displays changelog entries for debhelper which versions are greater than 8:
SELECT changelog
FROM changelogs
WHERE source = 'debhelper'
AND version > '8'; Working with projectb means working with real Debian archive data, so you are sure to always get solid information. You can also experience a limited (but functional) dak session on merkel. You cannot do much, data is read-only, but this is not a limit for a couple of commands. The most interesting is dak rm, where you can simulate a package removal, and discover reverse dependencies in advance, to save some work to your lovely FTP Team Just run the following:
dak rm -n -R -m "Don't worry, it's a Dry Run" your_behated_package Wait a little bit, and you will be prompted with removal report. Just remember to always use -n and -R flags, though. Want even more? Join FTP Team, and you will given the chance to learn a lot more about Debian! DISCLAIMER: if you plan to write a tool, or perform an activity which relies on current projectb implementation, PLEASE TALK TO FTP TEAM FIRST. Internal structure can change a lot over times, and it is way better let FTP Team to handle exporting/parsing its data for day-by-day tasks. ^[1] This is probably supposed to change, as merkel is going to be decommissioned soon. ^[2] If you need a new feature, just talk to FTP Team.

When I am writing software sometimes XKCD is accurate!

Richard Baraniuk gave an interesting TED talk about Open Source Learning [1]. His project named Connexions which is dedicated to the purpose of creating Creative Commons free textbooks is a leader in this space [2]. He spoke about Catherine Schmidt-Jones who wrote 197 modules and 12 courses on music [3], that s a very significant amount of work! He also mentioned the translation of the work into other languages. I wonder how well the changes get merged back across the language divide. We have ongoing disputes in the free software community about whether various organisations do enough work to send patches back upstream, this seems likely to be more of a problem in situations where most of the upstream authors can t even understand the language in which the changes are written and when the changes involve something a lot more subtle than an change to an algorithm. This would be particularly difficult for Chinese and Japanese as those languages seem to lack quality automatic translation. He mentioned Teachers Without Borders [4] in passing. Obviously an organisation that wants to bring education to some of the poorer parts of the world can t have a curriculum that involves $250 of text books per year for a high school student (which was about what my parents paid when I was in year 12) or $500 of text books per year for a university student (which might be a low estimate for some courses as a single text can cost more than $120). Free content and on-demand printing (or viewing PDF files on a OLPC system) can dramatically lower the cost of education. It s widely believed that free content without the ability to remix is cultural imperialism. This is apparently one of the reasons that the connexions project is based on the Creative Commons Attribution license [5]. So anyone anywhere can translate it, make a derivative work, or collate parts of it with other work. I expect that another factor is the great lack of success of all the various schemes that involve people contributing content for a share of the profits, the profits just don t match the amount of work involved. Philanthropy and reputation seem to be the only suitable motivating factors for contributing to such projects. One of the stated benefits of the project is to have computer based content with live examples of equations. Sometimes it is possible to just look at an equation and know what it means, but often more explanation is required. The ability to click on an equation, plug in different values and have them automatically calculated and possibly graphed if appropriate can make things a lot easier. Even if the result is merely what would be provided by reading a text book and spending a few minutes with a scientific calculator the result should be a lot better in terms of learning as the time required to operate a calculator can break the student s concentration. Even better it s possible to have dynamic explanations tailored to the user s demand. To try this out I searched on Ohm s Law (something that seems to be unknown by many people on the Internet who claim to understand electricity). I was directed to an off-site page which used Flash to display a tutorial on Ohm s Law, the tutorial was quite good but it does seem to depart from the free content mission of the project to direct people off-site to proprietary content which uses a proprietary delivery system. I think that the Connexions project could do without links to sites such as college-cram.com. One of the most important features of the project is peer review lenses . The High Performance Computing Lens [6] has some good content and will be of interest to many people in the free software community but again it requires Flash. One final nit is the search engine which is slow and not very useful. A search for engine returned lots of hits about engineering which isn t useful if you want to learn about how engines work. But generally this is a great project, it seems to be doing a lot of good and it s got enough content to encourage other people and organisations to get involved. It would be good to get some text books about free software on there!

• [1] http://www.ted.com/talks/lang/eng/richard_baraniuk_on_open_source_learning.html
• [2] http://cnx.org/
• [3] http://cnx.org/member_profile/Catherine
• [4] http://www.teacherswithoutborders.org/
• [5] http://creativecommons.org/licenses/by/2.0/

Debian s testing distribution is where Debian developers prepare the next stable distribution. While this is still its main purpose, many users have adopted this version of Debian because it offers them a good trade-off between stability and freshness. But there are downsides to using this distribution and the Constantly Usable Testing (CUT) project aims to resolve those. This article will present the project and the challenges involved to make it happen. About Debian unstable & testing Debian unstable is the distribution where developers upload new versions of their packages. It happens frequently that some packages are not installable due to changes in other packages or due to transitions not yet completed. Debian testing, on the contrary, is managed by a tool that ensures the consistency of the whole distribution: it picks updates from unstable only if the package has been enough tested (10 days usually), if it s free of new release-critical bugs, if it s available on all supported architectures, and if it doesn t break any other package already present in testing. The Release Team (RT) controls this tool and provide hints to help it find a set of packages that can flow from unstable to testing. Those rules also ensure that the packages that flow into testing are reasonably free of show-stopper bugs (like a system that doesn t boot, or X that doesn t work at all). This makes it very attractive to users who like to regularly get new upstream versions of their software without dealing with the biggest problems associated to them. This is all very attractive, yet several Debian developers advise people to not use testing. Why is that? Known problems with testing Disappearing software The release team use this distribution to prepare the next stable release and from time to time they remove packages from it. Either because it s needed to ensure that other packages can migrate from unstable to testing, or because they have long-standing release-critical bugs without progress towards a resolution. It also happens that they remove packages on request of the maintainers because they believe that the current version of the software cannot be supported (security-wise) for 2 years or more. The security team also regularly issues such requests. Long delays for security and important fixes Despite the 10-day delay in unstable, there are always some annoying bugs (and security bugs are no exceptions) that are only discovered when the package already has migrated to testing. The maintainer might be quick to upload a fixed package in unstable, and might even raise the urgency to allow the package to migrate sooner, but if the packages gets entangled in a large ongoing transition, it will not migrate before the transition is completed. Sometimes it can take weeks for that to happen. This delay can be avoided by doing direct uploads to testing (through testing-proposed-updates) but this is almost never used, except during a freeze, where targeted bugfixes are the norm. Not always installable With testing evolving daily, updates sometimes break the last installation images available (in particular netboot images that get everything from the network). The debian-installer (d-i) packages are usually quickly fixed but they don t move to testing automatically because the new combination of d-i packages has not necessarily been validated yet. Colin Watson sums up the problem:

Getting new installer code into testing takes too long, and problems remain unfixed in testing for too long. [...] The problem with d-i development at the moment is more that we re very slow at producing new d-i *releases*. [...] Your choices right now are to work with stable (too old), testing (would be nice except for the way sometimes it breaks and then it tends to take a week to fix anything), unstable (breaks all the time).

CUT s history CUT finds its root in an old proposal by Joey Hess: it introduces the idea that the stable release is not Debian s sole product and that testing could become with some work a suitable choice for end-users. Nobody took on that work and there was no visible progress in the last 3 years. But recently Joey brought up CUT again on the debian-devel mailing list and Stefano Zacchiroli (the Debian project leader) challenged him to setup a BoF on CUT for Debconf10. It turned out to be one of the most heavily attended BoF (video recording is here), there is clearly a lot of interest in the topic. There s now a dedicated wiki and an Alioth project with a mailing list. The rest of this article tries to summarize the various options discussed and how they re supposed to address the problems identified. The ideas behind CUT Among all the ideas, there are two main approaches that have been discussed. The first is to regularly snapshot testing at points where it is known to work reasonably well (those snapshots would be named cut ). The second is to build an improved testing distribution tailored to the needs of users who want a working distribution with daily updates, its name would be rolling . Regular snapshots of testing There s general agreement that regular snapshots of testing are required: it s the only way to ensure that the generated installation media will continue to work until the next snapshot. If tests of the snapshot do not reveal any major problem, then it becomes the latest cut . For clarity, the official codename would be date based: e.g. cut-2010-09 would be the cut taken during September 2010. While the frequency has not been fixed yet, the goal is clearly to be on the aggressive side: at the very least every 6 months, but every month has been suggested as well. In order to reach a decision, many aspects have to be balanced. One of them (and possibly the most important) is the security support. Given that the security team is already overworked, it s difficult to put more work on their shoulders by declaring that cuts will be supported like any stable release. No official security support sounds bad but it s not necessarily so problematic as one might imagine. Testing s security record is generally better than stable s one (see the security tracker) because fixes flow in naturally with new upstream versions. Stable still get fixes for very important security issues sooner than testing, but on the whole there are less known security-related problems in testing than in stable. Since it s only a question of time until the fixed version comes naturally from upstream, more frequent cut releases means that users get security fixes sooner. But Stefan Fritsch, who used to be involved in the Debian testing security team, has also experienced the downside for anyone who tries to contribute security updates:

The updates to testing-security usually stay useful only for a few weeks, until a fixed version migrates from unstable. In stable, the updates stay around for a few years, which gives a higher motivation to spend time on preparing them.

So if it s difficult to form a dedicated security team, the work of providing security updates comes back to the package maintainer. They are usually quite quick to upload fixed packages in unstable but tend to not monitor whether the packages migrate to testing. They can t be blamed because testing was created to prepare the next stable release and there is thus no urgency to get the fix in as long as it makes it before the release. CUT can help in this regard precisely because it changes this assumption: there will be users of the testing packages and they deserve to get security fixes much like the stable users. Another aspect to consider when picking a release frequency is the amount of associated work that comes with any official release: testing upgrades from the previous version, writing release notes and preparing installation images. It seems difficult to do this every month. With this frequency it s also impossible to have a new major kernel release for each cut (since they tend to come out only every 2 to 3 months) and the new hardware support that it brings is something worthwhile to many users. In summary, regular snapshots address the not always installable problem and changes the perception of maintainers towards testing, so that hopefully they care more of security updates in that distribution (and in cuts). But they do not solve the problem of disappearing packages. Something else is needed to fix that problem. A new rolling distribution? Lucas Nussbaum pointed out that regular snapshots of Debian is not really a new concept:

How would this differentiate from other distributions doing 6-month release cycles, and in particular Ubuntu, which can already be seen as Debian snapshots (+ added value)?

In Lucas s eyes, CUT becomes interesting if it can provide a rolling distribution (like testing) with a constant flux of new upstream releases . For him, that would be something quite unique in the Free Software world . The snapshots would be used as starting point for the initial installation, but the installed system would point to the rolling distribution and users would then upgrade as often as they want. In this scenario, security support for the snapshots is not so important, what matters is the state of the rolling distribution. If testing were used as the rolling distribution, the problem of disappearing packages would not be fixed. That s why there have been discussions of introducing a new distribution named rolling that would work like testing but with adapted rules, and the cuts would then be snapshots of rolling instead of testing. The basic proposal is to make a copy of testing and to re-add the packages which have been removed because they are not suited for a long term release while they are perfectly acceptable for a constantly updated release (the most recent example being Chromium). Then it s possible to go one step further: during freeze, testing is no longer automatically updated which makes it inappropriate to feed the rolling distribution. That s why rolling would be reconfigured to grab updates from unstable (but using the same rules than testing). Given the frequent releases, it s likely that only a subset of architectures would be officially supported. This is not a real problem because the users who want bleeding edge software tends to be desktop users on mainly i386/amd64 (and maybe armel for tablets and similar mobile products). This choice if made opens up the door to even more possibilities: if rolling is configured exactly like testing but with only a subset of the architectures, it s likely that some packages migrate to rolling before testing when non-mainstream architectures are lagging in terms of auto-building (or have toolchain problems). While being ahead of testing can be positive for the users, it s also problematic on several levels. First, managing rolling becomes much more complicated because the transition management work done by the release team can t be reused as-is. Then it introduces competition between both distributions which can make it more difficult to get a stable release out, for example if maintainers stop caring of the migration to testing once the migration to rolling has been completed. The rolling distribution is certainly a good idea but the rules governing it must be designed to avoid any conflict with the process of releasing a stable distribution. Lastly, the mere existence of rolling would finally fix the marketing problem plaguing testing: the name rolling does not suggest that the software is not yet ready for prime time. Conclusion Whether CUT will be implemented remains to be seen, but it s off for a good start: ftpmaster Joerg Jaspert said that the new archive server can cope with a new distribution, and there s now a proposal shaping up. The project might start quickly: there is already an implementation plan for the snapshot side of the project. The rolling distribution can always be introduced later, once it is ready. Both approaches can complement each other and provide something useful to different kind of users. The global proposal is certainly appealing: it would address the concerns of obsolescence of Debian s stable release by making intermediary releases. Anyone needing something more recent for hardware support can start by installing a cut and follow the subsequent releases until the next stable version. And users who always want the latest version of every software could use rolling after having installed a cut. From a user point of view, there are similarities with the mix of usual and long term releases of Ubuntu. But from the development side, the process followed would be quite different, and the constraints imposed by having a constantly usable distribution are stronger: any wide-scale change must be designed in a way that it can happen progressively in a transparent manner for the user.

This article was first published in Linux Weekly News. If you want to see more articles like this, join Flattr and click on the flattr button below every article that you like.

21 comments Support my work

I've recently returned from ICFP 2010 (International conference on functional programming). This was the second time I went there, and like last year, it was a very interesting conference. This year, I've even managed to do one keysigning with a fellow Debian Developer (hello Sylvain!) While there were many talks, I only mention a few key ones, that I have found very cool. Note: I'm a layman in this field. The text below is my own interpretation of the talks and contains many errors and wrong facts corrections are welcome! Workshop on ML I wasn't present at this workshop, but other people pointed me to this very cool talk: Mirage project The abstract, to be found at http://www.cs.rit.edu/~mtf/ml2010/program.html#Program (search for Mirage), says:

Our framework, dubbed Mirage, extends the Objective Caml language with I/O extensions and a custom runtime to emit binaries that execute as a paravirtual operating system directly under Xen. The custom run-time is significantly simpler than a general-purpose operating system, with a static single-process 64-bit address space, and "zero-copy" I/O that works directly with the OCaml garbage collector. As a result, Mirage applications exhibit significant performance speedups for I/O and memory handling versus the same code running under Linux/Xen.

It's extremely interesting, IMHO. You write your code as a more-or-less normal OCaml program, but you compile it with this special tool-chain, and you boot it directly on ton of Xen, as a domU, without the intermediate OS. The project Git repo is at http://github.com/avsm/mirage, and that links a bit older presentation (http://anil.recoil.org/papers/2010-hotcloud-lamp.pdf) which IMHO is a required read for anyone interested in cloud-related stuff. ICFP proper ICFP, the main conference, is as usual, full of math, designed to scare people away. Err, I mean, designed to tell about new developments in this field. The papers are gathered in this reddit post: http://www.reddit.com/r/haskell/comments/dji4v/papers_from_icfp_2010/. XenServer The paper http://anil.recoil.org/papers/2010-icfp-xen.pdf describes the XenServer project, and more specifically the management interface, which is written entirely in OCaml, by a team of ~10 people. They also have a bit of bin-packing code (http://xenbits.xen.org/xapi/xen-api.hg?file/95b9e4f1b9dd/ocaml/xapi/binpack.ml), but much simpler than the ganeti-htools project on which I'm working (no balancing or similar). On the other hand, it is interesting to see that they committed themselves to write the entire management suite in OCaml, as opposed to our Python plus just a few bits of Haskell. I envy them, very much so The fun tidbit was how stable they found the OCaml tool-chain. In three years, they had just one bug related to this (the tool-chain), a segfault due to wrong formatting code in the handling path of a seldom-triggered Python exception, which led to a %s to be translated as is to C code, which meant printf got the wrong specifiers. So in an OCaml project, a serious bug was due to lack of type checking in string formatting in a Python component Overall I have found interesting parallels between their experience and my own with Haskell; from is this language/tool-chain stable enough for use to concerns about using FP for a significant project (but of course, orders of magnitude difference between their case and mine). Super-compilation There were two papers on this topic; one at ICFP proper and one at the Haskell symposium. If I understood things right, super-compilation is the opposite of lazy evaluation. But let's start a bit back. First, programs written in a functional languages contain many pure function. Pure functions, being side-effect-free, exhibit many interesting properties, and one of them is that they permit decoupling of the computation of a value from the declaration of such computation. One technique derived from this is lazy computations. Lazy computation means that even though you write y = sin(x) + 1, the actual call to sin and the increment is not actually done until you need the actual value of y (not just for storing it in y, that is). Super-compilation is the opposite of this. While laziness postpones doing a computation to as late as possible, in the hope it won't be needed at all, super-compilation runs the actual computation way ahead of time, even before the program starts, that is at compile time. Since programs use input data at runtime, which is not known at compile time, the trick is to pre-compute as much as possible at this stage, given incomplete information; and the papers showed that it's possible to pre-compute much more than you'd usually believe. Optimising compilers already do some form of CAF (constant applicative form) optimisation, but the techniques presented in both papers are going beyond simple CAFs. The details are in the papers, and are complicated, but one simple example is converting this:

ones = map inc [1..]

(note that is an infinite list, and you don't want to fully compute that at any time, neither at compile nor at runtime), into:

ones = [2..]

The details on how this is done depend on the exact super-compilation implementation, and not all implementations can do the same optimisations, but in the presentations at this ICFP the idea was to split things into static/non-static parts, and build on that, or to do partial evaluation, etc. The result is that super-compilation shows significant increases in speed (in some synthetic benchmarks up to 80%), with a few benchmarks not being sped up at all, but most are. The downside is an explosion in code size, as many more things are pre-computed. Still, as a research topic, this is very interesting. My own paper Surprisingly, it seems I survived giving my talk (just an experience report, not actually fancy stuff like the rest). Either the people actually found it interesting (less likely), or they were very polite (more probably). Anyway, the paper is on the papers page (shameless self-promotion) (if you care). Haskell symposium This is co-located with ICFP, and is more practical than ICFP (which is, as I said, way too much math). The papers are again available on reddit: http://www.reddit.com/r/haskell/comments/dkzn4/papers_from_the_haskell_symposium_2010/. Orc: Concurrent Orchestration in Haskell How to easily write code for synchronising concurrent work? As an example, how complicated is to solve this problem:

[from the Orc literature] we might wish to contact two airlines simultaneously seeking price quotes. If either quote comes back below a threshold price, say $300, then let's buy a ticket immediately. On the other hand if both quotes exceed the threshold, then buy the cheapest ticket. Additionally, buy a ticket if the other airline does not give a timely quote, or notify the user if neither airline provides a timely quote.

Now, in my experience, this would require manual management and synchronisation of threads, and for example in Python the early abort of querying the price from airline B if airline A returned a good price would be non-trivial; in addition, adding two extra conditions would be again not a simple composition of rules. In Orc however, it takes all of 10 lines of code to do this, safely. I won't copy the code here, see the paper for details, or the API docs. And the best part is, this is just a cabal install orc away! I'll be very interested to see how I can use this in practice, as it is seems indeed a very powerful library. Nikola: Embedding Compiled GPU Functions in Haskell Interesting project. Using almost 100% Haskell code, and it marshalls the data to/from the GPU; in the example given, the performance is as good as hand-written CUDA code of course, this doesn't happen in all cases. It it's interesting how they manage to compile the GPU-related code to actual object code at Haskell compile time, which makes it transparent for the user. Given the prevalence of high-speed GPUs, this could be a nice way to utilise them without learning yet another language (or two, since NVidia and ATI have different stacks). They've also developed some interesting techniques (e.g. manual control of sharing, to prevent inlining) as part of their project. Scalable Event Handling for GHC This is a paper co-authored by Johan Tibell of Google Switzerland and Bryan O'Sullivan of Real World Haskell fame. Very interesting serving 20K QPS from a Haskell process while having 16K+ idle clients connected:

[ ] the performance of the epoll back end does not begin to fall off significantly until we have 50,000 idle connections open.

The talk started with the idea that we're on the path to serving 10 clients from a single machine, and how to improve Haskell's I/O manager to achieve (almost) 10 concurrent clients. The slides (which I don't have a link for) were a bit more cool on numbers than the paper. Oh, and this will come with GHC 7.0, for free An LLVM Backend For GHC The eternal fight to get better than the C backend in other words, to not have to translate Haskell to C It is interesting to see how many languages target LLVM nowadays, as opposed to plain old C via gcc. A fun trivia item was that, according to the author, there are many features in LLVM that are not actually used by 90% of the languages targeting it (e.g. the garbage collector) (if I heard correctly in the talk). The results are interesting, although not too much overall speedup is gained; this seems to be due to the hard-to-optimise Cmm intermediate language inside GHC. In any case, a few benchmarks get more than 10% speedup, even in the current form. This will also come in GHC 7.0, although not enabled by default; probably needs to mature some more Conclusion I've enjoyed this conference a lot, and I've learned a lot from it. Given that many of the talks are about research topics, it's harder to apply them in day-to-day work, but I think the pay-offs will be long term rather than immediate.

When I took over man-db in 2001, one of the major problems that became evident after maintaining it for a while was the way it handled subprocesses. The nature of man and friends means that it spends a lot of time calling sequences of programs such as zsoelim < input-file tbl nroff -mandoc -Tutf8. Back then, it was using C library facilities such as system and popen for all this, and I had to deal with several bugs where those functions were being called with untrusted input as arguments without properly escaping metacharacters. Of course it was possible to chase around every such call inserting appropriate escaping functions, but this was always bound to be error-prone and one of the tasks that rapidly became important to me was arranging to start subprocesses in a way that was fundamentally immune to this kind of bug. In higher-level languages, there are usually standard constructs which are safer than just passing a command line to the shell. For example, in Perl you can use system([$command, $arg1, $arg2, ...]) to invoke a program with arguments without the interference of the shell, and perlipc(1) describes various facilities for connecting them together. In Python, the subprocess module allows you to create pipelines easily and safely (as long as you remember the SIGPIPE gotcha). C has the fork and execve primitives, but assembling these to construct full-blown pipelines correctly is difficult and error-prone, so many programmers don't bother and use the simple but unsafe library facilities instead. I wrote a couple of thousand lines of library code in man-db to address this problem, loosely and now quite distantly based on code in groff. In the following examples, function names starting with command_, pipeline_, or decompress_ are real functions in the library, while any other function names are pseudocode. Constructing the simplified example pipeline from my first paragraph using this library looks like this:

pipeline *p;
int status;
p = pipeline_new ();
p->want_infile = "input-file";
pipeline_command_args (p, "zsoelim", NULL);
pipeline_command_args (p, "tbl", NULL);
pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
pipeline_start (p);
status = pipeline_wait (p);
pipeline_free (p);

You might want to construct a command more dynamically:

command *manconv = command_new_args ("manconv", "-f", from_code,
                                     "-t", "UTF-8", NULL);
if (quiet)
	command_arg (manconv, "-q");
pipeline_command (p, manconv);

Perhaps you want an environment variable set only while running a certain command:

command *less = command_new ("less");
command_setenv (less, "LESSCHARSET", lesscharset);

You might find yourself needing to pass the output of one pipeline to several other pipelines, in a "tee" arrangement:

pipeline *source, *sink1, *sink2;
source = make_source ();
sink1 = make_sink1 ();
sink2 = make_sink2 ();
pipeline_connect (source, sink1, sink2, NULL);
/* Pump data among these pipelines until there's nothing left. */
pipeline_pump (source, sink1, sink2, NULL);
pipeline_free (sink2);
pipeline_free (sink1);
pipeline_free (source);

Maybe one of your commands is actually an in-process function, rather than an external program:

command *inproc = command_new_function ("in-process", &func, NULL, NULL);
pipeline_command (p, inproc);

Sometimes your program needs to consume the output of a pipeline, rather than sending it all to some other subprocess:

pipeline *p = make_pipeline ();
const char *line;
line = pipeline_peekline (p);
if (!strstr (line, "coding: UTF-8"))
	printf ("Unicode text follows:\n");
while (line = pipeline_readline (p))
	printf ("  %s", line);
pipeline_free (p);

man-db deals with compressed files a lot, so I wrote an add-on library for opening compressed files (which is somewhat man-db-specific, but the implementation wasn't difficult given the underlying library):

pipeline *decomp_file = decompress_open (compressed_filename);
pipeline *decomp_stdin = decompress_fdopen (fileno (stdin));

This library has been in production in man-db for over five years now. The very careful signal handling code has been reviewed independently and the whole thing has been run through multiple static analysis tools, although I would always welcome more review; in particular I have no idea what it would take to make it safe for use in threaded programs since I generally avoid threading wherever possible. There have been a handful of bugs, which I've fixed promptly, and I've added various new features to support particular requirements of man-db (though in as general a way as possible). Every so often I see somebody asking about subprocess handling in C, and I wonder if I should split this library out into a standalone package so that it can be used elsewhere. Web searches for things like "pipeline library" and "libpipeline" don't reveal anything that's a particularly close match for what I have. The licensing would be GPLv2 or later; this isn't likely to be negotiable since some of the original code wasn't mine and in any case I don't feel particularly bad about giving an advantage to GPLed programs. For more details on the interface, the header file is well-commented. Is there enough interest in this to make the effort of producing a separate library package worthwhile? As well as the general effort of creating a new package, I'd need to do some work to disentangle it from a few bits and pieces specific to man-db. If you maintain a specific package that could use this and you're interested, please contact me with details, mentioning any extensions you think you'd need. I intentionally haven't enabled comments on my blog for various reasons, but you can e-mail me at cjwatson at debian.org or man-db-devel at nongnu.org.

I travel pretty regularly, about 35% so far in 2010. When it goes wrong, travel can be exhausting, frustrating, complicated, stressful and even debilitating. I m always looking for ways to make my trips run more smoothly. On a recent flight to Taipei, I wrote down a few of the techniques which I ve successfully put into practice and found helpful. This is not an exhaustive list; I ve omitted a lot of the common and obvious tips I ve seen elsewhere.

1. Make a packing list. This one may be obvious, but a lot of people neglect it. Perhaps they think making lists is boring and fussy, but really, it isn t. Without a packing list, it s easy to forget to do the things which will make your trip better. Use it every time, and bring a copy with you (or store it online) so you can add the things you wish you had brought or done. A simple, ever-improving packing list is the most effective technique I have found for making travel less stressful and more enjoyable.
2. Carry a water bottle with a tight-fitting lid. I use a 32oz Nalgene bottle, which fits nicely into the seat next to me or under an armrest, and gives me enough water for even the longest flights. I fill it up after passing through security, at a cafe, bar or lounge, and generally decline the beverages offered by the cabin crew. Staying hydrated helps me feel better during the flight, and leaves me with less malaise when I arrive. I don t need to manage a tray table or armrest full of cups and other debris, so I can sit more comfortably, with the tray table folded away.
3. Consolidate essential items using multipurpose equipment. For example, invest in a power adapter which has USB sockets onboard, and carry USB cables instead of wall chargers. Versatile items like this save on space and weight. I can charge two devices this way, but the equipment is smaller and lighter than even a single wall charger.
4. Learn how to sleep on an airplane. Getting some sleep on a long flight really helps to offset the effects of traveling. There are several resources out there with practical advice on how to do it. One thing which really helped me was to buy a high quality eye mask which blocks out all of the light in the cabin. The one I use looks a little funny and is not cheap, but is very comfortable and effective. It s made of memory foam with a soft, washable cover and works much better than the ones the airlines give away for free. I no longer bother with a neck pillow, and use the flaps built into the seat to lean my head against. I m surprised at how many people don t know about this common aricraft feature: virtually every long-haul seat has something like this, even in economy, though it may not be obvious how to use it.
5. Buy duplicates of things like toiletries, and keep them in your travel kit so you don t need to pack your everyday items (and risk forgetting them) each time. The less packing you need to do, the less time it will take, and the less opportunity there is for mistakes. This also saves time unpacking when you get home, and lets you buy a smaller size of the item where available.
6. Optimize border crossings Carry the forms you ll need for customs, immigration, etc. in your carry-on. They don t always provide them at the counter or on board the plane, and it s a hassle to rush to fill in the form at the last minute. If you have a few of them with you, you can fill them out early (perhaps even before you fly) and then hustle to the front of the queue. For countries you enter frequently (especially your home country), programs like Global Entry (US) and IRIS (UK) will save you a lot of time by allowing you to use an automated kiosk to cross the border.

A month or so ago, I published a paper with Leah Buechley that is mostly an analysis of how the LilyPad Arduino has been used. I read an earlier draft last year and loved it so, when the opportunity arose, I was honored to help out as the paper evolved. LilyPad is a microcontroller platform that Leah created a few years back and that is specifically designed to be more useful than other microcontroller platforms (like normal Arduino) in the context of crafting practices like textiles or painting. Leah's design goal with LilyPad was to create a sewable microcontroller that could be useful for making things that were qualitatively different from what most people made with microcontrollers and that, she hoped, would be of interest to women and girls. Our paper tries to measure the breadth of LilyPad's appeal and the degree to which it accomplished her goals. We used sales data from SparkFun (the largest retail source for both Arduino and LilyPad in the US) and a crowd-sourced dataset of high-visibility microcontroller projects. Our goal was to get a better sense of who it is that is using the two platforms and how these groups and their projects differ. We found evidence to support the suggestion that LilyPad is disproportionally appealing to women, as compared to Arduino (we estimated that about 9% of Arduino purchasers were female while 35% of LilyPad purchasers were). We found evidence that suggests that a very large proportion of people making high-visibility projects using LilyPad are female as compared to Arduino (65% for LilyPad, versus 2% for Arduino). Digging deeper, qualitative evidence suggests a reason. LilyPad users aren't just different. The projects they are making are different too. Although LilyPad and Arduino are the same chips and the same code, we suggest that LilyPad's design, and the way the platform is framed, leads to different types of projects that appeal to different types of people. For example, Arduino seems likely to find its way into an interaction design project or a fighting robot. LilyPad seems more likely to find its way into a smart and responsive textile. Very often, different types of people want to make these projects. Leah and I believe that there's a more general lesson to be learned about designing technologies for communities underrepresented in science, technology, engineering, and mathematics (STEM) -- and for women in particular. The dominant metaphor in the discussion on women in computer science is Margolis and Fisher's idea of "unlocking the clubhouse." The phrase provides a good description of the path that most projects aimed at broadening participation of women in computing projects seem to take. The metaphor is based around the idea that computing culture is a boys' club that is unfriendly to women. The solution is finding ways to make this club more accessible to those locked outside. It should go without saying that we share Margolis and Fisher's goal of increasing participation of women in STEM. That's LilyPad's point, after all. It it hopefully also clear that we're supportive of, and involved in, projects working to remove systematic barriers to participation by women and other groups. That work must continue. But I also think that Leah's work with LilyPad suggests another way forward based on addressing issues of self-selection that will affect even the most welcoming technological communities. Here's what we say in our paper:

Our experience suggests a different approach, one we call Building New Clubhouses. Instead of trying to fit people into existing engineering cultures, it may be more constructive to try to spark and support new cultures, to build new clubhouses. Our experiences have led us to believe that the problem is not so much that communities are prejudiced or exclusive but that they're limited in breadth--both intellectually and culturally. Some of the most revealing research in diversity in STEM found that women and other minorities don't join STEM communities not because they are intimidated or unqualified but rather because they're simply uninterested in these disciplines. One of our current research goals is thus to question traditional disciplinary boundaries and to expand disciplines to make room for more diverse interests and passions. To show, for example, that it is possible to build complex, innovative, technological artifacts that are colorful, soft, and beautiful. We want to provide alternative pathways to the rich intellectual possibilities of computation and engineering. We hope that our research shows that disciplines can grow both technically and culturally when we re-envision and re-contextualize them. When we build new clubhouses, new, surprising, and valuable things happen. As our findings on shared LilyPad projects seem to support, a new female-dominated electrical engineering/computer science community may emerge.

I have a strong belief that computing can be an empowering tool and that expanding users' control over technology is a critically important issue. Our paper argues that we should attempt to expand participation in computing by broadening the possibilities of computing, rather than only by broadening participation in extant, computing organizations, projects, and genres. Even if computing and electrical engineering communities were perfectly welcoming (which they are not) most people (both male and female, but disproportionately female) will choose not to participate. Building new clubhouses requires creativity of its proponents and risks charges of reinforcing stereotypes and existing status hierarchies. But, executed carefully and well (as I believe LilyPad has been), it suggests ways to reach the majority of people that no "unlocking" project will ever seem relevant to.

I today deactivated my Facebook account. It was some effort, because I had developed some Facebook applications, and their page to reassign them to a different developer was broken (actually the whole application edit page was broken).I have always been a "responsible" user, keeping good control over my data, and usually just posting stuff that I'd tell anyone anyway. Facebook has also always been some kind of "marketing" tool for me, being able to support some friends by inviting people to go to their events etc.However, there are plenty of reasons to abstain from Facebook:

• They track you on millions of web pages via the like button. See earlier posts on how to block some of this tracking.
• They collect tons of data on you even when you do not join, by collecting data on you via your friends email accounts, phone books etc. For example it has been shown that, when one of your friends uses Facebook for iPhone, then Facebook already has collected your phone number.
• Facebook is flooding you with information that is entertaining at first, but actually quite useless. (e.g. who became 'friends' with whom). FBpurity is a good way to remove much of the trivially useless information.
• Facebook is a huge time sink if you are not responsible, e.g. refrain from using all the games, use FBpurity to filter our cruft etc.
• Facebook is giving you the impression of having lots of friends and of being social - while in fact, you are probably even spending less time with your actual friends than ever before.
• Facebook is a great tool - for mobbing. You can easily tell things to everybody except a certain person. Invite them all to a party, but hide it from the target until its over, then brag on how great the party was. And so on - and in fact, that is what happens to non-Facebook users already; this is the kind of social pressure that made Facebook so big.
• Facebook is a lot about bragging rights. Having the most friends, the coolest photos, the best links, the prettiest girls posting on your wall.

Facebook makes things easier, but also a lot more shallow. When someone took note and remembered your birthday, there was some effort involved that shows that they cared. With Facebook reminding people to greet, what is a birthday greeting on Facebook more than an automated message? It's about as personal as the annual christmas card by some large company. Face it: the "social" part of the "social web" is pretty shallow.As you can see above, I've been caring about these issues for some time; I post selectively, I filter much of the tracking, I don't use their games. With my Hiatus, I want to further improve, in particular

• use my spare time for useful projects instead of reading trivialities posted on Facebook. There is just so much tiny stuff that you could do with your time instead.
• spend more time with my true friends
• be more self-aware of my time use in general
• remind people that they know people outside of Facebook, too