This is the second post about distcheck. I want to give a quick overview of the differences between edos-distcheck and the new version. First despite using the same sat solver and encoding of the problem, Distcheck has been re-written from scratch. Dose2 has several architectural problems and not very well documented. Adding new features had become too difficult and error-prone, so this was a natural choice (at least for me). Hopefully Dose3 will survive the Mancoosi project and provide a base for dependency reasoning. The framework is well documented and the architecture pretty modular. It's is written in ocaml, so sadly, I don't expect many people to join the development team, but we'll be very open to it. These are the main differences with edos-debcheck .

Performances distcheck is about two times faster than edos-debcheck (from dose2), but it is a "bit" slower then debcheck (say the original debcheck), that is the tool wrote by Jerome Vouillon and that was then superseded in debian by edos-debcheck. The original debcheck was a all-in-one tool that did the parsing, encoding and solving without converting the problem to any intermediate format. distcheck trades a bit of speed for generality. Since it is based on Cudf, it can handle different formats and can be easily adapted in a range of situation just by changing the encoding of the original problem to cudf. Below there are a couple of test I've performed on my machine (debian unstable). The numbers speak alone.

$time cat tmp/squeeze.packages edos-debcheck -failures > /dev/null
Completing conflicts... * 100.0%
Conflicts and dependencies... * 100.0%
Solving * 100.0%

real 0m19.515s
user 0m19.193s
sys 0m0.276s

$time ./distcheck.native -f deb://tmp/squeeze.packages > /dev/null

real 0m10.859s
user 0m10.669s
sys 0m0.172s

Input The second big difference is about different input format. In fact, at the moment, we have two different tools in debian, one edos-debcheck and the other edos-rpmcheck. Despite using the same underlying library these two tools have different code bases. distcheck basically is a multiplexer that convert different inputs to a common format and then uses it (agnostically) to solve the installation problem. It can be called in different ways (via symlinks) to behave similarly to its predecessors. At the moment we are able to handle 5 different formats

1. deb:// Packages 822 format for debian based distributions
2. hdlist:// a binary format used by rpm based distribution
3. synth:// a simplified format to describe rpm based package repositories
4. eclipse:// a 822 based format that encoded OSGi plugings metadata
5. cudf:// the native cudf format

distcheck handles gz and bz2 compressed file transparently . However if you care about performances, you should decompress your input file first and the parse it with distcheck and it often takes more time to decompress the file on the fly that run the installability test itself. There is also an experimental database backend that is not compiled by default at them moment.

Output Regarding the output, I've already explained the main differences in an old post. As a quick reminder, the old edos-debcheck had two output options. The first is a human readable - unstructured output - that was a handy source of information when running the tool interactively. The second was a xml based format (without a dtd or a schema, I believe) that was used for batch processing. distcheck has only one output type in the YAML format that aims to be human and machine readable. Hopefully this will cater for both needs. Moreover, just recently I've added the output of distcheck a summary of who is breaking what. The output of edos-debcheck was basically a map of packages to the reasons of the breakage. In addition to this information distcheck gives also a maps between reason (a missing dependency or a conflict) to the list of packages that are broken by this problem.This additional info is off by default, but I think it can be nice to know what is the missing dependency that is responsible for the majority of problems in a distribution... For example, calling distcheck with --summary :

$./distcheck.native --summary deb://tests/sid.packages
backgroud-packages: 29589
foreground-packages: 29589
broken-packages: 143
missing-packages: 138
conflict-packages: 5
unique-missing-packages: 52
unique-conflict-packages: 5
summary:
-
missing:
missingdep: libevas-svn-05-engines-x (>= 0.9.9.063)
packages:
-
package: enna-dbg
version: 0.4.0-4
architecture: amd64
source: enna (= 0.4.0-4)
-
package: enna
version: 0.4.0-4
architecture: amd64
source: enna (= 0.4.0-4)
-
missing:
missingdep: libopenscenegraph56 (>= 2.8.1)
packages:
-
package: libosgal1
version: 0.6.1-2+b3
architecture: amd64
source: osgal (= 0.6.1-2)
-
package: libosgal-dev
version: 0.6.1-2+b3
architecture: amd64
source: osgal (= 0.6.1-2)

Below I give a small example of the edos-debcheck output compared to the new yaml based output.

$cat tests/sid.packages edos-debcheck -failures -explain
Completing conflicts... * 100.0%
Conflicts and dependencies... * 100.0%
Solving * 100.0%
zope-zms (= 1:2.11.1-03-1): FAILED
zope-zms (= 1:2.11.1-03-1) depends on missing:
- zope2.10
- zope2.9
zope-tinytableplus (= 0.9-19): FAILED
zope-tinytableplus (= 0.9-19) depends on missing:
- zope2.11
- zope2.10
- zope2.9
...

And an extract from the distcheck output (the order is different. I cut and pasted parts of the output here...)

$./distcheck.native -f -e deb://tests/sid.packages
report:
-
package: zope-zms
version: 1:2.11.1-03-1
architecture: all
source: zope-zms (= 1:2.11.1-03-1)
status: broken
reasons:
-
missing:
pkg:
package: zope-zms
version: 1:2.11.1-03-1
architecture: all
missingdep: zope2.9 zope2.10
-
package: zope-tinytableplus
version: 0.9-19
architecture: all
source: zope-tinytableplus (= 0.9-19)
status: broken
reasons:
-
missing:
pkg:
package: zope-tinytableplus
version: 0.9-19
architecture: all
missingdep: zope2.9 zope2.10 zope2.11
...

Future The roadmap to release version 1.0 of distcheck is as follows:

1. add background and foreground package selection. This feature will allow the use to specify a larger universe (background packages), but check only a subset of this universe (foreground packages). This should allow users to select packages using grep-dctrl and then pipe them to discheck . At the moment we can select individual packages on the command line or we can use expression like bash (<= 2.7) to check all version of bash in the universe with version greater than 2.7.
2. code cleanup and a bit of refactoring between distcheck and buildcheck (that is a frontend for distcheck that allow us to report broken build dependencies)
3. consider essential packages while performing the installation test. Here there are few things we have to understand, but the idea would be to detect possible problems related the implicit presence of essential packages in the distribution. At the moment, distcheck performs the installation test in the empty universe, while ideally, the universe should contain all essential packages.
4. finish the documentation. The effort in underway and we hope to finalize shortly to release the debian package in experimental.

Tuesday, MIT Kerberos had a bad hair day one of those days where you re looking through your hair and realize that it s turned to Medusa s snakes while you weren t looking. Apparently, since the introduction of RC4, MIT Kerberos has had significant problems handling checksums. Recall that when Kerberos talks about checksums it s conflating two things: unkeyed checksums like SHA-1 and message authentication codes like HMAC-SHA1 used with an AES key derivation. The protocol doesn t have a well defined concept of an unkeyed checksum, although it does have the concept of checksums like CRC32 that ignore their keys and can be modified by an attacker. One way of looking at it is that checksums were over-abstracted and generalized. Around the time that 3DES was introduced, there was a belief that we d have a generalized mechanism for introducing new crypto systems. By the time RFC 3961 actually got written, we d realized that we could not abstract things quite as far as we d done for 3DES. The code however was written as part of adding 3DES support. There are two major classes of problem. The first is that the 3DES (and I believe AES) checksums don t actually depend on the crypto system: they re just HMACs. They do end up needing to perform encryption operations as part of key derivation. However the code permitted these checksums to be used with any key, not just the kind of key that was intended. In a nice abstract way, the operations of the crypto system associated with the key were used rather than those of the crypto system loosely associated with the checksum. I guess that s good: feeding a 128-bit key into 3DES might kind of confuse 3DES which expects a 168-bit key. On the other hand, RC4 has a block size of 1 because it is a stream cipher. For various reasons, that means that regardless of what RC4 key you start with, if you use the 3DES checksum with that key, there are only 256 possible outpus for the HMAC. Sadly, that s not a lot of work for the attacker. To make matters worse, one of the common interfaces for choosing the right checksum to use was to enumerate through the set of available checksums and pick the first one that would accept the kind of key in question. Unfortunately, 3DES came before RC4 and there are some cases where the wrong checksum would be used. Another serious set of problems stems from the handling of unkeyed checksums. It s important to check and make sure that a received checksum is keyed if you are in a context where an attacker could have modified it. Using an md5 outside of encrypted text to integrity protect a message doesn t make sense. Some of the code was not good about checking this. What worries me most about this set of issues is how many new vulnerabilities were introduced recently. The set of things you can do with 1.6 based on these errors was significant, but not nearly as impressive as 1.7. A whole new set of attacks were added for the 1.8 release. In my mind, the most serious attack was added for the 1.7 release. A remote attacker can send an integrity-protected GSS-API token using an unkeyed checksum. Since there s no key the attacker doesn t need to worry about not knowing it. However the checksum verifies, and the code is happy to go forward. I think we need to take a close look at how we got here and what went wrong. The fact that multiple future releases made the problem worse made it clear that we produced a set of APIs where doing the worng thing is easier than doing the right thing. It seems like there is something important to fix here about our existing APIs and documentation. It might be possible to add tests or things to look for when adding new crypto systems. However I also think there is an important lesson to take away at a design level. Right now I don t know what the answers our, but I encourage the community to think closely about this issue. I m speaking about MIT Kerberos because I m familiar with the details there. However it s my understanding that the entire Kerberos community has been thinking about checksums lately, and MIT is not the only implementation with improvements to make here.

LWN has an article by Valerie Aurora titled The dark side of open source conferences [1] which is about sexual harassment and sexual assault at Free Software conferences. Apparently some conferences create such a bad environment that some people won t attend, it s a well researched article that everyone in the community should read.The Autism DerailmentThe comments have the usual mix of insight, foolishness, and derailment that you expect from such discussions. One derailment thread that annoyed me is the discussion about men on the Autism Spectrum started by Joe Buck [2]. Joe seems to believe that the 1% of males on the Autism Spectrum (and something greater than 1% but a lot less than 50% in the Free Software community) are a serious part of the problem because they supposedly hit on women who aren t interested in them in spite of the fact that the article in question is about women who are being insulted, harassed, and groped at at open source conferences . The article had no mention of men who try to chat up women presumably this was a deliberate decision to focus on sexual assault and harassment rather than what Joe wanted to talk about.In response Mackenzie made the following insightful point:I don t think any autistic person who is high-functioning enough to A) contribute to open source B) want to be at an event with so many people and C) carry on any sort of conversation is low-functioning enough not to understand stop or no. If you can understand your patch has been rejected, you can likely understand don t do that again. Understanding how Other People FeelBruce Perens claimed What they [Aspies] don t understand is how the other person in the situation feels . Like many (possibly most) people Bruce doesn t seem to get the fact that no-one can really understand how other people feel. The best logical analysis of this seems to be the Changing Emotions article on Less Wrong [3]. While Less Wrong deals with Male to Female conversion as the example (which may be relevant to the discussion about the treatment of women) the same logic also applies to smaller changes. Anyone who even thinks that if they would always be able understand how their identical twin felt (if they had one) probably hasn t considered these issues much. As an aside, having a psychologist diagnose you as being on the Autism Spectrum and therefore by implication thinking differently to 99% of the population really makes you consider the ways in which other people might have different thought processes and experiences.Every time we have a discussion about issues related to sexism in the Free Software community we get a lot of documented evidence that there are many people who are apparently neuro-typical (IE not Autistic) who don t understand how other people think in many cases they go so far as to tell other people what their emotional state should be.What Really HappensNix said However, in that situation our natural reflex is to *get out of there*, not to jump on women like some sort of slobbering caveman which is a really good summary.In more detail, I think that the vast majority of guys who are on the Autism Spectrum and who are able to do things like attend computer conferences (*) realise that chatting up a random girl that they meet is something that just isn t going to work out. Generally people don t attempt things that they expect to fail so I don t think that Autistic guys are going to be hitting on girls at conferences.(*) Having never met any Autistic people who aren t capable of attending such conferences I can t speak for them. I really doubt that the Low Functioning Autistic guys are as much of a problem as some people claim, but lack evidence. In any case the actions of people who don t attend conferences aren t relevant to a discussion about things that happen at conferences.Update: It Keeps GoingDion claims that the misogyny at conferences is due to socially inept people, he also casually switches between discussing people who misunderstand when someone is flirting and people who hire almost-naked booth-babes (two very different classes of action) [4]. Several people asked for supporting evidence, naturally none was provided.In response njs posted a link to Marissa Lingen s blog post Don t blame autism, dammit [5]. Marissa points out that people who offend other people due to lacking social skills will tend to do so in times and places that are likely to get a bad reaction if you don t know that you are doing something wrong then there s no reason to hide it. If someone offends a senior manager at a corporate event then it could be because they are on the Autism Spectrum (I ve apparently done that). If someone offends junior people at a times and places where there are no witnesses but is always nice to managers and other powerful people then it s not related to Autism.One final note, I have little tolerance for anyone who claims to be an Aspie when they do something wrong. You are either on the Autism Spectrum all the time or none of it. Anyone who wants any sympathy for me for an occasion where they stuffed up due to being an Aspie can start by making a clear statement about where they are on the Autism Spectrum.Update2: Yet More from Bruce PerensBruce wrote IMO, the kind of men who go in to software engineering suffer a lack of healthy interaction with women who are their peers, and it may be that the high incidence of empathy disorders in our field is involved (which seems to be part of the inspiration for Joe Buck later in that thread) and now claims Nobody here was trying to connect Asperger s or autism with the touching incidents or violent crime .Matthew Garrett responded to that with If you weren t trying to say that the high incidence of empathy disorders in our field was related to a lack of healthy interaction with women who are their peers, and that that has something to do with incidents of sexual harassment or assault at conferences, what were you trying to say? Because that sounds awfully like We wouldn t have so many problems if it weren t for all the autists .Bruce s latest comment is If you choose to read something that nasty into my writing, that s your problem. Get therapy .Through this discussion I ve been unsure of whether to interpret the statements by Bruce et al the way Matthew does or whether I should consider them as merely a desperate attempt to derail the discussion. I can t imagine any possible way of interpreting such comments in connection with the discussion of sexual assault as anything other than either trivialising violent crimes against women (suggesting that they are no worse than asking out someone who s not interested) or claiming that anyone who lacks social skills should be treated as a violent sexual predator. It s just not reasonable to believe that every single person who wrote such comments referring to Autism was misunderstood and really meant something nice.As a general rule I don t think that it s the responsibility of other people to try and find a non-offensive interpretation of something that one might say. I don t think that all the people who strongly disagree with the most obvious and reasonable interpretations of Bruce s comments should get therapy. I think that Bruce should explain what he means clearly.

• [1] http://lwn.net/Articles/417952/
• [2] http://lwn.net/Articles/418209/
• [3] http://lesswrong.com/lw/xe/changing_emotions/
• [4] http://lwn.net/Articles/418281/
• [5] http://mrissa.livejournal.com/746160.html

Since (already) several weeks, bts-link has a new home: busoni.debian.org! That's one of the machines where BTS is running, and it helps to let bts-link complete its job in about 30 mins each time. Thanks goes to Don Armstrong and Peter Palfrader for their support in the migration (from merkel).

Ok, so micro rant time: this is the effect of not taking things upstream: hardware doesn t work Out Of The Box. Very briefly, I purchased a Vodafone prepaid mobile broadband package today, which comes with a modem and SIM. The modem is a K3571-Z, and Ubuntu *thinks* it knows how they work (it doesn t). So it fails to connect in NetworkManager with a rather opaque NO CARRIER message. Thanks to excellent assistance from Matt Trudel, we tracked this down to a theory that perhaps modemmanager is using the wrong serial port and voila, it is. From there, the config file (/lib/udev/rules.d/77-mm-zte-port-types.rules) was an obvious next step and indeed there is no entry in there for the 19d2:1010 the K3571-Z. Google found one immediately though, on a Vodafone research site. The awful shame is this: that was committed to the bcm project in March this year. If Vodafone had shipped off a patch to modemmanager, we could have had that in 10.10, and possibly even in 10.04. There are plenty of users having trouble on Whirlpool etc with this model who would have had a better experience helping Vodafone s users be happier. All it would have taken is an email I m sure Vodafone want a great experience for their users, but I think they re failing to separate out platform improvements share and share alike, and branding / custom facilities. The net impact is harmful, not helpful. Anyhow, Natty will support this modem.

A few weeks ago I noticed an image on my Froyo phone which appeared to be a screenshot. This came as a surprise, since I don t have any screenshot applications installed and am generally economic when it comes to the amount of fluff on my phone. I did a few searches and there doesn t seem to be any visible mention of this feature out there, only mentions of how to do it with the SDK or with specific apps. Since today I finally managed to discover how to actually do it again (holding the Back button and pressing Home), I took these screenshots using different launchers to be sure it wasn t some feature of ADW.Laucher. I still haven t beat my laziness to actually reset to factory settings and make sure it s not some other random app (which would be a bit unsettling, actually) and it could also be a Samsung specific feature, implemented as one of those pesky unremovable apps. Can anybody out there confirm this (not particularly useful) feature? Did I miss it being published somewhere? [UPDATE]: I found a couple of hidden apps that probably have something to do with this (through Settings Applications Manage applications All). The first one is an obvious suspect, but since it isn t named after Samsung (which admittedly doesn t mean much) I figured it might be activated by the second one, which in my overly-creative and not-really-investigative mind could be a sort of hotkey-daemon. [UPDATE 2: well, that's ignoring the obvious fact that SEC stands for "Samsung Electronics Corporation". D'oh!]
I don t think the feature is important or interesting enough for me to investigate much further, but still I thought I might leave some pointers if people want to get this on their ROMs.

We ve come a long way since the times when you needed to configure 2 X servers in XDM just to be able to use 2 X sessions at once. However there was still some way to go until recently. A number of bugs that could be wrongly attributed to bugs in the X server or in the desktop environment were actually caused by the display manager doing crap. GDM up to 2.20 Since the introduction of the flexible X servers feature, GDM hadn t evolved much on the matter of user switching. What it used to do was pretty straightforward:

• a specific protocol can be invoked by the gdmflexiserver command;
• the gdm daemon spawns a new X server on an empty console;
• it initiates another login process in it;
• when the session exits, or if the user clicks on Quit instead of logging in, the X server exits.

It is interesting to note that VT (console) switching is purely handled by the X server. When starting, the new server switches the current VT to where it is. When exiting, it automatically switches back to the VT from which it was launched. While very simple, this idea fails to work correctly every time you try to do something more complicated than starting a temporary session for a guest and exiting it. For example, if you start two of them, there is a chance that, when the X server switches back to the console it was run from, there is nothing left running in this console, leaving you with the funny Control-Alt-Fn shortcuts to find your way back to a X server. You will also meet interesting race conditions when trying to switch back to an existing session from the login window. GDM 2.28 and above In the process of rewriting the code entirely, the GDM developers tried to address a number of those shortcomings, making use of D-Bus and ConsoleKit. The new design is slightly more complicated, however.

• The gdmflexiserver tool will first try to look for an existing login window in another X server, and just switch to the VT it is in if it finds one.
• Otherwise, the daemon starts a new slave process with a new X server and a new login window, in a very similar way to what older versions did.
• When logging in as a user with an existing session, it switches to the VT it is in, but leaves the login window and its X server running.
• When going into a new session, the X server is simply left to die at the end of the session, and to switch back to the VT from which it was launched.

Not killing the X server in some cases partly addresses the problems caused by letting it switch back to the original VT when exiting. However in several ways the cure is worse than the disease.

• First of all, it will leave unused X servers, with all processes used by the login window - and that makes quite a number of them, with GDM now using a minimal GNOME session.
• When there is such a login window remaining, ConsoleKit will refuse to let you shut down your computer, being lured into thinking there is someone else using it.
• It doesn t solve the inconsistency issue. When you leave a session, you can find either of: a login window, a screensaver unlock dialog, or a black screen.

Getting it to work The modular architecture of GDM makes it possible to improve the situation. (Possible but not easy because of the millefeuille of classes.) However, it is merely a band-aid unless you fix the root issue: the X server knowing better than you which VT it should switch to when exiting. Fortunately Xorg now features an option to avoid that behavior: -novtswitch. So the first step is to enable it, and let the GDM daemon (or slave) handle VT switching through ConsoleKit. With that, the following changes are possible.

• When switching to an existing session, don t leave a X server behind. You can now kill it safely without risking a VT switch.
• On the opposite, when exiting a session, always respawn a login window on the same VT.
• The last step is to stop making a difference between the first launched X server (called a static server) and the flexible servers. The only remaining difference between a static display and a flexible one is that the static one honors automatic login/timed login settings.

The result With all these changes the behavior of the display manager is finally completely consistent.

• When exiting a session, regardless of it, you always find the same login window.
• There is never an unused process left.
• You will never find yourself facing a black screen with only keyboard shortcuts to leave you out of it.

Interestingly enough, this is very similar to what user switching looks like on Vista or MacOS X. So what now? These changes are stabilized for Debian squeeze, but of course it has been long overdue to get them accepted upstream, along with the very large number of Debian-specific changes that still lie in our packages.

Earlier in the year, Romain and I did a bunch of initial work on a wrapper from R to the GNU GSL by way of our Rcpp package for seamless R and C++ integration. But other work kept us busy and this fell a little to the side. We have now found some time to finish this work for a first release, together with a nicely detailed eleven page package vignette. As of today, the package is now a CRAN package, and Romain already posted a nice announcement on his blog and on the rcpp-devel list. So what does RcppGSL do? I gave the package its own webpage here as well and listed these points as key features of RcppGSL:

• templated vector and matrix classes: these are similar to Rcpp's own vector and matrix classes, but really are just smart pointers around the C structure expected by the library
• this means you can transfer data from R to your GSL-using programs in pretty much the same way you would in other C++ programs using Rcpp---by relying on the Rcpp::as() and Rcpp::wrap() converterrs
• at the C++ level you can use these GSL vectors in a more C++-alike way (using eg foo[i] to access an element at index i)
• yet at the same time you can just pass these vector and matrix objects to the GSL functions expecting its C objects: thanks to some cleverness in these classes they pass the right object on (see the example below)
• we also provide the lightweight views for vectors and matrices as the GSL API uses these in many places.

Also provided is a simple example which is a simple implementation of a column norm (which we could easily compute directly in R, but we are simply re-using an example from Section 8.4.14 of the GSL manual):

#include <RcppGSL.h>
#include <gsl/gsl_matrix.h>
#include <gsl/gsl_blas.h>
extern "C" SEXP colNorm(SEXP sM)  
  try  
        RcppGSL::matrix<double> M = sM;     // create gsl data structures from SEXP
        int k = M.ncol();
        Rcpp::NumericVector n(k);           // to store results
        for (int j = 0; j < k; j++)  
            RcppGSL::vector_view<double> colview = gsl_matrix_column (M, j);
            n[j] = gsl_blas_dnrm2(colview);
         
        M.free() ;
        return n;                           // return vector
    catch( std::exception &ex )  
        forward_exception_to_r( ex );
    catch(...)  
        ::Rf_error( "c++ exception (unknown reason)" );
   
  return R_NilValue; // -Wall
 

This example function is implemented in an example package contained in the RcppGSL package itself -- so that users have a complete stanza to use in their packages. This will then build a user package on Linux, OS X and Windows provided the GSL is installed (and on Windows you have to do all the extra steps of defining an environment variable pointing to and of course install Rtools to build in the first place---Linux and OS X are so much easier for development). Another complete example is in the package itself and provides a faster (compiled) alternative to the standard lm() function in R; this example is the continuation of the same example I had in several versions of my Intro to HPC with R tutorials and in the Rcpp package itself as an early example. We will try to touch base with CRAN package authors using both GSL and Rcpp to see how this can help them. The API in our package may well be incomplete, but we are always happy to try to respond to requests for additional features brought to our attention, preferably via the rcpp-devel list. More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

The new RQuantLib release 0.3.5 is now on CRAN and in Debian. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. Most of the changes were made two and four weeks ago: first in response to some warnings triggered by R 2.12.0 on the included manual pages which needed a brush-up, and then again is some consolidation of manual pages and some other minor tweaks. The release was then held back at CRAN as we noticed that manual pages, when collated to a single large document, triggered a segmentation fault in the latex compiler. Oddly enough only in Europe (if the a4paper option was used) and not here (where I use uspaper). Long story short, this turns out to be a bug in the latex toolchain (which we reported as Debian bug report 604754) which is apparently is known but has no known fix yet (a sample file was supplied with the bug report if you want to take a look). With that, special thanks go to Kurt Hornik and Brian Ripley on the R Core team who made a change to how R processes the manual which made it resilient to the latex bug so that normal release of the package could proceed (and the shiny manual is available too). Thanks to CRANberries, there is also a diff to the previous release 0.3.4. Full changelog details, examples and more details about this package are at my RQuantLib page.

Debugging a computer program is such an interesting activity because it's not really a matter of fixing a program. It's a matter of fixing your own understanding to the point that the cause of the bug becomes obvious. So debugging means constantly challenging your assumptions, constantly looking for the overlooked insignificant thing that turns out to be crucial. I mostly debug by print statements. I don't like it much, but it works. I add some print statements around the problem area and see if what's going on matches what the model in my head thinks should be going on. If it doesn't, I try to fix the model, and if it does, I focus the print statements in tighter and tighter, deeper and deeper, until I finally reach the bug. It's nasty and ah-hoc, but it works really well, mostly. And I know I'm in good company doing it. But that method can fail; I myself can end up caught in a loop. Like today, tracking down an obscure bug in Branchable's auditing subsystem, when I found myself at what was apparantly the point of a bug: Function A was supposed to call function B, but B never seemed to be called. So surely A was buggy, and I instrumented it to bits with print statements and stuff, and started wondering if the libraries it relied on were somehow breaking it in non-obvious ways before it could call B, or maybe there was a bug in the runtime language that was breaking the call to B. Going deeper and deeper didn't help, because the thing I was overlooking was up at the top level -- one of the first print statements I inserted showed that all this code was running twice. Why? Didn't seem relevant to the bug so I ignored it. Of course what I was ignoring was a symptom of the real problem: The first run was by another subsystem, that happened to redefine function B temporarily.. Actually, one of the nice things about Haskell is that print statements are not very helpful in debugging pure code. Pure code tends to get loaded up in ghci and tested interactively, and the bug is then clear. Or a quickcheck test case gets written with an invariant that the pure code should satisfy. I spent a whole day a few weeks ago writing an inverse form of a buggy function in git-annex just so I could feed them both into quickcheck and automate debugging. (Most of that time was spent wrestling with unicode decomposition .. ugh.) Being used to getting down in the buggy code with print statements, that at the time felt rather like a waste of time unrelated to getting on with making my program work, but it was very much worth it since I got my program working 100% in even crazy edge cases, and got a test suite for free, too. Most of my debugging of Haskell code so far has not involved bugs in pure code. The most memorable bug was actually a bug in ghc's IO manager, which I'd have never tracked down without Josh's help. The problem with finding bugs in the runtime environment or compiler is that after you've found enough over the years, you're more likely to go down false paths distrusting runtime environments, like I did with the bug today. The beginner's assumption is that the runtime environment just works, but I by now may have over-challenged that assumption. :)

At the end of September, things were quite exciting as we had our first project meeting. At that meeting those in the room saw a demonstration of the Moonshot GSS EAP mechanism and we discussed a number of open issues and began to plan for our test infrastructure. We ve made significant progress on the specification front and on explaining Moonshot to important communities since then. However there has been little public progress on the implementation front. Unfortunately, getting the necessary legal clearance and agreements to release code often takes longer than anyone would like; that is what is happening here. We re all eagerly awaiting final approval from the lawyers and JANET(UK) management. However, things have been moving behind the scenes. Throughout much of October, Luke Howard and Linus Nordberg were working on their respective parts of the code. I ve also been working on putting together the test and build infrastructure. As we discussed at the meeting, we re going to use Debian and Ubuntu as the basis for our testing. For example, we hope to release virtual machine images for these platforms for the major Moonshot components. Thus the primary build environment for our testing and virtualization will be for Debian. I ve been putting together that here. Right now, that branch will pull together packages of the SAML infrastructure that we need. I ve also been looking into virtualized test frameworks and believe I ve found one that meets our needs. I ve also put together some primitive build infrastructure that is independent of packaging available here. I ve set up a buildbot that builds both environments. So, as the code becomes available we ll be in a good position to start making it available.

Another thing I regard as useful on the commandline are colors. (Others commonly refer to my color preferences as eye cancer, but that s not relevant here. ;-) Colors help to easily distinguish between relevant and non-relevant things or to separate different things. One colored command line program I use nearly everyday is the colored fork of bsdmainutils s well know (or at least better known) cal program whose output looks like this:

$ cal
    November 2010    
Su Mo Tu We Th Fr Sa
    1  2  3  4  5  6
 7  8  9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30
$ 

Since the colored fork calls itself still cal (as the Debian source package is called), to avoid conflicts with bsdmainutils s cal, the binary package and the binary itself are called ccal (short for colored cal) and its output looks like this:

$ ccal
    November 2010     
 Mo Tu We Th Fr Sa Su 
  1  2  3  4  5  6  7 
  8  9 10 11 12 13 14 
 15 16 17 18 19 20 21 
 22 23 24 25 26 27 28 
 29<30>               
                      
$ 

ccal also use the locale definitions (LC_TIME), if available, that s why the one screenshot is Sun-Sat and the other one is Mon-Sun. According to the man page, ccal has some more improvements over the classic cal: If displaying the single-month format, ccal will look for a date file (the default file or whatever you specify with the -d option). If found, ccal will read the file, looking for special date descriptions for that month which will be displayed to the right of the calendar. By default, up to 24 appointments (number may be changed with -m) may be displayed per month. If the current date happens to fall on one of these special dates, it will be flagged by an asterisk. If there is room, appointments for the next month may also be displayed (next month s dates having definitions like 2nd Thursday will be skipped). But I must admit, I don t use that feature. I just use it as on-demand calendar sheet. The colored screenshot above is btw. generated with Adam Borowski s ansi2html which popped up after I published my Intent to package aha (for Ansi HTML Adapter), the first ANSI colors to HTML converter I found on the net. (And yes I searched for it because of this and some yet to come blog postings. No spoilers here though. ;-) While capturing colored output of ls and some other tools was easy, it needed a little hack to capture ccal s output, because if you pipe ccal output to anything, it drops its colors. Always. The solution was to run it in GNU Screen, log the output and then pipe the log file through aha or ansi2html :

$ screen -c /dev/null -L ccal; cat screenlog.0   ansi2html > ccal.html

Probably the output of every program which drops colors when writing to a pipe instead of a terminal can be fetched that way. Update, 2:04 CET: As Adam Borowski points out in message 62 of #605380, script is the better tool for making tools believe they talk to a terminal:

$ script -q /dev/null -c "ccal"   ansi2html

Catching the output of a whole year s calender ( ccal 2011 ) failed with GNU screen though, as it was truncated on the 24th of September by GNU screen (as it was already truncated in the screenlog.0), so no screenshot of that for the moment. And with script, I d get the whole output, but would have to manually fix the styles so they don t get posted literally to Planet Debian. So look for yourself how the output of ccal 2011 looks like by installing ccal. ;-)

Once again, there s been another story about how Google is having trouble retaining talent. Despite all Eric Schmidt s attempts to tell folks that Google s regretted attrition rate has not changed in seven years, this story just doesn t want to seem to die. (And those stories about Google paying $3.5 million and $7 million to keep an engineer from defecting to Facebook? As far as I know, total bull. I bet it s something made up by some Facebook recruiter who needed to explain how she let a live prospect get away. :-) At least for me, the complete opposite is true. There are very few companies where I can do the work that I want to do, and Google is one of them. A startup is totally the wrong place for me. Why? Because if you talk to any venture capitalist, a startup has one and only one reason to exist: to prove that it has a scalable, viable business model. Take diapers.com for example. As Business Week described, while they were proving that they had a business model that worked, they purchased their diapers at the local BJ s and shipped them via Fedex. Another startup, Chegg, proved its business model by using Amazon.com to drop ship text books to their first customers. (The venture capitalist Mark Maples talked about this in a brilliant talk at the Founders Showcase; the Chegg example starts around 20:50 minutes in, but I d recommend listening to the whole thing, since it s such a great talk.) You don t negotiate volume discounts with textbook publishers, or build huge warehouses to hold all of the diapers that you re going to buy until you prove that you have a business model that works. Similarly, you don t work on great technology at a startup. Startups, by and large, aren t about technology at least, not the Web 2.0 startups like Facebook, Foursquare, Twitter, Groupon, etc. They are about business model discovery. So if you are fundamentally a technologist at heart, whose heart sings when you re making a better file system, or fixing a kernel bug, you re not going to be happy at a startup. At least, not if the startup is run competently. If you have the heart of an entrepreneur, and you are willing to roll the dice (since 9 out of 10 startups go belly up; those are the ones that failed to find a viable business model), then sure, go for a startup. And understand that your job will be to make something that works well at a small scale, quick, dirty, and cheap. If that means using some proprietary software, then that s what you should do. Hopefully you ll get lucky and win the IPO lottery. But if your primary interest is to doing great engineering work, then you want go to company that has a proven business model. Google has enough scale that I can sleep well knowing that improvements I am making to Linux and the ext4 file system are saving enough money for the company that Google is making a profit on me. That is, the value Google is getting out of my work is worth many multiples of my total compensation. And that s a position that every engineer should strive for, since that s how you can be confident that you will remain gainfully employed. The fact that I can do what I love, and go to conferences, and all of my work is open source that s just total icing on the cake. :-) No related posts.

The Debian archive is known to be one of the largest software collections available in the free software world. With more than 16,000 source packages and 30,000 binary packages, users sometimes have trouble finding packages that are relevant to them. Debian developer Enrico Zini has been working on infrastructure to solve this problem. During the recent mini-debconf Paris, Enrico gave a talk presenting what he has been working on in the last few years, which hasn t gotten yet the attention it deserves . Enrico is known in the Debian community for the introduction of debtags, a system used to classify all packages using facets. Each facet describes a specific kind of property: type of user-interface, programming language it s written in, type of document manipulated, purpose of the software, etc. His most recent work builds on that. It is available in Debian and Ubuntu in the apt-xapian-index package. Its purpose is to allow advanced queries over the database of available packages. Users of apt-xapian-index He started by presenting some early users of the infrastructure. The most widely know is Ubuntu s software center. Its search feature provides results almost instantly thanks to apt-xapian-index. But it is a very simple interface that doesn t exploit many of the advanced features provided by the apt-xapian-index. Another early adopter, making use of some more advanced features, is GoPlay!. It s a graphical user interface to find games. It makes use of debtags to classify games so that you can browse, for example, all 3D action/arcade games related to cars. GoPlay has even been extended to be a more generic debtags based package browser and the package now also provides GoLearn!, GoAdmin!, GoNet!, GoOffice!, GoSafe!, and GoWeb!. Fuss-launcher is an application launcher and not a package browser, but by using apt-xapian-index, it s able to reuse information provided at the package level to make it easier to find installed applications. Package descriptions tend to be more verbose than those embedded in .desktop files. Enrico also showed another nice feature to the audience: if you drag a document onto its window, it will show you a list of applications that can open it. Last but not least, apt-xapian-index provides a command line search tool that is vastly superior to the traditional apt-cache search: it s axi-cache search (axi stands for apt-xapian-index). Enrico compared the output of a search on the letter r . While apt-cache spits out an infinite list of packages containing this letter somewhere in the description, axi-cache only listed packages related to GNU R. He also demonstrated the contextual tab completion. It makes it easy to use debtags and to refine your search. Once you have typed a first keyword, the tab-completion for the second one only contains keywords or debtags that are actually able to provide more restrictive results. Advanced queries with logical operations (AND, OR, NOT, XOR) are also supported. Features of the backend Enrico then dived into the internals. Xapian s search engine is at the root of this infrastructure. He likes it because it s a simple library (i.e. no daemon) and it has nice Python bindings. While apt-xapian-index s core work is to index the descriptions of all the packages, it actually stores much more and can be easily extended with plugins (written in Python). For instance, the information stored encompasses:

• words appearing in the description of the packages (including the translated descriptions if the user uses a non-English locale);
• their origin;
• their section;
• their size and installed size;
• the time they have been first seen;
• icons, categories, descriptions from the .desktop files they contain (through app-install-data);
• aliases for names of some popular applications that are not available on Linux (for instance excel maps to the debtag office::spreadsheet).

He already has plans to store more: adding popularity contest data (see wishlist bugs #602180 and #602182) will make it possible to sort query results in a useful way. The most widely used applications are good choices when it comes to community support, and they are likely of better quality due to the larger user base. Adding timestamps of the last installation/upgrade/removal, will make it easier to pin-point a regression to a specific package update. The generated index is world-readable and can be used from any application provided it can use the Xapian library which is written in C++ but has bindings for Perl, Python, PHP, Java, Tcl, C#, and Ruby. Call for experimentation Enrico believes that many useful applications have yet to be invented on top of apt-xapian-index s features. He s calling for experimentation and asking for new ideas. The only practical limit that he has encountered is the size of the index: currently it varies between 50 Mb (Debian unstable without translation) and 70 Mb (Debian stable/testing/unstable with one translation). He would like it to not grow over 100 Mb since it s installed by default (due to aptitude recommending it) and he s not comfortable with the idea of using more than 20% of the disk footprint of a basic install just for this service. That s why the index was configured to not store the position of the terms: it s thus not possible to find out packages whose description contains the word statistical immediately followed by the word computing . You can however find those which have both terms somewhere in their description. Enrico wondered if apt-xapian-index offers too much freedom. That could explain why few people experimented with it despite his numerous blog posts with code samples and information on how to get started using it. But it s not difficult to imagine use cases for this data. It could be used to extend tools like rc-alert or wnpp-alert, for example. They provide a long list of Debian packages that are looking for some help and are installed on the machine. With apt-xapian-index, it would be possible to restrict the results to the set of packages written in a specific programming language or for a particular desktop environment. The more likely explanation is that too few people know about the tool. There are many more itches to scratch where apt-xapian-index s features could be very useful, and my guess is that Enrico s wishes will eventually come true.

This article was first published in Linux Weekly News. Click here to subscribe to my newsletter and keep up with the Debian/Ubuntu news thanks to my monthly digest.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Sometimes it s nice being wrong. Contrary to what I predicted, ZFS will be supported in Debian Squeeze using the official installer. This means that Debian Squeeze will be one of the first GNU distributions to support ZFS. In fact, even though ZFS support didn t make it to Debian-Installer beta1 by the time it was released, it is now available in the netboot images (this happens because netboot images fetch newer installer components from the internet). As a consequence of this my unofficial installer can now be considered obsolete. So why did I say something that turned out to be grossly inaccurate? It s not due to anyone s fault really. At that time, the version of Parted that included ZFS detection hadn t migrated to Squeeze. The unblock policy didn t appear to allow this migration. However, the Release Team kindly decided to make an exception that allowed this, and after Parted had migrated the changes in Debian-Installer itself went in quite smoothly.

A few months ago I realised it has been 10 years already since Spacelabs ¹ was founded. Today we will celebrate that with a reunion and probably reminisce about past projects, events and experiences. Therefore, I thought it would be nice to accompany this with a post. The past Spacelabs was founded to create something that sits in between research and consumer appliances. Since we are associated to the Electro-Optical Communication group these things often had to do with bandwidth. For example, showing the ease of use of plastic optical fiber or participate in GPRS/UMTS trials. However, due to the loose structure of Spacelabs, we ventured into many other projects of our own such a information system-enabled fridge with a barcode reader, a dynamic DNS system, etc. At some point we got our own Internet connection with an accompanying subnet. It has been fun and rewarding to learn how to setup and run a mini ISP. Spacelabs doesn t have any formal structure: when you are on the mailing list, you are a member. There are no regular meetings, no hierarchy, no obligations. So in some ways, we are a group that is much like a free/open source software community. I think that this approach has provided us the drive to work both on projects during our free time as well as our own ideas. I feel that Spacelabs has contributed so much to my life. It might not be the typical study association, but it was a big supplement to my studies, it was fun and it was completely ours. Not only did it teach me many, many technical things from networks to program languages to software design but it also gave me experiences on other levels, such as very attuned group work and responsibilities for key systems. The future Soon, everyone connected to Spacelabs will have finished their Master s or PhD (in my case) and nobody will be left to continue the project. Although this is a bit sad, I think we can be proud of what we have accomplished, learnt and experienced. I will miss our location high in the Potentiaal/E-hoog building overlooking Eindhoven, and our being together as a group. We are however not gone yet, there is still a lot to plan and to work on if we want to rescue the huge amount of servers and services that we have accumulated over the years if we were to leave

¹ Yes, the website is very much from the past.

On the 5th of November Jono Bacon wrote a blog post titled MAKING OUR WORLD MORE RESPECTFUL [1]. He then registered the domain OpenRespect.org for his manifesto [2].Aaron Siego supports the general concept of being more respectful while listing some of the practical problems that occur in a multi-cultural world [3]. If you read nothing else about this issue I recommend reading Aaron s post (which I summarised very poorly, I can t think of a good one-sentence explanation).Sam Varghese wrote an article suggesting that it was an attempt to deflect criticism from Ubuntu [4]. I think that Ubuntu receives a lot of unfair criticism, but don t think it s really relevant to the issue at hand. Sam criticises some of the Open Respect supporters for failing to show respect in the past.Jono wrote a follow-up blog post titled REFLECTIONS ON RESPECT [5], in that post he said Sometimes we were disrespectful, and frankly, sometimes we were also inadvertently assholes. We never set out to be assholes, but we did set out to be edgy in a satirical way, but we sometimes went too far and I apologize for that. But you know what, we all grow and mature in different parts of our lives about his past LugRadio work. Maybe instead of focusing on being respectful of others it would give more benefit to focus on how to apologise, back down, and move on after being disrespectful. It seems that the situations of epic disrespect tend not to be one-off incidents but are instead arguments of greatly expanded scope that arose from someone refusing to admit any error after doing something that is considered to be offensive. [Someone has registered OpenApology.org, just in case you were wondering.]Maybe advocating Rogerian Argument would be another way of reducing some of the problems in the community.Finally Kirrily wrote an insightful post about the ineffectiveness of discussing offensive things [6]. In terms of issues such as Feminism (which Kirrily uses as an example and which is also a matter of significant interest for her) discussions about offense apparently tend to end up being about the person who was offended rather than the person who might be described as acting in an offensive manner. She suggests using terms such as marginalised, belittled, stereotyped, frustrated, humiliated, threatened, patronised, silenced, intimidated, misrepresented, etc instead. It seems to me that there may be a similar issue with respect . Objective claims about people being dismissed, ignored, patronised, etc can be evaluated more easily than a claim about being disrespected.

• [1] http://www.jonobacon.org/2010/11/05/making-our-world-more-respectful/
• [2] http://openrespect.org/
• [3] http://aseigo.blogspot.com/2010/11/commonality-and-community.html
• [4] http://www.itwire.com/opinion-and-analysis/open-sauce/43196-openrespectorg-a-bid-to-deflect-criticism-of-ubuntu
• [5] http://www.jonobacon.org/2010/11/17/reflections-on-respect/
• [6] http://infotrope.net/blog/2009/08/05/action-and-reaction-on-avoiding-offense/

Recently, I was considering the following set of statements: Vector graphics are great, especially in print. Program documentation needs screenshots. Usual screenshots are not bitmaps. Some programs use gtk. Gtk uses Cairo to render most of its display. Cairo can generate PDF. PDF is a vector graphics format.
And while I was doing so, for some reason, I thought: Wouldn t it be great if I my gtk programs could generate vector PDF files of their screen content? And after some experimentation, I was able to extract this PDF file from the buttonbox example program (see here if you do not see anything below): This works without any modification to the original program. Instead, I am loading some code using LD_PRELOAD that overrides the cairo_xlib_surface_create function and creates a PDF surface instead. It also monitors calls to XCopyArea to observe where on the window gtk (gdk actually) paints the resulting image. Then I combine these PDF files (6 in this example, theoretically a lot more) using a latex document and I must say, the result is pretty good for a start: All lines are vector lines, the text really is text (and not just rendered pixels), so you can actually copy n paste from it. Of course this is but a proof of concept and has several problems:

• You can not use the program with my library loaded, as all output is redirected. A solution would be to create a proxy cairo surface that renders both on-screen and to the PDF file, or to make the library react to a signal and only start intercepting the calls when the image has arrived.
• You need a gtk theme with a cairo-using engine such as Murrina, because
  
• any drawing done directly with gdk is lost. This includes icons, but also the clearing of parts of the screen. At the moment, all is just put on top of each other.
• It is not a turn-key tool.

I don t think I ll fix that myself, as I had my fun. But maybe it can inspire a more useful tool exploiting this idea, and maybe there are cleaner approaches (for example special gtk theme engines, or by patching gtk).

Have you ever wondered which tool you should use to set the background of your X11 desktop? There are tons of it and every single one does need different command line options. And if you picked one, you could be sure, you ll once come to a machine or operating system, where that one isn t available. When I started to play around with the tiling window manager awesome, I also noticed that the awesome Debian package contains awesome s own desktop background image setting tool named awsetbg . Yet another one! I thought. But I was wrong: Today I noticed the word wrapper in awsetbg s man page. I became curious and had a peek into its source code. And indeed, it s an awesome desktop background image setting POSIX bourne shell compatible shell-script: It knows about not less than 15 other desktop background image setting tools and can be used as frontend wrapper for those who don t want to remember each tool s command line syntax:

• Eterm s Esetroot
• habak
• feh
• hsetroot
• chbg
• FVWM s fvwm-root
• ImageMagick s display
• qiv
• xv (non-free, .deb packages available from Mowgli)
• xsri
• xli
• xsetbg from xloadimage
• WindowMaker s wmsetbg
• xsetroot from x11-xserver-utils
• IceWM s icewmbg (currently disabled by default with comment broken )

Needless to say that most of them are also available in Debian. :-) You may wonder what s useful with it. Well, it does something you no more need to do yourself or in one of your own scripts: Checking which desktop background image setting tools are available on the machine you re currently working and using the appropriate command line options for tiling, fullscreen, whatever. Only drawback I found so far: In Debian, you need the awesome window manager package to be installed, if you want to get the script via APT. But on the other hand, it s just a shell script. You can easily put it into your $HOME/bin or so. :-)