Receiving a full rejection for DebConf travel sponsorship reminds me of a time I was asked to nominate members for a corporate board of directors. At the time I replied that I would not be of much help because I value things which are irrelevant to such positions. Sure, I can identify worthless, grandstanding blowhards, but of the people who are not, I am unqualified to guess which ones make good board members and which do not. The person who had asked me was somewhat confused by my answer, because we come from different perspectives on how the world works and possibly how the world should work.

Maps and Coastlines in Debian #tags enso,climate,paleooceanography #postdate 2011-06-19 18:00 #ignore yes Thanks to Brian Lawrence for a pointer to an interesting review by George Philander: Where are you from? Why are you here? An African perspective on global warming. in the Annual Review of Earth and Planetary Sciences. Scroxton et al. at Oxford, in the latest edition of Paleooceanography published < href="http://www.agu.org/pubs/crossref/2011/2010PA002097.shtm"> Persistent El Ni o Southern Oscillation variation during the Pliocene Epoch Scroxton et al. study fossil foraminera, plankton remains from the mid-Pliacezian warm period (3.26 - 3.03 million years ago). By studying isotope ratios in the carbonate skeletons of these plankton, they can reconstruct the surface water temperatures in the Eastern Pacific, where the El Nino is today. They discover conditions similar to today, with El Nino-like events.

Well, every once in a while we re forced to do something that isn t particularly interesting or pleasant. Last week it happened again: I had to import a few pretty big PSTs (most 2Gig, one 10Gig, with about 100.000 emails) into our dovecot IMAP. Doing this with Outlook itself was out of the question: it took way too long, even on the local network (many hours for a 1G file) and was prone to hanging and crashes, which were obviously a pain to debug and start over.
Thunderbird was unfortunately not much better, since at least in our tests it didn t import the read status of the emails (they were all marked as unread) and also wasn t particularly good at handling folders with strange names, containing . , / or some more obscure characters. We had used it before for smaller files, where manually dealing with the problems was acceptable, but this time it required something a bit more elaborate, if we were to keep our sanity. Enter libpst. It includes the handy readpst utility which dumps all emails in usable formats in a directory tree, one directory per folder. Unfortunately the Debian version is somewhat outdated and doesn t support the newer Outlook formats, so I did some packaging and even a little bit of patching. It seems Thunderbird also uses this library, which would explain why it didn t handle the Read-Status (haven t confirmed this though; just read it somewhere). The last step was this not-so-little script, which uses the dumped directories from readpst and imports them in IMAP. It would have probably been a bit more elegant to use libpst directly, but I unfortunately didn t have the time to mess around with that. I did have to mess around a lot with encodings though, ergo the unholy chaos with unicode()s and str.encode()s thrown around like rice at a wedding (I could never really wrap my head around charset problems; the subject boggles my mind to this very day). Another useful automation is using the mappings file, which looks something like to automatically import in the right destinations, in case you have a standardized structure.

Even with the availability <video> tag in HTML5, most websites still require the use of Adobe Flash plugin to access some content. If you run a 64-bit (Intel architecture) version of Debian, you can still get this plugin through the flashplugin-nonfree package. Unfortunately, Adobe never commited to a decent support for the 64-bit version of its plugin. The first 64-bit version was provided as late as the fall of 2008. The support was discontinued. Then, we recently got a new preview release but it is still lagging behind the 32-bit version. The 64-bit plugin has two major drawbacks:

• it is not up-to-date and plagued by several security holes which are not planned to be corrected for the 64-bit version;
• it exhibits some bugs of its own, like some sound artifacts which come from the use of memcpy() in place of memmove() and only happens with recent GNU libc (where memcpy() has been optimized and is no longer equivalent to memmove()). Adobe has no plan to fix this. There are several way documented in the bug report to circumvent this bug.

Even if you fix the second point, you have no way to fix the first one. What are the alternatives?

Lightspark Lightspark is a modern, free, open-source Flash player implementation featuring JIT compilation (with the help of LLVM) and hardware acceleration using OpenGL. It only supports Actionscript 3 (introduced in Flash 9) but it now fallbacks to Gnash if an older version is required. The main problem of Lightspark is that it is optimized for modern hardware and uses OpenGL textures to display video frames. This allows to achieve video compositing with the help of the graphic card. Unfortunately, if you use a video card with poor OpenGL performance, like a NVidia card with Nouveau driver, Lightspark is slow as hell. It works well with an Intel card. Currently, Lightspark is only available in experimental for Debian. Install browser-plugin-lightspark.

Gnash Gnash is a more mature project but does not support Actionscript 3. Youtube works well with Gnash, as well as most other applications unless they use too recent features. If your use of Flash is limited, it is a nice alternative. Unfortunately, more and more content are delivered with an application requiring Actionscript 3 support. On Debian, you can get started by installing browser-plugin-gnash.

32-bit Flash plugin on Debian Ubuntu provides Flash as a 32-bit plugin with the help of nspluginwrapper which is a special plugin that will enable the use of a 32-bit plugin inside a 64-bit browser. This seems a sensible choice since Adobe only really supports the 32-bit version. You can also use nspluginwrapper with Debian. You need to download 32-bit version of Flash, uncompress it and put libflashplayer.so in ~/.mozilla/plugins.

# aptitude purge flashplugin-nonfree
# aptitude install nspluginwrapper lib32asound2-plugins
$ cd ~/.mozilla/plugins
$ nspluginwrapper -i $PWD/libflashplayer.so

Beware that you need to watch for updates yourself. Mozilla provides a website checking if plugins are up-to-date. It works with other browsers than Firefox. This is a pity to still need this kind of hack in 2011 but it seems that this is currently the best solution. UPDATED: In a comment, Paul Rufous tells debian-multimedia.org repository provides flashplayer-mozilla package which is the 32-bit plugin with nspluginwrapper.

HTML5 Thanks to the <video> tag available in HTML5, most videos should be available without the need of a special plugin. Unfortunately, HTML5 does not define a list of video codecs and container formats that should be supported by any browser. Therefore, some browsers support Ogg Theora, some others H264 and some of them also support WebM, the new royalty-free format from Google. You can find a good summary of the problem from Mark Pilgrim. With Chromium as found in Debian, you have Ogg Theora, WebM and H264. Youtube has an opt-in program to enable HTML5 video but most other video content are just unavailable without Flash. Web without Flash is not a reality yet.

Grab video with helper tools UPDATED: Josh Triplett pointed in a comment the existence of several command line tools that could be used to download Flash video from popular websites. Those tools include:

• youtube-dl
• get-flash-videos
• clive

Those tools can be useful even if you have installed a Flash plugin since they allow the use of the video player of your choice (better performance, working fullscreen mode). UPDATED: Another solution provided in a comment: Video DownloadHelper extension allows to extract videos from the current web page. In another comment, this is the extension FlashVideoReplacer which is featured. It replaces the Flash container with a standalone player.

My internship is now finished, this means that I will be able to work on my GSoC project a lot more than the previous weeks. I hope to make some good progress in the next days.

This last 2 weeks, I was still studying how Jigsaw works. Even if there is no progress on the Jigsaw packaging I have done some packages that will help later.

I ve spend a lot of time trying to package JT Harness and JTReg.
I ve successfully built JT Harness using CDBS but I ve finally rewritten the package using only debhelper and javahelper. Javahelper was really useful to build JAR files, move them to the right directory, put the Javadoc in the right place as well as the examples. The Debian directory of JT Harness can be found on the Java team SVN.

svn co svn+ssh://$ SVN_USER @svn.debian.org/svn/pkg-java/trunk/libjtharness-java

The source package of jtharness build 2 binary packages: libjtharness-java and libjtharness-java-doc. You can see that by looking at the control file in the debian directory. There are some interesting files in this package, let s take a look at them.

First of all, some of the source files needed to be patched for the package to build. So to accomplish this, there is a couple of patches in debian/patches :

• dependencies.diff, used to locate JAR files in /usr/share/java , and,
• rxtx.diff, done by Tom Marble used to change some imports from javax.comm to gnu.io .

To generate these patches, I ve used the edit-patch command which is a really awesome tool. This command uses a temporary shell where we can edit files we want to patch. When the job is done, CTRL + D terminates the temporary shell and the patch is generated.

There are 4 interesting files which are used to build the binary packages and put the files in the right directories. These files are:

• rules,
• libjtharness-java-doc.install,
• libjtharness-java.jlibs, and,
• libjtharness-java-doc.javadoc.

Of course rules is the most important one. This file tells us how packages are built. A good thing is that rules is quite simple in the jtharness case.

#!/usr/bin/make -f
JAVA_HOME = /usr/lib/jvm/default-java

override_dh_auto_build:
ant -f build/build.xml

%:
dh $@ --with javahelper

We just tell where Java is and to use javahelper ( with javahelper). Overriding dh_auto_build is interesting because since the JT Harness uses Ant as build system we need to give to Ant where the build.xml file is. By default Ant is searching it in the current directory but in jtharness the file is in the build directory. Since the Ant build file is correctly written, the build generates JAR files, Javadoc and examples code. And libjtharness-java* files are here to put the generated things in the right places.
The job of libjtharness-java.jlibs is to tell where to find the JAR libraries. In this case, the intesresting JAR files were javatest.jar and jt-junit.jar . So in the .jlibs file we just give where those files are after being built.
The job of libjtharness-java-doc.javadoc is to tell where the Javadoc is and where to put it. So there is only one line that says the generated Javadoc is in jar-build/javadoc and put it in /usr/share/doc/libjtharness-java/api .
And finally, libjtharness-java-doc.install is used to move the examples. The syntax is same as libjtharness-java-doc.javadoc .

The JT Harness package can be found in Sid now thanks to Sylvestre Ledru who uploaded it for me. So the ITP bug that I created is now closed.

Building the package for jtreg is quite similar. In fact, it probably easier but I m still facing a problem. jtharness is just a library, jtreg does have main classes and generate some binaries. The difficulty here is to generate the man pages for the 2 generated binaries: jtdiff and jtreg. These 2 files are shell scripts which just use jtreg.jar with the correct parameters.

I tried several tools to generate the man pages of each script. Sadly, I can t find a way that does not causes any problems.
My first choice was to use help2man. This tool looked like to be perfect for the job. But I couldn t find how to launch the scripts correctly. The scripts needs jtreg.jar but they also need javatest.jar from the libjtharness-java package. The classpath cannot be set correctly due how the scripts are written. So to use help2man I ll probably have to patch and completely rewrite jtdiff.sh and jtreg.sh in the sources.
My second choice was to use txt2man. During the build of jtreg there is a usage.txt file which is generated and which can be used as a man page. txt2man can take this text file and output a man page based on the text. I thought it was working but when building the package, lintian threw some warnings about syntax errors in the man pages.

W: jtreg: manpage-has-bad-whatis-entry usr/share/man/man1/jtdiff.1.gz
W: jtreg: manpage-has-errors-from-man usr/share/man/man1/jtdiff.1.gz 292: warning [p 4, 3.3i]: cannot adjust line

Finally, few days ago I decided to try rst2man. This tool looks cool and easy to use. But the text that I have to convert into a man page is not a reStructured text so it cannot without patching the usage.txt file for example. I was going to do it but there is no such file in the sources of jtreg. Actually this file is generated during the build but the text had to come from somewhere. And it was from two properties files in the source code.
I m still looking for a good solution to generate the man pages. If someone has any idea just let me know

The debian packaging of jtreg is not in the Java team SVN yet because of the man pages problem. I opened an ITP bug so people know that someone is working on jtreg.

My goal for the next two weeks is to finish the jtreg package, understand how jpkg (which is supposed to create Debian packages) works in Jigsaw and write more examples using the modular JVM.

The main release party in Finland was held at Tampere, Finland and organized by Ixonos Plc and COSS. It was a great success, as proven by the almost 200 participants and great speakers.

The first sessions were mostly about the basics of Ubuntu and its roots in Debian and elsewhere. Then Tuure Vartiainen from Tampere University of Technology shared with us the release from the official Finnish mirror perspective (fi.archive.ubuntu.com, fi.releases.ubuntu.com, ...). 10.04 LTS release date was very hectic and the transfer speeds were not constantly optimal, but quite good anyway. For 10.10 they hope to up the network connectivity to 10G. I also took the opportunity to thank him later personally for the Ubuntu Finnish Remix mirroring which was arranged right before the release as well.

The next-to-final session was about Ubuntu for senior people with brief demoing of how Ubuntu UI can be customized. To give a little different perspective to usage of Ubuntu, the final speaker was a theater director and dramatist Jotaarkka Pennanen from Interactive Film Productions. Blender among else was praised.

In addition to speakers, we had 300 Ubuntu Finnish Remix CD:s, Ubuntu posters, Free Software Foundation Europe flyers, COSS flyers et cetera. After the main program there was a dinner and some wine offered to participants, which was a great social ending to the event.

Now a few photos follow. Unfortunately they are from before the event actually began, so others have probably more crowded photos and photos of the speakers themselves.

Unfortunately, GLabels is currently unmaintained. Still, it's a very convenient tool, IMHO to print DIY business cards. Here's my glabels business card file, for those interested to borrow and reuse it.

I procrastinated to look up this for such a long time.. today was enough. What I want to achieve is reading a line from stdin and split it among several variables, naively I'd do:

$ echo "A BB CCC" read a b c
$ echo $a $b $c


(no output.) As well explained here the above code doesn't work, but this one does:

$ read a b c < <(echo "A BB CCC")
$ echo $a $b $c
A BB CCC

as a hint, <(cmds) is called process substitution.

I run some servers that use the DJB Daemontools to manage their daemons [1]. This is something I would have changed years ago if given a chance because DJB software always seems to do things in a different way to other Unix software and causes pain. Unfortunately when you have a lot of semi-embedded systems that have intermittent net access it s really not easy to change things, and having staff who aren t computer experts who have been trained to use certain DJB software makes it even more difficult. Daemontools uses multilog [2] to manage it s logging, this gives dates of the format @400000004deedcea1e4a18d4. While DJB has written a tool to parse this it s not always convenient, and I don t want to install DJB software on every system that might be used for reading logs (among other things DJB software is not included in any popular distribution). George Bernard Shaw says that All progress depends on the unreasonable man [3], of course he never participated in a large-scale software development project. In the modern age progress usually depends on people who can work with others, which is why DJB software doesn t get used much for every DJB program there is a similar program written by someone else that works about equally well on it s own and is more than 10* more popular because of better interoperability. So I wrote the following script to convert DJB dates to regular dates. It takes a DJB format date as the first command-line parameter as I generally just paste the relevant date into a different window. At some future time I may write a program to parse an entire log file and convert all the dates but I haven t had a need for it yet. I think that I ve done the hardest work involved in writing such a parser so someone else can use this as a starting point if they have such a need. #!/bin/bash
set -e
DATE=$(echo $1 cut -c 10-17)
SECS=$(echo -e ibase=16\\n$(echo $DATE tr "[a-z]" "[A-Z]") bc)
exec date -d "1970-01-01 $SECS sec utc"

• [1] http://cr.yp.to/daemontools.html
• [2] http://cr.yp.to/daemontools/multilog.html
• [3] http://elise.com/quotes/quotes/shawquotes.htm

Over the course of the last five years, some relatives of me and I have been working on a family book which lists all descendants of my grandgrandgrandgrandfather Isaak Riehm, born 1799, and his wife Charlotte. My task was everything related to maintaining the data, running the infrastructure and generating the final book. In the end, we have collected biographical details, often with the person s live story, of 703 descendants, 285 partners and 416 parents-in-law. Together with an extensive preface the book now has a respectable size of 401 pages, includes 144 pictures and maps and over 200 copies have been printed and sold through the family. Although I do not think that the details of the lives of my family is very interesting for the general public, or even suitable for general release, I do want to show the fruit of our work. Therefore, I have created a stripped-down on-line version that only covers the first four generations. I bundled the book body with the cover and the end sheets into one complete PDF file (caution, large: 54MB, and of course in German). Some nifty embellishments of the book that I especially like are the abstract looking descendant fan in the front-end and back-end sheets, the separating bars between the persons which indicate the livespan of that person between 1799 and 2011 and the maps of the birth places and Isaak s travel destinations in the preface. Now to the technical details: Naturally, I used only Free Software in the genesis of the work (The cover is an exception, as it was not created by me). The most important is definitely gramps, a free genealogy system that held all our data. It is written in python and extensible with plug-ins, so I was able to create a custom report that takes the data relevant for us, filters for people who have agreed to appear in the book, and then generate LaTeX code that typesets the book, ready to be printed. I learned a great deal of LaTeX for this project, which uses over 30 different packages and required me to create a font of my own (which, as a side product, lead to a publication in the Linux-Magazin). For the maps I found the Generic Mapping Tools (GMT) and the freely available data for that of great use. I still had to manually collect the coordinates for all the places I wanted to show up in the map. Here, I relied mostly on Wikipedia. The fan which appears in the end sheets is laid out using a Haskell program (naturally, I have to use Haskell somewhere). Of course, generating the book is only one side of the story; first, the data has to be collected and edited. For that, I wrote a plug-in to gramps that creates forms for each person. These forms contain the current known data and are used to correct and extend our knowledge. One variant of the form, created using LaTeX, was printed and mailed by snail-mail around the country and beyond, for those who are not reachable by mail. The other variant is an HTML form, hosted on a password-protected web site. Changes to that form cause the server to send an e-mail with a description of the changes (and only the changes, not the fields that are left untouched) to the mailing list of our team. There, the changes can be reviewed, e.g. for spelling and stylistic issues, and finally I copy n pasted them into gramps. These parts are implemented in python as well. Since genealogical data is never complete or up-to-date, I have set up a password-protected mediawiki based web site for the book where we collect corrections and additions to the data in the book. We have never anticipated that it would be so much work and that it would take us so long to finish this project, but we pulled it through and we are very happy with the result. Also, the reception has been very good and it was definitely a success. At this point I d like to express my gratitude to the hundreds of Free Software developers out there whose code we used and without whom it would not have been possible publish the book in the current form at all! Update: Due to some requests, I am making the code that I wrote available. You can download the tarball. Note that the code is a big mess and will not be usable right away. I guess if you can get the code to run and actually adjust to your needs, then you can as well write it yourself. Nevertheless it might be useful as inspiration or reference.

Last year the Psychologist and Baptist minister George Rekers who is famous for anti-homosexuality pseudo-science was discovered to be hiring gay escorts from Rentboy.com. Lots of LULZ there. But the story didn t end there. It turns out that George Rekers did some research on a child who ended up committing suicide as an adult, and the circumstantial evidence suggests that George s actions are directly related to the suicide [1]. The Rentboy.com affair doesn t seem so funny now. The Box Turtle Bulletin has a series of articles about Kirk Andrew Murphy s suicide and the roles of George Rekers and Richard Green in all of this [2], the articles are well written and generally appear to be well researched I recommend reading the articles if you can stomach them (lots of nasty stuff is described). The section answering the question of who s responsible for the mistreatment of Kirk Andrew Murphy [3] where they describe the use of ABA (AKA the Lovaas Technique) is interesting. Ivar Lovaas worked with George Rekers in such research and published a paper with him. The term ABA gets an immediate hostile reaction in the Autism community, but until now I hadn t realised why so many people hate it so much. It seems that to some extent I made the classic mistake of misjudging the reports of Autistic people who are unable to present their case well (as opposed to the psychologists who can present any position very well even if it s utterly insane). In the past I had the impression that ABA wasn t inherently bad, it was just implemented in a bad way in some cases now it seems that ABA was designed in an evil way right from the start. There is one massive problem with the Box Turtle analysis, he says Behavioral analysts don t dig around much into people s feelings, fears, dreams, family relationships or childhood memories. Indeed, in cases like autism, Lovaas s specialty, those avenues of exploration would be irrelevant . It could be that Jim Burroway (the Box Turtle writer) is merely quoting someone else without attribution, but even so saying that the feelings, fears, and dreams of a group of people are irrelevant is just awful, a statement that denies the humanity of a group of people can t be quoted without further explanation. In his article about ABA Jim refers to childhood Autism as a condition for which there is no hope for interior change [4]. I m not sure if he s just saying that Autistic children are incapable of learning or whether it s all Autistic people, in either case it s nonsense in terms of science and nasty as well. Generally I expect that members of various minority groups will show more sympathy to each other than they receive from the general population. Jim s posts are a great disappointment. I understand that he would be rather stressed about the horrible things that George Rekers et al did, but even so he should be able to avoid that sort of thing. Jim is obviously a very talented writer and can do better. One might think that Jim s posts use the word Autism to refer only to the people who are non-verbal (or in other ways less capable than the huge number of Autistic people who work for companies like Google and IBM). But that s no excuse either. You can find blogs and essays written by non-verbal Autistic people that describe their experiences if you care to search for them. It s obvious that they are people too and deserve to be treated as people not objects. Abusing Autistic children to try and make them impersonate NT children is no less evil than abusing children who don t fit gender norms.

• [1] http://edition.cnn.com/2011/US/06/08/rekers.sissy.boy.experiment/
• [2] http://www.boxturtlebulletin.com/what-are-little-boys-made-of-main
• [3] http://www.boxturtlebulletin.com/cuius_culpa
• [4] http://www.boxturtlebulletin.com/blind-mans-bluff

This is a follow-up on TDebs with some detail of how existing packages can be adapted to support TDebs.

$ cat debian/package-tdeb.tdeb

[gettextdir] po
[gettextname] package
[po4apodir] doc/po
[po4amandir] doc/package/man/
[po4acommand] po4a-build


What's happening then? Well, the first thing to take into account is that a lot of the values for these settings will be dictated by upstream, which is why it is necessary for maintainers to add this information as a support file in debian/. I tried determining all this information automatically but the script got far too long and still was far from complete.

• gettextdir - usually po but some upstream packages can support multiple po generation, so this allows for those.
• gettextname - the hardest thing to identify reliably, this is the name of the files to be installed in /usr/share/locale/lang/LC_MESSAGES/ and the name of the of the POT file.
• po4apodir - po4a is for documentation translation support, which is more variable, so a po4acommand field is available too.

Other changes
The (currently unsupported in Debian) XC-Package-Type: tdeb option is used in debian/control to mark a new or existing package as a TDeb. To use an existing package, it must comply with the rules for a TDeb including that it must be Architecture: all. The other rules centre around build issues. If any of the files in the existing package would require execution of any part of the normal build process of the package, these would have to be moved out of the package before that package could be a TDeb. The reason is that TDeb generation by translators needs to not require a full build environment for every individual package and must not risk modifying package contents in ways which are related to the build environment. So, unchanged images and other architecture-independent files are OK, any files which need build-dependencies are a problem. I don't know how many packages there are out there with such files. If you have a -data or -common package which contains translated data and that package also includes architecture-independent files, I'd be interested in knowing the name of the packages involved. If those packages also include files which are generated or modified within the package build, I'll be particularly interested in knowing about those source packages.
debconf
The most common use of TDebs during a release cycle is probably going to be debconf template updates by Christian and these are handled without needing a listing in the .tdeb file.
Other formats
I'm looking at QtLinguist support and it should be fairly straightforward using the .tdeb file arrangement.

Once that's done (in a branch, please - these can't be uploaded yet), the package can be tested with the soon-to-be-uploaded emdebian-tdeb package. The old content has been moved into the emdebian-grip-server package and the dependencies of the old version stripped back. The package contains two scripts dpkg-gentdeb and dh_gentdeb which I hope to get included into the respective packages in time for Wheezy. The idea will be that build systems (dh7, CDBS, debhelper-plain etc.) includes a call to the relevant script. (dh_ is just a wrapper around dpkg- in this case). This would allow maintainers to create the first TDeb, get that through NEW and then the fun starts. Once the translation mechanism is set up, I would like a nominated person (likely to be Christian if he agrees) to co-ordinate translation updates for particular packages to allow for a single upload including updates for many translations. This would be package_version+t1_all.tdeb and an appropriate .diff.gz or debian.tar.gz. No binary upload, no impact on testing migrations.

This is mostly theory right now but the scripts do exist in Emdebian SVN.

Still more testing to do but I hope to upload these scripts to unstable in the next few days.

As I'm asking for donations to help cover the traffic costs for the wine-unstable packages I provide and to support my Debian work in general if appreciated, I thought a "Thank you!" to all those who've donated something so far would be in order (don't worry, I won't disclose any names of donators or other personal information). Thus here we go: Thank you! It's much appreciated. For all those who just read this and think they'd like to donate something too, if just they could locate the links to do so, here they come:

• PayPal (general donation link, the wine-unstable specific link can be found on the package page
• Flattr:
  • general Debian work
  • wine-unstable packaging (traffic)
  • donate through Flattr (you need to be logged in, otherwise the previous link will redirect you to the main page)

Here is my first official Google Summer of Code report. As I still had classes, the first week of coding was rather slow. However, after my first phone call with my mentor, Matt Zimmerman, things started to pick up. I spent most of my time working on a dashboard/portal for the Debian Derivatives Exchange (DEX). We decided that the best approach to this would be to store as little information ourselves as possible. Instead, we will try to pull most of our information from other sources, such as Debian s BTS. The main advantage to this is that package maintainers do not need to change any of their workflows, but they still get to benefit from DEX s work. Currently, I have a python script that gets a list of usertags and bugs for debian-derivatives@lists.debian.org from the BTS. This script will end up getting run by cron. It stores the information it gets in static HTML files. Finally, JQuery and Javascript are used to present the final dashboard to the user. I created a mockup showing how this looks. I also have a functioning website that looks and acts like the mockup, but the code is not publicly available yet (see below). DEX uses Alioth to host its git branches and website. I have spent some time trying to familiarize myself with the recent changes that have taken place on Alioth, as well as trying to figure out the best way to store and execute my script to generate the website. So far, it looks like people are still deciding on the best way to handle having websites that are updated from VCS hooks (due to things being split between vasks and wagner). I will continue to look into this, and might try using an alternate host until things settle down. Next, I plan to work on updating the dashboard to provide a way to edit, store, and display certain extra fields of information. This is necessary for certain pieces of information that are not appropriate for adding to bugs and in a few other cases. I will also be taking some time to talk to a few people who were active contributors to the last DEX project. This will allow me to learn more about their experience, find out what changes they feel should be made to make things run smoother, and get feedback on my current Summer of Code work. Finally, although I have been posting updates on my Summer of Code work to my blog [2][3], Matt thinks that having all of this information in one place will be useful. As a result, I will be creating another wiki page about the project that will contain all of the information people need to know. This page will be updated as the summer progresses. I am also hoping to have my code available in a public VCS by the time of my next progress update so that people can check it out and provide feedback. In addition to the dashboard, I will be working on creating a few other tools to assist with the creation and tracking of new DEX projects. I will also be coming up with a way to generate graphs to visually allow us to track our progress on projects versus time as well as show the work done by people at the individual level. This last task is part of a goal to get more people involved in the DEX project. While we may only be a few weeks into the actual Google Summer of Code, I have already learned a lot of new things. I am looking forward to working more with Matt and other members of the community over the course of the summer.

Philipp is a Debian developer since 2005 and a member of the release team since 2008. Since he took the responsibility of Stable Release Manager, the process has evolved for the best. I asked him to explain how the release team decides what s fit for stable or not. His work on the buildd infrastructure is also admirable but I ll let him describe that. My questions are in bold, the rest is by Philipp. Who are you? I m a 24 year old Computer Science student, living in Karlsruhe, Germany. As a student assistant I m currently in two jobs: one is taking care of getting our university IPv6 network in shape, or rather generally getting fringe technologies into the network. The second is taking care of a s390x machine in the basement which my faculty got sponsored recently. In my spare time I tend my Debian duties and I m active in the student council (Fachschaft) as a sys-admin, software developer and until recently as treasurer. I got accepted as a Debian Developer in 2005. I m only really active since I was invited to join the Release Team in early 2008 after I contributed rewrites of some scripts that got lost in a disk crash of ftp-master. In late 2008 I also took on wanna-build/buildd duties. What s your notable achievement within Debian? For one I worked on the deployment of a single buildd/sbuild combination over all of our buildds and I rebased it on the sbuild in the archive several times already. All buildds basically look the same on all architectures, with only few variations. The chroots are now also created in a mostly predictable manner. Then we finally got build autosigning after years of constant poking. However the policy decision to allow it was made by the ftp-masters anyway, for which I m grateful, as it eases the workload of the buildd maintainers, the Security Team and the Release Team quite a bit. On the release side there s the integration of volatile into the proper structures of Debian. But there I m guilty for choosing the bad name squeeze-updates is, in comparision with security s squeeze/updates. Let s see if we can improve that for wheezy. The policy for stable updates have changed over time. Can you summarize what kind of updates are allowed nowadays? All the changes that were made to the policy were driven by the aim to keep stable (and to some lesser degree oldstable) usable throughout its lifetime. I have to admit that we got slightly more liberal in what we accept since I took over. The previous stable update policy included the fixes of critical bugs that break the system in interesting ways and security fixes. We also opened up the possibility of including important fixes for annoying bugs on a case-by-case basis. The whole don t update that much part of the stable release management is rooted in let s don t change the behaviour of a running system and let s have as few regressions as possible . We currently try to counter that with a review process that only allows self-contained fixes that were tested in unstable first, if applicable. In the future I d also like to have a feedback process that includes the users of the packages to report problems more early. In fact that s also why we now send calls for testing to debian-stable-announce one week in advance (see this mail as an example). So stable is updated through point releases that happen roughly every two months for stable and roughly every four for oldstable. They are accompanied by announcements and new CD/DVD sets. To be able to push some updates with less delay to our users and to avoid that they have to pick them from proposed-updates, there is stable-updates. The current policy for this suite is available in this post on debian-devel-announce. It boils down to everything urgent like tzdata, regression fixes and volatile packages. We noticed that some packages in stable just weren t useful anymore when they got updated through volatile. Hence those are folded into stable, too. We also relaxed the rules a bit so that leaf packages like clive, that rely on external websites not to change their URLs, can be updated too. If somebody wonders what happened to the and a half releases Those were intended to mark the inclusion of new hardware support (mostly by including a co-installable new kernel version). This kind of flag point release is thankfully no longer needed because our marvellous kernel team now backports certain hardware drivers to the kernel version in stable. They have some trouble soliciting test feedback for them, though. It would be cool if more people using stable could respond to their calls for testing. What are your plans for Debian Wheezy? Luckily the wanna-build/buildd side isn t much dependent on release cycles. The stable release management also just takes care of what gets dumped on them by the testing RMs. The near-term goals are certainly:

• getting d-i to use some development process in the archive that s compatible with the normal flow of packages into testing and
• hopefully getting wanna-build to use a sane SQL layout and hence better performance

Apart from that my work is mainly squashing the bugs on buildd.debian.org for the wanna-build part, so if you want something fixed, you need to report it, I m afraid. You have several roles within Debian, in particular stable release manager and member of the wanna-build team. Which role do you enjoy more and why? So the regular duties of the SRM, apart from setting up policies, are reviewing requests for uploads to proposed-updates, accepting them, chasing builds and scheduling point releases. It s mainly steady work. As for the Release Team it s working in a fabulous team where everyone s doing an awesome job. For me the wanna-build role is holding the things together and fixing up stuff that breaks while the world continues to evolve. And occassionally taking a stab at the ftp-masters. So rather different. My heart is probably siding with the buildd side because it s infrastructure but release work is fun, too. What s the biggest problem of Debian? Mainly that we re not quick in picking up new technologies and that we got slow on innovation. Distro-wide changes are usually a very big effort to coordinate and to finally complete. I guess people should be able to push fixes for release goals more eagerly into the archive. Also it seems to me that many DDs are actually disengaging from the release process, especially during the freeze but also before, which makes me a bit sad. However I m still not sure how we could counter that. As we re all volunteers you can t suddenly make others love the idea of releasing something together, I guess. Do you have wishes for Debian Wheezy? It s getting old, but there s certainly multiarch. Also a usable Python 3 is something I d really appreciate. Is there someone in Debian that you admire for their contributions? Peter Palfrader did an awesome job of getting a nicely working team together for DSA and got many tasks as automated as they are now. It s a pleasure to work with them and setting up a new buildd is, for example, a real breeze. He also developed a sane archive snapshot service, kudos for that.

---

Thank you to Philipp for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

Gearman is a distributed queue with several language bindings.

While Gearman has a nice Python implementation (python-gearman) of the client and worker, I chose to use the libgearman bindings (python-libgearman) directly since they are already packaged for Debian (as python-gearman.libgearman).

Unfortunately, these bindings are not very well documented, so here's the sample application I wished I had seen before I started.

Using the command-line toolsBefore diving into the Python bindings, you should make sure that you can get a quick application working on the command line (using the gearman-tools package).

Here's a very simple worker which returns verbatim the input it receives:

gearman -w -f myfunction cat

and here is the matching client:

gearman -f myfunction 'test'

You can have have a look at the status of the queues in the server by connecting to gearmand via telnet (port 4730) and issuing the status command.

Using the Python libgearman bindingsOnce your gearman setup is working (debugging is easier with the command-line tools), you can roll the gearman connection code into your application.

Here's a simple Python worker which returns what it receives:

#!/usr/bin/python

from gearman import libgearman

def work(job):
   workload = job.get_workload()
   return workload

gm_worker = libgearman.Worker()
gm_worker.add_server('localhost')
gm_worker.add_function('myfunction', work)

while True:
   gm_worker.work()

and a matching client:

#!/usr/bin/python

from gearman import libgearman

gm_client = libgearman.Client()
gm_client.add_server('localhost')

result = gm_client.do('myfunction', 'test')
print result

This should behave in exactly the same way as the command-line examples above.

Returning job errorsIf you want to expose to the client errors in the processing done by the worker, modify the worker like this:

#!/usr/bin/python

from gearman import libgearman

def work(job):
   workload = job.get_workload()
   if workload == 'fail':
       job.send_fail()
   return workload

gm_worker = libgearman.Worker()
gm_worker.add_server('localhost')
gm_worker.add_function('myfunction', work)

while True:
   gm_worker.work()

and the client this way:

#!/usr/bin/python

from gearman import libgearman

gm_client = libgearman.Client()
gm_client.add_server('localhost')

result = gm_client.do('myfunction', 'fail')
print result

LicenseThe above source code is released under the following terms:

To the extent possible under law, Francois Marier has waived all copyright and related or neighboring rights to this sample libgearman Python application. This work is published from: New Zealand.

Just added a TDeb test build branch for multistrap to test out the changes in the next version of the emdebian-tdeb binary package - specifically:

Move em_installtdeb into the emdebian-grip-server package so that the emdebian-tdeb package can be slimmed down to only contain what will become useful for other maintainers to use in their debian/rules files and for translators when the time comes to implement DEP-4. CDBS rules / patches to follow... bug reports against dpkg and debhelper to follow ...

DEP-4 itself is being restarted and refreshed but with some problems on alioth currently, I'm going to (try and) keep the document up to date via SGML and people.debian.org.

The changes are now in Emdebian SVN and I'm starting testing with my own packages first. Multistrap for PO and po4a-build testing, dpkg-cross for debconf testing, some others for testing with build systems other than CDBS, then the discussions will restart on debian-devel and I'll upload the TDeb support scripts to unstable.

There are a variety of things which need to happen before these scripts become useful, including:

• dpkg support for XC-Package-Type: tdeb
• debhelper support for dh_gentdeb,
• inevitable bug fixing in emdebian-tdeb

Why is the package called emdebian-tdeb? Simple - that's where the scripts have lived until now and the hope is that dpkg-gentdeb can migrate into dpkg-dev and dh_gentdeb into debhelper - eventually.

By all means play with the scripts and follow the example multistrap branch above to see what kind of changes might be needed in your own packages. I'll post again when I've worked out the changes needed in packages like dpkg-cross which only need TDebs for debconf support. (This is the easiest mode of TDeb usage but it's also the most useful during a release cycle - maintainers make a single change and then a nominated i18n boss-person - take a guess who - can update the all your debconf translations in one upload without a binary build - eventually possibly not even needing any bug reports, just a simple email advance warning.

What I've got to work on still is how the generation of an updated TDeb creates a new debian diff / debian.tar.gz to accompany the upload. For that I need XC-Package-Type: tdeb support in dpkg....

Sample output from the viewpoint of a translator updating a package already converted to support a TDeb:

neil@sylvester:tdeb-test$ dpkg-gentdeb -t

Set the changelog message to version 2.1.15+t0 first, closing any relevant bugs!

 dch -v 2.1.15+t0 

neil@sylvester:tdeb-test$ dch -v 2.1.15+t0 TDeb test

neil@sylvester:tdeb-test$ dpkg-gentdeb -t

Building TDeb update: 2.1.15+t0

 (192 entries)

Discard doc/pod/1/da/multistrap (65 of 177 strings; only 36.72% translated; need 50%).

Discard doc/pod/1/da/device-table.pl (5 of 19 strings; only 26.31% translated; need 50%).

Processing untranslated files for pod/multistrap (1) . . .

Processing untranslated files for device-table.pl (1) . . .

Processing de translations for pod/multistrap (1). . .

Processing de translations for device-table.pl (1). . .

Processing fr translations for pod/multistrap (1). . .

Processing fr translations for device-table.pl (1). . .

Processing pt translations for pod/multistrap (1). . .

Processing pt translations for device-table.pl (1). . .

dpkg-gencontrol -pmultistrap-tdeb -Pdebian/multistrap-tdeb -cdebian/control

dpkg --build debian/multistrap-tdeb ../multistrap-tdeb_2.1.15+t0_all.tdeb 

dpkg-deb: building package  multistrap-tdeb' in  ../multistrap-tdeb_2.1.15+t0_all.tdeb'.

neil@sylvester:tdeb-test$ dpkg -c ../multistrap-tdeb_2.1.15+t0_all.tdeb

drwxr-xr-x root/root         0 2011-06-08 21:06 ./

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/pt/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/pt/LC_MESSAGES/

-rw-r--r-- root/root      6427 2011-06-08 21:06 ./usr/share/locale/pt/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/vi/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/vi/LC_MESSAGES/

-rw-r--r-- root/root      6963 2011-06-08 21:06 ./usr/share/locale/vi/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/fr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/fr/LC_MESSAGES/

-rw-r--r-- root/root      6789 2011-06-08 21:06 ./usr/share/locale/fr/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/da/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/da/LC_MESSAGES/

-rw-r--r-- root/root     16311 2011-06-08 21:06 ./usr/share/locale/da/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/pt/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/pt/man1/

-rw-r--r-- root/root     33963 2011-06-08 21:06 ./usr/share/man/pt/man1/multistrap.1

-rw-r--r-- root/root      4471 2011-06-08 21:06 ./usr/share/man/pt/man1/device-table.pl.1

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/vi/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/fr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/fr/man1/

-rw-r--r-- root/root     33969 2011-06-08 21:06 ./usr/share/man/fr/man1/multistrap.1

-rw-r--r-- root/root      4708 2011-06-08 21:06 ./usr/share/man/fr/man1/device-table.pl.1

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/da/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/da/man1/

I'm also going to DebConf11



On the RoadTrip! all the way hrough France, Belgium, Netherlands, Germany, Austria, Slovenia, Croatia, Bosnia-Herzegovina (Banja Luka) in an MX5.

I've got a lot of things to talk to people about @ DebConf this year... you won't be able to tell if I'm just buying you a beer or trying to badger you about your packages!! Just note that TDebs have been already agreed (in terms of how TDebs work in the archive) with ftp-master at a previous meeting in Extremadura and I'm being badgered by a Release Manager and an ex-DPL to get TDebs implemented real soon now, so it is going to happen and hopefully in time for Wheezy - it's just a matter of fixing the bugs. The quicker everyone has a look at the new scripts (which have been completed re-written and immensely simplified since I first tried a dumb conversion of em_installtdeb) and the likely changes needed in packages, the quicker I can get the bugs out of the scripts.

More documentation to follow, I'll be updating the SGML whenever practical and the actual DEP just as soon as I get commit access back.

The world of augmented reality is quite fashinating.

I recently got interested into Natural Interaction due to the popularity of Microsoft's Kinect and decided to package OpenNI for Debian.

OpenNI is a framework aiming to making natural interaction transparent to the user/developer, publishing a unified API and abstracting the access to different sensors and modules. Currently it supports some Asus sensors and there is a branch on github supporting also Kinect (which is the one we're packaging).

Some cool projects are already using OpenNI: ROS, PointClouds and OpenCV come to my mind immediately.

We'are working on OpenNI package in the Multimedia Maintainers group on Alioth, the package is still not ready to be uploaded to unstable and upstream has some major problems like the lack of SONAME in its produced library, so some communication with upstream is needed but for preliminary use the packages are in a decent state: OpenNI, PrimeSense Sensor, NITE (non-free).

NITE is a closed source module set for advanced body and scene recognition, unfortunately PrimeSense didn't opensource them, thus I decided to use the same approach of flashplugin-nonfree and write a downloader.

For the sake of completeness, unstable already has libfreenect which purpose is similar but specifically oriented to Kinect.

I am studying both and really hope that in the next future some open source project will make (at least) skeleton and hand recognition available under an open license. People are already working on it, according to Google.

(First posting to Pl net Commandline! Tag: pcli) Vutral asked in IRC how to synchronise two shells environment while they re running. As you may know, POSIX systems cannot change a process environment vector after it has been started, only the process itself can. Well, the shell can, and we ll use a variety of things for this. This trick assumes you have $HISTFILE set to the same pathname in both shells (obviously, they run under the same user). It uses export -p to render the current list of exported variables, then transforms the list from newline-separated to a single big one-line export statement.
Then it transforms all remaining newlines (which will be part of a single-quoted string, since that s mksh(1) s export format) into the sequence '$'\n'' which means: terminate current single-quoted string, append $'\n' and open up a new single-quoted string immediately; concatenate these three.
Now, $'\n' is just a fancy way of saying newline, and part of mksh because David Korn (yes, the Korn in Korn Shell) strongly suggested to me that this functionality be included but, as we can see here, it pays off.
Finally, the so transformed string is prepended by unset \$(export); which, when executed, will cause the shell to unset (and unexport) all currently exported variables. The shell parameters that are not exported, i.e. not in the environment, are not affected by this code (except for $x and $nl, but whatever).
This string is then passed to read -s (plus -r and clearing IFS to enable raw mode), which means, read into the parameter $REPLY (which we conveniently don t use but it s trashed too, thus) but store into history at the same time. Ah hah! Now, the persistent history feature comes into effect! After running the below statement in the source shell, switch into the terminal running the destination shell, press Enter once on the empty line (Ctrl-U to empty it if it wasn t), then Cursor-Up ( ) to recall voil , an insanely large line with the previously created string sorta expanded and press Enter again to run it. Now your set of exported parameters is the exact same (minus if you exported IFS, nl, x or REPLY) as in the source shell.

I ve added extra spaces and a linewrap below, this is really just one big line:

Of course, this makes a nice function, for your ~/.mkshrc or somesuch.

Dropbox is a popular service to synchronize files between multiple computers. The service is entirely proprietary but the company is Linux friendly and provides Linux binaries ready to use. They even provide Ubuntu packages that wrap the dropbox client and provide integration with Nautilus. A bit of story Unfortunately for Debian users, those packages do not work on Debian due to a dependency that can t be satisfied (because Ubuntu introduced an epoch on the version of their nautilus package that Debian doesn t have). This was even reported in a Squeeze review in Linux Weekly News. At some point, Ivan Borzenkov introduced a dropbox package to Debian but it was not based on the above package, instead it packaged directly the proprietary binaries. This was a bad decision because the binaries bundle a set of LGPL libraries and nobody from Debian wanted to do the required work to provide the corresponding source code. So the package got dropped (see bug #610300). More recently several persons filed ITP (Intent To Package) bugs stating their willingness to re-introduce dropbox in Debian, but after many months lingering in the bug tracking system (see #544499, #613788), they have been turned back to RFP (Request For Package) because they changed their minds. What I did Being a dropbox user myself (despite the recent proof that data stored on dropbox is not 100% private), I offered sponsorship to the volunteers who wanted to package dropbox. But it turns out this was not enough to motivate someone to complete the task. In the mean time I was still using the old dropbox package that was removed (it used to be downloadable from snapshot.debian.org). While this was good enough for me, it s clearly not OK in the long term and way too difficult for the majority of users. So this week-end I spent some hours to create a proper package. It s loosely based on the package provided by Dropbox but I upgraded the packaging and changed the way it works. I patched the dropbox wrapper to provide a dropbox update command that downloads and updates the proprietary binaries. They are now stored in /var/lib/dropbox instead of having a copy in each user s home directory (~/.dropbox-dist/). This update command is run by the postinst so that installing the package immediately downloads the proprietary binaries. Get the Dropbox packages for Debian The package nautilus-dropbox has been uploaded to Debian unstable, it s currently in the NEW queue but will shortly reach the mirrors. Then you will be able to You can install the package with a simple apt-get install nautilus-dropbox (provided that you activated the non-free section since that s where the package is hosted, it can t be part of Debian since it requires the proprietary binaries to be useful). In the mean time you can get the packages from the links below:

• For Debian Squeeze: i386 amd64
• For Debian unstable/wheezy: i386 amd64
• For Debian unstable with GNOME3 from experimental: i386 amd64

Get the Dropbox packages for Ubuntu I have setup a PPA for nautilus-dropbox. Feel free to use it in place of the upstream packages.

$ sudo add-apt-repository ppa:hertzog/nautilus-dropbox
$ sudo apt-get update
$ sudo apt-get install nautilus-dropbox

It also includes packages for oneiric, but in theory once the package is accepted into Debian, it should appear in oneiric shortly after. Package maintenance I have done the initial packaging work but I don t really want to maintain it in the long term. I have more than enough to do with dpkg and my other packages. So if you are interested in maintaining this package, please get in touch with me. You should know a bit of python since there are Debian-specific patches of the upstream code. The package is maintained in a git repository. Feedback If you have encountered a problem with one of those packages, feel free to leave a comment. If you re an happy user of the above packages, click here to find out how you can thank me. I hope you enjoy the packages!

13 comments Liked this article? Click here. My blog is Flattr-enabled.