This week in brief:

I've rejoined the Debian Security Team

My first (recent) DSA was released earlier today, with assistance from various team members. (My memory of the process was poor, and some things have changed in my absence.)

BlogSpam gains a new user

The BlogSpam API is now available for users of Trac.

Finally, before I go, I've noticed several people on Planet Debian report their photo-challenges; either a picture a day or one a week. I too take pictures, and I'm happy if I get one session a month. I suspect some of my content might be a bit too racy for publication here. If you're not avoiding friendface-style sites you can follow "highlights" easily enough - or just look at the site. ObQuote: "Be strong and you will be renewed. Identify. " - Logan's Run (1976)

I have a Samsung CLP-315 printer. It works will with CUPS and foo2qpdl driver. Just ignore all the anti-distro junk written at driver s homepage, run aptitude install foo2zjs, and enjoy working printer on consistent system. But since the beginning there was a strange problem printing worked from everywhere, but not from iceweasel. If tried to print from the browser, error page was printed instead. So had to print to a pdf file, open it in kpdf, and print from there. Working but annoying. At last decided to do something with this. A quick search shows that problem is known, is not new, and as such has little chance to be quickly resolved. But then I found this hint. I immediately checked paper size in iceweasel s print dialog and yes, it was Letter! But the setting was greyed out, so unchangeable I ve tried to play with about:config (searching for print there), then tried to remove all printer setup from prefs.js no result, still Letter without option to change. But a web search helped again. Someone pointed that iceweasel has a separate menu item for page setup, at File::Page setup! That is where paper size has to be configured, tab within print dialog does not work! Mozilla people are wonderful UI designers. Many thanks to them for months of non-working printing. I ve told myself zillion of times that I should perform active steps to get rid of bugs on computers I use search the web for solutions, report bugs, fix code myself if nothing helps. That s the very point of using free software, after all. But still laziness lack of free time random real-life issues make me not to do so too often. So also many thanks to myself for months of non-working printing. But now printing from browser works, and that s good news . Update: full thanks above should be directed to, in order:

• Samsung firmware developers, who coded firmware that, when printing error page (btw wasting paper and toner), doesn t print what is actually wrong (paper size mismatch),
• Mozilla developers, who don t read default paper parameters from CUPS, but instead use own broken scheme,
• myself, for living with problem and not looking for solution for months,
• foo2qpdl upstream, who, by his anti-distro junk, turns people looking for solutions into wrong direction,
• Mozilla developers, who use braindead page setup UI.

And real thanks to the person (link above) for the hint about page size in applications that are unable to print

Lisa snagged us some sort of subscription to the CSO's jazz series and last night was a first concert resulting from this. In town were Allen Toussaint with a three piece rhythm section as well as Don Byron (also here) on clarinet and tenor saxophone, and Nicholas Payton (also here) on trumpet. Given Allen Toussaint's live-long association with New Orleans, there was a lot of music inspired by the town, including a modern rendition of an old Sidney Bechet tune as well as other material came from his most recent album The Bright Mississippi. And, given Allen Toussaint's fame in Rhythm and Blues circles, there was lot of R&B too. I found the program to be a little uneven and unbalanced that way---but it was still a great show, and he surely brought the house down. He is a charming old school performer, and maybe the audience warmed his heart right before the concert started with a quick Happy Birthday rendition as yesterday was in fact his this 73rd birthday! Don Byron and Nicholas Payton were, as expected, stellar in their own right. A nice night out.

Tomorrow, at this time, I'll probably be unsuccessfully trying to find a comfortable position on a seat of the Bilmanbus to Irun. Very early on Saturday, as soon as we get off the bus, Maria and I will quickly head to Hondarribia's beach in Cape Higer to symbolically wet our feet in the waters of the Cantabrian Sea. We won't have much time to enjoy the cold waters of the ocean, though. Soon after that, we'll have to take a deep breath, look East, and start walking if we want to achieve our utmost objective: take a bath in the beautiful beaches of the Cap de Creus, in the Mediterranean sea. In between, 30 days and 840 kilometres of thick woods, deep valleys, high peaks and cold waters, all of which shape incredible landscapes.

The Portella de Baiau, during our 2008 trip For a whole month, we'll be mostly disconnected from everything else that isn't our knee ache, our blisters, the Sun over our heads or where to get food. It's the first time I leave on a hiking trip as long and tough as this one, and I feel both uncertainty and eagerness. We've been so busy during the last few months that we've been unable to train at all for this, and I'm probably in the worst physical condition in a decade. It's too late to take care of that now, so we'll try to take good care of our legs and spine. In order to get back home in the Mediterranean, we'll have to be fast, some days joining two stages and skipping a few that we know are not that interesting (sections over asphalt, etc.). It's hard to make it in just one month, but we'll try our best. The plan is going to sleep not long after sunset, getting up at dawn, to be able to walk for a decent time before the heat starts being a handicap and just resting at midday, when the Sun is strongest. Of course, this means that I'm missing, yet again, this year's edition of DebConf in New York City, which is really sad because I was looking forward hanging around with Mako, Mika, Biella, micah, Clint and the rest of the NYC/ Boston gang, but when the idea of doing a long trip this summer popped up around January, it was clear DebConf seemed unlikely this year. I hope all of you have a lot of fun, and see you in a few weeks!

Two weeks ago, Maria and I completed one of our dreams when we arrived in Cap de Creus, where the Pyrenean range sinks in the Mediterranean. To get there, we walked hundreds of kilometres during a month, crossed dozens of steep valleys and enjoyed one of the richest experiences of our lives.

We managed to complete this challenge without facing major problems or pains, and after the first five our six days, our legs seemed to have gotten used to the daily effort and it started to be easier and easier. Our morale kept growing as days passed and we advanced east. When I started walking on a cold and rainy morning in Hondarribia, after barely no rest in the night bus to Ir n, I thought for myself that it was improbable that we'd manage to get anywhere near Catalunya, that one of us would get injuried way before, or we'd just give up and go for the easy beach vacation in the Basque Country.

Biad s refuge, under Posets But we didn't, and after a somewhat painful start, with our boots soaking wet during the stages that crossed the enchanting Selva de Irati which ended up with me getting the biggest blister I've ever seen, we started to walk farther every day, extending the stages when we felt strong after reaching their official end. When we crossed from Nafarroa to Aragonese territory, the mental wall that I had built over the toughest stages in our quest started to fall apart. Days later, we found ourselves climbing down to Pineta, leaving Ordesa behind and enjoying a feeling that our adventure could not go better. The weather had been perfect for over a week and our legs and back were strong to go all over the way to the sea.

Ca n de A isclo, on our way to Pineta When we finally saw that huge blue stain on the horizon, the day before getting to the last mark, we got really excited. We had made it, but as we walked towards the cape, happiness slowly got mixed with melancholy. An unforgettable adventure was about to end, and we didn't want to face our return to the city and our routines. It had been many days surrounded only by awesome landscapes, and living without watches, with only sunlight and weariness marking the time to get in our sleeping bags.

At the very last red and white mark of our journey We've had plenty of time to meet great people. Starting with the Navarrian brothers we met in the early stage, which provided us with a good pace to follow while our leg muscles were still building up; or Kike and Ana, who drove from Pamplona to visit us during the first Aragonese stages (thanks for the supplies!). Tom s and Roger, young hikers from Matar , surprised us with their maturity and experience as mountaineers; we had the pleasure to join forces during four days, walking a really cool variant through the Infiernos and Collado del Letrero which avoided going through the ghost city of Panticosa. Roger and Tom s, we really hope to meet you soon! Andreu, Manel, Ghandi and Ga la visited us at the Vall de N ria and were unlucky to suffer a frightening hail and thunder storm when they left our shelter on their way back to their car. Thanks! All of you have been a very special part of our experience!

This adventure through GR 11 has been incredibly positive for us for several reasons. We've learned a lot about ourselves and strengthened our relation, and now I know how powerful drive can be; to get somewhere, no matter how far, it's really easy if you just believe you can do it and desire to get there. Maria and I are proud of what we've done, and no wonder we're looking for new challenges. An obvious one would be repeating this experience, going through the French side of the Pyrenees, but for now, the Corsican GR 20 seems the most appealing. We'll see, next summer!

Yet another Ubuntu anecdote Some posts on planet made me remember of a little Canonical-related story of mine. Many years ago I shortly contracted for Canonical. It was interesting and fun. At the time I didn't have any experience of being temporarily hired by a foreign company, so I rang my labour union to get an appointment, to make sure with them that everything was allright. The phone call went more or less like this: Me:

Hello. I have received this contract for temporary employment by a foreign company and I wondered if I could book an appointment to come show it to you to see if it's all ok.

Their answer rather cut me short:

Hi. Be careful! People get temporary employment from obscure companies with the headquarters, like, in the Isle of Man, they do the job, the company disappears and they never get paid. There's bad stuff out there!

I looked at the contract, the heading said something like "Canonical ltd, Douglas, Isle of Man". I was certain that the union people would have never understood what was going on. I politely thanked them for their time and hung up. However, to this day I still regret that I didn't insist:

Uh, yes, the company is indeed in the Isle of Man. But what if I told you that it's owned by an astronaut?

I just signed the contract and had a good time.

I'm working on improving various bits of Apple hardware support, which includes trying to make EFI booting on Macs more sensible. One of my test machines is an 11" Macbook Air, which is a beautiful piece of hardware. If you want something small and light that's approximately a proper computer rather than a netbook, there's nothing else in the x86 market to touch it. Nothing.

So it's a shame that while Apple employs hardware designers who are absolutely the best in their field, their firmware developers are utterly incompetent. The EFI spec describes a protocol called GOP (Graphics Output Protocol) which allows the operating system to find out things like the screen resolution, framebuffer location and stride. This machine kindly passes everything back correctly, except for stride being wrong. Like, utterly wrong. Implausibly wrong. Obviously, terrifyingly wrong. So wrong that if you trust it, your screen image looks like it's been involved in some sort of awful pasta making accident. But never mind, that's a single line workaround.

But there's far, far worse. EFI has three modes of operation - boot services, runtime services, and runtime services with a virtual memory map. Boot services are, as the name implies, intended to be used by the bootloader. They're contained within memory marked with a type of either BOOT_SERVICES_CODE or BOOT_SERVICES_DATA. Once the bootloader has loaded the OS, just before handing over to it, it calls ExitBootServices() and the firmware is then at liberty to clear those areas of memory. The OS never executes boot services code and can use the memory that was allocated to them.

So then we have runtime services. EFI is, depending on how you want to look at it, either saner or much less sane than traditional BIOS access. The firmware gives you a bunch of function pointers and you then simply call them with native calling convention (this, incidentally, is why it's pretty much impossible to run a 32-bit OS on 64-bit EFI, or a 64-bit OS on a 64-bit system that happens to have 32-bit EFI). To begin with the firmware assumes flat physical mapping, which isn't entirely ideal when you're an OS with virtual address space - so there's a SetVirtualAddressMap() call which lets the OS pass a memory map to the firmware, and from then on everything just works.

Except on this machine. On this machine, you call SetVirtualAddressMap() and the kernel faults because it's just tried to execute code from a region of address space that's marked as non-executable. Closer examination revealed that, yes, this area of address space is marked as RAM and so the kernel's doing the right thing. Except, obviously, the right thing generally doesn't involve crashing the machine at boot time, so there was something else up.

That something else turned out to be SetVirtualAddressMap() calling into boot services code. The boot services code that's not supposed to be touched after ExitBootServices() has been called. The boot services code that's flagged as BOOT_SERVICES_CODE and was therefore mapped as RAM by us because, well, nothing's there any more. The boot services code that we're ignoring exactly as the spec says we should.

So, a quick change to the bootloader to pass us an e820 map[1] that marks the boot services regions as reserved, a quick change to the kernel to mark BOOT_SERVICES_CODE regions as executable and we're in business. Except that when we call SetVariable() to configure the bootloader, the firmware tries to read from an address that's approximately 54TB above anything else.

BlApple.

[1] Oh, yes, despite E820 being an x86 BIOS-specific thing, we use it for the in-kernel representation of the address map under EFI as well. So despite the EFI memory map being available to the kernel, the bootloader has to construct something that looks like an E820 map, losing information in the process. So even though it's been given the E820 map, the kernel has to look through the EFI memory map in order to find the EFI runtime services section in order to mark it executable. This is, arguably, insane, but so is the entirety of EFI so that's ok then.

Hello The new version of libconfig-model-perl 1.229 is now available in Debian/Sid. This new version provides a model of Debian s DEP-5 copyright file specification updated according to the CANDIDATE version. (Among other features, like editor/validator for debian/control). The DEP-5 editor must be run in the source package directory (I ll provide an example below) with the following command:

config-edit -application dpkg-copyright

Features:

• check copyright file (with option -ui none )
• migrate from old format to new format ( with option -ui none -save )
• graphical editor (no option, but libconfig-model-tkui-perl must be installed)

Known issues:

• some keywords from older versions of DEP-5 specs may have been missed. In this case, feel free to send a mail to config-model-users mailing list or to log a bug in BTS.
• the update of the content of the format string is silent. This behavior is a bit cavalier. This will change in the next version

Example: The copyright file of libconfig-model-backend-augeas-perl needs to be updated: Here s the original copyright file (slightly edited to protect my mail box and save electrons) :

$ cat debian/copyright
Format-Specification: http://wiki.debian.org/Proposals/CopyrightFormat?action=recall&rev=196 Upstream-Maintainer: Dominique Dumont (ddumont at cpan.org)
Upstream-Source: http://search.cpan.org/dist/Config-Model-Backend-Augeas/
Upstream-Name: Config-Model-Backend-Augeas Copyright: 2008-2009, Dominique Dumont (ddumont at cpan.org)
License: LGPL-2+ Files: debian/*
Copyright: 2009, Dominique Dumont
License: LGPL-2+ License: LGPL-2+
This program is free software; you can redistribute it and/or
[snip]

First, let s just check issues:

$ config-edit -application dpkg-copyright -ui none
Element Upstream-Maintainer of node Debian::Dpkg::Copyright is deprecated
Element Upstream-Source of node Debian::Dpkg::Copyright is deprecated
Element Upstream-Name of node Debian::Dpkg::Copyright is deprecated
Missing Files: specification at top of section number 2. Adding Files: * spec

Then, let s see what s config-edit can fix all by itself:

$ config-edit -application dpkg-copyright -ui none -save
[ same warnings displayed ]

Let s check the result of the migration:

$ config-edit -application dpkg-copyright -ui none
[ good: no more warnings ]

And here s the content of the upgraded file (drum rolls, please):

$ cat debian/copyright
Format: http://dep.debian.net/deps/dep5/
Upstream-Name: Config-Model-Backend-Augeas
Upstream-Contact: Dominique Dumont (ddumont at cpan.org)
Source: http://search.cpan.org/dist/Config-Model-Backend-Augeas/ Files: *
Copyright: 2008-2009, Dominique Dumont (ddumont at cpan.org)
License: LGPL-2+ Files: debian/*
Copyright: 2009, Dominique Dumont
License: LGPL-2+ License: LGPL-2+
This program is free software; you can redistribute it and/or
[snip]

You can see that:

• keywords were changed while keeping the correct content
• Format: was updated with the right URL
• Files: * line was added in the right place

Please provide feedback! If you have problem migrating DEP5 files, yell, and we ll fix them. If this migration works fine, don t be shy, please say so. Under the hood:

Most of the migration work is specified in DEP-5 model [2]. Using config-model s GUI [3] to update DEP-5 model, about one hour was spent to:

• flag old parameters as deprecated
• introduce new parameters from candidate version (mostly done by cut n'paste old parameters into their new name, done within the model editor GUI )
• cut n'paste the doc bits from the web page into the model editor GUI
• specify migrate_from instruction to copy semantic content from old parameters to new ones. (*)

This shows one important feature of Config::Model: maintenance and evolution of a model is not a huge task. Even though copyright files are not configuration files, the same approach can be used to specify model and migration of other configuration files. Now comes the question of the migration as seen by the user. Well, it s your turn now. Try it and tell us how it works! Hopefully, encouraged by your feedback, other people will apply config-model to other specifications or other configuration. All the best (*) E.g, in the text format produced by the model editor GUI: 'Upstream-Contact',

'value_type' => 'string',
'migrate_from' =>
'formula' => '$old_maintainer',
'variables' =>
'old_maintainer' => '- Maintainer'

,
'type' => 'leaf',
'description' => '* Syntax: line based list [snip]'
,


Today I discovered via digi.no that the Chrome developers, in a surprising announcement, yesterday announced plans to drop H.264 support for HTML5 <video> in the browser. The argument used is that H.264 is not a "completely open" codec technology. If you believe H.264 was free for everyone to use, I recommend having a look at the essay "H.264 Not The Kind Of Free That Matters". It is not free of cost for creators of video tools, nor those of us that want to publish on the Internet, and the terms provided by MPEG-LA excludes free software projects from licensing the patents needed for H.264. Some background information on the Google announcement is available from OSnews. A good read. :) Personally, I believe it is great that Google is taking a stand to promote equal terms for everyone when it comes to video publishing on the Internet. This can only be done by publishing using free and open standards, which is only possible if the web browsers provide support for these free and open standards. At the moment there seem to be two camps in the web browser world when it come to video support. Some browsers support H.264, and others support Ogg Theora and WebM (Dirac is not really an option yet), forcing those of us that want to publish video on the Internet and which can not accept the terms of use presented by MPEG-LA for H.264 to not reach all potential viewers. Wikipedia keep an updated summary of the current browser support. Not surprising, several people would prefer Google to keep promoting H.264, and John Gruber presents the mind set of these people quite well. His rhetorical questions provoked a reply from Thom Holwerda with another set of questions presenting the issues with H.264. Both are worth a read. Some argue that if Google is dropping H.264 because it isn't free, they should also drop support for the Adobe Flash plugin. This argument was covered by Simon Phipps in todays blog post, which I find to put the issue in context. To me it make perfect sense to drop native H.264 support for HTML5 in the browser while still allowing plugins. I suspect the reason this announcement make so many people protest, is that all the users and promoters of H.264 suddenly get an uneasy feeling that they might be backing the wrong horse. A lot of TV broadcasters have been moving to H.264 the last few years, and a lot of money has been invested in hardware based on the belief that they could use the same video format for both broadcasting and web publishing. Suddenly this belief is shaken. An interesting question is why Google is doing this. While the presented argument might be true enough, I believe Google would only present the argument if the change make sense from a business perspective. One reason might be that they are currently negotiating with MPEG-LA over royalties or usage terms, and giving MPEG-LA the feeling that dropping H.264 completely from Chroome, Youtube and Google Video would improve the negotiation position of Google. Another reason might be that Google want to save money by not having to pay the video tax to MPEG-LA at all, and thus want to move to a video format not requiring royalties at all. A third reason might be that the Chrome development team simply want to avoid the Chrome/Chromium split to get more help with the development of Chrome. I guess time will tell. Update 2011-01-15: The Google Chrome team provided more background and information on the move it a blog post yesterday.

After previous series of posts about Gammu usage in proprietary software, finding culprit and getting the sources I got number of questions and suggestions. So what are the suggestions? Lesson I've learned is that even though the website seems to be dead, the project itself does not have to be. This is what I thought about http://gpl-violations.org/ and I was proven wrong and these guys are now looking at my case. Another option is Software Freedom Law Center - http://www.softwarefreedom.org/. I've contacted them from the very beginning (about year ago) and got some useful hints, though now my contact seems to be gone (or simply too busy). FSF also provides information how to proceed in case of license violation. Maybe these hints will be also useful for somebody else and I hope they will help me as well.

Filed under: English Gammu Gpl 0 comments Flattr this!

As some of my readers may now, I'm a regular user of public transport. Even though I'm living in an area (far suburbs of Paris) where the use of cars for nearly everything is common, I managed up to now to not use my car for my daily commute. I was using the public bus line of our town, that starts close from our house and ends up at the local train station where I can commute to U line of Paris transport system (then C line, then another bus ride). You can compare this complicated 1h30 travel to the 35 minutes drive I would have if I was using my car. I *was*. I'll stop from tomorrow. The reason is simple: my U train start hour is 6h57. The bus formerly started at 6h36, then reached the train station at 6h51, which was a quite good commute. That made me leave our house at about 6h25, considering a 10min walk from the house to the bus stop. That's 32 minutes for....3 kilometers. Already needs some motivation, right? This changed today (with minimal notice). To catch the same train, I should have a bus at 6h31, thus leave 5 minutes earlier, so *nearly 40 minutes before the train schedule*. Considering that driving from our place to the train station is about 12 minutes, the deal is now made. I'll stop being the only non captive person using the bus in our neighbourhood (all others are people who can't use a car). So much for environment, etc. This is an 'interesting' way to understand why it is so easy to discourage proper use of public transport and minimize the impact on environment. A small change can lead to such effects. Of course, as you may expect, I won't stop there. Letters to the (public-funded) bus company and people in charge of public transport in our city (the bus line deficit is funded by the city), will follow very soon. Tehre are indeed many other reasons why people don't use our local buses that much. One being that, even though there are 2 trains every 15 minutes leaving the station (U and N lines), both alternated with a 7.5 minutes interval....there is only one bus every 15 minutes....on each bus line. Sometimes, even, you don't know why, there's a 23 minutes gap. Imagine waiting for your bus for more than 20 minutes at night in winter...when you're about 2-3 kilometers away from your home... Sure, that would need a lot of funding to have a really convenient system. But how about spending the billions of euros spent in super-highways, toll tunnels, etc., in public transport infrastructure? Funding my local bus line to have 2 buses every 15 minutes at peak hours would cost about the price of 100m of a 3-lane highway, for one year. Make the deal. (All that will indeed reinforce my backup system : run from my house to the train station, for 3km and 15 minutes...which involves another run at the end of my train ride, for 4km and 20 minutes. That, however, involves some "logistics" organization to be sure that I don't spend the day working in my stinking running suit. That also means up to 56km run every week, which I can't always do, particularly if I plan a race for the next week-end!) Another side effect: I will have less time for Debian as my bus commutes were used for hacking, while it will be much more difficult when driving my car to the train station, isn't it?

With help of LWN site, I ve discovered in-progress effort of writing a serious book about parallel programming, in form of a real open-source project where people can contribute. Book is titled Is Parallel Programming Hard, And, If So, What Can You Do About It?, edited by Paul E. McKenney. It is still far from being complete, but even in current state it definitely is a recommended reading. Many things are explained using Linux kernel code and environment. So it is also a good kernel resource. As for project things are managed in git repository, with commits from several people. Book sources are in TeX.

Older versions of Psi jabber client required manual changing status to online to leave auto-offline state. It was very handy for me: I connect to my jabber account from two locations (from home and from office), so after I left one of these locations, auto-offline soon happened, and Psi at that location did not return online until I returned and manually instructed it to re-login. Unfortunately newer versions of Psi (I guess, starting from commit 4a54500c on git://git.psi-im.org/psi.git, which was between 0.11 and 0.12 releases), and also Psi+, behave differently: if user activity is detected on the desktop, auto-offline state is left, and status becomes online again similar to leaving auto-away and auto-xa states. This change is not good for me, at least in two situations:

• vnc connection to home from office or to office from home I use this fairy often,
• occasional use of my home session by my wife for quick web browsing or similar.

In both these situations, Psi client not at my current location becomes online, which can cause routing incoming messages to wrong location. I don t state that new behaviour is somehow worse than old behaviour. But for my use cases, I want to get old behaviour back. I ve done some code reading. I think that point to fix is near the bottom of PsiAccount::secondsIdle() method, where any auto status is disabled if no timeout condition currently exists. This should be changed so that auto-offline state won t be left if some config key is set. I will try to implement this change soon (or at least I hope I will try ) P.S.
To make things even better, a check of some external condition could be performed to decide if client should return online from auto-offline or not. Then fancy things could be set up, e.g. return to online at office only if my mobile phone is currently registered in the office wifi network not really required but cool.

I use a lot of code in my lectures, in many different programming languages. I use highlight to generate HTML (via ikiwiki) for web pages. For class presentations, I mostly use the beamer LaTeX class. In order to simplify generating overlays, I wrote a perl script hl-beamer.pl to preprocess source code. An htmlification of the documention/man-page follows.

---

NAME hl-beamer - Preprocessor for hightlight to generate beamer overlays. SYNOPSIS hl-beamer -c // InstructiveExample.java highlight -S java -O latex > figure1.tex DESCRIPTION hl-beamer looks for single line comments (with syntax specified by -c) These comments can start with @ followed by some codes to specify beamer overlays or sections (just chunks of text which can be selectively included). OPTIONS

• -c commentstring Start of single line comments
• -k section1,section2 List of sections to keep (see @( below).
• -s number strip number spaces from the front of every line (tabs are first converted to spaces using Text::Tabs::expand)
• -S strip all directive comments.

CODES

• @( section named section. Can be nested. Pass -k section to include in output. The same name can (usefully) be re-used. Sections omit and comment are omitted by default.
• @) close most recent section.
• @< [overlaytype] [overlayspec] define a beamer overlay. overlaytype defaults to visibleenv if not specified. overlayspec defaults to +- if not specified.
• @> close most recent overlay

EXAMPLE Example input follows. I would probably process this with

hl-beamer -s 4 -k encodeInner

Sample Input

 // @( omit
 import java.io.BufferedReader;
 import java.io.FileReader;
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.Scanner;
 // @)
     // @( encoderInner
     private int findRun(int inRow, int startCol) 
         // @<
         int value=bits[inRow][startCol];
         int cursor=startCol;
         // @>
         // @<
         while(cursor<columns && 
               bits[inRow][cursor] == value) 
             //@<
             cursor++;
             //@>
         // @>
         // @<
         return cursor-1;
         // @>
      
     // @)

BUGS AND LIMITATIONS Currently overlaytype and section must consist of upper and lower case letters and or underscores. This is basically pure sloth on the part of the author. Tabs are always expanded to spaces.

At work we run a pair of ISC DHCP servers running Debian Lenny in a classical ISC DHCP failover setup which provide DHCP service to several subnets, some only with static IPs (e.g. for printers) and some with half static and half dynamic IPs. Today I got a call from a user that her laptop doesn t get an IP despite it s correctly registered in our MAC address database from which we generate the group sections of the dhcpd.conf. Everything looked fine, but every DHCPDISCOVER package got logged in the syslog on both servers like this:

Jan  7 14:34:39 dhcp1 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via eth2: peer holds all free leases
Jan  7 14:34:39 dhcp2 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via eth2: peer holds all free leases

Searching the web for this error message mostly results in mails which say If have this on one server but not the other, you soon run out of IP addresses , but none which mentions what happens if you got them on both sides. Following a coworker s idea of adding both servers to the search term, I found Debian bug #563449 (dhcp3-server: Incorrect peer holds all free leases log entries) which turned out as configuration error or at least unexpected configuration (machine was blocked from getting an IP on purpose) and misleading error messages. So I checked under which circumstances this computer would not get an IP despite it had a static IP configured:

  host somehost  
    hardware ethernet 01:23:45:67:89:ab;
    fixed-address 192.0.2.123;
   

That computer would not get an IP address in any subnet which has different IP range and no dynamic IP addresses. And even if I comment out the fixed-address setting, it wouldn t get an IP in any static-IPs-only subnet either. And *bingo*, that computer was plugged into the printer subnet which has only static IPs, e.g. in the 198.51.100.x range. So if you get the peer holds all free leases error message from both your DHCP servers, chances are very high that the mentioned MAC address should really not get an IP address on this network (as it does :-). The error messages are just somewhat misleading. Hope, this saves someone some time. :-)

It has been almost a full year since my last log entry. It has been a busy work year, I attended some nice conferences and did minimal FLOSS stuff. On the work side of things I was a third of an Australian VoIP startup that came and went. I setup Debian servers, installed OpenSIPS and associated software, wrote OpenSIPS scripts, wrote peripheral software and did customer support. We had a good thing going there for a while, some fans on the Whirlpool forums but in the end there wasn't enough money for the requisite marketing and local market circumstances were squeezing Australian VoIP providers anyway. On the conference side of things I went to LCA 2010, the Thai Mini-DebCamp 2010, DebConf10 and FOSSASIA 2010. Had a great time at all of them. At LCA 2010 in windy Wellington, New Zealand the distributions summit organised by Martin Krafft was one of the highlights. It was dominated by Debian/Ubuntu talks but there were some other interesting ones, especially the one on GoboLinux's integration of domain-specific package managers. Also excellent were the keynotes given by Gabriella Coleman (Best & worst of times), Mako Hill (Antifeatures) and others, which I felt gave LCA an improved and very welcome focus on software freedom. There were quite a few Debian folks at LCA, it was great to hang out with them during the week and afterwards. Monopedal sumo with mako and others was hilarious fun. At the Thailand Mini-DebCamp 2010 in Khon Kaen, I was glad to see Andrew Lee (Taiwan) and Christian Perrier (France) again and meet Yukiharu YABUKI (Japan) and Daiki Ueno (Japan). In addition to the five international folks, there were quite a few locals, including Thailand's currently sole Debian member, Theppitak Karoonboonyanan. The event was hosted at Khon Kaen University and opened with my talk about the Debian Social Contract and the Debian Free Software Guidelines. This was followed by a number of talks about Debian package building, a 3-day BSP where we touched 57 bugs, a great day of sightseeing and talks about i18n, derivative distros, keysigning, mirrors, contribution and a discussion about DebConf. During the week there was also the usual beersigning, combined with eating of unfamiliar and "interesting" Thai snacks. After the conference Andrew and I roamed some markets in Bangkok and got Thai massages. Beforehand I also visited a friend from my travels on the RV Heraclitus in Chiang Mai, once again experiencing the awesomeness of trains in Asia, unfortunately during the dry season this time. I took a lot of photos during my time in Thailand and ate a lot of great and spicy food. As a vegetarian I especially appreciated the organiser's efforts to accommodate this during the conference. At DebConf10 in New York City, by far the highlight was Eben Moglen's vision of the FreedomBox. Negotiating the hot rickety subways was fun, the party at the NYC Resistor space was most excellent, Coney Island was hot and the water a bit yuck, zack threw a ball, the food and campus was really nice. Really enjoyed the lintian BoF, ARM discussions, shy folks, GPLv3 question time, paulproteus' comments & insights, wiki BoF, puppet BoF, derivatives BoF, Sita, astronomy rooftop, cheese, virt BoF, Libravatar, DebConf11, Brave new Multimedia World, bagels for breakfast, CUT, OpenStreetMap & lightning talks. Having my power supply die was not fun at all. Afterwards I hung out with a couple of the exhausted organisers, ate awesome vegan food and fell asleep watching a movie about dreams. One weird thing about DebConf10 was that relatively few folks used the DebConf gallery to host their photos, months later only myself and Aigars Mahinovs posted any photos there. At FOSSASIA 2010 in H Ch Minh City (HCMC) was a mini-DebConf. I arrived at the HCMC airport and was greeted by Huyen (thanks!!), one of FOSSASIA's numerous volunteers, who bundled me into a taxi bound for the speakers accommodation and pre-event meetup at The Spotted Cow Bar. The next day the conference opened at the Raffles International College and after looking at the schedule I noticed that I was to give a talk about Debian that day. Since I didn't volunteer for such a talk and had nothing prepared, the schedule took me by surprise. So shortly after an awesome lunch of Vietnamese pancakes we gathered some Debian folks and a random Fedora dude and prepared a short intro to Debian. The rest of the day the highlights were the intro, video greetings and the fonts, YaCy and HTML5 talks. The next day the Debian MiniConf began with Arne Goetje and everyone trying to get Debian Live LXDE USB keys booted on as many machines in the classroom as possible (many didn't boot). Once people started showing up we kicked off with Thomas Goirand's introduction to the breadth of Debian. Others talked about Debian pure blends, Gnuk and building community and packages. The second last session was about showing the Vietnamese folks in the room how to do l10n and translation since Debian had only one Vietnamese translator (Clytie Siddall). After manually switching keyboard layouts (seems LXDE doesn't have a GUI for that) on the English LXDE installs, the two Cambodian folks were able to do some Khmer translation too. This was a great session and it resulted in two extra Vietnamese translators joining Debian. It went over time so I didn't end up doing my presentation about package reviewing. We rushed off to a university where the random Fedora ch^Wambassador was hosting a Fedora 14 release party in a huge packed classroom. There were a lot of excited faces, interesting and advanced questions and it was in general a success. Afterwards we had some food, joined up with some other speakers and ended up in a bar in the gross tourist zone. On the final day we hung around in the Debian room, went downstairs for the group photo and final goodbyes. Later we found a place with baked goods, coffee and juices and navigated the crazy traffic to a nice local restaurant. The next morning Arne & I went to the airport, others went on a Mekong Delta tour and Jonas hung out with the organisers. I took less photos than at other events but got a few interesting ones. I avoided doing a lot of FLOSS stuff over the last year, I hope to work on some things in the coming months;

• revive various (semi-)abandoned upstream projects
• do some more work on the Debian wiki
• contribute to RC bug squashing efforts
• push the VoIP company patches upstream
• contribute to debexpo/debshots/lintian/qa
• contribute to the Debian multimedia/FSO teams
• promote/organise/rejuvenate the Debian games team
• announce the Debian derivatives census more widely
• finish writing a SANE driver for my scanner
• finish the removal of defoma from Debian
• watch some videos from LCA and DebConf

I'm also planning some interesting travel and acquiring some new technological goods, more on those in some later posts.

Once again, a polemic regarding how to properly integrate the Ruby language and libraries with the Debian distribution has been ignited. Similar arguments were presented in November and December 2008 and September 2010 And excuse me for just refering to my own blog, but there are links to some other posts from there... and there will surely be many others I just missed. There were even nice collaboration attempts (such as DebGem, announced in January 2009... But which apparently never stuck on, as it is still listed as in its public Beta period and lists only support for Debian 4.0 and Ubuntu 8.04 and 8.10). Some days ago, while I was on vacation, I received a mail from Lucas Nussbaum expressing the burnout he had been suffering from this situation. Some days after that, he posted a corresponding blog post - Giving up on Ruby packaging. The comments following his post are most interesting The one comment I'd like to highlight (noting I skimmed a whole deal of them) is by Paul Brannan, one of the original RubyGems authors. Yes, possibly the design criteria for Gems included some mutually-conflicting goals, and they cover some of Debian's goals. Unfortunately, this conflicting criteria... Was resolved to the opposite way we would have wanted (specifically, that a sysadmin should be able to install software with the same tools he was already familiar with). And of course, there are deeper disagreements, which are rightly stemmed from different priorities: Debian (+derivatives) is meant to make user's and sysadmin's lifes simpler. Gems are meant to make developers' lifes simpler. And while all developers are some sort of users, and sysadminsthe reverse is (obviously) not so. A developer maintaining several different codebases will surely suffer (well, this particular developer has surely suffered) if he has to maintain them all using coherent versions of system libraries. When a system is programmed, its developers want to use the greatest, latest tools to offer the best experience and functionality. And it's natural for libraries to evolve over time. However, any sysadmin will grind their teeth at the prospect of having many different versions of libraries, as the Gem model proposes. Why? Well, I was bitten by a minor example of it Bug fixes do not get backported. Of course, if simple bugs are not backported, security and stability of the system as a whole suffers. Often, new software versions will not only add functionality, but improve on how things are done. Not only bugs get corrected, but we get better response times, better memory handling, etc. It might sound harsh to say this, and even more, as a developer it feels I am talking against myself But while development time should be minimized during the system design/implementation, once systems are in production, it should be used to allow for friendlier sysadmin relations. I have to wear both hats at my real-life job. So, what is my take on this? Of course, blaming myself for choosing the wrong version is a no-go. When evaluating a library before using it in my projects, I try to look at its history and API-stability. APIs that change often speak about immature libraries which are still trying to get the right way to implement their functionality; just-started projects might be great and revolutionary, but they do not yet show any kind of long-term committment... And can lead (and have often led me) to painful rewrites when the Next Big Thing is reached. As for the users: Even if your favorite language has the best and friendliest distribution method, users just do not care about what language was a particular piece of software implemented in. Users want to be able to install and uninstall with their system's usual tools. Directly using Gems, CPAN, PEAR and whatnot is just unnatural for users, however convenient they are for developers. Distributions offer a non-technical advantage in this regard as well: A human filter. As an example, as I write this there are 19392 Ruby Gems and 19092 Perl modules in CPAN (note that CPAN stores some older versions but discourages authors from keeping too many - So no, they are still very different in size). Debian has around 30,000 packages. Why is that? Because all Debian packages _must_ be human-generated, human-reviewed, human-submitted. This means, a person must think each packaged piece of code is worth packaging, is stable enough and provides value to users as it is, and is fit for being part of a stable release. I am not saying with this that 90% of CPAN and Gems are crap I am saying that they are probably early implementations, to be installed, tested and improved by developers, and still not apt for general public use. Or maybe not interesting enough to be packaged as a service for the non-techie public at large. Oh, and one last point: Ruby is not a new language anymore. It is a mature, powerful language with different implementations, every time more stable... But it is a language deeply affected by the (not so new anymore either) the appearance of Rails. Why do I say this? Because although the language is in no way tied to Web development, many of its strongest uses are Web-oriented. How does this affect the current discussion? Well, because many people argue that users are no longer needed to install the software. Web systems are installed by sysadmins and used via a Web browser, and sysadmins are expected be more skilled than casual users. Still, in Debian (and in other distributions, surely) we try to make sysadmin's lives simpler I have (again, talking out of personal experience) installed several webapps (and system tools, and whatnot) for which I never followed any instructions besides aptitude install foo Using different languages, frameworks, and so on. Can I troubleshoot their installs? Probably, as there is a common logic for how the distribution I have chosen and specialized in works. Can I find causes for bugs in them? Possibly, although there are some languages and frameworks I dare not touch. Can I get help on getting them out of a tight spot? Surely, as there is a central bug tracking system for my distribution And one of the maintainer's tasks is to separate the problems related to the distribution (packaging, installing, simple user questions and misconceptions) to those derived from real bugs upstream. Anyway I am not saying that our way is the best way. No, by a long shot. Again, developers should have an easy, convenient way for installing whatever they want to play with. And to publish it without jumping through hoops. With this post, I'm only trying to express again why Debian works the way it does. And hope for better cooperation in the future. And as for several comments of what I read in Lucas' post, I think that there is interest for this synergy among some of the most committed Ruby people.

Streaming JSON objects with python Dear Lazyweb, Suppose I'd like to create an HTTP service that gives you a looong list of JSON objects. For example, one JSON object for every package in Debian. One way I could do that is to just transfer a big, dozens of megabytes JSON array with one element per package. With the standard json module in python, this requires the client to buffer all the data in memory for decoding, as the JSON result would be something like this:

[
    info for package 1  ,
    info for package 2  ,
  ...dozens of megabytes...
    info for package 30000  ,
  ...
]

However, it would be reasonable to engineer things so that the client can process packages one at a time, as they are produced. This can dramatically reduce client memory usage, as well as remove a large latency between the start of the request and the start of processing. One trivial idea would be to just send JSON objects one after the other, and sort of call load multiple times. It doesn't work:

import json
from cStringIO import StringIO
a = StringIO()
json.dump( 1:3 , a)
json.dump( 2:4 , a)
a.reset()
print a.getvalue()
print json.load(a)

gives:

$ python z1.py 
 "1": 3 "2": 4 
Traceback (most recent call last):
  File "z1.py", line 8, in <module>
    print json.load(a)
  File "/usr/lib/python2.6/json/__init__.py", line 267, in load
    parse_constant=parse_constant, **kw)
  File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/json/decoder.py", line 322, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 8 - line 1 column 16 (char 8 - 16)

And the json module documentation does not provide any way of telling the "stream" decoder to just stop parsing after the first valid object. An alternative can be to use chunked transfer encoding and transfer each JSON object in a separate HTTP chunk. It would add the overhead of a little HTTP header per JSON object, which could be significant when generating lots of small JSON objects, but it would be an implementation that strictly follows HTTP and JSON standards to the letter in a clean way. Except:

1. How do you stream HTTP chunks in WSGI? The WSGI specification's only mention of chunks is that if you produce a result as a generator of "chunks", they are just sent one after the other (effectively, concatenated) in the response stream.
2. How do you read and process one chunk at a time with urllib/urllib2? I searched on Google but I only found people in tears.

Another alternative is to wrap the JSON objects in some custom container protocol, but then we are not talking standards anymore. Is this all you can do with the standard python library? Can this only be done by adding a dependency on a C library with two obscure separately maintained bindings, or by writing my own JSON parser? Ideas? Please mail enrico@enricozini.org and I'll add them here. Updates Wouter van Heyst mentions that he heard of ijson, which indeed has my exact use case in the synopsis, but after some research looks like a third binding for yajl. Aigars Mahinovs suggests using YAML instead of JSON. Indeed YAML supports streaming very well (it's even already supported in DDE, but last time I tried to do anything serious with it, PyYaml was slooow. Of course, this being the Python ecosystem, There Are Lots Other Ways To Do It And You Shall Be Patronised For Not Chosing The Proper One, but Debian only has PyYaml. Another itch with YAML is that there is no YAML parser in the standard python library, which requires clients to add an extra dependency. Jyrki Pulliainen has so far mailed my favourite solution: disable formatting in the encoder and separate JSON objects with newlines:

we built a streaming JSON app as a part of our platform. However, we
didn't stream a single JSON object as that was a bit cumbersome. When
having syntax like:
[
  <json-obj1>,
  <json-obj2>,
  ....
  <json-objN>
]
The reader has some problems. Should it read the whole stream to find
the closing bracket ] even if the stream would me tens of megabytes?
Should it try to split on , and parse single objects? What if the
objects contain , so should there be some regexp magic?
We ended up streaming separate JSON objects separated by newlines
(\n). This way the reader can read until hitting \n or end of stream
and then split the result on newlines. The new stream looked like
this:
<json-obj1>\n
<json-obj2>\n
...
<json-objN>\n
This way we could efficiently stream hundreds of thousands of JSON
objects from server to client without reserving a load of memory. For
the WSGI part how to do this, the best would probably to do a
generator that would yield a single JSON object with the newline (we
didn't use WSGI back then). Reading can be done using any reader
capable of handling streams. We used pycurl for reading, but I can't
really recommend it as it is way too complicated for a simple task
like this :p
Hopefully this helps!

This means that if the client has no access to a JSON reader that supports a stream of concatenated toplevel objects, they can still trivially code it on top of any JSON library. This means that working clients can be written based just on Squeeze's python standard library. That's what I'm going to do; I'll probably implement it as a separate 'jsons' output format for DDE, because producing a JSON response with multiple toplevel obejcts somehow breaks the JSON standard. Thanks to all who provided this excellent information!

Whilst my philosophy with mobile phones has always been "if it can make and receive calls and SMSes, it's good enough for me", the geek in me occasionally lusts for a smartphone. So I now have an Android phone. Once you have a smartphone, you need a plan that includes data. Once you've got that, you realise that your USB mobile internet dongle is somewhat redundant, as long as you can get your laptop talk through the phone. It turns out, with Android, it's pretty easy. First, root your phone. There are a couple of tethering apps that don't require rooting, but the tradeoff is you end up having to install proprietary drivers in your laptop, and if I'm going to have a security hole somewhere, I prefer it to be in my phone, not my laptop. Rooting an Android phone is trivially easy with Universal Androot. Despite warnings I've seen to the contrary, it didn't even wipe my phone's apps and settings, although that might be a lucky coincidence. Backups, people! Then we need the tethering app -- I use Barnacle Wifi Tether, because it's open source, easy to use, and fully featured. Trivial to install from the market, you just start it up and it's ready to go. It acts as an ad-hoc wireless router, with a DHCP server and NAT to make life easy. In theory it supports WPA and WEP, but that doesn't appear to work for me. If anyone on the train is desperate enough to leech off my wireless, they're welcome to it. The hardest part, ironically, was getting the laptop to connect. Having never played with ad-hoc mode before, it took me a little while to sort out exactly what was supposed to happen, but within 5 minutes I'd worked it out and was pinging The Internet. Win!

It's no secret to the people I around me that over the past two years I've become a Haskell fan, to put it mildly. I mean, what's not to like? Just in the past month:

• a conceptual one-line change (map parMap, but the actual commit is a big longer as it splits a foldl in two) and re-linking with the threaded runtime resulted in the balancing routines used in the ganeti-htools code to achieve between 60% and 80% increase with the number of cores; without any locking primitives, thread synchronisation, etc.; and yet the algorithm is not at all slower when run single-threaded, which is something very hard to do in other, mainstream languages
• a recent test project of mine introduced me to the web framework Yesod, which gives statically-type checked URL construction and referencing, without requiring me to do any value escaping, URL component construction, etc.; comparing that to what I'm used from other languages (though admitedly I don't have too much experience with web development), it's unbelievable how well the type system plays even with the untyped world of the Web applications

The downside is that whenever I feel Ok, now I kind of know my way around Haskell , I learn of another thing that completely resets how I feel about my Haskell understanding. The most recent example is Template Haskell. I mean, there are people who already blame Haskell for its arcane syntax, which I like. But Template Haskell goes a step forward and allows you to use a Haskell library to write Haskell code, with an even more contorted syntax or at least that how it looks for now to my eyes. Gladly this is not easily doable in a recursive way (meta-TH code writing TH code that will write Haskell code), otherwise heads would explode (at least mine). But even though it seems very complicated, I feel that the power that it enables (a macro system that generates code that will be statically checked) is removing one of the last big advantages that (for me) Python had over Haskell: it's easy to do class modification in Python and introspection (e.g. via metaclasses) to automate boiler plate away, but it's of course not type safe (so unittests galore). With TH, you can do the same customization of data types (and not only) and compile-time introspection (called reification) to alter (some) of the properties of the respective type, for example automatic instance derivation, so one less reason to say this particular problem is harder to do in Haskell than in other languages . I haven't yet used TH for my own code and until I actually use it extensively I won't be sure it's actually a good replacement for throw a __metaclass__ in there and automate the boring parts away . But in any case, it looks to be yet another awesome tool in the Haskell programmer's toolbox, which I can't wait to use.