In my professional life I am sometimes required to provide technical support to one of our salesmen. I find this an interesting change in pace though sometimes challenging.

Occasionally I fail to clearly convey the solution we are trying to sell because of my tendency to focus on detail the customer probably does not need to understand but I think is the interesting part of the problem.

Conversely sometimes the sales people gloss over important technology choices which have a deeper impact on the overall solution. I was recently in such a situation where as part of a larger project the subject of internationalisation (you can see why it gets abbreviated to i18n) was raised.

I had little direct personal experience with handling this within a project workflow so could not give any guidance but the salesman recommended the Transifex service as he had seen it used before, indicated integration was simple and we moved onto the next topic.

Unfortunately previous experience tells me that sometime in the near future someone is going to ask me hard technical questions about i18n and possibly how to integrate Transifex into their workflow (or at least give a good estimate on the work required).
LearningBeing an engineer I have few coping strategies available for situations when I do not know how something works. The approach I best know how to employ is to give myself a practical crash course and write up what I learned...so I did.

I proceeded to do all the usual things you do when approaching something unfamiliar (wikipedia, google, colleagues etc.) and got a basic understanding of internationalisation and localisation and how they fit together.

This enabled me to understand that the Transifex workflow proposed only covered the translation part of the problem and that, as Aldrich observed in my title quote, there is an awful lot more to translation than I suspected.
PlatformsMy research indicated that there are numerous translation platforms available for both open source and commercial projects and Transifex is one of many solutions.

Although the specific platform used was Transifex most of these observations apply to all these other platforms. The main lesson though is that all platforms are special snowflakes and once a project invests effort and time into one platform it will result in the dreaded lock in. The effort to move to another platform afterwards is at least as great as the initial implementation.

It became apparent to me that all of these services, regardless of their type, boil down to a very simple data structure. They appear to be a trivial table of Key:Language:Value wrapped in a selection of tools to perform format conversions and interfaces to manipulate the data.

There may be facilities to attach additional metadata to the table such as groupings for specific sets of keys (often referred to as resources) or translator hints to provide context but the fundamental operation is common.

The pseudo workflow is:

• Import a set of keys
• Provide a resource grouping for the keys.
• Import any existing translations for these keys.
• Use the services platform to provide additional translations
• Export the resources in the desired languages.

The Transifex service does at least allow for the Key values to be arbitrary although the resources are separated by language.

Even assuming a project uses gettext PO files and UTF-8 character encoding (and please can we kill every other character encoding and move the whole world to UTF-8) the tools to integrate the import/export into the project must be written.

A project must decide some pretty important policies, including:

• Will they use a single service to provide all their translations.
• Will they allow updates to the files in their revision control system and how those will be integrated.
• Will there be a verification step and if so who and how will that be performed. Especially important is the question of a reviewer understanding the translated language being integrated and how that is controlled.
• Will the project be paying for translations
• Will the project allow machine translations, if not can they be used as an initial hint (sometimes useful if the translators are weak in the "default" language

These are project policy decisions and, as I discovered, just as difficult to answer as the technical challenges.

Armed with my basic understanding it was time to move on and see how the transifex platform could be integrated into a real project workflow.
ImplementingProof of conceptMy first exercise was to take a trivial command line tool, use xgettext to generate a PO file and add the relevant libintl calls to produce gettext internationalised tool.

A transifex project was created and the english po file uploaded as the initial resource. A french language was added and the online editor used to provide translations for some strings. The PO resource file for french was exported and the tool executed with LANGUAGE=fr and the french translation seen.

This proved the trivial workflow was straightforward to implement. It also provided insight into the need to automate the process as the manual website operation would soon become exceptionally tedious and error prone.
Something more usefulTo get a better understanding of a real world workflow I needed a project that:

• Already internationalised but had limited language localisation
• Did not directly use gettext
• Had a code base I understood
• Could be modified reasonably easily.
• Might find the result useful rather than it being a purely academic exercise.

I selected the NetSurf web browser as it best fit this list.

Internally NetSurf keeps all the translated messages in a simple associative array this is serialised to an equally straightforward file named FatMessages. The file is UTF-8 encoded with keys separated from values by a colon. The Key is constrained to be ASCII characters with no colons and is structured as language.toolkit.identifier and is unique on identifier part alone.

This file is processed at build time into a simple identifier:value dictionary for each language and toolkit.

Transifex can import several resource formats similar to this, after experimenting with YAML and Android Resource format I immediately discovered a problem, the services import and export routines were somewhat buggy.

These routines coped ok with simple use cases but having more complex characters such as angle brackets and quotation marks in the translated strings would completely defeat the escaping mechanisms employed by both these formats (through entity escaping in android resource format XML is problematic anyway)

Finally the Java property file format was used (with UTF-8 encoding) which while having bugs in the import and export escaping these could at least be worked around. The existing tool that was used to process the FatMessages file was rewritten to cope with generating different output formats and a second tool to merge the java property format resources.

To create these two tools I enlisted the assistance of my colleague Vivek Dasmohapatra as his Perl language skills exceeded my own. He eventually managed to overcome the format translation issues and produce correct input and output.

I used the Transifex platforms free open source product, created a new project and configured it for free machine translation from the Microsoft service, all of which is pretty clearly documented by Transifex.

Once this was done the messages file was split up tinto resources for the supported languages and uploaded to the transifex system.

I manually marked all the uploaded translations as "verified" and then added a few machine translations to a couple of languages. I also created spanish as a new language and machine translated most of the keys.

The resources for each language were then downloaded and merged and the resulting FatMessages file checked for differences and verified only the changes I expected appeared.

I quickly determined that manually downloading the language resources every time was not going to work with any form of automation, so I wrote a perl script to retrieve the resources automatically (might be useful for other projects too).

Once these tools were written and integrated into the build system I could finally make an evaluation as to how successful this exercise had been.
ConclusionsThe main things I learned from this investigation were:

• Internationalisation has a number of complex areas
• Localisation to a specific locale is more than a mechanical process.
• The majority of platforms and services are oriented around textural language translation
• There is a concentration on the gettext mode of operation in many platforms
• Integration to any of these platforms requires both workflow and technical changes.
• At best tools to integrate existing resources into the selected platform need to be created
• Many project will require format conversion tools, necessitating additional developer time to create.
• The social issues within an open source project may require compromise on the workflow.
• The external platform may offer little benefits beyond a pretty user interface.
• External platforms introduce an external dependency unless the project is prepared and able to run its own platform instance.
• Real people are still required to do the translations and verify them.

Review: Asimov's Science Fiction, July 2011

Editor:	Sheila Williams
Issue:	Volume 35, No. 7
ISSN:	1065-2698
Pages:	112

Williams's editorial is a mildly interesting piece about story titles. Silverberg's column is a more interesting (and rather convincing) rebuttal of the joke that fiction authors are "professional liars," combined with an examination of a fake and fantastic 14th travelogue that (at least in Silverberg's telling) was widely believed at the time. The precis of Silverberg's argument is that lying requires an intent to deceive, which is a property of deceptive memoir writers but not of fiction authors. Di Filippo's review column, as usual, is devoted almost entirely to esoterica, although I was moderately interested to hear of Stableford's continued work on translating early French SF. None of it seems compelling enough to go buy, but good translations of early works seem like a good thing to have in the world. "Day 29" by Chris Beckett: The conceit of this novelette is an interstellar travel system akin to a transporter that allows near-instantaneous travel between worlds. The drawback is that all memories from somewhere between 40 and 29 days before transit up until transit are wiped. The progatonist is a data analyst who is about to travel, and therefore by agency rule is required to stop doing work on day 40 before transmission since he can't be held legally liable for anything he has no recollection of doing. (I would like to say that I find this implausible, since one could always keep records, but it's exactly the sort of ass-covering regulation that a human resources department would come up with.) The premise is quite interesting: what do you do during that period that you're going to forget? Beckett wisely mixes Stephen's current waiting period on the colony world with his diary of his original waiting period on Earth the first time he went through the transmission process, and the latter adds greatly to the reader's appreciation of the weirdness of the forgotten interval. Unfortunately, this is a story more about psychological exploration than about plot, and Stephen just isn't very interesting. The telepathic but possibly nonsentient aliens add weirdness but not much else, and the ending of the story provided little sense of closure or conclusion for me. A good idea, but not the execution I wanted. (5) "Pug" by Theodora Goss: Since I grew up with a pug, I have a soft spot for a story featuring one; sadly, though, this story has insufficient pug in it. This is a quiet fantasy (Asimov's calls it SF, presumably on the basis of parallel worlds and a hypothesized scientific explanation, but it reads like fantasy to me) featuring Victorian girls, including one with a bad heart. They discover a hidden door to other versions of their world and do some minor exploration. There's little or nothing in the way of plot; the story is more of an attempt to capture a mood. It's mildly diverting, but I wish it had gone somewhere more substantial. (5) "Dunyon" by Kristine Kathryn Rusch: A Rusch story is often the highlight of an issue, and this is no exception. The protagonist is the owner of a bar in a space station that's become a combination of a refugee camp and a slum. War and chaos have created desperate people, most of whom are attempting to find some way to resources and get out of the bottom of society. The story is about a rumor: a mythical system named Dunyon that's safe and far away. And it's about how people react to that rumor. There's nothing particularly surprising about the direction the story goes (it's fairly short), but Rusch is always a good storyteller. (7) "The Music of the Sphere" by Norman Spinrad: I've had mixed feelings about Spinrad's fiction (and some of his essays), but I liked this story, despite its implausibility. It's set in the near future, featuring an expert in cetaceans and dolphin perception and a composer obsessed with both loud music and classical musical style. Just from that description, you can probably predict much of the story, but I thought it had some neat ideas about dolphins, whales, and alternate perception and aesthetics. (Note: neat, not necessarily biologically plausible.) Enjoyable. (6) "Bring on the Rain" by Josh Roseman: In a change of pace from the rest of the issue, this is a post-apocalyptic story of caravans of wheeled ships traversing a scorched and ruined landscape in search of weather systems and rain. The feel is of an inverted Waterworld, but with more emphasis on military tactics and cooperating fleets. The transposition of fleet maneuvers to huge ground vehicles adds some extra fun. The plot has little to do with the background and is a fairly stock military adventure scenario, but it's reasonably well-told. The story feels like an excerpt from a larger military-SF-inspired adventure, but the length keeps the quantity of tactics and maneuvering below the threshold where I would get bored. (6) "Twelvers" by Leah Cypess: This is a sharp and occasionally mean story of adolescent cruelty and alienation. Darla is a "twelver," a child who was carried an extra three months in the womb using newly-invented medical technology because of a belief in the advantages this would bring in later life. Unfortunately for all those who used this technique, what it also brought was a preternatural calm and an unusual reaction to emotions. Darla finds it almost impossible to get upset at anything, and that, of course, prompts the cruelty and abuse of other children. Most of the story is a description of that abuse, leading up to Darla stumbling into a nasty solution to her immediate problem. It's all very believable (well, apart from the motivating biology), but I didn't enjoy reading about it, and I'm certainly not convinced that the ending will lead to anything good. (5) "The Messenger" by Bruce McAllister: This is a very short time travel story, where time travel is used to try to unwind old family pain. This world follows the unalterable history model: no changes to the past are possible, and anything you do in the past has already happened. The mechanics are mostly avoided. Instead, McAllister concentrates on his mother, his father, and their complex relationship. I would have needed a bit more background on the characters to care enough about them for the story to be fully effective, but while the heartstring-pulling is kind of obvious, it's still a solid story. (6) "The Copenhagen Interpretation" by Paul Cornell: This is the most ingenious of the stories in this issue. It's set in a future world that extends what seemed to me to be pre-World-War-I great power politics, although there may be a hint of the Cold War. Great nations have reached a careful balance of power, and spies and secret services work to sustain that balance. The progatonist is one of those agents, making use of advanced technology like space folds in the service of a cause that he doesn't entirely believe in. Cornell mixes in mental conditioning, artificial people, space travel, and even aliens (maybe) in a taut thriller plot that, for me, gained a great deal from the unexplained strangeness of its background. If you like diving into the deep end and following a fast-moving plot against a background of strangeness, this is the sort of SF you'll enjoy. (7) Rating: 6 out of 10

My last post I said that I had to remove my internet query tools due to some bugs that were a concern. Some of the code was hard to maintain and probably had holes and I had noticed that it looped at times. I m happy to say that I have restored some of those tools now, still located at http://enc.com.au/itools This code is completely re-written in Python using the TurboGears toolkit which means it is a lot cleaner in how it works and how it looks. Some of the lookup tables use a database rather than an array for ease of updating and querying. The downside is the backends will take time. It currently only does nslookup queries and whois only works for IPv4 addresses. The domain name queries will be a while off as these are the most complicated to handle. To give you an idea, all IPv4 and IPv6 address information comes from 5 sources with two formats while domain names come from over 200 sources with about 40 formats. This means the information from Regional Internet Registries will be done first.

This is a little note for anyone trying to get some debugging out of the puppetmaster when deploying with Foreman. The trick, much as it is, is that Foreman is running puppet via Apache; so if you're trying to start a puppet master daemon outside that it won't be able to bind to port 8140. You thus want to edit the config file Apache is using to launch puppet /etc/puppet/rack/config.ru. It's probably pretty obvious what's happening when you look in there; simply add

ARGV << "--debug"

and you will start to get debugging output. One issue is that this goes to syslog (/var/log/messages) by default and is a lot of output; so much so that it might get throttled. Although you can certainly reconfigure your syslog daemon to split out puppet logs, an easier way is to just skip syslog while you're debugging. Don't be fooled by config options; simply add

ARGV << "--logdest" << "/var/log/puppet/puppet-master.debug"

to the same file to get the logs going to a separate file. Don't forget to restart Apache so the changes stick.

Most OpenPGP User IDs look like this:

Jane Q. Public <jane@example.org>

This is clean, clear, and unambiguous. However, some tools (gpg, enigmail among others) ask the user to provide a "Comment:" field when they are choosing a new User ID (e.g. when making a new key). These UI prompts are evil. The savvy user knows to avoid entering anything in this field, so that they can end up with a User ID like the one above. The user who provides something here (perhaps even something inconsequential like "I like strawberries", due to not being sure what should go in this little box) will instead end up with a User ID like:

Jane Q. Public (I like strawberries) <jane@example.org>

This is bad. This means that Jane is asking the people who certify her key+userid to certify whether she actually likes strawberries (how could they know? what if she changes her mind? should they revoke their certifications?) and anywhere that she is referred to by name will include this mention of strawberries. This is not Jane's identity, and it doesn't belong in an OpenPGP User ID packet. Furthermore, since User IDs are atomic, if Jane wants to change the comment field (but leave her name and e-mail address the same), she will instead need to create a new User ID, publish it, get everyone who has certified her old key+userid to certify the key+newuserid, and then revoke the old one. It is difficult already to help people understand and participate in the certification network that forms that backbone of OpenPGP's so-called "web of trust". These bogus comment fields make an already-difficult task harder. And all because of strawberries! Tools like enigmail and gpg should not expose the "Comment:" field to users who are generating keys or choosing new User IDs. If they feel it absolutely must be present for some weird corner case that 0.1% of their users will have, they could require that the user enters some sort of "expert mode" before prompting the user to do something that is likely to be a mistake. There is almost no legitimate reason for anyone to use this field. Let's go through some examples of this people use, taken from some examples i have lying around (identifying marks have been changed to protect the innocent who were duped by this bad UI choice, but you can probably find them on the public keyserver network if you want to hunt around):

domain repetition

John Q. Public (Debian) <johnqpublic@debian.org>

We know you're with debian already from the @debian.org address. If this is in contrast to your other address (johnqpublic@example.org) so that people know where to send you debian-related e-mail, this is still not necessary. Lest you think i'm just calling out debian developers, people with @ubuntu.com addresses and (Ubuntu) comments (as well as @example.edu addresses and (Example University) comments and @example.com addresses and (Example Corp) comments) are out there too.

nicknames already evident

John Q. Public (Johnny) <johnqpublic@example.net>
John Q. Public (wackydude) <wackydude@example.net>

Again, the information these comments are providing offers no clear disambiguation from the info already contained in the name and e-mail address, and just muddies the water about what the people who certify this identity should actually be trying to verify before they make their certification.

"Work"

John Q. Public (Work) <johnqpublic@example.com>

if John's correspondents know that he works for Example Corp, then "Work" isn't helpful to them, because they already know this as the address that they're writing to him with. If they don't know that, then they probably aren't writing to him at work, so they don't need this comment either. The same problem appears (for example) with literal comments of (School) next to their @example.edu address.

This is my nth try at this crazy system!

John Q. Public (This is my second key) <johnqpublic@example.com>
John Q. Public (This is my primary key) <johnqpublic@example.com>
John Q. Public (No wait really use this one) <johnqpublic@example.com>

OpenPGP is confusing, and it can be tricky to get it right. We all know :) This is still not part of John's identity. If you want to designate a key as your preferred key, keep it up-to-date, get people to certify it, and revoke or expire your old keys. People who care can look at the timestamps on your keys and tell which ones are the most recent ones. You do have a revocation certificate for your key handy just in case you lose it, right?

Don't use this key

John Q. Public (Old key, do not use) <johnqpublic@example.com>
John Q. Public (Please only use this through September 2004) <johnqpublic@example.com>

This kind of sentiment is better expressed by revoking the key in question or setting an expiration time on the key or User ID self-sig directly. This sentiment is not part of John's identity, and shouldn't be included as though it were.

"none"

John Q. Public (none) <johnqpublic@example.com>

sigh. This is clearly someone getting mixed up by the UI.

I use strong crypto!

John Q. Public (3092 bits of RSA) <johnqpublic@example.com>

This comment refers to the strength of the key material, or the algorithms preferred by the user. Since the User ID is associated with the key material already, people who care about this information can get it from the key directly. This is also not part of the user's actual identity.

"no comment"

John Q. Public (no comment) <johnqpublic@example.com>

This is actually not uncommon (some keyservers reply "too many matches!"). It shows that the user is witty and can think on their feet (at least once), but it is still not part of the user's identity.

But wait (i hear you say)! I have a special case that actually is a legitimate use of the comment field that cannot be expressed in OpenPGP in any other way! I'm sure that such cases exist. I've even seen one or two of them. The fact that one or two cases exist does not excuse the fact that that overwhelming majority of these comments in OpenPGP User IDs are a mistake, caused only by bad UI design that prompts people to put something (anything!) in the empty box (or on the command prompt, depending on your preference). And this mistake is one of the thousand papercuts that inhibits the robust growth of the OpenPGP certification network that some people call the "web of trust". Let's avoid them so we can focus on the other 999 papercuts. Please don't use comments in your OpenPGP User ID. And if you make a user interface for OpenPGP that prompts the user to decide on a new User ID, please don't include a prompt for "Comment" unless the user has already certified that they are really and truly a special special snowflake. Thanks!Tags: openpgp, ui

The Debian Edu / Skolelinux project is making great progress and made its second Wheezy based release today. This is the release announcement: New features for Debian Edu 7.0.0 alpha1 released 2013-05-14 This is the release notes for for Debian Edu / Skolelinux 7.0.0 edu alpha1, based on <ahref>Debian with codename "Wheezy". About Debian Edu and Skolelinux Debian Edu, also known as Skolelinux, is a Linux distribution based on Debian providing an out-of-the box environment of a completely configured school network. Immediatly after installation a school server running all services needed for a school network is set up just waiting for users and machines being added via GOsa , a comfortable Web-UI. A netbooting environment is prepared using PXE, so after initial installation of the main server from CD, DVD or USB stick all other machines can be installed via the network. This is the first test release based on Wheezy (which currently is not released yet). Basically this is an updated and slightly improved version compared to the Squeeze release. Software updates

• Install freemind (0.9.0) by default, and stop installing vym by default.
• Install chromium (26.0.1410.43) by default.
• Install goplay (0.5-1.1) to make golearn available by default.
• Updated support for Japanese input methods, now based on ibus-anthy.

Other changes

• Switched default file system from ext3 to ext4 for speed and reliability improvements.
• Got rid of unwanted winbind daemon and PAM setup activated because of 706434.
• Extended and improved the testsuite tests to detect more possible problems.
• Corrected proxy handling to not set http_proxy to a bogus direct:// URL.
• Corrected proxy setup for diskless workstations.
• Corrected PXE setup to use our updated udebs during installation.
• Made installation handling of low entropy level more robust.
• Create larger partitions for Roaming workstations and Thin client servers, to make room for all the software installed.
• Fix bug in Roaming workstation PAM setup, making it impossible to log in (706753).

Known issues

• IP resolution for the local hostname give useless IPv6 address (705900). Only install libnss-myhostname on roaming workstations until it is fixed.
• DVD images are not yet ready.
• No mass import of user account data in GOsa (ldif or csv) available yet (698840).
• Missing artwork for the KDE desktop (and probably a few others).
• KDE Debian submenu lacks icons.
• LXDE menu lacks entry for changing GOsa password (website). Installing gosa-desktop will be an option.
• Backup configuration via web interface is impossible due to password submission problem (700257).

Where to get it To download the multiarch netinstall CD release you can use

• ftp://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu~7.0+edu0~a1-CD.iso
• http://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu~7.0+edu0~a1-CD.iso
• rsync -avzP ftp.skolelinux.org::skolelinux-cd/wheezy/debian-edu~7.0+edu0~a1-CD.iso debian-edu~7.0+edu0~a1-CD.iso

The MD5SUM of this image is: 685ed76c1aa8e44b12d3fde21faf450b The SHA1SUM of this image is: 6c874de157024da13e115bab29c068080a11ec4c How to report bugs http://wiki.debian.org/DebianEdu/HowTo/ReportBugs

Debian wheezy has been released (yay!), so I've updated my backport script to backport to wheezy by default and shuffled the meanings of stable and oldstable. The whole script badly needs a rewrite and needs to become more configuration-driven, but I sadly don't have the time at the moment, so will have to make do with this. If anyone else is using it, you can get the latest copy from my scripts page. Also done: suite names changed for local Stanford repositories. jessie added to our local Debian mirroring. reprepro pull rules changed accordingly. All local build chroots updated, with new ones created for wheezy and wheezy-backports. Still to do: update suite names and pull rules for the eyrie.org Debian repository (which isn't used much any more). Delete the old per-service lenny-based distributions, since we've gotten everything off of lenny that cared about them. Add a new jessie build chroot to our local build servers. Update our FAI installation to build wheezy by default and to use a wheezy NFS root. reprepro makes this whole process so massively easier than it was with debarchiver.

Today my main machine was down for about 8 hours. Oops. That meant when I got home, after a long and dull train journey, I received a bunch of mails from various hosts each saying:

• Failed to fetch slaughter policies from rsync://www.steve.org.uk/slaughter

Slaughter is my sysadmin utility which pulls policies/recipies from a central location and applies them to the local host. Slaughter has a bunch of different transports, which are the means by which policies and files are transferred from the remote "central host" to the local machine. Since git is supported I've now switched my policies to be fetched from the master github repository. This means:

• All my servers need git installed. Which was already the case.
• I can run one less service on my main box.
• We now have a contest: Is my box more reliable than github?

In other news I've fettled with lumail a bit this week, but I'm basically doing nothing until I've pondered my way out of the hole I've dug myself into. Like mutt lumail has the notion of "limiting" the display of things:

• Show all maildirs.
• Show all maildirs with new mail in them.
• Show all maildirs that match a pattern.
• Show all messages in the currently selected folder(s)
  • More than one folder may be selected :)
• Shall all unread messages in the currently selected folder(s).

Unfortunately the latter has caused an annoying, and anticipated, failure case. If you open a folder and cause it to only show unread messages all looks good. Until you read a message. At which point it is no longer allowed to be displayed, so it disappears. Since you were reading a message the next one is opened instead. WHich then becomes marked as read, and no longer should be displayed, because we've said "show me new/unread-only messages please". The net result is if you show only unread messages and make the mistake of reading one .. you quickly cycle through reading all of them, and are left with an empty display. As each message in turn is opened, read, and marked as non-new. There are solutions, one of which I documented on the issue. But this has a bad side-effect that message navigation is suddenly complicated in ways that are annoying. For the moment I'm mulling the problem over and I will only make trivial cleanup changes until I've got my head back in the game and a good solution that won't cause me more pain.

There's some exploit code going around that will let you get root on a range of Linux kernel versions by using a bug in the perf_event_open() syscall. The fix for this is in 3.2.45 and various other stable updates. As a workaround, until it's in Debian stable, you can set sysctl kernel.perf_event_paranoid=2. This blocks the exploit I've seen, but it is still possible to get around this restriction. Red Hat has a SystemTap script that should provide more complete mitigation. The debuginfo packages for Debian kernels are named e.g. linux-image-3.2.0-4-amd64-dbg. They are only provided for some architectures and flavours. Updated: Added the CVE ID. Added SystemTap information.

A week ago, I made the big step and presented my resignation letter at Google. It was not an easy decision, to leave a good job to pursue a blurry plan that sounds a bit infeasible, but I feel this what I want to do: it's a dream becoming reality. After the 31st of May, I will become self-employed, working as a freelancer, while travelling around the world. I plan to live with a small budget, working with my laptop from wherever I am, instead of stressing about getting many clients to keep an expensive lifestyle. I've had the travelling bug for some time, always thinking about my next trip, leaving for the airport just after finishing work, coming back on Monday and going directly to the office. You end up wishing for more vacation days all the time (and I had a fair amount of them). Now, for different reasons I want to spend some time in my old house in Nice, and in Argentina. There was no way I could do that with my current job, and that was the trigger for my decision. After that, I will come back to Ireland, just to think where my next destination will be. I know this is going to be a great experience, we'll see how well it works! If you think you -or your employer- might need my services, I'd be more than happy to talk! I'll be concentrating on the kind of work I've been doing at Google and before: finding creative solutions for difficult problems, be it systems administration, or (systems) programming. Think of hiring an SRE for just a few hours or days.

I've just released qcontrol 0.5.1. Changes since the last release:

• Add build targets to enable static linking (Original patch by Frans Pop).
• Wake-on-Lan and EUP control (by Michael Stapelberg, Debian bug #703888).
• Updated example configurations (based on Debian package).
• Support loading configuration snippets from a directory (Ian Campbell, Debian bug #697574).
• Various other bug fixes.

I also put together a very basic homepage. Get it from gitorious or http://www.hellion.org.uk/qcontrol/releases/0.5.1/. The Debian package will be uploaded shortly.

I was asked by some people to write a status report about the whole PK/AS/LI stuff sorry guys that it took so much time to write it . PackageKit (PackageKit is an abstraction layer for package-management systems, allowing applications to access the package-manager using simple DBus calls) PackageKit is an incredibly successful project. With the 0.8.x series, it received many performance improvements, and has now the same speed on my computer than the distribution s native tools. PackageKit is used in almost all major Linux distributions, except for Ubuntu[1]. But even Ubuntu has written some compatibility layer, so most calls to PackageKit will work. The only major distro where PackageKit is currently not available, seems to be Gentoo (and I am not sure about the shape of the Gentoo PackageKit backend too). Debian Wheezy includes PackageKit by default, and in Jessy we are going to replace some distribution-specific tools with PackageKit frontends (mostly the old and unmaintained update-notifier and Software-Updater no worries, we are not going for a Synaptic replacement (currently this won t be possible with PK anyway)) Unfortunately, some PackageKit backends are still not adjusted for the 0.8.x API and are only running on 0.7.x. This is bad, since 0.8.x is a huge step forward for PackageKit. But the situation is slowly improving, with the latest OpenSUSE release, the Zypper backend is now available on 0.8.x too. Being able to run a PackageKit from the 0.8.x series is a requirement for both AppStream and Listaller. AppStream (AppStream is a cross-distro effort for building Software-Center-like applications. It contains stuff like a screenshot-service, ratings&reviews etc. The most important component is a Xapian database, storing information about all available applications in the distribution s repositories. The Xapian DB is distro-agnostik, but distributors need to provide data to fill it. AppStream offers an application-centric way to look at a package database) The AppStream side doesn t look incredibly great, but the situation is improving. As far as I know, OpenSUSE is shipping AppStream XML to generate the database. Ubuntu ships the desktop-files, and I am working on AppStream support in Debian s Archive Kit. On the Fedora side, negotiations with the infrastructure-team are still going on. I haven t heard anything from Mageia and other AppStream participants yet. Unfortunately, at least for OpenSUSE, the AppStream efforts seem to be stalled, due to people having moved to different tasks. But efforts to add the remaining missing things exist. On the software side, Apper (KDE PackageKit frontend) has full support for AppStream. Apper just needs to be compiled with some extra flags to make it use the AppStream database. On the GNOME-side, GNOME-Software is being developed. The tool will make use of the AppStream database, on distributions where it is available. Also, a Software-Center for Elementary and other GTK+-based desktops is being developed, which is based on AppStream (already quite usable!). Using the Ubuntu Software Center on not-Ubuntu-based distributions ist still not much fun, but with the AppStream database available and a working PackageKit 0.8.x with a backend which supports parallel transactions, it is possible to use it. On the infrastructure side: I recently landed some patches in AppStream-Core, which will improve the search function a lot. AppStream-Core contains all tools necessary to generate the AppStream database. It also contains libappstream, a GObject-based library which can be used to access the AppStream database. Also, we discuss dropping PackageKit s internal desktop-file-cache in favour of using the AppStream database. If we do that, we will also add software descriptions to the AppStream db, to improve search results and to speed up search for applications. Because we have to deprecate API for that, I expect this change to happen with PackageKit 0.9.x. As soon as the Freedesktop-Wiki is alive again and my account is re-enabled, I will create compatibility-list, showing which distribution implements what of the PK/AS/LI stuff, especially focusing on components needed for AppStream. Only a few distributions package AppStream-Core so far. Although it is beta-software, creating packages for it and shipping the required data to generate the AppStream database would be a very huge step forward. Listaller (Listaller is a cross-distro 3rd-party software installer, which integrates into PackageKit and AppStream. It allows installing 3rd-party applications, which are not part of the distributor s repositories, using standard tools used also for native-package handling. Everything which uses PackageKit can make use of Listaller packages too. Listaller also allows sandboxing of new applications, and uses an AppDir-like approach for installing software.) Listaller is currently undergoing it s last transition before a release with stable API and specifications can be made. Dependency solving will be improved a lot during the current release-cycle, making it less powerful, but working on all distributions instead. (Fedora always had an advantage in dependency-solving, due to RPM providing more package metadata for Listaller to use) This change was delayed due to discussing a possible use of ZeroInstall-feeds to provide missing dependencies with the ZeroInstall team. We did not come to a conclusion about extending the XML, so Listaller will contain an own format to define a dependency, which can reference a ZeroInstall feed. That should be a good solution for everyone. All these changes will result in IPK1.2, a new version of the IPK spec with small changes in the pkoptions file syntax and huge changes in dependency-handling. The new code is slowly stabilizing in a separate branch, and will soon be merged into master. The next Listaller release will be the last one of the 0.5.x series, we will start 0.6.x then. KDE currently has support for Listaller through Apper, which is enabled on a few distributions. In GNOME, optional Listaller support is being developed and will be available in one of the upcoming releases. Currently, to my knowledge, only a few distributions package Listaller. This should improve, so it is easier for application developers to deploy IPK packages. The upcoming changes in KDE and GNOME to build stable developer platforms will help Listaller a lot in finding matching dependencies, and for stuff which only depends on one software frameworks, installations should be a matter of seconds. As you can see, lots of things are happening, and there is improvement in all components related to installing and presenting software on Linux machines. However, all these projects have a severe lack of manpower, especially AppStream and Listaller have the lowest number of developers working on the tools (at time, only two active developers). This is the main reason for the slow development. But I am confident that we will have something shipped in the next distribution releases. At least AppStream should be ready then. [1]: I don t blame Ubuntu for that during the time they wrote an own solution, PackageKit did not have all the required features. (This situation has changed now, fortunately.) NOTE: I might extend this post with feedback from the different distributions, as soon as I get it.

I ve recently been spending a bit of my spare time playing Ingress (see the Wikipedia page if you haven t heard of it). A quick summary is that Ingress is an Android phone game that involves geo-location of portals that you aim to control and most operations on a portal can only be performed when you are within 40 meters so you do a lot of travelling to get to portals at various locations. One reasonably common operation that can be performed remotely is recharging a portal by using it s key, after playing for a while you end up with a collection of keys which can be difficult to manage. Until recently the set of portal keys was ordered alphabetically. This isn t particularly useful given the fact that portal names are made up by random people who photograph things that they consider to be landmarks. If people tried to use a consistent geographic naming system (which was short enough to fit in large print on a phone display) then it would be really difficult to make it usable. But as joke names are accepted there s just no benefit in having a sort by name. A recent update to the Ingress client (the program which runs on the Android phone and is used for all game operations) changed the sort order to be by distance. This makes it really easy to see the portals which are near you (which is really useful) but also means that the order changes whenever you move which isn t such a good idea for use on a mobile phone. It s quite common for Ingress players to recharge portals while on public transport. But with the new Ingress client the list order will change as you move so anyone who does recharging on a train will find the order of the list changing during the process and it s really difficult to find items in a list which is in a different order each time you look at it. This problem of ordering by location has a much greater scope than Ingress. One example is collections of GPS tagged photographs, it wouldn t make any sense to mix the pictures of two different sets of holiday pictures because they were both taken in countries that are the same distance from my current location (as the current Ingress algorithm would do). It seems to me that the best way of sorting geo-tagged items (Ingress portals, photos, etc) is to base it on the distance from a fixed point which the user can select. It could default to the user s current location but in that case the order of the list should remain unchanged at least until the user returns to the main menu and I think it would be ideal for the order to remain unchanged until the user requests it. I think that most Ingress players would agree with me that fixing annoying mis-features of the Ingress client such as this one would be better for the game than adding new features. While most computer games have some degree of make-work (in almost every case a computer could do things better than a person) I don t think that finding things in a changing list should be part of the make-work. Also it would be nice if Google released some code for doing this properly to reduce the incidence of other developers implementing the same mistakes as the Ingress developers in this regard.

One of my recent innocent tweet about Gerrit vs Github triggered much more reponses and debate that I expected it to. I realize that it might be worth explaining a bit what I meant, in a text longer than 140 characters.

I'm having a hard time now contributing to projects not using Gerrit. Github isn't that good. Julien Danjou (@juldanjou) May 8, 2013

<script async="async" charset="utf-8" src="http://platform.twitter.com/widgets.js"></script> The problems with Github pull-requests I always looked at Github from a distant eye, mainly because I always disliked their pull-request handling, and saw no value in the social hype it brings. Why? One click away isn't one click effort The pull-request system looks like an incredible easy way to contribute to any project hosted on Github. You're a click away to send your contribution to any software. But the problem is that any worthy contribution isn't an effort of a single click. Doing any proper and useful contribution to a software is never done right the first time. There's a dance you will have to play. A slowly rhythmed back and forth between you and the software maintainer or team. You'll have to dance it until your contribution is correct and can be merged. But as a software maintainer, not everybody is going to follow you on this choregraphy, and you'll end up with pull-request you'll never get finished unless you wrap things up yourself. So the gain in pull-requests here, isn't really bigger than a good bug report in most cases. This is where the social argument of Github isn't anymore. As soon as you're talking about projects bigger than a color theme for your favorite text editor, this feature is overrated. Contribution rework If you're lucky enough, your contributor will play along and follow you on this pull-request review process. You'll make suggestions, he will listen and will modify his pull-request to follow your advice. At this point, there's two technics he can use to please you. Technic #1: the Topping Github's pull-requests invite you to send an entire branch, eclipsing the fact that it is composed of several commits. The problem is that a lot of one-click-away contributors do not masterize Git and/or do not make efforts to build a logical patchset, and nothing warns them that their branch history is wrong. So they tend to change stuff around, commit, make a mistake, commit, fix this mistake, commit, etc. This kind of branch is composed of the whole brain's construction process of your contributor, and is a real pain to review. To the point I quite often give up. <figure> <figcaption> A typical case: 3 commits to build a 4 lines long file. </figcaption> </figure> Without Github, the old method that all software used, and that many software still use (e.g. Linux), is to send a patch set over e-mail (or any other medium like Gerrit). This method has one positive effect, that it forces the contributor to acknowledge the list of commits he is going to publish. So, if the contributor he has fixup commits in his history, they are going to be seen as first class citizen. And nobody is going to want to see that, neither your contributor, nor the software maintainers. Therefore, such a system tend to push contributors to write atomic, logical and self-contained patchset that can be more easily reviewed. Technic #2: the History Rewriter This is actually the good way to build a working and logical patchset using Git. Rewriting history and amending problematic patches using the famous git rebase --interactive trick. The problem is that if your contributor does this and then repush the branch composing your pull-request to Github, you will both lose the previous review done, each time. There's no history on the different versions of the branch that has been pushed. In the old alternative system like e-mail, no information is lost when reworked patches are resent, obviously. This is far better because it eases the following of the iterative discussions that the patch triggered. Of course, it would be possible for Github to enhance this and fix it, but currently it doesn't handle this use case correctly.. <figure> <figcaption> Exercise for the doubtful readers: good luck finding all revisions of my patch in the pull-request #157 of Hy.</figcaption> </figure> A quick look at OpenStack workflow It's not a secret for anyone that I've been contributing to OpenStack as a daily routine for the last 18 months. The more I contribute, the more I like the contribution workflow and process. It's already well and longly described on the wiki, so I'll summarize here my view and what I like about it. Gerrit To send a contribution to any OpenStack project, you need to pass via Gerrit. This is way simpler than doing a pull-request on Github actually, all you have to do is do your commit(s), and type git review. That's it. Your patch will be pushed to Gerrit and available for review. Gerrit allows other developers to review your patch, add comments anywhere on it, and score your patch up or down. You can build any rule you want for the score needed for a patch to be merged; OpenStack requires one positive scoring from two core developers before the patch is merged. Until a patch is validated, it can be reworked and amended locally using Git, and then resent using git review again. That simple. The historic and the different version of the patches are available, with the whole comments. Gerrit doesn't lose any historic information on your workflow. Finally, you'll notice that this is actually the same kind of workflow projects use when they work by patch sent over e-mail. Gerrit just build a single place to regroup and keep track of patchsets, which is really handy. It's also much easier for people to actually send patch using a command line tool than their MUA or git send-email. Gate testing Testing is mandatory for any patch sent to OpenStack. Unit tests and functionnals test are run for each version of each patch of the patchset sent. And until your patch passes all tests, it will be impossible to merge it. Yes, this implies that all patches in a patchset must be working commits and can be merged on their own, without the entire patchset going in! With such a restricution, it's impossible to have "fixup commits" merged in your project and pollute the history and the testability of the project. Once your patch is validated by core developers, the system checks that there is not any merge conflicts. If there's not, tests are re-run, since the branch you are pushing to might have changed, and if everything's fine, the patch is merged. This is an uncredible force for the quality of the project. This implies that no broken patchset can ever sneak in, and that the project pass always all tests. Conclusion: accessibility vs code review In the end, I think that one of the key of any development process, which is code review, is not well covered by Github pull-request system. It is, along with history integrity, damaged by the goal of making contributions easier. Choosing between these features is probably a trade-off that each project should do carefully, considering what are its core goals and the quality of code it want to reach. I tend to find that OpenStack found one of the best trade-off available using Gerrit and plugging testing automation via Jenkins on it, and I would probably recommend it for any project taking seriously code reviews and testing.

This post was written with Andr s Monroy-Hern ndez. It is a summary of a paper just published in American Behavioral Scientist. You can also read the full paper: The remixing dilemma: The trade-off between generativity and originality. It is part of a series of papers I have written with Monroy-Hern ndez using data from Scratch. You can find the others on my academic website.

Remixing the reworking and recombination of existing creative artifacts represents a widespread, important, and controversial form of social creativity online. Proponents of remix culture often speak of remixing in terms of rich ecosystems where creative works are novel and highly generative. However, examples like this can be difficult to find. Although there is a steady stream of media being shared freely on the web, only a tiny fraction of these projects are remixed even once. On top of this, many remixes are not very different from the works they are built upon. Why is some content more attractive to remixers? Why are some projects remixed in deeper and more transformative ways?

We try to shed light on both of these questions using data from Scratch a large online remixing community. Although we find support for several popular theories, we also present evidence in support of a persistent trade-off that has broad practical and theoretical implications. In what we call the remixing dilemma, we suggest that characteristics of projects that are associated with higher rates of remixing are also associated with simpler and less transformative types of derivatives.

Our study is focused on two interrelated research questions. First, we ask why some projects shared in remixing communities are more or less generative than others. Generativity a term we borrow from Jonathan Zittrain describes creative works that are likely to inspire follow-on work. Several scholars have offered suggestions for why some creative works might be more generative than others. We focus on three central theories:

1. Projects that are moderately complicated are more generative. The free and open source software motto release early and release often suggests that simple projects will offer more obvious opportunities for contribution than more polished projects. That said, projects that are extremely simple (e.g., completely blank slates) may also uninspiring to would-be contributors.
2. Projects by prominent creators are more generative. The reasoning for this claim comes from the suggestion that remixing can act as a form of cultural conversation and that the work of popular creators can act like a common medium or language.
3. Projects that are remixes themselves are more generative. The reasoning for this final claim comes from the idea that remixing thrives through the accumulation of contributions from groups of people building on each other s work.

Our second question focuses on the originality of remixes and asks when more or less transformative remixing occurs. For example, highly generative projects may be less exciting if the projects produced based on them are all near-identical copies of antecedent projects. For a series of reasons including the fact that increased generativity might come by attracting less interested, skilled, or motivated individuals we suggest that each of the factors associated with generativity will also be associated with less original forms of remixing. We call this trade-off the remixing dilemma. We answer both of our research questions using a detailed dataset from Scratch, where young people build, share, and collaborate on interactive animations and video games. The community was built to support users of the Scratch programming environment, a desktop application with functionality similar to Flash created by the Lifelong Kindergarten Group at the MIT Media Lab. Scratch is designed to allow users to build projects by integrating images, music, sound, and other media with programming code. Scratch is used by more than a million users, most of them under 18 years old. To test our three theories about generativity, we measure whether or not, as well as how many times, Scratch projects were remixed in a dataset that includes every shared project. Although Scratch is designed as a remixing community, only around one tenth of all Scratch projects are ever remixed. Because more popular projects are remixed more frequently simply because of exposure, we control for the number of times each project is viewed. Our analysis shows at least some support for all three theories of generativity described above. (1) Projects with moderate amounts of code are remixed more often than either very simple or very complex projects. (2) Projects by more prominent creators are more generative. (3) Remixes are more likely to attract remixers than de novo projects. To test our theory that there is a trade-off between generativity and originality, we build a dataset that includes every Scratch remix and its antecedent. For each pair, we construct a measure of originality by comparing the remix to its antecedent and computing an edit distance (a concept we borrow from software engineering) to determine how much the projects differ. We find strong evidence of a trade-off: (1) Projects of moderate complexity are remixed more lightly than more complicated projects. (2) Projects by more prominent creators tend to be remixed in less transformative ways. (3) Cumulative remixing tends to be associated with shallower and less transformative derivatives. That said, our support for (1) is qualified in that we do not find evidence of the increased originality for the simplest projects as our theory predicted. We feel that our results raise difficult but important challenges, especially for the designers of social media systems. For example, many social media sites track and display user prominence with leaderboards or lists of aggregate views. This technique may lead to increased generativity by emphasizing and highlighting creator prominence. That said, it may also lead to a decrease in originality of the remixes elicited. Our results regarding the relationship of complexity to generativity and originality of remixes suggest that supporting increased complexity, at least for most projects, may have fewer drawbacks. As supporters and advocates of remixing, we feel that although highly generative works that lead to highly original derivatives may be rare and difficult for system designers to support, understanding remixing dynamics and encouraging these rare projects remain a worthwhile and important goal. Benjamin Mako Hill, Massachusetts Institute of Technology
Andr s Monroy-Hern ndez, Microsoft Research For more, see our full paper, The remixing dilemma: The trade-off between generativity and originality. Published in American Behavioral Scientist. 57-5, Pp. 643 663. (Official Link, Pay-Walled ).

One more step at being the universal operating system: getting Debian to space:

Specifically, the International Space Station astronauts will be using computers running Debian 6.

Filed under: Debian GNU/Linux