Collusion is "an experimental add-on for Firefox and allows you to see all the third parties that are tracking your movements across the Web."
I am very eager to see how this evolves given that things like Adsense, no less, belong to Google, and Google (i believe, and please correct me if I am wrong), is one of Mozilla's biggest, erm, how to put it, "sponsors". Will Chrome get a similar feature? I am very veruy curious. This might be the biggest White Rabbit I've seen in a long time.

Four and a half years after Sam Hocevar started a draft proposal on the Debian Wiki, we've published revision 1.0 of the machine-readable format for debian/copyright files. The format (as DEP-5) has been in widespread use for some time, but there have been multiple versions of the format and multiple transitional versioned URLs. Everyone can now update to the same version of the document and use a stable format URL. Below is the text of the annoucement that I just sent to debian-devel-announce. Version 1.0 of the machine-readable format for debian/copyright files (the culmination of the DEP-5 process) has now been published. The canonical URL (and also the URL to use in the Format field in such files) is:

http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/

For those who have adopted various iterations of the DEP-5 format for your packages, please consider updating to this format the next time you revise your package. Use of this format in Debian packages is optional. Thanks to all of the people who have contributed to this work over its long evolution. Particular thanks go to Steve Langasek, Charles Plessy, and Lars Wirzenius, who kept this document moving through its extended DEP discussion process, and Sam Hocevar, who started this work in 2007. The copyright format specification is now being maintained as part of the debian-policy package and will follow the same update process that's used for other parts of Policy. See:

http://wiki.debian.org/PolicyChangesProcess

for that process. For this specification, we're trying something new for Debian technical policies. This document is versioned, and older versions of the specification will be maintained to not invalidate older references in copyright files that have not been moved to newer versions. This is similar to the way that specifications are handled by some other projects, particularly the freedesktop.org standards. The plan, going forward, is that informative changes (changes that do not change the requirements or the contents of compliant copyright files) will be made following the same relaxed procedure as other informative changes to Policy, and new versions will only informative changes will be published with the same version number. So the existing 1.0 document may receive further improvements in wording, examples, or similar changes that don't modify its meaning. Normative changes (changes that change requirements for compliant files, which includes addition of new standardized license short names) will follow the normative Policy change process, including formal seconds. Once normative changes are accepted, the next release of this specification will receive a new version number (1.1, for example). The previous version, prior to the normative changes, will be frozen, and from that point forward will not be changed except as required to continue to publish it. This includes further informative changes. We will continue publishing these frozen older versions of the standard on www.debian.org and in the debian-policy package for some time: possibly forever, and at least as long as they're still referred to by packages distributed by the project. The hope is that this will provide a good balance between specification stability, editing for clarity, and specification changes to meet changing requirements, while not invalidating the URLs in older files and while continuing to provide the information required to interpret older files.

As many would know, Debian GNU/Linux is one of the oldest, and the largest Linux distributions that is available for free. Since it was first released in 1993, several people have analysed the size and produced cost estimates for the project. In 2001, Jes s M. Gonz lez-Barahona et al produced an article entitled Counting Potatoes , an analysis of Debian 2.2 (code named Potato). When Potato was released in June 2003, it contained 2,800 source packages of software, totalling around 55 million lines of source code. When using David A. Wheeler s sloccount tool to apply the COCOMO model of development, and an average developer salary of US$56,000, the projected development cost that Gonz lez-Barahona calculated to start-from-scratch and build Debian 2.2 in 2003 was US$1.9 billion. In 2007 an analysis entitled Macro-level software evolution: a case study of a large software compilation by Jes s M. Gonz lez-Barahona, Gregorio Robles, Martin Michlmayr, Juan Jos Amor and Daniel M. German was released. It found that Debian 4.0 (codename Etch released April 2007) had just over 10,000 source packages of software and 288 million lines of source code. This analysis also delved into the dependencies of software packages, and the update flow between Debian release (not all packages are updated with each release). Today (February 2012) the current development version of Debian, codenamed Wheezy, contains some 17,141 source packages of software, but as it s still in development this number may change over the coming months. I analysied the source code in Wheezy, looking at the content from the original software that Debian distributes from its upstream authors without including the additional patches that Debian Developers apply to this software, or the package management scripts (used to install, configure and de-install packages). One might argue that these patches and configuration scripts are the added value of Debian, however the in my analysis I only examined the pristine upstream source code. By using David A Wheeler s sloccount tool and average wage of a developer of US$72,533 (using median estimates from Salary.com and PayScale.com for 2011) I summed the individual results to find a total of 419,776,604 source lines of code for the pristine upstream sources, in 31 programming languages including 429 lines of Cobol and 1933 lines of Modula3! In my analysis the projected cost of producing Debian Wheezy in February 2012 is US$19,070,177,727 (AU$17.7B, EUR 14.4B, GBP 12.11B), making each package s upstream source code wrth an average of US$1,112,547.56 (AU$837K) to produce. Impressively, this is all free (of cost). Zooming in on the Linux Kernel In 2004 David A. Wheeler did a cost analysis of the Linux Kernel project by itself. He found 4,000,000 source lines of code (SLOC), and a projected cost between US$175M and US$611M depending on the complexity rating of the software. Within my analysis above, I used the standard (default) complexity with the adjusted salary for 2011 (US$72K), and deducted that Kernel version 3.1.8 with almost 10,000,000 lines of source code would be worth US$540M at standard complexity, or US$1,877M when rated as complex . Another Kernel Costing in 2011 put this figure at US$3 billion, so perhaps there s some more variance in here to play with. Individual Projects Other highlights by project included:

Project	Version	Thousands of SLOC	Projected cost at US$72,533/developer/year
Samba	3.6.1	2,000	US$101 (AU$93M)
Apache	2.2.9	693	US$33.5M (AU$31M)
MySQL	5.5.17	1,200	US$64.2M (AU$59.7M)
Perl	5.14.2	669	US$32.3M (AU$30M)
PHP	5.3.9	693	US$33.5M (AU$31.1M)
Bind	9.7.3	319	US$14.8M (AU$13.8M)
Moodle	1.9.9	396	US$18.6M (AU$17.3M)
Dasher	4.11	109	US$4.8M (AU$4.4M)
DVSwitch	0.8.3.6	6	US$250K (AU$232K)

Debian Wheezy by Programming Language The upstream code that Debian distributes is written in many different languages. ANSI C with 168,536,758 is the dominant language (40% of all lines), followed by C++ at 83,187,329 (20%) and Java with 34,698,990 (8%). If you are intersted in finding the line count and cost projections for any of the 17,000+ projects, you will find them in the raw data CSV. Other Tools and Comparisons Ohcount is another source code cost analysis tool. In March 2011 Ohcount was run across Debian Sid: its results are here. In comparison, its results appear much lower than the sloccount tool. There s also the Ohloh.net Debian Estimate which only finds 55 Million source lines of code and a projected cost of US$1B. However Ohloh uses Ohcount for its estimates, and seems to be to be around 370 million SLOC missing compared to my recent analysis. Summary Over the last 10 years the cost to develop Debian has increased ten-fold. It s intersting to know that US$19 billion of software is available to use, review, extend, and share, for the bargain price of $0. If we were to add in Debian patches and install scripts then this projected figure would increase. If only more organisations would realise the potential they have before them. Need help with Linux (including Debian), Perl, or AWS? See www.jamesbromberger.com.

You know ACTA and related topics have reached the mainstream when the German news show devotes a segment of several minutes(!) to ACTA, complete with background information about the sleazy backroom dealings, and the various political sunday talk shows debate controversially about it, as well. In a round of five 60-plus aged people, only one defenced ACTA; the rest was sceptic or dead against it. The several tens of thousands of people who walked the streets of Germany yesterday (I couldn't; I am ill) really managed to stir up Germany's political landscape. This. is. awesome. Now if only the population at large would understand the inherent differences between physical counterfeiting, physical theft, and copyright infrigement. That's where we all come in: Inform people about the background and actual issues. Now, more than ever, there's a chance that they will be willing to listen.

If you met Ana, you ll easily remember her. She has a great and pronounced Spanish accent I m glad that the existence of the Debian Women project helped her to join Debian because she has been doing a great job. From KDE packaging to publicity/marketing work, her interests shifted over the years but this allowed her to stay very involved. As she explains it very well, Debian is big enough so that you can stop doing something which is no longer fun for you, and still find something new to do in another part of Debian! Read on to learn more about Ana, the KDE team, Debian s participation to the Google Summer of Code, and more. Raphael: Who are you? Ana: I m Ana Guerrero L pez and I m in my early 30s. I was born and raised in the wonderful city of Sevilla, Spain and I live in Lyon, France. I share my life with another Debian Developer and my paid work is doing Debian support and integration, so you won t be surprised to read that Debian is a big part of my life. Raphael: How did you start contributing to Debian? Ana: Although I knew about the existence of Linux since 1997 or so, I didn t really start using Linux until the summer 2001 when I finally got a computer on my own and an Internet link at home. In the beginning, I was using Mandrake in a dual boot with Windows and later around 2003, I happily moved to only using Debian and ditching the Windows partition. Once settled as a Debian user, I knew anybody could help improve the distribution but I hesitated to join mostly due to two reasons, my perception of Debian was the one of a very elitist and aggressive club and who wants to join this kind of cult^wproject? And even if I wanted to join, I did not know how to get started. By the summer of 2004, the Debian Women project started, it made me seeing Debian as a more welcoming project, and I started maintaining my first packages. The following summer 2005, I attended akademy 2005 (the annual KDE conference) where I had the pleasure to meet there some of the people from the KDE team and this really made a difference for me. Christopher Martin and Adeodato Sim , with the help of other people, have started the maintenance of KDE as a team a few months before and by that time most of the KDE modules where under the maintenance umbrella of the team. This was a very good move since it allowed easily to share the KDE maintenance in a more coordinated way and also eased having non-DDs, like me at that time, to join in and help.

The Debian Women project started, it made me seeing Debian as a more welcoming project.

Raphael: You re part of the Debian KDE team. What s your role in the team and what are your plans for Wheezy? Ana: Nowadays, I am not as active in the KDE team as I used to be in the past. The KDE 3 to KDE 4 transition was quite tiring and changes on the KDE side like the successive marketing renames, the shorter 6 months schedule (it used to be at least 9) or the uncoordinated KDE releases mostly burnt me out. Currently, I am mostly working in helping others to get started within the team, some small fixes here and there, and helping with the uploads: an upload of the full KDE suite to the archive requires some building power and upload bandwidth not everybody have. For Wheezy, with the tentative freeze date in June, the plan is to try to ship the latest possible point release of the KDE 4.8 series. The first release of the series, 4.8.0 was released a couple of weeks ago and while writing these lines, the packaging work for 4.8 hasn t started yet. The next move for the team is getting 4.7.4 in unstable, currently sitting in experimental.

For Wheezy, [ ] the plan is to try to ship the latest possible point release of the KDE 4.8 series.

Besides the KDE packages, there is some software which users perceive as KDE, such as amarok, digikam, etc., which are not part of KDE but fall under its umbrella. These other programs have their own maintainers and their updates depend greatly in the availability of them. For the KDE office suite, we have right now KOffice in the archive. KOffice got a fork some time ago named Calligra and we should replace KOffice by Calligra in the archive before the release of Wheezy. Sadly there isn t yet a final release of Calligra to use. My personal goal for Wheezy was to finish the removal of all the remaining packages depending on KDE 3 and Qt 3 that Squeeze still contained. The removal of the KDE 3 libraries and all the packages using them was quickly achieved after the release of Squeeze. The removal of Qt 3 soon showed that it was task harder than expected since some popular packages (sometimes not in the Debian archive, e.g. third-party scientific software) depend on it, and also Qt 3 is a requirement for LSB compatibility. Right now, Qt 3 has been orphaned for 9 months and nobody has shown any interest in adopting it. Raphael: KDE, much like GNOME, has been forked by people who were unhappy by the direction that the project has taken since version 4 (cf Trinity). What s your personal opinion on KDE 4.x and what s the position of the Debian KDE team concerning this fork? Ana: I use KDE 4 on my laptop and I think it is a solid desktop environment and platform. However I am finding it less and less attractive for me. On one side, my usage of the computer has been slightly changing and on the other side, I do not like how the new developments in KDE are evolving, things like plasmoids or activities are not attractive for me. I have switched my other 2 systems to awesome although I continue to use mainly a bunch of KDE applications: dolphin, konsole, kate, juk, kmix, etc. So you might say my desktop environment is an awesome KDE. Regarding the Trinity project, a lot of users complained very loudly when KDE 3 got replaced by KDE 4 in testing/unstable, so I find quite laudable the decision of some users to act instead and try to continue with a forked development of KDE 3. However the Trinity team seems to be about 3 persons (funny for a project named Trinity ) while KDE 3 is big. In perspective, it does not look that big because KDE 4 is even larger, but it is still too much for such small team. In addition those developers need to maintain Qt3 that has been end-of-lifed years ago by Nokia/Trolltech . So my guess is that sooner or later the project will fade away. Nobody from the KDE team is interested in Trinity and in case someone wants to package it for Debian, they would have to make a new team. For the reasons mentioned above: Qt3 maintenance and reduced upstream group, this would be a bad idea. My advice if you do not like KDE 4 and you miss KDE 3, would be taking a look at razor-qt based on Qt4 and quite similar to KDE 3. I read they have plans to port it to Qt4, but frankly that could take some years same it took to the KDE project for KDE 4.0.0 Raphael: You used to maintain news.debian.net, a WordPress blog dedicated to Debian, but you stopped a while ago. A few months later you started to maintain a Debian page on Google+. Why did you stop the blog and what s your goal with the Google+ page? Ana: I blogged about the reasons I started news.debian.net. In short, I thought Debian needed a better system to publish news, something like a blog. I first tried to suggest the idea to the press/publicity team but they weren t interested, so I started the project alone. IMHO the blog worked quite well and I was feeling like it should be made official. I talked about this with some people but at the time I wasn t pushing it because I had other priorities and I knew pushing it to become official would need some extra time and energy. Stefano decided to start the discussion about making news.debian.net official (that s moving it to a debian.org domain) in its own initiative. After the public discussion and some private exchange of emails with DSA, the situation became frustrating and I decided to close news.debian.net after the release of Squeeze. Later, during DebConf, an officer from the press team announced they were launching a blog and I asked Stefano if he could try to have a discussion about this to see if it could still somehow fit my ideas, and maybe contributing myself, but nobody from the press team answered Stefano s email and the blog hasn t started yet either. Irony that communication didn t work when wanting to improve communication. About the Google+ page, everyday I follow what is going in Debian and quite often I find things I want to share. I do not want to clutter my own profiles with Debian stuff or have people following me because of that, so I decided to create the Debian page when Google+ made them available. I like the fact that people can follow that without having an account in Google+ although they can not comment anonymously. I am not happy about the fact that Google+ is a closed platform but hopefully the data will become easier to export in the near future. Right now, there are some services that provides RSS feeds of Google+ pages if you want to follow the page and you are not in Google+ (or I could setup one if several people ask me). Raphael: Last year you helped to manage Debian s participation to the Google Summer of Code. How did it went? Is there something that you can improve for this year? Ana: I think last year we managed to have people in Debian more aware about what the students were doing. That also helped students to get more feedback and therefore get to know more people in the project and get more integrated. Students were sending periodic public reports available to everybody interested in the status of the projects and some of them also held their own sessions in DebConf. We still failed to start looking for mentors early enough and to give them information about how the GSoC worked and how they could have a successful project. Having good projects in Debian is harder than in other projects because the GSoC mostly promotes having students started in Open Source *coding* for a project, while Debian is more a project about integrating software and we overall do not have so many parts that has to be coded. My personal goal for this year is to try getting the projects earlier to attract good students from the very beginning, even if that means we have less projects than in other years. Raphael: What motivates you to continue to contribute year after year? Ana: Three things. I like improving the OS I use, I like the friends I have made while working in Debian through the years and because I have fun. Also Debian is quite a big project, so if you become tired or burn out working in some area, you always can easily find interesting things to do somewhere else. Raphael: Is there someone in Debian that you admire for their contributions? Ana: Adeodato Sim , he is now in a long leave from the project, but it is one of those persons who made a difference in the project in his job in the release team some years ago. Aur lien Jarno because of his tireless work in (e)glibc and porting of several architectures. I also have special admiration for all those people who have been very active in the project for more than 7-8 years because I know it is not always easy to combine it with real life.

---

Thank you to Ana for the time spent answering my questions. I hope you enjoyed reading her answers as I did. Note that older interviews are indexed on wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

7 comments Liked this article? Click here. My blog is Flattr-enabled.

If you re not into packaging and if you asked how you could help Debian, someone probably suggested that you help to translate it. It s true that translating Debian is essential if we want to make Debian available to everybody on the world. There are many persons who are stuck as soon as they get a message in English, so it s important to aim for 100% coverage in terms of localization. Some vocabulary: localization vs internationalization Internationalization (i18n) is the work that makes it possible to translate messages in a given application. Localization (l10n) is the work of translating messages of said application. So as a translator, you ll be doing localization but some knowledge of internationalization is still useful because it will define how you re supposed to provide the translations. We ll come back to that later. Join your localization team Usually the translation work is shared among multiple translators within a localization team. Check out the Debian International page on www.debian.org to find out instructions for translators for each language. Many teams have a debian-l10n-*@lists.debian.org mailing list used for coordination, feel free to ask questions on those lists when you start (but make sure that you have read the relevant documentation before). Each team has its own workflow, so observe for a while to get used to what s happening before asking your first questions. What is there to translate? The translation of most of the software provided by Debian is not handled by Debian. The Debian translation teams only handle the translation of:

• the software that are specific to Debian (debian-installer, dpkg, APT, etc.) (*);
• the Debconf prompts in all Debian packages (*);
• the Debian documentation (*);
• the Debian website;
• the Debian wiki;
• the descriptions of packages.

Now before contributing to your first translation, I have to come back to internationalization to teach you a few things. In the above list, the projects marked with (*) do use PO files for their translation and the next sections will explain you how to work with those files. Introduction to Gettext The free software community has mostly standardized on a single internationalization infrastructure known as Gettext. With this tool, you re provided a POT file which contains all the translatable strings. It looks like this:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR Software in the Public Interest, Inc.
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <email>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: dpkg 1.16.1\n"
"Report-Msgid-Bugs-To: debian-dpkg@lists.debian.org\n"
"POT-Creation-Date: 2011-09-23 03:37+0200\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <email>\n"
"Language-Team: LANGUAGE <ll>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n"
#: lib/dpkg/ar.c:66
#, c-format
msgid "invalid character '%c' in archive '%.250s' member '%.16s' size"
msgstr ""
#: lib/dpkg/ar.c:81 lib/dpkg/ar.c:97 lib/dpkg/ar.c:108 lib/dpkg/ar.c:112
#: lib/dpkg/ar.c:134 utils/update-alternatives.c:1154
#, c-format
msgid "unable to write file '%s'"
msgstr ""
[ ]

The lines starting with #: are comments that indicate the source files where the (English) string is used. This can be useful if you want check the source to have more information about how the string is used. The lines starting with #, contain flags that can be important. If the fuzzy flag is set, the translated string is not used because it must be updated (or at least verified) since the original string evolved. The c-format flags indicates that the string must be a C format string, this has some implications in what s allowed in the string (in particular when it embeds conversion specifier for arguments submitted to printf-like functions). Another thing to note is that the translation of the empty string is used to store some meta-information about the translation itself. Contributing a translation as a PO file When you start a new translation, you copy that POT file to create a PO file for your own language (eg. fr.po for the French language). You replace some template values (identified with the upper case words in the POT file) and you replace all the empty strings on msgstr lines with the translation of the string that appears in the previous msgid line. The result could be something like this:

# translation of fr.po to French
# Messages fran ais pour dpkg (Linux-GNU Debian).
msgid ""
msgstr ""
"Project-Id-Version: fr\n"
"Report-Msgid-Bugs-To: debian-dpkg@lists.debian.org\n"
"POT-Creation-Date: 2011-09-23 03:37+0200\n"
"PO-Revision-Date: 2012-01-16 07:57+0100\n"
"Last-Translator: Christian Perrier <bubulle>\n"
"Language-Team: French <debian-l10n-french>\n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: Plural-Forms: nplurals=2; plural=n>1;\n"
"X-Generator: Lokalize 1.2\n"
#: lib/dpkg/ar.c:66
#, c-format
msgid "invalid character '%c' in archive '%.250s' member '%.16s' size"
msgstr "caract re invalide  %1$c  dans la taille du membre  %3$.16s  de l'archive  %2$.250s "
#: lib/dpkg/ar.c:81 lib/dpkg/ar.c:97 lib/dpkg/ar.c:108 lib/dpkg/ar.c:112
#: lib/dpkg/ar.c:134 utils/update-alternatives.c:1154
#, c-format
msgid "unable to write file '%s'"
msgstr "impossible d' crire le fichier  %s "
[ ]

If there s already a PO file for your language, there might still work to do: there might be strings that have not yet been translated and there might be fuzzy strings which have to be updated strings which were already translated but where the original string has been modified. There are software that can assist you to edit PO files: poedit, virtaal, lokalize, gtranslator. There are also special extensions for vim (packaged in vim-scripts) and for Emacs. Submit the translation for inclusion Once you have a complete PO file, you should submit it for inclusion. Sometimes you will have been granted commit rights to the source code repository so that you can include your translation by yourself. In the other cases, you should submit your translation with a bug report tagged l10n and someone else will include your work in the next release. Depending on the team, the workflow might require a review before the submission. In that case, you usually have to send a call for review on the coordination mailing list. Go ahead! Hopefully those explanations will be enough to get you started. There are many other things to learn but it s good to learn while practicing For example, can you find out why the French translation above changed %c in %1$c ?

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

No comment Liked this article? Click here. My blog is Flattr-enabled.

When you read this, GNOME Shell 3.2 will (hopefully!) have finally transitioned to Debian s testing suite. Planet GNOME readers might think Debian now has outdated versions of software even in their development versions, or the distribution s development marches at glacial pace. Wheezy GNOME users will finally have a Shell that matches the rest of their GNOME components, something that works with the Shell extensions website and much less problems and limitations compared to 3.0.2. The reality is that GNOME 3.2 s packaging was quite ready back when it was released in late September, but a number of not-so-desirable situations held GNOME Shell from transitioning to testing until today, four months later. So, what happened? TL;DR: transitioning from GNOME 2 GNOME 3 is not so easy if you want to keep testing in a sane state, and when you need to deal with dozens of indirectly related packages, for more than 10 architectures but it shouldn t take nearly a full year, either Let s go back to the last months of 2010. Debian squeeze is in very deep freeze, and the release team and many Debian developers are focusing on squashing as many release critical bugs as they can, in order to make Debian 6.0 the great release it ended up being. The GNOME project has recently delayed the big launch of GNOME 3.0 again, until March 2011; Debian has already settled on GNOME 2.28 for its release, although it will end up cherry-picking many updates from the 2.30 release modules. With most of the stabilization work being done, many Debian GNOME team members were at that time working on packaging very early versions of what would end up being GNOME 3.0 technology: GTK+3.0, GNOME Shell, Mutter and some brave users even tried to use it via the experimental archive. On February 6th, Debian 6.0 was released, and soon after, on April 6th, GNOME made a huge step forward with the much anticipated release of GNOME 3.0. At that time, Debian developers were busy breaking unstable as much as they could, as it s tradition on the weeks following a major release, and the Debian GNOME team was able to start moving some GNOME 3.0 libraries (those which were parallel-installable with their GTK+2.0 versions) to unstable. However, moving the bulk of GNOME 3.0 to unstable wasn t so easy. When you start doing that, you need to be sure you re ready to have all affected packages in a transitionable state as soon as possible, to minimise the chances of blocking transitions of unrelated packages via the dependencies they pick up with rebuilds. All the packages involved in a transition need to be ready to go in the same testing run , for all supported architectures. When you re dealing with dozens of GNOME source packages at the same time, many of which introduce new libraries, or worse, introduce incompatible APIs that affect many more unrelated packages, things get hairy, and you need a plan. So, Joss outlined what a sane approach to this monster transition could look like. The amount of work to do was what we call fun on #debian-gnome. In a nutshell, we had to deal with quite a few transitions, starting with having a newer version of libnotify in unstable, and a pre-requisite for that was making sure all the packages using libnotify1 were ready to use the source-incompatible libnotify4, and this meant preparing patches and NMUs for many of our packages, as well as many others not under our control. Before starting a controlled transition like this one, we had to get an ACK from the release team, who was busy enough handling other huge transitions like Perl 5.12, so by the time we got our own slot, we were well into Summer. With libnotify done in August, it was time to get our hands dirty with more exciting stuff, like getting Nautilus in testing. This meant bumping a soname and requiring all packages providing Nautilus extensions to migrate to GTK+3.0, or drop the extension entirely, as you can t mix GTK+2.0 and GTK+3.0 symbols in the same process. However, in GNOME 3.0, automounting code had moved from Nautilus to gnome-settings-daemon, so in order to not break filesystem automounting in testing for an unreasonable amount of time, both Nautilus and g-s-d needed to go in at the same time. The fun thing is that g-s-d dragged glib2.0, gvfs, gnome-control-center, gdm3, gnome-media, gnome-session and gnome-panel into the equation, so this transition needed extra planning and a lot more work than initially expected: migrating all nautilus extensions, plus ensuring all Panel applets had migrated to GTK+3.0 and the new libpanel-applet-4 interface. In short, this was the monster transition we were trying to avoid. By the time all this mess was sorted out, GNOME 3.2 had been released, and for what users said, it was a lot better than 3.0. We still had no more than a few bits and pieces of 3.0 in testing, and we were working hard to get 3.0 in wheezy. With all the excitement around 3.2, at times it was difficult to explain outsiders why we were beating a dead 3.0 horse Going back to our huge transition, it was just a matter of time before all the packages would be built and be ready to enter, on the same run, in testing. A few weeks later, in early November and after several rounds of mass-bug-filings, fixing unrelated FTBFS, many NMUs, package removal requests and dealing with any possible problem that could block our transition, everything seemed to be set, and our release team magicians had everything in place for the big magic to happen. However, our first clash with the rest of Debian happened a few hours before our victory, in the form of an unannounced ruby-gnome2 upload which resetted the count for everyone. It was fun to see the release team trying all sorts of black magic in an attempt to mitigate the damage. Fortunately, after a few tries they managed to fool britney (the script that handles package transitions from unstable to testing) somehow, and the hardest part of the job was done with just one day of delay. At last, the core of GNOME 3 was in testing, and testing users found soon after. The rest of the week saw a cascade of hate posts against GNOME 3 in Planet Debian, and personally I didn t find that especially motivating to keep on working on the rest of GNOME bits. With experimental clear of GNOME 3.0 stuff, we finally were able to focus on packaging whatever GNOME 3.2 components were not already done, and preparing for what should be a plain simple transition of GNOME 3.0 to 3.2. After our share of wait for a transition slot, as Perl 5.14, ICU and OpenSSL were in the line before us, and after dealing with a minor tracker 0.12 transition, we were ready for our next episode: evolution-data-server. At first sight, we thought this would be a lot easier, but it still got a bit hairy due to evo-data-server massive soname bumps. We were given our slot just before Christmas, after a few weeks of wait for others to finish their migration rounds, and most of the pack entered wheezy a few days before the new year. No rejoicing, though, as GNOME Shell 3.2 didn t make it. First, we discovered it was FTBFS on kFreeBSD architectures, as NetworkManager had been promoted from optional to required, for apparently no good reason, leaving the BSD world in the cold, including our exotic GNU/kFreeBSD architectures. Now, let s clarify that I m a supporter of the Debian kFreeBSD architectures and was really happy to see it accepted as a technology preview in squeeze. However, as you know, GNOME Shell currently requires hardware acceleration to run, a requirement hardly met in kFreeBSD, unless you re using a DRI1 X driver. We seriously doubted anyone had ever ran a GNOME 3 session on kfreebsd-*. However, if it didn t build, it was a blocker bug for GNOME Shell. We considered creating different meta-packages for kFreeBSD architectures, to conclude it d be a mess, so our awesome Michael Biebl ended up cooking up a patch that restored the ability to build the Shell without NetworkManager support. With this out of our way, we just needed to upload Michael s fix and watch the buildds do their part of the job. Or maybe not? Enter Iceweasel 9. In parallel, and with incredible bad timing, Iceweasel 9.0 was uploaded to Debian the very same day it was released by Mozilla. Again, it greeted us with a nasty surprise: yet another mozjs API change, which made gjs FTBFS, which meant our kFreeBSD fixes would be unusable until someone who knew Gjs internals well enough bit the bullet and worked around the new API changes. Again, Michael Biebl tried to be our saviour, but unfortunately wasn t able to fix all the problems, so we tried to focus on plan B. Mozilla had released a fork of the mozjs that is included in Firefox, so that embedders would have a bit less of a hard time with these recurrent API changes. This was based on Firefox 4, and was already being packaged by Ubuntu. Gjs would build using this older version just fine, so we just needed to get it in Debian as soon as possible. We just needed to find a sucke^Wvolunteer that would be inclined to maintain the beast. Only after a few weeks we managed to get Chris Coulson, the Ubuntu packager, to maintain the package directly through the Debian archive via package syncs. However, his package had only been auto-compiled in the three Ubuntu architectures, that is amd64, armel and i386. It s late January 2012, and we ve been fighting this war for 10 months. After getting some help from Michael to get the new package in shape for Debian standards, we were excited to sponsor it for Chris. Duh, after a few days in the NEW fridge, it was rejected by the ftp-masters. The license statement was missing quite a few details, so I went ahead and sacrificed a few hours of my copious free time to get this sorted out. A few days later, mozjs was accepted, but the result was horrible. It was very red. mozjs didn t build on half of our targets. Mike Hommey was quick to file a bug and point us to the most obvious fuckups. As he had dealt with this in the past as the Iceweasel maintainer, all of these issues were fixed and patches were ready to be applied verbatim or with minimal changes to our sources. With mozjs finally built successfully (although with severe problems on ia64), we were finally able to rebuild Gjs against it, upload GNOME Shell with our kFreeBSD fixes and wait until today for this mess to be over. Whew. I can t say I ve enjoyed all the stages of this ride. Some bumps on the road were clearly there to test our patience, but it has helped me get back in touch with non-leaf GNOME packaging, which was all I was doing for a while due to being super-busy lately with studies. It also reminds me of the privilege of working side by side with some awesome people, not only Joss, Michael, Sjoerd, Laurent or Gustavo, to name just a few Debian GNOME team members, but also the receptive release team members like Julien or Cyril, and NEW-processing record-breaking ftp-master Luca. Without them, we might be trying to figure out the Nautilus transition since last Summer. We really hope GNOME 3.4 will be a piece of cake compared to this. ;)

You are scaring away much more than that. I just came across an invitation for BugCon 2012. BugCon is a Mexican conference devoted to computer security I cannot comment on its level or value because, although it's a topic that has long interested me, I must recognize each day I feel less of an expert, nowadays finding myself at the level of a "sysadmin who tries not to be too dumb for his own job security". Oh, and also because it would be completely off-topic for this post. If you look at Vendetta's (the main organizer) blog post, it will probably give you the impression that the conference is just an excuse for the afterparty: Lets go see some b00bs! Do you think your fellow female hackers will have any interest in joining a bunch of sex-starved, hormone-infested teenagers who only want to pwn a website and grab more pr0n? Do you think females will feel welcome (or even mildly safe) between you? I would not think so. And I also think you are alienating any professional who might have any interest in joining your community, be it as a member, as a mentor, or whatnot. I cannot right now do a coherent post on this topic, but I can reference you to what I have seen (and read) over the last almost 10 years, when the issue was first brought up to our attention. I am very glad to see that, at least in the Free Software area, there has been a real change of mindset. I hope you are in time to think about it and rectify.

• Timeline of incidents in Geekfeminism. Note that while it seems we see more as time passes, I am almost sure it's because we are more aware of the problem, not because it occurs more often. I hope I'm not mistaken.
• Debconf ftp-masters talk. Myself a Debian person, my first contact with this problematic was being at the DebConf3 ftp-masters talk And the discussion and action that followed. This led to the creation of the Debian Women group, one of the most (socially, not technically) influent parts of Debian. Great thanks and admiration to their members, as well as to the (male and female alike) people who have worked to form it and make it heard.
  I think Debian Women sparked other similar projects such as GnomeWomen (and there is a list with further projects in there), but I cannot authoritatively say who was there first.
• Planet Fedora up-skirting photo (the original post is still available) showed the communit does no longer tolerate this behaviour. Good!
• The Open Source Boob Project. One of the most childlike attempts at humor that surely alienated many would-be female geeks.
• Another conference season, another dumb sexist, a post by Piers Cawley addressing this issue after attending the CouchDB + Ruby: Perform like a Pr0n star talk. Quoting him, Apparently, the difference between 80s truck salesmen and Matt s audience is that at least 80s salesmen had the grace to look embarrassed.
• Liz Keogh: "I am not a pr0n star: Avoiding unavoidable associations", a hacker woman that clearly felt offense by the CouchDB Pr0n Star joke, and did a thorough and interesting analysis, extending the effects to your work environment.
• Just Say You're Sorry Already (regarding the same incident on CouchDB+Ruby)
• Richard Stallman's EMACS virgins joke incident. It's sad how it's impossible to get Stallman to acknowledge he can also make mistakes and make feel people insulted.
• [update] And of course, MadameZou mentions the very important 2002 HOWTO: HOWTO encourage women in Linux?

Oh, and not the description of an incident, but a very interesting and thoughtful take on this: [pdf] Interesting analysis by Hannah Wallach on the numbers and motivations of women in Free Software groups. I don't know if Hannah has published this in article form, but many interesting points can be understood by looking at the presentation. My good friend Vendetta: I don't mean this post (longer than what I originally intended) as a way to say you and the conference you are organizing for the third year (IIRC) already is unprofessional or targetted to pimply teenagers. I know the work you have put in it. I hope you see the points I'm trying to drive You are of course free to have whatever afterparty you have. But, if as the main organizer, you are giving the images of nice chicks at Hooters more weight and relevance than to the conference itself... you are doing yourself a disservice. I hope you can rectify it, and make BugCon attractive to hacker women as well.

Hi folks!

We're almost done for the official Summer of Code program and I thought that I couldn't let it end without another update, so here we are (you may thank Daniel Burrows for additional poking).

The Gtk+ interface for Aptitude is making great progress. The product is not final yet but already implements many of the planned ideas, with others to come.




So, let's see the tasks presented in my previous update:

• A real user interface <- much progress here


• Being reliable <- check! we don't crash all the time anymore


• Packages dependencies, pre-dependencies, etc. <- check!


• Hyperlinks between packages <- check!


• Advanced package search <- check!


• Tags support (tag clouds ?) <- well, no clouds yet, but we have tags


• Linking with interesting data sources like popcon <- not done yet


• A lot of interesting things <- well..

In some more details: we are now much closer to the final interface. Some things are not done yet, for example, notification bars la Iceweasel (or is it Firefox ?) are in the works for unobtrusive notifications about important things and a little guidance.

We are much more reliable now. We eliminated lots of memory management problems that resulted from, mmh, imperfect coding discipline. We also lock the interface when important backend stuff reload to prevent the user from shooting himself in the foot when triggering an install when the APT cache is reloading for example.. The backend refactoring took a sizable part of this second term but it was worth it. The code is now much cleaner and uses no more (at least much fewer) ugly unscalable hacks.

One very important feature of Aptitude is the ability to easily display all kinds of dependencies and navigate through them by hyperlinking. It's now working in the Gtk+ interface as well. Feel free to click on whatever looks like a package or a package version.

We now support the powerful Aptitude package searching syntax. Daniel Burrows is talking with Enrico Zini to bring the power of apt-xapian-index to Aptitude: autocompletion! lightning fast full-text searches! tag clouds! ponies!

There is still a lot to be done. I'm not going away after the GSoC program. I'm here to stay, for Aptitude and Debian. Daniel Burrows has been very helpful and supportive. The Debian community is as awesome as I knew it before participating in the program.

Daniel Burrows is planning to merge the Gtk+ interface into the main branch in September and packages may appear into Experimental. With the very interesting evolutions of the upcoming Adept-3, Lenny+1 will show very interesting changes in the area of packages managers.

Now for you. I need you to test my interface. All necessary instructions are on the development trac : http://dev.graffit.net/aptitude/trac.

So, in 2002 I attended my first SPUG meeting. Also in attendance were Randal, Tim, J.P. (I think) and quite a few of the other usual suspects. The guest speaker was the inestimable Dr. Damian Conway. This night might be the single most important event in my career as a builder of the internets. I was already set in my ways regarding my operating system, and I knew enough perl to get by, but I wasn t in touch with any users groups until then. The crown jewel of the night was Damian s recital in full of his poem, I Am The Very Model of A Poster To CLPM (sung to the tune of that song from Pirates of Penzance). I must admit that I d never heard of CLPM and that it all sounded like gobble-de-gook at the time. If I recall correctly, this happened just after he took about 10 minutes to teach us all to pronounce the name of then-Pumpking, Jarkko Hietaniemi. This was the same year that the following video was made <iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="http://www.youtube.com/embed/XkU_GQVOCqo" width="420"></iframe> With that in mind, there s one more thing I need to share with you before disclosing the lyrics. This piece was developed during a time when it was common for members of the perl community to flame one another on news feeds, take their anger and frustration out on newbies and generally be the bane of most of the internet. I like to think that it helped to shape me into a kinder, gentler perl programmer, debian user and general netizen over the years. I would also like to think that it will do the same for future members of our communities. And so, without further ado, Damian Conway s

---

I Am The Very Model of a Poster to CLPM Now everything I need to know about the Perl community
I learned in kindergarten when I was a little lad of three
And in these next few minutes I ll relate those five apothegm
For I am the very model of a poster to c.l.p.m. The first truism I was ever taught upon my mother s knee
Was that our dad s a better dad than their s could ever hope to be.
His name is Larry Wall, and that s determinism nom native
Since of the mural attributes he demonstrates proof positive. He s eight feet tall and ten feet wide; of granite-like solidity.
He wards us from the serpent s tabs and coffee bean s stupidity.
He guards us from that Gallic tower s B&D urbanity,
And makes sure we don t see sharp code that might cut loose our sanity. He watches over each of us, like Odin of the Norse belief
The stern All-Father figure and, at times, the comical relief.
So, if you vaunt our patriarch s supreme linguistic acumen,
You are a model sychophantic poster to c.l.p.m. The second tenet that I learned concerns, of course, the fairer sex
And why they are in short supply, while geek-boy JAPHs are multiplex.
As every schoolboy knows each conversation has single aim:
To raise one s own significance, whilst razing others down in flame. And thus good men and true who hold this principle as sacrosanct
Will all pursue beligerence as though they were from Mars. Or tanked.
It is the tenor of your newgroup posts that she from Venus flies:
Because most threads devolve into comparisons of ..meanness, guys. When every post affronts their own collaborative sorority
It must exasperate them, for it renders the majority
Of female Perl geeks absent, till we re forced to rename blonde Dutch men
To act as female models of a poster to c.l.p.m. The third grand aphorism that was drummed into my youthful head
Was: if you can t say something nice about someone, you should instead
Say every nasty thing you can impart, imagine, or invent
In any public forum that it happens you should both frequent. This goes especially for the young, and new, and shy, and innocent.
To flame a newbie is a joy, t will make you feel omnipotent.
To watch them squirm and bleat and bleed and scurry back into their hole,
The surge of ecstasy it brings is worth the way it eats your soul. So never hesitate to turn the wrath of your vocabul ry
Upon the hapless supplicant for Perl epistomology
And if you flame each FAQ, and every question that s beneath you then
You ll be the very model of curmudg nly R.T.F.M. The next point I should like to make s the obverse of the previous
For while it s true that to be newbie is a sin most grieveous
It ain t as bad as being old and greyed and staid and tedious
And sitting at the keys all day dispensing thoughts invidious. What right have you to tell me not to ask about my CGI???
I thought that Perl was all about website development, and I
Expect that you d extend to me the simple human courtesy
Of dropping everything so you can do this homework now for me. The Internet exists so I need never read the manual
And so my inane questioning can form a stream continual
While pleading for an email since I never read this group, as well
As posting them in jeopardy, in caps, and in HTML. The rule I learned in kinder that explains where this approach springs from
Is fundamental to a certain class of Gen-X hackerdom
The single principle that guides their Internet activity
Is: me, me, me, me, me, me, me, me, me, me, me, me, me, me, ME! So if you are prepared to treat the Net as your god-given right
And suck the newsgroups dry like some goodwill-devouring parasite
And use them as a help-desk and a perldoc substitute, well then
You are the very model clueless poster to c.l.p.m. The final precept that my pre-school inculcated in my head
Is one I would commend as antithetical to all I ve said.
It is a noble concept that once used to garner more elan
The idea that, when I grow up ..I want to be a fireman. The fire-fighter s job s to help, despite the inconvenience
And spare the inexperienced the cost of their own ignorance
And put out fires when the flames ignite the common living space
And through it all maintain an air of competence and skill and grace. And this, my friends, I recommend to all of our community
That rather than a careless match or carefully lit incendiary
You spray out knowledge, patience, and a gentle stream of good advice
And save the newbies from themselves, and oldies from their prejudice. And be yea more as Larry is, a fire-Wall against the threat
Of ignorance and arrogance, with both of which we re still beset.
For if you choose to help your fellow JAPHs instead of scorching them,
You ll be the next burnt offering we spit-roast on c.l.p.m. ;-) [edit]

I've just finished reading Graham Joyce's haunting The Silent Land: an atmospheric and moving love story set in an Alpine ski resort. Husband and Wife Jake and Zoe head out to the slopes early one morning and are caught in an avalanche. They survive, narrowly, but upon returning to the resort, everything has changed Anyone who has read any quantity of SF or fantasy will probably make an early guess at what's going on, and you'd most likely get it wrong. Even if you know where you're going, the journey is often the point, rather than the destination, and this journey is pretty gripping. I do most of my reading on the train to and from work. I often listen to music when I'm reading and I try to choose things which aren't going to distract my attention from the book too much. Film soundtracks are good; so is classical; so is a lot of Eno's work. For this book I mostly listened to the score from the latest film version of The Girl with the Dragon Tattoo, by Trent Reznor and Atticus Ross. Clocking in at only seven minutes shy of three hours, this is a haunting score that evokes a cold, isolated Swedish winter, and thematically matched the book perfectly. The score is a little less accessible than their previous work, but is rewarding on repeat listen. You can get a free 6-track sampler at the website linked above.

Part of our work to make it possible to use UEFI Secure Boot on Linux has been to improve our EFI variable support code. Right now this has a hardcoded assumption that variables are 1024 bytes or smaller, which was true in pre-1.0 versions of the EFI spec. Modern implementations allow the maximum variable size to be determined by the hardware, and with implementations using large key sizes and hashes 1024 bytes isn't really going to cut it. My first attempt at this was a little ugly but also fell foul of the fact that sysfs only allows writes of up to the size of a page - so 4KB on most of the platforms we're caring about. So I've now reimplemented it as a filesystem[1], which is trickier but avoids this problem nicely.

Things were almost working fine - I could read variables of arbitrary sizes, and I could write to existing variables. I was just finishing hooking up new variable creation, but in the process accidentally set the contents of the Boot0002 variable to 0xffffffff 0xffffffff 0x00000000. Boot* variables provide the UEFI firmware with the different configured boot devices on the system - they can point either at a raw device or at a bootloader on a device, and they can do so using various different namespaces. They have a defined format, as documented in chapter 9 of the UEFI spec. At boot time the boot manager reads the variables and attempts to boot from them in a configured order as found in the BootOrder variable.

Now, obviously, 0xffffffff 0x00000000 is unlikely to conform to the specification. And when I rebooted the machine, it gave me a flashing cursor and did nothing. Fair enough - I should be able to choose another boot path from the boot manager. Except the boot manager behaves identically, and I get a flashing cursor and nothing else.

I reported this to the EDK2 development list, and Andrew Fish (who invented EFI back in the 90s) pointed me at the code that's probably responsible. It's in the BDS (Boot Device Selection) library that's part of the UEFI reference implementation from Intel, and you can find it here. The relevant function is BdsLibVariableToOption, which is as follows (with irrelevant bits elided):

BdsLibVariableToOption (
  IN OUT LIST_ENTRY                   *BdsCommonOptionList,
  IN  CHAR16                          *VariableName
  )
 
  UINT16                    FilePathSize;
  UINT8                     *Variable;
  UINT8                     *TempPtr;
  UINTN                     VariableSize;
  VOID                      *LoadOptions;
  UINT32                    LoadOptionsSize;
  CHAR16                    *Description;
  //
  // Read the variable. We will never free this data.
  //
  Variable = BdsLibGetVariableAndSize (
              VariableName,
              &gEfiGlobalVariableGuid,
              &VariableSize
              );
  if (Variable == NULL)  
    return NULL;
   

So so far so good - we read the variable from flash and put it in Variable, Variable is now 0xffffffff 0xffffffff 0x00000000. If it hadn't existed we'd have skipped over and continued. VariableSize is 12.

  //
  // Get the option attribute
  //
  TempPtr   =  Variable;
  Attribute =  *(UINT32 *) Variable;
  TempPtr   += sizeof (UINT32);

Attribute is now 0xffffffff and TempPtr points to Variable + 4.

  //
  // Get the option's device path size
  //
  FilePathSize =  *(UINT16 *) TempPtr;
  TempPtr      += sizeof (UINT16);

FilePathSize is 0xffff, TempPtr points to Variable + 6.

  //
  // Get the option's description string size
  //
  TempPtr     += StrSize ((CHAR16 *) TempPtr);

TempPtr points to 0xffff 0x0000, so StrSize (which is basically strlen) will be 4. TempPtr now points to Variable + 10.

  //
  // Get the option's device path
  //
  DevicePath =  (EFI_DEVICE_PATH_PROTOCOL *) TempPtr;
  TempPtr    += FilePathSize;

TempPtr now points to Variable + 65545 (FilePathSize is 0xffff).

  LoadOptions     = TempPtr;
  LoadOptionsSize = (UINT32) (VariableSize - (UINTN) (TempPtr - Variable));

LoadOptionsSize is now 12 - (Variable + 65545 - Variable), or 12 - 65545, or -65533. But it's cast to an unsigned 32 bit integer, so it's actually 4294901763.

  Option->LoadOptions = AllocateZeroPool (LoadOptionsSize);
  ASSERT(Option->LoadOptions != NULL);

We attempt to allocate just under 4GB of RAM. This probably fails - if it does the boot manager exits. This probably means game over. But if it somehow succeeds:

CopyMem (Option->LoadOptions, LoadOptions, LoadOptionsSize);

we then proceed to read almost 4GB of content from uninitialised addresses, and since Variable was probably allocated below 4GB that almost certainly includes all of your PCI space (which is typically still below 4GB) and bits of your hardware probably generate very unhappy signals on the bus and you lose anyway.

So now I have a machine that won't boot, and the clear CMOS jumper doesn't clear the flash contents so I have no idea how to recover from it. And because this code is present in the Intel reference implementation, doing the same thing on most other UEFI machines would probably have the same outcome. Thankfully, it's not something people are likely to do by accident - using any of the standard interfaces will always generate a valid entry, so you could only trigger this when modifying variables by hand. But now I need to set up another test machine.

[1] All code in Linux will evolve until the point where it's implemented as a filesystem.

comments

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1,SHA512

Hello,

I'm transitioning from my 2003 GPG key to a new one.

The old key will continue to be valid for some time, but I eventually 
plan to revoke it, so please use the new one from now on. I would also 
like this new key to be re-integrated into the web of trust. This message 
is signed by both keys to certify the transition.

The old key was:

pub   1024D/FC7F6D0F 2003-07-10
      Key fingerprint = E6A8 8BA0 D28A 3629 30A9  899F 82D7 DF6D FC7F 6D0F

The new key is:

pub   4096R/90BDD207 2012-01-06
      Key fingerprint = 6B85 4D46 E843 3CD7 CDC0  3630 E0F7 59F7 90BD D207

To fetch my new key from a public key server, you can simply do:

  gpg --keyserver pgp.mit.edu --recv-key 90BDD207

If you already know my old key, you can now verify that the new key is 
signed by the old one:

  gpg --check-sigs 90BDD207

If you don't already know my old key, or you just want to be double 
extra paranoid, you can check the fingerprint against the one above:

  gpg --fingerprint 90BDD207

If you are satisfied that you've got the right key, and the UIDs match 
what you expect, I'd appreciate it if you would sign my key:

  gpg --sign-key 90BDD207

Lastly, if you could send me these signatures, i would appreciate it. 
You can either send me an e-mail with the new signatures by attaching 
the following file:

  gpg --armor --export 90BDD207 > timojyrinki.asc

Or you can just upload the signatures to a public keyserver directly:

  gpg --keyserver pgp.mit.edu --send-key 90BDD207

Please let me know if there is any trouble, and sorry for the inconvenience.

(this post has been modified from the example at 
http://www.debian-administration.org/users/dkg/weblog/48)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk8GuZoACgkQgtffbfx/bQ9nqACglWyHnDTFQfdKmz8OCd3oL6iR
hcEAmgKJ7RZsgwxwkRGPhygy5y1Ztb+3iQIcBAEBCgAGBQJPBrmaAAoJEOD3WfeQ
vdIHdVQQAMT1yvIogzbtK6sUnWqwbrXI9pDEFk7AzJTb80R+wzxsw7gu9gcBDk8G
BL2O26GKUqKWA3ytuApSl42FJam/Lusi9npT3XNkmHs6FaBMNuLYrqEXmCwXwWr/
OrLyeeLiF4yxgbNWbv+600BqAWqFlo6NeTgQKsJWtCjR3RVMxX3R8nzjDnKJuF+z
c6+2JKBWyx/HVUKcJpJrFDDR36HRFvVJomTuma2JCQ/RAl9vzAguqNYOi1QkuuQv
EF1gXH7gLifukGuwquP1DHP6SWWkj77jtRWr5ewC0xymbrArzAwKbvMQl3VpKBHh
MmpJjYP3ECyL14AKi/TY2Lidi0Sf6yqFMcPcreoih01N0OU0NXmD4IrHMT24/ssb
okDUe1o3YImjGq1jTACvlzC8s54EfLsqDgSP98SGVpuoDqPJUwVk4nuHj8q0vDSs
qZox26gVwB2FAOUi1BFiZbIzM5rsyYfCGyWUGiAwBFf54lYRAeCDCt8iAOOL1Ov/
TumIGYdLoXnDuOJq1VjXLGx2OFDrpyU8SPGoa3zNEVz39tgxQ48ASJEqcqt7HvBy
IW+TTsMLdJ1Ait9aCM3mzzr1iwP8TrL0qUsdRLOE6AKdAqocIfqXY8OeDKhbUiOJ
CXWk5q3xheK3sDWUXX7J63bAAUH4jFnpQEOVMJKBUNMKsWa0iXDS
=mklN
-----END PGP SIGNATURE-----

Every year, on January 1st, new material ceases to be protected by copyright and enters the public domain. This means, every year, more knowledge, literature, paintings, music, movies and a long etcetera becomes collective property, instead of being artificially held by the current holders of their rights. As this image shows (source: http://publicdomainday.org/node/39 ), I have the honor(?) to live in the country with the longest copyright protection term in the world. Copyright in Mexico does not only last for 100 years It lasts for the natural life of the author plus 100 years. This means that the popular corridos that tell the stories of the 1910 revolution are still not in the public domain. La sucesi n presidencial, the book which Francisco I. Madero wrote to justify that a peaceful political change was needed for the 1910 elections, will not enter the public domain until 2014 (president Madero was killed during 1913). Does it make any sense to kidnap cultural, political or artistic works for over a century? Not only that: Material that is legally sold as public domain in other countries is illegal in ours. Take as an example the recordings of Enrico Carusso, the great Italian tenor who died in 1921. Over 15 years ago, I bought a couple of CDs with his recordings (even if the sources were quite low-quality, as they had been copied over from wax cylinders to magnetic tapes to optical media). I bought them surprisingly cheap, as they were genuine public domain. But they are still protected in my country. That means, I ilegally have some stolen(!) works of art which I lawfully bought outside my country. Copyright law needs to be revised to match reality. Technological advances have strongly changed reality since 1717's promulgation of the first copyright laws. The solution is not to extend the terms, but to rethink the whole process. (yes, this rant was mainly made as an excuse for me to copy this image and put it in a location I can easily refer to later. But I hope it is interesting to you!)

Summary: It is important that we (the Debian community that relies on OpenPGP through GNU Privacy Guard) stop using short key IDs. There is no vulnerability in OpenPGP and GPG. However, using short key IDs (like 0x70096AD1) is fundementally insecure; it is easy to generate collisions for short key IDs. We should always use 64-bit (or longer) key IDs, like: 0x37E1C17570096AD1 or 0xEC4B033C70096AD1. TL;DR: This now gives two results: gpg --recv-key 70096AD1

Some background, and my two keys Years ago, I read dkg's instructions on migrating the Debian OpenPGP infrastructure. It told me that the time and effort I had spent getting my key into the strong set wasn't as useful as I thought it had been. I felt deflated. I had put in quite a bit of effort over the years to strongly-connect my key to a variety of signatures, and I had helped people get their own keys into the strong set this way. If I migrated off my old key and revoked it, I'd be abandoning some people for whom I was their only link into the strong set. And what fun it was to first become part of the strong set! And all the eyebrows I raised when I told people I was going meet up with people I met on a website called Biglumber... I even made it my Facebook.com user ID. So if I had to generate a new key, I decided I had better really love the short key ID. But at that point, I already felt pretty attached to the number 0x70096AD1. And I couldn't come up with anything better. So that settled it: no key upgrade until I had a new key whose ID is the same as my old key. That dream has become a reality. Search for my old key ID, and you get two keys!

$ gpg --keyserver pgp.mit.edu --recv-key 0x70096AD1
gpg: requesting key 70096AD1 from hkp server pgp.mit.edu
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 2
gpg:               imported: 2  (RSA: 1)

I also saw it as an opportunity: I know that cryptography tools are tragically easy to mis-use. The use of 32-bit key IDs is fundamentally incorrect -- too little entropy. Maybe shocking people by creating two "identical" keys will help speed the transition away from this mis-use.

A neat stunt abusing --refresh-keys Thanks to a GNU Privacy Guard bug, it is super easy to get my new key. Let's say that, like many people, you only have my old key on your workstation:

$ gpg --list-keys   grep 70096AD1
pub   1024D/70096AD1 2005-12-28

Just ask GPG to refresh:

$ gpg --keyserver pgp.mit.edu --refresh-keys
gpg: refreshing 1 key from hkp://pgp.mit.edu
gpg: requesting key 70096AD1 from hkp server pgp.mit.edu
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: key 70096AD1: "Asheesh Laroia <asheesh@asheesh.org>" not changed
gpg: Total number processed: 2
gpg:               imported: 1  (RSA: 1)
gpg:              unchanged: 1
gpg: no ultimately trusted keys found

You can see that it set out to refresh just 1 key. It did that by querying the keyserver for the short ID. The keyserver provided two hits for that query. In the end, GPG refreshes one key and actually imports a new key into the keyring! Now you have two:

$ gpg --list-keys   grep 70096AD1
pub   1024D/70096AD1 2005-12-28
pub   4096R/70096AD1 2011-03-11

There is a bug filed in GNU Privacy Guard about this. It has a patch attached. There is, at the moment, no plan for a new release.

A faster attack, but nothing truly new My friend Venkatesh tells me there is an apocryphal old Perl script that could be used to generate key ID collisions. Here in the twenty-first century, l33t h4x0rz like Georgi Guninski are trying to create collisions. In May 2010, "halfdog" posted a note to the full-disclosure list that generates PGP keys with chosen short key IDs. I haven't benchmarked or tested that tool, but I have used a different tool (private for now) that can generate collisions in a similar fashion. It takes about 3 hours to loop through all key IDs on a dinky little netbook. You don't have to use any of these tools. You can just rent time on an elastic computing service or a botnet, or your own personal computer, and generate keys until you have a match. I think that it's easy to under-estimate the seriousness of this problem: tools like the PGP Key Pathfinder should be updated to only accept 64-bit (or longer) key IDs if we want to trust their output.

My offer: I will make you a key I've been spending some time wondering: What sort of exciting demonstration can I create to highlight that this is a real problem? Some ideas I've had:

• Publish a private/public key pair whose key ID is the same as Phil Zimmerman's, original author of PGP
• Publish a private/public key pair whose key ID is the same as Werner Koch's, maintainer of GNU Privacy Guard
• Publish a set of public keys that mimic the entire PGP strong set, except where I control the private key of all these keys

The last one would be extremely amusing, and would be a hat-tip to some work discussed in Raph Levien's Google Tech Talk about Advogato. For now, here is my offer: If you send me a request signed with a key in the strong set, I will create a 4096-bit RSA public/private key pair whose 32-bit key ID is one greater than yours. So if you are 0x517DD4E4 I will generate 0x517DD4E5. I will post the keys here, along a note about who requested it, and instructions on how to import them into your keyring. (Note: I will politely decline to create a new key whose 32-bit key ID would create a collision; apologies if your key ID is just one away from someone else's.) P.S. The prize for best sarcastic retort goes to Ian Jackson. He said, "I should go and create a lot of keys with your key ID. I'll set the real name to 'Not Asheesh Laroia' so everyone is totally clear about what is going on."

Calypso CalDAV/CardDAV/WebDAV for Android and Evolution Ever since I bought my first Palm Pilot in 1997, I ve relied upon a pocket-able device to carry a copy of my calendar and contacts, and for that same database to be present on my laptop. I went through a long list of Palm-compatible devices, including both the Palm Treo and Palm Centro telephones. I even wrote a number of my own Palm applications. Years ago, it was pretty obvious that I d have to find a new phone, but I was stuck looking for something that could provide the same hot-sync functionality. SyncML on the Series 40 I bought a Series 40 Nokia phone in Shanghai that promised SyncML support. Given that I had seen numerous SyncML implementations for Linux, it seemed like I should be able to get something to work. SyncML on the Series 40 is a disaster the phone couldn t actually store all of my contacts, and it couldn t hold half of the data fields I used. So, that phone landed in the box and back to the Centro I went. SyncML on the N900 Nokia kindly sent me an N900 at some point, so I gave SyncML another try. Given that the N900 runs evolution-data-server, and that I ve had evolution-data-server running on my laptop, it seemed like I should be in business. Well, almost. It took several days of hacking to fix bugs in the evolution SyncML back end, then another several days fussing with opensync. I ended up abandoning direct synchronization as unworkable opensync would sit in an infinite loop, or worse, trash my database completely. I finally found syncml-ds-tool , which is a debugging tool that comes with opensync. This tool simply synchronizes a set of disk files, one per contact or calendar entry, with the phone. That worked for well over a year. And then, a few months ago, Bluetooth on the laptop stopped connecting with the phone s SyncML server. I d get ECONNREFUSED every time I tried to use it. So much for the N900. DUN still worked, mostly, although it too would get ECONNREFUSED at times, but retrying seemed to make it work. However, While the N900 SyncML solution worked, I discovered another thing I wanted contacts and calendar entries stored in individual files and revision controlled with git. This makes it reasonable to delete stale calendar entries and know that they re never really gone, just left behind in an older version of the calendar. And, if you mess up, you can recover by poking at the database with git directly. I switched back to the venerable Palm Centro; it turns out that calendar and contacts are more important to me than being able to surf the web on my phone. Alas, my Centro went swimming in August and has passed on to the great electronics recycling house in the sky. I pulled the SIM out and switched back to the N900. I got my contacts imported on the N900 by copying files over the net work; not a long-term strategy, but at least I had phone numbers again. There was no hope for my calendar. I started looking for a solution in earnest. How about Android? At this point in my story, I m sure you re asking why I didn t just use one of the numerous Android phones that came through my hands. The answer is simple my calendar and contacts are probably some of my most personal data, and I m not willing to store them outside of my direct control, for reasons similar to those which are driving the development of the FreedomBox. When Android first came out, it could only talk to Google services, which didn t meet my hard requirement for personal data storage. One of my co-workers had his Google account suspended for violating the terms-of-service; he asked what he had done, but they wouldn t say. He asked if he could get his data back, and they said no . They invited him to create a new account, but it would not ever get any of the old data. A few days later, he got a nice apologetic email letting him know that they d made a mistake and that he hadn t, in fact, violated any of the terms-of-service, and that his old account was restored with all of his data intact. Wasn t that nice of Google? WebOS? I got a WebOS phone over the summer and discovered that while it had multiple contact/calendar back ends, none of them used standard protocols and so you only had the choice between multiple corporate data centers, which isn t actually a choice at all. Furthermore, the WebOS phone refused route PAN packets over the phone network, even though I have a data plan which allows this. It s not that it couldn t support it, it s that it refused. A couple of weeks after the WebOS phone arrived, HP canceled all of their WebOS hardware products, which made me less interested in trying to solve this problem. Android recovers About the same time the WebOS phone arrived, I discovered that Google had published enough information about the calendar/contacts internals for Marten Gajda to write CalDAV-Sync and CardDAV-Sync. And then, Andrew McMillan wrote aCal, which is a complete replacement for the built-in calendar and contacts applications and supports CalDAV and CardDAV. With two different standards-compliant solutions available, it seemed like it might be time to try Android again. I d love for CardDAV-Sync and CalDAV-Sync to become free software like aCal is. Andrew makes money from aCal by offering it for sale via the Android Market, while still publishing the sources for those who want to build their own copy. CalDAV/CardDAV on Linux I think the most widely known CalDAV server for Linux is probably DAViCal, a huge pile of PHP and SQL sitting on top of Apache. I m sure it s suitable for running on a server and being accessed over the internet, but I m not interested in that, nor am I interested in having my laptop run Apache and PostgreSQL. I found a tiny little CalDAV server, Radicale, which seemed like a lot better fit. It s written in Python and uses the usual Python HTTP server infrastructure, which provides SSL and authentication support along with some fairly convenient APIs for parsing and generating HTML. Before long, I discovered that Radicale was actually too simple for my needs. It stores the whole calendar in a single file, re-parsing it whenever a request is made, so a calendar with just hundreds of entries caused the server to slow down enough that evolution would time-out when talking to it. Also, Radicale doesn t actually parse the calendar entries completely, it has some ad-hoc code that finds various pieces of data, but without dealing with the whole syntax. I started hacking at Radicale to see how far I could get. I changed the storage code to store one event per file, then added hooks to use git for change management. Then, I found a full vcalendar/vcard parsing library in python, vobject, which I used to replace the ad-hoc parsing code. Finally, I added support for VCARD entries as well, allowing the system to store both calendar and contact information. Introducing Calypso, a CardDAV/CalDAV server With this much divergence from the original project, I ve figured I d best rename things to avoid confusion, so I decided to call it calypso , after a brief trip through the dictionary looking for names starting with ca . Calypso works with evolution, iceowl and the Android CalDAV/CardDAV plugins. It does not yet work with aCal; for some reason aCal cannot find any calendars on the server. Calypso also supports importing calendar changes from the command line, allowing you to integrate support into a text-based email application like notmuch or mutt. Calypso is available via git from git://keithp.com/git/calypso and is distributed under the GPL (v3 or later). I still consider it a work derived from Radicale, and so the code retains all of the Radicale copyrights along with my own. Using Calypso Initial setup Calypso runs as a regular user, all data are stored in ~/.config/calypso. To initialize calypso:

$ mkdir ~/.config/calypso ~/.config/calypso/calendars
$ cat > ~/.config/calypso/config << EOF
[server]
ssl=true
certificate=/home/keithp/.config/calypso/ssl/server.crt
key=/home/keithp/.config/calypso/ssl/server.key
[acl]
;type=htpasswd
type=fake
filename=/home/keithp/.config/calypso/passwd

Running calypso Then run calypso:

$ python ./calypso.py

No, I haven t figured out how to install it Creating new calendars To add a new database:

$ mkdir -p ~/.config/calypso/calendars/private/my_calendar
$ cd ~/.config/calypso/calendars/private/my_calendar
$ git init
$ git commit --allow-empty -m'initialize new calendar'

The new calendar should now be visible as https://localhost:5233/private/my_calendar You can add files to the directory at any time; calypso will check the directory mtime at each operation and update its internal state from that on disk automatically when the directory changes. Importing files Given a set of files with VCALENDAR or VCARD entries, you can import them with:

$ calypso --import private/my_calendar <filenames...>

This will update any changed entries and add any new ones. ToDo list for calypso

1. Document the config file contents.
2. Make it installable
3. Figure out what aCal wants
4. Support calendar creation

More Android info

• Synker. This provides a desktop widget which starts the sync process running on a list of accounts. This makes it easy to manually synchronize with the laptop when you are connected
• Contact Editor. This lets you fully edit contacts synchronized over CardDAV. The built-in contact editor doesn t let you change anything other than the name for some reason.

I m running cyanogenmod on my Nexus S as that provides PAN support. With PAN, I can create a network link between laptop and phone which doesn t depend on any local WiFi infrastructure and which gives both phone and laptop static IP addresses, allowing me to configure the sync URLs statically on the phone. I d use mDNS, but Android doesn t bother to support that.

29 and counting

Time flies... and very fast.

There are so many things I'd like to share and tell you about but that would make a long and boring post, so I'll divide it in future entries, so I can keep this diary alive, keep people informed and tell a little bit about my story. Eventually, I may split this diary into a technical and a non-technical blog, but for now, I will stick around mixing topics. :-) Last time I wrote was back in 2009 at DebConf9 in Spain and a lot of things happened that year. Not only for Debian, but also for me.


My last year studying Computer Science was 2009 and that same year I was working full time, I helped organizing a large event for Computer Students (ENECOMP 2009), we had some troubles and side effects with Influenza A (H1N1) in my city (Curitiba, PR) and towards the end of the year I got really sick (not from Influenza) and it took a while to know what was going, during this process I did hurt some people I really love and I'm truly sorry. The past, 2010 went on as busy and unstable as 2009, my laptop's hard disk broke twice (January and October) and I survived thanks to backup and recovery procedures. I took some wrong decisions but fixed it later and learned a lot from my mistakes, decisions and choices.


Then 2011 came, my laptop's hard disk broke again and I lost almost 6 months of emails, I find out that after the second crash in 2010 part of my backup got b0rk3d. Fine, I didn't die, everything seems to be fine, but I'm still catching up with a lot of work and backlog. At different times one of my parents also got really sick and I learned quite a bit on how to deal with it.


Due to the laptop and the different things going on in my life I didn't work for Debian as much as I want and I still feel guilty. :-( But that's fine, this is a kind of public apologize for my peers in Debian (i18n, mirrors, release), I still want to help all of you, but I still have quite a bit of backlog to clean and quite a few things to learn in the process. And in general, I'm really sorry if I didn't (or I couldn't) work for Debian as much as I want.


Although I'm taking the chance to apologize in this post, I still want to make clear that I'm quite happy. This last year was pretty good, I had the opportunity to attend DebConf again and also got to know Sarajevo and Zagreb. New lessons were learned, better work cycles, nicer solutions. Today I'm completing 29 years old. 29. This is quite some time, not a long ride, but for one reason or another, I have high hopes for 2012 and what awaits me. And I do plan to learn more, to help more and whenever possible, to contribute back.

Last time I wrote here I mentioned about my graduation and another post to talk about that, I'll write about it as soon as I get my diploma (because of different printing problems it was delayed for almost 2 years). And I do plan to write more frequently. :-)

My goal with the 3.0 (quilt) source format has always been to standardize the patch management in Debian source packages. This message seems to have been well understood. dbs and dpatch have been deprecated by their respective maintainers. I made numerous efforts to make this source format useful in as many use cases as possible (but some improvements are still possible) and I have added hints to encourage maintainers to switch. Thanks to this, the adoption rate of this new source format has been very good and it s now the most widely used source package format in Debian only two years after its introduction in Debian unstable. With 9829 source package using 3.0 (quilt) , it surpassed the number of source package still using 1.0 (7368). (Those numbers have been taken from http://upsilon.cc/~zack/stuff/dpkg-v3/ on december 13th 2011.) The number of source packages using 3.0 (quilt) doubled this year.

(Click on the picture to see it full size) Of the 7368 packages using the old format, 6816 packages trigger the missing-debian-source-format lintian tag. This means that only 552 source packages have explicitly opted to keep using the old format and that the bulk of the remaining packages are rarely updated packages that have not been switched yet.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

The route cache is a Linux kernel component enabling route lookups to be faster by caching the results in some table and checking it before issuing a regular lookup in the route tables. When using Linux as a router, the inefficiency of the route cache can hinder the performances of your box. The documentation on this component is scarce and it is difficult to find up-to-date bits on how the route cache works and how to tune it. The book Understanding Linux Network Internals from O Reilly is an exception and contains valuable information on how the route cache works. Even if the book is targeted at 2.6.12, the part on the route cache is still quite accurate. Unfortunately, it fails to provide appropriate tips on how to monitor and tune the route cache. I hope to provide here a concise view of the route cache, as it is implemented in Linux 3.1. It is protocol-dependent¹ and I will cover only the IPv4 version here.

Overview of the route cache subsystem To handle an incoming or outgoing IP datagram, the kernel needs to issue a lookup in the route tables. While it seems to be quite trivial, several questions need to be answered:

• Does the source address and the destination address appear to be valid?
• Is the source address a martian address²?
• Is the destination address mine or should I forward the packet?
• Which route table should I use?
• Does the destination address match this route? And this one?
• Can I currently contact the gateway that I should use?

Those checks can be a bit time consuming, even for small route tables. To avoid them for each packets, Linux maintains a route cache which is queried before doing a regular lookup and updated after each one. You can dump it with ip -s route show cache:

$ ip -s route show cache
198.51.100.7 from 203.0.113.2 via 192.0.2.1 dev eth1
  cache used 7 age 2sec ipid 0x1fce rtt 131ms rttvar 45ms cwnd 10
198.51.100.17 from 203.0.113.15 via 192.0.2.1 dev eth1
  cache used 3 age 0sec ipid 0xc3bd
local 192.0.2.18 from 203.0.113.15 dev lo src 192.0.2.18
  cache <local> used 154 age 1sec iif eth0

Here are two examples:

• If Linux receives a packet from 198.51.100.17 to 203.0.113.15, it will find this flow in the route cache. Therefore, it already knows that it should forward the packet to 192.0.2.1. No checks needed.
• If it receives a packet from 198.51.100.16 to 203.0.113.15, there is no appropriate entry in the route cache and therefore, the system will have to look at the route tables. It is likely to use the exact same entry than if the source was 198.51.100.17 but maybe there is some policy routing requesting the use of a special route table or 198.51.100.16 is a local address and the packet will therefore be classified as martian.

The schema below shows how this cache is implemented³. It uses a separate hash table (BSD systems keep the cache in the routing table). Each bucket is a chained list of route cache entries. Once an entry has been added to the route cache, there are several ways to remove it. Most entries are removed by the garbage collector which will scan the route cache and remove invalid and older entries. It will be triggered when the route cache is full or at regular interval, once a certain threshold has been met.

Available knobs There are several values you can tune. Most of them are available in sysctl.

• rhash_entries is the size of the hash table⁴. If you don t specify it on the kernel command line, it is computed dynamically based on the memory available on your system. You can view its value by looking at something like IP route cache hash table entries: 262144 (order: 9, 2097152 bytes) in the kernel logs.
• net.ipv4.route.max_size is the maximum number of entries in the route cache. Except under exceptional circumstances, this value is never exceeded.
• net.ipv4.route.gc_elasticity is the target average length of a chain in the route cache. The garbage collector will work harder if this value is exceeded. This means that if you multiply this value by the value of rhash_entries, you will get the target average number of entries in the route cache.
• net.ipv4.route.gc_min_interval_ms is the minimum delay between two runs of the garbage collector, except when the cache is full. The default value should be fine.
• net.ipv4.route.gc_thresh is a threshold triggering the garbage collector every net.ipv4.route.gc_min_interval_ms milliseconds.
• net.ipv4.route.gc_timeout is the base value to determine if an entry is old enough to be removed or not. Whatever its value, the garbage collector will attempt to remove the same number of entries. However, this value could potentially influence its efficiency. See below for more details on this.

You may find documentation about those obsolete sysctl values:

• net.ipv4.route.secret_interval has been removed in Linux 2.6.35; it was used to trigger an asynchronous flush at fixed interval to avoid to fill the cache.
• net.ipv4.route.gc_interval has been removed in Linux 2.6.38. It is still present until Linux 3.2 but has no effect. It was used to trigger an asynchronous cleanup of the route cache (but in a different way of what the garbage collector does). The garbage collector is now considered efficient enough for the job.

Statistics & monitoring Linux maintains some statistics about the use of the route cache. You can find them in /proc/net/stat/rt_cache. The command lnstat can print them nicely for you:

$ lnstat -s1 -i1 -c-1 -f rt_cache
rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache 
 entries   in_hit in_slow_ in_slow_ in_no_ro   in_brd in_marti in_marti  out_hit out_slow out_slow gc_total gc_ignor gc_goal_ gc_dst_o in_hlist out_hlis 
                       tot       mc      ute            an_dst   an_src              _tot      _mc                ed     miss  verflow  _search t_search 
 3096848    42309      686        0        0        0        0        0        3        0        0      674      672        0        0    27644        8 
 3096984    41405      636        0        0        0        0        0        3        0        0      623      621        0        0    28189        8 
 3097160    42483      700        0        0        0        0        0        5        0        0      694      692        0        0    29506       12 

Except for the first column, lnstat outputs values in units per second. Let s review some of those values:

• rt_cache_entries is the number of entries in the route cache. You should compare it with net.ipv4.route.max_size and ensure that the cache is never full to avoid triggering the garbage collector too often.
• rt_cache_in_hit and rt_cache_out_hit are the number of regular lookups avoided because the result was found in the cache for incoming and outgoing packets, respectively. When the system is a router, most lookups only happen on the incoming side. You should compare this value with rt_cache_in_slow_tot and rt_cache_in_out_slow_tot which are the number of lookups in the route tables. On this system, the efficiency of the route cache is more than 98% which is quite good.
• rt_cache_gc_total is how often the garbage collector was requested to be triggered while rt_cache_gc_ignored corresponds to how often it was finally not run because it has already been triggered shortly beforehand (less than net.ipv4.route.gc_min_interval_ms milliseconds). The difference between those two values should be very small to ensure that the garbage collector does not run more a handful of times per second.
• rt_cache_gc_goal_miss is how often the garbage collector was not able to fulfill its goal. This value should rarely be different of 0.
• rt_cache_gc_dst_overflow is how often the route cache is bigger than the allowed maximum size. This should never happen except when you try to shrink the cache.
• rt_cache_in_hlist_search and rt_cache_out_hlist_search is how often a lookup in the cache had to look at the next entry in the chain for the computed bucket: each time Linux has to follow the next pointer in a cache entry, it increments one of those counters. It is a clue on the average length of chains in the route cache hash table. Compare those values with the number of cache lookups (hit and miss).

As an illustration, here is a plot of the various statistics exposed above for a router whose route cache receives about 1000 routes par second: rhash_entries is 1,048,576, as is net.ipv4.route.gc_thresh. Therefore, the garbage collector was requested to be run when the number of entries goes above this level. Because the cache is not full, it is only really run twice per seconds (the value of net.ipv4.gc_interval_ms is 500). net.ipv4.gc_elasticity is set to 3. This explains why the garbage collector is aggressive when the number of entries reached 3,145,728. As you can see, the efficiency is near 100% all the time. The percentage of collisions is rt_cache_in_hlist_search ratio to the sum of rt_cache_in_hit and rt_cache_in_slow_tot.

Tuning Do you need to modify any of those values? You have two questions to ask yourself:

1. How much efficiency do you want to get from the route cache?
2. How much memory do you want to dedicate to the route cache?

As a rule of thumb, two millions entries eat about 500 MB of memory on a 64bit system. You should be able to compute the average memory usage and the maximum memory usage from the values of net.ipv4.route.max_size, rhash_entries and net.ipv4.route.gc_elasticity. For example, if the route cache hash table has 262,144 buckets, the maximum allowed number of entries in the cache is 4,194,304 and net.ipv4.route.gc_elasticity is set to 8, the memory usage will be 500 MB on average and 1 GB max. If this is too much, you will need to lower some values. Look at the previous section to compute the current efficiency of your route cache. It should be above 90%. If you are dissatisfied with that, you could increase the cache size. If you want to double the number of entries, double net.ipv4.route.max_size, net.ipv4.route.gc_thresh and rhash_entries but keep net.ipv4.route.gc_elasticity to 8. For the garbage collector to be efficient, the route cache should not be filled too fast. The garbage collector should be able to cope with this situation but this may impact performance because it needs to walk several times the route cache to find entries to expire. Watching gc_goal_miss may give you a hint about this: if this value starts to be different of 0, lower net.ipv4.gc_timeout.

In-depth look into the garbage collector Those different pieces of advice may have puzzled you. We need to understand how the garbage collector works to better cope with them. The garbage collector is triggered when a new entry needs to be added to the cache and the number of entries is superior to net.ipv4.route.gc_thresh. It is implemented in rt_garbage_collect() function. It will do nothing if it has been called less than net.ipv4.route.gc_interval_ms milliseconds and the cache is not full (more than net.ipv4.route.max_size entries).

Setting a goal The garbage collector will first assess the situation. It will look how the number of entries currently in the cache compares to the product of rhash_entries and net.ipv4.gc_elasticity:

1. Above this limit, it will try to remove at least rhash_entries entries.
2. Otherwise, it will try to remove no more than half the difference.

If you look at the previous plot, you can easily see the difference when the garbage collector is not aggressive (above 1,048,576 entries but below 3,145,728) and when it is (above 3,145,728 entries). If we assume the garbage collector is able to meet its goal, we can easily simulate its algorithm: The first plot shows what would happen if about 2000 routes are added per second with rhash_entries equal to 262,144 and net.ipv4.route.gc_elasticity set to 8. When net.ipv4.route.gc_threshold is met, the garbage collector has almost no effect. However, when the number of entries reaches 2,097,152, the garbage collector sets its goal to 262,144. From this point, the number of entries oscillates around 2 millions entries. On the second plot, rhash_entries is now equal to 1,048,576 but net.ipv4.route.gc_elasticity has been set to 2. Therefore, the aggressive part of the garbage collectors kicks at the same threshold than for the previous plot. However, its goal is now four times larger and the oscillations have a larger amplitude. The fact that the aggressive part of the garbage collector kicks less often is nullified by the fact that it needs to remove more entries each time. Because of the slight impact on cache efficiency, it seems better to keep net.ipv4.route.gc_elasticity around 8, or 4 if we want to keep shorter chains (but there seems to be no improvement to do so). The other plots show what happens when there is a sudden surge in the number of routes added or when there is a pause.

Meeting the goal Now that we understand how rhash_entries, net.ipv4.route.gc_elasticity and net.ipv4.route.gc_threshold interact, let s look at net.ipv4.route.gc_timeout. Once the garbage collector has set its goal and if it is positive, it needs to choose which entries to remove from the cache. It walks the hash table from the position of its last run and remove entries until its goal is met. If the entry is not current anymore (for example, the network interface associated to it has changed its IP configuration), it is removed. Otherwise, the system looks at the age of the entry and its position in the chain. If the entry is the first in the chain, it is kept only if its age is below net.ipv4.route.gc_timeout. If it is second, it is kept only if its age is below half of net.ipv4.route.gc_timeout. If it is third, the threshold is a quarter of net.ipv4.route.gc_timeout, and so on. The garbage collector will favor short chains. If after a full run of the hash table, the garbage collector was not able to meet its goal, it will start again but will behave more aggressively, as if net.ipv4.route.gc_timeout is set to half of its value. It will do as many passes as necessary until its goal is met or until there is no way to remove any entry (or until it has spent too much time). Once the garbage collector has switched to this more aggressive behavior, it will keep being aggressive for a few cycles (a bit less for each cycle). With a very large value of net.ipv4.route.gc_timeout, the garbage collector will have a hard time to find entries to expire. It will need to do several passes until it is able to expire some entries. On the other hand, if you choose a very small value, the garbage collector may remove entries that were just added, even if there are older entries further in the hash table. For a more in-depth coverage of how the route cache works, look at chapter 33 of Understanding Linux Network Internals. Just be aware that multipath route caching has been removed (and was never really used) and asynchronous cleanup (rt_periodic_timer) does not exist anymore.

Conclusion When tuning the route cache, rhash_entries, net.ipv4.route.gc_elasticity, net.ipv4.route.max_size and net.ipv4.route.gc_threshold are related and should not be modified independently. net.ipv4.route.gc_thresh should be below the product of net.ipv4.route.gc_elasticity and rhash_entries while net.ipv4.route.max_size should be above this value. Linux exposes several interesting metrics related to the route cache. Monitoring them allows one to watch the efficiency of the route cache and may uncover some anomalies (garbage collector running too often, difficulty to remove routes from the cache, ). The route cache subsystem still evolves and some old behaviors have been dismissed. The best documentation is, unfortunately, still the code and other documentations tend to become obsolete. The IPv6 route cache is quite different of the one presented here. Don t blindly apply the same tuning for IPv6.

---

1. Linux provides a protocol-independent destination cache subsystem (DST). This component is not a generic cache layer and only enables loose coupling with external subsystems.
2. Martian addresses are addresses that cannot be used as a source address, either because they are reserved for special-use (like a multicast address) or because of the use of reverse path filtering which checks if a packet received on one interface would be answered on the same interface, as defined in RFC 3704. This feature is enabled by setting rp_filter in Linux.
3. For more details, you may want to look at include/net/dst.h, include/net/route.h and net/ipv4/route.c.
4. In fact, the size of the hash table is always a power of two. If specified, rhash_entries is rounded to the next power of two and, internally, is stored as rt_hash_mask + 1.

It s been a couple days since I returned from this year s WebKitGTK+ hackfest in A Coru a, Spain. The weather was very nice, not too cold and not too rainy, we had great food, great drinks and I got to meet new people, and hang out with old friends, which is always great! I think this was a very productive hackfest, and as usual a very well organized one! Thanks to the GNOME Foundation for the travel sponsorship, to our friends at Igalia for doing an awesome job at making it happen, and to Collabora for sponsoring it and granting me the time to go there! We got a lot done, and although, as usual, our goals list had many items not crossed, we did cross a few very important ones. I took part in discussions about the new WebKit2 APIs, got to know the new design for GNOME s Web application, which looks great, discussed about Accelerated Compositing along with Joone, Alex, Nayan and Martin Robinson, hacked libsoup a bit to port the multipart/x-mixed-replace patch I wrote to the awesome gio-based infrastructure Dan Winship is building, and some random misc. The biggest chunk of time, though, ended up being devoted to a very uninteresting (to outsiders, at least), but very important task: making it possible to more easily reproduce our test results. TL;DR? We made our bots and development builds use jhbuild to automatically install dependencies; if you re using tarballs, don t worry, your usual autogen/configure/make/make install have not been touched. Now to the more verbose version! The need For a couple years now we have supported an increasingly complex and very demanding automated testing infrastructure. We have three buildbot slaves, one provided by Collabora (which I maintain), and two provided by Igalia (maintained by their WebKitGTK+ folks). Those bots build as many check ins as possible with 3 different configurations: 32 bits release, 64 bits release, and 64 bits debug. In addition to those, we have another bot called the EWS, or Early Warning System. There are two of those at this moment: one VM provided by Collabora and my desktop, provided by myself. These bots build every patch uploaded to the bugzilla, and report build failures or passes (you can see the green bubbles). They are very important to our development process because if the patch causes a build failure for our port people can often know that before landing, and try fixes by uploading them to bugzilla instead of doing additional commits. And people are usually very receptive to waiting for EWS output and acting on it, except when they take way too long. You can have an idea of what the life of an EWS bot looks like by looking at the recent status for the WebKitGTK+ bots. Maintaining all of those bots is at times a rather daunting task. The tests require a very specific set of packages, fonts, themes and icons to always report the same size for objects in a render. Upgrades, for instance, had to be synchronized, and usually involve generating new baselines for a large number of tests. You can see in these instructions, for instance, how strict the environment requirements are yes, we need specific versions of fonts, because they often cause layouts to change in size! At one point we had tests fail after a compiler upgrade, which made rounding act a bit different! So stability was a very important aspect of maintaining these bots. All of them have the same version of Debian, and most of the packages are pinned to the same version. On the other hand, and in direct contradition to the stability requirement, we often require bleeding edge versions of some libraries we rely on, such as libsoup. Since we started pushing WebKitGTK+ to be libsoup-only, its own progress has been pretty much driven by WebKitGTK+ s requirements, and Dan Winship has made it possible to make our soup backend much, much simpler and way more featureful. That meant, though, requiring very recent versions of soup. To top it off, for anyone not running Debian testing and tracking the exact same versions of packages as the bots it was virtually impossible to get the tests to pass, which made it very difficult for even ourselves to make sure all patches were still passing before committing something. Wow, what a mess. The explosion^Wsolution So a few weeks back Martin Robinson came up with a proposed solution, which, as he says, is the nuclear bomb solution. We would have a jhbuild environment which would build and install all of the dependencies necessary for reproducing the test expectations the bots have. So over the first three days of the hackfest Martin and myself hacked away in building scripts, buildmaster integration, a jhbuild configuration, a jhbuild modules file, setting up tarballs, and wiring it all in a way that makes it convenient for the contributors to get along with. You ll notice that our buildslaves now have a step just before compiling called updated gtk dependencies (gtk is the name we use for our port in the context of WebKit), which runs jhbuild to install any new dependencies or version bumps we added. You can also see that those instructions I mentioned above became a tad simpler. It took us way more time than we thought for the dust to settle, but it eventually began to. The great thing of doing it during the hackfest was that we could find and fix issues with weird configurations on the spot! Oh, you build with AR_FLAGS=cruT and something doesn t like it? OK, we fix it so that the jhbuild modules are not affected by that variable. Oh, turns out we missed a dependency, no problem, we add it to the modules file or install them on the bots, and then document the dependency. I set up a very clean chroot which we could use for trying out changes so as to not disrupt the tree too much for the other hackfest participants, and I think overall we did good. The aftermath By the time we were done our colleagues who ran other distributions such as Fedora were already being able to get a substantial improvements to the number of tests passing, and so did we! Also, the ability to seamlessly upgrade all the bots with a simple commit made it possible for us to very easily land a change that required a very recent (as in unreleased) version of soup which made our networking backend way simpler. All that red looks great, doesn t it? And we aren t done yet, we ll certainly be making more tweaks to this infrastructure to make it more transparent and more helpful to the users (contributors and other people interested in running the tests). If you ve been hit by the instability we caused, sorry about that, poke mrobinson or myself in the #webkitgtk+ IRC channel on FreeNode, and we ll help you out or fix any issues. If you haven t, we hope you enjoy all the goodness that a reproducible testing suite has to offer! That s it for now, folks, I ll have more to report on follow-up work started at the hackfest soon enough, hopefully =).