On the 1st of August I noticed that the server which runs my blog among other things was having an occasional SEGV from a sshd process. Unfortunately I was busy and didn t pay much attention to this, which turned out to be a big mistake. On the 12th of September I started investigating this properly and noticed that when someone tried to connect to ssh with password authentication sshd would SEGV after it was denied access to a shared memory region or a semaphore which had a SE Linux type of unconfined_t. I added some SE Linux auditallow rules and discovered that the memory region in question was created by the ssh client. Shortly after that I came to the conclusion that this wasn t some strange feature of ssh (or one of the many shared objects it uses) but hostile activity. The ssh client appeared to be storing passwords that it used in a shared memory region and sshd was also collecting passwords in the same region and presumably offering them to a ssh client which uses some extension to the ssh protocol. The sshd process was crashing because it couldn t handle EPERM on access to shared memory or semaphores. Presumably if the system in question wasn t running SE Linux then the exploit would have remained undetected for a lot longer. At this stage we don t know how the attacker got in. Presumably one of the people with root access ran a ssh client on a compromised system and had their password sniffed. One such client system was mysteriously reinstalled at about that time, the sysadmin of the system in question claimed to have no backups which made it impossible to determine if that system had been compromised. I believe that the sysadmin of the client system knew that their system was compromised, kept that information secret, and allowed other systems to become and remain compromised. The attacker made no good effort to conceal their presence, they replaced ssh, sshd, and ssh-add and didn t bother changing the Debian checksums so the debsums program flagged the files as modified. Note that I have kept copies of the files in question and am willing to share them with anyone who wants to analyse them. Steinar H. Gunderson has named this trojan Ebury [1]. Recovery By the evening of the 13th of September I had the system mostly working again. Jabber still isn t working because ejabberd is difficult to get working at the best of times, I am now investigating whether there is a better Jabber server to use, but as I don t use Jabber often this hasn t been a priority for me. Some of the WordPress plugins I use and all of the WordPress themes that are installed were outside the Debian packaging system, as I couldn t be sure that they hadn t been altered (because the people who wrote WordPress plugins don t keep old versions online) I had to upgrade to the newer versions. Of course the newer versions weren t entirely compatible so I had to use a different theme and I couldn t get all plugins working. Link Within no longer works, not that it ever worked properly [2], I wanted to try Outbrain again but their web site won t let me login (and they haven t responded to my support request). Does anyone know of a good WordPress plugin to provide links to related content? Either related content on my blog or on the Internet in general will be OK. Some people have asked me about the change in appearance of my blog. It was simply impossible (for someone with my PHP skills) to get my blog looking the same way as it did before the server was cracked. I think that the new look is OK and don t mind if people think it looks likw a VW advert VW make great cars, I was very satisfied with the VW Passat I used to drive. Future Plans I had bought some Yubikeys (USB devices that generate one-time passwords) [3] to control access to that server, if I had configured the software to use them then this might not have happened. The use of one-time password devices can prevent passive password-sniffing attacks. It would still allow active attacks (such as using ControlPath/ControlMaster options on the ssh client to allow a hostile party to connect later (EG the -M, -S, and -o ControlPersist options for the ssh client). It s a pity that there doesn t seem to be a way to configure the ssh server to disable ControlMaster. Conclusion It would be good to have some changes to sshd to allow more restrictions on what a client can request, as ControlMaster functionality isn t needed by most users it should be possible to disable it. SE Linux doesn t protect against a compromised client system or any other way of stealing passwords. It did do a good job of stopping Ebury from doing all the things it wanted to do and thus making me aware of the problem. So I count this as a win for SE Linux. Yubikeys are the cheapest and easiest way of managine one-time passwords. I had already bought some for use on the system in question but hadn t got around to configuring them. I have to make that a priority.

• [1] http://be-geek.com/linux/2011/11/15/steinar-h-gunderson-ebury-a-new-ssh-trojan
• [2] http://etbe.coker.com.au/2010/04/30/link-within/
• [3] http://www.yubico.com/

Backups are something I don't do well. I hope to write a more thorough article about them at some point. For now, here are some tips. Like most home users, I don't have access to a tape drive or jukebox for backups. I'm therefore looking for a solution that backs up to a local block device. I haven't ruled out a solution that backs up to DVD-Rs, but I haven't found a suitable one. I first tried rsnapshot, which uses hard link trees to represent increments. I found that this resulted in enormous filesystem metadata, sufficient to cause the machine I was running it on (an embedded ARM system) to slow to a crawl. I'd recommend to avoid anything using this technique. I more recently have been using rdiff-backup. This has appeared to work quite well for a number of years. However, it does not gracefully handle the backup volume being full, spewing python backtraces. This is compounded with some confusing use of temporary directies and not honouring $TMPDIR. This has been known about for years (1, 2). I've just independently discovered these problems for the second time. Aside from that, because it does not de-duplicate files, it is sensitive to large files being moved around in the source being backed up. It also cannot do a dry-run estimate of the disk space required for the next increment. So: time to move on. Back in 2010 I discovered bup, an intriguing backup tool that used a git-style backing store. It's interesting enough that I packaged it and use it in a few situations, but I wouldn't rely on it for my main backups, for two reasons: one, it's not tried-and-tested enough yet (but that is rapidly being resolved), two: you can't get rid of old increments, so your backup volume will always increase in size over time, and never throw away old data. So, for now: move on. Next on my list to try is Lars Wirzenius' obnam.

Summary: It is important that we (the Debian community that relies on OpenPGP through GNU Privacy Guard) stop using short key IDs. There is no vulnerability in OpenPGP and GPG. However, using short key IDs (like 0x70096AD1) is fundementally insecure; it is easy to generate collisions for short key IDs. We should always use 64-bit (or longer) key IDs, like: 0x37E1C17570096AD1 or 0xEC4B033C70096AD1. TL;DR: This now gives two results: gpg --recv-key 70096AD1

Some background, and my two keys Years ago, I read dkg's instructions on migrating the Debian OpenPGP infrastructure. It told me that the time and effort I had spent getting my key into the strong set wasn't as useful as I thought it had been. I felt deflated. I had put in quite a bit of effort over the years to strongly-connect my key to a variety of signatures, and I had helped people get their own keys into the strong set this way. If I migrated off my old key and revoked it, I'd be abandoning some people for whom I was their only link into the strong set. And what fun it was to first become part of the strong set! And all the eyebrows I raised when I told people I was going meet up with people I met on a website called Biglumber... I even made it my Facebook.com user ID. So if I had to generate a new key, I decided I had better really love the short key ID. But at that point, I already felt pretty attached to the number 0x70096AD1. And I couldn't come up with anything better. So that settled it: no key upgrade until I had a new key whose ID is the same as my old key. That dream has become a reality. Search for my old key ID, and you get two keys!

$ gpg --keyserver pgp.mit.edu --recv-key 0x70096AD1
gpg: requesting key 70096AD1 from hkp server pgp.mit.edu
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 2
gpg:               imported: 2  (RSA: 1)

I also saw it as an opportunity: I know that cryptography tools are tragically easy to mis-use. The use of 32-bit key IDs is fundamentally incorrect -- too little entropy. Maybe shocking people by creating two "identical" keys will help speed the transition away from this mis-use.

A neat stunt abusing --refresh-keys Thanks to a GNU Privacy Guard bug, it is super easy to get my new key. Let's say that, like many people, you only have my old key on your workstation:

$ gpg --list-keys   grep 70096AD1
pub   1024D/70096AD1 2005-12-28

Just ask GPG to refresh:

$ gpg --keyserver pgp.mit.edu --refresh-keys
gpg: refreshing 1 key from hkp://pgp.mit.edu
gpg: requesting key 70096AD1 from hkp server pgp.mit.edu
gpg: key 70096AD1: public key "Asheesh Laroia <asheesh@asheesh.org>" imported
gpg: key 70096AD1: "Asheesh Laroia <asheesh@asheesh.org>" not changed
gpg: Total number processed: 2
gpg:               imported: 1  (RSA: 1)
gpg:              unchanged: 1
gpg: no ultimately trusted keys found

You can see that it set out to refresh just 1 key. It did that by querying the keyserver for the short ID. The keyserver provided two hits for that query. In the end, GPG refreshes one key and actually imports a new key into the keyring! Now you have two:

$ gpg --list-keys   grep 70096AD1
pub   1024D/70096AD1 2005-12-28
pub   4096R/70096AD1 2011-03-11

There is a bug filed in GNU Privacy Guard about this. It has a patch attached. There is, at the moment, no plan for a new release.

A faster attack, but nothing truly new My friend Venkatesh tells me there is an apocryphal old Perl script that could be used to generate key ID collisions. Here in the twenty-first century, l33t h4x0rz like Georgi Guninski are trying to create collisions. In May 2010, "halfdog" posted a note to the full-disclosure list that generates PGP keys with chosen short key IDs. I haven't benchmarked or tested that tool, but I have used a different tool (private for now) that can generate collisions in a similar fashion. It takes about 3 hours to loop through all key IDs on a dinky little netbook. You don't have to use any of these tools. You can just rent time on an elastic computing service or a botnet, or your own personal computer, and generate keys until you have a match. I think that it's easy to under-estimate the seriousness of this problem: tools like the PGP Key Pathfinder should be updated to only accept 64-bit (or longer) key IDs if we want to trust their output.

My offer: I will make you a key I've been spending some time wondering: What sort of exciting demonstration can I create to highlight that this is a real problem? Some ideas I've had:

• Publish a private/public key pair whose key ID is the same as Phil Zimmerman's, original author of PGP
• Publish a private/public key pair whose key ID is the same as Werner Koch's, maintainer of GNU Privacy Guard
• Publish a set of public keys that mimic the entire PGP strong set, except where I control the private key of all these keys

The last one would be extremely amusing, and would be a hat-tip to some work discussed in Raph Levien's Google Tech Talk about Advogato. For now, here is my offer: If you send me a request signed with a key in the strong set, I will create a 4096-bit RSA public/private key pair whose 32-bit key ID is one greater than yours. So if you are 0x517DD4E4 I will generate 0x517DD4E5. I will post the keys here, along a note about who requested it, and instructions on how to import them into your keyring. (Note: I will politely decline to create a new key whose 32-bit key ID would create a collision; apologies if your key ID is just one away from someone else's.) P.S. The prize for best sarcastic retort goes to Ian Jackson. He said, "I should go and create a lot of keys with your key ID. I'll set the real name to 'Not Asheesh Laroia' so everyone is totally clear about what is going on."

"Static program analysis" is a technique whereby a program is verified for errors without actually running it. Finding bugs manually with a debugger and one's brain is tedious, so every shortcut that can help you avoid having to do so is great. There are some commercial tools available to do such analysis, some of which are rather expensive; but there are also some open source tools available to do similar things. One of these is built into clang, the C compiler of the LLVM project. Using it is fairly simple. Instead of compiling something with 'make', compile it with 'scan-build make'. This will set the CC (and similar) environment variable(s) so that before the compiler is ran, the clang static-analysis checker is run over the very same source code. The output of this checker is an HTML file with your source, but with comments added to explain the bugs which the tool found. What does that mean? well, let's look at an example, shall we? <iframe height="500px" src="http://barbershop.grep.be/~wouter/scan-view-example#LN1493" width="80%"></iframe> The iframe above contains one out of three reports produced by a scan-build run over the nbd source code (if there's no iframe, someone scrubbed some HTML in your RSS reader. Just follow the link just above instead). The other two are 'dead assignments', which might mean that I'm currently depending on some undefined behaviour (which would be bad), or it might mean that I'm being overly cautious (which makes my code future-proof, which would be good), or it might mean something else I still need to investigate. But this one is pretty interesting. In the example, there are eight, numbered, comments in the source code. The first seven show the code path which scan-build took through my code before getting at the eighth comment; and the eighth is where things go bad. In this case, when going through the function as shown, we have a NULL pointer dereference. When looking at the scan-build output, it's important to realize a few things. First, the code path shown may be just one of a number of possible code paths. For instance, the null pointer dereference would still happen if the phase function parameter would not contain the NEG_INIT bit with the client pointer set to NULL. However, clang does not show these other code paths; this is presumably an optimization ("if we've already shown that this kind of bug is possible at a particular location through one code path, don't bother recording future instances of that very same bug at the exact same location through another code path"). This means that sometimes, some of the branches shown may be completely irrelevant to the bug at hand. In this particular case, in fact, it's possible to show the bug with just the eighth comment; the first seven are in fact totally irrelevant. Second, the fact that the clang static analyser found a bug does not mean that it's possible to crash the application. Yet. In this particular case, the negotiate function will never be called with the NEG_INIT or NEG_MODERN bits not set, and with the client parameter set to NULL. That's an implicit assertion; there are a few ways in which this function may be called, but the client parameter may only be NULL if NEG_INIT and NEG_MODERN are both set at the same time. Since nbd-server doesn't currently call the negotiate function in that way, it is not currently possible to crash the server by exploiting this bug. But that doesn't mean it won't ever be possible, nor that there isn't a bug in the code. We may assume that the above rules are true, but we never check it. Adding an assertion to that effect should make sure that no future change to the code will accidentally introduce that error and cause a NULL pointer dereference. Is this a silly and useless precautionary measure? Not really. Usually, bugs happen in code not because someone wasn't thinking straight, but because there's so much going on inside a piece of software that it can be too much for any programmer to remember. If a function assumes that its parameters are within a given subset of all possible states, but does not check that this is in fact true, then when (not if) some future change incorrectly introduces a state that is outside of the assumed states, things will break. And that's Bad(TM).

29 and counting

Time flies... and very fast.

There are so many things I'd like to share and tell you about but that would make a long and boring post, so I'll divide it in future entries, so I can keep this diary alive, keep people informed and tell a little bit about my story. Eventually, I may split this diary into a technical and a non-technical blog, but for now, I will stick around mixing topics. :-) Last time I wrote was back in 2009 at DebConf9 in Spain and a lot of things happened that year. Not only for Debian, but also for me.


My last year studying Computer Science was 2009 and that same year I was working full time, I helped organizing a large event for Computer Students (ENECOMP 2009), we had some troubles and side effects with Influenza A (H1N1) in my city (Curitiba, PR) and towards the end of the year I got really sick (not from Influenza) and it took a while to know what was going, during this process I did hurt some people I really love and I'm truly sorry. The past, 2010 went on as busy and unstable as 2009, my laptop's hard disk broke twice (January and October) and I survived thanks to backup and recovery procedures. I took some wrong decisions but fixed it later and learned a lot from my mistakes, decisions and choices.


Then 2011 came, my laptop's hard disk broke again and I lost almost 6 months of emails, I find out that after the second crash in 2010 part of my backup got b0rk3d. Fine, I didn't die, everything seems to be fine, but I'm still catching up with a lot of work and backlog. At different times one of my parents also got really sick and I learned quite a bit on how to deal with it.


Due to the laptop and the different things going on in my life I didn't work for Debian as much as I want and I still feel guilty. :-( But that's fine, this is a kind of public apologize for my peers in Debian (i18n, mirrors, release), I still want to help all of you, but I still have quite a bit of backlog to clean and quite a few things to learn in the process. And in general, I'm really sorry if I didn't (or I couldn't) work for Debian as much as I want.


Although I'm taking the chance to apologize in this post, I still want to make clear that I'm quite happy. This last year was pretty good, I had the opportunity to attend DebConf again and also got to know Sarajevo and Zagreb. New lessons were learned, better work cycles, nicer solutions. Today I'm completing 29 years old. 29. This is quite some time, not a long ride, but for one reason or another, I have high hopes for 2012 and what awaits me. And I do plan to learn more, to help more and whenever possible, to contribute back.

Last time I wrote here I mentioned about my graduation and another post to talk about that, I'll write about it as soon as I get my diploma (because of different printing problems it was delayed for almost 2 years). And I do plan to write more frequently. :-)

Google Maps is now mapping the indoors. I saw an ad for this while passing through MSP yesterday (given how much time I spend on the Internet, it's strange and a little embarrassing to learn about new things on the Internet from airport billboards), since one of their initial targets is the infamous Mall of America.

Detailed floor plans automatically appear when you re viewing the map and zoomed in on a building where indoor map data is available. The familiar blue dot icon indicates your location within several meters, and when you move up or down a level in a building with multiple floors, the interface will automatically update to display which floor you re on. All this is achieved by using an approach similar to that of My Location for outdoor spaces, but fine tuned for indoors.

Thoughts about this:

• Okay, having a map of Ikea or an airport on my phone might be kind of nice.
• Shouldn't they have launched this years ago when people still went to malls?
• Shouldn't they have launched this at least earlier in the year, in time for people to know about it and have it before the one time each year when people who don't go to malls still maybe go to the mall?
• How does it know where you are, when cell and GPS so often don't work in these places? "fine tuned for indoors" is a very mysterious phrase. Based on nearby access points?
• Is your location also being transmitted if you use this service? In other words, will Google now know which stores you shop at, how many times you use the restroom, and whether you were actually shopping for gifts for other people?
• Relatedly, will this be free software? Or will it rely on proprietaryness in order to do things users would prefer it didn't do? (Answer: It's proprietary software, part of the existing proprietary Google Maps mobile application.)
• Will they sell data to the stores and malls?
• Will they start indicating various promotions on the maps?
• Will these promotions be targeted based on your past meandering behavior?
• Are they mapping less-commercial spaces, like museums?
• Are they making the map data available for others to use? Especially for truly public spaces?
• How do we get the functional part of this into OpenStreetMap?
• Is Google secretly working on computer-piloted mobility scooters?
• Where am I?

Title from John Ashbery

here's my RCBW report for this week. again, nothing spectacular, mostly uploading packages where patches were already in the BTS. thanks to all the patch creators and forwarders!

• #553968 src:blktrace: "FTBFS with binutils-gold"
  add patch from Ubuntu / Ilya Barygin, upload to DELAYED/2
• #554408 src:fuse-umfuse-iso9660: "FTBFS with binutils-gold"
  add patch from Luk Claes, upload to DELAYED/2
• #554755 src:gtodo: "FTBFS with binutils-gold"
  add patch from Bart Martens, upload to DELAYED/2
• #625319 chiark-tcl: "chiark-tcl: ftbfs with gcc-4.6 -Werror"
  add patch from peter green, upload to DELAYED/2
• #625327 filtergen: "filtergen: ftbfs with gcc-4.6 -Werror"
  add patch from peter green, upload to DELAYED/2
• #625343 gltron: "gltron: ftbfs with gcc-4.6 -Werror"
  add patch from peter green, upload to DELAYED/2
• #642709 src:ctsim: "ctsim: FTBFS: /bin/sh: 0: Can't open ./config.status"
  fix debian/rules (config. guess,sub handling and duplicate make clean), upload to DELAYED/2
• #643403 src:gtodo: "gtodo: FTBFS: callback.c:133:4: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Bart Martens, upload to DELAYED/2

This year is in its last throes and a few things, that don't really warrant their own blog post, have piled up here. To get them out of the door before 2012 starts, I'd like to go over them in the next few lines. First: there seem to be a group of rather cheap people who want to have the Humble Indie Bundle #4, but don't like to give at least 0.01 USD. Really guys, you seem to have enough money for a fast-enough computer and a (broadband) internet connection (otherwise downloading the circa 2.8 GB of games won't be a lot of fun), but you don't have the 0.01 USD to spare? If you don't like closed-source games (a sentiment I can truly understand, in fact, that was something I really pondered before buying) then just don't play them. But searching for a torrent for the HIB#4 while it is still being sold to just avoid paying the minimal fee, I can't understand. Now, some might wonder how I know this. The easy answer is, I blogged about buying the HIB#4 and mentioned, that I downloaded the games through a torrent (you can choose between direct downloads and torrents on your "login page") and a lot of people started hitting the blog entry with a referrer from a search engine, showing they'd searched for "humble indie bundle 4 torrent". Please make a decision: don't play the games or if you wish to, then pay something. The people behind those games make it easy and, in comparison to many other game studios, a fair offer. That should be worth something. For those of you who loved Josephine, there is a new awesome piece of art, Creative Commons licensed, by An-U (German homepage). The title (and the drawing) are derived from the equally awesome Katzenjammer song "On the Devil's Back" (in case you're still looking for a present, you might want to consider one of their CDs or a ticket to their next concert near you). Then, I wanted to thank all those people, who donated something over the course of 2011! Thanks a lot, especially for showing me, that you liked my services/work enough to even consider a donation! It means a lot to me, to know this. At year's end everybody is doing statistics and while I won't bore you with long in-depth analysis, I thought I throw out, what the top posts/things where (so far). Maybe you find yourself represented in there or maybe you find something new. In no particular order:

• My unofficial wine-unstable packages generated a lot of traffic and back links.
• The mini-tips series had many hits.
• My two posts (1, 2) about the Linux graphics stack garnered a lot of attention.
• The note about removing Sun's Java implementation was the first post to be mentioned in a Debian Project News issue.
• Right after my wine-unstable packages, my post about OSM and OpenLayers generated the most e-mails.
• The YouTube playlist fetcher is also pretty popular, though the Greasemonkey script collection has been picking up speed recently.

I hope your 2011 was a good year and I wish you all the best for 2012! I'm pretty sure Debian will continue to grow, but there is also a lot of work ahead, maybe you can join the effort and make Debian that much more awesome!

The route cache is a Linux kernel component enabling route lookups to be faster by caching the results in some table and checking it before issuing a regular lookup in the route tables. When using Linux as a router, the inefficiency of the route cache can hinder the performances of your box. The documentation on this component is scarce and it is difficult to find up-to-date bits on how the route cache works and how to tune it. The book Understanding Linux Network Internals from O Reilly is an exception and contains valuable information on how the route cache works. Even if the book is targeted at 2.6.12, the part on the route cache is still quite accurate. Unfortunately, it fails to provide appropriate tips on how to monitor and tune the route cache. I hope to provide here a concise view of the route cache, as it is implemented in Linux 3.1. It is protocol-dependent¹ and I will cover only the IPv4 version here.

Overview of the route cache subsystem To handle an incoming or outgoing IP datagram, the kernel needs to issue a lookup in the route tables. While it seems to be quite trivial, several questions need to be answered:

• Does the source address and the destination address appear to be valid?
• Is the source address a martian address²?
• Is the destination address mine or should I forward the packet?
• Which route table should I use?
• Does the destination address match this route? And this one?
• Can I currently contact the gateway that I should use?

Those checks can be a bit time consuming, even for small route tables. To avoid them for each packets, Linux maintains a route cache which is queried before doing a regular lookup and updated after each one. You can dump it with ip -s route show cache:

$ ip -s route show cache
198.51.100.7 from 203.0.113.2 via 192.0.2.1 dev eth1
  cache used 7 age 2sec ipid 0x1fce rtt 131ms rttvar 45ms cwnd 10
198.51.100.17 from 203.0.113.15 via 192.0.2.1 dev eth1
  cache used 3 age 0sec ipid 0xc3bd
local 192.0.2.18 from 203.0.113.15 dev lo src 192.0.2.18
  cache <local> used 154 age 1sec iif eth0

Here are two examples:

• If Linux receives a packet from 198.51.100.17 to 203.0.113.15, it will find this flow in the route cache. Therefore, it already knows that it should forward the packet to 192.0.2.1. No checks needed.
• If it receives a packet from 198.51.100.16 to 203.0.113.15, there is no appropriate entry in the route cache and therefore, the system will have to look at the route tables. It is likely to use the exact same entry than if the source was 198.51.100.17 but maybe there is some policy routing requesting the use of a special route table or 198.51.100.16 is a local address and the packet will therefore be classified as martian.

The schema below shows how this cache is implemented³. It uses a separate hash table (BSD systems keep the cache in the routing table). Each bucket is a chained list of route cache entries. Once an entry has been added to the route cache, there are several ways to remove it. Most entries are removed by the garbage collector which will scan the route cache and remove invalid and older entries. It will be triggered when the route cache is full or at regular interval, once a certain threshold has been met.

Available knobs There are several values you can tune. Most of them are available in sysctl.

• rhash_entries is the size of the hash table⁴. If you don t specify it on the kernel command line, it is computed dynamically based on the memory available on your system. You can view its value by looking at something like IP route cache hash table entries: 262144 (order: 9, 2097152 bytes) in the kernel logs.
• net.ipv4.route.max_size is the maximum number of entries in the route cache. Except under exceptional circumstances, this value is never exceeded.
• net.ipv4.route.gc_elasticity is the target average length of a chain in the route cache. The garbage collector will work harder if this value is exceeded. This means that if you multiply this value by the value of rhash_entries, you will get the target average number of entries in the route cache.
• net.ipv4.route.gc_min_interval_ms is the minimum delay between two runs of the garbage collector, except when the cache is full. The default value should be fine.
• net.ipv4.route.gc_thresh is a threshold triggering the garbage collector every net.ipv4.route.gc_min_interval_ms milliseconds.
• net.ipv4.route.gc_timeout is the base value to determine if an entry is old enough to be removed or not. Whatever its value, the garbage collector will attempt to remove the same number of entries. However, this value could potentially influence its efficiency. See below for more details on this.

You may find documentation about those obsolete sysctl values:

• net.ipv4.route.secret_interval has been removed in Linux 2.6.35; it was used to trigger an asynchronous flush at fixed interval to avoid to fill the cache.
• net.ipv4.route.gc_interval has been removed in Linux 2.6.38. It is still present until Linux 3.2 but has no effect. It was used to trigger an asynchronous cleanup of the route cache (but in a different way of what the garbage collector does). The garbage collector is now considered efficient enough for the job.

Statistics & monitoring Linux maintains some statistics about the use of the route cache. You can find them in /proc/net/stat/rt_cache. The command lnstat can print them nicely for you:

$ lnstat -s1 -i1 -c-1 -f rt_cache
rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache 
 entries   in_hit in_slow_ in_slow_ in_no_ro   in_brd in_marti in_marti  out_hit out_slow out_slow gc_total gc_ignor gc_goal_ gc_dst_o in_hlist out_hlis 
                       tot       mc      ute            an_dst   an_src              _tot      _mc                ed     miss  verflow  _search t_search 
 3096848    42309      686        0        0        0        0        0        3        0        0      674      672        0        0    27644        8 
 3096984    41405      636        0        0        0        0        0        3        0        0      623      621        0        0    28189        8 
 3097160    42483      700        0        0        0        0        0        5        0        0      694      692        0        0    29506       12 

Except for the first column, lnstat outputs values in units per second. Let s review some of those values:

• rt_cache_entries is the number of entries in the route cache. You should compare it with net.ipv4.route.max_size and ensure that the cache is never full to avoid triggering the garbage collector too often.
• rt_cache_in_hit and rt_cache_out_hit are the number of regular lookups avoided because the result was found in the cache for incoming and outgoing packets, respectively. When the system is a router, most lookups only happen on the incoming side. You should compare this value with rt_cache_in_slow_tot and rt_cache_in_out_slow_tot which are the number of lookups in the route tables. On this system, the efficiency of the route cache is more than 98% which is quite good.
• rt_cache_gc_total is how often the garbage collector was requested to be triggered while rt_cache_gc_ignored corresponds to how often it was finally not run because it has already been triggered shortly beforehand (less than net.ipv4.route.gc_min_interval_ms milliseconds). The difference between those two values should be very small to ensure that the garbage collector does not run more a handful of times per second.
• rt_cache_gc_goal_miss is how often the garbage collector was not able to fulfill its goal. This value should rarely be different of 0.
• rt_cache_gc_dst_overflow is how often the route cache is bigger than the allowed maximum size. This should never happen except when you try to shrink the cache.
• rt_cache_in_hlist_search and rt_cache_out_hlist_search is how often a lookup in the cache had to look at the next entry in the chain for the computed bucket: each time Linux has to follow the next pointer in a cache entry, it increments one of those counters. It is a clue on the average length of chains in the route cache hash table. Compare those values with the number of cache lookups (hit and miss).

As an illustration, here is a plot of the various statistics exposed above for a router whose route cache receives about 1000 routes par second: rhash_entries is 1,048,576, as is net.ipv4.route.gc_thresh. Therefore, the garbage collector was requested to be run when the number of entries goes above this level. Because the cache is not full, it is only really run twice per seconds (the value of net.ipv4.gc_interval_ms is 500). net.ipv4.gc_elasticity is set to 3. This explains why the garbage collector is aggressive when the number of entries reached 3,145,728. As you can see, the efficiency is near 100% all the time. The percentage of collisions is rt_cache_in_hlist_search ratio to the sum of rt_cache_in_hit and rt_cache_in_slow_tot.

Tuning Do you need to modify any of those values? You have two questions to ask yourself:

1. How much efficiency do you want to get from the route cache?
2. How much memory do you want to dedicate to the route cache?

As a rule of thumb, two millions entries eat about 500 MB of memory on a 64bit system. You should be able to compute the average memory usage and the maximum memory usage from the values of net.ipv4.route.max_size, rhash_entries and net.ipv4.route.gc_elasticity. For example, if the route cache hash table has 262,144 buckets, the maximum allowed number of entries in the cache is 4,194,304 and net.ipv4.route.gc_elasticity is set to 8, the memory usage will be 500 MB on average and 1 GB max. If this is too much, you will need to lower some values. Look at the previous section to compute the current efficiency of your route cache. It should be above 90%. If you are dissatisfied with that, you could increase the cache size. If you want to double the number of entries, double net.ipv4.route.max_size, net.ipv4.route.gc_thresh and rhash_entries but keep net.ipv4.route.gc_elasticity to 8. For the garbage collector to be efficient, the route cache should not be filled too fast. The garbage collector should be able to cope with this situation but this may impact performance because it needs to walk several times the route cache to find entries to expire. Watching gc_goal_miss may give you a hint about this: if this value starts to be different of 0, lower net.ipv4.gc_timeout.

In-depth look into the garbage collector Those different pieces of advice may have puzzled you. We need to understand how the garbage collector works to better cope with them. The garbage collector is triggered when a new entry needs to be added to the cache and the number of entries is superior to net.ipv4.route.gc_thresh. It is implemented in rt_garbage_collect() function. It will do nothing if it has been called less than net.ipv4.route.gc_interval_ms milliseconds and the cache is not full (more than net.ipv4.route.max_size entries).

Setting a goal The garbage collector will first assess the situation. It will look how the number of entries currently in the cache compares to the product of rhash_entries and net.ipv4.gc_elasticity:

1. Above this limit, it will try to remove at least rhash_entries entries.
2. Otherwise, it will try to remove no more than half the difference.

If you look at the previous plot, you can easily see the difference when the garbage collector is not aggressive (above 1,048,576 entries but below 3,145,728) and when it is (above 3,145,728 entries). If we assume the garbage collector is able to meet its goal, we can easily simulate its algorithm: The first plot shows what would happen if about 2000 routes are added per second with rhash_entries equal to 262,144 and net.ipv4.route.gc_elasticity set to 8. When net.ipv4.route.gc_threshold is met, the garbage collector has almost no effect. However, when the number of entries reaches 2,097,152, the garbage collector sets its goal to 262,144. From this point, the number of entries oscillates around 2 millions entries. On the second plot, rhash_entries is now equal to 1,048,576 but net.ipv4.route.gc_elasticity has been set to 2. Therefore, the aggressive part of the garbage collectors kicks at the same threshold than for the previous plot. However, its goal is now four times larger and the oscillations have a larger amplitude. The fact that the aggressive part of the garbage collector kicks less often is nullified by the fact that it needs to remove more entries each time. Because of the slight impact on cache efficiency, it seems better to keep net.ipv4.route.gc_elasticity around 8, or 4 if we want to keep shorter chains (but there seems to be no improvement to do so). The other plots show what happens when there is a sudden surge in the number of routes added or when there is a pause.

Meeting the goal Now that we understand how rhash_entries, net.ipv4.route.gc_elasticity and net.ipv4.route.gc_threshold interact, let s look at net.ipv4.route.gc_timeout. Once the garbage collector has set its goal and if it is positive, it needs to choose which entries to remove from the cache. It walks the hash table from the position of its last run and remove entries until its goal is met. If the entry is not current anymore (for example, the network interface associated to it has changed its IP configuration), it is removed. Otherwise, the system looks at the age of the entry and its position in the chain. If the entry is the first in the chain, it is kept only if its age is below net.ipv4.route.gc_timeout. If it is second, it is kept only if its age is below half of net.ipv4.route.gc_timeout. If it is third, the threshold is a quarter of net.ipv4.route.gc_timeout, and so on. The garbage collector will favor short chains. If after a full run of the hash table, the garbage collector was not able to meet its goal, it will start again but will behave more aggressively, as if net.ipv4.route.gc_timeout is set to half of its value. It will do as many passes as necessary until its goal is met or until there is no way to remove any entry (or until it has spent too much time). Once the garbage collector has switched to this more aggressive behavior, it will keep being aggressive for a few cycles (a bit less for each cycle). With a very large value of net.ipv4.route.gc_timeout, the garbage collector will have a hard time to find entries to expire. It will need to do several passes until it is able to expire some entries. On the other hand, if you choose a very small value, the garbage collector may remove entries that were just added, even if there are older entries further in the hash table. For a more in-depth coverage of how the route cache works, look at chapter 33 of Understanding Linux Network Internals. Just be aware that multipath route caching has been removed (and was never really used) and asynchronous cleanup (rt_periodic_timer) does not exist anymore.

Conclusion When tuning the route cache, rhash_entries, net.ipv4.route.gc_elasticity, net.ipv4.route.max_size and net.ipv4.route.gc_threshold are related and should not be modified independently. net.ipv4.route.gc_thresh should be below the product of net.ipv4.route.gc_elasticity and rhash_entries while net.ipv4.route.max_size should be above this value. Linux exposes several interesting metrics related to the route cache. Monitoring them allows one to watch the efficiency of the route cache and may uncover some anomalies (garbage collector running too often, difficulty to remove routes from the cache, ). The route cache subsystem still evolves and some old behaviors have been dismissed. The best documentation is, unfortunately, still the code and other documentations tend to become obsolete. The IPv6 route cache is quite different of the one presented here. Don t blindly apply the same tuning for IPv6.

---

1. Linux provides a protocol-independent destination cache subsystem (DST). This component is not a generic cache layer and only enables loose coupling with external subsystems.
2. Martian addresses are addresses that cannot be used as a source address, either because they are reserved for special-use (like a multicast address) or because of the use of reverse path filtering which checks if a packet received on one interface would be answered on the same interface, as defined in RFC 3704. This feature is enabled by setting rp_filter in Linux.
3. For more details, you may want to look at include/net/dst.h, include/net/route.h and net/ipv4/route.c.
4. In fact, the size of the hash table is always a power of two. If specified, rhash_entries is rounded to the next power of two and, internally, is stored as rt_hash_mask + 1.

(It s a new day, it s a new bug ) A regression appeared in the synaptics input driver, between the 1.4.1 and the 1.5.0 release, as reported in Debian #649002 by many users. Basically, no more touchpad on PowerPC (hi, iBook G4 users!). :-( What follows is how I tracked it down. The upstream bug report is fdo#43728. Anyone could have done so, there s no black magic involved. A little hint maybe: git bisect is really easy on X drivers, since one can build and test without the Debian packaging bits. 1st step: Getting started You need sources and build dependencies, nothing fancy. Since packages might be named a bit differently, there s a reference to the relevant upstream repositories in debian/watch for X packages.

# You need git here:
debcheckout xserver-xorg-input-synaptics synaptics.git
cd synaptics.git
# Let s get the upstream tags and branches:
cat debian/watch
git remote add upstream git://anongit.freedesktop.org/xorg/driver/xf86-input-synaptics
git fetch --all
# Let s install build dependencies:
sudo apt-get build-dep xserver-xorg-input-synaptics

2nd step: Reproducing It was said 1.5.0 was the first known buggy version, so let s make sure:

# Get the appropriate tag:
git checkout xf86-input-synaptics-1.5.0
# Prepare the build system:
autoreconf -vfi
# X finds its modules under /usr, not /usr/local:
./configure --prefix=/usr
# Build and install:
make && sudo make install

Now time to restart the display manager, and confirm that the pointer is not moving bug reproduced. Now let s check the 1.4.1 release isn t affected:

# Clean everything:
git clean -xdf
git checkout xf86-input-synaptics-1.4.1
autoreconf -vfi
./configure --prefix=/usr
make && sudo make install

And after a display manager restart, the pointer is moving again, good! For reference, the log has very different lines about appletouch, so it can be used to programmatically learn about the status (good or bad). 3rd step: Narrowing it down So, we have a known good version and a known bad version. Time for a binary search to find the guilty commit which introduced that regression!

git bisect start
git bisect good xf86-input-synaptics-1.4.1
git bisect bad  xf86-input-synaptics-1.5.0

You don t even need to know that 1.4.x and 1.5.x live in different branches, git figures that out for you:

Bisecting: a merge base must be tested
[de0dfb76444ad4160468d00515876c91a9fa20bf] synaptics 1.4.0

It s just a matter of running autoreconf && ./configure && make && sudo make install && sudo /etc/init.d/xdm restart every step of the way, so it s pretty easy. No wonder, 1.4.0 was working OK, so it can be recorded as such through git bisect good, and the binary search goes between that commit and 1.5.0. After a few iterations of git bisect good and git bisect bad, we get to the commit reported on the upstream bug. As an extra bonus, mentioning the upstream bug number/URL in the Debian bug means it can be marked as forwarded there and we ll receive automatic notifications when its status changes, thanks to bts-link. Now, if you want to provide a patch for this bug, you may want to try and revert this commit. 4th step: Providing a patch Let s go back to the affected release and try to revert the guilty commit:

git checkout xf86-input-synaptics-1.5.0
git revert 13543b156d78bc4d01a19844a5ee8f283269621b

Unfortunately, some later commits modified the affected areas so we re running into conflicts:

error: could not revert 13543b1... eventcomm: replace synaptics-custom TEST_BIT with server's BitIsOn.
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

Let s try to understand what the offending commit did: it replaced a set of macros with a single one, taken from the server. Both the old TEST_BIT() and the new BitIsOn() take 2 arguments, but the order is reversed. Reverting it should only be a matter of introducing the TEST_BIT() macro again (along with the two other ones it needs), and using it. The problem is replacing all BitIsOn(foo,bar) with TEST_BIT(bar,foo): foo and bar might not be trivial, they could be complex C expressions, and a replacement in your favourite editor might not get all occurrences right. Thankfully, there s a nice tool to handle such things, called coccinelle. I won t go into details, just show how it can help here. Basically you just need to craft a tiny patch, called a semantic patch, which describes what I wrote in the previous paragraph. You only need to declare two expressions, say a and b, and declare that all BitIsOn(a,b) must be replaced with TEST_BIT(b,a). Here s the syntax:

@@
expression a,b;
@@
-BitIsOn(a,b)
+TEST_BIT(b,a)

Save it under testbit.cocci and apply it through spatch (from the coccinelle package), asking it to perform in-place replacement (we re in a git checkout copy, remember):

# Apply:
spatch -sp_file testbit.cocci -in_place .
# Profit:
git diff

Then you can run git commit -s, write a nice commit message, send it upstream, and enjoy the rest of the night. Coming soon to a mirror near you: fixing commit. As mentioned on the upstream bug report, I got the Debian reference wrong in the commit message, I really meant #649002.

If you re having issues with video playback in unstable on Intel hardware, this post is for you. Some buffer freeing code was added in libdrm 2.4.28 to try and fix exhaustion of per-process limits. Some safeguards (assert()) were put in place to see whether some other components needed fixing (like unbalanced map()/unmap()). It s hard to miss those, since the server crashes when something s wrong. Good news: for affected users, downgrading to libdrm 2.4.27 (currently available in testing) should restore a functional setup. So far, patches have floated around for libva[1], xserver-xorg-video-intel[1,2] and libdrm[1,2,3] itself. Since libdrm has already been exposed to a wide audience for a while, a new upstream release is planned in a few days, disabling those assert()s. Other packages will then have some extra time to get new releases with the appropriate patches.

According to Eric S. Raymond, one rule for Unix programming is the Rule of Silence: When a program has nothing surprising to say, it should say nothing. Since return codes are a form of information, I would like to add: when a program has carried out its task with no surprise, it should exit(0).Take xscreensaver-command -lock, for instance. That command is meant to make a running XScreenSaver to lock the screen. I have never seen it fail in doing so, however I sometimes get a non-zero code:

% xscreensaver -lock
xscreensaver-command: could not frobnicate window 4212
zsh: exit 1   xscreensaver-command -lock

Well, xscreensaver-command, I have the deep regret to announce you that despite of your surprising return code, you successfully succeeded to lock my screen. But that return code is wrong and annoying. Wrong, because as a user, I do not care if xscreensaver-command -lock failed to frobnicate some window he would have loved to as long as it does its job and locks my screen. To the user's point of view, locking the screen is binary: either it is done and it is a success or it is not done and that is a failure. And it is annoying, because it makes the return code meaningless and prevents you from using it to do things like:

% # Lock screen and suspend to RAM (only if locking was successful)
% xscreensaver -lock && sudo pm-suspend

With xscreensaver-command sometimes returning a non-zero code with no apparent reason, you end with a locked screen and your computer still running, and have to unlock it and try again. This looks like this meaningless dialogue:

• Scott to bridge.
• Kirk here. Scotty, could you restore impulse power?
• I'm sorry captain, I was not able to align the coaxial stabilizers, we just sustained too much damage here.
• How long would it take to get that power back?
• But you have it captain! Impulse power is working all right, it's just that
• Good job Scotty, do you think will hold long enough?
• Well, I'd say it should hold until our next scheduled revision if we take care, but captain, I could not align

So please, when you write a utility, design your return code for the user, not for you. If your utility has done the job, it should exit(0). What happened inside does may interest you as the developer but it does not matter to the user as long as the job is done, so it should not be reported to him, especially not with a misleading failure code.

Today, I had some business in Bonn, so Janis Voigtl nder invited me to hold a guest lecture in his advanced functional programming course, maybe telling the students more about real world use of Haskell, given that he teaches mostly the theoretical foundations. I chose to discuss an experience I made while implementing SAT-Britney, namely that for the sake of good runtime and memory performance, it may be neither ideal to evaluate a list fully lazily, nor fully strict, but to have something in between. For the talk, I demonstrated this by a small example, showed how to use the GHCi debugger to get some insight in the evaluation order of things, how to benchmark the program and read the ghc statistics and a profiling chart, and finally how to get the most out of GHC s List Fusion. I wanted to also touch on QuickCheck, but skipped it to finish in time. The full text of my talk (including graphs) is available, but in German; if you want an English version convince your prof to invite me for a guest lecture as well. Afterwards I sat down with one of the students of the course, Helmut Grohne, who happened to take part in the Debian bug squashing party last weekend and had filed a release critical bug against the Haskell grammar generator frown (no better link available, homepage seems to be down) to debug the problem. It quickly became clear that the bug was introduced by me (can you spot it?) when trying to make it compile with a modern GHC slightly embarrassing for me.

Flattr this post

(including English!) The string freeze of Debian Installer officially ended at 23:59 yesterday (Sept. 20th). Indeed, this was extended a bit to today, with agreement by Otavio Salvador who I thank for this. That allowed Zak to "save" Tagalog and also the Welsh and Latvian translators to polish their work. We now have to decide about some of these languages: those that failed to meet the release criteria but were formerly activated in D-I. There are four such languages: Amharic, Welsh, Estonian and Northern Sami. Please find below the mail I just sent to debian-i18n and debian-boot. I promised that this discussion would happen in public. It will (but it will be short as we can't delay the release of the installer for ages....and I think that my proposals are reasonable!)

First of all, the numbers as of Sunday Sept. 21st 09:32 UTC (date of
the last commit with an l10n update):
Languages meeting the release criteria: 59
------------------------------------------
Already activated and complete for level 1: 51
 Arabic, Belarusian, Bulgarian, Bosnian, Catalan, Czech, Danish,
 German, Dzongkha, Greek, Esperanto, Spanish, Basque, Finnish, French,
 Galician, Gujarati, Hebrew, Hindi, Croatian, Hungarian, Indonesian,
 Italian, Japanese, Georgian, Khmer, Korean, Lithuanian, Latvian,
 Macedonian, Malayalam, Marathi, Norwegian Bokm l, Nepali, Dutch,
 Norwegian Nynorsk, Punjabi, Polish, Portuguese, Brazilian Portuguese,
 Romanian, Russian, Slovak, Albabian, Swedish, Tamil, Thai, Turkish,
 Vietnamese, Simplified Chinese, Traditional Chinese
Already activated and complete for sublevels 1 and 2: 6
Bengali, Kurdish, Slovenian, Tagalog, Ukrainian, Wolof
Not yet activated languages complete for sublevels 1 and 2: 2
 (the mail in -i18n and -boot says 3 but this is an error by me)
Irish, Serbian
Languages failing to meet the release criteria: 15
--------------------------------------------------
Activated languages: 4
Amharic, Welsh, Estonian, Northern Sami
Not yet activated languages: 11
Afrikaans, Persian, Armenian, Icelandic, Kazakh, Kannada,
Malagasy, Malay, Telugu, Urdu, Xhosa
Discussion
----------
(careful people will notice that I moved Welsh down to "failed to meet
the release criteria" as this is what is technically correct)
Nothing to discuss for the 57 already activated languages that meet
the defined criteria. They'll be kept or first activated in the RC1
release of Debian Installer.
Similarly, nothing to discuss for the 11 languages that were not
activated and haven't made it. They will remain unactivated.
Two languages should be activated as they have met the release
criteria for the first time during the string freeze: Irish and Serbian.
This adds more load (and size changes) to D-I but I really don't see
any reason to not follow our own rules there.
The discussion comes for the 4 languages that fail to meet the release
criteria. Here are my proposals with some rationale:
Amharic: 
  I would really dislike deactivating Amharic because it's highly
  symbolic to have the language of Ethiopia activated. We have so few
  African languages. Also, the translation is nearly complete and the
  translator was well coping with updates until July. The missing
  stuff for Amharic in sublevels 1 and 2 are messages about loading
  drivers or firmware from removable media, the rescue mode stuff for
  the graphical installer and some messages that briefly appear during
  finish-install. A little bit more important is the message warning
  that the boot partition is not ext2 or ext3, added in August by
  tbm. I think this is not enough to drop out one year of efforts for
  the translator
  As a consequence, I propose to KEEP Amharic.
Welsh:
  Only five strings are missing in sublevels 1 and 2 because of the
  small experience of PO files by the person who completed the
  translation during last week. One will make the regular user login
  name screen to be in English and others will make the GRUB password
  screen to be in English as well, that's all.
  Additionnally, we can safely assume that all potential users of
  Welsh have good skills in English...and will therefore very easily
  cope with these screens.
  As a consequence, I propose to KEEP Welsh.
Estonian:
  The translation had NO update since Etch. The last update is dated
  back to Feb. 17th 2007. I haven't got any sign of life from the
  translator and no Estonian users have volunteered to maintain the
  translation.
  Missing strings are in many places, including several screens that
  appear in default installs. Even though one can assume that the
  skills of the average Linux user in Estonia is fairly good, I think
  this is not enough to throw users in a big mix of English and
  Estonian.
  As a consequence, I propose to DROP Estonian.
Northern Sami:
  The translation is very incomplete. With about any other language,
  that would be a reason to drop the translation.
  However, a few reasons make me suggest keeping it:
   - Northern Sami is mostly used in Norway and D-I will fall back
     to Norwegian Bokm l which is understood by all potentials users
     as it is teached in all Norwegian schools. 
   - Users will be warned, *in Sami*, about this situation
   - The choice of Sami will be kept in localechooser even if the
     translations are dropped. This is on request of Debian Edu
     developers to avoid them to develop a special boot floppy
     to offer the choice of Sami (a requirement for Norwegian
     schools). I personnally think this is a reward to Debian Edu and
     its ancestor Skolelinux for their initial involvement in the
     development of D-I
  As a consequence, I propose to KEEP Northern Sami.
I understand that these choices may be debatable and some may sound
slightly subjective. I however think this is the best way to be fair
with translators' efforts without compromising the quality of D-I.
Please note that the final word on this will be by D-I release
managers...but advices are very much welcomed.

This week-end, after a few week-ends more targeted on real life activities (country house fall works, 70km ultra run preparation and ultra run week-end), I spent most of Sunday working on two important uploads of packages I'm involved in. So, as of Sunday evening, samba 3.6.1 is now in unstable. This is the first 3.6 version to land there, after earlier versions were in experimental while Steve Langasek was working on multi-arch support and dh7-like debian/rules transition in unstable, with 3.5 versions. Merging together changes in 3.5 and 3.6 branches has been a bit painful. I guess I had it right (thankfully, I merged 3.5 packaging changes from time to time to the 3.6 branch). If there are problems with 3.6 packages, anyway, you know what to do. I also worked to package the new version of pytrainer, a sport activity logging tool. pytrainer is maintained by the "pkg-running" team, aka No l K the, Ralf Treinen and /me. This is the software I use for keeping track of all my running activities. We have a very good link with our upstream and I even recently brought several translation updates by involving Debian translators in the loop. While working to integrate version 1.9.1 (we had 1.8.0 up to now in unstable), I also cleanedout things here orthereand I'mquite happy with the result. More testing will of course be appreciated.

The Comprehensive R Archive Network, or CRAN for short, has been a major driver in the success and rapid proliferation of the R statistical language and environment. CRAN currently hosts around 3400 packages, and is growing at a rapid rate. Not too long ago, John Fox gave a keynote lecture at the annual R conference and provided a lot of quantitative insight into R and CRAN---including an estimate of an incredible growth rate of 40% as a near-perfect straight line on a log-log chart! So CRAN does in fact grow exponentially. (His talk morphed into this paper in the R Journal, see figure 3 for this chart.) The success of CRAN is due to a lot of hard work by the CRAN maintainers, lead for many years and still today by Kurt Hornik whose dedication is unparalleled. Even at the current growth rate of several packages a day, all submissions are still rigorously quality-controlled using strong testing features available in the R system. And for all its successes, and without trying to sound ungrateful, there have always been some things missing at CRAN. It has always been difficult to keep a handle on the rapidly growing archive. Task Views for particular fields, edited by volunteers with specific domain knowledge (including yours truly) help somewhat, but still cannot keep up with the flow. What is missing are regular updates on packages. What is also missing is a better review and voting system (and while Hadley Wickham mentored a Google Summer of Code student to write CRANtastic, it seems fair to say that this subproject didn't exactly take off either). Following useR! 2007 in Ames, I decided to do something and noodled over a first design on the drive back to Chicago. A weekend of hacking lead to CRANberries. CRANberries uses existing R functions to learn which packages are available right now, and compares that to data stored in a local SQLite database. This is enough to learn two things: First, which new packages were added since the last run. That is very useful information, and it feeds a website with blog subscriptions (for the technically minded: an RSS feed, at this URL). Second, it can also compare current versions numbers with the most recent stored version number, and thereby learns about updated packages. This too is useful, and also feeds a website and RSS stream (at this URL; there is also a combined one for new and updated packages.) CRANberries writes out little summaries for both new packages (essentially copying what the DESCRIPTION file contains), and a quick diffstat summary for updated packages. A static blog compiler munges this into static html pages which I serve from here, and creates the RSS feed data at the same time. All this has been operating since 2007. Google Reader tells me the the RSS feed averages around 137 posts per week, and has about 160 subscribers. It does feed to Planet R which itself redistributes so it is hard to estimate the absolute number of readers. My weblogs also indicate a steady number of visits to the html versions. The most recent innovation was to add tweeting earlier in 2011 under the @CRANberriesFeed Twitter handle. After all, the best way to address information overload and too many posts in our RSS readers surely is to ... just generate more information and add some Twitter noise. So CRANberries now tweets a message for each new package, and a summary message for each set of new packages (or several if the total length exceeds the 140 character limit). As of today, we have sent 1723 tweets to what are currently 171 subscribers. Tweets for updated packages were added a few months later. Which leads us to today's innovation. One feature which has truly been missing from CRAN was updates about withdrawn packages. Packages can be withdrawn for a number of reasons. Back in the day, CRAN carried so-called bundles carrying packages inside. Examples were VR and gregmisc. Both had long been split into their component packages, making VR and gregmisc part of the set of packages no longer on the top page of CRAN, but only its archive section. Other examples are packages such as Design, which its author Frank Harrell renamed to rms to match to title of the book covering its methodology. And then there are of course package for which the maintainer disappeared, or lost interest, or was unable to keep up with quality requirements imposed by CRAN. All these packages are of course still in the Archive section of CRAN. But how many packages did disappear? Well, compared to the information accumulated by CRANberries over the years, as of today a staggering 282 packages have been withdrawn for various reasons. And at least I would like to know more regularly when this happens, if only so I have a chance to see if the retired package is one the 120+ packages I still look after for Debian (as happened recently with two Rmetrics packages). So starting with the next scheduled run, CRANberries will also report removed packages, in its own subtree of the website and its own RSS feed (which should appear at this URL). I made the required code changes (all of about two dozen lines), and did some light testing. To not overwhelm us all with line noise while we catch up to the current steady state of packages, I have (temporarily) lowered the frequency with which CRANberries is called by cron. I also put a cap on the number of removed packages that are reported in each run. As always with new code, there may be a bug or two but I will try to catch up in due course. I hope this is of interest and use to others. If so, please use the RSS feeds in your RSS readers, and subscribe to the @CRANberriesFeed, And keep using CRAN, and let's all say thanks to Kurt, Stefan, Uwe, and everybody who is working on CRAN (or has been in the past).

1. Non-resizable configuration windows. I have a 1920 1200 screen but the window manager provides no way to resize windows like the System Settings window. Result is that the window is quite small and, worse, has a scroll bar for no good reason when another hundred or so vertical pixels would easily display everything. Even another 30 or so horizontal pixels would allow more icons per line under Hardware, reducing the number of lines, and again allowing the entire list of available settings to be displayed without a scroll bar. Instead, to see all of the options I have to scroll down. And these people call themselves user-interface and usability experts. It's not up to the gnome developers to decide that I never need to resize certain windows or certain kinds of windows. It's up to me to decide that. They're not at my desktop, using my screen. I am.
2. No way to disable a shell extenstion from the command line. I installed the gnome-shell-extensions package and was trying out various extensions to see if they made gnome shell tolerable. Enabling one of them (i forget exactly which one), immediately displayed some "Oh no! Something has gone wrong. A problem has occured and the system can't recover. Please log out and try again." leaving me no option but to log out. An absolutely fucking useless error message. "Something" has gone wrong. A little fucking detail might help me to diagnose the problem and fix it. Like a trace of whatever the fuck it was doing when "Something went wrong". When i logged back in, it crashed again, with the same useless error screen. So, now I can't even log in to Gnome Shell. I've tried logging in to Fallback mode, but the Extensions tab is empty in gnome-tweak-tool (because, of course, nobody would want to configure Shell Extensions from fallback mode, would they?). I can't even find any way of listing which shell extensions are enabled, let alone disable one of them. Hell, I'd settle for disabling all of them, and then carefully re-enabling them one-by-one until I rediscovered the culprit.
3. Oh, wait. I finally found /org/gnome/shell/enabled-extensions in dconf-editor. it's a text field with an array of items. so far, so good. The "editor" (and i use that term very loosely) for text fields is garbage. It can't even wrap lines so that it displays the whole field without a horizontal scroll bar probably my all-time least favourite user-interface element. Have gnome devs never heard of multi-line text-edit fields? Rather than fuck around with a crappy GUI editor, it's far easier to:

   $ dconf read /org/gnome/shell/enabled-extensions > /tmp/extensions
   $ vi /tmp/extensions
   $ dconf write /org/gnome/shell/enabled-extensions "$(cat /tmp/extensions)"
   

   I guess now it's time to log out from fallback mode to find out if I can log back in to Gnome Shell again. [ nope. didn't work ]
4. Add non-resizable user-interface elements to item 1. above. So far, Gnome devs are doing a great job of regressing to the crap usability of the Windows GUI here s a useful rule of thumb for things like that: Every single window, every single user-interface element should be resizable and tweakable by the user. And the system should remember all user changes so that they don t have to waste a few minutes every time they run an app resizing things to suit their requirements. Gnome 2 mostly had that. Gnome 3 seems to have lost a large part of it along the way Second rule of thumb: If the user happens to replace their monitor or run it in a mode too small for the current settings, then they should revert to some sane default. Even better, it should remember the settings for each resolution choosing either sane defaults or the user s settings for the nearest similar resolution if the user hasn t created custom settings for that resolution yet.
5. Lack of documentation. here's a good example:

   $ gnome-shell-extension-tool help
   Usage: gnome-shell-extension-tool [options]
   $ man gnome-shell-extension-tool
   No manual entry for gnome-shell-extension-tool
   See 'man 7 undocumented' for help when manual pages are not available.

   What more needs to be said? At least /usr/bin/dconf has built-in help. Which is great, since it doesn't have a man page either.

Update: About three hours of stuffing around later, and I ve finally got Gnome Shell running again. I have NFI what actually fixed it, it could have any one of the dozens of settings I fucked around with. Nor do i have any idea what the actual problem was. And after all that, the extensions I tried still don t make Gnome Shell tolerable. They make it less *intolerable* (bottom-panel extension in particular), but that s still a long way from being tolerable i m really trying hard to find some way to come to terms with Gnome 3. I ve been trying for months now. But every time I give it another trial, I just end up hating it even more I m glad I have more than one desktop machine to use, so i can experiment with odious shit like Gnome Shell without fucking up my main working environment. So it s back to my main desktop machine and gnome2. Eventually I ll have to upgrade that, but xfce4 is looking to be my best option. I already switched to xfce4 on my desktop machine at work last week, and it s pretty good certainly far better than the Gnome 3 fallback mode i ve been using there for the last 5 or 6 months. Thanks, Gnome devs you re making a fantastic advertisement for XFCE. I wouldn t have bothered trying it again if I hadn t been forced to. ps: before anyone says wait for Gnome 3.2, it s much better . Todays waste of time *was* using Gnome 3.2 some more reasons to hate gnome shell is a post from: Errata

I want to blog frequently but usually don't seem to find the time for it. I'm not above tying myself to the mast if it means blogging more. Iron Blogger is a blogging and drinking club based on this premise. The rules are pretty simple:

• Blog at least once a week.
• If you fail to do so, pay $5 into a common pool.
• When the pool is big enough, the group uses it to pay for drinks and snacks at a meet-up for all the participants.

Nelson Elhage ran the original Iron Blogger for about a year before the effort ran out of steam. I've started a new instance with a couple people from the previous group and a bunch of folks from Berkman, MIT, and beyond. If you live in Boston and want to join, there are still a couple of spots available. I'm going to cap the current group, at least temporarily, at about 30 people because I think that's the maximum we'll fit into a local pub. Look over the site and send me an email if you're interested. If you don't live in Boston but want to organize your own Iron Blogger, you can use the software in Nelson's git repository (or my branch) to automate nearly the whole process of tracking posts, generating reports, and updating the ledger of debts.

as expected this week was characterised by fixing bugs around the perl 5.14 transition. besides that I've also uploaded a couple of NMUs with patches kindly provided in the BTS by some tireless bug fixers.

• #554047 src:cellwriter: "FTBFS with binutils-gold"
  add patch from Ubuntu / Mahyuddin Susanto, upload to DELAYED/2
• #628500 src:prima: "prima: FTBFS with perl 5.14: error: 'sv_undef' undeclared"
  patch to change functions that rely on removed PERL_POLLUTE, upload to DELAYED/2 (perl-5.14)
• #629284 src:libpod-constants-perl: "libpod-constants-perl: FTBFS with perl 5.14: Can't use string [...] as an ARRAY ref"
  patch Makefile.PL (older and newer EUMM) (pkg-perl, perl-5.14)
• #636268 src:libdbix-class-timestamp-perl: "libdbix-class-timestamp-perl: FTBFS with perl 5.14: test failures"
  upload new upstream release (pkg-perl, perl-5.14)
• #636271 src:libtest-www-declare-perl: "libtest-www-declare-perl: FTBFS with perl 5.14: tests fail"
  new patch to work with perl 5.14 regexp (pkg-perl, perl-5.14)
• #636517 src:libregexp-optimizer-perl: "libregexp-optimizer-perl: FTBFS with perl 5.14: test failures"
  new patch to work with perl 5.14 regexp (pkg-perl, perl-5.14)
• #636520 src:libtest-html-content-perl: "libtest-html-content-perl: FTBFS with perl 5.14: tests fail"
  send a patch to the BTS, uploaded by the maintainer (perl-5.14)
• #636521 src:libtest-log4perl-perl: "libtest-log4perl-perl: FTBFS with perl 5.14: tests fail"
  new patch to work with perl 5.14 regexp (pkg-perl, perl-5.14)
• #636522 src:libtest-simpleunit-perl: "libtest-simpleunit-perl: FTBFS with perl 5.14: tests fail"
  new patch to work with perl 5.14 regexp (pkg-perl, perl-5.14)
• #643040 src:abcm2ps: "abcm2ps: FTBFS: dpkg-buildpackage: error: dpkg-source -b abcm2ps-5.9.22 gave error exit status 2"
  backup/restore Makefile in debian/rules, upload to DELAYED/2
• #643359 src:blktool: "blktool: FTBFS: util.c:31:3: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Eric Alexander, upload to DELAYED/2
• #643362 src:cellwriter: "cellwriter: FTBFS: src/statusicon.c:215:17: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Eric Alexander, upload to DELAYED/2
• #643369 src:crack-attack: "crack-attack: FTBFS: callbacks.cxx:116:38: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Eric Alexander, upload to DELAYED/2
• #643370 src:d52: "d52: FTBFS: d52pass2.c:935:3: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from peter green, upload to DELAYED/2
• #643374 src:dis51: "dis51: FTBFS: pass2.c:186:4: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Eric Alexander, upload to DELAYED/2
• #643375 src:ebview: "ebview: FTBFS: dialog.c:72:6: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Eric Alexander, upload to DELAYED/2
• #643383 src:gfan: "gfan: FTBFS: application.cpp:548:22: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Eric Alexander, upload to DELAYED/2
• #643384 src:gfccore: "gfccore: FTBFS: error.cc:116:52: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Eric Alexander, upload to DELAYED/2
• #643413 src:jack-tools: "jack-tools: FTBFS: jack.dl.c:20:3: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from M nica Ram rez Arceda, upload to DELAYED/2
• #643417 src:libccscript3: "libccscript3: FTBFS: compiler.cpp:1147:38: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Ilya Barygin, upload to DELAYED/2
• #643448 src:nvramtool: "nvramtool: FTBFS: hexdump.c:94:7: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from peter green, upload to DELAYED/2
• #643457 src:pidgin-librvp: "pidgin-librvp: FTBFS: rvp.c:4271:3: error: format not a string literal and no format arguments [-Werror=format-security]"
  add patch from Eric Alexander, upload to DELAYED/2
• #646286 src:genders: "genders: FTBFS: crti.o: No such file or directory"
  send a patch to the BTS, uploaded by the maintainer (perl-5.14)
• #646297 src:spread: "spread: FTBFS: Spread.xs:463:4: error: format not a string literal and no format arguments [-Werror=format-security]"
  add format argument, do a QA upload (perl-5.14)
• #648858 src:libdbd-oracle-perl: "libdbd-oracle-perl: needs to be rebuilt against perl 5.14"
  manually rebuild against perl 5.14 (pkg-perl, perl-5.14)
• #649055 libgnome2-gconf-perl: "libgnome2-gconf-perl: FTBFS (Failed 1/1 test programs. 0/0 subtests failed)"
  add patch from Colin Watson (pkg-perl, perl-5.14)
• #649058 libdata-alias-perl: "libdata-alias-perl: FTBFS on mips powerpc s390 sparc"
  add patch from Niko Tyni (pkg-perl, perl-5.14)
• #649180 src:libmouse-perl: "libmouse-perl: swap libextutils-parsexs-perl/perl dep"
  upload fixed package prepared by Alessandro Ghedini (pkg-perl, perl-5.14)

Hello Most SDL packages are outdated, people are complaining in the BTS, and the team is mostly inactive. That s bad news for my project. What can I do ? Well, it s Debian project, I just have to step up. Thanks to Sam Hocevar, I m now admin on Debian Alioth SDL packaging project. I d like to revive the team. (Note that I don t have the bandwidth to be the team all by myself). Here are the first steps I plan (mostly inspired by the way Debian-perl team is working):

• create a SDL-team page on debian wiki (done)
• setup a SDL packaging git repo so that team members can collaborate more easily on packaging (on-going)
• call for volunteers to help. (this message is the first step)
• review and upload packages from team members

Now, there are some ground rules I wish to push forward:

• any team member can work on any package maintained by the team.
• just tell us on this list about any work you are doing (to avoid duplication)
• send a mail on this list if you need a package to be reviewed for upload.
• you can use SDL subversion repository to work on a package
• you can also use SDL git repo to work on a package (please discuss with us for the migration steps)
• once a package is migrated to git, it cannot go back to subversion.

If you don t agree with the above points, now is the time to say it. Feel free to join by:

1. Creating an account on Alioth
2. request to join sdl-team
3. Last but not least, send a short introduction mail to pkg-sdl-maintainers AT lists.alioth.debian.org so we can know each other.

See you soon