Lately I have been playing a little bit with different Platform as a Service providers. So far, the most developer-friendly one seems to be Heroku. In particular, Heroku provides an incredibly straightforward way to get started. After creating an account on their website, you just have to install the appropriate gem: sudo gem install heroku The command heroku help will then show you all the possible invocations of the command line client, just to give you an idea of what you can do with it (everything). Alright, that s not particularly interesting without an application to play with, is it? So let s create a new Sinatra application. In order to do that, we need a new directory containing a file like the following one:

# webapp.rb
require 'rubygems'
require 'sinatra'
# Add other gems you might need here
get '/' do
  "Your Sinatra app is working!"
end

Also add a Gemfile with all the required dependencies, in our trivial example sinatra is actually the only gem you need:

source :rubygems
gem "sinatra"

Lastly, you need a config.ru file where you can require our newly created Ruby program and specify that it is a Sinatra application. In this example, the Ruby file is called webapp.rb.

require './webapp'
run Sinatra::Application

A very useful program when developing a Sinatra application is shotgun. gem install shotgun and you will be able to run your new shiny app by executing the command shotgun. While developing on your machine, shotgun will take care of reloading the application for every request, so that you don t have to restart the server after every modification to the source code. Run bundle install to easily install all the required dependencies in your development environment, and shotgun to start the app locally. Pointing your browser to http://127.0.0.1:9393/ you can check if everything works as expected. If nothing went wrong, you can create a git repository to track changes to the source code:

git init
git add . 
git commit -m "Initial commit"

Perfect! Now, how do you create and deploy your application on Heroku?

heroku create
git push heroku master

Heroku will automatically find out that you deployed a Sinatra application, it will install all the required dependencies and start serving requests. A few interesting commands you might want to run to see what s going on are heroku info, heroku ps and heroku logs. The client also provides a command, heroku open, that opens your browser at the right production URL. So, you now have a working application under revision control. Dependencies are handled by specifying them into the Gemfile, and the required gems can be installed locally using bundle. Shotgun is our friend when it comes to use the application for development purposes. A git push heroku master is all we need to deploy a new version of the code. Of course there is way much more about Sinatra and Heroku than what I ve covered in this mini-intro. Please refer to the Sinatra Book and the Heroku Dev Center for more information!

Yesterday some confusion arised on my side when I saw a new facebook advertising campaign in my facebook account (yes, I am a member of facebook, although I'm aware of the privacy concerns). Basically it was saying that I should try the friends finder and that some of my friends (showing and naming me three of them) would have used it already.

Some background:
The friends finder feature of facebook is a feature that asks for the password of your e-mail account. It will then crawl through your emails to find contacts that might already be on facebook but not connected to you (in a facebook sense).

My first feeling was: Oh my god. How can it be that friends (and family) of mine are so naive? Especially since there were people included which I consider to be quiet clever. But honestly: Who would be so naive to give an unknown company direct unsupervised (you can't tell what they really do) access to your mail account? Would you give it to your friend? Your husband? Your father? I guess the answer will be "No" in the most cases and these people are most likely people you trust. Well, I know, you could state similar things for Googlemail who crawl your mails to show you personalized advertising. And in fact you are right. But if one decides to use gmail for your email hosting you have to trust them anyway. Like you have to trust anybody else you retain to host your mails (who could do the same but just don't tell you). But in this case its a third party, Facebook.

But if you now think that this becomes a rant against my friends: You are wrong. Over the day I found out that it basically shows up all my friends in a rotation. Every time I open the start page it randomly picks three people from my friend list and shows them to me, telling the same lie to me all over again.

So what do we have here? Facebook tries to advertise the most dubious feature they have in the most aggressive way one could imagine. By pretending wrong facts. Wouldn't that even be an element of crime in Germany ("Irref hrung" 5 UWG or maybe also 4 UWG, "unsachliche Beeinflussung")?

Facebook, you can do better.

Update: Stefano raised a good point. I didn't actually make clear that it has been verified that those people actually did not use the "feature":

1. I asked some of them. They said, they didn't use it and were in fact surprised that I asked.
2. Others told me they saw that ad stating that I used the feature. I definitely did not use this "feature".
3. Probably weak: IMHO its highly unlikely that all of my friends used that feature and at the point I've written this it had already shown my whole friend list.

Last night, the text below went out to r-sig-finance along with updates to the R/Finance website and its Call for Papers page; followed by some tweeting and Goggle+'ing (and please do feel free to retweet and share at will...)

Call for Papers: R/Finance 2012: Applied Finance with R
May 11 and 12, 2012
University of Illinois, Chicago, IL, USA
The fourth annual R/Finance conference for applied finance using R will be held on May 11 and 12, 2012 in Chicago, IL, USA on the campus of the University of Illinois at Chicago. The two-day conference will cover topics including portfolio management, time series analysis, advanced risk tools, high-performance computing, market microstructure, and econometrics. All will be discussed within the context of using R as a primary tool for financial risk management, portfolio construction, and trading. Over the past three years, R/Finance has included attendees from around the world and featured keynote presentations from prominent academics and practitioners. We anticipate another exciting line-up for 2012 --- including keynote presentations from Blair Hull, Paul Gilbert, Rob McCulloch, and Simon Urbanek. We invite you to submit complete papers or one-page abstracts (in txt or pdf format) for consideration. Academic and practitioner proposals related to R are encouraged. We welcome submissions for full talks, abbreviated "lightning talks", and for a limited number of (longer) pre-conference seminar sessions. Presenters are strongly encouraged to provide working R code to accompany the presentation/paper. Data sets should also be made public for the purposes of reproducibility (though we realize this may be limited due to contracts with data vendors). Preference may be given to presenters who have released R packages. Travel and accommodation grants may be available for selected presenters at the discretion of the committee. In addition, the conference will award prizes for best papers. To be eligible for a best paper award, a submission must be a full paper. Extended abstracts, even if a full paper by conference time, are not eligible for a best paper award. Please send submissions to: committee at RinFinance.com. The submission deadline is January 31, 2012. Submitters will be notified of acceptance via email by February 28, 2012. Notification of whether a presentation will be a long presentation or a lightning talk will also be made at that time. Additional details will be announced at this website as they become available. Information on previous year's presenters and their presentations are also at the conference website R/Finance 2009, 2010 and 2011. For the program committee:

Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson,
Dale Rosenthal, Jeffrey Ryan, Joshua Ulrich

So see you in Chicago in May!

The Good Every other year or so I feel the need to clean up my Debian system and remove the installed packages I m not interested in anymore. I remember there was a nice aptitude pattern to search for packages which I have manually installed (i.e. which were not installed to satisfy a dependency). Ideally I would then go through the (presumably short) list of packages and remove the ones I don t need any more. Since I always forget the aptitude pattern to search for those packages, I google for something like list of manually installed packages and find a solution like: aptitude search ~i !~M . Although this solution is not wrong, it is not quite what I was looking for. Sure, it will find you all packages which are installed and not installed to satisfy a dependency, but it also contains packages of priorities: required, important and standard which you usually don t want to deinstall. So the actual solution should also exclude those packages.

aptitude search '~i !~M !~pstandard !~pimportant !~prequired'

The above line will do the trick and present you a list of packages which are: installed and not dragged in to satisfy a dependency and not of priority standard, important or required. If you where only interested in the solution of the quest of getting a list of all manually installed packages you can stop reading here. The Bad If you re running your system for longer than, say, a few weeks you will probably notice that the list is very long and contains much more packages than you actually installed manually. You will probably notice a lot of libs and other packages in this list and wonder why they aren t marked auto as they should since you didn t install them on purpose. Unfortunately I don t have an answer for that question. I m using aptitude exclusively to install packages and rely on the fact that aptitude does install the dependencies automatically and more importantly: also removes them if not needed any more. But it seems like something is broken in aptitude (or somewhere else) and packages simply lose their auto status instead of being removed and thus stay on your system forever if you don t manually clean your system. The Ugly To solve that issue I started aptitude and filtered the package tree with the above pattern (press l in aptitude and enter ~i !~M !~pstandard !~pimportant !~prequired). Now the package view shows only packages which are safe to remove since they are not of of priority standard or higher and manually installed by you. Now I navigate with the cursor to the list of installed packages an mark all of them as auto by pressing M. For each package in this list, aptitude will try to find an installed package which depends on that package and mark that package auto. The remaining packages which no other packages depend on, aptitude will mark for deletion. If you now press g one time, aptitude will show you which packages will be removed. I then go through that list and mark every single of the remaining packages I really want on my system with + as manually installed. After I m done I proceed by pressing g again which will remove all remaining packages. In my case nearly 600 packages (1GB) where removed. Bugreport! I assume there is a bug in aptitude or some related package which sometimes(!) removes a packages auto status instead of removing it. Unfortunately I did not find a way to reproduce that (except that it obviously does happen). If someone could find a way to reproduce that problem, one could fill a bugreport and help to get that bug fixed. Right now, a Debian system suffers (like many other operating systems) from a slow but steadily growing bloat of installed software. In my case 600 packages with 1GB accumulated over one or two years.

The route cache is a Linux kernel component enabling route lookups to be faster by caching the results in some table and checking it before issuing a regular lookup in the route tables. When using Linux as a router, the inefficiency of the route cache can hinder the performances of your box. The documentation on this component is scarce and it is difficult to find up-to-date bits on how the route cache works and how to tune it. The book Understanding Linux Network Internals from O Reilly is an exception and contains valuable information on how the route cache works. Even if the book is targeted at 2.6.12, the part on the route cache is still quite accurate. Unfortunately, it fails to provide appropriate tips on how to monitor and tune the route cache. I hope to provide here a concise view of the route cache, as it is implemented in Linux 3.1. It is protocol-dependent¹ and I will cover only the IPv4 version here.

Overview of the route cache subsystem To handle an incoming or outgoing IP datagram, the kernel needs to issue a lookup in the route tables. While it seems to be quite trivial, several questions need to be answered:

• Does the source address and the destination address appear to be valid?
• Is the source address a martian address²?
• Is the destination address mine or should I forward the packet?
• Which route table should I use?
• Does the destination address match this route? And this one?
• Can I currently contact the gateway that I should use?

Those checks can be a bit time consuming, even for small route tables. To avoid them for each packets, Linux maintains a route cache which is queried before doing a regular lookup and updated after each one. You can dump it with ip -s route show cache:

$ ip -s route show cache
198.51.100.7 from 203.0.113.2 via 192.0.2.1 dev eth1
  cache used 7 age 2sec ipid 0x1fce rtt 131ms rttvar 45ms cwnd 10
198.51.100.17 from 203.0.113.15 via 192.0.2.1 dev eth1
  cache used 3 age 0sec ipid 0xc3bd
local 192.0.2.18 from 203.0.113.15 dev lo src 192.0.2.18
  cache <local> used 154 age 1sec iif eth0

Here are two examples:

• If Linux receives a packet from 198.51.100.17 to 203.0.113.15, it will find this flow in the route cache. Therefore, it already knows that it should forward the packet to 192.0.2.1. No checks needed.
• If it receives a packet from 198.51.100.16 to 203.0.113.15, there is no appropriate entry in the route cache and therefore, the system will have to look at the route tables. It is likely to use the exact same entry than if the source was 198.51.100.17 but maybe there is some policy routing requesting the use of a special route table or 198.51.100.16 is a local address and the packet will therefore be classified as martian.

The schema below shows how this cache is implemented³. It uses a separate hash table (BSD systems keep the cache in the routing table). Each bucket is a chained list of route cache entries. Once an entry has been added to the route cache, there are several ways to remove it. Most entries are removed by the garbage collector which will scan the route cache and remove invalid and older entries. It will be triggered when the route cache is full or at regular interval, once a certain threshold has been met.

Available knobs There are several values you can tune. Most of them are available in sysctl.

• rhash_entries is the size of the hash table⁴. If you don t specify it on the kernel command line, it is computed dynamically based on the memory available on your system. You can view its value by looking at something like IP route cache hash table entries: 262144 (order: 9, 2097152 bytes) in the kernel logs.
• net.ipv4.route.max_size is the maximum number of entries in the route cache. Except under exceptional circumstances, this value is never exceeded.
• net.ipv4.route.gc_elasticity is the target average length of a chain in the route cache. The garbage collector will work harder if this value is exceeded. This means that if you multiply this value by the value of rhash_entries, you will get the target average number of entries in the route cache.
• net.ipv4.route.gc_min_interval_ms is the minimum delay between two runs of the garbage collector, except when the cache is full. The default value should be fine.
• net.ipv4.route.gc_thresh is a threshold triggering the garbage collector every net.ipv4.route.gc_min_interval_ms milliseconds.
• net.ipv4.route.gc_timeout is the base value to determine if an entry is old enough to be removed or not. Whatever its value, the garbage collector will attempt to remove the same number of entries. However, this value could potentially influence its efficiency. See below for more details on this.

You may find documentation about those obsolete sysctl values:

• net.ipv4.route.secret_interval has been removed in Linux 2.6.35; it was used to trigger an asynchronous flush at fixed interval to avoid to fill the cache.
• net.ipv4.route.gc_interval has been removed in Linux 2.6.38. It is still present until Linux 3.2 but has no effect. It was used to trigger an asynchronous cleanup of the route cache (but in a different way of what the garbage collector does). The garbage collector is now considered efficient enough for the job.

Statistics & monitoring Linux maintains some statistics about the use of the route cache. You can find them in /proc/net/stat/rt_cache. The command lnstat can print them nicely for you:

$ lnstat -s1 -i1 -c-1 -f rt_cache
rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache rt_cache 
 entries   in_hit in_slow_ in_slow_ in_no_ro   in_brd in_marti in_marti  out_hit out_slow out_slow gc_total gc_ignor gc_goal_ gc_dst_o in_hlist out_hlis 
                       tot       mc      ute            an_dst   an_src              _tot      _mc                ed     miss  verflow  _search t_search 
 3096848    42309      686        0        0        0        0        0        3        0        0      674      672        0        0    27644        8 
 3096984    41405      636        0        0        0        0        0        3        0        0      623      621        0        0    28189        8 
 3097160    42483      700        0        0        0        0        0        5        0        0      694      692        0        0    29506       12 

Except for the first column, lnstat outputs values in units per second. Let s review some of those values:

• rt_cache_entries is the number of entries in the route cache. You should compare it with net.ipv4.route.max_size and ensure that the cache is never full to avoid triggering the garbage collector too often.
• rt_cache_in_hit and rt_cache_out_hit are the number of regular lookups avoided because the result was found in the cache for incoming and outgoing packets, respectively. When the system is a router, most lookups only happen on the incoming side. You should compare this value with rt_cache_in_slow_tot and rt_cache_in_out_slow_tot which are the number of lookups in the route tables. On this system, the efficiency of the route cache is more than 98% which is quite good.
• rt_cache_gc_total is how often the garbage collector was requested to be triggered while rt_cache_gc_ignored corresponds to how often it was finally not run because it has already been triggered shortly beforehand (less than net.ipv4.route.gc_min_interval_ms milliseconds). The difference between those two values should be very small to ensure that the garbage collector does not run more a handful of times per second.
• rt_cache_gc_goal_miss is how often the garbage collector was not able to fulfill its goal. This value should rarely be different of 0.
• rt_cache_gc_dst_overflow is how often the route cache is bigger than the allowed maximum size. This should never happen except when you try to shrink the cache.
• rt_cache_in_hlist_search and rt_cache_out_hlist_search is how often a lookup in the cache had to look at the next entry in the chain for the computed bucket: each time Linux has to follow the next pointer in a cache entry, it increments one of those counters. It is a clue on the average length of chains in the route cache hash table. Compare those values with the number of cache lookups (hit and miss).

As an illustration, here is a plot of the various statistics exposed above for a router whose route cache receives about 1000 routes par second: rhash_entries is 1,048,576, as is net.ipv4.route.gc_thresh. Therefore, the garbage collector was requested to be run when the number of entries goes above this level. Because the cache is not full, it is only really run twice per seconds (the value of net.ipv4.gc_interval_ms is 500). net.ipv4.gc_elasticity is set to 3. This explains why the garbage collector is aggressive when the number of entries reached 3,145,728. As you can see, the efficiency is near 100% all the time. The percentage of collisions is rt_cache_in_hlist_search ratio to the sum of rt_cache_in_hit and rt_cache_in_slow_tot.

Tuning Do you need to modify any of those values? You have two questions to ask yourself:

1. How much efficiency do you want to get from the route cache?
2. How much memory do you want to dedicate to the route cache?

As a rule of thumb, two millions entries eat about 500 MB of memory on a 64bit system. You should be able to compute the average memory usage and the maximum memory usage from the values of net.ipv4.route.max_size, rhash_entries and net.ipv4.route.gc_elasticity. For example, if the route cache hash table has 262,144 buckets, the maximum allowed number of entries in the cache is 4,194,304 and net.ipv4.route.gc_elasticity is set to 8, the memory usage will be 500 MB on average and 1 GB max. If this is too much, you will need to lower some values. Look at the previous section to compute the current efficiency of your route cache. It should be above 90%. If you are dissatisfied with that, you could increase the cache size. If you want to double the number of entries, double net.ipv4.route.max_size, net.ipv4.route.gc_thresh and rhash_entries but keep net.ipv4.route.gc_elasticity to 8. For the garbage collector to be efficient, the route cache should not be filled too fast. The garbage collector should be able to cope with this situation but this may impact performance because it needs to walk several times the route cache to find entries to expire. Watching gc_goal_miss may give you a hint about this: if this value starts to be different of 0, lower net.ipv4.gc_timeout.

In-depth look into the garbage collector Those different pieces of advice may have puzzled you. We need to understand how the garbage collector works to better cope with them. The garbage collector is triggered when a new entry needs to be added to the cache and the number of entries is superior to net.ipv4.route.gc_thresh. It is implemented in rt_garbage_collect() function. It will do nothing if it has been called less than net.ipv4.route.gc_interval_ms milliseconds and the cache is not full (more than net.ipv4.route.max_size entries).

Setting a goal The garbage collector will first assess the situation. It will look how the number of entries currently in the cache compares to the product of rhash_entries and net.ipv4.gc_elasticity:

1. Above this limit, it will try to remove at least rhash_entries entries.
2. Otherwise, it will try to remove no more than half the difference.

If you look at the previous plot, you can easily see the difference when the garbage collector is not aggressive (above 1,048,576 entries but below 3,145,728) and when it is (above 3,145,728 entries). If we assume the garbage collector is able to meet its goal, we can easily simulate its algorithm: The first plot shows what would happen if about 2000 routes are added per second with rhash_entries equal to 262,144 and net.ipv4.route.gc_elasticity set to 8. When net.ipv4.route.gc_threshold is met, the garbage collector has almost no effect. However, when the number of entries reaches 2,097,152, the garbage collector sets its goal to 262,144. From this point, the number of entries oscillates around 2 millions entries. On the second plot, rhash_entries is now equal to 1,048,576 but net.ipv4.route.gc_elasticity has been set to 2. Therefore, the aggressive part of the garbage collectors kicks at the same threshold than for the previous plot. However, its goal is now four times larger and the oscillations have a larger amplitude. The fact that the aggressive part of the garbage collector kicks less often is nullified by the fact that it needs to remove more entries each time. Because of the slight impact on cache efficiency, it seems better to keep net.ipv4.route.gc_elasticity around 8, or 4 if we want to keep shorter chains (but there seems to be no improvement to do so). The other plots show what happens when there is a sudden surge in the number of routes added or when there is a pause.

Meeting the goal Now that we understand how rhash_entries, net.ipv4.route.gc_elasticity and net.ipv4.route.gc_threshold interact, let s look at net.ipv4.route.gc_timeout. Once the garbage collector has set its goal and if it is positive, it needs to choose which entries to remove from the cache. It walks the hash table from the position of its last run and remove entries until its goal is met. If the entry is not current anymore (for example, the network interface associated to it has changed its IP configuration), it is removed. Otherwise, the system looks at the age of the entry and its position in the chain. If the entry is the first in the chain, it is kept only if its age is below net.ipv4.route.gc_timeout. If it is second, it is kept only if its age is below half of net.ipv4.route.gc_timeout. If it is third, the threshold is a quarter of net.ipv4.route.gc_timeout, and so on. The garbage collector will favor short chains. If after a full run of the hash table, the garbage collector was not able to meet its goal, it will start again but will behave more aggressively, as if net.ipv4.route.gc_timeout is set to half of its value. It will do as many passes as necessary until its goal is met or until there is no way to remove any entry (or until it has spent too much time). Once the garbage collector has switched to this more aggressive behavior, it will keep being aggressive for a few cycles (a bit less for each cycle). With a very large value of net.ipv4.route.gc_timeout, the garbage collector will have a hard time to find entries to expire. It will need to do several passes until it is able to expire some entries. On the other hand, if you choose a very small value, the garbage collector may remove entries that were just added, even if there are older entries further in the hash table. For a more in-depth coverage of how the route cache works, look at chapter 33 of Understanding Linux Network Internals. Just be aware that multipath route caching has been removed (and was never really used) and asynchronous cleanup (rt_periodic_timer) does not exist anymore.

Conclusion When tuning the route cache, rhash_entries, net.ipv4.route.gc_elasticity, net.ipv4.route.max_size and net.ipv4.route.gc_threshold are related and should not be modified independently. net.ipv4.route.gc_thresh should be below the product of net.ipv4.route.gc_elasticity and rhash_entries while net.ipv4.route.max_size should be above this value. Linux exposes several interesting metrics related to the route cache. Monitoring them allows one to watch the efficiency of the route cache and may uncover some anomalies (garbage collector running too often, difficulty to remove routes from the cache, ). The route cache subsystem still evolves and some old behaviors have been dismissed. The best documentation is, unfortunately, still the code and other documentations tend to become obsolete. The IPv6 route cache is quite different of the one presented here. Don t blindly apply the same tuning for IPv6.

---

1. Linux provides a protocol-independent destination cache subsystem (DST). This component is not a generic cache layer and only enables loose coupling with external subsystems.
2. Martian addresses are addresses that cannot be used as a source address, either because they are reserved for special-use (like a multicast address) or because of the use of reverse path filtering which checks if a packet received on one interface would be answered on the same interface, as defined in RFC 3704. This feature is enabled by setting rp_filter in Linux.
3. For more details, you may want to look at include/net/dst.h, include/net/route.h and net/ipv4/route.c.
4. In fact, the size of the hash table is always a power of two. If specified, rhash_entries is rounded to the next power of two and, internally, is stored as rt_hash_mask + 1.

Ben Hutchings is a rather unassuming guy but hiding behind his hat, there s a real kernel hacker who backports new drivers for the kernel in Debian stable so that our flagship release supports very recent hardware. Read on to learn more about Ben and the kernel team s projects for Debian Wheezy! Raphael: Who are you? Ben: I m a professional programmer, living in Cambridge, England with my long-suffering wife Nattie. In Debian, I mostly work on the Linux kernel and related packages. Raphael: How did you start contributing to Debian? Ben: I started using Debian in 1998 and at some point I subscribed to Debian Weekly News. So in 2003 I heard about the planned Debian 10th birthday party in Cambridge, and thought I would like to go to that. Somehow I persuaded Nattie that we should go, even though it was on the day of our wedding anniversary! We both enjoyed it; we made new friends and met some old ones (small world). From then on we have
both been socially involved in Debian UK. In 2004 there was a bug-squashing party in Cambridge, and we attended that as well. That s where I really started contributing fixing bugs and learning about Debian packaging. Then in 2005 I made my first package (sgt-puzzles), attended DebConf, and was persuaded to enter the New Maintainer process. NM involved a lot of waiting, but by the time I was given questions and tasks to do I had learned enough to get through quite quickly. In April 2006 I was approved as a Debian Developer. Meanwhile, I looked at the videos from DebConf 5 and thought that it would be useful to distribute them on a DVD. That led me to start writing video software and to get involved in the video team for the next year s DebConf. Raphael: You have been one the main driver behind the removal of non-free firmwares from the kernel. Explain us what you did and what s the status nowadays? Ben: That s giving me a bit more credit than I deserve. For a long time the easy way for drivers to load firmware programs was to include them as a blob in their static data, but more recently the kernel has included a simple method for drivers to request a named blob at run-time. These requests are normally handled by udev by reading from files on disk, although there is a build-time option to include blobs in the kernel. Several upstream and distribution developers worked to convert the older drivers to use this method. I converted the last few of these drivers that Debian included in its binary packages. In the upstream Linux source, those blobs have not actually been removed; they have been moved to a firmware subdirectory. The long-term plan is to remove this while still allowing the inclusion of blobs at build-time from the separate linux-firmware repository. For now, the Debian source package excludes this subdirectory from the upstream tarball, so it is all free software. There are still a few drivers that have not been converted, and in Debian we just exclude the firmware from them (so they cannot be built). And from time to time a driver will be added to the staging section of Linux that includes firmware in the old way. But it s understood in the kernel community that it s one of the bugs that will have to be fixed before the driver can move out of staging . Raphael: Do you believe that Debian has done enough to make it easy for users to install the non-free firmwares that they need? Ben: The installer, the Linux binary packages and initramfs-tools will warn about specific files that may be needed but are missing. Users who have enabled the non-free section should then be able to find the necessary package with apt-cache search, because each of the
binaries built from the firmware-nonfree source package includes driver and file names within its description. For the installer, there is a single tarball that provides everything. We could make this easier, but I think we have gone about as far as we can while following the Debian Social Contract and Debian policy. Raphael: At some point in the past, the Debian kernel team was not working very well. Did the situation improve? Ben: Back in 2008 when I started working on the Linux kernel package to sort out the firmware issues, I think there were some problems of communication and coordination, and quite possibly some members were burned-out. Since then, many of the most active kernel team members have been able to meet face-to-face to discuss future plans at LPC 2009 in Portland and the 2010 mini-DebConf in Paris. We generally seem to have productive discussions on the debian-kernel mailing list and elsewhere, and I think the team is working quite well. Several new contributors have joined after me. I would say our biggest problem today is that we just don t have enough time to do all we want to. Certainly, almost all my Debian time is now taken up with integrating upstream kernel releases and handling some fraction of the incoming bug reports. Occasionally I can take the time to work on actual features or the other packages I m neglecting!

Our biggest problem today is that we just don t have enough time to do all we want to.

Raphael: It is widely known that Linux is maintained in a git repository. But the Debian kernel team is using Subversion. I believe a switch is planned. Why was not git used from the start? Ben: The linux-2.6 source package dates from the time when Linus made his first release using git. I wasn t part of the team back then so I don t know for sure why it was imported to Subversion. However, at that time hardly anyone knew how to use git, no-one had experience hosting public git repositories, and Alioth certainly didn t offer that option. Today there are no real blockers: everyone on the kernel team is familiar with using git; Alioth is ready to host it; we don t have per-architecture patches that would require large numbers of branches. But it still takes time to plan such a conversion for what is a relatively complex source package (actually a small set of related source packages). Raphael: What are your plans for Debian Wheezy? Ben: Something I ve already done, in conjunction with the installer team, is to start generating udebs from the linux-2.6 source package. The kernel and modules have to be repacked into lots of little udebs to avoid using too much memory during installation. The configuration for this used to be in a bunch of separate source packages; these could get out of step with the kernel build configuration and this would only be noticed some time later. Now we can update them both at the same time, they are effectively cross-checked on every upload, and the installer can always be built from the latest kernel version in testing or unstable. I think that we should be encouraging PC users to install the 64-bit build (amd64), but many users will still use 32-bit (i386) for backward compatibility or out of habit. On i386, we ve slightly reduced the variety of kernel flavours by getting rid of 686 and making 686-pae the default (previously this was called 686-bigmem ). This means that the NX security feature will be used on all systems that support it. It should also mean that the first i386 CD can have suitable kernel packages for all systems. I have been trying to work on providing a full choice of Linux Security Modules (LSMs). Despite their name, they cannot be built as kernel modules, so every enabled LSM is a waste of memory on the systems that don t use it. This is a significant concern for smaller Debian systems. My intent is to allow all unused LSMs to be freed at boot time so that we can happily enable all of them. I recently proposed to drop support for older x86 systems, starting with 486-class processors in wheezy. In general, this would allow the use of more compiler optimisations throughout userland and the kernel. However it seems that there isn t that much to be gained unless we also drop 586-class processors, and there are still quite a few of those in use. So I think this will have to wait. Uwe Kleine-K nig has been working to include real-time support (also known as PREEMPT_RT). This can provide low and very predictable I/O latency, which is useful for live audio synthesis, for example. It still requires a number of patches and a build configuration change, resulting in a separate binary package. We re currently only building that for 64-bit PCs. (You may notice this is missing for Linux 3.1, because the real-time developers skipped this release.) Raphael: What s the biggest problem of Debian? Ben: I think we try too hard to accommodate every possible option, without regard for the cost to developers and users in general. As an example, we now have sysvinit, file-rc, upstart and systemd all in testing. Daemon maintainers can t rely on any advanced features of upstart or systemd because we refuse to choose between them. And the decision to support the FreeBSD kernel means that we cannot choose upstart or systemd as the only option. So all daemon maintainers will have to maintain those baroque init scripts for the indefinite future. We really should be able to decide as a distribution that when one option is technically good and popular then it can be made the only option. But no-one really has the authority to do that, so we muddle along with the pretence that all the options are equally valid and functional, while none of them is supported as well as they should be.

We try too hard to accommodate every possible option, without regard for the cost to developers and users in general.

We also try to build every package on every architecture, in general. I m quite sure there are many (package, architecture) combinations that have no users, ever. But if at some point that combination FTBFS, developers will waste time investigating and fixing that time that could have been spent working on bugs and features that users actually care about. Yes, sure, portability is good but you can t prove portability just by making a package compile on every architecture. This also applies to the selection of drivers for the kernel, by the way. Raphael: Is there someone in Debian that you admire for their contributions? Well, there are many people, but I will pick out just a few: Steve McIntyre, for his work as DPL to improve communication with the various Debian derivatives and to bring fresh blood into various core teams. Also for being a generous host for countless Debian social and bug-squashing events. Stefano Zacchiroli, for improving further on communications with both downstream and upstream projects, and for regularly exercising his power to lead discussions to the benefit of the project. Julien Cristau, for maintaining good humour while not only fighting against the tide of graphics driver regressions in X and the Linux kernel but also working on release management. Jonathan Nieder, for taking on the unglamorous and frustrating task of kernel bug triage as a non-maintainer and developing it to a fine art.

---

Thank you to Ben for the time spent answering my questions. I hope you enjoyed reading his answers as I did.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

4 comments Liked this article? Click here. My blog is Flattr-enabled.

<p><a href="http://www.physorg.com/news/2011-12-debunks-myths-gender-%0Amath.html">Study debunks myths about gender and math performance</a></p> <p><a href="/tags/math" class="tag">#math</a> , <a href="/tags/gender" class="tag">#gender</a>, <a href="/tags/physorg" class="tag">#physorg</a></p> <blockquote> <p>A major study of recent international data on school mathematics performance casts doubt on some common assumptions about gender and math achievement in particular, the idea that girls and women have less ability due to a difference in biology.</p> <p>&quot;We found that boys as well as girls tend to do better in math when raised in countries where females have better equality, and that&#39;s new and important,&quot; says Kane. &quot;It makes sense that when women are well-educated and earn a good income, the math scores of their children of both genders benefit.&quot;</p> <p>Mertz adds, &quot;Many folks believe gender equity is a win-lose zero-sum game: If females are given more, males end up with less. Our results indicate that, at least for math achievement, gender equity is a win-win situation.&quot;</p> </blockquote>

My home network has several devices that do not have large persistent storage to keep log files. For example, my wireless routers based on OpenWRT doesn t log to the limited local storage it has, and a Flukso energy metering device log power readings to a ramdisk. These devices log a fair amount of information that I ideally would like to keep for later analysis. I have never before seen a need to setup a syslogd server, thinking that storing logs locally and keeping regular backups of the machine is good enough. However, it appears like this situation calls for a syslogd server. I found an old NSLU2 in my drawer and installed Debian Squeeze on it following Martin Michlmayr s instructions. I m using a 4GB USB memory stick for storage, which should hold plenty of log data. I keep backups of the machine in case the USB memory stick wears out. After customizing the installation to my preferences (disable ssh passwords, disable portmap/rpc.statd/exim4, installing etckeeper, emacs23-nox, etc) I am ready to configure Rsyslog. I found what looked like the perfect configuration example, Storing messages from a remote system into a specific file , but it requires me to hard code a bit too much information in the configuration file for my taste. Instead, I found the DynFile concept. With a file /etc/rsyslogd.d/logger.conf as below I can point any new device to my log server and it will automatically create a new file for it. And since the dates are embedded into the filename, I get log rotation suitable for rsync-style backups for free.

$ModLoad imudp
$UDPServerRun 514 $ModLoad imtcp
$InputTCPServerRun 514 $template DynFile, /var/log/network-%HOSTNAME%-%$year%-%$month%-%$day%.log
:fromhost-ip, !isequal, 127.0.0.1 ?DynFile
:fromhost-ip, !isequal, 127.0.0.1 ~

Thanks to the longstading work by Paul Wise in nearly 3 years...and with some help from my side (and from other members of the font packaging team), we're getting closer of a longstanding goal : removing Defoma from the archive. Defoma is indeed obsoleted by fontconfig, which doesn't need these "hints" file, turning the packaging of fonts into a very simple process. On last Saturday, Paul sent the removal request for defoma. The request mentions blocking bugs that are, for most of them, requests for font packages to drop the dependency on defoma. As we already sent patches for these bugs, I started NMU'ing the remaining packages this week-end. Most of them were apparently nearly abandoned packages, so I had no shame doing that so quickly. Indeed, I no longer fear NMU'ing packages even for non critical issues and I even sometimes no longer fear adding a few other fixes when the packages are very obviously poorly maintained (things like source v3, debhelper compatibility bump, build- indep arch targets support, etc.). After this NMU wave, we should be left, in one week, with only five packages blocking the removal of defoma. I might even NMU some of them, though some are TeX-related packages providing T1 fonts additionnally to TrueType ones...something I'l less familiar with. Whatever happens, I now nearly sure that wheezy will not have defoma. Yay! Hats off to Paul.

(It s a new day, it s a new bug ) A regression appeared in the synaptics input driver, between the 1.4.1 and the 1.5.0 release, as reported in Debian #649002 by many users. Basically, no more touchpad on PowerPC (hi, iBook G4 users!). :-( What follows is how I tracked it down. The upstream bug report is fdo#43728. Anyone could have done so, there s no black magic involved. A little hint maybe: git bisect is really easy on X drivers, since one can build and test without the Debian packaging bits. 1st step: Getting started You need sources and build dependencies, nothing fancy. Since packages might be named a bit differently, there s a reference to the relevant upstream repositories in debian/watch for X packages.

# You need git here:
debcheckout xserver-xorg-input-synaptics synaptics.git
cd synaptics.git
# Let s get the upstream tags and branches:
cat debian/watch
git remote add upstream git://anongit.freedesktop.org/xorg/driver/xf86-input-synaptics
git fetch --all
# Let s install build dependencies:
sudo apt-get build-dep xserver-xorg-input-synaptics

2nd step: Reproducing It was said 1.5.0 was the first known buggy version, so let s make sure:

# Get the appropriate tag:
git checkout xf86-input-synaptics-1.5.0
# Prepare the build system:
autoreconf -vfi
# X finds its modules under /usr, not /usr/local:
./configure --prefix=/usr
# Build and install:
make && sudo make install

Now time to restart the display manager, and confirm that the pointer is not moving bug reproduced. Now let s check the 1.4.1 release isn t affected:

# Clean everything:
git clean -xdf
git checkout xf86-input-synaptics-1.4.1
autoreconf -vfi
./configure --prefix=/usr
make && sudo make install

And after a display manager restart, the pointer is moving again, good! For reference, the log has very different lines about appletouch, so it can be used to programmatically learn about the status (good or bad). 3rd step: Narrowing it down So, we have a known good version and a known bad version. Time for a binary search to find the guilty commit which introduced that regression!

git bisect start
git bisect good xf86-input-synaptics-1.4.1
git bisect bad  xf86-input-synaptics-1.5.0

You don t even need to know that 1.4.x and 1.5.x live in different branches, git figures that out for you:

Bisecting: a merge base must be tested
[de0dfb76444ad4160468d00515876c91a9fa20bf] synaptics 1.4.0

It s just a matter of running autoreconf && ./configure && make && sudo make install && sudo /etc/init.d/xdm restart every step of the way, so it s pretty easy. No wonder, 1.4.0 was working OK, so it can be recorded as such through git bisect good, and the binary search goes between that commit and 1.5.0. After a few iterations of git bisect good and git bisect bad, we get to the commit reported on the upstream bug. As an extra bonus, mentioning the upstream bug number/URL in the Debian bug means it can be marked as forwarded there and we ll receive automatic notifications when its status changes, thanks to bts-link. Now, if you want to provide a patch for this bug, you may want to try and revert this commit. 4th step: Providing a patch Let s go back to the affected release and try to revert the guilty commit:

git checkout xf86-input-synaptics-1.5.0
git revert 13543b156d78bc4d01a19844a5ee8f283269621b

Unfortunately, some later commits modified the affected areas so we re running into conflicts:

error: could not revert 13543b1... eventcomm: replace synaptics-custom TEST_BIT with server's BitIsOn.
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

Let s try to understand what the offending commit did: it replaced a set of macros with a single one, taken from the server. Both the old TEST_BIT() and the new BitIsOn() take 2 arguments, but the order is reversed. Reverting it should only be a matter of introducing the TEST_BIT() macro again (along with the two other ones it needs), and using it. The problem is replacing all BitIsOn(foo,bar) with TEST_BIT(bar,foo): foo and bar might not be trivial, they could be complex C expressions, and a replacement in your favourite editor might not get all occurrences right. Thankfully, there s a nice tool to handle such things, called coccinelle. I won t go into details, just show how it can help here. Basically you just need to craft a tiny patch, called a semantic patch, which describes what I wrote in the previous paragraph. You only need to declare two expressions, say a and b, and declare that all BitIsOn(a,b) must be replaced with TEST_BIT(b,a). Here s the syntax:

@@
expression a,b;
@@
-BitIsOn(a,b)
+TEST_BIT(b,a)

Save it under testbit.cocci and apply it through spatch (from the coccinelle package), asking it to perform in-place replacement (we re in a git checkout copy, remember):

# Apply:
spatch -sp_file testbit.cocci -in_place .
# Profit:
git diff

Then you can run git commit -s, write a nice commit message, send it upstream, and enjoy the rest of the night. Coming soon to a mirror near you: fixing commit. As mentioned on the upstream bug report, I got the Debian reference wrong in the commit message, I really meant #649002.

Mako's IronBlogger is a great idea. I often find myself postponing blog posts for a very long time, simply out of laziness. IronBlogger provides a nice community incentive to counter (my) laziness and blogging more often. As a related challenge, we have to face the fact that different subsets of our communities use different media to stay informed: mailing lists, blog (aggregators), social media, IRC, etc. Disparities in how they stay informed are a pity and can be countered using multiple medias at a time. Although I haven't blogged very often as of lately, I managed to keep the Debian (Developer) community informed of what happens in "DPL land" on a monthly basis, by the means of bits from the DPL mails sent to d-d-a. While the target of bits mails perfectly fits d-d-a, there is no reason to exclude a broader public from them. After all, who knows, maybe we'll find the next DPL victim^W candidate among Planet readers! Bonus point: blogging this also helped me realize that my mails are not as markdown-clean as I thought they were. I still have no IronBlogger squad, though. (And sharing beers with folks in the Boston area is not terribly handy for me ). Anyone interested in setting up a BloggeurDeFer in the Paris area? (SCNR)

---

Dear Project Members,
another month has passed, it's time to bother you again about what has happened in DPL land in November (this time, with even less delay than the last one, ah!). Call for Help: press/publicity team I'd like to highlight the call for help by the press / publicity teams. They are "hiring" and have sent out a call for new members a couple of weeks ago. The work they do is amazing and is very important for Debian, as important as maintaining packages or fixing RC bugs during a freeze. It is only by letting the world know what Debian is and what we do, that we can keep the Project thriving. And letting the world know is exactly what the publicity and press teams do. If you're into writing, blogging, or simply have a crush for social media, please read the call and "apply"! Interviews November has apparently been the "let's interview the DPL" month. I've spent quite some time giving interviews to interested journalists about various topics. For both my embarrassment and transparency on what I've said on behalf of Debian, here are the relevant links:

• audio interviewwith Karen Sandler for the Free as in Freedom oggcast by herself and Bradley M. Kuhn
• text interview with Raphael Hertzog for the People behind Debian series
• text interview for the Linux Certification Institute Italy (LICI) (note: the original of this one is in Italian; the English version is a translation by LICI)
• video interview with Amber Granger during last UDS

Assets

• the Video Team sprint approved in October has happened at beginning of November; report is still pending but the organizers are on it
• Christian Perrier reported about his trip to India mini-DebConf, even that he attended on behalf of the Project. I find amazing both the outcome of the event and Christian's report, a very good example of how to inform the Project when acting on behalf of it
• the hardware replacement for ftp,lists .d.o I've mentioned last month has been ordered, has arrived, and DSA have already put their hands on it for the setup. Many thanks to DSA and in particular to Martin Zobel-Helas for the logistics
• as an administratrivia, I've asked SPI to setup for Debian a "gift" credit card, i.e. a credit card with a limited, refillable budget. It is needed for using (with a safeguard) credits offered to the Project by public cloud providers (e.g. AWS' donation), as well as to use various kinds of services that we need from time to time

Legal advice (work in progress)

• I've sought and got from a former SPI lawyer a first legal advice on the relationships among trademark law and the procedures of a Free Software distro like Debian when distributing software subject to trademarks. I'm still waiting to seek a second advice from the current SPI lawyers (whom I've been keeping busy with other stuff). In the meantime I've pending a summary of what I've got thus far in the trademark-vs-DFSG thread. Stay tuned there!
• I've been working with SPI lawyers on a proposal of software patent policy that I think we should advertise to make explicit the position of the Debian Project on software patents. I see it as a continuation/completion of the FAQ we have already published on the subject. I'll let you know as soon as we reach a stable draft, that should be pretty soon.

Relationships with others

• I've been approached by members of the board of the OpenSource Initiative (OSI), about the soon to be open OSI affiliation program. They'd like to know if Debian is interested in being affiliated to OSI. I think we should and I'll eventually start a discussion on this topic. In the meantime, I welcome feedback from anyone who is interested in the topic.
• Thanks to the interest of Andrew Pollock, LaMont Jones, Florian Weimer and the Security Team, we are discussing with ISC to have Debian --- as a project, rather than as individuals who just happen to maintain ISC software in Debian --- become member of all forums relevant for software we distribute (BIND, NTP, DHCP). They are doing all the coordination work, but I've anticipated I'll be happy to pledge for Debian membership.
• I've taken the first step to reach out to companies that have a strategic interest in Debian. According to discussion I've had with representative of such companies, they often face similar challenges in offering Debian support and in contributing back to Debian. I think we should offer a forum for them to discuss and find synergies, as well as explain our needs to them. An official mailing list might be such a place. Once we have a suitable forum ready, I'll propose to the press team to send out a corresponding call for companies. In the meantime, if you happen to know, or even work for, companies that might be interested, please ask them to get in touch with me.

Miscellanea

• I've had a chance to discuss with the editors of the SPDX specification about its relationship with DEP-5. Thanks to the help of Dominique Dumont work is already in progress to have a mutual converter among the two formats. Also, although with some bumps, work on finalizing DEP-5 has finally restarted.
• an interesting thread of a while ago is in need of some work to produce a useful outcome, i.e. a set of security guidelines for DDs. If you're willing to help, have a look at the thread
• some more mediation-fu has been going on during November; have a look at my daily logs below if you care about the details

Thanks for reading thus far,
and happy hacking.

---

PS as usual, the boring day-to-day activity log is available at master:/srv/leader/news/bits-from-the-DPL.*

I left you 2.5 months ago with the last question asked by my applicaiton manager, Martin Michlmayr : "Please tell me about about yourself and what you intend to do for Debian". Interesting question to revisit now, indeed. Here is what I answered: About myself first. I'm a 40 year old project manager and system administrator working in French National Aerospace Research Center. My best definition of my skills in computing is "Know more or less about a Lot of Things and be a Specialist of Nothing"...:-). I'm definitely not a programmer, nor a real system administrator, nor a RDBMS administrator, nor a personal workstation designer, though I do all of these daily. I think I'm perfect for finding the good person for having a defined job done. Besides this, I'm a genealogist for several years now. This is what finally decided me to apply for becoming a package maintainer : there are some quite good free genealogy software for Unix, though for various reasons they are not used very widely, even Unix geeks (my main software for genealogy still runs on Another Operatin System and is evertythig but free).I think that I can bring something here to the Free Software World, by helping some of these good programs in getting into the best Linux distribution I know.... For me, this is a mean for giving back to the free software movement what I gives to me since I discovered Linux 6-7 years ago. My very first intention as soon as I get my way into the Debian Developers Heaven is adopting the Geneweb package currently maintained by Brent Flugham. I'm in close contact with the author (who happens to be french, which helps) as well as a daily user of it. The current package which is in the distribution is already my work for a great part. I gave it to Brent, the current maintainer and we both agreed that it would be better for me to apply to becoming anofficial maintainer. I also contributed to the package for lifelines, another genealogy software. The last version of the package is also 80% my work, acknowledged by Javier, the official maintainer. Concerning that package, I do not have "plans" for adopting it (we didn't discussed of this with Javier, and I'm not sure I could bring him that much things). I came to Linux thanks to a great friend of mine, Ren Cougnenc. Ren opened my eyes to the free software world when I still thought that it was only a variant of free beers. I got really involved into Linux when I forced me to remove any other Operating System from my computer at work and tried to do my daily job with Linux. I have now succeeded at ONERA in getting free software to be accepted as a credible alternative for important projects. At this time, especially for server and network-related projects. I absolutely cannot tell why and how I came to be a Debian user. I simply don't remember. But I know why I am still a Debian user : this is a distribution which is controlled by only one organisation--->its users. And I want to be part of it. Finally, I did not mention above the somewhat "political" nature of my personal involvment into free software. Except for the physical appearence, I think I mimic RMS on several points (though he probably speaks better french than I try to speak english....which does not help for expressing complex ideas like the ones above!). As anyone can see, I was already very verbose when writing, sorry for this. Funnily Martin summed this up in one paragraph when he posted his AM report about my application. From what I see, also, my English didn't improve that much since then. It seems this is a desperatecause, I'm afraid. Anyway, all this was apparently OK for Martin and, on July 21st 2001, he wrote and posted his AM report and, on July 30th 2001, I got a mail by James Troup: An account has been created for you on developer-accessible machines with username 'bubulle'. bubulle@debian.org was born. Now I can more easily destro^W contribute to my favourite Linux distro. Indeed, I don't remember that much about the 2001-2003 years. I was probably not that active in Debian. Mostly, I was maintaining geneweb, for which I polished the package to have it reach a quite decent state, with elaborated debconf configuration. Indeed, at that time, I was still also deeply involved in genealogy research and still contributing to several mutual help groups for this. This is about the time where I did setup my web site (including pages to keep the link with our US family, which we visited in 2002). I think that the major turn in my Debian activities happened around september 2002 when Denis Barbier contacted me to add support in geneweb for a new feature he introduced in Debian : po-debconf. At that time, I knew nearly nothing about localization and internationalization. Denis was definitely one of the "leaders" in this effort in Debian. During these years, he did a tremendous job setting up tools and infrastructure to make the translation work easier. One of his achievements was "po-debconf", this set of tools and scripts that allows translation debconf "templates", the questions asked to users when configuring packages. All this lead me to discover an entire new world : the world of translating software. As often when I discover something I like, I jumped into it very deeply. Indeed, in early January 2003, I did my very first contributions to debian-l10n-french and began working on systematic translation of debconf templates. Guess what was the goal : 100%, of course! Have ALL packages that have debconf templates...translated to French. We reached that goal.....on June 2nd 2008 in unstable (indeed "virtually" : all packages were either 100% translated...or had a bug report with a complete translation) and on December 21st 2010 for testing. Squeeze was indeed the first Debian release with full 100% for French. Something to learn with localization work: it's never finished and you have to be patient. So, back in 2003, we were starting this effort. Indeed, debian-l10n-french was, at that time, an incredibly busy list and the translation rate was very high: I still remember spending my summer holidays translating 2-3 packages debconf templates every day for two weeks. Meanwhile, my packaging activities were low: only geneweb and lifelines, that was all. Something suddenly changed this and it has been the other "big turn" in my Debian life. After summer 2003, I suddenly started coming on some strange packages that were needing translation: they were popping up daily in lists with funny names like "languagechooser", "countrychooser", "choose-mirror", etc. I knew nothing about them and started "translating" their strings too, and sending bug reports after a decent review on debian-l10n-french. Then, Denis Barbier mailed me and explained me that these things were belonging to a new shiny project named Debian Installer and meant to replace the good old boot-floppies. Denis explained me that it would maybe be more efficient to work directly in the "D-I" team and "commit" my work instead of sending bug reports. Commit? What's that? You mean this wizard tool that only Real Power Developers use, named "CVS"? But this is an incredibly complicated tool, Denis. Do you really want me, the nerd DD, to play with it? Oh, and in this D-I development, I see people who are close to be semi-gods. Names I read in mailing lists and always impress me with their Knowledge and Cleverness: Martin Michlmayr (my AM, doh), Tollef Fog Heen, Petter Reinholdtsen and so many others and, doh, this impressive person named "Joey Hess" who seems to be so clever and knowledgeable, and able to write things I have no clue about. Joey Hess, really? But this guy has been in Debian forever. Me, really? Work with the Elite of Debian? Doh, doh, doh. Anyway, in about two months time, I switched from the clueless guy status to the status of "the guy who nags people about l10n in D-I", along with another fellow named Denny "seppy" Stampfer". And then we started helping Joey to release well localized D-I alphas and betas at the end of 2003 (the release rate at the time was incredible: Sarge installer beta1 in November 2003, beta2 in January 2004). I really remember spending my 2003 Christmas holidays hunting for....100% completion of languages we were supporting, and helping new translators to work on D-I translation. Yes, 8 years ago, I was already doing all this..:-)...painting the world in red. All this leads up to the year 2004. Certainly the most important year in my Debian life because it has been....the year of my first DebConf. But you'll learn about this....in another post (hopefully not in 2.5 months).

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:

In Total:	1656
Affecting Wheezy:	1005
Wheezy only:	156
Remaining to be fixed in Wheezy:	849

Of these 849 bugs, the following tags are set:

Pending in Wheezy:	53
Patched in Wheezy:	157
Duplicates in Wheezy:	50
Can be fixed in a security Update:	31
Contrib or non-free in Wheezy:	9
Claimed in Wheezy:	0
Delayed in Wheezy:	9
Otherwise fixed in Wheezy:	67

Ignoring all the above (multiple tags possible) 551 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released. However, with the view of the Release Managers, 906 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

There is an interesting interview with Richard Wilkinson on ted.com about the damage to society that is caused by excessive inequality [1]. The conclusion is interesting, it seems that one part of the solution is to send the message that excessive greed is anti-social. The demonstrations against the greedy 1% seem likely to do some good in this regard. He also has a TED talk about this in which he shows that even the rich don t benefit from great inequality [2]. If nothing else the links between great inequality and high rates of mental illness and murder should convince people that they don t want to live in such a society. Buying Cameras Some of the radio news reports have claimed that some of the people in OWS camps in Australia are somehow unworthy because they have smart-phones and digital cameras. They seem to believe that people have to demonstrate great poverty to have standing to complain. The problem here is that owning a mobile phone is essential to getting employment in many industries, so the vast majority of people who are old enough to work but not near retirement age just have to own a mobile phone. Once you have decided to use a phone if you are going to use it a lot (which you probably will if living in a tent in a public place) then being on a contract usually makes economic sense. As the cheapest contracts include a free Android phone that means almost everyone who is old enough to be allowed out alone at night and young enough to be willing to sleep rough for a cause will have such a phone. A related complaint is about people owning good cameras. One reason for this is the fact that the huge number of cameras deters the police from using excessive force that s a good justification for spending money on a good camera. But then a good camera doesn t cost much nowadays, the specs on $100 cameras are quite impressive and a camera that costs $400 new will have features that are useful for professional photographers and cameras are even cheaper on ebay. In Australia social security payments for a single person who is out of work are as much as $486.80 per fortnight [3]. Therefore someone who is living on social security could buy a great camera if they saved up two weeks of social security payments or buy a similar camera on ebay for one week s worth of payments. When at a protest the widespread use of cameras is essential for protection against police brutality. It s worth saving up for a camera. But really they are missing the point, OWS is about representing ~99% of the population. Even people on the average income are suffering because of the greed of the rich. Is Australia such a Great Place? There have been claims that Australia is such a great place that OWS protests are not needed, supposedly we have a very egalitarian society. The ABS data shows that the level of inequality as expressed by numbers isn t that great when restricted to just income if you restrict the analysis to the top 20% [4]. The mean gross household income per week is $340 for the lowest quintile and $2,380 for the highest quintile giving a ratio of 1:7.0. But when you look at total household net worth it s $27,400 vs $1,720,700 a ratio of 1:62.8! The ratio for the richest 1% would surely be a lot greater but a quick search of abs.gov.au didn t turn up any statistics on them. Also Richard Wilkinson s analysis puts Australia quite close to the US. If the people who are in the top 1% earned their money, paid taxes, and didn t pervert the political process then there wouldn t be any OWS protests. The corruption in the political process and rent seeking by corporations isn t as bad here as it is in the US, but it s bad enough that we need some serious changes. Emily Manuel wrote an interesting article for Tiger Beatdown about the need for an Occupy movement in Australia [5]. Her description of the political problems in Australia is quite clear. But I wasn t convinced by her analysis which suggested that Australia is different from the US in any significant way. Really Comparing Income To properly compare income it doesn t make sense to compare a ratio of money earned, or disposable income which is often misleading due to disagreements about what is considered disposable. It s really about which of the things that you want to do can be achieved on your income. One possible way of comparing it is based on Maslow s hierarchy of human needs. The poorest 20% of households in Australia have a gross income of $340 per week. The ABS shows that the average of private rental prices in the 8 capital cities (where most of the population lives) is $218 per week [6]. Obviously the poorest households can t afford an average rent unless you have lots of people packed in one house. No matter how you look at the numbers it s going to be difficult for low income people to pay for rent and food. That puts them on the bottom level of Maslow s higherarchy. It seems to me that the benefits of increased income diminish and it becomes more of a contest to be richer than other people. Owning an old car is a significant benefit over having no car. Owning a new car has some real benefits over an old car. Luxury cars usually have safety features before they are introduced to cheaper cars, but apart from that the benefit of owning a luxury car is small. Owning multiple Rolls-Royces doesn t seem to provide any real benefit, someone who would want to do that would probably be better served by hiring a good psychologist. Update: In a comment Brendan Scott cited The Theory of the Leisure Class by Thorstein Veblen. That s a great point, and I ll have to read that book! I still maintain that owning two Rolls Royces is a symptom of psychological problems, people who can sort out such problems won t feel the need to try and prove that they have more money than others. Charles Stross has an interesting article about wealth disparity etc [7]. Being a sci-fi author he also explores what life might be like after Drexler boxes satisfy most of our current materiel needs. The Age has an informative article by Suzy Freeman-Greene contrasting the OWS protests in Melbourne which were broken up by police to the Catholic anti-abortion protests that have been going on for more than 20 years [8]. It seems that corporations aren t the only large organisations that are pwning democracy!

• [1] http://blog.ted.com/2011/10/26/we-quite-suddenly-realized-that-what-we-were-looking-at-was-a-general-pattern-qa-with-richard-wilkinson/
• [2] http://www.ted.com/talks/richard_wilkinson.html
• [3] http://www.centrelink.gov.au/internet/internet.nsf/payments/newstart_rates.htm
• [4] http://www.abs.gov.au/ausstats/abs@.nsf/0/4EABC33B47FEE100CA25773700169C75?opendocument
• [5] http://tigerbeatdown.com/2011/10/25/why-do-we-need-an-occupy-australia/
• [6] http://www.abs.gov.au/AUSSTATS/abs@.nsf/Lookup/1301.0Chapter10052009-10
• [7] http://www.antipope.org/charlie/blog-static/2011/10/a-cultural-experiment.html
• [8] http://www.theage.com.au/opinion/society-and-culture/some-occupy-to-protest-others-to-hamper-harangue-and-hurt-20111111-1nbn4.html

Today, I've released GePeS 0.2.0. The only major difference is that it should reduce power consumption of your phone while it is running. Reducing CPU usage when application is not active was definitely on my roadmap, though email from "Publisher Support Specialist for Nokia" about this actually pushed me to investigate the issue. It turned out to be quite easy to disable sensors while application is not active and it did the trick. While doing this I also moved to using AlignedTimer to synchronize wakeups with other possible wakeups in the device. As usual, the application sources can be downloaded on GePeS page, binary will be soon available in Nokia store.

Filed under: English Meego 0 comments Flattr this!

It s been a couple days since I returned from this year s WebKitGTK+ hackfest in A Coru a, Spain. The weather was very nice, not too cold and not too rainy, we had great food, great drinks and I got to meet new people, and hang out with old friends, which is always great! I think this was a very productive hackfest, and as usual a very well organized one! Thanks to the GNOME Foundation for the travel sponsorship, to our friends at Igalia for doing an awesome job at making it happen, and to Collabora for sponsoring it and granting me the time to go there! We got a lot done, and although, as usual, our goals list had many items not crossed, we did cross a few very important ones. I took part in discussions about the new WebKit2 APIs, got to know the new design for GNOME s Web application, which looks great, discussed about Accelerated Compositing along with Joone, Alex, Nayan and Martin Robinson, hacked libsoup a bit to port the multipart/x-mixed-replace patch I wrote to the awesome gio-based infrastructure Dan Winship is building, and some random misc. The biggest chunk of time, though, ended up being devoted to a very uninteresting (to outsiders, at least), but very important task: making it possible to more easily reproduce our test results. TL;DR? We made our bots and development builds use jhbuild to automatically install dependencies; if you re using tarballs, don t worry, your usual autogen/configure/make/make install have not been touched. Now to the more verbose version! The need For a couple years now we have supported an increasingly complex and very demanding automated testing infrastructure. We have three buildbot slaves, one provided by Collabora (which I maintain), and two provided by Igalia (maintained by their WebKitGTK+ folks). Those bots build as many check ins as possible with 3 different configurations: 32 bits release, 64 bits release, and 64 bits debug. In addition to those, we have another bot called the EWS, or Early Warning System. There are two of those at this moment: one VM provided by Collabora and my desktop, provided by myself. These bots build every patch uploaded to the bugzilla, and report build failures or passes (you can see the green bubbles). They are very important to our development process because if the patch causes a build failure for our port people can often know that before landing, and try fixes by uploading them to bugzilla instead of doing additional commits. And people are usually very receptive to waiting for EWS output and acting on it, except when they take way too long. You can have an idea of what the life of an EWS bot looks like by looking at the recent status for the WebKitGTK+ bots. Maintaining all of those bots is at times a rather daunting task. The tests require a very specific set of packages, fonts, themes and icons to always report the same size for objects in a render. Upgrades, for instance, had to be synchronized, and usually involve generating new baselines for a large number of tests. You can see in these instructions, for instance, how strict the environment requirements are yes, we need specific versions of fonts, because they often cause layouts to change in size! At one point we had tests fail after a compiler upgrade, which made rounding act a bit different! So stability was a very important aspect of maintaining these bots. All of them have the same version of Debian, and most of the packages are pinned to the same version. On the other hand, and in direct contradition to the stability requirement, we often require bleeding edge versions of some libraries we rely on, such as libsoup. Since we started pushing WebKitGTK+ to be libsoup-only, its own progress has been pretty much driven by WebKitGTK+ s requirements, and Dan Winship has made it possible to make our soup backend much, much simpler and way more featureful. That meant, though, requiring very recent versions of soup. To top it off, for anyone not running Debian testing and tracking the exact same versions of packages as the bots it was virtually impossible to get the tests to pass, which made it very difficult for even ourselves to make sure all patches were still passing before committing something. Wow, what a mess. The explosion^Wsolution So a few weeks back Martin Robinson came up with a proposed solution, which, as he says, is the nuclear bomb solution. We would have a jhbuild environment which would build and install all of the dependencies necessary for reproducing the test expectations the bots have. So over the first three days of the hackfest Martin and myself hacked away in building scripts, buildmaster integration, a jhbuild configuration, a jhbuild modules file, setting up tarballs, and wiring it all in a way that makes it convenient for the contributors to get along with. You ll notice that our buildslaves now have a step just before compiling called updated gtk dependencies (gtk is the name we use for our port in the context of WebKit), which runs jhbuild to install any new dependencies or version bumps we added. You can also see that those instructions I mentioned above became a tad simpler. It took us way more time than we thought for the dust to settle, but it eventually began to. The great thing of doing it during the hackfest was that we could find and fix issues with weird configurations on the spot! Oh, you build with AR_FLAGS=cruT and something doesn t like it? OK, we fix it so that the jhbuild modules are not affected by that variable. Oh, turns out we missed a dependency, no problem, we add it to the modules file or install them on the bots, and then document the dependency. I set up a very clean chroot which we could use for trying out changes so as to not disrupt the tree too much for the other hackfest participants, and I think overall we did good. The aftermath By the time we were done our colleagues who ran other distributions such as Fedora were already being able to get a substantial improvements to the number of tests passing, and so did we! Also, the ability to seamlessly upgrade all the bots with a simple commit made it possible for us to very easily land a change that required a very recent (as in unreleased) version of soup which made our networking backend way simpler. All that red looks great, doesn t it? And we aren t done yet, we ll certainly be making more tweaks to this infrastructure to make it more transparent and more helpful to the users (contributors and other people interested in running the tests). If you ve been hit by the instability we caused, sorry about that, poke mrobinson or myself in the #webkitgtk+ IRC channel on FreeNode, and we ll help you out or fix any issues. If you haven t, we hope you enjoy all the goodness that a reproducible testing suite has to offer! That s it for now, folks, I ll have more to report on follow-up work started at the hackfest soon enough, hopefully =).

Procps-ng, the debian, Fedora and OpenSuSE fork of procps had another release today. This is a bugfix release that fixes some important bugs that have cropped up in 3.3.0 pgrep crashes, pgrep -u not finding processes and a problem with top forest view have all been fixed. An important addition to this release is that test scripts have now been added using the Dejagnu framework. This framework tests most of the programs that ship with procps-ng. There is only 100 tests so far and they are only tested for Linux architecture, but we expect to increase this test count and sophistication with each release. The test scripts even found long-present but unknown bugs in the package. For example, running ps with output flags around the signal masks work as expected in Linux, but fail on kFreeBSD. This is due to the unusual output of the relevant lines on that architecture. This is not a new bug, bug has been around for quite some time; noone ever noticed (or reported) it. The Debian package has already been built and uploaded to the master FTP server. Oh, and it is the first package with my new GPG key!

At work we use .procmailrc files generated by CGIpaf to let non-technical users create forwards, out-of-office mails, etc. and any combination thereof. This also has the advantage that we can filter out double bounces and spam (which also prevents us from being listed in spammer blacklists). Unfortunately autofs (seems independent if autofs4 or autofs5 is used) seems to be unreliable if there are bursts of mount or umount requests, resulting either in File or directory not found error message while trying to access the home directory of a user, or Directory not empty error messages if the automounter tries to remove the mount point after unmounting. In that case a not mounted directory owned by root is left over. In the end both cases lead to procmail behaving as if that user does not have a .procmailrc which looks like sporadically lost mails to those who forward all mails. (The mails then can be found in the local default INBOX for that user.) Additionally there are similar issues when the NFS servers are not available. The most effective countermeasure we found so far was adding tests to the global /etc/procmailrc to check if the user s home directory exists and belongs to the correct user:

# -----------------
# Global procmailrc
# -----------------
# For debugging, turn off if everything works well
VERBOSE=1
LOGFILE=/var/log/procmail.log
# This only works with bourne shells, $SHELL defaults to the user's
# login shell. And by experience dash seems not work, so we use bash.
OLDSHELL=$SHELL
SHELL=/bin/bash
# temporary failure (see EX_TEMPFAIL in /usr/include/sysexits.h) if
# $LOGNAME is not set for some reason. (Just to be sure our paths
# later on are not senseless.
:0
* ? test -z "$LOGNAME"
 
    LOG="Expected variable LOGNAME not set. "
    EXITCODE=75
    :0
    /dev/null
 
# temporary failure (see EX_TEMPFAIL in /usr/include/sysexits.h) if
# $HOME is not readable. ~$LOGNAME does not seem to work, so this uses
# a hard wired /home/.
:0
* ? test ! -r /home/$LOGNAME
 
    LOG="Home of user $LOGNAME not readable: /home/$LOGNAME "
    EXITCODE=75
    :0
    /dev/null
 
# temporary failure (see EX_TEMPFAIL in /usr/include/sysexits.h) if
# $HOME has wrong owner. ~$LOGNAME does not seem to work, so this uses
# a hard wired /home/.
:0
* ? test ! -O /home/$LOGNAME
 
    LOG="Home of user $LOGNAME has wrong owner: /home/$LOGNAME "
    EXITCODE=75
    :0
    /dev/null
 
[ ]

If you want to store a copy of these mails for debugging purposes on every delivery attempt, replace /dev/null with some Maildir or mbox only accessible for root. One small but important part was to explicitly declare bash as shell for executing the tests, otherwise mails for users with tcsh or zsh as login shell filled up the mail queue and never get delivered (if the SHELL variable never gets fixed). Only drawback so far: This leads to more lagging e-mail on e-mail bursts also for those users who have no .procmailrc because procmail can t check if there s really no .procmailrc. Extensive procmail documentation can be found online at the Procmail Documentation Project as well as in the man pages procmail(1), procmailrc(5) and procmailex(5).

Thanks to some advice from Philipp Kern I have now got my Galaxy S running CyanogenMod 7.1.0 which is based on Android 2.3.7 [1]. CyanogenMod has lots of configuration options that seem to be lacking in the stock releases and also supports some advanced features such as OpenVPN and a command-line. I can t properly compare CyanogenMod to the stock Android as I ve only used versions 2.1 and 2.2 of the stock Android. Presumably some of the things that I like about CyanogenMod are in the stock Android 2.3.7 release. The process of updating a phone is difficult and has some risk. Fortunately Samsung provided Download mode in the BIOS to allow recovery. If you mess up the process of updating a Galaxy S and you can get Download mode by holding down volume-down, home, and then power buttons then you can almost certainly recover (so don t panic). The CyanogenMod people don t provide any documentation on upgrading from Android 2.2 (which is what Optus is still shipping AFAIK). So you will probably have some difficulty when upgrading a Galaxy S that you get in Australia (it seems that Optus is the only company shipping them in volume). As an aside if you want to buy a Gel Case for a Galaxy S in Australia then visit an Optus store. It seems that Optus is the only phone store that hasn t run out their Galaxy S accessories in favor of the Galaxy S2. I have previously written about the Galaxy S and Three Networking [2]. Now that I have the Galaxy S as my primary phone on the Virgin network all my data corruption problems are solved, the problem is entirely related to Three. With CyanogenMod there is an option to be able to toggle the LED Flash as a torch from the drop-down menu, this makes the lack of such a LED on the Galaxy S even more of an annoyance. I have also discovered that the Galaxy S apparently doesn t have a status LED! This makes it the only phone that I ve ever owned that has no clear way of informing me when the battery is charged! It s also really useful to have a flashing LED to indicate low battery when running a full screen app, and to have a flashing LED to indicate that email has been received. Someone should design a phone with multiple LEDs to indicate different things. I d like to have one LED to indicate charging status and another to indicate whether there is unread email or SMS. Whatever the cost of including a LED during manufacture it would have to be almost nothing compared to the ~$500 sale price of a phone. Wikipedia says The Samsung Galaxy S features a PowerVR graphics processor, yielding 20 million triangles per second, making it the fastest graphics processing unit in any smartphone at the time of release. Also, upon release, the Galaxy S was both the first Android phone to be certified for DivX HD, and at 9.9 mm was the thinnest smartphone available . I don t care about any of that, I want a phone with decent battery life, a LED Flash , and a status LED. The main benefit I get from the Galaxy S over the Xperia X10 is the greater storage. The Xperia X10 has a total of 1G of storage and only 465M of that is available for application install. My Galaxy S has 16G of internal storage of which 1.8G is available for phone apps and 13G is available for pictures and other mass storage. Having 1.8G for phone apps and internal phone storage used by such apps (which includes the offline IMAP cache) is a massive benefit, enough to outweigh the lack of a staus LED and a Flash LED. What I Really Want I d rather have a Samsung Galaxy Note. The Note has a LED flash, a 5.3 screen with 1280*800 resolution which is much better for running as a SSH client and also good for web browsing. I m not inclined to spend money on a phone now, so I ll probably use the Galaxy S until Virgin offers me a new phone or someone just gives me a new phone (I can always hope). One of the many nice features in the Galaxy Note is a built in stylus. When using my current phones for web browsing I sometimes find it difficult to have a touch registered to the desired part of the screen, this is a real problem with the Opera web browser which requires a long press to open a URL in a new tab.

• [1] http://www.cyanogenmod.com/
• [2] http://etbe.coker.com.au/2011/11/21/galaxy-xperia-android-network/