My project. This post assumes that the reader has read my last post. I ve been trying to find a reasonable method (which computer to use, single user or multi user) to benchmark for some time, and now an e-mail from Simon Marlow calmed me down. He suggests that in this stage it s not necessary to run a long benchmark, but only a short one, that will be enought to have an idea about what each change is causing to performance. For some time I was struggling with the benchmarks to get a reasonable result. I ve started more than once to write a blog post analysing the results I ve got, until in the middle of the post I notice the results are not consistent. This usually happens when there s a big distortion in benchmarks that shouldn t present no change. For instance, in a patch that only changes code that is executed with +RTS -w -RTS, there was a difference of 50% on execution time in default mode, that is, without +RTS -w -RTS. I think that only disabling CPU scaling and running a small number of tests, which make it easier to reproduce a possibly inconsistent result, will be enough to get a good data. I ve made a stack of patches, splitting each part of my work. The stack does not reflect my preferences between the patches, that is, the complete stack applied is not what I think that is the best change to GHC. It s only a set of patches applied linearly, in a way that it s easy to see the impact in performance of each change. The patches are: Don t check for swept blocks in -DS.

The checkHeap function assumed the allocated part of the block contained only alive objects and slops. This was not true for blocks that are collected using mark sweep. The code in this patch skip the test for this kind of blocks.

This patch is only needed to make it possible to run the binary with the parameters +RTS -w -DS -RTS. I ve already explained this patch. I didn t measure the difference in performance before this patch, since it only changes code that is not going to be executed in the benchmarks, since none of them are runned with -DS. Immix: allocate and free memory on lines. This is the main patch with the initial implementation of Immix. As it is the initial implementation, it has it s problems, that will be treated in the following patches. The comparison was done using 3 programs from nofib/gc: fibheaps, fulsom and constraints, as suggested by Simon Marlow. Two tests were made for each version of the compiler code: one using the default GC strategie (copy-collection with mark-compact if there s few heap space left), another one using mark-sweep (or Immix after my changes). When comparing the default GC strategie between before and after this change, the program got 0.4% slower, with the collection of the generation 0 being 0.1% slower and the collection of the generation 1 0.4% slower. Comparing the mark-sweep with Immix, it got 13.7% slower, 15% in GC0 and 7.2% in GC1. The change in memory used is irrelevant. Chose between allocating in line or blocks in todo_block_full The first improvement I ve made is to change the place where the allocation was searching for free lines. Most of the changes were done in the function alloc_for_copy(), in rts/sm/Evac.c. This is bad because this function is called a lot of times, and should be kept fast. So I moved them to ctodo_block_full() in rts/sm/GCUtils.c. It got about 9.9% and 9.6% faster in default and sweep, which makes it 1% faster and 2.8% slower than in the original code. Improvements in sweep() I was bothered that the code I ve written to free memory in lines was messy, and I thought it could be simplified and maybe even turn faster. The general execution time increased 0.4%, but the GC1 time, which is the only one that should be affected, was reduced by 0.6%. In the default GC strategie, there was also a 0.2% increase in time, so I guess this is inside the margin of change. I have to investigate better on this one, but it seems to have improved slightly the performance. Line before inscresing block size. In a conversatio with Simon Marlow, we thought about these two options: one is to search for a line after trying to increase the block size, and that s what I did in Chose between allocating in line or blocks in todo_block_full; the other is trying to search a line first, and just increase the block size if a block (and not a line) was being used for allocation and there is no line available. With this patch I ve changed it to use the second one. Using sweep, the GC time got 0.5% slower, and with default it got 0.1%. rts/sm/Sweep.c: Mark all BF_MEDIUM blocks as BF_FRAGMENTED In my first implementation, the blocks that contain objects bigger than a line (of medium size) are marked with BF_MEDIUM, and are treated as the usual block in mark-sweep: if the block is empty, it s freed; if it s very fragmented, it s marked to be collected using copy-collection. In this patch, instead of making it use the rules of mark-sweep, I just mark it as fragmented and make it be collected by copy-collection. This made the GC0 time be reduced by 3%, but caused an increase of 509.7% in the GC1 time. This was also the first time the memory used was reduced, by 25.4%. The results in the default mode are insignificant. What it seems to me is that the copy-collection code is more efficient with memory usage, but also more slow. Marking objects as BF_FRAGMENTED make this algorithm be used even for full objects. rts/sm/Sweep.c: Don t use 3/4 heuristics to mark as BF_FRAGMENTED The mark-sweep algorithm considered that an object was fragmented if more than 1/4 of the word groups are completely unmarked. I kept using this heuristic, and this change removes it, making more use of the Immix, and less use of copy-collection. This didn t change the memory used, and made the code 10.4% slower in sweep. In default GC mode, there was no relevant difference, as expected. Allocate in lines in minor GCs. In the initial implementation I was only allocating on lines only in major GCs, because I needed the mark stack, which was only available in these kind of allocations. In this change, I created the mark stack on all GCs, and used the allocation on lines. The results for the default mode are insignificant, and the code got 9.0% faster in sweep, using the same memory. Selection I ve made a selection with the presumably better patches, which are: Don t check for swept blocks in -DS, Immix: allocate and free memory on lines, Chose between allocating in line or blocks in todo_block_full, Improvements in sweep() and Allocate in lines in minor GCs. This is an attempt to achieve the best set of patches possible, to see how it improved the original code. Comparing the default strategie for this selection and the original code, it got 3.7% slower. Comparing both sweep, it got 3.9% slower. Comparing the original default with the final sweep, it got 4.2% slower and uses 4.7% more memory. There s a lot of room for improvement, and I m willing to hear suggestions of what I could change in the code to achieve this improvement.

---

The results are available. Every time I had doubt about a comparison, I d run both versions again to check. This is way there are some backup files. The data presented here is not from all the most recent versions of each measurement, but from the ones that I thought were more similar in conditions.

What do you do when you have a technical question that you're embarrassed to ask? The first Sunday of Debconf, I led a birds of a feather (BoF) session called Debian for Shy People. The conference team scheduled it on "Debian Day," a pre-conference day that was open to the public and still had plenty of Debian Developers in attendance. I just uploaded the slides to the "Penta" page for the talk. I led it because of my own experience. In 2004 or so, I saw Debian as the cool kids' club, that awesome project that I wished I could be a part of. By 2006, I managed to get over myself, read the New Maintainer's Guide, and find a way to get involved. As of mid 2009, I am a full-blown Debian Developer. I have real ultimate power. But I sometimes do still feel hesitation akin to "Imposter Syndrome". (A bunch of people at Debconf didn't really believe I'm "shy," since I asked a lot of questions at the conference. At core, I don't naturally believe that the things I say are worth hearing, but I patch over this hesitation. Sometimes I speak too much, and then I feel ashamed of burdening everyone. But anyway, this is about Debconf not, me -- so moving on....) In the past year of being a Developer, one thing I've seen is that other contributors ask me privately for help. Rather than blast the public lists like debian-mentors, they email or IRC private-message me, or SMS me, or find me at a Linux Users Group event. I'm lucky to know these people, and they're lucky to have me as a safe person to ask questions of. Moreover, Debian is better because these people could move past their confusion to make a technical contribution. I began the BoF session by talking about when someone asked me for help. Then I asked, "How many of you have someone you can ask embarrassing questions of?" Of the forty people crammed into Schapiro 414, two people' raised their hands. One person put it plainly, "I don't know anyone else who does Debian." It reminded me of a fact that Karen discovered when she was doing market research for us at OpenHatch: the vast majority of free software programmers know zero other people who do free software. I had seen the figure; we even used it in a talk to try and convince venture capitalists to fund OpenHatch last year. But I didn't really feel it until I heard it from a room full of Debian contributors. I structured the BoF in two parts: First, I talked in front of some slides to set the tone properly, and then we enjoyed open discussion. As I was preparing thoes slides, Daniel Morais asked me, "What's the point of having the session? Why not just come up with some ideas, implement them, and not bother also talking about it at the conference?" I had considered this; I decided I wasn't self-confident enough to start implementing ideas without talking to people to make sure I wasn't the only one who saw a problem. But I discovered another benefit of giving the talk: people who want to make Debian more welcoming knew to reach out to me. So here are some thoughts that came from our discussion (and later discussions during the conference):

• We need better mentorship in Debian. Truly, mentorship is a personal relationship. Perhaps matching people up into one-on-one groups would be a good idea. KiBi said that doing so using OpenHatch (if we add mentorship-oriented features!) would be a reasonable way to do it.
• I've heard from quite a few people who would be willing to have more one-on-one relationships with mentees.
• Everyone (except Manoj) knows that Debian contributors are not born experts. It takes a process of learning and teaching to bring them up to speed.

I set up an Etherpad document on cjb's OpenEtherpad.org. This is what we learned together:

• "If your level of understanding might be low, but you want development help, then you might not know if #debian-devel is only for people who are already super involved and know everything and therefore you should feel embarrassed."
• People get anxious over their ability to speak English.
• Bikeshedding over patches can be a major detractor.
• "Asheesh talks too fast."

One idea I had before the BoF was to create a discussion area that was safe for all questions, even if they seem silly. We talked for a while about what name that would take, if it were to become a new IRC channel. We reached something of a conclusion, but in the conference that followed Emmet Hickory offered to help make the debian-mentors IRC channel friendlyer. I think that's the best direction to take things, so the next step is for him and me to write up what we want and send a note to the debian-mentors email list explaining our vision. In the Etherpad document, people discussed the idea of doing Debian discussion over XMPP (also known as Google Talk, also known as Jabber). We weren't sure how such a place would get critical mass; someone briefly mentioned the idea of an IRC/XMPP gateway. I actually think this discussion is along a very reeasonable path, namely discovering what discussion method(s) Debian contributors want to use. (That might explain why I'm now an admin on forums.debian.net.) We also briefly discussed the idea of an anonymous question-answering service. I realize now that I'm not going to be able to have time to run that, but I still think it'd be a really cool idea. Biella would remind me that Debian is already successful at bringing in new contributors. I agree! As a free software project, we have an enormous number of participants. This is a really good thing, and we're clearly doing something right. The purpose of this talk was to figure out how to make contributing to Debian less stressful for those who participate. Truly, a "Debian for Shy People" effort isn't about shy people. It's about the moments of self-doubt we all have in which we don't know what to do and are too embarrassed to ask. I think that if the project more friendly, we can find more participants, make better use of our current ones, and see improvements to our diversity. Whew, that was long. What do you think of all this?

So, DebConf time is over once again. The two weeks worth of fifty weeks waiting are left behind once again, and it's back to get back to normal. DebConf was great Yes, it always is, and that's what we are all saying, but hey - Seriously! Being in the same building than 300 crazed developers is always fun, and it's always better than last year's fun. A good highlight this year is that, given the number of Free Software and Free Culture groups that exist in USA's north-eastern coast, we had the opportunity to join a large crowd which has never been part of DebConf. Also, I must agree that the USA bid for DebConf was aiming to attract as many Debian people (developers, maintainers, or just happy users) which had not yet been to a DebConf before as possible. And it was a great success! I finally met several people I have long read in the mailing lists, in blogs or in IRC. A much higher proportion than usual, I'd venture to say. Another interesting phenomenon /methinks is that this year's DebCamp started much more staffed than usual: I arrived on the first day, Sunday 25, and there were ~40 people there already; I don't have the actual numbers, but we quickly grew, and the number started to stabilize past mid-week, only to (sharply) rise in the weekend, in time for DebianDay and DebConf start. Great time! But, they say, nobody can go to the USA without buying some sweet toys, right? Well, being the proud owner of six very hairy cats, I have thought into entering the looming and weaving industry... But cat hair, while abundant, I have heard is untreadable... Maybe due to the indisciplined, natural and independent personality of the cats (catonality should I say?)... So I had two choices: Clean up my home quite often, or live in a literally hairy mess. Enter choice #3: The Roomba! I had been waiting to buy this thing for several years, as they refuse to send to Mexico or charge Mexican cards. So, I walked across Manhattan and got my very own robot cleaner! For my further surprise, although I have not yet tried it (I don't even have a suitable cable yet), I found this: Yay, the Roomba is actually hackable (via a 7 pin miniDIN serial port)! Wikipedia says that:

Roomba comes with a Mini-DIN TTL serial interface, which is incompatible with standard PC/Mac serial ports and cables, both electrically and physically. However, third-party adapters are available to access the Roomba's computer via Bluetooth, USB, or RS-232 (PC/Mac serial). New, 500-series, and 410/420 series Roombas upgraded with the OSMO hacker device allow the user to monitor Roomba's many sensors and modify its behavior. The Roomba Open Interface (formerly "Roomba Serial Command Interface") API allows programmers and roboticists to create their own enhancements to Roomba. ( )

My first impressions? Well, the Roomba lazily charged its battery throughout the day today, and was hungry and ready when I arrived home. It is a but louder than what I expected, and of course my cats were not thrilled by the presence of a eighth animated and apparently sentient being at home. Their initial reaction was of course to be verrry alert of the thing. Twelve eyes were constantly pointing at the Roomba while mine alternated between them. As they measured the thing's speed and (I guess) inferred its movement patterns, they started escaping upstairs A flat, round thing with no legs to be seen will not likely be able to climb the stairs. And they were completely right. At first, only Chupchic remained downstairs. After a bit, I went up to show them the Roomba didn't jump on us to eat our brains, and after a bit, Santa and Macusa joined. The Roomba roombed for maybe 90 minutes (this space is large, and decided it was enough... And slowly, the rest of them started coming down. I would not say Roomba's cleaning is perfect, of course. Its room discovery algorithm is funny, and it even seems it's based on the mere chance of covering most (never all) of the space it has to clean. I had, of course, not fully studied it (after a single run, how could I?). It does make a honestly good attempt at cleaning under coaches, chairs and tables. It collected a fair amount of dust (on a house that seemed quite clean to me, I cannot imagine what would happen on a messy one). I have not yet played with the virtual walls (infrared transmitters which limit rooms as if a door was closed), but given the size of this house (and that I don't want it to clean around the cats' designated bathroom area), I guess I will end up using them regularly. During DebConf, I heard one bad (stupid useless noisy thing) and two very good (it has radically changed my life) comments on the Roomba. I hope to shift the balance towards 3/4 and not towards 2/2! Anyways... Thanks to each and every one of you. DebConf is great. Always great. Always a success. I cannot even thank specific teams. Debian Rules, and DebConf Rocks!

You might remember that I started to work on closing RC bugs for stable. This effort hasn't died off so I joined the RC Bug Squashing Contest that was going on during the Debian conference (which I was unable to attend). The rules did permit it, even though I was aware that it can't (and shouldn't) be compared to the unstable/testing squashers and thus it was split of into its own Special category. There still seems to be a lot of confusion on how our BTS works, especially with respect to version tracking. I was even accused of falsely claiming bug closing because the bug has been closed by someone else already. This wasn't the case, otherwise the bugs would had been archived a long time ago already^[1] given that I had expected that person to know how bug archival works and also given the sheer amount of bugs that I simply had closed for stable, I guess it is needed to shine a light upon why some bugs won't get archived as expected. If you followed my IRC talk on BTS usage last month you might already have an idea of what's going on. There might be several different reasons why bugs aren't marked for archival yet. I'll try to explain them as I understand it (given that I neither have taken a look at the debbugs code nor am involved with its development) through my working on bugs over the last 10 years.

• Usually bugs are marked for archival when they are not affecting testing or unstable. That means, for a bug filed against a version that is only in unstable and closed in unstable, an immediate archival process is started (actually, it's not that immediate: It requires the package to be in sync on all architectures it's available for, too); otherwise the fix has to enter testing before bugs are considered for archival.
• Sometimes the bug is still considered affecting unstable though, even when the fixing package version already moved over to testing. This is the case when the hurd-i386 package is outdated in unstable and can be checked in the unstable overview of the package on packages.debian.org at the end of the page, it will have red entries next to the green ones (you can ignore red entries for debports architectures, though!). You have to file an arch specific removal request for those bugs to get them archived.
• For packages in experimental it's quite the similar situation, because that packages from experimental don't transition anywhere, so those will stay unarchived for the time being if they also affect unstable.
• So what can be the reason when the fix is already available in testing and unstable but the bug still doesn't get archived? Those cases mean that the bug is considered as affecting stable and has to get addressed there, too. Bugs that are considered for that are of a release-critical severity, bugs with lower severity aren't considered for stable because they are unlikely to get fixed in stable and thus get archived. So what can we do about them?
  • Do they actually really affect stable? Working on those is what I did over the last months. There are often enough bugs filed for library transition issues, FTBFS for toolchain changes, or other things that aren't applicable for stable. Those are easy to close in stable:
    • If the version information got lost (through reassigning or similar), the only thing needed is to re-add the found versions, given that it is a higher one than the one in stable.
    • If the version information is proper (because it had still the same version in unstable at the time of the reporting) tagging them with + squeeze sid will tell the BTS to consider them only for those releases, so not thinking it affects lenny, too.
  • Some do affect stable but still aren't severe enough to warrant an update in stable (like for documentation corrections in debian/copyright or similar). If you found one of those please talk to the release-team about getting them tagged lenny-ignore, don't do that yourself because that tag is meant to be set by the release-team themself only!
  • If they actually affect stable (like security issues that though are considered to be too minor by the security team to warrant a DSA, or other severe usability issues), please try to backport the relevant fix, propose a diff to the release-team on their mailinglist and have it uploaded to lenny-proposed-updates to be fixed in the next point release.

I hope this will get others also interested to fix stuff for stable. Actually when I see something that potential falls into the last group I ping the maintainers of those packages to let them know about having them fixed. Some of you might already have received such a ping and I am thankful for those that received them well and actually already fixed some of those, too. Thanks for making stable a better place! Coming back to the RCBC, it was an interesting small competition (if only in my brain) between myself and the testing/unstable RC squashers going on. It would had been nice to get at least as many bugs in stable addressed as in testing/unstable because the amount is still a lot higher, but I can still be happy about the things done. And it is good to know that not everything seems to think that it's wasted effort to work on getting the RC count lower for stable, too (like I also was told about my effort). So thanks from that point of view!

---

^[1] There are currently 2688 bugs still unarchived that were closed last year already! This UDD query helps you:
select count(*) from bugs where status = 'done' and last_modified <= '2009-12-31';

/debian permanent link Comments: 0

So, I have finally visited my first Debconf, and it was a phenomenal experience. To meet persons who have guided you, persons work with, and persons you respect face-to-face and have conversations with them about several topics ranging from their vision for Debian to their views on topics totally unrelated to software is something truly great. I am now convinced that events like this, which provide the right environment to socialize and discuss various issues concerning Debian, will go a long way into getting several aspects of the project moving along in the right directions, as well as create awareness to the outside world about what the project is about and how it benefits users.In terms of discussions, the Debian Science track was a nice way to see how contributors are putting Debian to good use for scientific research, and it was a nice opportunity to list out the things to be done to fill in the gaps and take Debian as a science platform forward. The Debian Python BoF was also a good place to meet many contributors face-to-face, and learn more about the technical concerns which people have in regard to the future of Python in Debian.While I didn't hack much during the conference, Christian's presence made me finish the Hindi translation of the Debian Installer. We can now shift focus to other aspects, such as getting packages in order for a short freeze-to-release cycle.Columbia University provides for a very nice venue, with easy access while maintaining a quaint environment within itself. Since this is my second visit to New York City, I had an idea of the common tourist locations, so I did not go around much during the conference. However, since I have a buffer day, I might try my hand at some photography around the place.Have a safe trip home, and a wonderful time ahead! I leave you with a photo I took when I went with a friend across the Brooklyn Bridge after Debconf.

My rule to see if a framework is worth of attention I came up with a little rule:

In order to be worth of any attention, a framework must be stable enough that I can charge money to train people to use it.

This probably applies to other kinds of software stacks, libraries, development environments and, well, to most software applications. In the context of python web frameworks, this means that:

• If it changes API all the time it is not worth of attention, because my customers won't get value for their money, as they'd continuously need retraining and rewriting their software.
• If I see lots of DeprecationWarnings it is not worth of attention, because my customers will see them and blame me for teaching them deprecated stuff.
• If fixes for bugs affecting the stable version are only distributed "in a recent git" or "in the next development version", and they are not backported into a new bugfix-only stable release, then it is not worth of attention, because:
  • my customers' business is to develop their own products based on the framework.
  • My customers' business is not to be maintaning in-house stable updates of the framework. Although if the framework's community is nice enough they might end up giving a hand.
• If it requires virtualenv or can only be obtained through easy_install it is not worth of attention, because:
  • my customers are not interested in maintaning custom deployment environments over time.
  • My customers are not interested in tracking each and every single library's upstream development to keep their production system free of bugs.
  • My customers are used to getting software through a proper distribution which also takes care of security updates.
  • I am paid to teach them how to use a framework, not a custom python-only package management system.
  • In my experience, if distributions have trouble keeping packages up to date, upstream is doing something fundamentally wrong.

In light of this rule, I regret to notice that I see very few python web frameworks worth of any attention.

When I created my on-line photo album, we had the year 2001 and there was nothing wrong with publishing ones photos in the Internet. It has been almost ten years now, and things have changed. Privacy is an issue, especially when it also affects your friends and family. The problem
So I needed to find some protection. These were my requirements:

1. I only want to protect parts of the site, as there are some pictures that I intentionally share with the public. This selection should be possible down to the individual image.
2. The solution should work without any dynamic server component besides the web server itself. This rules out any self-written CGI scripts as well.
3. One password, given to all my friends etc., is not sufficient as it might leak to unintended audience.
4. I do not want my visitors to have to register and remember yet another username and password just for my site. I also do not want to manage this user database.
5. I do not want to have do large changes to the file structure of my photo album.

This rules out most common options, e.g. protection by a .htpasswd file. Requirements 3 and 4 point to a solution based on OpenID. With OpenID, my visitors can authenticate against a service they already use (Google, Yahoo, etc.), relieving me from the burden of maintaining a user database and them from having to remember a password. There is a mod_auth_openid module for the Apache webserver, and it is even distributed with Debian in the libapache2-mod-auth-openid package. So requirement 2 is fulfilled. The tricky part is: How do we achieve OpenID protection for some images, and not for others. The solution I first played around with selectively enabling or disabling mod_auth_openid based on <FileMatch> directives in the Apache configuration, but it was not elegant and would not scale well. I have more than 20.000 pictures to manage, and have already selected over 5000 pictures to be shown without protection. My solution is based on a partial copy of the whole directory tree that contains all public files. To save disk space, these are just symbolic links to the real file in the protected location. Some mod_rewrite magic then takes care of giving the user the impression that all files are in the same location. I set up a small example of my solution, which has this directory structure:

.:
drwxr-xr-x 2 root root 4096  2. Aug 12:03 images
lrwxrwxrwx 1 root root   18  1. Aug 12:05 index.html -> private/index.html
lrwxrwxrwx 1 root root   18  1. Aug 12:05 login.html -> private/login.html
drwxr-xr-x 3 root root 4096  2. Aug 12:03 private
./images:
lrwxrwxrwx 1 root root 33  2. Aug 12:00 pleaselogin.png -> ../private/images/pleaselogin.png
lrwxrwxrwx 1 root root 28  2. Aug 12:03 public.png -> ../private/images/public.png
./private:
drwxr-xr-x 2 root root 4096  2. Aug 12:00 images
-rw-r--r-- 1 root root  267  2. Aug 12:03 index.html
-rw-r--r-- 1 root root   94  2. Aug 12:01 loggedin.html
-rw-r--r-- 1 root root 2091  2. Aug 12:03 login.html
-rw-r--r-- 1 root root   10 18. Nov 2009  protected.html
./private/images:
-rw-r--r-- 1 root root 4074  2. Aug 11:58 pleaselogin.png
-rw-r--r-- 1 root root 2670  2. Aug 11:58 private.png
-rw-r--r-- 1 root root 2043  2. Aug 11:58 public.png

As you can see, real files only reside in private/, outside of that, only symbolic links exist. The apache configuration protects the private directory and blends it into the main directory:

   <directory /var/www/nomeata.de/openid-test>
        RewriteEngine On
        # Abuse the login page as an error image
        RewriteCond % QUERY_STRING  \.(png jpg)
        RewriteRule ^login.html$ /openid-test/images/pleaselogin.png
        # Ship private files, if they exist, unless public files exist
        RewriteCond  $1 !^private
        RewriteCond  /var/www/nomeata.de/openid-test/$1 !-f
        RewriteCond  /var/www/nomeata.de/openid-test/private/$1 -f
        RewriteRule  ^(.+)$ /openid-test/private/$1
   </directory>
   <directory /var/www/nomeata.de/openid-test/private>
        AuthOpenIDEnabled        On
        AuthOpenIDDBLocation     /var/lib/apache2/mod_auth_openid/mod_auth_openid.db
        AuthOpenIDLoginPage      /openid-test/login.html
        AuthOpenIDTrustRoot      http://nomeata.de
        AuthOpenIDCookiePath     /
        AuthOpenIDCookieLifespan 2592000
    </directory>

A special trick handles the login page for protected images: If the login page is requested and the referrer indicates that the user tried to access a .png or .jpg file, apache will instead ship an image containing an error message. For my photo album I have a small Perl script that, given a directory with a private/ directory therein and a list of rules in form of glob patterns, will symlink matching files and remove symlinks that are not allowed any more.
What s next? As you can see, this does not actually protect the content. It only requires the user to authenticate, then everything is visible. To select which OpenIDs are allowed to access which code, some bugs will have to be fixed in mod_auth_openid first. There was little activity there recently, I hope that the project is not dead.

The Flattr FOSS project is quite young so very few free software are currently listed in the dedicated directories. It s still advocacy time: if there are free software that you d like to flattr, please tell them and point them to Flattr FOSS. It s too early for any conclusion, but the feedback from users is good and it looks like that people using flattr to support free software are more generous than the average flattr user. You re great people! Anyway, it s time for me to give a few suggestions of projects to flattr.

1. My first recommendation is Project Hamster, it s a time-tracking application for GNOME but it has come a long way thanks to the work of Toms Bau is. I use everyday and it rocks.
2. My second recommendation is verallt, it s a Firefox extension that allows usage of Flattr even in places where it was not designed to be used: forums, mailing list archives, etc. The persons using Flattr just put [Flattr=20693] in their signature and the Firefox extension automatically replaces this text with a real Flattr button ready to be clicked. Thanks to Paul Da Silva for the nice idea and the first implementation!
3. My third recommendation is Mustard, it s a StatusNet client for the Android platform. More concretely you can install it on your Android phone to send updates to identi.ca while traveling. I have no Android phone but Mustard seems to be popular among owners of such phones!
4. My fourth recommendation is moreutils from Joey Hess. It s set of unix (command line) tools that nobody else thought of writing when unix was young (according to Joey himself). They are all very useful, I use more or less regularly isutf8 (verify if a file is UTF-8 compliant), combine (logical operations on lines of two files), pee (send standard input to two programs, very useful in git hooks when you want to run two programs that both consume the same data), sponge (ever tried grep -v foo file >file and lost your data? now you can use grep -v foo file sponge file ). And Joey is a long time Debian developer who has truly shaped Debian, he s behind several major Debian projects (debconf, debian-installer, debhelper, ). Big kudos to him!
5. My last recommendation is to flattr Andreas Gohr for his work as dokuwiki lead developer. There s no flattr button on dokuwiki s website but you ll find some in Andreas blog, just pick a dokuwiki related article like this one: Dokuwiki Birthday Bughunting Weekend. Dokuwiki is a simple wiki that works without any database (the pages are stored in flat files) and it s even commonly used to setup simple websites thanks to the numerous themes available.

Okay, that s it for this month. I ll try to keep a broad panel of software (desktop, command line, mobile, web, etc.). Since I have promoted a GNOME desktop application, I would like to be able to promote a KDE or XFCE application next month but I did not discover any KDE/XFCE project using Flattr do you know any? Please share in the comments if you find some. Also I wonder if you have decided of a flattr strategy Personally I m likely to flattr my favorites projects/persons each month and to flattr a few other occasionally as I discover them. What s your own strategy? 4 comments Support my work

Here at DebConf10 the schedule is starting to get really packed of events, which is great! About a half of those events are BoFs (in the IETF sense), i.e. discussion sessions among people interested in specific topics. (Other conferences use "BoF" to refer to impromptu, non-scheduled, and spontaneous events, while at DebConf we have traditionally submitted, reviewed, and scheduled BoFs in advance.) As most DebConf regulars, I've attended quite some BoFs in the past and I share with others the impression that the quality and usefulness for the project of BoFs varies substantially from BoF to BoF. That's why I'd like to share a BoF recipe, which comes from possibly the best organized BoF I've ever attended. As it happens with all good recipes (and with free software), I've took the liberty of customizing it a bit.

---

DebConf BoF HOWTO have a coordinator A good BoF should have a coordinator, which is usually (but not necessarily) the person who proposed the BoF in the first place. The role of the coordinator encompasses at least 2 tasks: preparing the BoF in advance (read below) and moderating the discussion to ensure everybody get a chance to participate. prepare in advance A good BoF is more than just getting together, vaguely knowing the BoF topic, and ... discuss. Mind you, it might work very well that way too, but the chances of having a successful BoF are much higher if attendees arrive prepared. That is quite simple to achieve. It just takes the BoF coordinator to prepare some working questions and/or detailed discussion topics, and advertise them as such. That way potential participants can make up their minds, recall experiences that they want to share, note down comments they want to bring into the discussion, etc. In my experience, the BoF time will then be used in a much more productive, exciting, and ultimately fun way. Most BoFs can also benefit from a few introductory slides on the BoF topic, to ensure all participants start from a common ground. The last slide could contain a brief recap of the working questions, so that the audience have them handy. don't be exclusive Debian is not a company and Debian contributors are not employees, therefore we cannot expect all relevant people to attend a specific BoF, just if they had to. As usual, the risk of cutting off community members who cannot attend should be mitigated as much as possible. To that end, minutes are just great; after the BoF, minutes should be posted where appropriate, and possibly uploaded to Penta as event attachments. ... but taking minutes is just boring and not everyone is good at that (for instance, yours truly used to suck at taking minutes, even if I'm trying to get better ...). If you have a trusted participant which is good at taking minutes, great, go for it! If you don't, use collaboration. During past DebConfs as well as at UDS this year, I've been shown a nice way of doing that. It takes a dedicated projector in the BoF room showing some real-time, collaborative editing facility (e.g. gobby). With such a setup, people usually just start taking minutes collaboratively, although explicitly inviting the audience to do that is a very good idea. Ideally, the room should have two projectors, one for supporting slides, another for collaborative minutes taking. In case there is only one, as it is usually the case for DebConf, it's probably better to use it for minutes and switch to them as soon as supporting slides are over.

---

gobby.debian.net for BoF minutes at DebConf10 For DebConf10, the DebConf team has setup a gobby (infinote) server at gobby.debian.net. Just do the following in order to be ready for collaborative minutes taking:

    # apt-get install gobby-0.5
    $ gobby-0.5 -c gobby.debian.net

(Note: you shouldn't use the non versioned gobby package as it uses a different protocol and doesn't support undo, which is quite risky; see Debian bug #590753 for more background.) As the naming convention for DebConf10 minutes, using document names that start with dc10- sounds sensible.

---

comments welcome! So, what do you think of the above suggestions? Do you have further best practices to share on BoF organization at DebConf? Leave a comment or mail me. In the end it would be nice to author a proper "DebConf BoF HOWTO" document for further reference. Thanks to Gregor for his feedback on early versions of this post.

This spring I read a course on Python in Vilnius University, Department of Mathematics and informatics. The course was not mandatory, which I suspect is one of the reasons why my students were quite a bright and motivated bunch. I began the course with a brief introduction to the Python language and the standard library, then dedicated a few lectures to Django and in the remainder I covered various libraries (GUI toolkits, Pygame) and techniques (testing, debugging, optimization). One of the major tasks of the course (and the deciding factor in the final valuation) was to develop a project using Python. The students were given complete freedom over the topic of the project and the tools used, as long as Python was involved. The quality of the result varied, of course; had I given concrete tasks, the low bar would definitely have been higher. However, I am happy with how things turned out. Probably the best-executed project was 1000 Online, an online multiplayer card game (see rules). It was developed by Vytautas Karpavi ius. Vytautas apparently had prior experience with Python and Django, nevertheless, the quality of the app (and the code behind it) is impressive. Among the other projects, Gegute.lt, a Lithuanian clone of Reddit is still up and running. It's nothing too fancy, but looks quite clean for a university project. Other projects perhaps were not perfectly executed, but were very interesting conceptually. Jonas Keturka developed a bot (BitBucket) for the online strategy game Travian. Marius Damarackas developed a web-based guess-the-song game. He used YouTube for the song material and Last.fm (IIRC) to make you guess among similar songs, so that it would not be too easy. Andrius Chamentauskas developed a Puyo Pop clone (GitHub) with PyGame, complete with multiplayer and an AI. Paulius Budzinskas worked towards his course paper by developing a bacteria movement modelling and visualisation app (GitHub; Lithuanian only). David Abdurachmanov developed a spell-checker bot for WikiAnswers.com, with a really baroque and sophisticated architecture. I believe he has applied to Google Summer of Code with that project; I wonder how that is working out. For reference, you can find links to all student repositories on GitHub/BitBucket in this list. Beware: some of them may be documented/commented in Lithuanian. Another interesting effect of the course was that people started using Python in other related courses, for example in Mathematical Modelling and Numeric Methods. I even heard that the lecturer was impressed and expressed interest in learning Python. Another lecturer, this time a statistician, was interested in teaching and using SAGE at lab practice. Feeling such 'ripples' going through the department sure felt nice. By the way, the university also offers lectures on Ruby, read by Saulius Grigaitis. I took some inspiration from him for the course form and content.

Why use a Chroot environment? A large part of the use of chroot environments is for the purpose of security, it used to be the only way of isolating a user from a section of the files on a server. In many of the cases where a chroot used to be used for security it is now common practice to use a virtual server. Also another thing to note is that SE Linux provides greater access restrictions to most daemons than a chroot environment would so in many case using SE Linux with a sensible policy is a better option than using a chroot environment to restrict a daemon. So it seems to me that the security benefits that can be obtained by using a chroot environment have been dramatically decreased over the last 5+ years. One significant benefit of a chroot environment is that of running multiple different versions of software on one system. If for example you have several daemons that won t run correctly on the same distribution and if you don t want to have separate virtual machines (either because you don t run a virtualisation technology or because the resources/expense of having multiple virtual servers is unacceptable) then running multiple chroot environments is a reasonable option. The Simplest Solution The simplest case is when all the chroot environments are equally trusted, that means among many other things that they all have the latest security patches applied. Then you can run them all with the same labels, so every file in the chroot environment will have the same label as it s counterpart in the real root this will mean that for example a user from the real root could run /chroot/bin/passwd and possibly get results you don t desire. But it s generally regarded that the correct thing to do is to have a chroot environment on a filesystem that s mounted nosuid which will deal with most instances of such problems. One thing to note however is that the nosuid mount option also prevents SE Linux domain transitions, so it s not such a good option when you use SE Linux as domain transitions are often used to reduce the privileges assigned to the process. There are two programs for labeling files in SE Linux, restorecon is the most commonly used one but there is also setfiles which although being the same executable (restorecon is a symlink to setfiles) has some different command-line options. The following command on a default configuration of a Debian/Lenny system will label a chroot environment under /chroot with the same labels as the main environment: setfiles -r /chroot /etc/selinux/default/contexts/files/file_contexts /chroot I am considering adding an option to support chroot environments to restorecon, if I do that then I will probably back-port it to Lenny, but that won t happen for a while. For a simple chroot once the filesystem is labelled it s ready to go, then you can start daemons in the chroot environment in the usual way. Less trusted Chroot environments A reasonably common case is where the chroot environment is not as trusted. One example is when you run an image of an old server in a chroot environment. A good way of dealing with this is to selectively label parts of the filesystem as required. The following shell code instructs semanage to add file contexts entries for a chroot environment that is used for the purpose of running Apache. Note that I have given specific labels to device nodes null and urandom and the socket file log in the /dev directory of the chroot environment (these are the only things that are really required under /dev), and I have also put in a rule to specify that no other files or devices under /dev should be labelled. If /dev is bind mounted to /chroot/dev then it s important to not relabel all the devices to avoid messing up the real root environment and it s impractical to put in a specific rule for every possible device node. Note that the following is for a RHEL4 chroot environment, other distributions will vary a little some of the file names. semanage -i << END
fcontext -a -t root_t -f -d /chroot
fcontext -a -t bin_t /chroot/bin.*
fcontext -a -t usr_t /chroot/usr.*
fcontext -a -t usr_t /chroot/opt.*
fcontext -a -f -d /chroot/dev
fcontext -a -f -s -t devlog_t /chroot/dev/log
fcontext -a -f -c -t null_device_t /chroot/dev/null
fcontext -a -f -c -t urandom_device_t /chroot/dev/urandom
fcontext -a -t "<<none>>" "/chroot/dev/.*"
fcontext -a -t "<<none>>" "/chroot/proc.*"
fcontext -a -t lib_t /chroot/lib.*
fcontext -a -t lib_t /chroot/usr/lib.*
fcontext -a -t bin_t /chroot/usr/bin.*
fcontext -a -t httpd_exec_t -d /chroot/usr/bin/httpd
fcontext -a -t var_t /chroot/var.*
fcontext -a -t var_lib_t /chroot/var/lib.*
fcontext -a -t httpd_var_lib_t /chroot/var/lib/php.*
fcontext -a -t var_log_t /chroot/var/log.*
fcontext -a -t var_log_t -f /chroot/var/log/horde.log.*
fcontext -a -t httpd_log_t /chroot/var/log/httpd.*
fcontext -a -t var_run_t /chroot/var/run.*
fcontext -a -t httpd_var_run_t -f /chroot/var/run/httpd.pid
fcontext -a -t httpd_sys_content_t /chroot/var/www.*
END
You could create a shell script to run the above commands multiple times for multiple separate Apache chroot environments. If there is a need to isolate the various Apache instances from each other (as opposed to just protecting the rest of the system from a rogue Apache process) then you could start each copy of Apache with a different MCS sensitivity label which will provide adequate isolation for most purposes as long as no sensitivity label dominates the low level of any of the others. If you do that then the semanage commands require the -r option to specify the range. You could have one chroot environment under /chroot-0 with the sensitivity label of s0:c0 for it s files and another under /chroot-1 with the sensitivity label of s0:c1 for it s files. To start one environment you would use a command such as the following: runcon -l s0:c0 setsid chroot /chroot-0 /usr/sbin/httpd

Recipe for RCBC - Release Critical Bug squashing Contest: It has been a while since my last post in the RCBW tradition. Luckily, the tradition is in very good health of its own, thanks to many others that have picked up the habit of SPAM-ing planet with good news for Debian, encouraging others to do the same. With all that RC-obsessed people around and with the Squeeze release forthcoming, can DebConf10 be devoid of geeky RC-squashing activities? Of course not! Thanks to the orga team we expect the conference to contain a 2-week long RC bug squashing pride with tutorials, BoFs, a permanent bug squashing party, ... and a contest! I won't indulge much on the contest as the wiki page contains all rules and gory details. Obviously, all usual rules and best practices of bug squashing parties will apply; coordination will happen on #debian-bugs. Get involved, it starts today, and it's open to everyone (DebConf10 attendees as well as Debian enthusiasts abroad, regular RC squashers as well as casual bystanders, etc). All this wouldn't have been possible without the help of many people that love Debian, so many thanks to:

• the sponsors of RCBC:
  • Marvell, for offering a GuruPlug
    (of course, the winner of this device is expected to help with Debian support for it *g*)
  • HP, for offering t-shirts
  • Pearson, for offering books
• micah - who dragged me into this (it hasn't been hard)
• gregoa and mehdi - the almighty judges - who have been transitively dragged into this (it hasn't been hard either)

PS a corresponding announcement is in the debconf-announce pipeline already

This past week, the annual R user conference useR! 2010 took place at the National Institute of Standards and Technology (NIST) in Gaithersburg, MD (which is a tad northwest of Washington, DC). Kate Mullen and her team of local organizers did a truly tremendous job in putting together a very smooth conference attended by almost 500 people. It is always nice to meet so many other R contributors and users in person. And needless to say it's also just plain fun to hang out with these folks. As at the preceding useR! 2008 in Dortmund and useR! 2009 in Rennes, I presented a three-hour tutorial on high-performance computing with R. This covers scripting/automation, profiling, vectorisation, interfacing compiled code, parallel computing and large-memory approaches. The slides, as well as a condensed 2-up version, are now on my presentations page. On Wednesday, Romain and I had a chance to talk about recent work on Rcpp, our R and C++ integration. Thursday, we followed up with a presentation on RProtoBuf -- a project integrating Google's Protocol Buffers with R which much to our delight already seems to be in use at Google itself! It was quite fun to do these two talks jointly with Romain. But my other coauthor Khanh had to be at a conference related to his actual PhD work. So on Friday it was just me to give a presentation about RQuantLib which brings QuantLib to R. Slides from all these talks have now been added to my presentations page. I will also upload them via the conference form so that they can be part of the conference's collection of presentations which should be forthcoming.

Remember the one rc bug a days activities from Steinar and Zack? Well, maybe I won't make to one rc bug per day, but at least I could blog about some of my recent activities. Maybe it motivates some others? #553248 libnel-dev: missing-dependency-on-libc needed...
Trivial bug solved by adapting the depends-lines; uploaded.
#589344 missing symbols in library
Missing build depends; sponsored NMU by Davi Leal to delayed.
#582309 Links against libclamav
"solved" by disabling clamav support; uploaded to delayed 15.
#589819 Unmet Recommends on extremetuxracer-racer
Fixed typo in recommends (introduced by yours truly); uploaded.
#565805 FTBFS on kfreebsd-*: need to define type GLUTesselatorFunction
Applied patch by Felix Geyer; (doesn't help the release, package is experimental).
#568990 planetpenguin-racer-extras: Outdated; needs restructured and updated to be usefull
Upload fixed package prepared by Bertrand Marc.
#548084 aegir-provision: /etc/aegir/drushrc.php PHP error,
#548085 aegir-provision: /etc/aegir and /etc/aegir/drushrc.php file permissions
and unreported piuparts problem Uploaded to delayed.
#569177 kernel-patch-nfs-ngroups: doesn't apply anymore
Added an updated version of the patch; uploaded.
#580120 mediatomb allows anyone to browse and export the whole filesystem
Disabled user interface in configuration file; uploaded to delayed.
#588554 jamin: Ships files in /usr/lib64/
Patched configure-script to install to the correct place; with maintainers permission uploaded without delay.
#576901 init.d script fails under Squeeze with insserv due to lack of run level definitions
Updated LSB header of init script same was as upstream did in recent version; uploaded to delayed (got thanks from the maintainer for that).
#574624 dibbler-client: after removing package with dselect impossible to purge it.
Verified fix by Edward Welbourne and uploaded to delayed.
#586057 havp 0.91-1.1~volatile1 still haven't show up in lenny-volatile yet.
Seems to have been an infrastructure hick up, which got solved in the meantime. Closed without upload.
#518227 hplip: text missing at the bottom of a page on HP DJ 5550
Discussed with release team, severity got lowered.
As you can see, finding fixable rc bugs is certainly possible. So please step in :) BTW: If you don't have upload rights yourself, but prepared an NMU / patch, feel free to ping me and I'll take a look and try to sponsor your NMU. Just ask!

This week-end, I didn't work a lot on Debian stuff (mostly running and gardeing...) but still managed to finally take care of my pet package, the one I maintain since 2001, namely geneweb. The package suffered from a very longstanding RC bug that made it unusable on amd64 machines. I had to call for help Ocaml gurus and R mi Vanicat kindly provided a working patch. So, now, geneweb can be released and I won't have to ask for my pet package to be removed from testing. I even switched it to dh7 and "minimal" debian/rules file (something I planned to do at DC10 but finally turned out to be easy to do. That will be my last week-end of activity in Debian until DebConf10 as I'll be away for the upcoming week-end. So, for those of you who attend DC10, see you in NYC. I'll arrive on Wed 28th evening for part of Debcamp and I'll have the great pleasure to introduce my son Jean-Baptiste to the Debian world.

That s a recurring question I get asked: What does mraw mean? Over 2 years ago, I changed my mail signature from the traditional:

--
$FULLNAME

to:

Mraw,
KiBi.

I became a DD back then, so I put my now-become-my-login-as-well nickname there. About the mraw part, I kind of like felines in general, and felis catus in particular. Onomatop ias like maw or meow are nice, but that s even cuter when there s some purring happening at the same time, hence the r addition in there.

Finally we have the same exact features as the upstream precompiled one. YAY! Here's the story. JOSM supports the OAuth authentication method since the SVN snapshot rev. 3094. At the time (the package was uploaded on Mar 18, 2010), we didn't have some needed dependencies in Debian for that. So I had to disable it, by patching the source code. In the meanwhile, I started filing RFP/ITPs, to fulfill the missing dependency chain. And it turned to be a long chain. Here it is: Looking at it now, that it's all done, it doesn't seem that long. However, I had to face the infamous Maven buildsystem, which I didn't know at all and pretended to download every dependency in a "local repository". Then I found the maven-debian-helper and maven-repo-helper packages, and started understanding it. Unfortunately, it seems that maven is only being used by CDBS people, so I had to override some debian/rules targets to get the work done with dh7. Once the chain was fulfilled... JOSM didn't even compile. Then I discovered that JOSM had code working with oauth-signpost 1.1, but 1.2 (which I uploaded to Debian) broke the API. So I had to port it to the new API, and I could finally upload it. Now we're at the end of this adventure. Debian JOSM users will have the choice of OAuth re-enabled, and I learned to master a new (to me) beast. Enjoy

[ I meant to finish this article much earlier, but i had to leave it in a half-written draft state for quite some time due to other activities ] One particular issue that arises with big binary files is that of I/O patterns. It looks like it is not something under the usual scope for both ld.so and programs/libraries, but it can have a dramatic influence on startup times. My analysis is far from complete, would need to be improved by actually investigating ld.so code further, and so far has been limited to libxul.so built from mozilla-central, with the help of icegrind, on x86-64 linux. As I wrote recently, icegrind allows to track memory accesses within mmap()ed memory ranges. As ld.so does mmap() the program and library binaries, tracking the memory accesses within these mmap()s allows to get a better idea at I/O patterns when an ELF binary is initialized. I only focused on local accesses within a single given binary file (libxul.so) because the case of Firefox loading many files at startup is already being scrutinized, and because most of the non-Firefox files being read at Firefox startup (system and GNOME libraries) are most likely already in the page cache. I also focused on that because it was an interesting case that could be helpful to understand how big (and maybe less big) binaries may be affected by the toolchain (compiler, linker and dynamic linker) and possibly by some coding practices. For the record, what I did to get these results is to use icegrind and elflog, with further additions to the sections file with some output from objdump -h : I added sections that elflog wouldn t give (such as .plt, .hash, .dynsym, etc.), and even down to the entry level for the .rela.dyn section. The latter is particularly interesting because icegrind only outputs the first access for any given section. To see sequential accesses within that section, you need to split it in smaller pieces, which I did for .rela.dyn entries (each one being 24 bytes on x86-64). Uncommenting some parts of the icegrind code was also useful to track where some accesses were made from, code-wise. Now, the interesting data: One of the first accesses is a nullification of a few variables within the .bss section. The .bss section is usually an anonymous piece of memory, that is, a range of mmap()ed memory that is not backed by a file, and filled with zeroes by the kernel (I think it even does that lazily). It is used for e.g. variables that are initialized with zeroes in the code, and obviously, any code accessing these variables would be addressing at some offset of the .bss section. It means the section needs to start in memory at the offset it has been assigned by the linker at build time. This is actually where problems begin. When the .bss section offset as assigned by the linker doesn t align on a page (usually 4KB), the mmap()ed .bss can t be at that location, and really starts on the next page. The remainder is still mmap()ed from the binary file, and ld.so will itself fill that part. As the .bss section doesn t start on a page boundary, any write at this location will trigger the kernel reading the entire page. This means one of the first set of data being read from the file is the end of the section preceding .bss, and the beginning of the following one. Most likely, respectively the .data and .comment sections. While this probably doesn t matter much when the binary file is small, when it is big enough, reading in a non sequential manner will trigger hard disk seeks, and we all know how they can hurt performance. Although thankfully, cheap SSDs should be coming some day, in the meanwhile, we still need to cope with the bad performance. The interesting part is, the .bss section is really empty in the binary file, so its virtual memory address could be anywhere. Why the linker wouldn t align it at a page boundary without having to resort to a linker script is beyond me. The next accesses go back and forth between .dynamic, .gnu.hash, .dynstr, .gnu.version_r, .gnu.version and .dynsym. These are probably all related to symbol version resolution and DT_NEEDED library loading. While most of these sections are at the beginning of the file (not necessarily in the order they are read from), the .dynamic section is much nearer to the end, but way before .data, so that it won t even have been loaded as a by-product of the .bss section loading. After that, the .rela.dyn section is read, and for each relocation entry it contains, the relocations are being applied. Relocations is one of the mechanisms by which position independent code (PIC) is made possible. When a library is loaded in memory, it is not necessarily loaded at the same address every time. The code and data contained in the library thus need to cope with that constraint. Fortunately, while the base address where the library is mmap()ed in memory is not necessary constant, the offsets of the various sections are still as codified in the binary. The library code can thus directly access data at the right offset if it knows the base offset of the library (which is what is done on the x86 ABI), or if it knows where the current instruction is located (which is what is done on the x86-64 ABI). Static data (initialized in e.g. a const or a global variable, or, in C++ case, vtables), on the other hand, may contain pointers to other locations. This is where relocation enters the scene. The .rela.dyn section contains a set of rules describing where in the binary some pointers need to be adjusted depending on the base library offset (or some other information), and how they should be updated. ld.so thus reads all .rela.dyn entries, and applies each relocation, which means that while .rela.dyn is being read sequentially, reads and writes are also performed at various places of the binary, depending on the content of the .rela.dyn entries. This is where this gets ugly for Firefox: there are near 200000 such relocations. On x86-64 an entry is 24 bytes (12 on x86), and each of these is going to read/write a pointer (8 bytes on x86-64, 4 on x86) at some random (though mostly ordered) location. The whole .rela.dyn section not being read ahead, what actually happens is that it is read in small batches, with seeks and reads of other data at various locations in between. In libxul.so case, this spreads over .ctors, .data.rel.ro, .got, and .data. The relocation entries are somehow ordered by address to be rewritten, though they occasionally jump backwards. Some of these relocations also appear to be touching to .gnu.version, .dynsym and .dynstr, because their type involves a symbol resolution. Once .rela.dyn relocation have been dealt with comes .rela.plt s turn. The principle is the same for this section: entries describe what kind of relocation must be done where, and how the result must be calculated. The scope of this section, though, is apparently limited to .got.plt. But before explaining these relocations, I ll explain what happens with the PLT. The PLT (Procedure Linkage Table) is used when calling functions from external libraries. For example, in a Hello World example, the PLT would be used for calls to the puts function. The function making the call would in fact call the corresponding PLT location. The PLT itself, on x86 and x86-64, at least, consists of 3 instructions (I ll skip the gory details, especially for x86, where the caller needs to also set some register before calling the PLT). The first instruction is the only one to be called most of the time: it reads the final destination in the .got.plt section, and jumps there. That final destination, obviously, is not fixed in the library file, since it needs to be resolved by its symbol. This is why the two subsequent instructions exist : originally, the destination stored in the .got.plt section points back to the second instruction ; the first instruction will effectively be a nop (no operation), and the following instructions will be executed. They will jump into code responsible for symbol resolution, update of the .got.plt entry for the next call, and call of the real function. But pointing back to the second instruction is like the pointers in static data we saw above : it s not possible in position independent code. So, the .rela.plt relocations are actually filling the .got.plt section with these pointers back to the PLT. There are a tad more than 3000 such relocations. All these relocations should be going away when prelinking the binaries, but from my several experimentations, it looks like prelinking only avoids relative relocations, and not the others, while it technically could skip all of them. prelink even properly applies all the relocations in the binary, but executing the binary rewrites the same information at startup for all but relative relocations. That could well be a bug in either ld.so not skipping enough relocations or prelink not marking enough relocations to be skipped. I haven t dug deep enough in the code to know how prelinking works exactly. Anyways, prelinking is not a perfect solution, as it also breaks ASLR. Sure, prelink can randomize library locations, but it relies on the user or a cron job doing so at arbitrary times, but that s far from satisfying. An interesting thing to note, though, is that a good part of the relocations prelinking doesn t rid us of in libxul.so (more than 25000) are due to the __cxa_pure_virtual symbol, which is used for, well, pure virtual methods. In other words, virtual methods that don t have an implementation in a given class. The __cxa_pure_virtual function is set as method in the corresponding field(s) of the class VTABLE in the .data.rel.ro section. This function is provided by libstdc++, and as such, is dynamically linked. But this function is just a dummy function, doing nothing. Defining an empty __cxa_pure_virtual function to be included in libxul.so makes these relocations become relative, thus taken care of by prelinking. After all relocations occur, the library initialization itself can begin, and the content of the .init section is executed. That section, indirectly, executes all functions stored in the .ctors section. This includes static initializers, which are unfortunately called backwards, as Taras pointed out already. Each of these static initializers are also accessing various locations in the .data or .bss sections, which may or may not have already been loaded during the relocation phase. The execution of these initializers will also (obviously) read various pieces of the .text section (despite its name, it contains executable sections, i.e. functions code). The initialization of the library ends there, and no access should happen until a function from the library is called. In libxul.so case, XRE_main is first called, then many other functions, but that s another story. All that needs to be remembered about the startup process past this point is that the .text section will be read heavily, as well as the various data, got, plt and dynamic symbol sections, in a very scattered way. While most of these sections may have been retrieved in memory already, as a byproduct of the various initialization processes described above, some may have not, increasing even more the need to seek at all places in the binary file. Now the main problem with all these I/O patterns at startup, is that it seems the only way reorganizing the binary layout may have a visible impact is by considering all the above, and not only a part of it, because only addressing a part of it is very likely to only move part of the problem to a different layer. All in all, making sure the relevant sections of libxul.so are read by the kernel before ld.so enters the game is a good short-term solution to avoid many seeks at startup.

Qt 4.7.0 beta 2 and Qt Creator 2.0.0 have been released upstream. There s some
noticeable changes in our Qt packages compared to Qt 4.6.x series:
- legacy Qt Assistant (ADP) is removed
- new Qt declarative module is introduced
- Qt Multimedia and Qt Webkit aren t being built anymore In case your package depends on the deprecated Qt Assistant, it is still being
shipped in a separated Qt Assistant compatibility package called
qt-assistant-compat. Qt Multimedia is now shipped from Qt Mobility API instead of Qt. The module has
been renamed to Qt Mobility MultimediaKit. Qt Mobility delivers a set of APIs
for mobile device functionality. Qt WebKit is shipped standalone, separated from Qt. Hopefully, it will make
Qt WebKit security support easier. There are also a new component in the family: Qt Messaging Framework (QMF).
The Qt Messaging Framework, QMF, consists of a C++ library and daemon server
process that can be used to build email clients, and more generally software
that interacts with email and mail servers. Finally, I have started to ship a snapshot of a new Phonon backend: Phonon VLC,
one backend to rule them all ! All the packages can be found in Debian Experimental repository, except
Qt Simulator, a simulator for Qt applications running on Nokia devices.
This package should land soon in Debian archives. I encourage you to try building your Qt applications against these packages and
please, report any bugs you encounter. Enjoy. http://packages.qa.debian.org/q/qt4-x11.html
http://packages.qa.debian.org/q/qtcreator.html
http://packages.qa.debian.org/q/qt-assistant-compat.html
http://packages.qa.debian.org/q/qtmobility.html
http://packages.qa.debian.org/q/qtwebkit.html
http://packages.qa.debian.org/q/qmf.html
http://packages.qa.debian.org/p/phonon-backend-vlc.html
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587759

Apropos of my previous post, I see that dh has now overtaken CDBS as the most popular rules helper system of its kind in Debian unstable, and shows no particular sign of slowing its rate of uptake any time soon. The resolution of the graph is such that you can't see it yet, but dh drew dead level with CDBS on Thursday, and today 3836 packages are using dh as opposed to 3823 using CDBS.