[ I meant to finish this article much earlier, but i had to leave it in a half-written draft state for quite some time due to other activities ]
One particular issue that arises with big binary files is that of I/O patterns. It looks like it is not something under the usual scope for both
ld.so and programs/libraries, but it can have a dramatic influence on startup times. My analysis is far from complete, would need to be improved by actually investigating
ld.so code further, and so far has been limited to libxul.so built from mozilla-central, with the help of icegrind, on x86-64 linux.
As I
wrote recently, icegrind allows to track memory accesses within
mmap()ed memory ranges. As
ld.so does
mmap() the program and library binaries, tracking the memory accesses within these
mmap()s allows to get a better idea at I/O patterns when an ELF binary is initialized. I only focused on local accesses within a single given binary file (libxul.so) because the case of Firefox loading many files at startup is already being scrutinized, and because most of the non-Firefox files being read at Firefox startup (system and GNOME libraries) are most likely already in the page cache. I also focused on that because it was an interesting case that could be helpful to understand how big (and maybe less big) binaries may be affected by the toolchain (compiler, linker and dynamic linker) and possibly by some coding practices.
For the record, what I did to get these results is to use
icegrind and elflog, with further additions to the sections file with some output from
objdump -h : I added sections that elflog wouldn t give (such as
.plt,
.hash,
.dynsym, etc.), and even down to the entry level for the
.rela.dyn section. The latter is particularly interesting because icegrind only outputs the first access for any given section. To see sequential accesses within that section, you need to split it in smaller pieces, which I did for
.rela.dyn entries (each one being 24 bytes on x86-64). Uncommenting some parts of the icegrind code was also useful to track where some accesses were made from, code-wise.
Now, the interesting data:
One of the first accesses is a nullification of a few variables within the
.bss section. The
.bss section is usually an anonymous piece of memory, that is, a range of
mmap()ed memory that is not backed by a file, and filled with zeroes by the kernel (I think it even does that lazily). It is used for e.g. variables that are initialized with zeroes in the code, and obviously, any code accessing these variables would be addressing at some offset of the
.bss section. It means the section needs to start in memory at the offset it has been assigned by the linker at build time. This is actually where problems begin.
When the
.bss section offset as assigned by the linker doesn t align on a page (usually 4KB), the
mmap()ed
.bss can t be at that location, and really starts on the next page. The remainder is still
mmap()ed from the binary file, and
ld.so will itself fill that part. As the
.bss section doesn t start on a page boundary, any write at this location will trigger the kernel reading the entire page. This means one of the first set of data being read from the file is the end of the section preceding
.bss, and the beginning of the following one. Most likely, respectively the
.data and
.comment sections.
While this probably doesn t matter much when the binary file is small, when it is big enough, reading in a non sequential manner will trigger hard disk seeks, and we all know how they can hurt performance. Although thankfully, cheap SSDs should be coming some day, in the meanwhile, we still need to cope with the bad performance. The interesting part is, the
.bss section is really empty in the binary file, so its virtual memory address could be anywhere. Why the linker wouldn t align it at a page boundary without having to resort to a linker script is beyond me.
The next accesses go back and forth between
.dynamic,
.gnu.hash,
.dynstr,
.gnu.version_r,
.gnu.version and
.dynsym. These are probably all related to symbol version resolution and DT_NEEDED library loading. While most of these sections are at the beginning of the file (not necessarily in the order they are read from), the
.dynamic section is much nearer to the end, but way before
.data, so that it won t even have been loaded as a by-product of the
.bss section loading.
After that, the
.rela.dyn section is read, and for each relocation entry it contains, the relocations are being applied. Relocations is one of the mechanisms by which position independent code (PIC) is made possible. When a library is loaded in memory, it is not necessarily loaded at the same address every time. The code and data contained in the library thus need to cope with that constraint. Fortunately, while the base address where the library is
mmap()ed in memory is not necessary constant, the offsets of the various sections are still as codified in the binary. The library code can thus directly access data at the right offset if it knows the base offset of the library (which is what is done on the x86 ABI), or if it knows where the current instruction is located (which is what is done on the x86-64 ABI).
Static data (initialized in e.g. a
const or a global variable, or, in C++ case, vtables), on the other hand, may contain pointers to other locations. This is where relocation enters the scene. The
.rela.dyn section contains a set of rules describing where in the binary some pointers need to be adjusted depending on the base library offset (or some other information), and how they should be updated.
ld.so thus reads all
.rela.dyn entries, and applies each relocation, which means that while
.rela.dyn is being read sequentially, reads and writes are also performed at various places of the binary, depending on the content of the
.rela.dyn entries.
This is where this gets ugly for Firefox: there are near 200000 such relocations. On x86-64 an entry is 24 bytes (12 on x86), and each of these is going to read/write a pointer (8 bytes on x86-64, 4 on x86) at some random (though mostly ordered) location. The whole
.rela.dyn section not being read ahead, what actually happens is that it is read in small batches, with seeks and reads of other data at various locations in between. In libxul.so case, this spreads over
.ctors,
.data.rel.ro,
.got, and
.data. The relocation entries are somehow ordered by address to be rewritten, though they occasionally jump backwards. Some of these relocations also appear to be touching to
.gnu.version,
.dynsym and
.dynstr, because their type involves a symbol resolution.
Once
.rela.dyn relocation have been dealt with comes
.rela.plt s turn. The principle is the same for this section: entries describe what kind of relocation must be done where, and how the result must be calculated. The scope of this section, though, is apparently limited to
.got.plt. But before explaining these relocations, I ll explain what happens with the PLT.
The PLT (Procedure Linkage Table) is used when calling functions from external libraries. For example, in a Hello World example, the PLT would be used for calls to the
puts function. The function making the call would in fact call the corresponding PLT location. The PLT itself, on x86 and x86-64, at least, consists of 3 instructions (I ll skip the gory details, especially for x86, where the caller needs to also set some register before calling the PLT). The first instruction is the only one to be called most of the time: it reads the final destination in the
.got.plt section, and jumps there. That final destination, obviously, is not fixed in the library file, since it needs to be resolved by its symbol. This is why the two subsequent instructions exist : originally, the destination stored in the
.got.plt section points back to the second instruction ; the first instruction will effectively be a nop (no operation), and the following instructions will be executed. They will jump into code responsible for symbol resolution, update of the
.got.plt entry for the next call, and call of the real function.
But pointing back to the second instruction is like the pointers in static data we saw above : it s not possible in position independent code. So, the
.rela.plt relocations are actually filling the
.got.plt section with these pointers back to the PLT. There are a tad more than 3000 such relocations.
All these relocations should be going away when prelinking the binaries, but from my several experimentations, it looks like prelinking only avoids relative relocations, and not the others, while it technically could skip all of them.
prelink even properly applies all the relocations in the binary, but executing the binary rewrites the same information at startup for all but relative relocations. That could well be a bug in either
ld.so not skipping enough relocations or
prelink not marking enough relocations to be skipped. I haven t dug deep enough in the code to know how prelinking works exactly. Anyways, prelinking is not a perfect solution, as it also breaks
ASLR. Sure, prelink can randomize library locations, but it relies on the user or a cron job doing so at arbitrary times, but that s far from satisfying.
An interesting thing to note, though, is that a good part of the relocations prelinking doesn t rid us of in libxul.so (more than 25000) are due to the
__cxa_pure_virtual symbol, which is used for, well, pure virtual methods. In other words, virtual methods that don t have an implementation in a given class. The
__cxa_pure_virtual function is set as method in the corresponding field(s) of the class VTABLE in the
.data.rel.ro section. This function is provided by libstdc++, and as such, is dynamically linked. But this function is just a dummy function, doing nothing. Defining an empty
__cxa_pure_virtual function to be included in libxul.so makes these relocations become relative, thus taken care of by prelinking.
After all relocations occur, the library initialization itself can begin, and the content of the
.init section is executed. That section, indirectly, executes all functions stored in the
.ctors section. This includes static initializers, which are unfortunately called backwards, as
Taras pointed out already. Each of these static initializers are also accessing various locations in the
.data or
.bss sections, which may or may not have already been loaded during the relocation phase. The execution of these initializers will also (obviously) read various pieces of the
.text section (despite its name, it contains executable sections, i.e. functions code).
The initialization of the library ends there, and no access should happen until a function from the library is called. In libxul.so case, XRE_main is first called, then many other functions, but that s another story. All that needs to be remembered about the startup process past this point is that the
.text section will be read heavily, as well as the various data, got, plt and dynamic symbol sections, in a very scattered way. While most of these sections may have been retrieved in memory already, as a byproduct of the various initialization processes described above, some may have not, increasing even more the need to seek at all places in the binary file.
Now the main problem with all these I/O patterns at startup, is that it seems the only way reorganizing the binary layout may have a visible impact is by considering
all the above, and not only a part of it, because only addressing a part of it is very likely to only move part of the problem to a different layer.
All in all, making sure the relevant sections of libxul.so are read by the kernel before
ld.so enters the game is a good short-term solution to avoid many seeks at startup.
I've got a pretty packed August.
I just wrapped the Open and User Innovation Conference at MIT -- the
academic conference on user and open innovation connected to my
research. I organized the program and was MC for the 120+(!) talks and
research updates on the program so it's a huge relief to see it come off
successfully.
On Thursday, August 5th (at 14:30 UTC) I'll be giving a talk on
antifeatures at DebConf (the Annual Debian conference). It was
accidentally listed as "Revealing Errors" until a few minutes ago --
sorry about that! It will be streamed live (details on the DC site)
for those outside of New York City who might want to follow it.
As soon as DebConf is done on August 8th, I'm going to head to
Kor ula in Croatia to relax, read, and hopefully get a bit of
research done, before I head off to Outlaws and Inlaws in Split on
the 19th, a sort of piracy and (vs?) free software summit put on by
mi2 connected to the recurring Nothing Will Happen where, from
what I hear, quite a lot does.
I'm going to have to leave Nothing Will Happen a little early to head to
FrOSCon on the 21st where I'll be doing an antifeatures keynote again
on the 22nd. I tend not to like to do the same talk too many times, or
for more than a year, so this might be one of the last times I present
on antifeatures in this form.
After that, I'm going to head to Italy where I'll be between the 23rd
and the 3rd of September. I'll fly and in and out of Rome and plan to
spend some time in Rome, Tuscany, and Florence, but don't have a lot of
set plans and might travel to Bologna or elsewhere.
My schedule is pretty open. As always, I'm interested in meeting up for
coffee or a drink with like-minded hackers, Wikipedians, researchers,
activists, etc. If folks are interested in organizing talks or
presentations, that sounds fun too. I'm keeping a brief description of
my schedule updated alongside a bunch of ways to get in touch with me on
my contact page. Don't hesitate to drop me a line!
In the evening today (Tuesday), the Debconf organizers arranged for a screening of the Creative Commons licensed film "Sita Sings the Blues" by Nina Paley. I went to see it as I had missed it when it was first released among tales of trials and tribulation at the hands of copyright meanies.
So first let me say yay Free Culture!, boo Intellectual Property! etc. etc. Now we have that out of the way...
"Sita Sings the Blues" has two main narrative threads: The breakup of Ninas relationship with her boyfriend and the Ramayana particularly a story from the uttarakanda about how after Shri Rama returned from Lanka, Sita had to undergo the agnipravesha to prove her chastity and even after she passed that test was eventually banished to the forest. I had an adverse reaction to this film. In fact I stayed to the end to be polite but I left afterwards as soon as I could. I had almost a physical feeling of nausea and I was surprised by this. Since then I have been pondering why this is so.
It's not that its blasphemous. Shri Rama is not just a "perfect man", he is God (an avatar of Vishnu Bhagavan as is briefly touched upon in the film) for millions of people so, well, yes it is blasphemous but its more Rowan Atkinson style rather than Norwegian Black Metal style blasphemy. One never gets the impression that the author hates Hinduism or Indian culture the opposite in fact. And in fact it is not uncommon in our culture to speak of Sita and Rama in this way. They are not remote authority figures but as familiar to us as our own friends and family and the Ramayana is not just about events that happened long ago it is relevant to our lives right now. Even the "good woman done wrong by big dumb male jerk" narrative is not unknown. (See the Uttararamacharita of Bhavabhuti for instance.) though I would suggest reducing the Ramayana this way is akin to treating the Hitchhikers Guide to the Galaxy as a book about towels.
Even the milder criticism that it is not accurate is offbase because although the Sanskrit poem by Maharshi Valmiki is the canonical rendition of the Ramayana, there are other well-known versions. Not just in Sanskrit, the Hindi retelling by Sant Tulsidas called Ramacharitamanasa is more popular than Valmiki. Also speaking as someone who has preached Ramakatha "professionally" I can tell you that even the priests do not stick to the letter of the text. It is encouraged and expected that they will riff on the basic stories adding or removing details according to the needs and taste of the audience. And the device of the three narrators made me smile as this is exactly the kind of discussions that go on after hearing the Ramakatha. (I must say Paley picked a notably illiterate trio though.) One place where this fell flat was when they discussed the alleged discrepancy between Sita living simply in the forest and dropping a trail of jewellery on the way to Lanka. Although I can imagine someone at some point saying, "Do not question these stories" not at this point. Even the most retarded westernized Indian would know that removing ones ornaments is a sign of widowhood. A married woman would only do that in extremis certainly not just merely because she was a hermit. In a few cases some extra details might have helped the story. It was not explained that Ravana does not just have rakshasas in his command but he is one himself. (And he certainly wasn't a good guy until one day he got up and kidnapped Sita.) Shri Ramas brother Lakshamana accompanied the couple into the forest to Lanka and back but he only pops up (for no apparent reason) at the end. But look at me, I'm turning into one of those people who rant about Star Trek continuity errors or complain that the Romans in a gladiator movie have the wrong helmets for that time period etc. One has to cut a storyteller some slack and I entered the screening determined to be charitable.
It's also not because I don't really see much connection between the two stories. Getting dumped by someone who simply loses interest in you hardly seems equivalent to being trapped by a moral code that doesn't allow for individuals feelings. Paley could have easily made "Princess Leia Sings the Blues" and it wouldn't have changed the significance much. But in India people make far more tenuous connections between the epics and events in their own life so one can hardly complain about that.
Aesthetics are not the problem either. The first version of the Ramayana I ever read was the Amar Chitra Katha comicbook version which partly influenced the visual style of this film. (In fact you briefly see it in one scene when different editions of the Ramayana flash by.) Hindu popular culture is very gaudy. (See Ramanand Sagars '80s Ramayana Hindi TV serial, which was very reverential, insanely popular and tacky beyond belief.) The animation in this film is very good and quite artistic.
"So Jaldhar", you might be asking if you've read this far, "what exactly is your problem with this film?"
I have read the Valmiki Ramayana in Sanskrit with the commentaries from beginning to end. I've read other Ramayanas in Sanskrit, Gujarati, and English. I have read the comicbook, I've read anti-Ramayana polemics. When you learn Sanskrit in the traditional way the first lesson, in the declension of masculine gendered nouns ending in vowels is rAma.h rAmau rAmA.h ... I am well aware there are different emphases, different interpretations in various versions. Why can't I accept this as just another take on a story that has shown its fluidity over and over again? It's just...wrong on a subrational level. How can I put this? Imagine you are flipping through the TV channels and you see your grandparents amongst Lady Gaga's backup dancers. Or you came home one night to find everything that was on the floor meticulously glued to the ceiling. Over the course of my life (to hear my mother tell it my first exposure to the Ramayana was before I could speak.) I have developed a mental bond with this story a certain idea of how it should be. I think so has Nina Paley. And this is the problem. "Sita Sings the Blues" is Nina Paleys Ramayana not mine.
This week, I am attending 
On each last Friday of July the annual 
Yesterday late evening, Jean-Baptiste and I arrived in NYC after
about a 16 hours trip. Amazingly, he mentioned to me that we entered a
train in the station close to our home....and then never went outside
until we exited the 1 line subway at 116th street on Broadway Ave.:
While I have made good progress on many of my 
I had planned long for this Debconf and with the experience from the previous times I did all I could to reach one goal minimise stress. I think I got it right this time.
First of all I got my company (Accenture) pay for the whole thing, so I did not have to worry about sponsorship queue (but instead I must gather and deliver them back some knowledge gained from the conference should be easy). I also managed to get a direct flight there is a weekly flight Tashkent-Riga-New York on Sundays and I got a ticket for that both ways. This took many hours off the travel time and reduced the stress a ton. Then I also did all the prudent travelling things: mostly reading the New York page on Wikitravel. This told me (among other things) what metro line to take to get the best view of the city. And for luggage I took a backpack (for the laptop and walking around) and a roller (for the clothes). I learned a lot from the Up in the air movie, it really teaches one how to take stress out of travel.
This went very well in the beginning I took a bus to the airport early, checked in, breezed trough security and then waited and waited. The boarding only started half an hour after we were scheduled to depart. But I can not complain, because the flight attendant called my name (among a few others) just before boarding and changed my ticket I was bumped up to business class 
. It has been many years since I ve flown a regular (non-budget) airline and now first time ever in the business class! We got regular class meals, but at least we got to sit in the big chairs with lots of leg room. It was great! I bought a paper book (from Charles Stross) to read during the flight, because that is the only thing you can do during take-off and landing and I had some stuff to listen to in my iPhone. I tried sleeping and I think I managed to kill a couple hours that way too. It sure helped combat jet lag later on.
When I arrived, I had my immigration and customs forms filled out. Latvia is on the USA Visa Waiver program, so I just had to fill out a form online prior to the trip to get a permission to enter USA. I only had one printout of the Waiver confirmation page and my airline took that before giving me a boarding pass, so I had no paper copy to give to the guy at the USA border, but he did not even ask for it he only took the two forms I filled out, scanned my passport, took my fingerprints (all fingers) and a photo. Then he gave me back the customs for (stamped) and I could go get my luggage. After that it was just a matter of walking trough customs and giving the customs official the pre-filled and stamped customs form. I was not stopped further.
The AirTrain was a bit confusing you just get on the AirTrain and go where you need to go, you actually pay at the place where AirTrain connects to metro, so when you are coming from the airport you are actually paying ($5) when you are exiting the AirTrain system. There are plenty of AirTrain people that will give you an AirTrain ticked for $5 cash right there or you can use a machine to get it (where you can also pay with a card). The machines also take AmEx, but you must know the PIN code, if your don t know the PIN of you AmEx card (like I don t know the PIN of my corporate AmEx) you can only buy MetroCards at news stands, but there you will not find the unlimited ride tickets only 5 or 10 ride tickets are available.
Next I screwed up a bit and thus got a bit lost in the New York metro system. The Wikitravel recommends to take the AirTrain to Jamaica station and from there take J or Z line into the city for the best views. I went to Jamaica and got into the first metro train that came on to the platform. Unfortunately, that was the E line train and it does not cross with J or Z lines after that, so I was stuck. To complicate the situation E line is under repair, so after a few stops the trains diverted to the F line and I had to scramble to figure out where to get out to get to the 1 line where Columbia University is. I ended up spending more than an hour in the metro, but I got there in the end.
So I emerge from the 116th street metro station directly in front of the campus entrance gate and it starts raining. A thunderstorm swept across New York as soon as I stepped outside the metro station, so I took shelter a the side of the building. It kept raining. After a few minutes decided that I don t want to wait any more, so I put all my electronics into my backpack and went towards the dorm where my room was. The rain was warm, people around me were playing and running around in the wet grass, it was fun.
The guard showed me where to go to check in. It was really simple sign here, here is you magnetic stripe key, have fun. And the room was just as spartan bed, desk, chair, closet, bathroom and a large fan under the window. The first night I did not know that I could actually turn that fan down, so I slept to the sound of full fan blasting, much like on the airplane. The dorm looks ancient it looks like it was build in the 60s and never changed since then. I mean, I ve lived in worse places, but people would at least change wall sockets or faucets or ceiling paint every other decade or so. The newest things here were the magnetic locks on the doors and a TV and a mini fridge in the hallway.
After dropping my things I went off to the Mudd building across the campus to check in with people in the hacklab. It was just starting to take shape, but many great and familiar people were already there. It was like walking into a family home same warm welcome smiles and that great feel of belonging to the group.
After I quick chat I went off downtown. The plan was that I left my old Canon 400D camera back at home and that I would buy a new Canon 550D here in New York on arrival. Unfortunately I underestimated the travel time on subway and by the time I got back downtown most shops where already closed. I got to checking out a Best Buy and was told there that they don t have 550D in stock. Next morning I went for a full scale search B&H Photo Video store, J&R Express, another BestBuy none of them had the Canon 550D (aka T2i) in stock in a body-only configuration. BestBuy had a few sets with the kit lens, but I did not want paying 100$ extra for something I already have. The last option was Adorama and I got lucky I nabbed the last 550D they had and it was body-only.
Now I was back, with a camera in my hands and the Debconf10 could really begin!
I'm a terrible hoarder. I hang onto old stuff because I think it might be
fun to have a look at again later, when I've got nothing to do. The problem
is, I never have nothing to do, or when I do, I never think to go through
the stuff I've hoarded. As time goes by, the technology becomes more and
more obsolete to the point where it becomes impractical to look at it.
Today's example: the 3.5" floppy disk. I've got a disk holder thingy with
floppies in it dating back to the mid-nineties and earlier. Stuff from high
school, which I thought might be a good for a giggle to look at again some
time.
In the spirit of 
The other projects that will never all fit in the freed time: I want to spend more time working on dpkg. I do plan to blog more often too, but I m sure you ll notice that yourself soon. I would like to see my Debian book translated into English (another post coming on the topic sometimes soon). In my dreams, I could even start yet another software project, I have some ideas that I really would like to see implemented but I don t see how that could fit in this year s planning unless I can convince someone else to implement them! Maybe I should blog about them.
I ve got some interesting results using r1.1 of an example 
A few years ago, when I was still a member of the Jozef Pauly
ensemble from Ekeren, at some point I took over maintaining their
website. One of the things I did was converting it to drupal, mainly
because that would allow other members of the ensemble to help
maintaining it.
Of course, such things do not happen automatically; and had I spent
some more time on the project, it might've actually worked. But I
didn't, so it didn't. Oh well.
One of the main issues with the site as I set it up, was that I
hadn't really done a very good job integrating everything. The standard
drupal 5 theme looked pretty good, I thought. Slap on a few modules, and
done. Unfortunately, I skipped a bit of testing, and as a result, the
site didn't really function the way it should've. When this became
apparent, I didn't have the time to properly fix it.
During the last few years that I was a member of the ensemble,
someone else stepped up to help maintain it. I told him that one of the
things that I wanted to do was to upgrade the site to Drupal 6, so as to
make i18n a bit easier. But some of the modules which weren't in drupal
core made this somewhat harder than needed, and I didn't have the time
to investigate in-depth what the issues were.
In the end, I set up a secondary site on a different vhost on the
same server that ran drupal6, and gave that guy admin rights there to
help convert things to the new site. I planned to lead the effort to
make things work; but when I came back not so much later, it turned out
that he'd done a very good job already, and that it was probably a good
idea to let him finish.
This took much, much longer than expected; but eventually,
he's finalized the setup, and it's now live (since a few weeks) on