I did not write much code or merge much of my prototype code, but some things happened since the last blog post about APT2 specific things in August and I forgot to write about them. First of all, I dropped the GVariant-based cache. The format strings were simply getting ugly long and were not very understandable, performance was just much too slow (needing more than a few nanoseconds for a package lookup is obviously too slow for solving dependency problems); furthermore, building the cache was also slow and complicated because we needed all attributes of an object at once to pass them to GVariant, leading to ugly API. I replaced the GVariant cache with one that can be easily mmap()ed and is described completely in C. It s derived from APT s cache design (but more robust, as it includes the size of the cache and we can thus detect to small files, although that s scheduled for the next ABI break in APT as well), but has fewer duplicate data, and uses arrays where APT uses linked lists. The reason for arrays is simple: They take up less space and can be represented naturally in Python and other languages using array-based lists. The cache also contains a coalesced hash table which does use a linked list, but that one is a bit different, as it is for searching only and not exposed. Everything non-stringy is 64-bit aligned in order to keep things as simple as possible. All integers are fixed size, thus the format is architecture-independent if you fix byte orders. The format is described at http://people.debian.org/~jak/apt2-doc/apt-Cache-Format.html. I stole one more idea from cupt and changed the configuration system to verify types of variables. APT2 s configuration system knows more types than cupt s, though, including regular expressions, directory and filenames (i.e. it does not let you store a value /d/ in a file variable), strings (which store everything), unsigned and signed integers, and boolean options; all of which are checked when parsing files (producing warnings) or command-line options (producing errors). I have also simplified the type world by removing all iterator types except for one, replacing them with get_thing() and n_things() functions in the objects holding the arrays. Makes cool bindings slightly harder, but makes the C API much easier to use from C. Most things expected from a package manager are still missing, but what is there looks good in most cases (especially AptConfiguration has a nice API, and no complaints from valgrind anywhere). Currently I am working on Python bindings so I can interact with the functions easily and check things in an interactive fashion; and I am also writing a document explaining the concepts behind APT2, drafts at http://people.debian.org/~jak/midlevel.pdf. I also have some more code pending further thoughts (including complete index parsing), but it might still take some time before I have something usable in the wild. On other package managers: From time to time I also use Cupt, look at Cupt code, hack Cupt code, and report bugs against Cupt. I still do not really understand the (extreme) nesting of directory structures in the source code, why there are so (extremely) many source files split all over them, or the general concepts of Cupt, but I can hack together what I need for my personal testing. I also play with yum whenever I end up on a Fedora system (which happens from time to time).
Filed under: APT2

As everyone should know, underscores in C are not cool, as they cause undefined behavior per 7.1.3:

All identifiers that begin with an underscore and either an uppercase letter or another underscore are always reserved for any use.
[...]
If the program declares or defines an identifier in a context in which it is reserved (other than as allowed by 7.1.4), or defines a reserved identifier as a macro name, the behavior is undefined.

Yet, they are widely used everywhere. Here are some examples:

• inclusion guards in GLib: __G_VARIANT_H__
• internal Python functions: _PyUnicode_AsString
• various macros in APT: __deprecated, __hot

All of this triggers undefined behavior and is thus uncool. Of course in APT, it s most stupid, as we do not have any namespace and could thus
end up redefining things we should not much more likely then the other two. But why were those solutions chosen in the first place, and what is the alternative? I cannot answer the first question, but for the second one, the obvious alternative is to use trailing underscores:

• inclusion guards, defined behavior: G_VARIANT_H__
• internal functions, defined behavior: PyUnicode_AsString_
• various macros, defined behavior: deprecated__, hot__

Then there is another class of reserved identifiers with underscores:

All identifiers that begin with an underscore are always reserved for use as identifiers
with file scope in both the ordinary and tag name spaces.

Meaning that everything except for parameters, local variables and members of structs/unions that starts with an underscore is reserved. So, if you happen to create a variable _mylibrary_debug_flag, you trigger undefined behavior as well. And while we re at it, do not think you can create a type ending in _t: POSIX reserves all identifiers ending in _t for its own use. In summary, whenever you write C and want to be 100% safe of undefined-behavior-because-of-naming, do not start any identifier with an underscore and do not end any identifier with _t.
Filed under: General

(this post was going to be about something else, but after getting this far, I think it stands on its own as an introduction to dynamic linking) The shared library is an integral part of a modern system, but often the mechanisms behind the implementation are less well understood. There are, of course, many guides to this sort of thing. Hopefully this adds another perspective that resonates with someone. Let's start at the beginning - relocations are entries in binaries that are left to be filled in later -- at link time by the toolchain linker or at runtime by the dynamic linker. A relocation in a binary is a descriptor which essentially says "determine the value of X, and put that value into the binary at offset Y" each relocation has a specific type, defined in the ABI documentation, which describes exactly how "determine the value of" is actually determined. Here's the simplest example: The value of foo is not known at the time you make a.o, so the compiler leaves behind a relocation (of type R_386_32) which is saying "in the final binary, patch the value at offset 0x4 in this object file with the address of symbol foo". If you take a look at the output, you can see at offset 0x4 there are 4-bytes of zeros just waiting for a real address: That's link time; if you build another object file with a value of foo and build that into a final executable, the relocation can go away. But there is a whole bunch of stuff for a fully linked executable or shared-library that just can't be resolved until runtime. The major reason, as I shall try to explain, is position-independent code (PIC). When you look at an executable file, you'll notice it has a fixed load address This is not position-independent. The code section (with permissions R E; i.e. read and execute) must be loaded at virtual address 0x08048000, and the data section (RW) must be loaded above that at exactly 0x0805ff88. This is fine for an executable, because each time you start a new process (fork and exec) you have your own fresh address space. Thus it is a considerable time saving to pre-calculate addresses from and have them fixed in the final output (you can make position-independent executables, but that's another story). This is not fine for a shared library (.so). The whole point of a shared library is that applications pick-and-choose random permutations of libraries to achieve what they want. If your shared library is built to only work when loaded at one particular address everything may be fine until another library comes along that was built also using that address. The problem is actually somewhat tractable you can just enumerate every single shared library on the system and assign them all unique address ranges, ensuring that whatever combinations of library are loaded they never overlap. This is essentially what prelinking does (although that is a hint, rather than a fixed, required address base). Apart from being a maintenance nightmare, with 32-bit systems you rapidly start to run out of address-space if you try to give every possible library a unique location. Thus when you examine a shared library, they do not specify a particular base address to be loaded at: Shared libraries also have a second goal code sharing. If a hundred processes use a shared library, it makes no sense to have 100 copies of the code in memory taking up space. If the code is completely read-only, and hence never, ever, modified, then every process can share the same code. However, we have the constraint that the shared library must still have a unqiue data instance in each process. While it would be possible to put the library data anywhere we want at runtime, this would require leaving behind relocations to patch the code and inform it where to actually find the data destroying the always read-only property of the code and thus sharability. As you can see from the above headers, the solution is that the read-write data section is always put at a known offset from the code section of the library. This way, via the magic of virtual-memory, every process sees its own data section but can share the unmodified code. All that is needed to access data is some simple maths; address of thing I want = my current address + known fixed offset. Well, simple maths is all relative! "My current address" may or may not be easy to find. Consider the following: So foo will be in data, at a fixed offset from the code in function, and all we need to do is find it! On amd64, this is quite easy, check the disassembly: This says "put the value at offset 0x2002b2 from the current instruction pointer (%rip) into %eax. That's it we know the data is at that fixed offset so we're done. i386, on the other hand, doesn't have the ability to offset from the current instruction pointer. Some trickery is required there: The magic here is __i686.get_pc_thunk.cx. The architecture does not let us get the current instruction address, but we can get a known fixed address the value __i686.get_pc_thunk.cx pushes into cx is the return value from the call, i.e in this case 0x414. Then we can do the maths for the add instruction; 0x115c + 0x414 = 0x1570, the final move goes 0x18 bytes past that to 0x1588 ... checking the disassembly i.e., the value 100 in decimal, stored in the data section. We are getting closer, but there are still some issues to deal with. If a shared library can be loaded at any address, then how does an executable, or other shared library, know how to access data or call functions in it? We could, theoretically, load the library and patch up any data references or calls into that library; however as just described this would destroy code-sharability. As we know, all problems can be solved with a layer of indirection, in this case called global offset table or GOT. Consider the following library: Note this looks exactly like before, but in this case the foo is extern; presumably provided by some other library. Let's take a closer look at how this works, on amd64: The disassembly shows that the value to be returned is loaded from an offset of 0x200271 from the current %rip; i.e. 0x0200828. Looking at the section headers, we see that this is part of the .got section. When we examine the relocations, we see a R_X86_64_GLOB_DAT relocation that says "find the value of symbol foo and put it into address 0x200828. So, when this library is loaded, the dynamic loader will examine the relocation, go and find the value of foo and patch the .got entry as required. When it comes time for the code loads to load that value, it will point to the right place and everything just works; without having to modify any code values and thus destroy code sharability. This handles data, but what about function calls? The indirection used here is called a procedure linkage table or PLT. Code does not call an external function directly, but only via a PLT stub. Let's examine this: So, we see that function makes a call to code at 0x4d0. Disassembling this, we see an interesting call, we jump to the value stored in 0x200382 past the current %rip (i.e. 0x200858), which we can then see the relocation for the symbol foo. It is interesting to keep following this through; let's look at the initial value that is jumped to: Unscrambling 0x200858 we see its initial value is 0x4d6 i.e. the next instruction! Which then pushes the value 0 and jumps to 0x4c0. Looking at that code we can see it pushes a value from the GOT, and then jumps to a second value in the GOT: What's going on here? What's actually happening is lazy binding by convention when the dynamic linker loads a library, it will put an identifier and resolution function into known places in the GOT. Therefore, what happens is roughly this: on the first call of a function, it falls through to call the default stub, which loads the identifier and calls into the dynamic linker, which at that point has enough information to figure out "hey, this libtest.so is trying to find the function foo". It will go ahead and find it, and then patch the address into the GOT such that the next time the original PLT entry is called, it will load the actual address of the function, rather than the lookup stub. Ingenious! Out of this indirection falls another handy thing the ability to modify the symbol binding order. LD_PRELOAD, for example, simply tells the dynamic loader it should insert a library as first to be looked-up for symbols; therefore when the above binding happens if the preloaded library declares a foo, it will be chosen over any other one provided. In summary code should be read-only always, and to make it so that you can still access data from other libraries and call external functions these accesses are indirected through a GOT and PLT which live at compile-time known offsets. In a future post I'll discuss some of the security issues around this implementation, but that post won't make sense unless I can refer back to this one :)

While working on dictionaries for StarDict today, I've decided to finally bring them to openSUSE as well. First step was to fix my favorite tool to access them - sdcv. The fixed package is now in my home project and will be hopefully soon accepted to Education project so that it is more visible (see request 69901 if it is already the case). When I had the tool working, it was time to package dictionaries themselves. It was quite simple task and the rpms are available in my home project. To install packages on openSUSE 11.4 simply do:

zypper ar http://download.opensuse.org/repositories/home:/Nijel:/stardict/openSUSE_11.4/ stardict
zypper ref
zypper in  stardict-czech  stardict-english-czech  stardict-german-czech

Filed under: English Stardict Suse 0 comments Flattr this!

As announced on my French blog, I m attending Solutions Linux this week (10-12 May). I ll be on the Debian booth with a few other members of Debian France. If you re in Paris during those days, make sure to come by. I ll be pleased to meet you, and we ll be a bunch of Debian contributors ready to answer your questions. And if you feel like it, feel free to stop for a few hours and give us some help at the booth. It tends to be crowded and we re never enough to answer the questions. As usual, I ll come with a copy of the book Cahier de l Admin Debian (the one that I want to translate into English) to show, and I ll be glad to dedicate the book to whoever brings his copy with him/her. While speaking of conferences, this summer I m going to Banja Luka for DebConf 2011. I ve bought my plane tickets and I ll be there the whole week (24-31 july). Joining a DebConf is a completely different experience. DebConf is made by Debian people for Debian people. It s an opportunity to meet many of the people that you mainly know over IRC/email. And you don t need to be a hardcore Debian contributor to join. In fact, it s a great experience for anyone who just started contributing to Debian. Read this email of Asheesh Laroia as a proof. Note that the sponsored registration period has been extended up to the 17th of May. Register now! I m looking forward to DebConf11. On request of Stefano, I registered a Debian Rolling Bof. It will be a good occasion to see how far we are and to discuss future plans.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Today we had a very interesting demonstration. People who know me for some time know I like to be a part of demonstrations, adding my voice to the people. And even though I know most demonstrations, as massive as they might be, have no real and tangible effect... Well, I try to be there. Today we had a large demonstration in Mexico City of people angry with the government's tactics on the war on drugs Have you been to Mexico before 2007? You will remember we have always had our problems, but violence was basically limited to keeping an eye open, not flashing your money/jewels, and not looking too touristy. We were, all in all, a quiet country Known even as "the country where nothing happens, even if it happens". Five years later, a good portion of the country lives really in war-like conditions, and close to 45,000 people have been killed. At some point about a year ago (where the body count was at ~22,000), our oh-so-bright de-facto president said that "only" 10% of them were civilians. If that is still the case, and if proportions are held, we can now talk about ~4,500 people killed just because they were there Oh, but remember the country is not at war, and death penalty has very long been abolished, so we cannot give the other 90% a different status. We have had over 45,000 killed people. That means at least ~200,000 torn lifes. About six weeks ago, the cry from a father who lost his son started getting heard. And no, I do not want to talk about specific names or anything like that It is just the voice-bearer of many people who are fed up. Four days ago, he and a group of people left Cuernavaca city on a caravan and walked to Mexico city. Yesterday evening they arrived to the National University (UNAM), and were greeted by an amazing open-air concert: Mozart's Requiem. Today, our day started very early. My mother wanted to help, as many more anonymous citizens did, so we started making and packing sandwiches and apples at ~5:30AM, delivered them to the camp where they were staying and grouping with other people, and came back home for our breakfast. Later, at around 9AM, as the demonstration started going by our house (we live a block from UNAM), my girlfriend, father and me started walking with them. Why do I say this was a different demonstration than others I have been to?

1. Much longer. We walked for close to 16Km, for about five hours. Most demonstrations I have been part of are 3-5Km long
2. A different kind of support. I was surprised, even laughed at first when I saw a lady standing by us with a sign that read, I cannot join you today, but I'm there in spirit . But then I saw more, more people that just went out of their houses to a point close to where we walked on, and just stayed there on the streets. Not walking, yes, but being part of it.
3. The march was silent And yes, this is the first time I see it for true. At the very beginning we heard some people chanting, cheering, but the cheerers were requested not to be noisy. People wearing political parties' logos were requested not to wear them (as political parties are at an all-time low prestige). No, we were not silent for the whole 5 hours; we were talking with each other, but it felt... just very different from any other times.
4. Not only were people supporting, but this is the first time I saw this amount of people wanting to materially help. As I said, my mother delivered a little breakfast for 50 people, for those that slept at the camp, for those that had already walked ~70Km. But all along the road we were offered oranges, cookies, water, bread... Whatever people were able to offer, to somehow help. Very nice feeling of solidarity.
5. Contrary to all other times, we were really spread out, it was a very low density walk, os it was quite pleasant. Of course, I have not even seen an estimation of the number of participants... And I doubt anyboy can be decently close to it

Will this change anything? I doubt it. I am happy that for the first time, the de-facto president (remember a large percentage of Mexicans still believe he lost the presidential election, and it was only due to his lack of legitimacy that he pushed the army to the streets to start this war on drugs) is recognizing that this might not be the best strategy but it is the only one he has... People are still, despite what I saw many felt, very far from organized. And, yes, no short-term concrete proposals are made on how to stop the killing I feel the population agrees that drugs should be legalized and regulated, that would at least shift the problem, reduce the direct violence (and money) related to it... But, of course, it is plainly not feasible with our current reality. Anyway, after a 5hr walk today (and even after a nice nap and shower), I don't want to dig into hypotheses anymore. I just wanted to share about a very interesting and different experience I had the opportunity to be part of.

One of my last blog suggested a way to systematically clean up versioned dependencies in packages. Given the comments posted thereafter, it s clear that the one-size-fits-all strategy is ill-advised. Sorry about that. So, should these versioned dependencies be cleaned up or not ? Well It depends. Cleaning up versioned dependencies will hurt people when the package you re working on may be back-ported to old version of Debian. In corporate world, upgrading a machine may cost a lot, so you can still find production machine running etch. I would not be surprised to hear about older systems still used in production. In consumer world (mostly desktop or multimedia), people will mostly run more recent versions of Debian. All in all, the following elements must be considered before removing old versioned dependencies:

• Is the package likely to be installed on server ? or is it more targeted to consumer world ?
• What is the policy of the maintenance group of this package ? For instance, Debian perl group policy is to clean up versioned dependencies older than old-stable.

Unfortunately, there s no universal strategy. In case you want to clean up old dependencies, the latest version of Dpkg::Control model for Config::Model will be helpful. The last version of this models provides a bunch of meta Dpkg parameters. These parameters are used to customize the behavior of the Dpkg editor tool to suit your needs (For more details, see Config::Model::models::Debian::Dpkg::Meta). The dependency meta parameters will enable you to tune the behavior of config-edit to your needs. There are 3 dependency parameters, from the most global to the most specific ones:

• dependency-filter: your personal general policy
• group-dependency-filter: the policy of the maintainer group of your package
• package-dependency-filter: filter policy specified per package

Before applying a cleanup, config-edit will check the policies in the above order. The most specific one will be applied. Ie. the group policy will supersede your personal policy and the package specific policy will override the group policy. Here s a screenshot of config-edit GUI invoked on a package: From the parameters set in the GUI above, you can see that a warning will be raised for :

• perl package with versioned dependencies older or equals than lenny
• any other package with versioned dependencies older or equals than etch

Last but not least, if you want to use config-edit without any dependency filtering, just make sure that all these filter parameters are empty. You will still benefits from other features, like warning if a package dependency is unknown, which is useful to catch typos. All the best

It is a fundamental part of my belief structure that computers should be silent, unless told otherwise. A default install of the operating system should not make any sounds, unless there is a really good reason to. Thus, I object to the following:

• GDM beeping when it is ready to let me log in
• login sounds
• sound effects whenever I do anything, such as open or close windows
• sound effects whenever the computer wants to tell me something that isn't particularly important, such as new e-mail, IRC or IM messages to me, the networking going up or down, or a hard disk going missing
• a beep when I reboot or shutdown the computer

I'm sure there are situations when it would be OK for my computer to make noise to alert me. For example, supposed I'm asleep, and a zombie has penetrated my defensive perimeters and is about to chop off the top of my skull so it can devour my brain. In that case, I would be OK with my computer beeping. Once. Some remedies:

• With gdm3, edit /etc/gdm3/greeter.gconf-defaults to set /desktop/gnome/sound/event_sounds to false
• Configure GNOME's sound theme to be "No sounds". (It should not be necessary to set "alert sounds" to muted, but it won't hurt either.)
• Run amixer -q sset Beep off to shut up wall and other such tools. (I made an init.d script to make that happen automatically, and to make sure I don't forget it when I install new machines for myself.)

Now, it's possible that other people might like or need their computer to make sounds. I imagine, for example, that the GDM login-ready beep is to help people with vision problems. I'm all for that. However, I wish it be easier to shut things up in one simple place for those of us who don't need it. Edit: Thanks to NP237, noshadow, lindi-, flightplan, and others on #debian-devel with help on shutting up wall.

This June, soon after my seventh anniversary with Canonical, I will finish my job as Ubuntu CTO at Canonical. You can read my official announcement on the Canonical blog. It s sometimes hard to believe that it s been that long since we first envisioned a Linux desktop for human beings . When I look at how far we ve come, at the difference we ve made to so many people, it s easy to see how the time passed so quickly. It s been an incredible experience for me to play a part in building Ubuntu and Canonical. Whether building a decentralized company of hundreds of people, to a global community project of tens of thousands of members, I m grateful for all of the learning opportunities along the way. It has been a privilege to work with so many brilliant, passionate and thoughtful people under such auspicious circumstances. There is much I will miss, and I have many memories to enjoy, from all-night global hacking sessions driving toward a ship date, to casual singing and playing music at our many face-to-face events Nonetheless, it is time for me to seek out new challenges and stretch myself in new ways. I ll be moving back to the US, closer to old friends and family, and starting a new job with a different type of company. I am leaving behind a capable and dedicated team at Canonical, who I am confident will achieve even greater things in the years to come. I will remain active in the free software community as a volunteer. I intend to continue to participate in Ubuntu, and to serve on the Technical Board. I will also be continuing my work with Debian and the DEX project. This year, I ve accepted advisory positions with the Ada Initiative and the Freedom Box Foundation, and will continue to support those organizations and their missions. I ll be in Budapest at the Ubuntu Developer Summit for the next week, and look forward to seeing my friends in Canonical and the Ubuntu community. I ll have more to say soon about what I ll be doing next professionally. Watch this space for updates!

As previously announced, I ve been working on a Debian packaging tutorial. It is composed of about 60 slides providing a throughout overview of Debian packaging. It now reached the point where I consider it ready for use, and I am looking forward to reviews and comments.
The document is split into 4 different PDFs:

• the tutorial itself
• a practical session about modifying the grep package
• a practical session about packaging the gnujump game from scratch
• a practical session about packaging a Java library

And of course, it can be found on git.debian.org:
git clone git://git.debian.org/~lucas/packaging-tutorial.git

Since this has been a major request from users for a long time, I can only cool with the idea of seeing the Debian project support a rolling release. However I m not pleased with the proposed ideas, since they don t actually include any serious plan to make this happen. Sorry guys, but a big GR that says We want a pony rolling release to happen doesn t achieve anything. Let me elaborate. First of all, discussions have focused a lot on what to do when we re in a freeze phase. Numerous cool ideas have been proposed, including PPAs (which again, won t happen until someone implements them). This is all good, but this is only the tip of the iceberg. Above all, before wondering what can happen in a freeze that lasts 20% of the time, let s wonder what can happen for the 80% remaining time. Once you have something that works in the regular development phase, you can tune it to make it happen, even if in a less optimal way, when the distribution is frozen. So let s not put the cart before the horse. There are three options if you want to make a rolling release happen.

1. Make unstable usable. to make it happen, you have to prevent the disasters that rarely but unavoidably happen here. You don t want to make all rolling systems unusable because someone broke grub or uploaded a new version of udev that doesn t work with the kernel.
2. Make testing usable. This sounds easy since RC-buggy packages are already prevented from migrating, but actually it is not. A large number of RC bugs are discovered at the time of testing migration, when some packages migrate and others don t. Worst of all, they require several days to be fixed, and it is very often that they require several months, when one of the packages gets entangled in a transition.
3. Create a new suite for rolling usage.

The proponents of the CUT project obviously believe in option 2. Unfortunately, I haven t seen many things that could make it happen. A possible way to fix the situation would be to run large-scale regression testing on several upgrade paths. I don t know if there are volunteers for this, but that won t be me. That would also imply to make a lot of important bugs RC, since they could have a major effect on usability, but the release team will not be keen to make it happen. Because of the testing situation, when someone asks me for a rolling release, I point her to unstable with apt-listbugs. As of today, this is the closest thing we have to a rolling release, so we should probably examine more deeply option 1. Is it that complicated to write a tool to prevent upgrades to broken packages? A 2-day delay in mirror propagation and a simple list of broken packages/versions (like the #debian-devel topic, would be enough. Add an overlay archive, that works like experimental, and you can now handle freezes smoothly. Wait isn t that aptosid? We would probably gain a lot of insight from the people who invented this, instead of trying to reinvent the wheel. Finally, option 3 could open new horizons. There s a risk that it might drive users away from the testing and unstable suites, and this makes us wonder how we could have proper testing for our packages. Still, build a process that would (and that s really only an example) freeze unstable every month, give people 10 days to fix the most blatant issues, add a way to make security updates flow in from unstable, and you have a really nice rolling distribution. So overall, it only requires people to make things happen. You want option 2 to happen? Instead of working on GR drafts, start working with maintainers and release managers on ways to avoid breakage in testing. You want option 3 to happen? Start it as a new .debian.net service and see how it works. Personally, I d be in favor of offering aptosid developers to become DDs and offer their solution as a Debian service. It would bring in new people rather than driving away existing developers from working on our releases.

This is my monthly summary of my Debian related activities. If you re among the people who support my work, then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. GNOME 3 packaging Right after the GNOME 3 release, I was eager to try it out so I helped the pkg-gnome team to update some of the packages. I did some uploads of totem, totem-pl-parser, gvfs, mutter, gnome-shell, gnome-screensaver. I also kept people informed via my blog and prepared a pinning file for adventurous users who wanted to try it out from experimental (like me). One month later, I m still using GNOME 3. There are rough edges still, but not so many. And I m starting to get used to it. Debian Rolling planning Debian Rolling is a project on my TODO list for quite some time. I decided it was time to do something about it and started a series of articles to help clarify my ideas while getting some early feedback. My goal was to prepare a somewhat polished proposal before posting it to a Debian mailing list. But as usual with Murphy s law, my plan did not work out as expected. Almost immediately after my first post the discussion started on debian-devel:
At this point it s a discussion thread of several hundreds of messages (there are several screens of messages like the one above). Many of the sub-threads have been interesting, but the general discussions mixed too many different things so that there s no clear outcome yet. Lucas Nussbaum tried to make a summary. Obviously I must adjust my plan, there s lots of feedback to process. I accepted to drive a DEP together with Sean Finney to help structure the part of the discussion that focuses on allowing development to continue during freezes. But I m also eager to fix the marketing problem of testing and have the project recognize that testing is a product in itself and that end-users should be encouraged to use it. Package Tracking System maintenance The Package Tracking System is an important tool for Debian developers, and it has been broken by some change on the Bug Tracking System. I worked around it quite quickly so that few people noticed the problem but Cron kept reminding me that I had to properly fix it. I ended up doing it last week-end. While working on the PTS, I took the opportunity to merge a patch from Jan Dittberner to enhance the news RSS feed that the PTS provides. And I also integrated information from backports.debian.org (thanks to Mehdi Dogguy for reminding me #549115). Multiarch update Not much new this month. I fixed two bugs in the multiarch dpkg branch thanks to bug reports from Ubuntu users (LP 767634, LP 756381). I m still waiting on Guillem Jover finishing his review of the multiarch branch. I m pinging him from time to time but it looks like multi-arch is no longer in his short term priority list. I ve been running this code for more than 2 months and it works fine. I want to see it merged. I m ready to update my code should anything need to be changed to please Guillem. But without any feedback we re in a deadlock. Misc dpkg work While fixing a bug in update-alternatives (found in one of the valid reports on launchpad), I noticed that there was room for improvements in the error messages output by update-alternatives. I changed them to reuse the same strings that were already used in other parts of dpkg. The result is that there are a few strings less to translate (always a nice thing for the poor translators who have to deal with the thousands of strings that dpkg contains). I also tried to fix some of the most cryptic error messages in dpkg (see #621763) but that work is stalled at the request of Guillem. Book update We (me and Roland Mas) are almost done with the update of our French book for Debian Squeeze. It will hit the shelves in July or September. I m starting to prepare the fundraising campaign to make an English translation of it. We ll use ulule.com for this. On my blog I have been pleased to interview Meike Reichle, it s the first women that I have interviewed in the series but it s certainly not the last one. I also interviewed Adam D. Barratt, one of our tireless release managers. Thanks Many thanks to the people who gave me 180.35 in March and 235.37 in April. That represents 1.5 and 2 days of work for those months. See you next month for a new summary of my activities.

8 comments Liked this article? Click here. My blog is Flattr-enabled.

There seem to be lot of momentum in discussions about making Debian testing more attractive to users on debian-devel mailing list (see Lucas's summary for summary of the lengthy discussion). One of the raised points there is that Debian misses equivalent of Ubuntu's PPA or openSUSE build service. As Debian is usually about do-ocracy, it will be there once it will be implemented, but I still don't think it is always useful to have such thing. The problem with both (but this is especially the case for openSUSE build service) is that you need to add dozens of home: projects with questionable quality of the software. This usually leads to horrible mess, especially in case of incompatible libraries with different build options (anybody using wxWidgets?). It can definitely lower barrier for people to create packages for the distro, but is it really good to lower barrier that much? I'd really prefer new things to be uploaded to Debian itself without need to use any external repositories, it would ensure at least some level of quality and compatibility with other packages.

Filed under: Debian English Suse 5 comments Flattr this!

Here is my attempt at a summary of the rolling discussion currently happening on debian-devel@. It might not be complete, it s probably a bit biased, but I hope that it s still better than nothing. It was also posted on debian-devel@. If you are involved in Debian development, please discuss it on debian-devel@, rather than in the comments of this blog. If you want to provide feedback about the Debian rolling idea, use this Doodle poll and/or the comments of this blog.
Update : Someone had the good idea to vandalize the poll, and remove all the votes from people in favor of rolling. Before the vadalism (log), the results were:
37 votes: I am a Debian user, I use testing or unstable, and I would like rolling to happen.
6 votes: I am a Debian user, I use testing or unstable, and I DON T want rolling to happen.
17 votes: I am a Debian user, I use stable, and I would like rolling to happen.
3 votes: I am a Debian user, I use stable, and I DON T want rolling to happen.
7 votes: I am NOT a Debian user, but I would probably use it if rolling happened.
1 vote: I am NOT a Debian user, and I don t care about rolling. Motivations There s some user demand for rolling releases. For evidence, one can look at the usage of Debian testing or unstable which clearly goes further than the Debian development community. Or at the quickly growing market share of ArchLinux. Or at the interest in LinuxMint and Aptosid. Or at the DPL s report of his interactions with the press. Debian s only official product is its stable releases. While it s a clearly great and useful product, many users and developers don t recognize themselves in it: it contains software that is too old for their needs. The success of Ubuntu is related to this problem: Ubuntu proposes a different compromise between stability and up-to-date-ness. While concurrencing Ubuntu with more frequent stable releases (released every 6-months, for example) doesn t look like the right thing to do, Debian is in a perfect position to provide a rolling release with marginal additional work, since Debian already has testing (and unstable) to build on. The rolling discussion investigates how we could endorse the concept of a rolling release, provided as a product in addition to stable releases. This rolling release would be based on the current testing branch. Benefits for Debian:

• Attract users who think that testing is only a development branch, and want newer software than what one finds in stable. Those users are likely to be rather advanced users (free software developers and contributors), thus interesting to work with (able to submit good-quality bug reports, etc). Some of them could also become Debian contributors. And even if they don t, more users of testing/rolling means more testers of the next stable release [remember how the bug reporting rate of Ubuntu is higher than Debian's -- some areas of Debian could use more testers].
• Give back to the free software world by providing a platform where new upstream releases would quickly be available to users. Since users would be able to test new upstream releases earlier, they would be able to provide feedback to upstream devs earlier, contributing to a shorter feedback loop. Debian is often identified by upstream developers as the platform with releases from two years ago. I would love to see Debian in a position to contribute more to improviing the quality of the Free Software world.
• Get back some attention that is currently taken away from Debian by derivatives. This is important to carry our political or technical messages, which are not necessarily carried out by our derivatives.

Current proposed plan for rolling (Disclaimer: this is mostly my view. It is shared by others, but some details might not be) rolling is mostly about (external) communication. It is not expected to change our development processes fundamentally.
It would be a statement by the project (through a GR, for example). A very preliminary draft was proposed by Raphael Hertzog:

Title: Debian endorses usage of testing by end-users, and renames it to rolling The Debian project recognizes that the Debian testing distribution initially created to make it easier to prepare and test the next stable release has become a useful product of its own. It satisfies the needs of users who are looking for the latest stable versions of software and who can cope (or even appreciate) a system that s constantly evolving. The Debian project decides to endorse this usage and will strive to provide a good experience to users of testing . To better communicate this policy change to our users, testing will be renamed rolling .

Yes, it s mostly PR bullshit , and I don t expect it to significantly change Debian development processes. However, communication is necessary if we want to attract new users. What would change is more attention from developers to what happens in testing/rolling, which is a good thing since a better testing/rolling makes it easier to create stable releases from it. Open questions Most of the discussion is about the influence of the introduction of rolling on Debian development processes, and in particular, on the painful process resulting in stable releases. Many fear that, with rolling , it will be harder to get stable releases out. The root question is: if we do rolling, what do we do during freezes? Several options have been mentioned in the thread:

1. We could decide not to do anything special: just freeze rolling for ~6 months, as we used to freeze testing. That might bore people who like the constant flux of new upstream releases, but if we decide that it s the only way to ensure high-quality stable releases, so be it.
2. We could decide to fork a frozen branch when we freeze, and continue to manage rolling using migrations from unstable.
3. We could mix both solutions: freeze rolling for 3 months, so that most of the stabilization work occurs with a single active branch, and then, for the final release preparation, fork frozen off rolling, and unfreeze rolling.

Two kinds of objections have been raised:

• Those against the rolling concept:
  • It s only about PR, Debian isn t about PR. Answer: PR does matter sometimes, especially if we want to attract users.
  • There s usually no installer for it, other than installing the latest stable release and dist-upgrading, which doesn t always work. Answer: True; but it sounds like an acceptable problem. And if upgrades from the latest stable fail, it s an RC bug, so we would like to know. And if we do get d-i betas, it s a great way to get user testing for them.
  • It will split the developers base between supporting rolling and supporting stable releases (which also need to be supported after they have been released). Answer: already the case today.
  • Testing is not targeted at end users, but is a tool for the release team to create stable releases. It needs to stay that way. Answer: really, can t we do both?
  • Renaming testing to rolling will require changes in many parts of Debian infrastructure. Answer: some problems can be mitigated by keeping a testing symlink. The remaining impact needs to be evaluated.
• Those against the two development branches during freezes problem:
  • It splits the users and developers base between two branches (less users means less bug reports ; less developers means less bug fixing). Answer: true.
  • It requires the use of two different entry points for packages (unstable for packages targeted at rolling, frozen-proposed-updates for packages targeted at frozen). Answer: true.
  • All in all, huge overhead for the release team. Answer: true.
  • Overhead for developers, who need to support two targets. Answer: true.
  • Just after a stable release, we would start the next release from rolling, instead of stable. So we would start from a less clean base. Answer: true.
  • It s even worse if you consider staged freezes (freezing base packages earlier than the rest of the distribution, for example). Answer: true; but is it really a problem if some base packages are frozen in rolling for a few months?

Other things that were discussed:

• possible changes to processes around testing to make it more usable (reduce the set of architectures required for migration to testing ; allow/encourage usage of t-p-u to rebuild unstable packages that are ready to transition except for the fact that they are entangled in a transition ; have different level of RC bugs: there are RC bugs that are acceptable in rolling that are not acceptable in stable)
• PPAs for Debian
• Developer activity during freeze (developer temporarily stopping to work on Debian during freezes)
• Let s improve our packaging process or reduce the duration of our freezes before introducing rolling. Answer: but then, shouldn t we also stop doing stable releases for a while? ;)
• Setting up an official rolling instance . Answer: that can t work. detailed answer
• Using unstable instead of testing as the basis for rolling. Answer: there seems to be more demand for something similar to testing, than for something similar to unstable.

Tentative personal conclusion I have the impression that advertising testing as a rolling release usable by end-users is generally considered a good thing.
The renaming of testing to rolling is not as consensual, but most opponents have a whatever ; if you want position. How we deal with freezes is the hard point in this discussion. I m personnally in favor of the freeze rolling for 3 months, then fork frozen and unfreeze rolling plan, though it has some problems too (it is not clear whether the required manpower really decreases at the end of freezes). Where do we go from there? After another round of discussion, we might postpone the how to deal with freezes question to later, and proceed with a GR to measure the support for the testing for end-users + s/testing/rolling/ proposal.

This week I have a shot I took while visiting Beeston Castle. This is the start of the path leading up to the castle itself.

This week I have a shot I took while visiting Beeston Castle. This is the start of the path leading up to the castle itself.

Last week, I announced the activation of Uyghur language in Debian Installer. For once, I even went further: I rebuilt all D-I packages that include localization (at least for i386/amd64 architectures) and uploaded them. As a consequence, D-I daily builds now allow previewing how an install done in Uyghur can be like. For this, please go to the Debian Installer development page, download the netboot ISO image (not netinst) and launch D-I with "suite=unstable" passed as argument when booting the ISO image. You should end up with something like this, this or this. There are a few glitches, still: for instance, as the font we chose has no Bold variant, the screens "titles" do not use an Uyghur font but an Arabic or Persian one. As a consequence, glyphs ar enot properly joined together. I'm investigating this issue with Uyghur translators and we're trying to choose an appropriate font among the gazillion that have been created by UKIJ. Please notice that Uyghur looks like Arabic but, according to those who can read Arabic, is really different from Arabic (or Persian). I hope that this great work by Gheyret Tohti, Abduqadir Abliz and other people working on this will give ideas to other so-called "minorities", here or there in the world, who want to be able to install Debian in their language. There is still a lot of work to do, such as having end-user software localization for Uyghur and, then, a uyghur-desktop task, but this is at least the first language that gets added for wheezy. In case you would be wondering, the current count of activated languages in D-I is 71.

It s Autism Awareness Month April is Autism Awareness month, there are lots of articles about Autism, most of them are at best misleading if not wrong. The most common error seems to be describing non-verbal Autistic children and then claiming that Autistic people comprise about 1% of the population. The number of non-verbal Autistic children is a lot less than 1% of the population, and at least 1/3 of the people on the Autism Spectrum have good verbal skills and often talk more than average. To be precise, they are describing the symptoms of people who would be diagnosed with Severity 3 according to the American Psychiatric Association s Diagnostic and Statistical Manual (the DSM) as applying to all people who are on the Autism Spectrum even though of the currently diagnosed people probably about 1/3 might be classified as Severity 2 and 1/3 as Severity 1. It seems that severity 1 Autism is going to be the replacement for Asperger Syndrome in the new DSM-5. The problem with misleading information in regard to what comprises Autism is that parents really don t want to have a child who s abnormal and search for evidence to make their wishes come true. At the extreme there are parents who can convince themselves that a 4yo non-verbal child isn t on the Autism Spectrum. When people are given the impression that Autism is only about non-verbal children then they will probably think that a 6yo who can give a half-hour lecture on a topic of interest (regardless of whether the audience is interested) isn t on the Autism Spectrum. There is also a similar issue with undiagnosed adults, I would have probably got assessed at least a decade earlier if I had learned more about Autism instead of believing some of the stories. In a quick Google search for Autism Awareness Month I haven t found any clear and accurate information on what Autism is or what should be done about it, so it seems to me that the awareness project has largely been a failure. The Autism page on Wikipedia [1] and the page for Asperger Syndrome [2] are good places to start reading. By merely pointing to those two Wikipedia pages I ve made this post a lot more useful than every article I ve seen in the mainstream media. To find some useful and insightful posts about Autism awareness you have to add some search terms such as neurodiversity, which turns up posts such as a post by Leah Jane about Autism Acceptance [3]. It seems that the Autism Awareness articles that are most useful at informing people will only be found by people who are already aware of some of the issues. Most People Who Read this are Geeks The majority of the readers for my blog come from aggregators that cater to fairly geeky people. Geek culture is largely Aspie culture. So I think that most people who read my blog have a good general knowledge of what people who might be diagnosed with Asperger Syndrome or High Functioning Autism are like they just think that such people are merely a bit more geeky than is typical for the Free Software community. So it s not as if all Autistic people are hidden away somewhere because they can t live independently. Many of us work in the computer industry and in academia. Probably the majority of people who have a university degree related to science or engineering have had a professor who was on the Autism Spectrum, probably most people who have any significant experience in the computer industry have had colleagues who are on the Spectrum. After having become used to the geek culture I initially found it difficult to believe that I could be on the Autism Spectrum because I knew heaps of people in the Free Software community who are at least as geeky as me but apparently weren t on the Autism Spectrum. I now think that the majority of people who are as geeky as me would get diagnosed with Asperger Syndrome (or High Functioning Autism under DSM-5) if they were assessed. Adult Diagnosis If you are wondering whether you might be on the Autism Spectrum then it s worth reading the pages linked from the DSM 5 draft process have the full diagnostic criteria (both for DSM-4 and DSM-5) [4]. Also Glenn Rowe has a web page with tests designed by Professor Simon Baron-Cohen [5] and Leif Ekblad has a project to do long-term analysis of changes to personality test scores which starts with a quick Aspie quiz [6] (and also has lots of links to other relevant sites). If you think that you might be on the Autism Spectrum (which means Asperger Syndrome in the case of most people who read my blog) then I recommend that you get assessed if possible. As assessment by a psychologist isn t cheap there is the option of self-diagnosis in which case Chaotic Idealism s post about self-diagnosis is worth reading [7].

• [1] http://en.wikipedia.org/wiki/Autism
• [2] http://en.wikipedia.org/wiki/Asperger_syndrome
• [3] http://www.shiftjournal.com/2011/04/15/autism-acceptance-for-autism-awareness-month/
• [4] http://www.dsm5.org/ProposedRevisions/Pages/proposedrevision.aspx?rid=97
• [5] http://glennrowe.net/BaronCohen.aspx
• [6] http://www.rdos.net/eng/Aspie-quiz.php
• [7] http://chaoticidealism.livejournal.com/70133.html

forked-daapd v0.16 is a bugfix release, mainly addressing a bad regression introduced in v0.15. Fixes:

• DAAP songlist off-by-one with sort tags
• handling of non-UTF-8 tags under some circumstances
• file size in HTTP streaming

The DAAP bug is what causes an incorrect display and sorting in iTunes, the tags issue could cause a segfault in some cases, and finally fixing the file size in the HTTP streaming code may or may not fix some issues but was a bad bug anyway. Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ ; GPG signatures made with my Debian key F5D65169.

When building a user space binary, the -lc that gcc inserts into the final link seems pretty straight forward link the C library. As with all things system-library related there is more to investigate. If you look at /usr/lib/libc.so; the "library" that gets linked when you specify -lc, it is not a library as such, but a link script which specifies the libraries to link, which also includes the dynamic linker itself: On very old glibc's the AS_NEEDED didn't appear; so every binary really did have a DT_NEEDED entry for the dynamic linker itself. This can be somewhat confusing if you're ever doing forensics on an old binary which seems to have these entries for not apparent reason. However, we can see that that /lib/libc.so itself does actually require symbols from the dynamic linker: Although you're unlikely to encounter it, a broken toolchain can cause havoc if it gets the link order wrong here, because the glibc dynamic linker defines minimal versions of malloc and friends -- you've got a chicken-and-egg problem using libc's malloc before you've loaded it! You can simulate this havoc with something like: Depending on various versions of things, you might see that assert or possibly just strange, corrupt output in your logs as syslog calls the wrong malloc. You could debug something like this by asking the dynamic linker to show you its bindings as it resolves them: Above, because the dynamic loader comes first in the link order, libc.so.6's malloc has bound to the minimal implementation it provides, rather the full-featured one it provides internally. As an aside, AFAICT, there is really only one reason why a normal library will link against the dynamic loader -- for the thread-local storage support function __tls_get_addr. You can try this yourself: Thread-local storage is worthy of a book of its own, but the gist is that this support function says "hey, give me an address of foo in libtls.so", the magic being that if the current thread has never accessed foo then it may not actually have any storage for foo yet, so the dynamic linker can allocate some memory for it lazily and then return the right thing. Otherwise, every thread that started would need to reserve room for foo "just in case", even if it never cares about moo. But looking a little closer at the symbols of libc.so is also interesting. libc.so doesn't actually have many functions you can override. You can see what it is possible to override by checking the relocations against the procdure-lookup table (PLT). i.e. instead of jumping directly to the malloc defined in the libc code section, any internal calls will jump to this stub which, the first time, asks the dynamic linker to go out and find the address of malloc (it then saves it, so the second time the stub just jumps to the saved location). This is an interesting list, seemingly without much order. feof, for example, stands out as worth checking out a bit closer why would that be there when fopen isn't, say? We can track down where it comes from with a bit of detective work; knowing that the value of the symbol feof will be placed into 0x35b018 we can disassemble libc.so to see that this address is used by the feof PLT stub at 0x1e870 (luckily, objdump has done the math to offest from the rip for us; i.e. 0x1e876 + 0x33c7a2 = 0x35b018) From there we can search for anyone jumping to that address, and find out the caller: Which turns out to be part of a local patch which probably gets things a little wrong, as described below. The sysconf relocation is from a similar add-on patch (being used to find the page size, it seems). libc, like all sensible libraries, uses the hidden attribute on symbols to restrict what is exported by the library. The benefit of this is that when the linker knows you are referencing a hidden symbol it knows that the value can never be overridden, and thus does not need to emit extra code to do indirection just in case you ever wish to redirect the symbols. In the above, it appears that feof has never been marked as hidden, probably because no internal glibc functions used it until that add-on patch, and since it is not considered an internal function the linker must allow for the possibility that it will be overridden at run time and provide a PLT slot for it. There are consequences; if this was on a fast-path then the extra jumps required to go via the PLT may matter for performance and it may also cause strange behaviour if somebody had preloaded something that took over feof. Note, this is different from saying that your library can override symbols provided by libc.so; such as when you LD_PRELOAD a library to wrap an open call. What you can not override is the open call that say, the internal libc.so function getusershell does to read /etc/shells. Having the malloc related calls as preemptable seems intentional and sane; although I can not find a comment to the effect of "we deliberately leave these like this so that users may use alternative malloc implementations", it makes sense so that libc.so is internally using the same malloc as everything else if you choose to use something such as tc-malloc. tl;dr? Digging into your system libraries is always interesting. Be careful with your link order when creating toolchains, and be careful about symbol visibility when you're working with libraries.