This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (170 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. This month passed by very quickly since I attended both the Libre Software Meeting / RMLL and the DebConf. Libre Software Meeting / RMLL I attended only 3 days out of the 6 but that was a deliberate choice since I was also attending DebConf for a full week later in the month. During those 3 days I helped with the Debian booth that was already well taken care of by Fr d ric Perrenot and Arnaud Gambonnet. Unfortunately we did not have any goodies to sell. We (as in Debian France) should do better in this regard next time. One of the talks I attended presented EnVenteLibre. This website started as an online shop for two French associations (Ubuntu-fr, Framasoft). They externalize all the logistic to a company and only have to care about ordering goodies and delivering to the warehouse of the logistic company. They can also take some goodies from the warehouse and ship them for a conference, etc. We discussed a bit to see how Debian France could join, they are even ready to study what can be done to operate at the international level (that would be interesting for Debian with all the local associations that we have throughout the world). Back to the LSM, while I had 3 good days in Strasbourg, it seems to mee that the event is slowly fading out it s far from being an international event and the number of talks doesn t make for a better quality. BTW, do you remember that Debconf 0 and Debconf 1 were associated to this event while it was in Bordeaux? dpkg-source improvements During my time in Strasbourg (and in particular the travel to go there and back!) I implemented some changes to 3.0 (quilt) source format. It will now fail to build the source package if there are upstream changes that are not properly recorded in a quilt patch:

dpkg-source: info: local changes detected, the modified files are:
 2ping-1.1/README
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/2ping_1.1-1.diff.cki8YB

As the error message hints, there s a new --commit command supported by dpkg-source that will generate the required quilt patch to fix this. In the processe you will have to submit a name and edit the patch header (pre-formatted with DEP3 compatible fields). You can get back the old behavior with the --auto-commit option. Build flags changes Ever since we adopted the Ubuntu changes to let dpkg-buildpackage set some build related environment variables (see #465282), many Debian people expressed their concerns with this approach both because it broke some packages and because those variables are not set if you execute debian/rules directly. In the end, the change was not quickly reverted and we fixed the package that this change broke. Despite this we later decided that the correct approach to inject build flags would be a new interface: dpkg-buildflags. Before changing dpkg-buildpackage to no longer set the compilation flags, I wanted to ensure dpkg-buildflags had some decent coverage in the archive (to avoid breaking too many packages again). My criteria was that CDBS and dh (of debhelper) should be using it. With the recent debhelper change (see #544844) this has been reached so I changed dpkg-buildpackage accordingly. Makefile snippets provided by dpkg At the same time, I also wanted an easy way for maintainers not using dh or CDBS to be able to fix their package easily and go back to injecting the compilation flags in the environment but doing it from the rules files. Starting with the next version of dpkg, this will be possible with something like this:

DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

Without DPKG_EXPORT_BUILDFLAGS the variables are not exported in the environment and have no effect unless you use them somewhere. More than build flags, this will also provide a bunch of other variables that can be useful in a rules files: all the variables provided by dpkg-architecture, vendor related variables/macro and some basic package information (mainly version related). dpkg-buildflags improvements Given the renewed importance that dpkg-buildflags will take now that dpkg-buildpackage no longer sets the corresponding environment variables, I thought that I could give it some love by fixing all the open issues and implementing some suggestions I got. I also had a chat with a few members of the technical committee to discuss how hardening build flags could be enabled in Debian and this also resulted in a few ideas of improvements. In the end, here are the main changes implemented:

• new prepend directive to inject flags at the start (see commit);
• new strip directive to strip flags from the result returned by dpkg-buildflags (see commit);
• new environment variables DEB_flag_MAINT_directive that can be set by the maintainer to adjust what dpkg-buildflags will return (see commit);
• new --export=configure command to inject build flags on the ./configure command line (see commit);
• new --dump command that is the default (see #603435).

Will all those changes, the complete set of compilation flags can be returned by dpkg-buildflags (before it would only return the default flags and it was expected that the Debian packaging would add whatever else is required afterwards). Now the maintainer just has to use the new environment variables to ensure the returned values correspond to what the package needs. DebConf: rolling and hardening build flags I spent a full week in DebConf (from Sunday 24th to Sunday 31th) and as usual, it s been a pleasure to meet again all my Debian friends. It s always difficult to find a good balance between attending talks, working in the hacklab and socializing but I m pretty happy with the result. I did not have any goal when I arrived, except managing the Rolling Bof (slides and video here) but all the discussions during talks always lead to a growing TODO list. This year was no exception. The technical committee BoF resulted in some discussions of some of the pending issues, in particular one that interests me: how to enable hardening build flags in Debian (see #552688). We scheduled another discussion on the topic for Tuesday and the outcome is that dpkg-buildflags is the proper interface to inject hardening build flags provided that it offers a mean to drop unwanted flags and a practical way to inject them in the ./configure command line. Given this I got to work and implemented those new features and worked with Kees Cook to prepare a patch that enables the hardening build flags by default. It s not ready to be merged but it s working already (see my last update in the bug log). A few words about the Rolling BoF too. The room was pretty crowded: as usual the topic generates lots of interest. My goal with the BoF was very limited, I wanted to weigh the importance of the various opinions expressed in the last gigantic discussion on debian-devel. It turns out a vast majority of attendants believe that testing is already usable. But when you ask them if we must advertise it more, answers are relatively mixed. When asked if we can sustain lots of testing/rolling users, few people feel qualified to reply but those that do tend to say yes. More dpkg work Lots of small things done:

• I did again some bug triaging on Launchpad. But Brian Murray did a lot of it and the result is impressive, we re down to 154 bugs (from more than 300 a month ago!).
• I updated my multiarch branch multiple times. I was hoping to meet Guillem during DebConf to make some progress on this front but alas he did not attend. I have been asked a status update multiple times during my time in DebConf.
• I fixed a regression in update-alternatives (#633627), a test-suite failure when run as root (#634961), a segfault in findbreakcycle. There have been a bunch of minor improvements too (#634510, #633539, #608260, #632937).

Package Tracking System and DEHS Christoph Berg recently wrote a replacement for DEHS because the latter was not really reliable and not under control of the QA team. This is a centralized system that uses the watch files to detect new upstream versions of the software available in Debian. I updated the Package Tracking System to use this new tool instead of DEHS. The new thing works well but we re still lacking the mail notifications that DEHS used to send out. If someone wants to contribute it, that would be great! Misc packaging work I did some preliminary work to update the WordPress package to the latest upstream version (3.2). I still have to test the resulting package, replacing upstream shipped copies of javascript/PHP libraries is always a risk and unfortunately all of them had some changes in the integration process. I also updated nautilus-dropbox to version 0.6.8 released upstream. I also uploaded the previous version (that was in testing at that time) to squeeze-backports. So there s now an official package in all the Debian distributions (Squeeze, Wheezy, Sid and Experimental)! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

DebConf 11 -- Banja Luka, Bosnia and Herzegovina I'm writing this on my way back from my fifth DebConf, waiting in the Frankfurt Airport for my flight to Munich then another wait for a flight back to Los Angeles. DebConf is the annual convention of Debian Developers and other interested in how Debian Linux (or Debian BSD) works, and want to help make it better. Debian is an international volunteer project, and Debconf is held each year in a different city, normally not on the same Continent two years in a row. Only one so far has been held in the United States, last year Debconf 10 was in New York City. (Mexico and Canada have also hosted Debconfs.) DebConf is a week long conference, with one day designated as "Debian Day" with more introductory level talks, frequently in the local language, and local people are invited to learn about Debian. The rest of DebConf is held in English. This year had over 350 attendees from a couple of dozen countries. Unlike most computer conferences, DebConf is free to attend. (Professional and Corporate memberships are available for those who would like to financially support DebConf.) If you apply early enough, you can request help paying for food, lodging, and travel as well, but such support is limited and you have to justify it. This is all make possible by our generous sponsors, who include some major corporations who use Debian. This year DebConf was held in Banja Luka, Bosnia and Herzegovina. The Bosnian bid won partially based on the generous support of the Bosnian government, not only financial and letting us use one of their buildings for the conference, but also help with getting visas and other such aid. (Visas are not required from the US or most of Europe, but they are from some other countries.) I arrived on Saturday about 24 hours after I left home, starting with too little sleep. I actually managed to eat dinner before passing out in my hotel bed. The hotel room I had in Hotel Vidovic (accent on the c) was nicer than any I have stayed at in Europe (although I always chose by price), and the conference price was less than half what a similar room goes for at a US hotel. There was a nice breakfast buffet included in the hotel cost. (Hot omelets and sausages as well as the usual cold cuts, cheese, fruit, juice, coffee, tea, cereal, yogurt, etc.) The in-room ethernet connection worked all the time except one morning, although it was not particularly fast. The hotel is relatively new, the Lift (elevator) had a date of 2005 on it and I suspect that is when the building was constructed. Sunday was Debian Day, and I went to a couple of sessions before deciding to take a nap in the afternoon. My nap wound up being from 3pm to after 11pm, sleeping through dinner. I was awake for an hour or so checking my email, and then slept till 4am. That was the morning I made it to breakfast before 7am. This was the first time I ate breakfast at normal breakfast hours for a full week for many years. After breakfast Monday, I spent several hours at the front desk helping give out badges, conference bags, and t-shirts. Lunches and dinners were in the Hotel Bosna, across the street from venue. For lunches we were served soup, salad, a plate of food, and desert. For dinners there was a buffet. The food served at Bosna was fairly good, but repetitive from day to day. (I've had much worse hotel food in the US for much more money.) Tuesday was more or less a repeat of Monday, with me opening the front desk by myself and not nearly as many new arrivals. The sessions varied of course, and I played "Debconf Experimental 5-card Mao". (Mao is a card game, popular in Cambridge and at DebConf.) Wednesday was the Day Trip. One day at DebConf is dedicated to socializing while seeing some of the local culture of the hosting city/country. This year there was an option of rafting (with some white water) or visiting a historical monastery and hiking to a waterfall with several grain mills powered by diverted water. This is the first time I had seen one actually grinding grain. (It's mainly a historical tourist attraction, but I don't know what happens to the flower made.) We then met at the rafting location for a BBQ lunch. Apparently there was some miscommunicaiton and the rafting place was not prepared for 70 or 80 people who wished to go rafting, so some did not get to raft until after lunch, and I'm not sure all that wanted to did. Dinner was back at Bosna. Thursday was another normal conference day, with the new wrinkle of distributing food tickets since Bosna was not willing/able to follow our instructions on who should be allowed to eat at Debconf expense. Those who wished to dine with the other Debconf attendees could purchase the tickets from the front desk, although this was not well publicised and few people took us up on it. The special "Conference Dinner" was Thursday night. This was held at a restaurant on the 14th floor of a building a few blocks away, and was a fancier buffet. There didn't seem to be much difference between the first and second course, and they ran out of desert before many people got any. This one was not as memorable as the one in Mexico, but that one was for the wrong reasons, including an unplanned indoor waterfall. After the dinner I played Mao again, this time till 2am. Friday was the only day I did not arrive at the venue in time for the first session (10am). Saturday I opened the front desk by myself again. In the afternoon was the "Debbugs Skills Exchange" that was requested by several people, so Collin Watson, Ian Jackson and I gave some information about it and Don Armstrong (who has done most of the recent coding on debbugs) participated via voip. (Debbugs is software for the Debian Bug Tracking System, often called the BTS.) Collin and Ian are emeritus members of the BTS team, and not active in it, while I am handling the spam filtering and haven't done much with the rest of the BTS. This was in the small "Meeting Room" (rather than the Auditorium or the round room) so we don't have to worry about video archives of it being available. (All the sessions is the two main rooms were streamed live on the internet, and will be edited and archived on http://meetings-archive.debian.net/pub/debian-meetings/2011/debconf11/.) We created a mini intro to the BTS document at that session that should be cleaned up and published. Less than two hours after that talk, the conference was over and teardown was started. As usual, the teardown and cleanup of what we had set up over several days was done is several hours, with many people helping. Sunday was leaving day. I was one of the people on the 11am charter bus to Zagreb DebConf arranged. The two such charter busses were an excellent idea, since the normal busses and trains would not have been able to manage that many extra passengers. This actually took longer than the planned 4 hours due to the long lines at the border crossing on Sunday afternoon. Both Bosnia and Croatia check passports both coming and leaving, so it is checked on both sides of the river that is the border between the countries. The check is pretty cursory unless you are from a country that needs a visa. I then had a plane flight to Frankfurt, and am currently in the middle of my 11 hour layover here. After that I fly to Munich early in the morning, have an 8 hour layover, and fly to Los Angeles. This schedule was in order to get flights at something resembling reasonable cost when I did it. From what I saw of Banja Luka, it's a nice place to visit, although most of the people do not speak English. Vegetarians may have a hard time finding meals, and sometimes they try to serve fish to them. (Assuming vegetarian means "no meat".) Transportation to Banja Luka is a bit hard, there are only six or so flights per week to the local airport, and one of the airlines is not on the international ticket sites. Most DebConf attendees flew to Zagreb then took busses, trains, vans, or taxis to Banja Luka (a 2.5-5 hour ride depending on border lines, road construction, etc.). The weather was rainy off and on throughout Debconf, but the day of the Day trip the rain held off until most of us were back and we had a lovely day. Temperatures were warm but not overly hot, and I never needed my jacket. There are many large alarm clocks in Banja Luka, (such as the one pictured here, across the street from the venue) but there doesn't seem to be a way to change the time setting or put them on snooze.

Wouter Verhelst have some interesting comments and opinions on my blog post on the need to clean up /etc/rcS.d/ in Debian and my blog post about the default KDE desktop in Debian. I only have time to address one small piece of his comment now, and though it best to address the misunderstanding he bring forward:

Currently, a system admin has four options: [...] boot to a single-user system (by adding 'single' to the kernel command line; this runs rcS and rc1 scripts)

This make me believe Wouter believe booting into single user mode and booting into runlevel 1 is the same. I am not surprised he believe this, because it would make sense and is a quite sensible thing to believe. But because the boot in Debian is slightly broken, runlevel 1 do not work properly and it isn't the same as single user mode. I'll try to explain what is actually happing, but it is a bit hard to explain. Single user mode is defined like this in /etc/inittab: "~~:S:wait:/sbin/sulogin". This means the only thing that is executed in single user mode is sulogin. Single user mode is a boot state "between" the runlevels, and when booting into single user mode, only the scripts in /etc/rcS.d/ are executed before the init process enters the single user state. When switching to runlevel 1, the state is in fact not ending in runlevel 1, but it passes through runlevel 1 and end up in the single user mode (see /etc/rc1.d/S03single, which runs "init -t1 S" to switch to single user mode at the end of runlevel 1. It is confusing that the 'S' (single user) init mode is not the mode enabled by /etc/rcS.d/ (which is more like the initial boot mode). This summary might make it clearer. When booting for the first time into single user mode, the following commands are executed: "/etc/init.d/rc S; /sbin/sulogin". When booting into runlevel 1, the following commands are executed: "/etc/init.d/rc S; /etc/init.d/rc 1; /sbin/sulogin". A problem show up when trying to continue after visiting single user mode. Not all services are started again as they should, causing the machine to end up in an unpredicatble state. This is why Debian admins recommend rebooting after visiting single user mode. A similar problem with runlevel 1 is caused by the amount of scripts executed from /etc/rcS.d/. When switching from say runlevel 2 to runlevel 1, the services started from /etc/rcS.d/ are not properly stopped when passing through the scripts in /etc/rc1.d/, and not started again when switching away from runlevel 1 to the runlevels 2-5. I believe the problem is best fixed by moving all the scripts out of /etc/rcS.d/ that are not required to get a functioning single user mode during boot. I have spent several years investigating the Debian boot system, and discovered this problem a few years ago. I suspect it originates from when sysvinit was introduced into Debian, a long time ago.

Boy, the last three months have been very eventful, developmentally. First off, about 2 weeks after she turned 1, Zoe started walking in earnest. Within about 2 weeks of that there was no stopping her. It's funny to think she's only been walking for under 3 months and yet she's very stable on her feet. Secondly, she developed a mild egg allergy. Sarah was giving her some scrambled eggs (one egg's worth) for lunch one day, and she mixed in some cream cheese, and Zoe absolutely scoffed it all down, so she scrambled another egg and she scoffed that down too. She then broke out in some hives around her mouth and Sarah noticed that she was coughing a bit and a little wheezy when she put her down for her afternoon nap. Upon subsequent consultation with her paediatrician and a blood test, the allergy was confirmed and we now have to avoid eggs and Zoe was prescribed an Epi Pen Jr. We went and saw an allergist as a followup, and he did some skin tests that showed eggs were the only thing she's allergic to. He said her allergy was very mild in the grand scheme of things, and we're going to do another skin test when she's 2 to see if she's grown out of it. Eggs in baked goods are generally okay, as the offending protein is denatured at high temperatures. She's more allergic to the egg whites (1.56, which is in the "moderate" range) than the egg yolks (0.55, which is in the "low" range). Egg allergies are apparently something that one does eventually out grow, so that's a good thing. We'd also been keeping her off gluten and dairy because of a strong correlation between them and rather disgusting bowel motions. We've successfully reintroduced gluten with no adverse effects, and we've just started her back on cow's milk. We of course had our epic trip to the Grand Canyon, which Zoe handled absolutely marvelously. That was closely followed by a weekend down in LA, which was a bit more of a disaster as Sarah and I both came down with a 24 hour stomach bug that Zoe had had during the week. There's nothing quite like languishing in a non-childproofed hotel room, feeling like death, with a full of beans toddler busy running around jamming her fingers in drawers. Fortunately we were able to use a babysitter for a few hours respite, but it did rather completely mess up our plans for the weekend. Zoe did handle the drive to and from LA very well though. Zoe's been doing Saturday morning swim classes at the YMCA, which I've taken her to. I don't think she's enjoyed them as much as the indoor ones she did at the Sutton Swim School, where she went with Sarah previously, and I definitely don't feel like the classes are as structured. Unfortunately we just finished an 8 week session, and the next one filled up before we got a chance to book, so we're hoping to get back into something at the Sutton Swim School once an opening comes up. I might do something ad hoc with her on Saturdays at the Rinconada Pool in Palo Alto in the meantime. Zoe has also started going to regular daycare on Thursdays, so Sarah gets a day to focus on her University study. It's a home daycare that isn't too far away, and our friend's daughter goes there too, so there's someone she knows. The first few times Zoe got pretty upset when she was dropped off, but she barely makes a whimper now when Sarah drops her off. She's also usually down to one nap a day now. She has four teeth, top and bottom, with one molar definitely through and another three starting to make an appearance. I just bought an iBert safe-T-seat for my bike, because I've been itching to get out on the bike with Zoe on the weekends when I'm looking after her so Sarah can get some more study done. The weather is currently nice, and I've just learned about a bike trail that cuts across the bay in front of Moffett Field, so I'm keen to check that out. Apparently there's geese out there. I did my first decent ride with her last Sunday to the Mountain View Farmer's Markets and back. Zoe seemed to be okay with it. The helmet we got her drew a lot of comments from people at the markets. She's very much in the "point and grunt" stage at the moment. What's really cool is she'll nod or shake her head in response to yes/no questions. It's so cool when you can ask her a question and she solemnly nods her head in response. Often in the morning Sarah takes her to the park around the back of our home, and so now Zoe's gotten into that habit, so she'll scratch around at the front door and they head over there. The other day, Sarah opened the front door and she took off in the direction of the park, so Sarah just followed her to see where she went. Sure enough, she made her way over to the park. Tonight, Zoe fetched me my shoes and then dragged me over by the hand to the front door to demand that I take her out for a walk. She's also really getting into books now. She actively seeks out books, and has a bit of a flick through them, or brings one over and climbs into our laps, or backs up and sits in our laps if we're sitting on the floor, to have us read them to her. She's now strong enough to hold her bottle herself at bedtime, so I sit with her in my lap in the glider and read a story to her while she has her bedtime bottle. It's indescribably wonderful. She seems to understand quite a large array of words. We can go "Where's your <insert object here> and she'll look around the living room and generally find the item we're talking about. She also seems to understand questions about breakfast, lunch, going to the park, going to the bathroom, having a bath, etc as she seems to respond appropriately. In the speech department, I haven't really counted the words, but she's not saying a huge number of words. This morning, apparently she yelled "MAMA! MAMA! PARK!". "Park" being a new word for her. We bought an annual membership to the Children's Discovery Museum in San Jose, and we've probably taken her there half a dozen times or so since getting the membership. She particularly loves the water area. We finally got around to getting our back yard landscaped and made generally fit for a small child to run around in. Sarah's planted a butterfly garden in one corner, so hopefully once that's established we'll have some fun things for Zoe to watch from the window. I'm looking forward to planting some bulbs in the autumn. I expect the next 3 months will be more of the same. We've got some more overseas travel coming up, but a lot of relatives will be around, so hopefully we'll get a bit of downtime to ourselves.

It's a new month, so as usual for the beginning of the month I'm hoping to be a bit better about regularly updating, and ideally with more than just software release announcements and book reviews. Sadly, today has not been a good day for concentration despite a good early start, but I did manage to finally get through a long-delayed revision of my page of notes on using Git for Debian packaging. The revision drops the idea of using a separate debian branch to track Debian-specific changes so that master could be a pure feature branch. While appealing for its conceptual purity, this ended up confusing people horribly and was extra work for no apparent benefit. There are also a variety of other revisions to bring things up to date and a few new minor tips. Unfortunately, I haven't yet had a chance to write up how to maintain the upstream branch as a merge between the upstream Git repository and the released tarball, an idea from Sam Hartman that I'm now using very successfully for the openafs package, but at least now there's a pointer to it. I'm not really sure what to do with the initial part of that page that offers an introduction to Git. I've now written about three of these in separate places, and I'm still unconvinced of how much they actually help. And there are a ton of tools and basic techniques that I use all the time in Git that aren't mentioned in there (like git pull --rebase --stat, which I aliased to git update, or the whole concept of aliases, or how this all plays with 3.0 source packages). I've left the basic instructions for now, but I think at some point I may cede the tutorial aspect to other sites and just collect neat tricks that aren't part of the basic workflow that everyone describes.

Due to the midterm evaluation, this report will cover a span of 4 weeks. As a result, it will be quite a bit longer than my previous reports. One of the first things I did after my last report was create a FAQ for the project. At first, the FAQ was a plain text file living only in the git repository. Then, it moved to being maintained on the Debian wiki with a cronjob pulling it into my local repository for me to commit and push. Finally, the cronjob moved to wagner. The FAQ is not actually committed to the git repository, but it is available in the dextools instance running on wagner or on the wiki. There is also now a basic README file that is part of the git repository that simply points people to the FAQ on the wiki (so people who branch the repository can easily find the documentation). For those of you who have been reading my other reports and/or testing the dashboard, you have probably noticed the problems I was having getting the text boxes sized properly. After spending countless hours researching and testing different solutions, we finally decided to give up on this for the time being. Therefore, all of the size attributes have been removed from the text inputs, and the background color has been reset to its default value. This means that we will no longer experience the issue of very wide text boxes and a horizontally stretched table. However, it also means that long strings will get cut-off when displayed in a text input. Since my initial mockup design at the start of the summer, I have had the rows of the table zebra-striped. Based on some feedback from Paul Wise (pabs), I changed this so that the row colors are based on the status of the task. Open tasks are red, closed tasks are green, and in progress tasks are yellow. This makes it much easier to quickly see the progress of the project and find tasks you are interested in. My original dashboard consisted of one main dex.html file. This file used some javascript to pull in each project s task list. We felt that a much more common use case would involve a user only interested in one particular project. As a result, they would not be likely to use the select boxes to change projects. This ultimately led to me creating standalone dex.html dashboards for each project. These per-project dashboards were identical to the original dex.html, except they did not include the select boxes at the top for changing projects. Eventually, I decided to completely get rid of the original dex.html dashboard. That file now consists of a simple list of links to all of the projects sorted by distribution. Another advantage of this change is that it is much easier to link to a specific project. For example, you might now link to /projects/dex-ubuntu-ancient-patches/dex.html (you can get the URL by simply copying it from the address bar). Another advantage is that you can now link directly to a specific project s graphs (just append #graph). Another feature that I am still working on is per-project documentation. From the start, we wanted to make it easy for new contributors to get involved in DEX. One way we can do this is by having clear documentation about what the project is and how to deal with tasks. I decided that the easiest approach would be to have this documentation live on the wiki. This allows users to use the familiar wiki markup language to format their text, and it gives us all of the other benefits a wiki provides. Documentation will live at /DEX/<distribution>/ <project>. The dashboard will then take care of pulling in this documentation and displaying it at the top of the proper page. For a while, I was thinking that it would be necessary to filter the HTML produced by the wiki to prevent malicious code from ending up on the dashboard. However, after playing around for a bit, I think that it should be pretty safe to just use the HTML as-is (if someone spams the wiki, we can always revert the change). I am currently finishing up some of the code to handle this, and it should be live sometime this week (a partially working version is live on wagner). There are also two new checkboxes at the top of each dashboard. These allow users to add completed tasks and/or tasks with completed bugs. This is useful for projects that are close to being complete, as it allows users to quickly view the few remaining tasks. This feature was requested by several people, and I am glad to finally be able to implement it for them. There is also a second graph on each page. This graph is a bar graph that shows the number of tasks each person has completed. It gets the people from the assignee column for the project, and it only counts tasks with a status of Complete . The graph is updated whenever the getbugs.py script runs to update the task listing (currently every 10 minutes). The bars are sorted from tallest to shortest. The idea behind this graph is to recognize the people working on a project. The form processing script has not worked on wagner. This was mainly due to a permission issue. Thanks to a hint from Rapha l Hertzog, I was able to use setfacl to fix this. The form can now be submitted, and it should successfully save all changes made via the web form. This involved some updates to support the per-project dex.html files. The form processing script will also ignore any status that is not complete , in progress , or incomplete , which prevents people from messing up the dashboard by submitting invalid information. Finally, the script will now try to use the HTTP_REFERER header to send the user back to the correct page. If this header is not set, the user will be sent back to the main dex.html index page. Sorting the table has been partially broken since I started including text inputs. I finally got this issue resolved, and I should now be able to sort by text inputs, select boxes, or anything else I might end up including in the table. You can test this by clicking on any header in the table. I will probably add a tiny image to make it more clear which header is being used to sort the table. By default, it is sorted based on the status and then the task name. The status column now has a select box for setting the status. Since the row coloring is done via javascript, the minute you change the status in the select box, the row color will change to reflect the new status. Keep in mind, for the time being, any change you make via the web form will be lost if you do not hit the Submit button at the top of the form. That covers most of the dashboard changes that have taken place since my last report. However, there were also some other changes that are worth noting. Matt Zimmerman, my original mentor, settled down from his traveling and resumed serving as my mentor for the project. I would like to thank Stefano Zacchiroli for all of his help this summer. I look forward to working with him more in the future. I have also started using the issue tracker feature on alioth for dextools. Right now, it is mainly being used to help me keep track of bugs and features that I need to deal with. I am hoping that as dextools starts to be used by more people that they will also help report issues on the tracker. This issue tracker is a good source of information for people interested in knowing what I will be working on in the future. I will briefly talk about some of the more important features that I will be working on. The first issue is one that I have already touched upon in this report. I want to sort out the per-project documentation. This is rather important, as it will most likely be utilized by all projects using the dashboard. It is also essential if we are to get new contributors involved in DEX projects. Second, I want to setup some method for backing up the projects/ directory on wagner. This directory is not maintained in the git repository, and it contains all of the information submitted via the web form and generated by the scripts. My current plan is to have this directory maintained in its own VCS. Every time getbugs.py runs or the form is submitted, a new commit will be made. This will allow us to easily revert any changes (spam) made to the dashboard. Since we do not foresee the dashboard being the target of much abuse, any user will be able to modify data via the web form. I also want to add the ability for users to create new projects and task via the web. I envision this as a separate form where they enter the distribution name, project name, a list of BTS bug numbers, and a list of any additional tasks. The script will then take care of applying the necessary usertag to the bugs and/or creating a tasks file in the projects directory for the non-bts tasks. Doing this will make starting a new project trivial, and will eliminate the need for VCS access to create projects that do not utilize BTS bugs. Finally, I want to eliminate the need for the user to hit the Submit button to save their changes. Any change the user makes should be submitted and saved instantly using AJAX. This will also help eliminate some confusion that was caused by the row color instantly changing when the status is changed. Some users thought that the color change meant that the change was saved when it really was not. Matt asked me the other day what features I think need to get added before the dashboard is really ready for public use. While the features and issues I mentioned above will help make the dashboard easier to use, the dashboard is still perfectly usable without them. There are only a few more weeks left in the Google Summer of Code 2011. By that time, all of these features and bugs should be sorted out, and we should be able to properly announce the dashboard and make it available for use on all DEX projects. That does not mean that the dashboard will be done . Instead, it will probably result in many new bugs and feature requests being submitted. However, at that point, I will make a very strong effort to keep the dashboard stable and to prevent data loss. I am looking forward to the finally finishing up this project and seeing it used by the DEX team. As always, if you are interested in helping out with DEX or dextools, my code is available in a git repository on alioth. There is also the new issue tracker where you can report bugs and request new features. Finally, you can join the debian-derivatives mailing list, join #debian-derivatives on OFTC, or contact me directly via email or on IRC (nhandler).</project></distribution>

Fellow Debian Developer Petter Reinholdsen, who is also on Planet Debian, recently blogged about two things that I just have to respond to. I'd post a comment on his blog, but unfortunately his blog does not support comments (booh!). In his first blog post, titled "What is missing in the Debian desktop, or why my parents use Kubuntu", he complains about the lack of a "simple GUI based upgrade of packages". That already exists and is in Debian; if you install the default GNOME desktop, you'll get an icon in the (freedesktop.org) system tray (so it's not a gnome applet) that will tell you when there're updates available. It uses a system tray thingy, so it's not gnome-specific, and you can still use it if you use KDE instead of gnome though I should add that it's something my parents keep ignoring, even after I've told them. Oh well. The same blog post contains two more items that I don't disagree with, and is then followed with a complain about "Better browser handling of some MIME types". This is actually a bug in Firefox upstream, not Debian; ubuntu may have patched their Firefox, but feature requests to fix this issue have been posted in the Firefox Bugzilla. Meanwhile, a workaround exists. His second blog post is about the boot process. Petter is involved with the sysvinit system, and has some valid points in that some scripts in rcS.d really don't belong there. Petter wants to move many scripts, including those setting up the network, out of rcS.d. I don't think that's a very good idea. Currently, a system admin has four options: boot to a 'normal' login prompt (just boot, don't modify the kernel command line); boot to a single-user system (by adding 'single' to the kernel command line; this runs rcS and rc1 scripts); boot to a very minimally set up system (by adding 'emergency'; this just mounts the root filesystem, does some very minor setup (like doing the keyboard mapping) and allows the user to get a root shell); or just boot to a shell, period (by adding 'init=/bin/bash'). While it makes obvious sense that rcS.d shouldn't contain too much stuff (otherwise you might as well not bother with having it at all), I'm not sure that moving the mounting of 'network-based filesystems mounted by default at bootup time' away from rcS.d is a good idea. Sometimes things go horribly wrong after the networked filesystems are mounted, because some things are misconfigured on those networked file systems. We currently have two states that bring the system up to 'networked filesystems mounted, system not fully booted', and two that bring the system up to 'root filesystem mounted, you figure it out from there'. That seems well divided, and I don't think changing that is a good idea.

In the past week, I ran a lot of benchmarks using a Spirent Avalanche which is an appliance providing performance testing of network related products (a load balancer, a router, a web server, ). The reporting module does not provide a lot of flexibility and the plots are not the most beautiful ones. Fortunately, the results are also exported as CSV. matplotlib is a python plotting library which produces great figures without making things hard when they should be easy. It is a good replacement for GNU Plot and you don t need a lot of Python knowledge to use it. The documentation includes a fine user s guide that should get you started in less than 20 minutes.

Quick introduction You can use matplotlib from IPython to experiment:

$ ipython -pylab
Python 2.6.7 (r267:88850, Jul 10 2011, 08:11:54) 
Type "copyright", "credits" or "license" for more information.
  Welcome to pylab, a matplotlib-based Python environment.
  For more information, type 'help(pylab)'.
> In [1]: plot([1,1,4,5,10,11])

When you are ready, you can build a simple Python script:

#!/usr/bin/env python
from matplotlib.pylab import *
plot([1,1,4,5,10,11])
savefig("my-plot.pdf")

Grabbing results from CSV Data are contained into a file named realtime.csv. It contains the raw data as well as some general information about the benchmark (test name, description, paramaters ). We need to skip those. Fortunately, csv2rec() allows to load data from a CSV file into a record array and skip the first rows if necessary.

from matplotlib.pylab import *
import sys, os
import gzip
skip = 0
for line in gzip.open(sys.argv[1]):
    if line.startswith("Seconds Elapsed,"):
        break
    skip = skip + 1
ava = csv2rec(gzip.open(sys.argv[1]),skiprows=skip)

Now, the Elapsed seconds column can be accessed with ava['elapsed_seconds'].

General structure I need 4 plots:

• successful, unsuccesful and attempted transactions per second,
• minimum, average and maximum response time per page,
• Avalanche CPU usage,
• incoming and outgoing bandwidth.

The most important plot is the first one. The second one is less important and the last ones are here only to check we did not hit some bottleneck during the benchmark. We want to produce a page like this: matplotlib allows to plot subfigures. We create 4 subfigures sharing the same X axis (which the number of seconds elapsed since the beginning of the benchmark). We save the result to PDF.

# Create the figure (A4 format)
figure(num=None, figsize=(8.27, 11.69), dpi=100)
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
# [...]
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
# [...]
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
# [...]
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
# [...]
# Save to PDF
savefig("%s.pdf" % TITLE)

Bandwidth plot Let s start with the easiest plot: bandwidth usage.

# Plot 4: Bandwidth
ax4 = subplot2grid((4, 2), (3, 1), sharex=ax1)
plot(ava['seconds_elapsed'], ava['incoming_traffic_kbps']/1000.,
     'b-', label='Incoming traffic')
plot(ava['seconds_elapsed'], -ava['outgoing_traffic_kbps']/1000.,
     'r-', label='Outgoing traffic')
grid(True, which="both", linestyle="dotted")
ylabel("Mbps", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Here, in the suplot positioned in (3,1), we plot the number of seconds elapsed versus the incoming traffic with a blue line (b-). We also plot the seconds elapsed versus the outgoing traffic with a red line (r-). Here is the result: Most functions of matplotlib are exposed as a method of the object they refer to and as a global function. In the latest case, the function is applied to the latest created figure or plotting area. For example, plot() is called a function and therefore refer to the plotting area ax4. We could have written ax4.plot() instead.

CPU plot The average CPU utilization data available in the CSV file needs to be normalized. We assume the Avalanche to be mostly idle on start. The plotting part is pretty similar to our previous case.

# CPU
max = np.max(ava['average_cpu_utilization'])
order = 10**np.floor(np.log10(max))
max = np.ceil(max/order)*order
cpu = (max - ava['average_cpu_utilization'])*100/max
# Plot 3: CPU
ax3 = subplot2grid((4, 2), (3, 0), sharex=ax1)
plot(ava['seconds_elapsed'],
     cpu,
     'r-', label="Avalanche CPU")
grid(True, which="both", linestyle="dotted")
ylabel("Avalanche CPU%", fontsize=7)
xticks(fontsize=7)
yticks(fontsize=7)

Response time We would like to plot response time. We have three corresponding metrics: minimum response time, average response time, maximum response time. Because those metrics start from 0 ms up to several seconds, a logarithmic scale is used:

# Plot 2: response time
ax2 = subplot2grid((4, 2), (2, 0), colspan=2, sharex=ax1)
plot(ava['seconds_elapsed'], ava['minimum_response_time_per_page_msec'],
     'b-', label="Minimum response time")
plot(ava['seconds_elapsed'], ava['maximum_response_time_per_page_msec'],
     'r-', label="Maximum response time")
plot(ava['seconds_elapsed'], ava['average_response_time_per_page_msec'],
     'g-', linewidth=2, label="Average response time")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=8))
grid(True, which="major", linestyle="dotted")
yscale("log")
ylabel("Response time (msec)", fontsize=9)
xticks(fontsize=9)
yticks(fontsize=9)

This is also the first graphic with a legend.

Transactions per second The most important plot is the number of transactions per second.

# Plot 1: TPS
ax1 = subplot2grid((4, 2), (0, 0), rowspan=2, colspan=2)
plot(ava['seconds_elapsed'], ava['desired_load_transactionssec'],
     '-', color='0.7', label="Desired Load")
plot(ava['seconds_elapsed'], ava['successful_transactionssecond'],
     'g:', label="Successful")
plot(ava['seconds_elapsed'], smooth(ava['successful_transactionssecond']),
     'g-', linewidth=2)
plot(ava['seconds_elapsed'], ava['attempted_transactionssecond'],
     'b-', label="Attempted")
plot(ava['seconds_elapsed'], ava['aborted_transactionssecond'],
     'k-', label="Aborted")
plot(ava['seconds_elapsed'][:-1], ava['unsuccessful_transactionssecond'][:-1],
     'r-', label="Unsuccessful")
legend(loc='upper left', fancybox=True, shadow=True, prop=dict(size=10))
grid(True, which="both", linestyle="dotted")
ylabel("Transactions/s")

The number of successful transactions is plotted twice: when the benchmarked equipment becomes overloaded, we get a lot of noise in this metric and it can be difficult to read. Therefore, we plot a smoothed version with the help of NumPy:

import numpy as np
def smooth(x, win=4):
    s = np.r_[x[win-1:0:-1],x,x[-1:-win:-1]]
    w = np.ones(win, 'd')
    y = np.convolve(w/w.sum(),s,mode='valid')
    return y[(win-1)/2:-(win-1)/2]

np.r_() is just here to extend our data by the size of the window. np.ones() build a weight vector of the size of the window. If the window is 4, we get [0.25, 0.25, 0.25, 0.25]. We use this vector to apply a convolution to the original data. Here is the result: The original data is a green dotted line while the smoothed one is a green thick line. What about the three annotations? matplotlib allows to put annotations on a figure. Here is how this is done:

# Noticeable points
count = 0
def highlight(index, reason):
    global count
    if index and index > 0:
        x,y = (ava['seconds_elapsed'][index],
               smooth(ava['successful_transactionssecond'])[index])
        plot([x], [y], 'ko')
        annotate('%d TPS\n(%s)' % (y,reason), xy=(x,y),
                 xytext=(20, -(count+4.7)*22), textcoords='axes points',
                 arrowprops=dict(arrowstyle="-",
                 connectionstyle="angle,angleA=0,angleB=80,rad=10"),
                 horizontalalignment='left',
                 verticalalignment='bottom',
                 fontsize=8)
        count = count + 1
highlight(np.argmax(smooth(ava['successful_transactionssecond'])), "Max TPS")
highlight(np.argmax(cpu > 99), "CPU>99%")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 500), ">500ms")
highlight(np.argmax(ava['average_response_time_per_page_msec'] > 100), ">100ms")

np.argmax() returns the index of the first maximum value. The trick here is that when I write ava['average_response_time_per_page_msec'] > 100, I get an array with 1 when the value is more than 100 and 0 otherwise. Therefore, np.argmax() will return the first index where the value is superior to 100 ms. The highlight() function will add a point (plot([x], [y], 'ko')) on the smoothed successful transactions par second plot and add an annotation with some fancy arrow. Look at this benchmark of nginx as SSL termination for a complete output of this script.

DWARF-2 is a format to store debugging information. It is used on many ELF systems such as GNU/Linux. With the way things are compiled, there is a lot of redundant information in the DWARF-2 sections of an ELF binary. Fortunately, there is an option to gcc that helps dealing with the redundant information and downsizes the DWARF-2 sections of ELF binaries. This option is -feliminate-dwarf2-dups. Unfortunately, it doesn t work with C++. With -g alone, libxul.so is 468 MB. With -g -feliminate-dwarf2-dups, it is 1.5 GB. FAIL. The good news is that as stated in the message linked above, -gdwarf-4 does indeed help reducing debugging information size. libxul.so, built with -gdwarf-4 is 339 MB. This however requires gcc 4.6 and a pretty recent gdb.

While at Debconf11, I have several times during discussions mentioned the issues I believe should be improved in Debian for its desktop to be useful for more people. The use case for this is my parents, which are currently running Kubuntu which solve the issues. I suspect these four missing features are not very hard to implement. After all, they are present in Ubuntu, so if we wanted to do this in Debian we would have a source.

1. Simple GUI based upgrade of packages. When there are new packages available for upgrades, a icon in the KDE status bar indicate this, and clicking on it will activate the simple upgrade tool to handle it. I have no problem guiding both of my parents through the process over the phone. If a kernel reboot is required, this too is indicated by the status bars and the upgrade tool. Last time I checked, nothing with the same features was working in KDE in Debian.
2. Simple handling of missing Firefox browser plugins. When the browser encounter a MIME type it do not currently have a handler for, it will ask the user if the system should search for a package that would add support for this MIME type, and if the user say yes, the APT sources will be searched for packages advertising the MIME type in their control file (visible in the Packages file in the APT archive). If one or more packages are found, it is a simple click of the mouse to add support for the missing mime type. If the package require the user to accept some non-free license, this is explained to the user. The entire process make it more clear to the user why something do not work in the browser, and make the chances higher for the user to blame the web page authors and not the browser for any missing features.
3. Simple handling of missing multimedia codec/format handlers. When the media players encounter a format or codec it is not supporting, a dialog pop up asking the user if the system should search for a package that would add support for it. This happen with things like MP3, Windows Media or H.264. The selection and installation procedure is very similar to the Firefox browser plugin handling. This is as far as I know implemented using a gstreamer hook. The end result is that the user easily get access to the codecs that are present from the APT archives available, while explaining more on why a given format is unsupported by Ubuntu.
4. Better browser handling of some MIME types. When displaying a text/plain file in my Debian browser, it will propose to start emacs to show it. If I remember correctly, when doing the same in Kunbutu it show the file as a text file in the browser. At least I know Opera will show text files within the browser. I much prefer the latter behaviour.

There are other nice features as well, like the simplified suite upgrader, but given that I am the one mostly doing the dist-upgrade, it do not matter much. I really hope we could get these features in place for the next Debian release. It would require the coordinated effort of several maintainers, but would make the end user experience a lot better.

Many people still install Linux from CDs. But a growing number install from USB. In an ideal world you'd be able to download one image that would let you do either, but it turns out that that's quite difficult. Shockingly enough, it's another situation where the system firmware exists to make your life difficult.

Booting a hard drive is pretty easy. The BIOS reads the first 512 bytes off the drive, copies them to RAM and executes them. That code is then responsible for either starting your bootloader or identifying the currently active partition and jumping to its boot sector, but before too long you're in a happy place where you're executing whatever you want to. Life is good. So you'd think that CDs would work in a similar way. The ISO 9660 format even leaves a whole 32KB at the start of a filesystem, which is enough space for a pretty awesome bootloader. But no. This is not how CDs work. That would be far too easy.

Let's imagine we're back in the 90s. People want to be able to boot off CD without needing a boot floppy to do so. And you're a PC vendor with a BIOS that's been lovingly[1] forced into a tiny piece of flash and which has to execute out of an almost as tiny piece of RAM if you want your users to be able to play any games. Letting boot code read arbitrary content off the CD would mean adding a new set of interrupt hooks, and that's going to be even more complicated because CDs have a sector size of 2K while hard drives are 512 bytes[2] and who's going to pay to implement this and for the extra flash and RAM and look surely there has to be another way?

So, of course, another way was found. The El Torito specification defines a way for shoving a reference to some linear blocks into the ISO 9660 header. The BIOS reads those blocks into memory and then redirects either the floppy or hard drive access interrupts (depending on the El Torito type) to that region. The boot code can then proceed as if it had been read off a floppy without all the trouble of actually putting a floppy in the machine, and the extra code required in the system BIOS is minimal.

USB sticks, however, are treated as hard drives. The BIOS won't look for El Torito images on them. Instead, it'll try to execute a boot sector. That isn't there on a CD image. Sigh.

A few years ago a piece of software called isohybrid popped up and solved this problem nicely. isohybrid is a companion to isolinux, which itself is a bootloader that fits into an El Torito image and can then load your kernel and installer from CD. isohybrid takes an ISO image, adds an x86 boot sector and partition table and does some more fiddling to turn a valid ISO image into one that can be copied directly onto a USB stick and booted. The world suddenly becomes a better place.

But that's BIOS. EFI makes this easier, right? Right?

No. EFI does not make this easier.

Despite EFI being a modern firmware for the modern world[3], EFI implementations are not required to be able to understand ISO 9660. In fact, I've never seen one that does. FAT is all the spec requires, and FAT is typically all you get. Nor will EFI just execute some arbitrary boot code from the start of the CD. So, how does EFI boot off CD?

El Torito. Obviously.

It's not quite as bad as it sounds, merely almost as bad as it sounds. While the typical way of using El Torito for a long time was to use floppy or hard drive emulation, it also supports a "No emulation" mode. It also supports setting a type flag for your media, which means you can distinguish between images intended for BIOS booting and EFI booting. But the fact remains that your CD has to include an embedded FAT partition that then contains a bootloader that's able to read ISO 9660 because your firmware is too inept to handle that itself[4].

How about USB sticks? Thankfully, booting these on EFI doesn't require any boot sectors at all. Instead you just have to have a partition table, a FAT partition and a bootloader in a well known location in that FAT partition. The required partition is, in fact, identical to the one you need in an El Torito image. And so this is where we start introducing some extra hacks.

Like I said earlier, isohybrid fakes up an MBR and adds some boot code that points at the actual bootloader. It needs to do a little more on EFI. The first problem is that the isohybrid MBR partition has to cover the entire ISO 9660 filesystem on the USB stick so that the operating system can access it later, but the El Torito FAT image is inside that partition. A lot of MBR-based code becomes very unhappy if you try to set up a partition that's a subset of another partition. So we can't really use MBR. On to GPT.

GPT, or the GUID Partition Table, is the EFI era's replacement for MBR partitions. It has two main advantages over MBR - firstly it can cover partitions larger than 2TB without having to increase sector size, and secondly it doesn't have the primary/logical partition horror that still makes MBR more difficult than it has any right to be. The format is pretty simple - you have a header block 1 logical block into the media (so 512 bytes on a typical USB stick), and then a pointer to a list of partitions. There's then a secondary table one block from the end of the disk, which points at another list of partitions. Both blocks have multiple CRCs that guarantee that neither the header nor the partition list have been corrupted. It turns out to be a relatively straightforward modification of isohybrid to get it to look for a secondary EFI image and construct a GPT entry pointing at it. This works surprisingly well, and media prepared this way will boot EFI machines if burned to a CD or written to a USB stick.

There's a few quirks. Macs will show two boot icons for these CDs[6], one marked "EFI Boot" and one helpfully marked "Windows"[7], with the latter booting the BIOS El Torito image. That's a little irritating, but not insurmountable. The other issue is that older Macs won't look for boot loaders in the legacy locations. This is where things start getting horrible.

Back in the old days, Apple boot media used to have a special "blessed" folder. Attempting to boot would involve the firmware looking for such a folder and then using that to start itself up. Any folder in the filesystem could be blessed. Modern hardware doesn't use boot folders, but does use boot files. For an HFS+ filesystem, the inode of the bootloader is written to a specific offset in the filesystem superblock and the firmware simply finds that inode and executes it. And this appears to be all that older Macs support.

So, having written a small tool to bless an HFS+ partition, I tried the obvious first step of burning a CD with three El Torito images (one BIOS, one FAT, one HFS+). It failed. While Refit could see the bootloader in the HFS+ image, the firmware appeared to have no interest at all in booting off it. Yet Apple install media would boot. What was the difference?

The difference, obviously, was that these earlier Macs don't appear to support El Torito booting. The Apple install media contained an Apple partition map.

The Apple partition map (APM) is Apple's legacy partition table format. Apple mostly dropped it when they went to x86, where it's retained for two purposes. The first is for drives that need to be shared between Intel Macs and PPC ones. The second seems to be for their install DVDs. Some further playing revealed that burning a CD with an APM entry pointing at the HFS+ filesystem on the CD gave me a boot icon. Problem solved?

Not really. Remember how I earlier mentioned that ISO 9660 leaves 32KB at the start of the image, and that an isohybrid image then writes an MBR and boot sector in the first 512 bytes of that, and the GPT header starts 512 bytes into a drive? That means that it's easy to produce an ISO that has both a boot sector, MBR partition table and GPT. None of them overlap. APM, on the other hand, has a header that's located at byte 0 of the media, overlapping with the boot sector. And it has a partition listing that's located at sector 1, overlapping with the GPT. Is all lost?

No. Merely sanity.

The first thing to remember is that the boot sector is just raw assembler. It's a byte stream that's executed by the CPU. And there's a lot of things you can tell a CPU to do that result in nothing happening. Peter Jones pointed out that the only bits of the AFP header you actually need are the letters "ER", followed by the sector size as a two byte big endian integer. These disassemble to harmless instructions, so we can simply move the boot code down a little and stick these at the beginning. A PC that executes it will read straight through the bizarre (but harmless) Apple bytes and then execute the real boot code.

The second thing that's important here is that we were just given the opportunity to specify the sector size. The GPT is only relevant when the image is written to a USB stick, so assumes a sector size of 512 bytes. So when the GPT starts one sector into the drive, it's actually starting 512 bytes into the drive. APM also starts one sector into the drive, but we can simply put a different sector size into the header and suddenly we're able to choose where that's going to be. 2K seems like a good choice, and so the firmware will now look for the header at byte 2048.

That's still in them middle of the GPT partition listing, though. Except we can avoid that as well. GPT lets you specify where the partition listing starts and doesn't require it to be immediately after the header. So we can offset the partition listing to, say, byte 8192 and leave a hole for the Apple partition map.

And, shockingly, this works. Setting up a CD this way gives a boot icon on old Macs. On new Macs, it gives three - one for legacy boot, one for EFI boot via FAT and one for EFI boot via HFS. Less than ideal, but eh. The one remaining problem is that this doesn't work for USB sticks (the firmware sees the GPT and ignores the APM), so we also need to add a GPT entry for the HFS+ partition. Job done.

So, it is possible to produce install media that will work if burned to CD or written to a USB stick. It's even possible to produce a version that will work on Macs, as long as you're willing to put up with three partition tables and an x86 boot sector that doubles as an APM header. And patches to isohybrid to do all of this will be turning up as soon as I tidy the code to the point where it works without having to hack in offsets by hand.

[1] Insert some other adverb here if you feel like it
[2] Why yes, 15 years later BIOSes still tend to assume 512 bytes. Which is why your 4K sector disk is much harder to work with than you'd like it to be.
[3] Ever noticed how the modern world involves a great deal of suffering, misery and death? EFI fits into that world perfectly.
[4] Obviously if you want your media to be bootable via both BIOS and EFI you need to produce a CD with two El Torito images. BIOS systems should ignore the image that says it's for EFI, and EFI systems should ignore the BIOS one. Some especially creative BIOS authors[5] have decided that users shouldn't have their choices limited in such a way, and so pop up a screen that says:

1.
2.
Select CD-ROM boot type:

and wait for the user to press a key. The lack of labels after the numbers is not a typographical error on my part.
[5] Older (pre-2009, and some 2009 models) Apple hardware has this bug if a dual-El Torito CD is booted via the BIOS compatibility layer. This is especially unfortunate because said machines often fail to provide a working keyboard emulation at this stage, resulting in you being stuck forever at an impressively unhelpful screen. This isn't a Linux bug, since it's happening before we've run any of our code at all. It's not even limited to Linux. 64-bit install media for Vista SP1, Windows 7 and Server 2008 all have similar El Torito layout and all trigger the same bug on Apple hardware. Apple's aware of this, and has resolved the issue by declaring that these machines don't support 64 bit Windows.
[6] Even further investigation reveals that Apple will show you as many icons as there are El Torito images, which is a rare example of Apple giving the user the freedom to brutally butcher their extremities if they so desire
[7] "Windows" is Apple code for "Booting via BIOS compatibility". The Apple boot menu will call any filesystem with a BIOS boot sector Windows.

It is half-time for me at DebCamp/DebConf11 in Banja Luka, Bosnia-Herzegovina, and that is a good time to write about my work so far. I started with some finger exercises, such as the HTML copying for gnome-terminal (as blogged previously) and cosmetic changes to the buildd.debian.org/status website. After that, I began my main project: A SAT-based solver for the testing transition problem. In Debian, new software first enters a repository called unstable, where it is tested. Once some requirements are fulfilled (package has been in unstable for usually at least 10 days and has no new release critical bugs), it is entitled to enter the repository called testing, which will eventually form a new stable Debian release. But some packages need to migrate together, usually because either requires the new version of the other. Also, no packages in testing ought to become uninstallable by some seemingly unrelated change. The software that decides these things is called britney. It is sufficiently good in making sure nothing bad happens, but not smart enough to figure out what to do in case more than two packages need to migrate simultaneously. My goal is to improve this. Now, the the various requirements can all be expressed as predicative formulas, and a lot of research has been going into writing good solvers for such problems, called SAT-solvers. Hence my plan was to only specify what we expect from a testing migration, but leave the search for a solution to such a general purpose and highly optimized program. My initial progress was good and I had some result after two days of hacking, and the approach is promising. Because the data sets are pretty large (1.5GB of input data, the final SAT problem has 1.8 million clauses with 250.000 variables), so I learned quite a bit about profiling and optimizing Haskell programs, and that parsec is slower than working with BS.lines and BS.split to parse simple input. I also used the FFI to use dpkg code directly to compare Debian version numbers. At some point I noticed that I actually want to solve PMAX-SAT problems: Given two set of clauses, hard clauses and soft clauses, find a variable assignment that fulfills the hard clauses and as many of the soft clauses as possible. Unfortunately, there are no fast Free PMAX-SAT solvers around. The ones that I found and that were fast enough that I could use them, msuncore by Joao Marques-Silva at the University College Dublin and MiniMaxSat by Federico Heras at the Universitat Polit cnica de Catalunya, are only available as statically linked binaries. It is a shame that this is acceptable by the academic community; imagine mathematicians would stop including the proofs in their papers, and only share their theorems. So if you happen to have written a PMAX-SAT solver and can solve this instance (in weighted DIMACS format) in less than five minutes, and want to brag that the Debian project is using your code, then please release it under a Free license (e.g. GPL or BSD) and tell me about it! The code of my SAT-based solver is, of course, available, though slightly unpolished. Parts of it (notably the interface to picosat and the PMAX-SAT solvers) might be of general interest and I plan to put them on Hackage as a separate license. Update: I have contacted the authors of the SAT solvers mentioned above, and they have reconfirmed that they have no intention of releasing the source. Now I put my hope in maxsatz2009 by Chumin LI, which is GPL but was not able to cope with my large instance directly; I think I need to change the memory management.

Flattr this post

As the main reason why I got Nokia N950 is that I promised to write some applications for that, I started to look deeply at available options how to write them. Unfortunately there are too many options and all documentation I've found is pretty confusing. The short story is that you should use Qt for user interface. However it is not that easy as it looks, you can use regular Qt, which can be run on the phone, but looks ugly in the end (maybe I did something wrong, but it did look completely different to anything else running in the phone). You can use Qt Quick, which is easy to write (simple UI definition and JavaScript variant for code), but it does not seem to support much of the widgets. And then there is Meego Touch library, which seems like a best option, however not mentioned at all on Nokia developer website. Looks like I need to learn few more things before actually starting to code.

Filed under: English Meego 7 comments Flattr this!

If you thought Debian/m68k was dead and buried and where happy about it, I'm sorry to disappoint you. The reason m68k had died out slightly was not that we'd lost interest, or that we'd been kicked out of the archive, or anything of the sorts, as I've seen people believe. Instead, the reasons were fully technical: glibc past some particular version needs TLS (thread-local storage) support, which hadn't been implemented for m68k yet, and nobody within the Debian/m68k team had the expertise to work on this. That was all. Not so long ago, some people within RedHat had started doing a TLS implementation for m68k, however. This was paid for by one of their customers who needed the ColdFire to work, and therefore it was focused mostly on ColdFire. However, ColdFire is close enough to plain old m68k that it was possible to port this TLS support to plain old without too much effort. Unfortunately, most of our machines were pretty much nonfunctional by that time; so in order to get things to work, we'd have to

• Reinstall some of our machines, fix the others, without a functional installer,
• Do the necessary porting and package integration work,
• Rebootstrap the architecture,
• do other things necessary (such as providing a working installer, etc).

Not quite impossible, but still a lot of work that would need someone to put in quite an amount of time. This wasn't something that people in the m68k port could invest; and I'm also not sure we had the necessary skills amongst the team to do the necessary porting. So the work lay there, and it wasn't being done. Until one Thorsten Glaser, author of a shell called 'mksh' that I hadn't heard of until then, came along, and noted that his mksh package wasn't being built for m68k. Upon investigating, he discovered, as expected, that the port was almost dead. Now most people would leave it at that and decide that they couldn't build for an architecture that wasn't being maintained anymore and that would be perfectly fine but not Thorsten. On the brink of death, he almost single-handedly revived the m68k port, and got stuff to work to such an extent that it is now again possible to build a working and fairly up-to-date Linux/m68k ARAnyM virtual machine. As I'm typing this, my ARAnyM instance is checking out the d-i repository (so I can start building d-i), and building sudo at the same time. Let's see what comes from this.

If a single man can display so much hatred
think only of how much love we all can display together.

Stine Renate H heim

I wrote a Norwegian post explaining my experience at Ut ya. I had taken this blog for dead, and had entirely forgotten that it was syndicated on Planet Debian. I don t want to let Google Translate make this disaster any worse than it is the translation of bullets into balls being particularly bad so the international attention the massacre has garnered in consideration, I am writing an English translation of my experiences. I feel somehow duty-bound to make people aware of what happened, but I don t want to get into anything else but a sober description of the events and some very brief reflections. There are many details I have chosen to omit. Others have written their experiences of the events at Ut ya. I wanted to write mine down as well, and get it out there . Partly, I want to write this down because I m unsure if I will remember all the details at a later point in time, although I think I d prefer it if I couldn t. I m also writing this because people are asking about my experiences and it s much better to have an URL to give them, lest I have to keep going through the same spiel over and over again. Our former Prime Minister and current labour movement demigod Gro Harlem Brundtland had recently left the island. I had been the cameraman for a video interview of her talking about Ut ya, and I was in the media group room encoding the video into a file suitable for YouTube, when someone else in the room startled and said that Twitter was full of messages about a loud explosion in Oslo. As the newspapers brought us information about the extent of the damages, a consensus arose that an informational meeting was in order. As soon as the current round of talks finished, we were gathered into the main hall. The meeting was duly held, and after the statement was made that a TV feed would be made available, I took it upon myself as the local alpha geek to make it happen. Of course, the situation caused both the wireless network and the GPRS networks to become totally unusable. As I was waiting for someone to set up a password, I took the opportunity to face the consequences of having eaten two bits of a microwavable dish called Hold-It the local equivalent of a Hot Pocket and went to the toilet. As I was in there, I first heard agitated shouting, then screams, then gunshots coming from just outside the toilets. More than anything else, it sounded like a toy gun. I was convinced that someone was making a joke in incredibly bad taste and I stormed out of the booth with the intent of halting it. As I tore the door open, I saw two of my comrades hiding in a recessed corner. Their facial expressions left absolutely no doubt that this was no toy. They signalled for me to get back in the booth. I closed the door, did a mental double-take in utter, complete confusion, and opened it again. They were still signalling. Had they not stood there, I would have run straight into the gunman; they saved my life. I looked out into the hallway, and I made eye contact with a young boy lying in a pool of blood. He was motioning for me to help him. I heard more gunshots from inside the building and retreated back inside. As I was trying to think through my next move, I realized that the decidedly insubstantial wood-fiber door would not resist any kind of bullets. I made my way out into the hallway, with the intent of escaping outside. At that point, I was of course not aware that there was an intention to kill as many as possible, so I thought that the open spaces outside would be a place of relative safety. Of course, this proved to be wrong and my life was probably saved a second time by one of the caf volunteers taking me into a hard-to-spot employee s bathroom. We sat there for ninety minutes. Always ready to make a run for it, ready for just about anything. A peculiar group dynamic arose with these two people with whom I had barely previously spoken. We came to share a strange sense of common destiny and gallows humour. One of them had seen the shooter and described the police uniform. I perceived it to be realistic that we were the only ones aware of the wounded outside the toilet. I tried to reach the emergency services, but all their lines were busy; the terror attack in Oslo had probably clogged their lines. I finally got through to the fire services, who could inform me that the police did know about the situation and were on their way. This was to take 90 minutes and by the time we evacuated, the young boy outside my door had perished. The despair I first saw in his eyes as I passed him, fleeing from one room to the other and the empty, blank stare as we left, are burned into me and they are images I will never in my life forget. Finally, the real police arrived. We walked out. I chose the path through the minor conference hall something I now regret. The sight was simply beyond my capacity to describe fully, and so terrifying that I barely remember the sight only the terror it struck in me. There were several people bunched up in a corner, a big amorphous heap of bodies. Some were conscious and yelled at me not to do anything that could startle the police, others lay still. Their bodies were all covered in blood, and a thick pool of blood extended at least a half-metre in all directions around them. The policeman across the hall was screaming orders at me, but he was screaming so loudly that I couldn t make out his words at first. We were first moved into the camp newspaper s offices. There were about eight of us there, I think, in addition to one girl who lay wounded. Towards the end she was drifting in and out of consciousness. We covered her with sweaters to keep her warm and one of us tried to at least temper her bleeding. The bullet had missed her heart, but by the entry wound it was clear that it was not by far. I do not know who this girl was or how she is now. I sat behind and never saw her face. The wounded were evacuated first. I don t remember how long we remained; I had lost all concept of time. In spite of protests from the group who knew him, one kid was put in handcuffs. At the time I didn t understand why, and the policeman seemed to say something almost to the effect that there was no reason for it at all. I didn t see when they undid his cuffs, but I remember thinking that this treatment made a terrible experience even worse for him. I tried my best to comfort him but knew it would be little help. Later, when things stabilised a little, we were told that he was handcuffed because he had come from an unsecured area. The police was extremely good at carefully explaining what was happening and why; this was a big help and I am grateful for it. Eventually we were moved out into the main corridor of the building, where we joined up into a group of about fifty. Only when I saw the two people who saved my life did any emotion other than mild confusion arise. I broke down shivering in tears in one of their arms. After a few seconds, I came back to my senses and realised that this was not the time. I quickly gathered myself, got the shaking under control, and sat down. We were given some chocolate and soda from the kiosk. I remember making an offhand remark that an inability to find joy in free candy was a sure sign of a bad situation. We all laughed out loud. Gallows humour is a coping mechanism, but in retrospect one almost feels guilty for it. We were shown out in a single file with hands above our heads. I remember an intense concern that someone would slip in the steep, muddy slope and create a misunderstanding. Outside, there were more bodies. Some under improvised covers a tarpaulin from the waffle stand, the deflated bouncy castle but some simply lay there. Everyone I met displayed a courage, a mental discipline and unity of purpose far beyond anything one would ever wish to expect from people this young. Everyone conducted themselves with an attitude that could almost be described as stiff upper lip . Safely across the fjord we were offered blankets. I was asked if I was aware of any injuries, and asked to lift my shirt and show my abdominal region. We were shown into the bus which took us to the hotel used by the survivors and their family. I simply cannot describe in any words the relief I felt when I was able to embrace my living comrades. It was completely unlike anything I had ever felt before in my life. The euphoric feeling was tempered only by the realisation that there would be many I could never see again, comrades whom I had taken great pride in calling my friends, with futures in the service of all mankind, futures I had previously found such great joy in pondering and guessing about. The feeling which continues to upset me the most, is the feeling that so many of my comrades left behind grieving families and friends. Torn away senselessly. I do not know how much more than this relatively sober account of the events on Ut ya I can muster. I would, however, like to offer some reflections. First of all, from the bottom of my heart, I want to thank the police who saved the lives of so many still on the island, the holidaymakers who took aboard swimmers into their boats and the rescue services staffed primarily by volunteers who have spared no effort in trying to soften the blow as much as they can. The opportunity to spend time with those comrades who underwent the same experience as myself has also been an immeasurable aid. I was also so relieved to find my very closest friend among the survivors, which has also been an indescribable help. If I can name a single positive in this tragedy: Had he arrived with his automatic weapon fifteen or twenty minutes prior, he would have arrived during the informational meeting, at a time when the major hall was absolutely jam-packed the death toll would be many times what it ended up being. I am agonisingly aware of the meager comfort this provides to those who have been bereft of their closest, but I do find some solace in this. We cannot sweep under a rug that this was without question! a political attack on the labour movement. But it is thankfully also an attack which has been perceived by everyone as an attack on the Norwegian society, and on a symbol of the wide recruitment to the participatory democracy which lies at our very national soul. I cannot thank the Norwegian people, and indeed the people in other nations who have offered their condolences, enough for their shows of support and shared grief. It really has been a tremendous help to me knowing that so many people feel with us. I also want to thank from the bottom of my heart the rock-steadiness of everyone in both the national and local wings of the Labour Party and Labour Youth in supporting us, and the political milieu in general for their resolute steadfastness saving me from losing yet more that I cherish; our freedoms in a participatory democracy. Our Party has lost many of its very brightest youngsters. Personally I feel an angry spite, a deep restless urge to get the wheels of society going again. I want to show his kind that we will not be broken. We re stronger than that. I will not be frightened into silence and passivity. I want to remember the dead, and honor them by carrying on our common work. I want to end this with a request to everyone who reads this, echoing a statement I read by one of my good friends and comrades: Please, don t let me see any messages of hatred, wishes for the death penalty, anything like that. If anyone should be of the belief that anything will improve by murdering this sad little person, they would be profoundly wrong. All attention now should be plowed into caring for those victims and their relatives who did not share my luck, and not giving an audience to a perpetrator who wants one. Tore Sinding Bekkedal

When Marvell originally released the first plug computer, they created their own version of u-boot with support for their new devices. Unfortunately, this version of u-boot is fairly out of date nowadays compared to mainline u-boot and has several problems. Support for plug computers (such as SheevaPlug and GuruPlug) have been integrated into the mainline u-boot (also known as DENX u-boot) in the meantime and Clint Adams has packaged it for Debian. I finally found the time to test Clint's u-boot binary on my devices and have updated the SheevaPlug installation guide accordingly. If you're have installed Debian to a SheevaPlug according to my instructions, I suggest you upgrade. If you boot from a MMC/SD card, you should be aware that the mmcinit command has been renamed to mmc init in order to be consistent with the naming of other commands. You'll therefore have to update your bootcmd_mmc variable in u-boot like this:

Hello, world! So, now this blog is being syndicated on Planet Debian, and I decided to write something to introduce myself. I was born in Salvador, Bahia, Brazil. I am married to Josy, a wonderful person; we do not have any kids yet, but anyone can notice we both discretely drooling when close to small children. I have a passion for building things, and making things work. Programming happens to be the better building toy that someone already invented. I became am official Debian Developer a little more than one week ago, but I have been working in the project for some time already. In the very beginning I was involved with the Perl group, but now almost all my work is in the Ruby team. I am a PhD student at Federal University of Bahia under supervision of professor Christina Chavez. My research involves three topics I am just fascinated about: Free ("and Open Source") Software, Software Design and Empirical Software Engineering. I am investigating whether it is possible to explain the variation in Structural Complexity by analyzing developer attributes. I want to answer questions such as "do developers with more experience in a project produce more complex code?", or "do developers that focus on a single part of the project produce more complex code than developers who work on the entire project?". As part of my PhD research I have been working on analizo, a multi-language source code analysis and visualization toolkit. I plan to write specifically about it in a later opportunity. My other job is running my own company, Colivre, together with several other great people. Colivre is a cooperative, which basically means that everyone who works there is also an owner of the company. We provide web solutions, in special social networking environments, social media websites and the like. Working on (and with) free software is our premise at Colivre, and most of our work involves either Noosfero or Foswiki. Right now I am taking some time off from Colivre to be able to concentrate on (and finish!) my PhD. Until late September I live at the beautiful Vancouver, BC, Canada, where I am currenly a visiting reasearcher at the Software Practices Lab at UBC under supervision of professor Gail Murphy. So that's it, I hope to post interesting stuff here and to get your feedback whenever you find it's worth. I also maintain a microblog at identi.ca, where you'll find shorter (and more frequent) updates.

Earlier this month, for the first time in 4 years I unchecked the box "I want to attend this conference" in Penta, the DebConf conference management system. Despite this, I *do* want to attend DebConf11 but unfortunately due to work obligations I'm unable to attend this year. Hope to see you in Nicaragua!

average sprint speed: 1 sprint/month I've proposed a DebConf11 BoF on Debian sprints and, more generally, on how we have been using Debian money in the past 1.3 years. As part of the BoF preparation, I've taken the time to review the last 16 months of sprints and check how the Debian Sprint Program which we've recently streamlined and "marketed" quite a bit is going. In particular, I've finally done the homework of preparing the big table of sprints and their costs, in order to evaluate how sustainable the sprint program is. Without further ado, here is the table:

Debian sprints held from April 2010 to June 2011

No	Sprint	When (month)	Where	Attendees	Debian cost (EUR)
1	groupware (report)	04/2010	de	4	240
2	FAI (report)	06/2010	de	7	570
	(DebConf10)	08/2010	us
3	ftpmaster (report)	09/2010	de	3	350
4	DSA (report)	09/2010	de	3	400
5	release team (report)	10/2010	fr	6	0
6	kernel team (report)	10/2010	fr	3	120
7	www team	12/2010	at	5	1430
8	debian med	01/2011	de	25	460
9	security team	01/2011	de	7	1000
10	emdebian	02/2011	uk	17	0
11	ftpmaster	03/2011	de	5	2000
12	groupware	04/2011	de	6	300
13	alioth	05/2011	uk	4	500
14	debian edu	06/2011	de	9	760
15	release team	06/2011	be	5	1640
	(DebConf11)	07/2010	ba
Total	15 sprints	16 months			9770

To better understand the table, several comments are in order:

• numbers are approximate in various ways. In particular, 2010 costs predate a protocol I've agreed upon with the Debian auditors to do proper accounting, hence it is possible that some costs are slightly overestimated
• costs are Debian's costs and do not include reimbursements for non-Debian people that might have attended the sprint. This also explains why some sprints have been at 0 cost: they have been entirely sponsored by other entities interested in supporting Debian, such as companies or institutions.
• where needed, costs have been normalized to Euro
• we have streamlined the sprint process around October 2010 (you can find the underlying rationales in an old RFC). Sprints happened before that data had a different way of being announced and of reporting back to the project.
• the above data are not detailed enough to properly estimate per-person costs, as we lack a breakdown of Debian vs non-Debian attendees
• no costs for DebConf-s are included, as they have their own budgets which aims to be amortized 0 cost in the long run

Please note that the purpose of the table is not to be precise and transparent about Debian finances and how we use them. That is a (very!) worthwhile goal and I do think Debian should do much better in informing its community about how donated money are used to further Debian goals. But that is a broader topic on which the auditors are working; it is not up to me to discuss it here. If you are interested in that topic though, you might want to follow tbm's BoF at DebConf11. The purpose of the table is rather to find out some general figures about Debian sprints held in the recent past:

• we have had a total of 15 sprints in 16 months (about 1 sprint/month)
• we have spent about 10'000 EUR for sprints, for an average per-sprint cost of about 650 EUR
• all sprints have been held in Europe
  (given DD geographical distribution, I find quite surprising that no group of US-based Debian hackers have asked for sprint sponsoring)
• about half of the sprints have been held in Germany
  (probably thanks to DD-density there and also to the welcoming attitude of friendly venues such as the Linux Hotel)
• the variance of per-person costs is high: there are very cheap sprints (e.g. when people travel within Europe by train and stay at friends') and quite expensive sprints (e.g. when one or more of the participant needs to be flown oversea which has happened only once, up to now)

I'm personally quite happy about those figures. Enabling volunteer developers to meet and hack together in person is possibly the most valuable way of using donated money. Having 1 sprint/month is not bad, but in a project the size of Debian is quite possibly a minimum. Doing more than that is highly desirable. It is also financially sustainable, especially if we will be able to show by actually having more sprints and being transparent about them that we can put into good use donated money. Another, more subtle, aspect of sustainability is that related to sprint management. Processing sprint requests and ensuring that transparency guidelines are actually followed by the organizers is still quite some work. I've been mostly doing that myself up to now, which is all fine and well, but does not necessarily scale. Other organizations (such as KDE e.V.) have realized that to the point of having hired people specifically to manage sprints in an otherwise volunteer community. In Debian we are quite keen of maintaining the project running on a volunteer basis. At the same time I feel we should have more room for scalability in the number of sprints we could run. So if you are looking for a management task to help Debian with, think about becoming, err, "sprint master", and contact me. If otherwise you want to focus on Debian hacking, what are you waiting for? Check the guidelines and propose your sprint! To know more about sprints, Debian money, and how you could help with all that, be sure not to miss the Sprint and money BoF.