Executive summary Debian should consider:

• less fine-grained software packaging for overall simplification of the system and its development
• less flexibility at the lower levels of the software stack, again for overall simplicity
• reducing friction in the development workflow, especially by making it possible to branch and merge at the whole system level

Introduction I've taken a job with Codethink, as part of a team to develop a new embedded Linux system, called Baserock. Baserock is not a distribution as such, but deals with many of the same issues. I thought it might be interesting to write out my new thoughts related to this, since they might be useful for proper distributions to think about too. I'll hasten to add that many of these thoughts are not originally mine. I shamelessly adopt any idea that I think is good. The core ideas for Baserock come from Rob Taylor, Daniel Silverstone, and Paul Sherwood, my colleagues and bosses at Codethink. At the recent GNOME Summit in Montreal, I was greatly influenced by Colin Walters. This is also not an advertisment for Baserock, but since my of my current thinking come from that project, I'll discuss things in the context of Baserock. Finally, I'm writing this to express my personal opinion and thoughts. I'm not speaking for anyone else, least of all Codethink. On the package abstraction Baserock abandons the idea of packages for all individual programs. In the early 1990s, when Linux was new, and the number of packages in a distribution could be counted on two hands in binary, packages were a great idea. It was feasible to know at least something about every piece of software, and pick the exact set of software to install on a machine. It is still feasible to do so, but only in quite restricted circumstances. For example, picking the packages to install a DNS server, or an NFS server, or a mail server, by hand, without using meta packages or tasks (in Debian terminology), is still quite possible. On embedded devices, there's also usually only a handful of programs installed, and the people doing the install can be expected to understand all of them and decide which ones to install on each specific device. However, those are the exceptions, and they're getting rarer. For most people, manually picking software to install is much too tedious. In Debian, we've realised this a great many years ago, and developed meta packages (whose only purpose is to depend on other packages) and tasks (solves the same problem, but differently). These make it possible for a user to say "I want to have the GNOME desktop environment", and not have to worry about finding every piece that belongs in GNOME, and install that separately. For much of the past decade, computers have had sufficient hard disk space that it is no longer necessary to be quite so picky about what to install. A new cheap laptop will now typically come with at least 250 gigabytes of disk space. An expensive one, with an SSD drive, will have at least 128 gigabytes. A fairly complete desktop install uses less than ten gigabytes, so there's rarely a need to pick and choose between the various components. From a usability point of view, choosing from a list of a dozen or two options is much easier than from a list of thirty five thousand (the number of Debian packages as I write this). This is one reason why Baserock won't be using the traditional package abstraction. Instead, we'll collect programs into larger collections, called strata, which form some kind of logical or functional whole. So there'll be one stratum for the core userland software: init, shell, libc, perhaps a bit more. There'll be another for a development environment, one for a desktop environment, etc. Another, equally important reason, to move beyond packages is the problems caused by the combinatorial explosion of packages and versions. Colin Walters talks about this very well. When every system has a fairly unique set of packages, and versions of them, it becomes much harder to ensure that software works well for everyone, that upgrades work well, and that any problems get solved. When the number of possible package combinations is small, getting the interactions between various packages right is easier, QA has a much easier time to test all upgrade paths, and manual test coverage improves a lot when everyone is testing the same versions. Even debugging gets easier, when everyone can easily run the same versions. Grouping software into bigger collections does reduce flexibility of what gets installed. In some cases this is important: very constrainted embedded devices, for example, still need to be very picky about what software gets installed. However, even for them, the price of flash storage is low enough that it might not matter too much, anymore. The benefit of a simpler overall system may well outweigh the benefit of fine-grained software choice. Everything in git In Baserock, we'll be building everything from source in git. It will not be possible to build anything, unless the source is committed. This will allow us to track, for each binary blob we produce, the precise sources that were used to bulid it. We will also try to achieve something a bit more ambitious: anything that affects any bit in the final system image can be traced to files committed to git. This means tracking also all configuration settings for the build, and the whole build environment, in git. This is important for us so that we can reproduce an image used in the field. When a customer is deploying a specific image, and needs it to be changed, we want to be able to make the change with the minimal changes compared to the previous version of the image. This requires that we can re-create the original image, from source, bit for bit, so that when we make the actual change, only the changes we need to make affect the image. We will make it easy to branch and merge not just individual projects, but the whole system. This will make it easy to do large changes to the system, such as transitioning to a new version of GNOME, or the toolchain. Currently, in Debian, such large changes need to be serialised, so that they do not affect each other. It is easy, for example, for a GNOME transition to be broken by a toolchain transition. Branching and merging has long been considered the best available solution for concurrent development within a project. With Baserock, we want to have that for the whole system. Our build servers will be able to build images for each branch, without requiring massive hardware investment: any software that is shared between branches only gets built once. Launchpad PPAs and similar solutions provide many of the benefits of branching and merging on the system level. However, they're much more work than "git checkout -b gnome-3.4-transition". I believe that the git based approach will make concurrent development much more efficient. Ask me in a year if I was right. Git, git, and only git There are a lot of version control systems in active use. For the sake of simplicity, we'll use only git. When an upstream project uses something else, we'll import their code into git. Luckily, there are tools for that. The import and updates to it will be fully automatic, of course. Git is not my favorite version control system, but it's clearly the winner. Everything else will eventually fade away into obscurity. Or that's what we think. If it turns out that we're wrong about that, we'll switch to something else. However, we do not intend to have to deal with more than one at a time. Life's too short to use all possible tools at the same time. Tracking upstreams closely We will track upstream version control repositories, and we will have an automatic mechanism of building our binaries directly from git. This will, we hope, make it easy to follow closely upstream development, so that when, say, GNOME developers make commits, we want to be able to generate a new system image which includes those changes the same day, if not within minutes, rather than waiting days or weeks or months. This kind of closeness is greatly enhanced by having everything in version control. When upstream commits changes to their version control system, we'll mirror them automatically, and this then triggers a new system image build. When upstream makes changes that do not work, we can easily create a branch from any earlier commit, and build images off that branch. This will, we hope, also make it simpler to make changes, and give them back to upstream. Whenever we change anything, it'll be done in a branch, and we'll have system images available to test the change. So not only will upstream be able to easily get the change from our git repository, they'll also be easily verify, on a running system, that the change fixes the problem. Automatic testing, maybe even test driven development We will be automatically building system images from git commits for Baserock. This will potentially result in a very large number of images. We can't possibly test all of them manually, so we will implement some kind of automatic testing. The details of that are still under discussion. I hope to be able to start adding some test driven development to Baserock systems. In other words, when we are requested to make changes to the system, I want the specification to be provided as executable tests. This will probably be impossible in real life, but I can hope. I've talked about doing the same thing for Debian, but it's much harder to push through such changes in an established, large project. Solving the install, upgrade, and downgrade problems All mainstream Linux distributions are based on packages, and they all, pretty much, do installations and upgrades by unpacking packages onto a running system, and then maybe running some scripts from the packages. This works well for a completely idle system, but not so well on systems that are in active use. Colin Walters again talks about this. For installation of new software, the problem is that someone or something may invoke it before it is fully configured by the package's maintainer script. For example, a web application might unpack in such a way that the web server notices it, and a web browser may request the web app to run before it is configured to connect to the right database. Or a GUI program may unpack a .desktop file before the executable or its data files are unpacked, and a user may notice the program in their menu and start it, resulting in an error. Upgrades suffer from additonal problems. Software that gets upgraded may be running during the upgrade. Should the package manager replace the software's data files with new versions, which may be in a format that the old program does not understand? Or install new plugins that will cause the old version of the program to segfault? If the package manager does that, users may experience turbulence without having put on their seat belts. If it doesn't do that, it can't install the package, or it needs to wait, perhaps for a very long time, for a safe time to do the upgrade. These problems have usually been either ignored, or solved by using package specific hacks. For example, plugins might be stored in a directory that embeds the program's version number, ensuring that the old version won't see the new plugins. Some people would like to apply installs and upgrades only at shutdown or bootup, but that has other problems. None of the hacks solve the downgrade problem. The package managers can replace a package with an older version, and often this works well. However, in many cases, any package maintainer scripts won't be able to deal with downgrades. For example, they might convert data files to a new format or name or location upon upgrades, but won't try to undo that if the package gets downgraded. Given the combinatorial explosion of package versions, it's perhaps just as well that they don't try. For Baserock, we absolutely need to have downgrades. We need to be able to go back to a previous version of the system if an upgrade fails. Traditionally, this has been done by providing a "factory reset", where the current version of the system gets replaced with whatever version was installed in the factory. We want that, but we also want to be able to choose other versions, not just the factory one. If a device is running version X, and upgrades to X+1, but that version turns out to be a dud, we want to be able to go back to X, rather than all the way back to the factory version. The approach we'll be taking with Baserock relies on btrfs and subvolumes and snapshots. Each version of the system will be installed in a separate subvolume, which gets cloned from the previous version, using copy-on-write to conserve space. We'll make the bootloader be able to choose a version of the system to boot, and (waving hands here) add some logic to be able to automatically revert to the previous version when necessary. We expect this to work better and more reliably than the current package based one. Making choices Debian is pretty bad at making choices. Almost always, when faced with a need to choose between alternative solutions for the same problem, we choose all of them. For example, we support pretty much every init implementation, various implementations of /bin/sh, and we even have at least three entirely different kernels. Sometimes this is non-choice is a good thing. Our users may need features that only one of the kernels support, for example. And we certainly need to be able to provide both mysql and postresql, since various software we want to provide to our uses needs one and won't work with the other. At other times, the inability to choose causes trouble. Do we really need to support more than one implemenation of /bin/sh? By supporting both dash and bash for that, we double the load on testing and QA, and introduce yet another variable to deal with into any debugging situation involving shell scripts. Especially for core components of the system, it makes sense to limit the flexibility of users to pick and choose. Combinatorial explosion d j vu. Every binary choice doubles the number of possible combinations that need to be tested and supported and checked during debugging. Flexibility begets complexity, complexity begets problems. This is less of a problem at upper levels of the software stack. At the very top level, it doesn't really matter if there are many choices. If a user can freely choose between vi and Emacs, and this does not add complexity at the system level, since nothing else is affected by that choice. However, if we were to add a choice between glibc, eglibc, and uClibc for the system C library, then everything else in the system needs to be tested three times rather than once. Reducing the friction coefficient for system development Currently, a Debian developer takes upstream code, adds packaging, perhaps adds some patches (using one of several methods), builds a binary package, tests it, uploads it, and waits for the build daemons and the package archive and user-testers to report any problems. That's quite a number of steps to go through for the simple act of adding a new program to Debian, or updating it to a new version. Some of it can be automated, but there's still hoops to jump through. Friction does not prevent you from getting stuff done, but the more friction there is, the more energy you have to spend to get it done. Friction slows down the crucial hack-build-test cycle of software development, and that hurts productivity a lot. Every time a developer has to jump through any hoops, or wait for anything, he slows down. It is, of course, not just a matter of the number of steps. Debian requires a source package to be uploaded with the binary package. Many, if not most, packages in Debian are maintained using version control systems. Having to generate a source package and then wait for it to be uploaded is unnecessary work. The build daemon could get the source from version control directly. With signed commits, this is as safe as uploading a tarball. The above examples are specific to maintaining a single package. The friction that really hurts Debian is the friction of making large-scale changes, or changes that affect many packages. I've already mention the difficulty of making large transitions above. Another case is making policy changes, and then implementing them. An excellent example of that is in Debian is the policy change to use /usr/share/doc for documentation, instead of /usr/doc. This took us many years to do. We are, I think, perhaps a little better at such things now, but even so, it is something that should not take more than a few days to implement, rather than half a decade. On the future of distributions Occasionally, people say things like "distributions are not needed", or that "distributions are an unnecessary buffer between upstream developers and users". Some even claim that there should only be one distribution. I disagree. A common view of a Linux distribution is that it takes some source provided by upstream, compiles that, adds an installer, and gives all of that to the users. This view is too simplistic. The important part of developing a distribution is choosing the upstream projects and their versions wisely, and then integrating them into a whole system that works well. The integration part is particularly important. Many upstreams are not even aware of each other, nor should they need to be, even if their software may need to interact with each other. For example, not every developer of HTTP servers should need to be aware of every web application, or vice versa. (It they had to be, it'd be a combinatorial explosion that'd ruin everything, again.) Instead, someone needs to set a policy of how web apps and web servers interface, what their common interface is, and what files should be put where, for web apps to work out of the box, with minimal fuss for the users. That's part of the integration work that goes into a Linux distribution. For Debian, such decisions are recorded in the Policy Manual and its various sub-policies. Further, distributions provide quality assurance, particularly at the system level. It's not realistic to expect most upstream projects to do that. It's a whole different skillset and approach that is needed to develop a system, rather than just a single component. Distributions also provide user support, security support, longer term support than many upstreams, and port software to a much wider range of architectures and platforms than most upstreams actively care about, have access to, or even know about. In some cases, these are things that can and should be done in collaboration with upstreams; if nothing else, portability fixes should be given back to upstreams. So I do think distributions have a bright future, but the way they're working will need to change.

Maybe you've already heard, that Google is starting to charge a fee for using the Maps API (Google may decide you're an non-profit organization and waive the fees). That is, in my opinion, Google telling you to look for better alternatives. And thanks to the awesome effort of the OpenStreetMaps community, there is such an alternative. Thus today's addition to the mini-tips series will tell you, how you get an OpenStreetMaps-powered map on your website.

1. First you need to determine the coordinates, where you want to center, the map. You can do this with OpenRouteService.org. Just search for the address, right-click on the marker highlighting the result, and set it as e.g. your start point, that'll give you the coordinates in the start field on the left. Alternatively you can just move your pointer over a place on the map and have a look at the lower right corner, where the current coordinates for the pointer position are displayed.
2. Then you need to add a few things to the website, where the card is to be shown:
   • Include the OpenLayers script in the <head> of your webpage:

     <script src="http://www.openlayers.org/api/OpenLayers.js" type="text/javascript" />
     

   • Add a short ECMAScript snippet right after that, which will initialize your map:

     function osmInit()  
       document.getElementById("ol_map").style.display = 'block';
       var lonLat = new OpenLayers.LonLat(13.376096, 52.518598) // Center of the map
       .transform(
         new OpenLayers.Projection("EPSG:4326"),   // transform from WGS 1984
         new OpenLayers.Projection("EPSG:900913")  // to Spherical Mercator Projection
       );
       var map = new OpenLayers.Map("ol_map");
       var osmlayer = new OpenLayers.Layer.OSM();
       map.addLayer(osmlayer);
       map.setCenter(lonLat,
         16 // Zoom level
       );
      
     

     The first line is not needed, but I find it nice to hide the map on default, in case the user has disabled ECMAScript they haven't a large empty area somewhere on the website. The first line changes then the display value from "none" to "block". The rest is, I hope, already explained by the code itself.
   • Now only two more changes are needed, first you need to add onload="osmInit();" to your <body> tag. And afterwards an element (say a <p>) with the ID ol_map somewhere in the website <body>. You should set a width and height for that tag (style is your friend), otherwise the map might be bigger than you like.
3. Done. Enjoy your free (not just as in free beer) map!

You can, of course, customize your map further by adding additional layers and changing the available options (e.g. the zoom level). If you would want to mark a location on your map, you'd need to add something like

var markers = new OpenLayers.Layer.Markers("POI1");
map.addLayer(markers);
markers.addMarker(new OpenLayers.Marker(lonLat));

to your ECMAScript snipped after you added the OpenStreetMaps layer. These three lines would mark the location given by the variable lonLat. For further information, you can check out the OpenStreetMaps wiki and the OpenLayers documentation. In case you wonder, where my coordinates from this example point, you can check. Oh, and don't forget to consider a donation to OpenStreetMaps, so the project can pay the bills for its servers.

A few years ago I got up on stage and briefly talked about how the Linux community contained far too many people who were willing to engage in entirely inappropriate behaviour, how this discouraged people from getting involved and how we weren't very good at standing up against that sort of behaviour. Despite doing this in front of several hundred people, and despite the video of me doing so then being uploaded to the internet, this got me a sum total of:

• No death threats
• No discussion about any of my physical attributes or lack thereof
• No stalkers
• No accusations that I was selling out the Linux community
• No accusations that I was a traitor to my gender
• No real negative feedback at all[1]

Which is, really, what you'd expect, right? The internet seems intent on telling me otherwise:

Well, she didn't do herself any favors by talking at conferences about women in tech, or setting up a feminist movement. If you wanted to attract abuse, that's a good way to go about it. It should be expected.
(Source)

MikeeUSA is a troll. He has no means to actually harm anyone, and he does it purely for the lulz.

Thus, MikeeUSA trolled a woman, and she took the bait. I just don't get why this is news, I've been trolled before, I don't get a news story.
(Source)

I was going to start a rant about how this behavior is encouraged by the macho men online, but this was just one guy harassing her. "Due to harassment" reads as due to harassment from the community, but she gave in to one idiot. She let him win.
(Source)

The full comment thread has rather more examples. If you stand up and say anything controversial, you should expect abuse. And if you let that abuse change your behaviour in any way, you've let the trolls win.

These attitudes are problematic.

The immediate assumption underlying such advice is that the degree of abuse is related to what you've said, not who you are. I'm reasonably visible in the geek world. I've said a few controversial things. The worst thing that's happened to me has been Ryan Farmer deciding to subscribe me to several thousand mailing lists. Inconvenient, but not really threatening. I haven't, for instance, been sent death threats. Nobody has threatened to rape me. And even if they had done, I wouldn't need to worry too much - there's a rather stronger track record of violent antifeminism being targeted at women than men.

I don't have to worry about this kind of thing. That means I don't get to tell other people that they should have expected it. Nor do I get to tell them that they should ignore it, or that if they don't call the police then they have no grounds to complain. And nor does anyone else.

The trolls don't win because someone decides that getting out of the tech business is more appealing than continuing to face abuse. The trolls win because we consider their behaviour acceptable and then blame the victim for inviting them in the first place. That needs to change.

[1] It was justifiably pointed out that saying all this while standing on stage next to a mostly naked guy wearing a loincloth with a raccoon tail covering his penis may have weakened my message somewhat.

comments

Because of the very annoying new Mozilla release politics (which look like a pissing contest with the similar annoying Google Chrome/Chromium release schedule), Ubuntu kicked out Mozilla XULRunner with its recent release of 11.10 Oneiric. And with XULRunner, Ubuntu also kicked out Conkeror and all other XULRunner reverse dependencies, too. Meh. Sparked by this thread on the Conkeror mailing list, I extended the Debian package s /usr/bin/conkeror wrapper script so that it looks for firefox in the search path, too, if no xulrunner* is found, and added an alternative dependency on firefox versions greater or equal to 3.5, too. From now on, if the wrapper script finds no xulrunner but firefox in the search path, it calls firefox -app instead of xulrunner-$VERSION to start Conkeror. With the expection of the about:-page showing the orange-blue Firefox logo and claiming that this is Firefox $CONKEROR_VERSION , it works as expected on my Toshiba AC100 netbook running the armel port of Ubuntu 11.10. From version 1.0~~pre+git1110272207-~nightly1 on, the Conkeror Nightly Built Debian Packages will be installable on Ubuntu 11.10 Oneiric again without the need to install or keep XULRunner version from Ubuntu 11.04 Natty. For those who don t want to use the nightly builds, I created a (currently still empty) specific PPA for Conkeror where I ll probably upload all the conkeror packages I upload to Debian Unstable.

Wishing the entire Debian community worldwide a happy and prosperous Gujarati New Year (Vikram Samvat 2068 called Shobhan.) The picture above, is one that's been making the rounds of the Indian blogosphere. It purports to be a NASA satellite image of India on Diwali (which was last night.) Actually it is a composite of several night shots taken over a period of a few years. Also the red dots are people who have been injured by fireworks and the green dots those immobilized by the ingestion of too much baraphi and halwa. But I could be wrong about that. I've had some terrific news these past few days but alas I'm not at liberty to divulge it just yet...

The RISE co-op is the sole shareholder in the Social Enterprise Mark CIC and its members have been called to an Extraordinary General Meeting during the lunch break of next Tuesday s Knowing and Growing conference at UWE Bristol. The RISE board has proposed Four Special Resolutions that would dissolve the co-op and transfer all assets as windfalls to the SEM CIC and a trust, ignoring RISE Ltd s Memorandum of Association. software.coop is calling on other RISE members to attend the EGM and oppose this demutualisation attempt. Update: dissolved but not demutualised (see below) RISE is constituted as a common ownership co-op and its Memorandum of Association contains a clause that directs the assets to be transferred to another common ownership social enterprise organisation if the co-operative is dissolved. However, unlike the CIC asset lock, there is no independent regulator enforcing it and, unlike in many co-ops, there has been no requirement for new members to pledge to obey the RISE common ownership clause at an individual level and there has been no member education about common ownership in the last three years. This demutualisation is the wrong solution for RISE because:

• it has recieved much public money and money from members on the understanding that it would be kept in common ownership;
• there are common ownership social enterprises that could really benefit from the legacy assets at this time of budget cuts, but if those assets go to a CIC, they are lost from common ownership;
• the EGM is the day after the global launch of the United Nations International Year of Co-operatives, which is about promoting mutualism around the world and the RISE co-operative should support this;
• many RISE members are mutuals, including the world s largest consumer co-operative and demutualisation would be embarassing to them;
• demutualisation should be discussed seriously, through a proper democratic process before the regular Annual General Meeting, not sprung from the board to members in a short meeting during the lunch break of another event;
• the proposal has not been published on RISE s website or email newsletter;
• appointing the former directors of a dissolved co-op as trustees without member oversight seems unlikely to secure the assets in the long term.

software.coop will vote against the demutualisation, in favour of social enterprise, and calls on other RISE members to show solidarity with the co-operative and common ownership social enterprise movements. Update 2 November: rise has announced its dissolution but they ve got to give further consideration to where the assets go because the demutualisation resolutions were defeated. I fear that they ll still try some way to bail out the Social Enterprise Mark despite the rejection, but I hope they ll do the right thing and give the assets to good common ownership social enterprise like the RISE Memorandum of Association requires.

Finally! Last Friday, after two years worth of work, I finally got the first box of books for the Construcci n Colaborativa del Conocimiento (Collaborative Knowledge Construction) project I worked on as a coordinator together with Alejandro Miranda (pooka), and together with a large group of 11 authors: Translating over from the back cover text (and this is just a quick translation from me It reads better in Spanish ;-) ):

What defines us as humans is our ability, on one side, to
create knowledge, and on the other, to share or communicate it with our neighbors. Both features have worked together over tens of thousands of years, and, working together, have led the knowledge to transcend the individual, avoiding the need to rediscovery or reinvention of is already known. Sharing knowledge is what has taken our species to the dominant role it occupies today. But knowledge creation and sharing has seen a deep transformation in recent decades, thanks to the quick evolution of telecommunications, specially the massification of Internet and cellular telephony. We are transiting towards the so desired and at the same time so feared knowledge society. In this book, eleven authors from very different disciplinary backgrounds and geographic origins ellaborate on how a hyper-connected world has modified the basic rules of interaction in areas as diverse as artistic creation, social organizations, computer code development, education or the productive sector. This book is the result of a year worth of work for in the "Collaborative Construction of Knowledge" seminar, during which we
used the same new forms of knowledge production we have studied. The videos of the sessions, electronic participations and the full contents of this book are available under a permisive license at
http://seminario.edusol.info/seco3/

We will soon have the book ready in IIEc's e-store (which is mostly meant for national requests). I am also uploading the book to the lulu.com self-publishing service, and we are working on a epub-like edition. Right now it is still not available, but it should be there in some days. I will keep you posted. Meanwhile, the full contents can be read online at http://seminario.edusol.info/seco3

Reading Sylvestre Ledru's announcement, that sun-java6 was removed from the archives, I'd like to point out, that the recommended immediate purge of any binary packages built by sun-java6 should be postponed until you did have time to check, that important applications are working for you with the OpenJDK implementation. I need to retain sun-java6-plugin until upstream bug #588 for icedtea-plugin is fixed. If you have equal requirements (using ELSTER is required by tax law for me), make sure everything works with OpenJDK/IcedTea, otherwise you need to fetch the latest shipped version from snapshot.debian.org or install something directly from Oracle. Now, as security issues were mentioned in Sylvestre's post, I'm NOT recommending to have sun-java6-plugin running in your browser on default. Just switch the Java plugins before you really need it. For Iceweasel you can (de-)activate plugins in the add-on manager's plugin tab. Apart from that I'd recommend to not surf with any scripting or plugins active at all, NoScript! is a great way to do this conveniently.

As is natural for an LTS cycle, lots of people are thinking and talking about work focused on quality rather than features. With Canonical extending LTS support to five years on the desktop for 12.04, much of this is quite rightly focused on the desktop. I'm really not a desktop hacker in any way, shape, or form, though. I spent my first few years in Ubuntu working mainly on the installer - I still do, although I do some other things now too - and I used to say only half-jokingly that my job was done once X started. Of course there are plenty of bugs I can fix, but I wanted to see if I could do something with a bit more structure, so I got to thinking about projects we could work on at the foundations level that would make a big difference. Image build pipeline One difficulty we have is that quite a few of our bugs - especially installer bugs, although this goes for some other things too - are only really caught when people are doing coordinated image testing just before a milestone release. Now, it takes a while to do all the builds and then it takes a while to test them. The excellent work of the QA team has meant that testing is much quicker now than it used to be, and a certain amount of smoke-testing is automated (particularly for server images). On the other hand, the build phase has only got longer as we've added more flavours and architectures, particularly as some parts of the process are still serialised per architecture or subarchitecture so ARM builds in particular take a very long time indeed. Exact timings are a bit difficult to get for various reasons, but I think the minimum time between a developer uploading a fix and us having a full set of candidate images on all architectures including that fix is currently somewhere north of eight hours, and that's with people cutting corners and pulling strings which is a suboptimal thing to have to do around release time. This obviously makes us reluctant to respin for anything short of showstopper bugs. If we could get things down to something closer to two hours, respins would be a much less horrible proposition and so we might be able to fix a few bugs that are serious but not showstoppers, not to mention that the release team would feel less burned out. We discussed this problem at the release sprint, and came up with a laundry list of improvements; I've scheduled this for discussion at UDS in case we can think of any more. Please come along if you're interested! One thing in particular that I'm working on is refactoring Germinate, a tool which dates right back to our first meeting before Ubuntu was even called Ubuntu and whose job is to expand dependencies starting from our lists of "seed" packages; we use this, among other things, to generate Task fields in the archive and to decide which packages to copy into our images. This was acceptably quick in 2004, but now that we run it forty times (eight flavours multiplied by five architectures) at the end of every publisher run it's actually become rather a serious performance problem: cron.germinate takes about ten minutes, which is over a third of the typical publisher runtime. It parses Packages files eight times as often as it needs to, Sources files forty times as often as it needs to, and recalculates the dependency tree of the base system five times as often as it needs to. I am confident that we can significantly reduce the runtime here, and I think there's some hope that we might be able to move the publisher back to a 30-minute cycle, which would increase the velocity of Ubuntu development in general. Maintaining the development release Our release cycle always starts with syncing and merging packages from Debian unstable (or testing in the case of LTS cycles). The vast majority of packages in Ubuntu arrive this way, and generally speaking if we didn't do this we would fall behind in ways that would be difficult to recover from later. However, this does mean that we get a "big bang" of changes at the start of the cycle, and it takes a while for the archive to be usable again. Furthermore, even once we've taken care of this, we have a long-established rhythm where the first part of the cycle is mainly about feature development and the second part of the cycle is mainly about stabilisation. As a result, we've got used to the archive being fairly broken for the first few months, and we even tell people that they shouldn't expect things to work reliably until somewhere approaching beta. This makes some kind of sense from the inside. But how are you supposed to do feature development that relies on other things in the development release? In the first few years of Ubuntu, this question didn't matter very much. Nearly all the people doing serious feature development were themselves serious Ubuntu developers; they were capable of fixing problems in the development release as they went along, and while it got in their way a little bit it wasn't all that big a deal. Now, though, we have people focusing on things like Unity development, and we shouldn't assume that just because somebody is (say) an OpenGL expert or a window management expert that they should be able to recover from arbitrary failures in development release upgrades. One of the best things we could do to help the 12.04 desktop be more stable is to have the entire system be less unstable as we go along, so that developers further up the stack don't have to be distracted by things wobbling underneath them. Plus, it's just good software engineering to keep the basics working as you go along: it should always build, it should always install, it should always upgrade. Ubuntu is too big to do something like having everyone stop any time the build breaks, the way you might do in a smaller project, but we shouldn't let things slide for months either. I've been talking to Rick Spencer and the other Ubuntu engineering leads at Canonical about this. Canonical has a system of "rotations", where you can go off to another team for a while if you're in need of a change or want to branch out a bit; so I proposed that we allow our engineers to spend a month or two at a time on what I'm calling the +1 Maintenance Team, whose job is simply to keep the development release buildable, installable, and upgradeable at all times. Rick has been very receptive to this, and we're going to be running this as a trial throughout the 12.04 cycle, with probably about three people at a time. As well as being professional archive gardeners, these people will also work on developing infrastructure to help us keep better track of what we need to do. For instance, we could deploy better tools from Debian QA to help us track uninstallable packages, or we could enhance some of our many existing reports to have bug links and/or comment facilities, or we could spruce up the weather report; there are lots of things we could do to make our own lives easier. By 12.04, I would like, in no particular order:

• Precise to have been more or less continuously usable from Alpha 1 onward for people with reasonable general technical ability
• Canonical engineering teams outside Ubuntu (DX, Ubuntu One, Launchpad, etc.) to be comfortable with running the development release on at least one system from Alpha 2 onward
• Installability problems in daily image builds to be dealt with within one working day, or preferably before they even make it to daily builds
• The archive to be close to consistent as we start milestone preparation, rather than the release team having to scramble to make it so
• A very significant reduction in our long-term backlog of automatically-detected problems

Of course, this overlaps to a certain extent with the kinds of things that the MOTU team have been doing for years, not to mention with what all developers should be doing to keep their own houses in reasonable order, and I'd like us to work together on this; we're trying to provide some extra hands here to make Ubuntu better for everyone, not take over! I would love this to be an opportunity to re-energise MOTU and bring some new people on board. I've registered a couple of blueprints (priorities, infrastructure) for discussion at UDS. These are deliberately open-ended skeleton sessions, and I'll try to make sure they're scheduled fairly early in the week, so that we have time for break-out sessions later on. If you're interested, please come along and give your feedback!

My generous employer has kindly offered to host the first Bug Squashing Party of the now commencing Wheezy Release BSP Marathon. Yay! We'll meet during the first weekend in December (Friday 2nd to Sunday 4th) in Hildesheim, Germany. There's a wiki page listing all important information. Please also use this page to sign up if you want to join us.

Today I attended the Stop HRL demonstration [1]. The government plans to spend $100,000,000 of federal money and $50,000,000 of Victorian state money to build a new coal power station. The state government has imposed some unreasonable restrictions on renewable energy which includes allowing a single person who objects within 2Km of a wind turbine to block the project while also not allowing anyone to object to coal power plants or fracking (both of which have proven health implications). Today VCAT had a hearing about this issue, HRL wants to double the size of the proposed coal power plant while Environment Victoria, Doctors for the Environment Australia, local climate action group Locals Into Victoria s Environment Martin Shield from Climate Action Moreland are opposing the plan. What the government should be doing is permitting (if not encouraging) renewable energy production, particularly wind power which will drive down energy prices by undercutting the energy auctions due to it s almost zero marginal cost. The government should not be spending tax payer money on new coal projects due to climate change and air pollution in addition to the fact that coal is simply more expensive in the long term. As usual it s disappointing that the National party which supposedly represents farmers is supporting the government in such things. The pollution caused by coal power and fracking ends up damaging farmland. The Liberal party doesn t have a majority on it s own in either house of the state government, so this proposed coal power plant could be stopped immediately if the National party supported renewable power, or even if they merely opposed giving $50,000,000 of state tax money to HRL. The demonstration went reasonably well, there was about 80 people there when I arrived and by the time it ended there were more than 150 people. I took the above picture at the start and it missed many people who were off to the right. I think that s a good result for a demonstration that had little publicity and was held in business hours with really bad weather. Update: On Sunday the 13th of November at 13:00-14:30 there is a Going Backwards Under Baillieu demonstration at Parliament House. This is to mark the negative affects that the Liberal state government has had on the environment of Victoria.

• [1] http://www.stophrl.org/

There a many, many, e-book reader devices available these days, and they're quickly becoming pretty affordable. The currently cheapest device in Germany (that I know of) is the TrekStor eBook Reader 3.0, model number EBR30-a, at 59.- Euros via Weltbild or Hugendubel. The device has an 800x480 7" TFT (yep, no e-ink), 2100mAh battery, it can display PDFs, EPUB, and TXT files (and Adobe DRM crap, which I don't really care about), it has an accelerometer which allows for landscape/portrait switching, it can play MP3, OGG, WAV, and WMA audio files (headphone jack), it can display pictures (BMP, GIF, JPG, even PNG, though that's not mentioned in the vendor's specs), and it has 2GB internal storage for books/music/pictures. Uploading of (non-DRM) content is done by a simple file copy, it enumerates as a standard USB mass storage device with FAT filesystem. It's a relatively nice reader for the price, I've read a few PDFs (datasheets, presentations) on it in the subway/train while listening to music from the device and it's quite OK for my purposes. So much for the review part. However, I didn't really buy it for reading books on it, I was more interested in taking it apart, of course ;-) My hope was that it would turn out to be a really cheap device running Linux/U-Boot which would be perfect for playing around with embedded Linux stuff. Unfortunately, I wasn't so lucky (it seems). I've posted a few photos of the device and its hardware components on my flickr account and over at randomprojects.org, together with all the information I was able to find out so far. Here's a quick summary:

• Main CPU/SoC: FI E200 B6077BA 26P1
• RAM: MIRA P3S12D40ETP (512MBit / 64MByte DDR SDRAM, max. 200MHz)
• NAND flash: Samsung K9GAG08U0E (16GBit / 2GByte, x8, 3.3V)
• Battery management: KrossPower AXP199 A5004AB 36G
• RTC/clock/calender chip (I2C): H8563S
• Some accelerometer (to switch between landscape/portait mode), model unclear so far, maybe the chip labeled 605 132?

There are public datasheets for most of the hardware components (see randomprojects.org for links), but unfortunately the most important one (for the CPU) is not yet found/identified. I was told that the CPU/SoC is probably based on an ARM9 (ARM926EJ-S) core and the firmware running on it seems to be some uCos-based RTOS (not Linux, unfortunately). So far I was not able to find out the vendor name or website of the "FI E200" CPU/SoC (let alone any datasheets), any hints would be highly appreciated. I checked arm.com: Processor Licensees, but the only two companies whose name starts with "F" having licensed an ARM9 core are Fujitsu and Freescale, which doesn't fit, I think? I could (and probably will) check the PCB for RX/TX lines on an UART and/or JTAG pads (none are obviously labelled), and given that it's and ARM9 core there is a good chance that OpenOCD can be used and that a standard cross-gcc for ARM will work. However, that is all pretty pointless until it's clear which SoC exactly is used, and thus whether there is already Linux and/or U-Boot support for it and/or whether datasheets are available so that the respective code could be written. Without datasheets, this is going to be a pretty painful experience, not really worth investing much time, IMHO. If anyone knows more about the vendor/device and respective datasheets, please let me know. Thanks!

If you have been following Lintian s development closely, you will probably have noticed that I have not really done anything there for the past week. Instead I have turned my focus on our testing migration script, britney2. First, I have created a minimal test suite[1]. It started as 4 simple tests and by now it contains about 30 tests. The size of each test is rather small; the largest tests are about 1600 binary packages in total[2], but most are 2-20 binary packages in total. Thus the test suite is rather fast compared to a live data sample , which easily takes more than 10 minutes for a single run. Unfortunately, hand-crafting the test data is somewhat annoying and easy to get wrong. The test suite has a somewhat unfair focus on auto-hint [3] cases, so the current britney2 fails up to 14 tests. Some of these appears to fail because the auto-hinter (for some reason) receives incomplete information about the situation. To my knowledge we not been able to debug the situation, but Adam has a refactor branch that does not seem to have this issue. Personally I am hoping it will soon be merged into the master branch, especially because it seems to simplify a lot of common operations. Joachim Breitner (who has been working on a SAT-solver based britney) also contributed a couple of test cases[4]. Allegedly, SAT-britney does rather well on the test suite, failing only 2 tests as far as I can tell[5]. On the other hand, it does solve a some of the more interesting cases britney2 does not solve. On a more mathematical note, the britney2 implementation behaves like a function[6] with an attractive fixed point[7]. This is interesting, because for some cases it may take britney2 a couple of iterations to reach the right solution. This fixed point is somewhat simple to find by using the following steps (pseudo-code):

// Runtime complexity O(n * br * diff), where "n" is the number of iterations until
// a fixed point is reached, "br" is the complexity of "run_britney" and "diff" is
// the runtime of the "last != current" comparison.
function find_fixed_point(initial);
    last = run_britney(initial)
    current = run_britney(last)
    while last != current ; do
        last = current
        current = run_britney(last)
    od
    return current
end

This gives us a simple way to test if britney will eventually solve the issue herself (and when she will do it). Currently britney2 is automatically run twice a day, so for every 2 iterations (beyond the first) roughly translates to a 24-hours delay. So far the test suite does not have a lot of problems that requires more than one iteration. Personally I would be pleased if it turned out to stay that way as the test suite coverage grows. If you are interested in playing around with this, you can get sources from:

• britney2
  • Currently only works in stable (i.e. requires python2.5 and python-apt < 0.8 or so)
    • There some patches for this filed against release.debian.org.
  • See the INSTALL file for instructions
  • Adam s branch
    • use the p-u branch.
• SAT-britney
  • I haven t tested this one and I do not know the requirements here
• britney-tests
  • See the README file for instructions

Footnotes: [1] http://lists.debian.org/debian-release/2011/10/msg00178.html [2] These tests are auto-generated, so it is merely an up-scaled pattern . [3] Basically if two (or more) packages needs to migrate into testing at the exact same time, they need to be hinted in. [4] Not to mention all the copy-waste errors he pointed out in mine. Apparently, SAT-britney has stricter requirements to the data than britney2. [5] I assume the test called sat-britney-death (created by Joachim) was named that way for a reason. The second failure is caused by SAT britney not reading hints (yet?), so the approve tpu package test case should fail. [6] A function that maps an archive into another archive erh, I mean, it maps a set of packages into another set of packages [7] http://en.wikipedia.org/wiki/Fixed_point_%28mathematics%29 Assuming my claim to be true, the function will have more than one fixed point. The obtained fixed point depends on the initial state of testing. As an example:
y depends on x
x in testing has RC bugs If x is not in testing, it cannot migrate to testing (due to its RC bugs). If x is not in testing, then y cannot migrate into testing. But if x starts in testing, then y may be able to migrate. This can happen if x migrated to testing before an RC bug was filed against it. (Dis-)Proving my claim is an exercise left for the reader.

We all love stats, don t we? So, here we go! Let s start with a graph: It shows the number of packages in the NEW queue since last year. You can see a big drop during April 2011, and a reasonably low rate during the last six months. You could think fellow Debian Developers stopped to upload NEW packages. Sorry, you re wrong! Since Squeeze release, 3.832 .changes files with NEW components were processed by dak, with an average of 14,85 NEW packages per day. On the FTP Team side, we had 3.732 accepts (14,47 per day), 339 rejects (1,31 per day) and 178 comments to maintainers (0,69 per day).

---

Who were the most prolific maintainers who got a NEW processing? Here is our special top ten:

1. Debian Haskell Group (362 packages)
2. Debian Perl Group (343 packages)
3. Debian Java Maintainers (161 packages)
4. Debian Ruby Extras Maintainers (124 packages)
5. Debian Multimedia Maintainers (100 packages)
6. Debian Fonts Task Force (96 packages)
7. Debian Med Packaging Team (79 packages)
8. Debian Install System Team (61 packages)
9. Debian Javascript Maintainers (54 packages)
10. Debian Python Modules Team (50 packages)

That s bad packaging teams cannot bake cookies!
Let s do the same with Changed By, this time:

1. Ben Hutchings (159 packages)
2. Joachim Breitner (138 packages)
3. Clint Adams (134 packages)
4. Jonas Smedegaard (124 packages)
5. TANIGUCHI Takaki (97 packages)
6. Nicholas Bamber (61 packages)
7. Alessio Treglia (60 packages)
8. maximilian attems (54 packages)
9. David Paleino (51 packages)
10. Torsten Werner (45 packages)

Much better now go and heat up your ovens, we know who you are

---

Another nice aspect to look at is the speed of NEW processing. Some maintainers were very happy for a fast NEW processing, someone even complained for having been too quick! So, let s find out which upload was the quickest ever. Try to gamble a bit before reading the answer, to see whether you are near to the real value Alessio Treglia, you probably already know, because your gwc_0.21.16~dfsg-1 upload has been processed in 41 seconds (yes, forty-one seconds!). Here s an excerpt from ftp-master log to certify it:

20110516120252 process-upload dak Processing changes file gwc_0.21.16~dfsg-1_amd64.changes
20110516120258 process-upload dak Moving to new gwc_0.21.16~dfsg-1_amd64.changes
20110516120339 process-new tolimar NEW ACCEPT: gwc_0.21.16~dfsg-1_amd64.changes

Alex was the super-fast FTP Team member behind the quickest accept, do you want to beat him? Join FTP Team

Lindsay Holmwood has written about the benefits of a standing desk and how to buy one [1]. The case for avoiding sitting is strong, but I couldn t stand up all day. One thing that s been on my list of things to do if I had an unreasonably large amount of spare time or money is to make a reclining computer station. The idea would be to take a bed and mount a TFT display above it so that it can be viewed while lying down. Then a split keyboard would be required so that each hand can be used for half the keys (this would be difficult or expensive). If the keyboard halves were aligned correctly then it would reduce the carpal tunnel problems associated with computer use (which has been a big problem for me in the past and is something that I will probably never fully recover from [2]). As far as I am aware the risk of back problems is eliminated when lying down, so two of the major problems with regular computer desks would be avoided. I don t think that lying down all day would be that great and it wouldn t work for collaborative projects. But as monitors are so cheap nowadays it would be viable to have a second monitor at a desk connected to the same computer. Then I could spend about half my computer time lying down.

• [1] http://holmwood.id.au/~lindsay//2011/10/03/standing-desk-adventures/
• [2] http://etbe.coker.com.au/2010/06/16/carpal-tunnel-getting-better/

Before I start, I'll admit that I'm not a real RPM packager. Maype I'm approaching this from completely the wrong direction, what do I know? I'm in the process of packaging Varnish 3.0.2 which includes mangling the spec file. The top of the spec file reads:

%define v_rc
%define vd_rc % ?v_rc:-% ?v_rc 

Apparently, this is not legal, since we're trying to define v_rc as a macro with no body. It's however not possible to directly define it as an empty string which can later be tested on, you have to do something like:

%define v_rc % nil 
%define vd_rc % ?v_rc:-% ?v_rc 

Now, this doesn't work correctly either. % ?macro tests if macro is defined, not whether it's an empty string so instead of two lines, we have to write:

%define v_rc % nil 
%if 0% ?v_rc  != 0
%define vd_rc % ?v_rc:-% ?v_rc 
%endif

The 0 ?v_rc != 0 workaround is there so that we don't accidentially end up with == 0 which would be a syntax error. I think having four lines like that is pretty ugly, so I looked for a workaround and figured that, ok, I'll just rewrite every use of % vd_rc to % ?v_rc:-% ?v_rc . There are only a couple, so the damage is limited. Also, I'd then just comment out the v_rc definition, since that makes it clear what you should uncomment to have a release candidate version. In my naivety, I tried:

# %define v_rc ""

# is used as a comment character in spec files, but apparently not for defines. The define was still processed and the build process stopped pretty quickly. Luckily, doing # % define "" seems to work fine and is not processed. I have no idea how people put up with this or if I'm doing something very wrong. Feel free to point me at a better way of doing this, of course.

A common question about hosting is whether to use a dedicated server or a virtual server. Dedicated Servers If you use a dedicated server then you will face the risk of problems which interrupt the boot process. It seems that all the affordable dedicated server offerings lack any good remote management, so when the server doesn t boot you either have to raise a trouble ticket with the company running the Data-Center (DC) or use some sort of hack. Hetzner is a dedicated server company that I have had good experiences with [1], when a server in their DC fails to boot you can use their web based interface (at no extra charge or delay) to boot a Linux recovery environment which can then be used to fix whatever the problem may be. They also charge extra for hands-on support which could be used if the Linux recovery environment revealed no errors but the system just didn t work. This isn t nearly as good as using something like IPMI which permits remote console access to see error messages and more direct control of rebooting. The up-side of a dedicated server is performance. Some people think that avoiding virtualisation improves performance, but in practice most virtual servers use virtualisation technologies that have little overhead. A bigger performance issue than the virtualisation overhead is the fact that most companies running DCs have a range of hardware in their DC and your system (whether a virtual server or a dedicated server) will be on a random system from their DC. I have observed hosting companies to give different speed CPUs and for dedicated servers different amounts of RAM for the same price. I expect that the disk IO performance also varies a lot but I have no evidence. As long as the hosting company provides everything that they offered before you sign the contract you can t complain. It s worth noting that CPU performance is either poorly specified or absent in most offers and disk IO performance is almost never specified. One advantage of dedicated servers in this regard is that you get to know the details of the hardware and can therefore refuse certain low spec hardware. The real performance benefit of a dedicated server is that disk IO performance won t be hurt by other users of the same system. Disk IO is the real issue as CPU and RAM are easy to share fairly but disk performance is difficult to share and is also a significant bottleneck on many servers. Dedicated servers also have a higher minimum price due to the fact that a real server is being used which involves hardware purchase and rack space. Hetzner s offers which start at E29 per month are about as cheap as it s possible to get. But it appears that the E29 offer is for an old server new hardware starts at E49 per month which is still quite cheap. But no dedicated server compares to the virtual servers which can be rented for prices less than $10 per month. Virtual Servers A virtual server will typically have an effective management interface. You should expect to get web based access to the system console as well as ssh console access. If console access is not sufficient to recover the system then there is an option to boot from a recovery device. This allows you to avoid many situations that could potentially result in down-time and when things go wrong it allows you to recover faster. Linode is an example of a company that provides virtual servers and provides a great management interface [2]. It would take a lot of work with performance monitoring and graphing tools to give the performance overview that comes for free with the Linode interface. Disk IO performance can suck badly on virtual servers and it can happen suddenly and semi-randomly. If someone else who is using a virtual server on the same hardware is the target of a DoS attack then your performance can disappear. Performance for CPU is generally fairly reliable though. So a CPU bound server would be a better fit on the typical virtual server options than a disk IO bound server. Virtual servers are a lot cheaper at the low end so if you don t need the hardware capabilities of a minimal dedicated server (with 1G of RAM for Hetzner and a minimum of 8G of RAM for some other providers) then you can save a lot of money by getting a virtual server. Finally the options for running a virtual machine under a virtual machine aren t good, AFAIK the only options that would work on a commercial VPS offering are QEMU (an x86 CPU instruction emulator), Hercules (a S/370 S/390, and Z series IBM mainframe emulator), and similar CPU emulators. Please let me know if there are any other good options for running a virtual machine on a VPS. Now while these emulators are apparently good for debugging OS development they aren t something that are generally useful for running a virtual machine. I knew someone who ran his important servers under Hercules so that x86 exploits couldn t be used for attacking them, but apart from that CPU emulation isn t generally useful for servers. Summary If you want to have entire control of the hardware or if you want to run your own virtual machines that suit your needs (EG one with lots of RAM and another with lots of disk space) then a dedicated server is required. If you want to have minimal expense or the greatest ease of sysadmin use then a virtual server is a better option. But the cheapest option for virtual hosting is to rent a server from Hetzner, run Xen on it, and then rent out DomUs to other people. Apart from the inevitable pain that you experience if anything goes wrong with the Dom0 this is a great option. As an aside, if anyone knows of a reliable company that offers some benefits over Hetzner then please let me know. What I would Like to See There is no technical reason why a company like Linode couldn t make an offer which was a single DomU on a server taking up all available RAM, CPU, and disk space. Such an offer would be really compelling if it wasn t excessively expensive. That would give Linode ease of management and also a guarantee that no-one else could disrupt your system by doing a lot of disk IO. This would be really easy for Linode (or any virtual server provider) to implement. There is also no technical reason why a company like Linode couldn t allow their customers to rent all the capacity of a physical system and then subdivide it among DomUs as they wish. I have a few clients who would be better suited by Linode DomUs that are configured for their needs rather than stock Linode offerings (which never seem to offer the exact amounts of RAM, disk, or CPU that are required). Also if I had a Linode physical server that only had DomUs for my clients then I could make sure that none of them had excessive disk IO that affected the others. This would require many extra features in the Linode management web pages, so it seems unlikely that they will do it. Please let me know if there is someone doing this, it s obvious enough that someone must be doing it. Update: A Rimuhosting employee pointed out that they offer virtual servers on dedicated hardware which meets this criteria [3]. Rimuhosting allows you to use their VPS management system for running all the resources in a single server (so no-one else can slow your VM down) and also allow custom partitioning of a server into as many VMs as you desire.

• [1] http://www.hetzner.de/
• [2] http://www.linode.com/
• [3] https://rimuhosting.com/order/vps-on-dedicated.jsp

[UPDATE 2011-10-26] The story has developed.[/UPDATE] Recently I was talking with somebody about the WEB.DE toolbar, and today I finally got around to look something up I was told then. And while I searched for the information on the download page for the WEB.DE toolbar, I stumbled over the download URL for the installer:

http://wa.ui-portal.de/webde/webde/s?produkte.browserdownload.link.download.ff7&bd_mc=undef_undef&ns_type=pdf&ns_url=http://dl.web.de/browser/firefox/WEB.DE_MFF7_Setup.exe

There I was magically attracted by the ns_url parameter, because it smelled like something where you could just pass any URL along. And indeed you can: http://wa.ui-portal.de/webde/webde/s?produkte.browserdownload.link.download.ff7&bd_mc=undef_undef&ns_type=pdf&ns_url=http://bundestrojaner.net/inhalt-bundestrojaner-gratis-download-3.html. That should redirect you to http://bundestrojaner.net/inhalt-bundestrojaner-gratis-download-3.html instead of http://dl.web.de/browser/firefox/WEB.DE_MFF7_Setup.exe. This is a serious flaw, as it might allow an attacker to request a username and password combination from an unsuspecting user. With a harvest page designed to look like another WEB.DE page you should have a high return rate. This is, by the way, a flaw on the OWASP Top 10 (2010 edition). I've informed somebody working for United Internet and expect the bug to be resolved pretty soon, hopefully it won't work anylonger when you read this.

Red Hat s Matthew Garrett let the cat out of the bag about a month ago: when UEFI Secure Boot is adopted by mainboard manufacturers to satisfy Microsoft Windows 8 requirements, it may very well be the case that Linux and others (BSD, Haiku, Minix, OS/2, etc) will no longer boot. Matthew has written about it extensively and seems to know very well what the issues are (part I, part II), the details about signing binaries and why Linux does not support Secure Boot yet. The Free Software Foundation has also released a statement and started a campaign, which is, as usually, anti-Microsoft instead of pro-solutions. Now let me express my opinion on this matter: this is not Microsoft s fault. Facts Let s see what are the facts in this controversy:

• Secure Boot is here to stay. In my humble opinion, the idea is good and it will prevent and/or lessen malware effects, especially on Windows.
• Binaries need to be signed with a certificate from the binaries vendor (Microsoft, Apple, Red Hat, etc)
• The certificate that signs those binaries needs to be installed in the UEFI BIOS
• Everybody wants their certificate bundled with the UEFI BIOS so that their operating system works out of the box
• Given that there are many UEFI and mainboard manufacturers, getting your certificate included is not an easy task: it requires time, effort and money.

Problem The problem stems from the fact that most Linux vendors do not have the power to get their certificates in UEFI BIOS. Red Hat and Suse will for sure get their certificates bundled in server UEFI BIOS. Debian and Ubuntu? Maybe. NetBSD, OpenIndiana, Slackware, etc? No way. This is, in my humble opinion, a serious defect in the standard. A huge omission. Apparently while developing the Secure Boot specification everybody was busy talking about signed binaries, yet nobody thought for a second how the certificates will get into the UEFI BIOS. What should have been done The UEFI secure boot standard should have defined an organization (a Secure Boot Certification Authority ) that would issue and/or receive certificates from organizations/companies (Red Hat, Oracle, Ubuntu, Microsoft, Apple, etc) that want their binaries signed. This SBCA would also be in charge of verifying the background of those organizations. There is actually no need for a new organization: just use an existing one, such as Verisign, that carries on with this task for Microsoft for kernel-level binaries (AuthentiCode). Given that there is no Secure Boot Certification Authority, Microsoft asked BIOS (UEFI) developers and manufacturers to include their certificates, which looks 100% logical to me. The fact that Linux distributions do not have such power is unfortunate, but it is not Microsoft s fault at all. What can we do? Given its strong ties with Intel, AMD and others, maybe the Linux Foundation could start a task force and a Temporary Secure Boot Certification Authority to deal with UEFI BIOS manufacturers and developers. This task force and TSBCA would act as a proxy for minorities such as Linux, BSD, etc distributions. I am convinced this is our best chance to get something done in a reasonable amount of time. Complaining will not get us anything. Screaming at Microsoft will not get us anything. We need to propose solutions. Wait! Non-Microsoft certificates? Why? In addition to the missing Secure Boot Certification Authority, there is a second problem apparently nobody is talking about: what is the advantage mainboard manufacturers get from including non-Microsoft certificates? For instance: why would Gigabyte (or any other mainboard manufacturer) include the certificate for, say, Haiku? The benefit for Gigabyte would be negligible and if someone with ill-intentions gets Haiku s certificate, that piece of malware will be installable on all Gigabyte s mainboards.This would lead to manufacturer-targetted malware, which would be fatal to Gigabyte: oh, want to be immune to the-grandchild-of-Stuxnet? Buy (a computer with) an MSI mainboard, which does not include Haiku s certificate Given that 99% of desktops and laptops only run Windows, the result of this (yet unresolved) problem would be that manufacturers will only install Microsoft certificates, therefore they would be immune to malware signed with a Slackware certificate in the wild. If we are lucky, mainboard manufacturers will give us an utility to install more certificates under your own risk. The solution to the first problem looks easy to me. The solution to the second looks much more worrying to me.