There are six servers. They're meant to be used for the same purpose, and are mostly the same too, but not quite entirely. This being the case, mostly because of how they were maintained in the past: with cssh mostly, but it's inevitable that eventually there will be differences. So we're in the process of migrating them to puppet now. One of the things I need to do while doing that, however, is figure out what the differences are, and get rid of them where applicable. Differences such as, say, in the installed packages. Some of these servers have extra things installed, for a reason that isn't clear anymore, that others don't. So how does one figure out what the difference is? If this were two servers, I'd create a list of installed packages and use 'diff' on them. If there were three, I'd use 'diff3' instead. But if there are 6? Turns out that isn't too hard. First, of course, you need a list of packages installed on each individual host. Both rpm and dpkg can do this with a simple command line invocation. Assuming that's done, and the output is stored in files called installed-host1 through installed-host6, the procedure is:

$ comm -12 installed-host1 installed-host2 > installed-common
$ for i in $(seq 3 6)
> do
>   comm -12 installed-common installed-host$i > installed-tmp
>   mv installed-tmp installed-common
> done

And there, you now have a file "installed-common" containing everything which is installed everywhere. If you use "comm" or "diff" to compare that file against the installed-hostX files, you can easily see what the difference is. Granted, that doesn't give you the overview in just one file, but usually that's not really necessary. The 'installed-common' file contains the largest common denominator, which really is everything you need.

I realise it has been nearly two months since I last posted anything here. The good news is I'm still alive! Mostly the past couple of months has been full of cute victims to take pictures of, which has helped me setup a simplified portfolio site;

• Edinburgh Portraits

I still continue to prefer images of people and I was recently pleased with the delivery of my first "photobook". Over the past couple of years I've slowly decorated my flat with prints (4"x6" - A2) of my pictures, but seeing the pictures in a nicely bound book makes them feel so much more real. I've also been doing a little more software development, mostly relating to the archiving of images and the workflow of taking RAW images, converting them, and finally uploading via rsync. I suspect the tools I've put together are Steve-specific, but I did have some fun with duplicate image detection and eilimination - something I've written about in the past. ObQuote: "Better to write for yourself and have no public than to write for the public and have no self" - Cyril Connolly.

I was in Brazil last week and had the pleasure of participating in a teach-in at Occupy S o Paulo, a vibrant camp set up last week in the heart of the old down town. There are now under threat, please see below for details and spread the word. *** On October 15, a group of nearly 300 activists began an occupation of S o Paulo in the Valley of Anhagabau, one of the sites of the first rallies for direct elections during the end of the Brazilian military dictatorship in the 1980s. After a week of peaceful encampment, educational and cultural programs, and creating a sustainable community for not just themselves but many homeless people in downtown S o Paulo, the Occupy S o Paulo movement is coming under increased police threat. Today, Monday October 24, the governor of the state of S o Paulo, Geraldo Alckmin is holding two special events. First, he is hosting Florida Governor Rick Scott (R). Second, Alckmin has decided to hold a parade of 3000 military police right next to the encampment (see photo above). After a week of police harrassment and a pending court case for them to hold the right to pitch their tents, the Occupy S o Paulo movement sees this as an escalation of the harassment they have already faced by city police. Further, the presence of 3000 military police next to 300 occupiers is clearly meant to intimidate both occupiers and members of the public who have been coming up to the encampment and learning about the movement. Please take the time to call or email the governor and the secretary of public security of the state of S o Paulo to condemn this action. You can also send messages of solidarity to the S o Paulo occupation at occupysampaenglish@googlegroups.com. To contact Governor Geraldo Alckmin s office of citizen and organizational relations: Fill out a comment form at: http://www.saopaulo.sp.gov.br/en/fale/fale.php
Phone: 55-11-2193-8463 To contact the Secretary of Public Security, Ant nio Ferreiro Pinto:
Email: seguranca@sp.gov.br
Phone: 55-11-3291-8500

As is natural for an LTS cycle, lots of people are thinking and talking about work focused on quality rather than features. With Canonical extending LTS support to five years on the desktop for 12.04, much of this is quite rightly focused on the desktop. I'm really not a desktop hacker in any way, shape, or form, though. I spent my first few years in Ubuntu working mainly on the installer - I still do, although I do some other things now too - and I used to say only half-jokingly that my job was done once X started. Of course there are plenty of bugs I can fix, but I wanted to see if I could do something with a bit more structure, so I got to thinking about projects we could work on at the foundations level that would make a big difference. Image build pipeline One difficulty we have is that quite a few of our bugs - especially installer bugs, although this goes for some other things too - are only really caught when people are doing coordinated image testing just before a milestone release. Now, it takes a while to do all the builds and then it takes a while to test them. The excellent work of the QA team has meant that testing is much quicker now than it used to be, and a certain amount of smoke-testing is automated (particularly for server images). On the other hand, the build phase has only got longer as we've added more flavours and architectures, particularly as some parts of the process are still serialised per architecture or subarchitecture so ARM builds in particular take a very long time indeed. Exact timings are a bit difficult to get for various reasons, but I think the minimum time between a developer uploading a fix and us having a full set of candidate images on all architectures including that fix is currently somewhere north of eight hours, and that's with people cutting corners and pulling strings which is a suboptimal thing to have to do around release time. This obviously makes us reluctant to respin for anything short of showstopper bugs. If we could get things down to something closer to two hours, respins would be a much less horrible proposition and so we might be able to fix a few bugs that are serious but not showstoppers, not to mention that the release team would feel less burned out. We discussed this problem at the release sprint, and came up with a laundry list of improvements; I've scheduled this for discussion at UDS in case we can think of any more. Please come along if you're interested! One thing in particular that I'm working on is refactoring Germinate, a tool which dates right back to our first meeting before Ubuntu was even called Ubuntu and whose job is to expand dependencies starting from our lists of "seed" packages; we use this, among other things, to generate Task fields in the archive and to decide which packages to copy into our images. This was acceptably quick in 2004, but now that we run it forty times (eight flavours multiplied by five architectures) at the end of every publisher run it's actually become rather a serious performance problem: cron.germinate takes about ten minutes, which is over a third of the typical publisher runtime. It parses Packages files eight times as often as it needs to, Sources files forty times as often as it needs to, and recalculates the dependency tree of the base system five times as often as it needs to. I am confident that we can significantly reduce the runtime here, and I think there's some hope that we might be able to move the publisher back to a 30-minute cycle, which would increase the velocity of Ubuntu development in general. Maintaining the development release Our release cycle always starts with syncing and merging packages from Debian unstable (or testing in the case of LTS cycles). The vast majority of packages in Ubuntu arrive this way, and generally speaking if we didn't do this we would fall behind in ways that would be difficult to recover from later. However, this does mean that we get a "big bang" of changes at the start of the cycle, and it takes a while for the archive to be usable again. Furthermore, even once we've taken care of this, we have a long-established rhythm where the first part of the cycle is mainly about feature development and the second part of the cycle is mainly about stabilisation. As a result, we've got used to the archive being fairly broken for the first few months, and we even tell people that they shouldn't expect things to work reliably until somewhere approaching beta. This makes some kind of sense from the inside. But how are you supposed to do feature development that relies on other things in the development release? In the first few years of Ubuntu, this question didn't matter very much. Nearly all the people doing serious feature development were themselves serious Ubuntu developers; they were capable of fixing problems in the development release as they went along, and while it got in their way a little bit it wasn't all that big a deal. Now, though, we have people focusing on things like Unity development, and we shouldn't assume that just because somebody is (say) an OpenGL expert or a window management expert that they should be able to recover from arbitrary failures in development release upgrades. One of the best things we could do to help the 12.04 desktop be more stable is to have the entire system be less unstable as we go along, so that developers further up the stack don't have to be distracted by things wobbling underneath them. Plus, it's just good software engineering to keep the basics working as you go along: it should always build, it should always install, it should always upgrade. Ubuntu is too big to do something like having everyone stop any time the build breaks, the way you might do in a smaller project, but we shouldn't let things slide for months either. I've been talking to Rick Spencer and the other Ubuntu engineering leads at Canonical about this. Canonical has a system of "rotations", where you can go off to another team for a while if you're in need of a change or want to branch out a bit; so I proposed that we allow our engineers to spend a month or two at a time on what I'm calling the +1 Maintenance Team, whose job is simply to keep the development release buildable, installable, and upgradeable at all times. Rick has been very receptive to this, and we're going to be running this as a trial throughout the 12.04 cycle, with probably about three people at a time. As well as being professional archive gardeners, these people will also work on developing infrastructure to help us keep better track of what we need to do. For instance, we could deploy better tools from Debian QA to help us track uninstallable packages, or we could enhance some of our many existing reports to have bug links and/or comment facilities, or we could spruce up the weather report; there are lots of things we could do to make our own lives easier. By 12.04, I would like, in no particular order:

• Precise to have been more or less continuously usable from Alpha 1 onward for people with reasonable general technical ability
• Canonical engineering teams outside Ubuntu (DX, Ubuntu One, Launchpad, etc.) to be comfortable with running the development release on at least one system from Alpha 2 onward
• Installability problems in daily image builds to be dealt with within one working day, or preferably before they even make it to daily builds
• The archive to be close to consistent as we start milestone preparation, rather than the release team having to scramble to make it so
• A very significant reduction in our long-term backlog of automatically-detected problems

Of course, this overlaps to a certain extent with the kinds of things that the MOTU team have been doing for years, not to mention with what all developers should be doing to keep their own houses in reasonable order, and I'd like us to work together on this; we're trying to provide some extra hands here to make Ubuntu better for everyone, not take over! I would love this to be an opportunity to re-energise MOTU and bring some new people on board. I've registered a couple of blueprints (priorities, infrastructure) for discussion at UDS. These are deliberately open-ended skeleton sessions, and I'll try to make sure they're scheduled fairly early in the week, so that we have time for break-out sessions later on. If you're interested, please come along and give your feedback!

My generous employer has kindly offered to host the first Bug Squashing Party of the now commencing Wheezy Release BSP Marathon. Yay! We'll meet during the first weekend in December (Friday 2nd to Sunday 4th) in Hildesheim, Germany. There's a wiki page listing all important information. Please also use this page to sign up if you want to join us.

Twenty-four hours ago I was in heaven. Or to more properly recapitulate and situate this, fourty-eight hours ago I was in hell. Now this requires some context. Earlier in the year, I had tweeted briefly about a Kurt Elling live concert still streaming on NPR, and truth be told, I have listened to Kurt Elling a whole lot ever since. The most recent album The Gate is fantastic, the Grammy-winner Dedicated to You (which retakes the very famous Coltrane/Hartman album) is beyond words, and The Messenger is worth it just for 'Nature Boy' and the funkiest-ever 'April in Paris'. I also saw him as part of a larger and otherwise rather neat tour. But I needed to see the man himself. So following what one can see from his touring calendar, I jotted down two October dates. Kurt Elling at the Green Mill. Fine. That was earlier in the summer. Summer has a tendency of evaporating quickly, so here we were in October. And getting going late on Friday, so by the time we got the Green Mill, a long line had formed and we ended up paying tribute to the line for fourty-five minutes with nothing to show. Sad. Home we went, and no live music. But somehow we mustered the energy to go again last night and thought we were early, arriving about 20 minutes before the show was to start. Ha! Others had appeared at 3pm, by 5pm all seats were gone. We were literally the last ones the get in after a short spell in a shorter line, and standing room-only it was. Oh, but what a treat we got. The set for these two days was Kurt Elling with the fourteen piece Kl vers Big Band from Denmark. They played for three marvellous sets, and well over four hours (with two 25 minute breaks). Pieces often alternated between big band arrangements and smaller pieces by Elling's standard rhythm band, and covered standards as well as Elling's catalogue. As I mentioned earlier on a short post (with pictures) on Google+, it is essentially impossible to not fall for Elling's stage presence when being so close to the stage. The man is just that good. Elling and the Kl vers Big Band will now play for six nights at Birdland in New York followed by a night each in Washington, DC, and Boston. If you're around, do yourself a favour and go out to see them.

If you have been following Lintian s development closely, you will probably have noticed that I have not really done anything there for the past week. Instead I have turned my focus on our testing migration script, britney2. First, I have created a minimal test suite[1]. It started as 4 simple tests and by now it contains about 30 tests. The size of each test is rather small; the largest tests are about 1600 binary packages in total[2], but most are 2-20 binary packages in total. Thus the test suite is rather fast compared to a live data sample , which easily takes more than 10 minutes for a single run. Unfortunately, hand-crafting the test data is somewhat annoying and easy to get wrong. The test suite has a somewhat unfair focus on auto-hint [3] cases, so the current britney2 fails up to 14 tests. Some of these appears to fail because the auto-hinter (for some reason) receives incomplete information about the situation. To my knowledge we not been able to debug the situation, but Adam has a refactor branch that does not seem to have this issue. Personally I am hoping it will soon be merged into the master branch, especially because it seems to simplify a lot of common operations. Joachim Breitner (who has been working on a SAT-solver based britney) also contributed a couple of test cases[4]. Allegedly, SAT-britney does rather well on the test suite, failing only 2 tests as far as I can tell[5]. On the other hand, it does solve a some of the more interesting cases britney2 does not solve. On a more mathematical note, the britney2 implementation behaves like a function[6] with an attractive fixed point[7]. This is interesting, because for some cases it may take britney2 a couple of iterations to reach the right solution. This fixed point is somewhat simple to find by using the following steps (pseudo-code):

// Runtime complexity O(n * br * diff), where "n" is the number of iterations until
// a fixed point is reached, "br" is the complexity of "run_britney" and "diff" is
// the runtime of the "last != current" comparison.
function find_fixed_point(initial);
    last = run_britney(initial)
    current = run_britney(last)
    while last != current ; do
        last = current
        current = run_britney(last)
    od
    return current
end

This gives us a simple way to test if britney will eventually solve the issue herself (and when she will do it). Currently britney2 is automatically run twice a day, so for every 2 iterations (beyond the first) roughly translates to a 24-hours delay. So far the test suite does not have a lot of problems that requires more than one iteration. Personally I would be pleased if it turned out to stay that way as the test suite coverage grows. If you are interested in playing around with this, you can get sources from:

• britney2
  • Currently only works in stable (i.e. requires python2.5 and python-apt < 0.8 or so)
    • There some patches for this filed against release.debian.org.
  • See the INSTALL file for instructions
  • Adam s branch
    • use the p-u branch.
• SAT-britney
  • I haven t tested this one and I do not know the requirements here
• britney-tests
  • See the README file for instructions

Footnotes: [1] http://lists.debian.org/debian-release/2011/10/msg00178.html [2] These tests are auto-generated, so it is merely an up-scaled pattern . [3] Basically if two (or more) packages needs to migrate into testing at the exact same time, they need to be hinted in. [4] Not to mention all the copy-waste errors he pointed out in mine. Apparently, SAT-britney has stricter requirements to the data than britney2. [5] I assume the test called sat-britney-death (created by Joachim) was named that way for a reason. The second failure is caused by SAT britney not reading hints (yet?), so the approve tpu package test case should fail. [6] A function that maps an archive into another archive erh, I mean, it maps a set of packages into another set of packages [7] http://en.wikipedia.org/wiki/Fixed_point_%28mathematics%29 Assuming my claim to be true, the function will have more than one fixed point. The obtained fixed point depends on the initial state of testing. As an example:
y depends on x
x in testing has RC bugs If x is not in testing, it cannot migrate to testing (due to its RC bugs). If x is not in testing, then y cannot migrate into testing. But if x starts in testing, then y may be able to migrate. This can happen if x migrated to testing before an RC bug was filed against it. (Dis-)Proving my claim is an exercise left for the reader.

Before I start, I'll admit that I'm not a real RPM packager. Maype I'm approaching this from completely the wrong direction, what do I know? I'm in the process of packaging Varnish 3.0.2 which includes mangling the spec file. The top of the spec file reads:

%define v_rc
%define vd_rc % ?v_rc:-% ?v_rc 

Apparently, this is not legal, since we're trying to define v_rc as a macro with no body. It's however not possible to directly define it as an empty string which can later be tested on, you have to do something like:

%define v_rc % nil 
%define vd_rc % ?v_rc:-% ?v_rc 

Now, this doesn't work correctly either. % ?macro tests if macro is defined, not whether it's an empty string so instead of two lines, we have to write:

%define v_rc % nil 
%if 0% ?v_rc  != 0
%define vd_rc % ?v_rc:-% ?v_rc 
%endif

The 0 ?v_rc != 0 workaround is there so that we don't accidentially end up with == 0 which would be a syntax error. I think having four lines like that is pretty ugly, so I looked for a workaround and figured that, ok, I'll just rewrite every use of % vd_rc to % ?v_rc:-% ?v_rc . There are only a couple, so the damage is limited. Also, I'd then just comment out the v_rc definition, since that makes it clear what you should uncomment to have a release candidate version. In my naivety, I tried:

# %define v_rc ""

# is used as a comment character in spec files, but apparently not for defines. The define was still processed and the build process stopped pretty quickly. Luckily, doing # % define "" seems to work fine and is not processed. I have no idea how people put up with this or if I'm doing something very wrong. Feel free to point me at a better way of doing this, of course.

A common question about hosting is whether to use a dedicated server or a virtual server. Dedicated Servers If you use a dedicated server then you will face the risk of problems which interrupt the boot process. It seems that all the affordable dedicated server offerings lack any good remote management, so when the server doesn t boot you either have to raise a trouble ticket with the company running the Data-Center (DC) or use some sort of hack. Hetzner is a dedicated server company that I have had good experiences with [1], when a server in their DC fails to boot you can use their web based interface (at no extra charge or delay) to boot a Linux recovery environment which can then be used to fix whatever the problem may be. They also charge extra for hands-on support which could be used if the Linux recovery environment revealed no errors but the system just didn t work. This isn t nearly as good as using something like IPMI which permits remote console access to see error messages and more direct control of rebooting. The up-side of a dedicated server is performance. Some people think that avoiding virtualisation improves performance, but in practice most virtual servers use virtualisation technologies that have little overhead. A bigger performance issue than the virtualisation overhead is the fact that most companies running DCs have a range of hardware in their DC and your system (whether a virtual server or a dedicated server) will be on a random system from their DC. I have observed hosting companies to give different speed CPUs and for dedicated servers different amounts of RAM for the same price. I expect that the disk IO performance also varies a lot but I have no evidence. As long as the hosting company provides everything that they offered before you sign the contract you can t complain. It s worth noting that CPU performance is either poorly specified or absent in most offers and disk IO performance is almost never specified. One advantage of dedicated servers in this regard is that you get to know the details of the hardware and can therefore refuse certain low spec hardware. The real performance benefit of a dedicated server is that disk IO performance won t be hurt by other users of the same system. Disk IO is the real issue as CPU and RAM are easy to share fairly but disk performance is difficult to share and is also a significant bottleneck on many servers. Dedicated servers also have a higher minimum price due to the fact that a real server is being used which involves hardware purchase and rack space. Hetzner s offers which start at E29 per month are about as cheap as it s possible to get. But it appears that the E29 offer is for an old server new hardware starts at E49 per month which is still quite cheap. But no dedicated server compares to the virtual servers which can be rented for prices less than $10 per month. Virtual Servers A virtual server will typically have an effective management interface. You should expect to get web based access to the system console as well as ssh console access. If console access is not sufficient to recover the system then there is an option to boot from a recovery device. This allows you to avoid many situations that could potentially result in down-time and when things go wrong it allows you to recover faster. Linode is an example of a company that provides virtual servers and provides a great management interface [2]. It would take a lot of work with performance monitoring and graphing tools to give the performance overview that comes for free with the Linode interface. Disk IO performance can suck badly on virtual servers and it can happen suddenly and semi-randomly. If someone else who is using a virtual server on the same hardware is the target of a DoS attack then your performance can disappear. Performance for CPU is generally fairly reliable though. So a CPU bound server would be a better fit on the typical virtual server options than a disk IO bound server. Virtual servers are a lot cheaper at the low end so if you don t need the hardware capabilities of a minimal dedicated server (with 1G of RAM for Hetzner and a minimum of 8G of RAM for some other providers) then you can save a lot of money by getting a virtual server. Finally the options for running a virtual machine under a virtual machine aren t good, AFAIK the only options that would work on a commercial VPS offering are QEMU (an x86 CPU instruction emulator), Hercules (a S/370 S/390, and Z series IBM mainframe emulator), and similar CPU emulators. Please let me know if there are any other good options for running a virtual machine on a VPS. Now while these emulators are apparently good for debugging OS development they aren t something that are generally useful for running a virtual machine. I knew someone who ran his important servers under Hercules so that x86 exploits couldn t be used for attacking them, but apart from that CPU emulation isn t generally useful for servers. Summary If you want to have entire control of the hardware or if you want to run your own virtual machines that suit your needs (EG one with lots of RAM and another with lots of disk space) then a dedicated server is required. If you want to have minimal expense or the greatest ease of sysadmin use then a virtual server is a better option. But the cheapest option for virtual hosting is to rent a server from Hetzner, run Xen on it, and then rent out DomUs to other people. Apart from the inevitable pain that you experience if anything goes wrong with the Dom0 this is a great option. As an aside, if anyone knows of a reliable company that offers some benefits over Hetzner then please let me know. What I would Like to See There is no technical reason why a company like Linode couldn t make an offer which was a single DomU on a server taking up all available RAM, CPU, and disk space. Such an offer would be really compelling if it wasn t excessively expensive. That would give Linode ease of management and also a guarantee that no-one else could disrupt your system by doing a lot of disk IO. This would be really easy for Linode (or any virtual server provider) to implement. There is also no technical reason why a company like Linode couldn t allow their customers to rent all the capacity of a physical system and then subdivide it among DomUs as they wish. I have a few clients who would be better suited by Linode DomUs that are configured for their needs rather than stock Linode offerings (which never seem to offer the exact amounts of RAM, disk, or CPU that are required). Also if I had a Linode physical server that only had DomUs for my clients then I could make sure that none of them had excessive disk IO that affected the others. This would require many extra features in the Linode management web pages, so it seems unlikely that they will do it. Please let me know if there is someone doing this, it s obvious enough that someone must be doing it. Update: A Rimuhosting employee pointed out that they offer virtual servers on dedicated hardware which meets this criteria [3]. Rimuhosting allows you to use their VPS management system for running all the resources in a single server (so no-one else can slow your VM down) and also allow custom partitioning of a server into as many VMs as you desire.

• [1] http://www.hetzner.de/
• [2] http://www.linode.com/
• [3] https://rimuhosting.com/order/vps-on-dedicated.jsp

A month ago I started a new job, at Codethink, a British software development company. We work mainly on and with free software. I'm working on a new project called Baserock, a way to build embedded Linux systems that we think will avoid many of the pain points that people usually have. Baserock is still in its formative stages, not something you can really play with. Codethink has its main office in Manchester, so the past weekend I moved there with my fianc . Oh yeah, Soile and I got engaged. Rings of Palladium. While I've been having a lot of fun, I'd like to be bored for a while now. Instead, I'm going to Prague for ELC Europe.

A couple of times I have written code which basically does this:

1. accept() an incoming connection
2. do some magic to get another socket
3. start passing bit between the two sockets
4. perform a correct close down sequence

An example use case of this is a proxy implementing the HTTP CONNECT method, but in for some known hostnames it will log a message and mangle the hostname before proceeding. This has been used as a legacy fall back solution while changing a network setup. But my uses are not restricted to HTTP proxies I have done the same for a few legacy protocols where the magic has been of different complexities. The two final steps are quite general and it would be nice to have a module doing just that. Take two sockets and make it easy (or even automatic) to pass bytes between them. The na ve non-blocking solution would use a scalar string buffer for each direction and perform a select loop while maintaining the write vector depending on which buffers contain data. I have written this code multiple times. In development this is usually quite successful, in production less so. While Perl might be quite suited for the magic in step 2, the na ve way of passing bytes have quite an overhead for the buffer management. A less na ve way would use a array of strings for buffers, but I m not quite sure if this would be a win in all cases. You might be able to get away with some string operations on the read side of the buffer, but it might be more expensive on the write side. I have not benchmarked this. Most of the time I don t care about Perl level IO handles. I know that there is a real C level file descriptor beneath. So an even better POSIX compliant solution might be to use XS to have plain C strings and use readv()/writev() and a iovec structure as buffer. Can we do even better? At least on Linux we can. With the Linux splice() specific system call it is possible to us a pipe as buffer and never to have to copy data from and to user space. I have not been able to find any off the shelf solution on CPAN. So I think I need to write it myself, but what would the nice and general API be? I guess the basic interface would be something like:

    my $chain = IO::Splice->new($fh1, $fh2);
    $chain->pump(); # read and write from both handles if possible and needed
    $chain->read($fh1);  # read to buffer from one specific handle
    $chain->write($fh2); # write from buffer to one specific handle
    $chain->can_write(); # returns the handles it needs to write to

but it might be simpler to have two callbacks for setting a file handle in write or no-write state:

    my $readset  = IO::Select->new( $fh1, $fh2);
    my $writeset = IO::Select->new();
    my $chain = IO::Splice->new( $fh1, $fh2,
        writable => sub    $writeset->add( shift )  ,
        unwritable => sub   $writeset->remove( shift )  
    );
    while ( ... select ... )  
        $chain->pump();
     

As said, I think I have plenty of implementations of the na ve way but before releasing some code it would be nice to get some input on the API. But the best feedback would be a module that already have a usable API but might not implement the Linux specific way. That would allow me to steal the interface

Last weekend Dennis M. Ritchie, creator of the C Programming Language, the UNIX Operating System, The Plan 9 Operating System and many other key contributions to computing, passed away due an extended illness. I learned about it yesterday trough Rob Pike, former dmr s colleague and friend. I m writing this post to honor him and his legacy. While I ve seen people mourn his departure, I ve also noticed most technical people don t really understand why his work is important for today s computing, so I m here to give a refresh. Since I m good with visual tools (mindmaps) I ve made one to describe the extension of dmr s contribution to the world. You can get a glimpse of how this is important. The main topics are their key contributions, the sub-topics are the current technologies that were built upon dmr s work. Click on the image to see a larger resolution version.

The C Programming Language

The issue This morning (while I was running late for an appointement) I had a very weird stuff happening on my Thinkpad T61 laptop. Since I recently offered myself a shiny Thinkpad x201s, I have to admit I don't use much my T61 anymore. But this morning I had to print a page (for this appointement) and, as I didn't yet configured my printer on the x201s, I went to the T61. But I noticed that the network was down. I've tried quickly on wireless but, bad luck, my current wifi setup selects the channel automatically and it prefers choosing channels which aren't available in the US. Guess what, my T61 comes from the US and has those channels completely disabled, so no wireless available either. The investigation I first tried to

modprobe -r e1000e
modprobe e1000e to see if it fixed the problem, but it didn't. Worse, the interface disappeared and never reappeared. I tried to reboot but it didn't fix the problem, the link was still down. Running really late, I put the file on a usb key and printed it from the powerbook and postponed the fix for later. Now, this evening, I tried to investigate a bit more. Symptoms weren't only that the nic wasn't working, but there was a high load on the system (1-2 at idle), unresponsiveness every second or so, and watching top I could see spikes of high cpu usage for the kworker kernel thread. Typing that on google you can find a lot of people running on this issue, usually starting around kernel 2.6.36 or 2.6.37. Now, I might have upgraded the kernel recently to 3.0.0-4, but that didn't look related since the problem first appeared when the laptop was up and running. And I tried to reboot under 2.6.39, 2.6.38 and even 2.6.32 and the problem was still present. Each time, unloading the module would fix the problem, but loading it again wouldn't make the interface reappear. People advised to boot with pcie_ports=compat but that didn't do anything. I tried to boot without intel_iommu=force (disable Intel Vt-d) and pcie_aspm (Active State Power Management) but nothing either. Considering a userland issue, I've tried to boot a grml live distro (always keep a grml.iso in your /boot, extlinux-update will even put it in your menu automatically), and the problem was still present. So not a Debian kernel issue, not a userland issue, only thing left was the laptop. I didn't update the Bios recently, so I wondered exactly what could be the problem. I started to feel a little bad, since I still really like that laptop, and that I already decided to lend it to my sister since her own T61 is sitting with a dead system board in my shelf. I know she might have some negative waves, but she was not even landed when the problem first appear. The fix Then I had a flash. It's not mystery that I'm used to break network cards, and I had the bright idea to shutdown the laptop, disconnect AC and battery, then let it idle a bit. I even tried the secret Thinkpad power button code but I think it's unrelated. Then I re-plugged the battery, booted to grml and the issue was gone. I rebooted on the standard Debian and the link was up, network was working. So what happenned? The (tentative) explanation My guess is that, somehow, the network card firmware has an issue and choked on something (a network frame or an attack exactly like the one we demonstrated on ASF firmware). In fact, no, I don't think it's the e1000e firmware. My T61 comes with Intel vPro, which includes AMT (Active Management Technology), a remote management solution a bit like ASF but more advanced. As far as I know, AMT firmware always runs, even when it's disabled, it's just completely idle. Idle, but in this case I think it choked on something, and a reboot isn't enough to restart the AMT firmware. But a real hard reset without any power seems to do the trick. What next? Well, a part of me is pretty scared, but another is just bored. I mean, we know about that, that's exactly the kind of issue we are warning people of. I have no idea what exactly happened, and there's no way I'll be able to reproduce that, but I'm pretty sure it's something lying at a pretty low level in the platform, and which can severely disable your workstation. Now if it happens again I won't lose too much time on this. TL;DR: helping other people In case you came here because you searched on google terms like kworker cpu usage , e1000e, interrupts, it might be a good idea to first reboot on a live CD to eliminate installation issues, then shutdown the laptop, remove the battery and let it few seconds idle. This might be enough to reset something inside and fix the situation.

A month ago, I published an article on the compared performances of stunnel, nginx and stud as SSL terminators. The conclusion was to use stud on a 64-bit system, with session caching and AES. stunnel was unable to scale properly and nginx exhibited important latency issues. I got constructive comments on many aspects. Therefore, here is the second round. The protagonists are the same but both the conditions and the conclusions have changed a bit.

Benchmarks The hardware platform remains unchanged:

• HP DL 380 G7
• 6 GB RAM
• 2 L5630 @ 2.13GHz (8 cores), hyperthreading disabled
• 2 Intel 82576 NIC
• Spirent Avalanche 2900 appliance

We focus on the following environment:

• Ubuntu Lucid, featuring a patched version of OpenSSL 0.9.8k with AES-NI builtin and enabled by default
• 2.6.39 kernel (250 HZ)
• 64-bit
• 1 core for HAProxy, 2 cores for network cards¹, 4 cores for SSL, 1 core for the system
• system limit of 100,000 file descriptors per process

On the SSL side, the main components are:

• 2048-bit certificates
• AES128-SHA cipher suite
• 8 HTTP/1.0 requests per client
• HTTP response payload of 1024 bytes

If you are interested in different conditions (32-bit, number of cores, 1024-bit certificates, proportion of session reuses), have a look at the previous article whose results on the matter are still valid. I only consider 2048-bit certificates because most certificates issued today are 2048-bit. In SP800-57, NIST advises that 1024-bit RSA keys will no longer be viable after 2010 and advises moving to 2048-bit RSA keys:

If information is initially signed in 2009 and needs to remain secure for a maximum of ten years (i.e., from 2009 to 2019), a 1024-bit RSA key would not provide sufficient protection between 2011 and 2019 and, therefore, it is not recommended that 1024-bit RSA be used in this case.

And the protagonists are:

• stud, @e207dbc5a3, with session cache enabled for 20,000 sessions, listen backlog of 1000 connections,
• stunnel 4.44, configured with --disable-libwrap --with-threads=pthread (configuration),
• nginx 1.1.4, configured with --with-http_ssl_module (configuration).

UPDATED: Configuration of nginx has been updated to switch to off both multi_accept and accept_mutex thanks to the feedback of Maxim Dounin on latency issues that were present in the previous article (and still present in the early version of this article). HAProxy (configuration) is used for stunnel and stud to add load-balancing. For each test, we want to maximize the number of HTTP transactions per second (TPS) while minimizing latency and unsucessful transactions. The maximum number of allowed failed transactions is set at 1 transaction out of 1000. The maximum allowed average latency is 100 ms².

Control case Below is the first comparison between stud, nginx and stunnel on plain Ubuntu Lucid. Like in previous benchs, stunnel is not able to scale, with a maximum of 400 TPS. However, early latency issues of nginx has been solved, thanks to the change in configuration. stud and nginx gets similar performance but, as you can see on the bottom plot (whose scale is logarithmic) stud starts with an initial latency of 40 ms. This is pretty annoying. I have tried some micro optimization to solve this issue but without luck. If latency matters for you, stick with stunnel or nginx (or find where the problem lies in stud). UPDATED: After reporting this issue, a fix has been commited. The culprit was Nagle s algorithm which improves bandwidth at the expense of latency. With the fix, performances are mostly the same but there is no latency penalty any more.

OpenSSL 1.0.0e When I published the first round of benchmarks, Micha Trojnara was dubious about the bad results of stunnel. He told me on Twitter:

I solved it! You compiled stunnel against a 4 years old OpenSSL with MT-unsafe SSL_accept. Upgrade OpenSSL and rebuild stunnel. This bug was fixed in OpenSSL 1.0.0b. I implemented a (slow) workaround, so stunnel doesn t crash with earlier versions of OpenSSL.

Therefore, I backported Ubuntu Oneiric OpenSSL 1.0.0e to Ubuntu Lucid³ and, as you can see yourself on the plot below, there is a huge performance boost for stunnel: about 400%! stud features a slight improvement too (about 8%) but still starts with a latency of 40 ms while nginx does not improve.

Intel Accelerator Engine Most Linux distributions are still using OpenSSL 0.9.8. For example, Debian Squeeze ships with OpenSSL 0.9.8o, Ubuntu Lucid with 0.9.8k and RHEL 5.7 with 0.9.8e. Therefore, they are not eligible to recent performance improvements. To solve this, Intel released Intel Accelerator Engine, which also brings, among other things, AES-NI support:

This is essentially a collection of assembler modules from development OpenSSL branch, packed together as a standalone engine. Standalone means that it can be compiled outside OpenSSL source tree without patching the latter. Idea is to provide a way to utilize new code in already released OpenSSL versions constrained by support and release policies limiting their evolvement to genuine bug fixes.

Alas, in my tests, there was no improvement compared to OpenSSL 1.0.0e (already patched with AES-NI support). stunnel comes with out-of-the-box support for additional engines. Just add something like this to your stunnel.conf:

engine=dynamic
engineCtrl=SO_PATH:/WOO/intel-accel-1.4/libintel-accel.so
engineCtrl=ID:intel-accel
engineCtrl=LIST_ADD:1
engineCtrl=LOAD
engineCtrl=INIT

For stud, you should grab support for engine selection from GitHub and for nginx, just specify the engine with ssl_engine.

Intel Performance Primitives In his great article Accelerated SSL , Simon Kuhn shows huge performance improvements by using Intel s IPP library, a library aimed at leveraging power of multi-core processors and including some cryptographic primitives and a patch to make use of them in OpenSSL. Unfortunately, this library is not free, not even as in free beer . The patch provided by Intel only applies on OpenSSL 0.9.8j. Simon updated it for OpenSSL 1.0.0d. It also applies with a slight modification to OpenSSL 1.0.0e. Look at Simon s article for instructions on how to compile OpenSSL with IPP. Simon measured 220% improvement on RSA 1024-bit private key operations by using this library. Unfortunately, we don t get such a performance burst with stud which exhibits a 15% improvement (but remember that private key operations only happen once every 8 transactions thanks to session reuse). Worst, with stunnel, performances are reduced by one third and nginx just segfaults. I think such a library is risky to use in production.

Hyperthreading During the previous tests, only half the capacity of each core were used. For stud, adding more workers per core only slightly improves this situation (less than 10% with 4 workers per core). Therefore, I thought using hyperthreading could rectify the situation here. In this case, stud, stunnel and nginx are allocated 8 virtual cores instead of 4. stud only got a small improvement of 5%, nginx a moderate improvement of 11% while stunnel got a significant improvement of 26%.

Conclusion Here is a quick recap of the maximum TPS reached on each tests (while keeping average response time below 100 ms). All tests were run on 4 physical cores with 2048-bit certificates, AES128-SHA cipher suite. Each client makes 8 HTTP requests and reuse the first SSL session.

Context	nginx 1.1.4	stunnel 4.44	stud @e207dbc5a3
OpenSSL 0.9.8k	1489 TPS	400 TPS	1492 TPS
OpenSSL 1.0.0e	1501 TPS	1561 TPS	1609 TPS
OpenSSL 1.0.0e + IPP	-	1101 TPS	1857 TPS
OpenSSL 1.0.0e + HT	1718 TPS	1973 TPS	1694 TPS

I also would like to stress that below 1500 TPS, stunnel adds an overhead of a few milliseconds while stud adds 40 milliseconds. With OpenSSL 1.0.0e and hyperthreading enabled, stunnel has both good performance and low latency on low loads to moderate loads. You may just want to limit the number of allowed connections to avoid collapsing on high loads. While in the previous round, stud was the champion and stunnel lagged behind, the situation is now reversed : stunnel is on top! UPDATED: As said earlier, the latency issue with stud has been solved. Performances remain mostly the same.

---

1. HAProxy and network cards don t need dedicated core for this kind of load. However, if you also want to handle non SSL traffic, those dedicated cores can get busy.
2. The real condition is more complex. Each point is considered valid if the corresponding average latency is less than 100 ms or if the average on the next 8 points is less than 100 ms. Temporary latency bursts are therefore allowed. We are not so forgiveful with failed transactions.
3. This is pretty straighforward: I just dropped multiarch support. I chose to use Ubuntu version of OpenSSL instead of upstream because there are some interesting patches, like automatic support of AES-NI.

Our friends Andrew and Janice finally tied the knot last Sunday, in Langkawi, Malaysia. They're Aussies currently living in New York, and Janice is of Malaysian descent, so I think part of the reason was to accommodate some Malaysian relatives that wouldn't be able to travel, and the other reason was that it was going to mean a bunch of people had to travel anyway, so why not travel somewhere a bit exotic? As my sister is getting married later this month in Brisbane, we decided to spare Zoe two massive international flights in close succession, and Sarah and Zoe flew out a few days ahead of me, and went to Brisbane, and then Zoe, Sarah and her mum flew to Langkawi from there, and I flew there from San Francisco, with the plan being for Sarah and Zoe to hang out in Brisbane for the 4 weeks in between weddings. It was the longest amount of travel I think I've ever done. I hadn't sat down and calculated what the travel time was beforehand, and when I got on the flight from San Francisco to Hong Kong, and they said it was going to be nearly 14 hours just to Hong Kong, I nearly wept. The only good thing about it was I didn't have to wrangle a toddler as well. From there, I had to get to Singapore, with a several hour layover, and then another flight from Singapore to Langkawi. I left SFO at 1:20am on Monday morning, and finally arrived at about 7am (San Francisco time) the following day. I didn't even bother getting off the plane in Hong Kong, it was only an hour and a half layover. I think the layover in Singapore was 5 hours, and it was a separate flight. Singapore airport at least was pretty good to kill time in. It was funny, I was reading Wired while I was killing time in Singapore, and they had an article, which included a side bar on 10 Best Airports to Get Stuck In, and I observed that getting to Langkawi and back involved using 4 of them. The flight into Langkawi was more eventful than usual. I was dozing a bit as we were coming in to land, and I woke up to the sensation of taking off again. I wondered if we'd had an aborted landing? Sure enough, soon after, the captain got on the public address and said that they'd had to abort the landing due to bad weather in Langkawi. He had pretty bad Engrish, thanking us for our frustration and saying something about fuel, so I didn't get a lot of the details. We proceeded to circle for a while before finally landing. I think the flight ended up landing something like an hour or more later than scheduled. So that was mildly exciting. Langkawi airport was one of those little "walk across the tarmac" places. I think I stood the closest to an operating jet engine that I ever have, as another aeroplane taxied in to a stop between my plane and the terminal while I was walking to it. Loud. Anyway, that was all just me getting there. Sarah had a bit of fun getting to Brisbane on her own with Zoe. She's now too long to sleep in an in-flight bassinet, and doesn't seem to like sleeping vertically in the Ergo Baby Carrier (possibly she was too hot in it). I think she got a bit of sleep lying across Sarah's lap, but Sarah didn't get much sleep. I think there was similar fun for the Brisbane to Kuala Lumpur flight, but at least she had two laps to lie across for that one. Sarah, Zoe and Sarah's mum arrived in Langkawi earlier in the day on Tuesday, and I made it to the resort at about 10pm (local time) and pretty much immediately crashed for the night. The Westin Langkawi was just fabulous. I don't think we've stayed at a resort since our honeymoon, and this resort was a whole other level compared to Heron Island. The rooms were very comfortable, and the resort was very accommodating of a toddler. Their Kids Club menu was excellent. The breakfast buffet (included in our room rate) was the most comprehensive breakfast buffet I have ever seen in a hotel. The only thing missing was bacon, being a Muslim country, but I can live without that. It was pretty easy to just fill up on breakfast, skip lunch and just have a light dinner. The on-resort dining prices were also very reasonable. The off-resort prices were even cheaper. The resort facilities in general were very comprehensive. Three pools. A children's play room. A well equipped gym and recreational area including squash courts. An extremely nice day spa. The thing I liked best about the whole destination wedding thing was all of the activities leading up to the actual wedding. The wedding was on the Sunday, and we had the Hen's and Buck's parties on Friday. Apparently Australia is the only place that calls it a Buck's party. The rest of the world calls it a Stag party, how about that? There was some last-minute changes to the Buck's party, and we ended up chartering this huge $2 million yacht for 4 hours and going for a cruise and doing some jetskiing and water sports. The captain was Aussie, so we all had a good chat with him. Apparently it's a corporate vessel, with an exclusive chartering arrangement with the Westin. On Friday night, we all went out to a seafood buffet dinner at the Yellow Cafe over at Cenang Beach. Most of the guys and girls already knew each other courtesy of the day's activities, so dinner involved a lot of meeting the other halves. What I really liked about all of the pre-wedding stuff was that by the time the wedding rolled around, you really felt like you knew a bunch of Andrew and Janice's family and friends, which I think made the whole thing feel a lot more intimate. Saturday was a recovery day. Sarah's aunties Glenda (who was in Singapore on business) and Vicki (who lives in Thailand) had joined us at the resort by this stage, so we were all knocking around together checking out the island. We took the cable car to the top, but unfortunately the sky bridge was closed due to the inclement weather. There was a lot of touristy stuff at the bottom of where the cable car was, and Zoe got to have an encounter with a very large python. She even tried to give it a kiss. One of the things that was surprising was how popular Zoe was with all of the locals and other Asian tourists. We could probably have charged for photographs. There was a little girl, who I thought was possibly a similar age to Zoe (turned out she was two and a half) and I asked her parents if I could take a photo of the two girls together, and no sooner had they sat down together and a virtual paparazzi materialised out of nowhere taking photographs. Similarly, all of the staff at the Westin were totally taken by Zoe, and wanted to touch her. To her credit, Zoe handled all of the extra attention pretty well. The wedding wasn't until 6pm on Sunday, so we did a 4 hour island hopping boat tour in the morning. That was a bit of an experience. It was nothing short of sheer bedlam trying to board the right boat at the wharf (I use the term lightly) and then the boat's captain had to arrange the passengers appropriately so as to keep the boat on an even keel. It was a narrow motor boat with a canopy cover over the back two thirds of it. First stop was Dayang Bunting Island, home of Pregnant Maiden Lake, where we stopped for 45 minutes. More bedlam ensued getting off the boat at the wharf there (there was veritable flotilla of tour boats all trying to load and unload at the same time). Monkeys abounded. They seemed incredibly tame. We were practically tripping over them. Zoe didn't seem to be as taken by the monkeys as I thought she would. It was a short walk up a reasonable flight of stairs to get to the lake, though some incredibly humid jungle, and once we got there, there wasn't a huge amount to do in the time we had available. So we dunked Zoe's feet in the lake a few times and then made our way back to the chaos at the wharf and got back on our boat. We stopped for a similar amount of time at another island with a pretty nice beach, and Zoe had enough time to a quick swim in the ocean with Sarah, and then we made our way back to the boat again to another spot where we watched sea eagle feeding. Heaps of eagles circling overhead would take turns to swoop down and grab fish and fly off again. It was impressive. I think I managed to snap some good action shots with the long lens. Then we had the wedding itself. It was on the beach out the front of the Heavenly Spa on the resort. Let me say, the Westin knows how to put a wedding together. The ceremony was as lovely as the location. For the reception, the one 5 bedroom villa on the resort was booked for the night. Dinner was a buffet on the lawn out the front (briefly delayed thanks to a rain shower) and then speeches and general partying happened around the terrace out the back of the villa. There were some beautiful speeches. Janice's brought a tear to my eye. Part of the whole package included something like a 10 person video crew that were running around capturing everything. It felt a bit surreal. It was definitely the most high-end wedding I've ever been to (sorry Scott and Julie, your "Wedding of the Century" has been eclipsed by the "Wedding of the Millennium"). Monday was a wind down day. Zoe mercifully gave us a sleep in until 8am. Our flights didn't leave until the evening on Tuesday, so after we checked out we went for a drive around the island to try and get Zoe to take a nap in the car. We ended up at the fruit farm, which to the best of my understanding is run by the government. Zoe had fallen asleep by this stage, and Sarah said she'd stay behind in the car and have a nap too. So her mum and I went on the tour. It was a fun little tour. There were only four of us on it. They drive you around the farm on the back of a truck and show you the various crops they have. They stop at a few places and give you a bunch of fruit samples, attempt to answer your questions, and take funny pictures of you. The favourite seemed to be to dangle a fruit on a knife up close in front of the camera while having the subject(s) in the background look like they're carrying an enormous fruit. We got back to the car to an upset Sarah and small group of locals milling around. Sarah had managed to lock Zoe in the car for about 30 minutes when she got out to make her a sandwich after Zoe had woken up from her nap. Fortunately the engine was running and the air conditioning was on, so there was no risk of Zoe overheating (on the contrary, she was a bit on the cool side). Some kind locals had somehow figured out which car rental company to call (it's possible there's only one) and someone came out with a spare key, so the crisis was averted just before we got back to the car. It's the first time Zoe's been locked in the car (it's pretty much impossible in the Prius) and it was just our luck that it'd happen in a foreign country! One of the saddest things about Langkawi was the amount of garbage floating around in the ocean, and washing up on the beaches. I was swimming in the ocean off the yacht while we were out on the Buck's party, and it was disgusting. The jetski was temporarily disabled due to sucking in something. We visited Black Sand Beach, which was supposed to be something of a tourist attraction. Covered in trash. Ian Kiernan would be sad. The captain of the yacht we were on for the Buck's party said it was all from Thailand and got flushed out of all the rivers and streams by the recent rain. Just goes to show how connected everything is. The house is very quiet without Sarah and Zoe, but my friend Liam is visiting from Zurich for a couple of weeks, so that's helping break the silence. I'm using the time to get a heap of things done that I don't otherwise get time to get done. Unfortunately the photos from the trip are only available on Facebook at the moment, but they'll eventually get uploaded here.

Here in Norway the public libraries are debating with the publishing houses how to handle electronic books. Surprisingly, the libraries seem to be willing to accept digital restriction mechanisms (DRM) on books and renting e-books with artificial scarcity from the publishing houses. Time limited renting (2-3 years) is one proposed model, and only allowing X borrowers for each book is another. Personally I find it amazing that libraries are even considering such models. Anyway, while reading part of this debate, it occurred to me that someone should present a more sensible approach to the libraries, to allow its borrowers to get used to a better model. The idea is simple: Create a computer system for the libraries, either in the form of a Live DVD or a installable distribution, that provide a simple kiosk solution to hand out free e-books. As a start, the books distributed by Project Gutenberg (abount 36,000 books), Project Runenberg (1149 books) and The Internet Archive (3,033,748 books) could be included, but any book where the copyright has expired or with a free licence could be distributed. The computer system would make it easy to:

• Copy e-books into a USB stick, reading tablets, cell phones and other relevant equipment.
• Show the books for reading on the the screen in the library.

In addition to such kiosk solution, there should probably be a web site as well to allow people easy access to these books without visiting the library. The site would be the distribution point for the kiosk systems, which would connect regularly to fetch any new books available. Are there anyone working on a system like this? I guess it would fit any library in the world, and not just the Norwegian public libraries. :)

Mankind, it would appear, owe a great deal of their evolutionary advantage to using tools. This ability appears to have been massively amplified by our creation of machine tools.

The MeeGo community is frustrated with the news of the MeeGo brand being abandoned. Some are understandably angry or otherwise not happy about how Linux Foundation, Intel handled the Tizen announcement and community in general - or more like how they didn't handle it at all.Last week Openmind 2011 happened to be arranged in Tampere on the very same day as Tizen announcement came alive. It was good in the way that it lead to the fact that Nomovok's CEO Pasi Nieminen was able to initiate the "Reigniting MeeGo" session not just by talking vague things about future, but actually about the process which led to Tizen and the unfortunately brief initial PR about it. Pasi is intense on emphasizing the quality and role of Qt in Tizen as well, even though officially Tizen is all about HTML5 and apparently from Samsung's part at least EFL is provided as a native toolkit. However, the promise of Tizen compared to MeeGo is reportedly that the toolkit is not specified in compliancy documents, so HTML5 with WAC is the main/only "3rd party apps" layer whereas others can be offered case-by-case. This means that unlike before, the underlying system can be built on top of practically any distribution (theoretically) and using whatever toolkits and other techniques wanted. Obviously the "Nordic System Integrators" are probably all very keen of using Qt to produce more of Nokia N9 quality user experiences in various products.

Taking the corporate hat off, I as a community member am also puzzled. The only reason I was not completely blown by the news was that I didn't yet manage to get involved in MeeGo community on a daily basis, since I'm involved with a dozen communities already. Instead I've been more like scratching the surface with MeeGo Network Finland meetings, IRC activity, OBS usage for building a few apps for MeeGo Harmattan and MeeGo proper etc. But I can somewhat understand how people like Jarkko Moilanen from meego-fi feel. They have given a _lot_ to the MeeGo community and brand, all taken away without hearing or pre-notice.

So where to now for MeeGo community? Tizen is one obvious choice. However, for all the talks that even I started this post with, Tizen is still vaporware today, and the dislike of how community is being treated might make it easy to consider other options. Also, if Tizen's reference implementation has lesser meaning, it might also mean less to actually be "in" the Tizen community than in MeeGo. I met Jos Poortvliet at Openmind, and he invited people to openSUSE. There is a lot of common ground with MeeGo and openSUSE - strong OBS usage, RPM packaging, community side focused on KDE and therefore Qt.

I would like to now point similarly to Debian! If one is tired about corporate interests and not listening to community, there is no match for Debian's 15+ years history, purely volunteer based, trust based organization, and first of all scope. While openSUSE has traditionally focused on desktop (even though like Jos pointed out they are open to all new contributions and projects), Debian has always had the "universal" scope, ie. no boundaries besides producing free software operating system for various purposes. There are over 10 architectures maintained at the moment, including the ARM (different ports for ARMv4 and hard-float ARMv7) and x86 from MeeGo world. There are even alternative kernels to Linux, mainly the GNU/kFreeBSD port. There are multiple relevant plans and projects like the Smartphones wiki area, most noticeably Debian on Neo FreeRunner. I have run Debian on my primary mobile phone for over 2.5 years, although now in the recent months I've had dual-SIM in my Nokia N950 as well (Debian not yet running on Nokia N950 or Nokia N9 - but it can and will be done!).

What Debian may lack in both good and bad is corporate funding, if you don't count the still quite respectful contributions from Ubuntu to Debian (it's in Ubuntu's interests to contribute as much possible back to Debian, so that the delta remains small). For each and every aspect, it needs a volunteer - there are a thousand volunteer Debian Developers, and at least a double of that of people without the official DD status but who still maintain a package or two among the 25000+ packages in Debian. That means also that one my find it more lucrative to join a project that has paid people to do some of the "boring parts", more of fancy web tools, including for bug handling and build systems like the OBS (which I do love by the way). On the other hand, there is no other project in my opinion where what you do really matters as much.

To find out more about Debian from MeeGo perspective, please see the recent mailing list post Mobile UXes - From the DebConf11 BoF to the stars where I wrote most of the MeeGo (CE) part when I was asked to and known of my MeeGo involvement.

Last but not certainly least, there is the Mer project - originally "maemo reconstructed", ie. making Nokia's "not really distro" into a real distro by filling in the void places. Now it's obviously MeeGo reconstructed, and they aim to be the MeeGo they always wanted MeeGo to be! Read the post for details from Carsten Munk and other key Mer people. They share the love for Qt, and want the core to be as lean as possible. They also aim to incorporate the most community like aspect from MeeGo - MeeGo CE - as the reference vendor in Mer. They also aim to be Tizen compliant - and when Tizen comes alive, I wouldn't see why the Tizen reference implementation couldn't be used for saving resources. Maybe Nomovok and/or others could offer the Qt maintaining part.

So, it might be that Tizen itself is enough for most people's needs. The key point however in this post is not to fall in agony if one corporate based project takes big turns - it has happened before, it will happen in the future. There are always enough political and business reasons from some points of view to do Big Changes. But the wider community is out there, always, and it's bigger than you think. You should consider where you want to contribute by asking yourself why you are/were part of for example the MeeGo community. Aaron Seigo from KDE asked us all this question in the Openmind MeeGo Reignited session, and I think it's good to repeat.

A continuation of my intermittent series on Emdebian and prompted by a query from Paul Wise, I thought I'd cover some of the uses to which a device running Emdebian can be put. It is a bit long this one...

So, yes, there are general purpose computers which could run Emdebian but there are common features amongst general purpose computers:

1. Lots of different tasks must be supported - lots of different packages installed, each user could quite easily use a distinct set of packages from another user.


2. Multi-user support - even if any one machine is only used by one person, a general purpose computer must have support for adding another user - not just a system user, a real user with a login, shell, possibly bookmarks and browser history etc.


3. Connectivity - general purpose computers (and that includes mobile phones) must connect to a variety of other computers to be at all useful.


4. Multi-modal data input - there needs to be support for a mouse and a keyboard or touchscreen and just as likely, software support for accessibility interfaces like an on screen keyboard and switch inputs.


5. Storage for user data - the system admin can't know how much data users are going to add and if it is general purpose, it's probably going to support media players, image viewers, even office software. The data files managed and written by such software tend to be large and tend to accumulate into large collections. This means that a general purpose machine cannot anticipate the full storage requirements, so has to allow plenty of room for user data storage. If there's so much storage space available, why use an OS which is primarily designed to take up less space? Just use the full version, Debian.

Now compare with the common features of special purpose or single-purpose computers:

1. Single task only - a splash screen (without support for removal) covers all the boot information, the single task starts as a daemon and if the user quits the task, the task initiates a shutdown, not quit. Every effort is made to prevent the task from exposing the underlying OS, including auto-restarts if it does crash.


2. Single-user support - indeed, user is not the typical user. user in this case is a non-login user without a shell, created by the postinst of the task package purely to support running the task without root privileges (because if the task doesn't need privileges, it shouldn't retain privileges).


3. Single mode input - don't expect these machines to support 105 key keyboards. You'll be lucky to have 5 keys which are really just buttons. The button won't just enter 'J' into the machine, it will fire off a command to charge your bank or start a drill or shutdown a malfunctioning service. (Each button will often be multi-functional as well, depending on context). Some devices will replace the mouse with a touchscreen but many are fitted in places where touchscreens simply won't survive, where keys have to be able to withstand being submerged in water (or oil) for hours. Many machines simply won't have mouse support at all. Many will not have any typical external connectors. Indeed, units which go into dirty or wet environments certainly won't have standard external connectors of any kind. Think boats and factories. Internal battery packs with custom hot-fit replacements, buttons encased in 4mm of clear plastic, screens viewable only through protective mesh / filters.


4. Restricted connectivity - if the task doesn't need networking, no networking hardware is fitted. If the task doesn't need USB, no USB hardware is fitted. If the task doesn't need serial connections, there will probably be an internal serial connector but also a Warranty void if removed sticker. Don't expect an RS232 inside the case either, this will be a flat cable connector which you'll need to attach to a breakout board which correctly connects the cable wires to the RS232 port pins. That mapping is not standardised (why would it? only service engineers are approved to use it.)


5. Constrained user data - the task is in complete control and determines what the user can view, store and access. Storage can be expensive and a power drain, so if the task doesn't need Gb of storage, don't fit the machine with Gb of storage. Profit.

So what are these mysterious single-use computers running Emdebian? It's hard to tell because the task has no particular reason to tell the user anything about the OS. Perhaps a different way to phrase the same question would be: Why use a general purpose OS for a special purpose computer?.

• Single task does not mean single process - doing one thing can involve doing a couple of different things behind the scenes, so a multi-tasking kernel and userspace is extremely useful in providing a responsive machine without wasting money on unnecessarily powerful hardware. Multithreading support is also valuable.


• Software reuse makes good business sense - why write a single-usage operating system when the device needs a multi-tasking OS anyway and suitable ones already exist?


• Minimal software - no point packaging Xorg or GNOME when the user interface is a tiny QVGA framebuffer and the device only has 128Mb of RAM. By all means get the core system (cairo, pango, Qt, gettext, dpkg) from Emdebian but the actual code which performs the task of the device will generally be bespoke and, yes, that generally means proprietary, closed source, utterly non-free. This is where GNU/Linux wins over iOS and Windows - the graphical software is trivially separated from the core software and tested as such. If your device has only monochrome text output (and plenty of those still exist), what possible reason is there to use a graphical toolkit? Use the multi-tasking support of the OS to handle inter-process communication and understand the licensing - job done. Again, these devices have deliberately limited storage capacity, the smaller the OS, the lower the cost of the device, within certain constraints.


• Debugging compatibility - this is the big one. It's hard debugging the task software on device with the differences in architecture and connectivity. Why not use the same packages on the same OS on the development machines as on the devices? All the power and all the screen resources to properly investigate bugs, simple way to provide an emulator for translation support and other activities (like user interface training). There will always be bugs which have to be debugged on device - timing issues, hardware issues - but why not make at least a proportion of the likely bugs easier to fix by using binary compatible libraries on both the big desktop machines and the small devices?

This is why Emdebian Grip is popular for these machines - because Emdebian Grip is binary compatible with the equivalent Debian suite and when a bug appears in the high level user interface, it is much easier to debug that on the desktop than on device. Debian stable is a fantastic OS for commercial development - the stability of the OS makes detection of interface bugs much easier. It is rare that developers have to investigate whether bugs in their UI come down to a bug in the underlying OS. That saves time.

The machines themselves? I'm sure others can come up with other examples but these are some which come to my mind (no particular order):

• Control interfaces - solar power controllers, robot controllers - anywhere that a low power board can monitor responses from other machines and do something intelligent about reported errors. Temperature limit responses, pressure limit responses, encoding all of those in assembly or VHDL is hard, so use a system which provides for more creative result handling.


• Electric car charging points - very simple user interface, single task which doesn't complicate things by expecting the user to tap on something to start the main application - just process the card details, charge the customer and provide power.


• Similarlyelectronic point of sale devices - EPOS. The staff on the shop floor don't want to be waiting for an X server to start up and then using a menu. The device starts up and runs one task (checking barcodes, printing labels, counting stock), if the task is shutdown, the machine shuts down.


• Communication aids - my area, a machine which presents a simple UI on non-standard framebuffer hardware, lots of audio output but little other connectivity. Resilience is very important, so solid state is vital but why provide more solid state storage than the device can actually needs?

Anywhere where the device simply needs to DoTheRightThing in a variety of unpredictable circumstances, you're going to need a general purpose OS to gather the data and produce the right result. Anywhere where the device has only a single task, you're going to want to not provide access to the rest of the system. There's no point providing "general purpose access" when the device doesn't have "general purpose hardware". There's no point fitting general purpose hardware if the device does not use it. That is a waste of money, a waste of resources and causes unnecessary delays if one component or other becomes unobtainable or changes behaviour. Limit your exposure to hardware bugs and get the product out to the user more quickly and more reliably, everyone is happy. Fit non-standard hardware and you can fit custom hardware which better matches the profile of the device usage - why use standard speakers capable of CD quality across a vast frequency range when the device only makes very very loud alert noises which can be done at lower sample rates and with narrow range, high amplitude, speakers.

There are all sorts of estimates for the number of computers now in the average Western home. It's worth noting that the vast majority of those computers are not PCs or laptops or even game consoles and routers - all instances of general purpose machines with general purpose access / connectivity. A lot of the computers in your home are embedded in machines like white goods, media controllers (set top boxes and the like). Some of these will still use low level firmware written entirely in assembly or COBOL. More complex ones already have a general purpose OS inside yet constrain that OS with a simple, user-friendly interface which is tailored to the hardware actually fitted. It might be cool to connect to your set top box over Bluetooth but really, is that actually going to sell more units? Only if there are other high-end services written for the device which the average customer won't want to see pushing up the price of the base unit. Some machines (particularly in automotive uses) will need to be using Real Time operating systems and Safety Critical operating systems but even some of those are based on general purpose systems - just old versions which have been thoroughly tested.

Finally, consider that for a lot of these devices, the customer is not you. The customer is another business who then put the device into something else which is installed into a factory production line which produces something which goes into creating a consumer product you find in Tesco or on Amazon or bundled in with another service like your TV/broadband contract. There is absolutely no reason for these devices to provide general purpose computing to you, even if the device itself is part of something else which can provide such services.