A month ago, I published an article on the compared performances of stunnel, nginx and stud as SSL terminators. The conclusion was to use stud on a 64-bit system, with session caching and AES. stunnel was unable to scale properly and nginx exhibited important latency issues. I got constructive comments on many aspects. Therefore, here is the second round. The protagonists are the same but both the conditions and the conclusions have changed a bit.

Benchmarks The hardware platform remains unchanged:

• HP DL 380 G7
• 6 GB RAM
• 2 L5630 @ 2.13GHz (8 cores), hyperthreading disabled
• 2 Intel 82576 NIC
• Spirent Avalanche 2900 appliance

We focus on the following environment:

• Ubuntu Lucid, featuring a patched version of OpenSSL 0.9.8k with AES-NI builtin and enabled by default
• 2.6.39 kernel (250 HZ)
• 64-bit
• 1 core for HAProxy, 2 cores for network cards¹, 4 cores for SSL, 1 core for the system
• system limit of 100,000 file descriptors per process

On the SSL side, the main components are:

• 2048-bit certificates
• AES128-SHA cipher suite
• 8 HTTP/1.0 requests per client
• HTTP response payload of 1024 bytes

If you are interested in different conditions (32-bit, number of cores, 1024-bit certificates, proportion of session reuses), have a look at the previous article whose results on the matter are still valid. I only consider 2048-bit certificates because most certificates issued today are 2048-bit. In SP800-57, NIST advises that 1024-bit RSA keys will no longer be viable after 2010 and advises moving to 2048-bit RSA keys:

If information is initially signed in 2009 and needs to remain secure for a maximum of ten years (i.e., from 2009 to 2019), a 1024-bit RSA key would not provide sufficient protection between 2011 and 2019 and, therefore, it is not recommended that 1024-bit RSA be used in this case.

And the protagonists are:

• stud, @e207dbc5a3, with session cache enabled for 20,000 sessions, listen backlog of 1000 connections,
• stunnel 4.44, configured with --disable-libwrap --with-threads=pthread (configuration),
• nginx 1.1.4, configured with --with-http_ssl_module (configuration).

UPDATED: Configuration of nginx has been updated to switch to off both multi_accept and accept_mutex thanks to the feedback of Maxim Dounin on latency issues that were present in the previous article (and still present in the early version of this article). HAProxy (configuration) is used for stunnel and stud to add load-balancing. For each test, we want to maximize the number of HTTP transactions per second (TPS) while minimizing latency and unsucessful transactions. The maximum number of allowed failed transactions is set at 1 transaction out of 1000. The maximum allowed average latency is 100 ms².

Control case Below is the first comparison between stud, nginx and stunnel on plain Ubuntu Lucid. Like in previous benchs, stunnel is not able to scale, with a maximum of 400 TPS. However, early latency issues of nginx has been solved, thanks to the change in configuration. stud and nginx gets similar performance but, as you can see on the bottom plot (whose scale is logarithmic) stud starts with an initial latency of 40 ms. This is pretty annoying. I have tried some micro optimization to solve this issue but without luck. If latency matters for you, stick with stunnel or nginx (or find where the problem lies in stud). UPDATED: After reporting this issue, a fix has been commited. The culprit was Nagle s algorithm which improves bandwidth at the expense of latency. With the fix, performances are mostly the same but there is no latency penalty any more.

OpenSSL 1.0.0e When I published the first round of benchmarks, Micha Trojnara was dubious about the bad results of stunnel. He told me on Twitter:

I solved it! You compiled stunnel against a 4 years old OpenSSL with MT-unsafe SSL_accept. Upgrade OpenSSL and rebuild stunnel. This bug was fixed in OpenSSL 1.0.0b. I implemented a (slow) workaround, so stunnel doesn t crash with earlier versions of OpenSSL.

Therefore, I backported Ubuntu Oneiric OpenSSL 1.0.0e to Ubuntu Lucid³ and, as you can see yourself on the plot below, there is a huge performance boost for stunnel: about 400%! stud features a slight improvement too (about 8%) but still starts with a latency of 40 ms while nginx does not improve.

Intel Accelerator Engine Most Linux distributions are still using OpenSSL 0.9.8. For example, Debian Squeeze ships with OpenSSL 0.9.8o, Ubuntu Lucid with 0.9.8k and RHEL 5.7 with 0.9.8e. Therefore, they are not eligible to recent performance improvements. To solve this, Intel released Intel Accelerator Engine, which also brings, among other things, AES-NI support:

This is essentially a collection of assembler modules from development OpenSSL branch, packed together as a standalone engine. Standalone means that it can be compiled outside OpenSSL source tree without patching the latter. Idea is to provide a way to utilize new code in already released OpenSSL versions constrained by support and release policies limiting their evolvement to genuine bug fixes.

Alas, in my tests, there was no improvement compared to OpenSSL 1.0.0e (already patched with AES-NI support). stunnel comes with out-of-the-box support for additional engines. Just add something like this to your stunnel.conf:

engine=dynamic
engineCtrl=SO_PATH:/WOO/intel-accel-1.4/libintel-accel.so
engineCtrl=ID:intel-accel
engineCtrl=LIST_ADD:1
engineCtrl=LOAD
engineCtrl=INIT

For stud, you should grab support for engine selection from GitHub and for nginx, just specify the engine with ssl_engine.

Intel Performance Primitives In his great article Accelerated SSL , Simon Kuhn shows huge performance improvements by using Intel s IPP library, a library aimed at leveraging power of multi-core processors and including some cryptographic primitives and a patch to make use of them in OpenSSL. Unfortunately, this library is not free, not even as in free beer . The patch provided by Intel only applies on OpenSSL 0.9.8j. Simon updated it for OpenSSL 1.0.0d. It also applies with a slight modification to OpenSSL 1.0.0e. Look at Simon s article for instructions on how to compile OpenSSL with IPP. Simon measured 220% improvement on RSA 1024-bit private key operations by using this library. Unfortunately, we don t get such a performance burst with stud which exhibits a 15% improvement (but remember that private key operations only happen once every 8 transactions thanks to session reuse). Worst, with stunnel, performances are reduced by one third and nginx just segfaults. I think such a library is risky to use in production.

Hyperthreading During the previous tests, only half the capacity of each core were used. For stud, adding more workers per core only slightly improves this situation (less than 10% with 4 workers per core). Therefore, I thought using hyperthreading could rectify the situation here. In this case, stud, stunnel and nginx are allocated 8 virtual cores instead of 4. stud only got a small improvement of 5%, nginx a moderate improvement of 11% while stunnel got a significant improvement of 26%.

Conclusion Here is a quick recap of the maximum TPS reached on each tests (while keeping average response time below 100 ms). All tests were run on 4 physical cores with 2048-bit certificates, AES128-SHA cipher suite. Each client makes 8 HTTP requests and reuse the first SSL session.

Context	nginx 1.1.4	stunnel 4.44	stud @e207dbc5a3
OpenSSL 0.9.8k	1489 TPS	400 TPS	1492 TPS
OpenSSL 1.0.0e	1501 TPS	1561 TPS	1609 TPS
OpenSSL 1.0.0e + IPP	-	1101 TPS	1857 TPS
OpenSSL 1.0.0e + HT	1718 TPS	1973 TPS	1694 TPS

I also would like to stress that below 1500 TPS, stunnel adds an overhead of a few milliseconds while stud adds 40 milliseconds. With OpenSSL 1.0.0e and hyperthreading enabled, stunnel has both good performance and low latency on low loads to moderate loads. You may just want to limit the number of allowed connections to avoid collapsing on high loads. While in the previous round, stud was the champion and stunnel lagged behind, the situation is now reversed : stunnel is on top! UPDATED: As said earlier, the latency issue with stud has been solved. Performances remain mostly the same.

---

1. HAProxy and network cards don t need dedicated core for this kind of load. However, if you also want to handle non SSL traffic, those dedicated cores can get busy.
2. The real condition is more complex. Each point is considered valid if the corresponding average latency is less than 100 ms or if the average on the next 8 points is less than 100 ms. Temporary latency bursts are therefore allowed. We are not so forgiveful with failed transactions.
3. This is pretty straighforward: I just dropped multiarch support. I chose to use Ubuntu version of OpenSSL instead of upstream because there are some interesting patches, like automatic support of AES-NI.

I just uploaded a snapshot of GNU Screen to Debian Experimental. The package (4.1.0~20110819git450e8f3-1) is based on upstream s HEAD whose most recent commit currently dates to the 19th of August 2011. While the upload fixes tons of bugs which accumulated over the past two years in Debian s, Ubuntu s and upstream s bug tracker, I don t yet regard it as suitable for the next stable release (and hence for Debian Unstable) since there s one not so nice issue about it:

• #644788: screen 4.1.0 can t attach to a running/detached screen 4.0.3 session

Nevertheless it fixes a lot of open issues (of which the oldest is a wishlist bug report dating back to 1998 :-) and I didn t want to withhold it from the rest of the Debian community so I uploaded it to Debian Experimental. Issues closed in Debian Experimental

• #25096: digraph table should be run-time configurable
• #152961: lacks tsl/fsl/dsl caps
• #176626: mini-curses type of interface for screen -r w/ multiple screens? (Fixed by suggesting iselect, screenie or byobu)
• #223320: does not switch mouse mode
• #344759: mishandles xterm control string to set window title
• #353090: please enable the built-in telnet
• #361274: cannot reattach to sessionname if there is another session with similar sessionname
• #450421: please raise MAXWIN to at least 100 (merged with #499273)
• #461107: Requires test -t 0 even when opening a new window on existing screen
• #481411: window created with -d -m silently ignores -X exec
• #488619: Session name string escape
• #496750: screen -d -m and -D -m segfault if setenv given with no value in a configuration file
• #532240: screen with caption SEGVs when resized to 1 line tall
• #541793: C-a h (mis)documented twice
• #558724: breaks altscreen
• #560231: Please remove restriction on user/login name length
• #578729: outputs spaces when refreshing/attaching a window with defbce on
• #591624: segfault when running screen -d -m with layout save default in .screenrc
• #603009: Updating the screen Uploaders list
• #612990: /etc/init.d/screen-cleanup: should check for existence of screen binary
• #621704: Fix slow scrolling in vertical splits
• #630535: manpage typo
• #641867: version bump (this bug report sparked the upload :-)

Issues which will be closed in Ubuntu

• #183849: update to git version of screen
• #315237: crashes with certain options and terminal sizes
• #582153: doesn t accept login names longer than 20 chars
• #588846: slow when using vertical split
• #702094: Copying and pasting from mutt includes many trailing spaces
• #786292: segfaults if using layout saving with -D -m
• #788670: segfault in screen/byobu in natty

Please test the version from Experimental If you are affected by one of the issues mentioned above, please try the version from Debian Experimental and check if they re resolved for you, too. Thanks to all who contributed! A lot of the fixes have been made or applied upstream by Sadrul Habib Chowdhury who also industriously tagged Debian bug reports as fixed-upstream . Thanks! Thanks also to Brian P Kroth who gave the initial spark to this upload by packaging Fedora 15 s git snapshot for Debian and filing bug although the upload is based on the current HEAD version of GNU Screen as this fixes some more important issues than the snapshot Fedora 15 includes. That way also two patches from Fedora/RedHat s screen package are included in this upload. (Co-) Maintainer wanted! Oh, and if you care about the state of GNU Screen in Debian, I d really appreciate if you d join in and contribute to our collab-maint git repository there are still a lot of issues unresolved and I know that I won t be able to fix all of them myself. And since Hessophanes unfortunately currently has not enough time for the package, we definitely need more people maintaining this package. P.S. Yes, I know about tmux and tried to get some of my setups working with it, too. But I still prefer screen over tmux. :-)

For Ada Lovelace Day this year, I want to share my appreciation for Dr. Marian C. Diamond. In years past, I ve saluted women in the field of computing, which is my field as well. Dr. Diamond, however, is a biologist. Her research includes neuroanatomy, environment, immune functions, and hormones. In particular, she is interested in studying the effects of the external environment, aging, and immune responses on the cerebral neocortex. She has, in her words, had a love affair with the brain for about 70 years. I know very little about biology. The content and methods of her research are, frankly, beyond me, though some of her results have garnered popular attention. She has inspired me by demonstrating that rare combination of gifts: a deep understanding of a technical subject, and the ability to explain it to other people in an accessible way. In her interviews, articles and lectures, many of which are available online, Dr. Diamond displays these gifts in abundance. Her skill and enthusiasm for both learning and teaching is unmistakable. After applying her gifts in the classroom for many years, digital distribution has now enabled many more people to see and hear her, through millions of YouTube views. In 1960, she became the first female graduate student in UC Berkeley s anatomy department, and was apparently given the job of sewing a cover for a magnifying machine. I can only imagine the persistence required to continue from there to become a recognized leader in her field. She has gone on to help many other students along their way, and was named an unsung, everyday hero for the support she provided to students outside of the classroom or lab. As if that weren t enough, she has also traveled to Cambodia to apply her expertise in helping children injured by land mines. She still teaches today, just across the bay from where I write this, and will turn 85 next month.

After a little pause, I thought I add another entry to my ongoing tips series. Today we'll get rid of the black bars you can most likely observe on a widescreen display, when playing a video DVD with MPlayer. This little post assumes that you've already inserted the DVD you'd like to watch, your MPlayer setup works with dvd:// style addresses, you know which title from the DVD you want to watch (the examples will use the first title) and you know your monitor's aspect ratio (the examples uses 16:9). Now all that is needed are the following two steps:

1. Detect the cropping parameters:

   mplayer -vf cropdetect -monitoraspect 16:9 dvd://1

   The previous line is followed by the usual MPlayer header and output. The film should start playing and after a few seconds you can press q (just wait until the detected values stop changing). Now there should be a few lines looking like:

   [CROP] Crop area: X: 0..719  Y: 76..499  (-vf crop=720:416:0:80).
   A:   1.5 V:   1.5 A-V: -0.000 ct: -0.018  38/ 38  5%  7%  1.0% 0 0
   [CROP] Crop area: X: 0..719  Y: 75..500  (-vf crop=720:416:0:80).
   A:   1.6 V:   1.6 A-V: -0.000 ct: -0.018  39/ 39  5%  7%  1.1% 0 0
   [CROP] Crop area: X: 0..719  Y: 75..500  (-vf crop=720:416:0:80).
   A:   1.6 V:   1.6 A-V: -0.000 ct: -0.018  40/ 40  5%  7%  1.1% 0 0

2. Play the film with the detected crop values and the correct aspect for the video material (4:3 for PAL and 3:2 for NTSC):

   mplayer -vf crop=720:416:0:80 -monitoraspect 16:9 -aspect 4:3 dvd://1

Of course you can add any other options to the invocation as usual, though you only need to add those to the second invocation (e.g. select the correct audio stream or start in fullscreen mode). Also, you might want to set the monitor aspect permantly in your MPlayer configuration (just add monitoraspect = "16:9"), as the monitor rarely changes for the average machine /etc/mplayer/mplayer.conf.local is probably the best place (make sure your /etc/mplayer/mplayer.conf includes that file at the end). Now you should be able to enjoy all video DVDs on your widescreen display without black bars at the top and bottom!

Here in Norway the public libraries are debating with the publishing houses how to handle electronic books. Surprisingly, the libraries seem to be willing to accept digital restriction mechanisms (DRM) on books and renting e-books with artificial scarcity from the publishing houses. Time limited renting (2-3 years) is one proposed model, and only allowing X borrowers for each book is another. Personally I find it amazing that libraries are even considering such models. Anyway, while reading part of this debate, it occurred to me that someone should present a more sensible approach to the libraries, to allow its borrowers to get used to a better model. The idea is simple: Create a computer system for the libraries, either in the form of a Live DVD or a installable distribution, that provide a simple kiosk solution to hand out free e-books. As a start, the books distributed by Project Gutenberg (abount 36,000 books), Project Runenberg (1149 books) and The Internet Archive (3,033,748 books) could be included, but any book where the copyright has expired or with a free licence could be distributed. The computer system would make it easy to:

• Copy e-books into a USB stick, reading tablets, cell phones and other relevant equipment.
• Show the books for reading on the the screen in the library.

In addition to such kiosk solution, there should probably be a web site as well to allow people easy access to these books without visiting the library. The site would be the distribution point for the kiosk systems, which would connect regularly to fetch any new books available. Are there anyone working on a system like this? I guess it would fit any library in the world, and not just the Norwegian public libraries. :)

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (144.3 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg work While taking care of the last details for the hardening feature in dpkg 1.16.1, I have mailed debian-devel to find volunteers to handle a hardening release goal. The objective is to ensure a large number of packages have been converted/rebuilt to actually use the new hardening build flags. Then I prepared the draft of the announce of the dpkg 1.16.1 upload (aka Bits of dpkg maintainers sent to debian-devel-announce) which got expanded by Guillem to also cover new features since dpkg 1.15.7. update-alternatives got some refactoring by Guillem which resulted in a regression that has been fortunately discovered by Sven Joachim. I fixed that regression and did some further cleanup inspired by the root cause of this regression (see top 4 commits here). Note that Sven is one of the few persons who are running the git version of dpkg. Hopefully the number of tester will increase since I recently documented the APT repositories with autobuilt versions of dpkg in the wiki. At the end of the month, I started working on a bugfix release (what s going to be 1.16.1.1) by fixing some of the unavoidable problems discovered after an upload that accumulated more than 4 months worth of work (see top 4 commits here). The Debian Administrator s Handbook I spent countless hours finalizing the launch of the crowdfunding campaign for the Debian Administrator s Handbook and it went live on September 27th. So far it s on good track with more than 63% of the base funding already secured. But we still have a long way to go to reach the liberation goal (we re at 21%). It s still worth nothing that more than 55% of the money raised has been put in the liberation fund so there are many persons who care about getting the book freed. More than 250 persons are supporting the project currently with an average contribution of 38 EUR. I would have expected much less for the average contribution but many more supporters. I still hope we can get more people on board with the perspective of a good DFSG-free Debian ebook. Did you order your copy? If not, click here and fix this! By the way Paypal used to be required but it s no longer the case, you can support the project just with your usual credit card. Misc blog updates Over time, I have written many useful articles for Debian users and Debian contributors. But scattered in the history, they are somewhat difficult to find. To fix this I have created some index pages listing them. Check them out:

• Mastering Debian
• Contributing to Debian
• Debian Packaging Tutorials

Two new articles joined those pages this month: How to triage bugs in the Debian Bug Tracking System and Understand dpkg and don t get stuck with a maintainer script failure. While writing the first article, I noticed we lacked a good page showing the most buggy packages so I quickly created it (with the help of UDD): http://qa.debian.org/cgi-bin/bugs-by-source Misc packaging work I did a small update to the developer s reference. Luca Falavigna submitted a patch to clarify how one is supposed to deal with meta-packages (cf #569219), I improved it and integrated the result in the SVN repository. I upgraded nautilus-dropbox to version 0.6.9 and while doing this I discovered a bug in mergechanges (filed as #640782). I uploaded a new release of quilt mainly to add the Multi-Arch: foreign field so that it can satisfy dependencies of foreign packages (i.e. packages of a different architecture). Django released some security advisories (tracked in #641405) and since the maintainer did not deal with the issue, I stepped up to the task (I m a backup maintainer) and released the fixed version 1.3.1 to unstable. I took the opportunity to switch from python-support to dh_python2, and do some misc improvements to the packaging (see changelog). I wanted to update publican to a newer version but it turned out to be not possible because Debian doesn t have the latest version of docbook-xsl yet. I also discovered some bugs in the test suite and forwarded upstream the patch I created (see upstream bug). On top of this, fop was failing due to some java problem related to the introduction of multiarch. After having reported the bug, the java maintainers quickly released a fixed version. So now publican is ready in the git repository but it s waiting on the docbook-xsl update. I got in touch with the maintainer who said he would have the time to take care of it by mid-october. Thanks See you next month for a new summary of my activities.

One comment Liked this article? Click here. My blog is Flattr-enabled.

The Ubuntu Technical Board conducts a regular review of the most popular Ubuntu Brainstorm ideas (previous reviews conducted by Matt Zimmerman and Martin Pitt). This time it was my turn. Apologies for the late arrival of this review. Contact lens in the Unity Dash (#27584) Unity supports Lenses, which provide a consistent way for users to quickly search for information via the Dash. Current lenses include Applications, Files, and Music, but a number of people have asked for contacts to be accessible using the same interface. While Canonical's DX team isn't currently working on this for Ubuntu 11.10 or 12.04, we'd love somebody who's interested in this to get involved. Allison Randal explains how to get started, including some skeleton example code and several useful links. Displaying Ubuntu version information (#27460) Several people have asked for it to be more obvious what Ubuntu version they're running, as well as other general information about their system. John Lea, user experience architect on the Unity team, responds that in Ubuntu 11.10 the new LightDM greeter shows the Ubuntu version number, making that basic information very easily visible. For more detail, System Settings -> System Info provides a simple summary. Volume adjustments for headphone use (#27275) People often find that they need to adjust their sound volume when plugging in or removing headphones. It seems as though the computer ought to be able to remember this kind of thing and do it automatically; after all, a major goal of Ubuntu is to make the desktop Just Work. David Henningson, a member of Canonical's OEM Services group and an Ubuntu audio developer, responds on his blog with a summary of how PulseAudio jack detection has improved matters in Ubuntu 11.10, and what's left to do:

The good news: in the upcoming Ubuntu Oneiric (11.10), this is actually working. The bad news: it isn't working for everyone.

Making it easier to find software to handle a file (#28148) Ubuntu is not always as helpful as it could be when you don't have the right software installed to handle a particular file. Michael Vogt, one of the developers of the Ubuntu Software Center, responded to this. It seems that most of the pieces to make this work nicely are in place, but there are a few more bits of glue required:

Thanks a lot for this suggestion. I like the idea and it's something that software-center itself supports now. In the coming version 5.0 we will offer to "sort by top-rated" (based on the ratings&reviews data). It's also possible to search for an application based on its mime data. To search for a mime-type, you can enter "mime:text/html" or "mime:audio/ogg" into the search field. What is needed however is better integration into the file manager nautilus. I will make sure this gets attention at the next developer meeting and filed bug #860536 about it. In nautilus, there is now a button called "Find applications online" available as an option when opening an unknown file or when the user selects "open with...other application" in the context menu. But that will not use the data from software-center.

Show pop-up alert on low battery (#28037) Some users have reported on Brainstorm that they are not alerted frequently enough when their laptop's battery is low, as they clearly ought to be. This is an odd one, because there are already several power alert levels and this has been working well for us for some time. Nevertheless, enough people have voted for this idea that there must be something behind it, perhaps a bug that only affects certain systems. Martin Pitt, technical lead of the Ubuntu desktop team, has responded directly to the Brainstorm idea with a description of the current system and how to file a bug when it does not work as intended.

Mankind, it would appear, owe a great deal of their evolutionary advantage to using tools. This ability appears to have been massively amplified by our creation of machine tools.

I was due for another Google interview mail it seems. I have to say I wasn't expecting it, but this week I had a follow up to my polite mail from 6 months ago that said "No thanks, I'm not looking" asking if it was still the case. Normally I welcome this little bit of ego stroking; it's always nice to be wanted. Except that's not really the case, is it? It's an invitation to interview for something, not any indication that you've done more than tick some initial boxes. Google mails inevitably ask me if I'd like to work in SRE. It's always SRE. No one ever emails and asks if I want to work on self-replicating nanobots that will roam Mars searching for the perfect spot to build a beach house. And that's where things fall down. If someone currently has a job, then emailing them out of the blue to ask if they want to come and interview for something vague is hoping that they're either looking, and just haven't come to you yet, or not looking but unhappy enough with their current role that all they needed to start was an email asking them to submit a recent CV. For the former, if you're Google, do you really think that person doesn't know where to find you? For the latter, you're being quite presumptuous, aren't you? The act of updating my CV my be some effort (actually it's usually not, because the stuff that's not on it is the stuff I can't talk about because it's not released yet, or stuff that's specific and thus wouldn't go on a CV for a vague job spec). Even if it's not the act of interviewing is potentially a waste of time for both of us, if the role isn't clear. One argument used is that people will be placed according to the skills they show during the interview process. That's fine from the employers point of view, but if you're actively trying to get some interest from people who are gainfully employed you really need to grab their attention somehow. I can't remember the last time I had an unsolicited email interview offer that actually wowed me, or indeed even showed more than a passing sign of tailoring a spec to my profile. When I was running Black Cat I made a point of always replying to unsolicited CVs. How polite I was depended on how the covering emails were worded (a Word document with nothing else was likely to get short shrift, something well targeted in a Linux friendly format would normally get some comment about how we weren't hiring and were unlikely to be, but if that changed it would be mentioned on the website), but I felt people deserved a reply - I have been disappointed by not receiving responses myself to what I considered well targeted job inquiries. So far I've so far taken the same approach with mails from corporate recruiters (less so with recruiters that are associated with recruitment firms, rather than directly with the companies they are hiring for), but I'm starting to feel like changing that stance. Candidates are told to tailor CVs to the role being applied for, provide a decent cover letter, and in general make companies want to talk to them. Companies who are sending out recruitment emails should be held to the same standard. Even assuming you do a basic phone screen first, I can probably expect to need to take a day off work assuming that goes well. You need to convince me I can justify that before I'm going to feel like engaging at all. (And if I'm honest, based on what I've seen so far, it's unlikely to happen. All of the things I've considered have come from conversations with people I know directly about companies they own or work for, never some random contact via email. I try hard not to think of recruitment mail as spam, but I can how that line of thought follows through.) I should apologize to Google here. They got mentioned as an example, but I don't think they're particularly bad. I did interview with them at one point, and made the decision not to continue that process after deciding a different, more certain, path was better for me. So I've displayed interest. And in response to my reply today of "I know where to find you, so please assume I'll do so if I change my mind." they've said they'll make a note on their records.

Some years ago I came across a really peculiar newsgroup post. It was not about technicalities of any sort. It was about history. A beautifully written history of computers. From the Turing machine to the Free Software world, the original author managed to capture all the important events of the computer revolution with a great deal of humor. Re-posting it here, it is just brilliant. The Gospel of Tux (v1.0) In the beginning Turing created the Machine. And the Machine was crufty and bodacious, existing in theory only. And von Neumann looked upon the Machine, and saw that it was crufty. He divided the Machine into two Abstractions, the Data and the Code, and yet the two were one Architecture. This is a great Mystery, and the beginning of wisdom. And von Neumann spoke unto the Architecture, and blessed it, saying, Go forth and replicate, freely exchanging data and code, and bring forth all manner of devices unto the earth. And it was so, and it was cool. The Architecture prospered and was implemented in hardware and software. And it brought forth many Systems unto the earth. The first Systems were mighty giants; many great works of renown did they accomplish. Among them were Colossus, the codebreaker; ENIAC, the targeter; EDSAC and MULTIVAC and all manner of froody creatures ending in AC, the experimenters; and SAGE, the defender of the sky and father of all networks. These were the mighty giants of old, the first children of Turing, and their works are written in the Books of the Ancients. This was the First Age, the age of Lore. Now the sons of Marketing looked upon the children of Turing, and saw that they were swift of mind and terse of name and had many great and baleful attributes. And they said unto themselves, Let us go now and make us Corporations, to bind the Systems to our own use that they may bring us great fortune. With sweet words did they lure their customers, and with many chains did they bind the Systems, to fashion them after their own image. And the sons of Marketing fashioned themselves Suits to wear, the better to lure their customers, and wrote grave and perilous Licenses, the better to bind the Systems. And the sons of Marketing thus became known as Suits, despising and being despised by the true Engineers, the children of von Neumann. And the Systems and their Corporations replicated and grew numerous upon the earth. In those days there were IBM and Digital, Burroughs and Honeywell, Unisys and Rand, and many others. And they each kept to their own System, hardware and software, and did not interchange, for their Licences forbade it. This was the Second Age, the age of Mainframes. Now it came to pass that the spirits of Turing and von Neumann looked upon the earth and were displeased. The Systems and their Corporations had grown large and bulky, and Suits ruled over true Engineers. And the Customers groaned and cried loudly unto heaven, saying, Oh that there would be created a System mighty in power, yet small in size, able to reach into the very home! And the Engineers groaned and cried likewise, saying, Oh, that a deliverer would arise to grant us freedom from these oppressing Suits and their grave and perilous Licences, and send us a System of our own, that we may hack therein! And the spirits of Turing and von Neumann heard the cries and were moved, and said unto each other, Let us go down and fabricate a Breakthrough, that these cries may be stilled. And that day the spirits of Turing and von Neumann spake unto Moore of Intel, granting him insight and wisdom to understand the future. And Moore was with chip, and he brought forth the chip and named it 4004. And Moore did bless the Chip, saying, Thou art a Breakthrough; with my own Corporation have I fabricated thee. Thou thou art yet as small as a dust mote, yet shall thou grow and replicate unto the size of a mountain, and conquer all before thee. This blessing I give unto thee: every eighteen months shall thou double in capacity, until the end of the age. This is Moore s Law, which endures unto this day. And the birth of 4004 was the beginning of the Third Age, the age of Microchips. And as the Mainframes and their Systems and Corporations had flourished, so did the Microchips and their Systems and Corporations. And their lineage was on this wise: Moore begat Intel. Intel begat Mostech, Zilog and Atari. Mostech begat 6502, and Zilog begat Z80. Intel also begat 8800, who begat Altair; and 8086, mother of all PCs. 6502 begat Commodore, who begat PET and 64; and Apple, who begat 2. (Apple is the great Mystery, the Fruit that was devoured, yet bloomed again.) Atari begat 800 and 1200, masters of the game, who were destroyed by Sega and Nintendo. Xerox begat PARC. Commodore and PARC begat Amiga, creator of fine arts; Apple and PARC begat Lisa, who begat Macintosh, who begat iMac. Atari and PARC begat ST, the music maker, who died and was no more. Z80 begat Sinclair the dwarf, TRS-80 and CP/M, who begat many machines, but soon passed from this world. Altair, Apple and Commodore together begat Microsoft, the Great Darkness which is called Abomination, Destroyer of the Earth, the Gates of Hell. Now it came to pass in the Age of Microchips that IBM, the greatest of the Mainframe Corporations, looked upon the young Microchip Systems and was greatly vexed. And in their vexation and wrath they smote the earth and created the IBM PC. The PC was without sound and colour, crufty and bodacious in great measure, and its likeness was a tramp, yet the Customers were greatly moved and did purchase the PC in great numbers. And IBM sought about for an Operating System Provider, for in their haste they had not created one, nor had they forged a suitably grave and perilous License, saying, First we will build the market, then we will create a new System, one in our own image, and bound by our Licence. But they reasoned thus out of pride and not wisdom, not forseeing the wrath which was to come. And IBM came unto Microsoft, who licensed unto them QDOS, the child of CP/M and 8086. (8086 was the daughter of Intel, the child of Moore). And QDOS grew, and was named MS-DOS. And MS-DOS and the PC together waxed mighty, and conquered all markets, replicating and taking possession thereof, in accordance with Moore s Law. And Intel grew terrible and devoured all her children, such that no chip could stand before her. And Microsoft grew proud and devoured IBM, and this was a great marvel in the land. All these things are written in the Books of the Deeds of Microsoft. In the fullness of time MS-DOS begat Windows. And this is the lineage of Windows: CP/M begat QDOS. QDOS begat DOS 1.0. DOS 1.0 begat DOS 2.0 by way of Unix. DOS 2.0 begat Windows 3.11 by way of PARC and Macintosh. IBM and Microsoft begat OS/2, who begat Windows NT and Warp, the lost OS of lore. Windows 3.11 begat Windows 95 after triumphing over Macintosh in a mighty Battle of Licences. Windows NT begat NT 4.0 by way of Windows 95. NT 4.0 begat NT 5.0, the OS also called Windows 2000, The Millenium Bug, Doomsday, Armageddon, The End Of All Things. Now it came to pass that Microsoft had waxed great and mighty among the Microchip Corporations; mighter than any of the Mainframe Corporations before it had it waxed. And Gates heart was hardened, and he swore unto his Customers and their Engineers the words of this curse: Children of von Neumann, hear me. IBM and the Mainframe Corporations bound thy forefathers with grave and perilous Licences, such that ye cried unto the spirits of Turing and von Neumann for deliverance. Now I say unto ye: I am greater than any Corporation before me. Will I loosen your Licences? Nay, I will bind thee with Licences twice as grave and ten times more perilous than my forefathers. I will engrave my Licence on thy heart and write my Serial Number upon thy frontal lobes. I will bind thee to the Windows Platform with cunning artifices and with devious schemes. I will bind thee to the Intel Chipset with crufty code and with gnarly APIs. I will capture and enslave thee as no generation has been enslaved before. And wherefore will ye cry then unto the spirits of Turing, and von Neumann, and Moore? They cannot hear ye. I am become a greater Power than they. Ye shall cry only unto me, and shall live by my mercy and my wrath. I am the Gates of Hell; I hold the portal to MSNBC and the keys to the Blue Screen of Death. Be ye afraid; be ye greatly afraid; serve only me, and live. And the people were cowed in terror and gave homage to Microsoft, and endured the many grave and perilous trials which the Windows platform and its greatly bodacious Licence forced upon them. And once again did they cry to Turing and von Neumann and Moore for a deliverer, but none was found equal to the task until the birth of Linux. These are the generations of Linux: SAGE begat ARPA, which begat TCP/IP, and Aloha, which begat Ethernet. Bell begat Multics, which begat C, which begat Unix. Unix and TCP/IP begat Internet, which begat the World Wide Web. Unix begat RMS, father of the great GNU, which begat the Libraries and Emacs, chief of the Utilities. In the days of the Web, Internet and Ethernet begat the Intranet LAN, which rose to renown among all Corporations and prepared the way for the Penguin. And Linus and the Web begat the Kernel through Unix. The Kernel, the Libraries and the Utilities together are the Distribution, the one Penguin in many forms, forever and ever praised. Now in those days there was in the land of Helsinki a young scholar named Linus the Torvald. Linus was a devout man, a disciple of RMS and mighty in the spirit of Turing, von Neumann and Moore. One day as he was meditating on the Architecture, Linus fell into a trance and was granted a vision. And in the vision he saw a great Penguin, serene and well-favoured, sitting upon an ice floe eating fish. And at the sight of the Penguin Linus was deeply afraid, and he cried unto the spirits of Turing, von Neumann and Moore for an interpretation of the dream. And in the dream the spirits of Turing, von Neumann and Moore answered and spoke unto him, saying, Fear not, Linus, most beloved hacker. You are exceedingly cool and froody. The great Penguin which you see is an Operating System which you shall create and deploy unto the earth. The ice-floe is the earth and all the systems thereof, upon which the Penguin shall rest and rejoice at the completion of its task. And the fish on which the Penguin feeds are the crufty Licensed codebases which swim beneath all the earth s systems. The Penguin shall hunt and devour all that is crufty, gnarly and bodacious; all code which wriggles like spaghetti, or is infested with blighting creatures, or is bound by grave and perilous Licences shall it capture. And in capturing shall it replicate, and in replicating shall it document, and in documentation shall it bring freedom, serenity and most cool froodiness to the earth and all who code therein. Linus rose from meditation and created a tiny Operating System Kernel as the dream had foreshewn him; in the manner of RMS, he released the Kernel unto the World Wide Web for all to take and behold. And in the fulness of Internet Time the Kernel grew and replicated, becoming most cool and exceedingly froody, until at last it was recognised as indeed a great and mighty Penguin, whose name was Tux. And the followers of Linus took refuge in the Kernel, the Libraries and the Utilities; they installed Distribution after Distribution, and made sacrifice unto the GNU and the Penguin, and gave thanks to the spirits of Turing, von Neumann and Moore, for their deliverance from the hand of Microsoft. And this was the beginning of the Fourth Age, the age of Open Source. Now there is much more to be said about the exceeding strange and wonderful events of those days; how some Suits of Microsoft plotted war upon the Penguin, but were discovered on a Halloween Eve; how Gates fell among lawyers and was betrayed and crucified by his former friends, the apostles of Media; how the mercenary Knights of the Red Hat brought the gospel of the Penguin into the halls of the Corporations; and even of the dispute between the brethren of Gnome and KDE over a trollish Licence. But all these things are recorded elsewhere, in the Books of the Deeds of the Penguin and the Chronicles of the Fourth Age, and I suppose if they were all narrated they would fill a stack of DVDs as deep and perilous as a Usenet Newsgroup. Now may you code in the power of the Source; may the Kernel, the Libraries and the Utilities be with you, throughout all Distributions, until the end of the Epoch. Amen.

Woodchuck and FrOSCon

At the end of August, I attended FrOSCon in Bonn again, after skipping it last year. The evening before FrOSCon however, I visited Neal Walfield, his wife Isabel and their little son Noam in D sseldorf. Besides having a great time and a lovely dinner, I was most impressed by their collection of Maemo devices (they had at least two N770s, an N900 and, to my jealousy, an N950) which Neal is doing research on these days. He works on woodchuck, which is a project investigating how to improve data availability on mobile devices and our conversation prompted him to implement ATP Woodchuck, which makes smarter decisions when to run APT upgrade on your Maemo device then the standard updater. As part of the research, they also run a user behaviour study which I joined, where one installs a client which records various data off your N900 and sends them anonymized (he seems to be doing a good job at that) to figure out how people use their mobile devices and hopefully enhance the experience. So if you have a N900, you should consider joining the study so they get better data.
The next day, I picked up Martin Michlmayr nearby and we headed for FrOSCon. I was quite impressed by the Makerbot at the Tarent booth, but I still don't know what they are really doing and why they had it on display... In the afternoon, I attended a couple of talks in the PostgreSQL developer room and a talk about a big OpenVPN deployment, before ending the day with the excellent as always social event barbeque. On Sunday, I went to quite a few talks, but I thought that two of them were particularly interesting:
Michael "Monty" Widenius of MySQL gave a talk titled "Why going open source will improve your product" about starting businesses on an open source project, or how business can/should open-source their product. Besides a detailed discussion about the various forms of Open Source licenses and the Open-Core model, he proposed the idea of "Business Source" (see slide 20 of his presentation), where a startup would distribute the source code under a non-commercial (but otherwise open-source) license with the explicit guarantee that the license would be changed to a true FLOSS license at some defined point in the future, giving the company a head start to develop and nurture their project. I asked whether this has been already implemented in practise and how the community could be sure that e.g. lawyers after a hostile takeover would not just remove that part of the copyright notice, as long as a true distribution under a FLOSS license has not happened yet. Monty wasn't aware of any real-word cases, and he did not seem to be concerned about this and said the original intent would be clear in a possible court case. This was the first time I heard about this approach, I wonder how other people think about it, whether it would work in practise and be a useful thing to have?
Second, I attended a talk by Gregor Geiermann, a Ph.D. student in linguistics on "Perceptions of rudeness in Free Software communities". He conducted an online survey about the perceived rudeness of several forum thread posts on Ubuntu Forums. Survey participants were first asked a couple of generic questions about their gender, nationality etc. and were then presented with a series of posts. For each post, they were asked to rate how rude they thought it was on a scale of 1 to 5 and they also had the possibility to highlight the parts of the post they considered rude as well as add comments. He presented a neat web application for analyzing the results, which makes it possible to select different groups (he did male vs. female and Americans vs. Germans in the talk) and have their overall rudeness ratings as well as the highlighted texts visualized as different shades of blue. Comments can be easily accessed. There were quite a few interesting differences e.g. in how Germans perceived rudeness compared to Americans (RTFM comments were considered less rude by Germans for example, IIRC). In response to my question, he said he intended to release the web application as open source and this might be an interesting tool for FLOSS projects to analyze how their public communication channels are perceived by various groups. Unfortunately, I cannot find any other resources about this on the web as of today, so I should try to contact him about it at some point.

As grub is now working on the loongson 2e boxes as well (thanks to phcoder and Colin Watson), it is time to move the buildds running at my home to a data center (previously we couldn't remote manage the kernels / boot flags without VGA console, which means no data center usage). Also one of the boxes could be converted to a porter machine, so that we could get an mipsel porter box again. The machines are delivered only with 256M of ram, which is a bit too less for usage. Thanks to Zugschlus (Marc Haber) I got 1g ram for both machines going to Vienna (one buildd, one porter box), and thanks to Robert Grimm a 160g harddisk to replace the build-in 40g in the porter box (the buildd can cope with 40g fine). The additional ram modules and hard disk are visible on the following picture.
eysler.debian.org is now DSAed, online and building packages (including autosigning). I will shutdown monteverdi.ayous.org which is still at my place in the next days, reinstall the system and send it to Darmstadt as another buildd for data center hosting there. After that happens all mipsel buildds are DSAed as it should be, and are running in a data center and not via some DSL line. (In case you're looking for hardware at your place, there are a couple of loongson 2f-systems available to buy. 2f is the successor cpu of 2e. Some 2f systems get delivered with Debian installed on it, see e.g. http://www.aliexpress.com/product-gs/290632002-Yeeloong-8101-Laptop-10-inch-Debian-EU-Adaptor-version--wholesalers.html. However, for buildd usage, the 2e are fine as well, and we got them sponsored some time ago.)

Traditionally, when you start a program to view a document, for instance xpdf resume.pdf, it stays in foreground, and only returns when it finishes, when you are done with the document. Of course you can still run it in background if you want. Nowadays, several programs behave in a different way, for instance Evince. When you first start it, it stays in foreground as usual. However, if you launch it a second time while the first instance it still running, it will reuse this existing instance, asking it to open your document in a new window, and immediately return. Other examples are Firefox and LibreOffice.This behaviour is certainly more efficient in terms of memory usage and speed, but it is non-standard and causes subtle problems. When you open the attachment of an email message, or when you open a document from the Web without saving it permanently, the mailer or the browser saves the document to a temporary file, and then run a viewer with that file. As this is a temporary file, so they cannot leave it forever in /tmp; in fact they should delete it when you are done viewing it. I think the usual way to implement that is to delete it when the viewer returns: with a program that returns immediately, this means deleting it even before it has had a chance to load it. So, if you can no longer directly open a PDF attachment from Mutt, or if you are unable to view the compressed documentation of the LaTeX class Beamer with texdoc (which uncompresses it to a temporary file and calls a PDF viewer on it), you can blame your modern PDF viewer that returns before you are finished and start hacking your mailcap file. And if you are the author of such a program, please consider implementing a way to keep a lightweight process that does not return before the main process has closed the document or to disable process reuse.

The MeeGo community is frustrated with the news of the MeeGo brand being abandoned. Some are understandably angry or otherwise not happy about how Linux Foundation, Intel handled the Tizen announcement and community in general - or more like how they didn't handle it at all.Last week Openmind 2011 happened to be arranged in Tampere on the very same day as Tizen announcement came alive. It was good in the way that it lead to the fact that Nomovok's CEO Pasi Nieminen was able to initiate the "Reigniting MeeGo" session not just by talking vague things about future, but actually about the process which led to Tizen and the unfortunately brief initial PR about it. Pasi is intense on emphasizing the quality and role of Qt in Tizen as well, even though officially Tizen is all about HTML5 and apparently from Samsung's part at least EFL is provided as a native toolkit. However, the promise of Tizen compared to MeeGo is reportedly that the toolkit is not specified in compliancy documents, so HTML5 with WAC is the main/only "3rd party apps" layer whereas others can be offered case-by-case. This means that unlike before, the underlying system can be built on top of practically any distribution (theoretically) and using whatever toolkits and other techniques wanted. Obviously the "Nordic System Integrators" are probably all very keen of using Qt to produce more of Nokia N9 quality user experiences in various products.

Taking the corporate hat off, I as a community member am also puzzled. The only reason I was not completely blown by the news was that I didn't yet manage to get involved in MeeGo community on a daily basis, since I'm involved with a dozen communities already. Instead I've been more like scratching the surface with MeeGo Network Finland meetings, IRC activity, OBS usage for building a few apps for MeeGo Harmattan and MeeGo proper etc. But I can somewhat understand how people like Jarkko Moilanen from meego-fi feel. They have given a _lot_ to the MeeGo community and brand, all taken away without hearing or pre-notice.

So where to now for MeeGo community? Tizen is one obvious choice. However, for all the talks that even I started this post with, Tizen is still vaporware today, and the dislike of how community is being treated might make it easy to consider other options. Also, if Tizen's reference implementation has lesser meaning, it might also mean less to actually be "in" the Tizen community than in MeeGo. I met Jos Poortvliet at Openmind, and he invited people to openSUSE. There is a lot of common ground with MeeGo and openSUSE - strong OBS usage, RPM packaging, community side focused on KDE and therefore Qt.

I would like to now point similarly to Debian! If one is tired about corporate interests and not listening to community, there is no match for Debian's 15+ years history, purely volunteer based, trust based organization, and first of all scope. While openSUSE has traditionally focused on desktop (even though like Jos pointed out they are open to all new contributions and projects), Debian has always had the "universal" scope, ie. no boundaries besides producing free software operating system for various purposes. There are over 10 architectures maintained at the moment, including the ARM (different ports for ARMv4 and hard-float ARMv7) and x86 from MeeGo world. There are even alternative kernels to Linux, mainly the GNU/kFreeBSD port. There are multiple relevant plans and projects like the Smartphones wiki area, most noticeably Debian on Neo FreeRunner. I have run Debian on my primary mobile phone for over 2.5 years, although now in the recent months I've had dual-SIM in my Nokia N950 as well (Debian not yet running on Nokia N950 or Nokia N9 - but it can and will be done!).

What Debian may lack in both good and bad is corporate funding, if you don't count the still quite respectful contributions from Ubuntu to Debian (it's in Ubuntu's interests to contribute as much possible back to Debian, so that the delta remains small). For each and every aspect, it needs a volunteer - there are a thousand volunteer Debian Developers, and at least a double of that of people without the official DD status but who still maintain a package or two among the 25000+ packages in Debian. That means also that one my find it more lucrative to join a project that has paid people to do some of the "boring parts", more of fancy web tools, including for bug handling and build systems like the OBS (which I do love by the way). On the other hand, there is no other project in my opinion where what you do really matters as much.

To find out more about Debian from MeeGo perspective, please see the recent mailing list post Mobile UXes - From the DebConf11 BoF to the stars where I wrote most of the MeeGo (CE) part when I was asked to and known of my MeeGo involvement.

Last but not certainly least, there is the Mer project - originally "maemo reconstructed", ie. making Nokia's "not really distro" into a real distro by filling in the void places. Now it's obviously MeeGo reconstructed, and they aim to be the MeeGo they always wanted MeeGo to be! Read the post for details from Carsten Munk and other key Mer people. They share the love for Qt, and want the core to be as lean as possible. They also aim to incorporate the most community like aspect from MeeGo - MeeGo CE - as the reference vendor in Mer. They also aim to be Tizen compliant - and when Tizen comes alive, I wouldn't see why the Tizen reference implementation couldn't be used for saving resources. Maybe Nomovok and/or others could offer the Qt maintaining part.

So, it might be that Tizen itself is enough for most people's needs. The key point however in this post is not to fall in agony if one corporate based project takes big turns - it has happened before, it will happen in the future. There are always enough political and business reasons from some points of view to do Big Changes. But the wider community is out there, always, and it's bigger than you think. You should consider where you want to contribute by asking yourself why you are/were part of for example the MeeGo community. Aaron Seigo from KDE asked us all this question in the Openmind MeeGo Reignited session, and I think it's good to repeat.

I ve been a long-time user of GNU Screen, as almost every person I know that spend a lot of time in a terminal. I must admit I was happy, I really was. I was used to my workflow with 4 terminals 8 25 and a bunch of screens, with several nested-screens. I used awesome-wm before, so definitely I felt I wasn t using all the space in my laptop screen (my laptop is a ThinkPad X61, so 1024 768 in a 12 inches-screen) but that was OK, after all my workflow was, well, good. Until 2 days ago, I was told to try tmux, in fact, I was told and why are you using screen? use tmux . As a daily-user of Screen my first reaction was I m used to , I can use it as I want , and a lot of excuses for just not trying something new. But then I made a little research (on Google, of course). In about 30 minutes I ended up with a multiplexer working as I wanted and as I never got Screen to work. Finally I was using just a terminal and using all the space available in my tiny-laptop-screen and plus: saving a lot of bandwidth with my permanent SSH connection (a really big plus now that I m living with a capped link to the net with a GSM-modem). I changed the Ctrl-b shortcut to Ctrl-a (b is too far for using it with only one hand), configured so I could use nested tmux es like screen when I push Ctrl-a a, activated visual monitoring and even got to use the tmux s buffer along with my X11 buffer when I press Ctrl-a >!! It s all in my ~/.tmux.conf, use it at will. Later I will make a screencast about using tmux, so more people can get a working tmux easily. ATM, I m even more happy than I was with GNU Screen, so, any other recommendation, people?

A continuation of my intermittent series on Emdebian and prompted by a query from Paul Wise, I thought I'd cover some of the uses to which a device running Emdebian can be put. It is a bit long this one...

So, yes, there are general purpose computers which could run Emdebian but there are common features amongst general purpose computers:

1. Lots of different tasks must be supported - lots of different packages installed, each user could quite easily use a distinct set of packages from another user.


2. Multi-user support - even if any one machine is only used by one person, a general purpose computer must have support for adding another user - not just a system user, a real user with a login, shell, possibly bookmarks and browser history etc.


3. Connectivity - general purpose computers (and that includes mobile phones) must connect to a variety of other computers to be at all useful.


4. Multi-modal data input - there needs to be support for a mouse and a keyboard or touchscreen and just as likely, software support for accessibility interfaces like an on screen keyboard and switch inputs.


5. Storage for user data - the system admin can't know how much data users are going to add and if it is general purpose, it's probably going to support media players, image viewers, even office software. The data files managed and written by such software tend to be large and tend to accumulate into large collections. This means that a general purpose machine cannot anticipate the full storage requirements, so has to allow plenty of room for user data storage. If there's so much storage space available, why use an OS which is primarily designed to take up less space? Just use the full version, Debian.

Now compare with the common features of special purpose or single-purpose computers:

1. Single task only - a splash screen (without support for removal) covers all the boot information, the single task starts as a daemon and if the user quits the task, the task initiates a shutdown, not quit. Every effort is made to prevent the task from exposing the underlying OS, including auto-restarts if it does crash.


2. Single-user support - indeed, user is not the typical user. user in this case is a non-login user without a shell, created by the postinst of the task package purely to support running the task without root privileges (because if the task doesn't need privileges, it shouldn't retain privileges).


3. Single mode input - don't expect these machines to support 105 key keyboards. You'll be lucky to have 5 keys which are really just buttons. The button won't just enter 'J' into the machine, it will fire off a command to charge your bank or start a drill or shutdown a malfunctioning service. (Each button will often be multi-functional as well, depending on context). Some devices will replace the mouse with a touchscreen but many are fitted in places where touchscreens simply won't survive, where keys have to be able to withstand being submerged in water (or oil) for hours. Many machines simply won't have mouse support at all. Many will not have any typical external connectors. Indeed, units which go into dirty or wet environments certainly won't have standard external connectors of any kind. Think boats and factories. Internal battery packs with custom hot-fit replacements, buttons encased in 4mm of clear plastic, screens viewable only through protective mesh / filters.


4. Restricted connectivity - if the task doesn't need networking, no networking hardware is fitted. If the task doesn't need USB, no USB hardware is fitted. If the task doesn't need serial connections, there will probably be an internal serial connector but also a Warranty void if removed sticker. Don't expect an RS232 inside the case either, this will be a flat cable connector which you'll need to attach to a breakout board which correctly connects the cable wires to the RS232 port pins. That mapping is not standardised (why would it? only service engineers are approved to use it.)


5. Constrained user data - the task is in complete control and determines what the user can view, store and access. Storage can be expensive and a power drain, so if the task doesn't need Gb of storage, don't fit the machine with Gb of storage. Profit.

So what are these mysterious single-use computers running Emdebian? It's hard to tell because the task has no particular reason to tell the user anything about the OS. Perhaps a different way to phrase the same question would be: Why use a general purpose OS for a special purpose computer?.

• Single task does not mean single process - doing one thing can involve doing a couple of different things behind the scenes, so a multi-tasking kernel and userspace is extremely useful in providing a responsive machine without wasting money on unnecessarily powerful hardware. Multithreading support is also valuable.


• Software reuse makes good business sense - why write a single-usage operating system when the device needs a multi-tasking OS anyway and suitable ones already exist?


• Minimal software - no point packaging Xorg or GNOME when the user interface is a tiny QVGA framebuffer and the device only has 128Mb of RAM. By all means get the core system (cairo, pango, Qt, gettext, dpkg) from Emdebian but the actual code which performs the task of the device will generally be bespoke and, yes, that generally means proprietary, closed source, utterly non-free. This is where GNU/Linux wins over iOS and Windows - the graphical software is trivially separated from the core software and tested as such. If your device has only monochrome text output (and plenty of those still exist), what possible reason is there to use a graphical toolkit? Use the multi-tasking support of the OS to handle inter-process communication and understand the licensing - job done. Again, these devices have deliberately limited storage capacity, the smaller the OS, the lower the cost of the device, within certain constraints.


• Debugging compatibility - this is the big one. It's hard debugging the task software on device with the differences in architecture and connectivity. Why not use the same packages on the same OS on the development machines as on the devices? All the power and all the screen resources to properly investigate bugs, simple way to provide an emulator for translation support and other activities (like user interface training). There will always be bugs which have to be debugged on device - timing issues, hardware issues - but why not make at least a proportion of the likely bugs easier to fix by using binary compatible libraries on both the big desktop machines and the small devices?

This is why Emdebian Grip is popular for these machines - because Emdebian Grip is binary compatible with the equivalent Debian suite and when a bug appears in the high level user interface, it is much easier to debug that on the desktop than on device. Debian stable is a fantastic OS for commercial development - the stability of the OS makes detection of interface bugs much easier. It is rare that developers have to investigate whether bugs in their UI come down to a bug in the underlying OS. That saves time.

The machines themselves? I'm sure others can come up with other examples but these are some which come to my mind (no particular order):

• Control interfaces - solar power controllers, robot controllers - anywhere that a low power board can monitor responses from other machines and do something intelligent about reported errors. Temperature limit responses, pressure limit responses, encoding all of those in assembly or VHDL is hard, so use a system which provides for more creative result handling.


• Electric car charging points - very simple user interface, single task which doesn't complicate things by expecting the user to tap on something to start the main application - just process the card details, charge the customer and provide power.


• Similarlyelectronic point of sale devices - EPOS. The staff on the shop floor don't want to be waiting for an X server to start up and then using a menu. The device starts up and runs one task (checking barcodes, printing labels, counting stock), if the task is shutdown, the machine shuts down.


• Communication aids - my area, a machine which presents a simple UI on non-standard framebuffer hardware, lots of audio output but little other connectivity. Resilience is very important, so solid state is vital but why provide more solid state storage than the device can actually needs?

Anywhere where the device simply needs to DoTheRightThing in a variety of unpredictable circumstances, you're going to need a general purpose OS to gather the data and produce the right result. Anywhere where the device has only a single task, you're going to want to not provide access to the rest of the system. There's no point providing "general purpose access" when the device doesn't have "general purpose hardware". There's no point fitting general purpose hardware if the device does not use it. That is a waste of money, a waste of resources and causes unnecessary delays if one component or other becomes unobtainable or changes behaviour. Limit your exposure to hardware bugs and get the product out to the user more quickly and more reliably, everyone is happy. Fit non-standard hardware and you can fit custom hardware which better matches the profile of the device usage - why use standard speakers capable of CD quality across a vast frequency range when the device only makes very very loud alert noises which can be done at lower sample rates and with narrow range, high amplitude, speakers.

There are all sorts of estimates for the number of computers now in the average Western home. It's worth noting that the vast majority of those computers are not PCs or laptops or even game consoles and routers - all instances of general purpose machines with general purpose access / connectivity. A lot of the computers in your home are embedded in machines like white goods, media controllers (set top boxes and the like). Some of these will still use low level firmware written entirely in assembly or COBOL. More complex ones already have a general purpose OS inside yet constrain that OS with a simple, user-friendly interface which is tailored to the hardware actually fitted. It might be cool to connect to your set top box over Bluetooth but really, is that actually going to sell more units? Only if there are other high-end services written for the device which the average customer won't want to see pushing up the price of the base unit. Some machines (particularly in automotive uses) will need to be using Real Time operating systems and Safety Critical operating systems but even some of those are based on general purpose systems - just old versions which have been thoroughly tested.

Finally, consider that for a lot of these devices, the customer is not you. The customer is another business who then put the device into something else which is installed into a factory production line which produces something which goes into creating a consumer product you find in Tesco or on Amazon or bundled in with another service like your TV/broadband contract. There is absolutely no reason for these devices to provide general purpose computing to you, even if the device itself is part of something else which can provide such services.

The last month I was supposed to work on a OpenStack related project, but for administrative reasons it has been delayed and I've tried to do small tasks to be able to finish them quickly and start the work on the main project when the issues get solved. As the delay has been longer than expected last Wednesday I've realized than on the last weeks I did a lot of small system administration tasks:

• With a co-worker I started to work on a GNU/Linux version of our firewall based on Shorewall to handle the rules and conntrackd and keepalived to make it highly available (I had to stop my work on the Debian GNU/kFreeBSD based firewall a long time ago, and this summer the old firewalls' hardware started to fail, so a migration from Linux to Linux makes sense now, as it will be faster and a future migration will be simpler, as we will have a cleaner set of rules and better documentation),
• I installed and configured an instance of a web based File Exchange server (F*EX),
• I installed and configured an instance of a pastebin clone,
• I installed and configured an instance of ProFTPD that works only as a SFTP server using virtual users (without shell access),
• I also installed an instance of a web based event management system called indico that is being used to manage a conference and probably will be used for other events in the future,
• I installed and patched some plugins in our Trac servers,
• I tried a groupware system called SOGo that we will probable deploy in a week or two,
• And updated and fixed configurations of some other services,

With all the changes I did I noticed that I had to do something with our Intranet server; it is just a reverse proxy for a lot of different web services and its main page was one static HTML page with links to them, nothing else. In the long term maybe we will replace it with something based on Drupal or Lifeay, but for now I just wanted something to be able to organize the links and provide some information about the services for the new users without having to write HTML (I really like Agile Documentation Tools that let me focus on the content and forget about the markup), and started to look at some of them. My first idea was to use ikiwiki, as it has all the features I was looking for: I can use Markdown or reStructuredText to write the contents, the source pages are easily handled on a Version Control System, it supports the use of templates for the HTML, etc., but it seemed to me that using ikiwiki was like killing flies with a cannon (that's a Spanish say, I guess it's easy to understand it in English, no?) and I decided to review other tools to build static web sites. To make a long story short, I selected some tools that met my requirements and looked nice on their demo sites; after my first review I thought that Hyde was going to be my bet, as it uses technologies I'm already familiar with, but after trying it I saw that I was going to have a problem with documentation (the current Hyde version lacks it) and it was going to be more complicated that using ikiwiki. Before giving up I decided to review simpler tools, just in case, and after looking some of them I ended up using poole, a simple python script (the source is just one file and it only requires python-markdown to work). Before moving to the content I tried to adapt a couple of free themes to be used by the tool, but I didn't liked the result, so I went back to the plain style provided by the tool and added a logo and a background. With that simple look and feel I started to work with the content, splitting it into eight markdown files and a python macro to include a file that has all the links used on the site. While trying to make the main page look good I noticed how little I know about CSS, but using search engines I was able to build a two column block into the main page and publish the contents and with the help of some CSS enabled co-workers I changed the look and feel of the site in about 30 minutes. In summary, if you want a really simple website, you know a little bit of python and don't want to spend much time learning how to use a website generator then Poole is a good option. If you want something more complex I still think that ikiwiki is a good option, but YMMV.

3D programming is great, but when something goes wrong, it can be quite hard to debug! I was trying to launch two different OpenGL programs one after another, the first for the normal rendering, the second for a post-processing effect (motion blur, etc.). It seemed the first program's output was cleared as soon as the second one was activated. I eventually found a behavior that is well summed up at http://www.gamedev.net/topic/541571-glsl-changing-between-shaders/page__view__findpost__p__4495411. You don't write:

glUseProgram( shader-A );
modelA->Draw();
glUseProgram( shader-B );
modelB->Draw();
glutSwapBuffers();

You write:

modelA->Draw();
glUseProgram( shader-B );
modelB->Draw();
glUseProgram( shader-A );
glutSwapBuffers();

and since the code is called multiple times (at each frame), this will cycle OK. (There seem to be an GL_INVALID_OPERATION error the very first time I call glUseProgram, too.) Anybody has an idea on why we have this behavior?

Just a quick note to remind everyone that the Rcpp class in San Francisco, which I am holding together with Revolution Analytics, will take place a week from today. We are happy to report that the number of registrations has met our initial targets. But as a number of open slots remain, we have decided to offer a few places at discounts of 25% for academics (with code acad1) and 50% for students (with code student). Course details are at the Revolution course page, registration is at the Eventbrite page. And just for completeness, here is what I wrote in the previous announcement:

The format will follow the workshop Romain and I gave during the tutorial day preceding this year's R/Finance conference. The style will once again be hands-on, with copious concrete examples and solid coverage of most aspects of Rcpp and related packages such as RInside, RcppArmadillo and others. The eight-hour schedule contains about six hours of instruction, split into four sessions of around ninety minutes. This leaves ample time for both lunch and coffee breaks, and for informal discussions and Q+A. The one-day class will be offered in San Franciso on Saturday, October 8, 2011. Please see the official course page for more details, concrete location info and maps as well as registration details.

Feel free to contact me at the usual email address with questions. Or with suggestions for the after-party in San Francisco :)