Rapha l Hertzog recently announced a new dpkg-buildflags interface in dpkg that at long last gives the distribution, the package maintainers, and users the control they want over the build flags used when building packages. The announcement mail gives all the gory details about how to invoke dpkg-buildflags in your build to be compliant; but the nice thing is, if you're using dh(1) with debian/compat=9, debhelper does it for you automatically so long as you're using a build system that it knows how to pass compiler flags to. So for the first time, /usr/share/doc/debhelper/examples/rules.tiny can now be used as-is to provide a policy-compliant package by default (setting -g -O2 or -g -O0 for your build regardless of how debian/rules is invoked). Of course, none of my packages actually work that way; among other things I have a habit of liberally sprinkling DEB_MAINT_CFLAGS_APPEND := -Wall in my rules, and sometimes DEB_LDFLAGS_MAINT_APPEND := -Wl,-z,defs and DEB_CFLAGS_MAINT_APPEND := $(shell getconf LFS_CFLAGS) as well. And my upstreams' build systems rarely work 100% out of the box with dhauto* without one override or another somewhere. So in practice, the shortest debian/rules file in any of my packages seems to be 13 lines currently. But that's 13 lines of almost 100% signal, unlike the bad old days of cut'n'pasted dh_* command lists. The biggest benefit, though, isn't in making it shorter to write a rules file with the old, standard build options. The biggest benefit is that dpkg-buildflags now also outputs build-hardening compiler and linker flags by default on Debian. Specifically, using the new interface lets you pick up all of these hardening flags for free:

-fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -Wl,-z,relro

It also lets you get -fPIE and -Wl,-z,now by adding this one line to your debian/rules (assuming you're using dh(1) and compat 9):

export DEB_BUILD_MAINT_OPTIONS := hardening=+pie,+bindnow

Converting all my packages to use dh(1) has always been a long-term goal, but some packages are easier to convert than others. This was the tipping point for me, though. Even though debhelper compat level 9 isn't yet frozen, meaning there might still be other behavior changes to it that will make more work for me between now and release, over the past couple of weekends I've been systematically converting all my packages to use it with dh. In particular, pam and samba have been rebuilt to use the default hardening flags, and openldap uses these flags plus PIE support. (Samba already builds with PIE by default courtesy of upstream.) You can't really make samba and openldap out on the graph, but they're there (with their rules files reduced by 50% or more). I cannot overstate the significance of proactive hardening. There have been a number of vulnerabilities over the past few years that have been thwarted on Ubuntu because Ubuntu is using -fstack-protector by default. Debian has a great security team that responds quickly to these issues as soon as they're revealed, but we don't always get to find out about them before they're already being exploited in the wild. In this respect, Debian has lagged behind other distros. With dpkg-buildflags, we now have the tools to correct this. It's just a matter of getting packages to use the new interfaces. If you're a maintainer of a security sensitive package (such as a network-facing daemon or a setuid application), please enable dpkg-buildflags in your package for wheezy! (Preferably with PIE as well.) And if you don't maintain security sensitive packages, you can still help out with the hardening release goal.

I am currently running a crowdfunding campaign whose ultimate goal is to liberate the English translation of a French book that I have written. This book will be named The Debian Administrator s Handbook because it has primarily been written for Debian. Creating a new Ubuntu book based on The Debian Administrator s Handbook But since Ubuntu is based on Debian, a large part of its content applies equally well to Ubuntu. While discussing with Mark Shuttleworth, he suggested me to reuse those parts and to create a new book dedicated to Ubuntu. It would also cover the latest cloud technologies that Ubuntu has been delivering (since this is a topic that the current book does not cover). This is something that I have been envisioning for a while and something that I would be ready to try if we manage to complete the liberation of the current book. This project would then bring a truly free book to the Ubuntu ecosystem. Why? The official Ubuntu books are not really free There s a policy in place that ensures that official Ubuntu books use a free software/culture license and they are effectively available under the terms of a Creative Commons Share Alike license. But try to create a derivative book you won t find the sources (LaTeX or DocBook usually with most big books). You can only find a few PDF copies if you google for it. But this is really not the preferred form of modification for such a book. Those books are also not packaged. Ubuntu much like Debian deserves to have a good book embodying the values of free software that can be shipped together with its product. When I speak of liberation of the book, I really mean it in the way that free software hackers are used to: a public Git repository containing the DocBook sources, the pictures and the .dia files for the various schemas. Help Ubuntu by spreading the word I understand that at this point this proposed Ubuntu book is really hypothetical ( vaporware one could say) but we need to go step by step to make it a reality. And the first step is to ensure that we manage to liberate the Debian Administrator s Handbook. For this I am seeking the support of the Ubuntu community to promote the current fundraising campaign. If the perspective of the Ubuntu book is not enough to convince you, you ll be glad to learn that I also commit to give back to Ubuntu 15% of the money raised via the link below (once VAT has been subtracted). Click here to go to the crowdfunding campaign page and pledge a few euros. Then share this article (or the link http://debian-handbook.info/go/ulule-ubuntu/) and convince others to participate. At this point, the liberation target is entirely reachable with your help and the help of the community: the remaining 18 K needed in the liberation fund represent 720 persons giving 25 EUR each or 1800 persons giving 10 EUR each. Thank you very much for your support and your help in this project!

One comment Liked this article? Click here. My blog is Flattr-enabled.

With this, my NIN vinyl album collection is now complete. I don't have a comprehensive NIN vinyl collection by any account: I don't have all the singles nor the remix albums, rarities, etc., but I do have all the albums. This is the newly-remastered version of "Pretty Hate Machine", which sites proudly next to an original. I'm still waiting for the much-anticipated remaster of The Fragile, which turned 12 last month. It appears to have been sidelined by movie soundtrack work.

I really don t know what to make of this Cisco-made video What a Hackable World. It is sort of interesting that Anonymous and Lulzsec made it in but the video is not all that clever. And what up with the nerd-o-matic sys admin (most sys admin I know are more hoodie, less nerdie). But can anyone explain what/who the main featured dude is supposed to be? Hacker detective/destroyer? Cisco customer?

First, is that image really the best LOLcat referencing Brazil that exists out on the Internets? I sure hope not as it is sort of lame but it was the only one I could find (in the 30 seconds I looked, which I admit, is not all that much time). But all of this lolcat in Brazil stuff is really to say, I am heading to S o Paulo tomorrow to give a talk on Anon on Thursday at this conference. If you are an Anon in the city and can make it, please do. If you are not Anon and are still interested, come along as well ;=) If there are Debian developers as well in the city, it would be great to meet at some point. I am there from Tuesday morning to Friday night.

DIASPORA* as a blog I'm testing out DIASPORA* , the open-source social network, as a blogging platform. Apologies to those who see bad output on Planet Debian - I'm working on the issues as you read. It is possible to add comments to a post : it just requires a Diaspora account (on the post page, log in and create an account. This is on my home "pod", running on an old server at home, so don't be suprised at the speed). Failing that, my Debian email is available, mckinstry at debian.org.

Terah and I went to the Greek island Rhodes recently. This is the second in a series about it. I am one to enjoy history. There is something deeply, well, connecting, about standing in an old place. There is a timeless quality to it a feeling of being connected to so many people of the past, and yet still being connected to change, visible in things such as weathering of stones. To gaze at pottery that s 300 years old, walk past 700-year-old walls, or pass through what remains of the grand portico of an ancient temple to Athena stirs a feeling I can barely explain, of timelessness. Although Rhodes doesn t have the famous Greek sites such as the Parthenon or Delphi, I can t help but wonder why the Rhodes sites aren t better known. They were incredible and it is hard to condense all that we saw into a short blog post. I have to start with the medieval Rhodes Old Town. We got off the bus a few blocks from it one bright morning, and our first task was to find a gate across the moat. Oh yes, A GATE ACROSS THE MOAT. It s a dry moat, and that bridge off in the distance is the gate we were headed to. Outside of the outer wall is a nice quiet walking area. The moat and walls completely surround Old Town and, for the most part, date back about 500 years. The round stones you see on that picture, we were told, were likely surplus from catapults and other projectile weapons. Cross one line of walls and you come to another, with original canons still present. The Knights Hospitaller of St. John, which held Rhodes for a few centuries until the Ottomans captured it, sure knew how to build to impress. The gate we happened to use was Amboise, the Grand Master s Gate. Right there is the stunningly rebuilt landmark Palace of the Grand Master. It is absolutely impossible for any photograph to begin to do this building justice. Between its imported Greek and Roman floors, to the grand nature of everything in it, and the archaeological museum in one corner, it was a fitting start to a visit to Old Town. Here s one of the main staircases. Just near the Palace is quiet courtyard with an old door. Pass through that door and suddenly you re in the midst of the busy Old Town. And among the landmarks in Old Town, the most prominent is Ippoton, the Avenue of the Knights. Along this avenue are the buildings built by the various nationalities of knights, many of which are historical sites in their own. Taken together, it is quite clear why Rhodes is said to be one of the world s best-preserved medieval cities. Down at the other end of Ippoton is the Knights Hospital, which is now part of the archaeological museum. Step off the Avenue a few blocks and you get to some quieter narrow streets just as old, in many cases. On Sunday morning, we were able to visit Mount Filerimos. In contrast to the busy Rhodes, Filerimos had an air of quiet and still to it. It was the site of a monastery, two historic churches, and a landmark Italian cross on the mountaintop. We arrived, and begin our visit with a walk up the quiet stone path. When we got to the top, we walked past this peaceful church. As we walked past the outside, we heard the beautiful music of chant from indoors. We got to step in and listen to mass for a few minutes. In typical fashion, directly in front of the church are two much older sites: one, the ruins of a temple to Athena, and the other a 4th-century Christian bapistery. Rhodes is a popular tourist destination, and of course we saw plenty of popular sites (such as the grandmaster s palace). Filerimos had a few tourists too, but not as many. I frequently like to operate on the plan of going wherever all the tourists aren t. And so, on Filerimos, that meant seeing what was behind the monastery. It started with this peaceful tree-lined path. And the deserted, but intentionally open, gate led to the remains of a Byzantine fortress, which had been a staging area for both the Knights and the Ottomans before their campaigns to capture Rhodes. It also provided incredible views of the surrounding countryside. The first historic site we had visited on our trip was the Acropolis of Lindos, parts of which are 2300 years old. Here s a view of the mountain from the rooftop of the Kalypso, our favorite restaurant in Lindos. The columns of the temple to Athena Lindia are visible, and of course so are the walls. The road up to the acropolis is accessible only on foot or by donkey. It is apparently the only road that has ever been used to get to the acropolis. Here is the partially-restored grand portico to the temple. There s an old Christian church (4th century, if memory serves) at the Acropolis too. The Acropolis makes some pretty good use of natural defenses too. Here s a view from one level of it. There s a manmade wall up there at the very top. And, of course, the beautiful Aegean always in the background. There are lots of cats on Rhodes. Here is a kitten napping at the top of the Lindos Acropolis: Lindos itself is a beautiful town. Here s one of the quieter streets: Notice the pebble steps leading into the houses those intricate pieces of artwork are all over. This post won t be complete without the story of our visit to the Acropolis of Rhodes. We walked there from Old Town. At the Acropolis, there are the remains of a temple to Apollo, an ancient theater, and an ancient stadium where qualifying matches for the Olympics were held. As we got closer to the area, we were repeatedly passed by people dressed in uniforms of various types. And as we got there, we joined a stream of people entering the area. The ancient stadium had apparently thousands of people in it, country names were being read off over the loudspeakers, policemen wielding machine guns were standing by, and we had absolutely no idea what was going on. At this point, you can appreciate the difference between Terah and me. Terah thought that we have no idea what is happening, she was tired from the walk, and so thought we should just leave. I thought that we have no idea what is happening, which is a great reason to stay. So Terah opted to sit and read a bit under some trees while I explored. Here s a view of the stadium as it was emptying out, seen from the theater: I explored the temple and theater, and eventually we were ready to head back. We knew there was a bus back to the New Market (from where we could get a bus back to our hotel), but didn t know where the bus stop was. The obvious place to ask were the policemen, which I thought I would do. Terah thought she would just stay sitting under the trees, on the grounds that the policemen nearest us were all carrying machine guns and perhaps wouldn t like to be disturbed. This led to my cryptic tweet:

Only ONE of us is the kind of person that goes up to guys with machine guns to ask what s happening. Me to Terah today

They told me that it was the preparations for the opening ceremony for a global shooting contest, and also gave me directions to the bus stop.

Last weekend Dennis M. Ritchie, creator of the C Programming Language, the UNIX Operating System, The Plan 9 Operating System and many other key contributions to computing, passed away due an extended illness. I learned about it yesterday trough Rob Pike, former dmr s colleague and friend. I m writing this post to honor him and his legacy. While I ve seen people mourn his departure, I ve also noticed most technical people don t really understand why his work is important for today s computing, so I m here to give a refresh. Since I m good with visual tools (mindmaps) I ve made one to describe the extension of dmr s contribution to the world. You can get a glimpse of how this is important. The main topics are their key contributions, the sub-topics are the current technologies that were built upon dmr s work. Click on the image to see a larger resolution version.

The C Programming Language

Best said here Let not make the final statement and not a single newspaper cares " into a reality. PS: Do check out Against Nostalgia, a great jab at Jobs.

Greetings real ale fans, This most exciting time of year is almost upon us again. Yes, it's the Nottingham Robin Hood Beer Festival! This year I have followed in the footsteps of Alex and Karen and decided to trawl through the list of over 900 beers so that I can best sample the delights on offer. My selection is here. I'm attending on both the Thursday and Saturday, so there's plenty of time for me to sample a reasonable number of different ales. Beer aficionados, I'd welcome suggestions for alternative beers to try, if you'd recommend any of them. I promise to update the list after. I might even be tempted by the ciders, perries and wines. Generally I tend to prefer paler ales, but I also quite like a mild given half the chance. See you on the other side! (Image by Ant & Carrie Coleman, cc-by-nd 2.0)

Poor Sarah. Cat dramas always seem to befall us when she's in Australia. It looks as if Smudge has managed to escape the house. Being at work all day, it took me 24 hours to notice. Our friend Helen came over on Sunday night. Smudge was definitely around then, because she was climbing all over Liam. Helen used the downstairs bathroom before she went home, and I remember her having to shoo Smudge out because she was in there. I'm pretty sure she went home after that. If my memory serves, she saw herself out and I was in the kitchen cleaning up, so my current theory is that she escaped when Helen left. On Monday night, I had a lengthy video call with Sarah, who was at my parent's place setting up their new Mac Mini. I didn't realise until after that, when I was about to go to bed, that I hadn't been harassed by Smudge all evening. So I searched the house that night, and couldn't find her. Liam had already gone to bed for the night, so I hoped she was stuck in his room, and went to bed too. This morning, Liam and I both searched the house, and the immediate surrounds, without any luck. I called the Palo Alto Animal Services (I got the dispatch because the shelter hadn't opened yet), emailed the local neighbourhood association, and went to work. I called Palo Alto Animal Services again later in the day to report her missing to the shelter itself, and Betty-Ann there recognised my accent and name and asked me if I was Sarah's husband (Sarah used to volunteer there a lot, and we adopted Smudge from there), so they're now keeping a look out. Smudge is micro chipped, but wasn't wearing a collar. She's allowed in the back yard, because it's totally enclosed and can't escape, so she shouldn't be totally freaked out by being outdoors. Whenever she does escape out the front door, she usually makes a beeline to the right, which is mostly enclosed, but she can certainly get out of the complex. Unfortunately she often doesn't come when called, so for all I know she's in some obscure location 10 metres from the front door and I can't see her. My main concerns are that we're only two blocks from Highway 101, so if she does stray too far, she could get into serious trouble, and I'm also a bit worried about raccoons giving her grief. I spent some time tonight putting up posters in the immediate area and poking around bushes. I've left the two humane cat traps that we have out by the front door with some cat food in them, in the hope that she's roaming around and will walk into one of them. Oh and the other big concern I have is that Halloween is coming up. Shelters tend not to adopt out black cats around Halloween because bad things can happen to them. So now we wait. I'm really hoping she turns up safe and sound, soon.

I have several backup servers that run the excellent rsnapshot software, which uses Secure Shell (SSH) for remote access. The SSH private key of the backup server can be a weak link in the overall security. To see how it can be a problem, consider if someone breaks into your backup server and manages to copy your SSH private key, they will now have the ability to login to all machines that you take backups off (and that should be all of your machines, right?). The traditional way to mitigate SSH private key theft is by password protecting the private key. This works poorly in an unattended server environment because either the decryption password needs to be stored in disk (where the attacker can read it) or the decrypted private key has to be available in decrypted form in memory (where attacker can read it). A better way to deal with the problem is to move the SSH private key to a smartcard. The idea is that the private key cannot be copied by an attacker who roots your backup server. (Careful readers may have spotted a flaw here, and I need to explain one weakness with my solution: an attacker will still be able to login to all your systems by going through your backup server, however it will require an open inbound network connection to your backup server and the attacker will never know what your private key is. What this does is to allow you to more easily do damage control by removing the smartcard from the backup server.) In this writeup, I ll explain how to accomplish all this on a Debian/Ubuntu-system using a OpenPGP smartcard, a Gemalto USB Shell Token v2 with gpg-agent/scdaemon from GnuPG together with OpenSSH.


First we need to install some packages. The goal is to configure OpenSSH to talk to the gpg-agent which will start and talk to scdaemon which in turn talks to pcscd which talks to the smart card reader. For some strange reason, the scdaemon binary is shipped with GnuPG s S/MIME interface in the gpgsm package.

# apt-get install pcscd gnupg-agent gpgsm

The above command should install and start pcscd and if all works well, you should be able to check the status of the smartcard using GnuPG.

# gpg --card-status

You need to initialize the smartcard and generate a private key on it, again using GnuPG. If you trust GnuPG more than the smartcard to generate a good private key, you may generate the private key using GnuPG and then move it onto the smartcard (hint: use the keytocard command). Make sure you don t leave a copy of the private key on the same machine!

# gpg --card-edit
gpg: detected reader  Gemalto GemPC Key 00 00'
...
gpg/card> admin
Admin commands are allowed
gpg/card> name
Cardholder's surname:
Cardholder's given name: host.example.org
gpg: 3 Admin PIN attempts remaining before card is permanently locked
Please enter the Admin PIN
gpg: gpg-agent is not available in this session
gpg/card> lang
Language preferences: en
gpg/card> generate
Make off-card backup of encryption key? (Y/n) n
Please enter the PIN
What keysize do you want for the Signature key? (2048)
What keysize do you want for the Encryption key? (2048)
What keysize do you want for the Authentication key? (2048)
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: host.example.org
Email address:
Comment:
You selected this USER-ID:
    "host.example.org"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (33 seconds)
gpg: signatures created so far: 0
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (18 seconds)
gpg: signatures created so far: 1
gpg: signatures created so far: 2
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (23 seconds)
gpg: signatures created so far: 3
gpg: signatures created so far: 4
gpg: key 12345678 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/12345678 2011-09-19
      Key fingerprint = 1234 5678 1234 5678 1234  5678 1234 5678 1234 5678
uid                  host.example.org
sub   2048R/23456789 2011-09-19
sub   2048R/34567890 2011-09-19
gpg/card> quit
#

Now for the interesting part. OpenSSH talks to an agent for private key handling, and GnuPG s gpg-agent supports this protocol when the --enable-ssh-support parameter is given. During startup, gpg-agent will print some environment variables that needs to be set when ssh is run. Normally gpg-agent is invoked by the Xsession.d login scripts, so that the environment variables are inherited by all your processes. However, for an unattended machine without any normal login process, we need to write a script to start gpg-agent. First do these manual steps, to confirm that everything works.

# gpg-agent --daemon --enable-ssh-support > /var/run/gpg-agent-info.env
# . /var/run/gpg-agent-info.env
# ssh-add -L
ssh-rsa AAAAB3N... cardno:000500000BD8
#

If the final step printed a SSH public id, the (sometimes) tricky part in getting the hardware to work is (hopefully) complete. What remains is to script things so that gpg-agent is started on boot and to make sure that your backup scripts has the proper environment variables before launching whatever processes will launch ssh. Further, since we will be running unattended, we need a mechanism to unlock the smartcard using a PIN interactively once on each boot of the machine. I prefer manually entering the PIN on every boot over having the PIN stored in a file on the disk. I will use the /etc/rc.local mechanism to start gpg-agent, like this:

# cat> /etc/rc.local
#!/bin/sh -e
exec gpg-agent --daemon --enable-ssh-support \
    --pinentry-program /usr/local/sbin/pinentry-unattended \
    --write-env-file /var/run/gpg-agent-info.env
^D

The astute reader will now ask what /usr/local/sbin/pinentry-unattended is and why it is needed. Now here is the situation. scdaemon will normally query the user for a PIN using a tool called pinentry which reads and write to the user s TTY directly. This won t work in unattended mode, so we want the scdaemon to signal failure here unless we are actually unlocking the smartcard manually. Here is the entire script:

#!/bin/sh
# /usr/local/sbin/pinentry-unattended -- by Simon Josefsson
if test x"$PINENTRY_USER_DATA" = xinteractive; then
    exec pinentry "$@"
fi
exit 1

What remains is a script to unlock the smartcard by providing the PIN. This is typically invoked manually if the server has restarted for some reason. Don t worry, any ssh sessions invoked by cron until you have managed to unlock the smartcard will fail with an authentication error it won t hang waiting for a PIN to be entered.

#!/bin/sh
# /usr/local/sbin/unlock-smartcard -- by Simon Josefsson.
. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID
gpg-connect-agent 'scd killscd' /bye > /dev/null
while ! gpg-connect-agent 'scd serialno' /bye   grep -q SERIALNO; do
    sleep 1
done
PINENTRY_USER_DATA=interactive
export PINENTRY_USER_DATA
checkpin

And the script checkpin is as follows:

#!/bin/sh
# /usr/local/sbin/checkpin -- by Simon Josefsson.
id= gpg-connect-agent 'scd serialno' /bye   head -1   cut -d\  -f3 
gpg-connect-agent "scd checkpin $id" /bye   grep -q OK

At this point, you should have everything configured and installed. Don t forget to chmod +x the scripts. The typical use-pattern is as follows. After the machine has been started, gpg-agent is running but the smartcard is not unlocked with the PIN. You need to manually login to the machine and run unlock-smartcard and enter the PIN. In the script that runs the backup jobs, invoked via cron, make sure that the first line of the scripts reads (assuming Bourne shell script syntax):

. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID

To avoid needlessly attempting ssh connections if the smartcard is not unlocked, your backup script can also call the checkpin code and abort if it doesn t return true.

checkpin   exit 1

Some final words about debugging. A basic command to run to check that the GnuPG side is working is gpg --card-status, it should print some information about the smartcard if successful. To check that the SSH agent part is working, use ssh-add -L. If you get error messages, try killing the scdaemon process by running killall -9 scdaemon and let gpg-agent respawn a new scdaemon process. That s it! If you like my writeup, please flattr it.

Wouter described how to get Akonadi (the back-end for KDE PIM) to use PostgreSQL [1]. I don t agree with his MySQL is a toy sentiment. But inspired by his post I decided to convert some of my systems to use a MySQL instance running on a server instead of one instance for each user. In the default configuration you have 140M of disk space and 200M of RAM used by each user for a private MySQL installation which has about 24K of data (at least at the moment on systems I run, maybe more in future). Here s some pseudo shell script to dump the database and get a new config: mysqldump socket=$HOME/.local/share/akonadi/socket-$HOSTNAME/mysql.socket akonadi > dump.sql
akonadictl stop
rm -rf .config/akonadi
rm -rf .local/share/akonadi
mkdir .config/akonadi
cat > .config/akonadi/akonadiserverrc <<EOF
[%General]
Driver=QMYSQL
SizeThreshold=4096
ExternalPayload=false

[QMYSQL]
Name=$USER_akonadi
Host=IP_OR_HOST
User=$USER
Password=$PASS
StartServer=false
Options=
ServerPath=/usr/sbin/mysqld

[Debug]
Tracer=null
EOF Then with DBA privs you need to run the following in the mysql client: create database $USER_akonadi;
GRANT ALL PRIVILEGES ON $USER_akonadi.* to $USER @'% IDENTIFIED BY $PASS ; Then run the following to import the SQL data: mysql $USER_akonadi < dump.sql Ideally that would be it, but on my test installation (Debian/Squeeze MySQL server and Debian/Unstable KDE workstations) I needed to run the following SQL commands to deal with some sort of case problem.
rename table schemaversiontable to SchemaVersionTable;
rename table resourcetable to ResourceTable;
rename table collectionattributetable to CollectionAttributeTable;
rename table collectionmimetyperelation to CollectionMimeTypeRelation;
rename table collectionpimitemrelation to CollectionPimItemRelation;
rename table collectiontable to CollectionTable;
rename table flagtable to FlagTable;
rename table mimetypetable to MimeTypeTable;
rename table parttable to PartTable;
rename table pimitemflagrelation to PimItemFlagRelation;
rename table pimitemtable to PimItemTable; I am not using any PIM features other than the AddressBook (which hasn t been working well for a while), so I m not sure that this is working correctly. But I would prefer that something I don t use and is probably broken take up disk space and RAM on a server instead of a workstation anyway.

• [1] http://grep.be/blog/en/computer/debian/no_more_mysql?show_comments=yes

A month ago, I published an article on the compared performances of stunnel, nginx and stud as SSL terminators. The conclusion was to use stud on a 64-bit system, with session caching and AES. stunnel was unable to scale properly and nginx exhibited important latency issues. I got constructive comments on many aspects. Therefore, here is the second round. The protagonists are the same but both the conditions and the conclusions have changed a bit.

Benchmarks The hardware platform remains unchanged:

• HP DL 380 G7
• 6 GB RAM
• 2 L5630 @ 2.13GHz (8 cores), hyperthreading disabled
• 2 Intel 82576 NIC
• Spirent Avalanche 2900 appliance

We focus on the following environment:

• Ubuntu Lucid, featuring a patched version of OpenSSL 0.9.8k with AES-NI builtin and enabled by default
• 2.6.39 kernel (250 HZ)
• 64-bit
• 1 core for HAProxy, 2 cores for network cards¹, 4 cores for SSL, 1 core for the system
• system limit of 100,000 file descriptors per process

On the SSL side, the main components are:

• 2048-bit certificates
• AES128-SHA cipher suite
• 8 HTTP/1.0 requests per client
• HTTP response payload of 1024 bytes

If you are interested in different conditions (32-bit, number of cores, 1024-bit certificates, proportion of session reuses), have a look at the previous article whose results on the matter are still valid. I only consider 2048-bit certificates because most certificates issued today are 2048-bit. In SP800-57, NIST advises that 1024-bit RSA keys will no longer be viable after 2010 and advises moving to 2048-bit RSA keys:

If information is initially signed in 2009 and needs to remain secure for a maximum of ten years (i.e., from 2009 to 2019), a 1024-bit RSA key would not provide sufficient protection between 2011 and 2019 and, therefore, it is not recommended that 1024-bit RSA be used in this case.

And the protagonists are:

• stud, @e207dbc5a3, with session cache enabled for 20,000 sessions, listen backlog of 1000 connections,
• stunnel 4.44, configured with --disable-libwrap --with-threads=pthread (configuration),
• nginx 1.1.4, configured with --with-http_ssl_module (configuration).

UPDATED: Configuration of nginx has been updated to switch to off both multi_accept and accept_mutex thanks to the feedback of Maxim Dounin on latency issues that were present in the previous article (and still present in the early version of this article). HAProxy (configuration) is used for stunnel and stud to add load-balancing. For each test, we want to maximize the number of HTTP transactions per second (TPS) while minimizing latency and unsucessful transactions. The maximum number of allowed failed transactions is set at 1 transaction out of 1000. The maximum allowed average latency is 100 ms².

Control case Below is the first comparison between stud, nginx and stunnel on plain Ubuntu Lucid. Like in previous benchs, stunnel is not able to scale, with a maximum of 400 TPS. However, early latency issues of nginx has been solved, thanks to the change in configuration. stud and nginx gets similar performance but, as you can see on the bottom plot (whose scale is logarithmic) stud starts with an initial latency of 40 ms. This is pretty annoying. I have tried some micro optimization to solve this issue but without luck. If latency matters for you, stick with stunnel or nginx (or find where the problem lies in stud). UPDATED: After reporting this issue, a fix has been commited. The culprit was Nagle s algorithm which improves bandwidth at the expense of latency. With the fix, performances are mostly the same but there is no latency penalty any more.

OpenSSL 1.0.0e When I published the first round of benchmarks, Micha Trojnara was dubious about the bad results of stunnel. He told me on Twitter:

I solved it! You compiled stunnel against a 4 years old OpenSSL with MT-unsafe SSL_accept. Upgrade OpenSSL and rebuild stunnel. This bug was fixed in OpenSSL 1.0.0b. I implemented a (slow) workaround, so stunnel doesn t crash with earlier versions of OpenSSL.

Therefore, I backported Ubuntu Oneiric OpenSSL 1.0.0e to Ubuntu Lucid³ and, as you can see yourself on the plot below, there is a huge performance boost for stunnel: about 400%! stud features a slight improvement too (about 8%) but still starts with a latency of 40 ms while nginx does not improve.

Intel Accelerator Engine Most Linux distributions are still using OpenSSL 0.9.8. For example, Debian Squeeze ships with OpenSSL 0.9.8o, Ubuntu Lucid with 0.9.8k and RHEL 5.7 with 0.9.8e. Therefore, they are not eligible to recent performance improvements. To solve this, Intel released Intel Accelerator Engine, which also brings, among other things, AES-NI support:

This is essentially a collection of assembler modules from development OpenSSL branch, packed together as a standalone engine. Standalone means that it can be compiled outside OpenSSL source tree without patching the latter. Idea is to provide a way to utilize new code in already released OpenSSL versions constrained by support and release policies limiting their evolvement to genuine bug fixes.

Alas, in my tests, there was no improvement compared to OpenSSL 1.0.0e (already patched with AES-NI support). stunnel comes with out-of-the-box support for additional engines. Just add something like this to your stunnel.conf:

engine=dynamic
engineCtrl=SO_PATH:/WOO/intel-accel-1.4/libintel-accel.so
engineCtrl=ID:intel-accel
engineCtrl=LIST_ADD:1
engineCtrl=LOAD
engineCtrl=INIT

For stud, you should grab support for engine selection from GitHub and for nginx, just specify the engine with ssl_engine.

Intel Performance Primitives In his great article Accelerated SSL , Simon Kuhn shows huge performance improvements by using Intel s IPP library, a library aimed at leveraging power of multi-core processors and including some cryptographic primitives and a patch to make use of them in OpenSSL. Unfortunately, this library is not free, not even as in free beer . The patch provided by Intel only applies on OpenSSL 0.9.8j. Simon updated it for OpenSSL 1.0.0d. It also applies with a slight modification to OpenSSL 1.0.0e. Look at Simon s article for instructions on how to compile OpenSSL with IPP. Simon measured 220% improvement on RSA 1024-bit private key operations by using this library. Unfortunately, we don t get such a performance burst with stud which exhibits a 15% improvement (but remember that private key operations only happen once every 8 transactions thanks to session reuse). Worst, with stunnel, performances are reduced by one third and nginx just segfaults. I think such a library is risky to use in production.

Hyperthreading During the previous tests, only half the capacity of each core were used. For stud, adding more workers per core only slightly improves this situation (less than 10% with 4 workers per core). Therefore, I thought using hyperthreading could rectify the situation here. In this case, stud, stunnel and nginx are allocated 8 virtual cores instead of 4. stud only got a small improvement of 5%, nginx a moderate improvement of 11% while stunnel got a significant improvement of 26%.

Conclusion Here is a quick recap of the maximum TPS reached on each tests (while keeping average response time below 100 ms). All tests were run on 4 physical cores with 2048-bit certificates, AES128-SHA cipher suite. Each client makes 8 HTTP requests and reuse the first SSL session.

Context	nginx 1.1.4	stunnel 4.44	stud @e207dbc5a3
OpenSSL 0.9.8k	1489 TPS	400 TPS	1492 TPS
OpenSSL 1.0.0e	1501 TPS	1561 TPS	1609 TPS
OpenSSL 1.0.0e + IPP	-	1101 TPS	1857 TPS
OpenSSL 1.0.0e + HT	1718 TPS	1973 TPS	1694 TPS

I also would like to stress that below 1500 TPS, stunnel adds an overhead of a few milliseconds while stud adds 40 milliseconds. With OpenSSL 1.0.0e and hyperthreading enabled, stunnel has both good performance and low latency on low loads to moderate loads. You may just want to limit the number of allowed connections to avoid collapsing on high loads. While in the previous round, stud was the champion and stunnel lagged behind, the situation is now reversed : stunnel is on top! UPDATED: As said earlier, the latency issue with stud has been solved. Performances remain mostly the same.

---

1. HAProxy and network cards don t need dedicated core for this kind of load. However, if you also want to handle non SSL traffic, those dedicated cores can get busy.
2. The real condition is more complex. Each point is considered valid if the corresponding average latency is less than 100 ms or if the average on the next 8 points is less than 100 ms. Temporary latency bursts are therefore allowed. We are not so forgiveful with failed transactions.
3. This is pretty straighforward: I just dropped multiarch support. I chose to use Ubuntu version of OpenSSL instead of upstream because there are some interesting patches, like automatic support of AES-NI.

As somebody who often watches movies and TV shows I have the recurring problem of finding the right English subtitles for the video I am about to enjoy. Now, that is really a tedious thing to do. You go on a website such as opensubtitles.org and type the title of what you are looking for. Then you usually have to choose from a plethora of results, download the archive, unzip it, fire up VLC, go to Video Subtitles Track Open File, and finally open your subtitles file. And what you typically find out at that point is that the subtitles are completely out of sync with the audio, but rather than doing the whole process again you prefer to just sync them with a bit of VLC-fu. Which always takes longer than you thought, but hey. Now, what if there was an extremely simple way to just choose the subtitles for a given movie directly from VLC? The good news is that somebody wrote a beautiful VLC extension that does exactly that. He also wrote an introductory article about VLC extensions. Thank you, Jean-Philippe Andr . Installing the thing is as simple as downloading the subtitles extension and putting it in the VLC scripts folder. On Linux machines, the extensions folder can be found at ~/.local/share/vlc/lua/extensions/, on Windows APPDATA\VLC\lua\extensions. But seriously, don t use Windows. Once the extension is installed you can start watching a movie and then click View Subtitles to open the subtitles search interface. This is how it looks like: Now that the practical problem is solved, please take some time to look at the source code. VLC uses Lua as the language for writing extensions. I certainly cannot say that the web is full of resources about how to extend VLC using Lua, but the piece of code I wrote about in this post is certainly a good way to start hacking, and maybe scratch yet another itch.

For Ada Lovelace Day this year, I want to share my appreciation for Dr. Marian C. Diamond. In years past, I ve saluted women in the field of computing, which is my field as well. Dr. Diamond, however, is a biologist. Her research includes neuroanatomy, environment, immune functions, and hormones. In particular, she is interested in studying the effects of the external environment, aging, and immune responses on the cerebral neocortex. She has, in her words, had a love affair with the brain for about 70 years. I know very little about biology. The content and methods of her research are, frankly, beyond me, though some of her results have garnered popular attention. She has inspired me by demonstrating that rare combination of gifts: a deep understanding of a technical subject, and the ability to explain it to other people in an accessible way. In her interviews, articles and lectures, many of which are available online, Dr. Diamond displays these gifts in abundance. Her skill and enthusiasm for both learning and teaching is unmistakable. After applying her gifts in the classroom for many years, digital distribution has now enabled many more people to see and hear her, through millions of YouTube views. In 1960, she became the first female graduate student in UC Berkeley s anatomy department, and was apparently given the job of sewing a cover for a magnifying machine. I can only imagine the persistence required to continue from there to become a recognized leader in her field. She has gone on to help many other students along their way, and was named an unsung, everyday hero for the support she provided to students outside of the classroom or lab. As if that weren t enough, she has also traveled to Cambodia to apply her expertise in helping children injured by land mines. She still teaches today, just across the bay from where I write this, and will turn 85 next month.

It's become kind of a clich for me to claim that the reason I'm happy working on ACPI and UEFI and similarly arcane pieces of convoluted functionality is that no matter how bad things are there's at least some form of documentation and there's a well-understood language at the heart of them. My PhD was in biology, working on fruitflies. They're a poorly documented set of layering violations which only work because of side-effects at the quantum level, and they tend to die at inconvenient times. They're made up of 165 million bases of a byte code language that's almost impossible to bootstrap[1] and which passes through an intermediate representations before it does anything useful[2]. It's an awful field to try to do rigorous work in because your attempts to impose any kind of meaningful order on what you're looking at are pretty much guaranteed to be sufficiently naive that your results bear a resemblance to reality more by accident than design.

The field of bioinformatics is a fairly young one, and because of that it's very easy to be ignorant of its history. Crick and Watson (and those other people) determined the structure of DNA. Sanger worked out how to sequence proteins and nucleic acids. Some other people made all of these things faster and better and now we have huge sequence databases that mean we can get hold of an intractable quantity of data faster than we could ever plausibly need to, and what else is there to know?

Margaret Dayhoff graduated with a PhD in quantum chemistry from Columbia, where she'd performed computational analysis of various molecules to calculate their resonance energies[3]. The next few years involved plenty of worthwhile research that aren't relevant to the story, so we'll (entirely unfairly) skip forward to the early 60s and the problem of turning a set of sequence fragments into a single sequence. Dayhoff worked on a suite of applications called "Comprotein". The original paper can be downloaded here, and it's a charming look back at a rigorous analysis of a problem that anyone in the field would take for granted these days. Modern fragment assembly involves taking millions of DNA sequence reads and assembling them into an entire genome. In 1960, we were still at the point where it was only just getting impractical to do everything by hand.

This single piece of software was arguably the birth of modern bioinformatics, the creation of a computational method for taking sequence data and turning it into something more useful. But Dayhoff didn't stop there. The 60s brought a growing realisation that small sequence differences between the same protein in related species could give insight into their evolutionary past. In 1965 Dayhoff released the first edition of the Atlas of Protein Sequence and Structure, containing all 65 protein sequences that had been determined by then. Around the same time she developed computational methods for analysing the evolutionary relationship of these sequences, helping produce the first computationally generated phylogenetic tree. Her single-letter representation of amino acids was born of necessity[4] but remains the standard for protein sequences. And the atlas of 65 protein sequences developed into the Protein Information Resource, a dial-up database that allowed researchers to download the sequences they were interested in. It's now part of UniProt, the world's largest protein database.

Her contributions to the field were immense. Every aspect of her work on bioinformatics is present in the modern day larger, faster and more capable, but still very much tied to the techniques and concepts she pioneered. And so it still puzzles me that I only heard of her for the first time when I went back to write the introduction to my thesis. She's remembered today in the form of the Margaret Oakley Dayhoff award for women showing high promise in biophysics, having died of a heart attack at only 57.

I don't work on fruitflies any more, and to be honest I'm not terribly upset by that. But it's still somewhat disconcerting that I spent almost 10 years working in a field so defined by one person that I knew so little about. So my contribution to Ada Lovelace day is to highlight a pivotal woman in science who heavily influenced my life without me even knowing.

[1] You think it's difficult bringing up a compiler on a new architecture? Try bringing up a fruitfly from scratch.
[2] Except for the cases where the low-level language itself is functionally significant, and the cases where the intermediate representation is functionally significant.
[3] Something that seems to have involved a lot of putting punch cards through a set of machines, getting new cards out, and repeating. I'm glad I live in the future.
[4] The three-letter representation took up too much space on punch cards

comments

Would MTAs please stop sending hi-bit7 messages to other MTAs which do not advertise 8BITMIME! Recode it to QP or BASE64, damnit! The receiving MTA is entitled to strip the set bit7, which kinda makes things hard to read (while I know how to deal with blvde Stra_e, the advent of UTF-8 makes that blC6de StraC?e, introduces C0 control characters and makes typographic quotation marks into NUL-containing octet sequences (as their UTF-8 representation contains 0x80 octets) which let every sensible MDA terminate the line there). I even filed in the Debian BTS against the BTS (might be Drexim's fault, though). Would MUAs please default to Quoted-Printable! And mail hosters should use the same server when retrying delivery, to benefit greylisting. Or at least publish a list of outgoing IPv4 addresses they use for sending. Or use IPv6. Oh, and STARTTLS, while we are on my wishlist. It's a sad day when the percentage of correctly encoded eMail messages in my INBOX is smaller than that of my Spambox...

It doesn t look so old in hex; Zero, x, Two, Eight but I finally got there. So on this day, what other landmarks am I up to?

• I ve been programming, on and off, for 28 years (thanks for teaching me Chris anyone still use LOGO?)
• Writing Free Software for 17 years (thanks Terry for explaining the concepts and giving me my first Linux)
• A debian developer for 14 years (thanks for telling me and eventually adding me Bruce)

Some people get a little sad hitting this age, but it really is only a number, wether it is 0 28, \050 or even 40. As the saying goes: only the dead don t age.