I have several backup servers that run the excellent
rsnapshot software, which uses
Secure Shell (SSH) for remote access. The SSH private key of the backup server can be a weak link in the overall security. To see how it can be a problem, consider if someone breaks into your backup server and manages to copy your SSH private key, they will now have the ability to login to all machines that you take backups off (and that should be all of your machines, right?).
The traditional way to mitigate SSH private key theft is by password protecting the private key. This works poorly in an unattended server environment because either the decryption password needs to be stored in disk (where the attacker can read it) or the decrypted private key has to be available in decrypted form in memory (where attacker can read it).
A better way to deal with the problem is to move the SSH private key to a smartcard. The idea is that the private key cannot be copied by an attacker who roots your backup server. (Careful readers may have spotted a flaw here, and I need to explain one weakness with my solution: an attacker will still be able to login to all your systems by going through your backup server, however it will require an open inbound network connection to your backup server and the attacker will never know what your private key is. What this does is to allow you to more easily do damage control by removing the smartcard from the backup server.)
In this writeup, I ll explain how to accomplish all this on a
Debian/
Ubuntu-system using a
OpenPGP smartcard, a
Gemalto USB Shell Token v2 with gpg-agent/scdaemon from
GnuPG together with
OpenSSH.
First we need to install some packages. The goal is to configure
OpenSSH to talk to the
gpg-agent which will start and talk to
scdaemon which in turn talks to
pcscd which talks to the smart card reader. For some strange reason, the scdaemon binary is shipped with GnuPG s S/MIME interface in the
gpgsm package.
# apt-get install pcscd gnupg-agent gpgsm
The above command should install and start
pcscd and if all works well, you should be able to check the status of the smartcard using GnuPG.
# gpg --card-status
You need to initialize the smartcard and generate a private key on it, again using GnuPG. If you trust GnuPG more than the smartcard to generate a good private key, you may generate the private key using GnuPG and then move it onto the smartcard (hint: use the
keytocard command). Make sure you don t leave a copy of the private key on the same machine!
# gpg --card-edit
gpg: detected reader Gemalto GemPC Key 00 00'
...
gpg/card> admin
Admin commands are allowed
gpg/card> name
Cardholder's surname:
Cardholder's given name: host.example.org
gpg: 3 Admin PIN attempts remaining before card is permanently locked
Please enter the Admin PIN
gpg: gpg-agent is not available in this session
gpg/card> lang
Language preferences: en
gpg/card> generate
Make off-card backup of encryption key? (Y/n) n
Please enter the PIN
What keysize do you want for the Signature key? (2048)
What keysize do you want for the Encryption key? (2048)
What keysize do you want for the Authentication key? (2048)
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: host.example.org
Email address:
Comment:
You selected this USER-ID:
"host.example.org"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (33 seconds)
gpg: signatures created so far: 0
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (18 seconds)
gpg: signatures created so far: 1
gpg: signatures created so far: 2
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (23 seconds)
gpg: signatures created so far: 3
gpg: signatures created so far: 4
gpg: key 12345678 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 2048R/12345678 2011-09-19
Key fingerprint = 1234 5678 1234 5678 1234 5678 1234 5678 1234 5678
uid host.example.org
sub 2048R/23456789 2011-09-19
sub 2048R/34567890 2011-09-19
gpg/card> quit
#
Now for the interesting part. OpenSSH talks to an agent for private key handling, and GnuPG s gpg-agent supports this protocol when the
--enable-ssh-support parameter is given. During startup, gpg-agent will print some environment variables that needs to be set when
ssh is run. Normally gpg-agent is invoked by the Xsession.d login scripts, so that the environment variables are inherited by all your processes. However, for an unattended machine without any normal login process, we need to write a script to start gpg-agent. First do these manual steps, to confirm that everything works.
# gpg-agent --daemon --enable-ssh-support > /var/run/gpg-agent-info.env
# . /var/run/gpg-agent-info.env
# ssh-add -L
ssh-rsa AAAAB3N... cardno:000500000BD8
#
If the final step printed a SSH public id, the (sometimes) tricky part in getting the hardware to work is (hopefully) complete. What remains is to script things so that
gpg-agent is started on boot and to make sure that your backup scripts has the proper environment variables before launching whatever processes will launch
ssh. Further, since we will be running unattended, we need a mechanism to unlock the smartcard using a PIN interactively once on each boot of the machine. I prefer manually entering the PIN on every boot over having the PIN stored in a file on the disk.
I will use the
/etc/rc.local mechanism to start gpg-agent, like this:
# cat> /etc/rc.local
#!/bin/sh -e
exec gpg-agent --daemon --enable-ssh-support \
--pinentry-program /usr/local/sbin/pinentry-unattended \
--write-env-file /var/run/gpg-agent-info.env
^D
The astute reader will now ask what
/usr/local/sbin/pinentry-unattended is and why it is needed. Now here is the situation. scdaemon will normally query the user for a PIN using a tool called
pinentry which reads and write to the user s TTY directly. This won t work in unattended mode, so we want the scdaemon to signal failure here unless we are actually unlocking the smartcard manually. Here is the entire script:
#!/bin/sh
# /usr/local/sbin/pinentry-unattended -- by Simon Josefsson
if test x"$PINENTRY_USER_DATA" = xinteractive; then
exec pinentry "$@"
fi
exit 1
What remains is a script to unlock the smartcard by providing the PIN. This is typically invoked manually if the server has restarted for some reason. Don t worry, any ssh sessions invoked by cron until you have managed to unlock the smartcard will fail with an authentication error it won t hang waiting for a PIN to be entered.
#!/bin/sh
# /usr/local/sbin/unlock-smartcard -- by Simon Josefsson.
. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID
gpg-connect-agent 'scd killscd' /bye > /dev/null
while ! gpg-connect-agent 'scd serialno' /bye grep -q SERIALNO; do
sleep 1
done
PINENTRY_USER_DATA=interactive
export PINENTRY_USER_DATA
checkpin
And the script
checkpin is as follows:
#!/bin/sh
# /usr/local/sbin/checkpin -- by Simon Josefsson.
id= gpg-connect-agent 'scd serialno' /bye head -1 cut -d\ -f3
gpg-connect-agent "scd checkpin $id" /bye grep -q OK
At this point, you should have everything configured and installed. Don t forget to
chmod +x the scripts. The typical use-pattern is as follows. After the machine has been started, gpg-agent is running but the smartcard is not unlocked with the PIN. You need to manually login to the machine and run unlock-smartcard and enter the PIN. In the script that runs the backup jobs, invoked via cron, make sure that the first line of the scripts reads (assuming Bourne shell script syntax):
. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID
To avoid needlessly attempting ssh connections if the smartcard is not unlocked, your backup script can also call the
checkpin code and abort if it doesn t return true.
checkpin exit 1
Some final words about debugging. A basic command to run to check that the GnuPG side is working is
gpg --card-status, it should print some information about the smartcard if successful. To check that the SSH agent part is working, use
ssh-add -L. If you get error messages, try killing the
scdaemon process by running
killall -9 scdaemon and let
gpg-agent respawn a new
scdaemon process.
That s it! If you like my writeup, please flattr it.
Even though some three months have passed since DebConf11 has successfully ended, I still wanted to give you just a glimpse on how some parts were played through the eyes of a lunatic (read: organizer). Of course, blog post can t come close near of explaining anything but at least it should give some insight and hopefully some pointers to the future DebConf organizers.
If you don t feel like reading this much text you can listen to last episode of This Week In Debian podcast and after it just head to Beginning of the end part.
DebConf11
First of all I d like to apologize if I offended or hurt anyone in this whole process as it wasn t my intention and during DebConf organization this is nothing irregular, for you to get hurt or you hurting somebody else. One thing that got stuck with me this whole time is when Mart n Ferrari approached me after I did bid proposal of Bosnia/Herzegovina for DebConf11 back on DebConf9 and told me you have no idea what you just got yourself into I played cool and said something along the line of coure I do and boy did I lie.
During DebConf organization you re bound to make some of your decisions instantaneously in which you ll lose something, the most you can do is assess what s the thing you can cut the cord on, even though if that thing or a person for that matter might have meant something to you. You ll lose things, relationships, friendships, contacts at the times sense of humor and maybe even common logic. Some of it comes back to you, some the very next day and for some it may take longer. Some of it was lost irreversibly, but even in that case it wasn t lost and could even be traced to some other cause.
One thing that I m thankful for is that I didn t 
. Even this DebConf wasn t our first bid we lost DebConf7 bid to Edinburgh, but it is perseverance that got us here. The moral I m trying to tell here is if you have a goal never back down from it.
What I found to be our biggest problem and what I personally found as most painstaking in all of this was having and reporting to so many different sides and different groups while trying to keep them in same communication channel. Even my right hand (Velimir) was apart from me (Vienna), then place where two of us are coming from is not the place where DebConf is getting organized in. As in the end we ended up having all the parties separated and pulling to their own side, from Government of Republika Srpska to NGO to Debian to local team and to those who remained as undeclared . Option of cutting the cord in this case was missing as these were the components that DebCon11 was made of.
Of course this meant you ll be missing workforce and there were days you d wake up driving >= 200 km/h (literally) with one hand on the wheel, other on the phone and your other right hand (Velimir) holding your RedBull. But you wouldn t do this just because you had to and there was no one else to do it, you d do it because you re responsible for the sake of the project and won t be just sitting there because by some unwritten hierarchy rules you re supposed to sit and supervise things. I d advise you do the same thing when you see there s no one else to do something, even if it means making coffee for one of the teams.
I m not trying to scare you away from organizing your own DebConf, on the contrary after all I ve been through in past two years if someone asked me would I do it if I knew it was like this, without a doubt my answer would still be fsck yea . All the previous DebConf organizers told me about all of this, but why didn t I give up? For the same reason you won t give up and for the same reason why we believe in the same goal.
If anything this is one of the stories you ll want to tell your kids about so When I was your age I used to sleep naked in the snow would turn to When I was your age I organized DebConf11 in Maybe this blog post wasn t even necessary and some things weren t supposed to be said, I guess sometimes I just can t shut up 
For those who were on DebConf11 I hope you enjoyed and had a great time, otherwise I m saying all of this in vein. As organizer all you see is problems and you miss most of the confrence, but by what I know there are 11 candidate bids for DebConf13 and for me hearing that tells me the results of our efforts.
DebConf11 after effects
I could write a whole novel about this, but to keep it as short as possible, for last two years as a side project I was working on an idea of Government or some of its institutions migrating to Linux. At first I was somewhat loud about it, then after Microsoft heard about it and after they tried stopping the idea by trying to scare me by trying to interfere with my private life; as that didn t work its lobbyist came even near of obstructing the whole conference within the Government. For the sake of the conference, I convinced the Government that by supporting DebConf it doesn t mean they need to move to Linux and publicly stopped talking about it. I also convinced them that our only goal was to have successful conference and promote alternative options and open ideas. I wasn t lying as I saw this as new opportunity of them concluding on their own why they should or shouldn t not move, the better conference was the more chances of success we had.
That s why I tried pushing as many representatives from various companies as in this case we would use reverse psychology where basically no one or few know what Linux or Debian for that matter is, but everybody knows who Google is, so if you have participants from i.e: Google or Austrian E-Health care system talking about how they are using your technology is better way to explain what s it all about really. Eventually Microsoft even had their first ever conference in Bosnia/Herzegovina and you wanna take a wild guess where it was held? 
I've been wanting to sponsor a child in Africa for many, many years. I
remember seeing a photo of a sponsored child on a couple of sets of friends'
mantelpieces in Canberra, and saying to myself how I should really do that
too.
Of course, for no really good reason, fast-forwarding to 6+ years later, I
still hadn't gotten around to it. Until last month. I was walking out
of the Apple Store in Palo Alto. I was accosted by one of those people on
the street, with the clipboard and all that.
I usually manage to brush these sorts of people off. I really hate solicited
donations. I'd rather consciously make a donation to a charity, than do it
because someone on the street was paid to ask me to, or knocked on my door,
or whatever. But I was distracted, or in a good mood, or something, and so I
missed the initial opportunity to deflect her.
I'm really glad I did, because as I said, this is something that I've wanted
to do for years, so I barely hesitated, and signed up on the spot.
I received the welcome kit in the mail a couple of days ago. I've been
allocated a 6 year old boy named Armando, from Mozambique. I'm looking
forward to corresponding with him.
Our friends, and former neighbours, the Harvetts have moved back to South
Africa, and I hope one day to be able to visit them. Since we'll be in the
vicinity, I hope one day to be able to visit Armando as well.
I'm pleased to see that 
A couple of times I have written code which basically does this:
Good morning all!
It feels like forever since I last blogged . Which is extremely bad of me *slaps own wrist*. However, I m been horrendously stupidly busy. As some of you may know, I m very much an Android fan. Recently, I ve gotten myself a shiny Galaxy SII.
Given the fact that alongside my day job of being a Web Developer for an Online Retailer, I m also a 
The issue
This morning (while I was running late for an appointement) I
had a very weird stuff happening on my Thinkpad T61 laptop. Since I
recently offered myself a shiny Thinkpad x201s, I have to admit I
don't use much my T61 anymore. But this morning I had to print a
page (for this appointement) and, as I didn't yet configured my
printer on the x201s, I went to the T61. But I noticed that the
network was down. I've tried quickly on wireless but, bad luck, my
current wifi setup selects the channel automatically and it prefers
choosing channels which aren't available in the US. Guess what, my
T61 comes from the US and has those channels completely disabled,
so no wireless available either.
The investigation
I first tried to
Dear Lazyweb .
I ran into a very strange issue with MySQL authentication. The initial situation is as follows:
A month ago, I published an article on the compared
The 
It's become kind of a clich for me to claim that the reason I'm happy working on ACPI and UEFI and similarly arcane pieces of convoluted functionality is that no matter how bad things are there's at least some form of documentation and there's a well-understood language at the heart of them. My PhD was in biology, working on fruitflies. They're a poorly documented set of layering violations which only work because of side-effects at the quantum level, and they tend to die at inconvenient times. They're made up of 165 million bases of a byte code language that's almost impossible to bootstrap[1] and which passes through an intermediate representations before it does anything useful[2]. It's an awful field to try to do rigorous work in because your attempts to impose any kind of meaningful order on what you're looking at are pretty much guaranteed to be sufficiently naive that your results bear a resemblance to reality more by accident than design.
This is my monthly summary of my Debian related activities. If you re among the people who 
Dpkg work
While taking care of the last details for the hardening feature in dpkg 1.16.1, I have 
By the way Paypal used to be required but it s no longer the case, you can support the project just with your usual credit card.
Misc blog updates
Over time, I have written many useful articles for Debian users and Debian contributors. But scattered in the history, they are somewhat difficult to find. To fix this I have created some index pages listing them. Check them out:
