Over the coming week, there are a vast number of free software events taking place around the world. I'll be at the LinuxWochen Vienna and MiniDebConf Vienna, the events run over four days from Thursday, 28 April to Sunday, 1 May. At MiniDebConf Vienna, I'll be giving a talk on Saturday (schedule not finalized yet) about our progress with free Real-Time Communications (RTC) and welcoming 13 new GSoC students (and their mentors) working on this topic under the Debian umbrella. On Sunday, Iain Learmonth and I will be collaborating on a workshop/demonstration on Software Defined Radio from the perspective of ham radio and the Debian Ham Radio Pure Blend. If you want to be an active participant, an easy way to get involved is to bring an RTL-SDR dongle. It is highly recommended that instead of buying any cheap generic dongle, you buy one with a high quality temperature compensated crystal oscillator (TXCO), such as those promoted by RTL-SDR.com. Saturday, 30 April is also Linux Presentation Day in many places. There is an event in Switzerland organized by the local local FSFE group in Basel. DebConf16 is only a couple of months away now, Registration is still open and the team are keenly looking for additional sponsors. Sponsors are a vital part of such a large event, if your employer or any other organization you know benefits from Debian, please encourage them to contribute.

alberoVita The accelerated world of the Western or Westernized countries seems to be fed by an insidious food, which generates a kind of psychological dependence: anxiety. The economy of global markets cannot help it, it has a structural need of it to feed their iron logic of survival. The anxiety generated in the masses of consumers and in market competitors is crucial for Companies fighting each other and now they can only live if men are projected to objective targets continuously moving forward, without ever allowing them to achieve a stable destination.

The consumer is thus constantly maintained in a state of perpetual breathlessness, always looking for the fresh air of liberation that could eventually reduce his tension. It is a state of anxiety caused by false needs generated by advertising campaigns whose primary purpose is to create a need, to interpret to their advantage a still confused psychological demand leading to the destination decided by the market <Read More [by Fabio Marzocca]>

Switching from one GnuPG master key to the usage of subkeys was long on my list of things I wanted to do, but never came around. With the advent of a YubiKey NEO in my pocket I finally took the plunge: reading through lots of web pages (and adding one here for confusion), trying to understand the procedures, and above all, understanding my own requirements!

To sum up a long story, it was worth the plunge, and all over the security level of my working environment has improved considerable. While the advantages of subkeys are well documented (e.g., Debian Wiki), at the end of the day I was like probably many Debian Developers having one master key that was used for every action: mail decryption and signing, signing of uploads, etc. Traveling a lot I always felt uncomfortable. Despite a lengthy passphrase, I still didn t want my master key to get into wrong hands in case the laptop got stolen. Furthermore, I had my master key on several computers (work, laptop, mail server), which didn t help a lot either. With all this, I started to compile a list of requirements/objectives I wanted to have:

master key is only available on offline medium (USB sticks)
subkeys for signing, encryption, authentication
possibility to sign and decrypt my emails on the server where I read emails (ssh/mutt)
laptop does not contain any keys, instead use Yubikey
all keys with expiry date (1y)
mixture of gpg versions: local laptop: gpg2.1, mail server: gpg1

Warning Before we start a word of caution make backups, best is to make backups at every stage. You don t want that an erroneous operations wipes out your precious keys without a backup! Preparation In the following I will assume that MASTERKEY environment variable contains the id of the master key to be converted. Furthermore, I have followed some of the advice here, so key ids will be shown in long format. Let us start with the current situation:

$ gpg -K $MASTERKEY
sec   4096R/0x6CACA448860CDC13 2010-09-14
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                            Norbert Preining <norbert@preining.info>
uid                            Norbert Preining <preining@logic.at>
uid                            Norbert Preining <preining@debian.org>
uid                            Norbert Preining <preining@jaist.ac.jp>
ssb   4096R/0xD1D2BD14810F62B3 2010-09-14

In the following we will go through the following steps:

Prepare the Yubikey NEO (forthcoming blog>
Edit to current key: add expiry, add photo, and above all add subkeys
Create revocation certificate
Create gpg2.1 structure
Backup to USB media
Move subkeys to Yubikey NEO
Remove master keys
Separate gpg1 (for mail server) and gpg2 (for laptop)
Upload to key servers

Yubikey SmartCard setup There are several guides out there, but I will in very near future write one about using the NEO for various usage scenaria including GPG keys. Edit the current key The following can be done in one session or in different sessions, the screen logs are after starting with:

$ gpg --expert --edit-key $MASTERKEY

add expiry date Having an expiry date on your key serves two purposes: If you loose it, it will solve itself automatically, and furthermore, you are forced to deal with the key and refresh your gpg knowledge at least once a year. That are two perfect reasons to set expiry to one year. The following log selects each key in turn and sets its expiry date.

$ gpg --expert --edit-key $MASTERKEY
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: never       usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: never       usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
 
gpg> expire
Changing expiration time for the primary key.
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:09:16 PM JST
Is this correct? (y/N) y
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: never       usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
 
gpg> key 1
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: never       usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
 
gpg> expire
Changing expiration time for a subkey.
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:09:27 PM JST
Is this correct? (y/N) y
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>

Add a photo Not strictly necessary, but an interesting feature. gpg suggests 240 288, I resized a photo of my head, greyscaled it, and optimized it with jpegoptim -s -m40 my-photo.jpg. The parameter 40 is the quality, I played around a bit to find the best balance between size and quality. The size should not be too big as the photo will be part of the key!

gpg> addphoto
 
Pick an image to use for your photo ID.  The image must be a JPEG file.
Remember that the image is stored within your public key.  If you use a
very large picture, your key will become very large as well!
Keeping the image close to 240x288 is a good size to use.
 
Enter JPEG filename for photo ID: GPG/norbert-head.jpg
Is this photo correct (y/N/q)? y
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ unknown] (5)  [jpeg image of size 4185]

Add subkeys of 2048bit for signing/encryption/authentication Now comes the interesting part, adding three subkeys: one for signing, one for encrypting, and one for authentication. The one for signing is the one you will use for signing your uploads to Debian as well as emails. The authentication key will later be used to provide ssh authentication. Note that you have to use the --expert expert option to edit-key (as shown above), otherwise gpg does not allow to do this. As I want to move the subkeys to the Yubikey NEO, a keysize of 2048bits is necessary. First for the signing:

gpg> addkey
Key is protected.
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
Your selection? 4
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:10:06 PM JST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
....+++++
..........+++++
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
sub  2048R/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06  usage: S   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ unknown] (5)  [jpeg image of size 4185]

Now the same for encryption key:

gpg> addkey
Key is protected.
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
Your selection? 6
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:10:20 PM JST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
..+++++
........+++++
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
sub  2048R/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06  usage: S   
sub  2048R/0xBF361ED434425B4C  created: 2016-02-07  expires: 2017-02-06  usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ unknown] (5)  [jpeg image of size 4185]

Finally for the authentication key. Note that only here the --expert is necessary! We use (8) RSA (set your own capabilities) and then toggle sign and encryption capabilities off, and authentication on.

gpg> addkey
Key is protected.
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
Your selection? 8
 
Possible actions for a RSA key: Sign Encrypt Authenticate 
Current allowed actions: Sign Encrypt 
 
   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished
 
Your selection? s
 
Possible actions for a RSA key: Sign Encrypt Authenticate 
Current allowed actions: Encrypt 
 
   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished
 
Your selection? e
 
Possible actions for a RSA key: Sign Encrypt Authenticate 
Current allowed actions: 
 
   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished
 
Your selection? a
 
Possible actions for a RSA key: Sign Encrypt Authenticate 
Current allowed actions: Authenticate 
 
   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished
 
Your selection? q
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:10:34 PM JST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
......+++++
+++++
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
sub  2048R/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06  usage: S   
sub  2048R/0xBF361ED434425B4C  created: 2016-02-07  expires: 2017-02-06  usage: E   
sub  2048R/0x9C7CA4E294F04D49  created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ unknown] (5)  [jpeg image of size 4185]
 
gpg> save

Check the current status Good point to take a break and inspect the current status. We should have one main key and three subkeys, all with expiry dates of 1 year ahead, and a photo also attached to the key:

$ gpg --expert --edit-key $MASTERKEY
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
gpg: checking the trustdb
gpg: public key 0x0FC3EC02FBBB8AB1 is 58138 seconds newer than the signature
gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model
gpg: depth: 0  valid:   2  signed:  28  trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1  valid:  28  signed:  41  trust: 28-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2016-11-02
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
sub  2048R/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06  usage: S   
sub  2048R/0xBF361ED434425B4C  created: 2016-02-07  expires: 2017-02-06  usage: E   
sub  2048R/0x9C7CA4E294F04D49  created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg>

Create revocation certificate In case something happens, like all your backups are burned, your computers are destroyed, or all data stolen by the NSA, it is a good idea to have an old fashioned paper print out of a revocation certificate which allows you to revoke the key even if you are not in possession of it. This should be printed out and kept in a safe place.

$ gpg --gen-revoke $MASTERKEY > GPG/revoke-certificate-$MASTERKEY.txt
 
sec  4096R/0x6CACA448860CDC13 2010-09-14 Norbert Preining <norbert@preining.info>
 
Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
  0 = No reason specified
  1 = Key has been compromised
  2 = Key is superseded
  3 = Key is no longer used
  Q = Cancel
(Probably you want to select 1 here)
Your decision? 1
Enter an optional description; end it with an empty line:
> 
Reason for revocation: Key has been compromised
(No description given)
Is this okay? (y/N) y
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
ASCII armored output forced.
Revocation certificate created.

Please move it to a medium which you can hide away; if the NSA or KGB or Mossad gets access to this certificate, they can use it to make your key unusable. It is smart to print this certificate and store it away, just in case your media become unreadable. Create gpg 2.1 structure There are currently three versions of gpg available: classic (version 1) which is one static binary, perfect for servers or scripting tasks; stable (version 2.0) which is the modularized version supporting OpenPGP, S/MIME, and Secure Shell; and finally modern (version 2.1 and up) with enhanced features like support for Elliptic Curve cryptography. Debian currently ships version 1 as standard, and also the modern version (but there are traces in experimental of a pending transition). The newer versions of GnuPG are modularized and use an agent. For the following we need to kill any running instance of gpg-agent.

$ killall gpg-agent

After that a simple call to gpg2 to list the secret keys will convert the layout to the new standard:

$ gpg2 -K $MASTERKEY
gpg: keyserver option 'ca-cert-file' is obsolete; please use 'hkp-cacert' in dirmngr.conf
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/home/norbert/.gnupg/secring.gpg' to gpg-agent
gpg: key 0xD2BF4AA309C5B094: secret key imported
gpg: key 0x6CACA448860CDC13: secret key imported
gpg: migration succeeded
sec   rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                   [ultimate] Norbert Preining <norbert@preining.info>
uid                   [ultimate] Norbert Preining <preining@logic.at>
uid                   [ultimate] Norbert Preining <preining@debian.org>
uid                   [ultimate] Norbert Preining <preining@jaist.ac.jp>
uid                   [ultimate] [jpeg image of size 4185]
ssb   rsa4096/0xD1D2BD14810F62B3 2010-09-14 [E] [expires: 2017-02-06]
ssb   rsa2048/0xEC00B8DAD32266AA 2016-02-07 [S] [expires: 2017-02-06]
ssb   rsa2048/0xBF361ED434425B4C 2016-02-07 [E] [expires: 2017-02-06]
ssb   rsa2048/0x9C7CA4E294F04D49 2016-02-07 [A] [expires: 2017-02-06]

After this there will be new files/directories in the .gnupg directory, in particular: .gnupg/private-keys-v1.d/ which contains the private keys. Creating backup Now your .gnupg directory contains still all the keys, available for gpg1 and gpg2.1. You MUST MAKE A BACKUP NOW!!! on at least 3 USB sticks and maybe some other offline media. Keep them in a safe place, better in different and safe places, you will need them for extending the expiry date, signing other keys, etc. Warning concerning USB and vfat file systems gpg >= 2.1 requires gpg-agent which in turn needs a socket. If you have the backup on an USB drive (most often with vfat file system), you need to redirect the socket, as vfat does not support sockets! Edit /USBSTICK/gnupghome/S.gpg-agent and enter there

%Assuan%
socket=/dev/shm/S.gpg-agent

After that the socket will be created in /dev/shm/ instead and invoking gpg with gpg2 --homedir /USBSTICK/gnupghome will work. You have done your backups, right? Move sub keys to card As I mentioned, I want to have no keys on my laptop which I carry around to strange countries, instead I want to have them all on a Yubikey NEO. I will describe the setup and usage in details soon, but mention here only how to move the keys to the card. This requires a finished setup including change of pins. Note that when using gpg2 to move the keys to the card, the local copies are actually deleted, but only for the gpg2(.1) files. The gpg1 secret keys are still all in place.

$ gpg2 --edit-key $MASTERKEY
gpg (GnuPG) 2.1.11; Copyright (C) 2016 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> save

Note the repetition of selecting and deselecting keys. Current status After this procedure we are now in the following situation:

gpg1: all keys are still available
gpg2: sub keys are moved to yubikey (indicated below by ssb>), and master key is still available

In gpg words it looks like this:

$ gpg2 -K $MASTERKEY
gpg: keyserver option 'ca-cert-file' is obsolete; please use 'hkp-cacert' in dirmngr.conf
sec   rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                   [ultimate] Norbert Preining <norbert@preining.info>
uid                   [ultimate] Norbert Preining <preining@logic.at>
uid                   [ultimate] Norbert Preining <preining@debian.org>
uid                   [ultimate] Norbert Preining <preining@jaist.ac.jp>
uid                   [ultimate] [jpeg image of size 4185]
ssb   rsa4096/0xD1D2BD14810F62B3 2010-09-14 [E] [expires: 2017-02-06]
ssb>  rsa2048/0xEC00B8DAD32266AA 2016-02-07 [S] [expires: 2017-02-06]
ssb>  rsa2048/0xBF361ED434425B4C 2016-02-07 [E] [expires: 2017-02-06]
ssb>  rsa2048/0x9C7CA4E294F04D49 2016-02-07 [A] [expires: 2017-02-06]
 
$ gpg -K $MASTERKEY
sec   4096R/0x6CACA448860CDC13 2010-09-14 [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                            Norbert Preining <norbert@preining.info>
uid                            Norbert Preining <preining@logic.at>
uid                            Norbert Preining <preining@debian.org>
uid                            Norbert Preining <preining@jaist.ac.jp>
uid                            [jpeg image of size 4185]
ssb   4096R/0xD1D2BD14810F62B3 2010-09-14 [expires: 2017-02-06]
ssb   2048R/0xEC00B8DAD32266AA 2016-02-07 [expires: 2017-02-06]
ssb   2048R/0xBF361ED434425B4C 2016-02-07 [expires: 2017-02-06]
ssb   2048R/0x9C7CA4E294F04D49 2016-02-07 [expires: 2017-02-06]
 
$ gpg2 --card-status
 
....
Name of cardholder: Norbert Preining
....
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: 5871 F824 2DCC 3660 2362  BE7D EC00 B8DA D322 66AA
      created ....: 2016-02-07 11:10:06
Encryption key....: 2501 195C 90AB F4D2 3DEA  A303 BF36 1ED4 3442 5B4C
      created ....: 2016-02-07 11:10:20
Authentication key: 9CFB 3775 C164 0E99 F0C8  014C 9C7C A4E2 94F0 4D49
      created ....: 2016-02-07 11:10:34
General key info..: sub  rsa2048/0xEC00B8DAD32266AA 2016-02-07 Norbert Preining <norbert@preining.info>
sec   rsa4096/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06
ssb   rsa4096/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06
ssb>  rsa2048/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06
                                  card-no: 0006 03645719
ssb>  rsa2048/0xBF361ED434425B4C  created: 2016-02-07  expires: 2017-02-06
                                  card-no: 0006 03645719
ssb>  rsa2048/0x9C7CA4E294F04D49  created: 2016-02-07  expires: 2017-02-06
                                  card-no: 0006 03645719
$

Remove private master keys You are sure that you have a working backup? Did you try it with gpg --homedir ...? Only if you are really sure, continue. We are now removing the master key from both the gpg2 and gpg1 setup. removal for gpg2 gpg2 keeps the private keys in ~/.gnupg/private-keys-v1.d/KEYGRIP.key and the KEYGRIP can be found by adding --with-keygrip to the key listing. Be sure to delete the correct file, the one related to the master key.

$ gpg2 --with-keygrip --list-key $MASTERKEY
pub   rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
      Keygrip = 9DC1E90703856C1DE0EAC970CED7ABF5EE5EF79D
uid                   [ultimate] Norbert Preining <norbert@preining.info>
uid                   [ultimate] Norbert Preining <preining@logic.at>
uid                   [ultimate] Norbert Preining <preining@debian.org>
uid                   [ultimate] Norbert Preining <preining@jaist.ac.jp>
uid                   [ultimate] [jpeg image of size 4185]
sub   rsa4096/0xD1D2BD14810F62B3 2010-09-14 [E] [expires: 2017-02-06]
      Keygrip = 4B8FF57434DD989243666377376903281D861596
sub   rsa2048/0xEC00B8DAD32266AA 2016-02-07 [S] [expires: 2017-02-06]
      Keygrip = 39B14EF1392F2F251863A87AE4D44CE502755C39
sub   rsa2048/0xBF361ED434425B4C 2016-02-07 [E] [expires: 2017-02-06]
      Keygrip = E41C8DDB2A22976AE0DA8D7D11F586EA793203EA
sub   rsa2048/0x9C7CA4E294F04D49 2016-02-07 [A] [expires: 2017-02-06]
      Keygrip = A337DE390143074C6DBFEA64224359B9859B02FC
 
$ rm ~/.gnupg/private-keys-v1.d/9DC1E90703856C1DE0EAC970CED7ABF5EE5EF79D.key
$

After that the missing key is shown in gpg2 -K with an additional # meaning that the key is not available:

$ gpg2 -K $MASTERKEY
sec#  rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
...

removal for gpg1 Up to gpg v2.0 there is no simple way to delete only one part of the key. We export the subkeys, delete the private key, and reimport the subkeys:

$ gpg --output secret-subkeys --export-secret-subkeys $MASTERKEY
 
$ gpg --delete-secret-keys $MASTERKEY
 
sec  4096R/0x6CACA448860CDC13 2010-09-14 Norbert Preining <norbert@preining.info>
 
Delete this key from the keyring? (y/N) y
This is a secret key! - really delete? (y/N) y
 
$ gpg --import secret-subkeys
gpg: key 0x6CACA448860CDC13: secret key imported
gpg: key 0x6CACA448860CDC13: "Norbert Preining <norbert@preining.info>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
 
$

Current status We are basically at the stage we wanted to achieve: For gpg2.1 only the old encryption key is available, the master key is not, and the other sub keys are moved to the yubikey:

$ gpg2 -K $MASTERKEY
sec#  rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                   [ultimate] Norbert Preining <norbert@preining.info>
uid                   [ultimate] Norbert Preining <preining@logic.at>
uid                   [ultimate] Norbert Preining <preining@debian.org>
uid                   [ultimate] Norbert Preining <preining@jaist.ac.jp>
uid                   [ultimate] [jpeg image of size 4185]
ssb   rsa4096/0xD1D2BD14810F62B3 2010-09-14 [E] [expires: 2017-02-06]
ssb>  rsa2048/0xEC00B8DAD32266AA 2016-02-07 [S] [expires: 2017-02-06]
ssb>  rsa2048/0xBF361ED434425B4C 2016-02-07 [E] [expires: 2017-02-06]
ssb>  rsa2048/0x9C7CA4E294F04D49 2016-02-07 [A] [expires: 2017-02-06]
$

And for gpg <= 2.0 the old encryption key and the sub keys are available, but the master key is not:

$ gpg -K $MASTERKEY
sec#  4096R/0x6CACA448860CDC13 2010-09-14 [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                            Norbert Preining <norbert@preining.info>
uid                            Norbert Preining <preining@logic.at>
uid                            Norbert Preining <preining@debian.org>
uid                            Norbert Preining <preining@jaist.ac.jp>
uid                            [jpeg image of size 4185]
ssb   4096R/0xD1D2BD14810F62B3 2010-09-14 [expires: 2017-02-06]
ssb   2048R/0xEC00B8DAD32266AA 2016-02-07 [expires: 2017-02-06]
ssb   2048R/0xBF361ED434425B4C 2016-02-07 [expires: 2017-02-06]
ssb   2048R/0x9C7CA4E294F04D49 2016-02-07 [expires: 2017-02-06]
 
$

Split the .gnupg directory for mail server and laptop As mentioned, I want to have a gpg1 version available at the server where I read my emails, and be able to sign/encrypt emails there, while on my laptop no secret key is available. Thus I prepare two gnupg directories. For the mailserver the gpg2 specific files are removed:

$ cp -a .gnupg .gnupg-mail
$ cd .gnupg-mail
$ rm -rf private-keys-v1.d/ pubring.gpg~ reader_0.status
$ rm -rf S.gpg-agent* S.scdaemon .gpg-v21-migrated

On my laptop, where I did all this operation, I remove the gpg1 files, namely the outdated secring.gpg:

$ cd $HOME/.gnupg
$ rm secring.gpg

As a last step I move the .gnupg-mail directory to my mail server. Once could *expire* the old encryption key, but for now I leave it as is. Upload keys to keyservers If you are a Debian Developer, a simple update of your master key will suffice:

gpg --keyserver hkp://keyring.debian.org --send-key YOURMASTERKEYID

Note that the update from the keyring server to the actual Debian keyring takes up to one month. Until that time either do not upload anything, or use the (offline) master key for signing. After your key has been updated in the Debian keyring, signatures made with the signing subkey will be accepted for uploading to Debian. It might be also a good idea to upload your new keys to some keyservers like:

gpg --keyserver hkp://pool.sks-keyservers.net --send-key $MASTERKEY

Now you an also fix the configuration file skew between gpg1 and gpg2. Further remark I am currently trying to use the authentication key from my Yubikey NEO as ssh key, but bugs (see #795368 and #818969) prohibit it at the moment. Raphael Herzog gave a possible fix by killing the gpg-agent and restarting it with gpg-agent --daemon from an X terminal, and I can confirm that this worked. After one year before the key expires I need to extend the key validity for another year. For this you need the offline master key. I will describe the process when it becomes necessary. Reading list The following web sites have been useful in collecting the necessary information:

https://iain.learmonth.me/yubikey-neo-gpg/
https://iain.learmonth.me/yubikey-udev/
http://blog.josefsson.org/2014/06/23/offline-gnupg-master-key-and-subkeys-on-yubikey-neo-smartcard/
https://wiki.debian.org/Subkeys
https://jclement.ca/articles/2015/gpg-smartcard/ as modernized version of (3)
https://www.esev.com/blog/post/2015-01-pgp-ssh-key-on-yubikey-neo/ similar style, with ssh and gnome-keyring infos
http://karlgrz.com/2fa-gpg-ssh-keys-with-pass-and-yubikey-neo/ also good reading
https://help.riseup.net/en/security/message-security/openpgp/best-practices good and concise advise on gpg practices

My writing is mostly based on (5) with additions from (4). Please let me know of any errors, improvements, and fixes. I hope this walk-through might help others in the same situation.

It is days like today I am really happy to be a member of the Norwegian Unix User group, a member association for those of us believing in free software, open standards and unix-like operating systems. NUUG announced today it will try to bring the seizure of the DNS domain popcorn-time.no as unlawful, to stand up for the principle that writing about a controversial topic is not infringing copyrights, and censuring web pages by hijacking DNS domain should be decided by the courts, not the police. The DNS domain was seized by the Norwegian National Authority for Investigation and Prosecution of Economic and Environmental Crime a month ago. I hope this bring more paying members to NUUG to give the association the financial muscle needed to bring this case as far as it must go to stop this kind of DNS hijacking.

A few days ago a new upstream version "2.0" of CCTZ was released. See here for the corresponding post on the Google OpenSource Blog. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo. RcppCCTZ connects this library to R by relying on Rcpp. This new version adds more support to the notion of civil time representation -- i.e. independent of timezones -- which can then be mapped to different time zone representations. Changes in this version are summarized here:

Changes in version 0.0.4 (2016-04-17)

Synchronized with CCTZ v2 upstream.

Updated examples.cpp accordingly

A quick example is provided here where we look at the time when Neil Armstrong first stepped on the moon as an absolute ("civil") time and two local representations:

// from examples/hello.cc
// 
// [[Rcpp::export]]
int helloMoon()  
    cctz::time_zone syd;
    if (!cctz::load_time_zone("Australia/Sydney", &syd)) return -1;
    // Neil Armstrong first walks on the moon
    const auto tp1 = cctz::convert(cctz::civil_second(1969, 7, 21, 12, 56, 0), syd);
    const std::string s = cctz::format("%F %T %z", tp1, syd);
    Rcpp::Rcout << s << "\n";
    cctz::time_zone nyc;
    cctz::load_time_zone("America/New_York", &nyc);
    const auto tp2 = cctz::convert(cctz::civil_second(1969, 7, 20, 22, 56, 0), nyc);
    return tp2 == tp1 ? 0 : 1;

We can call this from R, and get the expected result (of equivalence between the dates):

R> library(RcppCCTZ)
R> helloMoon()
1969-07-21 12:56:00 +1000
[1] 0
R>

We also have a diff to the previous version thanks to CRANberries. More details, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Since mid-2015 we were using the dep11-generator in Debian to build AppStream metadata about available software components in the distribution. Getting rid of dep11-generator Unfortunately, the old Python-based dep11-generator was hitting some hard limits pretty soon. For example, using multiprocessing with Python was a pain, since it resulted in some very hard-to-track bugs. Also, the multiprocessing approach (as opposed to multithreading) made it impossible to use the underlying LMDB database properly (it was basically closed and reopened in each forked off process, since pickling the Python LMDB object caused some really funny bugs, which usually manifested themselves in the application hanging forever without any information on what was going on). Additionally to that, the Python-based generator forced me to maintain two implementations of the AppStream YAML spec, one in C and one in Python, which consumes quite some time. There were also some other issues (e.g. no unit-tests) in the implementation, which made me think about rewriting the generator. Adventures in Go / Rust / D Since I didn t want to write this new piece of software in C (or basically, writing it in C was my last option

), I explored Go and Rust for this purpose and also did a small prototype in the D programming language, when I was starting to feel really adventurous. And while I never intended to write the new generator in D (I was pretty fixated on Go ), this is what happened. The strong points for D for this particular project were its close relation to C (and ease of using existing C code), its super-flat learning curve for someone who knows and likes C and C++ and its pretty powerful implementations of the concurrent and parallel programming paradigms. That being said, not all is great in D and there are some pretty dark spots too, mainly when it comes to the standard library and compilers. I will dive into my experiences with D in a separate blogpost. What good to expect from appstream-generator? So, what can the new appstream-generator do for you? Basically, the same as the old dep11-generator: It will extract metadata from a distribution s package archive, download and resize screenshots, search for icons and size them properly and generate reports in JSON and HTML of found metadata and issues. LibAppStream-based parsing, generation of YAML or XML, multi-distro support, As opposed to the old generator, the new generator utilizes the metadata parsers and writers of libappstream. This allows it to return the extracted metadata as AppStream YAML (for Debian) or XML (everyone else) It is also written in a distribution-agnostic way, so if someone wants to use it in a different distribution than Debian, this is possible now. It just requires a very small distribution-specific backend to be written, all of the details of the metadata extraction are abstracted away (just two interfaces need to be implemented). While I do not expect anyone except Debian to use this in the near future (most distros have found a solution to generate metadata already), the frontend-backend split is a much cleaner design than what was available in the previous code. It also allows to unit-test the code properly, without providing a Debian archive in the testsuite. Feature Flags, Optipng, The new generator also allows to enable and disable certain sets of features in a standardized way. E.g. Ubuntu uses a language-pack system for translations, which Debian doesn t use. Features like this can be implemented as disableable separate modules in the generator. We use this at time to e.g. allow descriptions from packages to be used as AppStream descriptions, or for running optipng on the generated PNG images and icons. No more Contents file dependency Another issue the old generator had was that it used the Contents file from the Debian archive to find matching icons for an application. We could never be sure whether the contents in the Contents file actually matched the contents of the package we were currently dealing with. What made things worse is that at Ubuntu, the archive software is only updating the Contents file ~~weekly~~ daily (while the generator might run multiple times a day), which has lead to software being ignored in the metadata, because icons could not yet be found. Even on Debian, with its quickly-updated Contents file, we could immediately see the effects of an out-of-date Contents file when updating it failed once. In the new generator, we read the contents of each package ourselves now and store them in a LMDB database, bypassing the Contents file and removing the whole class of problems resulting from missing or wrong contents-data. It can t all be good, right? That is true, there are also some known issues the new generator has: Large amounts of RAM required The better speed of the new generator comes at the cost of holding more stuff in RAM. Much more. When processing data from 5 architectures initially on Debian, the amount of required RAM might lie above 4GB, with the OOM killer sometimes being quicker than the garbage collector That being said, on subsequent runs the amount of required memory is much lower. Still, this is something I am working on to improve. What are symbolic links? To be faster, the appstream-generator will read the md5sum file in .deb packages instead of extracting the payload archive and reading its contents. Since the md5sums file does not list symbolic links, symlinks basically don t exist for the new generator. This is a problem for software symlinking icons or even .desktop files around, like e.g. LibreOffice does. I am still investigating how widespread the use of symlinks for icons and .desktop files is, but it looks like fixing packages (making them not-symlink stuff and rather move the files) might be the better approach than investing additional computing power to find symlinks or even switch back to parsing the Contents file. Input on this is welcome! Deploying asgen I finished the last pieces of the appstream-generator (together with doing lots of other cool things and talking to great people) at the GNOME Software Hackfest in London last week (detailed blogposts about things that happened there will follow many thanks once again for the Ubuntu community for sponsoring my attendance!). Since today, the new generator is running on the Debian infrastructure. If bigger issues are found, we can still roll back to the old code. I decided to deploy this faster, so we can get some good testing done before the Stretch release. Please report any issues you may find!

Recently, Digital Ocean (which I am a customer of) asked me "how they were doing". Well, yet another survey... Let's ignore this one for now... I thought some days ago. And then yesterday, I added IPv6 support to my main mail server (which runs at Hetzner, Germany). All my hosted/rented/whatever systems report back to this main mailserver. Now that that main mail server finally has its AAAA record and its own IPv6 address, all associated systems try to reach this main mail server via IPv6. Of course. Crippling IPv6 support by adding Port Blocks But, then, I see messages like these in my syslog files on droplets hosted at Digital Ocean:

Apr 13 10:10:59 <do-droplet> postfix/smtp[23469]: connect to mail.<mydomain>[<ipv6-address>]:25: Connection timed out

After some more research [1], I realized that the folks out there at DO really apply some port blockings to IPv6 networks, but not to IPv4 networks. Pardon me? From my DO droplets, I can nmap any port on my mail server (25,80,143,443, 465, 587, etc.) via the IPv4 connection, but not over the IPv6 connection. Wait, not fully true: ports 80 and 443 are not blocked, but the other aforementioned ports are definitely blocked. Is Digital Ocean a professional ISP or a WiFi hotspot provider at my nearest coffee place? (This really makes me scratch my head...). Routing only the first 16 addresses of allocated /64 prefixes The above was the second IPv6 brokeness I learned about DO, recently. An earlier issue with DO's IPv6 support, I encountered while I was deploying an IPv6 capable OpenVPN internet gateway via a droplet hosted at DO. Digital Ocean assigns full IPv6 /64 prefixes to each individual droplet (which is great), but only properly routes the first 16 IP addresses of such a /64 prefix [2]. Urgh? I had to work around this flaw by adding an IPv6-over-IPv4 tunnel and attaching an IPv6 /56 prefix obtained from Hurricane Electrics' tunnel broker service [3] to the OpenVPN server. Thanks, Digital Ocean, for reminding me about giving feedback So, today, I luckily received a reminder mail about DO's yet-another-survey survey. My opportunity!!! Here is the feedback, I gave:

DO service is basically good.
BUT: You as a provider SUCK when it comes to IPv6.
(1) http://pixelschatten.net/blocked-ipv6-ports/
-> SMTP/IMAP traffic blocked over IPv6, but not over IPv4... WTF?). I normally have all my systems report back to my main mail server. I expect this to work as it is the default on all Linux hosts nowadays, and that is: prefer IPv6 over IPv4.
(2) https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestion...
-> Droplets get a full /64 prefix assigned, but only the first 16 addresses (or such) get routed properly. WTF?
Please do your homework on IPv6 and don't cripple your service by offering crippled IPv6 support.
I tell people, DO is great, but their IPv6 support is broken-by-design. Let me know, once this is about to change.
Mike Gabriel (aka sunweaver at debian dot org, Debian Developer)

Apology for the tone of the wording Now reading the feedback given, I realize that my tone has been quite impolite. I am sorry about this. However, the experienced IPv6 issues are indeed annoying. So please excuse me for having expressed my annoyance with such harsh words. And... I am still annoyed about myself paying an ISP for such a crippled IPv6 support. (I need to consider migrating the VMs to another hoster, unless there will be some dynamics observable in the near future). @Digital Ocean: Keep up the good work that you do in the realm of VM hosting. Evolve and grow up in the realm of IPv6 networking. Thank you! light+love
Mike [1] http://pixelschatten.net/blocked-ipv6-ports/
[2] https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestion...
[3] https://tunnelbroker.net/

What happened in the reproducible builds effort between March 20th and March 26th: Toolchain fixes

Sebastian Ramacher uploaded breathe/4.2.0-1 which makes its output deterministic. Original patch by Chris Lamb, merged upstream.
Rafael Laboissiere uploaded octave/4.0.1-1 which allows packages to be built in place and avoid unreproducible builds due to temporary build directories appearing in the .oct files.

Daniel Kahn Gillmor worked on removing build path from build symbols submitting a patch adding -fdebug-prefix-map to clang to match GCC, another patch against gcc-5 to backport the removal of -fdebug-prefix-map from DW_AT_producer, and finally by proposing the addition of a normalizedebugpath to the reproducible feature set of dpkg-buildflags that would use -fdebug-prefix-map to replace the current directory with . using -fdebug-prefix-map. Sergey Poznyakoff merged the --clamp-mtime option so that it will be featured in the next Tar release. This option is likely to be used by dpkg-deb to implement deterministic mtimes for packaged files. Packages fixed The following packages have become reproducible due to changes in their build dependencies: augeas, gmtkbabel, ktikz, octave-control, octave-general, octave-image, octave-ltfat, octave-miscellaneous, octave-mpi, octave-nurbs, octave-octcdf, octave-sockets, octave-strings, openlayers, python-structlog, signond. The following packages became reproducible after getting fixed:

atheme-services/7.0.7-1 by Antoine Beaupr .
boa-constructor/0.6.1-16 by Reiner Herrmann.
calculix-ccx/2.10-1 by Wolfgang F tterer, fixed upstream.
deap/1.0.2.post2-2 by Daniel Stender.
debian-keyring/2016.03.22 uploaded by Jonathan McDowell, fix by Niels Thykier, obsolete patch by Satyam Zode.
firefox-kwallet5/1.0-2 by Sandro Knau .
fox1.6/1.6.51-1 by Joachim Wiedorn.
gdnsd/2.2.3-1 by Faidon Liambotis.
glib2.0/2.48.0-1 uploaded by Iain Lane, original patch by Lunar, merged upstream.
jwm/2.3.4-1 uploaded by Samuel Henrique Oltramari Pinto, fix by Reiner Herrmann.
libasm4-java/5.0.4-2 by Emmanuel Bourg.
libjna-java/4.2.2-1 by Emmanuel Bourg.
libsynthesis/3.4.0.47.5+syncevolution-1.5.1-1 uploaded by Tino Mettler, patch by Reiner Herrmann.
logback/1:1.1.6-1 by Emmanuel Bourg.
nethack/3.6.0-2 uploaded by James Cowgill, patch by Reiner Herrmann.
php-htmlpurifier/4.7.0-2 by David Pr vot.
psocksxx/1.1.0-1 by Gianfranco Costamagna.
sbmltoolbox/4.1.0-3 uploaded by Afif Elghraoui, original patch by Reiner Herrmann.
spades/3.7.1+dfsg-2 by Sascha Steinbiss.
sprng/2.0a-10 uploaded by Dirk Eddelbuettel, original patch by Reiner Herrmann.

Some uploads fixed some reproducibility issues, but not all of them:

gle-graphics/4.2.5-2 by Christian T. Steigies.
kexec-tools/1:2.0.10-2 uploaded by Khalid Aziz, original patch by Lunar.
pdal/1.1.0-4 by Bas Couwenberg.
tcl8.5/8.5.19-2 uploaded by Sergei Golovan, original patch.
tcl8.6/8.6.5+dfsg-2 uploaded by Sergei Golovan, original patch.

Patches submitted which have not made their way to the archive yet:

#818742 on milkytracker by Reiner Herrmann: sorts the list of source files.
#818752 on tcl8.4 by Reiner Herrmann: sort source files using C locale.
#818753 on tk8.6 by Reiner Herrmann: sort source files using C locale.
#818754 on tk8.5 by Reiner Herrmann: sort source files using C locale.
#818755 on tk8.4 by Reiner Herrmann: sort source files using C locale.
#818952 on marionnet by ceridwen: dummy out build date and uname to make build reproducible.
#819334 on avahi by Reiner Herrmann: ship upstream changelog instead of the one generated by gettextize (although duplicate of #804141 by Santiago Vila).

tests.reproducible-builds.org i386 build nodes have been setup by converting 2 of the 4 amd64 nodes to i386. (h01ger) Package reviews 92 reviews have been removed, 66 added and 31 updated in the previous week. New issues: timestamps_generated_by_xbean_spring, timestamps_generated_by_mangosdk_spiprocessor. Chris Lamb filed 7 FTBFS bugs. Misc. On March 20th, Chris Lamb gave a talk at FOSSASIA 2016 in Singapore. The very same day, but a few timezones apart, h01ger did a presentation at LibrePlanet 2016 in Cambridge, Massachusetts. Seven GSoC/Outreachy applications were made by potential interns to work on various aspects of the reproducible builds effort. On top of interacting with several applicants, prospective mentors gathered to review the applications.

Started in 2008, lldpd is an implementation of IEEE 802.1AB-2005 (aka LLDP) written in C. While it contains some unit tests, like many other network-related software at the time, the coverage of those is pretty poor: they are hard to write because the code is written in an imperative style and tighly coupled with the system. It would require extensive mocking¹. While a rewrite (complete or iterative) would help to make the code more test-friendly, it would be quite an effort and it will likely introduce operational bugs along the way. To get better test coverage, the major features of lldpd are now verified through integration tests. Those tests leverage Linux network namespaces to setup a lightweight and isolated environment for each test. They run through pytest, a powerful testing tool.

pytest in a nutshell pytest is a Python testing tool whose primary use is to write tests for Python applications but is versatile enough for other creative usages. It is bundled with three killer features:

you can directly use the `assert` keyword,

you can inject fixtures in any test function, and

you can parametrize tests.

Assertions With unittest, the unit testing framework included with Python, and many similar frameworks, unit tests have to be encapsulated into a class and use the provided assertion methods. For example:

class testArithmetics(unittest.TestCase):
    def test_addition(self):
        self.assertEqual(1 + 3, 4)

The equivalent with pytest is simpler and more readable:

def test_addition():
    assert 1 + 3 == 4

pytest will analyze the AST and display useful error messages in case of failure. For further information, see Benjamin Peterson s article.

Fixtures A fixture is the set of actions performed in order to prepare the system to run some tests. With classic frameworks, you can only define one fixture for a set of tests:

class testInVM(unittest.TestCase):
    def setUp(self):
        self.vm = VM('Test-VM')
        self.vm.start()
        self.ssh = SSHClient()
        self.ssh.connect(self.vm.public_ip)
    def tearDown(self):
        self.ssh.close()
        self.vm.destroy()
    def test_hello(self):
        stdin, stdout, stderr = self.ssh.exec_command("echo hello")
        stdin.close()
        self.assertEqual(stderr.read(), b"")
        self.assertEqual(stdout.read(), b"hello\n")

In the example above, we want to test various commands on a remote VM. The fixture launches a new VM and configure an SSH connection. However, if the SSH connection cannot be established, the fixture will fail and the tearDown() method won t be invoked. The VM will be left running. Instead, with pytest, we could do this:

@pytest.yield_fixture
def vm():
    r = VM('Test-VM')
    r.start()
    yield r
    r.destroy()
@pytest.yield_fixture
def ssh(vm):
    ssh = SSHClient()
    ssh.connect(vm.public_ip)
    yield ssh
    ssh.close()
def test_hello(ssh):
    stdin, stdout, stderr = ssh.exec_command("echo hello")
    stdin.close()
    stderr.read() == b""
    stdout.read() == b"hello\n"

The first fixture will provide a freshly booted VM. The second one will setup an SSH connection to the VM provided as an argument. Fixtures are used through dependency injection: just give their names in the signature of the test functions and fixtures that need them. Each fixture only handle the lifetime of one entity. Whatever a dependent test function or fixture succeeds or fails, the VM will always be finally destroyed.

Parameters If you want to run the same test several times with a varying parameter, you can dynamically create test functions or use one test function with a loop. With pytest, you can parametrize test functions and fixtures:

@pytest.mark.parametrize("n1, n2, expected", [
    (1, 3, 4),
    (8, 20, 28),
    (-4, 0, -4)])
def test_addition(n1, n2, expected):
    assert n1 + n2 == expected

Testing lldpd The general plan for to test a feature in lldpd is the following:

Setup two namespaces.
Create a virtual link between them.
Spawn a lldpd process in each namespace.
Test the feature in one namespace.
Check with lldpcli we get the expected result in the other.

Here is a typical test using the most interesting features of pytest:

@pytest.mark.skipif('LLDP-MED' not in pytest.config.lldpd.features,
                    reason="LLDP-MED not supported")
@pytest.mark.parametrize("classe, expected", [
    (1, "Generic Endpoint (Class I)"),
    (2, "Media Endpoint (Class II)"),
    (3, "Communication Device Endpoint (Class III)"),
    (4, "Network Connectivity Device")])
def test_med_devicetype(lldpd, lldpcli, namespaces, links,
                        classe, expected):
    links(namespaces(1), namespaces(2))
    with namespaces(1):
        lldpd("-r")
    with namespaces(2):
        lldpd("-M", str(classe))
    with namespaces(1):
        out = lldpcli("-f", "keyvalue", "show", "neighbors", "details")
        assert out['lldp.eth0.lldp-med.device-type'] == expected

First, the test will be executed only if lldpd was compiled with LLDP-MED support. Second, the test is parametrized. We will execute four distinct tests, one for each role that lldpd should be able to take as an LLDP-MED-enabled endpoint. The signature of the test has four parameters that are not covered by the parametrize() decorator: lldpd, lldpcli, namespaces and links. They are fixtures. A lot of magic happen in those to keep the actual tests short:

lldpd is a factory to spawn an instance of lldpd. When called, it will setup the current namespace (setting up the chroot, creating the user and group for privilege separation, replacing some files to be distribution-agnostic, ), then call lldpd with the additional parameters provided. The output is recorded and added to the test report in case of failure. The module also contains the creation of the pytest.config.lldpd object that is used to record the features supported by lldpd and skip non-matching tests. You can read fixtures/programs.py for more details.
lldpcli is also a factory, but it spawns instances of lldpcli, the client to query lldpd. Moreover, it will parse the output in a dictionary to reduce boilerplate.
namespaces is one of the most interesting pieces. It is a factory for Linux namespaces. It will spawn a new namespace or refer to an existing one. It is possible to switch from one namespace to another (with with) as they are contexts. Behind the scene, the factory maintains the appropriate file descriptors for each namespace and switch to them with setns(). Once the test is done, everything is wipped out as the file descriptors are garbage collected. You can read fixtures/namespaces.py for more details. It is quite reusable in other projects².
links contains helpers to handle network interfaces: creation of virtual ethernet link between namespaces, creation of bridges, bonds and VLAN, etc. It relies on the pyroute2 module. You can read fixtures/network.py for more details.

You can see an example of a test run on the Travis build for 0.9.2. Since each test is correctly isolated, it s possible to run parallel tests with pytest -n 10 --boxed. To catch even more bugs, both the address sanitizer (ASAN) and the undefined behavior sanitizer (UBSAN) are enabled. In case of a problem, notably a memory leak, the faulty program will exit with a non-zero exit code and the associated test will fail.

A project like cwrap would definitely help. However, it lacks support for Netlink and raw sockets that are essential in lldpd operations.
There are three main limitations in the use of namespaces with this fixture. First, when creating a user namespace, only root is mapped to the current user. With lldpd, we have two users (root and _lldpd). Therefore, the tests have to run as root. The second limitation is with the PID namespace. It s not possible for a process to switch from one PID namespace to another. When you call setns() on a PID namespace, only children of the current process will be in the new PID namespace. The PID namespace is convenient to ensure everyone gets killed once the tests are terminated but you must keep in mind that /proc must be mounted in children only. The third limitation is that, for some namespaces (PID and user), all threads of a process must be part of the same namespace. Therefore, don t use threads in tests. Use multiprocessing module instead.

Let's see if I can scrounge through all of my now-organized directories of ebooks and figure out what I haven't recorded here yet. At least the paper books make that relatively easy, since I don't shelve them until I post them. (Yeah, yeah, I should actually make a database.) Hugh Aldersey-Williams Periodic Tales (nonfiction)
Sandra Ulbrich Almazan SF Women A-Z (nonfiction)
Radley Balko Rise of the Warrior Cop (nonfiction)
Peter V. Brett The Warded Man (sff)
Lois McMaster Bujold Gentleman Jole and the Red Queen (sff)
Fred Clark The Anti-Christ Handbook Vol. 2 (nonfiction)
Dave Duncan West of January (sff)
Karl Fogel Producing Open Source Software (nonfiction)
Philip Gourevitch We Wish to Inform You That Tomorrow We Will Be Killed With Our Families (nonfiction)
Andrew Groen Empires of EVE (nonfiction)
John Harris @ Play (nonfiction)
David Hellman & Tevis Thompson Second Quest (graphic novel)
M.C.A. Hogarth Earthrise (sff)
S.L. Huang An Examination of Collegial Dynamics... (sff)
S.L. Huang & Kurt Hunt Up and Coming (sff anthology)
Kameron Hurley Infidel (sff)
Kevin Jackson-Mead & J. Robinson Wheeler IF Theory Reader (nonfiction)
Rosemary Kirstein The Lost Steersman (sff)
Rosemary Kirstein The Language of Power (sff)
Merritt Kopas Videogames for Humans (nonfiction)
Alisa Krasnostein & Alexandra Pierce (ed.) Letters to Tiptree (nonfiction)
Mathew Kumar Exp. Negatives (nonfiction)
Ken Liu The Grace of Kings (sff)
Susan MacGregor The Tattooed Witch (sff)
Helen Marshall Gifts for the One Who Comes After (sff collection)
Jack McDevitt Coming Home (sff)
Seanan McGuire A Red-Rose Chain (sff)
Seanan McGuire Velveteen vs. The Multiverse (sff)
Seanan McGuire The Winter Long (sff)
Marc Miller Agent of the Imperium (sff)
Randal Munroe Thing Explainer (graphic nonfiction)
Marguerite Reed Archangel (sff)
J.K. Rowling Harry Potter: The Complete Collection (sff)
K.J. Russell Tides of Possibility (sff anthology)
Robert J. Sawyer Starplex (sff)
Bruce Schneier Secrets & Lies (nonfiction)
Mike Selinker (ed.) The Kobold Game to Board Game Design (nonfiction)
Douglas Smith Chimerascope (sff collection)
Jonathan Strahan Fearsome Journeys (sff anthology)
Nick Suttner Shadow of the Colossus (nonfiction)
Aaron Swartz The Boy Who Could Change the World (essays)
Caitlin Sweet The Pattern Scars (sff)
John Szczepaniak The Untold History of Japanese Game Developers I (nonfiction)
John Szczepaniak The Untold History of Japanese Game Developers II (nonfiction)
Jeffrey Toobin The Run of His Life (nonfiction)
Hayden Trenholm Blood and Water (sff anthology)
Coen Teulings & Richard Baldwin (ed.) Secular Stagnation (nonfiction)
Ursula Vernon Book of the Wombat 2015 (graphic nonfiction)
Ursula Vernon Digger (graphic novel) Phew, that was a ton of stuff. A bunch of these were from two large StoryBundle bundles, which is a great source of cheap DRM-free ebooks, although still rather hit and miss. There's a lot of just fairly random stuff that's been accumulating for a while, even though I've not had a chance to read very much. Vacation upcoming, which will be a nice time to catch up on reading.

Also titled minimal images - maximal effort! The OpenStack Infrastructure Team manages a large continuous-integration system that provides the broad range of testing the OpenStack project requires. Tests are run thousands of times a day across every project, on multiple platforms and on multiple cloud-providers. There are essentially no manual steps in any part of the process, with every component being automated via scripting, a few home-grown tools and liberal doses of Puppet and Ansible. More importantly, every component resides in the public git trees right alongside every other OpenStack project, with contributions actively encouraged. As with any large system, technical debt can build up and start to affect stability and long-term maintainability. OpenStack Infrastructure can see some of this debt accumulating as more testing environments across more cloud-providers are being added to support ever-growing testing demands. Thus a strong focus of recent work has been consolidating testing platforms to be smaller, better defined and more maintainable. This post illustrates some of the background to the issues and describes how these new platforms are more reliable and maintainable.

OpenStack CI Overview Before getting into details, it's a good idea to get a basic big-picture conceptual model of how OpenStack CI testing works. If you look at the following diagram and follow the numbers with the explanation below, hopefully you'll have all the context you need. Overview of OpenStack CI

The developer uploads their code to gerrit via the git-review tool. There is no further action required on their behalf and the developer simply waits for results.
Gerrit provides a JSON-encoded "firehose" output of everything happening to it. New reviews, votes, updates and more all get sent out over this pipe. Zuul is the overall scheduler that subscribes itself to this information and is responsible for managing the CI jobs appropriate for each change.
Zuul has a configuration that tells it what jobs to run for what projects. Zuul can do lots of interesting things, but for the purposes of this discussion we just consider that it puts the jobs it wants run into gearman for a Jenkins master to consume. gearman is a job-server; as they explain it "[gearman] provides a generic application framework to farm out work to other machines or processes that are better suited to do the work". Zuul puts into gearman basically a tuple (job-name, node-type) for each job it wants run, specifying the unique job name to run and what type of node it should be run on.
A group of Jenkins masters are subscribed to gearman as workers. It is these Jenkins masters that will consume the job requests from the queue and actually get the tests running. However, Jenkins needs two things to be able to run a job a job definition (what to actually do) and a slave node (somewhere to do it). The first part what to do is provided by job-definitions stored in external YAML files and processed by Jenkins Job Builder (jjb) in to job configurations for Jenkins. Each Jenkins master gets these definitions pushed to it constantly by Puppet, thus each Jenkins master instance knows about all the jobs it can run automatically. Zuul also knows about these job definitions; this is the job-name part of the tuple we said it put into gearman. The second part somewhere to run the test takes some more explaining. To the next point...
Several cloud companies donate capacity in their clouds for OpenStack to run CI tests. Overall, this capacity is managed by a customised orchestration tool called nodepool. Nodepool watches the gearman queue and sees what requests are coming out of Zuul. It looks at node-type of jobs in the queue and decides what types of nodes need to start and which cloud providers have capacity to satisfy demand. Nodepool will monitor the start-up of the virtual-machines and register the new nodes to the Jenkins master instances.
At this point, the Jenkins master has what it needs to actually get jobs started. When nodepool registers a host to a Jenkins master as a slave, the Jenkins master can now advertise its ability to consume jobs. For example, if a ubuntu-trusty node is provided to the Jenkins master instance by nodepool, Jenkins can now consume from gearman any job it knows about that is intended to run on an ubuntu-trusty slave. Jekins will run the job as defined in the job-definition on that host ssh-ing in, running scripts, copying the logs and waiting for the result. (It is a gross oversimplification, but for the purposes of OpenStack CI, Jenkins is pretty much used as a glorified ssh/scp wrapper. Zuul Version 3, under development, is working to remove the need for Jenkins to be involved at all).
Eventually, the test will finish. The Jenkins master will put the result back into gearman, which Zuul will consume. The slave will be released back to nodepool, which destroys it and starts all over again (slaves are not reused and also have no sensitive details on them, as they are essentially publicly accessible). Zuul will wait for the results of all jobs for the change and post the result back to Gerrit; it either gives a positive vote or the dreaded negative vote if required jobs failed (it also handles merges to git, but we'll ignore that bit for now).

In a nutshell, that is the CI work-flow that happens thousands-upon-thousands of times a day keeping OpenStack humming along.

Image builds So far we have glossed over how nodepool actually creates the images that it hands out for testing. Image creation, illustrated in step 8 above, contains a lot of important details. Firstly, what are these images and why build them at all? These images are where the "rubber hits the road" they are instantiated into the virtual-machines that will run DevStack, unit-testing or whatever else someone might want to test. The main goal is to provide a stable and consistent environment in which to run a wide-range of tests. A full OpenStack deployment results in hundreds of libraries and millions of lines of code all being exercised at once. The testing-images are right at the bottom of all this, so any instability or inconsistency affects everyone; leading to constant fire-firefighting and major inconvenience as all forward-progress stops when CI fails. We want to support a wide number of platforms interesting to developers such as Ubuntu, Debian, CentOS and Fedora, and we also want to and make it easy to handle new releases and add other platforms. We want to ensure this can be maintained without too much day-to-day hands-on. Caching is a big part of the role of these images. With thousands of jobs going on every day, an occasional network blip is not a minor annoyance, but creates constant and difficult to debug failures. We want jobs to rely on as few external resources as possible so tests are consistent and stable. This means caching things like the git trees tests might use (OpenStack just broke the 1000 repository mark), VM images, packages and other common bits and pieces. Obviously a cache is only as useful as the data in it, so we build these images up every day to keep them fresh.

Snapshot images If you log into almost any cloud-provider's interface, they almost certainly have a range of pre-canned images of common distributions for you to use. At first, the base images for OpenStack CI testing came from what the cloud-providers had as their public image types. However, over time, there are a number of issues that emerge:

No two images, even for the same distribution or platform, are the same. Every provider seems to do something "helpful" to the images which requires some sort of workaround.
Providers rarely leave these images alone. One day you would boot the image to find a bunch of Python libraries pip-installed, or a mount-point moved, or base packages removed (all happened).
Even if the changes are helpful, it does not make for consistent and reproducible testing if every time you run, you're on a slightly different base system.
Providers don't have some images you want (like a latest Fedora), or have different versions, or different point releases. All update asynchronously whenever they get around to it.

So the original incarnations of OpenStack CI images were based on these public images. Nodepool would start one of these provider images and then run a series of scripts on it these scripts would firstly try to work-around any quirks to make the images look as similar as possible across providers, and then do the caching, setup things like authorized keys and finish other configuration tasks. Nodepool would then snapshot this prepared image and start instantiating VM's from these snapshots into the pool for testing. If you hear someone talking about a "snapshot image" in OpenStack CI context, that's likely what they are referring to. Apart from the stability of the underlying images, the other issue you hit with this approach is that the number of images being built starts to explode when you take into account multiple providers and multiple regions. Even with just Rackspace and the (now defunct) HP Cloud we would end up creating snapshot images for 4 or 5 platforms across a total of about 8 regions meaning anywhere up to 40 separate image builds happening daily (you can see how ridiculous it was getting in the logging configuration used at the time). It was almost a fait accompli that some of these would fail every day nodepool can deal with this by reusing old snapshots but this leads to a inconsistent and heterogeneous testing environment. Naturally there was a desire for something more consistent a single image that could run across multiple providers in a much more tightly controlled manner.

Upstream-based builds Upstream distributions do provide "cloud-images", which are usually pre-canned .qcow2 format files suitable for uploading to your average cloud. So the diskimage-builder tool was put into use creating images for nodepool, based on these upstream-provided images. In essence, diskimage-builder uses a series of elements (each, as the name suggests, designed to do one thing) that allow you to build a completely customised image. It handles all the messy bits of laying out the image file, tries to be smart about caching large downloads and final things like conversion to qcow2 or vhd. nodepool has used diskimage-builder to create customised images based upon the upstream releases for some time. These are better, but still have some issues for the CI environment:

You still really have no control over what does or does not go into the upstream base images. You don't notice a change until you deploy a new image based on an updated version and things break.
The images still start with a fair amount of "stuff" on them. For example cloud-init is a rather large Python program and has a fair few dependencies. These dependencies can both conflict with parts of OpenStack or end up tacitly hiding real test requirements (the test doesn't specify it, but the package is there as part of another base dependency. Things then break when the base dependencies change). The whole idea of the CI is that (as much as possible) you're not making any assumptions about what is required to run your tests you want everything explicitly included.
An image that "works everywhere" across multiple cloud-providers is quite a chore. cloud-init hasn't always had support for config-drive and Rackspace's DHCP-less environment, for example. Providers all have their various different networking schemes or configuration methods which needs to be handled consistently.

If you were starting this whole thing again, things like LXC/Docker to keep "systems within systems" might come into play and help alleviate some of the packaging conflicts. Indeed they may play a role in the future. But don't forget that DevStack, the major CI deployment mechanism, was started before Docker existed. And there's tricky stuff with networking and Neutron going on. And things like iSCSI kernel drivers that containers don't support well. And you need to support Ubuntu, Debian, CentOS and Fedora. And you have hundreds of developers already relying on what's there. So change happens incrementally, and in the mean time, there is a clear need for a stable, consistent environment.

Minimal builds To this end, diskimage-builder now has a serial of "minimal" builds that are really that systems with essentially nothing on them. For Debian and Ubuntu this is achieved via debootstrap, for Fedora and CentOS we replicate this with manual installs of base packages into a clean chroot environment. We add on a range of important elements that make the image useful; for example, for networking, we have simple-init which brings up the network consistently across all our providers but has no dependencies to mess with the base system. If you check the elements provided by project-config you can see a range of specific elements that OpenStack Infra runs at each image build (these are actually specified by in arguments to nodepool, see the config file, particularly diskimages section). These custom elements do things like caching, using puppet to install the right authorized_keys files and setup a few needed things to connect to the host. In general, you can see the logs of an image build provided by nodepool for each daily build. So now, each day at 14:14 UTC nodepool builds the daily images that will be used for CI testing. We have one image of each type that (theoretically) works across all our providers. After it finishes building, nodepool uploads the image to all providers (p.s. the process of doing this is so insanely terrible it spawned shade; this deserves many posts of its own) at which point it will start being used for CI jobs. If you wish to replicate this entire process, the build-image.sh script, run on an Ubuntu Trusty host in a virtualenv with diskimage-builder will get you pretty close (let us know of any issues!).

DevStack and bare nodes There are two major ways OpenStack projects test their changes:

Running with DevStack, which brings up a small, but fully-functional, OpenStack cloud with the change-under-test applied. Generally tempest is then used to ensure the big-picture things like creating VM's, networks and storage are all working.
Unit-testing within the project; i.e. what you do when you type tox -e py27 in basically any OpenStack project.

To support this testing, OpenStack CI ended up with the concept of bare nodes and devstack nodes.

A bare node was made for unit-testing. While tox has plenty of information about installing required Python packages into the virtualenv for testing, it doesn't know anything about the system packages required to build those Python packages. This means things like gcc and library -devel packages which many Python packages use to build bindings. Thus the bare nodes had an ever-growing and not well-defined list of packages that were pre-installed during the image-build to support unit-testing. Worse still, projects didn't really know their dependencies but just relied on their testing working with this global list that was pre-installed on the image.
In contrast to this, DevStack has always been able to bootstrap itself from a blank system to a working OpenStack deployment by ensuring it has the right dependencies installed. We don't want any packages pre-installed here because it hides actual dependencies that we want explicitly defined within DevStack otherwise, when a user goes to deploy DevStack for their development work, things break because their environment differs slightly to the CI one. If you look at all the job definitions in OpenStack, by convention any job running DevStack has a dsvm in the job name this referred to running on a "DevStack Virtual Machine" or a devstack node. As the CI environment has grown, we have more and more testing that isn't DevStack based (puppet apply tests, for example) that rather confusingly want to run on a devstack node because they do not want dependencies installed. While it's just a name, it can be difficult to explain!

Thus we ended up maintaining two node-types, where the difference between them is what was pre-installed on the host and yes, the bare node had more installed than a devstack node, so it wasn't that bare at all!

Specifying Dependencies Clearly it is useful to unify these node types, but we still need to provide a way for the unit-test environments to have their dependencies installed. This is where a tool called bindep comes in. This tool gives project authors a way to specify their system requirements in a similar manner to the way their Python requirements are kept. For example, OpenStack has the concept of global requirements those Python dependencies that are common across all projects so version skew becomes somewhat manageable. This project now has some extra information in the other-requirements.txt file, which lists the system packages required to build the Python packages in the global-requirements list. bindep knows how to look at these lists provided by projects and get the right packages for the platform it is running on. As part of the image-build, we have a cache-bindep element that can go through every project and build a list of the packages it requires. We can thus pre-cache all of these packages onto the images, knowing that they are required by jobs. This both reduces the dependency on external mirrors and improves job performance (as the packages are locally cached) but doesn't pollute the system by having everything pre-installed. Package installation can now happen via the way we really should be doing it as part of the CI job. There is a job-macro called install-distro-packages which a test can use to call bindep to install the packages specified by the project before the run. You might notice the script has a "fallback" list of packages if the project does not specify it's own dependencies this essentially replicates the environment of a bare node as we transition to projects more strictly specifying their system requirements. We can now start with a blank image and all the dependencies to run the job can be expressed by and within the project leading to a consistent and reproducible environment without any hidden dependencies. Several things have broken as part of removing bare nodes this is actually a good thing because it means we have revealed areas where we were making assumptions in jobs about what the underlying platform provides. There's a few other job-macros that can do things like provide MySQL/Postgres instances for testing or setup other common job requirements. By splitting these types of things out from base-images we also improve the performance of jobs who don't waste time doing things like setting up databases for jobs that don't need it. As of this writing, the bindep work is new and still a work-in-progress. But the end result is that we have no more need for a separate bare node type to run unit-tests. This essentially halves the number of image-builds required and brings us to the goal of a single image for each platform running all CI.

Conclusion While dealing with multiple providers, image-types and dependency chains has been a great effort for the infra team, to everyone's credit I don't think the project has really noticed much going on underneath. OpenStack CI has transitioned to a situation where there is a single image type for each platform we test that deploys unmodified across all our providers and runs all testing environments equally. We have better insight into our dependencies and better tools to manage them. This leads to greatly decreased maintenance burden, better consistency and better performance; all great things to bring to OpenStack CI!

A large part of OpenStack Infrastructure teams recent efforts has been focused on moving towards more stable and maintainable CI environments for testing.

The developer uploads their code to gerrit via the git-review tool. They wait.
Gerrit provides a JSON-encoded "firehose" output of everything happening to it. New reviews, votes, updates and more all get sent out over this pipe. Zuul is the overall scheduler that subscribes itself to this information and is responsible for managing the CI jobs appropriate for each change.
Zuul has a configuration that tells it what jobs to run for what projects. Zuul can do lots of interesting things, but for the purposes of this discussion we just consider that it puts the jobs it wants run into gearman for a Jenkins host to consume. gearman is a job-server; as they explain it "[gearman] provides a generic application framework to farm out work to other machines or processes that are better suited to do the work".
A group of Jenkins hosts are subscribed to gearman as workers. It is these Jenkins hosts that will consume the job requests from the queue and actually get the tests running. Jenkins needs two things to be able to run a job -- a job definition (what to actually do) and a slave node (somewhere to do it). The first part -- what to do -- is provided by job-definitions stored in external YAML files and processed by Jenkins Job Builder (jjb) in to job configurations for Jenkins. Thus each Jenkins instance knows about all the jobs it might need to run. Zuul also knows about these job definitions, so you can see how we now have a mapping where Zuul can put a job into gearman saying "run test foo-bar-baz" and a Jenkins host can consume that request and know what to do. The second part -- somewhere to run the test -- takes some more explaining. To the next point...
Several cloud companies donate capacity in their clouds for OpenStack to run CI tests. Overall, this capacity is managed by nodepool -- a customised orchestration tool. Nodepool watches the gearman queue and sees what requests are coming out of Zuul, and decides what type of capacity to provide and in what clouds to satisfy the outstanding job queue. Nodepool will start-up virtual-machines as required, and register those nodes to the Jenkins instances.
At this point, Jenkins has what it needs to actually get jobs started. When nodepool registers a host to Jenkins as a slave, the Jenkins host can now advertise its ability to consume jobs. For example, if a ubuntu-trusty node is provided to the Jenkins instance by nodepool, Jenkins can now consume a job from the queue intended to run on an ubuntu-trusty host. It will run the job as defined in the job-definition -- ssh-ing into the host, running scripts, copying the logs and waiting for the result. (It is a gross oversimplification, but but Jenkins is pretty much a glorified ssh/scp wrapper to OpenStack CI. Zuul Version 3, under development, is working to remove the need for Jenkins to be involved at all).
Eventually, the test will finish. Jenkins will put the result back into gearman, which Zuul will consume. The slave will be released back to nodepool, which destroys it and starts all over again (slaves are not reused and also have no sensitive details on them, as they are essentially publicly accessible). Zuul will wait for the results of all jobs and post the result back to Gerrit and give either a positive vote or the dreaded negative vote if required jobs failed (it also handles merges to git, but we'll ignore that bit for now).

In a nutshell, that is the CI work-flow that happens thousands-upon-thousands of times a day keeping OpenStack humming along.

Image builds There is, however, another more asynchronous part of the process that hides a lot of details the rest of the system relies on. Illustrated in step 8 above, this is the management of the images that tests are being run upon. Above we we said that a test runs on a ubuntu-trusty, centos, fedora or some other type of node, but glossed over where these images come from. Firstly, what are these images, and why build them at all? These images are where the "rubber hits the road" -- where DevStack, functional testing or whatever else someone might want to test is actually run for real. Caching is a big part of the role of these images. With thousands of jobs going on every day, an occasional network blip is not a minor annoyance, but creates constant and difficult to debug CI failures. We want the images that CI runs on to rely on as few external resources as possible so test runs are as stable as possible. This means caching all the git trees tests might use, VM images consumed by various tests and other common bits and pieces. Obviously a cache is only as useful as the data in it, so we build these images up every day to keep them fresh.

Provider images If you log into almost any cloud-provider's interface, they almost certainly have a range of pre-canned images of common distributions for you to use. At first, the base images for OpenStack CI testing came from what the cloud-providers had as their public image types. However, over time, there are a number of issues that emerge:

No two images, even for the same distribution or platform, are the same. Every provider seems to do something "helpful" to the images which requires some sort of workaround.
Providers rarely leave these images alone. One day you would boot the image to find a bunch of Python libraries pip-installed, or a mount-point moved, or base packages removed (all happened).
Even if the changes are helpful, it does not make for consistent and reproducible testing if every time you run, you're on a slightly different base system.
Providers don't have some images you want (like a latest Fedora), or have different versions, or different point releases. All update asynchronously whenever they get around to it.

So the original incarnations of building images was that nodepool would start one of these provider images, run a bunch of scripts on it to make a base-image (do the caching, setup keys, etc), snapshot it and then start putting VM's based on these images into the pool for testing (if you hear someone talking about a "snapshot image" in OpenStack CI context, that's what it means). The first problem you hit here is that the number of images being built starts to explode when you take into account multiple providers and multiple regions. Even with just Rackspace and the (now defunct) HP Cloud we would end up snapshot images for 4 or 5 platforms across a total of about 8 regions -- meaning anywhere up to 40 separate image builds happening. It was almost a fait accompli that some images would fail every day -- nodepool can deal with this by reusing old snapshots; but this leads to a inconsistent and heterogeneous testing environment. OpenStack is like a gigantic Jenga tower, with a full DevStack deployment resulting in hundreds of libraries and millions of lines of code all being exercised at once. The testing images are right at the bottom of all this, and it doesn't take much to make the whole thing fall over (see points about providers not leaving images alone). This leads to constant fire-firefighting and everyone annoyed as all CI stops. Naturally there was a desire for something much more consistent -- a single image that could run across multiple providers in a much more tightly controlled manner.

Upstream-based builds Upstream distributions do provide their "cloud-images", which are usually pre-canned .qcow2 format files suitable for uploading to your average cloud. So the diskimage-builder tool was put into use creating images for nodepool, based on these upstream-provided images. In essence, diskimage-builder uses a series of elements (each, as the name suggests, designed to do one thing) that allow you to build a completely customised image. It handles all the messy bits of laying out the image file, tries to be smart about caching large downloads and final things like conversion to qcow2 or vhd or whatever your cloud requires. nodepool has used diskimage-builder to create customised images based upon the upstream releases for some time. These are better, but still have some issues for the CI environment:

You still really have no control over what does or does not go into the upstream base images. You don't notice a change until you deploy a new image based on an updated version and things break.
The images still have a fair amount of "stuff" on them. For example cloud-init is a rather large Python program and has a fair few dependencies. These dependencies can both conflict with parts of OpenStack or end up tacitly hiding real test requirements (the test doesn't specify it, but the package is there as part of another base dependency -- things break when the base dependencies change). The whole idea of the CI is that (as much as possible) you're not making any assumptions about what is required to run your tests -- you want everything explicitly included. (If you were starting this whole thing again, things like Docker to keep "systems within systems" might come into play. Indeed they may be in the future. But don't forget that DevStack, the major CI deployment mechanism, was started before Docker existed. And there's tricky stuff with networking and Neutron etc going on).
An image that "works everywhere" across multiple cloud-providers is quite a chore. cloud-init hasn't always had support for config-drive and Rackspace's DHCP-less environment, for example. Providers all have their various different networking schemes or configuration methods which needs to be handled consistently.

Minimal builds To this end, diskimage-builder now has a serial of "minimal" builds that are really that -- systems with essentially nothing on them. For Debian and Ubuntu, this is achieved via debootstrap, for Fedora and CentOS we replicate this with manual installs of base packages into a clean chroot environment. We add-on a range of important elements that make the image useful; for example, for networking, we have simple-init which brings up the network consistently across all our providers but has no dependencies to mess with the base system. If you check the elements provided by project-config you can see a range of specific elements that OpenStack Infra runs at each image build (these are actually specified by in arguments to nodepool, see the config file, particularly diskimages section). These custom elements do things like caching, using puppet to install the right authorized_keys files and setup a few needed things to connect to the host. In general, you can see the logs of an image build provided by nodepool for each daily build. So now, each day at 14:00 UTC nodepool builds the daily images that will be used for CI testing. We have one image of each type that (theoretically) works across all our providers. After it finishes building, nodepool uploads the image to all providers (p.s. the process of doing this is so insanely terrible it spawned shade; this deserves many posts of its own) at which point it will start being used for CI jobs. If you wish to replicate this entire process, the build-image.sh script, run on an Ubuntu Trusty host in a virtualenv with diskimage-builder will get you pretty close (let us know of any issues!).

Dependencies But guess what, there's more! Along the way, OpenStack CI ended up with the concept of bare nodes and devstack nodes. A bare node was one that was used for functional testing; i.e. what you do when you type tox -e py27 in basically any OpenStack project. The problem here is that tox has plenty of information about installing required Python packages into the virtualenv for testing; but it doesn't know anything about the system packages required to build the Python libraries. This means things like gcc and -devel packages which many Python libraries use to build library bindings. In contrast to this, DevStack has always been able to bootstrap itself from a blank system to a working OpenStack deployment, ensuring it has the right libraries, etc, installed to get everything working. If you look at all the job definitions, anything running DevStack has a dsvm in the job name; which referred to to "DevStack Virtual Machine"; or basically specifying what was installed on the host (and yes, the bare node had more installed, so it wasn't that bare at all!). Thus we don't want packages pre-installed for DevStack, because it hides actual devstack dependencies that we want explicitly defined. But the bare nodes, used for functional testing, were different -- there was an every-growing and not well-defined list of packages that were pre-installed on those nodes to make sure functional testing worked. In general, you don't want jobs relying on something like this; we want to be sure if jobs have a dependency, they require it explicitly. This is where a tool called bindep comes in. OpenStack has the concept of global requirements -- those Python dependencies that are common across all projects so version skew becomes somewhat manageable. This now has some extra information in the other-requirements.txt file, which lists the system-packages required to build the Python-packages in the requirements list. bindep knows how to look at this and get the right packages for the platform it is running on. Indeed -- remember how it was previously mentioned we want to minimise dependencies on external resources at runtime? Well we can pre-cache all of these packages onto the images, knowing that they are likely to be required by packages. How do we get the packages installed? The way we really should be doing it -- as part of the CI job. There is a macro called install-distro-packages which uses bindep to install those packages as required by the global-requirements list. The result -- no more need for the bare node type to run functional tests! In all cases we can start with essentially a blank image and all the dependencies to run the job are expressed by and within the job -- leading to a consistent and reproducible environment. Several things have broken as part of removing bare nodes -- this is actually a good thing because it means we have revealed areas where we were making assumptions in jobs about what the underlying platform provides; issues that get fixed by thinking about and ensuring we have correct dependencies bringing up jobs. There's a few other macros there that do things like provide MySQL/Postgres instances or setup other common job requirements. By splitting these out we also improve the performance of jobs who now only bring in the dependencies they need -- we don't waste time doing things like setting up databases for jobs that don't need it.

Conclusion While dealing with multiple providers, image-types and dependency chains has been a great effort for the infra team, to everyone's credit I don't think the project has really noticed much going on underneath. OpenStack CI has transitioned to a situation where there is a single image type for each platform we test that deploys unmodified across all our providers. We have better insight into our dependencies and better tools to manage them. This leads to greatly decreased maintenance burden, better consistency and better performance; all great things to bring to OpenStack CI!

The paths of life to guide us towards the knowledge are many and a straight; well-defined line does not always represent them. Among these, the theory of Fuzzy Logic can help us to better understand some of the methods that maybe for too long we have lost the habit to use, if not even forgotten, because of the residuals of the positivist scientism.

Brief mathematical introduction

blur2 All the elements making up the majority of the categories of everyday life (feelings, emotions, opinions, values, quality, etc.) cannot be defined in a unique or exact way. Let s try to define an event that causes severe pain: a hammer blow on the finger? The loss of a loved one? Tooth extraction? It is evident that the answer can vary in a very consistent way, based on the subjectivity of the individual, the environment, boundary conditions and the general context. This is because the Strong Pain is a fuzzy set (which we will call SP), characterized by a function of degree of membership which maps the elements of a universe in a real continuous interval between 0 and 1. In practical terms

Firefox has had support for Google's Safe Browsing since 2005 when it started as a stand-alone Firefox extension. At first it was only available in the USA, but it was opened up to the rest of the world in 2006 and moved to the Google Toolbar. It then got integrated directly into Firefox 2.0 before the public launch of the service in 2007. Many people seem confused by this phishing and malware protection system and while there is a pretty good explanation of how it works on our support site, it doesn't go into technical details. This will hopefully be of interest to those who have more questions about it.

Browsing Protection The main part of the Safe Browsing system is the one that watches for bad URLs as you're browsing. Browsing protection currently protects users from:

malware sites,

deceptive sites (including phishing and social engineering sites), and

sites hosting potentially unwanted software.

If a Firefox user attempts to visit one of these sites, a warning page will show up instead, which you can see for yourself here:

fake malware page

fake unwanted software page

fake phishing page

The first two warnings can be toggled using the `browser.safebrowsing.malware.enabled` preference (in `about:config`) whereas the last one is controlled by `browser.safebrowsing.enabled`.

List updates It would be too slow (and privacy-invasive) to contact a trusted server every time the browser wants to establish a connection with a web server. Instead, Firefox downloads a list of bad URLs every 30 minutes from the server (`browser.safebrowsing.provider.google.updateURL`) and does a lookup against its local database before displaying a page to the user. Downloading the entire list of sites flagged by Safe Browsing would be impractical due to its size so the following transformations are applied:

each URL on the list is canonicalized,

then hashed,

of which only the first 32 bits of the hash are kept.

The lists that are requested from the Safe Browsing server and used to flag pages as malware/unwanted or phishing can be found in `urlclassifier.malwareTable` and `urlclassifier.phishTable` respectively. If you want to see some debugging information in your terminal while Firefox is downloading updated lists, turn on `browser.safebrowsing.debug`. Once downloaded, the lists can be found in the cache directory:

`~/.cache/mozilla/firefox/XXXX/safebrowsing/` on Linux

`~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/` on Mac

`C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\` on Windows

Resolving partial hash conflicts Because the Safe Browsing database only contains partial hashes, it is possible for a safe page to share the same 32-bit hash prefix as a bad page. Therefore when a URL matches the local list, the browser needs to know whether or not the rest of the hash matches the entry on the Safe Browsing list. In order resolve such conflicts, Firefox requests from the Safe Browsing server (`browser.safebrowsing.provider.mozilla.gethashURL`) all of the hashes that start with the affected 32-bit prefix and adds these full-length hashes to its local database. Turn on `browser.safebrowsing.debug` to see some debugging information on the terminal while these "completion" requests are made. If the current URL doesn't match any of these full hashes, the load proceeds as normal. If it does match one of them, a warning interstitial page is shown and the load is canceled.

Download Protection The second part of the Safe Browsing system protects users against malicious downloads. It was launched in 2011, implemented in Firefox 31 on Windows and enabled in Firefox 39 on Mac and Linux. It roughly works like this:

Download the file.

Check the main URL, referrer and redirect chain against a local blocklist (`urlclassifier.downloadBlockTable`) and block the download in case of a match.

On Windows, if the binary is signed, check the signature against a local whitelist (`urlclassifier.downloadAllowTable`) of known good publishers and release the download if a match is found.

If the file is not a binary file then release the download.

Otherwise, send the binary file's metadata to the remote application reputation server (`browser.safebrowsing.downloads.remote.url`) and block the download if the server indicates that the file isn't safe.

Blocked downloads can be unblocked by right-clicking on them in the download manager and selecting "Unblock". While the download protection feature is automatically disabled when malware protection (`browser.safebrowsing.malware.enabled`) is turned off, it can also be disabled independently via the `browser.safebrowsing.downloads.enabled` preference. Note that Step 5 is the only point at which any information about the download is shared with Google. That remote lookup can be suppressed via the `browser.safebrowsing.downloads.remote.enabled` preference for those users concerned about sending that metadata to a third party.

Types of malware The original application reputation service would protect users against "dangerous" downloads, but it has recently been expanded to also warn users about unwanted software as well as software that's not commonly downloaded. These various warnings can be turned on and off in Firefox through the following preferences:

`browser.safebrowsing.downloads.remote.block_dangerous`

`browser.safebrowsing.downloads.remote.block_dangerous_host`

`browser.safebrowsing.downloads.remote.block_potentially_unwanted`

`browser.safebrowsing.downloads.remote.block_uncommon`

and tested using Google's test page. If you want to see how often each "verdict" is returned by the server, you can have a look at the telemetry results for Firefox Beta.

Privacy One of the most persistent misunderstandings about Safe Browsing is the idea that the browser needs to send all visited URLs to Google in order to verify whether or not they are safe. While this was an option in version 1 of the Safe Browsing protocol (as disclosed in their privacy policy at the time), support for this "enhanced mode" was removed in Firefox 3 and the version 1 server was decommissioned in late 2011 in favor of version 2 of the Safe Browsing API which doesn't offer this type of real-time lookup. Google explicitly states that the information collected as part of operating the Safe Browsing service "is only used to flag malicious activity and is never used anywhere else at Google" and that "Safe Browsing requests won't be associated with your Google Account". In addition, Firefox adds a few privacy protections:

Query string parameters are stripped from URLs we check as part of the download protection feature.

Cookies set by the Safe Browsing servers to protect the service from abuse are stored in a separate cookie jar so that they are not mixed with regular browsing/session cookies.

When requesting complete hashes for a 32-bit prefix, Firefox throws in a number of extra "noise" entries to obfuscate the original URL further.

On balance, we believe that most users will want to keep Safe Browsing enabled, but we also make it easy for users with particular needs to turn it off.

Learn More If you want to learn more about how Safe Browsing works in Firefox, you can find all of the technical details on the Safe Browsing and Application Reputation pages of the Mozilla wiki or you can ask questions on our mailing list. Google provides some interesting statistics about what their systems detect in their transparency report and offers a tool to find out why a particular page has been blocked. Some information on how phishing sites are detected is also available on the Google Security blog, but for more detailed information about all parts of the Safe Browsing system, see the following papers:

All Your IFrames Are Belong to Us

Content-Agnostic Malware Protection

Ghost in the Browser

DebConf16 will take place in Cape Town, South Africa in July 2016. We strive to provide an intense working environment and enable good progress for Debian and for Free Software in general. We extend an invitation to everyone to join us and to support this event. As a volunteer-run non-profit conference, we depend on our sponsors. Nine companies have already committed to sponsor DebConf16! Let's introduce them: Our first Platinum sponsor is Hewlett Packard Enterprise (HPE). HPE is one of the largest computer companies in the world, providing a wide range of products and services, such as servers, storage, networking, consulting and support, software, and financial services. HPE is also a development partner of Debian, and provides hardware for port development, Debian mirrors, and other Debian services (hardware donations are listed in the Debian machines page). Our first Gold sponsor is Valve, a company developing games, social entertainment platform, and game engine technologies. Our second Gold sponsor is Google, the technology company specialized in Internet-related services as online advertising and search engine. Rusbitech (developers of the Astra Linux Debian derivative), credativ (a service-oriented company focusing on open-source software and also a Debian development partner), Catalyst (a company offering IT solutions using open source software), the Bern University of Applied Sciences (with over 7,000 students enrolled, located in the Swiss capital), and Texas Instruments (the global semiconductor company) are our four Silver sponsors. And last but not least, the open source company Univention has agreed to support us as Bronze-level. Become a sponsor too! Would you like to become a sponsor? Do you know of or work in a company or organization that may consider sponsorship? Please have a look at our sponsorship brochure (or a summarized flyer), in which we outline all the details and describe the sponsor benefits. For further details, feel free to contact us through sponsors@debconf.org, and visit the DebConf16 website at https://debconf16.debconf.org.

With the (relatively, sorry) recent switch to ruby2.3 as default Ruby version, I realized that my plotting program ctioga2 crashed with ruby 2.3 (due to a rather ugly lazy hack). I have just fixed this in release 0.14.1. Update as usual with:

~ gem update ctioga2

If you're not using Ruby 2.3, you probably don't need to update soon, though. Enjoy !

This is a fusion recipe from a rather bland just stuff it with ricotta recipe I saw, David Scott s The Peniless Vegetarian , and my own mutations on those themes. I can t give you exact quantities, just make a little more than you will make the hollowed mound (grin), and the rest will make an excellent pasta sauce. Ingredients For an average sized butternut squash, you will need:
1 onion (I prefer red)
3 cloves of garlic
1 capsicum pepper (I prefer green, my ex- preferred red)
Some red lentils
Optional green or brown lentils for texture and flavour. I used some puy
The lentil quantity is hard to estimate, but I ratio 4 red to 1 optional.
Roughly one handful of chopped mushrooms i.e. when chopped, it is one handful
1 tin tinned tomatoes
Some tomato puree
A generous amout of garam masala garam masala is what brings out the flavout in lentils
Some paprike
Optional chilli if using chilli, I recommend fresh of course.
Optional Balsamic vinegar
Optional Marmite Preperation of the Squash
1. Cut the butternut squash in half, length ways. This is very hard, you will need a good large knife, and may require you jumping up and down into the air. This is the second most hard of the procedure. 2. For each half, scoop out the seeds, and pare back the bowl till it is no longer overly fibrous. Discard this, or find a use for the seeds. 3. For each half, scoop a channel of the softer flesh up from the baisin up near the top. This has to be done by feel, is hard and thankless work. Also experimentation required. Reserve this flesh. Preperation of the Filling This is just basically a nice lentil sauce that can be used with pasta, rice, toast etc. Important: this is not a stir fry, but a largish, heavy bottom pan is recommended. 1. Finely peel then chopp the onions and the garlic. Chopp the chillis if used (I am a chilli gal). Please observe Chilli Protocol[0] 2. Wash and chop the pepper and mushrooms. Not finely diced, but not crudite-sized slices. Remember that peppers shrivel down a little, mushrooms a lot. 3. Start frying the onions for a while in some oil (I prefer olive, but others are acceptible), until they just about to go translucent. Then add the garlic and optional chillis until the garlic is just cooking nicely. 4. Add the spices, turn over until all the containts of the pan are covered, and cook for another 30 seconds or so. Then add the tinned tomato, and then add half a can of cold water water which rinsed the tin out with. Stir this around, and make sure it is now at just at a simmer or pre-simmer. 5. Add the lentils. You want 0.5-1 cm of water above the lentils when you have added and stirred. Let these cook and expand for about 5 mins, stirring all the while, all the lentils will stick to the bottom. 6. Add the pepper, mushroom, reserved squash flesh, and optional dash of balsamic vinegar, and half a tea spoon of marmite. Cook and stir until the pepper goes soft. This is the hard part. Add boiling water if really too thick, or some tomato puree if too thin. There is no hard science to this, you want at the end of 10 minutes or so something resembling the thickness in texture of a stiff bolognaise sauce. Assembly
1. Have a baking tray. Whether you prefer to grease, line with foil, or line with baking parchment is up to you. I prefer baking parchment. 2. Stuff those two halves of butternut squash with that sauce you made. It should make a mound of about 1cm about the level. If you feel extravagent, and are not vegan, sprinkle a little grated cheese on top. 3. Place in a pre-heated oven of 200oC. Cooking time should be about 20 mins, but larger ones take longer. The acid test is to briefly take them out, and prod the lower side with a fork. It should go through the skin with little resistance. When ready, serve. It s really a dish in itself, but some people might like a bit of salad, or maybe a light green risotto.

Propellor was recently ported to FreeBSD, by Evan Cofsky. This new feature led me down a two week long rabbit hole to make it type safe. In particular, Propellor needed to be taught that some properties work on Debian, others on FreeBSD, and others on both. The user shouldn't need to worry about making a mistake like this; the type checker should tell them they're asking for something that can't fly.

-- Is this a Debian or a FreeBSD host? I can't remember, let's use both package managers!
host "example.com" $ props
    & aptUpgraded
    & pkgUpgraded

As of propellor 3.0.0 (in git now; to be released soon), the type checker will catch such mistakes. Also, it's really easy to combine two OS-specific properties into a property that supports both OS's:

upgraded = aptUpgraded  pickOS  pkgUpgraded

type level lists and functions The magick making this work is type-level lists. A property has a metatypes list as part of its type. (So called because it's additional types describing the type, and I couldn't find a better name.) This list can contain one or more OS's targeted by the property:

aptUpgraded :: Property (MetaTypes '[ 'Targeting 'OSDebian, 'Targeting 'OSBuntish ])
pkgUpgraded :: Property (MetaTypes '[ 'Targeting 'OSFreeBSD ])

In Haskell type-level lists and other DataKinds are indicated by the ' if you have not seen that before. There are some convenience aliases and type operators, which let the same types be expressed more cleanly:

aptUpgraded :: Property (Debian + Buntish)
pkgUpgraded :: Property FreeBSD

Whenever two properties are combined, their metatypes are combined using a type-level function. Combining aptUpgraded and pkgUpgraded will yield a metatypes that targets no OS's, since they have none in common. So will fail to type check. My implementation of the metatypes lists is hundreds of lines of code, consisting entirely of types and type families. It includes a basic implementation of singletons, and is portable back to ghc 7.6 to support Debian stable. While it takes some contortions to support such an old version of ghc, it's pretty awesome that the ghc in Debian stable supports this stuff. extending beyond targeted OS's Before this change, Propellor's Property type had already been slightly refined, tagging them with HasInfo or NoInfo, as described in making propellor safer with GADTs and type families. I needed to keep that HasInfo in the type of properties. But, it seemed unnecessary verbose to have types like Property NoInfo Debian. Especially if I want to add even more information to Property types later. Property NoInfo Debian NoPortsOpen would be a real mouthful to need to write for every property. Luckily I now have this handy type-level list. So, I can shove more types into it, so Property (HasInfo + Debian) is used where necessary, and Property Debian can be used everywhere else. Since I can add more types to the type-level list, without affecting other properties, I expect to be able to implement type-level port conflict detection next. Should be fairly easy to do without changing the API except for properties that use ports. singletons As shown here, pickOS makes a property that decides which of two properties to use based on the host's OS.

aptUpgraded :: Property DebianLike
aptUpgraded = property "apt upgraded" (apt "upgrade"  requires  apt "update")
pkgUpgraded :: Property FreeBSD
pkgUpgraded = property "pkg upgraded" (pkg "upgrade")
    
upgraded :: Property UnixLike
upgraded = (aptUpgraded  pickOS  pkgUpgraded)
     describe  "OS upgraded"

Any number of OS's can be chained this way, to build a property that is super-portable out of simple little non-portable properties. This is a sweet combinator! Singletons are types that are inhabited by a single value. This lets the value be inferred from the type, which came in handy in building the pickOS property combinator. Its implementation needs to be able to look at each of the properties at runtime, to compare the OS's they target with the actial OS of the host. That's done by stashing a target list value inside a property. The target list value is inferred from the type of the property, thanks to singletons, and so does not need to be passed in to property. That saves keyboard time and avoids mistakes. is it worth it? It's important to consider whether more complicated types are a net benefit. Of course, opinions vary widely on that question in general! But let's consider it in light of my main goals for Propellor:

Help save the user from pushing a broken configuration to their machines at a time when they're down in the trenches dealing with some urgent problem at 3 am.
Advance the state of the art in configuration management by taking advantage of the state of the art in strongly typed haskell.

This change definitely meets both criteria. But there is a tradeoff; it got a little bit harder to write new propellor properties. Not only do new properties need to have their type set to target appropriate systems, but the more polymorphic code is, the more likely the type checker can't figure out all the types without some help. A simple example of this problem is as follows.

foo :: Property UnixLike
foo = p  requires  bar
  where
    p = property "foo" $ do
        ...

The type checker will complain that "The type variable metatypes1 is ambiguous". Problem is that it can't infer the type of p because many different types could be combined with the bar property and all would yield a Property UnixLike. The solution is simply to add a type signature like p :: Property UnixLike Since this only affects creating new properties, and not combining existing properties (which have known types), it seems like a reasonable tradeoff. things to improve later There are a few warts that I'm willing to live with for now... Currently, Property (HasInfo + Debian) is different than

Property (Debian +
HasInfo)

, but they should really be considered to be the same type. That is, I need type-level sets, not lists. While there's a type level sets library for hackage, it still seems to require a specific order of the set items when writing down a type signature. Also, using ensureProperty, which runs one property inside the action of another property, got complicated by the need to pass it a type witness.

foo = Property Debian
foo = property' $ \witness -> do
    ensureProperty witness (aptInstall "foo")

That witness is used to type check that the inner property targets every OS that the outer property targets. I think it might be possible to store the witness in the monad, and have ensureProperty read it, but it might complicate the type of the monad too much, since it would have to be parameterized on the type of the witness. Oh no, I mentioned monads. While type level lists and type functions and generally bending the type checker to my will is all well and good, I know most readers stop reading at "monad". So, I'll stop writing. ;) thanks Thanks to David Miani who answered my first tentative question with a big hunk of example code that got me on the right track. Also to many other people who answered increasingly esoteric Haskell type system questions. Also thanks to the Shuttleworth foundation, which funded this work by way of a Flash Grant.

What happened in the reproducible builds effort between March 20th and March 26th:

Toolchain fixes

Sebastian Ramacher uploaded breathe/4.2.0-1 which makes its output deterministic. Original patch by Chris Lamb, merged uptream.

Rafael Laboissiere uploaded octave/4.0.1-1 which allows packages to be built in place and avoid unreproducible builds due to temporary build directories appearing in the `.oct` files.

Daniel Kahn Gillmor worked on removing build path from build symbols submitting a patch adding `-fdebug-prefix-map` to clang to match GCC, another patch against gcc-5 to backport the removal of `-fdebug-prefix-map` from `DW_AT_producer`, and finally by proposing the addition of a `normalizedebugpath` to the `reproducible` feature set of `dpkg-buildflags` that would use `-fdebug-prefix-map` to replace the current directory with `.` using `-fdebug-prefix-map`. As succesful result of lobbying at LibrePlanet 2016, the `--clamp-mtime` option will be featured in the next Tar release. This option is likely to be used by `dpkg-deb` to implement deterministic mtimes for packaged files.

Packages fixed The following packages have become reproducible due to changes in their build dependencies: augeas, gmtkbabel, ktikz, octave-control, octave-general, octave-image, octave-ltfat, octave-miscellaneous, octave-mpi, octave-nurbs, octave-octcdf, octave-sockets, octave-strings, openlayers, python-structlog, signond. The following packages became reproducible after getting fixed:

atheme-services/7.0.7-1 by Antoine Beaupr .

boa-constructor/0.6.1-16 by Reiner Herrmann.

calculix-ccx/2.10-1 by Wolfgang F tterer, fixed upstream.

deap/1.0.2.post2-2 by Daniel Stender.

debian-keyring/2016.03.22 uploaded by Jonathan McDowell, fix by Niels Thykier, obsolete patch by Satyam Zode.

firefox-kwallet5/1.0-2 by Sandro Knau .

fox1.6/1.6.51-1 by Joachim Wiedorn.

gdnsd/2.2.3-1 by Faidon Liambotis.

glib2.0/2.48.0-1 uploaded by Iain Lane, original patch by Lunar, merged upstream.

jwm/2.3.4-1 uploaded by Samuel Henrique Oltramari Pinto, fix by Reiner Herrmann.

libasm4-java/5.0.4-2 by Emmanuel Bourg.

libjna-java/4.2.2-1 by Emmanuel Bourg.

libsynthesis/3.4.0.47.5+syncevolution-1.5.1-1 uploaded by Tino Mettler, patch by Reiner Herrmann.

logback/1:1.1.6-1 by Emmanuel Bourg.

nethack/3.6.0-2 uploaded by James Cowgill, patch by Reiner Herrmann.

php-htmlpurifier/4.7.0-2 by David Pr vot.

psocksxx/1.1.0-1 by Gianfranco Costamagna.

sbmltoolbox/4.1.0-3 uploaded by Afif Elghraoui, original patch by Reiner Herrmann.

spades/3.7.1+dfsg-2 by Sascha Steinbiss.

sprng/2.0a-10 uploaded by Dirk Eddelbuettel, original patch by Reiner Herrmann.

Some uploads fixed some reproducibility issues, but not all of them:

gle-graphics/4.2.5-2 by Christian T. Steigies.

kexec-tools/1:2.0.10-2 uploaded by Khalid Aziz, original patch by Lunar.

pdal/1.1.0-4 by Bas Couwenberg.

tcl8.5/8.5.19-2 uploaded by Sergei Golovan, original patch.

tcl8.6/8.6.5+dfsg-2 uploaded by Sergei Golovan, original patch.

Patches submitted which have not made their way to the archive yet:

#818742 on milkytracker by Reiner Herrmann: sorts the list of source files.

#818752 on tcl8.4 by Reiner Herrmann: sort source files using C locale.

#818753 on tk8.6 by Reiner Herrmann: sort source files using C locale.

#818754 on tk8.5 by Reiner Herrmann: sort source files using C locale.

#818755 on tk8.4 by Reiner Herrmann: sort source files using C locale.

#818952 on marionnet by ceridwen: dummy out build date and uname to make build reproducible.

#819334 on avahi by Reiner Herrmann: ship upstream changelog instead of the one generated by gettextize (although duplicate of #804141 by Santiago Vila).

tests.reproducible-builds.org `i386` build nodes have been setup by converting 2 of the 4 `amd64` nodes to `i386`. (h01ger)

Package reviews 92 reviews have been removed, 66 added and 31 updated in the previous week. New issues: timestamps_generated_by_xbean_spring, timestamps_generated_by_mangosdk_spiprocessor. Chris Lamb filed 7 FTBFS bugs.

Misc. On March 20th, Chris Lamb gave a talk at FOSSASIA 2016 in Singapore. The very same day, but a few timezones apart, h01ger did a presentation at LibrePlanet 2016 in Cambridge, Massachusetts. Seven GSoC/Outreachy applications were made by potential interns to work on various aspects of the reproducible builds effort. On top of interacting with several applicants, prospective mentors gathered to review the applications. Huge thanks to Linda Naeun Lee for the new hackergotchi visible on Planet Debian.

Search Results: "zed"

24 April 2016

Daniel Pocock: LinuxWochen, MiniDebConf Vienna and Linux Presentation Day

21 April 2016

Alessio Treglia: Corporate Culture in the Transformative Enterprise

20 April 2016

Norbert Preining: GnuPG notes: subkeys, yubikey, gpg1 vs gpg2

18 April 2016

Petter Reinholdtsen: NUUG contests Norwegian police DNS seizure of popcorn-time.no

Dirk Eddelbuettel: RcppCCTZ 0.0.4

16 April 2016

Matthias Klumpp: Introducing AppStream-Generator

13 April 2016

Mike Gabriel: IPv6: Broken by Design; Digital Ocean - How are we doing?

Michal Čihař: Weekly phpMyAdmin contributions 2016-W14

12 April 2016

Reproducible builds folks: Reproducible builds: week 48 in Stretch cycle

10 April 2016

Vincent Bernat: Testing network software with pytest and Linux namespaces

Russ Allbery: Largish haul

5 April 2016

Ian Wienand: Image building in OpenStack CI

4 April 2016

Ian Wienand: Image building for OpenStack CI -- Minimal images, big effort

2 April 2016

Alessio Treglia: Fuzzy: a many-valued logic

1 April 2016

Francois Marier: How Safe Browsing works in Firefox

30 March 2016

Bits from Debian: DebConf16 welcomes its first nine sponsors!

29 March 2016

Vincent Fourmond: Release 0.14.1 of ctioga2

28 March 2016

Lucy Wayland: Stuffed Butternut Squash

Joey Hess: type safe multi-OS Propellor

27 March 2016

Lunar: Reproducible builds: week 48 in Stretch cycle

tests.reproducible-builds.org `i386` build nodes have been setup by converting 2 of the 4 `amd64` nodes to `i386`. (h01ger)

Package reviews 92 reviews have been removed, 66 added and 31 updated in the previous week. New issues: timestamps_generated_by_xbean_spring, timestamps_generated_by_mangosdk_spiprocessor. Chris Lamb filed 7 FTBFS bugs.

Search Results: "zed"

24 April 2016

21 April 2016

20 April 2016

18 April 2016

16 April 2016

13 April 2016

12 April 2016

10 April 2016

5 April 2016

4 April 2016

2 April 2016

1 April 2016

30 March 2016

29 March 2016

28 March 2016

27 March 2016

tests.reproducible-builds.org i386 build nodes have been setup by converting 2 of the 4 amd64 nodes to i386. (h01ger)

Package reviews 92 reviews have been removed, 66 added and 31 updated in the previous week. New issues: timestamps_generated_by_xbean_spring, timestamps_generated_by_mangosdk_spiprocessor. Chris Lamb filed 7 FTBFS bugs.

tests.reproducible-builds.org `i386` build nodes have been setup by converting 2 of the 4 `amd64` nodes to `i386`. (h01ger)