Over the last few months, I have been working casually on a new package to integrate MessagePack with R. What is MessagePack, you ask? To quote its website, "It's like JSON, but fast and small." Or in more extended terms:

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.

Now, serialization formats are a dime a dozen: JSON, BSON, Protocol Buffers, CaptnProto, Flatbuffer. The list goes on and on. So why another? In a nutshell: "software ecosystems". I happen to like working with Redis, and within the world of Redis, MessagePack is a first-class citizen supported by things close to the core like the embedded Lua interpreter, as well as fancy external add-ons such as the Redis Desktop Manager GUI. So nothing overly fundamentalist here, but a fairly pragmatic choice based on what happens to fit my needs. Plus, having worked on and off with Protocol Buffers for close to a decade, the chance of working with something not requiring a friggin' schema compiler seemed appealing for a chance. So far, we have been encoding a bunch of data streams at work via MessagePack into Redis (and of course back). It works really well---header-only C++11 libraries for the win. I'll provide an updated RcppRedis which uses this (if present) in due course. For now and the foreseeable future, this RcppMsgPack package will live only on the ghrr drat repository. To make RcppMsgPack work, I currently have to include the MessagePack 1.4.0 headers. A matching package for this version of the headers is in Debian but so far only in experimental. Once this hits the mainline repository I can depend on it, and upload a (lighter, smaller) RcppMsgPack to CRAN. Until then, please just do

## install drat if not present
if (!require(drat)) install.packages("drat")
## use drat to select ghrr repo
drat::addRepo("ghrr")
## install RcppMsgPack
install.packages("RcppMsgPack")

More details, issue tickets etc are at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

What happened in the reproducible builds effort between April 10th and April 16th 2016: Toolchain fixes

Roland Rosenfeld uploaded transfig/1:3.2.5.e-6 which honors SOURCE_DATE_EPOCH. Original patch by Alexis Bienven e.
Bill Allombert uploaded gap/4r8p3-2 which makes convert.pl honor SOURCE_DATE_EPOCH. Original patch by Jerome Benoit, duplicate patch by Dhole.
Emmanuel Bourg uploaded ant/1.9.7-1 which makes the Javadoc task use UTF-8 as the default encoding if none was specified and SOURCE_DATE_EPOCH is set.

Antoine Beaupr suggested that gitpkg stops recording timestamps when creating upstream archives. Antoine Beaupr also pointed out that git-buildpackage diverges from the default gzip settings which is a problem for reproducibly recreating released tarballs which were made using the defaults. Alexis Bienven e submitted a patch extending sphinx SOURCE_DATE_EPOCH support to copyright year. Packages fixed The following packages have become reproducible due to changes in their build dependencies: atinject-jsr330, avis, brailleutils, charactermanaj, classycle, commons-io, commons-javaflow, commons-jci, gap-radiroot, jebl2, jetty, libcommons-el-java, libcommons-jxpath-java, libjackson-json-java, libjogl2-java, libmicroba-java, libproxool-java, libregexp-java, mobile-atlas-creator, octave-econometrics, octave-linear-algebra, octave-odepkg, octave-optiminterp, rapidsvn, remotetea, ruby-rinku, tachyon, xhtmlrenderer. The following packages became reproducible after getting fixed:

autopkgtest/3.20.3 uploaded by Martin Pitt, original patch by Alexis Bienven e.
fop/1:2.1-3 uploaded by Mathieu Malaterre, original patch by Alexis Bienven e.
fwupdate/0.5-4 by Mario Limonciello.
gem/1:0.93.3-10 by IOhannes m zm lnig.
imview/1.1.9c-16 by Anton Gladky.
libappindicator/0.4.92-4 by Adam Borowski.
libjlha-java/0.0.20050504-9 by Emmanuel Bourg.
lwjgl/2.9.3+dfsg-1 by Markus Koschany.
lxc/1:2.0.0~rc15-1 uploaded by Evgeni Golov, original patch by Reiner Herrmann.
manpages-de/1.12-1 uploaded by Tobias Quathamer, original patch by Reiner Herrmann.
pd-mrpeach/0.1~svn17615-1 by IOhannes m zm lnig.
pyhoca-gui/0.5.0.6-1 uploaded by Mike Gabriel, original patch by Chris Lamb.
r-cran-spatstat/1.45-0-1 by Andreas Tille.
s5/1.1.dfsg.2-6 uploaded by Peter Pentchev, original patch by Juan Picca.
samba/2:4.3.8+dfsg-1 by Jelmer Vernoo .
sim4/0.0.20121010-3 uploaded by Andreas Tille, original patch by Alexis Bienven e.
svxlink/15.11-1 uploaded by Felix Lechner, fixed upstream.
tinc/1.0.28-1 by Guus Sliepen, fixed upstream.
ucblogo/6.0+dfsg-1 by Reiner Herrmann.

Some uploads fixed some reproducibility issues, but not all of them:

mm-common/0.9.10-1 by Michael Biebl.

Patches submitted which have not made their way to the archive yet:

#820603 on viking by Alexis Bienven e: fix icon headers inclusion order.
#820661 on nullmailer by Alexis Bienven e: fix the order in which files are included in the static archive.
#820668 on sawfish by Alexis Bienven e: fix file ordering in theme archives, strip hostname and username from the config.h file, and honour SOURCE_DATE_EPOCH when creating the config.h file.
#820740 on bless by Alexis Bienven e: always use /bin/sh as shell.
#820742 on gmic by Alexis Bienven e: strip the build date from help messages.
#820809 on wsdl4j by Alexis Bienven e: use a plain text representation of the copyright character.
#820815 on freefem++ by Alexis Bienven e: fix the order in which files are included in the .edp files, and honour SOURCE_DATE_EPOCH when using the build date.
#820869 on pyexiv2 by Alexis Bienven e: honour the SOURCE_DATE_EPOCH environment variable through the ustrftime function, to get a reproducible copyright year.
#820932 on fim by Alexis Bienven e: fix the order in which files are joined in header files, strip the build date from fim binary, make the embeded vim2html script honour SOURCE_DATE_EPOCH variable when building the documentation, and force language to be English when using bison to make a grammar that is going to be parsed using English keywords.
#820990 on grib-api by Santiago Vila: always call dh-buildinfo.

diffoscope development Zbigniew J drzejewski-Szmek noted in #820631 that diffoscope doesn't work properly when a file contains several cpio archives. Package reviews 21 reviews have been added, 14 updated and 22 removed in this week. New issue found: timestamps_in_htm_by_gap. Chris Lamb reported 10 new FTBFS issues. Misc. The video and the slides from the talk "Reproducible builds ecosystem" at LibrePlanet 2016 have been published now. This week's edition was written by Lunar and Holger Levsen. h01ger automated the maintenance and publishing of this weekly newsletter via git.

Switching from one GnuPG master key to the usage of subkeys was long on my list of things I wanted to do, but never came around. With the advent of a YubiKey NEO in my pocket I finally took the plunge: reading through lots of web pages (and adding one here for confusion), trying to understand the procedures, and above all, understanding my own requirements!

To sum up a long story, it was worth the plunge, and all over the security level of my working environment has improved considerable. While the advantages of subkeys are well documented (e.g., Debian Wiki), at the end of the day I was like probably many Debian Developers having one master key that was used for every action: mail decryption and signing, signing of uploads, etc. Traveling a lot I always felt uncomfortable. Despite a lengthy passphrase, I still didn t want my master key to get into wrong hands in case the laptop got stolen. Furthermore, I had my master key on several computers (work, laptop, mail server), which didn t help a lot either. With all this, I started to compile a list of requirements/objectives I wanted to have:

master key is only available on offline medium (USB sticks)
subkeys for signing, encryption, authentication
possibility to sign and decrypt my emails on the server where I read emails (ssh/mutt)
laptop does not contain any keys, instead use Yubikey
all keys with expiry date (1y)
mixture of gpg versions: local laptop: gpg2.1, mail server: gpg1

Warning Before we start a word of caution make backups, best is to make backups at every stage. You don t want that an erroneous operations wipes out your precious keys without a backup! Preparation In the following I will assume that MASTERKEY environment variable contains the id of the master key to be converted. Furthermore, I have followed some of the advice here, so key ids will be shown in long format. Let us start with the current situation:

$ gpg -K $MASTERKEY
sec   4096R/0x6CACA448860CDC13 2010-09-14
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                            Norbert Preining <norbert@preining.info>
uid                            Norbert Preining <preining@logic.at>
uid                            Norbert Preining <preining@debian.org>
uid                            Norbert Preining <preining@jaist.ac.jp>
ssb   4096R/0xD1D2BD14810F62B3 2010-09-14

In the following we will go through the following steps:

Prepare the Yubikey NEO (forthcoming blog>
Edit to current key: add expiry, add photo, and above all add subkeys
Create revocation certificate
Create gpg2.1 structure
Backup to USB media
Move subkeys to Yubikey NEO
Remove master keys
Separate gpg1 (for mail server) and gpg2 (for laptop)
Upload to key servers

Yubikey SmartCard setup There are several guides out there, but I will in very near future write one about using the NEO for various usage scenaria including GPG keys. Edit the current key The following can be done in one session or in different sessions, the screen logs are after starting with:

$ gpg --expert --edit-key $MASTERKEY

add expiry date Having an expiry date on your key serves two purposes: If you loose it, it will solve itself automatically, and furthermore, you are forced to deal with the key and refresh your gpg knowledge at least once a year. That are two perfect reasons to set expiry to one year. The following log selects each key in turn and sets its expiry date.

$ gpg --expert --edit-key $MASTERKEY
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: never       usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: never       usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
 
gpg> expire
Changing expiration time for the primary key.
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:09:16 PM JST
Is this correct? (y/N) y
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: never       usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
 
gpg> key 1
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: never       usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
 
gpg> expire
Changing expiration time for a subkey.
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:09:27 PM JST
Is this correct? (y/N) y
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>

Add a photo Not strictly necessary, but an interesting feature. gpg suggests 240 288, I resized a photo of my head, greyscaled it, and optimized it with jpegoptim -s -m40 my-photo.jpg. The parameter 40 is the quality, I played around a bit to find the best balance between size and quality. The size should not be too big as the photo will be part of the key!

gpg> addphoto
 
Pick an image to use for your photo ID.  The image must be a JPEG file.
Remember that the image is stored within your public key.  If you use a
very large picture, your key will become very large as well!
Keeping the image close to 240x288 is a good size to use.
 
Enter JPEG filename for photo ID: GPG/norbert-head.jpg
Is this photo correct (y/N/q)? y
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ unknown] (5)  [jpeg image of size 4185]

Add subkeys of 2048bit for signing/encryption/authentication Now comes the interesting part, adding three subkeys: one for signing, one for encrypting, and one for authentication. The one for signing is the one you will use for signing your uploads to Debian as well as emails. The authentication key will later be used to provide ssh authentication. Note that you have to use the --expert expert option to edit-key (as shown above), otherwise gpg does not allow to do this. As I want to move the subkeys to the Yubikey NEO, a keysize of 2048bits is necessary. First for the signing:

gpg> addkey
Key is protected.
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
Your selection? 4
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:10:06 PM JST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
....+++++
..........+++++
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
sub  2048R/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06  usage: S   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ unknown] (5)  [jpeg image of size 4185]

Now the same for encryption key:

gpg> addkey
Key is protected.
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
Your selection? 6
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:10:20 PM JST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
..+++++
........+++++
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
sub  2048R/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06  usage: S   
sub  2048R/0xBF361ED434425B4C  created: 2016-02-07  expires: 2017-02-06  usage: E   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ unknown] (5)  [jpeg image of size 4185]

Finally for the authentication key. Note that only here the --expert is necessary! We use (8) RSA (set your own capabilities) and then toggle sign and encryption capabilities off, and authentication on.

gpg> addkey
Key is protected.
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
Your selection? 8
 
Possible actions for a RSA key: Sign Encrypt Authenticate 
Current allowed actions: Sign Encrypt 
 
   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished
 
Your selection? s
 
Possible actions for a RSA key: Sign Encrypt Authenticate 
Current allowed actions: Encrypt 
 
   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished
 
Your selection? e
 
Possible actions for a RSA key: Sign Encrypt Authenticate 
Current allowed actions: 
 
   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished
 
Your selection? a
 
Possible actions for a RSA key: Sign Encrypt Authenticate 
Current allowed actions: Authenticate 
 
   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished
 
Your selection? q
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon 06 Feb 2017 08:10:34 PM JST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
......+++++
+++++
 
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub* 4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
sub  2048R/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06  usage: S   
sub  2048R/0xBF361ED434425B4C  created: 2016-02-07  expires: 2017-02-06  usage: E   
sub  2048R/0x9C7CA4E294F04D49  created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ unknown] (5)  [jpeg image of size 4185]
 
gpg> save

Check the current status Good point to take a break and inspect the current status. We should have one main key and three subkeys, all with expiry dates of 1 year ahead, and a photo also attached to the key:

$ gpg --expert --edit-key $MASTERKEY
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
gpg: checking the trustdb
gpg: public key 0x0FC3EC02FBBB8AB1 is 58138 seconds newer than the signature
gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model
gpg: depth: 0  valid:   2  signed:  28  trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1  valid:  28  signed:  41  trust: 28-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2016-11-02
pub  4096R/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06  usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06  usage: E   
sub  2048R/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06  usage: S   
sub  2048R/0xBF361ED434425B4C  created: 2016-02-07  expires: 2017-02-06  usage: E   
sub  2048R/0x9C7CA4E294F04D49  created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg>

Create revocation certificate In case something happens, like all your backups are burned, your computers are destroyed, or all data stolen by the NSA, it is a good idea to have an old fashioned paper print out of a revocation certificate which allows you to revoke the key even if you are not in possession of it. This should be printed out and kept in a safe place.

$ gpg --gen-revoke $MASTERKEY > GPG/revoke-certificate-$MASTERKEY.txt
 
sec  4096R/0x6CACA448860CDC13 2010-09-14 Norbert Preining <norbert@preining.info>
 
Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
  0 = No reason specified
  1 = Key has been compromised
  2 = Key is superseded
  3 = Key is no longer used
  Q = Cancel
(Probably you want to select 1 here)
Your decision? 1
Enter an optional description; end it with an empty line:
> 
Reason for revocation: Key has been compromised
(No description given)
Is this okay? (y/N) y
 
You need a passphrase to unlock the secret key for
user: "Norbert Preining <norbert@preining.info>"
4096-bit RSA key, ID 0x6CACA448860CDC13, created 2010-09-14
 
Enter passphrase:
 
ASCII armored output forced.
Revocation certificate created.

Please move it to a medium which you can hide away; if the NSA or KGB or Mossad gets access to this certificate, they can use it to make your key unusable. It is smart to print this certificate and store it away, just in case your media become unreadable. Create gpg 2.1 structure There are currently three versions of gpg available: classic (version 1) which is one static binary, perfect for servers or scripting tasks; stable (version 2.0) which is the modularized version supporting OpenPGP, S/MIME, and Secure Shell; and finally modern (version 2.1 and up) with enhanced features like support for Elliptic Curve cryptography. Debian currently ships version 1 as standard, and also the modern version (but there are traces in experimental of a pending transition). The newer versions of GnuPG are modularized and use an agent. For the following we need to kill any running instance of gpg-agent.

$ killall gpg-agent

After that a simple call to gpg2 to list the secret keys will convert the layout to the new standard:

$ gpg2 -K $MASTERKEY
gpg: keyserver option 'ca-cert-file' is obsolete; please use 'hkp-cacert' in dirmngr.conf
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/home/norbert/.gnupg/secring.gpg' to gpg-agent
gpg: key 0xD2BF4AA309C5B094: secret key imported
gpg: key 0x6CACA448860CDC13: secret key imported
gpg: migration succeeded
sec   rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                   [ultimate] Norbert Preining <norbert@preining.info>
uid                   [ultimate] Norbert Preining <preining@logic.at>
uid                   [ultimate] Norbert Preining <preining@debian.org>
uid                   [ultimate] Norbert Preining <preining@jaist.ac.jp>
uid                   [ultimate] [jpeg image of size 4185]
ssb   rsa4096/0xD1D2BD14810F62B3 2010-09-14 [E] [expires: 2017-02-06]
ssb   rsa2048/0xEC00B8DAD32266AA 2016-02-07 [S] [expires: 2017-02-06]
ssb   rsa2048/0xBF361ED434425B4C 2016-02-07 [E] [expires: 2017-02-06]
ssb   rsa2048/0x9C7CA4E294F04D49 2016-02-07 [A] [expires: 2017-02-06]

After this there will be new files/directories in the .gnupg directory, in particular: .gnupg/private-keys-v1.d/ which contains the private keys. Creating backup Now your .gnupg directory contains still all the keys, available for gpg1 and gpg2.1. You MUST MAKE A BACKUP NOW!!! on at least 3 USB sticks and maybe some other offline media. Keep them in a safe place, better in different and safe places, you will need them for extending the expiry date, signing other keys, etc. Warning concerning USB and vfat file systems gpg >= 2.1 requires gpg-agent which in turn needs a socket. If you have the backup on an USB drive (most often with vfat file system), you need to redirect the socket, as vfat does not support sockets! Edit /USBSTICK/gnupghome/S.gpg-agent and enter there

%Assuan%
socket=/dev/shm/S.gpg-agent

After that the socket will be created in /dev/shm/ instead and invoking gpg with gpg2 --homedir /USBSTICK/gnupghome will work. You have done your backups, right? Move sub keys to card As I mentioned, I want to have no keys on my laptop which I carry around to strange countries, instead I want to have them all on a Yubikey NEO. I will describe the setup and usage in details soon, but mention here only how to move the keys to the card. This requires a finished setup including change of pins. Note that when using gpg2 to move the keys to the card, the local copies are actually deleted, but only for the gpg2(.1) files. The gpg1 secret keys are still all in place.

$ gpg2 --edit-key $MASTERKEY
gpg (GnuPG) 2.1.11; Copyright (C) 2016 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> save

Note the repetition of selecting and deselecting keys. Current status After this procedure we are now in the following situation:

gpg1: all keys are still available
gpg2: sub keys are moved to yubikey (indicated below by ssb>), and master key is still available

In gpg words it looks like this:

$ gpg2 -K $MASTERKEY
gpg: keyserver option 'ca-cert-file' is obsolete; please use 'hkp-cacert' in dirmngr.conf
sec   rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                   [ultimate] Norbert Preining <norbert@preining.info>
uid                   [ultimate] Norbert Preining <preining@logic.at>
uid                   [ultimate] Norbert Preining <preining@debian.org>
uid                   [ultimate] Norbert Preining <preining@jaist.ac.jp>
uid                   [ultimate] [jpeg image of size 4185]
ssb   rsa4096/0xD1D2BD14810F62B3 2010-09-14 [E] [expires: 2017-02-06]
ssb>  rsa2048/0xEC00B8DAD32266AA 2016-02-07 [S] [expires: 2017-02-06]
ssb>  rsa2048/0xBF361ED434425B4C 2016-02-07 [E] [expires: 2017-02-06]
ssb>  rsa2048/0x9C7CA4E294F04D49 2016-02-07 [A] [expires: 2017-02-06]
 
$ gpg -K $MASTERKEY
sec   4096R/0x6CACA448860CDC13 2010-09-14 [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                            Norbert Preining <norbert@preining.info>
uid                            Norbert Preining <preining@logic.at>
uid                            Norbert Preining <preining@debian.org>
uid                            Norbert Preining <preining@jaist.ac.jp>
uid                            [jpeg image of size 4185]
ssb   4096R/0xD1D2BD14810F62B3 2010-09-14 [expires: 2017-02-06]
ssb   2048R/0xEC00B8DAD32266AA 2016-02-07 [expires: 2017-02-06]
ssb   2048R/0xBF361ED434425B4C 2016-02-07 [expires: 2017-02-06]
ssb   2048R/0x9C7CA4E294F04D49 2016-02-07 [expires: 2017-02-06]
 
$ gpg2 --card-status
 
....
Name of cardholder: Norbert Preining
....
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: 5871 F824 2DCC 3660 2362  BE7D EC00 B8DA D322 66AA
      created ....: 2016-02-07 11:10:06
Encryption key....: 2501 195C 90AB F4D2 3DEA  A303 BF36 1ED4 3442 5B4C
      created ....: 2016-02-07 11:10:20
Authentication key: 9CFB 3775 C164 0E99 F0C8  014C 9C7C A4E2 94F0 4D49
      created ....: 2016-02-07 11:10:34
General key info..: sub  rsa2048/0xEC00B8DAD32266AA 2016-02-07 Norbert Preining <norbert@preining.info>
sec   rsa4096/0x6CACA448860CDC13  created: 2010-09-14  expires: 2017-02-06
ssb   rsa4096/0xD1D2BD14810F62B3  created: 2010-09-14  expires: 2017-02-06
ssb>  rsa2048/0xEC00B8DAD32266AA  created: 2016-02-07  expires: 2017-02-06
                                  card-no: 0006 03645719
ssb>  rsa2048/0xBF361ED434425B4C  created: 2016-02-07  expires: 2017-02-06
                                  card-no: 0006 03645719
ssb>  rsa2048/0x9C7CA4E294F04D49  created: 2016-02-07  expires: 2017-02-06
                                  card-no: 0006 03645719
$

Remove private master keys You are sure that you have a working backup? Did you try it with gpg --homedir ...? Only if you are really sure, continue. We are now removing the master key from both the gpg2 and gpg1 setup. removal for gpg2 gpg2 keeps the private keys in ~/.gnupg/private-keys-v1.d/KEYGRIP.key and the KEYGRIP can be found by adding --with-keygrip to the key listing. Be sure to delete the correct file, the one related to the master key.

$ gpg2 --with-keygrip --list-key $MASTERKEY
pub   rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
      Keygrip = 9DC1E90703856C1DE0EAC970CED7ABF5EE5EF79D
uid                   [ultimate] Norbert Preining <norbert@preining.info>
uid                   [ultimate] Norbert Preining <preining@logic.at>
uid                   [ultimate] Norbert Preining <preining@debian.org>
uid                   [ultimate] Norbert Preining <preining@jaist.ac.jp>
uid                   [ultimate] [jpeg image of size 4185]
sub   rsa4096/0xD1D2BD14810F62B3 2010-09-14 [E] [expires: 2017-02-06]
      Keygrip = 4B8FF57434DD989243666377376903281D861596
sub   rsa2048/0xEC00B8DAD32266AA 2016-02-07 [S] [expires: 2017-02-06]
      Keygrip = 39B14EF1392F2F251863A87AE4D44CE502755C39
sub   rsa2048/0xBF361ED434425B4C 2016-02-07 [E] [expires: 2017-02-06]
      Keygrip = E41C8DDB2A22976AE0DA8D7D11F586EA793203EA
sub   rsa2048/0x9C7CA4E294F04D49 2016-02-07 [A] [expires: 2017-02-06]
      Keygrip = A337DE390143074C6DBFEA64224359B9859B02FC
 
$ rm ~/.gnupg/private-keys-v1.d/9DC1E90703856C1DE0EAC970CED7ABF5EE5EF79D.key
$

After that the missing key is shown in gpg2 -K with an additional # meaning that the key is not available:

$ gpg2 -K $MASTERKEY
sec#  rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
...

removal for gpg1 Up to gpg v2.0 there is no simple way to delete only one part of the key. We export the subkeys, delete the private key, and reimport the subkeys:

$ gpg --output secret-subkeys --export-secret-subkeys $MASTERKEY
 
$ gpg --delete-secret-keys $MASTERKEY
 
sec  4096R/0x6CACA448860CDC13 2010-09-14 Norbert Preining <norbert@preining.info>
 
Delete this key from the keyring? (y/N) y
This is a secret key! - really delete? (y/N) y
 
$ gpg --import secret-subkeys
gpg: key 0x6CACA448860CDC13: secret key imported
gpg: key 0x6CACA448860CDC13: "Norbert Preining <norbert@preining.info>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
 
$

Current status We are basically at the stage we wanted to achieve: For gpg2.1 only the old encryption key is available, the master key is not, and the other sub keys are moved to the yubikey:

$ gpg2 -K $MASTERKEY
sec#  rsa4096/0x6CACA448860CDC13 2010-09-14 [SC] [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                   [ultimate] Norbert Preining <norbert@preining.info>
uid                   [ultimate] Norbert Preining <preining@logic.at>
uid                   [ultimate] Norbert Preining <preining@debian.org>
uid                   [ultimate] Norbert Preining <preining@jaist.ac.jp>
uid                   [ultimate] [jpeg image of size 4185]
ssb   rsa4096/0xD1D2BD14810F62B3 2010-09-14 [E] [expires: 2017-02-06]
ssb>  rsa2048/0xEC00B8DAD32266AA 2016-02-07 [S] [expires: 2017-02-06]
ssb>  rsa2048/0xBF361ED434425B4C 2016-02-07 [E] [expires: 2017-02-06]
ssb>  rsa2048/0x9C7CA4E294F04D49 2016-02-07 [A] [expires: 2017-02-06]
$

And for gpg <= 2.0 the old encryption key and the sub keys are available, but the master key is not:

$ gpg -K $MASTERKEY
sec#  4096R/0x6CACA448860CDC13 2010-09-14 [expires: 2017-02-06]
      Key fingerprint = F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
uid                            Norbert Preining <norbert@preining.info>
uid                            Norbert Preining <preining@logic.at>
uid                            Norbert Preining <preining@debian.org>
uid                            Norbert Preining <preining@jaist.ac.jp>
uid                            [jpeg image of size 4185]
ssb   4096R/0xD1D2BD14810F62B3 2010-09-14 [expires: 2017-02-06]
ssb   2048R/0xEC00B8DAD32266AA 2016-02-07 [expires: 2017-02-06]
ssb   2048R/0xBF361ED434425B4C 2016-02-07 [expires: 2017-02-06]
ssb   2048R/0x9C7CA4E294F04D49 2016-02-07 [expires: 2017-02-06]
 
$

Split the .gnupg directory for mail server and laptop As mentioned, I want to have a gpg1 version available at the server where I read my emails, and be able to sign/encrypt emails there, while on my laptop no secret key is available. Thus I prepare two gnupg directories. For the mailserver the gpg2 specific files are removed:

$ cp -a .gnupg .gnupg-mail
$ cd .gnupg-mail
$ rm -rf private-keys-v1.d/ pubring.gpg~ reader_0.status
$ rm -rf S.gpg-agent* S.scdaemon .gpg-v21-migrated

On my laptop, where I did all this operation, I remove the gpg1 files, namely the outdated secring.gpg:

$ cd $HOME/.gnupg
$ rm secring.gpg

As a last step I move the .gnupg-mail directory to my mail server. Once could *expire* the old encryption key, but for now I leave it as is. Upload keys to keyservers If you are a Debian Developer, a simple update of your master key will suffice:

gpg --keyserver hkp://keyring.debian.org --send-key YOURMASTERKEYID

Note that the update from the keyring server to the actual Debian keyring takes up to one month. Until that time either do not upload anything, or use the (offline) master key for signing. After your key has been updated in the Debian keyring, signatures made with the signing subkey will be accepted for uploading to Debian. It might be also a good idea to upload your new keys to some keyservers like:

gpg --keyserver hkp://pool.sks-keyservers.net --send-key $MASTERKEY

Now you an also fix the configuration file skew between gpg1 and gpg2. Further remark I am currently trying to use the authentication key from my Yubikey NEO as ssh key, but bugs (see #795368 and #818969) prohibit it at the moment. Raphael Herzog gave a possible fix by killing the gpg-agent and restarting it with gpg-agent --daemon from an X terminal, and I can confirm that this worked. After one year before the key expires I need to extend the key validity for another year. For this you need the offline master key. I will describe the process when it becomes necessary. Reading list The following web sites have been useful in collecting the necessary information:

https://iain.learmonth.me/yubikey-neo-gpg/
https://iain.learmonth.me/yubikey-udev/
http://blog.josefsson.org/2014/06/23/offline-gnupg-master-key-and-subkeys-on-yubikey-neo-smartcard/
https://wiki.debian.org/Subkeys
https://jclement.ca/articles/2015/gpg-smartcard/ as modernized version of (3)
https://www.esev.com/blog/post/2015-01-pgp-ssh-key-on-yubikey-neo/ similar style, with ssh and gnome-keyring infos
http://karlgrz.com/2fa-gpg-ssh-keys-with-pass-and-yubikey-neo/ also good reading
https://help.riseup.net/en/security/message-security/openpgp/best-practices good and concise advise on gpg practices

My writing is mostly based on (5) with additions from (4). Please let me know of any errors, improvements, and fixes. I hope this walk-through might help others in the same situation.

flash that is used by Bosca Ceoil (but Bosca Ceoil is a free software).

What happened in the reproducible builds effort between April 3rd and April 9th 2016: Media coverage Emily Ratliff wrote an article for SecurityWeek called Establishing Correspondence Between an Application and its Source Code - How Combining Two Completely Separate Open Source Projects Can Make Us All More Secure. Tails have started work on a design for freezable APT repositories to make it easier and practical to perform reproductions of an entire distribution at a given point in time, which will be needed to create reproducible installation- or live-media. Toolchain fixes Alexis Bienven e submitted patches adding support for SOURCE_DATE_EPOCH in several tools: transfig, imagemagick, rdtool, and asciidoctor. boyska submitted one for python-reportlab. Packages fixed The following packages have become reproducible due to changes in their build dependencies: atinject-jsr330 brailleutils cglib3 gnugo libcobra-java libgnumail-java libjchart2d-java libjcommon-java libjfreechart-java libjide-oss-java liblaf-widget-java liblastfm-java liboptions-java octave-control octave-mpi octave-nan octave-parallel octave-stk octave-struct octave-tsa oar The following packages became reproducible after getting fixed:

apt-listchanges/2.88 by Robert Luberda.
cgal/4.8-1 by Joachim Reichel.
erlang-bitcask/2.0.2+dfsg-1 by Nobuhiro Iwamatsu.
geneweb/6.08+git20160228+dfsg-2 by Guillaume Brochu.
glance/2:12.0.0~rc2-1 uploaded by Thomas Goirand, original patch by Chris Lamb.
gle-graphics/4.2.5-4 by Christian T. Steigies.
maude uploaded by Andreas Tille, patch by Alexis Bienven e.
psqlodbc/1:09.05.0100-3 by Christoph Berg.
resource-agents/1:3.9.7-3 by Christoph Berg.
vgabios/0.7a-6 by Reiner Herrmann.
vim/2:7.4.1689-2 by James McCoy.
xmhtml/1.1.10-2 by Graham Inggs with Reiner Herrmann's help.
xscreensaver/5.34-2 uploaded by Tormod Volden, original patch by Sascha Steinbiss.

Several uploads fixed some reproducibility issues, but not all of them:

rkward/0.6.5-1 uploaded by Thomas Friedrichsmeier, original patch by Philip Rinn.
mailfilter/0.8.4-1 uploaded by Elimar Riesebieter, original patch by Chris Lamb.
bind9/1:9.10.3.dfsg.P4-6 uploaded by Michael Gilbert, original patch by Reiner Herrmann.
bzr/2.7.0- 3,4 by Jelmer Vernooij.
samba/2:4.3.6+dfsg-2 uploaded by Mathieu Parent, fix by Jelmer Vernooij.
fwupdate/0.5-3 by Mario Limonciello.
paraview/5.0.1+dfsg1-1 by Anton Gladky.

Patches submitted which have not made their way to the archive yet:

#819883 on debootstrap by Reiner Herrmann: tell tar to sort the archive members.
#819885 on chktex by Sascha Steinbiss: use the time of latest debian/changelog entry as documentation timestamp.
#819915 on kannel by Alexis Bienven e: use the time of latest debian/changelog entry as documentation timestamp.
#819921 on basket by Alexis Bienven e: remove build date from debug info.
#819965 on openarena-data by Alexandre Detiste: normalize file permissions before creating .pk3 archive.
#820016 on gabedit by Alexis Bienven e: sort object files used to build the executable.
#820032 on bibledit-gtk by Alexis Bienven e: remove useless included Makefile.
#820072 on synfig by Alexis Bienven e: remove build date from info output.
#820148 on autopkgtest by Alexis Bienven e: fix install order to cope with locales with case insensitive globbing.
#820152 on anope by Alexis Bienven e: remove build date from the version string.
#820179 on aodh by Alexis Bienven e: remove build date from the documentation.
#820183 on cython by Alexis Bienven e: add support SOURCE_DATE_EPOCH.
#820194 on nasm by rain1: sorts keys when traversing hash tables used to build the documentation.
#820226 on chrony by Alexis Bienven e: add support for SOURCE_DATE_EPOCH to preset the ntp_era_split parameter.
#820457 on recode by Alexis Bienven e: use system help2man.
#820522 on gtkspell by Alexis Bienven e: force shell to /bin/sh in example Makefile.

Other upstream fixes Alexander Batischev made a commit to make newsbeuter reproducible. tests.reproducible-builds.org

An architecture agnostic summary has been added to the reproducible-tracker.json by Valerie Young to make it easy to parse whether a package is unreproducible anywhere.
To find more reproducibility issues a new variation was added to the i386 builders, so that one build is done using a 32 bit kernel (686-PAE) and the other build is using a 64 bit kernel (amd64). (h01ger)
- Niko Tyni was the first to notice a bug due to this: #821182 perl: embeds kernel architecture information
The 2nd builds are now done in fr_CH on amd64, de_CH on i386 and it_CH on armhf. (h01ger)
The variation table has been updated to reflect the recent changes and various small bugs have been fixed. (h01ger)

Package reviews 93 reviews have been removed, 66 added and 21 updated in the previous week. 12 new FTBFS bugs have been reported by Chris Lamb and Niko Tyni. Misc. This week's edition was written by Lunar, Holger Levsen, Reiner Herrmann, Mattia Rizzolo and Ximin Luo. With the departure of Lunar as a full-time contributor, Reproducible Builds Weekly News (this thing you're reading) has moved from his personal Debian blog on Debian People to the Reproducible Builds team web site on Debian Alioth. You may want to update your RSS or Atom feeds. Very many thanks to Lunar for writing and publishing this weekly news for so long, well & continously!

Preparation for the release of TeX Live 2016 have started some time ago with the freeze of updates in TeX Live 2015. Yesterday we announced the official start of the pretest period. That means that we invite people to test the new release and help fixing bugs. At the same time I have uploaded the first set of packages of TeX Live 2016 for Debian to the experimental suite. texlive-2016-debian-pretest

Concerning the binaries we do expect a few further changes, but hopefully nothing drastic. The most invasive change on the tlmgr side is that cryptographic signatures are now verified to guarantee authenticity of the packages downloaded, but this is rather irrelevant for Debian users (though I will look into how that works in user mode). Other than that, many packages have been updated or added since the last Debian packages, here is the unified list: acro, animate, appendixnumberbeamer, arabluatex, asapsym, asciilist, babel-belarusian, bibarts, biblatex-bookinarticle, biblatex-bookinother, biblatex-caspervector, biblatex-chicago, biblatex-gost, biblatex-ieee, biblatex-morenames, biblatex-opcit-booktitle, bibtexperllibs, bxdvidriver, bxenclose, bxjscls, bxnewfont, bxpapersize, chemnum, cjk-ko, cochineal, csplain, cstex, datetime2-finnish, denisbdoc, dtx, dvipdfmx-def, ejpecp, emisa, fithesis, fnpct, font-change-xetex, forest, formation-latex-ul, gregoriotex, gzt, hausarbeit-jura, hyperxmp, imakeidx, jacow, l3, l3kernel, l3packages, latex2e, latex2e-help-texinfo-fr, latex-bib2-ex, libertinust1math, lollipop, lt3graph, lua-check-hyphen, lualibs, luamplib, luatexja, mathalfa, mathastext, mcf2graph, media9, metrix, nameauth, ndsu-thesis, newtx, normalcolor, noto, nucleardata, nwejm, ocgx2, pdfcomment, pdfpages, pkuthss, polyglossia, proposal, qcircuit, reledmac, rmathbr, savetrees, scanpages, stex, suftesi, svrsymbols, teubner, tex4ebook, tex-ini-files, tikzmark, tikzsymbols, titlesec, tudscr, typed-checklist, ulthese, visualtikz, xespotcolor, xetex-def, xetexko, ycbook, yinit-otf. Enjoy.

Since mid-2015 we were using the dep11-generator in Debian to build AppStream metadata about available software components in the distribution. Getting rid of dep11-generator Unfortunately, the old Python-based dep11-generator was hitting some hard limits pretty soon. For example, using multiprocessing with Python was a pain, since it resulted in some very hard-to-track bugs. Also, the multiprocessing approach (as opposed to multithreading) made it impossible to use the underlying LMDB database properly (it was basically closed and reopened in each forked off process, since pickling the Python LMDB object caused some really funny bugs, which usually manifested themselves in the application hanging forever without any information on what was going on). Additionally to that, the Python-based generator forced me to maintain two implementations of the AppStream YAML spec, one in C and one in Python, which consumes quite some time. There were also some other issues (e.g. no unit-tests) in the implementation, which made me think about rewriting the generator. Adventures in Go / Rust / D Since I didn t want to write this new piece of software in C (or basically, writing it in C was my last option

), I explored Go and Rust for this purpose and also did a small prototype in the D programming language, when I was starting to feel really adventurous. And while I never intended to write the new generator in D (I was pretty fixated on Go ), this is what happened. The strong points for D for this particular project were its close relation to C (and ease of using existing C code), its super-flat learning curve for someone who knows and likes C and C++ and its pretty powerful implementations of the concurrent and parallel programming paradigms. That being said, not all is great in D and there are some pretty dark spots too, mainly when it comes to the standard library and compilers. I will dive into my experiences with D in a separate blogpost. What good to expect from appstream-generator? So, what can the new appstream-generator do for you? Basically, the same as the old dep11-generator: It will extract metadata from a distribution s package archive, download and resize screenshots, search for icons and size them properly and generate reports in JSON and HTML of found metadata and issues. LibAppStream-based parsing, generation of YAML or XML, multi-distro support, As opposed to the old generator, the new generator utilizes the metadata parsers and writers of libappstream. This allows it to return the extracted metadata as AppStream YAML (for Debian) or XML (everyone else) It is also written in a distribution-agnostic way, so if someone wants to use it in a different distribution than Debian, this is possible now. It just requires a very small distribution-specific backend to be written, all of the details of the metadata extraction are abstracted away (just two interfaces need to be implemented). While I do not expect anyone except Debian to use this in the near future (most distros have found a solution to generate metadata already), the frontend-backend split is a much cleaner design than what was available in the previous code. It also allows to unit-test the code properly, without providing a Debian archive in the testsuite. Feature Flags, Optipng, The new generator also allows to enable and disable certain sets of features in a standardized way. E.g. Ubuntu uses a language-pack system for translations, which Debian doesn t use. Features like this can be implemented as disableable separate modules in the generator. We use this at time to e.g. allow descriptions from packages to be used as AppStream descriptions, or for running optipng on the generated PNG images and icons. No more Contents file dependency Another issue the old generator had was that it used the Contents file from the Debian archive to find matching icons for an application. We could never be sure whether the contents in the Contents file actually matched the contents of the package we were currently dealing with. What made things worse is that at Ubuntu, the archive software is only updating the Contents file ~~weekly~~ daily (while the generator might run multiple times a day), which has lead to software being ignored in the metadata, because icons could not yet be found. Even on Debian, with its quickly-updated Contents file, we could immediately see the effects of an out-of-date Contents file when updating it failed once. In the new generator, we read the contents of each package ourselves now and store them in a LMDB database, bypassing the Contents file and removing the whole class of problems resulting from missing or wrong contents-data. It can t all be good, right? That is true, there are also some known issues the new generator has: Large amounts of RAM required The better speed of the new generator comes at the cost of holding more stuff in RAM. Much more. When processing data from 5 architectures initially on Debian, the amount of required RAM might lie above 4GB, with the OOM killer sometimes being quicker than the garbage collector That being said, on subsequent runs the amount of required memory is much lower. Still, this is something I am working on to improve. What are symbolic links? To be faster, the appstream-generator will read the md5sum file in .deb packages instead of extracting the payload archive and reading its contents. Since the md5sums file does not list symbolic links, symlinks basically don t exist for the new generator. This is a problem for software symlinking icons or even .desktop files around, like e.g. LibreOffice does. I am still investigating how widespread the use of symlinks for icons and .desktop files is, but it looks like fixing packages (making them not-symlink stuff and rather move the files) might be the better approach than investing additional computing power to find symlinks or even switch back to parsing the Contents file. Input on this is welcome! Deploying asgen I finished the last pieces of the appstream-generator (together with doing lots of other cool things and talking to great people) at the GNOME Software Hackfest in London last week (detailed blogposts about things that happened there will follow many thanks once again for the Ubuntu community for sponsoring my attendance!). Since today, the new generator is running on the Debian infrastructure. If bigger issues are found, we can still roll back to the old code. I decided to deploy this faster, so we can get some good testing done before the Stretch release. Please report any issues you may find!

What happened in the reproducible builds effort between March 27th and April 2nd: Toolchain fixes

Emmanuel Bourg uploaded ant/1.9.6-2 which makes the Tstamp task support the SOURCE_DATE_EPOCH variable, and the Javadoc task use en as the default locale if none was specified and SOURCE_DATE_EPOCH is set.

Packages fixed The following packages have become reproducible due to changes in their build dependencies: ctioga2, erlang-bitcask, libcommons-collections3-java, libjgoodies-animation-java, libjide-oss-java, octave-gsl, octave-interval, octave-io, octave-quaternion, octave-signal, octave-stk, segment, service-wrapper-java, sqlline, svnkit, uddi4j, velocity-tools. The following packages became reproducible after getting fixed:

angular.js/1.3.20-3 uploaded by Laszlo Boszormenyi, patch by Dhole.
boa-constructor/0.6.1-16 by Reiner Herrmann.
fgetty/0.7-1 by Dmitry Bogatov.
hexchat/2.12.0-1 uploaded by Jesse Rhodes, fixed upstream.
littlewizard/1.2.2-4 uploaded by Kari Pahula, original patch by Reiner Herrmann.
logback/1:1.1.6-1 by Emmanuel Bourg.
pcsxr/1.9.94-2 by James Cowgill.
pybtex/0.19-2 by Daniel Stender.
python-structlog/16.0.0-1 by Filippo Giunchedi.
sbmltoolbox/4.1.0-3 by Afif Elghraoui, original patch by Reiner Herrmann.
tiles/2.2.2-7 by Emmanuel Bourg.

Some uploads fixed some reproducibility issues, but not all of them:

dash/0.5.8-2.2 uploaded by Niels Thykier, original patch by Lunar.
gle-graphics/4.2.5-4 by Christian T. Steigies.
prospector/0.11.7-6 by Daniel Stender.
therion/5.3.16-10 by Wookey.
vim/2:7.4.1689-1 uploaded by James McCoy, original patch by Reiner Herrmann.

Patches submitted which have not made their way to the archive yet:

#783239 on kexec-tools by Lunar: follow-up patch to cope with locale variations.
#819347 on starvoyager by Sascha Steinbiss: sort the list of input object files.
#819352 on xpdf by Sascha Steinbiss: sort the list of linked object files.
#819512 on breeze by Dhole: force grep to treat all files as text to avoid locale-related issues.
#819726 on ckbuilder by boyska: add support for SOURCE_DATE_EPOCH.
#819767 on libtool by rain1: removes extra timestamps from the build system, ensure a stable file order when creating the source archive, and replace uses of the hostname command with the fixed string "localhost".

tests.reproducible-builds.org The i386 builders are now testing packages on i386 for reproducibility. It will probably take 4 weeks until everything has been build twice, on this arch. (h01ger) Package reviews 52 reviews have been removed, 24 added and 4 updated in the previous week. Chris Lamb reported 13 new FTBFS. New issue: copyright_year_in_comments_generated_by_ckbuilder. Misc. This week's edition was mostly written by Lunar, with some help by Reiner Herrmann and h01ger.

Also titled minimal images - maximal effort! The OpenStack Infrastructure Team manages a large continuous-integration system that provides the broad range of testing the OpenStack project requires. Tests are run thousands of times a day across every project, on multiple platforms and on multiple cloud-providers. There are essentially no manual steps in any part of the process, with every component being automated via scripting, a few home-grown tools and liberal doses of Puppet and Ansible. More importantly, every component resides in the public git trees right alongside every other OpenStack project, with contributions actively encouraged. As with any large system, technical debt can build up and start to affect stability and long-term maintainability. OpenStack Infrastructure can see some of this debt accumulating as more testing environments across more cloud-providers are being added to support ever-growing testing demands. Thus a strong focus of recent work has been consolidating testing platforms to be smaller, better defined and more maintainable. This post illustrates some of the background to the issues and describes how these new platforms are more reliable and maintainable.

OpenStack CI Overview Before getting into details, it's a good idea to get a basic big-picture conceptual model of how OpenStack CI testing works. If you look at the following diagram and follow the numbers with the explanation below, hopefully you'll have all the context you need. Overview of OpenStack CI

The developer uploads their code to gerrit via the git-review tool. There is no further action required on their behalf and the developer simply waits for results.
Gerrit provides a JSON-encoded "firehose" output of everything happening to it. New reviews, votes, updates and more all get sent out over this pipe. Zuul is the overall scheduler that subscribes itself to this information and is responsible for managing the CI jobs appropriate for each change.
Zuul has a configuration that tells it what jobs to run for what projects. Zuul can do lots of interesting things, but for the purposes of this discussion we just consider that it puts the jobs it wants run into gearman for a Jenkins master to consume. gearman is a job-server; as they explain it "[gearman] provides a generic application framework to farm out work to other machines or processes that are better suited to do the work". Zuul puts into gearman basically a tuple (job-name, node-type) for each job it wants run, specifying the unique job name to run and what type of node it should be run on.
A group of Jenkins masters are subscribed to gearman as workers. It is these Jenkins masters that will consume the job requests from the queue and actually get the tests running. However, Jenkins needs two things to be able to run a job a job definition (what to actually do) and a slave node (somewhere to do it). The first part what to do is provided by job-definitions stored in external YAML files and processed by Jenkins Job Builder (jjb) in to job configurations for Jenkins. Each Jenkins master gets these definitions pushed to it constantly by Puppet, thus each Jenkins master instance knows about all the jobs it can run automatically. Zuul also knows about these job definitions; this is the job-name part of the tuple we said it put into gearman. The second part somewhere to run the test takes some more explaining. To the next point...
Several cloud companies donate capacity in their clouds for OpenStack to run CI tests. Overall, this capacity is managed by a customised orchestration tool called nodepool. Nodepool watches the gearman queue and sees what requests are coming out of Zuul. It looks at node-type of jobs in the queue and decides what types of nodes need to start and which cloud providers have capacity to satisfy demand. Nodepool will monitor the start-up of the virtual-machines and register the new nodes to the Jenkins master instances.
At this point, the Jenkins master has what it needs to actually get jobs started. When nodepool registers a host to a Jenkins master as a slave, the Jenkins master can now advertise its ability to consume jobs. For example, if a ubuntu-trusty node is provided to the Jenkins master instance by nodepool, Jenkins can now consume from gearman any job it knows about that is intended to run on an ubuntu-trusty slave. Jekins will run the job as defined in the job-definition on that host ssh-ing in, running scripts, copying the logs and waiting for the result. (It is a gross oversimplification, but for the purposes of OpenStack CI, Jenkins is pretty much used as a glorified ssh/scp wrapper. Zuul Version 3, under development, is working to remove the need for Jenkins to be involved at all).
Eventually, the test will finish. The Jenkins master will put the result back into gearman, which Zuul will consume. The slave will be released back to nodepool, which destroys it and starts all over again (slaves are not reused and also have no sensitive details on them, as they are essentially publicly accessible). Zuul will wait for the results of all jobs for the change and post the result back to Gerrit; it either gives a positive vote or the dreaded negative vote if required jobs failed (it also handles merges to git, but we'll ignore that bit for now).

In a nutshell, that is the CI work-flow that happens thousands-upon-thousands of times a day keeping OpenStack humming along.

Image builds So far we have glossed over how nodepool actually creates the images that it hands out for testing. Image creation, illustrated in step 8 above, contains a lot of important details. Firstly, what are these images and why build them at all? These images are where the "rubber hits the road" they are instantiated into the virtual-machines that will run DevStack, unit-testing or whatever else someone might want to test. The main goal is to provide a stable and consistent environment in which to run a wide-range of tests. A full OpenStack deployment results in hundreds of libraries and millions of lines of code all being exercised at once. The testing-images are right at the bottom of all this, so any instability or inconsistency affects everyone; leading to constant fire-firefighting and major inconvenience as all forward-progress stops when CI fails. We want to support a wide number of platforms interesting to developers such as Ubuntu, Debian, CentOS and Fedora, and we also want to and make it easy to handle new releases and add other platforms. We want to ensure this can be maintained without too much day-to-day hands-on. Caching is a big part of the role of these images. With thousands of jobs going on every day, an occasional network blip is not a minor annoyance, but creates constant and difficult to debug failures. We want jobs to rely on as few external resources as possible so tests are consistent and stable. This means caching things like the git trees tests might use (OpenStack just broke the 1000 repository mark), VM images, packages and other common bits and pieces. Obviously a cache is only as useful as the data in it, so we build these images up every day to keep them fresh.

Snapshot images If you log into almost any cloud-provider's interface, they almost certainly have a range of pre-canned images of common distributions for you to use. At first, the base images for OpenStack CI testing came from what the cloud-providers had as their public image types. However, over time, there are a number of issues that emerge:

No two images, even for the same distribution or platform, are the same. Every provider seems to do something "helpful" to the images which requires some sort of workaround.
Providers rarely leave these images alone. One day you would boot the image to find a bunch of Python libraries pip-installed, or a mount-point moved, or base packages removed (all happened).
Even if the changes are helpful, it does not make for consistent and reproducible testing if every time you run, you're on a slightly different base system.
Providers don't have some images you want (like a latest Fedora), or have different versions, or different point releases. All update asynchronously whenever they get around to it.

So the original incarnations of OpenStack CI images were based on these public images. Nodepool would start one of these provider images and then run a series of scripts on it these scripts would firstly try to work-around any quirks to make the images look as similar as possible across providers, and then do the caching, setup things like authorized keys and finish other configuration tasks. Nodepool would then snapshot this prepared image and start instantiating VM's from these snapshots into the pool for testing. If you hear someone talking about a "snapshot image" in OpenStack CI context, that's likely what they are referring to. Apart from the stability of the underlying images, the other issue you hit with this approach is that the number of images being built starts to explode when you take into account multiple providers and multiple regions. Even with just Rackspace and the (now defunct) HP Cloud we would end up creating snapshot images for 4 or 5 platforms across a total of about 8 regions meaning anywhere up to 40 separate image builds happening daily (you can see how ridiculous it was getting in the logging configuration used at the time). It was almost a fait accompli that some of these would fail every day nodepool can deal with this by reusing old snapshots but this leads to a inconsistent and heterogeneous testing environment. Naturally there was a desire for something more consistent a single image that could run across multiple providers in a much more tightly controlled manner.

Upstream-based builds Upstream distributions do provide "cloud-images", which are usually pre-canned .qcow2 format files suitable for uploading to your average cloud. So the diskimage-builder tool was put into use creating images for nodepool, based on these upstream-provided images. In essence, diskimage-builder uses a series of elements (each, as the name suggests, designed to do one thing) that allow you to build a completely customised image. It handles all the messy bits of laying out the image file, tries to be smart about caching large downloads and final things like conversion to qcow2 or vhd. nodepool has used diskimage-builder to create customised images based upon the upstream releases for some time. These are better, but still have some issues for the CI environment:

You still really have no control over what does or does not go into the upstream base images. You don't notice a change until you deploy a new image based on an updated version and things break.
The images still start with a fair amount of "stuff" on them. For example cloud-init is a rather large Python program and has a fair few dependencies. These dependencies can both conflict with parts of OpenStack or end up tacitly hiding real test requirements (the test doesn't specify it, but the package is there as part of another base dependency. Things then break when the base dependencies change). The whole idea of the CI is that (as much as possible) you're not making any assumptions about what is required to run your tests you want everything explicitly included.
An image that "works everywhere" across multiple cloud-providers is quite a chore. cloud-init hasn't always had support for config-drive and Rackspace's DHCP-less environment, for example. Providers all have their various different networking schemes or configuration methods which needs to be handled consistently.

If you were starting this whole thing again, things like LXC/Docker to keep "systems within systems" might come into play and help alleviate some of the packaging conflicts. Indeed they may play a role in the future. But don't forget that DevStack, the major CI deployment mechanism, was started before Docker existed. And there's tricky stuff with networking and Neutron going on. And things like iSCSI kernel drivers that containers don't support well. And you need to support Ubuntu, Debian, CentOS and Fedora. And you have hundreds of developers already relying on what's there. So change happens incrementally, and in the mean time, there is a clear need for a stable, consistent environment.

Minimal builds To this end, diskimage-builder now has a serial of "minimal" builds that are really that systems with essentially nothing on them. For Debian and Ubuntu this is achieved via debootstrap, for Fedora and CentOS we replicate this with manual installs of base packages into a clean chroot environment. We add on a range of important elements that make the image useful; for example, for networking, we have simple-init which brings up the network consistently across all our providers but has no dependencies to mess with the base system. If you check the elements provided by project-config you can see a range of specific elements that OpenStack Infra runs at each image build (these are actually specified by in arguments to nodepool, see the config file, particularly diskimages section). These custom elements do things like caching, using puppet to install the right authorized_keys files and setup a few needed things to connect to the host. In general, you can see the logs of an image build provided by nodepool for each daily build. So now, each day at 14:14 UTC nodepool builds the daily images that will be used for CI testing. We have one image of each type that (theoretically) works across all our providers. After it finishes building, nodepool uploads the image to all providers (p.s. the process of doing this is so insanely terrible it spawned shade; this deserves many posts of its own) at which point it will start being used for CI jobs. If you wish to replicate this entire process, the build-image.sh script, run on an Ubuntu Trusty host in a virtualenv with diskimage-builder will get you pretty close (let us know of any issues!).

DevStack and bare nodes There are two major ways OpenStack projects test their changes:

Running with DevStack, which brings up a small, but fully-functional, OpenStack cloud with the change-under-test applied. Generally tempest is then used to ensure the big-picture things like creating VM's, networks and storage are all working.
Unit-testing within the project; i.e. what you do when you type tox -e py27 in basically any OpenStack project.

To support this testing, OpenStack CI ended up with the concept of bare nodes and devstack nodes.

A bare node was made for unit-testing. While tox has plenty of information about installing required Python packages into the virtualenv for testing, it doesn't know anything about the system packages required to build those Python packages. This means things like gcc and library -devel packages which many Python packages use to build bindings. Thus the bare nodes had an ever-growing and not well-defined list of packages that were pre-installed during the image-build to support unit-testing. Worse still, projects didn't really know their dependencies but just relied on their testing working with this global list that was pre-installed on the image.
In contrast to this, DevStack has always been able to bootstrap itself from a blank system to a working OpenStack deployment by ensuring it has the right dependencies installed. We don't want any packages pre-installed here because it hides actual dependencies that we want explicitly defined within DevStack otherwise, when a user goes to deploy DevStack for their development work, things break because their environment differs slightly to the CI one. If you look at all the job definitions in OpenStack, by convention any job running DevStack has a dsvm in the job name this referred to running on a "DevStack Virtual Machine" or a devstack node. As the CI environment has grown, we have more and more testing that isn't DevStack based (puppet apply tests, for example) that rather confusingly want to run on a devstack node because they do not want dependencies installed. While it's just a name, it can be difficult to explain!

Thus we ended up maintaining two node-types, where the difference between them is what was pre-installed on the host and yes, the bare node had more installed than a devstack node, so it wasn't that bare at all!

Specifying Dependencies Clearly it is useful to unify these node types, but we still need to provide a way for the unit-test environments to have their dependencies installed. This is where a tool called bindep comes in. This tool gives project authors a way to specify their system requirements in a similar manner to the way their Python requirements are kept. For example, OpenStack has the concept of global requirements those Python dependencies that are common across all projects so version skew becomes somewhat manageable. This project now has some extra information in the other-requirements.txt file, which lists the system packages required to build the Python packages in the global-requirements list. bindep knows how to look at these lists provided by projects and get the right packages for the platform it is running on. As part of the image-build, we have a cache-bindep element that can go through every project and build a list of the packages it requires. We can thus pre-cache all of these packages onto the images, knowing that they are required by jobs. This both reduces the dependency on external mirrors and improves job performance (as the packages are locally cached) but doesn't pollute the system by having everything pre-installed. Package installation can now happen via the way we really should be doing it as part of the CI job. There is a job-macro called install-distro-packages which a test can use to call bindep to install the packages specified by the project before the run. You might notice the script has a "fallback" list of packages if the project does not specify it's own dependencies this essentially replicates the environment of a bare node as we transition to projects more strictly specifying their system requirements. We can now start with a blank image and all the dependencies to run the job can be expressed by and within the project leading to a consistent and reproducible environment without any hidden dependencies. Several things have broken as part of removing bare nodes this is actually a good thing because it means we have revealed areas where we were making assumptions in jobs about what the underlying platform provides. There's a few other job-macros that can do things like provide MySQL/Postgres instances for testing or setup other common job requirements. By splitting these types of things out from base-images we also improve the performance of jobs who don't waste time doing things like setting up databases for jobs that don't need it. As of this writing, the bindep work is new and still a work-in-progress. But the end result is that we have no more need for a separate bare node type to run unit-tests. This essentially halves the number of image-builds required and brings us to the goal of a single image for each platform running all CI.

Conclusion While dealing with multiple providers, image-types and dependency chains has been a great effort for the infra team, to everyone's credit I don't think the project has really noticed much going on underneath. OpenStack CI has transitioned to a situation where there is a single image type for each platform we test that deploys unmodified across all our providers and runs all testing environments equally. We have better insight into our dependencies and better tools to manage them. This leads to greatly decreased maintenance burden, better consistency and better performance; all great things to bring to OpenStack CI!

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. I skipped my monthly report last time so this one will cover two months. I will try to list only the most important things to not make it too long.

The Debian Handbook I worked with Ryuunosuke Ayanokouzi to prepare a paperback version of the Japanese translation of my book. Thanks to the efforts of everybody, it s now available. Unfortunately, Lulu declined to take it in distribution program so it won t be available on traditional bookstores (like Amazon, etc.). The reason is that they do not support non-latin character sets in the meta-data. I tried to cheat a little bit by inputting the description in English (still explaining that the book was in Japanese) but they rejected it nevertheless because the English title could mislead people. So the paperback is only available on lulu.com. Fortunately, the shipping costs are reasonable if you pick the most economic offer. Following this I invited the Italian, Spanish and Brazilian Portuguese translators to complete the work (they were close will all the strings already translated, mainly missing translated screenshots and some backcover content) so that we can also release paperback versions in those languages. It s getting close to completion for them. Hopefully we will have those available until next month. Distro Tracker In early February, I tweaked the configuration to send (by email) exceptions generated by incoming mails and by routine task. Before this they were logged but I did not take the time to look into them. This quickly brought a few issues into light and I fixed them as they appeared: for instance the bounce handling code was getting confused when the character case was not respected, and it appears that some emails come back to us after having been lowercased. Also the code was broken when the References field used more than one line on incoming control emails. This brought into light a whole class of problems with the database storing twice the same email with only differing case. So I did further work to merge all those duplicate entries behind a single email entry. Later, the experimental Sources files changed and I had to tweak the code to work with the removal of the Files field (relying instead on Checksums-* to find out the various files part of the entry). At some point, I also fixed the login form to not generate an exception when the user submits an empty form. I also decided that I no longer wanted to support Django 1.7 in distro tracker as Django 1.8 is the current LTS version. I asked the Debian system administrators to update the package on tracker.debian.org with the version in jessie-backports. This allowed me to fix a few deprecation warnings that I kept triggering because I wanted the code to work with Django 1.7. One of those warnings was generated by django-jsonfield though and I could not fix it immediately. Instead I prepared a pull request that I submitted to the upstream author. Oh, and a last thing, I tweaked the CSS to densify the layout on the package page. This was one of the most requested changes from the people who were still preferring packages.qa.debian.org over tracker.debian.org. Kali and new pkg-security team As part of my Kali work, I have been fixing RC bugs in Debian packages that we use in Kali. But in many cases, I stumbled upon packages whose maintainers were really missing in action (MIA). Up to now, we were only doing non-maintainers upload (NMU) but I want to be able to maintain those packages more effectively so we created a new pkg-security team (we re only two right now and we have no documentation yet, but if you want to join, you re welcome, in particular if you maintain a package which is useful in the security field). arm64 work. The first 3 packages that we took over (ssldump, sucrack, xprobe) are actually packages that were missing arm64 builds. We just started our arm64 port on Kali and we fixed them for that architecture. Since they were no longer properly maintained, in most cases it was just a matter of using dh_autoreconf to get up-to-date config. sub,guess files. We still miss a few packages on arm64: vboot-utils (that we will likely take over soon since it s offered for adoption), ruby-libv8 and ruby-therubyracer, ntopng (we have to wait a new luajit which is only in experimental right now). We also noticed that dh-make-golang was not available on arm64, after some discussion on #debian-buildd, I filed two bugs for this: #819472 on dh-make-golang and #819473 on dh-golang. RC bug fixing. hdparm was affected by multiple RC bugs and the release managers were trying to get rid of it from testing. This removed multiple packages that were used by Kali and its users. So I investigated the situation of that package, convinced the current maintainers to orphan it, asked for new maintainers on debian-devel, reviewed multiple updates prepared by the new volunteers and sponsored their work. Now hdparm is again RC-bug free and has the latest upstream version. We also updated jsonpickle to 0.9.3-1 to fix RC bug #812114 (that I forwarded upstream first). Systemd presets support in init-system-helpers. I tried to find someone (to hire) to implement the system preset feature I requested in #772555 but I failed. Still Andreas Henriksson was kind enough to give it a try and sent a first patch. I tried it and found some issues so I continued to improve it and simplify it I submitted an updated patch and pinged Martin Pitt. He pointed me to the DEP-8 test failures that my patch was creating. I quickly fixed those afterwards. This patch is in use in Kali and lets us disable network services by default. I would like to see it merged in Debian so that everybody can setup systemd preset file and have their desire respected at installation time. Misc bug reports. I filed #813801 to request a new upstream release of kismet. Same for masscan in #816644 and for wkhtmltopdf in #816714. We packaged (before Debian) a new upstream release of ruby-msgpack and found out that it was not building on armel/armhf so we filed two upstream tickets (with a suggested fix). In #814805, we asked the pyscard maintainer to reinstate python-pyscard that was dropped (keeping only the Python3 version) as we use the Python 2 version in Kali. And there s more: I filed #816553 (segfault) and #816554 against cdebootstrap. I asked for dh-python to have a better behaviour after having being bitten by the fact that dh with python3 was not doing what I expected it to do (see #818175). And I reported #818907 against live-build since it is failing to handle a package whose name contains an upper case character (it s not policy compliant but dpkg supports them). Misc packaging I uploaded Django 1.9.2 to unstable and 1.8.9 to jessie-backports. I provided the supplementary information that Julien Cristau asked me in #807654 but despite this, this jessie update has been ignored for the second point release in a row. It is now outdated until I update it to include the security fixes that have been released in the mean time but I m not yet sure that I will do it the lack of cooperation of the release team for that kind of request is discouraging. I sponsored multiple uploads of dolibarr (on security update notably) and tcpdf (to fix one RC bug). Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

A large part of OpenStack Infrastructure teams recent efforts has been focused on moving towards more stable and maintainable CI environments for testing.

The developer uploads their code to gerrit via the git-review tool. They wait.
Gerrit provides a JSON-encoded "firehose" output of everything happening to it. New reviews, votes, updates and more all get sent out over this pipe. Zuul is the overall scheduler that subscribes itself to this information and is responsible for managing the CI jobs appropriate for each change.
Zuul has a configuration that tells it what jobs to run for what projects. Zuul can do lots of interesting things, but for the purposes of this discussion we just consider that it puts the jobs it wants run into gearman for a Jenkins host to consume. gearman is a job-server; as they explain it "[gearman] provides a generic application framework to farm out work to other machines or processes that are better suited to do the work".
A group of Jenkins hosts are subscribed to gearman as workers. It is these Jenkins hosts that will consume the job requests from the queue and actually get the tests running. Jenkins needs two things to be able to run a job -- a job definition (what to actually do) and a slave node (somewhere to do it). The first part -- what to do -- is provided by job-definitions stored in external YAML files and processed by Jenkins Job Builder (jjb) in to job configurations for Jenkins. Thus each Jenkins instance knows about all the jobs it might need to run. Zuul also knows about these job definitions, so you can see how we now have a mapping where Zuul can put a job into gearman saying "run test foo-bar-baz" and a Jenkins host can consume that request and know what to do. The second part -- somewhere to run the test -- takes some more explaining. To the next point...
Several cloud companies donate capacity in their clouds for OpenStack to run CI tests. Overall, this capacity is managed by nodepool -- a customised orchestration tool. Nodepool watches the gearman queue and sees what requests are coming out of Zuul, and decides what type of capacity to provide and in what clouds to satisfy the outstanding job queue. Nodepool will start-up virtual-machines as required, and register those nodes to the Jenkins instances.
At this point, Jenkins has what it needs to actually get jobs started. When nodepool registers a host to Jenkins as a slave, the Jenkins host can now advertise its ability to consume jobs. For example, if a ubuntu-trusty node is provided to the Jenkins instance by nodepool, Jenkins can now consume a job from the queue intended to run on an ubuntu-trusty host. It will run the job as defined in the job-definition -- ssh-ing into the host, running scripts, copying the logs and waiting for the result. (It is a gross oversimplification, but but Jenkins is pretty much a glorified ssh/scp wrapper to OpenStack CI. Zuul Version 3, under development, is working to remove the need for Jenkins to be involved at all).
Eventually, the test will finish. Jenkins will put the result back into gearman, which Zuul will consume. The slave will be released back to nodepool, which destroys it and starts all over again (slaves are not reused and also have no sensitive details on them, as they are essentially publicly accessible). Zuul will wait for the results of all jobs and post the result back to Gerrit and give either a positive vote or the dreaded negative vote if required jobs failed (it also handles merges to git, but we'll ignore that bit for now).

In a nutshell, that is the CI work-flow that happens thousands-upon-thousands of times a day keeping OpenStack humming along.

Image builds There is, however, another more asynchronous part of the process that hides a lot of details the rest of the system relies on. Illustrated in step 8 above, this is the management of the images that tests are being run upon. Above we we said that a test runs on a ubuntu-trusty, centos, fedora or some other type of node, but glossed over where these images come from. Firstly, what are these images, and why build them at all? These images are where the "rubber hits the road" -- where DevStack, functional testing or whatever else someone might want to test is actually run for real. Caching is a big part of the role of these images. With thousands of jobs going on every day, an occasional network blip is not a minor annoyance, but creates constant and difficult to debug CI failures. We want the images that CI runs on to rely on as few external resources as possible so test runs are as stable as possible. This means caching all the git trees tests might use, VM images consumed by various tests and other common bits and pieces. Obviously a cache is only as useful as the data in it, so we build these images up every day to keep them fresh.

Provider images If you log into almost any cloud-provider's interface, they almost certainly have a range of pre-canned images of common distributions for you to use. At first, the base images for OpenStack CI testing came from what the cloud-providers had as their public image types. However, over time, there are a number of issues that emerge:

No two images, even for the same distribution or platform, are the same. Every provider seems to do something "helpful" to the images which requires some sort of workaround.
Providers rarely leave these images alone. One day you would boot the image to find a bunch of Python libraries pip-installed, or a mount-point moved, or base packages removed (all happened).
Even if the changes are helpful, it does not make for consistent and reproducible testing if every time you run, you're on a slightly different base system.
Providers don't have some images you want (like a latest Fedora), or have different versions, or different point releases. All update asynchronously whenever they get around to it.

So the original incarnations of building images was that nodepool would start one of these provider images, run a bunch of scripts on it to make a base-image (do the caching, setup keys, etc), snapshot it and then start putting VM's based on these images into the pool for testing (if you hear someone talking about a "snapshot image" in OpenStack CI context, that's what it means). The first problem you hit here is that the number of images being built starts to explode when you take into account multiple providers and multiple regions. Even with just Rackspace and the (now defunct) HP Cloud we would end up snapshot images for 4 or 5 platforms across a total of about 8 regions -- meaning anywhere up to 40 separate image builds happening. It was almost a fait accompli that some images would fail every day -- nodepool can deal with this by reusing old snapshots; but this leads to a inconsistent and heterogeneous testing environment. OpenStack is like a gigantic Jenga tower, with a full DevStack deployment resulting in hundreds of libraries and millions of lines of code all being exercised at once. The testing images are right at the bottom of all this, and it doesn't take much to make the whole thing fall over (see points about providers not leaving images alone). This leads to constant fire-firefighting and everyone annoyed as all CI stops. Naturally there was a desire for something much more consistent -- a single image that could run across multiple providers in a much more tightly controlled manner.

Upstream-based builds Upstream distributions do provide their "cloud-images", which are usually pre-canned .qcow2 format files suitable for uploading to your average cloud. So the diskimage-builder tool was put into use creating images for nodepool, based on these upstream-provided images. In essence, diskimage-builder uses a series of elements (each, as the name suggests, designed to do one thing) that allow you to build a completely customised image. It handles all the messy bits of laying out the image file, tries to be smart about caching large downloads and final things like conversion to qcow2 or vhd or whatever your cloud requires. nodepool has used diskimage-builder to create customised images based upon the upstream releases for some time. These are better, but still have some issues for the CI environment:

You still really have no control over what does or does not go into the upstream base images. You don't notice a change until you deploy a new image based on an updated version and things break.
The images still have a fair amount of "stuff" on them. For example cloud-init is a rather large Python program and has a fair few dependencies. These dependencies can both conflict with parts of OpenStack or end up tacitly hiding real test requirements (the test doesn't specify it, but the package is there as part of another base dependency -- things break when the base dependencies change). The whole idea of the CI is that (as much as possible) you're not making any assumptions about what is required to run your tests -- you want everything explicitly included. (If you were starting this whole thing again, things like Docker to keep "systems within systems" might come into play. Indeed they may be in the future. But don't forget that DevStack, the major CI deployment mechanism, was started before Docker existed. And there's tricky stuff with networking and Neutron etc going on).
An image that "works everywhere" across multiple cloud-providers is quite a chore. cloud-init hasn't always had support for config-drive and Rackspace's DHCP-less environment, for example. Providers all have their various different networking schemes or configuration methods which needs to be handled consistently.

Minimal builds To this end, diskimage-builder now has a serial of "minimal" builds that are really that -- systems with essentially nothing on them. For Debian and Ubuntu, this is achieved via debootstrap, for Fedora and CentOS we replicate this with manual installs of base packages into a clean chroot environment. We add-on a range of important elements that make the image useful; for example, for networking, we have simple-init which brings up the network consistently across all our providers but has no dependencies to mess with the base system. If you check the elements provided by project-config you can see a range of specific elements that OpenStack Infra runs at each image build (these are actually specified by in arguments to nodepool, see the config file, particularly diskimages section). These custom elements do things like caching, using puppet to install the right authorized_keys files and setup a few needed things to connect to the host. In general, you can see the logs of an image build provided by nodepool for each daily build. So now, each day at 14:00 UTC nodepool builds the daily images that will be used for CI testing. We have one image of each type that (theoretically) works across all our providers. After it finishes building, nodepool uploads the image to all providers (p.s. the process of doing this is so insanely terrible it spawned shade; this deserves many posts of its own) at which point it will start being used for CI jobs. If you wish to replicate this entire process, the build-image.sh script, run on an Ubuntu Trusty host in a virtualenv with diskimage-builder will get you pretty close (let us know of any issues!).

Dependencies But guess what, there's more! Along the way, OpenStack CI ended up with the concept of bare nodes and devstack nodes. A bare node was one that was used for functional testing; i.e. what you do when you type tox -e py27 in basically any OpenStack project. The problem here is that tox has plenty of information about installing required Python packages into the virtualenv for testing; but it doesn't know anything about the system packages required to build the Python libraries. This means things like gcc and -devel packages which many Python libraries use to build library bindings. In contrast to this, DevStack has always been able to bootstrap itself from a blank system to a working OpenStack deployment, ensuring it has the right libraries, etc, installed to get everything working. If you look at all the job definitions, anything running DevStack has a dsvm in the job name; which referred to to "DevStack Virtual Machine"; or basically specifying what was installed on the host (and yes, the bare node had more installed, so it wasn't that bare at all!). Thus we don't want packages pre-installed for DevStack, because it hides actual devstack dependencies that we want explicitly defined. But the bare nodes, used for functional testing, were different -- there was an every-growing and not well-defined list of packages that were pre-installed on those nodes to make sure functional testing worked. In general, you don't want jobs relying on something like this; we want to be sure if jobs have a dependency, they require it explicitly. This is where a tool called bindep comes in. OpenStack has the concept of global requirements -- those Python dependencies that are common across all projects so version skew becomes somewhat manageable. This now has some extra information in the other-requirements.txt file, which lists the system-packages required to build the Python-packages in the requirements list. bindep knows how to look at this and get the right packages for the platform it is running on. Indeed -- remember how it was previously mentioned we want to minimise dependencies on external resources at runtime? Well we can pre-cache all of these packages onto the images, knowing that they are likely to be required by packages. How do we get the packages installed? The way we really should be doing it -- as part of the CI job. There is a macro called install-distro-packages which uses bindep to install those packages as required by the global-requirements list. The result -- no more need for the bare node type to run functional tests! In all cases we can start with essentially a blank image and all the dependencies to run the job are expressed by and within the job -- leading to a consistent and reproducible environment. Several things have broken as part of removing bare nodes -- this is actually a good thing because it means we have revealed areas where we were making assumptions in jobs about what the underlying platform provides; issues that get fixed by thinking about and ensuring we have correct dependencies bringing up jobs. There's a few other macros there that do things like provide MySQL/Postgres instances or setup other common job requirements. By splitting these out we also improve the performance of jobs who now only bring in the dependencies they need -- we don't waste time doing things like setting up databases for jobs that don't need it.

Conclusion While dealing with multiple providers, image-types and dependency chains has been a great effort for the infra team, to everyone's credit I don't think the project has really noticed much going on underneath. OpenStack CI has transitioned to a situation where there is a single image type for each platform we test that deploys unmodified across all our providers. We have better insight into our dependencies and better tools to manage them. This leads to greatly decreased maintenance burden, better consistency and better performance; all great things to bring to OpenStack CI!

FTP assistant This month I marked 226 packages for accept and rejected 22. I also sent 5 emails to maintainers asking questions. It seems to be that a rather quiet month is behind us. As I have seen some packages with strange debian/copyright in binNEW, I wonder whether also the archive should be checked regularly. Maybe it is time to file some bugs Debian LTS This was my twenty-first month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. Due to outstanding hours that were redistributed, my all in all workload had been 14.25h. As Wheezy LTS didn t start yet and I am not able to do normal security uploads, I sent debdiffs to the security team. Btw. this can be done by everybody and the way to go is described in chapter 5.8.5 of the Debian Developer s Reference. Altogether I sent the following debdiffs for

extplorer to fix CVE-2015-0896
inspircd to fix CVE-2015-8702
libmatroska to fix CVE-2015-8792
libstruts1.2-java to fix CVE-2015-0899
fuseiso to fix two temporary issues
minissdpd to fix CVE-2016-3178 and CVE-2016-3179
tlslite to fix CVE-2015-3220

As the security team wants to update Wheezy and Jessie with only one DSA, whenever applicable I created debdiffs for both releases. Up to now the results can be seen in DSA 3526-1, DSA 3527-1 and DSA 3536-1. As tlslite has been removed from Wheezy during today s point release, I am afraid that was a wasted effort. Other stuff My node activities this month involved uploads of: node-component-consoler, node-generator-supported, node-xmlhttprequest-ssl, node-co, node-uid-umber, node-url-join, node-uri-path, node-read-file, node-nth-check, node-base62, node-require-dir, node-for-in, node-obj-util, node-normalize-it-url, node-delve, node-function-bind, node-seq, node-json-localizer, node-through, node-addressparser, node-ansi-regex, node-crypto-cacerts, node-decamelize, node-array-find-index, node-require-main-filename, node-invert-kv, node-starttls. To fix one or the other bug I also uploaded: node-connect, node-mysql. I also forwarded bug #809252, which is tagged as security relevant in the BTS, to the Node Security Project. I even got one answer stating that the report arrived. We will see what happens next. At least after 45 days another email might arrive

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

Submitted a number of pull requests to the Ansible sever configuration tool:
- Add the ability to specify the --allow-unauthenticated to APT. (#2023)
- Ignore EPIPE when flushing standard input and output to avoid unnecessary and ugly tracebacks. (#14774)
- Add an option for ansible-playbook to exit with an error if no plays were matched. (#14742)
Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
- Merged a patch from Joshua Blum to update the logging integration documentation. (#39)
- Fixed an issue when rendering various exceptions would in themselves raise an exception. (#40)
- Improved and merged a patch from Patrick Cloke to actually allow settings to be overriden using the @override_settings decorator. (#37)
Updated local-debian-mirror, a package that attempts to make it easy to maintain a partial local mirror of the Debian archive:
- Added an option to exclude DEP-11 files. (ee03b21)
- Added an option to exclude Translation-* files. (28bbf34)
- Corrected the phrasing of the debconf prompt to not ask a direct question to appease Lintian. (13fccc7b)
Updated django-keyerror a library to post exceptions to the KeyError.com error tracking service to support pushing tracebacks from the Celery queue processor using the task_failure signal. (#1)
Submitted a number of pull requests for django-zebra, a collection of utilities that make it easier to integrate the Stripe payment processor and the Django web development framework:
- Add the ability to test an incoming Stripe webhook. (#44)
- Prefer @require_POST over an explicit check in the view. (#43)
- Move away from the deprecated django.conf.urls.patterns method. (#45)
Updated django-template-tests, a tool to perform simple static analysis on Django templates prior to production to improve the robustness of dynamic test generation. (e98f9fc)
Fixed django-force-logout a library to allow administrators to log user's out of Django projects to not require a custom JSON serialiser. (c0e5d64)
Corrected the documentation for travis.debian.net my hosted script to easily test and build Debian packages on the Travis CI continuous integration platform to reflect the actual default mirror. (4c862f0)
Blogged about generating dynamic Python tests using metaclasses.

Debian

Presented Reproducible Builds - fulfilling the original promise of free software at FOSSASIA '16.
Uploaded libfiu (0.94-4), adding a patch from Logan Rose to fix a FTBFS with ld --as-needed.

My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#44, #45, #46, #47).

LTS

This month I have been paid to work 7 hours on Debian Long Term Support (LTS). Whilst the LTS team will take over support from the Security Team on April 26, 2016, in the meantime I did the following:

Archived the squeeze distribution (via the FTPteam).
Assisted in preparing updates for python-django.
Helping end-users migrate to wheezy now that squeeze LTS has reached end-of-life.

Bugs filed

debmirror: "Won't mirror without dists/experimental/main/binary-amd64/Packages.gz" due to .gz migration in experimental

I also filed 71 FTBFS bugs against ada-reference-manual, arachne-pnr, cdrkit, civicrm, cura-engine, dopewars, eclipse-cdt, fail2ban, fcitx-table-other, fleet, flint, flycheck, git-build-recipe, gitmagic, gnome-audio, golang-ginkgo, golang-github-armon-gomdb, golang-github-aws-aws-sdk-go, golang-github-gorilla-handlers, golang-github-hashicorp-raft-mdb, golang-github-jinzhu-gorm, golang-github-miekg-pkcs11, golang-goyaml, gstreamer0.10, gstreamer1.0, haskell-http2, hidrd, hplip, httpfs2, jedit, kodi-pvr-dvbviewer, kodi-pvr-mythtv, kodi-pvr-vuplus, kodi-pvr-wmc, libraw1394, libtest-file-contents-perl, metview, mpi4py, ndoutils, opensp, phpunit, python-glanceclient, python-oslo.messaging, python-phabricator, rrdtool, ruby-actionpack-xml-parser, ruby-acts-as-list, ruby-acts-as-list, ruby-acts-as-list, ruby-acts-as-list, ruby-amqp, ruby-buff-config, ruby-buff-extensions, ruby-clockwork, ruby-compass, ruby-em-synchrony, ruby-numru-misc, ruby-rabl, ruby-rails-deprecated-sanitizer, ruby-riddle, ruby-safe-yaml, ruby-solve, ruby-toml, ruby-twitter, ruby-typhoeus, ruby-varia-model, shared-mime-info, siproxd, stylish, sup-mail & uima-as.

FTP Team

As a Debian FTP assistant I ACCEPTed 143 packages: acme-tiny, berkshelf-api, circlator, cloud-utils, corsix-th, cronic, diaspora-installer, dub, dumb-init, firehol, firetools, flask-bcrypt, flask-oldsessions, flycheck, ganeti, geany-plugins, git-build-recipe, git-phab, gnome-shell-extension-caffeine, gnome-shell-extension-mediaplayer, golang-github-cheggaaa-pb, golang-github-coreos-ioprogress, golang-github-cyberdelia-go-metrics-graphite, golang-github-cznic-ql, golang-github-elazarl-goproxy, golang-github-hashicorp-hil, golang-github-mitchellh-go-wordwrap, golang-github-mvdan-xurls, golang-github-paulrosania-go-charset, golang-github-xeipuuv-gojsonreference, golang-github-xeipuuv-gojsonschema, grilo-plugins, gtk3-nocsd, herisvm, identity4c, lemonldap-ng, libisal, libmath-gsl-perl, libmemcached-libmemcached-perl, libplack-middleware-logany-perl, libplack-middleware-logwarn-perl, libpng1.6, libqmi, librdf-generator-http-perl, libtime-moment-perl, libvirt-php, libxml-compile-soap-perl, libxml-compile-wsdl11-perl, linux, linux-tools, mdk-doc, mesa, mpdecimal, msi-keyboard, nauty, node-addressparser, node-ansi-regex, node-argparse, node-array-find-index, node-base62, node-co, node-component-consoler, node-crypto-cacerts, node-decamelize, node-delve, node-for-in, node-function-bind, node-generator-supported, node-invert-kv, node-json-localizer, node-normalize-git-url, node-nth-check, node-obj-util, node-read-file, node-require-dir, node-require-main-filename, node-seq, node-starttls, node-through, node-uid-number, node-uri-path, node-url-join, node-xmlhttprequest-ssl, ocrmypdf, octave-netcdf, open-infrastructure-container-tools, osmose-emulator, pdal, pep8, pg-backup-ctl, php-guzzle, printrun, pydocstyle, pysynphot, python-antlr3, python-biom-format, python-brainstorm, python-django-adminsortable, python-feather-format, python-gevent, python-lxc, python-mongoengine, python-nameparser, python-pdal, python-pefile, python-phabricator, python-pika-pool, python-pynlpl, python-qtawesome, python-requests-unixsocket, python-saharaclient, python-stringtemplate3, r-cran-adegraphics, r-cran-assertthat, r-cran-bold, r-cran-curl, r-cran-data.table, r-cran-htmltools, r-cran-httr, r-cran-lazyeval, r-cran-mcmc, r-cran-openssl, r-cran-pbdzmq, r-cran-rncl, r-cran-uuid, rawtran, reel, ruby-certificate-authority, ruby-rspec-pending-for, ruby-ruby-engine, ruby-ruby-version, scribus-ng, specutils, symfony, tandem-mass, tdb, thrift, udfclient, vala, why3, wmaker, xdg-app & xiccd.

Today was my last day at Suomen Tilaajavastuu, where I worked on Qvarn. Tomorrow is my first day at my new job. The new job is for a new company, tentatively named QvarnLabs (registration is in process), to further develop and support Qvarn. The new company starts operation tomorrow, so you'll have to excuse me that there isn't a website yet. Qvarn provides a secure, RESTful JSON HTTP API for storing and retrieving data, with detailed access control (and I can provide more buzzwords if necessary). If you operate in the EU, and store information about people, you might want to read up about the General Data Protection Regulation, and Qvarn may be a possible part of a solution you want to look into, once we have the website up.

I attended GAME-ON last weedkend at Miraikan, The National Museum of Emerging Science and Innovation in Tokyo. There are many historical videogame hardware and software, and the bland-new PSVR exibition.

I can see the real DEC PDP-1 hardware. Unfortunately it is only haredware, not worked. And also PONG is exibited.

SPACEWAR description

There are many old hardware Atari 2600, Commodore 64, Spectrum, and so on.

Almost exhibition with English description, and officials can talk English, if you feel interested, go Miraikan until 30th May.

About one month has passed and here is the usual updated of TeX Live packages for Debian, this time also with an update to biber to accompany the updated version of biblatex. This will be (probably) the last upload before TeX Live 2015 gets frozen in preparation for 2016. After the freeze there will be some time of piece, and updates will got to experimental with the new binaries. Debian - TeX Live 2015

From the list of updated and new packages I want to mention that biblatex has been updated together with biber, and most biblatex styles should already by updated to work with the new biblatex, too. From the new packages one has in particular caught my attention: prooftrees. As I am writing a lot of proofs in my research, I am happy to see new package dealing with this, especially because this package is built on top of the excellent package forest (which in turn uses tikz), allowing for very compact proofs. Updated packages aastex, abntex2, academicons, acro, amsmath, animate, archaeologie, asciilist, babel, babel-french, babel-friulan, babel-russian, babel-spanish, beamertheme-metropolis, bibexport, biblatex, biblatex-apa, biblatex-bookinarticle, biblatex-caspervector, biblatex-chem, biblatex-fiwi, biblatex-gost, biblatex-manuscripts-philology, biblatex-nature, biblatex-philosophy, biblatex-phys, biblatex-publist, biblatex-realauthor, biblatex-science, bxjscls, cabin, caption, cbfonts-fd, celtic, chemmacros, computational-complexity, crimson, csplain, datetime2-english, diagbox, disser, droit-fr, dtk, dvips, ejpecp, emisa, factura, fibeamer, fithesis, forest, gost, hobby, hyperxmp, inconsolata, lualatex-math, mandi, mcf2graph, media9, nameauth, ocgx2, parades, pdftex, pkuthss, poetrytex, ptex, reledmac, roundrect, showhyphens, siunitx, spath3, splitindex, suftesi, tcolorbox, tetex, tex4ht, texinfo, texlive-scripts, thuthesis, titlesec, turabian-formatting, unicode-data, uptex, velthuis, venndiagram, xassoccnt, xint, xsavebox. New packages beamercolorthemeowl, bibletext, cochineal, formation-latex-ul, gobble, keyvaltable, lroundrect, luatex85, mathpartir, miama, multidef, parades, pgfornament, prooftrees, visualpstricks, visualtikz, xsavebox, ycbook. Enjoy.

Back in 2013 I proposed a way to make paper and PDF invoices easier to process electronically by adding a QR code with the key information about the invoice. I suggested using vCard field definition, to get some standard format for name and address, but any format would work. I did not do anything about the proposal, but hoped someone one day would make something like it. It would make it possible to efficiently send machine readable invoices directly between seller and buyer. This was the background when I came across a proposal and specification from the web based accounting and invoicing supplier Visma in Sweden called UsingQR. Their PDF invoices contain a QR code with the key information of the invoice in JSON format. This is the typical content of a QR code following the UsingQR specification (based on a real world example, some numbers replaced to get a more bogus entry). I've reformatted the JSON to make it easier to read. Normally this is all on one long line:

 
 "vh":500.00,
 "vm":0,
 "vl":0,
 "uqr":1,
 "tp":1,
 "nme":"Din Leverand r",
 "cc":"NO",
 "cid":"997912345 MVA",
 "iref":"12300001",
 "idt":"20151022",
 "ddt":"20151105",
 "due":2500.0000,
 "cur":"NOK",
 "pt":"BBAN",
 "acc":"17202612345",
 "bc":"BIENNOK1",
 "adr":"0313 OSLO"

The interpretation of the fields can be found in the format specification (revision 2 from june 2014). The format seem to have most of the information needed to handle accounting and payment of invoices, at least the fields I have needed so far here in Norway. Unfortunately, the site and document do not mention anything about the patent, trademark and copyright status of the format and the specification. Because of this, I asked the people behind it back in November to clarify. Ann-Christine Savlid (ann-christine.savlid (at) visma.com) replied that Visma had not applied for patent or trademark protection for this format, and that there were no copyright based usage limitations for the format. I urged her to make sure this was explicitly written on the web pages and in the specification, but unfortunately this has not happened yet. So I guess if there is submarine patents, hidden trademarks or a will to sue for copyright infringements, those starting to use the UsingQR format might be at risk, but if this happen there is some legal defense in the fact that the people behind the format claimed it was safe to do so. At least with patents, there is always a chance of getting sued... I also asked if they planned to maintain the format in an independent standard organization to give others more confidence that they would participate in the standardization process on equal terms with Visma, but they had no immediate plans for this. Their plan was to work with banks to try to get more users of the format, and evaluate the way forward if the format proved to be popular. I hope they conclude that using an open standard organisation like IETF is the correct place to maintain such specification. Update 2016-03-20: Via Twitter I became aware of some comments about this blog post that had several useful links and references to similar systems. In the Czech republic, the Czech Banking Association standard #26, with short name SPAYD, uses QR codes with payment information. More information is available from the Wikipedia page on Short Payment Descriptor. And in Germany, there is a system named BezahlCode, (specification v1.8 2013-12-05 available as PDF), which uses QR codes with URL-like formatting using "bank:" as the URI schema/protocol to provide the payment information. There is also the ZUGFeRD file format that perhaps could be transfered using QR codes, but I am not sure if it is done already. Last, in Bolivia there are reports that tax information since november 2014 need to be printed in QR format on invoices. I have not been able to track down a specification for this format, because of my limited language skill sets.

Last week was quite equally split between refactoring and bug fixing. As we're getting closer to 4.6.0 release more people give feedback on it and some corner case issues are being discovered. However the biggest challenge was strange bug that was reported long time ago - it affected only Italian translation on Windows server. After lot of debugging, I've realized that trying to encode Cardinalit </td> to json (using json_encode) causes this problem. Obviously there have to be more conditions met as this string only doesn't cause it, however removing it or placing x between and < fixes the problem. In the end I've workarounded it in our code and hopefully I will get to create proper bugreport for php so that it can be fixed upstream as well. The refactoring was mostly focused on encoding conversion part which is now all embedded in the Encoding class. It was quite simplified and should also speed up pages a bit as it does the initialization only when needed not on every phpMyAdmin's page. Handled issues:

Filed under: English phpMyAdmin 0 comments

What happened in the reproducible builds effort between March 6th and March 12th:

Packages fixed The following packages have become reproducible due to changes in their build dependencies: dfc, gap-openmath, gnubik, gplanarity, iirish, iitalian, monajat, openimageio, plexus-digest, ruby-fssm, vdr-plugin-dvd, vdr-plugin-spider. The following packages became reproducible after getting fixed:

adduser/3.114 by Niels Thykier.

bsdmainutils/9.0.7 by Michael Meskes.

criu/2.0-1 by Salvatore Bonaccorso.

genometools/1.5.8+ds-2 by Sascha Steinbiss.

gfs2-utils/3.1.8-1 uploaded by Bastian Blank, fix by Christoph Berg.

gmerlin/1.2.0~dfsg+1-5 by IOhannes m zm lnig.

heroes/0.21-14 by Stephen Kitt.

kmc/2.3+dfsg-3 by Sascha Steinbiss.

polyml/5.6-3 by James Clarke.

sed/4.2.2-7.1 by Niels Thykier.

snpomatic/1.0-3 by Sascha Steinbiss.

tantan/13-4 by Sascha Steinbiss.

Some uploads fixed some reproducibility issues, but not all of them:

apg/2.2.3.dfsg.1-3 uploaded by Marc Haber, original patch by Chris Lamb.

elastix/4.8-4 uploaded by Gert Wollny, reported by akira.

Patches submitted which have not made their way to the archive yet:

#817979 on modernizr by Sascha Steinbiss: sort list of files included in `feature-detects.js`.

#818027 on snapper by Sascha Steinbiss: always use `/bin/sh` as shell.

tests.reproducible-builds.org Always use all cores on `armhf` builders. (h01ger) Improve the look of Debian dashboard. (h01ger)

Package reviews 118 reviews have been removed, 114 added and 15 updated in the previous week. 15 FTBFS have been filled by Chris Lamb. New issues: xmlto_txt_output_locale_specific.

Misc. Lunar seeks new maintainers for diffoscope, several mailing lists, and these very weekly reports.

My recent rack design turned out to simply not be practical. It did not hold all the SBC I needed it to and most troubling accessing connectors was impractical. I was forced to remove the enclosure from the rack and go back to piles of SBC on a shelf.

View of the acrylic being laser cut through the heavily tinted window

This sent me back to the beginning of the design process. The requirement for easy access to connectors had been compromised on in my first solution because I wanted a compact 1U size. This time I returned to my initial toast rack layout but retaining the SBC inside their clip cases.

By facing the connectors downwards and providing basic cable management the design should be much more practical.

My design process is to use the QCAD package to create layered 2D outlines which are then converted from DXF into toolpaths with Lasercut CAM software. The toolpaths are then uploaded to the laser cutter directly from the PC running Lasercut.

Despite the laser cutters being professional grade systems the Lasercut software is a continuous cause of issues for many users, it is the only closed source piece of software in the production process and it has a pretty poor user interface. On this occasion my main issue with it was my design was quite large at 700mm by 400mm which caused the software to crash repeatedly. I broke the design down into two halves and this allowed me to continue.

Once I defeated the software the design was laser cut from 3mm clear extruded acrylic. The assembled is secured with 72 off M3 nuts and bolts. The resulting construction is very strong and probably contains much more material than necessary.

One interesting thing I discovered is that in going from a 1U enclosure holding 5 units to a 2U design holding 11 units I had increased the final weight from 320g to 980g and when all 11 SBC are installed that goes up to a whopping 2300g. Fortunately this is within the mechanical capabilities of the material but it is the heaviest thing I have ever constructed from 3mm acrylic.

Once installed in the rack with all SBC inserted and connected this finally actually works and provides a practical solution. The self is finally clear of SBC and has enough space for all the other systems I need to accommodate for various projects.

As usual the design files are all freely available though I really cannot see anyone else needing to replicate this.

Search Results: "js"

26 April 2016

Dirk Eddelbuettel: RcppMgsPack 0.1.0

20 April 2016

Reproducible builds folks: Reproducible builds: week 51 in Stretch cycle

Norbert Preining: GnuPG notes: subkeys, yubikey, gpg1 vs gpg2

19 April 2016

Rémi Vanicat: LudumDare35

18 April 2016

Reproducible builds folks: Reproducible builds: week 50 in Stretch cycle

Norbert Preining: TeX Live 2016 pretest and Debian packages

16 April 2016

Matthias Klumpp: Introducing AppStream-Generator

12 April 2016

Reproducible builds folks: Reproducible builds: week 49 in Stretch cycle

5 April 2016

Ian Wienand: Image building in OpenStack CI

4 April 2016

Raphaël Hertzog: My Free Software Activities in February and March 2016

Ian Wienand: Image building for OpenStack CI -- Minimal images, big effort

2 April 2016

Thorsten Alteholz: My Debian Activities in March 2016

31 March 2016

Chris Lamb: Free software activities in March 2016

Lars Wirzenius: New job: QvarnLabs

30 March 2016

NOKUBI Takatsugu: GAME-ON the historical videogame exhibition

20 March 2016

Norbert Preining: Debian/TeX Live 2015.20160320-1 and biber 2.4-1

19 March 2016

Petter Reinholdtsen: UsingQR - "Electronic" paper invoices using JSON and QR codes

15 March 2016

Michal Čihař: Weekly phpMyAdmin contributions 2016-W10

14 March 2016

Lunar: Reproducible builds: week 46 in Stretch cycle

tests.reproducible-builds.org Always use all cores on `armhf` builders. (h01ger) Improve the look of Debian dashboard. (h01ger)

Package reviews 118 reviews have been removed, 114 added and 15 updated in the previous week. 15 FTBFS have been filled by Chris Lamb. New issues: xmlto_txt_output_locale_specific.

Misc. Lunar seeks new maintainers for diffoscope, several mailing lists, and these very weekly reports.

13 March 2016

Vincent Sanders: I changed my mind, Erase and rewind

Search Results: "js"

26 April 2016

20 April 2016

19 April 2016

18 April 2016

16 April 2016

12 April 2016

5 April 2016

4 April 2016

2 April 2016

31 March 2016

30 March 2016

20 March 2016

19 March 2016

15 March 2016

14 March 2016

tests.reproducible-builds.org Always use all cores on armhf builders. (h01ger) Improve the look of Debian dashboard. (h01ger)

Package reviews 118 reviews have been removed, 114 added and 15 updated in the previous week. 15 FTBFS have been filled by Chris Lamb. New issues: xmlto_txt_output_locale_specific.

Misc. Lunar seeks new maintainers for diffoscope, several mailing lists, and these very weekly reports.

13 March 2016

tests.reproducible-builds.org Always use all cores on `armhf` builders. (h01ger) Improve the look of Debian dashboard. (h01ger)