If you ve perused the ActivityPub feed of certificates whose keys are known to be compromised, and clicked on the Show More button to see the name of the certificate issuer, you may have noticed that some issuers seem to come up again and again.
This might make sense after all, if a CA is issuing a large volume of certificates, they ll be seen more often in a list of compromised certificates.
In an attempt to see if there is anything that we can learn from this data, though, I did a bit of digging, and came up with some illuminating results.
The Procedure
I started off by finding all the unexpired certificates logged in Certificate Transparency (CT) logs that have a key that is in the pwnedkeys database as having been publicly disclosed.
From this list of certificates, I removed duplicates by matching up issuer/serial number tuples, and then reduced the set by counting the number of unique certificates by their issuer.
This gave me a list of the issuers of these certificates, which looks a bit like this:
/C=BE/O=GlobalSign nv-sa/CN=AlphaSSL CA - SHA256 - G4
/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Organization Validation Secure Server CA
/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./OU=http://certs.starfieldtech.com/repository//CN=Starfield Secure Certificate Authority - G2
/C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA
/C=BE/O=GlobalSign nv-sa/CN=GlobalSign GCC R3 DV TLS CA 2020
Rather than try to work with raw issuers (because, as Andrew Ayer says, The SSL Certificate Issuer Field is a Lie), I mapped these issuers to the organisations that manage them, and summed the counts for those grouped issuers together.
The Data
Insert obligatory "not THAT data" comment here
The end result of this work is the following table, sorted by the count of certificates which have been compromised by exposing their private key:
Issuer
Compromised Count
Sectigo
170
ISRG (Let's Encrypt)
161
GoDaddy
141
DigiCert
81
GlobalSign
46
Entrust
3
SSL.com
1
If you re familiar with the CA ecosystem, you ll probably recognise that the organisations with large numbers of compromised certificates are also those who issue a lot of certificates.
So far, nothing particularly surprising, then.
Let s look more closely at the relationships, though, to see if we can get more useful insights.
Volume Control
Using the issuance volume report from crt.sh, we can compare issuance volumes to compromise counts, to come up with a compromise rate .
I m using the Unexpired Precertificates colume from the issuance volume report, as I feel that s the number that best matches the certificate population I m examining to find compromised certificates.
To maintain parity with the previous table, this one is still sorted by the count of certificates that have been compromised.
Issuer
Issuance Volume
Compromised Count
Compromise Rate
Sectigo
88,323,068
170
1 in 519,547
ISRG (Let's Encrypt)
315,476,402
161
1 in 1,959,480
GoDaddy
56,121,429
141
1 in 398,024
DigiCert
144,713,475
81
1 in 1,786,586
GlobalSign
1,438,485
46
1 in 31,271
Entrust
23,166
3
1 in 7,722
SSL.com
171,816
1
1 in 171,816
If we now sort this table by compromise rate, we can see which organisations have the most (and least) leakiness going on from their customers:
Issuer
Issuance Volume
Compromised Count
Compromise Rate
Entrust
23,166
3
1 in 7,722
GlobalSign
1,438,485
46
1 in 31,271
SSL.com
171,816
1
1 in 171,816
GoDaddy
56,121,429
141
1 in 398,024
Sectigo
88,323,068
170
1 in 519,547
DigiCert
144,713,475
81
1 in 1,786,586
ISRG (Let's Encrypt)
315,476,402
161
1 in 1,959,480
By grouping by order-of-magnitude in the compromise rate, we can identify three bands :
The Super Leakers: Customers of Entrust and GlobalSign seem to love to lose control of their private keys.
For Entrust, at least, though, the small volumes involved make the numbers somewhat untrustworthy.
The three compromised certificates could very well belong to just one customer, for instance.
I m not aware of anything that GlobalSign does that would make them such an outlier, either, so I m inclined to think they just got unlucky with one or two customers, but as CAs don t include customer IDs in the certificates they issue, it s not possible to say whether that s the actual cause or not.
The Regular Leakers: Customers of SSL.com, GoDaddy, and Sectigo all have compromise rates in the 1-in-hundreds-of-thousands range.
Again, the low volumes of SSL.com make the numbers somewhat unreliable, but the other two organisations in this group have large enough numbers that we can rely on that data fairly well, I think.
The Low Leakers: Customers of DigiCert and Let s Encrypt are at least three times less likely than customers of the regular leakers to lose control of their private keys.
Good for them!
Now we have some useful insights we can think about.
Why Is It So?
If you don't know who Professor Julius Sumner Miller is, I highly recommend finding out
All of the organisations on the list, with the exception of Let s Encrypt, are what one might term traditional CAs.
To a first approximation, it s reasonable to assume that the vast majority of the customers of these traditional CAs probably manage their certificates the same way they have for the past two decades or more.
That is, they generate a key and CSR, upload the CSR to the CA to get a certificate, then copy the cert and key somewhere.
Since humans are handling the keys, there s a higher risk of the humans using either risky practices, or making a mistake, and exposing the private key to the world.
Let s Encrypt, on the other hand, issues all of its certificates using the ACME (Automatic Certificate Management Environment) protocol, and all of the Let s Encrypt documentation encourages the use of software tools to generate keys, issue certificates, and install them for use.
Given that Let s Encrypt has 161 compromised certificates currently in the wild, it s clear that the automation in use is far from perfect, but the significantly lower compromise rate suggests to me that lifecycle automation at least reduces the rate of key compromise, even though it doesn t eliminate it completely.
Sidebar: ACME Does Not Currently Rule The World
It is true that all of the organisations in this analysis also provide ACME issuance workflows, should customers desire it.
However, the traditional CA companies have been around a lot longer than ACME has, and so they probably acquired many of their customers before ACME existed.
Given that it s incredibly hard to get humans to change the way they do things, once they have a way that works , it seems reasonable to assume that most of the certificates issued by these CAs are handled in the same human-centric, error-prone manner they always have been.
If organisations would like to refute this assumption, though, by sharing their data on ACME vs legacy issuance rates, I m sure we d all be extremely interested.
Explaining the Outlier
The difference in presumed issuance practices would seem to explain the significant difference in compromise rates between Let s Encrypt and the other organisations, if it weren t for one outlier.
This is a largely traditional CA, with the manual-handling issues that implies, but with a compromise rate close to that of Let s Encrypt.
We are, of course, talking about DigiCert.
The thing about DigiCert, that doesn t show up in the raw numbers from crt.sh, is that DigiCert manages the issuance of certificates for several of the biggest hosted TLS providers, such as CloudFlare and AWS.
When these services obtain a certificate from DigiCert on their customer s behalf, the private key is kept locked away, and no human can (we hope) get access to the private key.
This is supported by the fact that no certificates identifiably issued to either CloudFlare or AWS appear in the set of certificates with compromised keys.
When we ask for all certificates issued by DigiCert , we get both the certificates issued to these big providers, which are very good at keeping their keys under control, as well as the certificates issued to everyone else, whose key handling practices may not be quite so stringent.
It s possible, though not trivial, to account for certificates issued to these hosted TLS providers, because the certificates they use are issued from intermediates branded to those companies.
With the crt.sh psql interface we can run this query to get the total number of unexpired precertificates issued to these managed services:
SELECT SUM(sub.NUM_ISSUED[2] - sub.NUM_EXPIRED[2])
FROM (
SELECT ca.name, max(coalesce(coalesce(nullif(trim(cc.SUBORDINATE_CA_OWNER), ''), nullif(trim(cc.CA_OWNER), '')), cc.INCLUDED_CERTIFICATE_OWNER)) as OWNER,
ca.NUM_ISSUED, ca.NUM_EXPIRED
FROM ccadb_certificate cc, ca_certificate cac, ca
WHERE cc.CERTIFICATE_ID = cac.CERTIFICATE_ID
AND cac.CA_ID = ca.ID
GROUP BY ca.ID
) sub
WHERE sub.name ILIKE '%Amazon%' OR sub.name ILIKE '%CloudFlare%' AND sub.owner = 'DigiCert';
The number I get from running that query is 104,316,112, which should be subtracted from DigiCert s total issuance figures to get a more accurate view of what DigiCert s regular customers do with their private keys.
When I do this, the compromise rates table, sorted by the compromise rate, looks like this:
Issuer
Issuance Volume
Compromised Count
Compromise Rate
Entrust
23,166
3
1 in 7,722
GlobalSign
1,438,485
46
1 in 31,271
SSL.com
171,816
1
1 in 171,816
GoDaddy
56,121,429
141
1 in 398,024
"Regular" DigiCert
40,397,363
81
1 in 498,732
Sectigo
88,323,068
170
1 in 519,547
All DigiCert
144,713,475
81
1 in 1,786,586
ISRG (Let's Encrypt)
315,476,402
161
1 in 1,959,480
In short, it appears that DigiCert s regular customers are just as likely as GoDaddy or Sectigo customers to expose their private keys.
What Does It All Mean?
The takeaway from all this is fairly straightforward, and not overly surprising, I believe.
The less humans have to do with certificate issuance, the less likely they are to compromise that certificate by exposing the private key.
While it may not be surprising, it is nice to have some empirical evidence to back up the common wisdom.
Fully-managed TLS providers, such as CloudFlare, AWS Certificate Manager, and whatever Azure s thing is called, is the platonic ideal of this principle: never give humans any opportunity to expose a private key.
I m not saying you should use one of these providers, but the security approach they have adopted appears to be the optimal one, and should be emulated universally.
The ACME protocol is the next best, in that there are a variety of standardised tools widely available that allow humans to take themselves out of the loop, but it s still possible for humans to handle (and mistakenly expose) key material if they try hard enough.
Legacy issuance methods, which either cannot be automated, or require custom, per-provider automation to be developed, appear to be at least four times less helpful to the goal of avoiding compromise of the private key associated with a certificate.
Humans Are, Of Course, The Problem
No thanks, Bender, I'm busy tonight
This observation that if you don t let humans near keys, they don t get leaked is further supported by considering the biggest issuers by volume who have not issued any certificates whose keys have been compromised: Google Trust Services (fourth largest issuer overall, with 57,084,529 unexpired precertificates), and Microsoft Corporation (sixth largest issuer overall, with 22,852,468 unexpired precertificates).
It appears that somewhere between most and basically all of the certificates these organisations issue are to customers of their public clouds, and my understanding is that the keys for these certificates are managed in same manner as CloudFlare and AWS the keys are locked away where humans can t get to them.
It should, of course, go without saying that if a human can never have access to a private key, it makes it rather difficult for a human to expose it.
More broadly, if you are building something that handles sensitive or secret data, the more you can do to keep humans out of the loop, the better everything will be.
Your Support is Appreciated
If you d like to see more analysis of how key compromise happens, and the lessons we can learn from examining billions of certificates, please show your support by buying me a refreshing beverage.
Trawling CT logs is thirsty work.
Appendix: Methodology Limitations
In the interests of clarity, I feel it s important to describe ways in which my research might be flawed.
Here are the things I know of that may have impacted the accuracy, that I couldn t feasibly account for.
Time Periods: Because time never stops, there is likely to be some slight mismatches in the numbers obtained from the various data sources, because they weren t collected at exactly the same moment.
Issuer-to-Organisation Mapping: It s possible that the way I mapped issuers to organisations doesn t match exactly with how crt.sh does it, meaning that counts might be skewed.
I tried to minimise that by using the same data sources (the CCADB AllCertificates report) that I believe that crt.sh uses for its mapping, but I cannot be certain of a perfect match.
Unwarranted Grouping: I ve drawn some conclusions about the practices of the various organisations based on their general approach to certificate issuance.
If a particular subordinate CA that I ve grouped into the parent organisation is managed in some unusual way, that might cause my conclusions to be erroneous.
I was able to fairly easily separate out CloudFlare, AWS, and Azure, but there are almost certainly others that I didn t spot, because hoo boy there are a lot of intermediate CAs out there.
What happened in the Reproducible
Builds effort between Sunday
September 11 and Saturday September 17 2016:
Toolchain developments
Ximin Luo started a new series of tools called (for now)
debrepatch, to
make it easier to automate checks that our old patches to Debian packages still
apply to newer versions of those packages, and still make these reproducible.
Ximin Luo updated one of our few remaining patches for dpkg in #787980
to make it cleaner and more minimal.
The following tools were fixed to produce reproducible output:
The following updated packages appear to be reproducible now, for reasons we
were not able to figure out. (Relevant changelogs did not mention reproducible
builds.)
The following 3 packages were not changed, but have become reproducible due to
changes in their build-dependencies: jaxrs-apipython-luazope-mysqlda.
Some uploads have addressed some reproducibility issues, but not all of them:
Reviews of unreproducible packages
462 package reviews have been added, 524 have been updated and 166 have been removed in this week,
adding to our knowledge about identified issues.
25 issue types have been updated:
Added a new annotation for issues called "fix-deterministic" to help us
update package reviews more easily. This indicates whether we expect that an
issue would always happen on Jenkins; i.e. if there is a successful build,
then we know the issue is fixed for that package and can update our notes.
diffoscope development
A new version of diffoscope 60 was
uploaded to unstable by Mattia
Rizzolo. It included
contributions
from:
Mattia Rizzolo:
Various packaging and testing improvements.
HW42:
minor wording fixes
Reiner Herrmann:
minor wording fixes
It also included from changes previous weeks; see either the changes or commits
linked above, or previous blog posts 727170.
strip-nondeterminism development
New versions of strip-nondeterminism 0.027-1 and 0.028-1 were uploaded to
unstable by Chris Lamb. It included
contributions
from:
Chris Lamb:
Testing improvements, including better handling of timezones.
disorderfs development
A new version of disorderfs 0.5.1 was
uploaded to unstable by Chris
Lamb. It included
contributions
from:
Andrew Ayer and Chris Lamb:
Support relative paths for ROOTDIR; it no longer needs to be an absolute path.
Chris Lamb:
Print the behaviour (shuffle/reverse/sort) on startup to stdout.
It also included from changes previous weeks; see either the changes or commits
linked above, or previous blog posts 70.
Misc.
This week's edition was written by Ximin Luo and reviewed by a bunch of
Reproducible Builds folks on IRC.
What happened in the Reproducible
Builds effort between June 5th and June 11th 2016:
Media coverage
Ed Maste gave a talk at BSDCan 2016 on
reproducible builds
(slides,
video).
GSoC and Outreachy updates
Weekly reports by our participants:
Scarlett Clark
worked on making some packages reproducible, focusing on KDE backend and
utility programs.
Ceridwen
published an initial design for the interface for reprotest, including a
discussion on different types of build variations and the difficulties of
specifying certain types of variations.
Valerie Young improved documentation for
building our tests website, began migrating Debian-specific pages into a new
namespace, and planned future work around its navigation.
Documentation update
- Ximin Luo proposed a modification
to our SOURCE_DATE_EPOCH spec explaining FORCE_SOURCE_DATE.
Some upstream build tools (e.g. TeX, see below) have expressed a desire to
control which cases of embedded timestamps should obey SOURCE_DATE_EPOCH.
They were not convinced by our arguments on why this is a bad idea, so we
agreed on an environment variable FORCE_SOURCE_DATE for them to implement
their desired behaviour - named generically, so that at least we can set it
centrally. For more details, see the text just linked. However, we strongly
urge most build tools not to use this, and instead obey SOURCE_DATE_EPOCHunconditionally in all cases.
Toolchain fixes
TeX Live 2016 released
with SOURCE_DATE_EPOCH support for all engines except LuaTeX and original TeX.
Continued discussion
(alternative archive)
with TeX upstream, about SOURCE_DATE_EPOCH corner cases, eventually
resulting in the FORCE_SOURCE_DATE proposal from above.
gcc-5/5.4.0-4 by Matthias Klose now avoids storing
-fdebug-prefix-map in DW_AT_producer, thanks to original patch by
Daniel Kahn Gillmor.
sphinx/1.4.3-1 by Dmitry Shachnev now drops Debian-specific patches
relating to SOURCE_DATE_EPOCH applied upstream, original patch by Alexis
Bienven e.
asciidoctor/1.5.4-2 by C dric Boutillier now supports
SOURCE_DATE_EPOCH, thanks to original patch by Alexis Bienven e.
dh-python/1.5.4-2 by Piotr O arowski now behaves better in some
cases, thanks to original patch by Chris Lamb.
Andrew Ayer uploaded strip-nondeterminism/0.018-2 fixing #826700, a packaging improvement for Multi-Arch to ease cross-build
situations.
2 days later Andrew released strip-nondeterminism/0.019; now
strip-nondeterminism is able to:
recursively normalize JAR files embedded within JAR files (#823917)
clamp the timestamp, the same way tar >=1.28-2.2 can (for now available only for gzip archives)
disorderfs development
Andrew Ayer released disorderfs/0.4.3, fixing a issue with umask handling (#826891)
tests.reproducible-builds.org
Valerie Young namespaced the Debian-specific pages to /debian/
namespace, with redirects to
for the previous URLs.
Holger Levsen improved the reliability of build jobs: the availability of
both build nodes (for a given build) is now being tested when a build job is
started, to better cope when one of the 25 build nodes go down for some reason.
Ximin Luo improved the index of identified
issues to
include the total popcon scores of each issue, which is now also used for
sorting that page.
Misc.
Steven Chamberlain submitted a patch to FreeBSD's
makefs
to allow reproducible builds of the kfreebsd installer.
Ed Maste committed a patch to FreeBSD's
binutils to
enable determinstic archives by default in GNU ar.
Helmut Grohne
experimentedwith
cross+native reproductions of dash with some success, using
rebootstrap.
This week's edition was written by Ximin Luo, Chris Lamb, Holger Levsen, Mattia
Rizzolo and reviewed by a bunch of Reproducible builds folks on IRC.
What happened in the reproducible
builds effort between December 20th to December 26th:
Toolchain fixes
Mattia Rizzolo rebased our experimental versions of debhelper (twice!) and dpkg on top of the latest releases.
Reiner Herrmann submited a patch for mozilla-devscripts to sort the file list in generated preferences.js files.
To be able to lift the restriction that packages must be built in the same path, translation support for the __FILE__ C pre-processor macro would also be required. Joerg Sonnenberger submitted a patch back in 2010 that would still be useful today.
Chris Lamb started work on providing a deterministic mode for debootstrap.
Packages fixed
The following packages have become reproducible due to changes in their
build dependencies:
bouncycastle,
cairo-dock-plug-ins,
darktable,
gshare,
libgpod,
pafy,
ruby-redis-namespace,
ruby-rouge,
sparkleshare.
The following packages became reproducible after getting fixed:
a7xpg/0.11.dfsg1-9 uploaded by Markus Koschany, original patch by Reiner Herrmann.
at/3.1.18-1 uploaded by Laurent Bigonville, original patch by Reiner Herrmann, merged by Jose M Calhariz.
yacas/1.3.6-1) uploaded by Muammar El Khatib, original patch by Reiner Herrmann.
Patches submitted which have not made their way to the archive yet:
#808459 on pywavelets by Chris Lamb: add support for SOURCE_DATE_EPOCH in the documentation generator.
#808652 on nexuiz-data by Reiner Herrmann: sorts with the locale set to C.
#808667 on libmouse-perl by Reiner Herrmann: sorts the list of filenames to be embedded.
#808679 on libcorelinux by Reiner Herrmann: sort the list of files in the generated Makefile.
#808711 on ca-certificates by Reiner Herrmann: sort the list of certificates before it is embedded.
reproducible.debian.net
Statistics for package sets are now visible for the armhf architecture. (h01ger)
The second build now has a longer timeout (18 hours) than the first build (12 hours). This should prevent wasting resources when a machine is loaded. (h01ger)
Builds of Arch Linux packages are now done using a tmpfs. (h01ger)
200 GiB have been added to jenkins.debian.net (thanks to ProfitBricks!) to make room for new jobs. The current count is at 962 and growing!
diffoscope development
Aside from some minor bugs that have been fixed, a one-line change made huge memory (and time) savings as the output of transformation tool is now streamed line by line instead of loaded entirely in memory at once.
disorderfs development
Andrew Ayer released disorderfs version 0.4.2-1 on December 22th. It fixes a memory corruption error when processing command line arguments that could cause command line options to be ignored.
Documentation update
Many small improvements for the documentation on reproducible-builds.org sent by Georg Koppen were merged.
Package reviews
666 (!) reviews have been removed, 189 added and 162 updated in the previous week.
151 new fail to build from source reports have been made by Chris West, Chris Lamb, Mattia Rizzolo, and Niko Tyni.
New issues identified: unsorted_filelist_in_xul_ext_preferences, nondeterminstic_output_generated_by_moarvm.
Misc.
Steven Chamberlain drew our attention to one analysis of the Juniper ScreenOS Authentication Backdoor: Whilst this may have been added in source code, it was well-disguised in the disassembly and just 7 instructions long. I thought this was a good example of the current state-of-the-art, and why we'd like our binaries and eventually, installer and VM images reproducible IMHO.
Joanna Rutkowska has mentioned possible ways for Qubes to become reproducible on their development mailing-list.
Niels Thykier uploaded debhelper/9.20151004 which exports SOURCE_DATE_EPOCH when running dh_auto_* (patch by Dhole). dh now also calls dh_strip_nondeterminism during build (patch by Andrew Ayer). The only remaining change regarding reproducibility in the custom debhelper is related to mtimes in data.tar which might be fixed directly in dpkg instead.
Niels Thykier made another upload of debhelper/9.20151005 the next day to sort Build IDs added by dh_strip in control files.
Scott Kitterman fixed an issue with non-deterministic Depends generated by dh-python identified by Santiago Vila and Chris Lamb.
Lunar updated the patch against dpkg which makes the order of files in control.tar.gz deterministic using the new --sort=name option available in GNU Tar 1.28.
josch released sbuild version 0.66.0-1 with several fixes and improvements. The most notable one for reproducible builds is the new --build-path option and $build_path configuration variable added by akira which allows to explicitly chose a given build path.
Reiner Herrmann wrote a new patch for dh-systemd to sort the list of unit files in the generated maintainer scripts.
Packages fixed
The following packages became reproducible due to changes in their
build dependencies:
aoeui,
apron,
camlmix,
cudf,
findlib,
glpk-java,
hawtjni,
haxe,
java-atk-wrapper,
llvm-py,
misery,
mtasc,
ocamldsort,
optcomp,
spamoracle.
The following packages became reproducible after getting fixed:
reproducible.debian.net
ProfitBricks once again increased their support for reproducible builds in Debian and in other free software projects by adding 58 new cores and 138 GiB of RAM to the already existing setup. Two new amd64 build nodes and 16 new amd64 build jobs have been added which doubles the build capacity per day and allows us to spot many kind of problems earlier. The size of the tmpfs where builds are performed has also been increased from 70 to 200 GiB on all amd64 build nodes. Huge thanks!
When examining a package, a link now points to a table listing all previous recorded tests for the same package. (Mattia) The menu on the package pages has also been improved. (h01ger)
Packages in the depwait state are now rescheduled automatically after five days. (h01ger)
Links to documentation and other projects being tested have been made more visible on the landing page. (h01ger)
To reduce noise on the team IRC channel five different types of notifications have been turned into mail notifications. The remaining ones have been shortened and the status changes have been limited to unstable and experimental. (h01ger)
Maintainer notifications about status changes in a package will only be sent out once per day, and not on each status change. (h01ger)
diffoscope development
Some more experiments of concurrent processing have been made. None were good and reliable enough to be shared, though.
Package reviews
48 reviews have been removed, 189 added and 23 updated this week.
9 FTBFS bugs were reported by Chris Lamb.
Misc.
h01ger met with Levente Polyak to discuss testing Arch Linux on Debian continuous test system with an easily extensible framework. The idea is to also allow testing of other distributions, and provide a nice package based view like the one for Debian.
Patches submitted which have not made their way to the archive yet:
#797027 on zyne by Chris Lamb: switch to pybuild to get rid of .pyc files.
#797180 on python-doit by Chris Lamb: sort output when creating completion script for bash and zsh.
#797211 on apt-dater by Chris Lamb: fix implementation of SOURCE_DATE_EPOCH.
#797215 on getdns by Chris Lamb: fix call to dpkg-parsechangelog in debian/rules.
#797254 on splint by Chris Lamb: support SOURCE_DATE_EPOCH for version string.
#797296 on shiro by Chris Lamb: remove username from build string.
#797408 on splitpatch by Reiner Herrmann: use SOURCE_DATE_EPOCH to set manpage date.
#797410 on eigenbase-farrago by Reiner Herrmann: sets the comment style to scm-safe which tells ResourceGen that no timestamps should be included.
#797415 on apparmor by Reiner Herrmann: sorting with the locale set to C. CAPABILITIES
#797419 on resiprocate by Reiner Herrmann: set the embedded hostname to a static value.
#797427 on jam by Reiner Herrmann: sorting with the locale set to C.
#797430 on ii-esu by Reiner Herrmann: sort source list using C locale.
#797431 on tatan by Reiner Herrmann: sort source list using C locale.
Chris Lamb also noticed that binaries shipped with libsilo-bindid not work.
Documentation update
Chris Lamb and Ximin Luo assembled a proper specification for SOURCE_DATE_EPOCH in the hope to convince more upstreams to adopt it. Thanks to Holger it is published under a non-Debian domain name.
Lunar documented easiest way to solve issues with file ordering and timestamps in tarballs that came with tar/1.28-1.
Some examples on how to use SOURCE_DATE_EPOCH have been improved to support systems without GNU date.
reproducible.debian.net
armhf is finally being tested, which also means the remote building of Debian packages finally works! This paves the way to perform the tests on even more architectures and doing variations on CPU and date. Some packages even produce the same binary Arch:all packages on different architectures (1, 2). (h01ger)
Tests for FreeBSD are finally running. (h01ger)
As it seems the gcc5 transition has cooled off, we schedule sid more often than testing again on amd64. (h01ger)
disorderfs has been built and installed on all build nodes (amd64 and armhf). One issue related to permissions for root and unpriviliged users needs to be solved before disorderfs can be used on reproducible.debian.net. (h01ger)
strip-nondeterminism
Version 0.011-1 has been released on August 29th. The new version updates dh_strip_nondeterminism to match recent changes in debhelper. (Andrew Ayer)
disorderfs
disorderfs, the new FUSE filesystem to ease testing of filesystem-related variations, is now almost ready to be used. Version 0.2.0 adds support for extended attributes. Since then Andrew Ayer also added support to reverse directory entries instead of shuffling them, and arbitrary padding to the number of blocks used by files.
Package reviews
142 reviews have
been removed, 48 added and 259 updated this week.
Santiago Vila renamed the not_using_dh_builddeb issue into varying_mtimes_in_data_tar_gz_or_control_tar_gz to align better with other tag names.
New issue identified this week: random_order_in_python_doit_completion.
37 FTBFS issues have been reported by Chris West (Faux) and Chris Lamb.
Misc.
h01ger gave a talk at FrOSCon on August 23rd. Recordings are already online.
These reports are being reviewed and enhanced every week by many people hanging out on #debian-reproducible. Huge thanks!
A good amount of the Debian reproducible
builds team had the chance to
enjoy face-to-face interactions during DebConf15.
Names in red and blue were all present at DebConf15
Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian.
A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive.
Toolchain fixes
Kenneth J. Pronovici uploaded epydoc/3.0.1+dfsg-12 which makes class and modules ordering predictable (#795835) and fixes __repr__ so memory addresses don't appear in docs (#795826). Patches by Val Lorentz.
Sergei Golovan uploaded erlang/1:18.0-dfsg-2 which adds support for SOURCE_DATE_EPOCH to erlc. Patch by Chris West (Faux) and Chris Lamb.
Dmitry Shachnev uploaded sphinx/1.3.1-5 which make grammar, inventory, and JavaScript locales generation deterministic. Original patch by Val Lorentz.
Enrico Tassi uploaded lua-ldoc/1.4.3-3 which now pass the -d option to txt2man and add the --date option to override the current date.
Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2htmlSOURCE_DATE_EPOCH aware.
#796330 on d-shlibs by Reiner Herrmann: sort with LC_ALL set to C.
#795985 on xorg by Dhole: set the timezone to UTC before calling asciidoc.
#795987 on pngcheck by Dhole: set the date in the man pages to the latest debian/changelog entry.
#795997 on python-babel by Val Lorentz: make build timestamp independent from the timezone and remove the name of the build system locale from the documentation.
#796092 on a7xpg by Reiner Herrmann: sort with LC_ALL set to C.
#796212 on bittornado by Chris Lamb: remove umask-varying permissions.
#796251 on liblucy-perl by Niko Tyni: generate lib/Lucy.xs in a deterministic order.
#796271 on tcsh by Reiner Herrmann: sort with LC_ALL set to C.
#796275 on hspell by Reiner Herrmann: remove timestamp from aff files generated by mk_he_affix.
#796324 on fftw3 by Reiner Herrmann: remove date from documentation files.
#796335 on nasm by Val Lorentz: remove extra timestamps from the build system.
#796360 on libical by Chris Lamb: removes randomess caused Perl in generated icalderivedvalue.c.
#796375 on wcd by Dhole: set the date in the man pages to the latest debian/changelog entry.
#796376 on mapivi by Dhole: set the date in the man pages to the latest debian/changelog entry.
#796527 on vserver-debiantools by Dhole: set the date in the man pages to the latest debian/changelog entry.
St phane Glondu reported two issues regarding embedded build date in omake and cduce.
Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar.
Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found.
reproducible.debian.net
Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays.
armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger)
The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger)
mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger)
Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output.
Connections to UDD have been made more robust. (Mattia Rizzolo)
diffoscope development
diffoscopeversion 31 was released on August 21st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep.
New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo).
jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb.
strip-nondeterminism development
Andrew Ayer released strip-nondeterminismversion 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added.
Testing directory ordering issues: disorderfs
During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random.
Documentation update
Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C.
Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it.
Package reviews
44 reviews have
been removed, 192 added and 77 updated this week.
New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex.
117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni.
Misc.
Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source!
Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.
#792673 on bup by use date from debian/changelog when generating version strings.
#792684 on cain by Chris Lamb: ensure stable permissions when creating source tarball.
#792709 on dict-jargon by Dhole: set timestamp in archive using the latest entry of debian/changelog.
#792727 on libaqbanking by Micha Lenk (upstream): sort source files in documentation.
#792763 on docbook-dsssl by Chris Lamb: sort input files when creating changelog.
#792770 on lynx-cur by Reiner Herrmann: use C locale when sorting configuration files.
#792771 on mu-cade by Reiner Herrmann: use C locale when sorting source files.
#792772 on titanion by Reiner Herrmann: use C locale when sorting source files.
#792783 on linuxlogo by Reiner Herrmann: use C locale when sorting source files.
#792821 on pkg-config by Juan Picca: use C locale when sorting source files.
#792828 on tiger by Daniel Kahn Gillmor: use C locale when listing soure files.
reproducible.debian.net
The statistics on the main page of reproducible.debian.net are now updated every five minutes. A random unreviewed package is suggested in the look at a package form on every build. (h01ger)
A new package set based new on the Core Internet Infrastructure census has been added. (h01ger)
Testing of FreeBSD has started, though no results yet. More details have been posted to the freebsd-hackers mailing list. The build is run on a new virtual machine running FreeBSD 10.1 with 3 cores and 6 GB of RAM, also sponsored by Profitbricks.
strip-nondeterminism development
Andrew Ayer released version 0.009 of strip-nondeterminism. The new version will strip locales from Javadoc, include the name of files causing errors, and ignore unhandled (but rare) zip64 archives.
debbindiff development
Lunar continued its major refactoring to enhance code reuse and pave the way to fuzzy-matching and parallel processing. Most file comparators have now been converted to the new class hierarchy.
In order to support for archive formats, work has started on packaging Python bindings for libarchive. While getting support for more archive formats with a common interface is very nice, libarchive is a stream oriented library and might have bad performance with how debbindiff currently works. Time will tell if better solutions need to be found.
Documentation update
Lunar started a Reproducible builds HOWTO intended to explain the different aspects of making software build reproducibly to the different audiences that might have to get involved like software authors, producers of binary packages, and distributors.
Package reviews
17 obsolete
reviews have
been removed, 212 added and 46 updated this week.
15 new bugs for packages failing to build from sources have been reported by Chris West (Faux), and Mattia Rizzolo.
Presentations
Lunar presented Debian efforts and some recipes on making software build reproducibly at Libre Software Meeting 2015. Slides and a video recording are available.
Misc.
h01ger, dkg, and Lunar attended a Core Infrastructure Initiative meeting. The progress and tools mode for the Debian efforts were shown. Several discussions also helped getting a better understanding of the needs of other free software projects regarding reproducible builds. The idea of a global append only log, similar to the logs used for Certificate Transparency, came up on multiple occasions. Using such append only logs for keeping records of sources and build results has gotten the name Binary Transparency Logs . They would at least help identifying a compromised software signing key. Whether the benefits in using such logs justify the costs need more research.
What happened about the reproducible
builds effort this week:
Media coverage
Daniel Stender published an English translation of the article which originally
appeared in Linux Magazin in Admin Magazine.
Toolchain fixes
Fixes landed in the Debian archive:
Lunar uploaded docbook-to-man/1:2.0.0-34 which removes a timestamp in generated manpages. Original patch by Chris Lamb.
Stefano Rivera uploaded dh-python/1.20150628-1 which now sorts namespace files. Original patch by Chris Lamb.
Christian Hofstaedtler uploaded ruby2.2/2.2.2-2 which now uses UTC for the dates in gemspec files. Original patch by Chris Lamb.
reproducible.debian.net
A new package set for the X Strike Force has been added. (h01ger)
Bugs tagged with locale are now visible in the statistics. (h01ger)
Some work has been done add tests for NetBSD. (h01ger)
Many changes by Mattia Rizzolo have been merged on the whole infrastructure:
IRC notifications when known reproducible packages stops buildig successfully.
What happened about the reproducible
builds effort for this week:
Toolchain fixes
Uploads that should help other packages:
Stephen Kitt uploaded mingw-w64/4.0.2-2 which avoids inserting timestamps in PE binaries, and specify dlltool's temp prefix so it generates reproducible files.
Stephen Kitt uploaded binutils-mingw-w64/6.1 which fixed dlltool to initialize its output's .idata$6 section, avoiding random data ending up there.
Patch submitted for toolchain issues:
#787159 on openjdk-7 by Emmanuel Bourg: sort the annotations and enums in package-tree.html produced by javadoc.
#787250 on python-qt4 by Reiner Herrmann: sort imported modules to get reproducible output.
#787251 on pyqt5 by Reiner Herrmann: sort imported modules to get reproducible output.
Some discussions have been started in Debian and with upstream:
#786927 on flowscan by Dhole: remove timestamps from gzip files and fix mtimes of packaged files.
#786959 on python3.5 by Lunar: set build date of binary and documentation to the time of latest debian/changelog entry, prevent gzip from storing a timestamp.
reproducible.debian.net
Holger Levsen added two new package sets: pkg-javascript-devel and pkg-php-pear. The list of packages with and without notes are now sorted by age of the latest build.
Mattia Rizzolo added support for email notifications so that maintainers can be warned when a package becomes unreproducible. Please ask Mattia or Holger or in the #debian-reproducible IRC channel if you want to be notified for your packages!
strip-nondeterminism development
Andrew Ayer fixed the gzip handler so that it skip adding a predetermined timestamp when there was none.
Documentation update
Lunar added documentation about mtimes of file extracted using unzip being timezone dependent. He also wrote a short example on how to test reproducibility.
Stephen Kitt updated the documentation about timestamps in PE binaries.
Documentation and scripts to perform weekly reports were published by Lunar.
Package reviews
50 obsolete
reviews have
been removed, 51 added and 29 updated this week. Thanks Chris West and Mathieu
Bridon amongst others.
New identified issues:
Misc.
Lunar will be talking (in French) about reproducible builds at Pas Sage en Seine on June 19th, at 15:00 in Paris.
Meeting will
happen this Wednesday, 19:00 UTC.
What happened about the reproducible
builds effort for this week:
Presentations
On May 26th,Holger Levsen presented reproducible builds in Debian at CCC Berlin for the Datengarten 52. The presentation was in German and the slides in English. Audio and video recordings are available.
Toolchain fixes
Dmitry Shachnev uploaded pyqt5/5.4.1+dfsg-3 which makes pyuic output imports in stable order. Original patch by Reiner Herrmann.
Lunar uploaded mozilla-devscripts/0.41 which uses the UTC timezone when calling zip or unzip.
Patches submitted which did not make their way to the archive yet:
#787327 on vim by Reiner Herrmann: set a constant user and set modified-by option.
#787650 on lush by Daniel Kahn Gillmor: remove __DATE__ and __TIME__ macros from source.
#787669 on cloc by Daniel Kahn Gillmor: use time of latest debian/changelog entry in manpage.
#787675 on ricochet by Daniel Kahn Gillmor: patch configure to allow an external build date and set it to the time of latest debian/changelog entry.
#787804 on clipper by akira: set HTML_TIMESTAMP to NO in Doxygen configuration.
#787829 on colobot by akira: set HTML_TIMESTAMP to NO in Doxygen configuration.
#787865 on fastjet by akira: call Doxygen with HTML_TIMESTAMP set to NO.
#787916 on cal3d by akira: remove $datetime from the file api_footer.html.
#787918 on dime by akira: remove $datetime from the file footer.html.
Daniel Kahn Gilmor also started discussions for emacs24 and the unsorted lists in generated .el files, the recording of a PID number in lush, and the reproducibility of ISO images in grub2.
reproducible.debian.net
Notifications are now sent when the build environment for a package has changed between two builds. This is a first step before automatically building the package once more. (Holger Levsen)
jenkins.debian.net was upgraded to Debian Jessie. (Holger Levsen)
A new variation is now being tested: $PATH. The second build will be done with a /i/capture/the/path added. (Holger Levsen)
Holger Levsen with the help of Alexander Couzens wrote extra job to test the reproducibility of coreboot. Thanks James McCoy for helping with certificate issues.
Mattia Rizollo made some more internal improvements.
strip-nondeterminism development
Andrew Ayer released strip-nondeterminism/0.008-1. This new version fixes the gzip handler so that it now skip adding a predetermined timestamp when there was none.
Holger Levsen sponsored the upload.
Documentation update
The pages about timestamps in manpages generated by Doxygen, GHC .hi files, and Jar files have been updated to reflect their status in upstream.
Markus Koschany documented an easy way to prevent Doxygen to write timestamps in HTML output.
Package reviews
83 obsolete
reviews have
been removed, 71 added and 48 updated this week.
Meetings
A meeting was held on 2015-06-03. Minutes and full logs are available.
It was agreed to hold such a meeting every two weeks for the time being. The time of the next meeting should be announced soon.
If you ve perused the ActivityPub feed of certificates whose keys are known to be compromised, and clicked on the Show More button to see the name of the certificate issuer, you may have noticed that some issuers seem to come up again and again.
This might make sense after all, if a CA is issuing a large volume of certificates, they ll be seen more often in a list of compromised certificates.
In an attempt to see if there is anything that we can learn from this data, though, I did a bit of digging, and came up with some illuminating results.
Insert obligatory "not THAT data" comment here
If you don't know who Professor Julius Sumner Miller is, I highly recommend finding out
No thanks, Bender, I'm busy tonight
What happened in the 
