Gunnar Wolf: We are GREAT at handling multimedia!


SOURCE_DATE_EPOCH
support:
tar --sort=name
we need to compile tar before downloading everythingCONFIG_AUTOREMOVE
to reduce required spaceopenwrt_get_banner()
: use staging_dir
instead of build_dir
because the former is persistent among the two builds.node_cleanup_tmpdirs
git reset/clean
like Jenkins doesWORKSPACE
dir names, as WORKSPACE cannot be generated from $0
as it's a temporary name.scheduling_args.message
.dpkg
1.18.14 has migrated to stretch.debdiff(1)
what patch(1)
is to diff(1)
. It is a
general tool that is not specific to Reproducible Builds. This week, Ximin Luo
worked on making it more "production-ready" and will soon submit it for
inclusion in devscripts.
reprotest development
Ximin Luo significantly improved reprotest, adding presets and auto-detection
of which preset to use. One can now run e.g. reprotest auto .
or reprotest
auto $pkg_$ver.dsc
instead of the long command lines that were needed before.
He also made it easier to set up build dependencies inside the virtual server
and made it possible to specify pre-build dependencies that reprotest itself
needs to set up the variations. Previously one had to manually edit the virtual
server to do that, which was not very usable to humans without an in-depth
knowledge of the building process.
These changes will be tested some more and then released in the near future as
reprotest 0.4
.
siamezzze: I've been accepted to #outreachy winter internship - going to work with Debian reproducible builds team. So excited about that! <3 DebianToolchain development and fixes dpkg:
62~bpo8+1
was
uploaded
to jessie-backports by Mattia Rizzolo.
Meanwhile in git, Ximin Luo greatly improved speed by fixing a O(n2)
lookup
which was causing diffs of large packages such as GCC and glibc to take many
more hours than was necessary. When this commit is released, we should
hopefully see full diffs for such
packages
again. Currently we have 197 source packages which - when built - diffoscope
fails to analyse.
buildinfo.debian.net development
dpkg-genbuildinfo
includes Build-Path
only
if it is under /build
. HW42 updated our jobs so this is the case for our
builds too, so you can see the build path in the .buildinfo files.$pkg-$version
directory naming scheme (which is what dpkg-source -x
gives but is not
mandated by Debian nor always-true) or that they're being built from a SCM.dpkg-genbuildinfo
also records a sanitised Environment
. This is
different in our builds, so HW42, Reiner and Holger updated our jobs to hide
these differences from diffoscope output.#!/bin/zsh
set -e
PATHS=$(tempfile)
NEWKEYS=$(tempfile)
NEWKEYRING=$(tempfile)
FARTHEST_TEN=$(tempfile)
trap "rm -f $ PATHS $ NEWKEYS $ NEWKEYRING $ FARTHEST_TEN " EXIT
keyring=$ 1:-ksp-dc16.gpg
myfpr=$ 2:-2100A32C46F895AF3A08783AF6D3495BB0AE9A02
#keyserver=$ 3:-http://pool.sks-keyservers.net:11371/
# this doesn't handle hokey fetch failures
#(for fpr in $(hkt list --keyring $ keyring --output-format JSON jq '.[].publickey.fpr')
#do
# hokey fetch --keyserver "$ keyserver " --validation-method MatchPrimaryKeyFingerprint "$ (Q)fpr "
#done) >$ NEWKEYS
#
#gpg2 --no-default-keyring --keyring $ NEWKEYRING --import $ NEWKEYS
cp "$ keyring " "$ NEWKEYRING "
gpg2 --no-default-keyring --keyring $ NEWKEYRING --refresh
hkt findpaths --keyring $ NEWKEYRING '' '' '' > $ PATHS
id=$(awk -F, "/$ myfpr )\$/ sub(/\(/,BLANKY,\$1);print \$1; " $ PATHS )
grep -e ",\[$ id ," -e ",$ id \]" $ PATHS sort -n tail -n 10 > $ FARTHEST_TEN
targetids=($ (f)"$ $((sed 's/^.*\[//;s/,.*$//;' $ FARTHEST_TEN ; sed 's/\])$//;s/.*,//;' $ FARTHEST_TEN ) sort -n -u grep -v "^$ id $") " )
targetfprs=($(for i in $ targetids ; do awk -F, "/\($ i ,[^[]/ sub(/\)/,BLANKY,\$2); print \$2 " $ PATHS ; done))
gpg2 --no-default-keyring --keyring $ NEWKEYRING --list-keys $ targetfprs
Output:
pub rsa4096/0x664F1238AA8F138A 2015-07-14 [SC]
Key fingerprint = 3575 0B8F B6EF 95FF 16B8 EBC0 664F 1238 AA8F 138A
uid [ unknown] Daniel Lange <dl.ml1@usrlocal.de>
sub rsa4096/0x03BEE1C11DB1954B 2015-07-14 [E]
pub rsa4096/0xDF23DA3396978EB3 2014-09-05 [SC]
Key fingerprint = BBBC 58B4 5994 CF9C CC56 BCDA DF23 DA33 9697 8EB3
uid [ undef ] Michael Meskes <michael@fam-meskes.de>
uid [ undef ] Michael Meskes <meskes@postgresql.org>
uid [ undef ] Michael Meskes <michael.meskes@credativ.com>
uid [ undef ] Michael Meskes <meskes@debian.org>
sub rsa4096/0x85C3AFFECF0BF9B5 2014-09-05 [E]
sub rsa4096/0x35D857C0BBCB3B25 2014-11-04 [S]
pub rsa4096/0x1E953E27D4311E58 2009-07-12 [SC]
Key fingerprint = C2FE 4BD2 71C1 39B8 6C53 3E46 1E95 3E27 D431 1E58
uid [ undef ] Chris Lamb <chris@chris-lamb.co.uk>
uid [ undef ] Chris Lamb <lamby@gnu.org>
uid [ undef ] Chris Lamb <lamby@debian.org>
sub rsa4096/0x72B3DBA98575B3F2 2009-07-12 [E]
pub rsa4096/0xDF6D76C44D696F6B 2014-08-15 [SC] [expires: 2017-06-03]
Key fingerprint = 1A6F 3E63 9A44 67E8 C347 6525 DF6D 76C4 4D69 6F6B
uid [ unknown] Sven Bartscher <sven.bartscher@weltraumschlangen.de>
uid [ unknown] Sven Bartscher <svenbartscher@yahoo.de>
uid [ unknown] Sven Bartscher <kritzefitz@debian.org>
sub rsa4096/0x9E83B071ED764C3A 2014-08-15 [E]
sub rsa4096/0xAEB25323217028C2 2016-06-14 [S]
pub rsa4096/0x83E33BD7D4DD4CA1 2015-11-12 [SC] [expires: 2017-11-11]
Key fingerprint = 0B5A 33B8 A26D 6010 9C50 9C6C 83E3 3BD7 D4DD 4CA1
uid [ unknown] Jerome Charaoui <jerome@riseup.net>
sub rsa4096/0x6614611FBD6366E7 2015-11-12 [E]
sub rsa4096/0xDB17405204ECB364 2015-11-12 [A] [expires: 2017-11-11]
pub rsa4096/0xF823A2729883C97C 2014-08-26 [SC]
Key fingerprint = 8ED6 C3F8 BAC9 DB7F C130 A870 F823 A272 9883 C97C
uid [ unknown] Lucas Kanashiro <kanashiro@debian.org>
uid [ unknown] Lucas Kanashiro <kanashiro.duarte@gmail.com>
sub rsa4096/0xEE6E5D1A9C2F5EA6 2014-08-26 [E]
pub rsa4096/0x2EC0FFB3B7301B1F 2014-08-29 [SC] [expires: 2017-04-06]
Key fingerprint = 76A2 8E42 C981 1D91 E88F BA5E 2EC0 FFB3 B730 1B1F
uid [ unknown] Niko Tyni <ntyni@debian.org>
uid [ unknown] Niko Tyni <ntyni@cc.helsinki.fi>
uid [ unknown] Niko Tyni <ntyni@iki.fi>
sub rsa4096/0x129086C411868FD0 2014-08-29 [E] [expires: 2017-04-06]
pub rsa4096/0xAA761F51CC10C92A 2016-06-20 [SC] [expires: 2018-06-20]
Key fingerprint = C9DE 2EA8 93EE 4C86 BE73 973A AA76 1F51 CC10 C92A
uid [ unknown] Roger Shimizu <rogershimizu@gmail.com>
sub rsa4096/0x2C2EE1D5DBE7B292 2016-06-20 [E] [expires: 2018-06-20]
sub rsa4096/0x05C7FD79DD03C4BB 2016-06-20 [S] [expires: 2016-09-18]
Note that this completely neglects potential victims who are
unconnected within the KSP set.
SOURCE_DATE_EPOCH
for manpage date.SOURCE_DATE_EPOCH
for dates in makeinfo output.--list-tools
, where a syntax error prevented proper listing of all tools.*.epub
files.
tests.reproducible-builds.org
SOURCE_DATE_EPOCH
.
Toolchain fixes
SOURCE_DATE_EPOCH
to generate reproducible manpages (original patch by Reiner Herrmann).SOURCE_DATE_EPOCH
(original patch by Alexis Bienven e).next
branch, which sorts file lists obtained with file(GLOB)
.--clamp-mtime
has been released upstream, closing #816072, which was the blocker for #759886 "dpkg-dev: please make mtimes of packaged files deterministic" which we now hope will be closed soon.SOURCE_DATE_EPOCH
for deterministic timestamp (merged upstream)SOURCE_DATE_EPOCH
as embedded build dateSOURCE_DATE_EPOCH
for embedded timestamp (similar patch by Bas Couwenberg already applied and forwarded upstream)SOURCE_DATE_EPOCH
for timestamp embedded into metadataDEB_BUILD_OPTIONS="nocheck"
SOURCE_DATE_EPOCH
in /usr/share/dpkg/*.mk.-fdebug-prefix-map
to the compilers options)echo
in a portable manner across shells.PERL_HASH_SEED=0
when calling the scripts that generate samples.debian/changelog
entry as documentation timestamp.debian/changelog
entry as documentation timestamp..pk3
archive.Makefile
.SOURCE_DATE_EPOCH
.SOURCE_DATE_EPOCH
to preset the ntp_era_split
parameter.help2man
./bin/sh
in example Makefile
.dcmd
supports .buildinfo files. Original patch by josch.qch
reproducible by using a fixed date instead of the current time. Original patch by Dhole.CreationDate
not appear in comments of DVI / PS files produced by TeX. He also mentioned that some timestamps can be replaced by using the -output-comment
option and that the next version of pdftex
will have patches inspired by reproducible build to mitigate the effects (see SOURCE_DATE_EPOCH patches) .
SOURCE_DATE_EPOCH
.armhf
build node has been added (thanks to Vagrant Cascadian) and integrated into the Jenkins setup for 4 new armhf
builder jobs. (h01ger)
All packages for Debian testing (Stretch) have been tested on armhf
in just 42 days. It took 114 days to get the same point for unstable back when the armhf
test infrastructure was much smaller.
Package sets have been enabled for testing on armhf
. (h01ger)
Packages producing architecture-independent ( Arch:all ) binary packages together with architecture dependent packages targeted for specific architectures will now only be tested on matching architectures. (Steven Chamberlain, h01ger)
As the Jenkins setup is now made of 252 different jobs, the overview has been split into 11 different smalller views. (h01ger)
data.tar
are reproducible, with the patches, dpkg-deb
uses the --clamp-mtime
option added in tar/1.28-1 when available. An updated package has been uploaded to the experimental repository. This removed the need for a modified debhelper as all required changes for reproducibility have been merged or are now covered by dpkg
.
armhf
build system, allowing to run 6 more armhf
builder jobs, right there. (h01ger)
Stop requiring a modified debhelper and adapt to the latest dpkg experimental version by providing a predetermined identifier for the .buildinfo
filename. (Mattia Rizzolo, h01ger)
New X.509 certificates were set up for jenkins.debian.net and reproducible.debian.net using Let's Encrypt!. Thanks to GlobalSign for providing certificates for the last year free of charge. (h01ger)
preferences.js
files.
To be able to lift the restriction that packages must be built in the same path, translation support for the __FILE__
C pre-processor macro would also be required. Joerg Sonnenberger submitted a patch back in 2010 that would still be useful today.
Chris Lamb started work on providing a deterministic mode for debootstrap.
Packages fixed
The following packages have become reproducible due to changes in their
build dependencies:
bouncycastle,
cairo-dock-plug-ins,
darktable,
gshare,
libgpod,
pafy,
ruby-redis-namespace,
ruby-rouge,
sparkleshare.
The following packages became reproducible after getting fixed:
SOURCE_DATE_EPOCH
in the documentation generator.Makefile
.tmpfs
. (h01ger)
200 GiB have been added to jenkins.debian.net (thanks to ProfitBricks!) to make room for new jobs. The current count is at 962 and growing!
diffoscope development
Aside from some minor bugs that have been fixed, a one-line change made huge memory (and time) savings as the output of transformation tool is now streamed line by line instead of loaded entirely in memory at once.
disorderfs development
Andrew Ayer released disorderfs version 0.4.2-1 on December 22th. It fixes a memory corruption error when processing command line arguments that could cause command line options to be ignored.
Documentation update
Many small improvements for the documentation on reproducible-builds.org sent by Georg Koppen were merged.
Package reviews
666 (!) reviews have been removed, 189 added and 162 updated in the previous week.
151 new fail to build from source reports have been made by Chris West, Chris Lamb, Mattia Rizzolo, and Niko Tyni.
New issues identified: unsorted_filelist_in_xul_ext_preferences, nondeterminstic_output_generated_by_moarvm.
Misc.
Steven Chamberlain drew our attention to one analysis of the Juniper ScreenOS Authentication Backdoor: Whilst this may have been added in source code, it was well-disguised in the disassembly and just 7 instructions long. I thought this was a good example of the current state-of-the-art, and why we'd like our binaries and eventually, installer and VM images reproducible IMHO.
Joanna Rutkowska has mentioned possible ways for Qubes to become reproducible on their development mailing-list.
-T
flag which will clamp superblock and file timestamps to a given time in epoch format.en_US
when generating the javadoc.%Y-%m-%d
for the dates.
Daniel Kahn Gillmor took another shot at the issue of source path captured in DWARF symbols. A patch has been sent for review by GCC upstream to add the ability to read an environment variable with -fdebug-prefix-map
.
Packages fixed
The following 24 packages have become reproducible due to changes in their
build dependencies:
gkeyfile-sharp,
gprbuild,
graphmonkey,
gthumb,
haskell-yi-language,
ion,
jackson-databind,
jackson-dataformat-smile,
jackson-dataformat-xml,
jnr-ffi,
libcommons-net-java,
libproxy,
maven-shared-utils,
monodevelop-database,
mydumper,
ndesk-dbus,
nini,
notify-sharp,
pixz,
protozero,
python-rtslib-fb,
slurm-llnl,
taglib-sharp,
tomboy-latex.
The following packages became reproducible after getting fixed:
SOURCE_DATE_EPOCH
(already fixed upstream).SOURCE_DATE_EPOCH
.SOURCE_DATE_EPOCH
.x86_64
. So far only 151 packages from the buildsys-build
group are tested (currently all unreproducible), but the plan is to build all 17,000 source packages in Fedora 23 and rawhide. The page presenting the results should also soon be improved. (h01ger, Dhiru Kholia)
For Arch Linux, all 2223 packages from the extra repository will also be tested from now on. Packages in extra" are tested every four weeks, while those from core every week. Statistics are now displayed alongside the results. (h01ger)
jenkins.debian.net has been updated to jenkins-job-builder version 1.3.0. Many job configurations have been simplified and refactored using features of the new version. This was another milestone for the jenkins.debian.org
migration. (Phil Hands, h01ger)
diffoscope development
Chris Lamb announced try.diffoscope.org: an online service that runs diffoscope on user provided files.
.buildinfo
files from being removed from the upload queue.
Toolchain fixes
dh_installinit
. Patch by Reiner Herrmann.debian/changelog
entry for LXC_GENERATE_DATE
.SOURCE_DATE_EPOCH
.armhf
. (h01ger)
Arch Linux packages in the multilib and community repositories (4,000 more source packages) are also being tested. All of these test results are better analyzed and nicely displayed together with each package. (h01ger)
For Fedora, build jobs can now run in parallel. Two are currently running, now testing reproducibility of 785 source packages from Fedora 23.
mock/1.2.3-1.1 has been uploaded to experimental to better build RPMs. (h01ger)
Work has started on having automatic build node pools to maximize use of armhf
build nodes. (Vagrant Cascadian)
diffoscope development
Version 43 has been released on December 15th. It has been dubbed as epic! as it contains many contributions that were written around the summit in Athens.
Baptiste Daroussin found that running diffoscope on some Tar archives could overwrite arbitrary files. This has been fixed by using libarchive instead of Python internal Tar library and adding a sanity check for destination paths. In any cases, until proper sandboxing is implemented, don't run diffosope on unstrusted inputs outside an isolated, throw-away system.
Mike Hommey identified that the CBFS comparator would needlessly waste time scanning big files. It will now not consider any files bigger than 24 MiB 8 MiB more than the largest ROM created by coreboot at this time. An encoding issue related to Zip files has also been fixed. (Lunar)
New comparators have been added: Android dex files (Reiner Herrmann), filesystem images using libguestfs (Reiner Herrmann), icons and JPEG images using libcaca (Chris Lamb), and OS X binaries (Clemens Lang). The comparator for Free Pascal Compilation Unit will now only be used when the unit version matches the compiler one. (Levente Polyak)
A new multi-file HTML output with on-demand loading of long diffs is available through the --html-dir
option. On-demand loading requires jQuery which path can be specified through the --jquery
option. The diffs can also be simply browsed for non-JavaScript users or when jQuery is not available. (Joachim Breitner)
en_US.UTF-8
(Ed Maste), the --list-tools
option can now support multiple systems (Mattia Rizzolo, Levente Polyak, Lunar).
Many internal changes and code clean-ups have been made, paving the way for parallel processing. (Lunar)
Version 44 was released on December 18th fixing an issue affecting .deb
lacking a md5sums
file introduced in a previous refactoring (Lunar). Support has been added for Mozilla optimized Zip files. (Mike Hommey). The HTML output has been optimized in size (Mike Hommey, Esa Peuha, Lunar), speed (Lunar), and will now properly number lines (Mike Hommey). A message will always be displayed when lines are ignored at the end of a diff (Lunar). For portability and consistency, Python os.walk()
function is now used instead of find
to perform directory listing. (Lunar)
Documentation update
Package reviews
143 reviews have been removed, 69 added and 22 updated in the previous week.
Chris Lamb reported 12 new FTBFS issues.
News issues identified this week: random_order_in_init_py_generated_by_python-genpy, timestamps_in_copyright_added_by_perl_dist_zilla, random_contents_in_dat_files_generated_by_chasen-dictutils_makemat, timestamps_in_documentation_generated_by_pandoc.
Chris West did some improvements on the scripts used to manage notes in the misc repository.
Misc.
Accounts of the reproducible builds summit in Athens were written by Thomas Klausner from NetBSD and Hans-Christoph Steiner from The Guardian Project.
Some openSUSE developers are working on a hackweek on reproducible builds which was discussed on the opensuse-packaging mailing-list.
.buildinfo
with different environment information can attest to the
same exact binary artifact..buildinfo
files can coexist for the same .deb
as long as the listed
checksums match the source and binary package in the archive..buildinfo
can be signed in-line to certify where a build comes from.metview
script.SOURCE_DATE_EPOCH
in grub image generator (#787795).
reproducible.debian.net
An issue with some armhf
build nodes was tracked down to a bad interaction between uname26 personality and new glibc (Vagrant Cascadian).
A Debian package was created for koji, the RPM building and tracking system used by Fedora amongst others. It is currently waiting for review in the NEW queue. (Ximin Luo, Marek Marczykowski-G recki)
diffoscope development
diffoscope now has a dedicated mailing list to better accommodate its growing user and developer base.
Going through diffoscope's guts together enabled several new contributors. Baptiste Daroussin, Ed Maste, Clemens Lang, Mike McQuaid, Joachim Breitner all contributed their first patches to improve portability or add new features. Regular contributors Chris Lamb, Reiner Herrmann, and Levente Polyak also submitted improvements.
dh_installinit
source files in a stable order.
Chris Lamb found how to make cython output deterministic by ordering the keys used to traverse a dict
.
Reiner Herrmann proposed a patch for pyside-tools to remove the timestamps embedded by rcc
in the generated Python code.
Mattia Rizzolo rebased our custom version of debhelper on version 9.20151126.
As no objections have been made so far, Mattia Rizzolo has filled #805872 asking -Wdate-time
to be turned on by default in dpkg-buildflag. Guillem has since sent a final warning before proceeding as such in the next dpkg upload.
Russ Allbery added support for SOURCE_DATE_EPOCH in podlators 4.00 which Niko Tyni intend to backport to Perl 5.22.
Packages fixed
The following packages have become reproducible due to changes in their
build dependencies:
fontforge,
golang-github-tinylib-msgp,
libpango-perl,
libparanamer-java,
libxaw,
sqljet,
stringtemplate4,
uzbl,
zope-mysqlda.
The following packages became reproducible after getting fixed:
help2man
instead of embedded copy.help2man
instead of embedded copy.LC_ALL
instead of LANG
to ensure how dd
output looks like.LC_ALL=C
when enumerating files to link.debian/changelog
entry.SOURCE_DATE_EPOCH
in VersionInfo.cmake
.armhf
build node has been added (resulting of two more armhf
build jobs). Thanks to Vagrant Cascadian for putting this Raspberry Pi 2B to help. (h01ger)
jenkins.debian.net has been made more robust against network and proxy failures. (h01ger)
A new 100 GB partition has been set up on reproducible.debian.net to prevent disk space issues. Thanks to ProfitBricks for its continuous support to our continuous test system. (h01ger)
New graphs showing usertagged bugs have been added on the dashboard to measure the progress without FTBFS issues. Please note that comparing the two graphs might be misleading as more than 1300 FTBFS bugs have been inventoried. (h01ger)
Package reviews
78 reviews have been removed, 116 added and 49 updated this week.
25 new FTBFS have been filed by Chris West, Chris Lamb and Santiago Vila.
New issues identified this week: timestamps_in_documentation_generated_with_libwibble, copyright_year_in_documentation_generated_by_sphinx, timestamps_in_documentation_generated_by_glib_genpod, random_order_of_tmpfiles_in_postinst, random_order_in_cython_output, timestamps_in_python_code_generated_by_pyside.
Reiner Herrmann and Lunar improved the prebuilder script: the script can now be called through a symlink, run parallel builds, calls diffoscope by its new name and ensure to install its recommends, and save the text output aside the HTML one.
Reiner also added a script to lookup the last update of notes for a given package.
Misc.
Santiago Villa has been recently working on making sure that Arch:all
packages were properly buildable by running dpkg-buildpackage -A
. This uncovered a question that is probably not currently addressed by the policy: on which architectures should architecture-independent be buildable?
Next.