169. This version includes the following changes:
[ Chris Lamb ] * Optimisations: - Use larger buffer/block sizes when extracting files from libarchive- based archives. - Use a much-shorter CSS class (instead of "diffponct") to dramatically reduce uncompressed HTML output. * Logging improvements: - Don't emit "Unable to stat file" warning/debug messages; we have entirely-artificial directory entries such as ELF sections which, of course, never exist as filesystem files. - Don't emit a "Returning a FooContainer" logging message - we already emit "Instantiating a FooContainer" one and are unlikely to fail in the middle. - Format the report size logging messages when generating HTML reports. - Add the target directory when logging which directory we are extracting containers to. * Miscellaneous: - Ignore "--debug" and similar arguments when creating a (hopefully useful) temporary directory. - Ensure all internal temporary directories have useful names. - Clarify a comment regarding diffoscope not extracting excluded files. [ Vagrant Cascadian ] * Skip a DEX-related test if the "procyon" tool is unavailable.
Welcome to the October 2020 report from the Reproducible Builds project. In our monthly reports, we outline the major things that we have been up to over the past month. As a brief reminder, the motivation behind the Reproducible Builds effort is to ensure flaws have not been introduced in the binaries we install on our systems. If you are interested in contributing to the project, please visit our main website.
The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.During the Reproducible Builds summit in Marrakesh in 2019, developers from the GNU Guix, NixOS and Debian distributions were able to produce a bit-for-bit identical GNU Mes binary despite using three different versions of GCC. Since this summit, additional work resulted in a bit-for-bit identical Mes binary using
tcc, and last month a fuller update was posted to this effect by the individuals involved. This month, however, David Wheeler updated his extensive page on Fully Countering Trusting Trust through Diverse Double-Compiling, remarking that:
GNU Mes rebuild is definitely an application of [Diverse Double-Compiling]. [..] This is an awesome application of DDC, and I believe it s the first publicly acknowledged use of DDC on a binaryThere was a small, followup discussion on our mailing list. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. This month, the Reproducible Builds project restarted our IRC meetings, managing to convene twice: the first time on October 12th (summary & logs), and later on the 26th (logs). As mentioned in previous reports, due to the unprecedented events throughout 2020, there will be no in-person summit event this year. On our mailing list this month El as Alejandro posted a request for help with a local configuration
reproducible=+fixfilepathDebian build flag by default. Enabling this
fixfilepathfeature will likely fix reproducibility issues in an estimated 500-700 packages. However, this month Vagrant Cascadian posted to the debian-devel mailing list:
It would be great to see theDebian Developer Stuart Prescott has been improving
reproducible=+fixfilepathfeature enabled by default in
dpkg-buildflags, and we would like to proceed forward with this soon unless we hear any major concerns or other outstanding issues. [ ] We would like to move forward with this change soon, so please raise any concerns or issues not covered already.
python-debian, a Python library that is used to parse Debian-specific files such as changelogs,
.dscs, etc. In particular, Stuart is working on adding support for
.buildinfofiles used for recording reproducibility-related build metadata:
This can mostly be a very thin layer around the existingA total of 159 Debian packages were categorised, 69 had their categorisation updated, and 33 had their classification removed this month, adding to our knowledge about identified issues. As part of this, Chris Lamb identified and classified two new issues:
Deb822types, using the existing
Changescode for the file listings, the existing
PkgRelationscode for the package listing and
gpg_*functions for signature handling.
go(version 1.15.3 has improved reproducibility over 1.14)
goxel(sort SCons-related filesystem ordering issue)
lal(rework an old date-related patch)
libsemigroups(build failure in single-CPU mode)
memcached(build failure in 2025 due to expired SSL certificate)
octant(SUSE-specific date issue)
openmpi4(date-related problem, revive old patch)
sbcl(datetime and hostname issue)
selinux-policy/policycoreutils(date-related issue in timezone)
161to Debian (later backported by Mattia Rizzolo), as well as made the following changes:
assert_diffhelper. [ ]
radare2to ensure our test pipelines continue to work [ ], and for the GNU Guix distribution Vagrant Cascadian diffoscope to version 161 [ ]. In related development, trydiffoscope is the web-based version of diffoscope. This month, Chris Lamb made the following changes:
--help-only test as being a superficial test. (#971506)
try.diffoscope.orgservice. [ ]
debhelpercompatibility level to 13 [ ] and bump
Standards-Versionto 4.5.0 [ ].
0.5.10-2was uploaded to Debian unstable by Holger Levsen, which enabled security hardening via
DEB_BUILD_MAINT_OPTIONS[ ] and dropped
dettrace[ ], and added yet another supply-chain security attack publication [ ].
relative_urlto fix missing translation icon on various pages. [ ]
tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
ath97. [ ]
sudocommand if we are not actually running
libvirt. [ ]
This is to say that anyone will be able to independently review Threema s security and verify that the published source code corresponds to the downloaded app.You can view the full announcement on Threema s website.
The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.During the Reproducible Builds summit in Marrakesh, GNU Guix, NixOS and Debian were able to produce a bit-for-bit identical binary when building GNU Mes, despite using three different major versions of GCC. Since the summit, additional work resulted in a bit-for-bit identical Mes binary using
tccand this month, a fuller update was posted by the individuals involved.
schroot(#902804) Last month, an issue was identified where a large number of Debian
.buildinfobuild certificates had been tainted on the official Debian build servers, as these environments had files underneath the
/usr/local/sbindirectory to prevent the execution of system services during package builds. However, this month, Aurelien Jarno and Wouter Verhelst fixed this issue in varying ways, resulting in a special
policy-rcd-declarative-deny-allpackage. Building on Chris Lamb s previous work on reproducible builds for Debian .ISO images, Roland Clobus announced his work in progress on making the Debian Live images reproducible. [ ] Lucas Nussbaum performed an archive-wide rebuild of packages to test enabling the
reproducible=+fixfilepathDebian build flag by default. Enabling the
fixfilepathfeature will likely fix reproducibility issues in an estimated 500-700 packages. The test revealed only 33 packages (out of 30,000 in the archive) that fail to build with
fixfilepath. Many of those will be fixed when the default LLVM/Clang version is upgraded. 79 reviews of Debian packages were added, 23 were updated and 17 were removed this month adding to our knowledge about identified issues. Chris Lamb added and categorised a number of new issue types, including packages that captures their build path via
quicktest.hand absolute build directories in documentation generated by Doxygen , etc. Lastly, Lukas Puehringer s uploaded a new version of the in-toto to Debian which was sponsored by Holger Levsen. [ ]
pgpdump, and check that the associated binary is actually installed before attempting to run it. (#969753)
guestfscleanup failure. [ ]
FALLBACK_FILE_EXTENSION_SUFFIX, otherwise we run
pgpdumpagainst all files that are recognised by
data. [ ]
jekyll-polyglotpackage is required [ ]. Lastly,
reproducible-builds.orgwere transferred to Software Freedom Conservancy. Many thanks to Brett Smith from Conservancy, J r my Bobbio (lunar) and Holger Levsen for their help with transferring and to Mattia Rizzolo for initiating this.
clutter(avoid a random ID in HTML from
kubernetes(1-bit order in manual page)
libint(merged, filesystem order)
-fprofile-arcsand code coverage)
libqb(date / copyright)
nauty(CPU type detection)
git2-rs(sort return ordering of
tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
systemctlstatus [ ] and the number of diffoscope processes [ ].
xzcompression format. [ ][ ][ ]
One hiccup we ve encountered in SecureDrop development is that not all Python wheels can be built reproducibly. We ship multiple (Python) projects in Debian packages, with Python dependencies included in those packages as wheels. In order for our Debian packages to be reproducible, we need that wheel build process to also be reproducibleParallel to this, transparencylog.com was also launched, a service that verifies the contents of URLs against a publicly recorded cryptographic log. It keeps an append-only log of the cryptographic digests of all URLs it has seen. (GitHub repo) On 18th September, Bernhard M. Wiedemann will give a presentation in German, titled Wie reproducible builds Software sicherer machen ( How reproducible builds make software more secure ) at the Internet Security Digital Days 2020 conference.
ftp.debian.orgare made from their claimed sources. It also served as a general update on the status of reproducible builds within Debian. The video (145 MB) and slides are available. There were also a number of other talks that involved Reproducible Builds too. For example, the Malayalam language mini-conference had a talk titled , ? ( I want to join Debian, what should I do? ) presented by Praveen Arimbrathodiyil, the Clojure Packaging Team BoF session led by Elana Hashman, as well as Where is Salsa CI right now? that was on the topic of Salsa, the collaborative development server that Debian uses to provide the necessary tools for package maintainers, packaging teams and so on. Jonathan Bustillos (Jathan) also gave a talk in Spanish titled Un camino verificable desde el origen hasta el binario ( A verifiable path from source to binary ). (Video, 88MB)
openwrt-develmailing list asking for clarification on when to raise the
PKG_RELEASEidentifier of a package. This is needed in order to successfully perform rebuilds in a reproducible builds context. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. Chris Lamb provided some comments and pointers on an upstream issue regarding the reproducibility of a Snap / SquashFS archive file. [ ]
.buildinfobuild certificates have been tainted on the official Debian build servers, as these environments have files underneath the
/usr/local/sbindirectory [ ]. He also filed against bug for
debrebuildafter spotting that it can fail to download packages from
snapshot.debian.org[ ]. This month, several issues were uncovered (or assisted) due to the efforts of reproducible builds. For instance, Debian bug #968710 was filed by Simon McVittie, which describes a problem with detached debug symbol files (required to generate a traceback) that is unlikely to have been discovered without reproducible builds. In addition, Jelmer Vernooij called attention that the new Debian Janitor tool is using the property of reproducibility (as well as diffoscope when applying archive-wide changes to Debian:
New merge proposals also include a link to the diffoscope diff between a vanilla build and the build with changes. Unfortunately these can be a bit noisy for packages that are not reproducible yet, due to the difference in build environment between the two builds. [ ]56 reviews of Debian packages were added, 38 were updated and 24 were removed this month adding to our knowledge about identified issues. Specifically, Chris Lamb added and categorised the
lessc_nondeterministic_keystoolchain issues. [ ][ ] Holger Levsen sponsored Lukas Puehringer s upload of the python-securesystemslib pacage, which is a dependency of in-toto, a framework to secure the integrity of software supply chains. [ ] Lastly, Chris Lamb further refined his merge request against the
debian-installercomponent to allow all arguments from
sources.listfiles (such as
[check-valid-until=no]) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure and sent a ping to the team that maintains that code.
getfem(embeds datetime and user, submitted via email)
getdp(hostname and user)
httpcomponents-client(Java documentation generator
lal(date and time issue, submitted via email)
OBS(discuss how to track old build
prjconfmetadata in buildinfo)
openblas(disable CPU detection)
python-eventlet(fails to build far in the future)
rna-star(date and hostname)
xz/b4(workaround CPU count influencing output, reported upstream)
<!ENTITY>declarations inside the Document Type Definition (DTD), or when a DTD or entity references an external resource. (#212)
pgpdump(1)can successfully parse some binary files, so check that the parsed output contains something sensible before accepting it. [ ]
gnumericfrom the Debian build-dependencies as it has been removed from the testing distribution. (#968742)
fallback_recognisesto prevent matching
.xsbbinary XML files.
ppudumpversion does not match our file header. [ ]
repr(object)output in Calling external command messages. [ ]
ppudumpversion 3.2.0 or higher. [ ]
setup.pythat diffoscope works with Python version 3.8 [ ] and Frazer Clews applied some Pylint suggestions [ ] and removed some deprecated methods [ ].
SOURCE_DATE_EPOCHage. [ ]
tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
arm64architecture. [ ]
armhf. [ ][ ][ ]
buildinfos.debian.net, etc.). [ ][ ][ ][ ][ ]
arm64architecture anymore. [ ]
If you think you know how to spread the word about reproducibility in the context of Bitcoin wallets through WalletScrutiny, your contributions are highly welcome on this PR [ ]Julien Lepiller posted to the list linking to a blog post by Tavis Ormandy titled You don t need reproducible builds. Morten Linderud (foxboron) responded with a clear rebuttal that Tavis was only considering the narrow use-case of proprietary vendors and closed-source software. He additionally noted that the criticism that reproducible builds cannot prevent against backdoors being deliberately introduced into the upstream source ( bugdoors ) are decidedly (and deliberately) outside the scope of reproducible builds to begin with. Chris Lamb included the Reproducible Builds mailing list in a wider discussion regarding a tentative proposal to include
.debpackages, adding his remarks regarding requiring a custom tool in order to determine whether generated build artifacts are identical in a reproducible context. [ ] Jonathan Bustillos (Jathan) posted a quick email to the list requesting whether there was a list of To do tasks in Reproducible Builds. Lastly, Chris Lamb responded at length to a query regarding the status of reproducible builds for Debian ISO or installation images. He noted that most of the technical work has been performed but there are at least four issues until they can be generally advertised as such . He pointed that the privacy-oriented Tails operation system, which is based directly on Debian, has had reproducible builds for a number of years now. [ ]
Welcome to the July 2020 report from the Reproducible Builds project. In these monthly reports, we round-up the things that we have been up to over the past month. As a brief refresher, the motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced from the original free software source code to the pre-compiled binaries we install on our systems. (If you re interested in contributing to the project, please visit our main website.)
ftp.debian.orgwere made from their claimed sources. Tavis Ormandy published a blog post making the provocative claim that You don t need reproducible builds , asserting elsewhere that the many attacks that have been extensively reported in our previous reports are fantasy threat models . A number of rebuttals have been made, including one from long-time contributor Reproducible Builds contributor Bernhard Wiedemann. On our mailing list this month, Debian Developer Graham Inggs posted to our list asking for ideas why the
openorienteering-mapperDebian package was failing to build on the Reproducible Builds testing framework. Chris Lamb remarked from the build logs that the package may be missing a build dependency, although Graham then used our own diffoscope tool to show that the resulting package remains unchanged with or without it. Later, Nico Tyni noticed that the build failure may be due to the relationship between the
FILEC preprocessor macro and the
-ffile-prefix-mapGCC flag. An issue in Zephyr, a small-footprint kernel designed for use on resource-constrained systems, around
.alibrary files not being reproducible was closed after it was noticed that a key part of their toolchain was updated that now calls
--enable-deterministic-archivesby default. Reproducible Builds developer kpcyrd commented on a pull request against the libsodium cryptographic library wrapper for Rust, arguing against the testing of CPU features at compile-time. He noted that:
I ve accidentally shipped broken updates to users in the past because the build system was feature-tested and the final binary assumed the instructions would be present without further runtime checksDavid Kleuker also asked a question on our mailing list about using
READMEfile [ ], marked the Alpine Linux continuous integration tests as currently disabled [ ] and linked the Arch Linux Reproducible Status page from our projects page [ ].
zipnote(1)to determine differences in a
.zipfile as we can use
libarchive. [ ]
--profileas a synonym for
--profile=-, ie. write profiling data to standard output. [ ]
strings(1)to eight characters to avoid unnecessary diff noise. [ ]
--no-exclude-directory-metadatahave been replaced with
--exclude-directory-metadata= yes,no. [ ]
xxd(1)and show bytes in groups of 4. [ ]
javap not found in pathif it is available in the path but it did not result in an actual difference. [ ]
... not available in pathmessages when looking for Java decompilers that used the Python class name instead of the command. [ ]
--debuglog noise by truncating the
has_some_contentmessages. [ ]
compare_fileslog message when the file does not have a literal name. [ ]
exit_if_paths_do_not_existto not check files multiple times. [ ][ ]
add_commenthelper method; don t mess with our internal list directly. [ ]
str.formatwith Python f-strings [ ] and make it easier to navigate to the
main.pyentry point [ ].
Nonein the failure case as we return a non-
Nonevalue in the success one. [ ]
NullChangesquasi-file to represent missing data in the Debian package comparator [ ] and clarify use of a null diff in order to remember an exit code. [ ]
diffoscope @args.txt. (!62)
objdump[ ][ ] and remove raw instructions from ELF tests [ ].
--verbose-level warning when the Archive::Cpio Perl module is missing. (!6) reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, Vagrant Cascadian made a number of changes to support diffoscope version 153 which had removed the (deprecated)
--no-exclude-directory-metadatacommand-line arguments, and updated the testing configuration to also test under Python version 3.8 [ ].
debhelperbuild tool impacting the reproducibility status of hundreds of packages that use the CMake build system. This month however, Niels Thykier uploaded
debhelperversion 13.2 that passes the
-DBUILD_RPATH_USE_ORIGIN=ONarguments to CMake when using the (currently-experimental) Debhelper compatibility level 14. According to Niels, this change:
should fix some reproducibility issues, but may cause breakage if packages run binaries directly from the build directory.34 reviews of Debian packages were added, 14 were updated and 20 were removed this month adding to our knowledge about identified issues. Chris Lamb added and categorised the
nondeterministic_order_of_debhelper_snippets_added_by_dh_fortran_mod[ ] and
gem2deb_install_mkmf_log[ ] toolchain issues. Lastly, Holger Levsen filed two more wishlist bugs against the
debrebuildDebian package rebuilder tool [ ][ ].
afl(fix an incorrectly built manual page varied from kernel boot options)
dnscrypt-proxy(sort the output of
graphviz(timezone issue, forwarded from Debian)
insighttoolkit(prevent CPU detection, forwarded upstream
ipopt(parallelism issue and use https://tracker.debian.org/pkg/strip-nondeterminism)
jboss-logging-tools(date, forwarded upstream)
lcov(date issue, already upstream)
multus(date issue, already upstream)
paperjam(date issue, forwarded upstream)
python-PyNaCl(sort Python glob/readdir)
python-enaml(workaround an open upstream Python issue)
sac(omit creation time from
sql-parser(sort, already upstream)
ugrep(CPU-related issue, already upstream)
unknown-horizons(filesystem ordering issue, already upstream)
unknown-horizons(filesystem ordering issue)
tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
sbuildexit code. [ ][ ]
php-hordepackages back to the
pkg-php-pearpackage set for the bullseye distribution. [ ]
debrebuild. [ ]
pbuilder[ ], NetBSD [ ], unkillable processes [ ], unresponsive nodes [ ][ ][ ][ ], proxy connection failures [ ], too many installed kernels [ ], etc.
systemdunits. [ ]
init_nodescript to suggest using sudo instead of explicit logout and logins [ ][ ] and the usual build node maintenance was performed by Holger Levsen [ ][ ][ ][ ][ ][ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ][ ].
.doctreesfrom installed files was created via Arch s TODO list mechanism. These
.doctreefiles are caches generated by the Sphinx documentation generator when developing documentation so that Sphinx does not have to reparse all input files across runs. They should not be packaged, especially as they lead to the package being unreproducible as their pickled format contains unreproducible data. Jelle van der Waa and Eli Schwartz submitted various upstream patches to fix projects that install these by default. Dimitry Andric was able to determine why the reproducibility status of FreeBSD s
base.txzdepended on the number of CPU cores, attributing it to an optimisation made to the Clang C compiler [ ]. After further detailed discussion on the FreeBSD bug it was possible to get the binaries reproducible again [ ]. For the GNU Guix operating system, Vagrant Cascadian started a thread about collecting reproducibility metrics and Jan janneke Nieuwenhuizen posted that they had further reduced their bootstrap seed to 25% which is intended to reduce the amount of code to be audited to avoid potential compiler backdoors. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update as well as made the following changes within the distribution itself:
carla(Timestamp in Windows Portable Executable executables)
fonttosfnt/xorg-x11-fonts(Address space layout randomization issue)
gcc10 C++(Link-time optimisation issue)
grep(Profile-guided optimisation issue)
kubernetes1.18(Remove Go build identifier)
stressapptest(Override date, user & host)
reproducible-checktool that reports on the reproducible status of installed packages on a running Debian system. They were subsequently all fixed by Chris Lamb [ ][ ][ ]. Timo R hling filed a wishlist bug against the
debhelperbuild tool impacting the reproducibility status of 100s of packages that use the CMake build system which led to a number of tests and next steps. [ ] Chris Lamb contributed to a conversation regarding the nondeterministic execution of order of Debian maintainer scripts that results in the arbitrary allocation of UNIX group IDs, referencing the Tails operating system s approach this [ ]. Vagrant Cascadian also added to a discussion regarding verification formats for reproducible builds. 47 reviews of Debian packages were added, 37 were updated and 69 were removed this month adding to our knowledge about identified issues. Chris Lamb identified and classified a new
uids_gids_in_tarballs_generated_by_cmake_kde_package_app_templatesissue [ ] and updated the
paths_vary_due_to_usrmerge as deterministicissue, and Vagrant Cascadian updated the
gcc_captures_build_pathissues. [ ][ ][ ]. Lastly, Debian Developer Bill Allombert started a mailing list thread regarding setting the
-fdebug-prefix-mapcommand-line argument via an environment variable and Holger Levsen also filed three bugs against the
debrebuildDebian package rebuilder tool (#961861, #961862 & #961864).
git logexample to another section [ ]. Chris Lamb also limited the number of news posts to avoid showing items from (for example) 2017 [ ]. strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. It is used automatically in most Debian package builds. This month, Mattia Rizzolo bumped the
debhelpercompatibility level to 13 [ ] and adjusted a related dependency to avoid potential circular dependency [ ].
libxml2(random data corruption)
frr(build fails on single-processor machines),
ghc-yesod-static/git-annex(a filesystem ordering issue) and
149to Debian and made the following changes:
jsondiffversion 1.2.0. (#159)
File.recognizesthat checks candidates against
file(1). [ ]
AbstractMissingTypetype instead of remembering to check for both types of missing files. [ ]
.buildinfocomparators. [ ]
f-stringsto tidy up code [ ][ ] and remove explicit
u"unicode"strings [ ].
--new-fileoption when comparing directories by merging
--html-dirpresenter format. [ ]
--html-dirformat. [ ][ ]
tlshfuzzy-matching library during tests [ ] and tweaked the build system to remove an unwanted
.builddirectory [ ]. For the GNU Guix distribution Vagrant Cascadian updated the version of diffoscope to version 147 [ ] and later 148 [ ].
tests.reproducible-builds.org. Amongst many other tasks, this tracks the status of our reproducibility efforts across many distributions as well as identifies any regressions that have been introduced. This month, Holger Levsen made the following changes:
rsync2buildinfos.debian.netevery night. [ ]
.buildinfofiles to include a fix regarding comparing source vs. binary package versions. [ ]
openwrt_rebuilder_futureto known broken jobs. [ ]
<meta>header to refresh the page every 5 minutes. [ ]
fixfilepathon bullseye, to get better data about the
ftbfs_due_to_f-file-prefix-mapcategorised issue. Lastly, the usual build node maintenance was performed by Holger Levsen [ ][ ], Mattia Rizzolo [ ] and Vagrant Cascadian [ ][ ][ ][ ][ ].
This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Eli Schwartz, Holger Levsen, Jelle van der Waa and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.
Welcome to the May 2020 report from the Reproducible Builds project. One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. Nonetheless, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into seemingly secure software during the various compilation and distribution processes. In these reports we outline the most important things that we and the rest of the community have been up to over the past month.
Recent years saw a number of supply chain attacks that leverage the increasing use of open source during software development, which is facilitated by dependency managers that automatically resolve, download and install hundreds of open source packages throughout the software life cycle.In related news, the LineageOS Android distribution announced that a hacker had access to the infrastructure of their servers after exploiting an unpatched vulnerability. Marcin Jachymiak of the Sia decentralised cloud storage platform posted on their blog that their
siadutilities can now be built reproducibly:
This means that anyone can recreate the same binaries produced from our official release process. Now anyone can verify that the release binaries were created using the source code we say they were created from. No single person or computer needs to be trusted when producing the binaries now, which greatly reduces the attack surface for Sia users.Synchronicity is a distributed build system for Rust build artifacts which have been published to crates.io. The goal of Synchronicity is to provide a distributed binary transparency system which is independent of any central operator. The Comparison of Linux distributions article on Wikipedia now features a Reproducible Builds column indicating whether distributions approach and progress towards achieving reproducible builds.
binutilspackage ships its own, unreproducible, log files in its binary packages. It was followed-up by replies from Chris Lamb and Matthias Klose.
.apkpackages. Allan McRae of the ArchLinux project posted their third Reproducible builds progress report to the
arch-dev-publicmailing list which includes the following call for help:
We also need help to investigate and fix the packages that fail to reproduce that we have not investigated as of yet.In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update.
146to Debian, PyPI, etc.
filenow supports recognising JSON data. (#106)
.buildinfohandling to show all details (including the GnuPG header and footer components) even when referenced files are not present. (#122)
BuildinfoFilecomparator (etc.) regardless of whether the associated files (such as the
.deb) are present. [ ]
.changes, etc. [ ]
differencestypo in the
id="foo"anchor reference twice in the HTML output, otherwise identically-named parts will not be able to linked to via a
#. [ ]
--jsonpresenter; it will usually be too complicated to be readable by the human anyway. [ ]
Command [ ] failed with exit codemessages to remove duplicate
exited with exitbut also to note that
diffoscopeis interpreting this as an error. [ ]
Command [ ] exited with 1messages. (#126)
debianPython module. [ ]
stderr fromif both commands emit the same output. [ ]
apksignertest failures due to lack of
binfmt_misc, eg. on Salsa CI and elsewhere. [ ]
.travis.ymlas we use Salsa instead. [ ]
.dockerignorefile to whitelist files we actually need in our container. (#105)
ENVwhen setting up the
DEBIAN_FRONTENDenvironment variable at runtime. (#103)
build-essentialduring build so we can install the recommended packages from Git. [ ]
shell=Falsekeyword argument to
subprocess.Popenso that the potentially-unsafe
shell=Trueis more obvious. [ ]
MissingFiles special handling of
deb822to prevent leaking through abstract layers. [ ][ ]
exceptblock when cleaning up temporary files with respect to the
flake8quality assurance tool. [ ]
dsc_in_same_dirto clarify the use of this variable. [ ]
debian_fallbackclass [ ] and add descriptions for the file types. [ ]
Opensslcommand class to
OpenSSLPKCS7to accommodate other command names with this prefix. [ ]
--debuggercommand-line argument to
--pdb. [ ]
stat(2)birth times (ie.
st_birthtime) in the same way we do with the
Change:times to fix a nondeterministic build failure in GNU Guix. (#74)
has_same_contentmethod was called regardless of the underlying type of file. [ ]
debian/py3dist-overridesto ensure the
rpm-pythonmodule is used in package dependencies (#89) and moved to using the new
execute_before_*Debhelper rules [ ].
relative_urlwhere possible [ ][ ] and move a number of configuration variables to
_config.yml[ ][ ].
golang-packaging(toolchain issue, affecting times in
jboss-logging-tools(toolchain issue, affecting date for
findoutput to avoid inheriting filesystem order)
moonjit(generate reproducible output by default if
vala(report ASLR nondeterminism)
1.8.1-1to Debian unstable and Bernhard M. Wiedemann fixed an off-by-one error when parsing PNG image modification times. (#16) In disorderfs, our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues, Chris Lamb replaced the term dirents in place of directory entries in human-readable output/log messages [ ] and used the astyle source code formatter with the default settings to the main
disorderfs.cppsource file [ ]. Holger Levsen bumped the
debhelper-compat levelto 13 in disorderfs [ ] and reprotest [ ], and for the GNU Guix distribution Vagrant Cascadian updated the versions of disorderfs to version 0.5.10 [ ] and diffoscope to version 145 [ ].
libtool. [ ]
_docssubdirectory to find the
_docs/index.mdfile after an internal move. (#27)
ltmain.shetc. in preformatted quotes. [ ]
SOURCE_DATE_EPOCHPython examples onto more lines to prevent visual overflow on the page. [ ]
tests.reproducible-builds.orgthat, amongst many other tasks, tracks the status of our reproducibility efforts as well as identifies any regressions that have been introduced. Holger Levsen made the following changes:
let VARIABLE=0exits with an error. [ ]
.buildinfofiles with the same name. [ ]
/usrmerge variation on Debian unstable. [ ]
molly-guard. [ ]
debrebuildscript. [ ][ ][ ][ ]
.buildinfofiles. [ ][ ]
alpine_schroot.shscript now that a patch for
abuildhad been released upstream. [ ]
bcm47xx. [ ]
jenkinsto run the
blacklistcommand [ ] and the usual build node maintenance was performed was performed by Holger Levsen [ ][ ][ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ].
To make the results accessible, storable and create tools around them, they should all follow the same schema, a reproducible builds verification format. The format tries to be as generic as possible to cover all open source projects offering precompiled source code. It stores the rebuilder results of what is reproducible and what not.Hans-Christoph Steiner of the Guardian Project also continued his previous discussion regarding making our website translatable. Lastly, Leo Wandersleb posted a detailed request for feedback on a question of supply chain security and other issues of software review; Leo is the founder of the Wallet Scrutiny project which aims to prove the security of Android Bitcoin Wallets:
Do you own your Bitcoins or do you trust that your app allows you to use your coins while they are actually controlled by them ? Do you have a backup? Do they have a copy they didn t tell you about? Did anybody check the wallet for deliberate backdoors or vulnerabilities? Could anybody check the wallet for those?Elsewhere, Leo had posted instructions on his attempts to reproduce the binaries for the BlueWallet Bitcoin wallet for iOS and Android platforms.
This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Jelle van der Waa and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.
Welcome to the April 2020 report from the Reproducible Builds project. In our regular reports we outline the most important things that we and the rest of the community have been up to over the past month. What are reproducible builds? One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. But whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into seemingly secure software during the various compilation and distribution processes.
atlas-clientin place of
atlas_client) that executed a script that intercepted Bitcoin payments. (Ars Technica report) Bernhard M. Wiedemann launched
ismypackagereproducibleyet.org, a service that takes a package name as input and displays whether the package is reproducible in a number of distributions. For example, it can quickly show the status of Perl as being reproducible on openSUSE but not in Debian. Bernhard also improved the documentation of his unreproducible package to add some example patches for hash issues. [ ]. There was a post on Chaos Computer Club s website listing Ten requirements for the evaluation of Contact Tracing apps in relation to the SARS-CoV-2 epidemic. In particular:
4. Transparency and verifiability: The complete source code for the app and infrastructure must be freely available without access restrictions to allow audits by all interested parties. Reproducible build techniques must be used to ensure that users can verify that the app they download has been built from the audited source code.Elsewhere, Nicolas Boulenguez wrote a patch for the Ada programming language component of the GCC compiler to skip
-f.*-prefix-mapoptions when writing Ada Library Information files. Amongst other properties, these
.alifiles embed the compiler flags used at the time of the build which results in the absolute build path being recorded via
-fdebug-prefix-map, etc. In the Arch Linux project, kpcyrd reported that they held their first rebuilder workshop . The session was held on IRC and participants were provided a document with instructions on how to install and use Arch s
reprotool. The meeting resulted in multiple people with no prior experience of Reproducible Builds validate their first package. Later in the month he also announced that it was now possible to run independent rebuilders under Arch in a hands-off, everything just works solution to distributed package verification. Mathias Lang submitted a pull request against
dmd, the canonical compiler for the D programming languageto add support for our
SOURCE_DATE_EPOCHenvironment variable as well the other C preprocessor tokens such
__TIMESTAMP__which was subsequently merged.
SOURCE_DATE_EPOCHdefines a distribution-agnostic standard for build toolchains to consume and emit timestamps in situations where they are deemed to be necessary. [ ] The Telegram instant-messaging platform announced that they had updated to version 5.1.1 continuing their claim that they are reproducible according to their full instructions and therefore verifying that its original source code is exactly the same code that is used to build the versions available on the Apple App Store and Google Play distribution platforms respectfully. Lastly, Herv Boutemy reported that 97% of the current development versions of various Maven packages appear to have a reproducible build. [ ]
debrebuild, a tool for rebuilding a Debian package given a
.buildinfofile, proposing to add
readdircall, rejected upstream)
guile/guix(parallelism race condition)
readdir, filesystem, toolchain)
readdir, filesystem, toolchain)
OBS(FTBFS in rebuild)
perl-Image-Sane(report hung build on a single core VM)
ruby2.7(date, already upstream)
[core]repository directly. The first rebuild has led to approximately 90% packages reproducible contrasting with 94% on the Reproducible Build s project own ArchLinux status page on
tests.reproducible-builds.orgthat continiously builds packages and does not verify Arch Linux packages. More information may be found on the corresponding wiki page and the underlying decisions were explained on our mailing list.
143to Debian which were subsequently uploaded to the backports repository):
.dexfiles can also serve as APK containers so restrict the narrower identification of
.dexfiles to files ending with this extension and widen the identification of APK files to when file(1) discovers a Dalvik file. (#28)
pdftotextas a requirement to run the PDF
debian/tests/control.into ensure that we have this module installed during a test run to generate the fixtures in these tests. [ ]
./setup.py test --pytest-argsarguments. [ ]
zipinfodirectly instead of piping input via
/dev/stdinin order to ensure portability to the BSD operating system [ ]. In addition, Ben Hutchings documented how
--excludearguments are matched against filenames [ ] and Jelle van der Waa updated the LLVM test fixture difference for LLVM version 10 [ ] as well as adding a reference to the name of the
h5dumptool in Arch Linux [ ]. Lastly, Mattia Rizzolo also fixed in incorrect build dependency [ ] and Vagrant Cascadian enabled diffoscope to locate the
h5dumppackages on GNU Guix [ ][ ], and updated diffoscope in GNU Guix to version 141 [ ] and 143 [ ].
*.symfiles as Java archives. (#15)
.zipfilename filtering and exclude two patterns of files generated by Maven projects in fork mode. (#13)
0.5.9-1to Debian unstable. Vagrant Cascadian subsequently refreshed disorderfs in GNU Guix to version 0.5.9 [ ].
acoular(report unknown non-determinism)
cri-o(report a date issue)
certtoolbeing unable to extend certificates beyond 2049)
gnutls(report copyright year variation)
libxslt(report a bug about non-deterministic output from data corruption)
python-astropy(report a future build failure in 2021)
rb-generalmailing list is probably a first step for contributors to take. [ ]
jekyll-redirect-fromover manual redirect pages [ ][ ] and add a redirect from
/news/namespace [ ] and improve formatting of archived news links [ ].
tests.reproducible-builds.orgthat, amongst many other tasks, tracks the status of our reproducibility efforts as well as identifies any regressions that have been introduced.
disorderfs-debugprefix in log output when we change non-disorderfs things in the file and, as it happens, do not run disorderfs at all. [ ]
<a>HTML elements under
<li>ones, which was causing a comma/bullet spacing issue. [ ]
arm64architecture, etc.. [ ][ ]
debrebuildto version from the latest version of
devscripts. [ ][ ]
sbuild. [ ][ ][ ][ ][ ]
.buildinfofile and attempt to build and compare the result. [ ][ ][ ][ ]
disorderfspackage did not pass its GPG verification which was also fixed by Chris Lamb. Hans-Christoph Steiner of the Guardian Project asked whether there would be interest in making our website translatable which resulted in a WIP merge request being filed against the website and a discussion on how to track translation updates.
This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Daniel Shahaf, Holger Levsen, Jelle van der Waa, kpcyrd, Mattia Rizzolo and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.
.jarbuild artefact is reproducible across builds. A practical and hands-on guide, it details how to avoid unnecessary differences between builds by explicitly declaring an encoding as the default value differs across Linux and MS Windows systems and ensuring that the generated
.jara variant of a
.ziparchive does not embed any nondeterministic filesystem metadata, and so on. Janneke gave a quick presentation on GNU Mes and reproducible builds during the lighting talk session at LibrePlanet 2020. [ ] Vagrant Cascadian presented There and Back Again, Reproducibly! video at SCaLE 18x in Pasadena in California which generated some attention on Twitter. Herv Boutemy mentioned on our mailing list in a thread titled Rebuilding and checking Reproducible Builds from Maven Central repository that since the update of a central build script (the parent POM ) every Apache project using the Maven build system should build reproducibly. A follow-up discussion regarding how to perform such rebuilds was also started on the Apache mailing list. The Telegram instant-messaging platform announced that they had updated their iOS and Android OS applications and claim that they are reproducible according to their full instructions, verifying that its original source code is exactly the same code that is used to build the versions available on the Apple App Store and Google Play distribution platforms respectfully. Herv Boutemy also reported about a new project called
reproducible-centralwhich aims to allow anyone to rebuild a component from the Maven Central Repository that is expected to be reproducible and check that the result is as expected. In last month s report we detailed Omar Navarro Leija s work in and around an academic paper titled Reproducible Containers which describes in detail the workings of a user-space container tool called
dettrace(PDF). Since then, the PhD student from the University Of Pennsylvania presented on this tool at the ASPLOS 2020 conference in Lausanne, Switzerland. Furthermore, there were contributions to
dettracefrom the Reproducible Builds community itself. [ ][ ]
avfs(report build problem in
arj(fix incorrect use of
strcpy, submitted upstream)
brickv(update get upstream fix)
fvwm-themes(delta between architectures in
libpeas(report build failure in single-CPU mode)
pmix(update to incoporate upstream fix)
pw3270(date variation, forwarded upstream)
python-mailmanclient(report build failure in single-CPU mode)
ripgrep(CPU, forwarded upstream)
tensorflow2(avoid random temporary directory path)
tesseract-ocr(drop native architecture optimisations)
vlc(fixed ghost file size and sort archive, already upstream)
debian-installercomponent to allow all arguments from
sources.listfiles (such as
[check-valid-until=no]) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure. (#13) Holger Levsen filed a number of bug reports against the
debrebuildtool that attempts to rebuild a Debian package given a
.buildinfofile as input, including:
nondeterministic_vo_files_generated_by_coq[ ] utput
buildinfos.debian.net(#955434) and Chris Lamb kept isdebianreproducibleyet.com up to date. [ ]
python3-pdfminerif we do not see any other differences from
pdftext, etc. (#92)
.rdxfiles directly as the
get_membermethod will return a file even if the file is missing. [ ]
--helpoutput or in the package long description. [ ]
--list-debian-substvarswhen we want them for
debian/tests/controlgeneration. [ ]
upstream-metadata-in-native-sourceas we are upstream. [ ]
RequiredToolNotFound.get_packagemethod s functionality as it is only used once. [ ]
py36 = [..]argument in the
pyproject.tomlfile. [ ]
calendar.monthrangePython method in a utility function. [ ]
tests.reproducible-builds.orgthat, amongst many other tasks, tracks the status of our reproducibility efforts as well as identifies any regressions that have been introduced. This month, Chris Lamb reworked the web-based package rescheduling tool to:
POSTmethod in the web-based scheduler as not only should HTTP GET requests be idempotent but this will allow many future improvements in the user interface. [ ][ ][ ]
ath97subtarget for the OpenWrt distribution.
i386architecture. [ ][ ]
This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.
Welcome to the February 2020 report from the Reproducible Builds project. One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the reproducible builds effort is to provide the ability to demonstrate these binaries originated from a particular, trusted, source release: if identical results are generated from a given source in all circumstances, reproducible builds provides the means for multiple third-parties to reach a consensus on whether a build was compromised via distributed checksum validation or some other scheme. In this month s report, we cover:
If you are interested in contributing to the project, please visit our Contribute page on our website.
All computation that occurs inside a DetTrace container is a pure function of the initial filesystem state of the container. Reproducible containers can be used for a variety of purposes, including replication for fault-tolerance, reproducible software builds and reproducible data analytics. We use DetTrace to achieve, in an automatic fashion, reproducibility for 12,130 Debian package builds, containing over 800 million lines of code, as well as bioinformatics and machine learning workflows.There was also considerable discussion on our mailing list regarding this research and a presentation based on the paper will occur at the ASPLOS 2020 conference between March 16th 20th in Lausanne, Switzerland. The many virtues of Reproducible Builds were touted as benefits for software compliance in a talk at FOSDEM 2020, debating whether the Careful Inventory of Licensing Bill of Materials Have Impact of FOSS License Compliance which pitted Jeff McAffer and Carol Smith against Bradley Kuhn and Max Sills. (~47 minutes in). Nobuyoshi Nakada updated the canonical implementation of the Ruby programming language a change such that filesystem globs (ie. calls to list the contents of filesystem directories) will henceforth be sorted in ascending order. Without this change, the underlying nondeterministic ordering of the filesystem is exposed to the language which often results in an unreproducible build. Vagrant Cascadian reported on our mailing list regarding a quick reproducible test for the GNU Guix distribution, which resulted in 81.9% of packages registering as reproducible in his installation:
Jeremiah Orians announced on our mailing list the release of a number of tools related to cross-compilation such as
$ guix challenge --verbose --diff=diffoscope ... 2,463 store items were analyzed: - 2,016 (81.9%) were identical - 37 (1.5%) differed - 410 (16.6%) were inconclusive
mescc-tools-seed. This project attemps a full bootstrap of a cross-platform compiler for the C programming language (written in C itself) from hex, the ultimate goal being able to demonstrate fully-bootstrapped compiler from hex to the GCC GNU Compiler Collection. This has many implications in and around Ken Thompson s Trusting Trust attack outlined in Thompson s 1983 Turing Award Lecture. Twitter user @TheYoctoJester posted an executive summary of reproducible builds in the Yocto Project: Finally, Reddit user
tofflosposted to the /r/Java subreddit asking about how to achieve reproducible builds with Maven and Chris Lamb noticed that the Linux kernel documentation about reproducible builds of it is available on the kernel.org homepages in an attractive HTML format.
debian-installerpackage to allow all arguments and options from
sources.listfiles (such as
[check-valid-until=no], etc.) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure. (#13) Thorsten Glaser followed-up to a bug filed against the
dpkg-sourcecomponent that was originally filed in late 2015 that claims that the build tool does not respect permissions when unpacking tarballs if the umask is set to
0002. Matthew Garrett posted to the
debian-develmailing list on the topic of Producing verifiable initramfs images as part of a wider conversation on being able to trust the entire software stack on our computers. 59 reviews of Debian packages were added, 30 were updated and 42 were removed this month adding to our knowledge about identified issues. Many issue types were noticed and categorised by Chris Lamb, including:
python-rpm-macros(do not save time-based
.pycfiles for tests)
solfege(filesystem ordering issue sent upstream via email; package is orphaned upstream)
DVDStyler(zip timestamps, submitted upstream)
sngimage utility appears to return with an exit code of 1 if there are even minor errors in the file. (#950806)
.apkfiles extracted by
str.formatif we are just returning the string. [ ]
Command.VALID_RETURNCODES. [ ]
Vcs-Gitto specify the
debianpackaging branch. [ ] reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, versions
0.7.14were uploaded to Debian unstable by Holger Levsen after Vagrant Cascadian added support for GNU Guix [ ].
SOURCE_DATE_EPOCHdocumentation [ ] and normalised various terms to unreproducible [ ]. Chris Lamb added a Meson.build example [ ] and improved the documentation for the CMake [ ] to the
SOURCE_DATE_EPOCHdocumentation, replaced anyone can with anyone may as, well, not everyone has the resources, skills, time or funding to actually do what it refers to [ ] and improved the pre-processing for our report generation [ ][ ][ ][ ] etc. In addition, Holger Levsen updated our news page to improve the list of reports [ ], added an explicit mention of the weekly news time span [ ] and reverted sorting of news entries to have latest on top [ ] and Mattia Rizzolo added Codethink as a non-fiscal sponsor [ ] and lastly Tianon Gravi added a Docker Images link underneath the Debian project on our Projects page [ ].
python-django(Always build the documentation in English)
.buildinfofiles. This has resulted in so that we now know that Debian bullseye contains 4,557 source packages for the
amd64architecture without corresponding
.buildinfofiles and 25,668 source packages with
.buildinfofiles. [ ][ ][ ][ ][ ]
rootuser. [ ]
devscriptsfrom buster-backports [ ].
i386architecture, to allow the latter to catch up a bit. [ ]
arm64architecture builders [ ]. The usual build node maintenance was performed by Holger Levsen, Mattia Rizzolo [ ][ ] and Vagrant Cascadian.
This month s report was written by Bernhard M. Wiedemann, Chris Lamb and Holger Levsen. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.
--auto-buildoption to try to determine which specific variations cause unreproducibility.
--source-patternoption to restrict copying of
source_root, and set this automatically in our presets.
build_commandthat doesn't exist
user_groupat the same time.
dpkg-sourceextract to different build dir if varying the build-path.
diffoscope(1)by default as this is the majority use-case.
odxu4aafter being reinstalled and renamed from
--env-buildoption for testing different env vars. (In-progress, requires the
python-rstrpackage awaiting entry into Debian.)
--source-patternoption to restrict copying of
dpkg-sourceextract to different build dir iff varying the build-path.
--debugto diffoscope if verbosity >= 2.
--exclude-directory-metadatato diffoscope(1) by default.
armhfbuild node that had been disabled due to performance issues, but works linux 4.14-rc1 now! #876212
__init__.pyand remove obsolete earlier code.
--variationsflag to support parameters to certain variations like
user_group, and document examples in README.
--varyflag for the new syntax and deprecate
artifact_patternto avoid arbitrary shell execution.
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.
the#debian-reproducible-changes IRC channel for unreproducible -> FTBFS transitions.
squid.conffor all nodes to 5.2.23 (and fixup some).
arm64nodes as well.
jenkins.debian.org, which affects tests.r-b.o as well.
--column-inserts's pg_dump option.
armhfbuild machines to stretch.
git log -1 > .htmlto node document environment().
postgres-9.4from jenkins, so we could test our backups