Here's what happened in the Reproducible Builds effort between Sunday February 5 and Saturday February 11 2017: Upcoming events

• Holger proposed a hackathon with various possible dates -- please reply with your preferred dates!
• "Reproducible builds: Status update" CfP submitted for Debconf17 in Montreal.

Patches sent upstream

• pnmixer (Chris Lamb)
• cloud-sptheme (Chris Lamb)
• python-hypothesis (Chris Lamb)
• cython (Jelmer Vernoo )

Packages reviewed and fixed, and bugs filed Chris Lamb:

• #854332 filed against cloud-sptheme.
• #854512 filed against ftpcopy.
• #854549 filed against python-hypothesis.

Daniel Shahaf:

• #854492 filed against xlsx2csv.
• #854541 filed against sogo.

"Z. Ren":

• #854293 filed against manpages-tr.
• #854294 filed against regina-rexx.
• #854362 filed against fonts-uralic.

Reviews of unreproducible packages 83 package reviews have been added, 8 have been updated and 32 have been removed in this week, adding to our knowledge about identified issues. 5 issue types have been added:

• randomness_in_swf_files_generated_by_as3compile
• randomness_in_t3g_files_generated_tslmendian
• absolute_build_paths_in_dot_packlist_file_generated_by_perl_extutils_packlist
• formatdb_from_ncbi_blastplus_captures_build_time
• timestamp_and_build_path_captured_by_python_cheetah

1 issue type has been updated:

• build_id_differences_only

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (7)
• gregory bahde (1)

diffoscope development diffoscope versions 71, 72, 73, 74 & 75 were uploaded to unstable by Chris Lamb:

• Chris Lamb:
  • New features:
    • Add --exclude option. (Closes: #854783)
    • Apply --max-report-size to --text reports. (Closes: #851147)
    • Add a machine-readable JSON output format. (Closes: #850791)
    • Show results from debugging packages last (Closes: #820427)
    • Specify lang="en" in HTML output. (re. #849411)
  • Bug fixes:
    • Fix errors when comparing directories with non-directories. (Closes: #835641)
    • Clean all temp files in signal handler thread instead of attempting to bubble exception back to the main thread. (Closes: #852013)
    • Correct logic of module_exists, ensuring we correctly skip tests when python3-debian is not installed. (Closes: #854745)
    • Extract archive members using an auto-incrementing integer, avoiding the need to sanitise filenames. (Closes: #854723)
    • Importing submodules (ie. parent.child) will attempt to import parent so we must catch that. (Closes: #854670)
    • Add missing Recommends for comparators. (Closes: #854655)
    • Device and RPM fallback comparisons needs xxd due to fixtures. (Closes: #854593)
    • Fix behaviour of setting report maximums to zero (ie. no limits)
  • Misc:
    • Don't uselessly run xxd(1) on non-directories.
    • Add .travis.yml from http://travis.debian.net.
    • Use diffoscope.tempfiles over tempfile.TemporaryDirectory to ensure correct cleanup at end of diffoscope run.
  • Tests:
    • Smoke test --progress output.
    • Test the --status-fd output.
    • Move many tests to use new @skip_unless_module_exists decorator.
    • Show local variables in pytest tracebacks.
• Mattia Rizzolo:
  • No need to do complex string formatting just to convert an integer in a string
• Holger Levsen:
  • README: Keep history, explain this was started in Debian.
• Ximin Luo:
  • Better way of performing the entry name sanitisation
  • Simplify call to subprocess.Popen
  • Remove pointless use of a thread
• Brett Smith:
  • diffoscope.diff: Improve FIFO writing robustness.

strip-nondeterminism development strip-nondeterminism 0.030-1 was uploaded to unstable by Chris Lamb:

• Print log entry when fixing a file. (Closes: #777239)
• dh_strip_nondeterminism: Use error() from Dh_Lib.pm over manual die().

buildinfo.debian.net development

• Chris Lamb:
  • Drop raw_text fields now; we've moved them to default_storage (S3)

reproducible-website development

• Joshua Lock:
  • Link to Yocto Project on projects page

Misc. This week's edition was written by Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday January 29 and Saturday February 4 2017: Media coverage Dennis Gilmore and Holger Levsen presented "Reproducible Builds and Fedora" (Video, Slides) at Devconf.cz on February 27th 2017. On February 1st, stretch/armhf reached 90% reproducible packages in our test framework, so that now all four tested architectures are 90% reproducible in stretch. Yay! For armhf this means 22472 reproducible source packages (in main); for amd64, arm64 and i386 these figures are 23363, 23062 and 22607 respectively. Chris Lamb appeared on the Changelog podcast to talk about reproducible builds: Holger Levsen pitched Reproducible Builds and our need for a logo in the "Open Source Design" room at FOSDEM 2017 (Video, 09:36 - 12:00). Upcoming Events

• The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd.
• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.
• Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Reproducible work in other projects We learned that the "slightly more secure" Heads firmware (a Coreboot payload) is now reproducibly built regardless of host system or build directory. A picture says more than a thousand words: Docker started preliminary work on making image builds reproducible. Toolchain development and fixes Ximin Luo continued to write code and test cases for the BUILD_PATH_PREFIX_MAP environment variable. He also did extensive research on cross-platform and cross-language issues with enviroment variables, filesystem paths, and character encodings, and started preparing a draft specification document to describe all of this. Chris Lamb asked CPython to implement an environment variable PYTHONREVERSEDICTKEYORDER to add an an option to reverse iteration order of items in a dict. However this was rejected because they are planning to formally fix this order in the next language version. Bernhard Wiedemann and Florian Festi added support for our SOURCE_DATE_EPOCH environment variable, to the RPM Package Manager. James McCoy uploaded devscripts 2.17.1 with a change from Guillem Jover for dscverify(1), adding support for .buildinfo files. (Closes: #852801) Piotr O arowski uploaded dh-python 2.20170125 with a change from Chris Lamb for a patch to fix #835805. Chris Lamb added documentation to diffoscope, strip-nondeterminism, disorderfs, reprotest and trydiffoscope about uploading signed tarballs when releasing. He also added a link to these on our website's tools page. Packages reviewed and bugs filed Bugs filed:

• "Z. Ren":
  • #854293 filed against manpages-tr.
  • #854294 filed against regina-rexx.
• Chris Lamb:
  • #853039 filed against fontypython.
  • #853912 filed against python-testfixtures, merged upstream as PR #56.
  • #854111 filed against aprx.
  • #854112 filed against pnmixer.
• Reiner Herrmann:
  • #854145 filed against daemontools.
  • #854146 filed against diploma.

Reviews of unreproducible packages 83 package reviews have been added, 86 have been updated and 276 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• captures_build_path_via_assert
• randomness_in_documentation_generated_by_epydoc

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (6)

diffoscope development Work on the next version (71) continued in git this week:

• Mattia Rizzolo:
  • Override a lintian warning.
• Chris Lamb:
  • Update and consolidate documentation
  • Many test additions and improvements
  • Various code quality and software architecture improvements
• anthraxx:
  • Update arch package, cdrkit -> cdrtools.

reproducible-website development Daniel Shahaf added more notes on our "How to chair a meeting" document. tests.reproducible-builds.org Holger unblacklisted pspp and tiledarray. If you think further packages should also be unblacklisted (possibly only on some architectures), please tell us. Misc. This week's edition was written by Ximin Luo, Holger Levsen and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday January 22 and Saturday January 28 2017: Media coverage

• The Reproducible Build Zoo was presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon on February 22nd.
• Dennis Gilmore and Holger Levsen presented "Reproducible Builds and Fedora" at Devconf.cz on February 27th.

Upcoming Events

• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.
• "Verifying Software Freedom with Reproducible Builds" will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Reproducible work in other projects John Gilmore wrote an interesting mail about how Cygnus.com worked on reproducible builds in the early 1990s. It's eye opening to see how the dealt with basically the very same problems we're dealing with today, how they solved them and then to realize that most of this has been forgotten and bit-rotted in the last 20 years. How will we prevent history repeating itself here? Toolchain development and fixes Christoph Biedl wrote a mail describing an interesting problem in to the way binNMUs are done in Debian. Guillem Jover made a number of changes to dpkg that affect the Reproducible Builds effort within Debian:

• Always set SOURCE_DATE_EPOCH in dpkg-buildpackage and dpkg-source. Use the current date if the changelog does not have one. Closes: #849081
• Add initial support for DEB_BUILD_OPTIONS to dpkg-genbuildinfo. This will make it possible to enable or disable specific features that should be recorded in the .buildinfo file. For now only all and path are supported. Closes: #848705
• Include .buildinfo files also for source-only uploads in dpkg-genchanges. Closes: #846164
• Add support for signed .buildinfo files to dpkg-buildpackage. Add new -ui and --unsigned-buildinfo options. Closes: #843925
• Make dpkg-buildpackage --unsigned-changes not sign .buildinfo either. This breaks the expectations of users and tools, because there was no way previously to request no signing at all. Closes: #852822

Packages reviewed and fixed, and bugs filed Chris Lamb:

• #852482 filed against flask-limiter.
• #853039 filed against fontypython.

Dhole:

• #852289 filed against python-passlib.

Reviews of unreproducible packages 17 package reviews have been added, 4 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• python_versioneer_uses_parentdir
• captures_kernel_version_via_CMAKE_SYSTEM

1 issue type has been removed:

• ftbfs_due_to_jenkins_semaphore_setup

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (6)
• Holger Levsen (1)

diffoscope development

• diffoscope 70 was uploaded to unstable by Mattia Rizzolo. It included contributions from:
• Chris Lamb:
  • tests.presenters: Prevent FTBFS by loading fixtures as UTF-8 in case surrounding terminal is not Unicode-aware. (Closes: #852926)
  • comparators: Tidy re_tests with list comprehensions and implicit "x, y" unpacking over indexing; lambda/filter is not idiomatic Python 3.
  • tests: Increase coverage by adding "# noqa" in relevant parts.
  • tests: Test that no arguments (beyond the filenames) prints the text output.
  • tests: Test --text-color output format.
  • tests: Add a test comparing two empty directories.
  • tests: Don't warn about coverage lines that raise NotImplementedError.
• anthraxx:
  • tools: switch Arch Linux dependency for pedump to mono
• Ximin Luo:
  • Fix lazy expression, filter is lazy in Python 3
  • Fix bug introduced in commit 36d1c964 that only worked "accidentally"

reprotest development

• reprotest 0.6 was uploaded to unstable by Holger Levsen. It included contributions:
• Ximin Luo:
  • Test the extra variations we added recently and ensure they don't get missed in the future
  • Better logging messages that actually get controlled by the verbosity flag
  • Add a man page using rst2man and help2man
  • Add a --config-file option and fix the loading of configs
  • Fix the reading of config options
  • Fix a bug where the sha256sum of a reproduction won't be displayed if --store-dir is not given

buildinfo.debian.net development

• Chris Lamb:
  • Show SHA256 checksums on larger displays
  • Move default storage to S3.
  • Store files directly onto S3.
  • Add tool to move data to S3.
  • Load data from S3 if we don't have it locally.

tests.reproducible-builds.org

• h01ger experimented with reusing SSH control connections but stopped that experiment when we ran into more network issues than before. To be continued, as we're having 10k SSH connections per day and saving 2 seconds each time would sum up, especially on the Jenkins host itself.
• h01ger made the scheduler run 3 times a day, 2.5h after dinstall runs, instead of every 3h as before.
• h01ger restructured the https://tests.reproducible-builds.org/debian/index_breakages.html and improved the corresponding Jenins job.
• h01ger also unblacklisted xmds2, sofia-sip and ck - if you think other packages should be unblacklisted (maybe only on some architectures), please do tell us.

Misc. This week's edition was written by Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday January 15 and Saturday January 21 2017: Media Coverage

• Valerie Young presented Reproducible Builds for a Better Future (video) at linux.conf.au 2017.
• Chris Lamb presented Reproducible Builds: Two years in the trenches (video) at linux.conf.au 2017.

Upcoming Events

• The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd.
• Dennis Gilmore and Holger Levsen will present on "Reproducible Builds and Fedora" at Devconf.cz on February 27th.
• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.
• Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Toolchain development and fixes Ximin Luo continued work on data formats, code, and test cases for SOURCE_PREFIX_MAP. He also continued to talk with the rustc team on the topic. Chris Lamb submitted a patch to implement SOURCE_DATE_EPOCH for wordwarvi, a game which gave extra points to people who built it from source within one hour. This fixes Debian #786593. Launchpad bug 1657704 was filed for them to start accepting buildinfo files. Bugs filed

• Chris Lamb:
  • #851809 filed against mono.
• Daniel Shahaf:
  • #851764 filed against hhsuite.

Reviews of unreproducible packages 10 package reviews have been added, 149 have been updated and 153 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• ftbfs_due_to_jenkins_semaphore_setup
• clilibs_line_order

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (3)
• Ond ej Kobli ek (1)

diffoscope development diffoscope 69 was uploaded to unstable by Chris Lamb. It included contributions from:

• Maria Glukhova:
  • Bug fixes: #850055, #850501.
  • Improve comparison of APK files: #850502.
  • Add comparison of image metadata: #849395.
• Chris Lamb:
  • Many bug fixes: #849411, #850485, #850807, #850850, #851588.
  • Add comparison of ICO files: #850730.
  • Optimisations: disable profiling unless needed, and pre-compile regexes.
  • Many code quality improvements.
• Mattia Rizzolo:
  • Deduplicate code for recognising file types based on RE_FILE_TYPE and RE_FILE_EXTENSION.
  • Improve code quality in tests.

Further development continued in Git, and will be released as version 70 next week:

• Chris Lamb:
  • Add tests for --html-dir output and improve code quality elsewhere in tests.
  • Add markdown and reStructuredText output, as well as tests for these.
  • Improve software architecture of presenters.
  • Fix error-checking in the Haskell comparator.
• James Clarke:
  • Haskell comparator: properly extract version from interface files.
• Mattia Rizzolo:
  • Improve some documentation.
• Brett Smith:
  • Improve documentation including --help output.

reproducible-builds.org website development

• Brett Smith:
  • berlin2016: List Conservancy consistently as a participant.
• Chris Lamb:
  • Add Valerie's talk to resources page.
• Daniel Shahaf:
  • Improved the "How to chair a meeting" section.

tests.reproducible-builds.org

• Holger added arm64 to https://tests.reproducible-builds.org/debian/index_variations.html
• Mattia improved our process for building the performance page so that stats for new architectures are computed correctly without manual intervention.
• Holger enhanced the build node maintenance scripts to correctly detect if /dev/shm is mounted incorrectly (due to #851427) and deployed an /etc/rc.local startup script to all systems which works around it. As a result, jenkins_semaphore_setup_issue should be obsolete.
• Mattia improved the diskspace monitoring visible at our munin page for the 44 nodes we're currently running.
• Holger added 6GB more RAM to jenkins.debian.net, for a total of 64GB RAM, to better cope with the new jobs due to arm64. As usual, thanks to profitbricks.com for the hardware resources enabling this work.

Misc. This week's edition was written by Ximin Luo, Vagrant Cascadian, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday January 8 and Saturday January 14 2017: Upcoming Events

• The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd
• Dennis Gilmore and Holger Levsen will present about "Reproducible Builds and Fedora" at Devconf.cz on February, 27th.
• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th

Reproducible work in other projects Reproducible Builds have been mentioned in the FSF high-priority project list. The F-Droid Verification Server has been launched. It rebuilds apps from source that were built by f-droid.org and checks that the results match. Bernhard M. Wiedemann did some more work on reproducibility for openSUSE. Bootstrappable.org (unfortunately no HTTPS yet) was launched after the initial work was started at our recent summit in Berlin. This is another topic related to reproducible builds and both will be needed in order to perform "Diverse Double Compilation" in practice in the future. Toolchain development and fixes Ximin Luo researched data formats for SOURCE_PREFIX_MAP and explored different options for encoding a map data structure in a single environment variable. He also continued to talk with the rustc team on the topic. Daniel Shahaf filed #851225 ('udd: patches: index by DEP-3 "Forwarded" status') to make it easier to track our patches. Chris Lamb forwarded #849972 upstream to yard, a Ruby documentation generator. Upstream has fixed the issue as of release 0.9.6. Alexander Couzens (lynxis) has made mksquashfs reproducible and is looking for testers. It compiles on BSD systems such as FreeBSD, OpenBSD and NetBSD. Bugs filed Chris Lamb:

• #850683 filed against gerbv.

Lucas Nussbaum:

• #850973 filed against shogun.

Nicola Corna:

• #851190 filed against gerbv.

Reviews of unreproducible packages 13 package reviews have been added and 13 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been added:

• help2man_puts_traceback_in_generated_man_page

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (3)
• Lucas Nussbaum (11)
• Nicola Corna (1)

diffoscope development Many bugs were opened in diffoscope during the past few weeks, which probably is a good sign as it shows that diffoscope is much more widely used than a year ago. We have been working hard to squash many of them in time for Debian stable, though we will see how that goes in the end

• Mattia Rizzolo:
  • Code quality and style improvements.
• Maria Glukhova:
  • Add image metadata comparison.
  • Zipinfo included in APK files comparison. (Closes: #850502)
  • Remove archive name from apktool.yml and rename it. (Closes: #850501)
• Chris Lamb:
  • Correctly escape value of href="" elements (re. #849411)
  • Support comparing .ico files using img2txt (Closes: #850730) and fixes and extra tests in subsequent commits.
  • comparators.utils.file: Include magic file type when we know the file format but can't find file-specific details. (Closes: #850850)
  • And other code quality and style improvements.

reproducible-website development

• Daniel Shahaf:
  • Add how-to-chair-a-meeting.md.
• Holger Levsen:
  • Add links to reports by several attendees of berlin2016

tests.reproducible-builds.org

• Ximin Luo and Holger Levsen worked on stricter tests to check that /dev/shm and /run/shm are both mounted with the correct permissions. Some of our build machines currently still fail this test, and the problem is probably the root cause of the FTBFS of some packages (which fails with issues regarding sem_open). The proper fix is still being discussed in #851427.
• Valerie Young worked on creating and linking autogenerated schema documentation for our database used to store the results.
• Holger Levsen added a graph with diffoscope crashes and timeouts.
• Holger also further improved the daily mail notifications about problems.

Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday January 1 and Saturday January 7 2017: GSoC and Outreachy updates

• Maria Glukhova blogged about Getting to know diffoscope better.

Toolchain development

• #849999 was filed: "dpkg-dev should not set SOURCE_DATE_EPOCH to the empty string"

Packages reviewed and fixed, and bugs filed Chris Lamb:

• #849968 filed against pciutils.
• #849972 filed against yard.
• #850556 filed against sed.

Dhole:

• #849926 filed against nxt-firmware.

Reviews of unreproducible packages 13 package reviews have been added, 4 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added/updated:

• Add new randomness_in_documentation_generated_by_yardoc toolchain issue.
• Add fix for randomness_in_documentation_generated_by_yardoc toolchain issue.

Upstreaming of reproducibility fixes Merged:

• calibre
• jna
• rocksndiamonds
• avfs
• jackaudio1
• jackaudio2
• gri

Opened:

• rockdodger

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (4)

diffoscope development diffoscope 67 was uploaded to unstable by Chris Lamb. It included contributions from :

[ Chris Lamb ]
* Optimisations:
  - Avoid multiple iterations over archive by unpacking once for an ~8X
    runtime optimisation.
  - Avoid unnecessary splitting and interpolating for a ~20X optimisation
    when writing --text output.
  - Avoid expensive diff regex parsing until we need it, speeding up diff
    parsing by 2X.
  - Alias expensive Config() in diff parsing lookup for a 10% optimisation.
* Progress bar:
  - Show filenames, ELF sections, etc. in progress bar.
  - Emit JSON on the the status file descriptor output instead of a custom
    format.
* Logging:
  - Use more-Pythonic logging functions and output based on __name__, etc.
  - Use Debian-style "I:", "D:" log level format modifier.
  - Only print milliseconds in output, not microseconds.
  - Print version in debug output so that saved debug outputs can standalone
    as bug reports.
* Profiling:
  - Also report the total number of method calls, not just the total time.
  - Report on the total wall clock taken to execute diffoscope, including
    cleanup.
* Tidying:
  - Rename "NonExisting" -> "Missing".
  - Entirely rework diffoscope.comparators module, splitting as many separate
    concerns into a different utility package, tidying imports, etc.
  - Split diffoscope.difference into diffoscope.diff, etc.
  - Update file references in debian/copyright post module reorganisation.
  - Many other cleanups, etc.
* Misc:
  - Clarify comment regarding why we call python3(1) directly. Thanks to J r my
    Bobbio <lunar@debian.org>.
  - Raise a clearer error if trying to use --html-dir on a file.
  - Fix --output-empty when files are identical and no outputs specified.
[ Reiner Herrmann ]
* Extend .apk recognition regex to also match zip archives (Closes: #849638)
[ Mattia Rizzolo ]
* Follow the rename of the Debian package "python-jsbeautifier" to
  "jsbeautifier".
[ siamezzze ]
* Fixed no newline being classified as order-like difference.

reprotest development reprotest 0.5 was uploaded to unstable by Chris Lamb. It included contributions from:

[ Ximin Luo ]
* Stop advertising variations that we're not actually varying.
  That is: domain_host, shell, user_group.
* Fix auto-presets in the case of a file in the current directory.
* Allow disabling build-path variations. (Closes: #833284)
* Add a faketime variation, with NO_FAKE_STAT=1 to avoid messing with
  various buildsystems. This is on by default; if it causes your builds
  to mess up please do file a bug report.
* Add a --store-dir option to save artifacts.

Other contributions (not yet uploaded):

• Chris Lamb:
  • setup.py: Specify a Python3 env shebang
  • Drop reprotest/ runner,tools from DEP-5 copyright wildcard matching.

reproducible-builds.org website development

• website.git notifications now go to#reproducible-builds. Thanks todanielsh and the KGB bot maintainers.
• Daniel Shahaf:
  • events: add IRC meeting
  • Move mailing list from 'who' to 'resources'.
• Holger Levsen:
  • link to meetbot.debian.net IRC meetings archives
• Chris Lamb:
  • tools: Underscore that disorderfs is deliberately adding such randomness.

tests.reproducible-builds.org

• Debian arm64 architecture was fully tested in all three suites in just 15 days. Thanks again to Codethink.co.uk for their support!
• Log diffoscope profiling info. (lamby)
• Run pg_dump with -O --column-inserts to make easier to import our main database dump into a non-PostgreSQL database. (mapreri)
• Debian armhf network: CPU frequency scaling was enabled for three Firefly boards, enabling the CPUs to run at full speed. (vagrant)
• Arch Linux and Fedora tests have been disabled (h01ger)
• Improve mail notifications about daily problems. (h01ger)

Misc. This week's edition was written by Chris Lamb, Holger Levsen and Vagrant Cascadian, reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

What happened in the Reproducible Builds effort between Sunday December 18 and Saturday December 24 2016: Media coverage 100% Of The 289 Coreboot Images Are Now Built Reproducibly by Phoronix, with more details in German by Pro-Linux.de. We have further reports on our Reproducible Builds World summit #2 in Berlin from Rok Garbas of NixOS as well as Clemens Lang of MacPorts Debian infrastructure work Dak now archives buildinfo files thanks to a patch from Chris Lamb. We also have mostly finalised a design of how they will be distributed by the Debian FTP mirror network which we will start implementing soon. This is great for the future of Debianb but unfortunately this also means that we won't have .buildinfo files for Stretch as Debian will not rebuild its source packages and because these binary packages currently in the archive were mostly built with dpkg > 1.18.11. reprepro/5.0.0-1 has added support for dealing with .buildinfo files that are included in .changes files. (Closes: #843402) Reproducible work in other projects The Chromium project is now working on making their build process (mostly) deterministic. Their motivation is to save both "[money] (less hardware is required) and developer time (reduced latency by having less work to do on the TS and CI)". Unreproducible bugs filed

• Chris Lamb:
  • #848721 filed against apt.
  • #848866 filed against libcorelinux.
  • #849314 filed against node-gulp.
  • #849333 filed against faker.
• Dhole:
  • #848633 filed against sugar-toolkit-gtk3.

Reviews of unreproducible packages 39 package reviews have been added, 75 have been updated and 44 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• Updated captures_build_path
• Added nondeterministic_ordering_in_desktop_files_by_python_sugar3

Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by:

• Adrian Bunk (1)
• Chris Lamb (7)
• Lucas Nussbaum (4)

diffoscope development diffoscope 66 was uploaded to unstable by Chris Lamb. It included contributions from:

• Emanuel Bronshtein:
  • Use ssh-keygen for comparing OpenSSH public keys
  • Use js-beautify as JavaScript code beautifier for .js files (with tests).
  • Many CSS & HTML improvements.
  • Change all HTTP URLs to HTTPS where applicable.
• anthraxx:
  • Enable the use of ssh-keygen on Arch Linux.
• Maria Glukhova:
  • Add detection of order-only difference in plain text format. (Closes: #848049)
  • Change icc-recognizing regexp to reflect changes in file type description. (Closes: #848814)
• Chris Lamb:
  • Update tests for compatibility with enjarify >= 1.0.3. (Closes: #849142)
  • When skipping tests because the version of an external is too low, print the detected version.
  • Avoid unpacking packages twice when comparing .changes. (Closes: #843531)
  • Add a simple profiling framework (enabled via --profile).
  • Various code quality and reliability improvements.
  • Document how to sign PyPI uploads.

strip-nondeterminism development strip-nondeterminism 0.029-1 was uploaded to unstable by Chris Lamb. It included no new content from this week, but rather included contributions from previous weeks. reproducible-website development The website is now also accessible via the https://www.reproducible-builds.org URL.

• Clemens Lang:
  • Add the definition of "reproducible", as drafted at the reproducible builds world summit in Berlin. Thanks to all participants in the sessions that worked these out!
• Valerie R Young:
  • Force ordering of titles.
  • Various formatting improvements.
• Holger Levsen:
  • Various usability and formatting improvements.
  • https://www.reproducible-builds.org
• Chris Lamb:
  • Various usability, style and wording improvements.
  • Add Debconf15, Skroutz.gz and MiniDebconfCambridge15 talks to resouces page.

tests.reproducible-builds.org

• We changed the data storage backend from a single sqlite3 database file (651 MB) to a PostgreSQL database. With this change we'll be able to scale a lot more and add testing of the arm64 architecture.
  • Valerie Young wrote most of the code, Mattia Rizzolo reviewed and helped improve the code and Holger deployed it and found some minor bugs which have been fixed.
• We are now testing the arm64 architecture for all packages on all suites, arranged by Holger. Many thanks to codethink for providing us with access to eight 8-core arm64 machines with 64GB memory, which allows us to rebuild Debian very fast!

Misc. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC and the mailing lists.

What happened in the Reproducible Builds effort between Sunday December 4 and Saturday December 10 2016: Toolchain development and fixes Anders Kaseorg opened a pull request to asciidoc upstream, to make it generate reproducible documentation. (#782294) Bugs filed Chris Lamb:

• #847768 filed against hoichess.

Clint Adams:

• #847008 filed against tircd.
• #847021 filed against oolite.
• #847024 filed against torrus.
• #847039 filed against jqueryui.

Dafydd Harries:

• #847033 filed against newlib.
• #847035 filed against bulletml.

Robbie Harwood:

• #846971 filed against maildrop.
• #847009 filed against amanda.

Valerie R Young:

• #846975 filed against libranlip.
• #847016 filed against magicfilter.
• #847020 filed against mailfront.
• #847022 filed against mimefilter.
• #847023 filed against netqmail.
• #847026 filed against psgml.
• #847028 filed against qmail-run.
• #847029 filed against qmail-tools.
• #847030 filed against skalibs.
• #847031 filed against socklog.
• #847032 filed against tinydyndns.
• #847034 filed against twoftpd.
• #847036 filed against ucspi-tcp.
• #847037 filed against ucspi-unix.

Reviews of unreproducible packages 47 package reviews have been added, 84 have been updated and 3 have been removed in this week, adding to our knowledge about identified issues. 1 new issue type has been added: lessc_captures_build_path Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (8)

diffoscope development Chris Lamb fixed a division-by-zero in the progress bar, split out trydiffoscope into a separate package, and made some performance enhancements. Maria Glukhova fixed build issues with Python 3.4 strip-nondeterminism development Anders Kaseorg added support for .par files, by allowing them to be treated as Zip archives; and Chris Lamb improved some documentation. reprotest development Ximin Luo added the ability to vary the build time using faketime, as well as other code quality improvements and cleanups. He also discovered a little-known fact about faketime - that it also modifies filesystem timestamps by default. He submitted a PR to libfaketime upstream to improve the documentation on this, which was quickly accepted, and also disabled this feature in reprotest's own usage of faketime. buildinfo.debian.net development There was further work on buildinfo.debian.net code. Chris Lamb added support for buildinfo format 0.2 and made rejection notices clearer; and Emanuel Bronshtein fixed some links to use HTTPS. Misc. This week's edition was written by Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC and via email.

What happened in the Reproducible Builds effort between Sunday November 27 and Saturday December 3 2016: Reproducible work in other projects

• Ducible is a new tool to make Windows builds reproducible.
• Manish Goregaokar wrote about Reflections on Rusting Trust.

Media coverage, etc.

• There was a Reproducible Builds hackathon in Boston with contributions from Dafydd, Valerie, Clint, Harlen, Anders, Robbie and Ben. (See the "Bugs filed" section below for the results).
• Distrowatch mentioned Webconverger's reproducible status.

Bugs filed Chris Lamb:

• #846588 filed against minicoredumper.
• #846647 filed against tinyeartrainer.
• #846842 filed against nethogs.

Clint Adams:

• #846892 filed against pkg-mozilla-archive-keyring.

Dafydd Harries:

• #846893 filed against flac.

Daniel Shahaf:

• #846832 filed against patat.

Reiner Herrmann:

• #845991 filed against pathogen.

Valerie R Young:

• #846878 filed against taggrepper.
• #846890 filed against ipsvd.
• #846891 filed against integrit.

Reviews of unreproducible packages 15 package reviews have been added, 4 have been updated and 26 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• nondeterminstic_ordering_in_casacore_tables
• nondeterminstic_output_from_uglifyjs

Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (5)
• Lucas Nussbaum (8)
• Santiago Vila (1)

diffoscope development

• diffoscope 63 was uploaded to unstable by Ximin Luo:
  • Greatly improve speed for large archives by fixing O(n^2) complexity for archive member lookup.
  • add +/- buttons to toggle visibility of parts of the diff
  • Output coloured diff using colordiff(1) via --text-color= never,auto,always

Is is available now in Debian, Archlinux and on PyPI. strip-nondeterminism development

• At the Reproducible Builds Boston hackathon Anders Kaseorg filed #846895 treat .par files as Zip archives, including a patch which was merged into master.

reprotest development

• reprotest 0.4 was uploaded to unstable by Ximin Luo:
  • disorderfs variation: don't query the testbed, put that in the script instead
  • Add a build_path_same variation to run builds from the same path
  • Fix auto-presets in the case of a file in the current directory
  • Fix d/control so reprotesting reprotest in sbuild works (6 reproductions)
  • Add util-linux to Recommends since we use it to vary some things

tests.reproducible-builds.org

• Holger made a couple of changes:
  • Group all "done" and all "open" usertagged bugs together in the bugs graphs and move the "done bugs" from the bottom of these gaps.
  • Update list of packages installed on .debian.org machines.
  • Made the maintenance jobs run every 2h instead of 3h.
  • Various bug fixes and minor improvements.
• After thorough review by Mattia, some patches by Valerie were merged in preparation of the switch from sqlite to Postgresql, most notably a conversion to the sqlalchemy expression language.
• Holger gave a talk at Profitbricks about how Debian is using 168 cores, 503 GB RAM and 5 TB storage to make jenkins.debian.net and tests.reproducible-builds.org run. Many thanks to Profitbricks for supporting jenkins.debian.net since August 2012!
• Holger created a Jenkins job to build reprotest from git master branch.
• Finally, the Jenkins Naginator plugin was installed to retry git cloning in case of Alioth/network failures, this will benefit all jobs using Git on jenkins.debian.net.

Misc. This week's edition was written by Chris Lamb, Valerie Young, Vagrant Cascadian, Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday November 20 and Saturday November 26 2016: Reproducible work in other projects

• Webconverger, the Debian-based kiosk software is now reproducible

Bugs filed Chris Lamb:

• #845194 filed against amd64-microcode.
• #845325 filed against wheel.
• #845475 filed against apktool.
• #845524 filed against cairo-5c.
• #845525 filed against zp.
• #845745 filed against node-rimraf.

Daniel Shahaf:

• #845191 filed against libhtml-parser-perl.
• #845229 filed against libnss-ldap.
• #845282 filed against pcsc-tools.
• #845426 filed against munin-c.

Reiner Herrmann:

• #845317 filed against bzflag.
• #845763 filed against hdmi2usb-mode-switch.
• #845768 filed against noiz2sa.
• #845780 filed against brotli.
• #845782 filed against hannah.
• #845790 filed against garmin-plugin.
• #845991 filed against pathogen.

Reviews of unreproducible packages 63 package reviews have been added, 73 have been updated and 41 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been added:

• timestamps_in_header_or_footer_by_htmldoc_in_documentation
• libtool_captures_shell_build-flags_build-path_path-env
• nondeterminstic_ordering_in_python_wheel_metadata_versions
• ftbfs_in_jenkins_setup_due_to_socket_binding

Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (9)
• Helmut Grohne (1)
• Peter De Wachter (1)

strip-nondeterminism development

• #845203 was fixed in git by Reiner Herrmann - the next release will be able to normalize NTFS timestamps in zip files.

debrepatch development

• Ximin Luo added much more documentation for debpatch and filed #845659 to include it in devscripts.

Continuous integration:

• Holger updated our jenkins jobs for disorderfs and strip-nondeterminism to build these from their respective git master branches, and removed the jobs that build them from other branches since we have none at the moment.

tests.reproducible-builds.org Debian: Since the stretch freeze is getting closer, Holger made the following changes:

• Schedule testing builds to be as equally-frequent as unstable, on all archs, so that testing's build results are more up-to-date.
• Adjust experimental builds scheduling frequency so that experimental results are not more recent than the ones in unstable.
• Disable our APT repository for the testing suite (stretch), but leave it active for the unstable and experimental suites. This is the repository where we uploaded patched toolchain packages from time to time, that are necessary to reproduce other packages with. Since recently, all our essential patches have been accepted into Debian stretch and this repository is currently empty. Debian stretch will soon become the next Debian stable, and we want to get an accurate impression of how many of its packages will be reproducible. Therefore, disabling this repository for stretch whilst leaving it activated for the Debian unstable and experimental suites, allows us to continue to experiment with new patches to toolchain packages, without affecting our knowledge of the next Debian stable.

Misc. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday November 13 and Saturday November 19 2016: Media coverage

• Chris Lamb and Holger Levsen presented a status update at MiniDebConf Cambridge. (Video, Slides).

Elsewhere in Debian

• dpkg 1.18.14 has migrated to stretch.
• Chris Lamb filed #844431 ("packages should build reproducibly") against debian-policy.
• Ximin worked on glibc reproducibility this week, catching some bugs in disorderfs, FUSE, as well as glibc itself.

Documentation update

• Webconverger was added to our list of projects working on reproducible builds..

Packages reviewed and fixed, and bugs filed

• #844491 filed against linux by Chris Lamb.
• #844992 filed against perlbrew by Chris Lamb.
• #844993 filed against qsynth by Chris Lamb.
• #845034 filed against initramfs-tools by Chris Lamb.
• #845055 filed against markdown by Chris Lamb.
• #844752 filed against perl by Niko Tyni.
• #845007 filed against qsampler by Reiner Herrmann.
• #845008 filed against qmidinet by Reiner Herrmann.
• #845010 filed against qxgedit by Reiner Herrmann.

Reviews of unreproducible packages 43 package reviews have been added, 4 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• captures_build_path, captures_build_path_via_assert
• issue records_build_flags

4 issue types have been added:

• timestamps_in_documentation_generated_by_asciidoctor
• randomness_in_property_annotations_generated_by_sphinx
• randomness_in_browserify_lite_output
• txt2man_dash_p

Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (26)
• Daniel Stender (1)
• Filip Pytloun (1)
• Lucas Nussbaum (28)
• Michael Biebl (1)

strip-nondeterminism development

• Don't make tests rely on Debian::Debhelper::Dh_Lib. (Chris Lamb)

disorderfs development

• #844498 ("disorderfs: using it for building kills the host")

debrebuild development debrebuild is new tool proposed by HW42 and josch (see #774415: "From srebuild sbuild-wrapper to debrebuild"). debrepatch development debrepatch is a set of scripts that we're currently developing to make it easier to track unapplied patches. We have a lot of those and we're not always sure if they still work. The plan is to set up jobs to automatically apply old reproducibility patches to newer versions of packages and notify the right people if they don't apply and/or no longer make the package reproducible. debpatch is a component of debrepatch that applies debdiffs to Debian source packages. In other words, it is to debdiff(1) what patch(1) is to diff(1). It is a general tool that is not specific to Reproducible Builds. This week, Ximin Luo worked on making it more "production-ready" and will soon submit it for inclusion in devscripts. reprotest development Ximin Luo significantly improved reprotest, adding presets and auto-detection of which preset to use. One can now run e.g. reprotest auto . or reprotest auto $pkg_$ver.dsc instead of the long command lines that were needed before. He also made it easier to set up build dependencies inside the virtual server and made it possible to specify pre-build dependencies that reprotest itself needs to set up the variations. Previously one had to manually edit the virtual server to do that, which was not very usable to humans without an in-depth knowledge of the building process. These changes will be tested some more and then released in the near future as reprotest 0.4.

• Ximin Luo:
  • Set uname to two explicitly different values
  • Add a --no-diffoscope option. (Closes: #844512)
  • Add "auto" presets and --testbed-init
  • Add --testbed-pre and --testbed-init arguments
  • Fix main script so that build commands starting with ENVVAR=val work after append_command()
  • Dramatically expand documentation (many commits)

tests.reproducible-builds.org

• Debian:
  • An index of our usertagged bugs page was added by Holger after a Q+A session in Cambridge.
  • Holger also setup two new i386 builders, build12+16, for >50% increased build performance. For this, we went from 18+17 cores on two 48GB machines to 10+10+9+9 cores on four 36GB ram machines and from 16 to 24 builder jobs. Thanks to Profitbricks for providing us with all these ressources once more!
  • h01ger also tried to enable disorderfs again, but hit #844498, which brought down the i386 builders, so he disabled it again. Next will be trying disorderfs on armhf or amd64, to see whether this bug also manifests there.

Misc. This week's edition was written by Chris Lamb, Holger Levsen, Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday November 6 and Saturday November 12 2016: Media coverage Matthew Garrett blogged about Tor, TPMs and service integrity attestation and how reproducible builds are the base for systems integrity. The Linux Foundation announced renewed funding for us as part of the Core Infrastructure Initiative. Thank you! Outreachy updates Maria Glukhova has been accepted into the Outreachy winter internship and will work with us the Debian reproducible builds team. To quote her words

siamezzze: I've been accepted to #outreachy winter internship - going to
work with Debian reproducible builds team. So excited about that! <3
Debian

Toolchain development and fixes dpkg:

• Thanks to a series of dpkg uploads by Guillem Jover, all our toolchain changes are now finally available in sid!
• This means your packages should now be reproducible without having to use our custom APT repository.
• Ximin Luo opened #843925 to remind the fact that dpkg-buildpackage should sign buildinfo files.
• We hope to have detailed post about the new dpkg and the new .buildinfo files for debian-devel-announce soon!

debrebuild:

• srebuild / debrebuild work was resumed by Johannes Schauer and others in #774415.

Bugs filed Chris Lamb:

• #844102 filed against lava-dispatcher
• #844103 filed against lava-server
• #844111 filed against python-defaults
• #843698 filed against tunnelx
• #843967 filed against asyncpg
• #843865 filed against keystone
• #844101 filed against suil

Daniel Shahaf:

• #844232 filed against daisy-player
• #844236 filed against libhtml-lint-perl
• #844228 filed against ebook-speaker

Niko Tyni:

• #843432 filed against libwww-curl-perl

Reiner Herrman:

• #844095 filed against fdm
• #844096 filed against gringo
• #843469 filed against dpkg

Reviews of unreproducible packages 136 package reviews have been added, 5 have been updated and 7 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated:

• Added: random_order_in_dh_pythonX_substvars, valac_permutes_get_type_calls
• Updated: timestamps_in_static_libraries

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (29)
• Niko Tyni (1)

diffoscope development A new version of diffoscope 62~bpo8+1 was uploaded to jessie-backports by Mattia Rizzolo. Meanwhile in git, Ximin Luo greatly improved speed by fixing a O(n²) lookup which was causing diffs of large packages such as GCC and glibc to take many more hours than was necessary. When this commit is released, we should hopefully see full diffs for such packages again. Currently we have 197 source packages which - when built - diffoscope fails to analyse. buildinfo.debian.net development

• Submissions with duplicate Installed-Build-Depends entries are rejected now that a bug in dpkg causing them has been fixed. Thanks to Guillem Jover.
• Add a new page for every (source, version) combination, for example diffoscope 62.
• DigitalOcean have generously offered to sponsor the hardware buildinfo.debian.net is running on.

tests.reproducible-builds.org Debian:

• For privacy reasons, the new dpkg-genbuildinfo includes Build-Path only if it is under /build. HW42 updated our jobs so this is the case for our builds too, so you can see the build path in the .buildinfo files.
• HW42 also updated our jobs to vary the basename of the source extraction directory. This detects packages that incorrectly assume a $pkg-$version directory naming scheme (which is what dpkg-source -x gives but is not mandated by Debian nor always-true) or that they're being built from a SCM.
• The new dpkg-genbuildinfo also records a sanitised Environment. This is different in our builds, so HW42, Reiner and Holger updated our jobs to hide these differences from diffoscope output.
• Package-set improvements:
  • Holger refactored the create_meta_pkg_sets job so that it's easier to add new package sets.
  • This job is now also using dose-extra from jessie-backports so that it can deal with versioned provides.
  • Added 4 new package sets: debian-edu, debian-edu_build-depends, maint_pkg-grass-devel, maint_debian-accessibility, maint_pkg-openstack.
  • Switched to using the new URL for tails manifests to generate the tails package set.
  • Renamed maint_lua to maint_debian-lua
• Valerie Young contributed four patches for our long-planned transition from SQLite to PostgreSQL.
• In anticipation of the freeze, already-tested packages from unstable and testing on amd64 are now scheduled with equal priority.

reproducible-builds.org website F-Droid was finally added to our list of partner projects. (This was an oversight and they had already been working with us for some time.) Misc. This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday October 30 and Saturday November 5 2016: Upcoming events

• Chris Lamb and Holger Levsen will be presenting at MiniDebConfCambridge 2016 in Cambridge, United Kingdom on November 10th-13th.
• Vagrant Cascadian will be presenting Introduction to Reproducible Builds at the SeaGL.org Conference in Seattle, USA on November 12th.
• The next IRC meeting will be held on November 15th.
• Reproducible Debian Hackathon - A small hackathon organized in Boston, USA on December 3rd and 4th. If you are interested in attending, please contact Valerie Young (spectranaut in the #debian-reproducible IRC channel on irc.oftc.net.)
• The second Reproducible Builds World Summit will be held in Berlin, Germany on December 13th-15th.

Reproducible work in other projects

• Patches to GCC to generate reproducible output independently of the build-path were submitted by Ximin Luo.
• The new Firefox Security Roadmap includes Reproducible Builds (via a tweet by Debian Developer Sylvestre).

Bugs filed

• #842708 filed against haskell-devscripts by Chris Lamb.
• #842864 filed against aewan by Reiner Herrmann.
• #842865 filed against atanks by Reiner Herrmann.
• #842867 filed against i3status by Reiner Herrmann.
• #842868 filed against gip by Reiner Herrmann.
• #842875 filed against lastpass-cli by Reiner Herrmann.
• #842986 filed against zathura-pdf-poppler by Reiner Herrmann.
• #842988 filed against masscan by Reiner Herrmann.
• #842991 filed against python-axolotl-curve25519 by Reiner Herrmann.
• #842992 filed against pyephem by Reiner Herrmann.
• #843097 filed against spacenavd by Reiner Herrmann.
• #843100 filed against python-clips by Reiner Herrmann.
• #843101 filed against python-hiredis by Reiner Herrmann.
• #843109 filed against zathura-ps by Reiner Herrmann.
• #843110 filed against greenbone-security-assistant by Reiner Herrmann.
• #843234 filed against vdr-plugin-sudoku by Reiner Herrmann.
• #843235 filed against vdr-plugin-spider by Reiner Herrmann.
• #843245 filed against distorm3 by Reiner Herrmann.

Reviews of unreproducible packages 81 package reviews have been added, 14 have been updated and 43 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated:

• sphinx_htmlhelp_readdir_sensitive
• random_order_in_dh_haskell_substvars
• sphinx_htmlhelp_readdir_sensitive

1 issue type has been removed:

• unsorted_pdl_output

1 issue type has been updated:

• document some reseach on docbook-to-man

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (17)

diffoscope development

• diffoscope 62 was uploaded to unstable by Mattia Rizzolo:
  • Add rudimentary support for OpenDocumentFormat files. (Michel Messerschmidt)
  • Detect JSON order-only differences and print them nicely. (Mattia Rizzolo)

buildinfo.debian.net development

• Chris Lamb continued work on our proof-of-concept .buildinfo file server:
  • Show key ID over node name now
  • Show submission timestamp.
  • Accept .udebs
  • Actually show latest submissions on homepage.
  • GPG signatures are now mandatory

tests.reproducible-builds.org Reproducible Debian:

• With thanks to Profitbricks continued sponsorship, Holger setup two new amd64 build nodes (and the associated Jenkins jobs) with 15/16 cores and 48GB RAM each for a total of four such amd64 nodes to double the build capacity of our amd64 build network.

Misc. Also with thanks to Profitbricks sponsoring the "hardware" resources, Holger created a 13 core machine with 24GB RAM and 100GB SSD based storage so that Ximin can do further tests and development on GCC and other software on a fast machine. This week's edition was written by Chris Lamb, Ximin Luo, Vagrant Cascadian, Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday October 23 and Saturday October 29 2016: Upcoming events The second Reproducible Builds World Summit will be held from December 13th-15th in Berlin! See the link for more details. Other events: Introduction to Reproducible Builds - Vagrant Cascadian will be presenting at the SeaGL.org Conference In Seattle, USA on November 12th, 2016. Reproducible Debian Hackathon - A small hackathon organized in Boston, USA on December 3rd and 4th. If you are interested in attending, contact Valerie Young - spectranaut in the #debian-reproducible IRC channel on irc.oftc.net. IRC meeting The next IRC meeting will be held on 2016-11-01 at 18:00 UTC. The meeting after that will be held on 2016-11-15 at 18:00 UTC. Reproducible work in other projects Ximin Luo has had his fix to bug 77985 accepted into GCC. This is needed to be able to write test cases for patches to make GCC produce debugging symbols that are reproducible regardless of the build path. There was continued discussion on the mailing list regarding our build path proposals. It has now been decided to use an environment variable SOURCE_PREFIX_MAP instead of the older proposal SOURCE_ROOT_DIR. This would be similar to GCC's existing -fdebug-prefix-map option, which allows for better disambiguation between paths from different packages. mandoc's makewhatis is now reproducible. It is used by all the BSDs, including FreeBSD, as well as Alpine Linux and Void Linux. Packages reviewed and fixed, and bugs filed Chris Lamb:

• #842477 filed against git
• #842115 filed against partclone
• #842356 filed against bio-eagle
• #842050 filed against tj3
• #842203 filed against seabios

Reiner Herrmann:

• #841777 filed against perl-tk
• #841783 filed against patchutils
• #841779 filed against netkit-rsh
• #842628 filed against mwparserfromhell
• #842629 filed against terminix
• #842622 filed against libvterm
• #842548 filed against libmsv
• #842624 filed against mdp-src
• #842625 filed against fuse-zip
• #842546 filed against rsbackup
• #842583 filed against paulstretch
• #842550 filed against the
• #842581 filed against cryptsetup

Reviews of unreproducible packages 145 package reviews have been added, 608 have been updated and 94 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated:

• Added 1 new issue docbook_to_man_one_byte_delta.
• Updated 2 existing issues randomness_in_r_rdb_rds_databases and r_base_appends_built_header_to_description_files.

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (17)
• Matthias Klose (2)

tests.reproducible-builds.org Debian:

• Valerie improved the SQL code so that the scheduler job again runs within minutes. She did the same to the job updating the notes about known issues, though this job still runs 12min and not 2min as it used to do
• Thanks to a patch from Chris, which was improved by dkg and h01ger after discussions on our list, the .buildinfo files submitted to buildinfo.debian.net are now GPG signed by our build nodes.

General:

• Holger fixed the NetBSD and coreboot jobs which were broken due to work on the LEDE+OpenWRT jobs.
• As squid on jessie/i386 (but not on jessie/amd64) crashes frequently, we have have monitoring for this and Holger fixed a subtile bug there.

diffoscope development

• Daniel Shahaf:
  • json: detect order-only differences and print them nicely
• Mattia Rizzolo:
  • tests: skip test using a format accepted by older pytest

Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday October 16 and Saturday October 22 2016: Media coverage

• Chris Lamb presented at Software Freedom Kosovo on reproducible builds on Saturday 22nd October.

Upcoming events

• J rgen Rose will be giving a talk on Enforcing reproducible builds with Eclipse Package Drone at EclipseCon Europe 2016 in Ludwigsburg, Germany on October 27th.

buildinfo.debian.net In order to build packages reproducibly, you not only need identical sources but also some external definition of the environment used for a particular build. This definition includes the inputs and the outputs and, in the Debian case, are available in a $package_$architecture_$version.buildinfo file. We anticipate the next dpkg upload to sid will create .buildinfo files by default. Whilst it's clear that we also need to teach dak to deal with them (#763822) its not actually clear how to handle .buildinfo files after dak has processed them and how to make them available to the world. To this end, Chris Lamb has started development on a proof-of-concept .buildinfo server to see what issues arise. Source Reproducible work in other projects

• Ximin Luo submitted a patch to GCC as a prerequisite for future patches to make debugging symbols reproducible.

Packages reviewed and fixed, and bugs filed

• #841274 filed against node-once by Chris Lamb.
• #841342 filed against zshdb by Chris Lamb.
• #841427 filed against unifdef by Chris Lamb.
• #841440 filed against rdp-alignment by Chris Lamb.
• #841497 filed against cf-python by Chris Lamb.
• #841694 filed against dvdauthor by Reiner Herrmann.
• #841698 filed against node-lodash by Chris Lamb.
• #841701 filed against libtext-charwidth-perl by Reiner Herrmann.
• #841702 filed against libapt-pkg-perl by Reiner Herrmann.
• #841703 filed against libio-pty-perl by Reiner Herrmann.
• #841707 filed against eximdoc4 by Chris Lamb.

Reviews of unreproducible packages 99 package reviews have been added, 3 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues. 6 issue types have been added:

• bin_sh_is_bash
• captures_build_arch
• captures_build_path_via_assert
• docbook_to_man_one_byte_delta_on_i386
• graphviz_nondeterminstic_output
• randomness_in_documentation_generated_by_scilab

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

• Chris Lamb (23)
• Daniel Reichelt (2)
• Lucas Nussbaum (1)
• Santiago Vila (18)

diffoscope development

• Mattia Rizzolo:
  • tests/ppu: skip some PPU tests if ppudump is < 3.0.0
  • ppu: ignore decoding errors from ppudump while filtering the output
  • ppu: don't do run a full ppudump while only looking for PPU file version
  • debian: bump debhelper compat level to 10, no changes needed.
• Michel Messerschmidt:
  • Add rudimentary support for OpenDocumentFormat files

tests.reproducible-builds.org

• h01ger increased the diskspace for reproducible content on Jenkins. Thanks to ProfitBricks.
• Valerie Young supplied a patch to make Python SQL interface more SQLite/PostgresSQL agnostic.
• lynxis worked hard to make LEDE and OpenWrt builds happen on two hosts.

Misc. Our poll to find a good time for an IRC meeting is still running until Tuesday, October 25st; please reply as soon as possible. We need a logo! Some ideas and requirements for a Reproducible Builds logo have been documented in the wiki. Contributions very welcome, even if simply by forwarding this information. This week's edition was written by Chris Lamb & Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday October 9 and Saturday October 15 2016: Media coverage

• despinosa wrote a blog post on Vala and reproducibility
• h01ger and lynxis gave a talk called "From Reproducible Debian builds to Reproducible OpenWrt, LEDE" (video, slides) at the OpenWrt Summit 2016 held in Berlin, together with ELCE, held by the Linux Foundation.
• A discussion on debian-devel@ resulted in a nice quotable comment from Paul Wise: "(Reproducible) builds from source (with continuous rechecking) is the only way to have enough confidence that a Debian user has the freedoms promised to them by the Debian social contract."
• Chris Lamb will present a talk at Software Freedom Kosovo on reproducible builds on Saturday 22nd October.

Documentation update After discussions with HW42, Steven Chamberlain, Vagrant Cascadian, Daniel Shahaf, Christopher Berg, Daniel Kahn Gillmor and others, Ximin Luo has started writing up more concrete and detailed design plans for setting SOURCE_ROOT_DIR for reproducible debugging symbols, buildinfo security semantics and buildinfo security infrastructure. Toolchain development and fixes Dmitry Shachnev noted that our patch for #831779 has been temporarily rejected by docutils upstream; we are trying to persuade them again. Tony Mancill uploaded javatools/0.59 to unstable containing original patch by Chris Lamb. This fixed an issue where documentation Recommends: substvars would not be reproducible. Ximin Luo filed bug 77985 to GCC as a pre-requisite for future patches to make debugging symbols reproducible. Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible - in our current test setup - after being fixed:

• cobbler/2.6.6+dfsg1-13 by Thomas Goirand, original patch by Chris Lamb.
• collectd/5.6.1-1 by Marc Fournier.
• fonts-tiresias/0.1-3 by G rkan Myczko, original patch by Chris Lamb.
• fntsample/4.0-2 by , original patch by Chris Lamb.
• fpga-icestorm/0~20160913git266e758-2 by Ruben Undheim, original patch by Chris Lamb.
• frog/0.13.5-1 by Maarten van Gompel, original patch by Chris Lamb.
• lambda-align/1.0.0-2 by Sascha Steinbiss, original patch by Chris Lamb.
• pleiades/1.7.0-2 by Hideki Yamane, original patch by Chris Lamb.
• sweethome3d/5.2+dfsg-1 by Markus Koschany, original fix by Gabriele Giacone.
• trac-subtickets/0.2.0-2 by W. Martin Borgert.

The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.)

• aodh/3.0.0-2 by Thomas Goirand.
• eog-plugins/3.16.5-1 by Michael Biebl.
• flam3/3.0.1-5 by Daniele Adriana Goulart Lopes.
• hyphy/2.2.7+dfsg-1 by Andreas Tille.
• libbson/1.4.1-1 by A. Jesse Jiryu Davis.
• libmongoc/1.4.1-1 by A. Jesse Jiryu Davis.
• lxc/1:2.0.5-1 by Evgeni Golov.
• spice-gtk/0.33-1 by Liang Guo.
• spice-vdagent/0.17.0-1 by Liang Guo.
• tnef/1.4.12-1 by Kevin Coyner.

Some uploads have addressed some reproducibility issues, but not all of them:

• chktex/1.7.6-1 by Thorsten Alteholz, original patch by Sascha Steinbiss.
• dbus/1.10.12-1 by Simon McVittie.
• doomsday/1.15.8-3 by Markus Koschany, #839338 by Lucas Nussbaum.
• emacs25/25.1+1-1 by Rob Browning.
• gpgme1.0/1.7.0-3 by Daniel Kahn Gillmor.
• monkeysign/2.2.0 by Antoine Beaupr .
• python-attrs/16.2.0-1 by Tristan Seligmann, original patch by Chris Lamb.
• shotwell/0.24.0-1 by J rg Frings-F rst, original patch by Alexis Bienven e.
• supple/1.0.6-2 by Daniel Silverstone.
• why/2.36-1 by Ralf Treinen, original patch by Valentin Lorentz.

Some uploads have addressed nearly all reproducibility issues, except for build path issues:

• palo/1.96 by Helge Deller, #778437 by Chris Lamb.
• rbdoom3bfg/1.1.0~preview3+dfsg+git20160807-1 by Tobias Frost.
• singular/4.0.3-p3+ds-1 by Jerome Benoit.
• varnish/5.0.0-3 by Stig Sandbeck Mathisen, original patch by Chris Lamb.
• yaml-cpp/0.5.2-4 by Paul Novotny, original patch by Reiner Herrmann.

Patches submitted that have not made their way to the archive yet:

• #840741 filed against http-icons by Chris Lamb.
• #840177 filed against qconf by Chris Lamb.
• #840845 filed against python-pygraphviz by Chris Lamb.
• #840346 filed against qjoypad by Chris Lamb.

Reviews of unreproducible packages 101 package reviews have been added, 49 have been updated and 4 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated:

• Added max_output_size_reached, ftbfs_due_to_jenkins_semaphore_setup, and build_id_differences_only.

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

• Anders Kaseorg (1)
• Chris Lamb (18)

tests.reproducible-builds.org Debian:

• h01ger has turned off the "Scheduled in testing+unstable+experimental" regular IRC notifications and turned them into emails to those running jenkins.d.n.
• Re-add opi2a armhf node and 3 new builder jobs for a total of 60 build jobs for armhf. (h01ger and vagrant)
• vagrant suggested to add a variation of init systems effecting the build, and h01ger added it to the TODO list.
• Steven Chamberlain submitted a patch so that now all buildinfo files are collected (unsigned yet) at submit@buildinfo.kfreebsd.eu.
• Holger enabled CPU type variation (Intel Haswell or AMD Opteron 62xx) for i386. Thanks to Profitbricks.com for their great and continued support!

Openwrt/LEDE/NetBSD/coreboot/Fedora/archlinux:

• Increase memory on the 2 build nodes from 12 to 16gb, thanks to profitbricks.com

Misc. We are running a poll to find a good time for an IRC meeting. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday September 25 and Saturday October 1 2016: Statistics For the first time, we reached 91% reproducible packages in our testing framework on testing/amd64 using a determistic build path. (This is what we recommend to make packages in Stretch reproducible.) For unstable/amd64, where we additionally test for reproducibility across different build paths we are at almost 76% again. IRC meetings We have a poll to set a time for a new regular IRC meeting. If you would like to attend, please input your available times and we will try to accommodate for you. There was a trial IRC meeting on Friday, 2016-09-31 1800 UTC. Unfortunately, we did not activate meetbot. Despite this participants consider the meeting a success as several topics where discussed (eg changes to IRC notifications of tests.r-b.o) and the meeting stayed within one our length. Upcoming events Reproduce and Verify Filesystems - Vincent Batts, Red Hat - Berlin (Germany), 5th October, 14:30 - 15:20 @ LinuxCon + ContainerCon Europe 2016. From Reproducible Debian builds to Reproducible OpenWrt, LEDE & coreboot - Holger "h01ger" Levsen and Alexander "lynxis" Couzens - Berlin (Germany), 13th October, 11:00 - 11:25 @ OpenWrt Summit 2016. Introduction to Reproducible Builds - Vagrant Cascadian will be presenting at the SeaGL.org Conference In Seattle (USA), November 11th-12th, 2016. Previous events GHC Determinism - Bartosz Nitka, Facebook - Nara (Japan), 24th September, ICPF 2016. Toolchain development and fixes Michael Meskes uploaded bsdmainutils/9.0.11 to unstable with a fix for #830259 based on Reiner Herrmann's patch. This fixed locale_dependent_symbol_order_by_lorder issue in the affected packages (freebsd-libs, mmh). devscripts/2.16.8 was uploaded to unstable. It includes a debrepro script by Antonio Terceiro which is similar in purpose to reprotest but more lightweight; specific to Debian packages and without support for virtual servers or configurable variations. Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible in our testing framework after being fixed:

• ara/1.0.32 by Chris Lamb, original patch by Chris Lamb.
• fracplanet/0.4.0-5 by Chris Lamb, original patch by Reiner Herrman.
• gnarwl/3.6.dfsg-8 by Bernhard Schmidt, original patch by Chris Lamb.
• kgb-bot/1.34-1 by gregor herrmann, original patch by gregor herrmann.
• survex/1.2.29-1 by Olly Betts.
• zpaq/1.10-3 by Chris Lamb, original patch by Reiner Herrman.
• fig2dev/1:3.2.6-3 by Roland Rosenfeld.
• luxio/11-1 by Daniel Silverstone.
• monkeysign/2.1.1 by Antoine Beaupr , original patch by Daniel Kahn Gillmor.
• openarena-085-data/0.8.5split-9 by Simon McVittie.
• openarena-088-data/0.8.8-7 by Simon McVittie.
• openarena-data/0.8.5split-9 by Simon McVittie.
• rc/1.7.4-1 by Reiner Herrmann, original patch by Chris Lamb.

The following updated packages appear to be reproducible now for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.)

• gkrellm/2.3.8-1 by Sandro Tosi
• glassfish/1:2.1.1-b31g+dfsg1-4 by Emmanuel Bourg

Some uploads have addressed some reproducibility issues, but not all of them:

• apache2/2.4.23-5 by Rapha l Hertzog
• freeradius/3.0.11+dfsg-1 by Michael Stapelberg
• libnss-ldap/265-4 by Chris Lamb
• lift/2.5.0-1 by Nicolas Delvaux
• linux/4.8~rc8-1~exp1 by Ben Hutchings
• nose2/0.6.5-2 by Barry Warsaw
• postgresql-9.6/9.6.0-1 Christoph Berg
• strace/4.13-0.1 by Nicolas Braud-Santoni
• yersinia/0.7.3-3 by No l K the

Patches submitted that have not made their way to the archive yet:

• #838888 filed against dh-haskell by Chris Lamb.
• #838971 filed against slang2 by Chris Lamb.
• #839587 filed against sympa by Chris Lamb.
• #839181 filed against transmission-remote-gtk by Chris Lamb.
• #838829 filed against vala by Sebastian Reichel.
• #838970 filed against webkit2pdf by Chris Lamb.
• #831569 filed against websockets by Chris Lamb.
• #839347 filed against xml-core by Lucas Nussbaum.
• #839526 filed against xml-core by Adrian Bunk.

Reviews of unreproducible packages 77 package reviews have been added, 178 have been updated and 80 have been removed in this week, adding to our knowledge about identified issues. 6 issue types have been updated:

• Added clilibs_line_order, records_build_flags and hevea_captures_build_path.
• Removed locale_dependent_symbol_order_by_lorder, fixed in bsdmainutils/9.0.11.
• Updated diffoscope_runs_forever, captures_build_path.

Weekly QA work As part of reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (3)
• Chris Lamb (12)
• Lucas Nussbaum (3)
• Sebastian Reichel (1)

diffoscope development A new version of diffoscope 61 was uploaded to unstable by Chris Lamb. It included contributions from:

• Ximin Luo:
  • Improve the CLI --help text and add an --output-empty option.
• Chris Lamb:
  • Add a progress bar and show it if stdout is a TTY. You can read more about it here. It can also be read by higher-level programs via the --status-fd CLI option.
• Maria Glukhova:
  • Behaviour improvements in the case of OS-level errors.
• Mattia Rizzolo:
  • Testing and packaging improvements.

Post-release there were further contributions from:

• Chris Lamb:
  • Code architecture improvements.
• Maria Glukhova:
  • Testing improvements.

reprotest development A new version of reprotest 0.3.2 was uploaded to unstable by Ximin Luo. It included contributions from:

• Ximin Luo:
  • Add a --diffoscope-arg CLI option to pass extra args to diffoscope.

Post-release there were further contributions from:

• Chris Lamb:
  • Code quality improvements.

tests.reproducible-builds.org

• Hans-Christoph Steiner continued work on setting up reproducible tests for F-Droid.
• Holger cleaned up the script creating the page showing breakages, so that it now also cleans up some of the breakage it finds.
• IRC notifications about diffoscope crashes and artifacts available for investigations have been dropped; instead the breakages page has a permanent pointer. (h01ger)
• IRC notifications from the automatic package scheduler and status changes for packages have been moved -- as a temporary trial -- to #debian-reproducible-changes on irc.oftc.net (Mattia).

Misc. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

Here is what happened in the Reproducible Builds effort between Sunday September 18 and Saturday September 24 2016: Outreachy We intend to participate in Outreachy Round 13 and look forward for new enthusiastic applications to contribute to reproducible builds. We're offering four different areas to work on:

• Improve test and debugging tools.
• Improving reproducibility of Debian packages.
• Improving Debian infrastructure.
• Help collaboration across distributions.

Reproducible Builds World summit #2 We are planning e a similar event to our Athens 2015 summit and expect to reveal more information soon. If you haven't been contacted yet but would like to attend, please contact holger. Toolchain development and fixes Mattia uploaded dpkg/1.18.10.0~reproducible1 to our experimental repository. and covered the details for the upload in a mailing list post. The most important change is the incorporation of improvements made by Guillem Jover (dpkg maintainer) to the .buildinfo generator. This is also in the hope that it will speed up the merge in the upstream. One of the other relevant changes from before is that .buildinfo files generated from binary-only builds will no longer include the hash of the .dsc file in Checksums-Sha256 as documented in the specification. Even if it was considered important to include a checksum of the source package in .buildinfo, storing it that way breaks other assumptions (eg. that Checksums-Sha256 contains only files part of that are part of a single upload, wheras the .dsc might not be part of that upload), thus we look forward for another solution to store the source checksum in .buildinfo. Bugs filed

• #838713 filed against python-xlib by Chris Lamb.
• #838754 filed against golang-google-grpc by Chris Lamb.
• #838188 filed against ocaml by Johannes Schauer.
• #838785 filed against funnelweb by Reiner Herrmann.

Reviews of unreproducible packages 250 package reviews have been added, 4 have been updated and 4 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been added:

• captures_users_gecos issue
• timestamps_in_org_mode_html_output toolchain issue.
• varnish_vmodtool_random_file_id
• gpg_keyring_magic_bytes_differ

3 issue types have been updated:

• timestamps_in_org_mode_html_output more generally.
• timestamps_in_documentation_generated_by_texi2html
• randomness_in_ocaml_preprocessed_files

Weekly QA work FTBFS bugs have been reported by:

• Chris Lamb (11)
• Santiago Vila (2)

Documentation updates h01ger created a new Jenkins job so that every commit pushed to the master branch for the website will update reproducible-builds.org. diffoscope development

• Mattia Rizzolo:
  • Skip rlib tests if the "nm" tool is missing
• Ximin Luo:
  • Give better advice about what envvar to set to make the console work
  • tests/basic-command-line: check exit code and use a more complex example
  • Add a script to check sizes of dependencies
  • Don't use unicode quotes to avoid breakage under LC_ALL=C

strip-nondeterminism development

• Chris Lamb:
  • .perltidyrc: Add from lintian.
  • .perltidyrc: We use tabs, not spaces.
  • Run perltidy

reprotest development

• Ximin Luo uploaded reprotest 0.3 and 0.3.1 to unstable with these changes:
  • Make some variations more reliable, so tests don't fail
  • Add a safety device to guard against typos
  • Address lintian warnings
  • Remove any existing artifact, in case the build script doesn't overwrite it
  • Fix the logic of some tests, and don't vary fileordering on Debian buildds
  • Use the magic of VIRTUALENV_DOWNLOAD=no, seen in tox's own autopkgtest tests
  • Flush so subprocess output is guaranteed to appear later
  • Don't error if the build command generates stderr
  • Default tests to run on "null" only since it takes effort to set up the others
  • hey dawg i herd u liek tests so i put some tests in ur tests so u can test while u test
  • Make no_clear_on_error optional; we don't want to pass it in everywhere e.g. tests
  • Output a nice big obvious summary at the end when successful
  • Don't repeat documentation in two different places, move it all to --help
  • More reliable build artifact pattern matching, and update docs
  • Use a list comprehension for slightly more idiomatic python than map/lambda
  • Make multi-component artifact patterns like '.deb .changes' work correctly
  • Add a --no-clean-on-error option so you can analyse what went wrong
  • More help text for virtual_server_args
  • Support shell patterns as the build_artifact
  • Use sys.exit inside main(), not check() as it's more idiomatic
  • Pass kind='build' to check_exec so it doesn't time out after 100s
  • adt_testbed: add stdout/stderr to the "auxverb failed" error message, if available
  • If no virtual_server is given then use "null"

tests.reproducible-builds.org

• The full rebuild of all packages in unstable (for all tested archs) with the new build path variation has been completed. This has had the result that we are down to ~75% reproducible packages in unstable now. In comparison, for testing (where we don't vary the build path) we are still at ~90%. IRC notifications for unstable have been enabled again. (Holger)
• Make the notes job robust about bad data (see #833695 and #833738). (Holger)
• Setup profitbricks-build7 running stretch as testing reproducible builds of F-Droid need to use a newer version of vagrant in order to support running vagrant VMs with kvm on kvm. (Holger)
• The misbehaving 'opi2a' armhf node has been replaced with a Jetson-TK1 board kindly donated by NVidia. This machine is using an NVIDIA tegra-k1 (cortex-a15) quad-core board. (vagrant and Holger)

Misc. This week's edition was written by Chris Lamb, Holger Levsen and Mattia Rizzolo and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday September 11 and Saturday September 17 2016: Toolchain developments Ximin Luo started a new series of tools called (for now) debrepatch, to make it easier to automate checks that our old patches to Debian packages still apply to newer versions of those packages, and still make these reproducible. Ximin Luo updated one of our few remaining patches for dpkg in #787980 to make it cleaner and more minimal. The following tools were fixed to produce reproducible output:

• naturaldocs/1.51-2 by Petter Reinholdtsen, original patch by Chris Lamb.

Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible - in our current test setup - after being fixed:

• elog/3.1.2-1-1 by Roger Kalt, original patch by Reiner Herrmann.
• eyed3/0.6.18-3 by Petter Reinholdtsen, original patch by Chris Lamb.
• frog/0.13.5-1 by Maarten van Gompel, original patch by Chris Lamb.
• gtranslator/2.91.7-3 by Andreas Henriksson, original patch by Reiner Herrmann.
• sozi/12.05-1.1 by Daniel Kahn Gillmor, original patch by Chris Lamb.

The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.)

• evince/3.21.92-1 by Michael Biebl.
• gnome-control-center/1:3.21.92-2 by Rapha l Hertzog.
• libipathverbs/1.3-2 by Ana Beatriz Guerrero Lopez.
• pagekite/0.5.8e-2 by Petter Reinholdtsen.

The following 3 packages were not changed, but have become reproducible due to changes in their build-dependencies: jaxrs-api python-lua zope-mysqlda. Some uploads have addressed some reproducibility issues, but not all of them:

• eurephia/1.1.0-6 by Alberto Gonzalez Iniesta, original patch by Chris Lamb.
• fdroidserver/0.7.0-1 by Hans-Christoph Steiner, original patch by Chris Lamb.
• mini-buildd/1.0.18 by Stephan S rken.
• nbc/1.2.1.r4+dfsg-3 by Petter Reinholdtsen, original patch by Chris Lamb.
• ncurses/6.0+20160910-1 by Sven Joachim, #818067 by Niels Thykier.
• python-kinterbasdb/3.3.0-4 by Santiago Vila, original patch by Chris Lamb.
• snapper/0.3.3-1 Hideki Yamane, original patch by Sascha Steinbiss.

Patches submitted that have not made their way to the archive yet:

• #838188 filed against ocaml by Johannes Schauer.

Reviews of unreproducible packages 462 package reviews have been added, 524 have been updated and 166 have been removed in this week, adding to our knowledge about identified issues. 25 issue types have been updated:

• Added a new annotation for issues called "fix-deterministic" to help us update package reviews more easily. This indicates whether we expect that an issue would always happen on Jenkins; i.e. if there is a successful build, then we know the issue is fixed for that package and can update our notes.
• Added random_order_in_sisu_javax_inject_named and too_much_input_for_diff.
• Removed timestamps_in_manpages_generated_by_ronn.
• Updated timestamps_in_allegro_dat_files. Additionally, 21 issues were marked with "fix-deterministic".

Weekly QA work FTBFS bugs have been reported by:

• Chris Lamb (10)
• Filip Pytloun (1)
• Santiago Vila (1)

diffoscope development A new version of diffoscope 60 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Mattia Rizzolo:
  • Various packaging and testing improvements.
• HW42:
  • minor wording fixes
• Reiner Herrmann:
  • minor wording fixes

It also included from changes previous weeks; see either the changes or commits linked above, or previous blog posts 72 71 70. strip-nondeterminism development New versions of strip-nondeterminism 0.027-1 and 0.028-1 were uploaded to unstable by Chris Lamb. It included contributions from:

• Chris Lamb:
  • Testing improvements, including better handling of timezones.

disorderfs development A new version of disorderfs 0.5.1 was uploaded to unstable by Chris Lamb. It included contributions from:

• Andrew Ayer and Chris Lamb:
  • Support relative paths for ROOTDIR; it no longer needs to be an absolute path.
• Chris Lamb:
  • Print the behaviour (shuffle/reverse/sort) on startup to stdout.

It also included from changes previous weeks; see either the changes or commits linked above, or previous blog posts 70. Misc. This week's edition was written by Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened in the Reproducible Builds effort between Sunday September 4 and Saturday September 10 2016: Reproducible work in other projects Python 3.6's dictonary type now retains the insertion order. Thanks to themill for the report. In coreboot, Alexander Couzens committed a change to make their release archives reproducible. Patches submitted

• #836609 filed against nostalgy by Chris Lamb.
• #836605 filed against torque by Chris Lamb.
• #836968 filed against erlang-p1-oauth2 by Chris Lamb.
• #836970 filed against erlang-p1-sqlite3 by Chris Lamb.
• #836817 filed against tj3 by Chris Lamb.
• #836969 filed against erlang-p1-xmlrpc by Chris Lamb.

Reviews of unreproducible packages We've been adding to our knowledge about identified issues. 3 issue types have been added:

• random_id_in_pdf_generated_by_dblatex
• captures_execution_time
• captures_home_dir

1 issue type has been updated:

• Expand comment for random_id_in_pdf_generated_by_dblatex

16 have been have updated:

• Add patch for nostalgy
• Add patch for torque
• Expand comment for babl.
• Tag pbuilder with random_id_in_pdf_generated_by_dblatex
• Tag uclibc with users_and_groups_in_tarball
• Tag realmd with different_encoding_in_html_by_docbook_xsl
• Tag mrmpi with timestamps_in_documentation_generated_by_htmldoc
• Tag dynare with leaks_path_environment_variable
• Tag sdlgfx with timestamps_in_tarball
• Tag pantomime1.2 with plist_weirdness
• Tag blockattack with different_due_to_umask
• Tag expeyes with different_pot_creation_date_in_gettext_mo_files
• Tag android-platform-frameworks-data-binding with random_ordering_in_pom
• Tag krunner with users_and_groups_in_tarball
• Tag libevocosm with leaks_path_environment_variable
• Tag gdbm with random_order_in_md5sums

13 have been removed, not including removed packages:

• htp fixed in/since 1.19-2
• Remove openni-sensor-pointclouds and openni-sensor-primesense; fixed by __DATE__ & __TIME__
• hsqldb fixed in/since 2.3.3+dfsg2-1
• linux-tools RM
• ksnapshot RM
• cobalt-panel-utils RM
• strigi RM
• libkdeedu RM
• erc RM
• ttf-atarismall RM
• gnupg-doc RM
• remove RMed django-localflavor
• easymp3gain RM

100s of packages have been tagged with the more generic captures_build_path, and many with captures_kernel_version, user_hostname_manually_added_requiring_further_investigation, user_hostname_manually_added_requiring_further_investigation, captures_shell_variable_in_autofoo_script, etc. Particular thanks to Emanuel Bronshtein for his work here. Weekly QA work FTBFS bugs have been reported by:

• Aaron M. Ucko (1)
• Chris Lamb (7)

diffoscope development

• Mattia Rizzolo:
  • Force LC_ALL=C.UTF-8 in basic-command-list autopkgtest so that diffoscope can always output something
• Ximin Luo:
  • html-dir: use 4 * the given max-diff-block-lines
  • difference: Don't truncate diff output by default, and use a separate flag when doing so
  • html-dir: In html-dir output, split large diffs across several child pages
  • Rename no-max-limits to no-default-limits, a more accurate name
  • Increase default max_diff_input_lines, should be OK based on what we've seen
  • Fix behaviour for max-diff-block-limit=1 and don't report when 0 lines are removed
  • Set limits to inf when parsing args, instead of checking for "> 0" everywhere
  • Suppress decimal line numbers for xxd output

strip-nondeterminism development

• Chris Lamb:
  • handlers/jar.pm: Reintroduce subroutine prototype on _jar_filename_cmp to fix ordering regression in .jar files.
  • bin/strip-nondeterminism: die if file does not exist (Closes: #800159)
  • t/fixtures/jar/*: Add a testcase for jar file ordering and normalisation.
  • t/fixtures.t:
    • Check that output matches expected before comparing attributes to make it less confusing on failure.
    • Don't test whether inode number has changed either.
    • Add a lookup table for descriptions of stat(1) indices.
    • Re-order @fixtures setup for global->specific ordering.
    • Re-add top-level "plan" call dropped in previous commit.
    • Align size check with atime, etc.
    • Make it clearer that we are checking for attributes
    • Add comment why we use a per-fixture temporary directory.
  • Drop all calls to unlink_on_destroy now that we copy the file. (Closes: #836670)
  • Extend testsuite to ensure no extra files are leftover again.

tests.reproducible-builds.org:

• F-Droid:
  • Hans-Christoph Steiner found after extensive debugging that for kvm-on-kvm, vagrant from stretch is needed (or a backport, but that seems harder than setting up a new VM).
• FreeBSD:
  • Holger updated the VM for testing FreeBSD to FreeBSD 10.3.

Misc. This week's edition was written by Chris Lamb and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.