Search Results: "Holger Levsen"

17 November 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, October 2021

A Debian LTS logo
Every month we review the work funded by Freexian s Debian LTS offering. Please find the report for October below. Debian project funding We re looking forward to receiving more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In October 12 contributors were paid to work on Debian LTS, their reports are available below. Evolution of the situation In October we released 34 DLAs.

Also, we would like to remark once again that we are constantly looking for new contributors. Please contact Jeremiah if you are interested! The security tracker currently lists 37 packages with a known CVE and the dla-needed.txt file has 22 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

6 November 2021

Reproducible Builds: Reproducible Builds in October 2021

Welcome to the October 2021 report from the Reproducible Builds project!
This month Samanta Navarro posted to the oss-security security mailing on a novel category of exploit in the .tar archive format, where a single .tar file contains different contents depending on the tar utility being used. Naturally, this has consequences for reproducible builds as Samanta goes onto reply:

Arch Linux uses libarchive (bsdtar) in its build environment. The default tar program installed is GNU tar. It is possible to create a source distribution which leads to different files seen by the build environment than compared to a careful reviewer and other Linux distributions.
Samanta notes that addressing the tar utilities themselves will not be a sufficient fix:
I have submitted bug reports and patches to some projects but eventually I had to conclude that the problem itself cannot be fixed by these implementations alone. The best choice for these tools would be to only allow archives which are fully compatible to standards but this in turn would render a lot of archives broken.
Reproducible builds, with its twin ideas of reaching consensus on the build outputs as well as precisely recording and describing the build environment, would help address this problem at a higher level.
Codethink announced that they had achieved ISO-26262 ASIL D Tool Certification, a way of determining specific safety standards for software. Codethink used open source tooling to achieve this, but they also leverage:
Reproducibility, repeatability and traceability of builds, drawing heavily on best-practices championed by the Reproducible Builds project.

Elsewhere on the internet, according to a comment on Hacker News, Microsoft are now comparing NPM Javascript packages with their original source repositories:
I got a PR in my repository a few days ago leading back to a team trying to make it easier for packages to be reproducible from source.

Lastly, Martin Monperrus started an interesting thread on our mailing list about Github, specifically that their autogenerated release tarballs are not deterministic . The thread generated a significant number of replies that are worth reading.

Events and presentations

Community news On our mailing list this month:
There were quite a few changes to the Reproducible Builds website and documentation this month as well, including Feng Chai updating some links on our publications page [ ] and marco updated our project metadata around the Bitcoin Core building guide [ ].
Lastly, we ran another productive meeting on IRC during October. A full set of notes from the meeting is available to view.

Distribution work Qubes was heavily featured in the latest edition of Linux Weekly News, and a significant section was dedicated to discussing reproducibility. For example, it was mentioned that the Qubes project has been working on incorporating reproducible builds into its continuous integration (CI) infrastructure . But the LWN article goes on to describe that:
The current goal is to be able to build the Qubes OS Debian templates solely from packages that can be built reproducibly. Templates in Qubes OS are VM images that can be used to start an application qube quickly based on the template. The qube will have read-only access to the root filesystem of the template, so that the same root filesystem can be shared with multiple application qubes. There are official templates for several variants of both Fedora and Debian, as well as community maintained templates for several other distributions.
You can view the whole article on LWN, and Fr d ric also published a lengthy summary about their work on reproducible builds in Qubes as well for those wishing to learn more.
In Debian this month, 133 reviews of Debian packages were added, 81 were updated and 24 were removed this month, adding to Debian s ever-growing knowledge about identified issues. A number of issues were categorised and added by Chris Lamb and Vagrant Cascadian too [ ][ ][ ]. In addition, work on alternative snapshot service has made progress by Fr d ric Pierret and Holger Levsen this month, including moving from the existing host (snapshot.notset.fr) to snapshot.reproducible-builds.org (more info) thanks to OSUOSL for the machine and hosting and Debian for the disks.
Finally, Bernhard M. Wiedemann posted his monthly reproducible builds status report.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made the following changes, including preparing and uploading versions 186, 187, 188 and 189 to Debian
  • New features:
    • Add support for Python Sphinx inventory files (usually named objects.inv on-disk). [ ]
    • Add support for comparing .pyc files. Thanks to Sergei Trofimovich for the inspiration. [ ]
    • Try some alternative suffixes (e.g. .py) to support distributions that strip or retain them. [ ][ ]
  • Bug fixes:
    • Fix Python decompilation tests under Python 3.10+ [ ] and for Python 3.7 [ ].
    • Don t raise a traceback if we cannot unmarshal Python bytecode. This is in order to support Python 3.7 failing to load .pyc files generated with newer versions of Python. [ ]
    • Skip Python bytecode testing where we do not have an expected diff. [ ]
  • Codebase improvements:
    • Use our file_version_is_lt utility instead of accepting both versions of uImage expected diff. [ ]
    • Split out a custom call to assert_diff for a .startswith equivalent. [ ]
    • Use skipif instead of manual conditionals in some tests. [ ]
In addition, Jelle van der Waa added external tool references for Arch Linux for ocamlobjinfo, openssl and ffmpeg [ ][ ][ ] and added Arch Linux as a Continuous Integration (CI) test target. [ ] and Vagrant Cascadian updated the testsuite to skip Python bytecode comparisons when file(1) is older than 5.39. [ ] as well as added external tool references for the Guix distribution for dumppdf and ppudump. [ ][ ]. Vagrant Cascadian also updated the diffoscope package in GNU Guix [ ][ ]. Lastly, Guangyuan Yang updated the FreeBSD package name on the website [ ], Mattia Rizzolo made a change to override a new Lintian warning due to the new test files [ ], Roland Clobus added support to detect and log if the GNU_BUILD_ID field in an ELF binary been modified [ ], Sandro J ckel updated a number of helpful links on the website [ ] and Sergei Trofimovich made the uImage test output support file() version 5.41 [ ].

reprotest reprotest is the Reproducible Build s project end-user tool to build same source code twice in widely differing environments, checking the binaries produced by the builds for any differences. This month, reprotest version 0.7.18 was uploaded to Debian unstable by Holger Levsen, which also included a change by Holger to clarify that Python 3.9 is used nowadays [ ], but it also included two changes by Vasyl Gello to implement realistic CPU architecture shuffling [ ] and to log the selected variations when the verbosity is configured at a sufficiently high level [ ]. Finally, Vagrant Cascadian updated reprotest to version 0.7.18 in GNU Guix.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix unreproducible packages. We try to send all of our patches upstream where appropriate. We authored a large number of such patches this month, including:

Testing framework The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
  • Holger Levsen:
    • Debian-related changes:
      • Incorporate a fix from bremner into builtin-pho related to binary-NMUs. [ ]
      • Keep bullseye environments around longer, in an attempt to fix a Jenkins issue. [ ]
      • Improve the documentation of buildinfos.debian.net. [ ]
      • Improve documentation for the builtin-pho setup. [ ][ ]
    • OpenWrt-related changes:
      • Also use -j1 for better debugging. [ ]
      • Document that that Python 3.x is now used. [ ]
      • Enable further debugging for the toolchain build. [ ]
    • New snapshot.reproducible-builds.org service:
      • Actually add new node. [ ][ ]
      • Install xfsprogs on snapshot.reproducible-builds.org. [ ]
      • Create account for fpierret on new node. [ ]
      • Run node_health_check job on new node too. [ ]
  • Mattia Rizzolo:
    • Debian-related changes:
      • Handle schroot errors when invoking diffoscope instead of masking them. [ ][ ]
      • Declare and define some variables separately to avoid masking the subshell return code. [ ]
      • Fix variable name. [ ]
      • Improve log reporting. [ ]
      • Execute apt-get update with the -q argument to get more decent logs. [ ]
      • Set the Debian HTTP mirror and proxy for snapshot.reproducible-builds.org. [ ]
      • Install the libarchive-tools package (instead of bsdtar) when updating Jenkins nodes. [ ]
    • Be stricter about errors when starting the node agent [ ] and don t overwrite NODE_NAME so that we can expect Jenkins to properly set for us [ ].
    • Explicitly warn if the NODE_NAME is not a fully-qualified domain name (FQDN). [ ]
    • Document whether a node runs in the future. [ ]
    • Disable postgresql_autodoc as it not available in bullseye. [ ]
    • Don t be so eager when deleting schroot internals, call to schroot -e to terminate the schroots instead. [ ]
    • Only consider schroot underlays for deletion that are over a month old. [ ][ ]
    • Only try to unmount /proc if it s actually mounted. [ ]
    • Move the db_backup task to its own Jenkins job. [ ]
Lastly, Vasyl Gello added usage information to the reproducible_build.sh script [ ].

Contributing If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

19 October 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, September 2021

A Debian LTS logo
Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding Folks from the LTS team, along with members of the Debian Android Tools team and Phil Morrel, have proposed work on the Java build tool, gradle, which is currently blocked due to the need to build with a plugin not available in Debian. The LTS team reviewed the project submission and it has been approved. After approval we ve created a Request for Bids which is active now. You ll hear more about this through official Debian channels, but in the meantime, if you feel you can help with this project, please submit a bid. Thanks! This September, Freexian set aside 2550 EUR to fund Debian projects. We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In September, 15 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In September we released 30 DLAs. September was also the second month of Jeremiah coordinating LTS contributors. Also, we would like say that we are always looking for new contributors to LTS. Please contact Jeremiah if you are interested! The security tracker currently lists 33 packages with a known CVE and the dla-needed.txt file has 26 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

6 October 2021

Reproducible Builds: Reproducible Builds in September 2021

The goal behind reproducible builds is to ensure that no deliberate flaws have been introduced during compilation processes via promising or mandating that identical results are always generated from a given source. This allowing multiple third-parties to come to an agreement on whether a build was compromised or not by a system of distributed consensus. In these reports we outline the most important things that have been happening in the world of reproducible builds in the past month:
First mentioned in our March 2021 report, Martin Heinz published two blog posts on sigstore, a project that endeavours to offer software signing as a public good, [the] software-signing equivalent to Let s Encrypt . The two posts, the first entitled Sigstore: A Solution to Software Supply Chain Security outlines more about the project and justifies its existence:
Software signing is not a new problem, so there must be some solution already, right? Yes, but signing software and maintaining keys is very difficult especially for non-security folks and UX of existing tools such as PGP leave much to be desired. That s why we need something like sigstore - an easy to use software/toolset for signing software artifacts.
The second post (titled Signing Software The Easy Way with Sigstore and Cosign) goes into some technical details of getting started.
There was an interesting thread in the /r/Signal subreddit that started from the observation that Signal s apk doesn t match with the source code:
Some time ago I checked Signal s reproducibility and it failed. I asked others to test in case I did something wrong, but nobody made any reports. Since then I tried to test the Google Play Store version of the apk against one I compiled myself, and that doesn t match either.

BitcoinBinary.org was announced this month, which aims to be a repository of Reproducible Build Proofs for Bitcoin Projects :
Most users are not capable of building from source code themselves, but we can at least get them able enough to check signatures and shasums. When reputable people who can tell everyone they were able to reproduce the project s build, others at least have a secondary source of validation.

Distribution work Fr d ric Pierret announced a new testing service at beta.tests.reproducible-builds.org, showing actual rebuilds of binaries distributed by both the Debian and Qubes distributions. In Debian specifically, however, 51 reviews of Debian packages were added, 31 were updated and 31 were removed this month to our database of classified issues. As part of this, Chris Lamb refreshed a number of notes, including the build_path_in_record_file_generated_by_pybuild_flit_plugin issue. Elsewhere in Debian, Roland Clobus posted his Fourth status update about reproducible live-build ISO images in Jenkins to our mailing list, which mentions (amongst other things) that:
  • All major configurations are still built regularly using live-build and bullseye.
  • All major configurations are reproducible now; Jenkins is green.
    • I ve worked around the issue for the Cinnamon image.
    • The patch was accepted and released within a few hours.
  • My main focus for the last month was on the live-build tool itself.
Related to this, there was continuing discussion on how to embed/encode the build metadata for the Debian live images which were being worked on by Roland Clobus.
Ariadne Conill published another detailed blog post related to various security initiatives within the Alpine Linux distribution. After summarising some conventional security work being done (eg. with sudo and the release of OpenSSH version 3.0), Ariadne included another section on reproducible builds: The main blocker [was] determining what to do about storing the build metadata so that a build environment can be recreated precisely . Finally, Bernhard M. Wiedemann posted his monthly reproducible builds status report.

Community news On our website this month, Bernhard M. Wiedemann fixed some broken links [ ] and Holger Levsen made a number of changes to the Who is Involved? page [ ][ ][ ]. On our mailing list, Magnus Ihse Bursie started a thread with the subject Reproducible builds on Java, which begins as follows:
I m working for Oracle in the Build Group for OpenJDK which is primary responsible for creating a built artifact of the OpenJDK source code. [ ] For the last few years, we have worked on a low-effort, background-style project to make the build of OpenJDK itself building reproducible. We ve come far, but there are still issues I d like to address. [ ]

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 183, 184 and 185 as well as performed significant triaging of merge requests and other issues in addition to making the following changes:
  • New features:
    • Support a newer format version of the R language s .rds files. [ ]
    • Update tests for OCaml 4.12. [ ]
    • Add a missing format_class import. [ ]
  • Bug fixes:
    • Don t call close_archive when garbage collecting Archive instances, unless open_archive definitely returned successfully. This prevents, for example, an AttributeError where PGPContainer s cleanup routines were rightfully assuming that its temporary directory had actually been created. [ ]
    • Fix (and test) the comparison of R language s .rdb files after refactoring temporary directory handling. [ ]
    • Ensure that RPM archives exists in the Debian package description, regardless of whether python3-rpm is installed or not at build time. [ ]
  • Codebase improvements:
    • Use our assert_diff routine in tests/comparators/test_rdata.py. [ ]
    • Move diffoscope.versions to diffoscope.tests.utils.versions. [ ]
    • Reformat a number of modules with Black. [ ][ ]
However, the following changes were also made:
  • Mattia Rizzolo:
    • Fix an autopkgtest caused by the androguard module not being in the (expected) python3-androguard Debian package. [ ]
    • Appease a shellcheck warning in debian/tests/control.sh. [ ]
    • Ignore a warning from h5py in our tests that doesn t concern us. [ ]
    • Drop a trailing .1 from the Standards-Version field as it s required. [ ]
  • Zbigniew J drzejewski-Szmek:
    • Stop using the deprecated distutils.spawn.find_executable utility. [ ][ ][ ][ ][ ]
    • Adjust an LLVM-related test for LLVM version 13. [ ]
    • Update invocations of llvm-objdump. [ ]
    • Adjust a test with a one-byte text file for file version 5.40. [ ]
And, finally, Benjamin Peterson added a --diff-context option to control unified diff context size [ ] and Jean-Romain Garnier fixed the Macho comparator for architectures other than x86-64 [ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Testing framework The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
  • Holger Levsen:
    • Drop my package rebuilder prototype as it s not useful anymore. [ ]
    • Schedule old packages in Debian bookworm. [ ]
    • Stop scheduling packages for Debian buster. [ ][ ]
    • Don t include PostgreSQL debug output in package lists. [ ]
    • Detect Python library mismatches during build in the node health check. [ ]
    • Update a note on updating the FreeBSD system. [ ]
  • Mattia Rizzolo:
    • Silence a warning from Git. [ ]
    • Update a setting to reflect that Debian bookworm is the new testing. [ ]
    • Upgrade the PostgreSQL database to version 13. [ ]
  • Roland Clobus (Debian live image generation):
    • Workaround non-reproducible config files in the libxml-sax-perl package. [ ]
    • Use the new DNS for the snapshot service. [ ]
  • Vagrant Cascadian:
    • Also note that the armhf architecture also systematically varies by the kernel. [ ]

Contributing If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

4 October 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, August 2021

A Debian LTS logo
Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In August, we put aside 2460 EUR to fund Debian projects. We received a new project proposal that got approved and there s an associated bid request if you feel like proposing yourself to implement this project. We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In August, 14 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In August we released 30 DLAs.

This is the first month of Jeremiah coordinating LTS contributors. We would like to thank Holger Levsen for his work on this role up to now.

Also, we would like to remark once again that we are constantly looking for new contributors. Please contact Jeremiah if you are interested! The security tracker currently lists 73 packages with a known CVE and the dla-needed.txt file has 29 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

30 September 2021

Holger Levsen: 20210930-Debian-Reunion-Hamburg-2021

Debian Reunion Hamburg 2021 is almost over... The Debian Reunion Hamburg 2021 is almost over now, half the attendees have already left for Regensburg, while five remaining people are still busy here, though tonight there will be two concerts at the venue, plus some lovely food and more. Together with the day trip tomorrow (involving lots of water but hopefully not from above...) I don't expect much more work to be done, so that I feel comfortable publishing the following statistics now, even though I expect some more work will be done while travelling back or due to renewed energy from the event! So I might update these numbers later :-) Together we did: I think that's a pretty awesome and am very happy we did this event! Debian Reunion / MiniDebConf Hamburg 2022 - save the date, almost! Thus I think we should have another Debian event at Fux in 2022, and after checking suitable free dates with the venue I think what could work out is an event from Monday May 23rd until Sunday May 29th 2022. What do you think? For now these dates are preliminary. If you know any reasons why these dates could be less than optimal for such an event, please let me know. Assuming there's no feedback indicating this is a bad idea, the dates shall be finalized by November 1st 2021. Obviously assuming having physical events is still and again a thing! ;-)

28 September 2021

Holger Levsen: 20210928-Debian-Reunion-Hamburg-2021

Debian Reunion Hamburg 2021, klein aber fein / small but beautiful So the Debian Reunion Hamburg 2021 has been going on for not yet 48h now and it appears people are having fun, enjoying discussions between fellow Debian people and getting some stuff done as well. I guess I'll write some more about it once the event is over... Sharing android screens... For now I just want to share one little gem I learned about yesterday on the hallway track:
$ sudo apt install scrcpy
$ scrcpy
And voila, once again I can type on my phone with a proper keyboard and copy and paste URLs between the two devices. One can even watch videos on the big screen with it :) (This requires ADB debugging enabled on the phone, but doesn't require root access.)

20 September 2021

Holger Levsen: 20210920-Debian-Reunion-Hamburg-2021

Debian Reunion Hamburg 2021, we still have free beds We still have some free slots and beds available for the "Debian Reunion Hamburg 2021" taking place in Hamburg at the venue of the 2018 & 2019 MiniDebConfs from Monday, Sep 27 2021 until Friday Oct 1 2021, with Sunday, Sep 26 2021 as arrival day. So again, Debian people will meet in Hamburg. The exact format is less defined and structured than previous years, probably we will just be hacking from Monday to Wednesday, have talks on Thursday and a nice day trip on Friday. Please read https://wiki.debian.org/DebianEvents/de/2021/DebianReunionHamburg and if you intend to attend, please register there. If additionally you would like to stay on site (in a single room or shared with one another person), please mail me. I'm looking forward to this event, even though (or maybe also because) it will be much smaller than last years. I suppose this will lead to more personal interactions and more intense hacking, though of course it is to be seen how this works out exactly!

5 September 2021

Reproducible Builds: Reproducible Builds in August 2021

Welcome to the latest report from the Reproducible Builds project. In this post, we round up the important things that happened in the world of reproducible builds in August 2021. As always, if you are interested in contributing to the project, please visit the Contribute page on our website.
There were a large number of talks related to reproducible builds at DebConf21 this year, the 21st annual conference of the Debian Linux distribution (full schedule):
PackagingCon (@PackagingCon) is new conference for developers of package management software as well as their related communities and stakeholders. The virtual event, which is scheduled to take place on the 9th and 10th November 2021, has a mission is to bring different ecosystems together: from Python s pip to Rust s cargo to Julia s Pkg, from Debian apt over Nix to conda and mamba, and from vcpkg to Spack we hope to have many different approaches to package management at the conference . A number of people from reproducible builds community are planning on attending this new conference, and some may even present. Tickets start at $20 USD.
As reported in our May report, the president of the United States signed an executive order outlining policies aimed to improve the cybersecurity in the US. The executive order comes after a number of highly-publicised security problems such as a ransomware attack that affected an oil pipeline between Texas and New York and the SolarWinds hack that affected a large number of US federal agencies. As a followup this month, however, a detailed fact sheet was released announcing a number large-scale initiatives and that will undoubtedly be related to software supply chain security and, as a result, reproducible builds.
Lastly, We ran another productive meeting on IRC in August (original announcement) which ran for just short of two hours. A full set of notes from the meeting is available.

Software development kpcyrd announced an interesting new project this month called I probably didn t backdoor this which is an attempt to be:
a practical attempt at shipping a program and having reasonably solid evidence there s probably no backdoor. All source code is annotated and there are instructions explaining how to use reproducible builds to rebuild the artifacts distributed in this repository from source. The idea is shifting the burden of proof from you need to prove there s a backdoor to we need to prove there s probably no backdoor . This repository is less about code (we re going to try to keep code at a minimum actually) and instead contains technical writing that explains why these controls are effective and how to verify them. You are very welcome to adopt the techniques used here in your projects. ( )
As the project s README goes on the mention: the techniques used to rebuild the binary artifacts are only possible because the builds for this project are reproducible . This was also announced on our mailing list this month in a thread titled i-probably-didnt-backdoor-this: Reproducible Builds for upstreams. kpcyrd also wrote a detailed blog post about the problems surrounding Linux distributions (such as Alpine and Arch Linux) that distribute compiled Python bytecode in the form of .pyc files generated during the build process.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made a number of changes, including releasing version 180), version 181) and version 182) as well as the following changes:
  • New features:
    • Add support for extracting the signing block from Android APKs. [ ]
    • If we specify a suffix for a temporary file or directory within the code, ensure it starts with an underscore (ie. _ ) to make the generated filenames more human-readable. [ ]
    • Don t include short GCC lines that differ on a single prefix byte either. These are distracting, not very useful and are simply the strings(1) command s idea of the build ID, which is displayed elsewhere in the diff. [ ][ ]
    • Don t include specific .debug-like lines in the ELF-related output, as it is invariably a duplicate of the debug ID that exists better in the readelf(1) differences for this file. [ ]
  • Bug fixes:
    • Add a special case to SquashFS image extraction to not fail if we aren t the superuser. [ ]
    • Only use java -jar /path/to/apksigner.jar if we have an apksigner.jar as newer versions of apksigner in Debian use a shell wrapper script which will be rejected if passed directly to the JVM. [ ]
    • Reduce the maximum line length for calculating Wagner-Fischer, improving the speed of output generation a lot. [ ]
    • Don t require apksigner in order to compare .apk files using apktool. [ ]
    • Update calls (and tests) for the new version of odt2txt. [ ]
  • Output improvements:
    • Mention in the output if the apksigner tool is missing. [ ]
    • Profile diffoscope.diff.linediff and specialize. [ ][ ]
  • Logging improvements:
    • Format debug-level messages related to ELF sections using the diffoscope.utils.format_class. [ ]
    • Print the size of generated reports in the logs (if possible). [ ]
    • Include profiling information in --debug output if --profile is not set. [ ]
  • Codebase improvements:
    • Clarify a comment about the HUGE_TOOLS Python dictionary. [ ]
    • We can pass -f to apktool to avoid creating a strangely-named subdirectory. [ ]
    • Drop an unused File import. [ ]
    • Update the supported & minimum version of Black. [ ]
    • We don t use the logging variable in a specific place, so alias it to an underscore (ie. _ ) instead. [ ]
    • Update some various copyright years. [ ]
    • Clarify a comment. [ ]
  • Test improvements:
    • Update a test to check specific contents of SquashFS listings, otherwise it fails depending on the test systems user ID to username passwd(5) mapping. [ ]
    • Assign seen and expected values to local variables to improve contextual information in failed tests. [ ]
    • Don t print an orphan newline when the source code formatting test passes. [ ]

In addition Santiago Torres Arias added support for Squashfs version 4.5 [ ] and Felix C. Stegerman suggested a number of small improvements to the output of the new APK signing block [ ]. Lastly, Chris Lamb uploaded python-libarchive-c version 3.1-1 to Debian experimental for the new 3.x branch python-libarchive-c is used by diffoscope.

Distribution work In Debian, 68 reviews of packages were added, 33 were updated and 10 were removed this month, adding to our knowledge about identified issues. Two new issue types have been identified too: nondeterministic_ordering_in_todo_items_collected_by_doxygen and kodi_package_captures_build_path_in_source_filename_hash. kpcyrd published another monthly report on their work on reproducible builds within the Alpine and Arch Linux distributions, specifically mentioning rebuilderd, one of the components powering reproducible.archlinux.org. The report also touches on binary transparency, an important component for supply chain security. The @GuixHPC account on Twitter posted an infographic on what fraction of GNU Guix packages are bit-for-bit reproducible: Finally, Bernhard M. Wiedemann posted his monthly reproducible builds status report for openSUSE.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: Elsewhere, it was discovered that when supporting various new language features and APIs for Android apps, the resulting APK files that are generated now vary wildly from build to build (example diffoscope output). Happily, it appears that a patch has been committed to the relevant source tree. This was also discussed on our mailing list this month in a thread titled Android desugaring and reproducible builds started by Marcus Hoffmann.

Website and documentation There were quite a few changes to the Reproducible Builds website and documentation this month, including:
  • Felix C. Stegerman:
    • Update the website self-build process to not use the buster-backports suite now that Debian Bullseye is the stable release. [ ]
  • Holger Levsen:
    • Add a new page documenting various package rebuilder solutions. [ ]
    • Add some historical talks and slides from DebConf20. [ ][ ]
    • Various improvements to the history page. [ ][ ][ ]
    • Rename the Comparison protocol documentation category to Verification . [ ]
    • Update links to F-Droid documentation. [ ]
  • Ian Muchina:
    • Increase the font size of titles and de-emphasize event details on the talk page. [ ]
    • Rename the README file to README.md to improve the user experience when browsing the Git repository in a web browser. [ ]
  • Mattia Rizzolo:
    • Drop a position:fixed CSS statement that is negatively affecting with some width settings. [ ]
    • Fix the sizing of the elements inside the side navigation bar. [ ]
    • Show gold level sponsors and above in the sidebar. [ ]
    • Updated the documentation within reprotest to mention how ldconfig conflicts with the kernel variation. [ ]
  • Roland Clobus:
    • Added a ticket number for the issue with the live Cinnamon image and diffoscope. [ ]

Testing framework The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
  • Holger Levsen:
    • Debian-related changes:
      • Make a large number of changes to support the new Debian bookworm release, including adding it to the dashboard [ ], start scheduling tests [ ], adding suitable Apache redirects [ ] etc. [ ][ ][ ][ ][ ]
      • Make the first build use LANG=C.UTF-8 to match the official Debian build servers. [ ]
      • Only test Debian Live images once a week. [ ]
      • Upgrade all nodes to use Debian Bullseye [ ] [ ]
      • Update README documentation for the Debian Bullseye release. [ ]
    • Other changes:
      • Only include rsync output if the $DEBUG variable is enabled. [ ]
      • Don t try to install mock, a tool used to build Fedora packages some time ago. [ ]
      • Drop an unused function. [ ]
      • Various documentation improvements. [ ][ ]
      • Improve the node health check to detect zombie jobs. [ ]
  • Jessica Clarke (FreeBSD-related changes):
    • Update the location and branch name for the main FreeBSD Git repository. [ ]
    • Correctly ignore the source tarball when comparing build results. [ ]
    • Drop an outdated version number from the documentation. [ ]
  • Mattia Rizzolo:
    • Block F-Droid jobs from running whilst the setup is running. [ ]
    • Enable debugging for the rsync job related to Debian Live images. [ ]
    • Pass BUILD_TAG and BUILD_URL environment for the Debian Live jobs. [ ]
    • Refactor the master_wrapper script to use a Bash array for the parameters. [ ]
    • Prefer YAML s safe_load() function over the unsafe variant. [ ]
    • Use the correct variable in the Apache config to match possible existing files on disk. [ ]
    • Stop issuing HTTP 301 redirects for things that not actually permanent. [ ]
  • Roland Clobus (Debian live image generation):
    • Increase the diffoscope timeout from 120 to 240 minutes; the Cinnamon image should now be able to finish. [ ]
    • Use the new snapshot service. [ ]
    • Make a number of improvements to artifact handling, such as moving the artifacts to the Jenkins host [ ] and correctly cleaning them up at the right time. [ ][ ][ ]
    • Where possible, link to the Jenkins build URL that created the artifacts. [ ][ ]
    • Only allow only one job to run at the same time. [ ]
  • Vagrant Cascadian:
    • Temporarily disable armhf nodes for DebConf21. [ ][ ]

Lastly, if you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. You can get in touch with us via:

1 September 2021

Holger Levsen: 20210901-Debian-Reunion-Hamburg-2021

Debian Reunion Hamburg 2021 Moin! I'm glad to finally be able to send out this invitation for the "Debian Reunion Hamburg 2021" taking place at the venue of the 2018 & 2019 MiniDebConfs! The event will run from Monday, Sep 27 2021 until Friday Oct 1 2021, with Sunday, Sep 26 2021 as arrival day. IOW, Debian people meet again in Hamburg. The exact format is less defined and structured than previous years, probably we will just be hacking from Monday to Wednesday, have talks on Thursday and a nice day trip on Friday. Please read https://wiki.debian.org/DebianEvents/de/2021/DebianReunionHamburg if you intend to attend, especially Probably having some video coverage would be very nice to have, though due to this very late announcement I'm not sure we'll really have talks and the need for video. The event is in 3.5 weeks and will take place, either as a very small hack meeting, or somewhat bigger. We certainly want videoing if we have talks - and if you could help with this that would be very great! Last and definitly not least, financial sponsors for the event would be great. If you can support the "Debian Reunion Hamburg 2021", please contact me directly! Now, late, after weeks of wondering if and how to do this event, I'm finally and very much looking forward to it, to meet some Debian folks at least & for some shared Debian hacking. Definitly not the 2021 event I had in mind after the 2019 one, but something I feel I can responsibly do & enjoy. So, hoping to see some of you soon & most of you later! ;-) Sad but true, and at least something for some people. We should all do more local events. And more online events too, eg I think this is a great idea too: https://wiki.debian.org/DebianEvents/internet/2021/MiniDebConfOnlineBookworm See you!

25 August 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, July 2021

A Debian LTS logo
Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In July, we put aside 2400 EUR to fund Debian projects. We haven t received proposals of projects to fund in the last months, so we have scheduled a discussion during Debconf to try to to figure out why that is and how we can fix that. Join us on August 26th at 16:00 UTC on this link. We are pleased to announce that Jeremiah Foster will help out to make this initiative a success : he can help Debian members to come up with solid proposals, he can look for people willing to do the work once the project has been formalized and approved, and he will make sure that the project implementation keeps on track when the actual work has begun. We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In July, 12 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In July we released 30 DLAs. Also we were glad to welcome Neil Williams and Lee Garrett who became active contributors. The security tracker currently lists 63 packages with a known CVE and the dla-needed.txt file has 17 packages needing an update. We would like to thank Holger Levsen for the years of work where he managed/coordinated the paid LTS contributors. Jeremiah Foster will take over his duties. Thanks to our sponsors Sponsors that joined recently are in bold.

17 July 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, June 2021

A Debian LTS logo
Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In June, we put aside 5775 EUR to fund Debian projects for which we re looking forward to receive more projects from various
Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In June, 12 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In June we released 30 DLAs. As already written last month we are looking for a Debian LTS project manager and team coordinator.
Finally, we would like to remark once again that we are constantly looking for new contributors. Please contact Holger if you are interested! The security tracker currently lists 41 packages with a known CVE and the dla-needed.txt file has 23 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

15 June 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, May 2021

A Debian LTS logo
Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In May, we again put aside 2100 EUR to fund Debian projects. There was no proposals for new projects received, thus we re looking forward to receive more projects from various Debian teams! Please do not hesitate to submit a proposal, if there is a project that could benefit from the funding! We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In May, 12 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In May we released 33 DLAs and mostly skipped our public IRC meeting and the end of the month. In June we ll have another team meeting using video as lined out on our LTS meeting page.
Also, two months ago we announced that Holger would step back from his coordinator role and today we are announcing that he is back for the time being, until a new coordinator is found.
Finally, we would like to remark once again that we are constantly looking for new contributors. Please contact Holger if you are interested! The security tracker currently lists 41 packages with a known CVE and the dla-needed.txt file has 21 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

28 May 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, April 2021

A Debian LTS logo
Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In April, we put aside 5775 EUR to fund Debian projects. There was no proposals for new projects received, thus we re looking forward to receive more projects from various Debian teams! Please do not hesitate to submit a proposal, if there is a project that could benefit from the funding! Debian LTS contributors In April, 11 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In April we released 33 DLAs and held a LTS team meeting using video conferencing. The security tracker currently lists 53 packages with a known CVE and the dla-needed.txt file has 26 packages needing an update. We are please to welcome VyOS as a new gold sponsor! Thanks to our sponsors Sponsors that joined recently are in bold.

30 April 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, March 2021

A Debian LTS logo
Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In March, we put aside 3225 EUR to fund Debian projects but sadly nobody picked up anything, so this one of the many reasons Raphael posted as series of blog posts titled Challenging times for Freexian , posted in 4 parts on the last two days of March and the first two of April. [Part one, two, three and four] So we re still looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article! Debian LTS contributors In March, 11 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In March we released 28 DLAs and held our second LTS team meeting for 2021 on IRC, with the next public IRC meeting coming up at the end of May. At that meeting Holger announced that after 2.5 years he wanted to step back from his role helping Rapha l in coordinating/managing the LTS team. We would like to thank Holger for his continuous work on Debian LTS (which goes back to 2014) and are happy to report that we already found a successor which we will introduce in the upcoming April report from Freexian. Finally, we would like to remark once again that we are constantly looking for new contributors. For a last time, please contact Holger if you are interested! The security tracker currently lists 42 packages with a known CVE and the dla-needed.txt file has 28 packages needing an update. We are also pleased to report that we got 4 new sponsors over the last 2 months : thanks to sipgate GmbH, OVH US LLC, Tilburg University and Observatoire des Sciences de l Univers de Grenoble ! Thanks to our sponsors Sponsors that joined recently are in bold.

22 March 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, February 2020

A Debian LTS logo
Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In February, we put aside 5475 EUR to fund Debian projects. The first project from this initiative was finished and thus Carles Pina was able to issue the first invoice! We are looking forward to receive more projects from various Debian teams and contributors. Learn more about the rationale behind this initiative in this article. Debian LTS contributors In February, 12 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In February we released 28 DLAs (including one regression update) and we held an internal team meeting using video chat.
Finally, as every month we would like to remark once again that we are constantly looking for new contributors. Please contact Holger if you are interested! The security tracker currently lists 46 packages with a known CVE and the dla-needed.txt file has 34 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

21 February 2021

Louis-Philippe V ronneau: dput-ng or: How I Learned to Stop Worrying and Love the Hooks

As my contributions to Debian continue to grow in number, I find myself uploading to the archive more and more often. Although I'm pretty happy with my current sbuild-based workflow, twice in the past few weeks I inadvertently made a binary upload instead of a source-only one.1 As it turns out, I am not the only DD who has had this problem before. As Nicolas Dandrimont kindly pointed to me, dput-ng supports pre and post upload hooks that can be used to lint your uploads. Even better, it also ships with a check-debs hook that lets you block binary uploads. Pretty neat, right? In a perfect world, enabling the hook would only be a matter of adding it in the hook list of /etc/dput.d/metas/debian.json and using the following defaults:
"check-debs":  
    "enforce": "source",
    "skip": false
 ,
Sadly, bug #983160 currently makes this whole setup more complex than it should be and forces me to use two different dput-ng profiles pointing to two different files in /etc/dput.d/metas: a default source-only one (ftp-master) and a binary upload one (ftp-master-binary). Otherwise, one could use a single profile that disallows binary uploads and when needed, override the hook using something like this:
$ dput --override "check-debs.enforce=debs" foo_1.0.0-1_amd64.changes
I did start debugging the --override issue in dput-ng, but I'm not sure I'll have time to submit a patch anytime soon. In the meantime, I'm happy to report I shouldn't be uploading the wrong .changes file by mistake again!

  1. Thanks to Holger Levsen and Adrian Bunk for catching those and notifying me.

15 February 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, January 2020

A Debian LTS logo Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In January, we put aside 2175 EUR to fund Debian projects. As part of this Carles Pina i Estany started to work on better no-dsa support for the PTS which recently resulted in two merge requests which will hopefully be deployed soon. We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In January, 13 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation In January we released 28 DLAs and held our first LTS team meeting for 2021 on IRC, with the next public IRC meeting coming up at the end of March. During that meeting Utkarsh shared that after he rolled out the python-certbot update (on December 8th 2020) the maintainer told him: I just checked with Let s Encrypt, and the stats show that you just saved 142,500 people from having their certificates start failing next month. I didn t know LTS was still that used!

Finally, we would like to welcome sipgate GmbH as a new silver sponsor. Also remember that we are constantly looking for new contributors. Please contact Holger if you are interested. The security tracker currently lists 43 packages with a known CVE and the dla-needed.txt file has 23 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

20 January 2021

Raphaël Hertzog: Freexian s report about Debian Long Term Support, December 2020

A Debian LTS logo Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In December, we put aside 2100 EUR to fund Debian projects. The first project proposal (a tracker.debian.org improvement for the security team) was received and quickly approved by the paid contributors, then we opened a request for bids and the bid winner was announced today (it was easy, we had only one candidate). Hopefully this first project will be completed until our next report. We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In December, 12 contributors have been paid to work on Debian LTS, their reports are available: Evolution of the situation December was a quiet month as we didn t have a team meeting nor any other unusual activity and we released 43 DLAs. The security tracker currently lists 30 packages with a known CVE and the dla-needed.txt file has 25 packages needing an update. This month we are pleased to welcome Deveryware as new sponsor! Thanks to our sponsors Sponsors that joined recently are in bold.

31 December 2020

Holger Levsen: 20201231-no-source-change-source-uploads

On doing 540 no-source-change source-only uploads in two weeks So I've been doing 540 no-source-change source-only uploads in the last two weeks and am planning to do 3000 more in January 2021. We'll see how that goes ;) Let me explain what I have been doing and why. So, starting with the Bullseye release cycle the Release Team changed policy: only packages which were build on buildds are allowed to migrate to testing. Which is pretty nice for reproducible builds as this also ensures that a .buildinfo file is available for anyone wanting to reproduce the binaries of that package. However, there are many binary (and source) packages in Debian which were uploaded before 2016 (which is when .buildinfo files were introduced) or were uploaded with binaries until that change in release policy July 2019. Then Ivo De Decker scheduled binNMUs for all the affected packages but due to the way binNMUs work, he couldn't do anything about arch:all packages as they currently cannot be rebuilt with binNMUs. Ivo and myself discussed what could be done about the remaining packages and (besides long complicated changes to Debian's workflows) the only thing deemed possible was doing many many source uploads with just a new changelog entry:
  * Non maintainer upload by the Reproducible Builds team.
  * No source change upload to rebuild on buildd with .buildinfo files.
These packages are all inherently buggy, because Debian policy mandates that packages should be reproducible and without .buildinfo files one cannot reproducibly rebuild packages. So instead of filing many many bugs we've decided to just fix these bugs by doing a no-source-change source uploads. One nice aspect of these uploads is that there's no follow-up work imposed on the maintainer: whether they keep that changelog entry or whether they discard it, it does not matter. So Ivo had developed an SQL query which showed 570 packages needing an update roughly two weeks ago, on December 18 and so I started slowly. This is the amount of NMUs I did in the last days:
for i in $(seq 18 30) ; do echo -n "Dec $i: " ; ls -lart1 done/*upload grep -c "Dec $i" ; done
Dec 18: 12
Dec 19: 0
Dec 20: 3
Dec 21: 13
Dec 22: 13
Dec 23: 16
Dec 24: 4
Dec 25: 28
Dec 26: 0
Dec 27: 38
Dec 28: 198
Dec 29: 206
Dec 30: 9
About ten packages had FTBFS bugs preventing an upload and seven packages were uploaded by the maintainer before me. I've seen two cases of sudden maintainer uploads after 8 and 10 years of no activity! So what did I do for each upload? Much to my surprise I didn't get much feedback, there were like 6 people on the #debian-reproducible channel cheering and one on #debian-qa, though that person is a Release Team member so that was kind of important cheering. And I've seen some maintainer uploads to packages which haven't seen uploads since some years. And really nice: no-one complained so far. I hope this will stay this way with the plan to do 3000 more uploads of this kind: Those 570 packages were only key packages but there are 3000 more source packages which have a binary in bullseye for which no .buildinfo file exists. So I plan to upload them all in January 2021 and you can help me doing so, by uploading your packages before me - and maybe fixing some other bugs in the process! I've posted the list of packages (sorted by ddlist) to debian-devel@lists.d.o, see https://lists.debian.org/debian-devel/2020/12/msg00419.html. Many thanks to Ivo and the whole Release Team for their support of Reproducible Builds and generally speaking for the many many enhancements to the release process we've seen over the years. Debian in 2021 will rock'n'roll more than ever! So thank you all, once again, for making Debian what it is and what it will be!

Next.