What is HMAC? HMAC stands for Hash-Based Message Authentication Code. It s a specific way to use a cryptographic hash function (like SHA-1, SHA-256, etc.) along with a secret key to produce a unique fingerprint of some data. This fingerprint allows someone else with the same key to verify that the data hasn t been tampered with. How HMAC Works Keyed Hashing: The core idea is to incorporate the secret key into the hashing process. This is done in a specific way to prevent clever attacks that might try to bypass the security.
Inner and Outer Hashing: HMAC uses two rounds of hashing. First, the message and a modified version of the key are hashed together. Then, the result of that hash, along with another modified version of the key, are hashed again. This two-step process adds an extra layer of protection. HMAC in OpenSSH OpenSSH uses HMAC to ensure the integrity of messages sent back and forth during an SSH session. This prevents an attacker from subtly modifying data in transit. HMAC-SHA1 with OpenSSH: Is it Weak? SHA-1 itself is considered cryptographically broken. This means that with enough computing power, it s possible to find collisions (two different messages that produce the same hash). However, HMAC-SHA1 is generally still considered secure for most purposes. This is because exploiting weaknesses in SHA-1 to break HMAC-SHA1 is much more difficult than just finding collisions in SHA-1. Should you use it? While HMAC-SHA1 might still be okay for now, it s best practice to move to stronger alternatives like HMAC-SHA256 or HMAC-SHA512. OpenSSH supports these, and they provide a greater margin of safety against future attacks. In Summary HMAC is a powerful tool for ensuring data integrity. Even though SHA-1 has weaknesses, HMAC-SHA1 in OpenSSH is likely still safe for most users. However, to be on the safe side and prepare for the future, switching to HMAC-SHA256 or HMAC-SHA512 is recommended. Following are instructions for creating dataproc clusters with sha1 mac support removed: I can appreciate an excess of caution, and I can offer you some code to produce Dataproc instances which do not allow HMAC authentication using sha1. Place code similar to this in a startup script or an initialization action that you reference when creating a cluster with gcloud dataproc clusters create:

#!/bin/bash
# remove mac specification from sshd configuration
sed -i -e 's/^macs.*$//' /etc/ssh/sshd_config
# place a new mac specification at the end of the service configuration
ssh -Q mac   perl -e \
  '@mac=grep  chomp; ! /sha1/  ; print("macs ", join(",",@mac), $/)' >> /etc/ssh/sshd_config
# reload the new ssh service configuration
systemctl reload ssh.service

If this code is hosted on GCS, you can refer to it with

--initialization-actions=CLOUD_STORAGE_URI,[...]

or

--metadata startup-script-url=CLOUD_STORAGE_URI,[...]

Welcome to the December 2024 report from the Reproducible Builds project! Our monthly reports outline what we ve been up to over the past month and highlight items of news from elsewhere in the world of software supply-chain security when relevant. As ever, however, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. Table of contents:

1. reproduce.debian.net
2. debian-repro-status
3. On our mailing list
4. Enhancing the Security of Software Supply Chains
5. diffoscope
6. Supply-chain attack in the Solana ecosystem
7. Website updates
8. Debian changes
9. Other development news
10. Upstream patches
11. Reproducibility testing framework

---

reproduce.debian.net Last month saw the introduction of reproduce.debian.net. Announced at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project. rebuilderd is our server designed monitor the official package repositories of Linux distributions and attempts to reproduce the observed results there. This month, however, we are pleased to announce that not only does the service now produce graphs, the reproduce.debian.net homepage itself has become a start page of sorts, and the amd64.reproduce.debian.net and i386.reproduce.debian.net pages have emerged. The first of these rebuilds the amd64 architecture, naturally, but it also is building Debian packages that are marked with the no architecture label, all. The second builder is, however, only rebuilding the i386 architecture. Both of these services were also switched to reproduce the Debian trixie distribution instead of unstable, which started with 43% of the archive rebuild with 79.3% reproduced successfully. This is very much a work in progress, and we ll start reproducing Debian unstable soon. Our i386 hosts are very kindly sponsored by Infomaniak whilst the amd64 node is sponsored by OSUOSL thank you! Indeed, we are looking for more workers for more Debian architectures; please contact us if you are able to help.

debian-repro-status Reproducible builds developer kpcyrd has published a client program for reproduce.debian.net (see above) that queries the status of the locally installed packages and rates the system with a percentage score. This tool works analogously to arch-repro-status for the Arch Linux Reproducible Builds setup. The tool was packaged for Debian and is currently available in Debian trixie: it can be installed with apt install debian-repro-status.

On our mailing list On our mailing list this month:

• Bernhard M. Wiedemann wrote a detailed post on his long journey towards a bit-reproducible Emacs package. In his interesting message, Bernhard goes into depth about the tools that they used and the lower-level technical details of, for instance, compatibility with the version for glibc within openSUSE.
• Shivanand Kunijadar posed a question pertaining to the reproducibility issues with encrypted images. Shivanand explains that they must use a random IV for encryption with AES CBC. The resulting artifact is not reproducible due to the random IV used. The message resulted in a handful of replies, hopefully helpful!
• User Danilo posted an in interesting question related to their attempts in trying to achieve reproducible builds for Threema Desktop 2.0. The question resulted in a number of replies attempting to find the right combination of compiler and linker flags (for example).
• Longstanding contributor David A. Wheeler wrote to our list announcing the release of the Census III of Free and Open Source Software: Application Libraries report written by Frank Nagle, Kate Powell, Richie Zitomer and David himself. As David writes in his message, the report attempts to answer the question what is the most popular Free and Open Source Software (FOSS)? .
• Lastly, kpcyrd followed-up to a post from September 2024 which mentioned their desire for someone to implement a hashset of allowed module hashes that is generated during the kernel build and then embedded in the kernel image , thus enabling a deterministic and reproducible build. However, they are now reporting that somebody implemented the hash-based allow list feature and submitted it to the Linux kernel mailing list . Like kpcyrd, we hope it gets merged.

Enhancing the Security of Software Supply Chains: Methods and Practices Mehdi Keshani of the Delft University of Technology in the Netherlands has published their thesis on Enhancing the Security of Software Supply Chains: Methods and Practices . Their introductory summary first begins with an outline of software supply chains and the importance of the Maven ecosystem before outlining the issues that it faces that threaten its security and effectiveness . To address these:

First, we propose an automated approach for library reproducibility to enhance library security during the deployment phase. We then develop a scalable call graph generation technique to support various use cases, such as method-level vulnerability analysis and change impact analysis, which help mitigate security challenges within the ecosystem. Utilizing the generated call graphs, we explore the impact of libraries on their users. Finally, through empirical research and mining techniques, we investigate the current state of the Maven ecosystem, identify harmful practices, and propose recommendations to address them.

A PDF of Mehdi s entire thesis is available to download.

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 283 and 284 to Debian:

• Update copyright years. [ ]
• Update tests to support file 5.46. [ ][ ]
• Simplify tests_quines.py::test_ differences,differences_deb to simply use assert_diff and not mangle the test fixture. [ ]

Supply-chain attack in the Solana ecosystem A significant supply-chain attack impacted Solana, an ecosystem for decentralised applications running on a blockchain. Hackers targeted the @solana/web3.js JavaScript library and embedded malicious code that extracted private keys and drained funds from cryptocurrency wallets. According to some reports, about $160,000 worth of assets were stolen, not including SOL tokens and other crypto assets.

Website updates Similar to last month, there was a large number of changes made to our website this month, including:

• Chris Lamb:
  • Make the landing page hero look nicer when the vertical height component of the viewport is restricted, not just the horizontal width.
  • Rename the Buy-in page to Why Reproducible Builds? [ ]
  • Removing the top black border. [ ][ ]
• Holger Levsen:
  • Fixed a number of issues on the 2024 Summit page, including fixing the path to a sponsor logo [ ] but also added the event documentation from Aspiration [ ].
  • Check and cleanup a presentation formerly linked from the About page on the Debian wiki. [ ]
  • Link to reproduce.debian.net on the Involved Projects page. [ ]
  • Fix a number of links on the Talks & Resources page. [ ][ ][ ][ ]
• hulkoba:
  • Remove the sidebar-type layout and move to a static navigation element. [ ][ ][ ][ ]
  • Create and merge a new Success stories page, which highlights the success stories of Reproducible Builds, showcasing real-world examples of projects shipping with verifiable, reproducible builds. These stories aim to enhance the technical resilience of the initiative by encouraging community involvement and inspiring new contributions. . [ ]
  • Further changes to the homepage. [ ]
  • Remove the translation icon from the navigation bar. [ ]
  • Remove unused CSS styles pertaining to the sidebar. [ ]
  • Add sponsors to the global footer. [ ]
  • Add extra space on large screens on the Who page. [ ]
  • Hide the side navigation on small screens on the Documentation pages. [ ]

Debian changes There were a significant number of reproducibility-related changes within Debian this month, including:

• Santiago Vila uploaded version 0.11+nmu4 of the dh-buildinfo package. In this release, the dh_buildinfo becomes a no-op ie. it no longer does anything beyond warning the developer that the dh-buildinfo package is now obsolete. In his upload, Santiago wrote that We still want packages to drop their [dependency] on dh-buildinfo, but now they will immediately benefit from this change after a simple rebuild.
• Holger Levsen filed Debian bug #1091550 requesting a rebuild of a number of packages that were built with a very old version of dpkg.
• Fay Stegerman contributed to an extensive thread on the debian-devel development mailing list on the topic of Supporting alternative zlib implementations . In particular, Fay wrote about her results experimenting whether zlib-ng produces identical results or not.
• kpcyrd uploaded a new rust-rebuilderd-worker, rust-derp, rust-in-toto and debian-repro-status to Debian, which passed successfully through the so-called NEW queue.
• Gioele Barabucci filed a number of bugs against the debrebuild component/script of the devscripts package, including:
  • #1089087: Address a spurious extra subdirectory in the build path.
  • #1089201: Extra zero bytes added to .dynstr when rebuilding CMake projects.
  • #1089088: Some binNMUs have a 1-second offset in some timestamps.
• Gioele Barabucci also filed a bug against the dh-r package to report that the Recommends and Suggests fields are missing from rebuilt R packages. At the time of writing, this bug has no patch and needs some help to make over 350 binary packages reproducible.
• Lastly, 8 reviews of Debian packages were added, 11 were updated and 11 were removed this month adding to our knowledge about identified issues.

Other development news In other ecosystem and distribution news:

• Jan-Benedict Glaw published the 6th NetBSD Reproducibility Report and reported on our mailing list as well.
• Developer unmush wrote a long post on the GNU Guix blog on the topic of Adding a fully-bootstrapped Mono to the distribution.
• The Glasgow Haskell Compiler (GHC) has released a new version of their compiler. This release introduces a new experimental flag, -fobject-determinism, which enables deterministic object code generation .
• The IzzyOnDroid Android APK repository published an extensive Review of 2024 and Outlook for 2025 which includes statistics and future plans related to reproducible builds (including having passed the 30% mark this month).
• The historic Arch Linux reproducibility tests that were hosted at tests.reproducible-builds.org/archlinux now redirect to reproducible.archlinux.org instead. In fact, everything Arch-related has now been removed from the jenkins.debian.net.git repository, as those continuous integration tests have been disabled for some time.
• reprotest version 0.7.29 was uploaded to Debian unstable by Vagrant Cascadian. It included contributions already covered in previous months as well as new ones from Rebecca N. Palmer, such as:
  • Stop using pkg_resources. [ ][ ]
  • The as_file attribute is not a method. [ ]
  • Use a non-constant object to test memory address capture. [ ]
• rebuilderd was updated as follows by kpcyrd:
  • Migrate diesel dependency from 1.x to 2.x. [ ]
  • Migrate clap dependency from 2 to 4. [ ]
  • Refactor reqwest code, and the replace openssl dependency with the memory-safe rustls. [ ][ ]

• Lastly, in openSUSE, Bernhard M. Wiedemann published another report for the distribution. There, Bernhard reports about the success of building R-B-OS , a partial fork of openSUSE with only 100% bit-reproducible packages. This effort was sponsored by the NLNet NGI0 initiative.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann: cargo-packaging/rusty_v8, cockpit, collectd, deepin-daemon, deepin-file-manager, esbuild, grpc, hyperkitty, icedtea-web, java-atk-wrapper, kdenetwork-filesharing, kicad, kompare, librespeed-cli, lincity-ng, mraa, ollama, opa-fmgui, opencryptoki, opencryptoki, openmpi4:gnu-hpc, openwsman, patterns-microos, portmidi, presage, procps, sad, scons/nst, sendmail, static-initrd, suse-hpc, swtpm, tiny, vtk, xdg-desktop-portal and yast.
• Chris Lamb:
  • #1089011 filed against pyorbital.
  • #1089095 filed against python-pbcore.
• Gioele Barabucci:
  • #1089088 filed against sbuild.
• James Addison:
  • #1090395 filed against binutils.
• Johannes Schauer Marin Rodrigues:
  • #1089092 filed against hurd.
• Moritz Schlarb:
  • #1090078 filed against firehol.
• Roland Clobus:
  • #1090981 filed against dictionaries-common.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In November, a number of changes were made by Holger Levsen, including:

• reproduce.debian.net-related:
  • Add a new i386.reproduce.debian.net rebuilder. [ ][ ][ ][ ][ ][ ]
  • Make a number of updates to the documentation. [ ][ ][ ][ ][ ]
  • Run i386.reproduce.debian.net run on a public port to allow external workers. [ ]
  • Add a link to the /api/v0/pkgs/list endpoint. [ ]
  • Add support for a statistics page. [ ][ ][ ][ ][ ][ ]
  • Limit build logs to 20 MiB and diffoscope output to 10 MiB. [ ]
  • Improve the frontpage. [ ][ ]
  • Explain that we re testing arch:any and arch:all on the amd64 architecture, but only arch:any on i386. [ ]
• Misc:
  • Remove code for testing Arch Linux, which has moved to reproduce.archlinux.org. [ ][ ]
  • Don t install dstat on Jenkins nodes anymore as its been removed from Debian trixie. [ ]
  • Prepare the infom08-i386 node to become another rebuilder. [ ]
  • Add debug date output for benchmarking the reproducible_pool_buildinfos.sh script. [ ]
  • Install installation-birthday everywhere. [ ]
  • Temporarily disable automatic updates of pool links on buildinfos.debian.net. [ ]
  • Install Recommends by default on Jenkins nodes. [ ]
  • Rename rebuilder_stats.py to rebuilderd_stats.py. [ ]
  • r.d.n/stats: minor formatting changes. [ ]
  • Install files under /etc/cron.d/ with the correct permissions. [ ]

and Jochen Sprickerhof made the following changes:

• Always prefer official .buildinfo on buildinfos.debian.net files. [ ][ ][ ]
• Add a rebuilder_stats.py scripts. [ ]

Lastly, Gioele Barabucci also classified packages affected by 1-second offset issue filed as Debian bug #1089088 [ ][ ][ ][ ], Chris Hofstaedtler updated the URL for Grml s dpkg.selections file [ ], Roland Clobus updated the Jenkins log parser to parse warnings from diffoscope [ ] and Mattia Rizzolo banned a number of bots and crawlers from the service [ ][ ].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org
• Twitter: @ReproBuilds

Welcome to the 44th post in the $R^4 series. A few weeks ago, and following an informal call for talks by James Lamb, I had an opportunity to talk about r2u to the Chicago ML and MLops meetup groups. You can find the slides here. Over the last 2 1/2 years, r2u has become a widely-deployed mechanism in a number of settings, including (but not limited to) software testing via continuous integration, deployment on cloud servers besides of course to more standard use on local laptops or workstation. 30 million downloads illustrate this. My thesis for the talk was that this extends equally to ML(ops) where no surprises, no hickups automated deployments are key for large-scale model training, evaluation and of course production deployments. In this context, I introduce r2u while giving credit both to what came before it, the existing alternatives (or competitors for mindshare if one prefers that phrasing), and of course what lies underneath it. The central takeaway, I argue, is that r2u can and does take advantage of a unique situation in that we can join the package manager task for the underlying (operating) system and and the application domain, here R and its unique CRAN repository network. Other approaches can, and of course do, provide binaries, but by doing this outside the realm of the system package manager can only arrive at a lesser integration (and I show a common error arising in that case). So where r2u is feasible, it dominates the alternatives (while the alternatives may well provide deployment on more platforms which, even when less integrated, may be of greater importance for some). As always, it all depends. But the talk, and its slides, motivate and illustrate why we keep calling r2u by its slogan of r2u: Fast. Easy. Reliable. Pick All Three.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub. Please report excessive re-aggregation in third-party for-profit settings.

Welcome to the November 2024 report from the Reproducible Builds project! Our monthly reports outline what we ve been up to over the past month and highlight items of news from elsewhere in the world of software supply-chain security where relevant. As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. Table of contents:

1. Reproducible Builds mourns the passing of Lunar
2. Introducing reproduce.debian.net
3. New landing page design
4. SBOMs for Python packages
5. Debian updates
6. Reproducible builds by default in Maven 4
7. PyPI now supports digital attestations
8. Dependency Challenges in OSS Package Registries
9. Zig programming language demonstrated reproducible
10. Website updates
11. Upstream patches
12. Misc development news
13. Reproducibility testing framework

---

Reproducible Builds mourns the passing of Lunar The Reproducible Builds community sadly announced it has lost its founding member, Lunar. J r my Bobbio aka Lunar passed away on Friday November 8th in palliative care in Rennes, France. Lunar was instrumental in starting the Reproducible Builds project in 2013 as a loose initiative within the Debian project. He was the author of our earliest status reports and many of our key tools in use today are based on his design. Lunar s creativity, insight and kindness were often noted. You can view our full tribute elsewhere on our website. He will be greatly missed.

Introducing reproduce.debian.net In happier news, this month saw the introduction of reproduce.debian.net. Announced at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project. rebuilderd is our server designed monitor the official package repositories of Linux distributions and attempts to reproduce the observed results there. In November, reproduce.debian.net began rebuilding Debian unstable on the amd64 architecture, but throughout the MiniDebConf, it had attempted to rebuild 66% of the official archive. From this, it could be determined that it is currently possible to bit-for-bit reproduce and corroborate approximately 78% of the actual binaries distributed by Debian that is, using the .buildinfo files hosted by Debian itself. reproduce.debian.net also contains instructions how to setup one s own rebuilderd instance, and we very much invite everyone with a machine to spare to setup their own version and to share the results. Whilst rebuilderd is still in development, it has been used to reproduce Arch Linux since 2019. We are especially looking for installations targeting Debian architectures other than i386 and amd64.

New landing page design As part of a very productive partnership with the Sovereign Tech Fund and Neighbourhoodie, we are pleased to unveil our new homepage/landing page. We are very happy with our collaboration with both STF and Neighbourhoodie (including many changes not directly related to the website), and look forward to working with them in the future.

SBOMs for Python packages The Python Software Foundation has announced a new cross-functional project for SBOMs and Python packages . Seth Michael Larson writes that the project is specifically looking to solve these issues :

• Enable Python users that require SBOM documents (likely due to regulations like CRA or SSDF) to self-serve using existing SBOM generation tools.
• Solve the phantom dependency problem, where non-Python software is bundled in Python packages but not recorded in any metadata. This makes the job of software composition analysis (SCA) tools difficult or impossible.
• Make the adoption work by relevant projects such as build backends, auditwheel-esque tools, as minimal as possible. Empower users who are interested in having better SBOM data for the Python projects they are using to be able to contribute engineering time towards that goal.

A GitHub repository for the initiative is available, and there are a number of queries, comments and remarks on Seth s Discourse forum post.

Debian updates There was significant development within Debian this month. Firstly, at the recent MiniDebConf in Toulouse, France, Holger Levsen gave a Debian-specific talk on rebuilding packages distributed from ftp.debian.org that is to say, how to reproduce the results from the official Debian build servers: Holger described the talk as follows:

For more than ten years, the Reproducible Builds project has worked towards reproducible builds of many projects, and for ten years now we have build Debian packages twice with maximal variations applied to see if they can be build reproducible still. Since about a month, we ve also been rebuilding trying to exactly match the builds being distributed via ftp.debian.org. This talk will describe the setup and the lessons learned so far, and why the results currently are what they are (spoiler: they are less than 30% reproducible), and what we can do to fix that.

The Debian Project Leader, Andreas Tille, was present at the talk and remarked later in his Bits from the DPL update that:

It might be unfair to single out a specific talk from Toulouse, but I d like to highlight the one on reproducible builds. Beyond its technical focus, the talk also addressed the recent loss of Lunar, whom we mourn deeply. It served as a tribute to Lunar s contributions and legacy. Personally, I ve encountered packages maintained by Lunar and bugs he had filed. I believe that taking over his packages and addressing the bugs he reported is a meaningful way to honor his memory and acknowledge the value of his work.

Holger s slides and video in .webm format are available.
Next, rebuilderd is the server to monitor package repositories of Linux distributions and attempt to reproduce the observed results. This month, version 0.21.0 released, most notably with improved support for binNMUs by Jochen Sprickerhof and updating the rebuilderd-debian.sh integration to the latest debrebuild version by Holger Levsen. There has also been significant work to get the rebuilderd package into the Debian archive, in particular, both rust-rebuilderd-common version 0.20.0-1 and rust-rust-lzma version 0.6.0-1 were packaged by kpcyrd and uploaded by Holger Levsen. Related to this, Holger Levsen submitted three additional issues against rebuilderd as well:

• rebuildctl should be more verbose when encountering issues. [ ]
• Please add an option to used randomised queues. [ ]
• Scheduling and re-scheduling multiple packages at once. [ ]

and lastly, Jochen Sprickerhof submitted one an issue requested that rebuilderd downloads the source package in addition to the .buildinfo file [ ] and kpcyrd also submitted and fixed an issue surrounding dependencies and clarifying the license [ ]
Separate to this, back in 2018, Chris Lamb filed a bug report against the sphinx-gallery package as it generates unreproducible content in various ways. This month, however, Dmitry Shachnev finally closed the bug, listing the multiple sub-issues that were part of the problem and how they were resolved.
Elsewhere, Roland Clobus posted to our mailing list this month, asking for input on a bug in Debian s ca-certificates-java package. The issue is that the Java key management tools embed timestamps in its output, and this output ends up in the /etc/ssl/certs/java/cacerts file on the generated ISO images. A discussion resulted from Roland s post suggesting some short- and medium-term solutions to the problem.
Holger Levsen uploaded some packages with reproducibility-related changes:

• devscripts versions 2.24.3, 2.24.4 and 2.24.5 were uploaded, including several fixes for the debrebuild and debootsnap and scripts.
• cdbs version 0.4.167 uploaded in order to drop dh_buildinfo support, as dpkg has generated .buildinfo files since 2016 and the results of dh_buildinfo are typically unreproducible. Related to this a mass bug filing by Helmut Grohne intended to remove the obsolete and deprecated dh-buildinfo package from the archive. At the time of writing, this still affects 311 packages in Debian unstable.

Lastly, 12 reviews of Debian packages were added, 5 were updated and 21 were removed this month adding to our knowledge about identified issues in Debian.

Reproducible builds by default in Maven 4 On our mailing list this month, Herv Boutemy reported the latest release of Maven (4.0.0-beta-5) has reproducible builds enabled by default. In his mailing list post, Herv mentions that this story started during our Reproducible Builds summit in Hamburg , where he created the upstream issue that builds on a multi-year effort to have Maven builds configured for reproducibility.

PyPI now supports digital attestations Elsewhere in the Python ecosystem and as reported on LWN and elsewhere, the Python Package Index (PyPI) has announced that it has finalised support for PEP 740 ( Index support for digital attestations ). Trail of Bits, who performed much of the development work, has an in-depth blog post about the work and its adoption, as well as what is left undone:

One thing is notably missing from all of this work: downstream verification. [ ] This isn t an acceptable end state (cryptographic attestations have defensive properties only insofar as they re actually verified), so we re looking into ways to bring verification to individual installing clients. In particular, we re currently working on a plugin architecture for pip that will enable users to load verification logic directly into their pip install flows.

There was an in-depth discussion on LWN s announcement page, as well as on Hacker News.

Dependency Challenges in OSS Package Registries At BENEVOL, the Belgium-Netherlands Software Evolution workshop in Namur, Belgium, Tom Mens and Alexandre Decan presented their paper, An Overview and Catalogue of Dependency Challenges in Open Source Software Package Registries . The abstract of their paper is as follows:

While open-source software has enabled significant levels of reuse to speed up software development, it has also given rise to the dreadful dependency hell that all software practitioners face on a regular basis. This article provides a catalogue of dependency-related challenges that come with relying on OSS packages or libraries. The catalogue is based on the scientific literature on empirical research that has been conducted to understand, quantify and overcome these challenges. [ ]

A PDF of the paper is available online.

Zig programming language demonstrated reproducible Motiejus Jak ty posted an interesting and practical blog post on his successful attempt to reproduce the Zig programming language without using the pre-compiled binaries checked into the repository, and despite the circular dependency inherent in its bootstrapping process. As a summary, Motiejus concludes that:

I can now confidently say (and you can also check, you don t need to trust me) that there is nothing hiding in zig1.wasm [the checked-in binary] that hasn t been checked-in as a source file.

The full post is full of practical details, and includes a few open questions.

Website updates Notwithstanding the significant change to the landing page (screenshot above), there were an enormous number of changes made to our website this month. This included:

• Alex Feyerke and Mariano Gim nez:
  • Dramatically overhaul the website s landing page with new benefit cards tailored to the expected visitors to our website and a reworking of the visual hierarchy and design. [ ][ ][ ][ ][ ][ ][ ][ ][ ][ ]
• Bernhard M. Wiedemann:
  • Update the System images page to document the e2fsprogs approach. [ ]
• Chris Lamb:
  • Cachebust every CSS file per-release. [ ]
  • Replace some inline markdown with HTML. [ ]
  • Use spaces on the Publications page. [ ]
  • Add a news article about the passing of Lunar. [ ][ ][ ][ ]
  • Add a black memorial band to the top of the page. [ ]
• FC (Fay) Stegerman:
  • Replace more inline markdown with HTML on the Success stories page. [ ]
  • Add some links, fix some other links and correct some spelling errors on the Tools page. [ ]
• Holger Levsen:
  • Add a historical presentation ( Reproducible builds everywhere eg. in Debian, OpenWrt and LEDE ) from October 2016. [ ]
  • Add jochensp and Oejet to the list of known contributors. [ ][ ]
• Julia Kr ger:
  • Add a new Stripping of unreproducible information page to the documentation. [ ]
• Ninette Adhikari & hulkoba:
  • Add/rework the list of success stories into a new page that clearly shows milestones in Reproducible Builds. [ ][ ][ ][ ][ ][ ]
• Philip Rinn:
  • Import 47 historical weekly reports. [ ]
• hulkoba:
  • Add alt text to almost all images (!). [ ][ ]
  • Fix a number of links on the Talks . [ ][ ]
  • Avoid so-called ghost buttons by not using <button> elements as links, as the affordance of a <button> implies an action with (potentially) a side effect. [ ][ ]
  • Center the sponsor logos on the homepage. [ ]
  • Move publications and generate them instead from a data.yml file with an improved layout. [ ][ ]
  • Make a large number of small but impactful stylisting changes. [ ][ ][ ][ ]
  • Expand the Tools to include a number of missing tools, fix some styling issues and fix a number of stale/broken links. [ ][ ][ ][ ][ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • clisp (fix contributed by Bruno Haible)
  • conky (date-related issue)
  • emacs-auctex (date-related gzip issue)
  • javadoc (filesystem ordering issue)
  • jboss-websocket-1.0-api (embeds uname -r)
  • lcms2 (CPU issue)
  • LiE (ASLR-related issue)
  • make_ext4fs (toolchain-related issue for for VM images)
  • obs-build (issue when running builds with certain CPU types or core numbers)
  • perl-Time-modules (fails to build far in the future)
  • python-bson (fails to build far in the future)
  • python-exiv2 (fails to build far in the future)
  • python-moto (date-related gzip issue)
  • python-pyhanko-certvalidator (fails to build far in the future)
  • python-python-gvm (concurrency-related issue)
  • python310 (fails to build far in the future)
  • python313 (fails to build far in the future)
  • reproducible-faketools (toolchain for emacs)
  • shadowsocks-rust (date-related issue)
  • swipl (fails to build far in the future)
• Chris Lamb:
  • #1087330 filed against python-pydash.
  • #1087485 filed against fritzconnection.
  • #1087486 filed against tracy.
  • #1088238 filed against rust-broot.
  • #1088353 filed against python-aiovlc.
  • #1088742 filed against python-aiohomekit.
• James Addison:
  • #1088144 filed against cdbs.

Misc development news

• Bernhard M. Wiedemann published another report for the openSUSE distribution.
• Martin Abente Lahaye updated diffoscope to fix a crash when objdump is missing. [ ]
• On our mailing list, Jan-Benedict Glaw announced the publication of the fifth NetBSD Reproducibility Report

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In November, a number of changes were made by Holger Levsen, including:

• reproduce.debian.net-related changes:
  • Create and introduce a new reproduce.debian.net service and subdomain [ ]
  • Make a large number of documentation changes relevant to rebuilderd. [ ][ ][ ][ ][ ]
  • Explain a temporary workaround for a specific issue in rebuilderd. [ ]
  • Setup another rebuilderd instance on the o4 node and update installation documentation to match. [ ][ ]
  • Make a number of helpful/cosmetic changes to the interface, such as clarifying terms and adding links. [ ][ ][ ][ ][ ]
  • Deploy configuration to the /opt and /var directories. [ ][ ]
  • Add an infancy (or alpha ) disclaimer. [ ][ ]
  • Add more notes to the temporary rebuilderd documentation. [ ]
  • Commit an nginx configuration file for reproduce.debian.net s Stats page. [ ]
  • Commit a rebuilder-worker.conf configuration for the o5 node. [ ]
• Debian-related changes:
  • Grant jspricke and jochensp access to the o5 node. [ ][ ]
  • Build the qemu package with the nocheck build flag. [ ]
• Misc changes:
  • Adapt the update_jdn.sh script for new Debian trixie systems. [ ]
  • Stop installing the PostgreSQL database engine on the o4 and o5 nodes. [ ]
  • Prevent accidental reboots of the o4 node because of a long-running job owned by josch. [ ][ ]

In addition, Mattia Rizzolo addressed a number of issues with reproduce.debian.net [ ][ ][ ][ ]. And lastly, both Holger Levsen [ ][ ][ ][ ] and Vagrant Cascadian [ ][ ][ ][ ] performed node maintenance.
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org
• Twitter: @ReproBuilds

Notes to self. And anyone else who might find them useful. Following are some ssacli commands which I use infrequently enough that they fall out of cache. This may repeat information in other blogs, but since I search my posts first when commands slip my mind, I thought I d include them here, too. hpacucli is the wrong command. Use ssacli instead.

$ KR='/usr/share/keyrings/hpe.gpg'
$ for fingerprint in \
  882F7199B20F94BD7E3E690EFADD8D64B1275EA3 \
  57446EFDE098E5C934B69C7DC208ADDE26C2B797 \
  476DADAC9E647EE27453F2A3B070680A5CE2D476 ; do \
    curl "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x$ fingerprint " \
        gpg --no-default-keyring --keyring "$ KR " --import ; \
  done
$ gpg --list-keys --no-default-keyring --keyring "$ KR " 
/usr/share/keyrings/hpe.gpg
---------------------------
pub   rsa2048 2012-12-04 [SC] [expired: 2022-12-02]
      476DADAC9E647EE27453F2A3B070680A5CE2D476
uid           [ expired] Hewlett-Packard Company RSA (HP Codesigning Service)
pub   rsa2048 2014-11-19 [SC] [expired: 2024-11-16]
      882F7199B20F94BD7E3E690EFADD8D64B1275EA3
uid           [ expired] Hewlett-Packard Company RSA (HP Codesigning Service) - 1
pub   rsa2048 2015-12-10 [SCEA] [expires: 2025-12-07]
      57446EFDE098E5C934B69C7DC208ADDE26C2B797
uid           [ unknown] Hewlett Packard Enterprise Company RSA-2048-25 
$ echo "deb [signed-by=$ KR ] http://downloads.linux.hpe.com/SDR/repo/mcp bookworm/current non-free" \
    sudo dd of=/etc/apt/sources.list.d status=none
$ sudo apt-get update
$ sudo apt-get install -y -qq ssacli > /dev/null 2>&1
$ sudo ssacli ctrl all show status
HPE Smart Array P408i-p SR Gen10 in Slot 3
   Controller Status: OK
   Cache Status: OK
   Battery/Capacitor Status: OK
$ sudo ssacli ctrl all show detail
HPE Smart Array P408i-p SR Gen10 in Slot 3
   Bus Interface: PCI
   Slot: 3
   Serial Number: PFJHD0ARCCR1QM
   RAID 6 Status: Enabled
   Controller Status: OK
   Hardware Revision: B
   Firmware Version: 2.65
   Firmware Supports Online Firmware Activation: True
   Driver Supports Online Firmware Activation: True
   Rebuild Priority: High
   Expand Priority: Medium
   Surface Scan Delay: 3 secs
   Surface Scan Mode: Idle
   Parallel Surface Scan Supported: Yes
   Current Parallel Surface Scan Count: 1
   Max Parallel Surface Scan Count: 16
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Write Cache Bypass Threshold Size: 1040 KiB
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 15 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 10% Read / 90% Write
   Configured Drive Write Cache Policy: Disable
   Unconfigured Drive Write Cache Policy: Default
   Total Cache Size: 2.0
   Total Cache Memory Available: 1.8
   Battery Backed Cache Size: 1.8
   No-Battery Write Cache: Disabled
   SSD Caching RAID5 WriteBack Enabled: True
   SSD Caching Version: 2
   Cache Backup Power Source: Batteries
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True
   Spare Activation Mode: Activate on physical drive failure (default)
   Controller Temperature (C): 53
   Cache Module Temperature (C): 43
   Capacitor Temperature  (C): 40
   Number of Ports: 2 Internal only
   Encryption: Not Set
   Express Local Encryption: False
   Driver Name: smartpqi
   Driver Version: Linux 2.1.18-045
   PCI Address (Domain:Bus:Device.Function): 0000:11:00.0
   Negotiated PCIe Data Rate: PCIe 3.0 x8 (7880 MB/s)
   Controller Mode: Mixed
   Port Max Phy Rate Limiting Supported: False
   Latency Scheduler Setting: Disabled
   Current Power Mode: MaxPerformance
   Survival Mode: Enabled
   Host Serial Number: 2M20040D1Q
   Sanitize Erase Supported: True
   Sanitize Lock: None
   Sensor ID: 0
      Location: Capacitor
      Current Value (C): 40
      Max Value Since Power On: 42
   Sensor ID: 1
      Location: ASIC
      Current Value (C): 53
      Max Value Since Power On: 55
   Sensor ID: 2
      Location: Unknown
      Current Value (C): 43
      Max Value Since Power On: 45
   Sensor ID: 3
      Location: Cache
      Current Value (C): 43
      Max Value Since Power On: 44
   Primary Boot Volume: None
   Secondary Boot Volume: None
$ sudo ssacli ctrl all show config
HPE Smart Array P408i-p SR Gen10 in Slot 3  (sn: PFJHD0ARCCR1QM)
   Internal Drive Cage at Port 1I, Box 2, OK
   Internal Drive Cage at Port 2I, Box 2, OK
   Port Name: 1I (Mixed)
   Port Name: 2I (Mixed)
   Array A (SAS, Unused Space: 0  MB)
      logicaldrive 1 (1.64 TB, RAID 6, OK)
      physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS HDD, 300 GB, OK)
      physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS HDD, 1.2 TB, OK)
      physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS HDD, 300 GB, OK)
      physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS HDD, 1.2 TB, OK)
      physicaldrive 2I:2:5 (port 2I:box 2:bay 5, SAS HDD, 300 GB, OK)
      physicaldrive 2I:2:6 (port 2I:box 2:bay 6, SAS HDD, 300 GB, OK)
      physicaldrive 2I:2:7 (port 2I:box 2:bay 7, SAS HDD, 1.2 TB, OK)
      physicaldrive 2I:2:8 (port 2I:box 2:bay 8, SAS HDD, 1.2 TB, OK)
   SEP (Vendor ID HPE, Model Smart Adapter) 379  (WWID: 51402EC013705E88, Port: Unknown)
$ sudo ssacli ctrl slot=3 pd 2I:2:7 show detail
HPE Smart Array P408i-p SR Gen10 in Slot 3
   Array A
      physicaldrive 2I:2:7
         Port: 2I
         Box: 2
         Bay: 7
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 1.2 TB
         Drive exposed to OS: False
         Logical/Physical Block Size: 512/512
         Rotational Speed: 10000
         Firmware Revision: U850
         Serial Number: KZGN1BDE
         WWID: 5000CCA01D247239
         Model: HGST    HUC101212CSS600
         Current Temperature (C): 46
         Maximum Temperature (C): 51
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         PHY Physical Link Rate: 6.0Gbps, Unknown
         PHY Maximum Link Rate: 6.0Gbps, 6.0Gbps
         Drive Authentication Status: OK
         Carrier Application Version: 11
         Carrier Bootloader Version: 6
         Sanitize Erase Supported: False
         Shingled Magnetic Recording Support: None
         Drive Unique ID: 5000CCA01D247238

2024-12-09 update I was experiencing a disk failure or predictive failure when I walked into my office this afternoon. I checked this blog post and found the command I used to check the status of the sas array.

cjac@server0:~$ sudo ssacli ctrl all show config
HPE Smart Array P408i-p SR Gen10 in Slot 3  (sn: PFJHD0ARCCR1QM)
   Internal Drive Cage at Port 1I, Box 2, OK
   Internal Drive Cage at Port 2I, Box 2, OK
   Port Name: 1I (Mixed)
   Port Name: 2I (Mixed)
   Array A (SAS, Unused Space: 0  MB)
      logicaldrive 1 (1.64 TB, RAID 6, OK)
      physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS HDD, 300 GB, OK)
      physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS HDD, 1.2 TB, Predictive Failure)
      physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS HDD, 300 GB, OK)
      physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS HDD, 1.2 TB, OK)
      physicaldrive 2I:2:5 (port 2I:box 2:bay 5, SAS HDD, 300 GB, OK)
      physicaldrive 2I:2:6 (port 2I:box 2:bay 6, SAS HDD, 300 GB, OK)
      physicaldrive 2I:2:7 (port 2I:box 2:bay 7, SAS HDD, 1.2 TB, OK)
      physicaldrive 2I:2:8 (port 2I:box 2:bay 8, SAS HDD, 300 GB, OK)
   SEP (Vendor ID HPE, Model Smart Adapter) 379  (WWID: 51402EC013705E88, Port: Unknown)

At this point I engaged the under-study to have the flashing disk replaced

cjac@server0:~$ sudo ssacli ctrl all show config
[sudo] password for cjac: 
HPE Smart Array P408i-p SR Gen10 in Slot 3  (sn: PFJHD0ARCCR1QM)
   Internal Drive Cage at Port 1I, Box 2, OK
   Internal Drive Cage at Port 2I, Box 2, OK
   Port Name: 1I (Mixed)
   Port Name: 2I (Mixed)
   Array A (SAS, Unused Space: 0  MB)
      logicaldrive 1 (1.64 TB, RAID 6, Recovering, 56.43% complete)
      physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS HDD, 300 GB, OK)
      physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS HDD, 300 GB, Rebuilding)
      physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS HDD, 300 GB, OK)
      physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS HDD, 1.2 TB, OK)
      physicaldrive 2I:2:5 (port 2I:box 2:bay 5, SAS HDD, 300 GB, OK)
      physicaldrive 2I:2:6 (port 2I:box 2:bay 6, SAS HDD, 300 GB, OK)
      physicaldrive 2I:2:7 (port 2I:box 2:bay 7, SAS HDD, 1.2 TB, OK)
      physicaldrive 2I:2:8 (port 2I:box 2:bay 8, SAS HDD, 300 GB, OK)
   SEP (Vendor ID HPE, Model Smart Adapter) 379  (WWID: 51402EC013705E88, Port: Unknown)

In the next and future output, I will elide duplicate lines

cjac@server0:~$ date ; sudo ssacli ctrl all show config
Mon Dec  9 07:36:44 PM PST 2024 ... logicaldrive 1 (1.64 TB, RAID 6, Recovering, 66.85% complete)...
Mon Dec  9 07:39:25 PM PST 2024 ... logicaldrive 1 (1.64 TB, RAID 6, Recovering, 68.97% complete) ...
cjac@server0:~$ date ; sudo ssacli ctrl all show config   grep Recovering
Mon Dec  9 07:43:05 PM PST 2024 ... logicaldrive 1 (1.64 TB, RAID 6, Recovering, 71.85% complete)
Mon Dec  9 07:43:57 PM PST 2024 ... logicaldrive 1 (1.64 TB, RAID 6, Recovering, 72.55% complete)
cjac@server0:~$ sleep 5m ; date ; sudo ssacli ctrl all show config   grep Recovering
Mon Dec  9 07:49:43 PM PST 2024 ... logicaldrive 1 (1.64 TB, RAID 6, Recovering, 77.00% complete)
cjac@server0:~$ date ; sudo ssacli ctrl all show config   grep -e 1I:2:2 -e logicaldrive
Mon Dec  9 11:36:14 PM PST 2024 ... logicaldrive 1 (1.64 TB, RAID 6, OK)
physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS HDD, 300 GB, OK)

In 2004, I was living in London, and decided it was time I had my own little virtual private server somewhere online. As a Debian developer since the start of 2000, it had to be Debian, and it still is This was before cloud as we know it today. Virtual Private Servers (VPS) was a Continue reading "My own little server"

Welcome to the October 2024 report from the Reproducible Builds project. Our reports attempt to outline what we ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. Table of contents:

1. Beyond bitwise equality for Reproducible Builds?
2. Two Ways to Trustworthy at SeaGL 2024
3. Number of cores affected Android compiler output
4. On our mailing list
5. diffoscope
6. IzzyOnDroid passed 25% reproducible apps
7. Distribution work
8. Website updates
9. Reproducibility testing framework
10. Supply-chain security at Open Source Summit EU
11. Upstream patches

---

Beyond bitwise equality for Reproducible Builds? Jens Dietrich, Tim White, of Victoria University of Wellington, New Zealand along with Behnaz Hassanshahi and Paddy Krishnan of Oracle Labs Australia published a paper entitled Levels of Binary Equivalence for the Comparison of Binaries from Alternative Builds :

The availability of multiple binaries built from the same sources creates new challenges and opportunities, and raises questions such as: Does build A confirm the integrity of build B? or Can build A reveal a compromised build B? . To answer such questions requires a notion of equivalence between binaries. We demonstrate that the obvious approach based on bitwise equality has significant shortcomings in practice, and that there is value in opting for alternative notions. We conceptualise this by introducing levels of equivalence, inspired by clone detection types.

A PDF of the paper is freely available.

Two Ways to Trustworthy at SeaGL 2024 On Friday 8th November, Vagrant Cascadian will present a talk entitled Two Ways to Trustworthy at SeaGL in Seattle, WA. Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture. Vagrant s talk:

[ ] delves into how two project[s] approaches fundamental security features through Reproducible Builds, Bootstrappable Builds, code auditability, etc. to improve trustworthiness, allowing independent verification; trustworthy projects require little to no trust. Exploring the challenges that each project faces due to very different technical architectures, but also contextually relevant social structure, adoption patterns, and organizational history should provide a good backdrop to understand how different approaches to security might evolve, with real-world merits and downsides.

Number of cores affected Android compiler output Fay Stegerman wrote that the cause of the Android toolchain bug from September s report that she reported to the Android issue tracker has been found and the bug has been fixed.

the D8 Java to DEX compiler (part of the Android toolchain) eliminated a redundant field load if running the class s static initialiser was known to be free of side effects, which ended up accidentally depending on the sharding of the input, which is dependent on the number of CPU cores used during the build.

To make it easier to understand the bug and the patch, Fay also made a small example to illustrate when and why the optimisation involved is valid.

On our mailing list On our mailing list this month:

• Following-up to previous work, James Addison informed the list that the recently-released Sphinx documentation generator includes improvements to the next copyright notice substitutions.
• Pol Dellaiera wrote to the list in order to seek advice around introducing the concept of reproducibility to computer science Masters students at the University of Mons, Belgium.
• James Addison also followed-up to a previous thread on CONFIG_MODULE_SIG and the unreproducible Linux Kernel to add: I wonder whether it would be possible to use the Linux kernel s Integrity Policy Enforcement to deploy a policy that would prevent loading of anything except a set of expected kernel modules. [ ]
• There were also two informative replies from David Wheeler to a broad-based discussion on Reproducible Builds being defined in various standards. [ ][ ]

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 279, 280, 281 and 282 to Debian:

• Ignore errors when listing .ar archives (#1085257). [ ]
• Don t try and test with systemd-ukify in the Debian stable distribution. [ ]
• Drop Depends on the deprecated python3-pkg-resources (#1083362). [ ]

In addition, Jelle van der Waa added support for Unified Kernel Image (UKI) files. [ ][ ][ ] Furthermore, Vagrant Cascadian updated diffoscope in GNU Guix to version 282. [ ][ ]

IzzyOnDroid passed 25% reproducible apps The IzzyOnDroid project has reached a good milestone by reaching over 25% of the ~1,200 Android apps provided by their repository (of official APKs built by the original application developers) having been confirmed to be reproducible by a rebuilder.

Distribution work In Debian this month:

• Holger Levsen uploaded devscripts version 2.24.2, including many changes to the debootsnap, debrebuild and reproducible-check scripts. This is the first time that debrebuild actually works (using sbuild s unshare backend). As part of this, Holger also fixed an issue in the reproducible-check script where a typo in the code led to incorrect results [ ]
• Recently, a news entry was added to snapshot.debian.org s homepage, describing the recent changes that made the system stable again:

  The new server has no problems keeping up with importing the full archives on every update, as each run finishes comfortably in time before it s time to run again. [While] the new server is the one doing all the importing of updated archives, the HTTP interface is being served by both the new server and one of the VM s at LeaseWeb.

  The entry list a number of specific updates surrounding the API endpoints and rate limiting.
• Lastly, 12 reviews of Debian packages were added, 3 were updated and 18 were removed this month adding to our knowledge about identified issues.

Elsewhere in distribution news, Zbigniew J drzejewski-Szmek performed another rebuild of Fedora 42 packages, with the headline result being that 91% of the packages are reproducible. Zbigniew also reported a reproducibility problem with QImage. Finally, in openSUSE, Bernhard M. Wiedemann published another report for that distribution.

Website updates There were an enormous number of improvements made to our website this month, including:

• Alba Herrerias:
  • Improve consistency across distribution-specific guides. [ ]
  • Fix a number of links on the Contribute page. [ ]
• Chris Lamb:
  • Correct the name of Civil Infrastructure Platform name and update image on the Projects page. [ ]
  • Update broken link on the Value Initialization page. [ ]
  • Try and make pipeline/branch builds of the website easier to browse. [ ][ ][ ][ ]
• hulkoba
  • Contribute to the new Success stories page. [ ]
• James Addison:
  • Huge and significant work on a (as-yet-merged) quickstart guide to be linked from the homepage [ ][ ][ ][ ][ ]
  • On the homepage, link directly to the Projects subpage. [ ]
  • Relocate dependency-drift notes to the Volatile inputs page. [ ]
• Ninette Adhikari:
  • Add a brand new Success stories page that highlights the success stories of Reproducible Builds, showcasing real-world examples of projects shipping with verifiable, reproducible builds . [ ][ ][ ][ ][ ][ ]
• Pol Dellaiera:
  • Update the website s README page for building the website under NixOS. [ ][ ][ ][ ][ ]
  • Add a new academic paper citation. [ ]

Lastly, Holger Levsen filed an extensive issue detailing a request to create an overview of recommendations and standards in relation to reproducible builds.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, a number of changes were made by Holger Levsen, including:

• Add a basic index.html for rebuilderd. [ ]
• Update the nginx.conf configuration file for rebuilderd. [ ]
• Document how to use a rescue system for Infomaniak s OpenStack cloud. [ ]
• Update usage info for two particular nodes. [ ]
• Fix up a version skew check to fix the name of the riscv64 architecture. [ ]
• Update the rebuilderd-related TODO. [ ]

In addition, Mattia Rizzolo added a new IP address for the inos5 node [ ] and Vagrant Cascadian brought 4 virt nodes back online [ ].

Supply-chain security at Open Source Summit EU The Open Source Summit EU took place recently, and covered plenty of topics related to supply-chain security, including:

• Public Sector & OpenSSF: Principles for Package Repository Security
• The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency and Usability in AI
• Structured Scorecard Results: Tailor Your Own Supply-Chain Security Policies
• Lightning Talk: Elephant in the Room: How Supply Chain Security Standards Are Not Standard and What to Do About It
• Lightning Talk: Charting the Course for Secure Software Supply Chain with Guac-AI-Mole!
• TPMs, Merkle Trees and TEEs: Enhancing SLSA with Hardware-Assisted Build Environment Verification
• Accountability Taxonomy for AI Software Bill of Materials
• Securing Your Supply Chain with an Open Source Ecosystem
• OSS Supply Chain Threats and Why You Need a Holistic Security Strategy
• A Step Closer to in-Toto lly Secure: Using in-Toto and OPA Gatekeeper to Verify Artifact Integrity
• Panel Discussion: Improving Supply Chain Integrity with OpenSSF Technologies
• Case Study: 10+ Years of Developing an SBOM System and the Dos and Don ts
• SBOM in SaaS Environments: An Update
• Securing Git Repositories with Gittuf

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann
  • apache-ivy (.zip modification time)
  • ccache (build failure)
  • colord (CPU)
  • efivar (CPU/march=native)
  • gsl (no check)
  • libcamera (date/copyright year)
  • libreoffice (possible rpm/build toolchain corruption bug)
  • moto (.gz modification time)
  • openssl-1_1 (date-related issue)
  • python-pygraphviz (benchmark)
  • sphinx/python-pygraphviz (benchmark)
  • python-panel (package.lock has random port)
  • python-propcache (random temporary path)
  • python314 (.gz-related modification time)
  • rusty_v8 (random .o files)
  • scapy (date)
  • wine (parallelism)
  • ibmtss (FTBFS-2026)
  • pymol (date)
  • pandas (ASLR)
  • linutil (drop date)
  • lsof (also filed in openSUSE: uname -r in LSOF_VSTR)
  • schily (also filed in openSUSE: uname -r)
  • superlu (nocheck)
  • util (random test failure)
  • ceph (year-2038 variation from embedded boost)
• Chris Lamb:
  • #1085097 filed against python-roborock.
  • #1085280 filed against pywayland.
  • #1085283 filed against readsb.
  • #1085381 filed against xraylarch.
• James Addison:
  • #1085112 filed against distro-info.
• Zbigniew J drzejewski-Szmek:
  • calibre (two sort issues) [ ][ ]

---

Finally, If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org
• Twitter: @ReproBuilds

Welcome to the September 2024 report from the Reproducible Builds project! Our reports attempt to outline what we ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. Table of contents:

1. New binsider tool to analyse ELF binaries
2. Unreproducibility of GHC Haskell compiler 95% fixed
3. Mailing list summary
4. Towards a 100% bit-for-bit reproducible OS
5. Two new reproducibility-related academic papers
6. Distribution work
7. diffoscope
8. Other software development
9. Android toolchain core count issue reported
10. New Gradle plugin for reproducibility
11. Website updates
12. Upstream patches
13. Reproducibility testing framework

---

New binsider tool to analyse ELF binaries Reproducible Builds developer Orhun Parmaks z has announced a fantastic new tool to analyse the contents of ELF binaries. According to the project s README page:

Binsider can perform static and dynamic analysis, inspect strings, examine linked libraries, and perform hexdumps, all within a user-friendly terminal user interface!

More information about Binsider s features and how it works can be found within Binsider s documentation pages.

Unreproducibility of GHC Haskell compiler 95% fixed A seven-year-old bug about the nondeterminism of object code generated by the Glasgow Haskell Compiler (GHC) received a recent update, consisting of Rodrigo Mesquita noting that the issue is:

95% fixed by [merge request] !12680 when -fobject-determinism is enabled. [ ]

The linked merge request has since been merged, and Rodrigo goes on to say that:

After that patch is merged, there are some rarer bugs in both interface file determinism (eg. #25170) and in object determinism (eg. #25269) that need to be taken care of, but the great majority of the work needed to get there should have been merged already. When merged, I think we should close this one in favour of the more specific determinism issues like the two linked above.

Mailing list summary On our mailing list this month:

• Fay Stegerman let everyone know that she started a thread on the Fediverse about the problems caused by unreproducible zlib/deflate compression in .zip and .apk files and later followed up with the results of her subsequent investigation.
• Long-time developer kpcyrd wrote that there has been a recent public discussion on the Arch Linux GitLab [instance] about the challenges and possible opportunities for making the Linux kernel package reproducible , all relating to the CONFIG_MODULE_SIG flag. [ ]
• Bernhard M. Wiedemann followed-up to an in-person conversation at our recent Hamburg 2024 summit on the potential presence for Reproducible Builds in recognised standards. [ ]
• Fay Stegerman also wrote about her worry about the possible repercussions for RB tooling of Debian migrating from zlib to zlib-ng as reproducibility requires identical compressed data streams. [ ]
• Martin Monperrus wrote the list announcing the latest release of maven-lockfile that is designed aid building Maven projects with integrity . [ ]
• Lastly, Bernhard M. Wiedemann wrote about potential role of reproducible builds in combatting silent data corruption, as detailed in a recent Tweet and scholarly paper on faulty CPU cores. [ ]

Towards a 100% bit-for-bit reproducible OS Bernhard M. Wiedemann began writing on journey towards a 100% bit-for-bit reproducible operating system on the openSUSE wiki:

This is a report of Part 1 of my journey: building 100% bit-reproducible packages for every package that makes up [openSUSE s] minimalVM image. This target was chosen as the smallest useful result/artifact. The larger package-sets get, the more disk-space and build-power is required to build/verify all of them.

This work was sponsored by NLnet s NGI Zero fund.

Two new reproducibility-related academic papers Marvin Strangfeld published his bachelor thesis, Reproducibility of Computational Environments for Software Development from RWTH Aachen University. The author offers a more precise theoretical definition of computational environments compared to previous definitions, which can be applied to describe real-world computational environments. Additionally, Marvin provide a definition of reproducibility in computational environments, enabling discussions about the extent to which an environment can be made reproducible. The thesis is available to browse or download in PDF format. In addition, Shenyu Zheng, Bram Adams and Ahmed E. Hassan of Queen s University, ON, Canada have published an article on hermeticity in Bazel-based build systems:

A hermetic build system manages its own build dependencies, isolated from the host file system, thereby securing the build process. Although, in recent years, new artifact-based build technologies like Bazel offer build hermeticity as a core functionality, no empirical study has evaluated how effectively these new build technologies achieve build hermeticity. This paper studies 2,439 non-hermetic build dependency packages of 70 Bazel-using open-source projects by analyzing 150 million Linux system file calls collected in their build processes. We found that none of the studied projects has a completely hermetic build process, largely due to the use of non-hermetic top-level toolchains. [ ]

Distribution work In Debian this month, 14 reviews of Debian packages were added, 12 were updated and 20 were removed, all adding to our knowledge about identified issues. A number of issue types were updated as well. [ ][ ] In addition, Holger opened 4 bugs against the debrebuild component of the devscripts suite of tools. In particular:

• #1081047: Fails to download .dsc file.
• #1081048: Does not work with a proxy.
• #1081050: Fails to create a debrebuild.tar.
• #1081839: Fails with E: mmdebstrap failed to run error.

Last month, an issue was filed to update the Salsa CI pipeline (used by 1,000s of Debian packages) to no longer test for reproducibility with reprotest s build_path variation. Holger Levsen provided a rationale for this change in the issue, which has already been made to the tests being performed by tests.reproducible-builds.org. This month, this issue was closed by Santiago R. R., nicely explaining that build path variation is no longer the default, and, if desired, how developers may enable it again. In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading version 278 to Debian:

• New features:
  • Add a helpful contextual message to the output if comparing Debian .orig tarballs within .dsc files without the ability to fuzzy-match away the leading directory. [ ]
• Bug fixes:
  • Drop removal of calculated os.path.basename from GNU readelf output. [ ]
  • Correctly invert X% similar value and do not emit 100% similar . [ ]
• Misc:
  • Temporarily remove procyon-decompiler from Build-Depends as it was removed from testing (via #1057532). (#1082636)
  • Update copyright years. [ ]

For trydiffoscope, the command-line client for the web-based version of diffoscope, Chris Lamb also:

• Added an explicit python3-setuptools dependency. (#1080825)
• Bumped the Standards-Version to 4.7.0. [ ]

Other software development disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into system calls to reliably flush out reproducibility issues. This month, version 0.5.11-4 was uploaded to Debian unstable by Holger Levsen making the following changes:

• Replace build-dependency on the obsolete pkg-config package with one on pkgconf, following a Lintian check. [ ]
• Bump Standards-Version field to 4.7.0, with no related changes needed. [ ]

In addition, reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, version 0.7.28 was uploaded to Debian unstable by Holger Levsen including a change by Jelle van der Waa to move away from the pipes Python module to shlex, as the former will be removed in Python version 3.13 [ ].

Android toolchain core count issue reported Fay Stegerman reported an issue with the Android toolchain where a part of the build system generates a different classes.dex file (and thus a different .apk) depending on the number of cores available during the build, thereby breaking Reproducible Builds:

We ve rebuilt [tag v3.6.1] multiple times (each time in a fresh container): with 2, 4, 6, 8, and 16 cores available, respectively:

• With 2 and 4 cores we always get an unsigned APK with SHA-256 14763d682c9286ef .
• With 6, 8, and 16 cores we get an unsigned APK with SHA-256 35324ba4c492760 instead.

New Gradle plugin for reproducibility A new plugin for the Gradle build tool for Java has been released. This easily-enabled plugin results in:

reproducibility settings [being] applied to some of Gradle s built-in tasks that should really be the default. Compatible with Java 8 and Gradle 8.3 or later.

Website updates There were a rather substantial number of improvements made to our website this month, including:

• Chris Lamb:
  • Attempt to use GitLab CI to artifact the website; hopefully useful for testing branches. [ ]
  • Correct the linting rule whilst building the website. [ ]
  • Make a number of small changes to Kees post written by Vagrant. [ ][ ][ ]
  • Add the Civil Infrastructure Platform to the Projects page. [ ]
  • Miscellaneous administration of misfiled images. [ ][ ]
• Evangelos Tzaras made a huge number of changes related to the recent Hamburg 2024 summit [ ][ ][ ][ ][ ] as well as proposed an infographic about which question Reproducible Builds is trying to answer.
• Holger Levsen added his two presentations (Reproducible Builds: The First Eleven Years and Preserving *other* build artifacts) to the website. [ ]
• Jelle van der Waa completely modernised the System Images documentation, noting that a lot has changed since 2017(!); ext4, erofs and FAT filesystems can now be made reproducible . [ ]
• Developer RyanSquared replaced the continuous integration test link for Arch Linux on our Projects page with an external instance [ ][ ] as well as updated the documentation to reflect the dependencies required to build the website [ ].
• Vagrant Cascadian pushed a lengthy interview with Linux developer Kees Cook. [ ][ ][ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • agama-integration-tests (random)
  • contrast (FTBFS-nocheck)
  • cpython (FTBFS-2038)
  • crash (parallelism, race)
  • ghostscript (toolchain date)
  • glycin-loaders (FTBFS -j1)
  • gstreamer-plugins-rs (date, other)
  • kernel-doc/Sphinx (toolchain bug, parallelism/race)
  • kernel (parallelism in BTF)
  • libcamera (random key)
  • libgtop (uname -r)
  • libsamplerate (random temporary directory)
  • lua-luarepl (FTBFS)
  • meson (toolchain)
  • netty (modification time in .a)
  • nvidia-persistenced (date)
  • nvidia-xconfig (date-related issue)
  • obs-build (build-tooling corruption)
  • perl (Perl records kernel version)
  • pinentry (make efl droppable)
  • python-PyGithub (FTBFS 2024-11-25)
  • python-Sphinx (parallelism/race)
  • python-chroma-hnswlib (CPU)
  • python-libcst
  • python-pygraphviz (random timing)
  • python312 (.pyc embeds modification time)
  • python312 (drop .pyc from documentation time)
  • scap-security-guide (date)
  • seahorse (parallelism)
  • subversion (minor Java .jar modification times)
  • xen/acpica (date-related issue in toolchain)
  • xmvn (random)
• Fridrich Strba:
  • ant (jar mtime)
  • xmvn (various fixes)
• Chris Lamb:
  • #1082702 filed against magic-wormhole-transit-relay.
  • #1082706 filed against python-sphobjinv.
  • #1082707 filed against lomiri-content-hub.
  • #1082796 filed against python-mt-940.
  • #1082806 filed against tree-puzzle.
  • #1083053 filed against muon-meson.
• James Addison:
  • guake (fix a parallelism/timing-race build bug)
  • sphobjinv (duplicates fix from Debian bug #1082706)

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In September, a number of changes were made by Holger Levsen, including:

• Debian-related changes:
  • Upgrade the osuosl4 node to Debian trixie in anticipation of running debrebuild and rebuilderd there. [ ][ ][ ]
  • Temporarily mark the osuosl4 node as offline due to ongoing xfs_repair filesystem maintenance. [ ][ ]
  • Do not warn about (very old) broken nodes. [ ]
  • Add the risc64 architecture to the multiarch version skew tests for Debian trixie and sid. [ ][ ][ ]
  • Mark the virt 32,64 b nodes as down. [ ]
• Misc changes:
  • Add support for powercycling OpenStack instances. [ ]
  • Update the fail2ban to ban hosts for 4 weeks in total [ ][ ] and take care to never ban our own Jenkins instance. [ ]

In addition, Vagrant Cascadian recorded a disk failure for the virt32b and virt64b nodes [ ], performed some maintenance of the cbxi4a node [ ][ ] and marked most armhf architecture systems as being back online.

---

Finally, If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org
• Twitter: @ReproBuilds

A new release 0.0.4 of the RcppFastAD package by James Yang and myself is now on CRAN. RcppFastAD wraps the FastAD header-only C++ library by James which provides a C++ implementation of both forward and reverse mode of automatic differentiation. It offers an easy-to-use header library (which we wrapped here) that is both lightweight and performant. With a little of bit of Rcpp glue, it is also easy to use from R in simple C++ applications. This release updates the quick fix in release 0.0.3 from a good week ago. James took a good look and properly disambiguated the statement that lead clang to complain, so we are back to compiling as C++17 under all compilers which makes for a slightly wider reach. The NEWS file for this release follows.

Changes in version 0.0.4 (2024-09-24)

• The package now properly addresses a clang warning on empty variadic macros arguments and is back to C++17 (James in #10)

Courtesy of my CRANberries, there is also a diffstat report for the most recent release. More information is available at the repository or the package page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

A new release 0.0.3 of the RcppFastAD package by James Yang and myself is now on CRAN. RcppFastAD wraps the FastAD header-only C++ library by James which provides a C++ implementation of both forward and reverse mode of automatic differentiation. It offers an easy-to-use header library (which we wrapped here) that is both lightweight and performant. With a little of bit of Rcpp glue, it is also easy to use from R in simple C++ applications. This release turns compilation to the C++20 standard as newer clang++ versions complained about a particular statement (it took to be C++20) when compiled under C++17. So we obliged. The NEWS file for these two initial releases follows.

Changes in version 0.0.3 (2024-09-15)

• The package now compiles under the C++20 standard to avoid a warning under clang++-18 (Dirk addressing #9)
• Minor updates to continuous integration and badges have been made as well

Courtesy of my CRANberries, there is also a diffstat report for the most recent release. More information is available at the repository or the package page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Welcome to the August 2024 report from the Reproducible Builds project! Our reports attempt to outline what we ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. Table of contents:

1. LWN: The history, status, and plans for reproducible builds
2. Intermediate Autotools build artifacts removed from PostgreSQL distribution tarballs
3. Distribution news
4. Mailing list news
5. diffoscope
6. Website updates
7. Upstream patches
8. Reproducibility testing framework

---

LWN: The history, status, and plans for reproducible builds The free software newspaper of record, Linux Weekly News, published an in-depth article based on Holger Levsen s talk, Reproducible Builds: The First Eleven Years which was presented at the recent DebConf24 conference in Busan, South Korea. Titled The history, status, and plans for reproducible builds and written by Jake Edge, LWN s article not only summarises Holger s talk and clarifies its message but it links to external information as well. Holger s original talk can also be watched on the DebConf24 webpage (direct .webm link and his HTML slides are available also). There are also a significant number of comments on LWN s page as well. Holger Levsen also headed a scheduled discussion session at DebConf24 on Preserving *other* build artifacts addressing a topic where a number of Debian packages are (or would like to) produce results that are neither the .deb files, the build logs nor the logs of CI tests. This is an issue for reproducible builds as this 4th type of build artifact are typically shipped within the binary .deb packages, and are invariably non-deterministic; thus making the .deb files unreproducible. (A direct .webm link and HTML slides are available).

Intermediate Autotools build artifacts removed from PostgreSQL distribution tarballs Peter Eisentraut wrote a detailed blog post on the subject of The new PostgreSQL 17 make dist . Like many projects, the PostgreSQL database has previously pre-built parts of its GNU Autotools build system: the reason for this is a mix of convenience and traditional practice . Peter astutely notes that this arrangement in the build system is quite tricky as:

You need to carefully maintain the different states of clean source code , partially built source code , and fully built source code , and the commands to transition between them.

However, Peter goes on to mention that:

a lot more attention is nowadays paid to the software supply chain. There are security and legal reasons for this. When users install software, they want to know where it came from, and they want to be sure that they got the right thing, not some fake version or some version of dubious legal provenance.

And cites the XZ Utils backdoor as a reason to care about transparent and reproducible ways of distributing and communicating a source tarball and provenance. Because of this, intermediate build artifacts are now henceforth essentially disallowed from PostgreSQL distribution tarballs.

Distribution news In Debian this month, 30 reviews of Debian packages were added, 17 were updated and 10 were removed this month adding to our knowledge about identified issues. One issue type was added by Chris Lamb, too. [ ] In addition, an issue was filed to update the Salsa CI pipeline (used by 1,000s of Debian packages) to no longer test for reproducibility with reprotest s build_path variation. Holger Levsen provided a rationale for this change in the issue, which has already been made to the tests being performed by tests.reproducible-builds.org.
In Arch Linux this month, Jelle van der Waa published a short blog post on the topic of Investigating creating reproducible images with mkosi, motivated by the desire to make it possible for anyone to re-recreate the official Arch cloud image bit-by-bit identical on their own machine as per [the] reproducible builds definition. In addition, Jelle filed a patch for pacman, the Arch Linux package manager, to respect the SOURCE_DATE_EPOCH environment variable when installing a package.
In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.
In Android news, the IzzyOnDroid project added 49 new rebuilder recipes and now features 256 total reproducible applications representing 21% of the total offerings in the repository. IzzyOnDroid is an F-Droid style repository for Android apps[:] applications in this repository are official binaries built by the original application developers, taken from their resp. repositories (mostly GitHub).

Mailing list news From our mailing list this month:

• Bernhard M. Wiedemann posted a brief message to the list with some helpful information regarding nondeterminism within Rust binaries, positing the use of the codegen-units = 16 default and resulting in a bug being filed in the Rust issue tracker. [ ]
• Bernhard also wrote to the list, following up to a thread in November 2023, on attempts to make the LibreOffice suite of office applications build reproducibly. In the thread from this month, Bernhard could announce that the four patches previously mentioned have landed in LibreOffice upstream.
• Fay Stegerman linked the mailing list to a thread she made on the Signal issue tracker regarding whether device-specific binaries [can] ever be considered meaningfully reproducible . In particular: the whole part about allow[ing] multiple third parties to come to a consensus on a correct result breaks down completely when correct is device-specific and not something everyone can agree on. [ ]
• Developer kpcyrd posted an update for source code indexing project, whatsrc.org. Announcing that it now importing packages from live-bootstrap ( a usable Linux system [that is] created with only human-auditable, and wherever possible, human-written, source code ) into its database of provenance data.
• Lastly, Mechtilde Stehmann posted an update to an earlier thread about how Java builds are not reproducible on the armhf architecture, enquiring how they might gain temporary access to such a machine in order to perform some deeper testing. [ ]

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb released versions 274, 275, 276 and 277, uploaded these to Debian, and made the following changes as well:

• New features:
  • Strip ANSI escapes usually colour codes from the output of the Procyon Java decompiler. [ ]
  • Factor out a method for stripping ANSI escapes. [ ]
  • Append output from dumppdf(1) in more cases, avoiding situations where we fallback to a binary diff. [ ]
  • Add support for versions of Perl s IO::Compress::Zip version 2.212. [ ]
• Bug fixes:
  • Also catch RuntimeError exceptions when importing the PyPDF library so that it, or, crucially, its transitive dependencies, cannot not cause diffoscope to traceback at runtime and build time. [ ]
  • Do not call marshal.load( ) of precompiled Python bytecode as it, alas, inherently unsafe. Replace for now with a brief summary of the code section of .pyc. [ ][ ]
  • Don t include excessive debug output when calling dumppdf(1). [ ]
• Testsuite-related changes:
  • Don t bother to check version number in test_python.py: the fixture for this test is fixed. [ ][ ]
  • Update test_zip text fixtures and definitions to support new changes to the Perl IO::Compress library. [ ]

In addition, Mattia Rizzolo updated the available architectures for a number of test dependencies [ ] and Sergei Trofimovich fixed an issue to avoid diffoscope crashing when hashing directory symlinks [ ] and Vagrant Cascadian proposed GNU Guix updates for diffoscope versions 275 and 276 and 277.

Website updates There were a rather substantial number of improvements made to our website this month, including:

• Alba Herrerias:
  • Substantially extend the guidance on the Contribute page. [ ]
• Chris Lamb:
  • Set the future: true configuration value so we render all files and documents in the website, regardless of whether they have a date property in the future. After all, we don t re-generate the website on a timer, and have other ways of making unpublished, draft posts. [ ][ ]
• Fay Stegerman:
  • Add IzzyOnDroid (IoD) to the Projects page. [ ]
• hulkoba:
  • Considerably overhaul the History page in the documentation, linking strip-nondeterminism and SOURCE_DATE_EPOCH [ ], fixing the test statistics link [ ], adjusting the Google Summer of Code application link [ ], a link to a Debian bug [ ], and removed a dead link to the debhelper utility [ ].
  • Use the jekyll-sitemap plugin to create a sitemap for the website. [ ]
  • Use raw HTML to avoid a literal .lead directive appearing in the page. [ ]
  • Fix a number of issues on the Virtual machine drivers page, such as keeping the Gitian info, linking (and then removing) an issue on the Bitcoin issue tracker [ ] and fixing a link to the Bazel website [ ].
  • Address a broken footnote link on the Timestamps page. [ ]
  • Unify the style on the Commandments of Reproducible Builds page in order to match other documentation entries. [ ]
  • Add a table of contents to the main Documentation page. [ ]
  • Avoid a number of so-called here links on the Variations in the build environment page. [ ]
  • Fix a link to the man2html patch on the SOURCE_DATE_EPOCH documentation page. [ ]
  • Fix a link to sources.debian.org on the Randomness page. [ ]
• kpcyrd:
  • Fix a typo on the Variations in the build environment page. [ ]
• Mattia Rizzolo:
  • Add all the sponsors to the Hamburg 2024 summit page. [ ][ ][ ][ ][ ]
  • Fix a link in the summit sponsor prospectus PDF. [ ]
• Pol Dellaiera:
  • Fix the DoI for their thesis on the Publications page. [ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • agama-integration-tests (uses a random TCP-port number in .lock file)
  • ca-certificates-mozilla:ca-certificates-mozilla-prebuilt
  • cosmic (hash order issue)
  • openSUSE (meta-issue to test reproducibility in the openSUSE Build Service)
  • pop-launcher (parallelism-related issue)
  • post (toolchain-issue, avoiding Rust parallelism)
  • rpm-config-SUSE (date-related issue)
  • rust (Rust toolchain issue)
  • weblate (build gets stuck)
• Chris Lamb:
  • #1078048 filed against scikit-optimize.
  • #1078153 filed against pan.
  • #1078687 filed against tetzle.
  • #1079041 filed against apg.
  • #1079312 filed against receptor.
• James Addison:
  • #1064782 forwarded and merged in bind9-doc
  • #1066083 forwarded and merged in gnome-maps

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In August, a number of changes were made by Holger Levsen, including:

• Temporarily install the openssl-provider-legacy package for the Debian unstable environments for running diffoscope due to Debian bug #1078944. [ ][ ][ ][ ]
• Mark Debian armhf architecture nodes as being down due to proxy down. [ ][ ]
• Detect proxy failures. [ ][ ][ ]
• Run the index-buildinfo for the builtin-pho script with the -q switch. [ ]
• Disable all Arch Linux reproducible jobs. [ ]

In addition, Mattia Rizzolo updated the website configuration to install the ruby-jekyll-sitemap package as it is now used in the website [ ], Roland Clobus updated the script to build Debian live images to treat openQA issues as warnings [ ], and Vagrant Cascadian marked the cbxi4b node as down [ ].

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org
• Twitter: @ReproBuilds

Welcome to the July 2024 report from the Reproducible Builds project! In our reports, we outline what we ve been up to over the past month and highlight news items in software supply-chain security more broadly. As always, if you are interested in contributing to the project, please visit our Contribute page on our website. Table of contents:

1. Reproducible Builds Summit 2024
2. Pulling Linux up by its bootstraps
3. Towards Idempotent Rebuilds?
4. AROMA: Automatic Reproduction of Maven Artifacts
5. Community updates
6. Android Reproducible Builds at IzzyOnDroid with rbtlog
7. Extending the Scalability, Flexibility and Responsiveness of Secure Software Update Systems
8. Development news
9. Website updates
10. Upstream patches
11. Reproducibility testing framework

---

Reproducible Builds Summit 2024 Last month, we were very pleased to announce the upcoming Reproducible Builds Summit, set to take place from September 17th 19th 2024 in Hamburg, Germany. We are thrilled to host the seventh edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving. If you re interesting in joining us this year, please make sure to read the event page, which has more details about the event and location. We are very much looking forward to seeing many readers of these reports there.

Pulling Linux up by its bootstraps (LWN) In a recent edition of Linux Weekly News, Daroc Alden has written an article on bootstrappable builds. Starting with a brief introduction that

a bootstrappable build is one that builds existing software from scratch for example, building GCC without relying on an existing copy of GCC. In 2023, the Guix project announced that the project had reduced the size of the binary bootstrap seed needed to build its operating system to just 357-bytes not counting the Linux kernel required to run the build process.

The article goes onto to describe that now, the live-bootstrap project has gone a step further and removed the need for an existing kernel at all. and concludes:

The real benefit of bootstrappable builds comes from a few things. Like reproducible builds, they can make users more confident that the binary packages downloaded from a package mirror really do correspond to the open-source project whose source code they can inspect. Bootstrappable builds have also had positive effects on the complexity of building a Linux distribution from scratch [ ]. But most of all, bootstrappable builds are a boon to the longevity of our software ecosystem. It s easy for old software to become unbuildable. By having a well-known, self-contained chain of software that can build itself from a small seed, in a variety of environments, bootstrappable builds can help ensure that today s software is not lost, no matter where the open-source community goes from here

Towards Idempotent Rebuilds? Trisquel developer Simon Josefsson wrote an interesting blog post comparing the output of the .deb files from our tests.reproducible-builds.org testing framework and the ones in the official Debian archive. Following up from a previous post on the reproducibility of Trisquel, Simon notes that typically [the] rebuilds do not match the official packages, even when they say the package is reproducible , Simon correctly identifies that the purpose of [these] rebuilds are not to say anything about the official binary build, instead the purpose is to offer a QA service to maintainers by performing two builds of a package and declaring success if both builds match. However, Simon s post swiftly moves on to announce a new tool called debdistrebuild that performs rebuilds of the difference between two distributions in a GitLab pipeline and displays diffoscope output for further analysis.

AROMA: Automatic Reproduction of Maven Artifacts Mehdi Keshani, Tudor-Gabriel Velican, Gideon Bot and Sebastian Proksch of the Delft University of Technology, Netherlands, have published a new paper in the ACM Software Engineering on a new tool to automatically reproduce Apache Maven artifacts:

Reproducible Central is an initiative that curates a list of reproducible Maven libraries, but the list is limited and challenging to maintain due to manual efforts. [We] investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of Maven packages. Overall, our approach achieves an accuracy of up to 99.5% when compared field-by-field to the existing manual approach [and] we reveal that automatic reproducibility is feasible for 23.4% of the Maven packages using AROMA, and 8% of these packages are fully reproducible.

Community updates On our mailing list this month:

• Nichita Morcotilo reached out to the community, first to share their efforts to build reproducible packages cross-platform with a new build tool called rattler-build, noting that as you can imagine, building packages reproducibly on Windows is the hardest challenge (so far!) . Nichita goes onto mention that the Apple ecosystem appears to be using ZERO_AR_DATE over SOURCE_DATE_EPOCH. [ ]
• Roland Clobus announced that the Debian bookworm 12.6 live images are nearly reproducible , with more detail in the post itself and input in the thread from other contributors.
• As reported in last month s report, Pol Dellaiera completed his master thesis on Reproducibility in Software Engineering at the University of Mons, Belgium. This month, Pol announced this on the list with more background info. Since the master thesis sources have been available, it has received some feedback and contributions. As a result, an updated version of the thesis has been published containing those community fixes.
• Daniel Gr ber asked for help in getting the Yosys documentation to build reproducibly, citing issues in inter alia the PDF generation causing differing CreationDate metadata values.
• James Addison continued his long journey towards getting the Sphinx documentation generator to build reproducible documentation. In this thread, James concerns himself with the problem that even when SOURCE_DATE_EPOCH is configured, Sphinx projects that have configured their copyright notices using dynamic elements can produce nonsensical output under some circumstances. James query ended up generating a number of replies.
• Allen gunner Gunner posted a brief update on the progress the core team is making towards introducing a Code of Conduct (CoC) such that it is in place in time for the RB Summit in Hamburg in September . In particular, gunner asks if you are interested in helping with CoC design and development in the weeks ahead, simply email rb-core@lists.reproducible-builds.org and let us know . [ ]

Android Reproducible Builds at IzzyOnDroid with rbtlog On our mailing list, Fay Stegerman announced a new Reproducible Builds collaboration in the Android ecosystem:

We are pleased to announce Reproducible Builds, special client support and more in our repo : a collaboration between various independent interoperable projects: the IzzyOnDroid team, 3rd-party clients Droid-ify & Neo Store, and rbtlog (part of my collection of tools for Android Reproducible Builds) to bring Reproducible Builds to IzzyOnDroid and the wider Android ecosystem.

Extending the Scalability, Flexibility and Responsiveness of Secure Software Update Systems Congratulations to Marina Moore of the New York Tandon School of Engineering who has submitted her PhD thesis on Extending the Scalability, Flexibility and Responsiveness of Secure Software Update Systems. The introduction outlines its contributions to the field:

[S]oftware repositories are a vital component of software development and release, with packages downloaded both for direct use and to use as dependencies for other software. Further, when software is updated due to patched vulnerabilities or new features, it is vital that users are able to see and install this patched version of the software. However, this process of updating software can also be the source of attack. To address these attacks, secure software update systems have been proposed. However, these secure software update systems have seen barriers to widespread adoption. The Update Framework (TUF) was introduced in 2010 to address several attacks on software update systems including repository compromise, rollback attacks, and arbitrary software installation. Despite this, compromises continue to occur, with millions of users impacted by such compromises. My work has addressed substantial challenges to adoption of secure software update systems grounded in an understanding of practical concerns. Work with industry and academic communities provided opportunities to discover challenges, expand adoption, and raise awareness about secure software updates. [ ]

Development news In Debian this month, 12 reviews of Debian packages were added, 13 were updated and 6 were removed this month adding to our knowledge about identified issues. A new toolchain issue type was identified as well, specifically ordering_differences_in_pkg_info.
Colin Percival filed a bug against the LLVM compiler noting that building i386 binaries on the i386 architecture is different when building i386 binaries under amd64. The fix was narrowed down to x87 excess precision, which can result in slightly different register choices when the compiler is hosted on x86_64 or i386 and a fix committed. [ ]
Fay Stegerman performed some in-depth research surrounding her apksigcopier tool, after some Android .apk files signed with the latest apksigner could no longer be verified as reproducible. Fay identified the issue as follows:

Since build-tools >= 35.0.0-rc1, backwards-incompatible changes to apksigner break apksigcopier as it now by default forcibly replaces existing alignment padding and changed the default page alignment from 4k to 16k (same as Android Gradle Plugin >= 8.3, so the latter is only an issue when using older AGP). [ ]

She documented multiple available workarounds and filed a bug in Google s issue tracker.
Lastly, diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb uploaded version 272 and Mattia Rizzolo uploaded version 273 to Debian, and the following changes were made as well:

• Chris Lamb:
  • Ensure that the convert utility is from ImageMagick version 6.x. The command-line interface has seemingly changed with the 7.x series of ImageMagick. [ ]
  • Factor out version detection in test_jpeg_image. [ ]
  • Correct the import of the identify_version method after a refactoring change in a previous commit. [ ]
  • Move away from using DSA OpenSSH keys in tests as support has been deprecated and removed in OpenSSH version 9.8p1. [ ]
  • Move to assert_diff in the test_openssh_pub_key package. [ ]
  • Update copyright years. [ ]
• Mattia Rizzolo:
  • Add support for ffmpeg version 7.x which adds some extra context to the diff. [ ]
  • Rework the handling of OpenSSH testing of DSA keys if OpenSSH is strictly 9.7, and add an OpenSSH key test with a ed25519-format key [ ][ ][ ]
  • Temporarily disable a few packages that are not available in Debian testing. [ ][ ]
  • Stop ignoring the results of Debian testing in the continuous integration system. [ ]
  • Adjust options in debian/source to make sure not to pack the Python sdist directory into the binary Debian package. [ ]
  • Adjust Lintian overrides. [ ]

Website updates There were a number of improvements made to our website this month, including:

• Bernhard M. Wiedemann updated the SOURCE_DATE_EPOCH page to include instructions on how to create reproducible .zip files from within Python using the zipfile module. [ ]
• Chris Lamb fixed a potential duplicate heading on the Projects page. [ ]
• Fay Stegerman added rbtlog to the Tools page [ ] and IzzyOnDroid to the Projects page [ ], also ensuring that the latter page was always sorted regardless of the ordering within the input data files. [ ]
• Holger Levsen added Linus Nordberg to our global list of contributors [ ] as well as made a number of changes to the page for the upcoming Reproducible Builds summit later this year [ ][ ][ ][ ].
• Mattia Rizzolo updated the Civil Infrastructure Platform logo [ ] and also updated the 2024 summit page [ ][ ].
• Nichita Morcotilo added rattler-build to the Projects page. [ ][ ][ ]
• Pol Dellaiera updated the Academic Publications page, adding two publications. [ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • armagetron (date)
  • blaspp (hostname)
  • cligen (GnuTLSs date)
  • cloudflared (date)
  • dpdk (Sphinx doctrees)
  • fonttosfnt/xorg-x11-fonts (toolchain, date)
  • gegl (build machine details)
  • gettext-runtime (jar mtime)
  • kf6-kirigami+kf6-qqc2-desktop-style (race-condition)
  • kubernetes1.26 (backport upstream fix for random path)
  • lapackpp (hostname)
  • latex2html (nochecks)
  • libdb-4_8 (.jar modification time)
  • librcc (already merged upstream)
  • libreoffice (strip .jar mtimes + clucene-core toolchain)
  • maliit-keyboard (nocheck)
  • nautilus (date)
  • openblas (CPU type, fixed)
  • openssl-3 (random-related issue)
  • python-ruff (ASLR)
  • python3 (date, parallelism/race)
  • reproducible-faketools (0.5.2)
  • sphinx (GZip modification time)
  • sphinxcontrib (gzip mtime)
• Chris Lamb:
  • #1076368 filed against nautilus.
  • #1076507 filed against mccode.
  • #1076806 filed against meson-python.
  • #1077479 filed against debcraft.
  • #1077485 filed against pytest.
  • #1077601 filed against setuptools.
• Fridrich Strba:
  • javapackages-tools
  • ant
  • java-21-openjdk
• Evangelos Ribeiro Tzaras:
  • buffybox

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In July, a number of changes were made by Holger Levsen, including:

• Grant bremner access to the ionos7 node. [ ][ ]
• Perform a dummy change to force update of all jobs. [ ][ ]

In addition, Vagrant Cascadian performed some necessary node maintenance of the underlying build hosts. [ ]

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org
• Twitter: @ReproBuilds

podlators contains the Perl modules and scripts used to convert Perl's documentation language, POD, to text and manual pages. This is another small bug fix release that is part of iterating on getting the new podlators incorproated into Perl core. The bug fixed in this release was another build system bug I introduced in recent refactorings, this time breaking the realclean target so that some generated scripts were not removed. Thanks to James E Keenan for the report. You can get the latest version from CPAN or from the podlators distribution page.

This is a small bug-fix release to remove use of autodie from the build system for the module. podlators is included in Perl core, and at the point when it is built during the core build, the prerequisites of the autodie module are not yet met, so the module is not available. This release reverts to explicit error checking in all the files used by the build system. Thanks to James E Keenan for the report and the analysis. You can get the latest version from CPAN or the podlators distribution page.

Welcome to the June 2024 report from the Reproducible Builds project! In our reports, we outline what we ve been up to over the past month and highlight news items in software supply-chain security more broadly. As always, if you are interested in contributing to the project, please visit our Contribute page on our website. Table of contents:

1. Next Reproducible Builds Summit dates announced
2. GNU Guix patch review session for reproducibility
3. New reproducibility-related academic papers
4. Misc development news
5. Website updates
6. Reproducibility testing framework

---

Next Reproducible Builds Summit dates announced We are very pleased to announce the upcoming Reproducible Builds Summit, set to take place from September 17th 19th 2024 in Hamburg, Germany. We are thrilled to host the seventh edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving. If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. We are very much looking forward to seeing many readers of these reports there.

GNU Guix patch review session for reproducibility Vagrant Cascadian will be holding a Reproducible Builds session as part of the monthly Guix patch review series on July 11th at 17:00 UTC. These online events are intended to encourage everyone everyone becoming a patch reviewer and the goal of reviewing patches is to help Guix project accept contributions while maintaining our quality standards and learning how to do patch reviews together in a friendly hacking session.

New reproducibility-related academic papers Multiple scholarly papers related to Reproducible Builds were published this month: An Industry Interview Study of Software Signing for Supply Chain Security was published by Kelechi G. Kalu, Tanmay Singla, Chinenye Okafor, Santiago Torres-Arias and James C. Davis of Electrical and Computer Engineering department of Purdue University, Indiana, USA, and is concerned with:

To understand software signing in practice, we interviewed 18 high-ranking industry practitioners across 13 organizations. We provide possible impacts of experienced software supply chain failures, security standards, and regulations on software signing adoption. We also study the challenges that affect an effective software signing implementation.

DiVerify: Diversifying Identity Verification in Next-Generation Software Signing was written by Chinenye L. Okafor, James C. Davis and Santiago Torres-Arias also of Purdue University and is interested in:

Code signing enables software developers to digitally sign their code using cryptographic keys, thereby associating the code to their identity. This allows users to verify the authenticity and integrity of the software, ensuring it has not been tampered with. Next-generation software signing such as Sigstore and OpenPubKey simplify code signing by providing streamlined mechanisms to verify and link signer identities to the public key. However, their designs have vulnerabilities: reliance on an identity provider introduces a single point of failure, and the failure to follow the principle of least privilege on the client side increases security risks. We introduce Diverse Identity Verification (DiVerify) scheme, which strengthens the security guarantees of next-generation software signing by leveraging threshold identity validations and scope mechanisms.

Felix Lagn hed published their thesis on the Integration of Reproducibility Verification with Diffoscope in GNU Make. This work, amongst some other results:

[ ] resulted in an extension of GNU make which is called rmake, where diffoscope a tool for detecting differences between a large number of file types was integrated into the workflow of make. rmake was later used to answer the posed research questions for this thesis. We found that different build paths and offsets are a big problem as three out of three tested Free and Open Source Software projects all contained these variations. The results also showed that gcc s optimisation levels did not affect reproducibility, but link-time optimisation embeds a lot of unreproducible information in build artefacts. Lastly, the results showed that build paths, build ID s and randomness are the three most common groups of variations encountered in the wild and potential solutions for some variations were proposed.

Lastly, Pol Dellaiera completed his master thesis on Reproducibility in Software Engineering at the University of Mons, Belgium, under the supervision of Dr. Tom Mens, professor and director of the Software Engineering Lab.

The thesis serves as an introduction to the concept of reproducibility in software engineering, offering a comprehensive overview of formalizations using mathematical notations for key concepts and an empirical evaluation of several key tools. By exploring various case studies, methodologies and tools, the research aims to provide actionable insights for practitioners and researchers alike.

Development news In Debian this month, 4 reviews of Debian packages were added, 11 were updated and 14 were removed this month adding to our knowledge about identified issues. Only one issue types was updated, though, explaining that we don t vary the build path anymore.
On our mailing list this month, Bernhard M. Wiedemann wrote that whilst he had previously collected issues that introduce non-determinism he has now moved on to discuss about mitigations , in the sense of how can we avoid whole categories of problem without patching an infinite number of individual packages . In addition, Janneke Nieuwenhuizen announced the release of two versions of GNU Mes. [ ][ ]
In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.
In NixOS, with the 24.05 release out, it was again validated that our minimal ISO is reproducible by building it on a virtual machine with no access to the binary cache.
What s more, we continued to write patches in order to fix specific reproducibility issues, including Bernhard M. Wiedemann writing three patches (for qutebrowser, samba and systemd), Chris Lamb filing Debian bug #1074214 against the fastfetch package and Arnout Engelen proposing fixes to refind and for the Scala compiler [ .
Lastly, diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb uploaded two versions (270 and 271) to Debian, and made the following changes as well:

• Drop Build-Depends on liblz4-tool in order to fix Debian bug #1072575. [ ]
• Update tests to support zipdetails version 4.004 that is shipped with Perl 5.40. [ ]

Website updates There were a number of improvements made to our website this month, including Akihiro Suda very helpfully making the <h4> elements more distinguishable from the <h3> level [ ][ ] as well as adding a guide for Dockerfile reproducibility [ ]. In addition Fay Stegerman added two tools, apksigcopier and reproducible-apk-tools, to our Tools page.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In June, a number of changes were made by Holger Levsen, including:

• Marking the virt(32 64)c-armhf nodes as down. [ ]
• Granting a developer access to the osuosl4 node in order to debug a regression on the ppc64el architecture. [ ]
• Granting a developer access to the osuosl4 node. [ ][ ]

In addition, Mattia Rizzolo re-aligned the /etc/default/jenkins file with changes performed upstream [ ] and changed how configuration files are handled on the rb-mail1 host. [ ], whilst Vagrant Cascadian documented the failure of the virt32c and virt64c nodes after initial investigation [ ].

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org
• Twitter: @ReproBuilds

Hello folks, I ve been working this year on better integrating NVIDIA hardware with the Google Cloud Dataproc product (Hadoop on Google Cloud) running the default cluster node image. We have an open bug[1] in the initialization-actions repo regarding creation failures upon enabling secure boot. This is because with secure boot, kernel driver code has its signature verified before insmod places the symbols into kernel memory. The verification process involves reading trust root certificates from EFI variables, and validating that the signatures on the kernel driver either a) were made directly by one of the certificates in the boot sector or b) were made by certificates which chain up to one of them. This means that Dataproc disk images must have a certificate installed into them. My work on the internals will likely start producing images which have certificates from Google in them. In the meantime, however, our users are left without a mechanism to have both secure boot enabled and install out-of-tree kernel modules such as the NVIDIA GPU drivers. To that end, I ve got PR #83[2] open with the GoogleCloudDataproc/custom-images github repository. This PR introduces a new argument to the custom image creation script, trusted-cert , the argument of which is the path to a DER-encoded certificate to be included in the certificate database in the EFI variables of the disk s boot sector. I ve written up the instructions on creating a custom image with a trusted certificate here: https://github.com/cjac/custom-images/blob/secure-boot-custom-image/examples/secure-boot/README.md Here is a set of commands that can be used to create a Dataproc custom image with certificate installed to the EFI s db variable. You can run these commands from the root directory of a checkout such as this:

git clone https://github.com/cjac/custom-images.git --branch secure-boot-custom-image --single-branch
pushd custom-images
PROJECT_ID=your-project-here
PROJECT_NUMBER=your-project-nnnn-here
my_bucket=your-bucket-here
custom_image_zone=your-zone-here
gcloud projects add-iam-policy-binding $ PROJECT_ID  \
        --member=serviceAccount:$ PROJECT_NUMBER -compute@developer.gserviceaccount.com \
        --role=roles/secretmanager.secretAccessor
gcloud config set project $ PROJECT_ID 
gcloud auth login
eval $(bash examples/secure-boot/create-key-pair.sh)
metadata="public_secret_name=$ public_secret_name "
metadata="$ metadata ,private_secret_name=$ private_secret_name "
metadata="$ metadata ,secret_project=$ secret_project "
metadata="$ metadata ,secret_version=$ secret_version "
#dataproc_version=2.1-debian11
dataproc_version=2.2-debian12
#customization_script=examples/secure-boot/install-nvidia-driver-debian11.sh
customization_script=examples/secure-boot/install-nvidia-driver-debian12.sh
#image_name="nvidia-open-kernel-bullseye-$(date +%F)"
image_name="nvidia-open-kernel-bookworm-$(date +%F)"
disk_size_gb="50"
python generate_custom_image.py \
    --image-name $ image_name  \
    --dataproc-version $ dataproc_version  \
    --trusted-cert "tls/db.der" \
    --customization-script $ customization_script  \
    --metadata "$ metadata " \
    --zone "$ custom_image_zone " \
    --disk-size "$ disk_size_gb " \
    --no-smoke-test \
    --gcs-bucket "$ my_bucket "
popd

I d love to hear your feedback! [1] https://github.com/GoogleCloudDataproc/initialization-actions/issues/1058
[2] https://github.com/GoogleCloudDataproc/custom-images/pull/83

#
# Debian / libvirt
#
# By default, qemu guests network interfaces are created with the
# trustGuestRxFilters attribute set to no.  This setting blocks
# neighbor solicitation and other ipv6 multicast traffic.  To enable
# neighbor solicitations, modify your interface definition in your
# guest definition XML.
# For example if you have a guest named bird0 with an interface
# definition that looks like this:
$ sudo virsh dumpxml bird0   grep -A7 "type='direct'"
    <interface type='direct' trustGuestRxFilters='no'>
      <mac address='52:54:00:AA:BB:CC'/>
      <source dev='ens2f0np0' mode='bridge'/>
      <target dev='macvtap3'/>
      <model type='virtio'/>
      <alias name='net4'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </interface>
# Then use 'virsh edit bird0' change the attribute in the <interface>
# element to have the value 'yes' instead:
    <interface type='direct' trustGuestRxFilters='yes'>
      <mac address='52:54:00:AA:BB:CC'/>
      <source dev='ens2f0np0' mode='bridge'/>
      <target dev='macvtap3'/>
      <model type='virtio'/>
      <alias name='net4'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </interface>
# If the trustGuestRxFilters attribute is missing, it defaults to no,
# so be sure that the attribute exists and that the attribute has a
# value of 'yes'

Welcome to the May 2024 report from the Reproducible Builds project! In these reports, we try to outline what we have been up to over the past month and highlight news items in software supply-chain security more broadly. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. Table of contents:

1. A peek into build provenance for Homebrew
2. Distribution news
3. Mailing list news
4. Miscellaneous news
5. Two new academic papers
6. diffoscope
7. Website updates
8. Upstream patches
9. Reproducibility testing framework

---

A peek into build provenance for Homebrew Joe Sweeney and William Woodruff on the Trail of Bits blog wrote an extensive post about build provenance for Homebrew, the third-party package manager for MacOS. Their post details how each bottle (i.e. each release):

[ ] built by Homebrew will come with a cryptographically verifiable statement binding the bottle s content to the specific workflow and other build-time metadata that produced it. [ ] In effect, this injects greater transparency into the Homebrew build process, and diminishes the threat posed by a compromised or malicious insider by making it impossible to trick ordinary users into installing non-CI-built bottles.

The post also briefly touches on future work, including work on source provenance:

Homebrew s formulae already hash-pin their source artifacts, but we can go a step further and additionally assert that source artifacts are produced by the repository (or other signing identity) that s latent in their URL or otherwise embedded into the formula specification.

Distribution news In Debian this month, Johannes Schauer Marin Rodrigues (aka josch) noticed that the Debian binary package bash version 5.2.15-2+b3 was uploaded to the archive twice. Once to bookworm and once to sid but with differing content. This is problem for reproducible builds in Debian due its assumption that the package name, version and architecture triplet is unique. However, josch highlighted that

This example with bash is especially problematic since bash is Essential:yes, so there will now be a large portion of .buildinfo files where it is not possible to figure out with which of the two differing bash packages the sources were compiled.

In response to this, Holger Levsen performed an analysis of all .buildinfo files and found that this needs almost 1,500 binNMUs to fix the fallout from this bug. Elsewhere in Debian, Vagrant Cascadian posted about a Non-Maintainer Upload (NMU) sprint to take place during early June, and it was announced that there is now a #debian-snapshot IRC channel on OFTC to discuss the creation of a new source code archiving service to, perhaps, replace snapshot.debian.org. Lastly, 11 reviews of Debian packages were added, 15 were updated and 48 were removed this month adding to our extensive knowledge about identified issues. A number of issue types have been updated by Chris Lamb as well. [ ][ ]
Elsewhere in the world of distributions, deep within a larger announcement from Colin Percival about the release of version 14.1-BETA2, it was mentioned that the FreeBSD kernels are now built reproducibly.
In Fedora, however, the change proposal mentioned in our report for April 2024 was approved, so, per the ReproduciblePackageBuilds wiki page, the add-determinism tool is now running in new builds for Fedora 41 ( rawhide ). The add-determinism tool is a Rust program which, as its name suggests, adds determinism to files that are given as input by attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances . This is essentially the Fedora version of Debian s strip-nondeterminism. However, strip-nondeterminism is written in Perl, and Fedora did not want to pull Perl in the buildroot for every package. The add-determinism tool eliminates many causes of non-determinism and work is ongoing to continue the scope of packages it can operate on.

Mailing list news On our mailing list this month, regular contributor kpcyrd wrote to the list with an update on their source code indexing project, whatsrc.org. The whatsrc.org project, which was launched last month in response to the XZ Utils backdoor, now contains and indexes almost 250,000 unique source code archives. In their post, kpcyrd gives an example of its intended purpose, noting that it shown that whilst there seems to be consensus about [the] source code for zsh 5.9 in various Linux distributions, it does not align with the contents of the zsh Git repository . Holger Levsen also posted to the list with a pre-announcement of sorts for the 2024 Reproducible Builds summit. In particular:

[Whilst] the dates and location are not fixed yet, however if you don help us with finding a suitable location soon, it is very likely that we ll meet again in Hamburg in the 2nd half of September 2024 [ ].

Lastly, Frederic-Emmanuel Picca wrote to the list asking for help understanding the non-reproducible status of the Debian silx package and received replies from both Vagrant Cascadian and Chris Lamb.

Miscellaneous news strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month strip-nondeterminism version 1.14.0-1 was uploaded to Debian unstable by Chris Lamb chiefly to incorporate a change from Alex Muntada to avoid a dependency on Sub::Override to perform monkey-patching and break circular dependencies related to debhelper [ ]. Elsewhere in our tooling, Jelle van der Waa modified reprotest because the pipes module will be removed in Python version 3.13 [ ].
It was also noticed that a new blog post by Daniel Stenberg detailing How to verify a Curl release mentions the SOURCE_DATE_EPOCH environment variable. This is because:

The [curl] release tools document also contains another key component: the exact time stamp at which the release was done using integer second resolution. In order to generate a correct tarball clone, you need to also generate the new version using the old version s timestamp. Because the modification date of all files in the produced tarball will be set to this timestamp.

Furthermore, Fay Stegerman filed a bug against the Signal messenger app for Android to report that their reproducible builds cannot, in fact, be reproduced. However, Fay is quick to note that she has:

found zero evidence of any kind of compromise. Some differences are yet unexplained but everything I found seems to be benign. I am disappointed that Reproducible Builds have been broken for months but I have zero reason to doubt Signal s security in any way.

Lastly, it was observed that there was a concise and diagrammatic overview of supply chain threats on the SLSA website.

Two new academic papers Two new scholarly papers were published this month. Firstly, Mathieu Acher, Beno t Combemale, Georges Aaron Randrianaina and Jean-Marc J z quel of University of Rennes on Embracing Deep Variability For Reproducibility & Replicability. The authors describe their approach as follows:

In this short [vision] paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g., language, libraries, compiler, virtual machine, OS, environment variables, etc.). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

(A PDF of this article is available.)
Secondly, Ludovic Court s, Timothy Sample, Simon Tournier and Stefano Zacchiroli have collaborated to publish a paper on Source Code Archiving to the Rescue of Reproducible Deployment. Their paper was motivated because:

The ability to verify research results and to experiment with methodologies are core tenets of science. As research results are increasingly the outcome of computational processes, software plays a central role. GNU Guix is a software deployment tool that supports reproducible software deployment, making it a foundation for computational research workflows. To achieve reproducibility, we must first ensure the source code of software packages Guix deploys remains available.

(A PDF of this article is also available.)

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions 266, 267, 268 and 269 to Debian, making the following changes:

• New features:
  • Use xz --list to supplement output when comparing .xz archives; essential when metadata differs. (#1069329)
  • Include xz --verbose --verbose (ie. double) output. (#1069329)
  • Strip the first line from the xz --list output. [ ]
  • Only include xz --list --verbose output if the xz has no other differences. [ ]
  • Actually append the xz --list after the container differences, as it simplifies a lot. [ ]
• Testing improvements:
  • Allow Debian testing to fail right now. [ ]
  • Drop apktool from Build-Depends; we can still test APK functionality via autopkgtests. (#1071410)
  • Add a versioned dependency for at least version 5.4.5 for the xz tests as they fail under (at least) version 5.2.8. (#374)
  • Fix tests for 7zip 24.05. [ ][ ]
  • Fix all tests after additon of xz --list. [ ][ ]
• Misc:
  • Update copyright years. [ ]

In addition, James Addison fixed an issue where the HTML output showed only the first difference in a file, while the text output shows all differences [ ][ ][ ], Sergei Trofimovich amended the 7zip version test for older 7z versions that include the string [64] [ ][ ] and Vagrant Cascadian relaxed the versioned dependency to allow version 5.4.1 for the xz tests [ ] and proposed updates to guix for versions 267, 268 and pushed version 269 to Guix. Furthermore, Eli Schwartz updated the diffoscope.org website in order to explain how to install diffoscope on Gentoo [ ].

Website updates There were a number of improvements made to our website this month, including Chris Lamb making the print CSS stylesheet nicer [ ]. Fay Stegerman made a number of updates to the page about the SOURCE_DATE_EPOCH environment variable [ ][ ][ ] and Holger Levsen added some of their presentations to the Resources page. Furthermore, IOhannes zm lnig stipulated support for SOURCE_DATE_EPOCH in clang version 16.0.0+ [ ], Jan Zerebecki expanded the Formal definition page and fixed a number of typos on the Buy-in page [ ] and Simon Josefsson fixed the link to Trisquel GNU/Linux on the Projects page [ ].

Upstream patches This month, we wrote a number of patches to fix specific reproducibility issues, including:

• Bernhard M. Wiedemann:
  • nauty (CPU-detection issue)
  • emacs (ASLR)
• Chris Lamb:
  • #1070754 filed against gensio.
  • #1071064 filed against tkgate.
  • #1072094 filed against ruby-pgplot.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In May, a number of changes were made by Holger Levsen:

• Debian-related changes:
  • Enable the rebuilder-snapshot API on osuosl4. [ ]
  • Schedule the i386 architecture a bit more often. [ ]
  • Adapt cleanup_nodes.sh to the new way of running our build services. [ ]
  • Add 8 more workers for the i386 architecture. [ ]
  • Update configuration now that the infom07 and infom08 nodes have been reinstalled as real i386 systems. [ ]
  • Make diffoscope timeouts more visible on the #debian-reproducible-changes IRC channel. [ ]
  • Mark the cbxi4a-armhf node as down. [ ][ ]
  • Only install the hdmi2usb-mode-switch package only on Debian bookworm and earlier [ ] and only install the haskell-platform package on Debian bullseye [ ].
• Misc:
  • Install the ntpdate utility as we need it later. [ ]
  • Document the progress on the i386 architecture nodes at Infomaniak. [ ]
  • Drop an outdated and unnoticed notice. [ ]
  • Add live_setup_schroot to the list of so-called zombie jobs. [ ]

In addition, Mattia Rizzolo reinstalled the infom07 and infom08 nodes [ ] and Vagrant Cascadian marked the cbxi4a node as online [ ].

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

I attended the school Yarra Valley Grammer (then Yarra Valley Anglican School which I will refer to as YV ) and completed year 12 in 1990. The school is currently in the news for a spreadsheet some boys made rating girls where unrapeable was one of the ratings. The school s PR team are now making claims like Respect for each other is in the DNA of this school . I d like to know when this DNA change allegedly occurred because respect definitely wasn t in the school DNA in 1990! Before I go any further I have to note that if the school threatens legal action against me for this post it will be clear evidence that they don t believe in respect. The actions of that school have wronged me, several of my friends, many people who aren t friends but who I wish they hadn t had to suffer and I hadn t had to witness it, and presumably countless others that I didn t witness. If they have any decency they would not consider legal action but I have learned that as an institution they have no decency so I have to note that they should read the Wikipedia page about the Streisand Effect [1] and keep it in mind before deciding on a course of action. I think it is possible to create a school where most kids enjoy being there and enjoy learning, where hardly any students find it a negative experience and almost no-one finds it traumatic. But it is not possible to do that with the way schools tend to be run. When I was at high school there was a general culture that minor sex crimes committed by boys against boys weren t a problem, this probably applied to all high schools. Things like ripping a boy s pants off (known as dakking ) were considered a big joke. If you accept that ripping the pants off an unwilling boy is a good thing (as was the case when I was at school) then that leads to thinking that describing girls as unrapeable is acceptable. The Wikipedia page for Pantsing [2] has a reference for this issue being raised as a serious problem by the British Secretary of State for Education and Skills Alan Johnson in 2007. So this has continued to be a widespread problem around the world. Has YV become better than other schools in dealing with it or is Dakking and Wedgies as well accepted now as it was when I attended? There is talk about schools preparing kids for the workforce, but grabbing someone s underpants without consent will result in instant dismissal from almost all employment. There should be more tolerance for making mistakes at school than at work, but they shouldn t tolerate what would be serious crimes in other contexts. For work environments there have been significant changes to what is accepted, so it doesn t seem unreasonable to expect that schools can have a similar change in culture. One would hope that spending 6 years wondering who s going to grab your underpants next would teach boys the importance of consent and some sympathy for victims of other forms of sexual assault. But that doesn t seem to happen, apparently it s often the opposite. When I was young Autism wasn t diagnosed for anyone who was capable of having a normal life. Teachers noticed that I wasn t like other kids, some were nice, but some encouraged other boys to attack me as a form of corporal punishment by proxy not a punishment for doing anything wrong (detentions were adequate for that) but for being different. The lesson kids will take from that sort of thing is that if you are in a position of power you can mistreat other people and get away with it. There was a girl in my year level at YV who would probably be diagnosed as Autistic by today s standards, the way I witnessed her being treated was considerably worse than what was described in the recent news reports but it is quite likely that worse things have been done recently which haven t made the news yet. If this issue is declared to be over after 4 boys were expelled then I ll count that as evidence of a cover-up. These things don t happen in a vacuum, there s a culture that permits and encourages it. The word respect has different meanings, it can mean treat a superior as the master or treat someone as a human being . The phrase if you treat me with respect I ll treat you with respect usually means if you treat me as the boss then I ll treat you as a human being . The distinction is very important when discussing respect in schools. If teachers are considered the ultimate bosses whose behaviour can never be questioned then many boys won t need much help from Andrew Tate in developing the belief that they should be the boss of girls in the same way. Do any schools have a process for having students review teachers? Does YV have an ombudsman to take reports of misbehaving teachers in the way that corporations typically have an ombudsman to take reports about bad managers? Any time you have people whose behaviour is beyond scrutiny or oversight you will inevitably have bad people apply for jobs, then bad things will happen and it will create a culture of bad behaviour. If teachers can treat kids badly then kids will treat other kids badly, and this generally ends with girls being treated badly by boys. My experience at YV was that kids barely had the status of people. It seemed that the school operated more as a caretaker of the property of parents than as an organisation that cares for people. The current YV website has a Whistleblower policy [3] that has only one occurrence of the word student and that is about issues that endanger the health or safety of students. Students are the people most vulnerable to reprisal for complaining and not being listed as an eligible whistleblower shows their status. The web site also has a flowchart for complaints and grievances [4] which doesn t describe any policy for a complaint to be initiated by a student. One would hope that parents would advocate for their children but that often isn t the case. When discussing the possibility of boys being bullied at school with parents I ve had them say things like my son wouldn t be so weak that he would be bullied , no boy will tell his parents about being bullied if that s their attitude! I imagine that there are similar but different issues of parents victim-blaming when their daughter is bullied (presumably substituting immoral for weak) but don t have direct knowledge of the topic. The experience of many kids is being disrespected by their parents, the school system, and often siblings too. A school can t solve all the world s problems but can ideally be a refuge for kids who have problems at home. When I was at school the culture in the country and the school was homophobic. One teacher when discussing issues such as how students could tell him if they had psychological problems and no-one else to talk to said some things like the Village People make really good music which was the only time any teacher said anything like It s OK to be gay (the Village People were the gayest pop group at the time). A lot of the bullying at school had a sexual component to it. In addition to the wedgies and dakking (which while not happening often was something you had to constantly be aware of) I routinely avoided PE classes where a shower was necessary because of a thug who hung around by the showers and looked hungrily at my penis, I don t know if he had a particular liking to mine or if he stared at everyone that way. Flashing and perving was quite common in change rooms. Presumably as such boy-boy sexual misbehaviour was so accepted that led to boys mistreating girls. I currently work for a company that is active in telling it s employees about the possibility of free psychological assistance. Any employee can phone a psychologist to discuss problems (whether or not they are work related) free of charge and without their manager or colleagues knowing. The company is billed and is only given a breakdown of the number of people who used the service and roughly what the issue was (work stress, family, friends, grief, etc). When something noteworthy happens employees are given reminders about this such as if you need help after seeing a homeless man try to steal a laptop from the office then feel free to call the assistance program . Do schools offer something similar? With the school fees paid to a school like YV they should be able to afford plenty of psychologist time. Every day I was at YV I saw something considerably worse than laptop theft, most days something was done to me. The problems with schools are part of larger problems with society. About half of the adults in Australia still support the Liberal party in spite of their support of Christian Porter, Cardinal Pell, and Bruce Lehrmann. It s not logical to expect such parents to discourage their sons from mistreating girls or to encourage their daughters to complain when they are mistreated. The Anglican church has recently changed it s policy to suggesting that victims of sexual abuse can contact the police instead of or in addition to the church, previously they had encouraged victims to only contact the church which facilitated cover-ups. One would hope that schools associated with the Anglican church have also changed their practices towards such things. I approve of the respect is in our DNA concept, it s like Google s former slogan of Don t be evil which is something that they can be bound to. Here s a list of questions that could be asked of schools (not just YV but all schools) by journalists when reporting on such things:

1. Do you have a policy of not trying to silence past students who have been treated badly?
2. Do you take all sexual assaults seriously including wedgies and dakking?
3. Do you take all violence at school seriously? Even if there s no blood? Even if the victim says they don t want to make an issue of it?
4. What are your procedures to deal with misbehaviour from teachers? Do the students all know how to file complaints? Do they know that they can file a complaint if they aren t the victim?
5. Does the school have policies against homophobia and transphobia and are they enforced?
6. Does the school offer free psychological assistance to students and staff who need it? NB This only applies to private schools like YV that have huge amounts of money, public schools can t afford that.
7. Are serious incidents investigated by people who are independent of the school and who don t have a vested interest in keeping things quiet?
8. Do you encourage students to seek external help from organisations like the ones on the resources list of the Grace Tame Foundation [5]? Having your own list of recommended external organisations would be good too.

Counter Arguments I ve had practice debating such things, here s some responses to common counter arguments.

• Teachers are nice people how dare you criticise them. Teachers like any other large group of people includes good and bad people. The issue is how well the good people are supported in doing good things, how much effort is spent on tracking down and removing the bad people, and how much effort is spent training people to be the best version of themselves. Also my father worked as a teacher so I really don t think that all teachers are bad.
• Teachers are overworked and underpaid and you shouldn t criticise them. When a school has 25 students in a class whose parents each pay $30,000 per annum the school can afford to pay as much as is necessary. Arguments about teachers being overworked and underpaid are a criticism of the organisation of private school and of government priorities for public schools not a counter argument to criticisms of the way schools operate.
• When I went to school no bad things happened. Did you go to YV? If not then your experience isn t relevant to this post.
• I was a prefect at YV and didn t see any bad things, if you saw bad things you should have reported it to me. I was not aware of any prefect who had a history of opposing bullying in previous years, I can think of some who had a history of encouraging it. Prefects were selected on the basis of supporting the system so anyone who would be expected to try to change things would have been rejected.
• Children will make false and frivolous claims so we should ignore most of what they say, therefore complaints should only come from parents. Children have considerably less ability to lie than adults and the senior teachers are much better at detecting lies than most people. Sorting out accurate claims from false ones shouldn t be difficult but if you reject all criticism as false claims then you will definitely miss reports of bad things and allow problems to continue.
• I had a hard time at school and I turned out fine. If having bad things done to you doesn t make you want to protect others from the same things then you didn t turn out fine at all.
• Kids need to toughen up to survive the real world. The real world that I live in doesn t involve much violence at all, even having someone raise their voice at work is uncommon. Of the situations where being tough due to my experience at YV has been useful almost all of them involve me choosing to help someone I don t know in a dangerous situation while other men pretend that they didn t even notice it. The real solution is to create a world with less violence and a large part of that involves improving schools.

Conclusion I don t think that YV is necessarily worse than other schools, although I m sure that representatives of other private schools are now working to assure parents of students and prospective students that they are. I don t think that all the people who were employed as teachers there when I attended were bad people, some of them were nice people who were competent teachers. But a few good people can t turn around a bad system. I will note that when I attended all the sports teachers were decent people, it was the only department I could say such things about. But sports involves situations that can lead to a bad result, issues started at other times and places can lead to violence or harassment in PE classes regardless of how good the teachers are. Teachers who know that there are problems need to be able to raise issues with the administration. When a teacher quits teaching to join the clergy and another teacher describes it as a loss for the clergy but a gain for YV it raises the question of why the bad teacher in question couldn t have been encouraged to leave earlier. A significant portion of the population will do whatever is permitted. If you say no teacher would ever bully a student so we don t need to look out for that then some teacher will do exactly that. I hope that this will lead to changes both in YV and in other schools. But if they declare this issue as resolved after expelling 4 students then something similar or worse will happen again. At least now students know that when this sort of thing happens they can send evidence to journalists to get some action.

• [1] https://en.wikipedia.org/wiki/Streisand_effect
• [2] https://en.wikipedia.org/wiki/Pantsing
• [3] https://www.yvg.vic.edu.au/siteassets/Whistleblower-Policy-FINAL.pdf
• [4] https://tinyurl.com/254crjqr
• [5] https://www.thegracetamefoundation.org.au/resources

I've been on Chromebooks for a while. However, since I had to recently try a Mac, I figured it was time to give Firefox a try again. After two weeks of trying, I've given up. At least for myself, I figured I'd write down the reasons I've given up.Reasons:

• There is no way from the tab context menu to move a tab between windows. I typically try to keep no more than 3 windows open at a time. Ideally one, but maybe a second. Without the ability to through the context menu to move a tab, I need a very large screen (not a laptop screen) to move tabs between windows.
• I couldn't find a way to take a URL and turn it into a custom search. This really is a critical feature as it allows me to use short names to access specific searches. E.g. search code (cs), show a calendar (c), etc.