$ git diff --shortstat v1.11.5
 493 files changed, 25015 insertions(+), 21135 deletions(-)

New outlet service The major change in Akvorado 2.0 is splitting the inlet service into two parts: the inlet and the outlet. Previously, the inlet handled all flow processing: receiving, decoding, and enrichment. Flows were then sent to Kafka for storage in ClickHouse:

Akvorado flow processing before the introduction of the outlet service

Network flows reach the inlet service using UDP, an unreliable protocol. The inlet must process them fast enough to avoid losing packets. To handle a high number of flows, the inlet spawns several sets of workers to receive flows, fetch metadata, and assemble enriched flows for Kafka. Many configuration options existed for scaling, which increased complexity for users. The code needed to avoid blocking at any cost, making the processing pipeline complex and sometimes unreliable, particularly the BMP receiver.¹ Adding new features became difficult without making the problem worse.² In Akvorado 2.0, the inlet receives flows and pushes them to Kafka without decoding them. The new outlet service handles the remaining tasks:

Akvorado flow processing after the introduction of the outlet service

This change goes beyond a simple split:³ the outlet now reads flows from Kafka and pushes them to ClickHouse, two tasks that Akvorado did not handle before. Flows are heavily batched to increase efficiency and reduce the load on ClickHouse using ch-go, a low-level Go client for ClickHouse. When batches are too small, asynchronous inserts are used (e20645). The number of outlet workers scales dynamically (e5a625) based on the target batch size and latency (50,000 flows and 5 seconds by default). This new architecture also allows us to simplify and optimize the code. The outlet fetches metadata synchronously (e20645). The BMP component becomes simpler by removing cooperative multitasking (3b9486). Reusing the same RawFlow object to decode protobuf-encoded flows from Kafka reduces pressure on the garbage collector (8b580f). The effect on Akvorado s overall performance was somewhat uncertain, but a user reported 35% lower CPU usage after migrating from the previous version, plus resolution of the long-standing BMP component issue.

Other changes This new version includes many miscellaneous changes, such as completion for source and destination ports (f92d2e), and automatic restart of the orchestrator service (0f72ff) when configuration changes to avoid a common pitfall for newcomers. Let s focus on some key areas for this release: observability, documentation, CI, Docker, Go, and JavaScript.

Observability Akvorado exposes metrics to provide visibility into the processing pipeline and help troubleshoot issues. These are available through Prometheus HTTP metrics endpoints, such as /api/v0/inlet/metrics. With the introduction of the outlet, many metrics moved. Some were also renamed (4c0b15) to match Prometheus best practices. Kafka consumer lag was added as a new metric (e3a778). If you do not have your own observability stack, the Docker Compose setup shipped with Akvorado provides one. You can enable it by activating the profiles introduced for this purpose (529a8f). The prometheus profile ships Prometheus to store metrics and Alloy to collect them (2b3c46, f81299, and 8eb7cd). Redis and Kafka metrics are collected through the exporter bundled with Alloy (560113). Other metrics are exposed using Prometheus metrics endpoints and are automatically fetched by Alloy with the help of some Docker labels, similar to what is done to configure Traefik. cAdvisor was also added (83d855) to provide some container-related metrics. The loki profile ships Loki to store logs (45c684). While Alloy can collect and ship logs to Loki, its parsing abilities are limited: I could not find a way to preserve all metadata associated with structured logs produced by many applications, including Akvorado. Vector replaces Alloy (95e201) and features a domain-specific language, VRL, to transform logs. Annoyingly, Vector currently cannot retrieve Docker logs from before it was started. Finally, the grafana profile ships Grafana, but the shipped dashboards are broken. This is planned for a future version.

Documentation The Docker Compose setup provided by Akvorado makes it easy to get the web interface up and running quickly. However, Akvorado requires a few mandatory steps to be functional. It ships with comprehensive documentation, including a chapter about troubleshooting problems. I hoped this documentation would reduce the support burden. It is difficult to know if it works. Happy users rarely report their success, while some users open discussions asking for help without reading much of the documentation. In this release, the documentation was significantly improved.

$ git diff --shortstat v1.11.5 -- console/data/docs
 10 files changed, 1873 insertions(+), 1203 deletions(-)

The documentation was updated (fc1028) to match Akvorado s new architecture. The troubleshooting section was rewritten (17a272). Instructions on how to improve ClickHouse performance when upgrading from versions earlier than 1.10.0 was added (5f1e9a). An LLM proofread the entire content (06e3f3). Developer-focused documentation was also improved (548bbb, e41bae, and 871fc5). From a usability perspective, table of content sections are now collapsable (c142e5). Admonitions help draw user attention to important points (8ac894).

Example of use of admonitions in Akvorado's documentation

Continuous integration This release includes efforts to speed up continuous integration on GitHub. Coverage and race tests run in parallel (6af216 and fa9e48). The Docker image builds during the tests but gets tagged only after they succeed (8b0dce).

GitHub workflow to test and build Akvorado

End-to-end tests (883e19) ensure the shipped Docker Compose setup works as expected. Hurl runs tests on various HTTP endpoints, particularly to verify metrics (42679b and 169fa9). For example:

## Test inlet has received NetFlow flows
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_flow_input_udp_packets_total job="akvorado-inlet",listener=":2055" )
HTTP 200
[Captures]
inlet_receivedflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_receivedflows" > 10
## Test inlet has sent them to Kafka
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_kafka_sent_messages_total job="akvorado-inlet" )
HTTP 200
[Captures]
inlet_sentflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_sentflows" >=   inlet_receivedflows  

Docker Akvorado ships with a comprehensive Docker Compose setup to help users get started quickly. It ensures a consistent deployment, eliminating many configuration-related issues. It also serves as a living documentation of the complete architecture. This release brings some small enhancements around Docker:

• an example to configure Traefik for TLS with Let s Encrypt (1a27bb),
• HTTP compression (bee9a5), and
• support of IPv6 (a74a41).

Previously, many Docker images were pulled from the Bitnami Containers library. However, VMWare acquired Bitnami in 2019 and Broadcom acquired VMWare in 2023. As a result, Bitnami images were deprecated in less than a month. This was not really a surprise⁴. Previous versions of Akvorado had already started moving away from them. In this release, the Apache project s Kafka image replaces the Bitnami one (1eb382). Thanks to the switch to KRaft mode, Zookeeper is no longer needed (0a2ea1, 8a49ca, and f65d20). Akvorado s Docker images were previously compiled with Nix. However, building AArch64 images on x86-64 is slow because it relies on QEMU userland emulation. The updated Dockerfile uses multi-stage and multi-platform builds: one stage builds the JavaScript part on the host platform, one stage builds the Go part cross-compiled on the host platform, and the final stage assembles the image on top of a slim distroless image (268e95 and d526ca).

# This is a simplified version
FROM --platform=$BUILDPLATFORM node:20-alpine AS build-js
RUN apk add --no-cache make
WORKDIR /build
COPY console/frontend console/frontend
COPY Makefile .
RUN make console/data/frontend
FROM --platform=$BUILDPLATFORM golang:alpine AS build-go
RUN apk add --no-cache make curl zip
WORKDIR /build
COPY . .
COPY --from=build-js /build/console/data/frontend console/data/frontend
RUN go mod download
RUN make all-indep
ARG TARGETOS TARGETARCH TARGETVARIANT VERSION
RUN make
FROM gcr.io/distroless/static:latest
COPY --from=build-go /build/bin/akvorado /usr/local/bin/akvorado
ENTRYPOINT [ "/usr/local/bin/akvorado" ]

When building for multiple platforms with --platform linux/amd64,linux/arm64,linux/arm/v7, the build steps until the highlighted line execute only once for all platforms. This significantly speeds up the build. Akvorado now ships Docker images for these platforms: linux/amd64, linux/amd64/v3, linux/arm64, and linux/arm/v7. When requesting ghcr.io/akvorado/akvorado, Docker selects the best image for the current CPU. On x86-64, there are two choices. If your CPU is recent enough, Docker downloads linux/amd64/v3. This version contains additional optimizations and should run faster than the linux/amd64 version. It would be interesting to ship an image for linux/arm64/v8.2, but Docker does not support the same mechanism for AArch64 yet (792808).

Go This release includes many changes related to Go but not visible to the users.

Toolchain In the past, Akvorado supported the two latest Go versions, preventing immediate use of the latest enhancements. The goal was to allow users of stable distributions to use Go versions shipped with their distribution to compile Akvorado. However, this became frustrating when interesting features, like go tool, were released. Akvorado 2.0 requires Go 1.25 (77306d) but can be compiled with older toolchains by automatically downloading a newer one (94fb1c).⁵ Users can still override GOTOOLCHAIN to revert this decision. The recommended toolchain updates weekly through CI to ensure we get the latest minor release (5b11ec). This change also simplifies updates to newer versions: only go.mod needs updating. Thanks to this change, Akvorado now uses wg.Go() (77306d) and I have started converting some unit tests to the new test/synctest package (bd787e, 7016d8, and 159085).

Testing When testing equality, I use a helper function Diff() to display the differences when it fails:

got := input.Keys()
expected := []int 1, 2, 3 
if diff := helpers.Diff(got, expected); diff != ""  
    t.Fatalf("Keys() (-got, +want):\n%s", diff)
 

This function uses kylelemons/godebug. This package is no longer maintained and has some shortcomings: for example, by default, it does not compare struct private fields, which may cause unexpectedly successful tests. I replaced it with google/go-cmp, which is stricter and has better output (e2f1df).

Another package for Kafka Another change is the switch from Sarama to franz-go to interact with Kafka (756e4a and 2d26c5). The main motivation for this change is to get a better concurrency model. Sarama heavily relies on channels and it is difficult to understand the lifecycle of an object handed to this package. franz-go uses a more modern approach with callbacks⁶ that is both more performant and easier to understand. It also ships with a package to spawn fake Kafka broker clusters, which is more convenient than the mocking functions provided by Sarama.

Improved routing table for BMP To store its routing table, the BMP component used kentik/patricia, an implementation of a patricia tree focused on reducing garbage collection pressure. gaissmai/bart is a more recent alternative using an adaptation of [Donald Knuth s ART algorithm][] that promises better performance and delivers it: 90% faster lookups and 27% faster insertions (92ee2e and fdb65c). Unlike kentik/patricia, gaissmai/bart does not help efficiently store values attached to each prefix. I adapted the same approach as kentik/patricia to store route lists for each prefix: store a 32-bit index for each prefix, and use it to build a 64-bit index for looking up routes in a map. This leverages Go s efficient map structure. gaissmai/bart also supports a lockless routing table version, but this is not simple because we would need to extend this to the map storing the routes and to the interning mechanism. I also attempted to use Go s new unique package to replace the intern package included in Akvorado, but performance was worse.⁷

Miscellaneous Previous versions of Akvorado were using a custom Protobuf encoder for performance and flexibility. With the introduction of the outlet service, Akvorado only needs a simple static schema, so this code was removed. However, it is possible to enhance performance with planetscale/vtprotobuf (e49a74, and 8b580f). Moreover, the dependency on protoc, a C++ program, was somewhat annoying. Therefore, Akvorado now uses buf, written in Go, to convert a Protobuf schema into Go code (f4c879). Another small optimization to reduce the size of the Akvorado binary by 10 MB was to compress the static assets embedded in Akvorado in a ZIP file. It includes the ASN database, as well as the SVG images for the documentation. A small layer of code makes this change transparent (b1d638 and e69b91).

JavaScript Recently, two large supply-chain attacks hit the JavaScript ecosystem: one affecting the popular packages chalk and debug and another impacting the popular package @ctrl/tinycolor. These attacks also exist in other ecosystems, but JavaScript is a prime target due to heavy use of small third-party dependencies. The previous version of Akvorado relied on 653 dependencies. npm-run-all was removed (3424e8, 132 dependencies). patch-package was removed (625805 and e85ff0, 69 dependencies) by moving missing TypeScript definitions to env.d.ts. eslint was replaced with oxlint, a linter written in Rust (97fd8c, 125 dependencies, including the plugins). I switched from npm to Pnpm, an alternative package manager (fce383). Pnpm does not run install scripts by default⁸ and prevents installing packages that are too recent. It is also significantly faster.⁹ Node.js does not ship Pnpm but it ships Corepack, which allows us to use Pnpm without installing it. Pnpm can also list licenses used by each dependency, removing the need for license-compliance (a35ca8, 42 dependencies). For additional speed improvements, beyond switching to Pnpm and Oxlint, Vite was replaced with its faster Rolldown version (463827). After these changes, Akvorado only pulls 225 dependencies.

Next steps I would like to land three features in the next version of Akvorado:

• Add Grafana dashboards to complete the observability stack. See issue #1906 for details.
• Integrate OVH s Grafana plugin by providing a stable API for such integrations. Akvorado s web console would still be useful for browsing results, but if you want to build and share dashboards, you should switch to Grafana. See issue #1895.
• Move some work currently done in ClickHouse (custom dictionaries, GeoIP and IP enrichment) back into the outlet service. This should give more flexibility for adding features like the one requested in issue #1030. See issue #2006.

---

I started working on splitting the inlet into two parts more than one year ago. I found more motivation in recent months, partly thanks to Claude Code, which I used as a rubber duck. Almost none of the produced code was kept:¹⁰ it is like an intern who does not learn.

---

1. Many attempts were made to make the BMP component both performant and not blocking. See for example PR #254, PR #255, and PR #278. Despite these efforts, this component remained problematic for most users. See issue #1461 as an example.
2. Some features have been pushed to ClickHouse to avoid the processing cost in the inlet. See for example PR #1059.
3. This is the biggest commit:

   $ git show --shortstat ac68c5970e2c   tail -1
   231 files changed, 6474 insertions(+), 3877 deletions(-)
   

4. Broadcom is known for its user-hostile moves. Look at what happened with VMWare.
5. As a Debian developer, I dislike these mechanisms that circumvent the distribution package manager. The final straw came when Go 1.25 spent one month in the Debian NEW queue, an arbitrary mechanism I don t like at all.
6. In the early years of Go, channels were heavily promoted. Sarama was designed during this period. A few years later, a more nuanced approach emerged. See notably Go channels are bad and you should feel bad.
7. This should be investigated further, but my theory is that the intern package uses 32-bit integers, while unique uses 64-bit pointers. See commit 74e5ac.
8. This is also possible with npm. See commit dab2f7.
9. An even faster alternative is Bun, but it is less available.
10. The exceptions are part of the code for the admonition blocks, the code for collapsing the table of content, and part of the documentation.

Changes in version 0.1.14 (2025-09-13)

• simdjson was upgraded to version 4.0.0 (Dirk in #96
• Continuous integration now relies a token for codecov.io

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Changes in RcppSMC version 0.2.9 (2025-09-09)

• Adjust to RcppArmadillo 15.0.* by setting ARMA_USE_CURRENT and updating two expressions from deprecated code
• Rely on r-ci GitHub Action which includes the bootstrap step

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

Welcome to the August 2025 report from the Reproducible Builds project! Welcome to the latest report from the Reproducible Builds project for August 2025. These monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. If you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website. In this report:

1. Reproducible Builds Summit 2025
2. Reproducible Builds and live-bootstrap at WHY2025
3. DALEQ Explainable Equivalence for Java Bytecode
4. Reproducibility regression identifies issue with AppArmor security policies
5. Rust toolchain fixes
6. Distribution work
7. diffoscope
8. Website updates
9. Reproducibility testing framework
10. Upstream patches

Reproducible Builds Summit 2025 Please join us at the upcoming Reproducible Builds Summit, set to take place from October 28th 30th 2025 in Vienna, Austria!** We are thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving. If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. Registration is open until 20th September 2025, and we are very much looking forward to seeing many readers of these reports there!

Reproducible Builds and live-bootstrap at WHY2025 WHY2025 (What Hackers Yearn) is a nonprofit outdoors hacker camp that takes place in Geestmerambacht in the Netherlands (approximately 40km north of Amsterdam). The event is organised for and by volunteers from the worldwide hacker community, and knowledge sharing, technological advancement, experimentation, connecting with your hacker peers, forging friendships and hacking are at the core of this event . At this year s event, Frans Faase gave a talk on live-bootstrap, an attempt to provide a reproducible, automatic, complete end-to-end bootstrap from a minimal number of binary seeds to a supported fully functioning operating system . Frans talk is available to watch on video and his slides are available as well.

DALEQ Explainable Equivalence for Java Bytecode Jens Dietrich of the Victoria University of Wellington, New Zealand and Behnaz Hassanshahi of Oracle Labs, Australia published an article this month entitled DALEQ Explainable Equivalence for Java Bytecode which explores the options and difficulties when Java binaries are not identical despite being from the same sources, and what avenues are available for proving equivalence despite the lack of bitwise correlation:

[Java] binaries are often not bitwise identical; however, in most cases, the differences can be attributed to variations in the build environment, and the binaries can still be considered equivalent. Establishing such equivalence, however, is a labor-intensive and error-prone process.

Jens and Behnaz therefore propose a tool called DALEQ, which:

disassembles Java byte code into a relational database, and can normalise this database by applying Datalog rules. Those databases can then be used to infer equivalence between two classes. Notably, equivalence statements are accompanied with Datalog proofs recording the normalisation process. We demonstrate the impact of DALEQ in an industrial context through a large-scale evaluation involving 2,714 pairs of jars, comprising 265,690 class pairs. In this evaluation, DALEQ is compared to two existing bytecode transformation tools. Our findings reveal a significant reduction in the manual effort required to assess non-bitwise equivalent artifacts, which would otherwise demand intensive human inspection. Furthermore, the results show that DALEQ outperforms existing tools by identifying more artifacts rebuilt from the same code as equivalent, even when no behavioral differences are present.

Jens also posted this news to our mailing list.

Reproducibility regression identifies issue with AppArmor security policies Tails developer intrigeri has tracked and followed a reproducibility regression in the generation of AppArmor policy caches, and has identified an issue with the 4.1.0 version of AppArmor. Although initially tracked on the Tails issue tracker, intrigeri filed an issue on the upstream bug tracker. AppArmor developer John Johansen replied, confirming that they can reproduce the issue and went to work on a draft patch. Through this, John revealed that it was caused by an actual underlying security bug in AppArmor that is to say, it resulted in permissions not (always) matching what the policy intends and, crucially, not merely a cache reproducibility issue. Work on the fix is ongoing at time of writing.

Rust toolchain fixes Rust Clippy is a linting tool for the Rust programming language. It provides a collection of lints (rules) designed to identify common mistakes, stylistic issues, potential performance problems and unidiomatic code patterns in Rust projects. This month, however, Sosth ne Gu don filed a new issue in the GitHub requesting a new check that would lint against non deterministic operations in proc-macros, such as iterating over a HashMap .

Distribution work In Debian this month:

• Holger made extensive updates to Debian package reproducibility testing infrastructure this month, including:
  • Upgrading all of the nodes to Debian trixie.
  • Adding tests for the new Debian forky release.
  • Dropping tests for Debian bookworm.
  • Dropping support for the armhf architecture. From July 2015, Vagrant Cascadian has been hosting a zoo of approximately 35 armhf systems which were used for building Debian packages for that architecture.
• Holger Levsen also uploaded strip-nondeterminism, our program that improves reproducibility by stripping out non-deterministic information such as timestamps or other elements introduced during packaging. This new version, 1.14.2-1, adds some metadata to aid the deputy tool. ( #1111947)
• 8 reviews of Debian packages were added, 5 were updated and 5 were removed this month adding to our knowledge about identified issues.
• Marc Haber posted to our mailing list this month asking for assistance with the duperemove package in Debian, which appears to be an issue where the order the object files are linked together is dependent on the underlying filesystem . Chris Lamb provided a detailed analysis, including a suggestion that this can be resolved by adding a locale-agnostic sort.

Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their work there.

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions, 303, 304 and 305 to Debian:

• Improvements:
  • Use sed(1) backreferences when generating debian/tests/control to avoid duplicating ourselves. [ ]
  • Move from a mono-utils dependency to versioned mono-devel mono-utils dependency, taking care to maintain the [!riscv64] architecture restriction. [ ]
  • Use sed over awk to avoid mangling dependency lines containing = (equals) symbols such as version restrictions. [ ]
• Bug fixes:
  • Fix a test after the upload of systemd-ukify version 258~rc3. [ ]
  • Ensure that Java class files are named .class on the filesystem before passing them to javap(1). [ ]
  • Do not run jsondiff on files over 100KiB as the algorithm runs in O(n^2) time. [ ]
  • Don t check for PyPDF version 3 specifically; check for >= 3. [ ]
• Misc:
  • Update copyright years. [ ][ ]

In addition, Martin Joerg fixed an issue with the HTML presenter to avoid crash when page limit is None [ ] and Zbigniew J drzejewski-Szmek fixed compatibility with RPM 6 [ ]. Lastly, John Sirois fixed a missing requests dependency in the trydiffoscope tool. [ ]

Website updates Once again, there were a number of improvements made to our website this month including:

• Chris Lamb:
  • Write and publish a news entry for the upcoming summit. [ ]
  • Add some assets used at FOSSY, such as the badges and the paper handouts. [ ]
• Holger Levsen:
  • Restructure the new project history pages pages [ ] and add some recent news entries [ ].
  • Mark the OpenWrt tests as disabled on the Who is Involved. [ ]
  • Various changes to the upcoming Reproducible Builds Summit page. [ ]
• Jochen Sprickerhof made various improvements to the Vienna summit page. [ ][ ]
• Mattia Rizzolo also made various improvements to the Vienna summit page. [ ][ ][ ][ ][ ][ ][ ]
• kpcyrd made a number of changes to the new project history pages [ ][ ]

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In August, however, a number of changes were made by Holger Levsen, including:

• reproduce.debian.net-related:
  • Run 4 workers on the o4 node again in order to speed up testing. [ ][ ][ ][ ]
  • Also test trixie-proposed-updates and trixie-updates etc. [ ][ ]
  • Gather seperate statistics for each tested release. [ ]
  • Support sources from all Debian suites. [ ]
  • Run new code from the prototype database rework branch for the amd64-pull184 pseudo-architecture. [ ][ ]
  • Add a number of helpful links. [ ][ ][ ][ ][ ][ ][ ][ ][ ]
  • Temporarily call debrebuild without the --cache argument to experiment with a new version of devscripts. [ ][ ][ ]
  • Update public TODO. [ ]
• Installation tests:
  • Add comments to explain structure. [ ]
  • Mark more old jobs as old or dead . [ ][ ][ ]
  • Turn the maintenance job into a no-op. [ ]
• Jenkins node maintenance:
  • Increase penalties if the osuosl5 or ionos7 nodes are down. [ ]
  • Stop trying to fix network automatically. [ ]
  • Correctly mark ppc64el architecture nodes when down. [ ]
  • Upgrade the remaining arm64 nodes to Debian trixie in anticipation of the release. [ ][ ]
  • Allow higher SSD temperatures on the riscv64 architecture. [ ]
• Debian-related:
  • Drop the armhf architecture; many thanks to Vagrant for physically hosting the nodes for ten years. [ ][ ]
  • Add Debian forky, and archive bullseye. [ ][ ][ ][ ][ ][ ][ ]
  • Document the filesystem space savings from dropping the armhf architecture. [ ]
  • Exclude i386 and armhfr from JSON results. [ ]
  • Update TODOs for when Debian trixie and forky have been released. [ ][ ]
• tests.reproducible-builds.org-related:
  • Add a link to reproduce.debian.net. [ ]
  • Improve the dashboard graphs. [ ][ ][ ]
• Misc:
  • Detect errors with openQA erroring out. [ ]
  • Drop the long-disabled openwrt_rebuilder jobs. [ ]
  • Use qa-jenkins-dev@alioth-lists.debian.net as the contact for jenkins.debian.net. [ ]
  • Redirect reproducible-builds.org/vienna25 to reproducible-builds.org/vienna2025. [ ]
  • Disable all OpenWrt reproducible CI jobs, in coordination with the OpenWrt community. [ ][ ]
  • Make reproduce.debian.net accessable via IPv6. [ ]
  • Ignore that the megacli RAID controller requires packages from Debian bookworm. [ ]

In addition,

• James Addison migrated away from deprecated toplevel deb822 Python module in favour of debian.deb822 in the bin/reproducible_scheduler.py script [ ] and removed a note on reproduce.debian.net note after the release of Debian trixie [ ].
• Jochen Sprickerhof made a huge number of improvements to the reproduce.debian.net statistics calculation [ ][ ][ ][ ][ ][ ] as well as to the reproduce.debian.net service more generally [ ][ ][ ][ ][ ][ ][ ][ ].
• Mattia Rizzolo performed a lot of work migrating scripts to SQLAlchemy version 2.0 [ ][ ][ ][ ][ ][ ] in addition to making some changes to the way openSUSE reproducibility tests are handled internally. [ ]
• Lastly, Roland Clobus updated the Debian Live packages after the release of Debian trixie. [ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • cpython
  • hashcat
  • neochat
  • pocketbase
• Chris Lamb:
  • #1111497 filed against neon27.
• Jochen Sprickerhof:
  • #1111620 filed against pcbasic.
  • #1111624 filed against shellia.
  • #1111625 filed against bup.
• Mark Johnston:
  • FreeBSD
  • FreeBSD
  • FreeBSD
  • FreeBSD
• Robin Candau:
  • syd

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

$ grep-dctrl -v -F Testsuite --regex -s Package -n . trixie   cut -d - -f 1   uniq -c   sort -n  tail -20
     50 apertium
     50 kodi
     51 lomiri
     53 maven
     55 libjs
     57 globus
     66 cl
     67 pd
     72 lua
     79 php
     88 puppet
     91 r
    111 gnome
    124 ruby
    140 ocaml
    152 rust
    178 golang
    341 fonts
    557 python
   1072 haskell

#!/bin/sh
set -eu
extract()  
  local release
  local url
  release="$1"
  url="$2"
  if [ ! -f "$ release " ]; then
    rm -f "$ release .gz"
    curl --silent -o $ release .gz "$ url "
    gunzip "$ release .gz"
  fi
  local with_tests
  local total
  with_tests="$(grep-dctrl -c -F Testsuite --regex . $release)"
  total="$(grep-dctrl -c -F Package --regex . $release)"
  echo "  $ release    $ with_tests    $ total    $((100*with_tests/total))%  "
 
echo "  **Release**   **Packages with tests**   **Total number of packages**   **% of packages with tests**  "
echo " ------------- ------------------------- ------------------------------ ------------------------------ "
for release in wheezy jessie stretch buster; do
  extract "$ release " "http://archive.debian.org/debian/dists/$ release /main/source/Sources.gz"
done
for release in bullseye bookworm trixie; do
  extract "$ release " "http://ftp.br.debian.org/debian/dists/$ release /main/source/Sources.gz"
done

• aioftp
• aiosignal (building on work by IanLucca)
• audioop-lts
• celery
• djangorestframework
• djoser
• fpylll
• frozenlist
• git-repo-updater
• ipykernel
• klepto
• kombu
• multipart
• netmiko (sponsoring work by Eduardo Silva; contributed supporting fix upstream)
• pathos
• ppft
• pydantic
• pydantic-core
• pydantic-settings
• pylsqpack
• pymssql
• pytest-mock
• pytest-pretty
• pytest-repeat
• pytest-rerunfailures
• python-a2wsgi
• python-apptools (sponsoring work by Kathlyn Lara Murussi)
• python-asgiref
• python-asyncssh
• python-bitarray
• python-bitstring
• python-bytecode
• python-channels-redis
• python-charset-normalizer
• python-daphne
• python-django-analytical
• python-django-guid
• python-django-health-check
• python-django-pgbulk
• python-django-pgtrigger
• python-django-postgres-extra
• python-django-storages
• python-holidays
• python-httpx-sse
• python-icalendar
• python-lazy-model
• python-line-profiler
• python-lz4
• python-marshmallow-dataclass
• python-mastodon
• python-model-bakery
• python-oauthlib
• python-parse-type
• python-pathvalidate
• python-pgspecial
• python-processview
• python-pytest-subtests
• python-roman
• python-semantic-release
• python-testfixtures
• python-time-machine
• python-tokenize-rt
• python-typeguard
• python-typing-extensions
• python-urllib3
• pyupgrade
• requests (fixing CVE-2024-47081)
• responses
• zope.deferredimport
• zope.schema
• zope.testrunner

• billiard
• lazr.config
• python-timeline
• zope.sqlalchemy
• zope.testing

• aiosmtplib
• blinker
• ipykernel
• ipyparallel
• quart

• austin: binutils now has a libsframe-dev package, please add an explicit build dependency
• python-django-pgbulk: Missing dependency on python3-typing-extensions (contributed upstream)
• python-maturin: Upcoming rust-which update

• rust-serde
• rust-serde-derive
• rust-serde-json
• rust-smallvec
• rust-speedate
• rust-time
• rust-time-core
• rust-time-macros

• [1] https://tinyurl.com/2a7jfbbu
• [2] https://www.youtube.com/watch?v=znl1WhLCnhE
• [3] https://www.youtube.com/watch?v=URDjsHupqUM
• [4] https://tinyurl.com/243vo432
• [5] https://www.youtube.com/watch?v=_A_nq18uTR0
• [6] https://danglingpointer.fun/posts/GFWHistory
• [7] https://tinyurl.com/28htt7tg
• [8] https://www.youtube.com/watch?v=AFXLZ7FEJc4
• [9] https://tinyurl.com/26p4gdhm
• [10] https://www.salon.com/2002/12/17/tolkien_brin/

Debian LTS contributors In July, 17 contributors have been paid to work on Debian LTS, their reports are available:

• Adrian Bunk did 19.0h (out of 19.0h assigned).
• Andrej Shadura did 5.0h (out of 0.0h assigned and 8.0h from previous period), thus carrying over 3.0h to the next month.
• Bastien Roucari s did 18.5h (out of 18.75h assigned), thus carrying over 0.25h to the next month.
• Ben Hutchings did 12.5h (out of 3.25h assigned and 15.5h from previous period), thus carrying over 6.25h to the next month.
• Carlos Henrique Lima Melara did 10.0h (out of 10.0h assigned).
• Chris Lamb did 18.0h (out of 18.0h assigned).
• Daniel Leidert did 18.75h (out of 17.25h assigned and 1.5h from previous period).
• Emilio Pozuelo Monfort did 18.75h (out of 18.75h assigned).
• Guilhem Moulin did 15.0h (out of 14.0h assigned and 1.0h from previous period).
• Jochen Sprickerhof did 2.0h (out of 16.5h assigned and 2.25h from previous period), thus carrying over 16.75h to the next month.
• Lee Garrett did 7.0h (out of 0.0h assigned and 23.25h from previous period), thus carrying over 16.25h to the next month.
• Markus Koschany did 9.0h (out of 18.75h assigned), thus carrying over 9.75h to the next month.
• Roberto C. S nchez did 10.25h (out of 18.5h assigned and 2.75h from previous period), thus carrying over 11.0h to the next month.
• Santiago Ruano Rinc n did 7.25h (out of 12.75h assigned and 2.25h from previous period), thus carrying over 7.75h to the next month.
• Sylvain Beucler did 18.75h (out of 18.75h assigned).
• Thorsten Alteholz did 15.0h (out of 15.0h assigned).
• Utkarsh Gupta did 15.0h (out of 1.0h assigned and 14.0h from previous period).

Evolution of the situation In July, we released 24 DLAs.

• Notable security updates:
  • angular.js, prepared by Bastien Roucari s, fixes multiple vulnerabilities including input sanitization and potential regular expression denial of service (ReDoS)
  • tomcat9, prepared by Markus Koschany, fixes an assortment of vulnerabilities
  • mediawiki, prepared by Guilhem Moulin, fixes several information disclosure and privilege escalation vulnerabilities
  • php7.4, prepared by Guilhem Moulin, fixes several server side request forgery and denial of service vulnerabilities

This month s contributions from outside the regular team include an update to thunderbird, prepared by Christoph Goehre (the package maintainer). LTS Team members also contributed updates of the following packages:

• commons-beanutils (to stable and unstable), prepared by Adrian Bunk
• djvulibre (to oldstable, stable, and unstable), prepared by Adrian Bunk
• git (to stable), prepared by Adrian Bunk
• redis (to oldstable), prepared by Chris Lamb
• libxml2 (to oldstable), prepared by Guilhem Moulin
• commons-vfs (to oldstable), prepared by Daniel Leidert

Additionally, LTS Team member Santiago Ruano Rinc n proposed and implemented an improvement to the debian-security-support package. This package is available so that interested users can quickly determine if any installed packages are subject to limited security support or are excluded entirely from security support. However, there was not previously a way to identify explicitly supported packages, which has become necessary to note exceptions to broad exclusion policies (e.g., those which apply to substantial package groups, like modules belonging to the Go and Rust language ecosystems). Santiago s work has enabled the notation of exceptions to these exclusions, thus ensuring that users of debian-security-support have accurate status information concerning installed packages.

DebCamp 25 Security Tracker Sprint The previously announced security tracker sprint took place at DebCamp from 7-13 July. Participants included 8 members of the standing LTS Team, 2 active Debian Developers with an interest in LTS, 3 community members, and 1 member of the Debian Security Team (who provided guidance and reviews on proposed changes to the security tracker); participation was a mix of in person at the venue in Brest, France and remote. During the days of the sprint, the team tackled a wide range of bugs and improvements, mostly targeting the security tracker. The sprint participants worked on the following items:

• Completed during the sprint:
  • Implementation of a resource which provides an alternate view of the CVE history contained in the main security tracker
  • Implementation of a feature which identifies CVEs that have been fixed via a DLA but which remain unfixed in more recent releases (associated tests are still a work in progress)
  • A minor bug fix to the LTS Team s CVE triage tooling
  • Removal of some dead code
• Still in progress as of the end of the sprint:
  • Proposed implementation of support for vulnerabilities that don t affect the binaries (only in the sources)
  • Proposed implementation of support for tracking uploads to proposed-updates
  • Continued work (which was in progress prior to the sprint) on tooling to export security tracker data in CSAF and VEX formats
  • Proposed implementation of visual distinction between vulnerable/unimportant/ignored CVEs
  • Proposed implementation of support for identifying CVEs that have been fixed in older and newer releases but which remain unfixed in LTS
  • Proposed implementation of tooling that checks the consistency of the list of CVEs associated with a specific security update which is being prepared
  • Draft documentation of the security tracker s JSON data export schema
  • Proposed clean-up of inconsistent historical entries in the DSA index
  • Proposed improvement to how the security tracker handles requests for non-existent resources
  • Proposed bug fix for inconsistencies in the security tracker JSON data export
  • Proposed improvement to more accurate display of CVE states that are currently all shown as fixed (1 2)
  • Proposed bug fix for turning URLs from text into clickable links
  • A minor bug fix to the security tracker s linkage to Ubuntu security resources
  • Proposed implementation of the ability to identify CVEs for re-triage by the LTS Team
  • Continued work (which was in progress prior to the sprint) on improved tooling to support security releases of packages from language ecosystems that rely heavily on static linking

As can be seen from the above list, only a small number of changes were brought to completion during the sprint week itself. Given the very compressed timeframe involved, the broad scope of tasks which were under consideration, and the highly sensitive data managed by the security tracker, this is not entirely unexpected and in no way diminishes the great work done by the sprint participants. The LTS Team would especially like to thank Salvatore Bonaccorso of the Debian Security Team for making himself available throughout the sprint to answer questions, for providing guidance on the work, and for helping the work by reviewing and merging the MRs which were able to merged during the sprint itself. In the weeks that follow the sprint, the team will continue working towards completing the in progress items.

Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • Toshiba Corporation (for 118 months)
  • Civil Infrastructure Platform (CIP) (for 86 months)
  • VyOS Inc (for 50 months)
• Gold sponsors:
  • Roche Diagnostics International AG (for 128 months)
  • Akamai - Linode (for 123 months)
  • Babiel GmbH (for 112 months)
  • Plat Home (for 111 months)
  • University of Oxford (for 68 months)
  • Deveryware (for 55 months)
  • EDF SA (for 40 months)
  • Dataport A R (for 15 months)
  • CERN (for 13 months)
• Silver sponsors:
  • Domeneshop AS (for 133 months)
  • Nantes M tropole (for 127 months)
  • Univention GmbH (for 119 months)
  • Universit Jean Monnet de St Etienne (for 119 months)
  • Ribbon Communications, Inc. (for 113 months)
  • Exonet B.V. (for 103 months)
  • Leibniz Rechenzentrum (for 97 months)
  • Minist re de l Europe et des Affaires trang res (for 81 months)
  • Cloudways by DigitalOcean (for 70 months)
  • Dinahosting SL (for 68 months)
  • Platform.sh SAS (for 62 months)
  • Moxa Inc. (for 56 months)
  • sipgate GmbH (for 54 months)
  • OVH US LLC (for 52 months)
  • Tilburg University (for 52 months)
  • GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 43 months)
  • THINline s.r.o. (for 16 months)
  • Copenhagen Airports A/S (for 10 months)
• Bronze sponsors:
  • Evolix (for 133 months)
  • Seznam.cz, a.s. (for 133 months)
  • Intevation GmbH (for 130 months)
  • Linuxhotel GmbH (for 130 months)
  • Daevel SARL (for 129 months)
  • Megaspace Internet Services GmbH (for 128 months)
  • Greenbone AG (for 127 months)
  • NUMLOG (for 127 months)
  • WinGo AG (for 126 months)
  • Entr ouvert (for 118 months)
  • Adfinis AG (for 115 months)
  • Laboratoire LEGI - UMR 5519 / CNRS (for 110 months)
  • Tesorion (for 110 months)
  • Bearstech (for 101 months)
  • LiHAS (for 101 months)
  • Catalyst IT Ltd (for 96 months)
  • Demarcq SAS (for 90 months)
  • Universit Grenoble Alpes (for 76 months)
  • TouchWeb SAS (for 68 months)
  • SPiN AG (for 65 months)
  • CoreFiling (for 61 months)
  • Institut des sciences cognitives Marc Jeannerod (for 56 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 52 months)
  • Tem Innovations GmbH (for 47 months)
  • WordFinder.pro (for 47 months)
  • CNRS DT INSU R sif (for 45 months)
  • Soliton Systems K.K. (for 41 months)
  • Alter Way (for 38 months)
  • Institut Camille Jordan (for 28 months)
  • SOBIS Software GmbH (for 13 months)
  • Tuxera Inc. (for 4 months)

1. Install Chrome
2. Add the BrowserMCP extension
3. Install gemini-cli npm install -g @google/gemini-cli
4. Retrieve a Gemini API key via AI Studio
5. Export API key for gemini-cli export GEMINI_API_KEY=2342
6. Start BrowserMCP extension, see manual, an info box will appear that it's active with a cancel button.
7. Add mcp server to gemini-cli gemini mcp add browsermcp npx @browsermcp/mcp@latest
8. Start gemini-cli, let it use the mcp server and task it to open a website.

• [DLA 4255-1] audiofile security update of two CVEs related to an integer overflow and a memory leak.
• [DLA 4256-1] libetpan security update to fix one CVE related to prevent a null pointer dereference.
• [DLA 4257-1] libcaca security update to fix two CVEs related to heap buffer overflows.
• [DLA 4258-1] libfastjson security update to fix one CVE related to an out of bounds write.
• [#1106867] kmail-account-wizard was marked as accepted

• sane-airscan to experimental.

• supernovas (sponsored upload to experimental)
• calceph (sponsored upload to experimental)

1. Reproducible Builds Summit 2025
2. Reproducible Builds an official goal for SUSE Enterprise Linux
3. Reproducible Builds at FOSSY 2025
4. New OSS Rebuild project from Google
5. New extension of Python setuptools to support reproducible builds
6. diffoscope
7. New library to patch system functions for reproducibility
8. Independently Reproducible Git Bundles
9. Website updates
10. Distribution work
11. Reproducibility testing framework
12. Upstream patches

Reproducible Builds Summit 2025 We are extremely pleased to announce the upcoming Reproducible Builds Summit, set to take place from October 28th 30th 2025 in Vienna, Austria! We are thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving. If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. Registration is open until 20th September 2025, and we are very much looking forward to seeing many readers of these reports there!

Reproducible Builds an official goal for SUSE Enterprise Linux On our mailing list this month, Bernhard M. Wiedemann revealed the big news that reproducibility is now an official goal for SUSE Linux Enterprise Server (SLES) 16:

[Everything] changed earlier this year when reproducible-builds for SLES-16 became an official goal for the product. More people are talking about digital sovereignty and supply-chain security now. [ ] Today, only 9 of 3319 (source) packages have significant problems left (plus 7 with pending fixes), so 99.5% of packages have reproducible builds.

Reproducible Builds at FOSSY 2025 On Saturday 2nd August, Vagrant Cascadian and Chris Lamb presented at this year s FOSSY 2025. Their talk, titled Never Mind the Checkboxes, Here s Reproducible Builds!, was introduced as follows:

There are numerous policy compliance and regulatory processes being developed that target software development but do they solve actual problems? Does it improve the quality of software? Do Software Bill of Materials (SBOMs) actually give you the information necessary to verify how a given software artifact was built? What is the goal of all these compliance checklists anyways or more importantly, what should the goals be? If a software object is signed, who should be trusted to sign it, and can they be trusted forever?

Hosted by the Software Freedom Conservancy and taking place in Portland, Oregon, USA, FOSSY aims to be a community-focused event: Whether you are a long time contributing member of a free software project, a recent graduate of a coding bootcamp or university, or just have an interest in the possibilities that free and open source software bring, FOSSY will have something for you . More information on the event is available on the FOSSY 2025 website, including the full programme schedule. Vagrant and Chris also staffed a table, where they will be available to answer any questions about Reproducible Builds and discuss collaborations with other projects.

New OSS Rebuild project from Google The Google Open Source Security Team (GOSST) published an article this month announcing OSS Rebuild, a new project to strengthen trust in open source package ecosystems by reproducing upstream artifacts. As the post itself documents, the new project comprises four facets:

• Automation to derive declarative build definitions for existing PyPI (Python), npm (JS/TS), and Crates.io (Rust) packages.
• SLSA Provenance for thousands of packages across our supported ecosystems, meeting SLSA Build Level 3 requirements with no publisher intervention.
• Build observability and verification tools that security teams can integrate into their existing vulnerability management workflows.
• Infrastructure definitions to allow organizations to easily run their own instances of OSS Rebuild to rebuild, generate, sign, and distribute provenance.

One difference with most projects that aim for bit-for-bit reproducibility, OSS Rebuild aims for a kind of semantic reproducibility:

Through automation and heuristics, we determine a prospective build definition for a target package and rebuild it. We semantically compare the result with the existing upstream artifact, normalizing each one to remove instabilities that cause bit-for-bit comparisons to fail (e.g. archive compression).

The extensive post includes examples about how to access OSS Rebuild attestations using the Go-based command-line interface.

New extension of Python setuptools to support reproducible builds Wim Jeantine-Glenn has written a PEP 517 Build backend in order to enable reproducible builds when building Python projects that use setuptools. Called setuptools-reproducible, the project s README file contains the following:

Setuptools can create reproducible wheel archives (.whl) by setting SOURCE_DATE_EPOCH at build time, but setting the env var is insufficient for creating reproducible sdists (.tar.gz). setuptools-reproducible [therefore] wraps the hooks build_sdist build_wheel with some modifications to make reproducible builds by default.

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 301, 302 and 303 to Debian:

• Improvements:
  • Use Difference.from_operation in an attempt to pipeline the output of the extract-vmlinux script, potentially avoiding it all in memory. [ ]
  • Memoize a number of calls to --version, saving a very large number of external subprocess calls.
• Bug fixes:
  • Don t check for PyPDF version 3 specifically, check for versions greater than 3. [ ]
  • Ensure that Java class files are named .class on the filesystem before passing them to javap(1). [ ]
  • Mask stderr from extract-vmlinux script. [ ][ ]
  • Avoid spurious differences in h5dump output caused by exposure of absolute internal extraction paths. (#1108690)
• Misc:
  • Use our_check_output in the ODT comparator. [ ]
  • Update copyright years. [ ]

In addition:

• Siva Mahadevan made a change to use the --print-armap long option when calling nm(1) for wider compatibility. [ ]
• Vagrant Diffoscope updated diffoscope in GNU Guix to 300 [ ], 301 [ ], 302 and fixed the execute bit on the extract-vmlinux script [ ].

Lastly, Chris Lamb added a tmpfs to try.diffoscope.org so that diffoscope has a non-trivial temporary area to unpack archives, etc. [ ] Elsewhere in our tooling, however, reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, reprotest version 0.7.30 was uploaded to Debian unstable by Holger Levsen, chiefly including a change by Rebecca N. Palmer to not call sudo with the -h flag in order to fix Debian bug #1108550. [ ]

New library to patch system functions for reproducibility Nicolas Graves has written and published libfate, a simple collection of tiny libraries to patch system functions deterministically using LD_PRELOAD. According to the project s README:

libfate provides deterministic replacements for common non-deterministic system functions that can break reproducible builds. Instead of relying on complex build systems or apps or extensive patching, libfate uses the LD_PRELOAD trick to intercept system calls and return fixed, predictable values.

Describing why he wrote it, Nicolas writes:

I originally used the OpenSUSE dettrace approach to make Emacs reproducible in Guix. But when Guix switch to GCC@14, dettrace stopped working as expected. dettrace is a complex piece of software, my need was much less heavy: I don t need to systematically patch all sources of nondetermism, just the ones that make a process/binary unreproducible in a container/chroot.

Independently Reproducible Git Bundles Simon Josefsson has published another interesting article this month. Titled Independently Reproducible Git Bundles, the blog post describes the advantages of why you might a reproducible bundle, and the pitfalls that can arise when trying to create them:

One desirable property is that someone else should be able to reproduce the same git bundle, and not only that a single individual is able to reproduce things on one machine. It surprised me to see that when I ran the same set of commands on a different machine (started from a fresh git clone), I got a different checksum. The different checksums occurred even when nothing had been committed on the server side between the two runs.

Website updates Once again, there were a number of improvements made to our website this month including:

• Bernhard M. Wiedemann added an entry related to R-B-OS on the History page. [ ]
• Chris Lamb:
  • Replaced rbtlog run by Fay by rbtlog run by Benl on the Who is involved page. [ ]
  • Added a new, centered version of the RB logo. [ ]
• Holger Levsen:
  • Added and worked on the page for the Vienna 2025 summit. [ ][ ]
  • Add Jarl Gullberg to the Who is involved? page. [ ]
  • Add a talk from the recent MiniDebConf in Hamburg to our database of talks and presentations. [ ]
• Julien Cristau fixed a link to the diffoci repository on the [Tools](/tools/ page. [ ]

Distribution work In Debian this month:

• 5 reviews of Debian packages were added, 4 were updated and 2 were removed this month adding to our ever-increasing knowledge about identified issues.
• The release notes for the upcoming Debian Trixie release feature the following paragraph related to Reproducible Builds, thanks to the collaboration of Justin B Rye, Chris Hofstaedtler, Vagrant Cascadian and Holger Levsen:

Debian contributors have made significant progress toward ensuring package builds produce byte-for-byte reproducible results. You can check the status for packages installed on your system using the new package debian-repro-status, or visit reproduce.debian.net for Debian s overall statistics for trixie and later. You can contribute to these efforts by joining #debian-reproducible on IRC to discuss fixes, or verify the statistics by installing the new rebuilderd package and setting up your own instance.

The IzzyOnDroid Android APK repository made further progress in July, crossing the 50% reproducibility threshold congratulations. Furthermore, a new release of the Neo Store was released, which exposes the reproducible status directly next to the version of each app.
In GNU Guix, a series of patches intended to fix the reproducibility for the Mono programming language was merged, fixing reproducibility in Mono versions 1.9 [ ], 2.4 [ ] and 2.6 [ ].
Lastly, in addition to the news that openSUSE Enterprise Linux now [has an official goal of reproducibility]((https://lists.reproducible-builds.org/pipermail/rb-general/2025-July/003846.html), Bernhard M. Wiedemann posted another monthly update for their work there.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In July, however, a number of changes were made by Holger Levsen, including:

• Switch the URL for the Tails package set. [ ]
• Make the dsa-check-packages output more useful. [ ]
• Setup the ppc64el architecture again, has it has returned this time with a 2.7 GiB database instead of 72 GiB. [ ]

In addition, Jochen Sprickerhof improved the reproducibility statistics generation:

• Enable caching of statistics. [ ][ ][ ]
• Add some common non-reproducible patterns. [ ]
• Change output to directory. [ ]
• Add a page sorted by diffoscope size. [ ][ ]
• Switch to Python s argparse module and separate output(). [ ]

Holger also submitted a number of Debian bugs against rebuilderd and rebuilderd-worker:

• Config files and scripts for a simple one machine setup. [ ][ ]
• Create a rebuilderd user. [ ]
• Create rebuilderd-worker user with sbuild. [ ]

Lastly, Mattia Rizzolo added a scheduled job to renew some SSL certificates [ ] and Vagrant Cascadian performed some node maintenance [ ][ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • dpdk
  • HTTP-CookieJar
  • ibus-libzhuyin
  • java-21-openjdk
  • kea
  • libcamera
  • libpinyin
  • libsemigroups
  • llvm, llvm20 , llvm20, llvm19, llvm15, llvm17 (and all backports to llvm7-19)
  • mumble
  • python-Babel/pympress
  • python-numpy
  • python-numpy
  • python-paho-mqtt
  • python-rencode
  • rabbitmq-server
  • rage
  • re-flex
  • shadow-rs
• Chris Lamb:
  • #1109484 filed against rust-microformats
  • #1109731 filed against piglit
• Rafa Mikrut:
  • cargo-i18n
• Robin Candau:
  • netdata

There were a number of other patches from openSUSE developers:

• BCI-dockerfile-generator (Dirk Mueller)
• agama (Imobach Gonzalez Sosa)
• emacs (Werner Fink)
• firecracker (Andrea Manzini)
• maven-archiver (Fridrich Strba)
• moditect (Fridrich Strba)
• obs-build (Adrian Schroeter)
• ovmf (Richard Lyu)
• python-rdflib (Daniel Garcia)
• python313:doc (Daniel Garcia)

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

Get a list of current display outputs. You can also just check it in display settings of your DE/WM with wdisplays

rajudev@sanganak ~> swaymsg -t get_outputs
Output LVDS-1 &aposSeiko Epson Corporation 0x3047 Unknown&apos (focused)
  Current mode: 1366x768 @ 60.002 Hz
  Power: on
  Position: 0,0
  Scale factor: 1.000000
  Scale filter: nearest
  Subpixel hinting: rgb
  Transform: normal
  Workspace: 2
  Max render time: off
  Adaptive sync: disabled
  Allow tearing: no
  Available modes:
    1366x768 @ 60.002 Hz

Single physical display of the laptop

$ git clone https://github.com/odincat/sway-vdctl.git
$ cd sway-vdctl
$ cargo build --release

$ sudo cp target/release/main /usr/local/bin/vdctl

$ vdctl --help
Usage: vdctl [OPTIONS] <ACTION> [VALUE]
Arguments:
  <ACTION>
          Possible values:
          - create:      Create new output based on a preset
          - kill:        Terminate / unplug an active preset
          - list:        List out active presets
          - next-number: Manually set the next output number, in case something breaks
          - sync-number: Sync the next output number using &aposswaymsg -t get_outputs&apos
  [VALUE]
          Preset name to apply, alternatively a value
          
          [default: ]
Options:
      --novnc
          do not launch a vnc server, just create the output
  -h, --help
          Print help (see a summary with &apos-h&apos)

 
    "host": "0.0.0.0",
    "presets": [
         
            "name": "pad6",
            "scale_factor": 2,
            "port": 9901,
            "resolution":  
                "width": 2800,
                "height": 1800
             
       
    ]
 

$ vdctl create pad6
Created output, presumably &aposHEADLESS-1&apos
Set resolution of &aposHEADLESS-1&apos to 2800x1800
Set scale factor of &aposHEADLESS-1&apos to 2
Preset &apospad6&apos (&aposHEADLESS-1&apos: 2800x1800) is now active on port 9901

$ swaymsg -t get_outputs
Output LVDS-1 &aposSeiko Epson Corporation 0x3047 Unknown&apos
  Current mode: 1366x768 @ 60.002 Hz
  Power: on
  Position: 0,0
  Scale factor: 1.000000
  Scale filter: nearest
  Subpixel hinting: rgb
  Transform: normal
  Workspace: 2
  Max render time: off
  Adaptive sync: disabled
  Allow tearing: no
  Available modes:
    1366x768 @ 60.002 Hz
Output HEADLESS-1 &aposUnknown Unknown Unknown&apos (focused)
  Current mode: 2800x1800 @ 0.000 Hz
  Power: on
  Position: 1366,0
  Scale factor: 2.000000
  Scale filter: nearest
  Subpixel hinting: unknown
  Transform: normal
  Workspace: 3
  Max render time: off
  Adaptive sync: disabled
  Allow tearing: no

Display settings on Wayland with physical and virtual monitor output

$ sudo apt install wayvnc

$ wayvnc -o HEADLESS-1 0.0.0.0 5900

AVNC interface with the connection details to connect to the virtual monitor.

Debian

This was my 79th month of actively contributing to Debian. I became a DM in late March 2019 and a DD on Christmas 19! \o/ Debian was in freeze throughout so whilst I didn t do many uploads, there s a bunch of other things I did:

• Attended DebConf25 in Brest, France.
  • Lead the bursary BOF and discussions.
  • Participated in other sessions, especially around the FTP masters.
  • I ve started to look at things with my trainee hat on.
  • Participated in the Debian Security Tracker sprints during DebCamp. More on that below.
• Mentoring for newcomers.
• Moderation of -project mailing list.

---

Ubuntu

This was my 54th month of actively contributing to Ubuntu. I joined Canonical to work on Ubuntu full-time back in February 2021. Whilst I can t give a full, detailed list of things I did (there s so much and some of it might not be public yet!), here s a quick TL;DR of what I did:

• Released Questing snapshot 3! \o/
• EOL d Oracular. o/
• Participated in the mid-cycle sprints.
• Got a recognition award for leading 24.04.2 LTS release and leading the Release Management team.
• Preparing for the 24.04.3 LTS release early next month.

---

Debian (E)LTS

Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the stretch and jessie release (+2 years after LTS support). This was my 70th month as a Debian LTS and 57th month as a Debian ELTS paid contributor.
I only worked for 15.00 hours for LTS and 5.00 hours for ELTS and did the following things:

• [LTS] Released DLA 4263-1 for ruby-graphql.
  • Coordinated with upstream due to lack of clarity on 1.11.4 being affected & not having a clear reproducer.
  • As 1.11.4 was still partially vulnerable and the backport was non-trivial, it was probably conveinent to bump the upstream version to 1.11.12 instead, fixing:
  • CVE-2025-27407): a remote code execution.
  • Salsa repository: https://salsa.debian.org/lts-team/packages/ruby-graphql.
  • Coordinated with the Security team for a p-u fix or a DSA.
• [E/LTS] Frontdesk duty from 28th July to 04th August.
  • Triaged a bunch of CVEs.
  • Raised an inconsistency about issues being fixed in buster & bookworm but not in bullseye.
  • Helped Bastien with ca-certificates bit & coordinating with fellow Debian contributors.
  • Also triaged some newly supported packages for ELTS.
• [LTS] Attended the monthly LTS meeting on IRC. Summary here.

Debian Security Tracker sprint 2025 Thanks to the LTS team for also organizing a security tracker sprint during DebCamp25. I attended the sprint and spent 10 hours working on the following tasks:

• JSON API documentation:
  • Issue: https://salsa.debian.org/security-tracker-team/security-tracker/-/issues/15
  • MR: https://salsa.debian.org/security-tracker-team/security-tracker/-/merge_requests/237
  • This also includes actioning of Roberto s and Salvatore s review.
• Missing -1 from DSA entries causing web redirects to fail:
  • Issue: https://salsa.debian.org/security-tracker-team/security-tracker/-/issues/28
  • MR: https://salsa.debian.org/security-tracker-team/security-tracker/-/merge_requests/224
  • This includes some coordination with the web team and some back and forth with them to ensure both the Web team and the Security team would be happy with the fix. It also fixes more issues as mentioned in the description of the MR.
• Show packages from next-point-release.txt in source package overview
  • Bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989065
  • Did some initial brainstorming on the issue, haven t had time to do the actual implementation.
  • I intend to continue working on it as time allows in the next weeks & months. Will update if & when I make progress.

That s all. A quicky shoutout to Roberto for organizing the sprints remotely and being awake at odd hours. <3

---

Until next time.
:wq for today.

Achieving full disk encryption using FIPS, TCG OPAL and LUKS to encrypt UEFI ESP on bare-metal and in VMs

If one has a lot of estate and a lot of encryption keys to keep track off a key management solution is likely needed. The most popular solution is likely the one from Thales Group marketed under ChiperTrust Data Security Platform (previously Vormetric), but there are many others including OEM / Vendor / Hardware / Cloud specific or agnostic solutions.

I hope this crash course guide piques your interest to learn and discover modern confidentially and integrity solutions, and to re-affirm or change your existing controls w.r.t. to data protection at rest.

Full disk encryption, including UEFI ESP /boot/efi is now widely achievable by default on both baremetal machines and in VMs including with FIPS certification. To discuss more let's connect on Linkedin.

tl;dr: there is an attack in the wild which is triggering dangerous-but-seemingly-intended behaviour in the Oj JSON parser when used in the default and recommended manner, which can lead to everyone s favourite kind of security problem: object deserialization bugs! If you have the oj gem anywhere in your Gemfile.lock, the quickest mitigation is to make sure you have Oj.default_options = mode: :strict somewhere, and that no library is overwriting that setting to something else.

Prologue As a sensible sysadmin, all the sites I run send me a notification if any unhandled exception gets raised. Mostly, what I get sent is error-handling corner cases I missed, but now and then things get more interesting. In this case, it was a PG::UndefinedColumn exception, which looked something like this:

PG::UndefinedColumn: ERROR:  column "xyzzydeadbeef" does not exist

This is weird on two fronts: firstly, this application has been running for a while, and if there was a schema problem, I d expect it to have made itself apparent long before now. And secondly, while I don t profess to perfection in my programming, I m usually better at naming my database columns than that. Something is definitely hinky here, so let s jump into the mystery mobile!

The column name is coming from outside the building! The exception notifications I get sent include a whole lot of information about the request that caused the exception, including the request body. In this case, the request body was JSON, and looked like this:

 "name":":xyzzydeadbeef", ... 

The leading colon looks an awful lot like the syntax for a Ruby symbol, but it s in a JSON string. Surely there s no way a JSON parser would be turning that into a symbol, right? Right?!? Immediately, I thought that that possibly was what was happening, because I use Sequel for my SQL database access needs, and Sequel treats symbols as database column names. It seemed like too much of a coincidence that a vaguely symbol-shaped string was being sent in, and the exact same name was showing up as a column name. But how the flying fudgepickles was a JSON string being turned into a Ruby symbol, anyway? Enter Oj.

Oj? I barely know aj A long, long time ago, the standard Ruby JSON library had a reputation for being slow. Thus did many competitors flourish, claiming more features and better performance. Strong amongst the contenders was oj (for Optimized JSON ), touted as The fastest JSON parser and object serializer . Given the history, it s not surprising that people who wanted the best possible performance turned to Oj, leading to it being found in a great many projects, often as a sub-dependency of a dependency of a dependency (which is how it ended up in my project). You might have noticed in Oj s description that, in addition to claiming fastest , it also describes itself as an object serializer . Anyone who has kept an eye on the security bug landscape will recall that object deserialization is a rich vein of vulnerabilities to mine. Libraries that do object deserialization, especially ones with a history that goes back to before the vulnerability class was well-understood, are likely to be trouble magnets. And thus, it turns out to be with Oj. By default, Oj will happily turn any string that starts with a colon into a symbol:

>> require "oj"
>> Oj.load(' "name":":xyzzydeadbeef","username":"bob","answer":42 ')
=>  "name"=>:xyzzydeadbeef, "username"=>"bob", "answer"=>42 

How that gets exploited is only limited by the creativity of an attacker. Which I ll talk about more shortly but first, a word from my rant cortex.

Insecure By Default is a Cancer While the object of my ire today is Oj and its fast-and-loose approach to deserialization, it is just one example of a pervasive problem in software: insecurity by default. Whether it s a database listening on 0.0.0.0 with no password as soon as its installed, or a library whose default behaviour is to permit arbitrary code execution, it all contributes to a software ecosystem that is an appalling security nightmare. When a user (in this case, a developer who wants to parse JSON) comes across a new piece of software, they have by definition no idea what they re doing with that software. They re going to use the defaults, and follow the most easily-available documentation, to achieve their goal. It is unrealistic to assume that a new user of a piece of software is going to do things the right way , unless that right way is the only way, or at least the by-far-the-easiest way. Conversely, the developer(s) of the software is/are the domain experts. They have knowledge of the problem domain, through their exploration while building the software, and unrivalled expertise in the codebase. Given this disparity in knowledge, it is tantamount to malpractice for the experts the developer(s) to off-load the responsibility for the safe and secure use of the software to the party that has the least knowledge of how to do that (the new user). To apply this general principle to the specific case, take the Using section of the Oj README. The example code there calls Oj.load, with no indication that this code will, in fact, parse specially-crafted JSON documents into Ruby objects. The brand-user user of the library, no doubt being under pressure to Get Things Done, is almost certainly going to look at this Using example, get the apparent result they were after (a parsed JSON document), and call it a day. It is unlikely that a brand-new user will, for instance, scroll down to the Further Reading section, find the second last (of ten) listed documents, Security.md , and carefully peruse it. If they do, they ll find an oblique suggestion that parsing untrusted input is never a good idea . While that s true, it s also rather unhelpful, because I d wager that by far the majority of JSON parsed in the world is untrusted , in one way or another, given the predominance of JSON as a format for serializing data passing over the Internet. This guidance is roughly akin to putting a label on a car s airbags that driving at speed can be hazardous to your health : true, but unhelpful under the circumstances. The solution is for default behaviours to be secure, and any deviation from that default that has the potential to degrade security must, at the very least, be clearly labelled as such. For example, the Oj.load function should be named Oj.unsafe_load, and the Oj.load function should behave as the Oj.safe_load function does presently. By naming the unsafe function as explicitly unsafe, developers (and reviewers) have at least a fighting chance of recognising they re doing something risky. We put warning labels on just about everything in the real world; the same should be true of dangerous function calls. OK, rant over. Back to the story.

But how is this exploitable? So far, I ve hopefully made it clear that Oj does some Weird Stuff with parsing certain JSON strings. It caused an unhandled exception in a web application I run, which isn t cool, but apart from bombing me with exception notifications, what s the harm? For starters, let s look at our original example: when presented with a symbol, Sequel will interpret that as a column name, rather than a string value. Thus, if our save an update to the user code looked like this:

# request_body has the JSON representation of the form being submitted
body = Oj.load(request_body)
DB[:users].where(id: user_id).update(name: body["name"])

In normal operation, this will issue an SQL query along the lines of UPDATE users SET name='Jaime' WHERE id=42. If the name given is Jaime O Dowd , all is still good, because Sequel quotes string values, etc etc. All s well so far. But, imagine there is a column in the users table that normally users cannot read, perhaps admin_notes. Or perhaps an attacker has gotten temporary access to an account, and wants to dump the user s password hash for offline cracking. So, they send an update claiming that their name is :admin_notes (or :password_hash). In JSON, that ll look like "name":":admin_notes" , and Oj.load will happily turn that into a Ruby object of "name"=>:admin_notes . When run through the above update the user code fragment, it ll produce the SQL UPDATE users SET name=admin_notes WHERE id=42. In other words, it ll copy the contents of the admin_notes column into the name column which the attacker can then read out just by refreshing their profile page.

But Wait, There s More! That an attacker can read other fields in the same table isn t great, but that s barely scratching the surface. Remember before I said that Oj does object serialization ? That means that, in general, you can create arbitrary Ruby objects from JSON. Since objects contain code, it s entirely possible to trigger arbitrary code execution by instantiating an appropriate Ruby object. I m not going to go into details about how to do this, because it s not really my area of expertise, and many others have covered it in detail. But rest assured, if an attacker can feed input of their choosing into a default call to Oj.load, they ve been handed remote code execution on a platter.

Mitigations As Oj s object deserialization is intended and documented behaviour, don t expect a future release to make any of this any safer. Instead, we need to mitigate the risks. Here are my recommended steps:

1. Look in your Gemfile.lock (or SBOM, if that s your thing) to see if the oj gem is anywhere in your codebase. Remember that even if you don t use it directly, it s popular enough that it is used in a lot of places. If you find it in your transitive dependency tree anywhere, there s a chance you re vulnerable, limited only by the ingenuity of attackers to feed crafted JSON into a deeply-hidden Oj.load call.
2. If you depend on oj directly and use it in your project, consider not doing that. The json gem is acceptably fast, and JSON.parse won t create arbitrary Ruby objects.
3. If you really, really need to squeeze the last erg of performance out of your JSON parsing, and decide to use oj to do so, find all calls to Oj.load in your code and switch them to call Oj.safe_load.
4. It is a really, really bad idea to ever use Oj to deserialize JSON into objects, as it lacks the safety features needed to mitigate the worst of the risks of doing so (for example, restricting which classes can be instantiated, as is provided by the permitted_classes argument to Psych.load). I d make it a priority to move away from using Oj for that, and switch to something somewhat safer (such as the aforementioned Psych). At the very least, audit and comment heavily to minimise the risk of user-provided input sneaking into those calls somehow, and pass mode: :object as the second argument to Oj.load, to make it explicit that you are opting-in to this far more dangerous behaviour only when it s absolutely necessary.
5. To secure any unsafe uses of Oj.load in your dependencies, consider setting the default Oj parsing mode to :strict, by putting Oj.default_options = mode: :strict somewhere in your initialization code (and make sure no dependencies are setting it to something else later!). There is a small chance that this change of default might break something, if a dependency is using Oj to deliberately create Ruby objects from JSON, but the overwhelming likelihood is that Oj s just being used to parse ordinary JSON, and these calls are just RCE vulnerabilities waiting to give you a bad time.

Is Your Bacon Saved? If I ve helped you identify and fix potential RCE vulnerabilities in your software, or even just opened your eyes to the risks of object deserialization, please help me out by buying me a refreshing beverage. I would really appreciate any support you can give. Alternately, if you d like my help in fixing these (and many other) sorts of problems, I m looking for work, so email me.