Welcome to the October 2025 report from the Reproducible Builds project! Welcome to the very latest report from the Reproducible Builds project. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website. In this report:

1. Farewell from the Reproducible Builds Summit 2025
2. Google s Play Store breaks reproducible builds for Signal
3. Mailing list updates
4. The Original Sin of Computing that no one can fix
5. Reproducible Builds at the Transparency.dev summit
6. Supply Chain Security for Go
7. Three new academic papers published
8. Distribution work
9. Upstream patches
10. Website updates
11. Tool development

Farewell from the Reproducible Builds Summit 2025 Thank you to everyone who joined us at the Reproducible Builds Summit in Vienna, Austria! We were thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. During this event, participants had the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim was to create an inclusive space that fosters collaboration, innovation and problem-solving. The agenda of the three main days is available online however, some working sessions may still lack notes at time of publication. One tangible outcome of the summit is that Johannes Starosta finished their rebuilderd tutorial, which is now available online and Johannes is actively seeking feedback.

Google s Play Store breaks reproducible builds for Signal On the issue tracker for the popular Signal messenger app, developer Greyson Parrelli reports that updates to the Google Play store have, in effect, broken reproducible builds:

The most recent issues have to do with changes to the APKs that are made by the Play Store. Specifically, they add some attributes to some .xml files around languages are resources, which is not unexpected because of how the whole bundle system works. This is trickier to resolve, because unlike current expected differences (like signing information), we can t just exclude a whole file from the comparison. We have to take a more nuanced look at the diff. I ve been hesitant to do that because it ll complicate our currently-very-readable comparison script, but I don t think there s any other reasonable option here.

The full thread with additional context is available on GitHub.

Mailing list updates On our mailing list this month:

• kpcyrd forwarded a fascinating tidbit regarding so-called ninja and samurai build ordering, that uses data structures in which the pointer values returned from malloc are used to determine some order of execution.
• Arnout Engelen, Justin Cappos, Ludovic Court s and kpcyrd continued a conversation started in September regarding the Minimum Elements for a Software Bill of Materials . (Full thread)
• Felix Moessbauer of Siemens posted to the list reporting that he had recently stumbled upon a couple of Debian source packages on the snapshot mirrors that are listed multiple times (same name and version), but each time with a different checksum . The thread, which Felix titled, Debian: what precisely identifies a source package is about precisely that what can be axiomatically relied upon by consumers of the Debian archives, as well as indicating an issue where we can t exactly say which packages were used during build time (even when having the .buildinfo files).
• Luca DiMaio posted to the list announcing the release of xfsprogs 6.17.0 which specifically includes a commit that implements the functionality to populate a newly created XFS filesystem directly from an existing directory structure which makes it easier to create populated filesystems without having to mount them [and thus is] particularly useful for reproducible builds . Luca asked the list how they might contribute to the docs of the System images page.

The Original Sin of Computing that no one can fix Popular YouTuber @laurewired published a video this month with an engaging take on the Trusting Trust problem. Titled The Original Sin of Computing that no one can fix, the video touches on David A. Wheeler s Diverse Double-Compiling dissertation. GNU developer Janneke Nieuwenhuizen followed-up with an email (additionally sent to our mailing list) as well, underscoring that GNU Mes s current solution [to this issue] uses ancient softwares in its bootstrap path, such as gcc-2.95.3 and glibc-2.2.5 . (According to Colby Russell, the GNU Mes bootstrapping sequence is shown at 18m54s in the video.)

Reproducible Builds at the Transparency.dev summit Holger Levsen gave a talk at this year s Transparency.dev summit in Gothenburg, Sweden, outlining the achievements of the Reproducible Builds project in the last 12 years, covering both upstream developments as well as some distribution-specific details. As mentioned on the talk s page, Holger s presentation concluded with an outlook into the future and an invitation to collaborate to bring transparency logs into Reproducible Builds projects . The slides of the talk are available, although a video has yet to be released. Nevertheless, as a result of the discussions at Transparency.dev there is a new page on the Debian wiki with the aim of describing a potential transparency log setup for Debian.

Supply Chain Security for Go Andrew Ayer has setup a new service at sourcespotter.com that aims to monitor the supply chain security for Go releases. It consists of four separate trackers:

1. A tool to verify that the Go Module Mirror and Checksum Database is behaving honestly and has not presented inconsistent information to clients.
2. A module monitor that records every module version served by the Go Module Mirror and Checksum Database, allowing you to monitor for unexpected versions of your modules.
3. A tool to verifies that the Go toolchains published in the Go Module Mirror can be reproduced from source code, making it difficult to hide backdoors in the binaries downloaded by the go command.
4. A telemetry config tracker that tracks the names of telemetry counters uploaded by the Go toolchain, to ensure that Go telemetry is not violating users privacy.

As the homepage of the service mentions, the trackers are free software and do not rely on Google infrastructure.

Three new academic papers published Julien Malka of the Institut Polytechnique de Paris published an exciting paper this month on How NixOS could have detected the XZ supply-chain attack for the benefit of all thanks to reproducible-builds. Julien outlines his paper as follows:

In March 2024, a sophisticated backdoor was discovered in xz, a core compression library in Linux distributions, covertly inserted over three years by a malicious maintainer, Jia Tan. The attack, which enabled remote code execution via ssh, was only uncovered by chance when Andres Freund investigated a minor performance issue. This incident highlights the vulnerability of the open-source supply chain and the effort attackers are willing to invest in gaining trust and access. In this article, I analyze the backdoor s mechanics and explore how bitwise build reproducibility could have helped detect it.

A PDF of the paper is available online.
Iy n M ndez Veiga and Esther H nggi (of the Lucerne University of Applied Sciences and Arts and ETH Zurich) published a paper this month on the topic of Reproducible Builds for Quantum Computing. The abstract of their paper mentions the following:

Although quantum computing is a rapidly evolving field of research, it can already benefit from adopting reproducible builds. This paper aims to bridge the gap between the quantum computing and reproducible builds communities. We propose a generalization of the definition of reproducible builds in the quantum setting, motivated by two threat models: one targeting the confidentiality of end users data during circuit preparation and submission to a quantum computer, and another compromising the integrity of quantum computation results. This work presents three examples that show how classical information can be hidden in transpiled quantum circuits, and two cases illustrating how even minimal modifications to these circuits can lead to incorrect quantum computation results.

A full PDF of their paper is available.
Congratulations to Georg Kofler who submitted their Master s thesis for the Johannes Kepler University of Linz, Austria on the topic of Reproducible builds of E2EE-messengers for Android using Nix hermetic builds:

The thesis focuses on providing a reproducible build process for two open-source E2EE messaging applications: Signal and Wire. The motivation to ensure reproducibility and thereby the integrity of E2EE messaging applications stems from their central role as essential tools for modern digital privacy. These applications provide confidentiality for private and sensitive communications, and their compromise could undermine encryption mechanisms, potentially leaking sensitive data to third parties.

A full PDF of their thesis is available online.
Shawkot Hossain of Aalto University, Finland has also submitted their Master s thesis on the The Role of SBOM in Modern Development with a focus on the extant tooling:

Currently, there are numerous solutions and techniques available in the market to tackle supply chain security, and all claim to be the best solution. This thesis delves deeper by implementing those solutions and evaluates them for better understanding. Some of the tools that this thesis implemented are Syft, Trivy, Grype, FOSSA, dependency-check, and Gemnasium. Software dependencies are generated in a Software Bill of Materials (SBOM) format by using these open-source tools, and the corresponding results have been analyzed. Among these tools, Syft and Trivy outperform others as they provide relevant and accurate information on software dependencies.

A PDF of the thesis is also available.

Distribution work Michael Plura published an interesting article on Heise.de on the topic of Trust is good, reproducibility is better:

In the wake of growing supply chain attacks, the FreeBSD developers are relying on a transparent build concept in the form of Zero-Trust Builds. The approach builds on the established Reproducible Builds, where binary files can be rebuilt bit-for-bit from the published source code. While reproducible builds primarily ensure verifiability, the zero-trust model goes a step further and removes trust from the build process itself. No single server, maintainer, or compiler can be considered more than potentially trustworthy.

The article mentions that this goal has now been achieved with a slight delay and can be used in the current development branch for FreeBSD 15 .
In Debian this month, 7 reviews of Debian packages were added, 5 were updated and 11 were removed this month adding to our knowledge about identified issues. For the Debian CI tests Holger fixed #786644 and set nocheck in DEB_BUILD_OPTIONS for the 2nd build..
Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their work there.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Chris Lamb:
  • #1117494 filed against python-can.
  • #1117614 filed against rsbackup.
  • #1117742 filed against mobilitydb.
  • #1118160 filed against pyraf.
  • #1118596 filed against ne.
• Bernhard M. Wiedemann:
  • qt6-lottie, plasma6-print-manager, plasma6-nm (avoid race in qmlcachegen)
  • xfishtank (date, regression)
  • gstreamer-plugins-rs
  • gpg2 (FTBFS-2038)
  • rocclr (PID)
  • kf6-breeze-icons (parallelism)
  • opencloud-server (random tmp path)
  • python-awscrt (FTBFS-j1)
  • glib-macros/contrast/fractal/Fragments/identity/mousai/loupe/gstreamer-plugins-rs (rust HashMap)
  • deno (rust order)
• Robin Candau:
  • treetop

Website updates Once again, there were a number of improvements made to our website this month including:

• Arnout Engelen added a note on using git archive to the Archive metadata page. [ ]
• James Addison updated the user stories that feature on the homepage [ ][ ][ ][ ] as well as a new Reproducibility Troubleshooting that functions as an excellent getting started guide [ ][ ].
• Zbigniew J drzejewski-Szmek added a link on the Tools page for add-determinism and linkdupes [ ] as well as added a link to Fedora s reproducibility efforts to the Contribute page [ ].
• Bernhard Wiedemann and Zbigniew J drzejewski-Szmek extended ismypackagereproducibleyet.org with initial support for Fedora [ ].

In addition, a number of contributors added a series of notes from our recent summit to the website, including Alexander Couzens [ ], Robin Candau [ ][ ][ ][ ][ ][ ][ ][ ][ ] and kpcyrd [ ].

Tool development diffoscope version 307 was uploaded to Debian unstable by Chris Lamb, who made a number of changes including fixing compatibility with LLVM version 21 [ ], an attempt to automatically attempt to deploy to PyPI by liaising with the PyPI developers/maintainers (with this experimental feature). [ ] In addition, Vagrant Cascadian updated diffoscope in GNU Guix to version 307.

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

Changes in version 0.2.14 (2024-11-03)

• The rbenchmark package is now a Suggests: as it appears in demo
• The continuous integration setup now uses r-ci with its embedded setup step
• The URL used for the GPL-2 is now the R Project copy

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

As the last chime fades we drop neatly on to the balcony's rusting hand rail, folding our wings with a soft shuffle. Noon, on the ninth day of the eighth month, 1531. Neema Kraa's lodgings. We are here, exactly where we should be, at exactly the right moment, because we are the Raven, and we are magnificent.

Respect, that's all we demand. Recognition of our magnificence. Offerings. Love. Fear. Trembling awe. Worship. Shiny things. Blood sacrifice, some of us very much enjoy blood sacrifice. Truly, we ask for so little.

• Switch back to default them when disabling automatic HighContrast (MR)
• Hande gnome-session 49 changes so OSK can still start up (MR)
• Release 0.50.0, 0.50.1
• Don't forget to apply corner-shift to gear icon (MR)
• Fix startup warning (MR)
• Update doap (MR)
• DBus codegen cleanups (MR, MR, MR)
• Add Autobrightness handling (MR), (MR), (MR)

• Dispatch idle loop in prepare (MR)
• Release 0.50.0

• Allow to hide plugins (MR)
• Release 0.50~rc1, 0.50.0
• Hide demo plugins by default (MR)
• Sink floating refs properly (MR)
• Simplify includes (MR)
• Allow to configure alarm sound if clock app likely supports it (MR)
• Use shared check CI images (MR)
• Release 0.50.1
• Fix ringtone role (MR)

• Ship sytemd user unit so things work with gnome-session 49 (MR)
• Fix fallback to default OSK size with multiple outputs (MR)
• Improve character styling a bit (MR)
• Consolidate input surface creation (MR)
• Release 0.50.0, 0.50.1
• Don't trigger backspace key repeat in keypad or emoji layouts (MR)
• Better restore layout after swipe closing (MR)

• Release 0.50.0

• Build shared CI image MR

• Use pfs subproject for Rust portal (MR)

• Fix doc build (MR)

• Release 49.1
• Fix plugin loading (MR)
• Fix debug logging (MR)

• Allow OSKs to run with gnome-session 49 (MR)
• Release 0.50.0 (MR)

• Fix build (MR)
• Simplify and cleanup docs (MR)
• Fix mkosi build (MR)
• Install Recommends: (MR)
• Draft: Add support for initial-setup (MR)
• Add version option (MR)
• Drop debos build (MR, MR)

• Fix compatiblility with systemd >= 258 (MR)
• Use meson.options (MR)
• Add phone role (MR)

• Add support for Google Pixel 3A (MR)

• Fix failing CI build (MR)

• Add systemd unit to start with gnome-session 49 (MR)

• Upload phosh-tour 0.50.0
• Upload stevia 0.50.0, 0.50.1
• Upload phosh-mobile-settings 0.50.0
• Upload feedbackd 0.8.6
• Prepare phrog upload (MR)
• gnome-session Breaks (MR, MR)
• cellbroadcastd: Backport our upstream deadlock fix (MR)
• Upload wlroots 0.19.2
• Upload iio-sensor-proxy 3.8 (MR)

• Release 0.0.3
• Fix deadlock on start MR)

• Fix brightness values (MR)

• Ignore role based loopbacks (MR)

• Use AdwSwitchRow to reduce horizonatal allocation (MR)
• Quit when done (and not under GDM) (MR)

• shift6mq: Switch to mainline panel driver (MR)

• RequiredComponents is now gone (MR)

• stevia: Add hunspell-en-us dependency (MR)

• Install sensor firmware for SHIFT6mq (MR)
• Install sensor firmware for Oneplus 6T (MR)
• Fix clippy complaints (MR)

• Mention mirror (MR)
• Update for hugo 150 (MR, MR)
• Embed some ouf our peertube videos (MR)
• Add donations item (MR)
• Update osk post to mention systemd unit (MR)
• Update list of distributions and users (MR)
• Switch nightly builds to forky (MR)
• Lint more markdown (MR)
• Release 0.50.1 (MR)
• Notes on audio (MR)

• Switch to Debian forky (MR)
• Add g-i-s (MR)

• shift6mq: Add missing panel driver dependency (MR)
• shift6mq: Fix DTS warning (MR)

• Update media-role volume MR (MR)
• Allow to set a default target for e.g. alerts and alarms (MR)

• Add demo videos (MR)
• Add demo epubs (MR)

• phoc: Use correct format specifiers (MR)
• phoc: seat: Use the getter to access focused layer's output (MR)
• phosh: Upcoming events empty state (MR)
• phosh: Caffeine duration (MR)
• phosh: Location service quick setting (MR)
• phosh: UI check fix (MR)
• phosh: Autobrightness toggle (MR)
• p-m-s: Empty tweaks page test (MR)
• p-m-s: Symlink backend (MR)
• p-m-s: Return boolean from value setters (MR)
• p-m-s: conf-tweaks: Make setting_data a prop in Xresources backend (MR)
• p-m-s: conf-tweaks: Add gtk3settings backend (MR)
• gmobile: xiaomi-sweet support (MR)
• xdg-d-p: Use clippy (MR), MR)
• libcmatrix: various tweaks ([MR (https://source.puri.sm/Librem5/libcmatrix/-/merge_requests/110))
• libcmatrix: Release 0.0.4 (MR](https://source.puri.sm/Librem5/libcmatrix/-/merge_requests/115)
• meta-phosh: Add cellbroadcasts (MR)
• calls: Unload plugin test (MR)
• calls: Sip proxy support (MR)
• calls: Build system cleanups (MR)
• m-b-p-i: JP MVNO (MR)

• Debian packages:
  • firmware-nonfree:
    • Bugs:
      • closed #1118156: Missing Build-Depends on python3-dacite, python3-jinja2
    • Merge requests:
      • closed !129: Retire vorlon from uploaders
      • merged !130: Update firmware-nonfree to 20250917
      • reviewed and merged !131: Update to 20251011
      • merged !132: misc-nonfree: Include all Arm Mali firmware
      • opened and merged !133: Fix another broken symlink and avoid creating more of them
    • Uploads:
      • uploaded version 20250917-1 to unstable
  • gdm3:
    • Bugs:
      • opened #1119809: Black screen after upgrading from 48
  • linux:
    • Bugs:
      • replied to #1116358: linux-image-6.12.48+deb13-amd64: LVM snapshots causing I/O errors in KVM guest with aio=io_uring set
      • replied to #1117256: linux-image-6.12.48+deb13-amd64: Kernel can no longer activate ethernet on Dell WD19S docking station
      • (LTS) replied to #1118465: Regression: linux-image-5.10.0-36-arm64 fails to bring up LAN7800 USB NIC: NO-CARRIER and state DOWN
      • replied to #1118653: linux-image-6.17.2-amd64: Please restore CONFIG_NETFILTER_XT_TARGET_MASQUERADE
    • Merge requests:
      • reviewed !1577: udeb: Add USB TYPE-C modules and Mux for usb-modules.
      • reviewed and merged !1636: d/b/test-patches: Stop handling rt featureset for test builds
      • reviewed !1664: Use more debhelper files
      • reviewed !1667: Drop some Secure Boot backward compatibility
      • (LTS) opened and merged !1675: CI: Fix more issues introduced by the removal of the extract-source job
      • opened !1685: netfilter: Enable NETFILTER_XTABLES_LEGACY
    • (LTS) Updated the bullseye-security branch to 5.10.246, but did not upload it
  • linux-base:
    • Bugs:
      • closed #1117908: linux-sysctl-defaults should restart systemd-sysctl.service in the postinst
  • rust-sequoia-sop:
    • Bugs:
      • opened and replied to #1118154: sqopv does not seem to support some signing keys
      • opened #1118155: The debug option to verify does nothing
  • wireless-regdb:
    • Uploads:
      • uploaded version 2025.10.07-1 to unstable
• Mailing lists:
  • debian-devel:
    • replied to uscan downloads wrong source tarball from Github
  • debian-kernel:
    • posted Agenda items for kernel-team meeting on 2025-10-15
    • replied to slow memory I/O on AMD EPYC 9334
  • debian-lts:
    • replied to Please enable udebs for the 6.1 backport
  • debian-lts-announce:
    • posted [SECURITY] [DLA 4327-1] linux security update
    • posted [SECURITY] [DLA 4328-1] linux-6.1 security update
  • stable:
    • replied to [PATCH 5.10 013/122] compiler.h: drop fallback overflow checkers
    • replied to [PATCH 5.10 171/332] lib/crypto/curve25519-hacl64: Disable KASAN with clang-17 and older

• [1] https://etbe.coker.com.au/2021/06/01/internode-nbn-arris-cm8200/
• [2] https://etbe.coker.com.au/2024/09/15/kogan-ax1800-wifi6-mesh/

I had entered Iraq supporting the war on the grounds that we could at least produce a better society than Saddam Hussein's. It was one of the greatest mistakes in my life. We attempted to impose programmes made up by Washington think tanks, and reheated in air-conditioned palaces in Baghdad a new taxation system modelled on Hong Kong; a system of ministers borrowed from Singapore; and free ports, modelled on Dubai. But we did it ultimately at the point of a gun, and our resources, our abstract jargon and optimistic platitudes could not conceal how much Iraqis resented us, how much we were failing, and how humiliating and degrading our work had become. Our mission was a grotesque satire of every liberal aspiration for peace, growth and democracy.

Neither they, nor indeed any of the 12,000 peacekeepers and policemen who had been posted to South Sudan from sixty nations, had spent a single night in a rural house, or could complete a sentence in Dinka, Nuer, Azande or Bande. And the international development strategy written jointly between the donor nations resembled a fading mission statement found in a new space colony, whose occupants had all been killed in an alien attack.

David Cameron, I was beginning to realise, had put in charge of environment, food and rural affairs a Secretary of State who openly rejected the idea of rural affairs and who had little interest in landscape, farmers or the environment. I was beginning to wonder whether he could have given her any role she was less suited to apart perhaps from making her Foreign Secretary. Still, I could also sense why Cameron was mesmerised by her. Her genius lay in exaggerated simplicity. Governing might be about critical thinking; but the new style of politics, of which she was a leading exponent, was not. If critical thinking required humility, this politics demanded absolute confidence: in place of reality, it offered untethered hope; instead of accuracy, vagueness. While critical thinking required scepticism, open-mindedness and an instinct for complexity, the new politics demanded loyalty, partisanship and slogans: not truth and reason but power and manipulation. If Liz Truss worried about the consequences of any of this for the way that government would work, she didn't reveal it.

A month earlier, they had been anticipating every nuance of Minister Rogerson's diary, supporting him on shifts twenty-four hours a day, seven days a week. But it was already clear that there would be no pretence of a handover no explanation of my predecessor's strategy, and uncompleted initiatives. The arrival of a new minister was Groundhog Day. Dan Rogerson was not a ghost haunting my office, he was an absence, whose former existence was suggested only by the black plastic comb.

But Cameron, who was rumoured to be sceptical about nation-building projects, only nodded, and then looking confidently up and down the table said, "Well, at least we all agree on one extremely straightforward and simple point, which is that our troops are doing very difficult and important work and we should all support them." It was an odd statement to make to civilians running humanitarian operations on the ground. I felt I should speak. "No, with respect, we do not agree with that. Insofar as we have focused on the troops, we have just been explaining that what the troops are doing is often futile, and in many cases making things worse." Two small red dots appeared on his cheeks. Then his face formed back into a smile. He thanked us, told us he was out of time, shook all our hands, and left the room. Later, I saw him repeat the same line in interviews: "the purpose of this visit is straightforward... it is to show support for what our troops are doing in Afghanistan". The line had been written, in London, I assumed, and tested on focus groups. But he wanted to convince himself it was also a position of principle. "David has decided," one of his aides explained, when I met him later, "that one cannot criticise a war when there are troops on the ground." "Why?" "Well... we have had that debate. But he feels it is a principle of British government." "But Churchill criticised the conduct of the Boer War; Pitt the war with America. Why can't he criticise wars?" "British soldiers are losing their lives in this war, and we can't suggest they have died in vain." "But more will die, if no one speaks up..." "It is a principle thing. And he has made his decision. For him and the party." "Does this apply to Iraq too?" "Yes. Again he understands what you are saying, but he voted to support the Iraq War, and troops are on the ground." "But surely he can say he's changed his mind?" The aide didn't answer, but instead concentrated on his food. "It is so difficult," he resumed, "to get any coverage of our trip." He paused again. "If David writes a column about Afghanistan, we will struggle to get it published." "But what would he say in an article anyway?" I asked. "We can talk about that later. But how do you get your articles on Afghanistan published?" I remembered how the US politicians and officials had shown their mastery of strategy and detail. I remembered the earnestness of Gordon Brown when I had briefed him on Iraq. Cameron seemed somehow less serious. I wrote as much in a column in the New York Times, saying that I was afraid the party of Churchill was becoming the party of Bertie Wooster.

This post is a review for Computing Reviews for LLM Hallucinations in Practical Code Generation Phenomena, Mechanism, and Mitigation , a article published in Proceedings of the ACM on Software Engineering, Volume 2, Issue ISSTA

• Why didn t any Linux distro packagers notice anything odd when importing the new XZ version 5.6.0/5.6.1 from upstream?
• Is the current software supply-chain in the most popular Linux distros easy to audit?
• Could we have similar backdoors lurking that haven t been detected yet?

Downloading Debian and upstream source packages Let s start by working backwards from what the Debian package repositories offer for download. As auditing binaries is extremely complicated, we skip that, and assume the Debian build hosts are trustworthy and reliably building binaries from the source packages, and the focus should be on auditing the source code packages. As with everything in Debian, there are multiple tools and ways to do the same thing, but in this post only one (and hopefully the best) way to do something is presented for brevity. The first step is to download the latest version and some past versions of the package from the Debian archive, which is easiest done with debsnap. The following command will download all Debian source packages of xz-utils from Debian release 5.2.4-1 onwards:

Copy

$ debsnap --verbose --first 5.2.4-1 xz-utils Getting json https://snapshot.debian.org/mr/package/xz-utils/ ... Getting dsc file xz-utils_5.2.4-1.dsc: https://snapshot.debian.org/file/a98271e4291bed8df795ce04d9dc8e4ce959462d Getting file xz-utils_5.2.4.orig.tar.xz.asc: https://snapshot.debian.org/file/59ccbfb2405abe510999afef4b374cad30c09275 Getting file xz-utils_5.2.4-1.debian.tar.xz: https://snapshot.debian.org/file/667c14fd9409ca54c397b07d2d70140d6297393f source-xz-utils/xz-utils_5.2.4-1.dsc: Good signature found validating xz-utils_5.2.4.orig.tar.xz validating xz-utils_5.2.4.orig.tar.xz.asc validating xz-utils_5.2.4-1.debian.tar.xz All files validated successfully.

$ debsnap --verbose --first 5.2.4-1 xz-utils
Getting json https://snapshot.debian.org/mr/package/xz-utils/
...
Getting dsc file xz-utils_5.2.4-1.dsc: https://snapshot.debian.org/file/a98271e4291bed8df795ce04d9dc8e4ce959462d
Getting file xz-utils_5.2.4.orig.tar.xz.asc: https://snapshot.debian.org/file/59ccbfb2405abe510999afef4b374cad30c09275
Getting file xz-utils_5.2.4-1.debian.tar.xz: https://snapshot.debian.org/file/667c14fd9409ca54c397b07d2d70140d6297393f
source-xz-utils/xz-utils_5.2.4-1.dsc:
Good signature found
validating xz-utils_5.2.4.orig.tar.xz
validating xz-utils_5.2.4.orig.tar.xz.asc
validating xz-utils_5.2.4-1.debian.tar.xz
All files validated successfully.

Once debsnap completes there will be a subfolder source-<package name> with the following types of files:

• *.orig.tar.xz: source code from upstream
• *.orig.tar.xz.asc: detached signature (if upstream signs their releases)
• *.debian.tar.xz: Debian packaging source, i.e. the debian/ subdirectory contents
• *.dsc: Debian source control file, including signature by Debian Developer/Maintainer

Example:

Copy

$ ls -1 source-xz-utils/ ... xz-utils_5.6.4.orig.tar.xz xz-utils_5.6.4.orig.tar.xz.asc xz-utils_5.6.4-1.debian.tar.xz xz-utils_5.6.4-1.dsc xz-utils_5.8.0.orig.tar.xz xz-utils_5.8.0.orig.tar.xz.asc xz-utils_5.8.0-1.debian.tar.xz xz-utils_5.8.0-1.dsc xz-utils_5.8.1.orig.tar.xz xz-utils_5.8.1.orig.tar.xz.asc xz-utils_5.8.1-1.1.debian.tar.xz xz-utils_5.8.1-1.1.dsc xz-utils_5.8.1-1.debian.tar.xz xz-utils_5.8.1-1.dsc xz-utils_5.8.1-2.debian.tar.xz xz-utils_5.8.1-2.dsc

$ ls -1 source-xz-utils/
...
xz-utils_5.6.4.orig.tar.xz
xz-utils_5.6.4.orig.tar.xz.asc
xz-utils_5.6.4-1.debian.tar.xz
xz-utils_5.6.4-1.dsc
xz-utils_5.8.0.orig.tar.xz
xz-utils_5.8.0.orig.tar.xz.asc
xz-utils_5.8.0-1.debian.tar.xz
xz-utils_5.8.0-1.dsc
xz-utils_5.8.1.orig.tar.xz
xz-utils_5.8.1.orig.tar.xz.asc
xz-utils_5.8.1-1.1.debian.tar.xz
xz-utils_5.8.1-1.1.dsc
xz-utils_5.8.1-1.debian.tar.xz
xz-utils_5.8.1-1.dsc
xz-utils_5.8.1-2.debian.tar.xz
xz-utils_5.8.1-2.dsc

Verifying authenticity of upstream and Debian sources using OpenPGP signatures As seen in the output of debsnap, it already automatically verifies that the downloaded files match the OpenPGP signatures. To have full clarity on what files were authenticated with what keys, we should verify the Debian packagers signature with:

Copy

$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc gpg: Signature made Fri Oct 3 22:04:44 2025 UTC gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported gpg: Total number processed: 1 gpg: imported: 1 gpg: Good signature from "Sebastian Andrzej Siewior" [unknown] gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown] gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1 Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B

$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc
gpg: Signature made Fri Oct 3 22:04:44 2025 UTC
gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B
gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org
gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Sebastian Andrzej Siewior" [unknown]
gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown]
gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1
Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B

The upstream tarball signature (if available) can be verified with:

Copy

$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz' gpg: Signature made Thu Apr 3 11:38:23 2025 UTC gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620 gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported gpg: Total number processed: 1 gpg: imported: 1 gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620

$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc
gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz'
gpg: Signature made Thu Apr 3 11:38:23 2025 UTC
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620

Note that this only proves that there is a key that created a valid signature for this content. The authenticity of the keys themselves need to be validated separately before trusting they in fact are the keys of these people. That can be done by checking e.g. the upstream website for what key fingerprints they published, or the Debian keyring for Debian Developers and Maintainers, or by relying on the OpenPGP web-of-trust .

Verifying authenticity of upstream sources by comparing checksums In case the upstream in question does not publish release signatures, the second best way to verify the authenticity of the sources used in Debian is to download the sources directly from upstream and compare that the sha256 checksums match. This should be done using the debian/watch file inside the Debian packaging, which defines where the upstream source is downloaded from. Continuing on the example situation above, we can unpack the latest Debian sources, enter and then run uscan to download:

Copy

$ tar xvf xz-utils_5.8.1-2.debian.tar.xz ... debian/rules debian/source/format debian/source.lintian-overrides debian/symbols debian/tests/control debian/tests/testsuite debian/upstream/signing-key.asc debian/watch ... $ uscan --download-current-version --destdir /tmp Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1 gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620 gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz.

$ tar xvf xz-utils_5.8.1-2.debian.tar.xz
...
debian/rules
debian/source/format
debian/source.lintian-overrides
debian/symbols
debian/tests/control
debian/tests/testsuite
debian/upstream/signing-key.asc
debian/watch
...
$ uscan --download-current-version --destdir /tmp
Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1
gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC
gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>"
Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz.

The original files downloaded from upstream are now in /tmp along with the files renamed to follow Debian conventions. Using everything downloaded so far the sha256 checksums can be compared across the files and also to what the .dsc file advertised:

Copy

$ ls -1 /tmp/ xz-5.8.1.tar.xz xz-5.8.1.tar.xz.sig xz-utils_5.8.1.orig.tar.xz xz-utils_5.8.1.orig.tar.xz.asc $ sha256sum xz-utils_5.8.1.orig.tar.xz /tmp/xz-5.8.1.tar.xz 0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e xz-utils_5.8.1.orig.tar.xz 0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e /tmp/xz-5.8.1.tar.xz $ grep -A 3 Sha256 xz-utils_5.8.1-2.dsc Checksums-Sha256: 0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e 1461872 xz-utils_5.8.1.orig.tar.xz 4138f4ceca1aa7fd2085fb15a23f6d495d27bca6d3c49c429a8520ea622c27ae 833 xz-utils_5.8.1.orig.tar.xz.asc 3ed458da17e4023ec45b2c398480ed4fe6a7bfc1d108675ec837b5ca9a4b5ccb 24648 xz-utils_5.8.1-2.debian.tar.xz

$ ls -1 /tmp/
xz-5.8.1.tar.xz
xz-5.8.1.tar.xz.sig
xz-utils_5.8.1.orig.tar.xz
xz-utils_5.8.1.orig.tar.xz.asc
$ sha256sum xz-utils_5.8.1.orig.tar.xz /tmp/xz-5.8.1.tar.xz
0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e xz-utils_5.8.1.orig.tar.xz
0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e /tmp/xz-5.8.1.tar.xz
$ grep -A 3 Sha256 xz-utils_5.8.1-2.dsc
Checksums-Sha256:
0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e 1461872 xz-utils_5.8.1.orig.tar.xz
4138f4ceca1aa7fd2085fb15a23f6d495d27bca6d3c49c429a8520ea622c27ae 833 xz-utils_5.8.1.orig.tar.xz.asc
3ed458da17e4023ec45b2c398480ed4fe6a7bfc1d108675ec837b5ca9a4b5ccb 24648 xz-utils_5.8.1-2.debian.tar.xz

In the example above the checksum 0b54f79df85... is the same across the files, so it is a match.

Repackaged upstream sources can t be verified as easily Note that uscan may in rare cases repackage some upstream sources, for example to exclude files that don t adhere to Debian s copyright and licensing requirements. Those files and paths would be listed under the Files-Excluded section in the debian/copyright file. There are also other situations where the file that represents the upstream sources in Debian isn t bit-by-bit the same as what upstream published. If checksums don t match, an experienced Debian Developer should review all package settings (e.g. debian/source/options) to see if there was a valid and intentional reason for divergence.

Reviewing changes between two source packages using diffoscope Diffoscope is an incredibly capable and handy tool to compare arbitrary files. For example, to view a report in HTML format of the differences between two XZ releases, run:

Copy

diffoscope --html-dir xz-utils-5.6.4_vs_5.8.0 xz-utils_5.6.4.orig.tar.xz xz-utils_5.8.0.orig.tar.xz browse xz-utils-5.6.4_vs_5.8.0/index.html

diffoscope --html-dir xz-utils-5.6.4_vs_5.8.0 xz-utils_5.6.4.orig.tar.xz xz-utils_5.8.0.orig.tar.xz
browse xz-utils-5.6.4_vs_5.8.0/index.html

If the changes are extensive, and you want to use a LLM to help spot potential security issues, generate the report of both the upstream and Debian packaging differences in Markdown with:

Copy

diffoscope --markdown diffoscope-debian.md xz-utils_5.6.4-1.debian.tar.xz xz-utils_5.8.1-2.debian.tar.xz diffoscope --markdown diffoscope.md xz-utils_5.6.4.orig.tar.xz xz-utils_5.8.0.orig.tar.xz

diffoscope --markdown diffoscope-debian.md xz-utils_5.6.4-1.debian.tar.xz xz-utils_5.8.1-2.debian.tar.xz
diffoscope --markdown diffoscope.md xz-utils_5.6.4.orig.tar.xz xz-utils_5.8.0.orig.tar.xz

The Markdown files created above can then be passed to your favorite LLM, along with a prompt such as:

Based on the attached diffoscope output for a new Debian package version compared with the previous one, list all suspicious changes that might have introduced a backdoor, followed by other potential security issues. If there are none, list a short summary of changes as the conclusion.

Reviewing Debian source packages in version control As of today only 93% of all Debian source packages are tracked in git on Debian s GitLab instance at salsa.debian.org. Some key packages such as Coreutils and Bash are not using version control at all, as their maintainers apparently don t see value in using git for Debian packaging, and the Debian Policy does not require it. Thus, the only reliable and consistent way to audit changes in Debian packages is to compare the full versions from the archive as shown above. However, for packages that are hosted on Salsa, one can view the git history to gain additional insight into what exactly changed, when and why. For packages that are using version control, their location can be found in the Git-Vcs header in the debian/control file. For xz-utils the location is salsa.debian.org/debian/xz-utils. Note that the Debian policy does not state anything about how Salsa should be used, or what git repository layout or development practices to follow. In practice most packages follow the DEP-14 proposal, and use git-buildpackage as the tool for managing changes and pushing and pulling them between upstream and salsa.debian.org. To get the XZ Utils source, run:

Copy

$ gbp clone https://salsa.debian.org/debian/xz-utils.git gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'

$ gbp clone https://salsa.debian.org/debian/xz-utils.git
gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'

At the time of writing this post the git history shows:

Copy

$ git log --graph --oneline * bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2 * 4b769547 d: Remove the symlinks from -dev package. * a39f3428 Correct the nocheck build profile * 1b806b8d Import Debian changes 5.8.1-1.1 * b1cad34b Prepare 5.8.1-1 * a8646015 Import 5.8.1 * 2808ec2d Update upstream source from tag 'upstream/5.8.1' \ * fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1 * a522a226 Bump version and soname for 5.8.1 * 1c462c2a Add NEWS for 5.8.1 * 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h * 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder * 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding * 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread * d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115) * c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state * 831b55b9 liblzma: mt dec: Fix a comment * b9d168ee liblzma: Add assertions to lzma_bufcpy() * c8e0a489 DOS: Update Makefile to fix the build * 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers * 7ce38b31 Update THANKS * 688e51bd Translations: Update the Croatian translation * a6b54dde Prepare 5.8.0-1. * 77d9470f Add 5.8 symbols. * 9268eb66 Import 5.8.0 * 6f85ef4f Update upstream source from tag 'upstream/5.8.0' \ \ * afba662b New upstream version 5.8.0 / * 173fb5c6 doc/SHA256SUMS: Add 5.8.0 * db9258e8 Bump version and soname for 5.8.0 * bfb752a3 Add NEWS for 5.8.0 * 6ccbb904 Translations: Run "make -C po update-po" * 891a5f05 Translations: Run po4a/update-po * 4f52e738 Translations: Partially fix overtranslation in Serbian man pages * ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage * 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()

$ git log --graph --oneline
* bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2
* 4b769547 d: Remove the symlinks from -dev package.
* a39f3428 Correct the nocheck build profile
* 1b806b8d Import Debian changes 5.8.1-1.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
 \
  * fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
  * a522a226 Bump version and soname for 5.8.1
  * 1c462c2a Add NEWS for 5.8.1
  * 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h
  * 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder
  * 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding
  * 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread
  * d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
  * c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state
  * 831b55b9 liblzma: mt dec: Fix a comment
  * b9d168ee liblzma: Add assertions to lzma_bufcpy()
  * c8e0a489 DOS: Update Makefile to fix the build
  * 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers
  * 7ce38b31 Update THANKS
  * 688e51bd Translations: Update the Croatian translation
*   a6b54dde Prepare 5.8.0-1.
*   77d9470f Add 5.8 symbols.
*   9268eb66 Import 5.8.0
*   6f85ef4f Update upstream source from tag 'upstream/5.8.0'
 \ \
  *   afba662b New upstream version 5.8.0
   /
  * 173fb5c6 doc/SHA256SUMS: Add 5.8.0
  * db9258e8 Bump version and soname for 5.8.0
  * bfb752a3 Add NEWS for 5.8.0
  * 6ccbb904 Translations: Run "make -C po update-po"
  * 891a5f05 Translations: Run po4a/update-po
  * 4f52e738 Translations: Partially fix overtranslation in Serbian man pages
  * ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage
  * 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()

This shows both the changes on the debian/unstable branch as well as the intermediate upstream import branch, and the actual real upstream development branch. See my Debian source packages in git explainer for details of what these branches are used for. To only view changes on the Debian branch, run git log --graph --oneline --first-parent or git log --graph --oneline -- debian. The Debian branch should only have changes inside the debian/ subdirectory, which is easy to check with:

Copy

$ git diff --stat upstream/v5.8 debian/README.source 16 +++ debian/autogen.sh 32 +++++ debian/changelog 949 ++++++++++++++++++++++++++ ... debian/upstream/signing-key.asc 52 +++++++++ debian/watch 4 + debian/xz-utils.README.Debian 47 ++++++++ debian/xz-utils.docs 6 + debian/xz-utils.install 28 +++++ debian/xz-utils.postinst 19 +++ debian/xz-utils.prerm 10 ++ debian/xzdec.docs 6 + debian/xzdec.install 4 + 33 files changed, 2014 insertions(+)

$ git diff --stat upstream/v5.8
debian/README.source   16 +++
debian/autogen.sh   32 +++++
debian/changelog   949 ++++++++++++++++++++++++++
...
debian/upstream/signing-key.asc   52 +++++++++
debian/watch   4 +
debian/xz-utils.README.Debian   47 ++++++++
debian/xz-utils.docs   6 +
debian/xz-utils.install   28 +++++
debian/xz-utils.postinst   19 +++
debian/xz-utils.prerm   10 ++
debian/xzdec.docs   6 +
debian/xzdec.install   4 +
33 files changed, 2014 insertions(+)

All the files outside the debian/ directory originate from upstream, and for example running git blame on them should show only upstream commits:

Copy

$ git blame CMakeLists.txt 22af94128 (Lasse Collin 2024-02-12 17:09:10 +0200 1) # SPDX-License-Identifier: 0BSD 22af94128 (Lasse Collin 2024-02-12 17:09:10 +0200 2) 7e3493d40 (Lasse Collin 2020-02-24 23:38:16 +0200 3) ############### 7e3493d40 (Lasse Collin 2020-02-24 23:38:16 +0200 4) # 426bdc709 (Lasse Collin 2024-02-17 21:45:07 +0200 5) # CMake support for building XZ Utils

$ git blame CMakeLists.txt
22af94128 (Lasse Collin 2024-02-12 17:09:10 +0200 1) # SPDX-License-Identifier: 0BSD
22af94128 (Lasse Collin 2024-02-12 17:09:10 +0200 2)
7e3493d40 (Lasse Collin 2020-02-24 23:38:16 +0200 3) ###############
7e3493d40 (Lasse Collin 2020-02-24 23:38:16 +0200 4) #
426bdc709 (Lasse Collin 2024-02-17 21:45:07 +0200 5) # CMake support for building XZ Utils

If the upstream in question signs commits or tags, they can be verified with e.g.:

Copy

$ git verify-tag v5.6.2 gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620 gpg: issuer "lasse.collin@tukaani.org" gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired] gpg: Note: This key has expired!

$ git verify-tag v5.6.2
gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: issuer "lasse.collin@tukaani.org"
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired]
gpg: Note: This key has expired!

The main benefit of reviewing changes in git is the ability to see detailed information about each individual change, instead of just staring at a massive list of changes without any explanations. In this example, to view all the upstream commits since the previous import to Debian, one would view the commit range from afba662b New upstream version 5.8.0 to fa1e8796 New upstream version 5.8.1 with git log --reverse -p afba662b...fa1e8796. However, a far superior way to review changes would be to browse this range using a visual git history viewer, such as gitk. Either way, looking at one code change at a time and reading the git commit message makes the review much easier.

Comparing Debian source packages to git contents As stated in the beginning of the previous section, and worth repeating, there is no guarantee that the contents in the Debian packaging git repository matches what was actually uploaded to Debian. While the tag2upload project in Debian is getting more and more popular, Debian is still far from having any system to enforce that the git repository would be in sync with the Debian archive contents. To detect such differences we can run diff across the Debian source packages downloaded with debsnap earlier (path source-xz-utils/xz-utils_5.8.1-2.debian) and the git repository cloned in the previous section (path xz-utils):

diff Copy

$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog --- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700 +++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700 @@ -5,7 +5,7 @@ * Remove the symlinks from -dev, pointing to the lib package. (Closes: #1109354) - -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200 + -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200

$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/
diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog
--- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700
+++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700
@@ -5,7 +5,7 @@
 * Remove the symlinks from -dev, pointing to the lib package.
 (Closes: #1109354)

- -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200

In the case above diff revealed that the timestamp in the changelog in the version uploaded to Debian is different from what was committed to git. This is not malicious, just a mistake by the maintainer who probably didn t run gbp tag immediately after upload, but instead some dch command and ended up with having a different timestamps in the git compared to what was actually uploaded to Debian.

Creating syntetic Debian packaging git repositories If no Debian packaging git repository exists, or if it is lagging behind what was uploaded to Debian s archive, one can use git-buildpackage s import-dscs feature to create synthetic git commits based on the files downloaded by debsnap, ensuring the git contents fully matches what was uploaded to the archive. To import a single version there is gbp import-dsc (no s at the end), of which an example invocation would be:

Copy

$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'

$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc
Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'

Example commit history from a repository with commits added with gbp import-dsc:

Copy

$ git log --graph --oneline * 86aed07b (HEAD -> debian/unstable, tag: debian/5.8.1-2, origin/debian/unstable) Import Debian changes 5.8.1-2 * f111d93b (tag: debian/5.8.1-1.1) Import Debian changes 5.8.1-1.1 * 1106e19b (tag: debian/5.8.1-1) Import Debian changes 5.8.1-1 \ * 08edbe38 (tag: upstream/5.8.1, origin/upstream/v5.8, upstream/v5.8) Import Upstream version 5.8.1 \ * a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1 * 1c462c2a Add NEWS for 5.8.1 * 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h

$ git log --graph --oneline
* 86aed07b (HEAD -> debian/unstable, tag: debian/5.8.1-2, origin/debian/unstable) Import Debian changes 5.8.1-2
* f111d93b (tag: debian/5.8.1-1.1) Import Debian changes 5.8.1-1.1
* 1106e19b (tag: debian/5.8.1-1) Import Debian changes 5.8.1-1
 \
  * 08edbe38 (tag: upstream/5.8.1, origin/upstream/v5.8, upstream/v5.8) Import Upstream version 5.8.1
   \
    * a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
    * 1c462c2a Add NEWS for 5.8.1
    * 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h

An online example repository with only a few missing uploads added using gbp import-dsc can be viewed at salsa.debian.org/otto/xz-utils-2025-09-29/-/network/debian%2Funstable An example repository that was fully crafted using gbp import-dscs can be viewed at salsa.debian.org/otto/xz-utils-gbp-import-dscs-debsnap-generated/-/network/debian%2Flatest. There exists also dgit, which in a similar way creates a synthetic git history to allow viewing the Debian archive contents via git tools. However, its focus is on producing new package versions, so fetching a package with dgit that has not had the history recorded in dgit earlier will only show the latest versions:

Copy

$ dgit clone xz-utils canonical suite name for unstable is sid starting new git history last upload to archive: NO git hash downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz... downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc... downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz... dpkg-source: info: extracting xz-utils in unpacked dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz synthesised git commit from .dsc 5.8.1-2 HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium dgit ok: ready for work in xz-utils $ dgit/sid git log --graph --oneline * f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid) \ * 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago * 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago

$ dgit clone xz-utils
canonical suite name for unstable is sid
starting new git history
last upload to archive: NO git hash
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz...
dpkg-source: info: extracting xz-utils in unpacked
dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz
dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz
synthesised git commit from .dsc 5.8.1-2
HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium
dgit ok: ready for work in xz-utils
$ dgit/sid   git log --graph --oneline
* f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid)
 \
  * 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago
* 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago

Unlike git-buildpackage managed git repositories, the dgit managed repositories cannot incorporate the upstream git history and are thus less useful for auditing the full software supply-chain in git.

Comparing upstream source packages to git contents Equally important to the note in the beginning of the previous section, one must also keep in mind that the upstream release source packages, often called release tarballs, are not guaranteed to have the exact same contents as the upstream git repository. Projects might strip out test data or extra development files from their release tarballs to avoid shipping unnecessary files to users, or projects might add documentation files or versioning information into the tarball that isn t stored in git. While a small minority, there are also upstreams that don t use git at all, so the plain files in a release tarball is still the lowest common denominator for all open source software projects, and exporting and importing source code needs to interface with it. In the case of XZ, the release tarball has additional version info and also a sizeable amount of pregenerated compiler configuration files. Detecting and comparing differences between git contents and tarballs can of course be done manually by running diff across an unpacked tarball and a checked out git repository. If using git-buildpackage, the difference between the git contents and tarball contents can be made visible directly in the import commit. In this XZ example, consider this git history:

Copy

* b1cad34b Prepare 5.8.1-1 * a8646015 Import 5.8.1 * 2808ec2d Update upstream source from tag 'upstream/5.8.1' \ * fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1 * a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1 * 1c462c2a Add NEWS for 5.8.1

* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
 \
  * fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
  * a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
  * 1c462c2a Add NEWS for 5.8.1

The commit a522a226 was the upstream release commit, which upstream also tagged v5.8.1. The merge commit 2808ec2d applied the new upstream import branch contents on the Debian branch. Between these is the special commit fa1e8796 New upstream version 5.8.1 tagged upstream/v5.8. This commit and tag exists only in the Debian packaging repository, and they show what is the contents imported into Debian. This is generated automatically by git-buildpackage when running git import-orig --uscan for Debian packages with the correct settings in debian/gbp.conf. By viewing this commit one can see exactly how the upstream release tarball differs from the upstream git contents (if at all). In the case of XZ, the difference is substantial, and shown below in full as it is very interesting:

Copy

$ git show --stat fa1e8796 commit fa1e8796dabd91a0f667b9e90f9841825225413a (debian/upstream/v5.8, upstream/v5.8) Author: Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Date: Thu Apr 3 22:58:39 2025 +0200 New upstream version 5.8.1 .codespellrc 30 - .gitattributes 8 - .github/workflows/ci.yml 163 - .github/workflows/freebsd.yml 32 - .github/workflows/netbsd.yml 32 - .github/workflows/openbsd.yml 35 - .github/workflows/solaris.yml 32 - .github/workflows/windows-ci.yml 124 - .gitignore 113 - ABOUT-NLS 1 + ChangeLog 17392 +++++++++++++++++++++ Makefile.in 1097 +++++++ aclocal.m4 1353 ++++++++ build-aux/ci_build.bash 286 -- build-aux/compile 351 ++ build-aux/config.guess 1815 ++++++++++ build-aux/config.rpath 751 +++++ build-aux/config.sub 2354 +++++++++++++ build-aux/depcomp 792 +++++ build-aux/install-sh 541 +++ build-aux/ltmain.sh 11524 ++++++++++++++++++++++ build-aux/missing 236 ++ build-aux/test-driver 160 + config.h.in 634 ++++ configure 26434 ++++++++++++++++++++++ debug/Makefile.in 756 +++++ doc/SHA256SUMS 236 -- doc/man/txt/lzmainfo.txt 36 + doc/man/txt/xz.txt 1708 ++++++++++ doc/man/txt/xzdec.txt 76 + doc/man/txt/xzdiff.txt 39 + doc/man/txt/xzgrep.txt 70 + doc/man/txt/xzless.txt 36 + doc/man/txt/xzmore.txt 31 + lib/Makefile.in 623 ++++ m4/.gitignore 40 - m4/build-to-host.m4 274 ++ m4/gettext.m4 392 +++ m4/host-cpu-c-abi.m4 529 +++ m4/iconv.m4 324 ++ m4/intlmacosx.m4 71 + m4/lib-ld.m4 170 + m4/lib-link.m4 815 +++++ m4/lib-prefix.m4 334 ++ m4/libtool.m4 8488 +++++++++++++++++++++ m4/ltoptions.m4 467 +++ m4/ltsugar.m4 124 + m4/ltversion.m4 24 + m4/lt~obsolete.m4 99 + m4/nls.m4 33 + m4/po.m4 456 +++ m4/progtest.m4 92 + po/.gitignore 31 - po/Makefile.in.in 517 +++ po/Rules-quot 66 + po/boldquot.sed 21 + po/ca.gmo Bin 0 -> 15587 bytes po/cs.gmo Bin 0 -> 7983 bytes po/da.gmo Bin 0 -> 9040 bytes po/de.gmo Bin 0 -> 29882 bytes po/en@boldquot.header 35 + po/en@quot.header 32 + po/eo.gmo Bin 0 -> 15060 bytes po/es.gmo Bin 0 -> 29228 bytes po/fi.gmo Bin 0 -> 28225 bytes po/fr.gmo Bin 0 -> 10232 bytes

$ git show --stat fa1e8796
commit fa1e8796dabd91a0f667b9e90f9841825225413a
(debian/upstream/v5.8, upstream/v5.8)
Author: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Date: Thu Apr 3 22:58:39 2025 +0200
New upstream version 5.8.1
.codespellrc   30 -
.gitattributes   8 -
.github/workflows/ci.yml   163 -
.github/workflows/freebsd.yml   32 -
.github/workflows/netbsd.yml   32 -
.github/workflows/openbsd.yml   35 -
.github/workflows/solaris.yml   32 -
.github/workflows/windows-ci.yml   124 -
.gitignore   113 -
ABOUT-NLS   1 +
ChangeLog   17392 +++++++++++++++++++++
Makefile.in   1097 +++++++
aclocal.m4   1353 ++++++++
build-aux/ci_build.bash   286 --
build-aux/compile   351 ++
build-aux/config.guess   1815 ++++++++++
build-aux/config.rpath   751 +++++
build-aux/config.sub   2354 +++++++++++++
build-aux/depcomp   792 +++++
build-aux/install-sh   541 +++
build-aux/ltmain.sh   11524 ++++++++++++++++++++++
build-aux/missing   236 ++
build-aux/test-driver   160 +
config.h.in   634 ++++
configure   26434 ++++++++++++++++++++++
debug/Makefile.in   756 +++++
doc/SHA256SUMS   236 --
doc/man/txt/lzmainfo.txt   36 +
doc/man/txt/xz.txt   1708 ++++++++++
doc/man/txt/xzdec.txt   76 +
doc/man/txt/xzdiff.txt   39 +
doc/man/txt/xzgrep.txt   70 +
doc/man/txt/xzless.txt   36 +
doc/man/txt/xzmore.txt   31 +
lib/Makefile.in   623 ++++
m4/.gitignore   40 -
m4/build-to-host.m4   274 ++
m4/gettext.m4   392 +++
m4/host-cpu-c-abi.m4   529 +++
m4/iconv.m4   324 ++
m4/intlmacosx.m4   71 +
m4/lib-ld.m4   170 +
m4/lib-link.m4   815 +++++
m4/lib-prefix.m4   334 ++
m4/libtool.m4   8488 +++++++++++++++++++++
m4/ltoptions.m4   467 +++
m4/ltsugar.m4   124 +
m4/ltversion.m4   24 +
m4/lt~obsolete.m4   99 +
m4/nls.m4   33 +
m4/po.m4   456 +++
m4/progtest.m4   92 +
po/.gitignore   31 -
po/Makefile.in.in   517 +++
po/Rules-quot   66 +
po/boldquot.sed   21 +
po/ca.gmo   Bin 0 -> 15587 bytes
po/cs.gmo   Bin 0 -> 7983 bytes
po/da.gmo   Bin 0 -> 9040 bytes
po/de.gmo   Bin 0 -> 29882 bytes
po/en@boldquot.header   35 +
po/en@quot.header   32 +
po/eo.gmo   Bin 0 -> 15060 bytes
po/es.gmo   Bin 0 -> 29228 bytes
po/fi.gmo   Bin 0 -> 28225 bytes
po/fr.gmo   Bin 0 -> 10232 bytes

To be able to easily inspect exactly what changed in the release tarball compared to git release tag contents, the best tool for the job is Meld, invoked via git difftool --dir-diff fa1e8796^..fa1e8796. To compare changes across the new and old upstream tarball, one would need to compare commits afba662b New upstream version 5.8.0 and fa1e8796 New upstream version 5.8.1 by running git difftool --dir-diff afba662b..fa1e8796. With all the above tips you can now go and try to audit your own favorite package in Debian and see if it is identical with upstream, and if not, how it differs.

Should the XZ backdoor have been detected using these tools? The famous XZ Utils backdoor (CVE-2024-3094) consisted of two parts: the actual backdoor inside two binary blobs masqueraded as a test files (tests/files/bad-3-corrupt_lzma2.xz, tests/files/good-large_compressed.lzma), and a small modification in the build scripts (m4/build-to-host.m4) to extract the backdoor and plant it into the built binary. The build script was not tracked in version control, but generated with GNU Autotools at release time and only shipped as additional files in the release tarball. The entire reason for me to write this post was to ponder if a diligent engineer using git-buildpackage best practices could have reasonably spotted this while importing the new upstream release into Debian. The short answer is no . The malicious actor here clearly anticipated all the typical ways anyone might inspect both git commits, and release tarball contents, and masqueraded the changes very well and over a long timespan. First of all, XZ has for legitimate reasons for several carefully crafted .xz files as test data to help catch regressions in the decompression code path. The test files are shipped in the release so users can run the test suite and validate that the binary is built correctly and xz works properly. Debian famously runs massive amounts of testing in its CI and autopkgtest system across tens of thousands of packages to uphold high quality despite frequent upgrades of the build toolchain and while supporting more CPU architectures than any other distro. Test data is useful and should stay. When git-buildpackage is used correctly, the upstream commits are visible in the Debian packaging for easy review, but the commit cf44e4b that introduced the test files does not deviate enough from regular sloppy coding practices to really stand out. It is unfortunately very common for git commit to lack a message body explaining why the change was done, and to not be properly atomic with test code and test data together in the same commit, and for commits to be pushed directly to mainline without using code reviews (the commit was not part of any PR in this case). Only another upstream developer could have spotted that this change is not on par to what the project expects, and that the test code was never added, only test data, and thus that this commit was not just a sloppy one but potentially malicious. Secondly, the fact that a new Autotools file appeared (m4/build-to-host.m4) in the XZ Utils 5.6.0 is not suspicious. This is perfectly normal for Autotools. In fact, starting from XZ Utils version 5.8.1 it is now shipping a m4/build-to-host.m4 file that it actually uses now. Spotting that there is anything fishy is practically impossible by simply reading the code, as Autotools files are full custom m4 syntax interwoven with shell script, and there are plenty of backticks ( ) that spawn subshells and evals that execute variable contents further, which is just normal for Autotools. Russ Cox s XZ post explains how exactly the Autotools code fetched the actual backdoor from the test files and injected it into the build. There is only one tiny thing that maybe a very experienced Autotools user could potentially have noticed: the serial 30 in the version header is way too high. In theory one could also have noticed this Autotools file deviates from what other packages in Debian ship with the same filename, such as e.g. the serial 3, serial 5a or 5b versions. That would however require and an insane amount extra checking work, and is not something we should plan to start doing. A much simpler solution would be to simply strongly recommend all open source projects to stop using Autotools to eventually get rid of it entirely.

Not detectable with reasonable effort While planting backdoors is evil, it is hard not to feel some respect to the level of skill and dedication of the people behind this. I ve been involved in a bunch of security breach investigations during my IT career, and never have I seen anything this well executed. If it hadn t slowed down SSH by ~500 milliseconds and been discovered due to that, it would most likely have stayed undetected for months or years. Hiding backdoors in closed source software is relatively trivial, but hiding backdoors in plain sight in a popular open source project requires some unusual amount of expertise and creativity as shown above.

Is the software supply-chain in Debian easy to audit? While maintaining a Debian package source using git-buildpackage can make the package history a lot easier to inspect, most packages have incomplete configurations in their debian/gbp.conf, and thus their package development histories are not always correctly constructed or uniform and easy to compare. The Debian Policy does not mandate git usage at all, and there are many important packages that are not using git at all. Additionally the Debian Policy also allows for non-maintainers to upload new versions to Debian without committing anything in git even for packages where the original maintainer wanted to use git. Uploads that bypass git unfortunately happen surpisingly often. Because of the situation, I am afraid that we could have multiple similar backdoors lurking that simply haven t been detected yet. More audits, that hopefully also get published openly, would be welcome! More people auditing the contents of the Debian archives would probably also help surface what tools and policies Debian might be missing to make the work easier, and thus help improve the security of Debian s users, and improve trust in Debian.

Is Debian currently missing some software that could help detect similar things? To my knowledge there is currently no system in place as part of Debian s QA or security infrastructure to verify that the upstream source packages in Debian are actually from upstream. I ve come across a lot of packages where the debian/watch or other configs are incorrect and even cases where maintainers have manually created upstream tarballs as it was easier than configuring automation to work. It is obvious that for those packages the source tarball now in Debian is not at all the same as upstream. I am not aware of any malicious cases though (if I was, I would report them of course). I am also aware of packages in the Debian repository that are misconfigured to be of type 1.0 (native) packages, mixing the upstream files and debian/ contents and having patches applied, while they actually should be configured as 3.0 (quilt), and not hide what is the true upstream sources. Debian should extend the QA tools to scan for such things. If I find a sponsor, I might build it myself as my next major contribution to Debian. In addition to better tooling for finding mismatches in the source code, Debian could also have better tooling for tracking in built binaries what their source files were, but solutions like Fraunhofer-AISEC s supply-graph or Sony s ESSTRA are not practical yet. Julien Malka s post about NixOS discusses the role of reproducible builds, which may help in some cases across all distros.

Or, is Debian missing some policies or practices to mitigate this? Perhaps more importantly than more security scanning, the Debian Developer community should switch the general mindset from anyone is free to do anything to valuing having more shared workflows. The ability to audit anything is severely hampered by the fact that there are so many ways to do the same thing, and distinguishing what is a normal deviation from a malicious deviation is too hard, as the normal can basically be almost anything. Also, as there is no documented and recommended default workflow, both those who are old and new to Debian packaging might never learn any one optimal workflow, and end up doing many steps in the packaging process in a way that kind of works, but is actually wrong or unnecessary, causing process deviations that look malicious, but turn out to just be a result of not fully understanding what would have been the right way to do something. In the long run, once individual developers workflows are more aligned, doing code reviews will become a lot easier and smoother as the excess noise of workflow differences diminishes and reviews will feel much more productive to all participants. Debian fostering a culture of code reviews would allow us to slowly move from the current practice of mainly solo packaging work towards true collaboration forming around those code reviews. I have been promoting increased use of Merge Requests in Debian already for some time, for example by proposing DEP-18: Encourage Continuous Integration and Merge Request based Collaboration for Debian packages. If you are involved in Debian development, please give a thumbs up in dep-team/deps!21 if you want me to continue promoting it.

Can we trust open source software? Yes and I would argue that we can only trust open source software. There is no way to audit closed source software, and anyone using e.g. Windows or MacOS just have to trust the vendor s word when they say they have no intentional or accidental backdoors in their software. Or, when the news gets out that the systems of a closed source vendor was compromised, like Crowdstrike some weeks ago, we can t audit anything, and time after time we simply need to take their word when they say they have properly cleaned up their code base. In theory, a vendor could give some kind of contractual or financial guarantee to its customer that there are no preventable security issues, but in practice that never happens. I am not aware of a single case of e.g. Microsoft or Oracle would have paid damages to their customers after a security flaw was found in their software. In theory you could also pay a vendor more to have them focus more effort in security, but since there is no way to verify what they did, or to get compensation when they didn t, any increased fees are likely just pocketed as increased profit. Open source is clearly better overall. You can, if you are an individual with the time and skills, audit every step in the supply-chain, or you could as an organization make investments in open source security improvements and actually verify what changes were made and how security improved. If your organisation is using Debian (or derivatives, such as Ubuntu) and you are interested in sponsoring my work to improve Debian, please reach out.

Debian LTS contributors In September, 20 contributors have been paid to work on Debian LTS, their reports are available:

• Abhijith PA did 10.0h (out of 10.0h assigned and 4.0h from previous period), thus carrying over 4.0h to the next month.
• Andreas Henriksson did 1.0h (out of 0.0h assigned and 20.0h from previous period), thus carrying over 19.0h to the next month.
• Bastien Roucari s did 20.0h (out of 20.0h assigned).
• Ben Hutchings did 20.0h (out of 21.0h assigned), thus carrying over 1.0h to the next month.
• Carlos Henrique Lima Melara did 10.0h (out of 12.0h assigned), thus carrying over 2.0h to the next month.
• Chris Lamb did 18.0h (out of 18.0h assigned).
• Daniel Leidert did 21.0h (out of 21.0h assigned).
• Emilio Pozuelo Monfort did 39.75h (out of 40.0h assigned), thus carrying over 0.25h to the next month.
• Guilhem Moulin did 15.0h (out of 15.0h assigned).
• Jochen Sprickerhof did 12.0h (out of 9.25h assigned and 11.75h from previous period), thus carrying over 9.0h to the next month.
• Lee Garrett did 13.5h (out of 21.0h assigned), thus carrying over 7.5h to the next month.
• Lucas Kanashiro did 8.0h (out of 20.0h assigned), thus carrying over 12.0h to the next month.
• Markus Koschany did 15.0h (out of 3.25h assigned and 17.75h from previous period), thus carrying over 6.0h to the next month.
• Paride Legovini did 6.0h (out of 8.0h assigned), thus carrying over 2.0h to the next month.
• Roberto C. S nchez did 7.25h (out of 7.75h assigned and 13.25h from previous period), thus carrying over 13.75h to the next month.
• Santiago Ruano Rinc n did 13.25h (out of 13.5h assigned and 1.5h from previous period), thus carrying over 1.75h to the next month.
• Sylvain Beucler did 17.0h (out of 7.75h assigned and 13.25h from previous period), thus carrying over 4.0h to the next month.
• Thorsten Alteholz did 21.0h (out of 21.0h assigned).
• Tobias Frost did 5.0h (out of 0.0h assigned and 8.0h from previous period), thus carrying over 3.0h to the next month.
• Utkarsh Gupta did 16.5h (out of 14.25h assigned and 6.75h from previous period), thus carrying over 4.5h to the next month.

Evolution of the situation In September, we released 38 DLAs.

• Notable security updates:
  • modsecurity-apache prepared by Adrian Bunk, fixes a cross-site scripting vulnerability
  • cups, prepared by Thorsten Alteholz, fixes authentication bypass and denial of service vulnerabilities
  • jetty9, prepared by Adrian Bunk, fixes the MadeYouReset vulnerability (a recent, well-known denial of service vulnerability)
  • python-django, prepared by Chris Lamb, fixes a SQL injection vulnerability
  • firefox-esr and thunderbird, prepared by Emilio Pozuelo Monfort, were updated from the 128.x ESR series to the 140.x ESR series, fixing a number of vulnerabilities as well
• Notable non-security updates:
  • wireless-regdb prepared by Ben Hutchings, updates information reflecting changes to radio regulations in many countries

There was one package update contributed by a Debian Developer outside of the LTS Team: an update of node-tar-fs, prepared by Xavier Guimard (a member of the Node packaging team). Finally, LTS Team members also contributed updates of the following packages:

• libxslt (to stable and oldstable), prepared by Guilhem Moulin, to address a regression introduced in a previous security update
• libphp-adodb (to stable and oldstable), prepared by Abhijith PA
• cups (to stable and oldstable), prepared by Thorsten Alteholz
• u-boot (to oldstable), prepared by Daniel Leidert and Jochen Sprickerhof
• libcommongs-lang3-java (to stable and oldstable), prepared by Daniel Leidert
• python-internetarchive (to oldstable), prepared by Daniel Leidert

One other notable contribution by a member of the LTS Team is that Sylvain Beucler proposed a fix upstream for CVE-2025-2760 in gimp2. Upstream no longer supports gimp2, but it is still present in Debian LTS, and so proposing this fix upstream is of benefit to other distros which may still be supporting the older gimp2 packages.

Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • Toshiba Corporation (for 120 months)
  • Civil Infrastructure Platform (CIP) (for 88 months)
  • VyOS Inc (for 52 months)
• Gold sponsors:
  • F. Hoffmann-La Roche AG (for 130 months)
  • Akamai - Linode (for 124 months)
  • Babiel GmbH (for 114 months)
  • Plat Home (for 113 months)
  • University of Oxford (for 70 months)
  • Deveryware (for 57 months)
  • EDF SA (for 42 months)
  • Dataport A R (for 17 months)
  • CERN (for 15 months)
• Silver sponsors:
  • Domeneshop AS (for 135 months)
  • Nantes M tropole (for 129 months)
  • Univention GmbH (for 121 months)
  • Universit Jean Monnet de St Etienne (for 121 months)
  • Ribbon Communications, Inc. (for 115 months)
  • Exonet B.V. (for 105 months)
  • Leibniz Rechenzentrum (for 99 months)
  • Minist re de l Europe et des Affaires trang res (for 83 months)
  • Cloudways by DigitalOcean (for 72 months)
  • Dinahosting SL (for 70 months)
  • Platform.sh SAS (for 64 months)
  • Moxa Inc. (for 58 months)
  • sipgate GmbH (for 56 months)
  • OVH US LLC (for 54 months)
  • Tilburg University (for 54 months)
  • GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 45 months)
  • THINline s.r.o. (for 18 months)
  • Copenhagen Airports A/S (for 12 months)
• Bronze sponsors:
  • Evolix (for 135 months)
  • Seznam.cz, a.s. (for 135 months)
  • Intevation GmbH (for 132 months)
  • Linuxhotel GmbH (for 132 months)
  • Daevel SARL (for 131 months)
  • Megaspace Internet Services GmbH (for 130 months)
  • Greenbone AG (for 129 months)
  • NUMLOG (for 129 months)
  • WinGo AG (for 128 months)
  • Entr ouvert (for 120 months)
  • Adfinis AG (for 117 months)
  • Laboratoire LEGI - UMR 5519 / CNRS (for 112 months)
  • Tesorion (for 112 months)
  • Bearstech (for 103 months)
  • LiHAS (for 103 months)
  • Catalyst IT Ltd (for 98 months)
  • Demarcq SAS (for 92 months)
  • Universit Grenoble Alpes (for 78 months)
  • TouchWeb SAS (for 70 months)
  • SPiN AG (for 67 months)
  • CoreFiling (for 63 months)
  • Institut des sciences cognitives Marc Jeannerod (for 58 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 54 months)
  • Tem Innovations GmbH (for 49 months)
  • WordFinder.pro (for 48 months)
  • CNRS DT INSU R sif (for 47 months)
  • Soliton Systems K.K. (for 42 months)
  • Alter Way (for 40 months)
  • Institut Camille Jordan (for 30 months)
  • SOBIS Software GmbH (for 15 months)
  • Tuxera Inc. (for 6 months)

• Debian packages:
  • firmware-nonfree:
    • Uploads:
      • uploaded version 20250808-1~bpo13+1 to trixie-backports
  • initramfs-tools:
    • Merge requests:
      • closed !188: Use portable pathnames
  • iw:
    • Bugs:
      • closed #499868: udev: multiple mac80211 interfaces created on the same physical device are incorrectly renamed
      • closed #930271: iw: cannot scan if two cells on 2.4Ghz and 5Gz have same SSID
      • closed #943988: Usage message jumbled
    • Uploads:
      • uploaded version 6.17-1 to unstable
  • linux:
    • Bugs:
      • replied to #1112541: linux: initrd grew significantly from 6.12.38+deb13 to 6.16.3+deb14 causing ENOSPACE
      • replied to #1114773: linux-image-6.1.0-39-amd64: Kernel panic when booting with kernel commandline parameter fips=1
      • opened #1114857: Switch default initramfs builder to dracut
    • Merge requests:
      • reviewed !1548: Merge the extract-source job into the build s job script
      • merged !1625: [rt] Add new signing key for Daniel Wagner
      • merged !1633: drivers/leds/trigger: Enable LEDS_TRIGGER_INPUT_EVENTS as module
      • reviewed !1634: Disable bcachefs filesystem support
      • reviewed !1636: d/b/test-patches: Stop handling rt featureset for test builds
      • opened !1637: Revert to using RSA for module signatures
      • reviewed !1644: Backport Merge the extract-source job into the build s job script
      • opened and merged !1662: Update CI configuration for salsa-ci switch to sbuild
      • opened !1663: d/salsa-ci.yml: Adjust filenames to allow source package name suffix
    • Uploads:
      • uploaded version 6.12.43-1~bpo12+1 to bookworm-backports
      • uploaded version 6.16.3-1~bpo13+1 to trixie-backports
      • (LTS) uploaded version 5.10.244-1 to bullseye-security
  • (LTS) linux-6.1:
    • Uploads:
      • uploaded version 6.1.148-1~deb11u1 to bullseye-security
      • uploaded version 6.1.153-1~deb11u1 to bullseye-security
  • linux-base:
    • Bugs:
      • closed #1114659: linux-run-hooks depends on recent run-parts
  • wireless-regdb:
    • Uploads:
      • (LTS) uploaded version 2025.07.10-1~deb11u1 to bullseye-security
• Mailing lists:
  • debian-devel:
    • replied to Inquiry about armel support for Debian upgrades post-Trixie
  • debian-kernel:
    • posted Agenda items for kernel-team meeting on 2025-09-24
    • replied to Clarification on Subpackage Mapping for CVE Fixes in Debian Linux Kernel
    • replied to [RFC] Reorganizing Linux packages
  • debian-lts-announce:
    • posted [SECURITY] [DLA 4293-1] wireless-regdb new upstream version
  • debian-rust:
    • posted and replied to Questions about Rust team for Kangrejos
  • debian-www:
    • replied to Stale copy of Debian Kernel Handbook
  • stable:
    • replied to [PATCH 5.10 26/52] e1000e: fix heap overflow in e1000_set_eeprom

• [1] https://tinyurl.com/2dztdg77
• [2] https://tinyurl.com/2b2yvgfx
• [3] https://tinyurl.com/yeppy78q
• [4] https://philwip.com/2025/04/14/galaxian-3-project-revival/
• [5] https://tinyurl.com/25ylubnc
• [6] https://tinyurl.com/27jnzjkt
• [7] https://tinyurl.com/29fl4q39
• [8] https://tinyurl.com/2ye367yn
• [9] https://tinyurl.com/24bx8nds
• [10] https://pluralistic.net/2025/09/27/econopocalypse/

$ git diff --shortstat v1.11.5
 493 files changed, 25015 insertions(+), 21135 deletions(-)

New outlet service The major change in Akvorado 2.0 is splitting the inlet service into two parts: the inlet and the outlet. Previously, the inlet handled all flow processing: receiving, decoding, and enrichment. Flows were then sent to Kafka for storage in ClickHouse:

Akvorado flow processing before the introduction of the outlet service

Network flows reach the inlet service using UDP, an unreliable protocol. The inlet must process them fast enough to avoid losing packets. To handle a high number of flows, the inlet spawns several sets of workers to receive flows, fetch metadata, and assemble enriched flows for Kafka. Many configuration options existed for scaling, which increased complexity for users. The code needed to avoid blocking at any cost, making the processing pipeline complex and sometimes unreliable, particularly the BMP receiver.¹ Adding new features became difficult without making the problem worse.² In Akvorado 2.0, the inlet receives flows and pushes them to Kafka without decoding them. The new outlet service handles the remaining tasks:

Akvorado flow processing after the introduction of the outlet service

This change goes beyond a simple split:³ the outlet now reads flows from Kafka and pushes them to ClickHouse, two tasks that Akvorado did not handle before. Flows are heavily batched to increase efficiency and reduce the load on ClickHouse using ch-go, a low-level Go client for ClickHouse. When batches are too small, asynchronous inserts are used (e20645). The number of outlet workers scales dynamically (e5a625) based on the target batch size and latency (50,000 flows and 5 seconds by default). This new architecture also allows us to simplify and optimize the code. The outlet fetches metadata synchronously (e20645). The BMP component becomes simpler by removing cooperative multitasking (3b9486). Reusing the same RawFlow object to decode protobuf-encoded flows from Kafka reduces pressure on the garbage collector (8b580f). The effect on Akvorado s overall performance was somewhat uncertain, but a user reported 35% lower CPU usage after migrating from the previous version, plus resolution of the long-standing BMP component issue.

Other changes This new version includes many miscellaneous changes, such as completion for source and destination ports (f92d2e), and automatic restart of the orchestrator service (0f72ff) when configuration changes to avoid a common pitfall for newcomers. Let s focus on some key areas for this release: observability, documentation, CI, Docker, Go, and JavaScript.

Observability Akvorado exposes metrics to provide visibility into the processing pipeline and help troubleshoot issues. These are available through Prometheus HTTP metrics endpoints, such as /api/v0/inlet/metrics. With the introduction of the outlet, many metrics moved. Some were also renamed (4c0b15) to match Prometheus best practices. Kafka consumer lag was added as a new metric (e3a778). If you do not have your own observability stack, the Docker Compose setup shipped with Akvorado provides one. You can enable it by activating the profiles introduced for this purpose (529a8f). The prometheus profile ships Prometheus to store metrics and Alloy to collect them (2b3c46, f81299, and 8eb7cd). Redis and Kafka metrics are collected through the exporter bundled with Alloy (560113). Other metrics are exposed using Prometheus metrics endpoints and are automatically fetched by Alloy with the help of some Docker labels, similar to what is done to configure Traefik. cAdvisor was also added (83d855) to provide some container-related metrics. The loki profile ships Loki to store logs (45c684). While Alloy can collect and ship logs to Loki, its parsing abilities are limited: I could not find a way to preserve all metadata associated with structured logs produced by many applications, including Akvorado. Vector replaces Alloy (95e201) and features a domain-specific language, VRL, to transform logs. Annoyingly, Vector currently cannot retrieve Docker logs from before it was started. Finally, the grafana profile ships Grafana, but the shipped dashboards are broken. This is planned for a future version.

Documentation The Docker Compose setup provided by Akvorado makes it easy to get the web interface up and running quickly. However, Akvorado requires a few mandatory steps to be functional. It ships with comprehensive documentation, including a chapter about troubleshooting problems. I hoped this documentation would reduce the support burden. It is difficult to know if it works. Happy users rarely report their success, while some users open discussions asking for help without reading much of the documentation. In this release, the documentation was significantly improved.

$ git diff --shortstat v1.11.5 -- console/data/docs
 10 files changed, 1873 insertions(+), 1203 deletions(-)

The documentation was updated (fc1028) to match Akvorado s new architecture. The troubleshooting section was rewritten (17a272). Instructions on how to improve ClickHouse performance when upgrading from versions earlier than 1.10.0 was added (5f1e9a). An LLM proofread the entire content (06e3f3). Developer-focused documentation was also improved (548bbb, e41bae, and 871fc5). From a usability perspective, table of content sections are now collapsable (c142e5). Admonitions help draw user attention to important points (8ac894).

Example of use of admonitions in Akvorado's documentation

Continuous integration This release includes efforts to speed up continuous integration on GitHub. Coverage and race tests run in parallel (6af216 and fa9e48). The Docker image builds during the tests but gets tagged only after they succeed (8b0dce).

GitHub workflow to test and build Akvorado

End-to-end tests (883e19) ensure the shipped Docker Compose setup works as expected. Hurl runs tests on various HTTP endpoints, particularly to verify metrics (42679b and 169fa9). For example:

## Test inlet has received NetFlow flows
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_flow_input_udp_packets_total job="akvorado-inlet",listener=":2055" )
HTTP 200
[Captures]
inlet_receivedflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_receivedflows" > 10
## Test inlet has sent them to Kafka
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_kafka_sent_messages_total job="akvorado-inlet" )
HTTP 200
[Captures]
inlet_sentflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_sentflows" >=   inlet_receivedflows  

Docker Akvorado ships with a comprehensive Docker Compose setup to help users get started quickly. It ensures a consistent deployment, eliminating many configuration-related issues. It also serves as a living documentation of the complete architecture. This release brings some small enhancements around Docker:

• an example to configure Traefik for TLS with Let s Encrypt (1a27bb),
• HTTP compression (bee9a5), and
• support of IPv6 (a74a41).

Previously, many Docker images were pulled from the Bitnami Containers library. However, VMWare acquired Bitnami in 2019 and Broadcom acquired VMWare in 2023. As a result, Bitnami images were deprecated in less than a month. This was not really a surprise⁴. Previous versions of Akvorado had already started moving away from them. In this release, the Apache project s Kafka image replaces the Bitnami one (1eb382). Thanks to the switch to KRaft mode, Zookeeper is no longer needed (0a2ea1, 8a49ca, and f65d20). Akvorado s Docker images were previously compiled with Nix. However, building AArch64 images on x86-64 is slow because it relies on QEMU userland emulation. The updated Dockerfile uses multi-stage and multi-platform builds: one stage builds the JavaScript part on the host platform, one stage builds the Go part cross-compiled on the host platform, and the final stage assembles the image on top of a slim distroless image (268e95 and d526ca).

# This is a simplified version
FROM --platform=$BUILDPLATFORM node:20-alpine AS build-js
RUN apk add --no-cache make
WORKDIR /build
COPY console/frontend console/frontend
COPY Makefile .
RUN make console/data/frontend
FROM --platform=$BUILDPLATFORM golang:alpine AS build-go
RUN apk add --no-cache make curl zip
WORKDIR /build
COPY . .
COPY --from=build-js /build/console/data/frontend console/data/frontend
RUN go mod download
RUN make all-indep
ARG TARGETOS TARGETARCH TARGETVARIANT VERSION
RUN make
FROM gcr.io/distroless/static:latest
COPY --from=build-go /build/bin/akvorado /usr/local/bin/akvorado
ENTRYPOINT [ "/usr/local/bin/akvorado" ]

When building for multiple platforms with --platform linux/amd64,linux/arm64,linux/arm/v7, the build steps until the highlighted line execute only once for all platforms. This significantly speeds up the build. Akvorado now ships Docker images for these platforms: linux/amd64, linux/amd64/v3, linux/arm64, and linux/arm/v7. When requesting ghcr.io/akvorado/akvorado, Docker selects the best image for the current CPU. On x86-64, there are two choices. If your CPU is recent enough, Docker downloads linux/amd64/v3. This version contains additional optimizations and should run faster than the linux/amd64 version. It would be interesting to ship an image for linux/arm64/v8.2, but Docker does not support the same mechanism for AArch64 yet (792808).

Go This release includes many changes related to Go but not visible to the users.

Toolchain In the past, Akvorado supported the two latest Go versions, preventing immediate use of the latest enhancements. The goal was to allow users of stable distributions to use Go versions shipped with their distribution to compile Akvorado. However, this became frustrating when interesting features, like go tool, were released. Akvorado 2.0 requires Go 1.25 (77306d) but can be compiled with older toolchains by automatically downloading a newer one (94fb1c).⁵ Users can still override GOTOOLCHAIN to revert this decision. The recommended toolchain updates weekly through CI to ensure we get the latest minor release (5b11ec). This change also simplifies updates to newer versions: only go.mod needs updating. Thanks to this change, Akvorado now uses wg.Go() (77306d) and I have started converting some unit tests to the new test/synctest package (bd787e, 7016d8, and 159085).

Testing When testing equality, I use a helper function Diff() to display the differences when it fails:

got := input.Keys()
expected := []int 1, 2, 3 
if diff := helpers.Diff(got, expected); diff != ""  
    t.Fatalf("Keys() (-got, +want):\n%s", diff)
 

This function uses kylelemons/godebug. This package is no longer maintained and has some shortcomings: for example, by default, it does not compare struct private fields, which may cause unexpectedly successful tests. I replaced it with google/go-cmp, which is stricter and has better output (e2f1df).

Another package for Kafka Another change is the switch from Sarama to franz-go to interact with Kafka (756e4a and 2d26c5). The main motivation for this change is to get a better concurrency model. Sarama heavily relies on channels and it is difficult to understand the lifecycle of an object handed to this package. franz-go uses a more modern approach with callbacks⁶ that is both more performant and easier to understand. It also ships with a package to spawn fake Kafka broker clusters, which is more convenient than the mocking functions provided by Sarama.

Improved routing table for BMP To store its routing table, the BMP component used kentik/patricia, an implementation of a patricia tree focused on reducing garbage collection pressure. gaissmai/bart is a more recent alternative using an adaptation of [Donald Knuth s ART algorithm][] that promises better performance and delivers it: 90% faster lookups and 27% faster insertions (92ee2e and fdb65c). Unlike kentik/patricia, gaissmai/bart does not help efficiently store values attached to each prefix. I adapted the same approach as kentik/patricia to store route lists for each prefix: store a 32-bit index for each prefix, and use it to build a 64-bit index for looking up routes in a map. This leverages Go s efficient map structure. gaissmai/bart also supports a lockless routing table version, but this is not simple because we would need to extend this to the map storing the routes and to the interning mechanism. I also attempted to use Go s new unique package to replace the intern package included in Akvorado, but performance was worse.⁷

Miscellaneous Previous versions of Akvorado were using a custom Protobuf encoder for performance and flexibility. With the introduction of the outlet service, Akvorado only needs a simple static schema, so this code was removed. However, it is possible to enhance performance with planetscale/vtprotobuf (e49a74, and 8b580f). Moreover, the dependency on protoc, a C++ program, was somewhat annoying. Therefore, Akvorado now uses buf, written in Go, to convert a Protobuf schema into Go code (f4c879). Another small optimization to reduce the size of the Akvorado binary by 10 MB was to compress the static assets embedded in Akvorado in a ZIP file. It includes the ASN database, as well as the SVG images for the documentation. A small layer of code makes this change transparent (b1d638 and e69b91).

JavaScript Recently, two large supply-chain attacks hit the JavaScript ecosystem: one affecting the popular packages chalk and debug and another impacting the popular package @ctrl/tinycolor. These attacks also exist in other ecosystems, but JavaScript is a prime target due to heavy use of small third-party dependencies. The previous version of Akvorado relied on 653 dependencies. npm-run-all was removed (3424e8, 132 dependencies). patch-package was removed (625805 and e85ff0, 69 dependencies) by moving missing TypeScript definitions to env.d.ts. eslint was replaced with oxlint, a linter written in Rust (97fd8c, 125 dependencies, including the plugins). I switched from npm to Pnpm, an alternative package manager (fce383). Pnpm does not run install scripts by default⁸ and prevents installing packages that are too recent. It is also significantly faster.⁹ Node.js does not ship Pnpm but it ships Corepack, which allows us to use Pnpm without installing it. Pnpm can also list licenses used by each dependency, removing the need for license-compliance (a35ca8, 42 dependencies). For additional speed improvements, beyond switching to Pnpm and Oxlint, Vite was replaced with its faster Rolldown version (463827). After these changes, Akvorado only pulls 225 dependencies.

Next steps I would like to land three features in the next version of Akvorado:

• Add Grafana dashboards to complete the observability stack. See issue #1906 for details.
• Integrate OVH s Grafana plugin by providing a stable API for such integrations. Akvorado s web console would still be useful for browsing results, but if you want to build and share dashboards, you should switch to Grafana. See issue #1895.
• Move some work currently done in ClickHouse (custom dictionaries, GeoIP and IP enrichment) back into the outlet service. This should give more flexibility for adding features like the one requested in issue #1030. See issue #2006.

---

I started working on splitting the inlet into two parts more than one year ago. I found more motivation in recent months, partly thanks to Claude Code, which I used as a rubber duck. Almost none of the produced code was kept:¹⁰ it is like an intern who does not learn.

---

1. Many attempts were made to make the BMP component both performant and not blocking. See for example PR #254, PR #255, and PR #278. Despite these efforts, this component remained problematic for most users. See issue #1461 as an example.
2. Some features have been pushed to ClickHouse to avoid the processing cost in the inlet. See for example PR #1059.
3. This is the biggest commit:

   $ git show --shortstat ac68c5970e2c   tail -1
   231 files changed, 6474 insertions(+), 3877 deletions(-)
   

4. Broadcom is known for its user-hostile moves. Look at what happened with VMWare.
5. As a Debian developer, I dislike these mechanisms that circumvent the distribution package manager. The final straw came when Go 1.25 spent one month in the Debian NEW queue, an arbitrary mechanism I don t like at all.
6. In the early years of Go, channels were heavily promoted. Sarama was designed during this period. A few years later, a more nuanced approach emerged. See notably Go channels are bad and you should feel bad.
7. This should be investigated further, but my theory is that the intern package uses 32-bit integers, while unique uses 64-bit pointers. See commit 74e5ac.
8. This is also possible with npm. See commit dab2f7.
9. An even faster alternative is Bun, but it is less available.
10. The exceptions are part of the code for the admonition blocks, the code for collapsing the table of content, and part of the documentation.

Changes in RcppArmadillo version 15.0.2-2 (2025-09-18)

• Minor update to skeleton Makevars,Makevars.win
• Update README.md to mention ldlasb2 repository
• Minor documentation update (#487)
• Synchronized with Armadillo upstream (#488)
• Refine Armadillo version selection in coordination with CRAN maintainers to support transition towards Armadillo 15.0.*

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

1. Separate application-specific users with granular permissions allowing only necessary access and no more.
2. Distributing and storing passwords and credentials securely
3. Ensuring all remote connections are properly encrypted

How encrypting database connections with TLS differs from web server HTTP(S) Even though MariaDB (and other databases) use the same SSL/TLS protocol for encrypting remote connections as web servers and HTTPS, the way it is implemented is significantly different, and the different security assumptions are important for a database administrator to grasp. Firstly, most HTTP requests to a web server are unauthenticated, meaning the web server serves public web pages and does not require users to log in. Traditionally, when a user logs in over a HTTP connection, the username and password were transmitted in plaintext as a HTTP POST request. Modern TLS, which was previously called SSL, does not change how HTTP works but simply encapsulates it. When using HTTPS, a web browser and a web server will start an encrypted TLS connection as the very first thing, and only once established, do they send HTTP requests and responses inside it. There are no passwords or other shared secrets needed to form the TLS connection. Instead, the web server relies on a trusted third party, a Certificate Authority (CA), to vet that the TLS certificate offered by the web server can be trusted by the web browser. For a database server like MariaDB, the situation is quite different. All users need to first authenticate and log in to the server before getting being allowed to run any SQL and getting any data out of the server. The database server and client programs have built-in authentication methods, and passwords are not, and have never been, sent in plaintext. Over the years, MySQL and its successor, MariaDB, have had multiple password authentication methods: the original SHA-1-based hashing, later double SHA-1-based mysql_native_password, followed by sha256_password and caching_sha256_password in MySQL and ed25519 in MariaDB. The MariaDB.org blog post by Sergei Golubchik recaps the history of these well. Even though most modern MariaDB installations should be using TLS to encrypt all remote connections in 2025, having the authentication method be as secure as possible still matters, because authentication is done before the TLS connection is fully established. To further harden the authentication agains man-in-the-middle attacks, a new password the authentication method PARSEC was introduced in MariaDB 11.8, which builds upon the previous ed25519 public-key-based verification (similar to how modern SSH does), and also combines key derivation with PBKDF2 with hash functions (SHA512,SHA256) and a high iteration count. At first it may seem like a disadvantage to not wrap all connections in a TLS tunnel like HTTPS does, but actually not having the authentication done in a MitM resistant way regardless of the connection encryption status allows a clever extra capability that is now available in MariaDB: as the database server and client already have a shared secret that is being used by the server to authenticate the user, it can also be used by the client to validate the server s TLS certificate and no third parties like CAs or root certificates are needed. MariaDB 11.8 was the first LTS version to ship with this capability for zero-configuration TLS. Note that the zero-configuration TLS also works on older password authentication methods and does not require users to have PARSEC enabled. As PARSEC is not yet the default authentication method in MariaDB, it is recommended to enable it in installations that use zero-configuration TLS encryption to maximize the security of the TLS certificate validation.

Why the root user in MariaDB has no password and how it makes the database more secure Relying on passwords for security is problematic, as there is always a risk that they could leak, and a malicious user could access the system using the leaked password. It is unfortunately far too common for database passwords to be stored in plaintext in configuration files that are accidentally committed into version control and published on GitHub and similar platforms. Every application or administrative password that exists should be tracked to ensure only people who need it know it, and rotated at regular intervals to ensure old employees etc won t be able to use old passwords. This password management is complex and error-prone. Replacing passwords with other authentication methods is always advisable when possible. On a database server, whoever installed the database by running e.g. apt install mariadb-server, and configured it with e.g. nano /etc/mysql/mariadb.cnf, already has full root access to the operating system, and asking them for a password to access the MariaDB database shell is moot, since they could circumvent any checks by directly accessing the files on the system anyway. Therefore, MariaDB, since version 10.4 stopped requiring the root user to enter a password when connecting locally, and instead checks using socket authentication whether the user is the operating-system root user or equivalent (e.g. running sudo). This is an elegant way to get rid of a password that was actually unnecessary to begin with. As there is no root password anymore, the risk of an external user accessing the database as root with a leaked password is fully eliminated. Note that socket authentication only works for local connections on the same server. If you want to access a MariaDB server remotely as the root user, you would need to configure a password for it first. This is not generally recommended, as explained in the next section.

Create separate database users for normal use and keep root for administrative use only Out of the box a MariaDB installation is already secure by default, and only the local root user can connect to it. This account is intended for administrative use only, and for regular daily use you should create separate database users with access limited to the databases they need and the permissions required. The most typical commands needed to create a new database for an app and a user the app can use to connect to the database would be the following:

sql Copy

CREATE DATABASE app_db; CREATE USER 'app_user'@'%' IDENTIFIED BY 'your_secure_password'; GRANT ALL PRIVILEGES ON app_db.* TO 'app_user'@'%'; FLUSH PRIVILEGES;

CREATE DATABASE app_db;
CREATE USER 'app_user'@'%' IDENTIFIED BY 'your_secure_password';
GRANT ALL PRIVILEGES ON app_db.* TO 'app_user'@'%';
FLUSH PRIVILEGES;

Alternatively, if you want to use the parsec authentication method, run this to create the user:

sql Copy

CREATE OR REPLACE USER 'app_user'@'%' IDENTIFIED VIA parsec USING PASSWORD('your_secure_password');

CREATE OR REPLACE USER 'app_user'@'%'
 IDENTIFIED VIA parsec
 USING PASSWORD('your_secure_password');

Note that the plugin auth_parsec is not enabled by default. If you see the error message ERROR 1524 (HY000): Plugin 'parsec' is not loaded fix this by running INSTALL SONAME 'auth_parsec';. In the CREATE USER statements, the @'%' means that the user is allowed to connect from any host. This needs to be defined, as MariaDB always checks permissions based on both the username and the remote IP address or hostname of the user, combined with the authentication method. Note that it is possible to have multiple user@remote combinations, and they can have different authentication methods. A user could, for example, be allowed to log in locally using the socket authentication and over the network using a password. If you are running a custom application and you know exactly what permissions are sufficient for the database users, replace the ALL PRIVILEGES with a subset of privileges listed in the MariaDB documentation. For new permissions to take effect, restart the database or run FLUSH PRIVILEGES.

Allow MariaDB to accept remote connections and enforce TLS Using the above 'app_user'@'%' is not enough on its own to allow remote connections to MariaDB. The MariaDB server also needs to be configured to listen on a network interface to accept remote connections. As MariaDB is secure by default, it only accepts connections from localhost until the administrator updates its configuration. On a typical Debian/Ubuntu system, the recommended way is to drop a new custom config in e.g. /etc/mysql/mariadb.conf.d/99-server-customizations.cnf, with the contents:

Copy

[mariadbd] # Listen for connections from anywhere bind-address = 0.0.0.0 # Only allow TLS encrypted connections require-secure-transport = on

[mariadbd]
# Listen for connections from anywhere
bind-address = 0.0.0.0
# Only allow TLS encrypted connections
require-secure-transport = on

For settings to take effect, restart the server with systemctl restart mariadb. After this, the server will accept connections on any network interface. If the system is using a firewall, the port 3306 would additionally need to be allow-listed. To confirm that the settings took effect, run e.g. mariadb -e "SHOW VARIABLES LIKE 'bind_address';" , which should now show 0.0.0.0. When allowing remote connections, it is important to also always define require-secure-transport = on to enforce that only TLS-encrypted connections are allowed. If the server is running MariaDB 11.8 and the clients are also MariaDB 11.8 or newer, no additional configuration is needed thanks to MariaDB automatically providing TLS certificates and appropriate certificate validation in recent versions. On older long-term-supported versions of the MariaDB server one would have had to manually create the certificates and configure the ssl_key, ssl_cert and ssl_ca values on the server, and distribute the certificate to the clients as well, which was cumbersome, so good it is not required anymore. In MariaDB 11.8 the only additional related config that might still be worth setting is tls_version = TLSv1.3 to ensure only the latest TLS protocol version is used. Finally, test connections to ensure they work and to confirm that TLS is used by running e.g.:

shell Copy

mariadb --user=app_user --password=your_secure_password \ --host=192.168.1.66 -e '\s'

mariadb --user=app_user --password=your_secure_password \
 --host=192.168.1.66 -e '\s'

The result should show something along:

Copy

-------------- mariadb from 11.8.3-MariaDB, client 15.2 for debian-linux-gnu (x86_64) ... Current user: app_user@192.168.1.66 SSL: Cipher in use is TLS_AES_256_GCM_SHA384, cert is OK ...

--------------
mariadb from 11.8.3-MariaDB, client 15.2 for debian-linux-gnu (x86_64)
...
Current user: app_user@192.168.1.66
SSL: Cipher in use is TLS_AES_256_GCM_SHA384, cert is OK
...

If running a Debian/Ubuntu system, see the bundled README with zcat /usr/share/doc/mariadb-server/README.Debian.gz to read more configuration tips.

Should TLS encryption be used also on internal networks? If a database server and app are running on the same private network, the chances that the connection gets eavesdropped on or man-in-the-middle attacked by a malicious user are low. However, it is not zero, and if it happens, it can be difficult to detect or prove that it didn t happen. The benefit of using end-to-end encryption is that both the database server and the client can validate the certificates and keys used, log it, and later have logs audited to prove that connections were indeed encrypted and show how they were encrypted. If all the computers on an internal network already have centralized user account management and centralized log collection that includes all database sessions, reusing existing SSH connections, SOCKS proxies, dedicated HTTPS tunnels, point-to-point VPNs, or similar solutions might also be a practical option. Note that the zero-configuration TLS only works with password validation methods. This means that systems configured to use PAM or Kerberos/GSSAPI can t use it, but again those systems are typically part of a centrally configured network anyway and are likely to have certificate authorities and key distribution or network encryption facilities already set up. In a typical software app stac however, the simplest solution is often the best and I recommend DBAs use the end-to-end TLS encryption in MariaDB 11.8 in most cases. Hopefully with these tips you can enjoy having your MariaDB deployments both simpler and more secure than before!

• The cost of enabling data checksums on this bad-cache-ratio workload is about 0.25 %.
• The cost of enabling both TDE encryption and data checksums on this workload is about 0.5%.