Gunnar Wolf: 404 not found
Found this grafitti on the wall behind my house today:
Found this grafitti on the wall behind my house today:
Found this grafitti on the wall behind my house today:
Found this grafitti on the wall behind my house today:
autopkgtest. For some
time, we ve seen occasional failures in the expect, pip, and
template_jinja2_non_native tests that usually go away before anyone has a
chance to look into them properly. Colin found that these were blocking an
openssh upgrade and so decided to track them down.
It turns out that these failures happened exactly when the libpython3.13-stdlib
package had different versions in testing and unstable. A setup script removed
/usr/lib/python3*/EXTERNALLY-MANAGED in order that pip can install system
packages for some of the tests, but if a package shipping that file were ever
upgraded then that customization would be undone, and the same setup script
removed apt pins in a way that caused problems when autopkgtest was invoked
in certain ways. In combination with this, one of the integration tests
attempted to disable system apt sources while testing the behaviour of the
ansible.builtin.apt module, but it failed to do so comprehensively enough and
so that integration test accidentally upgraded the testbed from testing to
unstable in the middle of the test. Chaos ensued.
Colin fixed this in Debian
and contributed the relevant part upstream.
gnome-shell-extension-hamster for GNOME 49. It is a GNOME
Shell integration for the Hamster time tracker.debvm dropping support for running on bookworm . There
are two trixie features improving the operation. mkfs.ext4 can now consume a
tar archive to populate the filesystem via libarchive and dash now supports
set -o pipefail. Beyond this change in operation, a number of robustness and
quality issues have been resolved.brlaser and ifhp. Moreover he uploaded a new upstream version of
cups.xorg-server to the latest security release and helped with
various transitions.reverse-depends to Python 3 (yes, it had been running on 2.7) and migrated it
to git from bzr.beautifulsoup4,
mkdocs-macros-plugin, python-pipx.pypy3, fixing an AST Compiler Assertion error.distro-info-data, including data for two
additional Debian derivatives: eLxr and Devuan.dh-python, the python packaging tool, merging
several contributed patches and resolving some bugs.
xev to gather lowlevel input from the mouse. To my surprise xev showed that this thumb button press was actually sending Control and Alt keypress events:
$ xev
KeyPress event, serial 52, synthetic NO, window 0x2c00001,
root 0x413, subw 0x0, time 3233018, (58,87), root:(648,579),
state 0x10, keycode 37 (keysym 0xffe9, Alt_L), same_screen YES,
XLookupString gives 0 bytes:
XmbLookupString gives 0 bytes:
XFilterEvent returns: False
KeyPress event, serial 52, synthetic NO, window 0x2c00001,
root 0x413, subw 0x0, time 3233025, (58,87), root:(648,579),
state 0x18, keycode 64 (keysym 0xffe3, Control_L), same_screen YES,
XLookupString gives 0 bytes:
XmbLookupString gives 0 bytes:
XFilterEvent returns: False
xinput:
$ xinput --list
Virtual core pointer id=2 [master pointer (3)]
...
Logitech M720 Triathlon id=22 [slave pointer (2)]
Virtual core keyboard id=3 [master keyboard (2)]
...
Logitech M720 Triathlon id=23 [slave keyboard (3)]
xinput --disable-device with id 23 disabled the problematic behaviour, but I was wondering how to put that in X11 startup script, and if this Ctrl and Alt combination was not simply triggering a window manager keyboard shortcut that I could disable.
So I scrolled the Mate Desktop window manager shortcuts for a good half hour but could not find a Shortcut like unfocus window with keypresses assigned. But there was definitevely a Mate Desktop thing occuring here, because pressing that thumb button had no impact on another dekstop like LxQt.
Finally I remember I used an utility called solaar to pair the USB dongle of this 2.4Ghz wireless mouse. I could maybe use it to inspect the mouse profile.
Then bingo !
$ solaar show 'M720 Triathlon' grep --after 1 12:
12: PERSISTENT REMAPPABLE ACTION 1C00 V0
Mappage touche/bouton persistant : Left Button:Mouse Button Left, Right Button:Mouse Button Right, Middle Button:Mouse Button Middle, Back Button:Mouse Button Back, Forward Button:Mouse Button Forward, Left Tilt:Horizontal Scroll Left, Right Tilt:Horizontal Scroll Right, MultiPlatform Gesture Button:Alt+Cntrl+TAB
If you read the above title, you might wonder how the switch to wayland (yes, the graphical stack replacing the venerable X11) can possibly relate to SSH agents. The answer is easy.
For as long as I can remember, as a long time user of gpg-agent as SSH agent (because my SSH key is a GPG sub-key) I relied on /etc/X11/Xsession.d/90gpg-agent that would configure the SSH_AUTH_SOCK environment variable (pointing to gpg-agent s socket) provided that I added enable-ssh-support in ~/.gnupg/gpg-agent.conf.
Now when I switched to Wayland, that shell script used in the startup sequence of Xorg was no longer used. During a while I cheated a bit by setting SSH_AUTH_SOCK directly in my ~/.bashrc. But that only works for terminals, and not for other applications that are started by the session manager (which is basically systemd --user).
So how is that supposed to work out of the box nowadays? The SSH agents (as packaged in Debian) have all adopted the same trick, their .socket unit have an ExecStartPost setting which runs systemctl --user set-environment SSH_AUTH_SOCK=some-value. This command dynamically modifies the environment of the running systemd daemon and thus influences the environment for the future units started. Putting this in a socket unit ensures an early run, before most of the applications are started so it s a good choice. They tend to also explicitly ensure this with a directive like Before=graphical-session-pre.target.
However, in a typical installation you end up with multiple SSH agents (right now I have ssh-agent, gpg-agent, and gcr-ssh-agent), which one is the one that the user ends up using? Well, that is not clearly defined, the one that wins is the one that runs last because each of them overwrites the value in the systemd environment.
Some of them fight to have that place (cf #1079246 for gcr-ssh-agent) by setting explicit After directives. In the above bug I argue that we should let gpg-agent.socket have the priority since that s the only one that is not enabled by default and that requires the user to opt-in. However, ultimately there will always be cases where you will want to be explicit about the SSH agent that should win.
You could rely on systemd overrides to add/remove ordering directives but that s pretty fragile. Instead the right way to deal with this is to mask the socket units of the SSH agents that you don t want. Note that disabling (i.e. systemctl --user disable) either will not work[1] or will not be sufficient[2]. In my case, I wanted to keep gpg-agent.socket so I masked gcr-ssh-agent.socket and ssh-agent.socket:
$ systemctl --user mask ssh-agent.socket gcr-ssh-agent.socket
Created symlink '/home/rhertzog/.config/systemd/user/ssh-agent.socket' '/dev/null'.
Created symlink '/home/rhertzog/.config/systemd/user/gcr-ssh-agent.socket' '/dev/null'.sudo systemctl --global mask ssh-agent.socket gcr-ssh-agent.socket. Now on next login, you will only get a single ssh agent socket unit that runs and the SSH_AUTH_SOCK value will thus be predictable again!
Hopefully you will find that useful as it s already the second time that I stumble upon this either for me or for a relative. Next time, I will know where to look it up. 
[1]: If you try to run systemctl --user disable gcr-ssh-agent.socket, you will get a message saying that it will not work because the unit is enabled for all users at the global level. You can do it with --global instead of --user but it doesn t help, cf below.
[2]: Disabling an unit basically means stopping to explicitely schedule its startup as part of a desired target. However, the unit can still be started as a dependency of other units and that s the case here because a socket unit will typically be pulled in by its corresponding service unit.
About 95% of my Debian contributions this month were
sponsored by Freexian.
You can also support my work directly via
Liberapay or GitHub
Sponsors.
OpenSSH
OpenSSH upstream released
10.1p1 this month, so I
upgraded to that. In the process, I reverted a Debian patch that changed IP
quality-of-service defaults, which made sense at the
time but has since been reworked upstream
anyway, so it makes sense to find out whether we still have similar
problems. So far I haven t heard anything bad in this area.
10.1p1 caused a regression in the ssh-agent-filter package s tests, which I
bisected and chased up with
upstream.
10.1p1 also had a few other user-visible regressions
(#1117574,
#1117594,
#1117638,
#1117720); I upgraded to
10.2p1 which fixed some of
these, and contributed some upstream debugging
help to clear up the
rest. While I was there, I also fixed ssh-session-cleanup: fails due to
wrong $ssh_session_pattern in our packaging.
Finally, I got all this into trixie-backports, which I intend to keep up to
date throughout the forky development cycle.
Python packaging
For some time, ansible-core has had occasional autopkgtest failures that
usually go away before anyone has a chance to look into them properly. I
ran into these via openssh recently and decided to track them down. It
turns out that they only happened when the libpython3.13-stdlib package
had different versions in testing and unstable, because an integration test
setup script made a change that would be reverted if that package was ever
upgraded in the testbed, and one of the integration tests accidentally
failed to disable system apt sources comprehensively enough while testing
the behaviour of the ansible.builtin.apt module. I fixed this in
Debian
and contributed the relevant part
upstream.
We ve started working on enabling Python 3.14 as a supported version in
Debian. I fixed or helped to fix a number of packages for this:
nocheck build
profile, and I fixed several of
these (generally just a matter of adjusting build-dependencies):
/usr/bin/env: 'python': No such file or
directoryWelcome to the October 2025 report from the Reproducible Builds project!
Welcome to the very latest report from the Reproducible Builds project. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
In this report:
We were thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. During this event, participants had the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim was to create an inclusive space that fosters collaboration, innovation and problem-solving.
The agenda of the three main days is available online however, some working sessions may still lack notes at time of publication.
One tangible outcome of the summit is that Johannes Starosta finished their rebuilderd tutorial, which is now available online and Johannes is actively seeking feedback.
On the issue tracker for the popular Signal messenger app, developer Greyson Parrelli reports that updates to the Google Play store have, in effect, broken reproducible builds:
The most recent issues have to do with changes to the APKs that are made by the Play Store. Specifically, they add some attributes to some .xml files around languages are resources, which is not unexpected because of how the whole bundle system works. This is trickier to resolve, because unlike current expected differences (like signing information), we can t just exclude a whole file from the comparison. We have to take a more nuanced look at the diff. I ve been hesitant to do that because it ll complicate our currently-very-readable comparison script, but I don t think there s any other reasonable option here.
The full thread with additional context is available on GitHub.
malloc are used to determine some order of execution.
.buildinfo files).
xfsprogs 6.17.0 which specifically includes a commit that implements the functionality to populate a newly created XFS filesystem directly from an existing directory structure which makes it easier to create populated filesystems
without having to mount them [and thus is] particularly useful for reproducible builds . Luca asked the list how they might contribute to the docs of the System images page.
Popular YouTuber @laurewired published a video this month with an engaging take on the Trusting Trust problem. Titled The Original Sin of Computing that no one can fix, the video touches on David A. Wheeler s Diverse Double-Compiling dissertation.
GNU developer Janneke Nieuwenhuizen followed-up with an email (additionally sent to our mailing list) as well, underscoring that GNU Mes s current solution [to this issue] uses ancient softwares in its bootstrap path, such as gcc-2.95.3 and glibc-2.2.5 . (According to Colby Russell, the GNU Mes bootstrapping sequence is shown at 18m54s in the video.)
Holger Levsen gave a talk at this year s Transparency.dev summit in Gothenburg, Sweden, outlining the achievements of the Reproducible Builds project in the last 12 years, covering both upstream developments as well as some distribution-specific details. As mentioned on the talk s page, Holger s presentation concluded with an outlook into the future and an invitation to collaborate to bring transparency logs into Reproducible Builds projects .
The slides of the talk are available, although a video has yet to be released. Nevertheless, as a result of the discussions at Transparency.dev there is a new page on the Debian wiki with the aim of describing a potential transparency log setup for Debian.
Andrew Ayer has setup a new service at sourcespotter.com that aims to monitor the supply chain security for Go releases. It consists of four separate trackers:
go command.
Julien Malka of the Institut Polytechnique de Paris published an exciting paper this month on How NixOS could have detected the XZ supply-chain attack for the benefit of all thanks to reproducible-builds. Julien outlines his paper as follows:
In March 2024, a sophisticated backdoor was discovered in xz, a core compression library in Linux distributions, covertly inserted over three years by a malicious maintainer, Jia Tan. The attack, which enabled remote code execution via ssh, was only uncovered by chance when Andres Freund investigated a minor performance issue. This incident highlights the vulnerability of the open-source supply chain and the effort attackers are willing to invest in gaining trust and access. In this article, I analyze the backdoor s mechanics and explore how bitwise build reproducibility could have helped detect it.A PDF of the paper is available online.
Iy n M ndez Veiga and Esther H nggi (of the Lucerne University of Applied Sciences and Arts and ETH Zurich) published a paper this month on the topic of Reproducible Builds for Quantum Computing. The abstract of their paper mentions the following:
Although quantum computing is a rapidly evolving field of research, it can already benefit from adopting reproducible builds. This paper aims to bridge the gap between the quantum computing and reproducible builds communities. We propose a generalization of the definition of reproducible builds in the quantum setting, motivated by two threat models: one targeting the confidentiality of end users data during circuit preparation and submission to a quantum computer, and another compromising the integrity of quantum computation results. This work presents three examples that show how classical information can be hidden in transpiled quantum circuits, and two cases illustrating how even minimal modifications to these circuits can lead to incorrect quantum computation results.A full PDF of their paper is available.
Congratulations to Georg Kofler who submitted their Master s thesis for the Johannes Kepler University of Linz, Austria on the topic of Reproducible builds of E2EE-messengers for Android using Nix hermetic builds:
The thesis focuses on providing a reproducible build process for two open-source E2EE messaging applications: Signal and Wire. The motivation to ensure reproducibility and thereby the integrity of E2EE messaging applications stems from their central role as essential tools for modern digital privacy. These applications provide confidentiality for private and sensitive communications, and their compromise could undermine encryption mechanisms, potentially leaking sensitive data to third parties.A full PDF of their thesis is available online.
Shawkot Hossain of Aalto University, Finland has also submitted their Master s thesis on the The Role of SBOM in Modern Development with a focus on the extant tooling:
Currently, there are numerous solutions and techniques available in the market to tackle supply chain security, and all claim to be the best solution. This thesis delves deeper by implementing those solutions and evaluates them for better understanding. Some of the tools that this thesis implemented are Syft, Trivy, Grype, FOSSA, dependency-check, and Gemnasium. Software dependencies are generated in a Software Bill of Materials (SBOM) format by using these open-source tools, and the corresponding results have been analyzed. Among these tools, Syft and Trivy outperform others as they provide relevant and accurate information on software dependencies.A PDF of the thesis is also available.
Michael Plura published an interesting article on Heise.de on the topic of Trust is good, reproducibility is better:
In the wake of growing supply chain attacks, the FreeBSD developers are relying on a transparent build concept in the form of Zero-Trust Builds. The approach builds on the established Reproducible Builds, where binary files can be rebuilt bit-for-bit from the published source code. While reproducible builds primarily ensure verifiability, the zero-trust model goes a step further and removes trust from the build process itself. No single server, maintainer, or compiler can be considered more than potentially trustworthy.The article mentions that this goal has now been achieved with a slight delay and can be used in the current development branch for FreeBSD 15 .
In Debian this month, 7 reviews of Debian packages were added, 5 were updated and 11 were removed this month adding to our knowledge about identified issues.
For the Debian CI tests Holger fixed #786644 and set nocheck in DEB_BUILD_OPTIONS for the 2nd build..
python-can.rsbackup.mobilitydb.pyraf.ne.qt6-lottie, plasma6-print-manager, plasma6-nm (avoid race in qmlcachegen)xfishtank (date, regression)gstreamer-plugins-rsgpg2 (FTBFS-2038)rocclr (PID)kf6-breeze-icons (parallelism)opencloud-server (random tmp path)python-awscrt (FTBFS-j1)glib-macros/contrast/fractal/Fragments/identity/mousai/loupe/gstreamer-plugins-rs (rust HashMap)deno (rust order)
Once again, there were a number of improvements made to our website this month including:
git archive to the Archive metadata page. [ ]
307 was uploaded to Debian unstable by Chris Lamb, who made a number of changes including fixing compatibility with LLVM version 21 [ ], an attempt to automatically attempt to deploy to PyPI by liaising with the PyPI developers/maintainers (with this experimental feature). [ ] In addition, Vagrant Cascadian updated diffoscope in GNU Guix to version 307.
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
drag_lock
to disabled.
I also uploaded new releases of cage (a
kiosk for Wayland), labwc, the
window-stacking Wayland compositor that is inspired by Openbox, and
wf-recorder, a tool for creating
screen recordings of wlroots-based Wayland compositors.
If I don t forget I try to update the watch file of the packages I touch to the
new version 5
format.
Simon Ser announced
vali, a C library for
Varlink. The blog post also mentions that this will be
a dependency of the next version of the kanshi Wayland output management
daemon and the PR to do so is now already merged. So I created ITP: vali A
Varlink C implementation and code
generator, packaged
the library and it is now waiting in NEW. In addition to libscfg this is now
the second dependency of kanshi that is in NEW.
On the Rust side of things I fixed a
bug in
carl. The fix introduces new date
properties which can be use to highlight a calendar date. I also updated
all the dependencies and plan to create a new release soon.
Later I dug up a Rust project that I started a couple of years ago, where I try
to use wasm-bindgen to
implement interactive web components. There is a lot I have to refactor in this
code base, but I will work on that and try to publish something in the next few
months.
<dialog>,
which made the case for using standardized HTML elements instead of resorting
to JavaScript libraries.
I finally managed to update my shell Server to Debian 13.
I created an issue for the nextcloud-news android
client because I moved
to a new phone and my starred articles did not show up in the news app, which
is a bit annoying.
I got my ticket for 39C3.
In my dayjob I continued to work on the refactoring of the import logic of our
apis-core-rdf app. I released version 0.56 which also introduced the
#snackbar as the container for the toast message, as described in the
<dialog> block post. At the end of the month I released version 0.57
of apis-core-rdf, which got rid of the remaining leftovers of the old
import logic.
A couple of interesting articles I stumbled upon (or finally had the time to read):
Welcome to post 54 in the R4 series.
The topic of continuous integration has been a recurrent theme here
at the R4 series. Post #32
introducess r-ci,
while post #41
brings r2u to r-ci, but does not show a
matrix deployment. Post #45
describes the updated r-ci setup that is now
the default and contains a macOS and Ubuntu matrix, where the latter
relies on r2u to keep
things fast, easy, reliable . Last but not least more recent post #52
shares a trick for ensuring coverage reports.
Following #45,
use of r-ci at for
example GitHub Actions has seen steady use and very reliable
performance. With the standard setup, a vanilla Ubuntu setup is changed
into one supported by r2u. This requires
downloading and installating a few Ubuntu packages, and has
generally been fairly quick on the order of around fourty
seconds. Now, the general variability of run-times for identical
tasks in GitHub Actions is well documented by the results of the
setup described in post #39
which still runs weekly. It runs the identical SQL query against a
remote backend using two different package families. And lo and behold,
the intra-method variability on unchanged code or setup and
therefore due solely to system variability is about as large as the
inter-method variability. In short, GitHub Actions performance varies
randomly with significant variability. See the repo README.md for
chart that updates weekly (and see #39
for background).
Of late, this variability became more noticeable during standard
GitHub Actions runs where it would regularly take more than two minutes
of setup time before actual continuous integration work was done. Some
caching seems to be in effect, so subsequent runs in the same
repo seem faster and often came back to one minute or less. For
lightweight and small packages, loosing two minutes to setup when the
actual test time is a minute or less gets old fast.
Looking around, we noticed that container use can be combined with
matrix use. So we have now been deploying the following setup (not
always over all the matrix elements though)
jobs:
ci:
strategy:
matrix:
include:
- name: container, os: ubuntu-latest, container: rocker/r2u4ci
- name: macos, os: macos-latest
- name: ubuntu, os: ubuntu-latest
runs-on: $ matrix.os
container: $ matrix.container NULL for
container in the two other cases, so
container: $ matrix.container is ignored there. But
when container is set as here for the ci-enhanced version
of r2u (which adds a few binaries commonly needed such as
git, curl, wget etc needed for
CI) then the CI jobs runs inside the container. And thereby skips most
of the setup time as the container is already prepared.
This also required some small adjustments in the underlying shell
script doing the work. To not disrupt standard deployment, we placed
these into a release candidate / development version one can op into
via an new variable dev_version
Everything else remains the same and works as before. But faster as
much less time is spent on setup. You can see the actual full yaml file
and actions in my repositories for rcpparmadillo and
rcppmlpack-examples.
Additional testing would be welcome, so feel free to deploy this in your
actions now. Otherwise I will likely carry this over and make it the
defaul in a few weeks time. It will still work as before but when
the added container: line is used will run much faster
thanks to rocker/r2u4ci being already set up for CI.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.
| Series: | White Space #1 |
| Publisher: | Saga Press |
| Copyright: | 2019 |
| ISBN: | 1-5344-0300-0 |
| Format: | Kindle |
| Pages: | 501 |
Eventually, I realized that I was wasting my time, and if I wanted to hide from humanity in a bottle, I was better off making it a titanium one with a warp drive and a couple of carefully selected companions.Halmey does salvage: finding ships lost in white space and retrieving them. One of her partners is Connla, a pilot originally from a somewhat atavistic world called Spartacus. The other is their salvage tug.
The boat didn't have a name. He wasn't deemed significant enough to need a name by the authorities and registries that govern such things. He had a registration number 657-2929-04, Human/Terra and he had a class, salvage tug, but he didn't have a name. Officially. We called him Singer. If Singer had an opinion on the issue, he'd never registered it but he never complained. Singer was the shipmind as well as the ship or at least, he inhabited the ship's virtual spaces the same way we inhabited the physical ones but my partner Connla and I didn't own him. You can't own a sentience in civilized space.As Ancestral Night opens, the three of them are investigating a tip of a white space anomoly well off the beaten path. They thought it might be a lost ship that failed a transition. What they find instead is a dead Ativahika and a mysterious ship equipped with artificial gravity. The Ativahikas are a presumed sentient race of living ships that are on the most alien outskirts of the Synarche confederation. They don't communicate, at least so far as Halmey is aware. She also wasn't aware they died, but this one is thoroughly dead, next to an apparently abandoned ship of unknown origin with a piece of technology beyond the capabilities of the Synarche. The three salvagers get very little time to absorb this scene before they are attacked by pirates. I have always liked Bear's science fiction better than her fantasy, and this is no exception. This was great stuff. Halmey is a talkative, opinionated infodumper, which is a great first-person protagonist to have in a fictional universe this rich with delightful corners. There are some Big Dumb Object vibes (one of my favorite parts of salvage stories), solid character work, a mysterious past that has some satisfying heft once it's revealed, and a whole lot more moral philosophy than I was expecting from the setup. All of it is woven together with experienced skill, unsurprising given Bear's long and prolific career. And it's full of delightful world-building bits: Halmey's afthands (a surgical adaptation for zero gravity work) and grumpiness at the sheer amount of gravity she has to deal with over the course of this book, the Culture-style ship names, and a faster-than-light travel system that of course won't pass physics muster but provides a satisfying quantity of hooky bits for plot to attach to. The backbone of this book is an ancient artifact mystery crossed with a murder investigation. Who killed the Ativahika? Where did the gravity generator come from? Those are good questions with interesting answers. But the heart of the book is a philosophical conflict: What are the boundaries between identity and society? How much power should society have to reshape who we are? If you deny parts of yourself to fit in with society, is this necessarily a form of oppression? I wrote a couple of paragraphs of elaboration, and then deleted them; on further thought, I don't want to give any more details about what Bear is doing in this book. I will only say that I was not expecting this level of thoughtfulness about a notoriously complex and tricky philosophical topic in a full-throated adventure science fiction novel. I think some people may find the ending strange and disappointing. I loved it, and weeks after finishing this book I'm still thinking about it. Ancestral Night has some pacing problems. There is a long stretch in the middle of the book that felt repetitive and strained, where Bear holds the reader at a high level of alert and dread for long enough that I found it enervating. There are also a few political cheap shots where Bear picks the weakest form of an opposing argument instead of the strongest. (Some of the cheap shots are rather satisfying, though.) The dramatic arc of the book is... odd, in a way that I think was entirely intentional given how well it works with the thematic message, but which is also unsettling. You may not get the catharsis that you're expecting. But all of this serves a purpose, and I thought that purpose was interesting. Ancestral Night is one of those books that I liked more a week after I finished it than I did when I finished it.
Epiphanies are wonderful. I m really grateful that our brains do so much processing outside the line of sight of our consciousnesses. Can you imagine how downright boring thinking would be if you had to go through all that stuff line by line?Also, for once, I think Bear hit on exactly the right level of description rather than leaving me trying to piece together clues and hope I understood the plot. It helps that Halmey loves to explain things, so there are a lot of miniature infodumps, but I found them interesting and a satisfying throwback to an earlier style of science fiction that focused more on world-building than on interpersonal drama. There is drama, but most of it is internal, and I thought the balance was about right. This is solid, well-crafted work and a good addition to the genre. I am looking forward to the rest of the series. Followed by Machine, which shifts to a different protagonist. Rating: 8 out of 10
The discovery of a backdoor in XZ Utils in the spring of 2024 shocked the open source community, raising critical questions about software supply chain security. This post explores whether better Debian packaging practices could have detected this threat, offering a guide to auditing packages and suggesting future improvements.
The XZ backdoor in versions 5.6.0/5.6.1 made its way briefly into many major Linux distributions such as Debian and Fedora, but luckily didn t reach that many actual users, as the backdoored releases were quickly removed thanks to the heroic diligence of Andres Freund. We are all extremely lucky that he detected a half a second performance regression in SSH, cared enough to trace it down, discovered malicious code in the XZ library loaded by SSH, and reported promtly to various security teams for quick coordinated actions.
This episode makes software engineers pondering the following questions:
$ debsnap --verbose --first 5.2.4-1 xz-utils
Getting json https://snapshot.debian.org/mr/package/xz-utils/
...
Getting dsc file xz-utils_5.2.4-1.dsc: https://snapshot.debian.org/file/a98271e4291bed8df795ce04d9dc8e4ce959462d
Getting file xz-utils_5.2.4.orig.tar.xz.asc: https://snapshot.debian.org/file/59ccbfb2405abe510999afef4b374cad30c09275
Getting file xz-utils_5.2.4-1.debian.tar.xz: https://snapshot.debian.org/file/667c14fd9409ca54c397b07d2d70140d6297393f
source-xz-utils/xz-utils_5.2.4-1.dsc:
Good signature found
validating xz-utils_5.2.4.orig.tar.xz
validating xz-utils_5.2.4.orig.tar.xz.asc
validating xz-utils_5.2.4-1.debian.tar.xz
All files validated successfully.source-<package name> with the following types of files:
*.orig.tar.xz: source code from upstream*.orig.tar.xz.asc: detached signature (if upstream signs their releases)*.debian.tar.xz: Debian packaging source, i.e. the debian/ subdirectory contents*.dsc: Debian source control file, including signature by Debian Developer/Maintainer$ ls -1 source-xz-utils/
...
xz-utils_5.6.4.orig.tar.xz
xz-utils_5.6.4.orig.tar.xz.asc
xz-utils_5.6.4-1.debian.tar.xz
xz-utils_5.6.4-1.dsc
xz-utils_5.8.0.orig.tar.xz
xz-utils_5.8.0.orig.tar.xz.asc
xz-utils_5.8.0-1.debian.tar.xz
xz-utils_5.8.0-1.dsc
xz-utils_5.8.1.orig.tar.xz
xz-utils_5.8.1.orig.tar.xz.asc
xz-utils_5.8.1-1.1.debian.tar.xz
xz-utils_5.8.1-1.1.dsc
xz-utils_5.8.1-1.debian.tar.xz
xz-utils_5.8.1-1.dsc
xz-utils_5.8.1-2.debian.tar.xz
xz-utils_5.8.1-2.dscdebsnap, it already automatically verifies that the downloaded files match the OpenPGP signatures. To have full clarity on what files were authenticated with what keys, we should verify the Debian packagers signature with:
$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc
gpg: Signature made Fri Oct 3 22:04:44 2025 UTC
gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B
gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org
gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Sebastian Andrzej Siewior" [unknown]
gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown]
gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1
Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc
gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz'
gpg: Signature made Thu Apr 3 11:38:23 2025 UTC
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620debian/watch file inside the Debian packaging, which defines where the upstream source is downloaded from. Continuing on the example situation above, we can unpack the latest Debian sources, enter and then run uscan to download:
$ tar xvf xz-utils_5.8.1-2.debian.tar.xz
...
debian/rules
debian/source/format
debian/source.lintian-overrides
debian/symbols
debian/tests/control
debian/tests/testsuite
debian/upstream/signing-key.asc
debian/watch
...
$ uscan --download-current-version --destdir /tmp
Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1
gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC
gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>"
Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz./tmp along with the files renamed to follow Debian conventions. Using everything downloaded so far the sha256 checksums can be compared across the files and also to what the .dsc file advertised:
$ ls -1 /tmp/
xz-5.8.1.tar.xz
xz-5.8.1.tar.xz.sig
xz-utils_5.8.1.orig.tar.xz
xz-utils_5.8.1.orig.tar.xz.asc
$ sha256sum xz-utils_5.8.1.orig.tar.xz /tmp/xz-5.8.1.tar.xz
0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e xz-utils_5.8.1.orig.tar.xz
0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e /tmp/xz-5.8.1.tar.xz
$ grep -A 3 Sha256 xz-utils_5.8.1-2.dsc
Checksums-Sha256:
0b54f79df85912504de0b14aec7971e3f964491af1812d83447005807513cd9e 1461872 xz-utils_5.8.1.orig.tar.xz
4138f4ceca1aa7fd2085fb15a23f6d495d27bca6d3c49c429a8520ea622c27ae 833 xz-utils_5.8.1.orig.tar.xz.asc
3ed458da17e4023ec45b2c398480ed4fe6a7bfc1d108675ec837b5ca9a4b5ccb 24648 xz-utils_5.8.1-2.debian.tar.xz0b54f79df85... is the same across the files, so it is a match.
uscan may in rare cases repackage some upstream sources, for example to exclude files that don t adhere to Debian s copyright and licensing requirements. Those files and paths would be listed under the Files-Excluded section in the debian/copyright file. There are also other situations where the file that represents the upstream sources in Debian isn t bit-by-bit the same as what upstream published. If checksums don t match, an experienced Debian Developer should review all package settings (e.g. debian/source/options) to see if there was a valid and intentional reason for divergence.
diffoscope --html-dir xz-utils-5.6.4_vs_5.8.0 xz-utils_5.6.4.orig.tar.xz xz-utils_5.8.0.orig.tar.xz
browse xz-utils-5.6.4_vs_5.8.0/index.html
If the changes are extensive, and you want to use a LLM to help spot potential security issues, generate the report of both the upstream and Debian packaging differences in Markdown with:
diffoscope --markdown diffoscope-debian.md xz-utils_5.6.4-1.debian.tar.xz xz-utils_5.8.1-2.debian.tar.xz
diffoscope --markdown diffoscope.md xz-utils_5.6.4.orig.tar.xz xz-utils_5.8.0.orig.tar.xzBased on the attached diffoscope output for a new Debian package version compared with the previous one, list all suspicious changes that might have introduced a backdoor, followed by other potential security issues. If there are none, list a short summary of changes as the conclusion.
Git-Vcs header in the debian/control file. For xz-utils the location is salsa.debian.org/debian/xz-utils.
Note that the Debian policy does not state anything about how Salsa should be used, or what git repository layout or development practices to follow. In practice most packages follow the DEP-14 proposal, and use git-buildpackage as the tool for managing changes and pushing and pulling them between upstream and salsa.debian.org.
To get the XZ Utils source, run:
$ gbp clone https://salsa.debian.org/debian/xz-utils.git
gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'$ git log --graph --oneline
* bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2
* 4b769547 d: Remove the symlinks from -dev package.
* a39f3428 Correct the nocheck build profile
* 1b806b8d Import Debian changes 5.8.1-1.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h
* 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder
* 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding
* 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread
* d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
* c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state
* 831b55b9 liblzma: mt dec: Fix a comment
* b9d168ee liblzma: Add assertions to lzma_bufcpy()
* c8e0a489 DOS: Update Makefile to fix the build
* 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers
* 7ce38b31 Update THANKS
* 688e51bd Translations: Update the Croatian translation
* a6b54dde Prepare 5.8.0-1.
* 77d9470f Add 5.8 symbols.
* 9268eb66 Import 5.8.0
* 6f85ef4f Update upstream source from tag 'upstream/5.8.0'
\ \
* afba662b New upstream version 5.8.0
/
* 173fb5c6 doc/SHA256SUMS: Add 5.8.0
* db9258e8 Bump version and soname for 5.8.0
* bfb752a3 Add NEWS for 5.8.0
* 6ccbb904 Translations: Run "make -C po update-po"
* 891a5f05 Translations: Run po4a/update-po
* 4f52e738 Translations: Partially fix overtranslation in Serbian man pages
* ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage
* 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()debian/unstable branch as well as the intermediate upstream import branch, and the actual real upstream development branch. See my Debian source packages in git explainer for details of what these branches are used for.
To only view changes on the Debian branch, run git log --graph --oneline --first-parent or git log --graph --oneline -- debian.
The Debian branch should only have changes inside the debian/ subdirectory, which is easy to check with:
$ git diff --stat upstream/v5.8
debian/README.source 16 +++
debian/autogen.sh 32 +++++
debian/changelog 949 ++++++++++++++++++++++++++
...
debian/upstream/signing-key.asc 52 +++++++++
debian/watch 4 +
debian/xz-utils.README.Debian 47 ++++++++
debian/xz-utils.docs 6 +
debian/xz-utils.install 28 +++++
debian/xz-utils.postinst 19 +++
debian/xz-utils.prerm 10 ++
debian/xzdec.docs 6 +
debian/xzdec.install 4 +
33 files changed, 2014 insertions(+)debian/ directory originate from upstream, and for example running git blame on them should show only upstream commits:
$ git blame CMakeLists.txt
22af94128 (Lasse Collin 2024-02-12 17:09:10 +0200 1) # SPDX-License-Identifier: 0BSD
22af94128 (Lasse Collin 2024-02-12 17:09:10 +0200 2)
7e3493d40 (Lasse Collin 2020-02-24 23:38:16 +0200 3) ###############
7e3493d40 (Lasse Collin 2020-02-24 23:38:16 +0200 4) #
426bdc709 (Lasse Collin 2024-02-17 21:45:07 +0200 5) # CMake support for building XZ Utils$ git verify-tag v5.6.2
gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: issuer "lasse.collin@tukaani.org"
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired]
gpg: Note: This key has expired!git log --reverse -p afba662b...fa1e8796. However, a far superior way to review changes would be to browse this range using a visual git history viewer, such as gitk. Either way, looking at one code change at a time and reading the git commit message makes the review much easier.
source-xz-utils/xz-utils_5.8.1-2.debian) and the git repository cloned in the previous section (path xz-utils):
$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/
diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog
--- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700
+++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700
@@ -5,7 +5,7 @@
* Remove the symlinks from -dev, pointing to the lib package.
(Closes: #1109354)
- -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200
diff revealed that the timestamp in the changelog in the version uploaded to Debian is different from what was committed to git. This is not malicious, just a mistake by the maintainer who probably didn t run gbp tag immediately after upload, but instead some dch command and ended up with having a different timestamps in the git compared to what was actually uploaded to Debian.
$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc
Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'gbp import-dsc:
$ git log --graph --oneline
* 86aed07b (HEAD -> debian/unstable, tag: debian/5.8.1-2, origin/debian/unstable) Import Debian changes 5.8.1-2
* f111d93b (tag: debian/5.8.1-1.1) Import Debian changes 5.8.1-1.1
* 1106e19b (tag: debian/5.8.1-1) Import Debian changes 5.8.1-1
\
* 08edbe38 (tag: upstream/5.8.1, origin/upstream/v5.8, upstream/v5.8) Import Upstream version 5.8.1
\
* a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.hgbp import-dsc can be viewed at salsa.debian.org/otto/xz-utils-2025-09-29/-/network/debian%2Funstable
An example repository that was fully crafted using gbp import-dscs can be viewed at salsa.debian.org/otto/xz-utils-gbp-import-dscs-debsnap-generated/-/network/debian%2Flatest.
There exists also dgit, which in a similar way creates a synthetic git history to allow viewing the Debian archive contents via git tools. However, its focus is on producing new package versions, so fetching a package with dgit that has not had the history recorded in dgit earlier will only show the latest versions:
$ dgit clone xz-utils
canonical suite name for unstable is sid
starting new git history
last upload to archive: NO git hash
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz...
dpkg-source: info: extracting xz-utils in unpacked
dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz
dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz
synthesised git commit from .dsc 5.8.1-2
HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium
dgit ok: ready for work in xz-utils
$ dgit/sid git log --graph --oneline
* f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid)
\
* 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago
* 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1git import-orig --uscan for Debian packages with the correct settings in debian/gbp.conf. By viewing this commit one can see exactly how the upstream release tarball differs from the upstream git contents (if at all).
In the case of XZ, the difference is substantial, and shown below in full as it is very interesting:
$ git show --stat fa1e8796
commit fa1e8796dabd91a0f667b9e90f9841825225413a
(debian/upstream/v5.8, upstream/v5.8)
Author: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Date: Thu Apr 3 22:58:39 2025 +0200
New upstream version 5.8.1
.codespellrc 30 -
.gitattributes 8 -
.github/workflows/ci.yml 163 -
.github/workflows/freebsd.yml 32 -
.github/workflows/netbsd.yml 32 -
.github/workflows/openbsd.yml 35 -
.github/workflows/solaris.yml 32 -
.github/workflows/windows-ci.yml 124 -
.gitignore 113 -
ABOUT-NLS 1 +
ChangeLog 17392 +++++++++++++++++++++
Makefile.in 1097 +++++++
aclocal.m4 1353 ++++++++
build-aux/ci_build.bash 286 --
build-aux/compile 351 ++
build-aux/config.guess 1815 ++++++++++
build-aux/config.rpath 751 +++++
build-aux/config.sub 2354 +++++++++++++
build-aux/depcomp 792 +++++
build-aux/install-sh 541 +++
build-aux/ltmain.sh 11524 ++++++++++++++++++++++
build-aux/missing 236 ++
build-aux/test-driver 160 +
config.h.in 634 ++++
configure 26434 ++++++++++++++++++++++
debug/Makefile.in 756 +++++
doc/SHA256SUMS 236 --
doc/man/txt/lzmainfo.txt 36 +
doc/man/txt/xz.txt 1708 ++++++++++
doc/man/txt/xzdec.txt 76 +
doc/man/txt/xzdiff.txt 39 +
doc/man/txt/xzgrep.txt 70 +
doc/man/txt/xzless.txt 36 +
doc/man/txt/xzmore.txt 31 +
lib/Makefile.in 623 ++++
m4/.gitignore 40 -
m4/build-to-host.m4 274 ++
m4/gettext.m4 392 +++
m4/host-cpu-c-abi.m4 529 +++
m4/iconv.m4 324 ++
m4/intlmacosx.m4 71 +
m4/lib-ld.m4 170 +
m4/lib-link.m4 815 +++++
m4/lib-prefix.m4 334 ++
m4/libtool.m4 8488 +++++++++++++++++++++
m4/ltoptions.m4 467 +++
m4/ltsugar.m4 124 +
m4/ltversion.m4 24 +
m4/lt~obsolete.m4 99 +
m4/nls.m4 33 +
m4/po.m4 456 +++
m4/progtest.m4 92 +
po/.gitignore 31 -
po/Makefile.in.in 517 +++
po/Rules-quot 66 +
po/boldquot.sed 21 +
po/ca.gmo Bin 0 -> 15587 bytes
po/cs.gmo Bin 0 -> 7983 bytes
po/da.gmo Bin 0 -> 9040 bytes
po/de.gmo Bin 0 -> 29882 bytes
po/en@boldquot.header 35 +
po/en@quot.header 32 +
po/eo.gmo Bin 0 -> 15060 bytes
po/es.gmo Bin 0 -> 29228 bytes
po/fi.gmo Bin 0 -> 28225 bytes
po/fr.gmo Bin 0 -> 10232 bytesgit difftool --dir-diff fa1e8796^..fa1e8796.
To compare changes across the new and old upstream tarball, one would need to compare commits afba662b New upstream version 5.8.0 and fa1e8796 New upstream version 5.8.1 by running git difftool --dir-diff afba662b..fa1e8796.
With all the above tips you can now go and try to audit your own favorite package in Debian and see if it is identical with upstream, and if not, how it differs.
tests/files/bad-3-corrupt_lzma2.xz, tests/files/good-large_compressed.lzma), and a small modification in the build scripts (m4/build-to-host.m4) to extract the backdoor and plant it into the built binary. The build script was not tracked in version control, but generated with GNU Autotools at release time and only shipped as additional files in the release tarball.
The entire reason for me to write this post was to ponder if a diligent engineer using git-buildpackage best practices could have reasonably spotted this while importing the new upstream release into Debian. The short answer is no . The malicious actor here clearly anticipated all the typical ways anyone might inspect both git commits, and release tarball contents, and masqueraded the changes very well and over a long timespan.
First of all, XZ has for legitimate reasons for several carefully crafted .xz files as test data to help catch regressions in the decompression code path. The test files are shipped in the release so users can run the test suite and validate that the binary is built correctly and xz works properly. Debian famously runs massive amounts of testing in its CI and autopkgtest system across tens of thousands of packages to uphold high quality despite frequent upgrades of the build toolchain and while supporting more CPU architectures than any other distro. Test data is useful and should stay.
When git-buildpackage is used correctly, the upstream commits are visible in the Debian packaging for easy review, but the commit cf44e4b that introduced the test files does not deviate enough from regular sloppy coding practices to really stand out. It is unfortunately very common for git commit to lack a message body explaining why the change was done, and to not be properly atomic with test code and test data together in the same commit, and for commits to be pushed directly to mainline without using code reviews (the commit was not part of any PR in this case). Only another upstream developer could have spotted that this change is not on par to what the project expects, and that the test code was never added, only test data, and thus that this commit was not just a sloppy one but potentially malicious.
Secondly, the fact that a new Autotools file appeared (m4/build-to-host.m4) in the XZ Utils 5.6.0 is not suspicious. This is perfectly normal for Autotools. In fact, starting from XZ Utils version 5.8.1 it is now shipping a m4/build-to-host.m4 file that it actually uses now.
Spotting that there is anything fishy is practically impossible by simply reading the code, as Autotools files are full custom m4 syntax interwoven with shell script, and there are plenty of backticks ( ) that spawn subshells and evals that execute variable contents further, which is just normal for Autotools. Russ Cox s XZ post explains how exactly the Autotools code fetched the actual backdoor from the test files and injected it into the build.
There is only one tiny thing that maybe a very experienced Autotools user could potentially have noticed: the serial 30 in the version header is way too high. In theory one could also have noticed this Autotools file deviates from what other packages in Debian ship with the same filename, such as e.g. the serial 3, serial 5a or 5b versions. That would however require and an insane amount extra checking work, and is not something we should plan to start doing. A much simpler solution would be to simply strongly recommend all open source projects to stop using Autotools to eventually get rid of it entirely.
debian/gbp.conf, and thus their package development histories are not always correctly constructed or uniform and easy to compare. The Debian Policy does not mandate git usage at all, and there are many important packages that are not using git at all. Additionally the Debian Policy also allows for non-maintainers to upload new versions to Debian without committing anything in git even for packages where the original maintainer wanted to use git. Uploads that bypass git unfortunately happen surpisingly often.
Because of the situation, I am afraid that we could have multiple similar backdoors lurking that simply haven t been detected yet. More audits, that hopefully also get published openly, would be welcome! More people auditing the contents of the Debian archives would probably also help surface what tools and policies Debian might be missing to make the work easier, and thus help improve the security of Debian s users, and improve trust in Debian.
debian/watch or other configs are incorrect and even cases where maintainers have manually created upstream tarballs as it was easier than configuring automation to work. It is obvious that for those packages the source tarball now in Debian is not at all the same as upstream. I am not aware of any malicious cases though (if I was, I would report them of course).
I am also aware of packages in the Debian repository that are misconfigured to be of type 1.0 (native) packages, mixing the upstream files and debian/ contents and having patches applied, while they actually should be configured as 3.0 (quilt), and not hide what is the true upstream sources. Debian should extend the QA tools to scan for such things. If I find a sponsor, I might build it myself as my next major contribution to Debian.
In addition to better tooling for finding mismatches in the source code, Debian could also have better tooling for tracking in built binaries what their source files were, but solutions like Fraunhofer-AISEC s supply-graph or Sony s ESSTRA are not practical yet. Julien Malka s post about NixOS discusses the role of reproducible builds, which may help in some cases across all distros.
[2025-10-03 12:37:17,525 - pika.adapters.utils.connection_workflow - ERROR] AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: timeout("TCP connection attempt timed out: ''/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP>', 5672))")
[2025-10-03 12:37:17,526 - pika.adapters.utils.connection_workflow - ERROR] AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: timeout("TCP connection attempt timed out: ''/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP>', 5672))"); first exception - None.
[2025-10-03 12:37:17,526 - pika.adapters.utils.connection_workflow - ERROR] AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: timeout("TCP connection attempt timed out: ''/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP>', 5672))"); first exception - None$ docker run \
--name envoy --rm \
--network host \
-v $(pwd)/envoy.yaml:/etc/envoy/envoy.yaml \
envoyproxy/envoy:v1.33-latestenvoy.yaml configurations: one that uses Envoy's TCP proxy,
and another that uses Envoy's HTTP connection manager.
Here's the simplest Envoy TCP proxy setup: a listener on port 8000
forwarding traffic to a backend running on port 8080.
static_resources:
listeners:
- name: go_server_listener
address:
socket_address:
address: 0.0.0.0
port_value: 8000
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: go_server_tcp
cluster: go_server_cluster
clusters:
- name: go_server_cluster
connect_timeout: 1s
type: static
load_assignment:
cluster_name: go_server_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080package main
import (
"fmt"
"net/http"
"time"
)
func helloHandler(w http.ResponseWriter, r *http.Request)
w.Write([]byte("Hello from Go!"))
func main()
http.HandleFunc("/", helloHandler)
server := http.Server
Addr: ":8080",
IdleTimeout: 3 * time.Second,
fmt.Println("Starting server on :8080")
panic(server.ListenAndServe())
oha is the perfect tool to generate the HTTP
requests for this test. The Load test is not meant to stress this setup,
the idea is to wait long enough so that some requests are closed. The
burst-delay feature will help with that:
$ oha -z 30s -w --burst-delay 3s --burst-rate 100 http://localhost:8000-w option to wait
for ongoing requests when the duration is reached. The result looks like
this:
We had 886 responses with status code 200 and 64 connections closed.
The backend terminated 64 connections while the load balancer still had
active requests directed to it.
Let's change the Load Balancer idle_timeout to two
seconds.
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: go_server_tcp
cluster: go_server_cluster
idle_timeout: 2s # <--- NEW LINE
Great! Now all the requests worked.
This is a common issue, not specific to Envoy Proxy or the setup
shown earlier. Major cloud providers have all documented it.
AWS
troubleshoot guide for Application Load Balancers says this:
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.
Google
troubleshoot guide for Application Load Balancers mention this as
well:
Verify that the keepalive configuration parameter for the HTTP server software running on the backend instance is not less than the keepalive timeout of the load balancer, whose value is fixed at 10 minutes (600 seconds) and is not configurable.
The load balancer generates an HTTP 5XX response code when the connection to the backend has unexpectedly closed while sending the HTTP request or before the complete HTTP response has been received. This can happen because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed keepalive timeout of the load balancer. Ensure that the keepalive timeout configuration for HTTP server software on each backend is set to slightly greater than 10 minutes (the recommended value is 620 seconds).
RabbitMQ
docs also warn about this:
Certain networking tools (HAproxy, AWS ELB) and equipment (hardware load balancers) may terminate "idle" TCP connections when there is no activity on them for a certain period of time. Most of the time it is not desirable.
Most of them are talking about Application Load Balancers and the
test I did was using a Network Load Balancer. For the sake of
completeness, I will do the same test but using Envoy's HTTP connection
manager.
The updated envoy.yaml:
static_resources:
listeners:
- name: listener
address:
socket_address:
address: 0.0.0.0
port_value: 8000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: go_server_http
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: http_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: go_server_cluster
clusters:
- name: go_server_cluster
type: STATIC
load_assignment:
cluster_name: go_server_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 0.0.0.0
port_value: 8080
Even thought the success rate is 100%, the status code distribution
show some responses with status code 503. This is the case where is not
that obvious that the problem is related to idle timeout.
However, it's clear when we look the Envoy access logs:
[2025-10-10T13:32:26.617Z] "GET / HTTP/1.1" 503 UC 0 95 0 - "-" "oha/1.10.0" "9b1cb963-449b-41d7-b614-f851ced92c3b" "localhost:8000" "0.0.0.0:8080"UC is the short name for
UpstreamConnectionTermination. This means the upstream,
which is the golang server, terminated the connection.
To fix this once again, the Load Balancer idle timeout needs to
change:
clusters:
- name: go_server_cluster
type: STATIC
typed_extension_protocol_options: # <--- NEW BLOCK
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
common_http_protocol_options:
idle_timeout: 2s # <--- NEW VALUE
explicit_http_config:
http_protocol_options: 
Key Takeaways:
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1273 other packages on CRAN, downloaded 41.8 million
times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 651 times according
to Google Scholar.
Armadillo 15 brought changes. We mentioned these in the 15.0.2-1
and 15.0.2-1
release blog posts:
#definearma::is_finite to std::isfinite) change. I
did not encounter anything much more complicated yet.
The number of affected packages approximated by looking at all
packages with a reverse dependency on RcppArmadillo and having
a deadline can be computed as
suppressMessages(library(data.table))
D <- setDT(tools::CRAN_package_db())
P <- data.table(Package=tools::package_dependencies("RcppArmadillo", reverse=TRUE, db=D)[[1]])
W <- merge(P, D, all.x=TRUE)[is.na(Deadline)==FALSE,c(1:2,38,68)]
W
W[, nrevdep := length(tools::package_dependencies(Package, reverse=TRUE, recursive=TRUE, db=D)[[1]]), by=Package]
W[order(-nrevdep)]This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
md_in_html extension and use the
markdown attribute for it to kick in on the <details> html tag.
Add the extension to your mkdocs.yaml:
markdown_extensions:
- md_in_html
<details markdown>
<summary>Long Table</summary>
Foo Bar
- -
Fizz Buzz
</details>
clang. In such situations, the best first step is often to
replicate the issue. As I pointed out on the list, the LLVM team behind
clang maintains an apt repo
at apt.llvm.org/ making it a good resource to add to Debian-based
container such as Rocker r-base or the offical r-base (the two
are in fact interchangeable, and I take care of both).
A small pothole, however, is that the documentation at the top of apt.llvm.org site is a bit stale and
behind two aspects that changed on current Debian systems
(i.e. unstable/testing as used for r-base). First,
apt now prefers files ending in .sources (in a
nicer format) and second, it now really requires a key (which is good
practice). As it took me a few minutes to regather how to meet both
requirements, I reckoned I might as well script this.
Et voil the following script does that:
~/.R/Makevars to use that
compilerclang-22 (and clang++-22)
(still using the g++ C++ library)#!/bin/sh
## Update does not hurt but is not strictly needed
#apt update --quiet --quiet
#apt upgrade --yes
## wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
## or as we are root in container
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key > /etc/apt/trusted.gpg.d/apt.llvm.org.asc
cat <<EOF >/etc/apt/sources.list.d/llvm-dev.sources
Types: deb
URIs: http://apt.llvm.org/unstable/
# for clang-21
# Suites: llvm-toolchain-21
# for current clang
Suites: llvm-toolchain
Components: main
Signed-By: /etc/apt/trusted.gpg.d/apt.llvm.org.asc
EOF
test -d ~/.R mkdir ~/.R
cat <<EOF >~/.R/Makevars
CLANGVER=-22
# CLANGLIB=-stdlib=libc++
CXX=clang++\$(CLANGVER) \$(CLANGLIB)
CXX11=clang++\$(CLANGVER) \$(CLANGLIB)
CXX14=clang++\$(CLANGVER) \$(CLANGLIB)
CXX17=clang++\$(CLANGVER) \$(CLANGLIB)
CXX20=clang++\$(CLANGVER) \$(CLANGLIB)
CC=clang\$(CLANGVER)
SHLIB_CXXLD=clang++\$(CLANGVER) \$(CLANGLIB)
EOF
apt update
apt install --yes clang-22clang-22 and clang++-22. This may help
R package developers. The script is also generic enough for other
development communities who can ignore (or comment-out / delete) the bit
about ~/.R/Makevars and deploy the compiler differently.
Updating the softlink as apt-preferences does is one way
and done in many GitHub Actions recipes. As we only need
wget here a basic Debian container should work, possibly
with the addition of wget. For R users r-base
hits a decent sweet spot.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.
libscfg is a C
implementation of the scfg configuration
file format and it is a dependency of recent version of
kanshi. kanshi is a tool
similar to autorandr which allows you define output profiles and kanshi
switches to the correct output profile on hotplug events. Once libscfg is in
unstable I can finally update kanshi to the latest version.
A lot of time this month in finalizing a redesign of the output rendering of
carl. carl is a small rust program I wrote
that provides a calendar view similar to cal, but it comes with colors and
ical file integration. That means that you can not only display a simple
calendar, but also colorize/highlight dates based on various attributes or
based on events on that day. In the initial versions of carl the output
rendering was simply hardcoded into the app.
carl to use templates. I chose the
minijinja Rust library which is
based on the syntax and behavior of the Jinja2 template engine for Python .
There are others out there, like tera, but
minijinja seems to be more active in development currently. I worked on this
implementation on and off for the last year and finally had the time to finish
it up and write some additional tests for the outputs. It is easier to maintain
templates than Rust code that uses write!() to format the output. I also
implemented a configuration option for users to override the templates.
Additional to the output refactoring I also fixed couple of bugs and finally
released v0.4.0 of carl.
In my dayjob I released version 0.53 of apis-core-rdf which contains the place
lookup field which I implemented in August. A couple of weeks later we released
version 0.54 which comes with a middleware to show pass on messages from the
Django messages
framework via
response header to HTMX to trigger message popups. This implementation is based
on the blog post Using the Django messages framework with
HTMX. Version
0.55 was the last release in September. It contained preparations for
refactoring the import logic as well as a couple of UX improvements.
Welcome to post 51 in the R4 series.
A while back I realized I should really just post a little more as
not all post have to be as deep and introspective as for example the
recent-ish two
cultures post #49.
So this post is a neat little trick I (somewhat belatedly) realized
somewhat recently. The context is the ongoing transition from
(Rcpp)Armadillo 14.6.3 and earlier to (Rcpp)Armadillo 15.0.2 or later.
(I need to write a bit more about that, but that may require a bit more
time.) (And there are a total of seven (!!) issue tickets managing the
transition with issue
#475 being the main parent issue, please see there for more
details.)
In brief, the newer and current Armadillo no longer allows C++11
(which also means it no longer allowes suppression of deprecation
warnings ). It so happens that around a decade ago packages were
actively encouraged to move towards C++11 so many either set an
explicit SystemRequirements: for it, or set CXX_STD=CXX11
in src/Makevars .win . CRAN has for some time now issued
NOTEs asking for this to be removed, and more recently enforced this
with actual deadlines. In RcppArmadillo I opted to accomodate old(er)
packages (using this by-now anti-pattern) and flip to Armadillo 14.6.3
during a transition period. That is what the package does now: It gives
you either Armadillo 14.6.3 in case C++11 was detected (or this legacy
version was actively selected via a compile-time #define),
or it uses Armadillo 15.0.2 or later.
So this means we can have either one of two versions, and may want to
know which one we have. Armadillo carries its own version macros, as
many libraries or projects do (R of course included). Many many years
ago (git blame points to sixteen and twelve for a revision)
we added the following helper function to the package (full source here,
we show it here without the full roxygen2 comment header)
// [[Rcpp::export]]
Rcpp::IntegerVector armadillo_version(bool single)
// These are declared as constexpr in Armadillo which actually does not define them
// They are also defined as macros in arma_version.hpp so we just use that
const unsigned int major = ARMA_VERSION_MAJOR;
const unsigned int minor = ARMA_VERSION_MINOR;
const unsigned int patch = ARMA_VERSION_PATCH;
if (single)
return Rcpp::wrap(10000 * major + 100 * minor + patch) ;
else
return Rcpp::IntegerVector::create(Rcpp::Named("major") = major,
Rcpp::Named("minor") = minor,
Rcpp::Named("patch") = patch);
> library(RcppArmadillo)
> armadillo_version(FALSE)
major minor patch
15 0 2
> armadillo_version(TRUE)
[1] 150002
>package_version. (Full version of one such variant here)
// [[Rcpp::export]]
Rcpp::List armadilloVersion()
// create a vector of major, minor, patch
auto ver = Rcpp::IntegerVector::create(ARMA_VERSION_MAJOR, ARMA_VERSION_MINOR, ARMA_VERSION_PATCH);
// and place it in a list (as e.g. packageVersion() in R returns)
auto lst = Rcpp::List::create(ver);
// and class it as 'package_version' accessing print() etc methods
lst.attr("class") = Rcpp::CharacterVector::create("package_version", "numeric_version");
return lst;
package_version inheriting from
numeric_version can directly compare against a (human- but
not normally machine-readable) string like 15.0.0 because the simple
S3 class defines appropriate operators, as well as print()
/ format() methods as the first expression shows. It is
these little things that make working with R so smooth, and we can
easily (three statements !!) do so from Rcpp-based packages too.
The underlying object really is merely a list containing a
vector:
but the S3 glue around it makes it behave nicely.
So next time you are working with an object you plan to return to R,
consider classing it to take advantage of existing infrastructure (if it
exists, of course). It s easy enough to do, and may smoothen the
experience at the R side.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.
A shot of Jewel Changi. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
A shot of Sentosa Boardwalk. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
This is the hawker center we went to. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Table littering at the hawker center was prohibited by law. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Merlion from behind, giving a good view of Marina Bay Sands. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Next.