Activity summary
During the month of October, 21 contributors have been
paid to work on Debian LTS (links to individual
contributor reports are located below).
The team released 37 DLAs fixing 893 CVEs.
The team has continued in its usual rhythm, preparing and uploading security
updates targeting LTS and ELTS, as well as helping with updates to oldstable,
stable, testing, and unstable. Additionally, the team received several
contributions of LTS uploads from Debian Developers outside the standing LTS
Team.
Notable security updates:
https-everywhere, prepared by Markus Koschany, deals with a problem created by ownership of the https-rulesets.org domain passing to a malware operator
openjdk-17 and openjdk-11, prepared by Emilio Pozuelo Monfort, fixes XML external entity and certificate validation vulnerabilities
intel-microcode, prepared by Tobias Frost, fixes a variety of privilege escalation and denial of service vulnerabilities
Notable non-security updates:
distro-info-data, prepared by Stefano Rivera, updates information concerning current and upcoming Debian and Ubuntu releases
Contributions from outside the LTS Team:
Lukas M rdian, a Debian Developer, provided an update of log4cxx
Andrew Ruthven, one of the request-tracker4 maintainers, provided an update of request-tracker4
Christoph Goehre, co-maintainer of thunderbird, provided an update of thunderbird
Beyond the typical LTS updates, the team also helped the Debian community more broadly:
Guilhem Moulin prepared oldstable/stable updates of libxml2, and an unstable update of libxml2.9
Bastien Roucari s prepared oldstable/stable updates of imagemagick
Daniel Leidert prepared an oldstable update of python-authlib, oldstable update of libcommons-lang-java and stable update of libcommons-lang3-java
Utkarsh Gupta prepared oldstable/stable/testing/unstable updates of ruby-rack
The LTS Team is grateful for the opportunity to contribute to making LTS a high quality for sponsors and users. We are also particularly grateful for the collaboration from others outside the time; their contributions are important to the success of the LTS effort.
Transferring a Signal account between two Android devices
I spent far too much time recently trying to get a Signal Private Messenger account to transfer from one device to another.
What I eventually found worked was a very finicky path to enable functioning "Wi-Fi Direct", which I go into below.
I also offer some troubleshooting and recovery-from-failure guidance.
All of this blogpost uses "original device" to refer to the Android pocket supercomputer that already has Signal installed and set up, and "new device" to mean the Android device that doesn't yet have Signal on it.
Why Transfer?
Signal Private Messenger is designed
with the expectation that the user has a "primary device",
which is either an iPhone or an Android pocket supercomputer.
If you have an existing Signal account,
and try to change your primary device
by backing up and restoring from backup,
it looks to me like Signal will cause your
long-term identity keys to be changed.
This in turn causes your peers to see
a message like "Your safety number with Alice has changed."
These warning messages are the same messages that they would get if an adversary were to take over your account.
So it's a good idea to minimize them when there isn't an account takeover false alarms train people to ignore real alarms.
You can avoid "safety number changed" warnings by using signal's "account transfer" process during setup,
at least if you're transferring between two Android devices.
However, my experience was that the transfer between two Android devices was very difficult to get to happen at all.
I ran into many errors trying to do this, until I finally found a path that worked.
Dealing with Failure
After each failed attempt at a transfer, my original device's Signal installation would need to be re-registered.
Having set a PIN meant that i could re-register the device without needing to receive a text message or phone call.
Set a PIN before you transfer!
Also, after a failure, you need to re-link any "linked device" (i.e. any Signal Desktop or iPad installation).
If any message came in during the aborted transfer, the linked device won't get a copy of that message.
Finally, after a failed transfer, i recommend completely uninstalling Signal from the new device, and starting over with a fresh install on the new device.
Permissions
My understanding is that Signal on Android uses Wi-Fi Direct to accomplish the transfer.
But to use Wi-Fi Direct, Signal needs to have the right permissions.
On each device:
Entirely stop the Signal app
Go to Settings Apps Signal Permissions
Ensure that the following permissions are all enabled whenever the app is running:
Location
Nearby Devices
Network
Preparing for Wi-Fi Direct
The transfer process depends on "Wi-Fi Direct", which is a bit of a disaster on its own.
I found that if i couldn't get Wi-Fi Direct to work between the two devices, then the Signal transfer was guaranteed to fail.
So, for clearer debugging, i first tried to establish a Wi-Fi Direct link on Android, without Signal being involved at all.
Setting up a Wi-Fi Direct connection directly failed, multiple times, until i found the following combination of steps, to be done on each device:
Turn off Bluetooth
Ensure Wi-Fi is enabled
Disconnect from any Wi-Fi network you are connected to (go to the "Internet" or "Wi-Fi" settings page, long-press on the currently connected network, and choose "Disconnect"). If your device knows how to connect to multiple local Wi-Fi networks, disconnct from each of them in turn until you are in a stable state where Wi-Fi is enabled, but no network is connected.
Close to the bottom of the "Inteernet" or "Wi-Fi" settings page, choose "Network Preferences" and then "Wi-Fi Direct"
if there are any entries listed under "Remembered groups", tap them and choose to "Forget this group"
If there are Peer devices that say "Invited", tap them and choose to "Cancel invitation"
I found that this configuration is the most likely to enable a successful Wi-Fi Direct connection, where clicking "invite" on one device would pop up an alert on the other asking to accept the connection, and result in a "Connected" state between the two devices.
Actually Transferring
Start with both devices fully powered up and physically close to one another (on the same desk should be fine).
On the new device:
Reboot the device, and log into the profile you want to use
Enable Internet access via Wi-Fi.
Remove any old version of Signal.
Install Signal, but DO NOT OPEN IT!
Set up the permissions for the Signal app as described above
Open Signal, and choose "restore or transfer" -- you still need to be connected to the network at this point.
The new device should display a QR code.
On the original device:
Reboot the device, and log into the profile that has the Signal account you're looking to transfer
Enable Internet access via Wi-Fi, using the same network that the old device is using.
Make sure the permissions for Signal are set up as described above
Open Signal, and tap the camera button
Point the camera at the new device's QR code
Now tap the "continue" choices on both devices until they both display a message that they are searching for each other.
You might see the location indicator (a green dot) turn on during this process.
If you see an immediate warning of failure on either device, you probably don't have the permissions set up right.
You might see an alert (a "toast") on one of the devices that the other one is trying to connect.
You should click OK on that alert.
In my experience, both devices are likely to get stuck "searching" for each other.
Wait for both devices to show Signal's warning that the search has timed out.
At this point, leave Signal open on both devices, and go through all the steps described above to prepare for Wi-Fi Direct.
Your Internet access will be disabled.
Now, tap "Try again" in Signal on both devices, pressing the buttons within a few seconds of each other.
You should see another alert that one device is trying to connect to the other.
Press OK there.
At this point, the transfer should start happening!
The old device will indicate what percentag has been transferred, and the new device will indicate how many messages hav been transferred.
When this is all done, re-connect to Wi-Fi on the new device.
Temporal gap for Linked Devices
Note that during this process, if new messages are arriving, they will be queuing up for you.
When you reconnect to wi-fi, the queued messages will flow to your new device.
But the process of transferring automatically unlinks any linked devices.
So if you want to keep your instance of Signal Desktop with as short a gap as possible, you should re-link that installation promptly after the transfer completes.
Clean-up
After all this is done successfully, you probably want to go into the Permissions settings and turn off the Location and Nearby Devices permissions for Signal on both devices.
I recommend also going into Wi-Fi Direct and removing any connected devices and forgetting any existing connections.
Conclusion
This is an abysmally clunky user experience, and I'm glad I don't have to do it often.
It would have been much simpler to make a backup and restore from it, but I didn't want to freak out my contacts with a safety number change.
By contrast, when i wanted extend a DeltaChat account across two devices, the transfer was prompt and entirely painless -- i just had to make sure the devices were on the same network, and then scanned a QR code from one to the other.
And there was no temporal gap for any other deviees.
And i could use Delta on both devices simultaneously until i was convinced that it would work on the new device -- Delta doesn't have the concept of a primary account.
I wish Signal made it that easy!
Until it's that easy, i hope the processes described here are useful to someone.
Upstreaming cPython patches, by Stefano Rivera
Python 3.14.0 (final) released in early October, and Stefano uploaded it to
Debian unstable. The transition to support 3.14 has begun in Ubuntu, but hasn t
started in Debian, yet.
While build failures in Debian s non-release ports are typically not a concern
for package maintainers, Python is fairly low in the stack. If a new minor
version has never successfully been built for a Debian port by the time we start
supporting it, it will quickly become a problem for the port. Python 3.14 had
been failing to build on two Debian ports architectures (hppa and m68k), but
thankfully their porters provided patches. These were applied and uploaded, and
Stefano forwarded the hppa one upstream.
Getting it into shape for upstream approval took some work, and shook out
severalotherregressionsfor the Python hppa port.
Debugging these on slow hardware takes a while.
These two ports aren t successfully autobuilding 3.14 yet (they re both timing
out in tests), but they re at least manually buildable, which unblocks the ports.
Docutils 0.22 also landed in Debian around this time, and Python
needed some work to build its
docs with it. The upstream isn t quite comfortable with distros using newer
docutils, so there isn t a clear path forward for these patches, yet.
The start of the Python 3.15 cycle was also a good time to renew submission
attempts on our other outstanding python patches, most importantly
multiarch tuples for stable ABI extension filenames.
ansible-core autopkgtest robustness, by Colin Watson
The ansible-core package runs its integration tests via autopkgtest. For some
time, we ve seen occasional failures in the expect, pip, and
template_jinja2_non_native tests that usually go away before anyone has a
chance to look into them properly. Colin found that these were blocking an
openssh upgrade and so decided to track them down.
It turns out that these failures happened exactly when the libpython3.13-stdlib
package had different versions in testing and unstable. A setup script removed
/usr/lib/python3*/EXTERNALLY-MANAGED in order that pip can install system
packages for some of the tests, but if a package shipping that file were ever
upgraded then that customization would be undone, and the same setup script
removed apt pins in a way that caused problems when autopkgtest was invoked
in certain ways. In combination with this, one of the integration tests
attempted to disable system apt sources while testing the behaviour of the
ansible.builtin.apt module, but it failed to do so comprehensively enough and
so that integration test accidentally upgraded the testbed from testing to
unstable in the middle of the test. Chaos ensued.
Colin fixed this in Debian
and contributed the relevant part upstream.
Miscellaneous contributions
Carles kept working on the missing-relations (packages which Recommends or
Suggests packages that are not available in Debian). He improved the tooling to
detect Suggested packages
that are not available in Debian because they were removed (or changed names).
Carles improved po-debconf-manager
to send translations for packages that are not in Salsa. He also improved the UI
of the tool (using rich for some of the output).
Carles, using po-debconf-manager, reviewed and submitted 38 debconf template
translations.
Carles created a merge request
for distro-tracker to align text and input-field (postponed until distro-tracker
uses Bootstrap 5).
Rapha l updated gnome-shell-extension-hamster for GNOME 49. It is a GNOME
Shell integration for the Hamster time tracker.
Rapha l merged a couple of trivial merge requests, but he did not yet find the
time to properly review and test the bootstrap 5 related merge requests that are
still waiting on salsa.
Helmut sent patches for 20 cross build failures.
Helmut refactored debvm dropping support for running on bookworm . There
are two trixie features improving the operation. mkfs.ext4 can now consume a
tar archive to populate the filesystem via libarchive and dash now supports
set -o pipefail. Beyond this change in operation, a number of robustness and
quality issues have been resolved.
Thorsten fixed some bugs in the printing software and uploaded improved
versions of brlaser and ifhp. Moreover he uploaded a new upstream version of
cups.
Emilio updated xorg-server to the latest security release and helped with
various transitions.
Santiago supported the work on the DebConf 26 organisation, particularly
helping with an implemented method to count the votes to choose the conference
logo.
Stefano reviewed Python PEP-725 and
PEP-804, which hope to provide a mechanism
to declare external (e.g. APT) dependencies in Python packages. Stefano engaged
in discussion and provided feedback to the authors.
Stefano prepared for Berkeley DB removal in Python.
Stefano ported the backend to
reverse-depends to Python 3 (yes, it had been running on 2.7) and migrated it
to git from bzr.
Stefano updated miscellaneous packages, including beautifulsoup4,
mkdocs-macros-plugin, python-pipx.
Stefano uploaded an update to distro-info-data, including data for two
additional Debian derivatives: eLxr and Devuan.
Stefano prepared an update to dh-python, the python packaging tool, merging
several contributed patches and resolving some bugs.
Colin upgraded OpenSSH to 10.1p1, helped upstream to chase down some
regressions, and further upgraded to 10.2p1. This is also now in trixie-backports.
Colin fixed several build regressions with Python 3.14, scikit-learn 1.7, and
other transitions.
Continuing from where Badri and I left off in the last post. On the 7th of December 2024, we boarded a bus from Singapore to the border town of Johor Bahru in Malaysia. The bus stopped at the Singapore emigration for us to get off for the formalities.
The process was similar to the immigration at the Singapore airport. It was automatic, and we just had to scan our passports for the gates to open. Here also, we didn t get Singapore stamps on our passports.
After we were done with the emigration, we had to find our bus. We remembered the name of the bus company and the number plate, which helped us recognize our bus. It wasn t there already after we came out of the emigration, but it arrived soon enough, and we boarded it promptly.
From the Singapore emigration, the bus travelled a few kilometers and dropped us at Johor Bahru Sentral (JB Sentral) bus station, where we had to go through Malaysian immigration. The process was manual, unlike Singapore, and there was an immigration officer at the counter who stamped our passports (which I like) and recorded our fingerprints.
At the bus terminal, we exchanged rupees at an exchange shop to get Malaysian ringgits. We could not find any free drinking water sources on the bus terminal, so we had to buy water.
Badri later told me that Johor Bahru has a lot of data centers, which need a lot of water for cooling. When he read about it later, he immediately connected it with the fact that there was no free drinking water, and we had to buy water. Such data centers can lead to scarcity of water for others in the area.
From JB Sentral, we took a bus to Larkin Terminal, as our hotel was nearby. It was 1.5 ringgits per person (30 rupees). In order to pay for the fare, we had to put cash in a box near the driver s seat.
Around half-an-hour later, we reached our hotel. The time was 23:30 hours. The hotel room was hot as it didn t have air-conditioning. The weather in Malaysia is on the hotter side throughout the year. It was a budget hotel, and we paid 70 ringgits for our room.
Badri slept soon after we checked-in. I went out during the midnight at around 00:30. I was hungry, so I entered a small scale restaurant nearby, which was quite lively for the midnight hours. At the restaurant, I ordered a coffee and an omelet. I also asked for drinking water. The unique thing about that was that they put ice in hot water to make its temperature normal.
My bill from the restaurant looked like the below-mentioned table, as the items names were in the local language Malay:
Item
Price (Malaysian ringgits)
Conversion to Indian rupees
Comments
Nescafe Tarik
2.50
50
Coffee
Ais Kosong
0.50
10
Water
Telur Dadar
2.00
40
Omelet
SST Tax (6%)
0.30
6
Total
5.30
106
After checking out from the restaurant, I explored nearby shops. I also bought some water before going back to the hotel room.
The next day, we had a (pre-booked) bus to Kuala Lumpur. We checked out from the hotel 10 minutes after the check-out time (which was 14:00 hours). However, within those 10 minutes, the hotel staff already came up three times asking us to clear out (which we were doing as fast as possible). And finally on the third time they said our deposit was forfeit, even though it was supposed to be only for keys and towels.
The above-mentioned bus for Kuala Lumpur was from the nearby Larkin Bus Terminal. The bus terminal was right next to our hotel, so we walked till there.
Upon reaching there, we found out that the process of boarding a bus in Malaysia resembled with taking a flight. We needed to go to a counter to get our boarding passes, followed by reporting at our gate half-an-hour before the scheduled time. Furthermore, they had a separate waiting room and boarding gates. Also, there was a terminal listing buses with their arrival and departure signs. Finally, to top it off, the buses had seatbelts.
We got our boarding pass for 2 ringgits (40 rupees). After that, we proceeded to get something to eat as we were hungry. We went to a McDonald s, but couldn t order anything because of the long queue. We didn t have a lot of time, so we proceeded towards our boarding gate without having anything.
The boarding gate was in a separate room, which had a vending machine. I tried to order something using my card, but the machine wasn t working. In Malaysia, there is a custom of queueing up to board buses even before the bus has arrived. We saw it in Johor Bahru as well. The culture is so strong that they even did it in Singapore while waiting for the Johor Bahru bus!
Our bus departed at 15:30 as scheduled. The journey was around 5 hours. A couple of hours later, our bus stopped for a break. We got off the bus and went to the toilet. As we were starving (we didn t have anything the whole day), we thought it was a good opportunity to get some snack. There was a stall selling some food. However, I had to determine which options were vegetarian. We finally settled on a cylindrical box of potato chips, labelled Mister Potato. They were 7 ringgits.
We didn t know how long the bus is going to stop. Furthermore, eating inside buses in Malaysia is forbidden. When we went to get some coffee from the stall, our bus driver was standing there and made a face. We got an impression that he doesn t want us to have coffee.
However, after we got into the bus, we had to wait for a long time for it to resume its journey as the driver was taking his sweet time to drink his coffee.
During the bus journey, we saw a lot of palm trees on the way. The landscape was beautiful, with good road infrastructure throughout the journey. Badri also helped me improve my blog post on obtaining Luxembourg visa in the bus.
The bus dropped us at the Terminal Bersepadu Selatan (TBS in short) in Kuala Lumpur at 21:30 hours.
Finally, we got something at the TBS. We also noticed that the TBS bus station had lockers. This gave us the idea of putting some of our luggage in the lockers later while we will be in Brunei. We had booked a cheap Air Asia ticket which doesn t allow check-in luggage. Further, keeping the checked-in luggage in lockers for three days was cheaper than paying the excess luggage penalty for Air Asia.
We followed it up by taking a metro as our hotel was closer to a metro station. This was a bad day due to our deposit being forfeited unfairly, and got nothing to eat.
We took the metro to reach our hostel, which was located in the Bukit Bintang area. The name of this hostel was Manor by Mingle. I had stayed here earlier in February 2024 for two nights. Back then, I paid 1000 rupees per day for a dormitory bed. However, this time the same hostel was much cheaper. We got a private room for 800 rupees per day, with breakfast included. Earlier it might have been pricier due to my stay falling on weekends or maybe February has more tourists in Kuala Lumpur.
That s it for this post. Stay tuned for our adventures in Malaysia!
Welcome to the October 2025 report from the Reproducible Builds project!
Welcome to the very latest report from the Reproducible Builds project. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
In this report:
Farewell from the Reproducible Builds Summit 2025
Thank you to everyone who joined us at the Reproducible Builds Summit in Vienna, Austria!
We were thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. During this event, participants had the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim was to create an inclusive space that fosters collaboration, innovation and problem-solving.
The agenda of the three main days is available online however, some working sessions may still lack notes at time of publication.
One tangible outcome of the summit is that Johannes Starosta finished their rebuilderd tutorial, which is now available online and Johannes is actively seeking feedback.
Google s Play Store breaks reproducible builds for Signal
On the issue tracker for the popular Signal messenger app, developer Greyson Parrelli reports that updates to the Google Play store have, in effect, broken reproducible builds:
The most recent issues have to do with changes to the APKs that are made by the Play Store. Specifically, they add some attributes to some .xml files around languages are resources, which is not unexpected because of how the whole bundle system works. This is trickier to resolve, because unlike current expected differences (like signing information), we can t just exclude a whole file from the comparison. We have to take a more nuanced look at the diff. I ve been hesitant to do that because it ll complicate our currently-very-readable comparison script, but I don t think there s any other reasonable option here.
kpcyrd forwarded a fascinating tidbit regarding so-called ninja and samurai build ordering, that uses data structures in which the pointer values returned from malloc are used to determine some order of execution.
Arnout Engelen, Justin Cappos, Ludovic Court s and kpcyrd continued a conversation started in September regarding the Minimum Elements for a Software Bill of Materials . (Full thread)
Felix Moessbauer of Siemens posted to the list reporting that he had recently stumbled upon a couple of Debian source packages on the snapshot mirrors that are listed multiple times (same name and version), but each time with a different checksum . The thread, which Felix titled, Debian: what precisely identifies a source package is about precisely that what can be axiomatically relied upon by consumers of the Debian archives, as well as indicating an issue where we can t exactly say which packages were used during build time (even when having the .buildinfo files).
Luca DiMaio posted to the list announcing the release of xfsprogs 6.17.0 which specifically includes a commit that implements the functionality to populate a newly created XFS filesystem directly from an existing directory structure which makes it easier to create populated filesystems
without having to mount them [and thus is] particularly useful for reproducible builds . Luca asked the list how they might contribute to the docs of the System images page.
Reproducible Builds at the Transparency.dev summit
Holger Levsen gave a talk at this year s Transparency.dev summit in Gothenburg, Sweden, outlining the achievements of the Reproducible Builds project in the last 12 years, covering both upstream developments as well as some distribution-specific details. As mentioned on the talk s page, Holger s presentation concluded with an outlook into the future and an invitation to collaborate to bring transparency logs into Reproducible Builds projects .
The slides of the talk are available, although a video has yet to be released. Nevertheless, as a result of the discussions at Transparency.dev there is a new page on the Debian wiki with the aim of describing a potential transparency log setup for Debian.
Supply Chain Security for Go
Andrew Ayer has setup a new service at sourcespotter.com that aims to monitor the supply chain security for Go releases. It consists of four separate trackers:
A tool to verify that the Go Module Mirror and Checksum Database is behaving honestly and has not presented inconsistent information to clients.
A module monitor that records every module version served by the Go Module Mirror and Checksum Database, allowing you to monitor for unexpected versions of your modules.
A tool to verifies that the Go toolchains published in the Go Module Mirror can be reproduced from source code, making it difficult to hide backdoors in the binaries downloaded by the go command.
A telemetry config tracker that tracks the names of telemetry counters uploaded by the Go toolchain, to ensure that Go telemetry is not violating users privacy.
As the homepage of the service mentions, the trackers are free software and do not rely on Google infrastructure.
In March 2024, a sophisticated backdoor was discovered in xz, a core compression library in Linux distributions, covertly inserted over three years by a malicious maintainer, Jia Tan. The attack, which enabled remote code execution via ssh, was only uncovered by chance when Andres Freund investigated a minor performance issue. This incident highlights the vulnerability of the open-source supply chain and the effort attackers are willing to invest in gaining trust and access. In this article, I analyze the backdoor s mechanics and explore how bitwise build reproducibility could have helped detect it.
Although quantum computing is a rapidly evolving field of research, it can already benefit from adopting reproducible builds. This paper aims to bridge the gap between the quantum computing and reproducible builds communities. We propose a generalization of the definition of reproducible builds in the quantum setting, motivated by two threat models: one targeting the confidentiality of end users data during circuit preparation and submission to a quantum computer, and another compromising the integrity of quantum computation results. This work presents three examples that show how classical information can be hidden in transpiled quantum circuits, and two cases illustrating how even minimal modifications to these circuits can lead to incorrect quantum computation results.
The thesis focuses on providing a reproducible build process for two open-source E2EE messaging applications: Signal and Wire. The motivation to ensure reproducibility and thereby the integrity of E2EE messaging applications stems from their central role as essential tools for modern digital privacy. These applications provide confidentiality for private and sensitive communications, and their compromise could undermine encryption mechanisms, potentially leaking sensitive data to third parties.
Currently, there are numerous solutions and techniques available in the market to tackle supply chain security, and all claim to be the best solution. This thesis delves deeper by implementing those solutions and evaluates them for better understanding. Some of the tools that this thesis implemented are Syft, Trivy, Grype, FOSSA, dependency-check, and Gemnasium. Software dependencies are generated in a Software Bill of Materials (SBOM) format by using these open-source tools, and the corresponding results have been analyzed. Among these tools, Syft and Trivy outperform others as they provide relevant and accurate information on software dependencies.
In the wake of growing supply chain attacks, the FreeBSD developers are relying on a transparent build concept in the form of Zero-Trust Builds. The approach builds on the established Reproducible Builds, where binary files can be rebuilt bit-for-bit from the published source code. While reproducible builds primarily ensure verifiability, the zero-trust model goes a step further and removes trust from the build process itself. No single server, maintainer, or compiler can be considered more than potentially trustworthy.
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Bernhard Wiedemann and Zbigniew J drzejewski-Szmek extended ismypackagereproducibleyet.org with initial support for Fedora [].
In addition, a number of contributors added a series of notes from our recent summit to the website, including Alexander Couzens [], Robin Candau [][][][][][][][][] and kpcyrd [].
Tool development
diffoscope version 307 was uploaded to Debian unstable by Chris Lamb, who made a number of changes including fixing compatibility with LLVM version 21 [], an attempt to automatically attempt to deploy to PyPI by liaising with the PyPI developers/maintainers (with this experimental feature). [] In addition, Vagrant Cascadian updated diffoscope in GNU Guix to version 307.
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
I m pleased to share that my career transition has been successful! I ve joined our local county assessor s office, beginning a new path in property assessment for taxation and valuation. While the compensation is modest, it offers the stability I was looking for.
My new schedule consists of four 10-hour days with an hour commute each way, which means Monday through Thursday will be largely devoted to work and travel. However, I ll have Fridays available for open source contributions once I ve completed my existing website maintenance commitments.
Open Source Priorities
Going forward, my contribution focus will be:
Ubuntu Community Council
Kubuntu/Debian
Snap packages (as time permits)
Regarding the snap packages: my earlier hope of transitioning them to Carl hasn t worked out as planned. He s taken on maintaining KDE Neon single-handedly, and understandably, adding snap maintenance on top of that proved unfeasible. I ll do what I can to help when time allows.
Looking for Contributors
If you re interested in contributing to Kubuntu or helping with snap packages, I d love to hear from you! Feel free to reach out community involvement is what makes these projects thrive.
Thanks for your patience and understanding as I navigate this transition.
Those Who Wait is a stand-alone self-published sapphic romance
novel. Given the lack of connection between political figures named in
this book and our reality, it's also technically an alternate history, but
it will be entirely unsatisfying to anyone who reads it in that genre.
Sutton Spencer is an English grad student in New York City. As the story
opens, she has recently realized that she's bisexual rather than straight.
She certainly has not done anything about that revelation; the very
thought makes her blush. Her friend and roommate Regan, not known for
either her patience or her impulse control, decides to force the issue by
stealing Sutton's phone, creating a profile on a lesbian dating app, and
messaging the first woman Sutton admits being attracted to.
Charlotte Thompson is a highly ambitious politician, current deputy mayor
of New York City for health and human services, and granddaughter of the
first female president of the United States. She fully intends to become
president of the United States herself. The next step on that path is an
open special election for a seat in the House of Representatives. With her
family political connections and the firm support of the mayor of New York
City (who is also dating her brother), she thinks she has an excellent
shot of winning.
Charlotte is also a lesbian, something she's known since she was a
teenager and which still poses serious problems for a political career.
She is therefore out to her family and a few close friends, but otherwise
in the closet. Compared to her political ambitions, Charlotte considers
her love life almost irrelevant, and therefore has a strict policy of
limiting herself to anonymous one-night stands arranged on dating apps.
Even that is about to become impossible given her upcoming campaign, but
she indulges in one last glance at SapphicSpark before she deletes her
account.
Sutton is as far as possible from the sort of person who does one-night
stands, which is a shame as far as Charlotte is concerned. It would have
been a fun last night out. Despite that, both of them find the other
unexpectedly enjoyable to chat with. (There are a lot of text message
bubbles in this book.) This is when Sutton has her brilliant idea:
Charlotte is charming, experienced, and also kind and understanding of
Sutton's anxiety, at least in app messages. Maybe Charlotte can be her
mentor? Tell her how to approach women, give her some guidance, point her
in the right directions.
Given the genre, you can guess how this (eventually) turns out.
I'm going to say a lot of good things about this book, so let me get the
complaints over with first.
As you might guess from that introduction, Charlotte's political career
and the danger of being outed are central to this story. This is a bit
unfortunate because you should not, under any circumstances, attempt to
think deeply about the politics in this book.
In 550 pages, Charlotte does not mention or expound a single meaningful
political position. You come away from this book as ignorant about what
Charlotte wants to accomplish as a politician as you entered. Apparently
she wants to be president because her grandmother was president and she
thinks she'd be good at it. The closest the story comes to a position is
something unbelievably vague about homeless services and Charlotte's
internal assertion that she wants to help people and make real change.
There are even transcripts of media interviews, later in the book, and
they somehow manage to be more vacuous than US political talk shows, which
is saying something. I also can't remember a single mention of fundraising
anywhere in this book, which in US politics is absurd (although I will be
generous and say this is due to Cass's alternate history).
I assume this was a deliberate choice and Cass didn't want politics to
distract from the romance, but as someone with a lot of opinions about
concrete political issues, the resulting vague soft-liberal squishiness
was actively off-putting. In an actual politician, this would be an entire
clothesline of red flags. Thankfully, it's ignorable for the same reason;
this is so obviously not the focus of the book that one can mostly perform
the same sort of mental trick that one does when ignoring the backdrop in
a cheap theater.
My second complaint is that I don't know what Sutton does outside
of the romance. Yes, she's an English grad student, and she does some
grading and some vaguely-described work and is later referred to a
prestigious internship, but this is as devoid of detail as Charlotte's
political positions. It's not quite as jarring because Cass does
eventually show Sutton helping concretely with her mother's work (about
which I have some other issues that I won't get into), but it deprives
Sutton of an opportunity to be visibly expert in something. The romance
setup casts Charlotte as the experienced one to Sutton's naivete, and I
think it would have been a better balance to give Sutton something
concrete and tangible that she was clearly better at than Charlotte.
Those complaints aside, I quite enjoyed this. It was a recommendation from
the same
BookTuber who recommended Delilah Green
Doesn't Care, so her recommendations are quickly accumulating more
weight. The chemistry between Sutton and Charlotte is quite believable;
the dialogue sparkles, the descriptions of the subtle cues they pick up
from each other are excellent, and it's just fun to read about how they
navigate a whole lot of small (and sometimes large) misunderstandings and
mismatches in personality and world view.
Normally, misunderstandings are my least favorite part of a romance novel,
but Sutton and Charlotte come from such different perspectives that their
misunderstandings feel more justified than is typical. The characters are
also fairly mature about working through them: Main characters who track
the other character down and insist on talking when something happens they
don't understand! Can you imagine! Only with the third-act breakup is the
reader dragged through multiple chapters of both characters being
miserable, and while I also usually hate third-act breakups, this one is
so obviously coming and so clearly advertised from the initial setup that
I couldn't really be mad. I did wish the payoff make-up scene at the end
of the book had a bit more oomph, though; I thought Sutton's side of it
didn't have quite the emotional catharsis that it could have had.
I particularly enjoyed the reasons why the two characters fall in love,
and how different they are. Charlotte is delighted by Sutton because she's
awkward and shy but also straightforward and frequently surprisingly
blunt, which fits perfectly with how much Charlotte is otherwise living in
a world of polished politicians in constant control of their personas.
Sutton's perspective is more physical, but the part I liked was the way
that she treats Charlotte like a puzzle. Rather than trying to change how
Charlotte expresses herself, she instead discovers that she's remarkably
good at reading Charlotte if she trusts her instincts. There was something
about Sutton's growing perceptiveness that I found quietly delightful.
It's the sort of non-sexual intimacy that often gets lost among the big
emotions in romance novels.
The supporting cast was also great. Both characters have deep support
networks of friends and family who are unambiguously on their side. Regan
is pure chaos, and I would not be friends with her, but Cass shows her
deep loyalty in a way that makes her dynamic with Sutton make sense. Both
characters have thoughtful and loving families who support them but don't
make decisions for them, which is a nice change of pace from the usually
more mixed family situations of romance novel protagonists. There's a lot
of emotional turbulence in the main relationship, and I think that only
worked for me because of how rock-solid and kind the supporting cast is.
This is, as you might guess from the title, a very slow burn, although the
slow burn is for the emotional relationship rather than the physical one
(for reasons that would be spoilers). As usual, I have no calibration for
spiciness level, but I'd say that this was roughly on par with the later
books in the Bright Falls series.
If you know something about politics (or political history) and try to
take that part of this book seriously, it will drive you to drink, but if
you can put that aside and can deal with misunderstandings and emotional
turmoil, this was both fun and satisfying. I liked both of the characters,
I liked the timing of the alternating viewpoints, and I believed in the
relationship and chemistry, as improbable and chaotic as some of the setup
was. It's not the greatest thing I ever read, and I wish the ending was a
smidgen stronger, but it was an enjoyable way to spend a few reading days.
Recommended.
Rating: 7 out of 10
Welcome to post 54 in the R4 series.
The topic of continuous integration has been a recurrent theme here
at the R4 series. Post #32
introducess r-ci,
while post #41
brings r2u to r-ci, but does not show a
matrix deployment. Post #45
describes the updated r-ci setup that is now
the default and contains a macOS and Ubuntu matrix, where the latter
relies on r2u to keep
things fast, easy, reliable . Last but not least more recent post #52
shares a trick for ensuring coverage reports.
Following #45,
use of r-ci at for
example GitHub Actions has seen steady use and very reliable
performance. With the standard setup, a vanilla Ubuntu setup is changed
into one supported by r2u. This requires
downloading and installating a few Ubuntu packages, and has
generally been fairly quick on the order of around fourty
seconds. Now, the general variability of run-times for identical
tasks in GitHub Actions is well documented by the results of the
setup described in post #39
which still runs weekly. It runs the identical SQL query against a
remote backend using two different package families. And lo and behold,
the intra-method variability on unchanged code or setup and
therefore due solely to system variability is about as large as the
inter-method variability. In short, GitHub Actions performance varies
randomly with significant variability. See the repo README.md for
chart that updates weekly (and see #39
for background).
Of late, this variability became more noticeable during standard
GitHub Actions runs where it would regularly take more than two minutes
of setup time before actual continuous integration work was done. Some
caching seems to be in effect, so subsequent runs in the same
repo seem faster and often came back to one minute or less. For
lightweight and small packages, loosing two minutes to setup when the
actual test time is a minute or less gets old fast.
Looking around, we noticed that container use can be combined with
matrix use. So we have now been deploying the following setup (not
always over all the matrix elements though)
GitHub Actions is smart enough to provide NULL for
container in the two other cases, so
container: $ matrix.container is ignored there. But
when container is set as here for the ci-enhanced version
of r2u (which adds a few binaries commonly needed such as
git, curl, wget etc needed for
CI) then the CI jobs runs inside the container. And thereby skips most
of the setup time as the container is already prepared.
This also required some small adjustments in the underlying shell
script doing the work. To not disrupt standard deployment, we placed
these into a release candidate / development version one can op into
via an new variable dev_version
Everything else remains the same and works as before. But faster as
much less time is spent on setup. You can see the actual full yaml file
and actions in my repositories for rcpparmadillo and
rcppmlpack-examples.
Additional testing would be welcome, so feel free to deploy this in your
actions now. Otherwise I will likely carry this over and make it the
defaul in a few weeks time. It will still work as before but when
the added container: line is used will run much faster
thanks to rocker/r2u4ci being already set up for CI.
I ve written down a new rule (no name, sorry) that I ll be repeating to myself
and those around me. If you can replace DNS with key value store mapping
a name to an ip and it still makes sense, it was not, in fact, DNS. Feel
free to repeat it along with me.
Sure, the It s always DNS meme is funny the first few hundred times you see
it but what s less funny is when critical thinking ends because a DNS query
is involved. DNS failures are often the first observable problem because
it s one of the first things that needs to be done. DNS is fairly complicated,
implementation-dependent, and at times frustrating to debug but it is not
the operational hazard it s made out to be. It s at best a shallow take, and at
worst actively holding teams back from understanding their true operational
risks.
IP connectivity failures between a host and the rest of the network is not a
reason to blame DNS. This would happen no matter how you distribute the updated
name to IP mappings. Wiping out
all the records during the course of operations due to an automation bug
is not a reason to blame DNS. This, too, would happen no matter how you
distribute the name to IP mappings. Something made the choice to delete all the
mappings, and it did what you asked it to do
There s plenty of annoying DNS specific sharp edges to blame when things do
go wrong (like 8.8.8.8 and 1.1.1.1 disagreeing about resolving a domain
because of DNSSEC, or since we re on the topic, a
DNSSEC rollout bricking prod for hours)
for us to be cracking jokes anytime a program makes a DNS request.
We can do better.
Yesterday, I had my first successful AI coding
experience.
I ve used AI coding tools before and come away disappointed. The
results were underwhelming: low-quality code, inconsistent abstraction
levels, and subtle bugs that take longer to fix than it would take to
write the whole thing from scratch.
Those problems haven t vanished. The code quality this time was still
disappointing. As I asked the AI to refined its work, it would randomly
drop important constraints or refactor things in unhelpful ways. And
yet, this experience was different and genuinely valuable for
two reasons.
The first benefit was the obvious one: the AI helped me get over the
blank-page problem. It produced a workable skeleton for
the project imperfect, but enough to start building on.
The second benefit was more surprising. I was working on a problem in
odds-ratio preference optimization specifically,
finding a way to combine similar examples in datasets for AI training. I
wanted an ideal algorithm, one that extracted every ounce of
value from the data.
The AI misunderstood my description. Its first attempt was laughably
simple it just concatenated two text strings. Thanks, but I can call
strcat or the Python equivalent without help.
However, the second attempt was different. It was still not what I
had asked for but as I thought about it, I realized it was good enough.
The AI had created a simpler algorithm that would probably solve my
problem in practice.
In trying too hard to make the algorithm perfect, I d overlooked that
the simpler approach might be the right one. The AI, by
misunderstanding, helped me see that.
This experience reminded me of something that happened years ago when
I was mentoring a new developer. They came to me asking how to solve a
difficult problem. Rather than telling them it was impossible, I
explained what would be required: a complex authorization framework,
intricate system interactions, and a series of political and
organizational hurdles that would make deployment nearly impossible.
A few months later, they returned and said they d found a solution. I
was astonished until I looked more closely. What they d built wasn t the
full, organization-wide system I had envisioned. Instead, they d
reframed the problem. By narrowing the scope reducing the need for
global trust and deep integration they d built a local solution that
worked well enough within their project.
They succeeded precisely because they didn t see all the
constraints I did. Their inexperience freed them from assumptions that
had trapped me.
That s exactly what happened with the AI. It didn t know which
boundaries not to cross. In its simplicity, it found a path forward that
I had overlooked.
My conclusion isn t that AI coding is suddenly great. It s that
working with someone or something that thinks differently can
open new paths forward. Whether it s an AI, a peer, or a less
experienced engineer, that collaboration can bring fresh perspectives
that challenge your assumptions and reveal simpler, more practical ways
to solve problems.
Rory Stewart is a former British diplomat, non-profit executive, member of
Parliament, and cabinet minister. Politics on the Edge is a memoir
of his time in the UK Parliament from 2019 to 2019 as a Tory
(Conservative) representing the Penrith and The Border constituency in
northern England. It ends with his failed run against Boris Johnson for
leader of the Conservative Party and Prime Minister.
This book provoked many thoughts, only some of which are about the book.
You may want to get a beverage; this review will be long.
Since this is a memoir told in chronological order, a timeline may be
useful. After Stewart's time as a regional governor in occupied Iraq
(see The Prince of the Marshes), he
moved to Kabul to found and run an NGO to preserve traditional Afghani
arts and buildings (the
Turquoise Mountain Foundation, about which I know nothing except what
Stewart wrote in this book). By his telling, he found that work deeply
rewarding but thought the same politicians who turned Iraq into a mess
were going to do the same to Afghanistan. He started looking for ways to
influence the politics more directly, which led him first to Harvard and
then to stand for Parliament.
The bulk of this book covers Stewart's time as MP for Penrith and The
Border. The choice of constituency struck me as symbolic of Stewart's
entire career: He was not a resident and had no real connection to the
district, which he chose for political reasons and because it was the
nearest viable constituency to his actual home in Scotland. But once he
decided to run, he moved to the district and seems sincerely earnest in
his desire to understand it and become part of its community. After five
years as a backbencher, he joined David Cameron's government in a minor
role as Minister of State in the Department for Environment, Food, and
Rural Affairs. He then bounced through several minor cabinet positions
(more on this later) before being elevated to Secretary of State for
International Development under Theresa May. When May's government
collapsed during the fight over the Brexit agreement, he launched a
quixotic challenge to Boris Johnson for leader of the Conservative Party.
I have enjoyed Rory Stewart's writing ever since The Places in Between. This book is no exception. Whatever one's
other feelings about Stewart's politics (about which I'll have a great
deal more to say), he's a talented memoir writer with an understated and
contemplative style and a deft ability to shift from concrete description
to philosophical debate without bogging down a story. Politics on
the Edge is compelling reading at the prose level. I spent several
afternoons happily engrossed in this book and had great difficulty putting
it down.
I find Stewart intriguing since, despite being a political conservative,
he's neither a neoliberal nor any part of the new right. He is instead an
apparently-sincere throwback to a conservatism based on epistemic
humility, a veneration of rural life and long-standing traditions, and a
deep commitment to the concept of public service. Some of his principles
are baffling to me, and I think some of his political views are obvious
nonsense, but there were several things that struck me throughout this
book that I found admirable and depressingly rare in politics.
First, Stewart seems to learn from his mistakes. This goes beyond
admitting when he was wrong and appears to include a willingness to
rethink entire philosophical positions based on new experience.
I had entered Iraq supporting the war on the grounds that we could at
least produce a better society than Saddam Hussein's. It was one of
the greatest mistakes in my life. We attempted to impose programmes
made up by Washington think tanks, and reheated in air-conditioned
palaces in Baghdad a new taxation system modelled on Hong Kong; a
system of ministers borrowed from Singapore; and free ports, modelled
on Dubai. But we did it ultimately at the point of a gun, and our
resources, our abstract jargon and optimistic platitudes could not
conceal how much Iraqis resented us, how much we were failing, and how
humiliating and degrading our work had become. Our mission was a
grotesque satire of every liberal aspiration for peace, growth and
democracy.
This quote comes from the beginning of this book and is a sentiment
Stewart already expressed in The Prince of the Marshes, but he
appears to have taken this so seriously that it becomes a theme of his
political career. He not only realized how wrong he was on Iraq, he
abandoned the entire neoliberal nation-building project without abandoning
his belief in the moral obligation of international aid. And he, I think
correctly, identified a key source of the error: an ignorant,
condescending superiority that dismissed the importance of deep expertise.
Neither they, nor indeed any of the 12,000 peacekeepers and policemen
who had been posted to South Sudan from sixty nations, had spent a
single night in a rural house, or could complete a sentence in Dinka,
Nuer, Azande or Bande. And the international development strategy
written jointly between the donor nations resembled a fading mission
statement found in a new space colony, whose occupants had all been
killed in an alien attack.
Second, Stewart sincerely likes ordinary people. This shone through
The Places in Between and recurs here in his descriptions of his
constituents. He has a profound appreciation for individual people who
have spent their life learning some trade or skill, expresses thoughtful
and observant appreciation for aspects of local culture, and appears to
deeply appreciate time spent around people from wildly different social
classes and cultures than his own. Every successful politician can at
least fake gregariousness, and perhaps that's all Stewart is doing, but
there is something specific and attentive about his descriptions of other
people, including long before he decided to enter politics, that makes me
think it goes deeper than political savvy.
Third, Stewart has a visceral hatred of incompetence. I think this is the
strongest through-line of his politics in this book: Jobs in government
are serious, important work; they should be done competently and well; and
if one is not capable of doing that, one should not be in government.
Stewart himself strikes me as an insecure overachiever: fiercely
ambitious, self-critical, a bit of a micromanager (I suspect he would be
difficult to work for), but holding himself to high standards and appalled
when others do not do the same. This book is scathing towards multiple
politicians, particularly Boris Johnson whom Stewart clearly despises, but
no one comes off worse than Liz Truss.
David Cameron, I was beginning to realise, had put in charge of
environment, food and rural affairs a Secretary of State who openly
rejected the idea of rural affairs and who had little interest in
landscape, farmers or the environment. I was beginning to wonder
whether he could have given her any role she was less suited to
apart perhaps from making her Foreign Secretary. Still, I could also
sense why Cameron was mesmerised by her. Her genius lay in exaggerated
simplicity. Governing might be about critical thinking; but the new
style of politics, of which she was a leading exponent, was not. If
critical thinking required humility, this politics demanded absolute
confidence: in place of reality, it offered untethered hope; instead
of accuracy, vagueness. While critical thinking required scepticism,
open-mindedness and an instinct for complexity, the new politics
demanded loyalty, partisanship and slogans: not truth and reason but
power and manipulation. If Liz Truss worried about the consequences of
any of this for the way that government would work, she didn't reveal
it.
And finally, Stewart has a deeply-held belief in state capacity and
capability. He and I may disagree on the appropriate size and role of the
government in society, but no one would be more disgusted by an
intentional project to cripple government in order to shrink it than
Stewart.
One of his most-repeated criticisms of the UK political system in this
book is the way the cabinet is formed. All ministers and secretaries come
from members of Parliament and therefore branches of government are led by
people with no relevant expertise. This is made worse by constant cabinet
reshuffles that invalidate whatever small amounts of knowledge a minister
was able to gain in nine months or a year in post. The center portion of
this book records Stewart's time being shuffled from rural affairs to
international development to Africa to prisons, with each move
representing a complete reset of the political office and no transfer of
knowledge whatsoever.
A month earlier, they had been anticipating every nuance of Minister
Rogerson's diary, supporting him on shifts twenty-four hours a day,
seven days a week. But it was already clear that there would be no
pretence of a handover no explanation of my predecessor's strategy,
and uncompleted initiatives. The arrival of a new minister was
Groundhog Day. Dan Rogerson was not a ghost haunting my office, he was
an absence, whose former existence was suggested only by the black
plastic comb.
After each reshuffle, Stewart writes of trying to absorb briefings, do
research, and learn enough about his new responsibilities to have the hope
of making good decisions, while growing increasingly frustrated with the
system and the lack of interest by most of his colleagues in doing the
same. He wants government programs to be successful and believes success
requires expertise and careful management by the politicians, not only by
the civil servants, a position that to me both feels obviously correct and
entirely at odds with politics as currently practiced.
I found this a fascinating book to read during the accelerating collapse
of neoliberalism in the US and, to judge by current polling results, the
UK. I have a theory that the political press are so devoted to a
simplistic left-right political axis based on seating arrangements during
the French Revolution that they are missing a significant minority whose
primary political motivation is contempt for arrogant incompetence. They
could be convinced to vote for Sanders or Trump, for Polanski or Farage,
but will never vote for Biden, Starmer, Romney, or Sunak.
Such voters are incomprehensible to those who closely follow and debate
policies because their hostile reaction to the center is not about
policies. It's about lack of trust and a nebulous desire for justice.
They've been promised technocratic competence and the invisible hand of
market forces for most of their lives, and all of it looks like lies.
Everyday living is more precarious, more frustrating, more abusive and
dehumanizing, and more anxious, despite (or because of) this wholehearted
embrace of economic "freedom." They're sick of every complaint about the
increasing difficulty of life being met with accusations about their
ability and work ethic, and of being forced to endure another round of
austerity by people who then catch a helicopter ride to a party on some
billionaire's yacht.
Some of this is inherent in the deep structural weaknesses in neoliberal
ideology, but this is worse than an ideological failure. The degree to
which neoliberalism started as a project of sincere political thinkers is
arguable, but that is clearly not true today. The elite class in politics
and business is now thoroughly captured by people whose primary skill is
the marginal manipulation of complex systems for their own power and
benefit. They are less libertarian ideologues than narcissistic
mediocrities. We are governed by management consultants. They are firmly
convinced their organizational expertise is universal, and consider the
specific business of the company, or government department, irrelevant.
Given that context, I found Stewart's instinctive revulsion towards David
Cameron quite revealing. Stewart, later in the book, tries to give Cameron
some credit by citing several policy accomplishments and comparing him
favorably to Boris Johnson (which, true, is a bar Cameron probably flops
over). But I think Stewart's baffled astonishment at Cameron's vapidity
says a great deal about how we have ended up where we are. This last quote
is long, but I think it provides a good feel for Stewart's argument in
this book.
But Cameron, who was rumoured to be sceptical about nation-building
projects, only nodded, and then looking confidently up and down the
table said, "Well, at least we all agree on one extremely
straightforward and simple point, which is that our troops are doing
very difficult and important work and we should all support them."
It was an odd statement to make to civilians running humanitarian
operations on the ground. I felt I should speak. "No, with respect, we
do not agree with that. Insofar as we have focused on the troops, we
have just been explaining that what the troops are doing is often
futile, and in many cases making things worse." Two small red dots
appeared on his cheeks. Then his face formed back into a smile. He
thanked us, told us he was out of time, shook all our hands, and left
the room.
Later, I saw him repeat the same line in interviews: "the purpose of
this visit is straightforward... it is to show support for what our
troops are doing in Afghanistan". The line had been written, in
London, I assumed, and tested on focus groups. But he wanted to
convince himself it was also a position of principle.
"David has decided," one of his aides explained, when I met him later,
"that one cannot criticise a war when there are troops on the ground."
"Why?"
"Well... we have had that debate. But he feels it is a principle of
British government."
"But Churchill criticised the conduct of the Boer War; Pitt the war
with America. Why can't he criticise wars?"
"British soldiers are losing their lives in this war, and we can't
suggest they have died in vain."
"But more will die, if no one speaks up..."
"It is a principle thing. And he has made his decision. For him and
the party."
"Does this apply to Iraq too?"
"Yes. Again he understands what you are saying, but he voted to
support the Iraq War, and troops are on the ground."
"But surely he can say he's changed his mind?"
The aide didn't answer, but instead concentrated on his food. "It is
so difficult," he resumed, "to get any coverage of our trip." He
paused again. "If David writes a column about Afghanistan, we will
struggle to get it published."
"But what would he say in an article anyway?" I asked.
"We can talk about that later. But how do you get your articles on
Afghanistan published?"
I remembered how the US politicians and officials had shown their
mastery of strategy and detail. I remembered the earnestness of Gordon
Brown when I had briefed him on Iraq. Cameron seemed somehow less
serious. I wrote as much in a column in the New York Times,
saying that I was afraid the party of Churchill was becoming the party
of Bertie Wooster.
I don't know Stewart's reputation in Britain, or in the constituency that
he represented. I know he's been accused of being a self-aggrandizing
publicity hound, and to some extent this is probably true. It's hard to
find an ambitious politician who does not have that instinct. But whatever
Stewart's flaws, he can, at least, defend his politics with more substance
than a corporate motto. One gets the impression that he would respond
favorably to demonstrated competence linked to a careful argument, even if
he disagreed. Perhaps this is an illusion created by his writing, but even
if so, it's a step in the right direction.
When people become angry enough at a failing status quo, any option that
promises radical change and punishment for the current incompetents will
sound appealing. The default collapse is towards demagogues who are
skilled at expressing anger and disgust and are willing to promise simple
cures because they are indifferent to honesty. Much of the political
establishment in the US, and possibly (to the small degree that I can
analyze it from an occasional news article) in the UK, can identify the
peril of the demagogue, but they have no solution other than a return to
"politics as usual," represented by the amoral mediocrity of a McKinsey
consultant. The rare politicians who seem to believe in something, who
will argue for personal expertise and humility, who are disgusted by
incompetence and have no patience for facile platitudes, are a breath of
fresh air.
There are a lot of policies on which Stewart and I would disagree, and
perhaps some of his apparent humility is an affectation from the
rhetorical world of the 1800s that he clearly wishes he were inhabiting,
but he gives the strong impression of someone who would shoulder a
responsibility and attempt to execute it with competence and attention to
detail. He views government as a job, where coworkers should cooperate to
achieve defined goals, rather than a reality TV show. The arc of this
book, like the arc of current politics, is the victory of the reality TV
show over the workplace, and the story of Stewart's run against Boris
Johnson is hard reading because of it, but there's a portrayal here of a
different attitude towards politics that I found deeply rewarding.
If you liked Stewart's previous work, or if you want an inside look at
parliamentary politics, highly recommended. I will be thinking about this
book for a long time.
Rating: 9 out of 10
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1270 other packages on CRAN, downloaded 42 million times
(per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 650 times according
to Google Scholar.
This versions updates to the 15.2.0 upstream release made today. It
brings a few changes over Armadillo 15.0 (see below for more). It
follows the most recent RcppArmadillo
15.0.2-2 release and the Armadillo
15 upstream transition with its dual focus on moving on from C++11
and deprecation of a number of API access points. As we had a few
releases last month to manage the transition, we will sit this upgrade
out and not upload to CRAN in order to normalize our
update cadence towards the desired about six in six months (that the
CRAN Policy asks for). One can of course install as usual directly from
the GitHub
repository as well as from r-universe
which also offers binaries for all CRAN platforms.
The transition to Armadillo 15 appears to be going slowly but
steadily. We had well over 300 packages with either a need to relax the
C++11 setting and/or update away from now-deprecated API access points.
That number has been cut in half thanks to a lot of work from
a lot of package maintainers which is really appreciated! Of
course, a lot remains to be done. Issues #489 and
#491
contain the over sixty PRs and patches I prepared for all packages with
at least one reverse dependency. Most (but not all) have aided in CRAN updates, some packages are
still outstanding in terms of updates. As before meta-issue #475
regroups all the resources for the transition. If you, dear
reader, have a package that is affected and I could be of assistance
please do reach out.
The other change we made is to greatly simplify the detection and
setup of OpenMP. As before, we rely on configure to attempt
compilation of a minimal OpenMP-using program in order to pass the
success or failure onto Armadillo as a can-or-cannot use OpenMP. In
the year 2025 one of the leading consumer brands still cannot ship an OS
where this works out of the box, so we try to aide there. For all others
systems, R actually covers this pretty well and has a reliable
configuration variable that we rely upon. Just as we recommend for
downstream users of the package. This setup should be robust, but is a
change so by all means if you knowingly rely on OpenMP please test and
report back.
The detailed changes since the last CRAN release follow.
Changes
in RcppArmadillo version 15.2.0-0 (2025-10-20) (GitHub Only)
Upgraded to Armadillo release 15.2.0 (Medium Roast Deluxe)
Added rande() for generating matrices with elements
from exponential distributions
shift() has been deprecated in favour of
circshift(), for consistency with Matlab/Octave
Reworked detection of aliasing, leading to more efficient
compiled code
The discovery of a backdoor in XZ Utils in the spring of 2024 shocked the open source community, raising critical questions about software supply chain security. This post explores whether better Debian packaging practices could have detected this threat, offering a guide to auditing packages and suggesting future improvements.
The XZ backdoor in versions 5.6.0/5.6.1 made its way briefly into many major Linux distributions such as Debian and Fedora, but luckily didn t reach that many actual users, as the backdoored releases were quickly removed thanks to the heroic diligence of Andres Freund. We are all extremely lucky that he detected a half a second performance regression in SSH, cared enough to trace it down, discovered malicious code in the XZ library loaded by SSH, and reported promtly to various security teams for quick coordinated actions.
This episode makes software engineers pondering the following questions:
Why didn t any Linux distro packagers notice anything odd when importing the new XZ version 5.6.0/5.6.1 from upstream?
Is the current software supply-chain in the most popular Linux distros easy to audit?
Could we have similar backdoors lurking that haven t been detected yet?
As a Debian Developer, I decided to audit the xz package in Debian, share my methodology and findings in this post, and also suggest some improvements on how the software supply-chain security could be tightened in Debian specifically.
Note that the scope here is only to inspect how Debian imports software from its upstreams, and how they are distributed to Debian s users. This excludes the whole story of how to assess if an upstream project is following software development security best practices. This post doesn t discuss how to operate an individual computer running Debian to ensure it remains untampered as there are plenty of guides on that already.
Downloading Debian and upstream source packages
Let s start by working backwards from what the Debian package repositories offer for download. As auditing binaries is extremely complicated, we skip that, and assume the Debian build hosts are trustworthy and reliably building binaries from the source packages, and the focus should be on auditing the source code packages.
As with everything in Debian, there are multiple tools and ways to do the same thing, but in this post only one (and hopefully the best) way to do something is presented for brevity.
The first step is to download the latest version and some past versions of the package from the Debian archive, which is easiest done with debsnap. The following command will download all Debian source packages of xz-utils from Debian release 5.2.4-1 onwards:
Verifying authenticity of upstream and Debian sources using OpenPGP signatures
As seen in the output of debsnap, it already automatically verifies that the downloaded files match the OpenPGP signatures. To have full clarity on what files were authenticated with what keys, we should verify the Debian packagers signature with:
$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc
gpg: Signature made Fri Oct 3 22:04:44 2025 UTC
gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B
gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org
gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Sebastian Andrzej Siewior" [unknown]
gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown]
gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1
Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B
$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc
gpg: Signature made Fri Oct 3 22:04:44 2025 UTC
gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B
gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org
gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Sebastian Andrzej Siewior" [unknown]
gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown]
gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1
Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B
The upstream tarball signature (if available) can be verified with:
$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc
gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz'
gpg: Signature made Thu Apr 3 11:38:23 2025 UTC
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620
$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc
gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz'
gpg: Signature made Thu Apr 3 11:38:23 2025 UTC
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620
Note that this only proves that there is a key that created a valid signature for this content. The authenticity of the keys themselves need to be validated separately before trusting they in fact are the keys of these people. That can be done by checking e.g. the upstream website for what key fingerprints they published, or the Debian keyring for Debian Developers and Maintainers, or by relying on the OpenPGP web-of-trust .
Verifying authenticity of upstream sources by comparing checksums
In case the upstream in question does not publish release signatures, the second best way to verify the authenticity of the sources used in Debian is to download the sources directly from upstream and compare that the sha256 checksums match.
This should be done using the debian/watch file inside the Debian packaging, which defines where the upstream source is downloaded from. Continuing on the example situation above, we can unpack the latest Debian sources, enter and then run uscan to download:
$ tar xvf xz-utils_5.8.1-2.debian.tar.xz
...
debian/rules
debian/source/format
debian/source.lintian-overrides
debian/symbols
debian/tests/control
debian/tests/testsuite
debian/upstream/signing-key.asc
debian/watch
...
$ uscan --download-current-version --destdir /tmp
Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1
gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC
gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>"
Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz.
$ tar xvf xz-utils_5.8.1-2.debian.tar.xz
...
debian/rules
debian/source/format
debian/source.lintian-overrides
debian/symbols
debian/tests/control
debian/tests/testsuite
debian/upstream/signing-key.asc
debian/watch
...
$ uscan --download-current-version --destdir /tmp
Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1
gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC
gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>"
Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz.
The original files downloaded from upstream are now in /tmp along with the files renamed to follow Debian conventions. Using everything downloaded so far the sha256 checksums can be compared across the files and also to what the .dsc file advertised:
In the example above the checksum 0b54f79df85... is the same across the files, so it is a match.
Repackaged upstream sources can t be verified as easily
Note that uscan may in rare cases repackage some upstream sources, for example to exclude files that don t adhere to Debian s copyright and licensing requirements. Those files and paths would be listed under the Files-Excluded section in the debian/copyright file. There are also other situations where the file that represents the upstream sources in Debian isn t bit-by-bit the same as what upstream published. If checksums don t match, an experienced Debian Developer should review all package settings (e.g. debian/source/options) to see if there was a valid and intentional reason for divergence.
Reviewing changes between two source packages using diffoscope
Diffoscope is an incredibly capable and handy tool to compare arbitrary files. For example, to view a report in HTML format of the differences between two XZ releases, run:
If the changes are extensive, and you want to use a LLM to help spot potential security issues, generate the report of both the upstream and Debian packaging differences in Markdown with:
The Markdown files created above can then be passed to your favorite LLM, along with a prompt such as:
Based on the attached diffoscope output for a new Debian package version compared with the previous one, list all suspicious changes that might have introduced a backdoor, followed by other potential security issues. If there are none, list a short summary of changes as the conclusion.
Reviewing Debian source packages in version control
As of today only 93% of all Debian source packages are tracked in git on Debian s GitLab instance at salsa.debian.org. Some key packages such as Coreutils and Bash are not using version control at all, as their maintainers apparently don t see value in using git for Debian packaging, and the Debian Policy does not require it. Thus, the only reliable and consistent way to audit changes in Debian packages is to compare the full versions from the archive as shown above.
However, for packages that are hosted on Salsa, one can view the git history to gain additional insight into what exactly changed, when and why. For packages that are using version control, their location can be found in the Git-Vcs header in the debian/control file. For xz-utils the location is salsa.debian.org/debian/xz-utils.
Note that the Debian policy does not state anything about how Salsa should be used, or what git repository layout or development practices to follow. In practice most packages follow the DEP-14 proposal, and use git-buildpackage as the tool for managing changes and pushing and pulling them between upstream and salsa.debian.org.
To get the XZ Utils source, run:
$ gbp clone https://salsa.debian.org/debian/xz-utils.git
gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'
$ gbp clone https://salsa.debian.org/debian/xz-utils.git
gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'
At the time of writing this post the git history shows:
$ git log --graph --oneline
* bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2
* 4b769547 d: Remove the symlinks from -dev package.
* a39f3428 Correct the nocheck build profile
* 1b806b8d Import Debian changes 5.8.1-1.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h
* 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder
* 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding
* 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread
* d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
* c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state
* 831b55b9 liblzma: mt dec: Fix a comment
* b9d168ee liblzma: Add assertions to lzma_bufcpy()
* c8e0a489 DOS: Update Makefile to fix the build
* 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers
* 7ce38b31 Update THANKS
* 688e51bd Translations: Update the Croatian translation
* a6b54dde Prepare 5.8.0-1.
* 77d9470f Add 5.8 symbols.
* 9268eb66 Import 5.8.0
* 6f85ef4f Update upstream source from tag 'upstream/5.8.0'
\ \
* afba662b New upstream version 5.8.0
/
* 173fb5c6 doc/SHA256SUMS: Add 5.8.0
* db9258e8 Bump version and soname for 5.8.0
* bfb752a3 Add NEWS for 5.8.0
* 6ccbb904 Translations: Run "make -C po update-po"
* 891a5f05 Translations: Run po4a/update-po
* 4f52e738 Translations: Partially fix overtranslation in Serbian man pages
* ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage
* 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()
$ git log --graph --oneline
* bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2
* 4b769547 d: Remove the symlinks from -dev package.
* a39f3428 Correct the nocheck build profile
* 1b806b8d Import Debian changes 5.8.1-1.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h
* 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder
* 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding
* 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread
* d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
* c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state
* 831b55b9 liblzma: mt dec: Fix a comment
* b9d168ee liblzma: Add assertions to lzma_bufcpy()
* c8e0a489 DOS: Update Makefile to fix the build
* 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers
* 7ce38b31 Update THANKS
* 688e51bd Translations: Update the Croatian translation
* a6b54dde Prepare 5.8.0-1.
* 77d9470f Add 5.8 symbols.
* 9268eb66 Import 5.8.0
* 6f85ef4f Update upstream source from tag 'upstream/5.8.0'
\ \
* afba662b New upstream version 5.8.0
/
* 173fb5c6 doc/SHA256SUMS: Add 5.8.0
* db9258e8 Bump version and soname for 5.8.0
* bfb752a3 Add NEWS for 5.8.0
* 6ccbb904 Translations: Run "make -C po update-po"
* 891a5f05 Translations: Run po4a/update-po
* 4f52e738 Translations: Partially fix overtranslation in Serbian man pages
* ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage
* 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()
This shows both the changes on the debian/unstable branch as well as the intermediate upstream import branch, and the actual real upstream development branch. See my Debian source packages in git explainer for details of what these branches are used for.
To only view changes on the Debian branch, run git log --graph --oneline --first-parent or git log --graph --oneline -- debian.
The Debian branch should only have changes inside the debian/ subdirectory, which is easy to check with:
If the upstream in question signs commits or tags, they can be verified with e.g.:
$ git verify-tag v5.6.2
gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: issuer "lasse.collin@tukaani.org"
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired]
gpg: Note: This key has expired!
$ git verify-tag v5.6.2
gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: issuer "lasse.collin@tukaani.org"
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired]
gpg: Note: This key has expired!
The main benefit of reviewing changes in git is the ability to see detailed information about each individual change, instead of just staring at a massive list of changes without any explanations. In this example, to view all the upstream commits since the previous import to Debian, one would view the commit range from afba662b New upstream version 5.8.0 to fa1e8796 New upstream version 5.8.1 with git log --reverse -p afba662b...fa1e8796. However, a far superior way to review changes would be to browse this range using a visual git history viewer, such as gitk. Either way, looking at one code change at a time and reading the git commit message makes the review much easier.
Comparing Debian source packages to git contents
As stated in the beginning of the previous section, and worth repeating, there is no guarantee that the contents in the Debian packaging git repository matches what was actually uploaded to Debian. While the tag2upload project in Debian is getting more and more popular, Debian is still far from having any system to enforce that the git repository would be in sync with the Debian archive contents.
To detect such differences we can run diff across the Debian source packages downloaded with debsnap earlier (path source-xz-utils/xz-utils_5.8.1-2.debian) and the git repository cloned in the previous section (path xz-utils):
diff$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/
diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog
--- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700
+++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700
@@ -5,7 +5,7 @@
* Remove the symlinks from -dev, pointing to the lib package.
(Closes: #1109354)
- -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200
$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/
diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog
--- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700
+++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700
@@ -5,7 +5,7 @@
* Remove the symlinks from -dev, pointing to the lib package.
(Closes: #1109354)
- -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200
In the case above diff revealed that the timestamp in the changelog in the version uploaded to Debian is different from what was committed to git. This is not malicious, just a mistake by the maintainer who probably didn t run gbp tag immediately after upload, but instead some dch command and ended up with having a different timestamps in the git compared to what was actually uploaded to Debian.
Creating syntetic Debian packaging git repositories
If no Debian packaging git repository exists, or if it is lagging behind what was uploaded to Debian s archive, one can use git-buildpackage s import-dscs feature to create synthetic git commits based on the files downloaded by debsnap, ensuring the git contents fully matches what was uploaded to the archive. To import a single version there is gbp import-dsc (no s at the end), of which an example invocation would be:
$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc
Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'
$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc
Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'
Example commit history from a repository with commits added with gbp import-dsc:
An online example repository with only a few missing uploads added using gbp import-dsc can be viewed at salsa.debian.org/otto/xz-utils-2025-09-29/-/network/debian%2Funstable
An example repository that was fully crafted using gbp import-dscs can be viewed at salsa.debian.org/otto/xz-utils-gbp-import-dscs-debsnap-generated/-/network/debian%2Flatest.
There exists also dgit, which in a similar way creates a synthetic git history to allow viewing the Debian archive contents via git tools. However, its focus is on producing new package versions, so fetching a package with dgit that has not had the history recorded in dgit earlier will only show the latest versions:
$ dgit clone xz-utils
canonical suite name for unstable is sid
starting new git history
last upload to archive: NO git hash
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz...
dpkg-source: info: extracting xz-utils in unpacked
dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz
dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz
synthesised git commit from .dsc 5.8.1-2
HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium
dgit ok: ready for work in xz-utils
$ dgit/sid git log --graph --oneline
* f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid)
\
* 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago
* 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago
$ dgit clone xz-utils
canonical suite name for unstable is sid
starting new git history
last upload to archive: NO git hash
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz...
dpkg-source: info: extracting xz-utils in unpacked
dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz
dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz
synthesised git commit from .dsc 5.8.1-2
HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium
dgit ok: ready for work in xz-utils
$ dgit/sid git log --graph --oneline
* f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid)
\
* 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago
* 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago
Unlike git-buildpackage managed git repositories, the dgit managed repositories cannot incorporate the upstream git history and are thus less useful for auditing the full software supply-chain in git.
Comparing upstream source packages to git contents
Equally important to the note in the beginning of the previous section, one must also keep in mind that the upstream release source packages, often called release tarballs, are not guaranteed to have the exact same contents as the upstream git repository. Projects might strip out test data or extra development files from their release tarballs to avoid shipping unnecessary files to users, or projects might add documentation files or versioning information into the tarball that isn t stored in git. While a small minority, there are also upstreams that don t use git at all, so the plain files in a release tarball is still the lowest common denominator for all open source software projects, and exporting and importing source code needs to interface with it.
In the case of XZ, the release tarball has additional version info and also a sizeable amount of pregenerated compiler configuration files. Detecting and comparing differences between git contents and tarballs can of course be done manually by running diff across an unpacked tarball and a checked out git repository. If using git-buildpackage, the difference between the git contents and tarball contents can be made visible directly in the import commit.
In this XZ example, consider this git history:
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
The commit a522a226 was the upstream release commit, which upstream also tagged v5.8.1. The merge commit 2808ec2d applied the new upstream import branch contents on the Debian branch. Between these is the special commit fa1e8796 New upstream version 5.8.1 tagged upstream/v5.8. This commit and tag exists only in the Debian packaging repository, and they show what is the contents imported into Debian. This is generated automatically by git-buildpackage when running git import-orig --uscan for Debian packages with the correct settings in debian/gbp.conf. By viewing this commit one can see exactly how the upstream release tarball differs from the upstream git contents (if at all).
In the case of XZ, the difference is substantial, and shown below in full as it is very interesting:
To be able to easily inspect exactly what changed in the release tarball compared to git release tag contents, the best tool for the job is Meld, invoked via git difftool --dir-diff fa1e8796^..fa1e8796.
To compare changes across the new and old upstream tarball, one would need to compare commits afba662b New upstream version 5.8.0 and fa1e8796 New upstream version 5.8.1 by running git difftool --dir-diff afba662b..fa1e8796.
With all the above tips you can now go and try to audit your own favorite package in Debian and see if it is identical with upstream, and if not, how it differs.
Should the XZ backdoor have been detected using these tools?
The famous XZ Utils backdoor (CVE-2024-3094) consisted of two parts: the actual backdoor inside two binary blobs masqueraded as a test files (tests/files/bad-3-corrupt_lzma2.xz, tests/files/good-large_compressed.lzma), and a small modification in the build scripts (m4/build-to-host.m4) to extract the backdoor and plant it into the built binary. The build script was not tracked in version control, but generated with GNU Autotools at release time and only shipped as additional files in the release tarball.
The entire reason for me to write this post was to ponder if a diligent engineer using git-buildpackage best practices could have reasonably spotted this while importing the new upstream release into Debian. The short answer is no . The malicious actor here clearly anticipated all the typical ways anyone might inspect both git commits, and release tarball contents, and masqueraded the changes very well and over a long timespan.
First of all, XZ has for legitimate reasons for several carefully crafted .xz files as test data to help catch regressions in the decompression code path. The test files are shipped in the release so users can run the test suite and validate that the binary is built correctly and xz works properly. Debian famously runs massive amounts of testing in its CI and autopkgtest system across tens of thousands of packages to uphold high quality despite frequent upgrades of the build toolchain and while supporting more CPU architectures than any other distro. Test data is useful and should stay.
When git-buildpackage is used correctly, the upstream commits are visible in the Debian packaging for easy review, but the commit cf44e4b that introduced the test files does not deviate enough from regular sloppy coding practices to really stand out. It is unfortunately very common for git commit to lack a message body explaining why the change was done, and to not be properly atomic with test code and test data together in the same commit, and for commits to be pushed directly to mainline without using code reviews (the commit was not part of any PR in this case). Only another upstream developer could have spotted that this change is not on par to what the project expects, and that the test code was never added, only test data, and thus that this commit was not just a sloppy one but potentially malicious.
Secondly, the fact that a new Autotools file appeared (m4/build-to-host.m4) in the XZ Utils 5.6.0 is not suspicious. This is perfectly normal for Autotools. In fact, starting from XZ Utils version 5.8.1 it is now shipping a m4/build-to-host.m4 file that it actually uses now.
Spotting that there is anything fishy is practically impossible by simply reading the code, as Autotools files are full custom m4 syntax interwoven with shell script, and there are plenty of backticks () that spawn subshells and evals that execute variable contents further, which is just normal for Autotools. Russ Cox s XZ post explains how exactly the Autotools code fetched the actual backdoor from the test files and injected it into the build.
There is only one tiny thing that maybe a very experienced Autotools user could potentially have noticed: the serial 30 in the version header is way too high. In theory one could also have noticed this Autotools file deviates from what other packages in Debian ship with the same filename, such as e.g. the serial 3, serial 5a or 5b versions. That would however require and an insane amount extra checking work, and is not something we should plan to start doing. A much simpler solution would be to simply strongly recommend all open source projects to stop using Autotools to eventually get rid of it entirely.
Not detectable with reasonable effort
While planting backdoors is evil, it is hard not to feel some respect to the level of skill and dedication of the people behind this. I ve been involved in a bunch of security breach investigations during my IT career, and never have I seen anything this well executed.
If it hadn t slowed down SSH by ~500 milliseconds and been discovered due to that, it would most likely have stayed undetected for months or years. Hiding backdoors in closed source software is relatively trivial, but hiding backdoors in plain sight in a popular open source project requires some unusual amount of expertise and creativity as shown above.
Is the software supply-chain in Debian easy to audit?
While maintaining a Debian package source using git-buildpackage can make the package history a lot easier to inspect, most packages have incomplete configurations in their debian/gbp.conf, and thus their package development histories are not always correctly constructed or uniform and easy to compare. The Debian Policy does not mandate git usage at all, and there are many important packages that are not using git at all. Additionally the Debian Policy also allows for non-maintainers to upload new versions to Debian without committing anything in git even for packages where the original maintainer wanted to use git. Uploads that bypass git unfortunately happen surpisingly often.
Because of the situation, I am afraid that we could have multiple similar backdoors lurking that simply haven t been detected yet. More audits, that hopefully also get published openly, would be welcome! More people auditing the contents of the Debian archives would probably also help surface what tools and policies Debian might be missing to make the work easier, and thus help improve the security of Debian s users, and improve trust in Debian.
Is Debian currently missing some software that could help detect similar things?
To my knowledge there is currently no system in place as part of Debian s QA or security infrastructure to verify that the upstream source packages in Debian are actually from upstream. I ve come across a lot of packages where the debian/watch or other configs are incorrect and even cases where maintainers have manually created upstream tarballs as it was easier than configuring automation to work. It is obvious that for those packages the source tarball now in Debian is not at all the same as upstream. I am not aware of any malicious cases though (if I was, I would report them of course).
I am also aware of packages in the Debian repository that are misconfigured to be of type 1.0 (native) packages, mixing the upstream files and debian/ contents and having patches applied, while they actually should be configured as 3.0 (quilt), and not hide what is the true upstream sources. Debian should extend the QA tools to scan for such things. If I find a sponsor, I might build it myself as my next major contribution to Debian.
In addition to better tooling for finding mismatches in the source code, Debian could also have better tooling for tracking in built binaries what their source files were, but solutions like Fraunhofer-AISEC s supply-graph or Sony s ESSTRA are not practical yet. Julien Malka s post about NixOS discusses the role of reproducible builds, which may help in some cases across all distros.
Or, is Debian missing some policies or practices to mitigate this?
Perhaps more importantly than more security scanning, the Debian Developer community should switch the general mindset from anyone is free to do anything to valuing having more shared workflows. The ability to audit anything is severely hampered by the fact that there are so many ways to do the same thing, and distinguishing what is a normal deviation from a malicious deviation is too hard, as the normal can basically be almost anything.
Also, as there is no documented and recommended default workflow, both those who are old and new to Debian packaging might never learn any one optimal workflow, and end up doing many steps in the packaging process in a way that kind of works, but is actually wrong or unnecessary, causing process deviations that look malicious, but turn out to just be a result of not fully understanding what would have been the right way to do something.
In the long run, once individual developers workflows are more aligned, doing code reviews will become a lot easier and smoother as the excess noise of workflow differences diminishes and reviews will feel much more productive to all participants. Debian fostering a culture of code reviews would allow us to slowly move from the current practice of mainly solo packaging work towards true collaboration forming around those code reviews.
I have been promoting increased use of Merge Requests in Debian already for some time, for example by proposing DEP-18: Encourage Continuous Integration and Merge Request based Collaboration for Debian packages. If you are involved in Debian development, please give a thumbs up in dep-team/deps!21 if you want me to continue promoting it.
Can we trust open source software?
Yes and I would argue that we can only trust open source software. There is no way to audit closed source software, and anyone using e.g. Windows or MacOS just have to trust the vendor s word when they say they have no intentional or accidental backdoors in their software. Or, when the news gets out that the systems of a closed source vendor was compromised, like Crowdstrike some weeks ago, we can t audit anything, and time after time we simply need to take their word when they say they have properly cleaned up their code base.
In theory, a vendor could give some kind of contractual or financial guarantee to its customer that there are no preventable security issues, but in practice that never happens. I am not aware of a single case of e.g. Microsoft or Oracle would have paid damages to their customers after a security flaw was found in their software. In theory you could also pay a vendor more to have them focus more effort in security, but since there is no way to verify what they did, or to get compensation when they didn t, any increased fees are likely just pocketed as increased profit.
Open source is clearly better overall. You can, if you are an individual with the time and skills, audit every step in the supply-chain, or you could as an organization make investments in open source security improvements and actually verify what changes were made and how security improved.
If your organisation is using Debian (or derivatives, such as Ubuntu) and you are interested in sponsoring my work to improve Debian, please reach out.
A side project I have been working on a little since last winter and
which explores extending duckdb with
mlpack is now public at the duckdb-mlpack
repo.
duckdb is an excellent small (as
in runs as a self-contained binary ) database engine with both a focus
on analytical payloads (OLAP rather than OLTP) and an impressive number
of already bolted-on extensions (for example for cloud data access)
delivered as a single-build C++ executable (or of course as a library
used from other front-ends). mlpack is
an excellent C++ library containing many/most machine learning
algorithms, also built in a self-contained manner (or library) making it
possible to build compact yet powerful binaries, or to embed (as opposed
to other ML framework accessed from powerful but not lightweight
run-times such as Python or R). The compact build aspect as well as the
common build tools (C++, cmake) make these two a natural candidate for
combining them. Moreover, duckdb is a
champion of data access, management and control and the complementary
machine learning insights and predictions offered by mlpack are fully complementary and hence
fit this rather well.
duckdb also has a very robust and
active extension system. To use it, one starts from a template
repository and its use this template button, runs a script and can
then start experimenting. I have now grouped my initial start and test
functions into a separate repository duckdb-example-extension
to keep the duckdb-mlpack
one focused on the extend to mlpack aspect.
duckdb-mlpack
is right an MVP , i.e. a minimally viable product (or demo). It just
runs the adaboost classifier but does so on any dataset
fitting the rectangular setup with columns of features (real valued)
and a final column (integer valued) of labels. I had hope to use two
select queries for both features and then labels but it
turns a table function (returning a table of data from a query) can
only run one select *. So the basic demo, also on the repo
README is now to run the following script (where the
SELECT * FROM mlpack_adaboost((SELECT * FROM D)); is the
key invocation of the added functionality):
#!/bin/bashcat<<EOFbuild/release/duckdbSET autoinstall_known_extensions=1;SET autoload_known_extensions=1; # for httpfsCREATE TEMP TABLE Xd AS SELECT * FROM read_csv("https://mlpack.org/datasets/iris.csv");CREATE TEMP TABLE X AS SELECT row_number() OVER () AS id, * FROM Xd;CREATE TEMP TABLE Yd AS SELECT * FROM read_csv("https://mlpack.org/datasets/iris_labels.csv");CREATE TEMP TABLE Y AS SELECT row_number() OVER () AS id, CAST(column0 AS double) as label FROM Yd;CREATE TEMP TABLE D AS SELECT * FROM X INNER JOIN Y ON X.id = Y.id;ALTER TABLE D DROP id;ALTER TABLE D DROP id_1;CREATE TEMP TABLE A AS SELECT * FROM mlpack_adaboost((SELECT * FROM D));SELECT COUNT(*) as n, predicted FROM A GROUP BY predicted;EOF
(Note that this requires the httpfs extension. So when
you build from a freshly created extension repository you may be ahead
of the most recent release of duckdb by a few commits. It
is easy to check out the most recent release tag (or maybe the one you
are running for your local duckdb binary) to take advantage
of the extensions you likely already have for that version. So here, and
in the middle of October 2025, I picked v1.4.1 as I run
duckdb version 1.4.1 on my box.)
There are many other neat duckdb
extensions. The core ones are regrouped here
while a list of community extensions is here
and here.
For this (still more minimal) extension, I added a few TODO items to
the README.md:
More examples of model fitting and prediction
Maybe set up model serialization into table to predict on new
data
Ideally: Work out how to SELECT from multiple tabels,
or else maybe SELECT into temp. tables and pass temp. table
names into routine
Maybe add mlpack as a git submodule
Please reach out if you are interested in working on any of this.
The seventeenth release of the qlcal package
arrivied at CRAN today, once
again following a QuantLib
release as 1.40 came out this morning.
qlcal
delivers the calendaring parts of QuantLib. It is provided (for the R
package) as a set of included files, so the package is self-contained
and does not depend on an external QuantLib library (which can be
demanding to build). qlcal covers
over sixty country / market calendars and can compute holiday lists, its
complement (i.e. business day lists) and much more. Examples
are in the README at the repository, the package page,
and course at the CRAN package
page.
This releases mainly synchronizes qlcal with
the QuantLib release 1.40. Only
one country calendar got updated; the diffstat
looks larger as the URL part of the copyright got updated throughout. We
also updated the URL for the GPL-2 badge: when CRAN checks this, they always hit
a timeout as the FSF server possibly keeps track of incoming requests;
we now link to version from the R Licenses page to avoid
this.
Changes in version 0.0.17
(2025-07-14)
Synchronized with QuantLib 1.40 released today
Calendar updates for Singapore
URL update in README.md
Courtesy of my CRANberries, there
is a diffstat report for this
release. See the project page
and package documentation for more details, and more examples.
Avoiding
5XX errors by adjusting Load Balancer Idle Timeout
Recently I faced a problem in production where a client was running a
RabbitMQ server behind the Load Balancers we provisioned and the TCP
connections were closed every minute.
My team is responsible for the LBaaS (Load Balancer as a Service)
product and this Load Balancer was an Envoy proxy provisioned by our
control plane.
The error was similar to this:
At first glance, the issue is simple: the Load Balancer's idle
timeout is shorter than the RabbitMQ heartbeat interval.
The idle timeout is the time at which a downstream or upstream
connection will be terminated if there are no active streams. Heartbeats
generate periodic network traffic to prevent idle TCP connections from
closing prematurely.
Adjusting these timeout settings to align properly solved the
issue.
However, what I want to explore in this post are other similar
scenarios where it's not so obvious that the idle timeout is the
problem. Introducing an extra network layer, such as an Envoy proxy, can
introduce unpredictable behavior across your services, like intermittent
5XX errors.
To make this issue more concrete, let's look at a minimal,
reproducible setup that demonstrates how adding an Envoy proxy can lead
to sporadic errors.
Reproducible setup
I'll be using the following tools:
I'll be running experiments with two different
envoy.yaml configurations: one that uses Envoy's TCP proxy,
and another that uses Envoy's HTTP connection manager.
Here's the simplest Envoy TCP proxy setup: a listener on port 8000
forwarding traffic to a backend running on port 8080.
The default idle timeout if not otherwise specified is 1 hour, which
is the case here.
The backend setup is simple as well:
package mainimport("fmt""net/http""time")func helloHandler(w http.ResponseWriter, r *http.Request) w.Write([]byte("Hello from Go!"))func main() http.HandleFunc("/", helloHandler) server := http.Server Addr:":8080", IdleTimeout:3* time.Second, fmt.Println("Starting server on :8080")panic(server.ListenAndServe())
The IdleTimeout is set to 3 seconds to make it easier to test.
Now, oha is the perfect tool to generate the HTTP
requests for this test. The Load test is not meant to stress this setup,
the idea is to wait long enough so that some requests are closed. The
burst-delay feature will help with that:
I'm running the Load test for 30 seconds, sending 100 requests at
three-second intervals. I also use the -w option to wait
for ongoing requests when the duration is reached. The result looks like
this:
We had 886 responses with status code 200 and 64 connections closed.
The backend terminated 64 connections while the load balancer still had
active requests directed to it.
Let's change the Load Balancer idle_timeout to two
seconds.
filter_chains:-filters:-name: envoy.filters.network.tcp_proxytyped_config:"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxystat_prefix: go_server_tcpcluster: go_server_clusteridle_timeout: 2s # <--- NEW LINE
Run the same test again.
Great! Now all the requests worked.
This is a common issue, not specific to Envoy Proxy or the setup
shown earlier. Major cloud providers have all documented it.
AWS
troubleshoot guide for Application Load Balancers says this:
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.Google
troubleshoot guide for Application Load Balancers mention this as
well:
Verify that the keepalive configuration parameter for the HTTP server software running on the backend instance is not less than the keepalive timeout of the load balancer, whose value is fixed at 10 minutes (600 seconds) and is not configurable.The load balancer generates an HTTP 5XX response code when the connection to the backend has unexpectedly closed while sending the HTTP request or before the complete HTTP response has been received. This can happen because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed keepalive timeout of the load balancer. Ensure that the keepalive timeout configuration for HTTP server software on each backend is set to slightly greater than 10 minutes (the recommended value is 620 seconds).RabbitMQ
docs also warn about this:
Certain networking tools (HAproxy, AWS ELB) and equipment (hardware load balancers) may terminate "idle" TCP connections when there is no activity on them for a certain period of time. Most of the time it is not desirable.
Most of them are talking about Application Load Balancers and the
test I did was using a Network Load Balancer. For the sake of
completeness, I will do the same test but using Envoy's HTTP connection
manager.
The updated envoy.yaml:
The yaml above is an example of a service proxying HTTP from
0.0.0.0:8000 to 0.0.0.0:8080. The only difference from a minimal
configuration is that I enabled access logs.
Let's run the same tests with oha.
Even thought the success rate is 100%, the status code distribution
show some responses with status code 503. This is the case where is not
that obvious that the problem is related to idle timeout.
However, it's clear when we look the Envoy access logs:
UC is the short name for
UpstreamConnectionTermination. This means the upstream,
which is the golang server, terminated the connection.
To fix this once again, the Load Balancer idle timeout needs to
change:
clusters:-name: go_server_clustertype: STATICtyped_extension_protocol_options: # <--- NEW BLOCKenvoy.extensions.upstreams.http.v3.HttpProtocolOptions:"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptionscommon_http_protocol_options:idle_timeout: 2s # <--- NEW VALUEexplicit_http_config:http_protocol_options:
Finally, the sporadic 503 errors are over:
To Sum Up
Here's an example of the values my team recommends to our
clients:
Key Takeaways:
The Load Balancer idle timeout should be less than the backend
(upstream) idle/keepalive timeout.
When we are working with long lived connections, the client
(downstream) should use a keepalive smaller than the LB idle
timeout.
Debian LTS
This was my hundred-thirty-fifth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:
[DLA 4168-2] openafs regression update to fix an incomplete patch in the previous upload.
[DSA 5998-1] cups security update to fix two CVEs related to a authentication bypass and a denial of service.
[DLA 4298-1] cups security update to fix two CVEs related to a authentication bypass and a denial of service.
[DLA 4304-1] cjson security update to fix one CVE related to an out-of-bounds memory access.
[DLA 4307-1] jq security update to fix one CVE related to a heap buffer overflow.
[DLA 4308-1] corosync security update to fix one CVE related to a stack-based buffer overflow.
An upload of spim was not needed, as the corresponding CVE could be marked as ignored.
I also started to work on an open-vm-tools and attended the monthly LTS/ELTS meeting.
Debian ELTS
This month was the eighty-sixth ELTS month. During my allocated time I uploaded or worked on:
[ELA-1512-1] cups security update to fix two CVEs in Buster and Stretch, related to a authentication bypass and a denial of service.
[ELA-1520-1] jq security update to fix one CVE in Buster and Stretch, related to a heap buffer overflow.
[ELA-1524-1] corosync security update to fix one CVE in Buster and Stretch, related to a stack-based buffer overflow.
[ELA-1527-1] mplayer security update to fix ten CVEs in Stretch, distributed all over the code.
The CVEs for open-vm-tools could be marked as not-affeceted as the corresponding plugin was not yet available. I also attended the monthly LTS/ELTS meeting.
Debian Printing
This month I uploaded a new upstream version or a bugfix version of:
misc
The main topic of this month has been gcc15 and cmake4, so my upload rate was extra high. This month I uploaded a new upstream version or a bugfix version of:
I wonder what MBF will happen next, I guess the /var/lock-issue will be a good candidate.
On my fight against outdated RFPs, I closed 30 of them in September. Meanwhile only 3397 are still open, so don t hesitate to help closing one or another.
FTP master
This month I accepted 294 and rejected 28 packages. The overall number of packages that got accepted was 294.
Deep Black is a far-future science fiction novel and the direct
sequel to Artifact Space. You do not
want to start here. I regretted not reading the novels closer together and
had to refresh my memory of what happened in the first book.
The shorter fiction in Beyond the Fringe
takes place between the two series novels and leads into some of the
events in this book, although reading it is optional.
Artifact Space left Marca Nbaro at the farthest point of the voyage
of the Greatship Athens, an unexpected heroine and now
well-integrated into the crew. On a merchant ship, however, there's always
more work to be done after a heroic performance. Deep Black opens
with that work: repairs from the events of the first book, the
never-ending litany of tasks required to keep the ship running smoothly,
and of course the trade with aliens that drew them so far out into the
Deep Black.
We knew early in the first book that this wouldn't be the simple, if long,
trading voyage that most of the crew of the Athens was expecting,
but now they have to worry about an unsettling second group of aliens on
top of a potential major war between human factions. They don't yet have
the cargo they came for, they have to reconstruct their trading post, and
they're a very long way from home. Marca also knows, at this point in the
story, that this voyage had additional goals from the start. She will
slowly gain a more complete picture of those goals during this novel.
Artifact Space was built around one of the most satisfying plots in
military science fiction (at least to me): a protagonist who benefits
immensely from the leveling effect and institutional inclusiveness of the
military slowly discovering that, when working at its best, the military
can be a true meritocracy. (The merchant marine of the Athens is
not military, precisely, since it's modeled on the trading ships of
Venice, but it's close enough for the purposes of this plot.) That's not a
plot that lasts into a sequel, though, so Cameron had to find a new spine
for the second half of the story. He chose first contact (of a sort) and
space battle.
The space battle parts are fine. I read a ton of children's World War II
military fiction when I was a boy, and I always preferred the naval
battles to the land battles. This part of Deep Black reminded me of
those naval battles, particularly a book whose title escapes me about the
Arctic convoys to the Soviet Union. I'm more interested in character than
military adventure these days, but every once in a while I enjoy reading
about a good space battle. This was not an exemplary specimen of the
genre, but it delivered on all the required elements.
The first contact part was more original, in part because Cameron chose an
interesting medium ground between total incomprehensibility and universal
translators. He stuck with the frustrations of communication for
considerably longer than most SF authors are willing to write, and it
worked for me. This is the first book I've read in a while where
superficial alien fluency with the mere words of a human language masks
continuing profound mutual incomprehension. The communication difficulties
are neither malicious nor a setup for catastrophic misunderstanding, but
an intrinsic part of learning about a truly alien species. I liked this,
even though it makes for slower and more frustrating progress. It felt
more believable than a lot of first contact, and it forced the characters
to take risks and act on hunches and then live with the consequences.
One of the other things that Cameron does well is maintain the steady
rhythm of life on a working ship as a background anchor to the story. I've
read a lot of science fiction that shows the day-to-day routine only until
something more interesting and plot-focused starts happening and then
seems to forget about it entirely. Not here. Marca goes through intense
and adrenaline-filled moments requiring risk and fast reactions, and then
has to handle promotion write-ups, routine watches, and studying for
advancement. Cameron knows that real battles involve long periods of
stressful waiting and incorporates them into the book without making them
too boring, which requires a lot of writing skill.
I prefer the emotional magic of finding a place where one belongs, so I
was not as taken with Deep Black as I was with Artifact
Space, but that's the inevitable result of plot progression and not
really a problem with this book. Marca is absurdly central to the story in
ways that have a whiff of "chosen one" dynamics, but if one can suspend
one's disbelief about that, the rest of the book is solid. This is,
fundamentally, a book about large space battles, so save it when you're in
the mood for that sort of story, but it was a satisfying continuation of
the series. I will definitely keep reading.
Recommended if you enjoyed Artifact Space. If you didn't,
Deep Black isn't going to change your mind.
Followed by Whalesong, which is not yet released (and is currently
in some sort of limbo for pre-orders in the US, which I hope will clear
up).
Rating: 7 out of 10
In December 2024, I went on a trip through four countries - Singapore, Malaysia, Brunei, and Vietnam - with my friend Badri. This post covers our experiences in Singapore.
I took an IndiGo flight from Delhi to Singapore, with a layover in Chennai. At the Chennai airport, I was joined by Badri. We had an early morning flight from Chennai that would land in Singapore in the afternoon. Within 48 hours of our scheduled arrival in Singapore, we submitted an arrival card online. At immigration, we simply needed to scan our passports at the gates, which opened automatically to let us through, and then give our address to an official nearby. The process was quick and smooth, but it unfortunately meant that we didn t get our passports stamped by Singapore.
Before I left the airport, I wanted to visit the nature-themed park with a fountain I saw in pictures online. It is called Jewel Changi, and it took quite some walking to get there. After reaching the park, we saw a fountain that could be seen from all the levels. We roamed around for a couple of hours, then proceeded to the airport metro station to get to our hotel.
A shot of Jewel Changi. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
There were four ATMs on the way to the metro station, but none of them provided us with any cash. This was the first country (outside India, of course!) where my card didn t work at ATMs.
To use the metro, one can tap the EZ-Link card or bank cards at the AFC gates to get in. You cannot buy tickets using cash. Before boarding the metro, I used my credit card to get Badri an EZ-Link card from a vending machine. It was 10 Singapore dollars ( 630) - 5 for the card, and 5 for the balance. I had planned to use my Visa credit card to pay for my own fare. I was relieved to see that my card worked, and I passed through the AFC gates.
We had booked our stay at a hostel named Campbell s Inn, which was the cheapest we could find in Singapore. It was 1500 per night for dorm beds. The hostel was located in Little India. While Little India has an eponymous metro station, the one closest to our hostel was Rochor.
On the way to the hostel, we found out that our booking had been canceled.
We had booked from the Hostelworld website, opting to pay the deposit in advance and to pay the balance amount in person upon reaching. However, Hostelworld still tried to charge Badri s card again before our arrival. When the unauthorized charge failed, they sent an automatic message saying we tried to charge and to contact them soon to avoid cancellation, which we couldn t do as we were in the plane.
Despite this, we went to the hostel to check the status of our booking.
The trip from the airport to Rochor required a couple of transfers. It was 2 Singapore dollars (approx. 130) and took approximately an hour.
Upon reaching the hostel, we were informed that our booking had indeed been canceled, and were not given any reason for the cancelation. Furthermore, no beds were available at the hostel for us to book on the spot.
We decided to roam around and look for accommodation at other hostels in the area. Soon, we found a hostel by the name of Snooze Inn, which had two beds available. It was 36 Singapore dollars per person (around 2300) for a dormitory bed. Snooze Inn advertised supporting RuPay cards and UPI. Some other places in that area did the same. We paid using my card. We checked in and slept for a couple of hours after taking a shower.
By the time we woke up, it was dark. We met Praveen s friend Sabeel to get my FLX1 phone. We also went to Mustafa Center nearby to exchange Indian rupees for Singapore dollars. Mustafa Center also had a shopping center with shops selling electronic items and souvenirs, among other things. When we were dropping off Sabeel at a bus stop, we discovered that the bus stops in Singapore had a digital board mentioning the bus routes for the stop and the number of minutes each bus was going to take.
In addition to an organized bus system, Singapore had good pedestrian infrastructure. There were traffic lights and zebra crossings for pedestrians to cross the roads. Unlike in Indian cities, rules were being followed. Cars would stop for pedestrians at unmanaged zebra crossings; pedestrians would in turn wait for their crossing signal to turn green before attempting to walk across. Therefore, walking in Singapore was easy.
Traffic rules were taken so seriously in Singapore I (as a pedestrian) was afraid of unintentionally breaking them, which could get me in trouble, as breaking rules is dealt with heavy fines in the country. For example, crossing roads without using a marked crossing (while being within 50 meters of it) - also known as jaywalking - is an offence in Singapore.
Moreover, the streets were litter-free, and cleanliness seemed like an obsession.
After exploring Mustafa Center, we went to a nearby 7-Eleven to top up Badri s EZ-Link card. He gave 20 Singapore dollars for the recharge, which credited the card by 19.40 Singapore dollars (0.6 dollars being the recharge fee).
When I was planning this trip, I discovered that the World Chess Championship match was being held in Singapore. I seized the opportunity and bought a ticket in advance. The next day - the 5th of December - I went to watch the 9th game between Gukesh Dommaraju of India and Ding Liren of China. The venue was a hotel on Sentosa Island, and the ticket was 70 Singapore dollars, which was around 4000 at the time.
We checked out from our hostel in the morning, as we were planning to stay with Badri s aunt that night. We had breakfast at a place in Little India. Then we took a couple of buses, followed by a walk to Sentosa Island. Paying the fare for the buses was similar to the metro - I tapped my credit card in the bus, while Badri tapped his EZ-Link card. We also had to tap it while getting off.
If you are tapping your credit card to use public transport in Singapore, keep in mind that the total amount of all the trips taken on a day is deducted at the end. This makes it hard to determine the cost of individual trips. For example, I could take a bus and get off after tapping my card, but I would have no way to determine how much this journey cost.
When you tap in, the maximum fare amount gets deducted. When you tap out, the balance amount gets refunded (if it s a shorter journey than the maximum fare one). So, there is incentive for passengers not to get off without tapping out. Going by your card statement, it looks like all that happens virtually, and only one statement comes in at the end. Maybe this combining only happens for international cards.
We got off the bus a kilometer away from Sentosa Island and walked the rest of the way. We went on the Sentosa Boardwalk, which is itself a tourist attraction. I was using Organic Maps to navigate to the hotel Resorts World Sentosa, but Organic Maps route led us through an amusement park. I tried asking the locals (people working in shops) for directions, but it was a Chinese-speaking region, and they didn t understand English. Fortunately, we managed to find a local who helped us with the directions.
A shot of Sentosa Boardwalk. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Following the directions, we somehow ended up having to walk on a road which did not have pedestrian paths. Singapore is a country with strict laws, so we did not want to walk on that road. Avoiding that road led us to the Michael Hotel. There was a person standing at the entrance, and I asked him for directions to Resorts World Sentosa. The person told me that the bus (which was standing at the entrance) would drop me there! The bus was a free service for getting to Resorts World Sentosa. Here I parted ways with Badri, who went to his aunt s place.
I got to the Resorts Sentosa and showed my ticket to get in. There were two zones inside - the first was a room with a glass wall separating the audience and the players. This was the room to watch the game physically, and resembled a zoo or an aquarium. :) The room was also a silent room, which means talking or making noise was prohibited. Audiences were only allowed to have mobile phones for the first 30 minutes of the game - since I arrived late, I could not bring my phone inside that room.
The other zone was outside this room. It had a big TV on which the game was being broadcast along with commentary by David Howell and Jovanka Houska - the official FIDE commentators for the event. If you don t already know, FIDE is the authoritative international chess body.
I spent most of the time outside that silent room, giving me an opportunity to socialize. A lot of people were from Singapore. I saw there were many Indians there as well. Moreover, I had a good time with Vasudevan, a journalist from Tamil Nadu who was covering the match. He also asked questions to Gukesh during the post-match conference. His questions were in Tamil to lift Gukesh s spirits, as Gukesh is a Tamil speaker.
Tea and coffee were free for the audience. I also bought a T-shirt from their stall as a souvenir.
After the game, I took a shuttle bus from Resorts World Sentosa to a metro station, then travelled to Pasir Ris by metro, where Badri was staying with his aunt. I thought of getting something to eat, but could not find any caf s or restaurants while I was walking from the Pasir Ris metro station to my destination, and was positively starving when I got there.
Badri s aunt s place was an apartment in a gated community. On the gate was a security guard who asked me the address of the apartment. Upon entering, there were many buildings. To enter the building, you need to dial the number of the apartment you want to go to and speak to them. I had seen that in the TV show Seinfeld, where Jerry s friends used to dial Jerry to get into his building.
I was afraid they might not have anything to eat because I told them I was planning to get something on the way. This was fortunately not the case, and I was relieved to not have to sleep with an empty stomach.
Badri s uncle gave us an idea of how safe Singapore is. He said that even if you forget your laptop in a public space, you can go back the next day to find it right there in the same spot. I also learned that owning cars was discouraged in Singapore - the government imposes a high registration fee on them, while also making public transport easy to use and affordable. I also found out that 7-Eleven was not that popular among residents in Singapore, unlike in Malaysia or Thailand.
The next day was our third and final day in Singapore. We had a bus in the evening to Johor Bahru in Malaysia. We got up early, had breakfast, and checked out from Badri s aunt s home. A store by the name of Cat Socrates was our first stop for the day, as Badri wanted to buy some stationery. The plan was to take the metro, followed by the bus. So we got to Pasir Ris metro station. Next to the metro station was a mall. In the mall, Badri found an ATM where our cards worked, and we got some Singapore dollars.
It was noon when we reached the stationery shop mentioned above. We had to walk a kilometer from the place where the bus dropped us. It was a hot, sunny day in Singapore, so walking was not comfortable. We had to go through residential areas in Singapore. We saw some non-touristy parts of Singapore.
After we were done with the stationery shop, we went to a hawker center to get lunch. Hawker centers are unique to Singapore. They have a lot of shops that sell local food at cheap prices. It is similar to a food court. However, unlike the food courts in malls, hawker centers are open-air and can get quite hot.
This is the hawker center we went to. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
To have something, you just need to buy it from one of the shops and find a table. After you are done, you need to put your tray in the tray-collecting spots. I had a kaya toast with chai, since there weren t many vegetarian options. I also bought a persimmon from a nearby fruit vendor. On the other hand, Badri sampled some local non-vegetarian dishes.
Table littering at the hawker center was prohibited by law. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Next, we took a metro to Raffles Place, as we wanted to visit Merlion, the icon of Singapore. It is a statue having the head of a lion and the body of a fish. While getting through the AFC gates, my card was declined. Therefore, I had to buy an EZ-Link card, which I had been avoiding because the card itself costs 5 Singapore dollars.
From the Raffles Place metro station, we walked to Merlion. The place also gave a nice view of Marina Bay Sands. It was filled with tourists clicking pictures, and we also did the same.
Merlion from behind, giving a good view of Marina Bay Sands. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
After this, we went to the bus stop to catch our bus to the border city of Johor Bahru, Malaysia. The bus was more than an hour late, and we worried that we had missed the bus. I asked an Indian woman at the stop who also planned to take the same bus, and she told us that the bus was late. Finally, our bus arrived, and we set off for Johor Bahru.
Before I finish, let me give you an idea of my expenditure. Singapore is an expensive country, and I realized that expenses could go up pretty quickly. Overall, my stay in Singapore for 3 days and 2 nights was approx. 5500 rupees. That too, when we stayed one night at Badri s aunt s place (so we didn t have to pay for accomodation for one of the nights) and didn t have to pay for a couple of meals. This amount doesn t include the ticket for the chess game, but includes the costs of getting there. If you are in Singapore, it is likely you will pay a visit to Sentosa Island anyway.
Stay tuned for our experiences in Malaysia!
Credits: Thanks to Dione, Sahil, Badri and Contrapunctus for reviewing the draft. Thanks to Bhe for spotting a duplicate sentence.
Akvorado 2.0 was released today! Akvorado collects network flows with
IPFIX and sFlow. It enriches flows and stores them in a ClickHouse database.
Users can browse the data through a web console. This release introduces an
important architectural change and other smaller improvements. Let s dive in!
New outlet service
The major change in Akvorado 2.0 is splitting the inlet service into two
parts: the inlet and the outlet. Previously, the inlet handled all flow
processing: receiving, decoding, and enrichment. Flows were then sent to Kafka
for storage in ClickHouse:
Akvorado flow processing before the introduction of the outlet service
Network flows reach the inlet service using UDP, an unreliable protocol. The
inlet must process them fast enough to avoid losing packets. To handle a
high number of flows, the inlet spawns several sets of workers to receive flows,
fetch metadata, and assemble enriched flows for Kafka. Many configuration
options existed for scaling, which increased complexity for users. The code
needed to avoid blocking at any cost, making the processing pipeline complex
and sometimes unreliable, particularly the BMP receiver.1 Adding new
features became difficult without making the problem worse.2
In Akvorado 2.0, the inlet receives flows and pushes them to Kafka without
decoding them. The new outlet service handles the remaining tasks:
Akvorado flow processing after the introduction of the outlet service
This change goes beyond a simple split:3 the outlet now reads flows from
Kafka and pushes them to ClickHouse, two tasks that Akvorado did not handle
before. Flows are heavily batched to increase efficiency and reduce the load
on ClickHouse using ch-go, a low-level Go client for ClickHouse. When
batches are too small, asynchronous inserts are used (e20645). The number of
outlet workers scales dynamically (e5a625) based on the target batch
size and latency (50,000 flows and 5 seconds by default).
This new architecture also allows us to simplify and optimize the code. The
outlet fetches metadata synchronously (e20645). The BMP component becomes
simpler by removing cooperative multitasking (3b9486). Reusing the same
RawFlow object to decode protobuf-encoded flows from Kafka reduces pressure on
the garbage collector (8b580f).
The effect on Akvorado s overall performance was somewhat uncertain, but a
user reported 35% lower CPU usage after migrating from the previous
version, plus resolution of the long-standing BMP component issue.
Other changes
This new version includes many miscellaneous changes, such as completion for
source and destination ports (f92d2e), and automatic restart of the
orchestrator service (0f72ff) when configuration changes to avoid a common
pitfall for newcomers.
Let s focus on some key areas for this release: observability,
documentation, CI,
Docker, Go, and JavaScript.
Observability
Akvorado exposes metrics to provide visibility into the processing pipeline and
help troubleshoot issues. These are available through Prometheus HTTP metrics
endpoints, such as /api/v0/inlet/metrics. With the introduction
of the outlet, many metrics moved. Some were also renamed (4c0b15) to match
Prometheus best practices. Kafka consumer lag was added as a new metric
(e3a778).
If you do not have your own observability stack, the Docker Compose setup
shipped with Akvorado provides one. You can enable it by activating the profiles
introduced for this purpose (529a8f).
The prometheus profile ships Prometheus to store metrics and Alloy
to collect them (2b3c46, f81299, and 8eb7cd). Redis and Kafka
metrics are collected through the exporter bundled with Alloy (560113).
Other metrics are exposed using Prometheus metrics endpoints and are
automatically fetched by Alloy with the help of some Docker labels, similar to
what is done to configure Traefik. cAdvisor was also added (83d855) to
provide some container-related metrics.
The loki profile ships Loki to store logs (45c684). While Alloy
can collect and ship logs to Loki, its parsing abilities are limited: I could
not find a way to preserve all metadata associated with structured logs produced
by many applications, including Akvorado. Vector replaces Alloy (95e201)
and features a domain-specific language, VRL, to transform logs. Annoyingly,
Vector currently cannot retrieve Docker logs from before it was
started.
Finally, the grafana profile ships Grafana, but the shipped dashboards are
broken. This is planned for a future version.
Documentation
The Docker Compose setup provided by Akvorado makes it easy to get the web
interface up and running quickly. However, Akvorado requires a few mandatory
steps to be functional. It ships with comprehensive documentation, including
a chapter about troubleshooting problems. I hoped this documentation would
reduce the support burden. It is difficult to know if it works. Happy users
rarely report their success, while some users open discussions asking for help
without reading much of the documentation.
In this release, the documentation was significantly improved.
The documentation was updated (fc1028) to match Akvorado s new architecture.
The troubleshooting section was rewritten (17a272). Instructions on how to
improve ClickHouse performance when upgrading from versions earlier than 1.10.0
was added (5f1e9a). An LLM proofread the entire content (06e3f3).
Developer-focused documentation was also improved (548bbb, e41bae, and
871fc5).
From a usability perspective, table of content sections are now collapsable
(c142e5). Admonitions help draw user attention to important points
(8ac894).
Example of use of admonitions in Akvorado's documentation
Continuous integration
This release includes efforts to speed up continuous integration on GitHub.
Coverage and race tests run in parallel (6af216 and fa9e48). The Docker
image builds during the tests but gets tagged only after they succeed
(8b0dce).
GitHub workflow to test and build Akvorado
End-to-end tests (883e19) ensure the shipped Docker Compose setup works as
expected. Hurl runs tests on various HTTP endpoints, particularly to verify
metrics (42679b and 169fa9). For example:
## Test inlet has received NetFlow flowsGEThttp://127.0.0.1:8080/prometheus/api/v1/query[Query]query:sum(akvorado_inlet_flow_input_udp_packets_total job="akvorado-inlet",listener=":2055" )
HTTP200[Captures]inlet_receivedflows:jsonpath "$.data.result[0].value[1]" toInt
[Asserts]variable"inlet_receivedflows">10## Test inlet has sent them to KafkaGEThttp://127.0.0.1:8080/prometheus/api/v1/query[Query]query:sum(akvorado_inlet_kafka_sent_messages_total job="akvorado-inlet" )
HTTP200[Captures]inlet_sentflows:jsonpath "$.data.result[0].value[1]" toInt
[Asserts]variable"inlet_sentflows">=inlet_receivedflows
Docker
Akvorado ships with a comprehensive Docker Compose setup to help users get
started quickly. It ensures a consistent deployment, eliminating many
configuration-related issues. It also serves as a living documentation of the
complete architecture.
This release brings some small enhancements around Docker:
Previously, many Docker images were pulled from the Bitnami Containers
library. However, VMWare acquired Bitnami in 2019 and Broadcom acquired
VMWare in 2023. As a result, Bitnami images were deprecated in less than a
month. This was not really a surprise4. Previous versions of Akvorado
had already started moving away from them. In this release, the Apache project s
Kafka image replaces the Bitnami one (1eb382). Thanks to the switch to KRaft
mode, Zookeeper is no longer needed (0a2ea1, 8a49ca, and f65d20).
Akvorado s Docker images were previously compiled with Nix. However, building
AArch64 images on x86-64 is slow because it relies on QEMU userland emulation.
The updated Dockerfile uses multi-stage and multi-platform builds: one
stage builds the JavaScript part on the host platform, one stage builds the Go
part cross-compiled on the host platform, and the final stage assembles the
image on top of a slim distroless image (268e95 and d526ca).
# This is a simplified versionFROM--platform=$BUILDPLATFORMnode:20-alpineASbuild-js
RUNapkadd--no-cachemake
WORKDIR/buildCOPYconsole/frontendconsole/frontend
COPYMakefile.
RUNmakeconsole/data/frontend
FROM--platform=$BUILDPLATFORMgolang:alpineASbuild-go
RUNapkadd--no-cachemakecurlzip
WORKDIR/buildCOPY..
COPY--from=build-js/build/console/data/frontendconsole/data/frontend
RUNgomoddownload
RUNmakeall-indep
ARGTARGETOSTARGETARCHTARGETVARIANTVERSION
RUNmake
FROMgcr.io/distroless/static:latestCOPY--from=build-go/build/bin/akvorado/usr/local/bin/akvorado
ENTRYPOINT["/usr/local/bin/akvorado"]
When building for multiple platforms with --platform
linux/amd64,linux/arm64,linux/arm/v7, the build steps until the highlighted
line execute only once for all platforms. This significantly speeds up the
build.
Akvorado now ships Docker images for these platforms: linux/amd64,
linux/amd64/v3, linux/arm64, and linux/arm/v7. When requesting
ghcr.io/akvorado/akvorado, Docker selects the best image for the current CPU.
On x86-64, there are two choices. If your CPU is recent enough, Docker
downloads linux/amd64/v3. This version contains additional optimizations and
should run faster than the linux/amd64 version. It would be interesting to
ship an image for linux/arm64/v8.2, but Docker does not support the same
mechanism for AArch64 yet (792808).
Go
This release includes many changes related to Go but not visible to the users.
Toolchain
In the past, Akvorado supported the two latest Go versions, preventing immediate
use of the latest enhancements. The goal was to allow users of stable
distributions to use Go versions shipped with their distribution to compile
Akvorado. However, this became frustrating when interesting features, like go
tool, were released. Akvorado 2.0 requires Go 1.25 (77306d) but can be
compiled with older toolchains by automatically downloading a newer one
(94fb1c).5 Users can still override GOTOOLCHAIN to revert this
decision. The recommended toolchain updates weekly through CI to ensure we get
the latest minor release (5b11ec). This change also simplifies updates to
newer versions: only go.mod needs updating.
Thanks to this change, Akvorado now uses wg.Go() (77306d) and I have
started converting some unit tests to the new test/synctest package
(bd787e, 7016d8, and 159085).
Testing
When testing equality, I use a helper function Diff() to display the
differences when it fails:
This function uses kylelemons/godebug. This package is
no longer maintained and has some shortcomings: for example, by default, it does
not compare struct private fields, which may cause unexpectedly successful
tests. I replaced it with google/go-cmp, which is stricter
and has better output (e2f1df).
Another package for Kafka
Another change is the switch from Sarama to franz-go to interact with
Kafka (756e4a and 2d26c5). The main motivation for this change is to
get a better concurrency model. Sarama heavily relies on channels and it is
difficult to understand the lifecycle of an object handed to this package.
franz-go uses a more modern approach with callbacks6 that is both more
performant and easier to understand. It also ships with a package to spawn fake
Kafka broker clusters, which is more convenient than the mocking functions
provided by Sarama.
Improved routing table for BMP
To store its routing table, the BMP component used
kentik/patricia, an implementation of a patricia tree
focused on reducing garbage collection pressure.
gaissmai/bart is a more recent alternative using an
adaptation of [Donald Knuth s ART algorithm][] that promises better
performance and delivers it: 90% faster lookups and 27% faster
insertions (92ee2e and fdb65c).
Unlike kentik/patricia, gaissmai/bart does not help efficiently store values
attached to each prefix. I adapted the same approach as kentik/patricia to
store route lists for each prefix: store a 32-bit index for each prefix, and use
it to build a 64-bit index for looking up routes in a map. This leverages Go s
efficient map structure.
gaissmai/bart also supports a lockless routing table version, but this is not
simple because we would need to extend this to the map storing the routes and to
the interning mechanism. I also attempted to use Go s new unique package to
replace the intern package included in Akvorado, but performance was
worse.7
Miscellaneous
Previous versions of Akvorado were using a custom Protobuf encoder for
performance and flexibility. With the introduction of the outlet service,
Akvorado only needs a simple static schema, so this code was removed. However,
it is possible to enhance performance with
planetscale/vtprotobuf (e49a74, and 8b580f).
Moreover, the dependency on protoc, a C++ program, was somewhat annoying.
Therefore, Akvorado now uses buf, written in Go, to convert a Protobuf
schema into Go code (f4c879).
Another small optimization to reduce the size of the Akvorado binary by
10 MB was to compress the static assets embedded in Akvorado in a ZIP file. It
includes the ASN database, as well as the SVG images for the documentation. A
small layer of code makes this change transparent (b1d638 and e69b91).
JavaScript
Recently, two large supply-chain attacks hit the JavaScript ecosystem: one
affecting the popular packages chalk and debug and another
impacting the popular package @ctrl/tinycolor. These attacks also
exist in other ecosystems, but JavaScript is a prime target due to heavy use of
small third-party dependencies. The previous version of Akvorado relied on 653
dependencies.
npm-run-all was removed (3424e8, 132 dependencies). patch-package was
removed (625805 and e85ff0, 69 dependencies) by moving missing
TypeScript definitions to env.d.ts. eslint was replaced with oxlint, a
linter written in Rust (97fd8c, 125 dependencies, including the plugins).
I switched from npm to Pnpm, an alternative package manager (fce383).
Pnpm does not run install scripts by default8 and prevents installing
packages that are too recent. It is also significantly faster.9 Node.js
does not ship Pnpm but it ships Corepack, which allows us to use Pnpm
without installing it. Pnpm can also list licenses used by each dependency,
removing the need for license-compliance (a35ca8, 42 dependencies).
For additional speed improvements, beyond switching to Pnpm and Oxlint, Vite
was replaced with its faster Rolldown version (463827).
After these changes, Akvorado only pulls 225 dependencies.
Next steps
I would like to land three features in the next version of Akvorado:
Add Grafana dashboards to complete the observability stack. See issue
#1906 for details.
Integrate OVH s Grafana plugin by providing a stable API for such
integrations. Akvorado s web console would still be useful for browsing
results, but if you want to build and share dashboards, you should switch to
Grafana. See issue #1895.
Move some work currently done in ClickHouse (custom dictionaries, GeoIP and IP
enrichment) back into the outlet service. This should give more flexibility
for adding features like the one requested in issue #1030. See issue #2006.
I started working on splitting the inlet into two parts more than one year ago.
I found more motivation in recent months, partly thanks to Claude Code,
which I used as a rubber duck. Almost none of the produced code was
kept:10 it is like an intern who does not learn.
Many attempts were made to make the BMP component both performant and
not blocking. See for example PR #254, PR #255, and PR #278.
Despite these efforts, this component remained problematic for most users.
See issue #1461 as an example.
Some features have been pushed to ClickHouse to avoid the
processing cost in the inlet. See for example PR #1059.
Broadcom is known for its user-hostile moves. Look at what happened
with VMWare.
As a Debian developer, I dislike these mechanisms that circumvent
the distribution package manager. The final straw came when Go 1.25 spent one month in the Debian NEW queue, an arbitrary mechanism I
don t like at all.
In the early years of Go, channels were heavily promoted. Sarama
was designed during this period. A few years later, a more nuanced approach
emerged. See notably Go channels are bad and you should feel bad.
This should be investigated further, but my theory is that the
intern package uses 32-bit integers, while unique uses 64-bit pointers.
See commit 74e5ac.
This is also possible with npm. See commit dab2f7.
An even faster alternative is Bun, but it is less available.
The exceptions are part of the code for the admonition blocks,
the code for collapsing the table of content, and part of the documentation.
The Debian LTS Team, funded by Freexian s Debian LTS
offering, is pleased to report
its activities for October.
Welcome to post 54 in the 
I ve written down a new rule (no name, sorry) that I ll be repeating to myself
and those around me. If you can replace DNS with key value store mapping
a name to an ip and it still makes sense, it was not, in fact, DNS. Feel
free to repeat it along with me.
Sure, the It s always DNS meme is funny the first few hundred times you see
it but what s less funny is when critical thinking ends because a DNS query
is involved. DNS failures are often the first observable problem because
it s one of the first things that needs to be done. DNS is fairly complicated,
implementation-dependent, and at times frustrating to debug but it is not
the operational hazard it s made out to be. It s at best a shallow take, and at
worst actively holding teams back from understanding their true operational
risks.
IP connectivity failures between a host and the rest of the network is not a
reason to blame DNS. This would happen no matter how you distribute the updated
name to IP mappings. Wiping out
The discovery of a backdoor in XZ Utils in the spring of 2024 shocked the open source community, raising critical questions about software supply chain security. This post explores whether better Debian packaging practices could have detected this threat, offering a guide to auditing packages and suggesting future improvements.
The XZ backdoor in versions 5.6.0/5.6.1 made its way briefly into many major Linux distributions such as Debian and Fedora, but luckily didn t reach that many actual users, as the backdoored releases were quickly removed thanks to the heroic diligence of 
If the changes are extensive, and you want to use a LLM to help spot potential security issues, generate the report of both the upstream and Debian packaging differences in Markdown with:
To compare changes across the new and old upstream tarball, one would need to compare commits afba662b New upstream version 5.8.0 and fa1e8796 New upstream version 5.8.1 by running 
With all the above tips you can now go and try to audit your own favorite package in Debian and see if it is identical with upstream, and if not, how it differs.
There is only one tiny thing that maybe a very experienced Autotools user could potentially have noticed: the 
We had 886 responses with status code 200 and 64 connections closed.
The backend terminated 64 connections while the load balancer still had
active requests directed to it.
Let's change the Load Balancer 
Great! Now all the requests worked.
This is a common issue, not specific to Envoy Proxy or the setup
shown earlier. Major cloud providers have all documented it.
Even thought the success rate is 100%, the status code distribution
show some responses with status code 503. This is the case where is not
that obvious that the problem is related to idle timeout.
However, it's clear when we look the Envoy access logs:
Key Takeaways:
A shot of Jewel Changi. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
A shot of Sentosa Boardwalk. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
This is the hawker center we went to. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Table littering at the hawker center was prohibited by law. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Merlion from behind, giving a good view of Marina Bay Sands. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
