In September 2025, I attended the LibreOffice Conference in Budapest, Hungary, on the 4th and the 5th, and a community meeting on the 3rd. Thanks to The Document Foundation (TDF) for sponsoring my travel and accommodation costs. The conference venue was Faculty of Informatics, E tv s Lor nd University (ELTE).
The conference was planned to be held from the 4th to the 6th, but the program for the 6th of September had to be canceled due to the venue being unavailable because of a marathon in Budapest. So, all the talks got squeezed into just two days, making the schedule a bit hectic.
The TDF had booked my room at the Corvin Hotel. It was a double bedroom with a window. The breakfast was included in the hotel booking. The hotel was walking distance from the conference venue. One could also take a tram from the hotel to reach the venue.
A shot of my room. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
A tram in Budapest. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
3rd of September
On the 3rd of September, we had a community meeting at the above-mentioned venue. I walked with my friend Dione to the venue. Upon reaching there, I noticed that the university had no boundaries and gates. This reminded me of the previous year s conference venue in Luxembourg, which also had no boundaries or gates.
In contrast, Indian universities and institutes typically have walls and gates serving as boundaries to separate them from the rest of the city. Many of these institutes also have security guards at the entrance, who may ask attendees to present proof of admission before allowing them inside. I was surprised to find that institutes in Europe, like the one where the conference was held, did not have such boundaries.
The building where the conference was held was red, which happened to be the same color as the building for the previous year s conference venue. I remember joking with Dione that the criteria for the conference venue might have been the color of the building.
The red building in the picture served as the conference venue. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
During the community meeting, we shared ideas on how to spread the word about LibreOffice. The meeting lasted for a couple of hours.
After the community meeting, we went to the hotel for dinner sponsored by the TDF.
These Esterh zy cake bites were really yummy. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Raspberry Currant cake slices. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
4th of September
On the first day of the conference, attendees were given swag bags containing a pad, sticky notes, a pen, a conference T-shirt, and a bottle.
Conference swag. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The talks started early in the morning with Eliane Domingos, Chairperson of TDF s Board of Directors, giving the inauguration talk. As always, I found Italo Vignoli s talk on the importance of document freedom interesting.
During the snack break, I noticed that there were three types of milk available for coffee: cow s milk, lactose-free milk, and almond milk. Almond milk is rare in India, but I have managed to get it, but I have never seen lactose-free milk in India.
Since I run fundraisers in my projects, such as Prav, I could relate to Lothar K. Becker s talk. He discussed the issue that certain implementations in LibreOffice require a budget that is too large for any single interested entity to fund independently. Furthermore, The Document Foundation (TDF) cannot legally receive funds from government entities. Therefore, there is no organization or entity to pool resources from all the interested entities to finance the implementation.
Lothar giving his presentation. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Another talk was by the Austrian Armed Forces on their migration to LibreOffice. I wanted to know why they migrated, and I found out that they did it for their digital sovereignty, and not for saving on the license costs. Another point presented in the talk was that LibreOffice is available on all the operating systems, while the Microsoft Office suite is not that widely available. The migration was systematic and was performed over a few years. They started working on it in 2021, and the migration was finished recently. In addition, it also required training their staff in using LibreOffice.
Presentation on migration to LibreOffice by Austrian Armed Forces. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The lunch was inside the university canteen. We were provided lunch coupons by the TDF. I got a vegan coupon with 4000 Ft written on it, which meant I could take lunch for up to 4000 Hungarian forints.
My lunch ticket for the conference. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The lunch I had on the first day. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
During the evening, it was my turn for the presentation. I was done with preparing my slides ten days before my talk. I also got my slides reviewed by friends.
My talk was finished in 20 minutes, while I was given a 30-minute slot. This helped us catch up on the schedule. Furthermore, I made my talk interactive by asking questions and making sure that the audience was not asleep. During my talk, my friend Dione took my pictures with my camera.
My talk was on how free software projects could give users a say in freedom to modify the software. I illustrated this using the Prav project that I am a part of.
After the talks were over, we were treated to a conference dinner at Trofea Grill. It had a great selection of desserts, which helped me sample some Hungarian desserts. The sponge cake was especially good.
Desserts at Tofea Grill. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
5th of September
The next day the 5th of September I went with Dione to the venue early in the morning, as her talk was the first one of the day. Her talk was titled Managing Tasks with Nextcloud Deck. Later that day, I also attended a talk on Collabora. At lunch, I found the egg white salad quite tasty.
Dione giving her presentation. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Egg white salad. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
After the lunch break, we had the conference group photo. I had a Nikon camera, which we used to take the group photo. I requested a university student to take our group photo and also taught her how to operate the camera.
Group photo
By the evening, the conference ended, after which we went to a pub, which was again sponsored by TDF. I had beer, but that one really tasted bad, so I couldn t finish it. The only vegetarian option was goat cheeseburger, which my friend Manish and I opted for. The burger tasted awful. Apparently, I don t like goat cheese.
The next day I went sightseeing with Dione in Budapest. Stay tuned for our adventures!
Credits: Thanks to Dione and Richard for proofreading.
tl;dr
This is an experimental feature that, for the first time, brings full ECH
support to curl on Debian using OpenSSL.
Starting with curl 8.19.0-3+exp2 (Debian Experimental), you can now use
ECH, with HTTPS-RR and DoH for maximum privacy.
curl 8.19.0-3+exp2 is quite fresh at the time of writing, bear in mind that your
repository might not have synced the package yet, all mirrors should have it by
March 27th 14:00 UTC.
# defo.ie is a test server that confirms whether ECH was successfully usedcurl -v --ech hard https://defo.ie/ech-check.php# For Encrypted Client Hello (ECH) + DNS over HTTPS (DoH)curl -v --ech hard --doh-url https://1.1.1.1/dns-query https://defo.ie/ech-check.php
"--ech hard" tells curl to refuse the connection entirely if ECH cannot be negotiated.
Or, if you would like to try it out in a container:
podman run debian:experimental /bin/bash -c 'apt install --update -t experimental -y curl && curl -v --ech hard --doh-url https://1.1.1.1/dns-query https://defo.ie/ech-check.php'
(in case you haven't noticed, apt now has the --update option for the
upgrade and install commands)
For Privacy
CloudFlare calls it "the last puzzle piece to privacy" in their must-read
announcement: https://blog.cloudflare.com/announcing-encrypted-client-hello/.
Encrypted Client Hello (rfc9849) encrypts the
"which website are you connecting to?" part of the TLS handshake that was
previously visible in plaintext.
HTTPS-RR (rfc9460) is a DNS record type that
publishes connection parameters for a service, including the public key clients
need to perform ECH.
DNS Over HTTPS (rfc8484) encrypts DNS queries
by tunneling them over HTTPS, hiding what domains you're looking up from
network observers.
When all three operate together over a CDN with shared IP space, the target
domain name is hidden from passive observers; the HTTPS-RR record is queried
over DoH in order to retrieve the ECH key
(rfc9848) for the TLS handshake.
Seems like quite an important feature, and in fact the major browsers have it
enabled for some time now, the trick is that they do not use OpenSSL (Chrome
uses BoringSSL and Firefox uses NSS).
For everyone else, the only option is to patch OpenSSL or wait until 4.0.0 is
released, and so part of the reason Debian is the first distro to enable it
(curl + OpenSSL + ECH) is that the OpenSSL maintainer (Sebastian Andrzej
Siewior) packaged the alpha release just 3 days after it was published.
Do not forget that ECH support is experimental and currently relies on the
alpha release of OpenSSL.
wcurl Gets It Too
Considering wcurl is just a wrapper on curl, it gets
the feature for free:
wcurl --curl-options="--ech hard --doh-url https://1.1.1.1/dns-query" $URL
If you're using wcurl, you don't want to have to set parameters, this is just
to show that the feature is there and if you have a .curlrc file, it can
enable the feature seamlessly.
Other Debian Releases
Given the ECH feature requires OpenSSL >= 4, it will not make it to Debian 13,
having a small chance of going to Debian 13 Backports (emphasis on small).
It should get to Debian Unstable and Debian Testing within the next couple of
months as the OpenSSL GA release happens and gets packaged, but you should be
able to install the package from Experimental in your Unstable and Testing
systems without issues. It will also be in Debian 14 once it becomes the new Stable.
Shoulders of Giants
Stephen Farrell's presentation from OpenSSL Conference 2025 has a lot of
background on the work involved:
Encrypted Client Hello Lessons learned from trying to do something that was
probably too complicated
They have been working on implementing ECH in open-source projects for years,
something as big as this doesn't happen without lots of people dedicating both
their paid and free times over it.
I ended up being the person who enabled it on Debian, which was pretty much the
least amount of work between everyone involved, but hey it's fun flipping the
switch and telling you about it.
Background
Since 2025, the curl developers started organizing an yearly meeting with all
maintainers of curl in Operating Systems. The 2026 edition happened in March
26th:
https://github.com/curl/curl/wiki/curl-distro-discussion-2026.
Attendance was really good, and as you can imagine one of the topics of
discussion was ECH, in which it was pointed out that having OpenSSL 4 was
the main requirement but besides it nothing unusual was needed.
In Debian Experimental, we have been enabling HTTPS-RR since March 2025, and
OpenSSL 4.0.0 alpha was packaged just recently (2026-03-13) by Sebastian
Andrzej Siewior, it's time for the next step.
The curl distro meeting was just the motivation I needed to go ahead and
enable it in Debian Experimental, so as part of our Debian Brasil Weekly
Meetings I've prepared and uploaded the changes, while Carlos Henrique Lima
Melara worked on addressing a recent test regression for Debian Unstable.
Unfortunately sergiodj couldn't join and I'm sure he's jealous of the hacking
session now.
Appendix
While writing this, I've noticed one of the authors of the CloudFlare blogpost
is the previous curl maintainer on Debian; Alessandro Ghedini let me take over
the maintenance back in 2021 and today curl is maintained by a team of 4
people, it's nice to see Alessandro's involvement.
A lot of hardware runs non-free software. Sometimes that non-free software is in ROM. Sometimes it s in flash. Sometimes it s not stored on the device at all, it s pushed into it at runtime by another piece of hardware or by the operating system. We typically refer to this software as firmware to differentiate it from the software run on the CPU after the OS has started1, but a lot of it (and, these days, probably most of it) is software written in C or some other systems programming language and targeting Arm or RISC-V or maybe MIPS and even sometimes x862. There s no real distinction between it and any other bit of software you run, except it s generally not run within the context of the OS3. Anyway. It s code. I m going to simplify things here and stop using the words software or firmware and just say code instead, because that way we don t need to worry about semantics.
A fundamental problem for free software enthusiasts is that almost all of the code we re talking about here is non-free. In some cases, it s cryptographically signed in a way that makes it difficult or impossible to replace it with free code. In some cases it s even encrypted, such that even examining the code is impossible. But because it s code, sometimes the vendor responsible for it will provide updates, and now you get to choose whether or not to apply those updates.
I m now going to present some things to consider. These are not in any particular order and are not intended to form any sort of argument in themselves, but are representative of the opinions you will get from various people and I would like you to read these, think about them, and come to your own set of opinions before I tell you what my opinion is.
THINGS TO CONSIDER
Does this blob do what it claims to do? Does it suddenly introduce functionality you don t want? Does it introduce security flaws? Does it introduce deliberate backdoors? Does it make your life better or worse?
You re almost certainly being provided with a blob of compiled code, with no source code available. You can t just diff the source files, satisfy yourself that they re fine, and then install them. To be fair, even though you (as someone reading this) are probably more capable of doing that than the average human, you re likely not doing that even if you are capable because you re also likely installing kernel upgrades that contain vast quantities of code beyond your ability to understand4. We don t rely on our personal ability, we rely on the ability of those around us to do that validation, and we rely on an existing (possibly transitive) trust relationship with those involved. You don t know the people who created this blob, you likely don t know people who do know the people who created this blob, these people probably don t have an online presence that gives you more insight. Why should you trust them?
If it s in ROM and it turns out to be hostile then nobody can fix it ever
The people creating these blobs largely work for the same company that built the hardware in the first place. When they built that hardware they could have backdoored it in any number of ways. And if the hardware has a built-in copy of the code it runs, why do you trust that that copy isn t backdoored? Maybe it isn t and updates would introduce a backdoor, but in that case if you buy new hardware that runs new code aren t you putting yourself at the same risk?
Designing hardware where you re able to provide updated code and nobody else can is just a dick move5. We shouldn t encourage vendors who do that.
Humans are bad at writing code, and code running on ancilliary hardware is no exception. It contains bugs. These bugs are sometimes very bad. This paper describes a set of vulnerabilities identified in code running on SSDs that made it possible to bypass encryption secrets. The SSD vendors released updates that fixed these issues. If the code couldn t be replaced then anyone relying on those security features would need to replace the hardware.
Even if blobs are signed and can t easily be replaced, the ones that aren t encrypted can still be examined. The SSD vulnerabilities above were identifiable because researchers were able to reverse engineer the updates. It can be more annoying to audit binary code than source code, but it s still possible.
Vulnerabilities in code running on other hardware can still compromise the OS. If someone can compromise the code running on your wifi card then if you don t have a strong IOMMU setup they re going to be able to overwrite your running OS.
Replacing one non-free blob with another non-free blob increases the total number of non-free blobs involved in the whole system, but doesn t increase the number that are actually executing at any point in time.
Ok we re done with the things to consider. Please spend a few seconds thinking about what the tradeoffs are here and what your feelings are. Proceed when ready.
I trust my CPU vendor. I don t trust my CPU vendor because I want to, I trust my CPU vendor because I have no choice. I don t think it s likely that my CPU vendor has designed a CPU that identifies when I m generating cryptographic keys and biases the RNG output so my keys are significantly weaker than they look, but it s not literally impossible. I generate keys on it anyway, because what choice do I have? At some point I will buy a new laptop because Electron will no longer fit in 32GB of RAM and I will have to make the same affirmation of trust, because the alternative is that I just don t have a computer. And in any case, I will be communicating with other people who generated their keys on CPUs I have no control over, and I will also be relying on them to be trustworthy. If I refuse to trust my CPU then I don t get to computer, and if I don t get to computer then I will be sad. I suspect I m not alone here.
Why would I install a code update on my CPU when my CPU s job is to run my code in the first place? Because it turns out that CPUs are complicated and messy and they have their own bugs, and those bugs may be functional (for example, some performance counter functionality was broken on Sandybridge at release, and was then fixed with a microcode blob update) and if you update it your hardware works better. Or it might be that you re running a CPU with speculative execution bugs and there s a microcode update that provides a mitigation for that even if your CPU is slower when you enable it, but at least now you can run virtual machines without code in those virtual machines being able to reach outside the hypervisor boundary and extract secrets from other contexts. When it s put that way, why would I not install the update?
And the straightforward answer is that theoretically it could include new code that doesn t act in my interests, either deliberately or not. And, yes, this is theoretically possible. Of course, if you don t trust your CPU vendor, why are you buying CPUs from them, but well maybe they ve been corrupted (in which case don t buy any new CPUs from them either) or maybe they ve just introduced a new vulnerability by accident, and also you re in a position to determine whether the alleged security improvements matter to you at all. Do you care about speculative execution attacks if all software running on your system is trustworthy? Probably not! Do you need to update a blob that fixes something you don t care about and which might introduce some sort of vulnerability? Seems like no!
But there s a difference between a recommendation for a fully informed device owner who has a full understanding of threats, and a recommendation for an average user who just wants their computer to work and to not be ransomwared. A code update on a wifi card may introduce a backdoor, or it may fix the ability for someone to compromise your machine with a hostile access point. Most people are just not going to be in a position to figure out which is more likely, and there s no single answer that s correct for everyone. What we do know is that where vulnerabilities in this sort of code have been discovered, updates have tended to fix them - but nobody has flagged such an update as a real-world vector for system compromise.
My personal opinion? You should make your own mind up, but also you shouldn t impose that choice on others, because your threat model is not necessarily their threat model. Code updates are a reasonable default, but they shouldn t be unilaterally imposed, and nor should they be blocked outright. And the best way to shift the balance of power away from vendors who insist on distributing non-free blobs is to demonstrate the benefits gained from them being free - a vendor who ships free code on their system enables their customers to improve their code and enable new functionality and make their hardware more attractive.
It s impossible to say with absolute certainty that your security will be improved by installing code blobs. It s also impossible to say with absolute certainty that it won t. So far evidence tends to support the idea that most updates that claim to fix security issues do, and there s not a lot of evidence to support the idea that updates add new backdoors. Overall I d say that providing the updates is likely the right default for most users - and that that should never be strongly enforced, because people should be allowed to define their own security model, and whatever set of threats I m worried about, someone else may have a good reason to focus on different ones.
Code that runs on the CPU before the OS is still usually described as firmware - UEFI is firmware even though it s executing on the CPU, which should give a strong indication that the difference between firmware and software is largely arbitrary
Because UEFI makes everything more complicated, UEFI makes this more complicated. Triggering a UEFI runtime service involves your OS jumping into firmware code at runtime, in the same context as the OS kernel. Sometimes this will trigger a jump into System Management Mode, but other times it won t, and it s just your kernel executing code that got dumped into RAM when your system booted.
I don t understand most of the diff between one kernel version and the next, and I don t have time to read all of it either.
There s a bunch of reasons to do this, the most reasonable of which is probably not wanting customers to replace the code and break their hardware and deal with the support overhead of that, but not being able to replace code running on hardware I own is always going to be an affront to me.
2026 already! The winter weather here has really been beautiful and I always enjoy
this time of year. Writing this yearly musical retrospective has now become a
beloved tradition of mine1 and I enjoy retracing the year's various
events through albums I listened to and concerts I went to.
Albums
In 2025, I added 141 new albums to my collection, around 60% more than last
year's haul. I think this might have been too much? I feel like I didn't have
time to properly enjoy all of them and as such, I decided to slow down my
acquisition spree sometimes in early December, around the time I normally do the
complete opposite.
This year again, I bought the vast majority of my music on Bandcamp. Most of
the other albums I bought as CDs and ripped them.
Concerts
In 2025, I went to the following 25 (!!) concerts:
January 17th: Uzu, Young Blades, She came to quit, Fever Visions
February 1st: Over the Hill, Jail, Mortier, Ain't Right
February 7th: B ton Arm , Mulchulation II, Ooz
February 15th: The Prowlers, Ultra Razzia, Sistema de Muerte,
Trauma Bond
February 28th: Francb tards
March 28th: Conflit Majeur, to Even Exist, Crachat
April 12th: Jetsam, Mortier, NIIVI, Canette
April 26th-27th (Montreal Oi! Fest 2025): The
Buzzers, Bad Terms, Sons of Pride, Liberty and Justice, Flafoot 56, The
Beltones, Mortier, Street Code, The Stress, Alternate Action
May 1st: Bauxite, Atomic threat, the 351's
May 30th: Uzu, Tenaz, Extra a Humana, Sistema de muerte
June 7th: Ordures Ioniques, Tulaviok, Fucking Raymonds, Voyou
June 18th: Tiken Jah Fakoly
June 21st: Sa an Supa Celebration
June 26th: Taxi Girls, Death Proof, Laura Krieg
July 4th: Frente Cumbiero
July 12th: Montreal's Big Fiesta DJ Set
August 16th: Guerilla Poubelle
September 11th: No Suicide Act, Mortier
September 20th: Hors Contr le, Union Thugs, Barricade Mentale
October 20th: Ezra Furman, The Golden Dregs
October 24th: Overbass, Hommage B rurier Noir, Self Control,
Vermin Kaos
November 6th: B ton Arm , Faze, Slash Need, Chain Block
November 28th (Blood Moon Ritual 2025): Bhatt, Channeler,
Pyrocene Death Cult, Masse d'Armes
December 13th (Stomp Records' 30th Anniversary Bash):
The Planet Smashers, The Flatliners, Wine Lips, The Anti-Queens, Crash ton
rock
Although I haven't touched metalfinder's code in a good while, my instance
still works very well and I get the occasional match when a big-name artist in
my collection comes in town. Most the venues that advertise on Bandsintown are
tied to Ticketmaster though, which means most underground artists (i.e. most of
the music I listen to) end up playing elsewhere.
As such, shout out again to the Gancio project and to the folks running the
Montreal instance. It continues to be a smash hit and most of
the interesting concerts end up being advertised there.
See you all in 2026!
The discovery of a backdoor in XZ Utils in the spring of 2024 shocked the open source community, raising critical questions about software supply chain security. This post explores whether better Debian packaging practices could have detected this threat, offering a guide to auditing packages and suggesting future improvements.
The XZ backdoor in versions 5.6.0/5.6.1 made its way briefly into many major Linux distributions such as Debian and Fedora, but luckily didn t reach that many actual users, as the backdoored releases were quickly removed thanks to the heroic diligence of Andres Freund. We are all extremely lucky that he detected a half a second performance regression in SSH, cared enough to trace it down, discovered malicious code in the XZ library loaded by SSH, and reported promtly to various security teams for quick coordinated actions.
This episode makes software engineers pondering the following questions:
Why didn t any Linux distro packagers notice anything odd when importing the new XZ version 5.6.0/5.6.1 from upstream?
Is the current software supply-chain in the most popular Linux distros easy to audit?
Could we have similar backdoors lurking that haven t been detected yet?
As a Debian Developer, I decided to audit the xz package in Debian, share my methodology and findings in this post, and also suggest some improvements on how the software supply-chain security could be tightened in Debian specifically.
Note that the scope here is only to inspect how Debian imports software from its upstreams, and how they are distributed to Debian s users. This excludes the whole story of how to assess if an upstream project is following software development security best practices. This post doesn t discuss how to operate an individual computer running Debian to ensure it remains untampered as there are plenty of guides on that already.
Downloading Debian and upstream source packages
Let s start by working backwards from what the Debian package repositories offer for download. As auditing binaries is extremely complicated, we skip that, and assume the Debian build hosts are trustworthy and reliably building binaries from the source packages, and the focus should be on auditing the source code packages.
As with everything in Debian, there are multiple tools and ways to do the same thing, but in this post only one (and hopefully the best) way to do something is presented for brevity.
The first step is to download the latest version and some past versions of the package from the Debian archive, which is easiest done with debsnap. The following command will download all Debian source packages of xz-utils from Debian release 5.2.4-1 onwards:
Verifying authenticity of upstream and Debian sources using OpenPGP signatures
As seen in the output of debsnap, it already automatically verifies that the downloaded files match the OpenPGP signatures. To have full clarity on what files were authenticated with what keys, we should verify the Debian packagers signature with:
$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc
gpg: Signature made Fri Oct 3 22:04:44 2025 UTC
gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B
gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org
gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Sebastian Andrzej Siewior" [unknown]
gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown]
gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1
Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B
$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc
gpg: Signature made Fri Oct 3 22:04:44 2025 UTC
gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B
gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org
gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Sebastian Andrzej Siewior" [unknown]
gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown]
gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1
Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B
The upstream tarball signature (if available) can be verified with:
$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc
gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz'
gpg: Signature made Thu Apr 3 11:38:23 2025 UTC
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620
$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc
gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz'
gpg: Signature made Thu Apr 3 11:38:23 2025 UTC
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620
Note that this only proves that there is a key that created a valid signature for this content. The authenticity of the keys themselves need to be validated separately before trusting they in fact are the keys of these people. That can be done by checking e.g. the upstream website for what key fingerprints they published, or the Debian keyring for Debian Developers and Maintainers, or by relying on the OpenPGP web-of-trust .
Verifying authenticity of upstream sources by comparing checksums
In case the upstream in question does not publish release signatures, the second best way to verify the authenticity of the sources used in Debian is to download the sources directly from upstream and compare that the sha256 checksums match.
This should be done using the debian/watch file inside the Debian packaging, which defines where the upstream source is downloaded from. Continuing on the example situation above, we can unpack the latest Debian sources, enter and then run uscan to download:
$ tar xvf xz-utils_5.8.1-2.debian.tar.xz
...
debian/rules
debian/source/format
debian/source.lintian-overrides
debian/symbols
debian/tests/control
debian/tests/testsuite
debian/upstream/signing-key.asc
debian/watch
...
$ uscan --download-current-version --destdir /tmp
Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1
gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC
gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>"
Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz.
$ tar xvf xz-utils_5.8.1-2.debian.tar.xz
...
debian/rules
debian/source/format
debian/source.lintian-overrides
debian/symbols
debian/tests/control
debian/tests/testsuite
debian/upstream/signing-key.asc
debian/watch
...
$ uscan --download-current-version --destdir /tmp
Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1
gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC
gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>"
Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz.
The original files downloaded from upstream are now in /tmp along with the files renamed to follow Debian conventions. Using everything downloaded so far the sha256 checksums can be compared across the files and also to what the .dsc file advertised:
In the example above the checksum 0b54f79df85... is the same across the files, so it is a match.
Repackaged upstream sources can t be verified as easily
Note that uscan may in rare cases repackage some upstream sources, for example to exclude files that don t adhere to Debian s copyright and licensing requirements. Those files and paths would be listed under the Files-Excluded section in the debian/copyright file. There are also other situations where the file that represents the upstream sources in Debian isn t bit-by-bit the same as what upstream published. If checksums don t match, an experienced Debian Developer should review all package settings (e.g. debian/source/options) to see if there was a valid and intentional reason for divergence.
Reviewing changes between two source packages using diffoscope
Diffoscope is an incredibly capable and handy tool to compare arbitrary files. For example, to view a report in HTML format of the differences between two XZ releases, run:
If the changes are extensive, and you want to use a LLM to help spot potential security issues, generate the report of both the upstream and Debian packaging differences in Markdown with:
The Markdown files created above can then be passed to your favorite LLM, along with a prompt such as:
Based on the attached diffoscope output for a new Debian package version compared with the previous one, list all suspicious changes that might have introduced a backdoor, followed by other potential security issues. If there are none, list a short summary of changes as the conclusion.
Reviewing Debian source packages in version control
As of today only 93% of all Debian source packages are tracked in git on Debian s GitLab instance at salsa.debian.org. Some key packages such as Coreutils and Bash are not using version control at all, as their maintainers apparently don t see value in using git for Debian packaging, and the Debian Policy does not require it. Thus, the only reliable and consistent way to audit changes in Debian packages is to compare the full versions from the archive as shown above.
However, for packages that are hosted on Salsa, one can view the git history to gain additional insight into what exactly changed, when and why. For packages that are using version control, their location can be found in the Git-Vcs header in the debian/control file. For xz-utils the location is salsa.debian.org/debian/xz-utils.
Note that the Debian policy does not state anything about how Salsa should be used, or what git repository layout or development practices to follow. In practice most packages follow the DEP-14 proposal, and use git-buildpackage as the tool for managing changes and pushing and pulling them between upstream and salsa.debian.org.
To get the XZ Utils source, run:
$ gbp clone https://salsa.debian.org/debian/xz-utils.git
gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'
$ gbp clone https://salsa.debian.org/debian/xz-utils.git
gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'
At the time of writing this post the git history shows:
$ git log --graph --oneline
* bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2
* 4b769547 d: Remove the symlinks from -dev package.
* a39f3428 Correct the nocheck build profile
* 1b806b8d Import Debian changes 5.8.1-1.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h
* 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder
* 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding
* 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread
* d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
* c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state
* 831b55b9 liblzma: mt dec: Fix a comment
* b9d168ee liblzma: Add assertions to lzma_bufcpy()
* c8e0a489 DOS: Update Makefile to fix the build
* 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers
* 7ce38b31 Update THANKS
* 688e51bd Translations: Update the Croatian translation
* a6b54dde Prepare 5.8.0-1.
* 77d9470f Add 5.8 symbols.
* 9268eb66 Import 5.8.0
* 6f85ef4f Update upstream source from tag 'upstream/5.8.0'
\ \
* afba662b New upstream version 5.8.0
/
* 173fb5c6 doc/SHA256SUMS: Add 5.8.0
* db9258e8 Bump version and soname for 5.8.0
* bfb752a3 Add NEWS for 5.8.0
* 6ccbb904 Translations: Run "make -C po update-po"
* 891a5f05 Translations: Run po4a/update-po
* 4f52e738 Translations: Partially fix overtranslation in Serbian man pages
* ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage
* 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()
$ git log --graph --oneline
* bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2
* 4b769547 d: Remove the symlinks from -dev package.
* a39f3428 Correct the nocheck build profile
* 1b806b8d Import Debian changes 5.8.1-1.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h
* 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder
* 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding
* 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread
* d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
* c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state
* 831b55b9 liblzma: mt dec: Fix a comment
* b9d168ee liblzma: Add assertions to lzma_bufcpy()
* c8e0a489 DOS: Update Makefile to fix the build
* 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers
* 7ce38b31 Update THANKS
* 688e51bd Translations: Update the Croatian translation
* a6b54dde Prepare 5.8.0-1.
* 77d9470f Add 5.8 symbols.
* 9268eb66 Import 5.8.0
* 6f85ef4f Update upstream source from tag 'upstream/5.8.0'
\ \
* afba662b New upstream version 5.8.0
/
* 173fb5c6 doc/SHA256SUMS: Add 5.8.0
* db9258e8 Bump version and soname for 5.8.0
* bfb752a3 Add NEWS for 5.8.0
* 6ccbb904 Translations: Run "make -C po update-po"
* 891a5f05 Translations: Run po4a/update-po
* 4f52e738 Translations: Partially fix overtranslation in Serbian man pages
* ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage
* 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()
This shows both the changes on the debian/unstable branch as well as the intermediate upstream import branch, and the actual real upstream development branch. See my Debian source packages in git explainer for details of what these branches are used for.
To only view changes on the Debian branch, run git log --graph --oneline --first-parent or git log --graph --oneline -- debian.
The Debian branch should only have changes inside the debian/ subdirectory, which is easy to check with:
If the upstream in question signs commits or tags, they can be verified with e.g.:
$ git verify-tag v5.6.2
gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: issuer "lasse.collin@tukaani.org"
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired]
gpg: Note: This key has expired!
$ git verify-tag v5.6.2
gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: issuer "lasse.collin@tukaani.org"
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired]
gpg: Note: This key has expired!
The main benefit of reviewing changes in git is the ability to see detailed information about each individual change, instead of just staring at a massive list of changes without any explanations. In this example, to view all the upstream commits since the previous import to Debian, one would view the commit range from afba662b New upstream version 5.8.0 to fa1e8796 New upstream version 5.8.1 with git log --reverse -p afba662b...fa1e8796. However, a far superior way to review changes would be to browse this range using a visual git history viewer, such as gitk. Either way, looking at one code change at a time and reading the git commit message makes the review much easier.
Comparing Debian source packages to git contents
As stated in the beginning of the previous section, and worth repeating, there is no guarantee that the contents in the Debian packaging git repository matches what was actually uploaded to Debian. While the tag2upload project in Debian is getting more and more popular, Debian is still far from having any system to enforce that the git repository would be in sync with the Debian archive contents.
To detect such differences we can run diff across the Debian source packages downloaded with debsnap earlier (path source-xz-utils/xz-utils_5.8.1-2.debian) and the git repository cloned in the previous section (path xz-utils):
diff$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/
diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog
--- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700
+++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700
@@ -5,7 +5,7 @@
* Remove the symlinks from -dev, pointing to the lib package.
(Closes: #1109354)
- -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200
$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/
diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog
--- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700
+++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700
@@ -5,7 +5,7 @@
* Remove the symlinks from -dev, pointing to the lib package.
(Closes: #1109354)
- -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200
In the case above diff revealed that the timestamp in the changelog in the version uploaded to Debian is different from what was committed to git. This is not malicious, just a mistake by the maintainer who probably didn t run gbp tag immediately after upload, but instead some dch command and ended up with having a different timestamps in the git compared to what was actually uploaded to Debian.
Creating syntetic Debian packaging git repositories
If no Debian packaging git repository exists, or if it is lagging behind what was uploaded to Debian s archive, one can use git-buildpackage s import-dscs feature to create synthetic git commits based on the files downloaded by debsnap, ensuring the git contents fully matches what was uploaded to the archive. To import a single version there is gbp import-dsc (no s at the end), of which an example invocation would be:
$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc
Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'
$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc
Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'
Example commit history from a repository with commits added with gbp import-dsc:
An online example repository with only a few missing uploads added using gbp import-dsc can be viewed at salsa.debian.org/otto/xz-utils-2025-09-29/-/network/debian%2Funstable
An example repository that was fully crafted using gbp import-dscs can be viewed at salsa.debian.org/otto/xz-utils-gbp-import-dscs-debsnap-generated/-/network/debian%2Flatest.
There exists also dgit, which in a similar way creates a synthetic git history to allow viewing the Debian archive contents via git tools. However, its focus is on producing new package versions, so fetching a package with dgit that has not had the history recorded in dgit earlier will only show the latest versions:
$ dgit clone xz-utils
canonical suite name for unstable is sid
starting new git history
last upload to archive: NO git hash
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz...
dpkg-source: info: extracting xz-utils in unpacked
dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz
dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz
synthesised git commit from .dsc 5.8.1-2
HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium
dgit ok: ready for work in xz-utils
$ dgit/sid git log --graph --oneline
* f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid)
\
* 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago
* 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago
$ dgit clone xz-utils
canonical suite name for unstable is sid
starting new git history
last upload to archive: NO git hash
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz...
dpkg-source: info: extracting xz-utils in unpacked
dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz
dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz
synthesised git commit from .dsc 5.8.1-2
HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium
dgit ok: ready for work in xz-utils
$ dgit/sid git log --graph --oneline
* f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid)
\
* 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago
* 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago
Unlike git-buildpackage managed git repositories, the dgit managed repositories cannot incorporate the upstream git history and are thus less useful for auditing the full software supply-chain in git.
Comparing upstream source packages to git contents
Equally important to the note in the beginning of the previous section, one must also keep in mind that the upstream release source packages, often called release tarballs, are not guaranteed to have the exact same contents as the upstream git repository. Projects might strip out test data or extra development files from their release tarballs to avoid shipping unnecessary files to users, or projects might add documentation files or versioning information into the tarball that isn t stored in git. While a small minority, there are also upstreams that don t use git at all, so the plain files in a release tarball is still the lowest common denominator for all open source software projects, and exporting and importing source code needs to interface with it.
In the case of XZ, the release tarball has additional version info and also a sizeable amount of pregenerated compiler configuration files. Detecting and comparing differences between git contents and tarballs can of course be done manually by running diff across an unpacked tarball and a checked out git repository. If using git-buildpackage, the difference between the git contents and tarball contents can be made visible directly in the import commit.
In this XZ example, consider this git history:
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
The commit a522a226 was the upstream release commit, which upstream also tagged v5.8.1. The merge commit 2808ec2d applied the new upstream import branch contents on the Debian branch. Between these is the special commit fa1e8796 New upstream version 5.8.1 tagged upstream/v5.8. This commit and tag exists only in the Debian packaging repository, and they show what is the contents imported into Debian. This is generated automatically by git-buildpackage when running git import-orig --uscan for Debian packages with the correct settings in debian/gbp.conf. By viewing this commit one can see exactly how the upstream release tarball differs from the upstream git contents (if at all).
In the case of XZ, the difference is substantial, and shown below in full as it is very interesting:
To be able to easily inspect exactly what changed in the release tarball compared to git release tag contents, the best tool for the job is Meld, invoked via git difftool --dir-diff fa1e8796^..fa1e8796.
To compare changes across the new and old upstream tarball, one would need to compare commits afba662b New upstream version 5.8.0 and fa1e8796 New upstream version 5.8.1 by running git difftool --dir-diff afba662b..fa1e8796.
With all the above tips you can now go and try to audit your own favorite package in Debian and see if it is identical with upstream, and if not, how it differs.
Should the XZ backdoor have been detected using these tools?
The famous XZ Utils backdoor (CVE-2024-3094) consisted of two parts: the actual backdoor inside two binary blobs masqueraded as a test files (tests/files/bad-3-corrupt_lzma2.xz, tests/files/good-large_compressed.lzma), and a small modification in the build scripts (m4/build-to-host.m4) to extract the backdoor and plant it into the built binary. The build script was not tracked in version control, but generated with GNU Autotools at release time and only shipped as additional files in the release tarball.
The entire reason for me to write this post was to ponder if a diligent engineer using git-buildpackage best practices could have reasonably spotted this while importing the new upstream release into Debian. The short answer is no . The malicious actor here clearly anticipated all the typical ways anyone might inspect both git commits, and release tarball contents, and masqueraded the changes very well and over a long timespan.
First of all, XZ has for legitimate reasons for several carefully crafted .xz files as test data to help catch regressions in the decompression code path. The test files are shipped in the release so users can run the test suite and validate that the binary is built correctly and xz works properly. Debian famously runs massive amounts of testing in its CI and autopkgtest system across tens of thousands of packages to uphold high quality despite frequent upgrades of the build toolchain and while supporting more CPU architectures than any other distro. Test data is useful and should stay.
When git-buildpackage is used correctly, the upstream commits are visible in the Debian packaging for easy review, but the commit cf44e4b that introduced the test files does not deviate enough from regular sloppy coding practices to really stand out. It is unfortunately very common for git commit to lack a message body explaining why the change was done, and to not be properly atomic with test code and test data together in the same commit, and for commits to be pushed directly to mainline without using code reviews (the commit was not part of any PR in this case). Only another upstream developer could have spotted that this change is not on par to what the project expects, and that the test code was never added, only test data, and thus that this commit was not just a sloppy one but potentially malicious.
Secondly, the fact that a new Autotools file appeared (m4/build-to-host.m4) in the XZ Utils 5.6.0 is not suspicious. This is perfectly normal for Autotools. In fact, starting from XZ Utils version 5.8.1 it is now shipping a m4/build-to-host.m4 file that it actually uses now.
Spotting that there is anything fishy is practically impossible by simply reading the code, as Autotools files are full custom m4 syntax interwoven with shell script, and there are plenty of backticks () that spawn subshells and evals that execute variable contents further, which is just normal for Autotools. Russ Cox s XZ post explains how exactly the Autotools code fetched the actual backdoor from the test files and injected it into the build.
There is only one tiny thing that maybe a very experienced Autotools user could potentially have noticed: the serial 30 in the version header is way too high. In theory one could also have noticed this Autotools file deviates from what other packages in Debian ship with the same filename, such as e.g. the serial 3, serial 5a or 5b versions. That would however require and an insane amount extra checking work, and is not something we should plan to start doing. A much simpler solution would be to simply strongly recommend all open source projects to stop using Autotools to eventually get rid of it entirely.
Not detectable with reasonable effort
While planting backdoors is evil, it is hard not to feel some respect to the level of skill and dedication of the people behind this. I ve been involved in a bunch of security breach investigations during my IT career, and never have I seen anything this well executed.
If it hadn t slowed down SSH by ~500 milliseconds and been discovered due to that, it would most likely have stayed undetected for months or years. Hiding backdoors in closed source software is relatively trivial, but hiding backdoors in plain sight in a popular open source project requires some unusual amount of expertise and creativity as shown above.
Is the software supply-chain in Debian easy to audit?
While maintaining a Debian package source using git-buildpackage can make the package history a lot easier to inspect, most packages have incomplete configurations in their debian/gbp.conf, and thus their package development histories are not always correctly constructed or uniform and easy to compare. The Debian Policy does not mandate git usage at all, and there are many important packages that are not using git at all. Additionally the Debian Policy also allows for non-maintainers to upload new versions to Debian without committing anything in git even for packages where the original maintainer wanted to use git. Uploads that bypass git unfortunately happen surpisingly often.
Because of the situation, I am afraid that we could have multiple similar backdoors lurking that simply haven t been detected yet. More audits, that hopefully also get published openly, would be welcome! More people auditing the contents of the Debian archives would probably also help surface what tools and policies Debian might be missing to make the work easier, and thus help improve the security of Debian s users, and improve trust in Debian.
Is Debian currently missing some software that could help detect similar things?
To my knowledge there is currently no system in place as part of Debian s QA or security infrastructure to verify that the upstream source packages in Debian are actually from upstream. I ve come across a lot of packages where the debian/watch or other configs are incorrect and even cases where maintainers have manually created upstream tarballs as it was easier than configuring automation to work. It is obvious that for those packages the source tarball now in Debian is not at all the same as upstream. I am not aware of any malicious cases though (if I was, I would report them of course).
I am also aware of packages in the Debian repository that are misconfigured to be of type 1.0 (native) packages, mixing the upstream files and debian/ contents and having patches applied, while they actually should be configured as 3.0 (quilt), and not hide what is the true upstream sources. Debian should extend the QA tools to scan for such things. If I find a sponsor, I might build it myself as my next major contribution to Debian.
In addition to better tooling for finding mismatches in the source code, Debian could also have better tooling for tracking in built binaries what their source files were, but solutions like Fraunhofer-AISEC s supply-graph or Sony s ESSTRA are not practical yet. Julien Malka s post about NixOS discusses the role of reproducible builds, which may help in some cases across all distros.
Or, is Debian missing some policies or practices to mitigate this?
Perhaps more importantly than more security scanning, the Debian Developer community should switch the general mindset from anyone is free to do anything to valuing having more shared workflows. The ability to audit anything is severely hampered by the fact that there are so many ways to do the same thing, and distinguishing what is a normal deviation from a malicious deviation is too hard, as the normal can basically be almost anything.
Also, as there is no documented and recommended default workflow, both those who are old and new to Debian packaging might never learn any one optimal workflow, and end up doing many steps in the packaging process in a way that kind of works, but is actually wrong or unnecessary, causing process deviations that look malicious, but turn out to just be a result of not fully understanding what would have been the right way to do something.
In the long run, once individual developers workflows are more aligned, doing code reviews will become a lot easier and smoother as the excess noise of workflow differences diminishes and reviews will feel much more productive to all participants. Debian fostering a culture of code reviews would allow us to slowly move from the current practice of mainly solo packaging work towards true collaboration forming around those code reviews.
I have been promoting increased use of Merge Requests in Debian already for some time, for example by proposing DEP-18: Encourage Continuous Integration and Merge Request based Collaboration for Debian packages. If you are involved in Debian development, please give a thumbs up in dep-team/deps!21 if you want me to continue promoting it.
Can we trust open source software?
Yes and I would argue that we can only trust open source software. There is no way to audit closed source software, and anyone using e.g. Windows or MacOS just have to trust the vendor s word when they say they have no intentional or accidental backdoors in their software. Or, when the news gets out that the systems of a closed source vendor was compromised, like Crowdstrike some weeks ago, we can t audit anything, and time after time we simply need to take their word when they say they have properly cleaned up their code base.
In theory, a vendor could give some kind of contractual or financial guarantee to its customer that there are no preventable security issues, but in practice that never happens. I am not aware of a single case of e.g. Microsoft or Oracle would have paid damages to their customers after a security flaw was found in their software. In theory you could also pay a vendor more to have them focus more effort in security, but since there is no way to verify what they did, or to get compensation when they didn t, any increased fees are likely just pocketed as increased profit.
Open source is clearly better overall. You can, if you are an individual with the time and skills, audit every step in the supply-chain, or you could as an organization make investments in open source security improvements and actually verify what changes were made and how security improved.
If your organisation is using Debian (or derivatives, such as Ubuntu) and you are interested in sponsoring my work to improve Debian, please reach out.
Some progress upstream
Recently Sebastian Reichel at Collabora [1] has made a few related commits, apparently inspired in part by my kvetching on this blog.
Disconnecting and reconnecting PCI busses
At some point I noticed error message about the nvme device on
resume. I then learned how to disconnect and reconnect PCI buses in
Linux. I ended up with something like the following. At least the PCI
management seems to work. I can manually disconnect all the PCI busses
and rescan to connect them again on a running system. It presumably
helps that I am not using the nvme device in this system.
Minimal changes to upstream
With the ongoing work at collabora I decided to try a minimal patch
stack to get the pocket reform to boot. I added the following 3
commits (available from [3]).
This does look like some progress, probably thanks to Sebastien. Comparing with the logs in hibernate-pocket-12, the resume process is not bailing out complaining about PHY.
Attempt to reapply PCI reset patches
Following the procedure in hibernate-pocket-12, I attempted to
re-apply the pci reset patches [2]. In particular I followed the hints
output by b4.
Unfortunately there are too many conflicts now for me to sensibly
resolve.
even I managed to migrate my last setup to sway a few weeks ago. I was the last one you've been
waiting for dear X Strike Force, right?
Multi display support just works, no more
modeline hackery.
Oh and we can also remove those old
clipboard manager.
One oddity with sway I could not yet solve is that I had to delete the default
wallpaper /usr/share/backgrounds/sway/Sway_Wallpaper_Blue_1920x1080.png to allow it to load
the Debian wallpaper via
output * bg /usr/share/desktop-base/active-theme/wallpaper/contents/images/1920x1080.svg fill
Update: Thanks to Birger and Sebastian who easily could explain that. The sway-backgrounds package
ships a config snippet in /etc/sway/config.d and if that's included e.g. via include /etc/sway/config.d/*after setting the background in your ~/.config/sway/config it does the obvious and overrides
your own background configuration again. Didn't expect that but makes sense. So the right fix
is to just remove the sway-backgrounds package.
I also had a bit of fist fight with sway to make sure I've as much screen space available
as possible. So I tried to shrink fonts and remove borders.
Rest I guess is otherwise well documented. I settled on wofi as menu tool, cliphist for
clipboard access, of course waybar to be able to use the nm-applet, swayidle and swaylock
are probably also more or less standard for screen locking.
Having
Welcome to our 5th report from the Reproducible Builds project in 2025! Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. If you are interested in contributing to the Reproducible Builds project, please do visit the Contribute page on our website.
In this report:
Security audit of Reproducible Builds tools published
The Open Technology Fund s (OTF) security partner Security Research Labs recently an conducted audit of some specific parts of tools developed by Reproducible Builds. This form of security audit, sometimes called a whitebox audit, is a form testing in which auditors have complete knowledge of the item being tested. They auditors assessed the various codebases for resilience against hacking, with key areas including differential report formats in diffoscope, common client web attacks, command injection, privilege management, hidden modifications in the build process and attack vectors that might enable denials of service.
The audit focused on three core Reproducible Builds tools: diffoscope, a Python application that unpacks archives of files and directories and transforms their binary formats into human-readable form in order to compare them; strip-nondeterminism, a Perl program that improves reproducibility by stripping out non-deterministic information such as timestamps or other elements introduced during packaging; and reprotest, a Python application that builds source code multiple times in various environments in order to to test reproducibility.
OTF s announcement contains more of an overview of the audit, and the full 24-page report is available in PDF form as well.
[Colleagues] approached me to talk about a reproducibility issue they d been having with some R code. They d been running simulations that rely on generating samples from a multivariate normal distribution, and despite doing the prudent thing and using set.seed() to control the state of the random number generator (RNG), the results were not computationally reproducible. The same code, executed on different machines, would produce different random numbers. The numbers weren t just a little bit different in the way that we ve all wearily learned to expect when you try to force computers to do mathematics. They were painfully, brutally, catastrophically, irreproducible different. Somewhere, somehow, something broke.
present attestable builds, a new paradigm to provide strong source-to-binary correspondence in software artifacts. We tackle the challenge of opaque build pipelines that disconnect the trust between source code, which can be understood and audited, and the final binary artifact, which is difficult to inspect. Our system uses modern trusted execution environments (TEEs) and sandboxed build containers to provide strong guarantees that a given artifact was correctly built from a specific source code snapshot. As such it complements existing approaches like reproducible builds which typically require time-intensive modifications to existing build configurations and dependencies, and require independent parties to continuously build and verify artifacts.
The authors compare attestable builds with reproducible builds by noting an attestable build requires only minimal changes to an existing project, and offers nearly instantaneous verification of the correspondence between a given binary and the source code and build pipeline used to construct it , and proceed by determining that t he overhead (42 seconds start-up latency and 14% increase in build duration) is small in comparison to the overall build time.
Timo Pohl, Pavel Nov k, Marc Ohm and Michael Meier have published a paper called Towards Reproducibility for Software Packages in Scripting Language Ecosystems. The authors note that past research into Reproducible Builds has focused primarily on compiled languages and their ecosystems, with a further emphasis on Linux distribution packages:
However, the popular scripting language ecosystems potentially face unique issues given the systematic difference in distributed artifacts. This Systemization of Knowledge (SoK) [paper] provides an overview of existing research, aiming to highlight future directions, as well as chances to transfer existing knowledge from compiled language ecosystems. To that end, we work out key aspects in current research, systematize identified challenges for software reproducibility, and map them between the ecosystems.
Ultimately, the three authors find that the literature is sparse , focusing on few individual problems and ecosystems, and therefore identify space for more critical research.
Distribution work
In Debian this month:
Ian Jackson filed a bug against the debian-policy package in order to delve into an issue affecting Debian s support for cross-architecture compilation, multiple-architecture systems, reproducible builds SOURCE_DATE_EPOCH environment variable and the ability to recompile already-uploaded packages to Debian with a new/updated toolchain (binNMUs). Ian identifies a specific case, specifically in the libopts25-dev package, involving a manual page that had interesting downstream effects, potentially affecting backup systems. The bug generated a large number of replies, some of which have references to similar or overlapping issues, such as this one from 2016/2017.
There is now a Reproducibility Status link for each app on f-droid.org, listed on every app s page. Our verification server shows or based on its build results, where means our rebuilder reproduced the same APK file and means it did not. The IzzyOnDroid repository has developed a more elaborate system of badges which displays a for each rebuilder. Additionally, there is a sketch of a five-level graph to represent some aspects about which processes were run.
Hans compares the approach with projects such as Arch Linux and Debian that provide developer-facing tools to give feedback about reproducible builds, but do not display information about reproducible builds in the user-facing interfaces like the package management GUIs.
Arnout Engelen of the NixOS project has been working on reproducing the minimal installation ISO image. This month, Arnout has successfully reproduced the build of the minimal image for the 25.05 release without relying on the binary cache. Work on also reproducing the graphical installer image is ongoing.
In openSUSE news, Bernhard M. Wiedemann posted another monthly update for their work there.
Lastly in Fedora news, Jelle van der Waa opened issues tracking reproducible issues in Haskell documentation, Qt6 recording the host kernel and R packages recording the current date. The R packages can be made reproducible with packaging changes in Fedora.
diffoscope & disorderfsdiffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 295, 296 and 297 to Debian:
Don t rely on zipdetails --walk argument being available, and only add that argument on newer versions after we test for that. []
Review and merge support for NuGet packages from Omair Majid. []
Update copyright years. []
Merge support for an lzma comparator from Will Hollywood. [][]
Chris also merged an impressive changeset from Siva Mahadevan to make disorderfs more portable, especially on FreeBSD. disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues []. This was then uploaded to Debian as version 0.6.0-1.
Lastly, Vagrant Cascadian updated diffoscope in GNU Guix to version 296 [][] and 297 [][], and disorderfs to version 0.6.0 [][].
Website updates
Once again, there were a number of improvements made to our website this month including:
Chris Lamb:
Merged four or five suggestions from Guillem Jover for the GNU Autotools examples on the SOURCE_DATE_EPOCH example page []
Incorporated a number of fixes for the JavaScript SOURCE_DATE_EPOCH snippet from Sebastian Davis, which did not handle non-integer values correctly. []
Remove the JavaScript example that uses a fixed timezone on the SOURCE_DATE_EPOCH page. []
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility.
However, Holger Levsen posted to our mailing list this month in order to bring a wider awareness to funding issues faced by the Oregon State University (OSU) Open Source Lab (OSL). As mentioned on OSL s public post, recent changes in university funding makes our current funding model no longer sustainable [and that] unless we secure $250,000 in committed funds, the OSL will shut down later this year . As Holger notes in his post to our mailing list, the Reproducible Builds project relies on hardware nodes hosted there. Nevertheless, Lance Albertson of OSL posted an update to the funding situation later in the month with broadly positive news.
Separate to this, there were various changes to the Jenkins setup this month, which is used as the backend driver of for both tests.reproducible-builds.org and reproduce.debian.net, including:
Migrating the central jenkins.debian.net server AMD Opteron to Intel Haswell CPUs. Thanks to IONOS for hosting this server since 2012.
After testing it for almost ten years, the i386 architecture has been dropped from tests.reproducible-builds.org. This is because that, with the upcoming release of Debian trixie, i386 is no longer supported as a regular architecture there will be no official kernel and no Debian installer for i386 systems. As a result, a large number of nodes hosted by Infomaniak have been retooled from i386 to amd64.
Another node, ionos17-amd64.debian.net, which is used for verifying packages for all.reproduce.debian.net (hosted by IONOS) has had its memory increased from 40 to 64GB, and the number of cores doubled to 32 as well. In addition, two nodes generously hosted by OSUOSL have had their memory doubled to 16GB.
Lastly, we have been granted access to more riscv64 architecture boards, so now we have seven such nodes, all with 16GB memory and 4 cores that are verifying packages for riscv64.reproduce.debian.net. Many thanks to PLCT Lab, ISCAS for providing those.
Outside of this, a number of smaller changes were also made by Holger Levsen:
Disable testing of the i386 architecture. [][][][][]
Document the current disk usage. [][]
Address some image placement now that we only test three architectures. []
Keep track of build performance. []
Misc:
Fix a (harmless) typo in the multiarch_versionskew script. []
In addition, Jochen Sprickerhof made a series of changes related to reproduce.debian.net:
Add out of memory detection to the statistics page. []
Reverse the sorting order on the statistics page. [][][][]
Improve the spacing between statistics groups. []
Update a (hard-coded) line number in error message detection pertaining to a debrebuild line number. []
Support Debian unstable in the rebuilder-debian.sh script. []]
Rely on rebuildctl to sync only arch-specific packages. [][]
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. This month, we wrote a large number of such patches, including:
0xFFFF: Use SOURCE_DATE_EPOCH for date in manual pages.
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
How long has it been since you last saw a conversation over different blogs
syndicated at the same planet? Well, it s one of the good memories of the
early 2010s. And there is an opportunity to re-engage!
I came across Evgeni s post naming things is
hard in Planet
Debian. So, what names have I given my
computers?
I have had many since the mid-1990s I also had several during the decade
before that, but before Linux, my computers didn t hve a formal
name. Naming my computers something nice Linux gave me.
I have forgotten many. Some of the names I have used:
My years in Iztacala: I worked as a sysadmin
between 1999 and 2003. When I arrived, we already had two servers,
campus and tlali, and one computer pending installation, ollin. The
credit for their names is not mine.
campus: A mighty SPARCstation
5! Because it was the
main (and for some time, the only!) server in our campus.
tlali: A regular PC used as a Linux server. Tlali means something
like lands in n huatl, the prehispanic language spoken in central
Mexico. My workplace was Iztacala, which translates as the place
where there are white houses ; tlali and cali are related words.
ollin: was a big IBM RS/6000 system running AIX. It came to us,
probably already obsolete, as a (useless) donation from Fundaci n
UNAM; I don t recall the exact model, but it looked very much like
this one. Ran on
AIX. We had no software for it, and frankly never really got it to
be productive. Funnily, its name Ollin means movement in N huatl.
I added some servers to the lineup during the two years I was in Iztacala:
tlamantli: An Alpha
21164 server that doubled
as my desktop. Given the tradition in Iztacala of naming things in
N huatl, but trying to be somewhat funny, tlamantli just means a
thing; I understand the word is usually bound to a quantifier.
tepancuate: A regular PC system we set up with OpenBSD as a
firewall. It means wall in N huatl.
Following the first CONSOL (National Free Software Conference), I was
invited to work as a programmer at UPN, Universidad Pedag gica
Nacional in 2003 2004. There I was not directly
in charge of any of the servers (I mostly used ajusco, managed by
V ctor, named after the mountain on whose
slopes our campus was). But my only computer there was:
shmate: , meaning old rag in yiddish. The word shmate is used like
thingy, although it would usually mean old and slightly worn-out
thingy. It was a quite nice machine, though. I had a Pentium 4 with
512MB RAM, not bad for 2003!
I started my present work at Instituto de Investigaciones Econ micas,
UNAM 20 years ago(!), in 2005. Here I am a
systems administrator, so naturally I am in charge of the servers. And
over the years, we have had a fair share of machines:
mosca: is my desktop. It has changed hardware several times (of
course) over the years, but it s still the same Debian Sid install I
did in January 2005 (I must have reinstalled once, when I got it
replaced by an AMD64). Its name is the Spanish name for the common
fly. I have often used it to describe my work, since I got in the early
1990s an automated bilingual translator called TRANSLATE; it came on
seven 5.25 floppies. As a teenager, I somehow got my hands on a copy,
and installed it in my 80386SX. Fed it its own README to see how it
fared. And the first sentence made me burst in laughter: TRANSLATE
performs on the fly translation TRADUCE realiza traducci n sobre la
mosca . Starting then, I always think of on the fly as sobre la
mosca . As Groucho said, I guess Time flies like an arrow, but
fruit flies like a banana.
lafa When I got there, we didn t have any servers; for some time, I
took one of the computer lab s systems to serve our web page and
receive mail. But when we got some budget approved, we bought a
fsckin-big server. Big as in four-rack-units. Double CPUs (not
multicore, but two independent early Xeon CPUs, if I m not
mistaken. Still, it was still a 32 bits system). (lafa) is a
big, more flexible kind of Arab bread than pita; I loved it when I
lived in Israel. And there is an album (and song) by
Teapacks, an Israeli group I am very
fond of, hajaim shelja belafa (your life in a lafa), saying, hey,
brother! Your life is in a lafa. You throw everything in a big
pita. You didn t have time to chew, you already swallowed it .
joma: Our firewall. means wall in Hebrew.
baktun: lafa was great, but over the years, it got old. After many
years, I finally got the Institute to buy a second server. We got it in
December 2012. There was a lot of noise around then because the world
was supposed to end on 2012.12.21, as the Mayan calendar reached a
full long cycle. This long cycle is called /baktun/. So, it was fitting
as the name of the new server.
teom: As lafa was almost immediately decomissioned and turned into
a virtual machine in the much bigger baktun,, I wanted to split
services, make off-hardware backups, and such. Almost two years later,
my request was approved and we bought a second server. But instead of
buying it from a regular provider, we got it off a stash of machines
bought by our university s central IT
entity. To my surprise, it had the exact same
hardware configuration as baktun, bought two years earlier. Even the
serial number was absurdly close. So, I had it as baktun s long-lost
twin. Hence, (transliterated as teom), the Hebrew word for
twin. About a year after teom arrived to my life, my twin children
were also born, but their naming followed a completely different logic
process than my computers
At home or on the road: I am sure I am missing several systems over the
years.
pato: The earliest system I had that I remember giving a name to. I
built a 80386SX in 1991, buying each component separately. The box had
a 1-inch square for integrators to put their branding And after some
time, I carefully printed and applied a label that said Catarm quina
PATO (the first word, very small). Pato (duck) is how we d call a
no-brand system. Catarm quina because it was the system where I ran
my BBS, CatarSYS (1992-1994).
malenkaya: In 2008 I got a 9 Acer Aspire One
netbook
(Atom N270 i386, 1GB RAM). I really loved that machine! Although it was
quite limited, it was my main computer while on the road for almost
five years. malenkaya means small (for female) in Russian.
matlalli: After malenkaya started being too limited for my regular
use, I bought its successor Acer Aspire One
model. This
one was way larger (10.1 inches screen) and I wasn t too happy about
it at the beginning, but I ended up loving it. So much, in fact, that
we bought at least four very similar such computers for us and our
family members. This computer was dirt cheap, and endured five further
years of lugging everywhere. matlalli is due to its turquoise color:
it is the N huatl word for blue or green.
cajita: In 2014 I got a beautiful Cubox i4
Pro
computer. It took me some time to get it to boot and be generally
useful, but it ended up being my home server for many years, until I
had a power supply malfunction which bricked it. cajita means little
box in Spanish.
pitentzin: Another 10.1 Acer Aspire One (the last in the lineup; the
CPU is a Celeron 877, so it does run AMD64, and it supports up to 16GB
RAM, I think I have it with 12). We originally bought it for my family
in Argentina, but they didn t really use it much, and after a couple of
years we got it back. We decided it would be the computer for the kids,
at least for the time being. And although it is a 2013 laptop, it s
still our everyday media station driver. Oh, and the name pitentzin?
N huatl for /children/.
tliltik: In 2018, I bought a second-hand Thinkpad X230. It was my
daily driver for about three years. I reflashed its firmware with
CoreBoot, and repeated the experience for seven people IIRC in
DebConf18. With it, I learned to love the Thinkpad keyboard. Naturally
for a thinkpad, tliltik means black in N huatl.
uesebe: When COVID struck, we were all sent home, and my university
lent me a nice recently bought Intel i7 HP laptop. At first, I didn t
want to mess up its Windows install (so I set up a USB-drive-based
installation, hence the name uesebe); when it was clear the lockdown
was going to be long (and that tliltik had too many aches to be used
for my daily work), I transferred the install to its HDD and used it
throughout the pandemic, until mid 2022.
bolex: I bought this computer for my father in 2020. After he passed
away in May 2022, I took his computer, and named it bolex because
that s the brand of the 8mm cinema camera he loved and had since 1955,
and with which he created most of his
films. It is really an entry-level machine,
though (a single-core, dual-threaded Celeron), and it was too limited
when I started distance-teaching again, so I had to store it as an
emergency system.
yogurtu: During the pandemics, I spent quite a bit of time fiddling
with the Raspberry Pi family. But all in all, while they are nice
machines for many uses, they are too limited to be daily drivers. Or
even enough for taking i.e. to Debconf and have them be my conference
computer. I bought an almost-new-but-used ( 2 year old) Yoga C630 ARM
laptop. I
often brag about my happy experience with it, and how it brings a
reasonably powerful ARM Linux system to my everyday life. In our last
DebConf, I didn t even pick up my USB-C power connector every day; the
battery just lasts over ten hours of active work. But I m not here
doing ads, right? yogurtu naturally is derived from the Yoga brand
it has, but is taken from Yogurtu
Ngh , a fictional
character by the Argentinian comical-musical group Les
Luthiers, that has marked my life.
misnenet: Towards mid 2023, when it was clear that bolex would not
be a good daily driver, and considering we would be spending six months
in Argentina, I bought a new desktop system. It seems I have something
for small computers: I decided for a refurbished HP EliteDesk 800 G5
Mini
i7
system. I picked it because, at close to 18 18 3.5cm it perfectly fits
in my DebConf18 bag. A laptop, it is clearly not, but it can easily
travel with me when needed. Oh, and the name? Because for this model,
HP uses different enclosures based on the kind of processor: The i3
model has a flat, black aluminum top But mine has lots of tiny
holes, covering two areas of roughly 15 7cm, with a tiny hole every
~2mm, and with a solid strip between them. Of course,
(misnenet, in Hebrew) means strainer.
Most of my Debian contributions this month were
sponsored by Freexian.
You can also support my work directly via
Liberapay.
OpenSSH
Changes in dropbear 2025.87 broke OpenSSH s regression
tests. I cherry-picked the fix.
I reviewed and merged patches from Luca
Boccassi to
send and accept the COLORTERM and NO_COLOR environment variables.
Python team
Following up on last month, I fixed some
more uscan errors:
python-ewokscore
python-ewoksdask
python-ewoksdata
python-ewoksorange
python-ewoksutils
python-processview
python-rsyncmanager
I upgraded these packages to new upstream versions:
python-mastodon (in the course of which I found
#1101140 in blurhash-python and
proposed a small
cleanup to slidge)
python-model-bakery
python-multidict
python-pip
python-rsyncmanager
python-service-identity
python-setproctitle
python-telethon
python-trio
python-typing-extensions
responses
setuptools-scm
trove-classifiers
zope.testrunner
In bookworm-backports, I updated python-django to 3:4.2.19-1.
Although Debian s upgrade to python-click 8.2.0 was
reverted for the time being, I fixed a
number of related problems anyway since we re going to have to deal with it eventually:
dh-python dropped its dependency on python3-setuptools in 6.20250306, which
was long overdue, but it had quite a bit of fallout; in most cases this was
simply a question of adding build-dependencies on python3-setuptools, but in
a few cases there was a missing build-dependency on
python3-typing-extensions which had previously been pulled in as a
dependency of python3-setuptools. I fixed these bugs resulting from this:
We agreed to remove python-pytest-flake8.
In support of this, I removed unnecessary build-dependencies from
pytest-pylint, python-proton-core, python-pyzipper, python-tatsu,
python-tatsu-lts, and python-tinycss, and filed #1101178 on
eccodes-python and #1101179 on
rpmlint.
There was a dnspython autopkgtest regression on
s390x. I independently tracked that down
to a pylsqpack bug and came up with a reduced test case before realizing
that Pranav P had already been working on it; we then worked together on it
and I uploaded their patch to Debian.
I fixed various other build/test failures:
Hey everyone!
The 2024 Linux Display Next
hackfest concluded
in May, and its outcomes continue to shape the Linux Display stack. Igalia
hosted this year s event in A Coru a, Spain, bringing together leading experts
in the field. Samuel Iglesias and I
organized this year s edition and this blog post summarizes the experience and
its fruits.
One of the highlights of this year s hackfest was the wide range of backgrounds
represented by our 40 participants (both on-site and remotely). Developers and
experts from various companies and open-source projects came together to
advance the Linux Display ecosystem. You can find the list of participants
here.
The event covered a broad spectrum of topics affecting the development of Linux
projects, user experiences, and the future of display technologies on Linux.
From cutting-edge topics to long-term discussions, you can check the event
agenda
here.
Organization Highlights
The hackfest was marked by in-depth discussions and knowledge sharing among
Linux contributors, making everyone inspired, informed, and connected to the
community. Building on feedback from the previous year, we refined the
unconference format to enhance participant preparation and engagement.
Structured Agenda and Timeboxes: Each session had a defined scope, time
limit (1h20 or 2h10), and began with an introductory talk on the topic.
Participant-Led Discussions: We pre-selected in-person participants to
lead discussions, allowing them to prepare introductions, resources, and
scope.
Transparent Scheduling: The schedule was shared in advance as GitHub
issues, encouraging participants to review and prepare for sessions of
interest.
Engaging Sessions: The hackfest featured a variety of topics, including
presentations and discussions on how participants were addressing specific
subjects within their companies.
No Breakout Rooms, No Overlaps: All participants chose to attend all
sessions, eliminating the need for separate breakout rooms. We also adapted
run-time schedule to keep everybody involved in the same topics.
Real-time Updates: We provided notifications and updates through
dedicated emails and the event matrix room.
Strengthening Community Connections: The hackfest offered ample
opportunities for networking among attendees.
Social Events: Igalia sponsored coffee breaks, lunches, and a dinner at
a local restaurant.
Museum Visit: Participants enjoyed a sponsored visit to the Museum of
Estrela Galicia Beer (MEGA).
Fruitful Discussions and Follow-up
The structured agenda and breaks allowed us to cover multiple topics during the
hackfest. These discussions have led to new display feature development and
improvements, as evidenced by patches, merge requests, and implementations in
project repositories and mailing lists.
With the KMS color management API taking shape, we discussed refinements and
best approaches to cover the variety of color pipeline from different
hardware-vendors. We are also investigating techniques for a performant
SDR<->HDR content reproduction and reducing latency and power consumption when
using the color blocks of the hardware.
Color Management/HDR
Color Management and HDR continued to be the hottest topic of the hackfest. We
had three sessions dedicated to discuss Color and HDR across Linux Display
stack layers.
Color/HDR (Kernel-Level)
Harry Wentland (AMD) led this session.
Here, kernel Developers shared the Color Management pipeline of AMD, Intel and
NVidia. We counted with diagrams and explanations from HW-vendors developers
that discussed differences, constraints and paths to fit them into the KMS
generic color management properties such as advertising modeset needs,
IN\_FORMAT, segmented LUTs,
interpolation types, etc. Developers from Qualcomm and ARM also added
information regarding their hardware.
Upstream work related to this session:
Color/HDR (Compositor-Level)
Sebastian Wick (RedHat) led this session.
It started with Sebastian s presentation covering Wayland color protocols and
compositor implementation. Also, an explanation of APIs provided by Wayland and
how they can be used to achieve better color management for applications and
discussions around ICC profiles and color representation metadata. There was
also an intensive Q&A about LittleCMS with Marti Maria.
Upstream work related to this session:
Color/HDR (Use Cases and Testing)
Christopher Cameron (Google) and Melissa Wen (Igalia) led this session.
In contrast to the other sessions, here we focused less on implementation and
more on brainstorming and reflections of real-world SDR and HDR transformations
(use and validation) and gainmaps. Christopher gave a nice presentation
explaining HDR gainmap images and how we should think of HDR. This presentation
and Q&A were important to put participants at the same page of how to
transition between SDR and HDR and somehow emulating HDR.
We also discussed on the usage of a kernel background color property.
Finally, we discussed a bit about
Chamelium
and the future of VKMS (future work and maintainership).
Power Savings vs Color/Latency
Mario Limonciello (AMD) led this session.
Mario gave an introductory presentation about AMD ABM (adaptive backlight
management) that is similar to Intel DPST. After some discussions, we agreed on
exposing a kernel property for power saving policy. This work was already
merged on kernel and the userspace support is under development.
Upstream work related to this session:
Strategy for video and gaming use-cases
Leo Li (AMD) led this session.
Miguel Casas (Google) started this session with a presentation of Overlays in
Chrome/OS Video, explaining the main goal of power saving by switching off GPU
for accelerated compositing and the challenges of different colorspace/HDR for
video on Linux.
Then Leo Li presented different strategies for video and gaming and we
discussed the userspace need of more detailed feedback mechanisms to understand
failures when offloading. Also, creating a debugFS interface came up as a tool
for debugging and analysis.
Real-time scheduling and async KMS API
Xaver Hugl (KDE/BlueSystems) led this session.
Compositor developers have exposed some issues with doing real-time scheduling
and async page flips. One is that the Kernel limits the lifetime of realtime
threads and if a modeset takes too long, the thread will be killed and thus the
compositor as well. Also, simple page flips take longer than expected and
drivers should optimize them.
Another issue is the lack of feedback to compositors about hardware programming
time and commit deadlines (the lastest possible time to commit). This is
difficult to predict from drivers, since it varies greatly with the type of
properties. For example, color management updates take much longer.
In this regard, we discusssed implementing a hw_done callback to timestamp
when the hardware programming of the last atomic commit is complete. Also an
API to pre-program color pipeline in a kind of A/B scheme. It may not be
supported by all drivers, but might be useful in different ways.
VRR/Frame Limit, Display Mux, Display Control, and more and beer
We also had sessions to discuss a new KMS API to mitigate headaches on VRR
and Frame Limit as different brightness level at different refresh rates,
abrupt changes of refresh rates, low frame rate compensation (LFC) and precise
timing in VRR more.
On Display Control we discussed features missing in the current KMS
interface for HDR mode, atomic backlight settings, source-based tone mapping,
etc. We also discussed the need of a place where compositor developers can post
TODOs to be developed by KMS people.
The Content-adaptive Scaling and Sharpening session focused on sharpening
and scaling filters. In the Display Mux session, we discussed proposals to
expose the capability of dynamic mux switching display signal between discrete
and integrated GPUs.
In the last session of the 2024 Display Next Hackfest, participants
representing different compositors summarized current and future work and built
a Linux Display wish list , which includes: improvements to VTTY and HDR
switching, better dmabuf API for multi-GPU support, definition of tone mapping,
blending and scaling sematics, and wayland protocols for advertising to clients
which colorspaces are supported.
We closed this session with a status update on feature development by
compositors, including but not limited to: plane offloading (from libcamera to
output) / HDR video offloading (dma-heaps) / plane-based scrolling for web
pages, color management / HDR / ICC profiles support, addressing issues such as
flickering when color primaries don t match, etc.
After three days of intensive discussions, all in-person participants went to a
guided tour at the Museum of Extrela Galicia beer (MEGA), pouring and tasting
the most famous local beer.
Feedback and Future Directions
Participants provided valuable feedback on the hackfest, including suggestions
for future improvements.
Schedule and Break-time Setup: Having a pre-defined agenda and schedule
provided a better balance between long discussions and mental refreshments,
preventing the fatigue caused by endless discussions.
Action Points: Some participants recommended explicitly asking for action
points at the end of each session and assigning people to follow-up tasks.
Remote Participation: Remote attendees appreciated the inclusive setup
and opportunities to actively participate in discussions.
Technical Challenges: There were bandwidth and video streaming issues
during some sessions due to the large number of participants.
Thank you for joining the 2024 Display Next Hackfest
We can t help but thank the 40 participants, who engaged in-person or virtually
on relevant discussions, for a collaborative evolution of the Linux display
stack and for building an insightful agenda.
A big thank you to the leaders and presenters of the nine sessions: Christopher
Cameron (Google), Harry Wentland (AMD), Leo Li (AMD), Mario Limoncello (AMD),
Sebastian Wick (RedHat) and Xaver Hugl (KDE/BlueSystems) for the effort in
preparing the sessions, explaining the topic and guiding discussions. My
acknowledge to the others in-person participants that made such an effort to
travel to A Coru a: Alex Goins (NVIDIA), David Turner (Raspberry Pi), Georges
Stavracas (Igalia), Joan Torres (SUSE), Liviu Dudau (Arm), Louis Chauvet
(Bootlin), Robert Mader (Collabora), Tian Mengge (GravityXR), Victor Jaquez
(Igalia) and Victoria Brekenfeld (System76). It was and awesome opportunity to
meet you and chat face-to-face.
Finally, thanks virtual participants who couldn t make it in person but
organized their days to actively participate in each discussion, adding
different perspectives and valuable inputs even remotely: Abhinav Kumar
(Qualcomm), Chaitanya Borah (Intel), Christopher Braga (Qualcomm), Dor Askayo
(Red Hat), Jiri Koten (RedHat), Jonas dahl (Red Hat), Leandro Ribeiro
(Collabora), Marti Maria (Little CMS), Marijn Suijten, Mario Kleiner, Martin
Stransky (Red Hat), Michel D nzer (Red Hat), Miguel Casas-Sanchez (Google),
Mitulkumar Golani (Intel), Naveen Kumar (Intel), Niels De Graef (Red Hat),
Pekka Paalanen (Collabora), Pichika Uday Kiran (AMD), Shashank Sharma (AMD),
Sriharsha PV (AMD), Simon Ser, Uma Shankar (Intel) and Vikas Korjani (AMD).
We look forward to another successful Display Next hackfest, continuing to
drive innovation and improvement in the Linux display ecosystem!
Hey everyone!
The 2024 Linux Display Next
hackfest concluded
in May, and its outcomes continue to shape the Linux Display stack. Igalia
hosted this year s event in A Coru a, Spain, bringing together leading experts
in the field. Samuel Iglesias and I
organized this year s edition and this blog post summarizes the experience and
its fruits.
One of the highlights of this year s hackfest was the wide range of backgrounds
represented by our 40 participants (both on-site and remotely). Developers and
experts from various companies and open-source projects came together to
advance the Linux Display ecosystem. You can find the list of participants
here.
The event covered a broad spectrum of topics affecting the development of Linux
projects, user experiences, and the future of display technologies on Linux.
From cutting-edge topics to long-term discussions, you can check the event
agenda
here.
Organization Highlights
The hackfest was marked by in-depth discussions and knowledge sharing among
Linux contributors, making everyone inspired, informed, and connected to the
community. Building on feedback from the previous year, we refined the
unconference format to enhance participant preparation and engagement.
Structured Agenda and Timeboxes: Each session had a defined scope, time
limit (1h20 or 2h10), and began with an introductory talk on the topic.
Participant-Led Discussions: We pre-selected in-person participants to
lead discussions, allowing them to prepare introductions, resources, and
scope.
Transparent Scheduling: The schedule was shared in advance as GitHub
issues, encouraging participants to review and prepare for sessions of
interest.
Engaging Sessions: The hackfest featured a variety of topics, including
presentations and discussions on how participants were addressing specific
subjects within their companies.
No Breakout Rooms, No Overlaps: All participants chose to attend all
sessions, eliminating the need for separate breakout rooms. We also adapted
run-time schedule to keep everybody involved in the same topics.
Real-time Updates: We provided notifications and updates through
dedicated emails and the event matrix room.
Strengthening Community Connections: The hackfest offered ample
opportunities for networking among attendees.
Social Events: Igalia sponsored coffee breaks, lunches, and a dinner at
a local restaurant.
Museum Visit: Participants enjoyed a sponsored visit to the Museum of
Estrela Galicia Beer (MEGA).
Fruitful Discussions and Follow-up
The structured agenda and breaks allowed us to cover multiple topics during the
hackfest. These discussions have led to new display feature development and
improvements, as evidenced by patches, merge requests, and implementations in
project repositories and mailing lists.
With the KMS color management API taking shape, we discussed refinements and
best approaches to cover the variety of color pipeline from different
hardware-vendors. We are also investigating techniques for a performant
SDR<->HDR content reproduction and reducing latency and power consumption when
using the color blocks of the hardware.
Color Management/HDR
Color Management and HDR continued to be the hottest topic of the hackfest. We
had three sessions dedicated to discuss Color and HDR across Linux Display
stack layers.
Color/HDR (Kernel-Level)
Harry Wentland (AMD) led this session.
Here, kernel Developers shared the Color Management pipeline of AMD, Intel and
NVidia. We counted with diagrams and explanations from HW-vendors developers
that discussed differences, constraints and paths to fit them into the KMS
generic color management properties such as advertising modeset needs,
IN\_FORMAT, segmented LUTs,
interpolation types, etc. Developers from Qualcomm and ARM also added
information regarding their hardware.
Upstream work related to this session:
Color/HDR (Compositor-Level)
Sebastian Wick (RedHat) led this session.
It started with Sebastian s presentation covering Wayland color protocols and
compositor implementation. Also, an explanation of APIs provided by Wayland and
how they can be used to achieve better color management for applications and
discussions around ICC profiles and color representation metadata. There was
also an intensive Q&A about LittleCMS with Marti Maria.
Upstream work related to this session:
Color/HDR (Use Cases and Testing)
Christopher Cameron (Google) and Melissa Wen (Igalia) led this session.
In contrast to the other sessions, here we focused less on implementation and
more on brainstorming and reflections of real-world SDR and HDR transformations
(use and validation) and gainmaps. Christopher gave a nice presentation
explaining HDR gainmap images and how we should think of HDR. This presentation
and Q&A were important to put participants at the same page of how to
transition between SDR and HDR and somehow emulating HDR.
We also discussed on the usage of a kernel background color property.
Finally, we discussed a bit about
Chamelium
and the future of VKMS (future work and maintainership).
Power Savings vs Color/Latency
Mario Limonciello (AMD) led this session.
Mario gave an introductory presentation about AMD ABM (adaptive backlight
management) that is similar to Intel DPST. After some discussions, we agreed on
exposing a kernel property for power saving policy. This work was already
merged on kernel and the userspace support is under development.
Upstream work related to this session:
Strategy for video and gaming use-cases
Leo Li (AMD) led this session.
Miguel Casas (Google) started this session with a presentation of Overlays in
Chrome/OS Video, explaining the main goal of power saving by switching off GPU
for accelerated compositing and the challenges of different colorspace/HDR for
video on Linux.
Then Leo Li presented different strategies for video and gaming and we
discussed the userspace need of more detailed feedback mechanisms to understand
failures when offloading. Also, creating a debugFS interface came up as a tool
for debugging and analysis.
Real-time scheduling and async KMS API
Xaver Hugl (KDE/BlueSystems) led this session.
Compositor developers have exposed some issues with doing real-time scheduling
and async page flips. One is that the Kernel limits the lifetime of realtime
threads and if a modeset takes too long, the thread will be killed and thus the
compositor as well. Also, simple page flips take longer than expected and
drivers should optimize them.
Another issue is the lack of feedback to compositors about hardware programming
time and commit deadlines (the lastest possible time to commit). This is
difficult to predict from drivers, since it varies greatly with the type of
properties. For example, color management updates take much longer.
In this regard, we discusssed implementing a hw_done callback to timestamp
when the hardware programming of the last atomic commit is complete. Also an
API to pre-program color pipeline in a kind of A/B scheme. It may not be
supported by all drivers, but might be useful in different ways.
VRR/Frame Limit, Display Mux, Display Control, and more and beer
We also had sessions to discuss a new KMS API to mitigate headaches on VRR
and Frame Limit as different brightness level at different refresh rates,
abrupt changes of refresh rates, low frame rate compensation (LFC) and precise
timing in VRR more.
On Display Control we discussed features missing in the current KMS
interface for HDR mode, atomic backlight settings, source-based tone mapping,
etc. We also discussed the need of a place where compositor developers can post
TODOs to be developed by KMS people.
The Content-adaptive Scaling and Sharpening session focused on sharpening
and scaling filters. In the Display Mux session, we discussed proposals to
expose the capability of dynamic mux switching display signal between discrete
and integrated GPUs.
In the last session of the 2024 Display Next Hackfest, participants
representing different compositors summarized current and future work and built
a Linux Display wish list , which includes: improvements to VTTY and HDR
switching, better dmabuf API for multi-GPU support, definition of tone mapping,
blending and scaling sematics, and wayland protocols for advertising to clients
which colorspaces are supported.
We closed this session with a status update on feature development by
compositors, including but not limited to: plane offloading (from libcamera to
output) / HDR video offloading (dma-heaps) / plane-based scrolling for web
pages, color management / HDR / ICC profiles support, addressing issues such as
flickering when color primaries don t match, etc.
After three days of intensive discussions, all in-person participants went to a
guided tour at the Museum of Extrela Galicia beer (MEGA), pouring and tasting
the most famous local beer.
Feedback and Future Directions
Participants provided valuable feedback on the hackfest, including suggestions
for future improvements.
Schedule and Break-time Setup: Having a pre-defined agenda and schedule
provided a better balance between long discussions and mental refreshments,
preventing the fatigue caused by endless discussions.
Action Points: Some participants recommended explicitly asking for action
points at the end of each session and assigning people to follow-up tasks.
Remote Participation: Remote attendees appreciated the inclusive setup
and opportunities to actively participate in discussions.
Technical Challenges: There were bandwidth and video streaming issues
during some sessions due to the large number of participants.
Thank you for joining the 2024 Display Next Hackfest
We can t help but thank the 40 participants, who engaged in-person or virtually
on relevant discussions, for a collaborative evolution of the Linux display
stack and for building an insightful agenda.
A big thank you to the leaders and presenters of the nine sessions: Christopher
Cameron (Google), Harry Wentland (AMD), Leo Li (AMD), Mario Limoncello (AMD),
Sebastian Wick (RedHat) and Xaver Hugl (KDE/BlueSystems) for the effort in
preparing the sessions, explaining the topic and guiding discussions. My
acknowledge to the others in-person participants that made such an effort to
travel to A Coru a: Alex Goins (NVIDIA), David Turner (Raspberry Pi), Georges
Stavracas (Igalia), Joan Torres (SUSE), Liviu Dudau (Arm), Louis Chauvet
(Bootlin), Robert Mader (Collabora), Tian Mengge (GravityXR), Victor Jaquez
(Igalia) and Victoria Brekenfeld (System76). It was and awesome opportunity to
meet you and chat face-to-face.
Finally, thanks virtual participants who couldn t make it in person but
organized their days to actively participate in each discussion, adding
different perspectives and valuable inputs even remotely: Abhinav Kumar
(Qualcomm), Chaitanya Borah (Intel), Christopher Braga (Qualcomm), Dor Askayo,
Jiri Koten (RedHat), Jonas dahl (Red Hat), Leandro Ribeiro
(Collabora), Marti Maria (Little CMS), Marijn Suijten, Mario Kleiner, Martin
Stransky (Red Hat), Michel D nzer (Red Hat), Miguel Casas-Sanchez (Google),
Mitulkumar Golani (Intel), Naveen Kumar (Intel), Niels De Graef (Red Hat),
Pekka Paalanen (Collabora), Pichika Uday Kiran (AMD), Shashank Sharma (AMD),
Sriharsha PV (AMD), Simon Ser, Uma Shankar (Intel) and Vikas Korjani (AMD).
We look forward to another successful Display Next hackfest, continuing to
drive innovation and improvement in the Linux display ecosystem!
por Allythy
O Debian Day um evento anual que celebra o anivers rio do Debian, uma das
distribui es GNU/Linux mais importante do Software Livre, criada em 16 de
Agosto de 1993, por Ian Murdock.
No ltimo s bado (17/08/2024) no Sebrae-RN comemoramos os 31 anos Debian em
Natal, no Rio Grande do Norte. A celebra o, foi organizada pela
PotiLivre(Comunidade Potiguar de Software Livre), destacou os 31 anos de
hist ria do Debian. O evento contou com algumas palestras e muitas discuss es
sobre Software Livre. Tivemos 70 inscri es, 40 estiverem presentes.
O Debian Day em Natal foi uma ocasi o para celebrar a trajet ria do Debian e
refor ar a import ncia do Software Livre.
Palestrantes
Agradecemos imensamente a Isaque Barbosa Martins, Eduardo de Souza Paix o,
Fernando Guisso,que palestraram nessa edi o! Obrigado por compartilhar tanto
conhecimento com a comunidade. Esperamos ver voc s novamente em futuros
encontros!
10:40 - 11:20 - Analisando a aplica o de algoritmos criptogr ficos em pacotes de redes - Isaque Barbosa Martins
11:20 - 12:00 - Introdu o a escala o de privil gio em sistemas GNU/Linux - Eduardo de Souza Paix o
Link dos slides do Debian Day
Participantes
Um grande obrigado tamb m a todos os participantes, n s fazemos isso por voc s!
Esperamos que tenham aprendido, se divertido e feito novas conex es entre a
comunidade
Essa edi o do Debina Day Natal foi organizada por: Allythy, Clara Nobre,
Gabriel Damazio e Marcel Ribeiro.
Welcome to the July 2024 report from the Reproducible Builds project!
In our reports, we outline what we ve been up to over the past month and highlight news items in software supply-chain security more broadly. As always, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
Reproducible Builds Summit 2024
Last month, we were very pleased to announce the upcoming Reproducible Builds Summit, set to take place from September 17th 19th 2024 in Hamburg, Germany. We are thrilled to host the seventh edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.
If you re interesting in joining us this year, please make sure to read the event page, which has more details about the event and location. We are very much looking forward to seeing many readers of these reports there.
a bootstrappable build is one that builds existing software from scratch for example, building GCC without relying on an existing copy of GCC. In 2023, the Guix project announced that the project had reduced the size of the binary bootstrap seed needed to build its operating system to just 357-bytes not counting the Linux kernel required to run the build process.
The article goes onto to describe that now, the live-bootstrap project has gone a step further and removed the need for an existing kernel at all. and concludes:
The real benefit of bootstrappable builds comes from a few things. Like reproducible builds, they can make users more confident that the binary packages downloaded from a package mirror really do correspond to the open-source project whose source code they can inspect. Bootstrappable builds have also had positive effects on the complexity of building a Linux distribution from scratch [ ]. But most of all, bootstrappable builds are a boon to the longevity of our software ecosystem. It s easy for old software to become unbuildable. By having a well-known, self-contained chain of software that can build itself from a small seed, in a variety of environments, bootstrappable builds can help ensure that today s software is not lost, no matter where the open-source community goes from here
Towards Idempotent Rebuilds?Trisquel developer Simon Josefsson wrote an interesting blog post comparing the output of the .deb files from our tests.reproducible-builds.org testing framework and the ones in the official Debian archive. Following up from a previous post on the reproducibility of Trisquel, Simon notes that typically [the] rebuilds do not match the official packages, even when they say the package is reproducible , Simon correctly identifies that the purpose of [these] rebuilds are not to say anything about the official binary build, instead the purpose is to offer a QA service to maintainers by performing two builds of a package and declaring success if both builds match.
However, Simon s post swiftly moves on to announce a new tool called debdistrebuild that performs rebuilds of the difference between two distributions in a GitLab pipeline and displays diffoscope output for further analysis.
Reproducible Central is an initiative that curates a list of reproducible Maven libraries, but the list is limited and challenging to maintain due to manual efforts. [We] investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of Maven packages. Overall, our approach achieves an accuracy of up to 99.5% when compared field-by-field to the existing manual approach [and] we reveal that automatic reproducibility is feasible for 23.4% of the Maven packages using AROMA, and 8% of these packages are fully reproducible.
Nichita Morcotilo reached out to the community, first to share their efforts to build reproducible packages cross-platform with a new build tool called rattler-build, noting that as you can imagine, building packages reproducibly on Windows is the hardest challenge (so far!) . Nichita goes onto mention that the Apple ecosystem appears to be using ZERO_AR_DATE over SOURCE_DATE_EPOCH. []
Roland Clobus announced that the Debian bookworm 12.6 live images are nearly reproducible , with more detail in the post itself and input in the thread from other contributors.
Daniel Gr ber asked for help in getting the Yosys documentation to build reproducibly, citing issues in inter alia the PDF generation causing differing CreationDate metadata values.
James Addison continued his long journey towards getting the Sphinx documentation generator to build reproducible documentation. In this thread, James concerns himself with the problem that even when SOURCE_DATE_EPOCH is configured, Sphinx projects that have configured their copyright notices using dynamic elements can produce nonsensical output under some circumstances. James query ended up generating a number of replies.
Allen gunner Gunner posted a brief update on the progress the core team is making towards introducing a Code of Conduct (CoC) such that it is in place in time for the RB Summit in Hamburg in September . In particular, gunner asks if you are interested in helping with CoC design and development in the weeks ahead, simply email rb-core@lists.reproducible-builds.org and let us know . []
[S]oftware repositories are a vital component of software development and release, with packages downloaded both for direct use and to use as dependencies for other software. Further, when software is updated due to patched vulnerabilities or new features, it is vital that users are able to see and install this patched version of the software. However, this process of updating software can also be the source of attack. To address these attacks, secure software update systems have been proposed. However, these secure software update systems have seen barriers to widespread adoption. The Update Framework (TUF) was introduced in 2010 to address several attacks on software update systems including repository compromise, rollback attacks, and arbitrary software installation. Despite this, compromises continue to occur, with millions of users impacted by such compromises. My work has addressed substantial challenges to adoption of secure software update systems grounded in an understanding of practical concerns. Work with industry and academic communities provided opportunities to discover challenges, expand adoption, and raise awareness about secure software updates. [ ]
Since build-tools >= 35.0.0-rc1, backwards-incompatible changes to apksigner break apksigcopier as it now by default forcibly replaces existing alignment padding and changed the default page alignment from 4k to 16k (same as Android Gradle Plugin >= 8.3, so the latter is only an issue when using older AGP). []
She documented multiple available workarounds and filed a bug in Google s issue tracker.
Lastly, diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb uploaded version 272 and Mattia Rizzolo uploaded version 273 to Debian, and the following changes were made as well:
Chris Lamb:
Ensure that the convert utility is from ImageMagick version 6.x. The command-line interface has seemingly changed with the 7.x series of ImageMagick. []
Factor out version detection in test_jpeg_image. []
Correct the import of the identify_version method after a refactoring change in a previous commit. []
Move away from using DSA OpenSSH keys in tests as support has been deprecated and removed in OpenSSH version 9.8p1. []
Move to assert_diff in the test_openssh_pub_key package. []
Update copyright years. []
Mattia Rizzolo:
Add support for ffmpeg version 7.x which adds some extra context to the diff. []
Rework the handling of OpenSSH testing of DSA keys if OpenSSH is strictly 9.7, and add an OpenSSH key test with a ed25519-format key [][][]
Temporarily disable a few packages that are not available in Debian testing. [][]
Stop ignoring the results of Debian testing in the continuous integration system. []
Adjust options in debian/source to make sure not to pack the Python sdist directory into the binary Debian package. []
Adjust Lintian overrides. []
Website updates
There were a number of improvements made to our website this month, including:
Bernhard M. Wiedemann updated the SOURCE_DATE_EPOCH page to include instructions on how to create reproducible .zip files from within Python using the zipfile module. []
Chris Lamb fixed a potential duplicate heading on the Projects page. []
Fay Stegerman added rbtlog to the Tools page [] and IzzyOnDroid to the Projects page [], also ensuring that the latter page was always sorted regardless of the ordering within the input data files. []
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In July, a number of changes were made by Holger Levsen, including:
Grant bremner access to the ionos7 node. [][]
Perform a dummy change to force update of all jobs. [][]
In addition, Vagrant Cascadian performed some necessary node maintenance of the underlying build hosts. []
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
A short status update of what happened on my side last month. A broken
gcovr in Debian triggered a bit of busy work but 0.39.0 came out
nicely nevertheless. We also reduced build time quiet a bit in phosh
and phoc.
I ended 2022 with a musical retrospective and very much enjoyed writing
that blog post. As such, I have decided to do the same for 2023! From now on,
this will probably be an annual thing :)
Albums
In 2023, I added 73 new albums to my collection nearly 2 albums every three
weeks! I listed them below in the order in which I acquired them.
I purchased most of these albums when I could and borrowed the rest at
libraries. If you want to browse though, I added links to the album covers
pointing either to websites where you can buy them or to Discogs when digital
copies weren't available.
Once again this year, it seems that Punk (mostly O !) and Metal dominate my
list, mostly fueled by Angry Metal Guy and the amazing Montr al
Skinhead/Punk concert scene.
Concerts
A trend I started in 2022 was to go to as many concerts of artists I like as
possible. I'm happy to report I went to around 80% more concerts in 2023 than
in 2022! Looking back at my list, April was quite a busy month...
Here are the concerts I went to in 2023:
March 8th: Godspeed You! Black Emperor
April 11th: Alexandra Str liski
April 12th: Bikini Kill
April 21th: Brigada Flores Magon, Union Thugs
April 28th: Komintern Sect, The Outcasts, Violent Way, Ultra Razzia, Over the
Hill
May 3rd: First Fragment
May 12th: Rhapsody of Fire, Wind Rose
May 13th: Aeternam
June 2nd: Mortier, La Gachette
June 17th: Ultra Razzia, Total Nada, BLEMISH
June 30th: Avishai Cohen Trio
July 9th: Richard Galliano
August 18th: Gojira, Mastodon, Lorna Shore
September 14th: Jinjer
September 22nd: CUIR, Salvaje Punk, Hysteric Polemix, Perestroika, Ultra Razzia, Ilusion, Over the Hill, Asbestos
October 6th: Rancoeur, Street Code, Tenaz, Mortimer, Guernica, High Anxiety
Although metalfinder continues to work as intended, I'm very glad to have
discovered the Montr al underground scene has departed from Facebook/Instagram
and adopted en masseGancio, a FOSS community agenda that supports
ActivityPub. Our local instance, askapunk.net
is pretty much all I could ask for :)
That's it for 2023!
It was pointed out to me that I have not blogged about this, so better now than never:
Since 2021 I am together with four other hosts producing a regular podcast about Haskell, the Haskell Interlude. Roughly every two weeks two of us interview someone from the Haskell Community, and we chat for approximately an hour about how they came to Haskell, what they are doing with it, why they are doing it and what else is on their mind. Sometimes we talk to very famous people, like Simon Peyton Jones, and sometimes to people who maybe should be famous, but aren t quite yet.
For most episodes we also have a transcript, so you can read the interviews instead, if you prefer, and you should find the podcast on most podcast apps as well. I do not know how reliable these statistics are, but supposedly we regularly have around 1300 listeners. We don t get much feedback, however, so if you like the show, or dislike it, or have feedback, let us know (for example on the Haskell Disourse, which has a thread for each episode).
At the time of writing, we released 40 episodes. For the benefit of my (likely hypothetical) fans, or those who want to train an AI voice model for nefarious purposes, here is the list of episodes co-hosted by me:
Can t decide where to start? The one with Ryan Trinkle might be my favorite.
Thanks to the Haskell Foundation and its sponsors for supporting this podcast (hosting, editing, transscription).
An exciting new release 0.4.21 of RProtoBuf
arrived on CRAN earlier today.
RProtoBuf
provides R with bindings for the
Google Protocol Buffers
( ProtoBuf ) data encoding and serialization library used and
released by Google, and deployed very widely in numerous projects as a
language and operating-system agnostic protocol.
ProtoBuf
development, following what seemed like a multi-year lull, all of a
sudden picked up again with a vengeance a little while ago. And the
library releases we rely on for convenience and provided by the Linux
distributions are lagging. So last summer we received an excellent, and
focussed, pull request
#93 offering to update the package to the newer ProtoBuf 22.0 and beyond.
(Aside: When a library ditches its numbering scheme you know changes are
for real . My Ubuntu 23.10 box is still at 3.21 in a different counting
scheme .) But it wasn t until last weekend the issue ticket
#95 by Sebastian
ran into the same issue, but recognized it and contained a container
recipe! So now all of a sudden we were able to build under a newer ProtoBuf which made
accepting the PR #93 much
easier! We added this as an additional continuous unit test, and made a
few other smaller updates to documentation and style.
The following section from the NEWS.Rd file has full details.
Changes in
RProtoBuf version 0.4.21 (2022-12-13)
Package now builds with ProtoBuf >= 22.x thanks to Matteo
Gianella (#93
addressing #92).
An Alpine 3.19-based workflow was added to test this in
continuous integration thanks to a suggestion by Sebastian
Meyer.
A large number of old-style .Call were updated (#96).
Several packaging, dcoumentation and testing items were
updated.
The new-ish package RcppInt64
(announced earlier this fall in this
post, with two small updates following) arrived on CRAN minutes ago as relase 0.0.4.
RcppInt64
collects some of the previous conversions between 64-bit integer values
in R and C++, and regroups them in a single package. It offers two
interfaces: both a more standard as<>() converter
from R values along with its companions wrap() to return to
R, as well as more dedicated functions from and to .
This release addresses an issues Sebastian
reported a few hours and which is reported by newer, pickier compilers:
We need to include <cstdint> so that
int64_t is declared. CRAN was at its usual best
processing this efficiently including tests of the by now two reverse
dependencies. Twenty two minutes total, all automated:
The brief NEWS entry follows:
Welcome to the November 2023 report from the Reproducible Builds project! In these reports we outline the most important things that we have been up to over the past month. As a rather rapid recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries (more).
Reproducible Builds Summit 2023
Between October 31st and November 2nd, we held our seventh Reproducible Builds Summit in Hamburg, Germany! Amazingly, the agenda and all notes from all sessions are all online many thanks to everyone who wrote notes from the sessions.
As a followup on one idea, started at the summit, Alexander Couzens and Holger Levsen started work on a cache (or tailored front-end) for the snapshot.debian.org service. The general idea is that, when rebuilding Debian, you do not actually need the whole ~140TB of data from snapshot.debian.org; rather, only a very small subset of the packages are ever used for for building. It turns out, for amd64, arm64, armhf, i386, ppc64el, riscv64 and s390 for Debian trixie, unstable and experimental, this is only around 500GB ie. less than 1%. Although the new service not yet ready for usage, it has already provided a promising outlook in this regard. More information is available on https://rebuilder-snapshot.debian.net and we hope that this service becomes usable in the coming weeks.
The adjacent picture shows a sticky note authored by Jan-Benedict Glaw at the summit in Hamburg, confirming Holger Levsen s theory that rebuilding all Debian packages needs a very small subset of packages, the text states that 69,200 packages (in Debian sid) list 24,850 packages in their .buildinfo files, in 8,0200 variations. This little piece of paper was the beginning of rebuilder-snapshot and is a direct outcome of the summit!
The Reproducible Builds team would like to thank our event sponsors who include Mullvad VPN, openSUSE, Debian, Software Freedom Conservancy, Allotropia and Aspiration Tech.
Beyond Trusting FOSS presentation at SeaGL
On November 4th, Vagrant Cascadian presented Beyond Trusting FOSS at SeaGL in Seattle, WA in the United States. Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture. The summary of Vagrant s talk mentions that it will:
[ ] introduce the concepts of Reproducible Builds, including best practices for developing and releasing software, the tools available to help diagnose issues, and touch on progress towards solving decades-old deeply pervasive fundamental security issues Learn how to verify and demonstrate trust, rather than simply hoping everything is OK!
Germane to the contents of the talk, the slides for Vagrant s talk can be built reproducibly, resulting in a PDF with a SHA1 of cfde2f8a0b7e6ec9b85377eeac0661d728b70f34 when built on Debian bookworm and c21fab273232c550ce822c4b0d9988e6c49aa2c3 on Debian sid at the time of writing.
Human Factors in Software Supply Chain Security
Marcel Fourn , Dominik Wermke, Sascha Fahl and Yasemin Acar have published an article in a Special Issue of the IEEE s Security & Privacy magazine. Entitled A Viewpoint on Human Factors in Software Supply Chain Security: A Research Agenda, the paper justifies the need for reproducible builds to reach developers and end-users specifically, and furthermore points out some under-researched topics that we have seen mentioned in interviews. An author pre-print of the article is available in PDF form.
Bernhard M. Wiedemann posted a positive LibreOffice success story documenting that, after some work:
[ ] today I hold in my hands the first two bit-identical LibreOffice rpm packages. And this is the success I wanted to share with you all today [and] it makes me feel as if we can solve anything.
I chose the esp32c3 [board] because it has good Rust support from the esp-rs project, and you can get a dev board for about 6-8 . To document my build environment I used repro-env together with Arch Linux because its archive is very reliable and contains all the different Rust development tools I needed.
Separate to their work on LibreOffice however, Bernhard M. Wiedemann also requested assistance with a number of packages that so far refuse to build reproducibly. He writes that the common theme around them is that they use scheme or lisp to produce binaries with a dump command and hopes that someone may be able to help.
openSUSE updates
Bernhard M. Wiedemann has created a wiki page outlining an proposal to create a general-purpose Linux distribution which consists of 100% bit-reproducible packages albeit minus the embedded signature within RPM files. It would be based on openSUSE Tumbleweed or, if available, its Slowroll-variant.
In addition, Bernhard posted another monthly update for his work elsewhere in openSUSE.
Reproducibility-related changes in Debian
As recently reported in the most recent Debian Developer News, Paul Gevers has integrated a package s reproducibility status into the way Debian migrates packages into the next stable release. For the amd64, arm64, i386 and armhf architectures, data is collected from the Reproducible Builds testing framework is collected by this migration software even though, at the time of writing, it neither causes nor migration bonuses nor blocks migration. Indeed, the information only results are visible on Britney s excuses as well as on individual packages pages on tracker.debian.org.
Ubuntu Launchpad now supports .buildinfo files
Back in 2017, Steve Langasek filed a bug against Ubuntu s Launchpad code hosting platform to report that .changes files (artifacts of building Ubuntu and Debian packages) reference .buildinfo files that aren t actually exposed by Launchpad itself. This was causing issues when attempting to process .changes files with tools such as Lintian. However, it was noticed last month that, in early August of this year, Simon Quigley had resolved this issue, and .buildinfo files are now available from the Launchpad system.
PHP reproducibility updates
There have been two updates from the PHP programming language this month.
Firstly, the widely-deployed PHPUnit framework for the PHP programming language have recently released version 10.5.0, which introduces the inclusion of a composer.lock file, ensuring total reproducibility of the shipped binary file. Further details and the discussion that went into their particular implementation can be found on the associated GitHub pull request.
In addition, the presentation Leveraging Nix in the PHP ecosystem has been given in late October at the PHP International Conference in Munich by Pol Dellaiera. While the video replay is not yet available, the (reproducible) presentation slides and speaker notes are available.
diffoscope changes
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including:
Improving DOS/MBR extraction by adding support for 7z. []
Adding a missing RequiredToolNotFound import. []
As a UI/UX improvement, try and avoid printing an extended traceback if diffoscope runs out of memory. []
Website updates
A huge number of notes were added to our website that were taken at our recent Reproducible Builds Summit held between October 31st and November 2nd in Hamburg, Germany. In particular, a big thanks to Arnout Engelen, Bernhard M. Wiedemann, Daan De Meyer, Evangelos Ribeiro Tzaras, Holger Levsen and Orhun Parmaks z.
In addition to this, a number of other changes were made, including:
Holger Levsen also made a large number of notes pages from our 2022 summit in Venice [], migrated the website s syntax highlighter from [Pygments]https://pygments.org/() to Rouge [], fixed some grammar on our donate page [][][] and did a lot of updates to the Hamburg Summit s general information page [][].
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In October, a number of changes were made by Holger Levsen:
Debian-related changes:
Track packages marked as Priority: important in a new package set. [][]
Stop scheduling packages that fail to build from source in bookworm [] and bullseye. [].
Add old releases dashboard link in web navigation. []
Permit re-run of the pool_buildinfos script to be re-run for a specific year. []
Grant jbglaw access to the osuosl4 node [][] along with lynxis [].
Increase RAM on the amd64 Ionos builders from 48 GiB to 64 GiB; thanks IONOS! []
Move buster to archived suites. [][]
Reduce the number of arm64 architecture workers from 24 to 16 in order to improve stability [], reduce the workers for amd64 from 32 to 28 and, for i386, reduce from 12 down to 8 [].
Show the entire build history of each Debian package. []
Stop scheduling already tested package/version combinations in Debian bookworm. []
Other changes were made as well, however, including Mattia Rizzolo fixing rc.local s Bash syntax so it can actually run [], commenting away some file cleanup code that is (potentially) deleting too much [] and fixing the html_brekages page for Debian package builds []. Finally, diagnosed and submitted a patch to add a AddEncoding gzip .gz line to the tests.reproducible-builds.org Apache configuration so that Gzip files aren t re-compressed as Gzip which some clients can t deal with (as well as being a waste of time). []
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via: