I believe that, in the context of Debian voting, we are better off when we
know the opinion of our peers, however, since the 2022-001
vote, it is no longer the
case. Still, some DDs have disclosed the way they are voting on the
2026-002 General Resolution currently in progress, regarding LLM usage in
Debian. So, here goes my vote
and reasoning as briefly as possible. This is the ballot I sent to
devotee, the Debian Vote Engine:
-=-=-=-=-=- Don't Delete Anything Between These Lines =-=-=-=-=-=-=-=-
d69f9187-ed2f-40b6-a2eb-4211d3f84d86
[9] Choice 1: Ban LLM contributions from Debian via Social Contract
[3] Choice 2: Allow AI-Assisted Contributions with conditions
[6] Choice 3: Reject LLMs as far as practical, update Code of Conduct
[2] Choice 4: Accept AI contributions for Debian specific work
[4] Choice 5: Responsible Use of Generative AI
[1] Choice 6: A cautious approach to generative AI
[8] Choice 7: Debian is created by humans
[5] Choice 8: Avoid the use of LLM: climate destruction is a deal breaker
[7] Choice 9: None of the above
-=-=-=-=-=- Don't Delete Anything Between These Lines =-=-=-=-=-=-=-=-
This is the first time I can recall I delay my voting until after receiving
the final call for votes (the vote will be over two days from now). I had
some participation in the discussion, so I guess my position will be of no
big surprise to anybody. I was also a seconder for choices D and F (4 and
6 in the vote text). This does not necessarily mean I believe they are
the best (although I did rank them as 2 and 1, meaning I do): sometimes
you agree a given text needs to be in the ballot, and second it even
though you don t intend to vote for it.
Ranking this ballot was a mess due to the complex array of options it
encodes. I warmly thank Lucas Nussbaum for coming up with the LLM usage in
Debian: ballot option
comparison
(URL shown with my particular ballot ordering).
How do you read a complex Debian ballot like this one? I rank with [1] my
favorite option, [2] for the next one, etc. We can encode options to be
tied (i.e. setting more than options to the same value), and we can
implicitly push options to the worst position by leaving them blank (so,
with[ ]); I chose not to do any of those.
What were my voting guidelines?
First, I don t want anything banning or that threatens with disciplinary
action, so I push them below the special none of the above
marker. Second Some time ago I published a review in my blog (and in
Computing Reviews) about the unfeasibility and unfairness of detecting LLM
output on students
assignments. I
strongly believe we ought to appeal to the human responsibility and
professionalism in all Debian contributors. This is the reason I proposed
this amendment paragraph, that was accepted in choice F (6), which I ranked
as my favorite:
The Debian project has always recognized the commitment and
professionalism of its members. All contributions are under the
responsibility of the Debian Contributor making it, no matter the
technology they have behind. We trust all Debian Developers,
Maintainers and Contributors will continue to uphold the high quality
values that have distinguished our project from its onset.
Other than that I do not consider myself to be in any way an LLM
fanboy nor anything like that. I distrust and dislike the excessive use
of this technology, and continue to warn about the dangers and bad points
of its abuse. But in my day-to-day professional work, I am also starting to
rely on it for some tasks. I recognize it needs a lot of human
oversight and lets call it hand-holding to produce anything worth
it, at least in my experience. But I do benefit from it and always
disclose its use to people who might be affected by it. I would like
Debian to adopt such a stance.
Of course, I recgnize proposal H/8 as important (Avoid the use of LLM:
climate destruction is a deal breaker). Some people have argued it s not
bad at all. I do not buy such claims: LLMs are f*cking expensive to
train. But training can be seen as a once-per-model cost, and fine-tuning
a good model to be run locally can be really worth it. It still pains me
somewhat, but I cannot push this option higher than its #5 position in my
list.
Welcome to the March 2026 report from the Reproducible Builds project!
These reports outline what we ve been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
Linux kernel hash-based integrity checking proposed
Eric Biggers posted to the Linux Kernel Mailing List in response to a patch series posted by Thomas Wei schuh to introduce a calculated hash-based system of integrity checking to complement the existing signature-based approach. Thomas original post mentions:
The current signature-based module integrity checking has some drawbacks in combination with reproducible builds. Either the module signing key is generated at build time, which makes the build unreproducible, or a static signing key is used, which precludes rebuilds by third parties and makes the whole build and packaging process much more complicated.
I think this actually undersells the feature. It s also much simpler than the signature-based module authentication. The latter relies on PKCS#7, X.509, ASN.1, OID registry, crypto_sig API, etc in addition to the implementations of the actual signature algorithm (RSA / ECDSA / ML-DSA) and at least one hash algorithm.
Distribution work
In Debian this month,
Lucas Nussbaum announced Debaudit, a new service to verify the reproducibility of Debian source packages :
debaudit complements the work of the Reproducible Builds project. While reproduce.debian.net focuses on ensuring that binary packages can be bit-for-bit reproduced from their source packages, debaudit focuses on the preceding step: ensuring that the source package itself is a faithful and reproducible representation of its upstream source or Vcs-Git repository.
Lastly, Bernhard M. Wiedemann posted another openSUSEmonthly update for their work there.
Tool development
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions, 314 and 315 to Debian.
Chris Lamb:
Don t run test_code_is_black_clean test in the autopkgtests. (#1130402). []
Adjust the PGP file detection regular expression. []
Michael R. Crusoe:
Reformat the source code using Black version 26.1.0 [][]
In addition, Vagrant Cascadian updated diffoscope in GNU Guix to version 315.
rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there; it powers, amongst other things, reproduce.debian.net.
A new version, 0.26.0, was released this month, with the following improvements:
Much smoother onboarding/installation.
Complete database redesign with many improvements.
New REST HTTP API.
It s now possible to artificially delay the first reproduce attempt. This gives archive infrastructure more time to catch up.
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Two new academic papers
Marc Ohm, Timo Pohl, Ben Swierzy and Michael Meier published a paper on the threat of cache poisoning in the Python ecosystem:
Attacks on software supply chains are on the rise, and attackers are becoming increasingly creative in how they inject malicious code into software components.
This paper is the first to investigate Python cache poisoning, which manipulates bytecode cache files to execute malicious code without altering the human-readable source code.
We demonstrate a proof of concept, showing that an attacker can inject malicious bytecode into a cache file without failing the Python interpreter s integrity checks.
In a large-scale analysis of the Python Package Index, we find that about 12,500 packages are distributed with cache files.
Through manual investigation of cache files that cannot be reproduced automatically from the corresponding source files, we identify classes of reasons for irreproducibility to locate malicious cache files.
While we did not identify any malware leveraging this attack vector, we demonstrate that several widespread package managers are vulnerable to such attacks.
A PDF of the paper is available online.
Mario Lins of the University of Linz, Austria, has published their PhD doctoral thesis on the topic of Software supply chain transparency:
We begin by examining threats to the software distribution stage the point at which artifacts (e.g., mobile apps) are delivered to end users with an emphasis on mobile ecosystems [and] we next focus on the operating system on mobile devices, with an emphasis on mitigating bootloader-targeted attacks. We demonstrate how to compensate lost security guarantees on devices with an unlocked bootloader. This allows users to flash custom operating systems on devices that no longer receive security updates from the original manufacturer without compromising security. We then move to the source code stage. [Also,] we introduce a new architecture to ensure strong source-to-binary correspondence by leveraging the security guarantees of Confidential Computing technology. Finally, we present The Supply Chain Game, an organizational security approach that enhances standard risk-management methods. We demonstrate how game-theoretic techniques, combined with common risk management practices, can derive new criteria to better support decision makers.
A PDF of the paper is available online.
Holger Levsen announced that this year s Reproducible Builds summit will almost certainly be held in Gothenburg, Sweden, from September 22 until 24, followed by two days of hacking. However, these dates are preliminary and not 100% final an official announcement is forthcoming.
Mark Wielaard posted to our list asking a question on the difference between debugedit and relative debug paths based on a comment on the Build path page: Have people tried more modern versions of debugedit to get deterministic (absolute) DWARF paths and found issues with it?
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
Welcome to the December 2025 from the Reproducible Builds project!
Our monthly reports outline what we ve been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
New orig-check service to validate Debian upstream tarballs
This month, Debian Developer Lucas Nussbaum announced the orig-check service, which attempts to automatically reproduce the generation upstream tarballs (ie. the original source component of a Debian source package), comparing that to the upstream tarball actually shipped with Debian.
As of the time of writing, it is possible for a Debian developer to upload a source archive that does not actually correspond to upstream s version. Whilst this is not inherently malicious (it typically indicates some tooling/process issue), the very possibility that a maintainer s version may differ potentially permits a maintainer to make (malicious) changes that would be misattributed to upstream.
This service therefore nicely complements the whatsrc.org service, which was reported in our reports for both April and August. The orig-check is dedicated to Lunar, who sadly passed away a year ago.
disorderfs updated to FUSE 3
disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into system calls to reliably flush out reproducibility issues.
This month, however, Roland Clobus upgraded disorderfs* from FUSE 2 to FUSE 3 after its package automatically got removed from Debian testing. Some tests in Debian currently require disorderfs to make the Debian live images reproducible, although disorderfs is not a Debian-specific tool.
Luca Di Maio announced stampdalf, a filesystem timestamp preservation tool that wraps arbitrary commands and ensures filesystem timestamp reproducibility :
stampdalf allows you to run any command that modifies files in a directory tree, then automatically resets all timestamps back to their original values. Any new files created during command execution are set to [the UNIX epoch] or a custom timestamp via SOURCE_DATE_EPOCH.
The project s GitHub page helpfully reveals that the project is pronounced: stamp-dalf (stamp like time-stamp, dalf like Gandalf the wizard) as it s a wizard of time and stamps .)
Lastly, Reproducible Builds developer cen1 posted to our list announcing that early/experimental/alpha support for FreeBSD was added to rebuilderd. In their post, cen1 reports that the initial builds are in progress and look quite decent . cen1 also interestingly notes that since the upstream is currently not technically reproducible I had to relax the bit-for-bit identical requirement of rebuilderd [ ] I consider the pkg to be reproducible if the tar is content-identical (via diffoscope), ignoring timestamps and some of the manifest files. .
Most package managers also generate a lockfile, which records the exact set of resolved dependency versions. Lockfiles are used to reduce build times; to verify the integrity of resolved packages; and to support build reproducibility across environments and time. Despite these beneficial features, developers often struggle with their maintenance, usage, and interpretation. In this study, we unveil the major challenges related to lockfiles, such that future researchers and engineers can address them. [ ]
While existing studies primarily focus on software supply chain attacks prevention and detection methods, there is a need for a broad overview of attacks and comprehensive risk assessment for software supply chain security. This study conducts a systematic literature review to fill this gap. By analyzing 96 papers published between 2015-2023, we identified 19 distinct SSC attacks, including 6 novel attacks highlighted in recent studies. Additionally, we developed 25 specific security controls and established a precisely mapped taxonomy that transparently links each control to one or more specific attacks. [ ]
[Achieving] reproducibility at scale remains difficult, especially in Java, due to a range of non-deterministic factors and caveats in the build process. In this work, we focus on reproducibility in Java-based software, archetypal of enterprise applications. We introduce a conceptual framework for reproducible builds, we analyze a large dataset from Reproducible Central, and we develop a novel taxonomy of six root causes of unreproducibility. [ ]
Robert Stupp made a number of useful changes, fixing and reorganising the Groovy / Kotlin pages [][][] as well adding a note about potential non-deterministic behaviour to the JVM page [].
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
It's Sunday and I'm now sitting in the train from Brest to Paris where I will be changing to Germany, on the way back from the annual Debian conference. A full week of presentations, discussions, talks and socializing is laying behind me and my head is still spinning from the intensity.
Pollito and the gang of DebConf mascots wearing their conference badges (photo: Christoph Berg)
Table of Contents
Sunday, July 13th
It started last Sunday with traveling to the conference. I got on the Eurostar in Duisburg and we left on time, but even before reaching Cologne, the train was already one hour delayed for external reasons, collecting yet another hour between Aachen and Liege for its own technical problems. "The train driver is working on trying to fix the problem." My original schedule had well over two hours for changing train stations in Paris, but being that late, I missed the connection to Brest in Montparnasse. At least in the end, the total delay was only one hour when finally arriving at the destination. Due to the French julliet quatorze fireworks approaching, buses in Brest were rerouted, but I managed to catch the right bus to the conference venue, already meeting a few Debian people on the way.
The conference was hosted at the IMT Atlantique Brest campus, giving the event a nice university touch. I arrived shortly after 10 in the evening and after settling down a bit, got on one of the "magic" buses for transportation to the camping site where half of the attendees where stationed. I shared a mobile home with three other Debianites, where I got a small room for myself.
Monday, July 14th
Next morning, we took the bus back to the venue with a small breakfast and the opening session where Enrico Zini invited me to come to his and Nicolas Dandrimont's session about Debian community governance and curation, which I gladly did. Many ideas about conflict moderation and community steering were floated around. I hope some of that can be put into effect to make flamewars on the mailing lists less heated and more directed. After that, I attended Olly Betts' "Stemming with Snowball" session, which is the stemmer used also in PostgreSQL. Text search is one of the areas in PostgreSQL that I never really looked closely at, including the integration into the postgresql-common package, so it was nice to get more information about that.
In preparation for the conference, a few of us Ham radio operators in Debian had decided to bring some radio gear to DebConf this year in order to perhaps spark more interest for our hobby among the fellow geeks. In the afternoon after the talks, I found a quieter spot just outside of the main hall and set up a shortwave antenna by attaching a 10m mast to one of the park benches there. The 40m band was still pretty much closed, but I could work a few stations from England, just across the channel from Bretagne, answering questions from interested passing-by Debian people between the contacts. Over time, the band opened and more European stations got into the log.
F/DF7CB in Brest (photo: Evangelos Ribeiro Tzaras)
Tuesday, July 15th
Tuesday started with Helmut Grohne's session about "Reviving (un)schroot". The schroot program has been Debian's standard way of managing build chroots for a long time, but it is more and more being regarded as obsolete with all kinds of newer containerization and virtualization technologies taking over. Since many bits of Debian infrastructure depend on schroot, and its user interface is still very useful, Helmut reimplemented it using Linux namespaces and the "unshare" systemcall. I had already worked with him at the Hamburg Minidebconf to replace the apt.postgresql.org buildd machinery with the new system, but we were not quite there yet (network isolation is nice, but we still sometimes need proper networking), so it was nice to see the effort is still progressing and I will give his new scripts a try when I'm back home.
Next, Stefano Rivera and Colin Watson presented Debusine, a new package repository and workflow management system. It looks very promising for anyone running their own repository, so perhaps yet another bit of apt.postgresql.org infrastructure to replace in the future. After that, I went to the Debian LTS BoF session by Santiago Ruano Rinc n and Bastien Roucari s - Debian releases plus LTS is what we are covering with apt.postgresql.org. Then there were bits from the DPL (Debian Project Leader), and a session moderated by Stefano Rivera interesting to me as a member of the Debian Technical Committee on the future structure of the packages required for cross-building in Debian, a topic which had been brought to TC a while ago. I am happy that we could resolve the issue without having to issue a formal TC ruling as the involved parties (kernel, glibc, gcc and the cross-build people) found a promising way forward themselves. DebConf is really a good way to get such issues unstuck.
Ten years ago at the 2015 Heidelberg DebConf, Enrico had given a seminal "Semi-serious stand-up comedy" talk, drawing parallels between the Debian Open Source community and the BDSM community - "People doing things consensually together". (Back then, the talk was announced as "probably unsuitable for people of all ages".) With his unique presentation style and witty insights, the session made a lasting impression on everyone attending. Now, ten years later (and he and many in the audience being ten years older), he gave an updated version of it. We are now looking forward to the sequel in 2035. The evening closed with the famous DebConf tradition of the Cheese & Wine party in a old fort next to the coast, just below the conference venue. Even when he's a fellow Debian Developer, Ham and also TC member, I had never met Paul Tagliamonte in person before, but we spent most of the evening together geeking out on all things Debian and Ham radio.
The northern coast of Ushant (photo: Christoph Berg)
Wednesday, July 16th
Wednesday already marked the end of the first half of the week, the day of the day trips. I had chosen to go to Ouessant island (Ushant in English) which marks the Western end of French mainland and hosts one of the lighthouses yielding the way into the English channel. The ferry trip included surprisingly big waves which left some participants seasick, but everyone recovered fast. After around one and a half hours we arrived, picked up the bicycles, and spent the rest of the day roaming the island. The weather forecast was originally very cloudy and 18 C, but over noon this turned into sunny and warm, so many got an unplanned sunburn. I enjoyed the trip very much - it made up for not having time visiting the city during the week. After returning, we spent the rest of the evening playing DebConf's standard game, Mao (spoiler alert: don't follow the link if you ever intend to play).
Having a nice day (photo: Christoph Berg)
Thursday, July 17th
The next day started with the traditional "Meet the Technical Committee" session. This year, we trimmed the usual slide deck down to remove the boring boilerplate parts, so after a very short introduction to the work of the committee by our chairman Matthew Vernon, we opened up the discussion with the audience, with seven (out of 8) TC members on stage. I think the format worked very well, with good input from attendees. Next up was "Don't fear the TPM" by Jonathan McDowell. A common misconception in the Free Software community is that the TPM is evil DRM hardware working against the user, but while it could be used in theory that way, the necessary TPM attestations seem to impossible to attain in practice, so that wouldn't happen anyway. Instead, it is a crypto coprocessor present in almost all modern computers that can be used to hold keys, for example to be used for SSH. It will also be interesting to research if we can make use of it for holding the Transparent Data Encryption keys for CYBERTEC's PostgreSQL Enterprise Edition.
Aigars Mahinovs then directed everyone in place for the DebConf group picture, and Lucas Nussbaum started a discussion about archive-wide QA tasks in Debian, an area where I did a lot of work in the past and that still interests me. Antonio Terceiro and Paul Gevers followed up with techniques to track archive-wide rebuilding and testing of packages and in turn filing a lot of bugs to track the problems. The evening ended with the conference dinner, again in the fort close by the coast. DebConf is good for meeting new people, and I incidentally ran into another Chris, who happened to be one of the original maintainers of pgaccess, the pre-predecessor of today's pgadmin. I admit still missing this PostgreSQL frontend for its simplicity and ability to easily edit table data, but it disappeared around 2004.
Friday, July 18th
On Friday, I participated in discussion sessions around contributors.debian.org (PostgreSQL is planning to set up something similar) and the New Member process which I had helped to run and reform a decade or two ago. Agathe Porte (also a Ham radio operator, like so many others at the conference I had no idea of) then shared her work on rust-rewriting the slower parts of Lintian, the Debian package linter. Craig Small talked about "Free as in Bytes", the evolution of the Linux procps free command. Over the time and many kernel versions, the summary numbers printed became better and better, but there will probably never be a version that suits all use cases alike. Later over dinner, Craig (who is also a TC member) and I shared our experiences with these numbers and customers (not) understanding them. He pointed out that for PostgreSQL and looking at used memory in the presence of large shared memory buffers, USS (unique set size) and PSS (proportional set size) should be more realistic numbers than the standard RSS (resident set size) that the top utility is showing by default.
Antonio Terceiro and Paul Gevers again joined to lead a session, now on ci.debian.net and autopkgtest, the test driver used for running tests on packages after then have been installed on a system. The PostgreSQL packages are heavily using this to make sure no regressions creep in even after builds have successfully completed and test re-runs are rescheduled periodically. The day ended with Bdale Garbee's electronics team BoF and Paul Tagliamonte and me setting up the radio station in the courtyard, again answering countless questions about ionospheric conditions and operating practice.
Saturday, July 19th
Saturday was the last conference day. In the first session, Nikos Tsipinakis and Federico Vaga from CERN announced that the LHC will be moving to Debian for the accelerator's frontend computers in their next "long shutdown" maintenance period in the next year. CentOS broke compatibility too often, and Debian trixie together with the extended LTS support will cover the time until the next long shutdown window in 2035, until when the computers should have all been replaced with newer processors covering higher x86_64 baseline versions. The audience was very delighted to hear that Debian is now also being used in this prestige project.
Ben Hutchings then presented new Linux kernel features. Particularly interesting for me was the support for atomic writes spanning more than one filesystem block. When configured correctly, this would mean PostgreSQL didn't have to record full-page images in the WAL anymore, increasing throughput and performance. After that, the Debian ftp team discussed ways to improve review of new packages in the archive, and which of their processes could be relaxed with new US laws around Open Source and cryptography algorithms export. Emmanuel Arias led a session on Salsa CI, Debian's Gitlab instance and standard CI pipeline. (I think it's too slow, but the runners are not under their control.) Julian Klode then presented new features in APT, Debian's package manager. I like the new display format (and a tiny bit of that is also from me sending in wishlist bugs).
In the last round of sessions this week, I then led the Ham radio BoF with an introduction into the hobby and how Debian can be used. Bdale mentioned that the sBitx family of SDR radios is natively running Debian, so stock packages can be used from the radio's touch display. We also briefly discussed his involvement in ARDC and the possibility to get grants from them for Ham radio projects. Finally, DebConf wrapped up with everyone gathering in the main auditorium and cheering the organizers for making the conference possible and passing Pollito, the DebConf mascot, to the next organizer team.
Pollito on stage (photo: Christoph Berg)
Sunday, July 20th
Zoom back to the train: I made it through the Paris metro and I'm now on the Eurostar back to Germany. It has been an intense week with all the conference sessions and meeting all the people I had not seen so long. There are a lot of new ideas to follow up on both for my Debian and PostgreSQL work. Next year's DebConf will take place in Santa Fe, Argentina. I haven't yet decided if I will be going, but I can recommend the experience to everyone!
The post The Debian Conference 2025 in Brest appeared first on CYBERTEC PostgreSQL Services & Support.
Dear Debian community,
This is bits from the DPL for June.
The Challenge of Mentoring Newcomers
In June there was an extended discussion about the ongoing challenges
around mentoring newcomers in Debian. As many of you know, this is a
topic I ve cared about deeply--long before becoming DPL. In my view, the
issue isn t just a matter of lacking tools or needing to try harder to
attract contributors. Anyone who followed the discussion will likely
agree that it s more complex than that.
I sometimes wonder whether Debian s success contributes to the problem.
From the outside, things may appear to just work , which can lead to
the impression: Debian is doing fine without me--they clearly have
everything under control. But that overlooks how much volunteer effort
it takes to keep the project running smoothly.
We should make it clearer that help is always needed--not only in
packaging, but also in writing technical documentation, designing web
pages, reaching out to upstreams about license issues, finding sponsors,
or organising events. (Speaking from experience, I would have
appreciated help in patiently explaining Free Software benefits to
upstream authors.) Sometimes we think too narrowly about what newcomers
can do, and also about which tasks could be offloaded from overcommitted
contributors.
In fact, one of the most valuable things a newcomer can contribute is
better documentation. Those of us who ve been around for years may be
too used to how things work--or make assumptions about what others
already know. A person who just joined the project is often in the best
position to document what s confusing, what s missing, and what they
wish they had known sooner.
In that sense, the recent
"random new contributor s experience"
posts might be a useful starting point for further reflection. I think
we can learn a lot from positive user stories, like this recent
experience of a newcomer
adopting the courier package. I'm absolutely convinced that those who just
found their way into Debian have valuable perspectives--and that we stand to
learn the most from listening to them.
We should also take seriously what Russ Allbery noted in the discussion:
"This says bad things about the project's sustainability and I think
everyone knows that."
Volunteers move on--that s normal and expected. But it makes it all the
more important that we put effort into keeping Debian's contributor base at
least stable, if not growing.
Project-wide LLM budget for helping people
Lucas Nussbaum has volunteered to handle the paperwork and submit a
request on Debian s behalf to LLM providers,
aiming to secure project-wide access for Debian Developers. If successful,
every DD will be free to use this access--or not--according to their own
preferences.
Kind regards
Andreas.
All but about four hours of my Debian contributions this month were
sponsored by
Freexian. (I ended up going a bit over my 20% billing limit this month.)
You can also support my work directly via
Liberapay.
man-db and friends
I released libpipeline
1.5.8 and
man-db 2.13.0.
Since autopkgtests are great for making sure we spot regressions caused by
changes in dependencies, I added one to man-db that runs the upstream tests
against the installed package. This required some preparatory work
upstream, but otherwise was surprisingly easy to do.
OpenSSH
I fixed the various 9.8 regressions I mentioned last
month: socket activation, libssh2, and
Twisted. There were a few other regressions reported too: TCP wrappers
support,
openssh-server-udeb,
and xinetd were all broken by changes
related to the listener/per-session binary split, and I fixed all of those.
Once all that had made it through to testing, I finally uploaded the first
stage of my plan to split out GSS-API
support: there
are now openssh-client-gssapi and openssh-server-gssapi packages in
unstable, and if you use either GSS-API authentication or key exchange then
you should install the corresponding package in order for upgrades to
trixie+1 to work correctly. I ll write a release note once this has reached testing.
Multiple identical results from getaddrinfo
I expect this is really a bug in a chroot creation script somewhere, but I
haven t been able to track down what s causing it yet. My sbuild chroots,
and apparently Lucas Nussbaum s as well, have an /etc/hosts that looks
like this:
The last line clearly ought to be ::1 rather than 127.0.0.1; but things
mostly work anyway, since most code doesn t really care which protocol it
uses to talk to localhost. However, a few things try to set up test
listeners by calling getaddrinfo("localhost", ...) and binding a socket
for each result. This goes wrong if there are duplicates in the resulting
list, and the test output is typically very confusing: it looks just like
what you d see if a test isn t tearing down its resources correctly, which
is a much more common thing for a test suite to get wrong, so it took me a
while to spot the problem.
I ran into this in both python-asyncssh
(#1052788, upstream
PR) and Ruby
(ruby3.1/#1069399,
ruby3.2/#1064685,
ruby3.3/#1077462, upstream
PR). The latter took a while
since Ruby isn t one of my languages, but hey, I ve tackled much harder
side quests. I
NMUed ruby3.1 for this since it was showing up as a blocker for openssl
testing migration, but haven t done the other active versions (yet, anyway).
OpenSSL vs. cryptography
I tend to care about openssl migrating to testing promptly, since openssh
uploads have a habit of getting stuck on it otherwise.
Debian s OpenSSL packaging recently split out some legacy code (cryptography
that s no longer considered a good idea to use, but that s sometimes needed
for compatibility) to an openssl-legacy-provider package, and added a
Recommends on it. Most users install Recommends, but package build
processes don t; and the Python cryptography package requires this code
unless you set the CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 environment variable,
which caused a bunch of packages that build-depend on it to fail to build.
After playing whack-a-mole setting that environment variable in a few
packages build process, I decided I didn t want to be caught in the middle
here and filed an upstream
issue to see if I could
get Debian s OpenSSL team and cryptography s upstream talking to each other
directly. There was some moderately spirited discussion and the issue
remains open, but for the time being the OpenSSL team has effectively
reverted the
change
so it s no longer a pressing problem.
GCC 14 regressions
Continuing from last month, I fixed build
failures in pccts (NMU) and
trn4.
Python team
I upgraded alembic, automat, gunicorn, incremental, referencing, pympler
(fixing compatibility with Python >=
3.10), python-aiohttp, python-asyncssh
(fixing CVE-2023-46445,
CVE-2023-46446, and
CVE-2023-48795), python-avro,
python-multidict (fixing a build failure with GCC
14), python-tokenize-rt, python-zipp,
pyupgrade, twisted (fixing CVE-2024-41671
and CVE-2024-41810), zope.exceptions,
zope.interface, zope.proxy, zope.security, and zope.testrunner to new
upstream versions. In the process, I added myself to Uploaders for
zope.interface; I m reasonably comfortable with the Zope Toolkit and I seem
to be gradually picking up much of its maintenance in Debian.
A few of these required their own bits of yak-shaving:
Core Python Packages, by Stefano Rivera
Just before the freeze, pip added support for PEP-668.
This is a scheme devised by Debian with other distributions and the Python
Packaging Authority, to allow distributors to mark Python installations as
being managed by a distribution package manager. When this EXTERNALLY-MANAGED
flag is present, installers like pip will refuse to install packages outside
a virtual environment. This protects users from breaking unrelated software
on their systems, when installing packages with pip or similar tools. Stefano
quickly got this version of pip into the archive, marked Debian s Python
interpreters as EXTERNALLY-MANAGED, and worked with the upstream to add a
mechanism to allow users to override the restriction. Debian bookworm will
likely be the first distro release to implement this change.
The transition from Python 3.10 to 3.11 was one of the last to complete before
the bookworm freeze (as 3.11 only released at the end of October 2022). Stefano
helped port some Python packages to 3.11, in January, and also kicked off the
final transition to remove Python 3.10 support.
Stefano did a big round of bug triage in the cPython interpreter (and related)
packages, applying some provided patches, and fixing some long-standing minor
bugs in the packaging.
To allow Debian packages to more accurately reflect upstream-specified
dependencies that only apply under specific Python interpreter versions, in the
future, Stefano
added more metadata
to the python3 binary package.
Python s unittest runner would successfully exit with 0 passed tests, if it
couldn t find any tests. This means that configuration / layout changes can
cause test failures to go unnoticed, because the tests aren t being run any
more in Debian packages. Stefano
proposed a change to Python
3.12 to change this behavior and treat 0 tests as a kind of failure.
debvm, by Helmut Grohne
With support from Johannes Schauer Marin Rodrigues, and Jochen Sprickerhof,
Helmut Grohne wrote debvm, a tool for
quickly creating and running Debian virtual machine images for various
architectures and Debian and Ubuntu releases. This is meant for development
and testing purposes and has already identified a number of bugs in e.g.
fakechroot (#1029490), Linux
(#1029270), and runit
(#1028181).
Rails 6 and Redmine 5 available in bullseye-backports, by Utkarsh Gupta
Bullseye users can now upgrade to the latest 6.1 branch of Rails, v6.1.7, and
the latest Redmine version, v5.0.4. The Ruby team received numerous requests
to backport the latest version of Rails and Redmine, especially since there was
no redmine shipped in the bullseye release itself. So this is big news for all
users as we ve not only successfully backported both the packages, but also
fixed all the CVEs and RC bugs in the process!
This work was sponsored by Entrouvert.
Patches metadata in the Package Tracker, by Rapha l Hertzog
Building on the great Ultimate Debian Database work of Lucas Nussbaum and on
his suggestion, Rapha l enhanced the
Debian Package Tracker to display action items
when the patches metadata indicate that some patches were not forwarded
upstream, or when the metadata were invalid. One can now also browse the
patches metadata from the Links panel on the right.
Fixed kernel bug that broke debian-installer on computers with Mediatek wifi devices, by Helmut Grohne
As part of our regular work on Kali Linux for OffSec,
they funded Helmut s work to fix the MT7921e driver. When being loaded without
firmware available, it would not register itself, but upon module release it
would unregister itself causing a kernel oops.
This was commonly observed in Kali Linux when reloading the module to add
firmware. Helmut Grohne identified the cause and
sent a patch, a
different variant of which is now
heading into Linux
and available from Kali Linux.
Printing in Debian, by Thorsten Alteholz
There are about 40 packages in Debian that take care of sending output to
printers, scan documents, or even send documents to fax machines. In the light
of the upcoming/already ongoing freeze, these packages had to be updated to
the latest version and bugs had to be fixed. Basically this applies to large
packages like cups, cups-filters, hplip but also the smaller ones that
shouldn t be neglected. All in all Thorsten uploaded 13 packages with new
upstream versions or improved packaging and could resolve 14 bugs. Further
triaging led to 35 bugs that could be closed, either because they were already
fixed and not closed in an earlier upload or they could not be reproduced with
current software versions.
There is also work to do to prepare for the future. Historically, printing on
Linux required finding a PPD file for your printer and finding some software
that is able to render your documents with this PPD. These days, driverless
printing is becoming more common and the use of PPD files has decreased.
In the upcoming version 3.0 of cups, PPD files are no longer supported and so
called printer applications need to be used. In order not to lose the ability
to print documents, this big transition needs to be carefully planned. This
started in the beginning of 2023 and will hopefully be finished with the
release of Debian Trixie. More information can be found in
this Debian Printing Wiki article.
In preparation for this transition Thorsten created three new packages.
Yade update, by Anton Gladky
Last month, Anton updated the yade package to the newest 2023.02a version,
which includes new features.
Yade is a software package for discrete element method (DEM) simulations, which
are widely used in scientific and engineering fields for the simulation of
granular systems. Yade is an open-source project that is being used worldwide
for different tasks, such as geomechanics, civil engineering, mining, and
materials science.
The Yade package in Debian supports different precision levels for its
simulations. This means that researchers and engineers can select the needed
precision level without recompiling the package, saving time and effort.
Miscellaneous contributions
Helmut Grohne continues to improve cross building (mostly Qt) and
architecture bootstrap (mostly loong64 and musl).
The Ruby team is working now on transitioning to ruby 3.0. Even though most
packages will work just fine, there is substantial amount of packages that
require some work to adapt. We have been doing test rebuilds for a while during
transitions, but usually triaged the problems manually.
This time I decided to try
collab-qa-tools, a set of
scripts Lucas Nussbaum uses when he does archive-wide rebuilds. I'm really glad
that I did, because those tols save a lot of time when processing a large
number of build failures. In this post, I will go through how to triage a set
of build logs using collab-qa-tools.
I have
somesomeimprovements
to the code. Given my last merge request is very new and was not merged yet,
a few of the things I mention here may apply only to
my own ruby3.0 branch.
collab-qa-tools also contains a few tools do perform the builds in the
cloud, but since we already had the builds done, I will not be mentioning that
part and will write exclusively about the triaging tools.
Installing collab-qa-tools
The first step is to clone the git repository. Make sure you have the
dependencies from debian/control installed (a few Ruby libraries).
One of the patches I sent, and was already accepted, is the ability to run it
without the need to install:
source /path/to/collab-qa-tools/activate.sh
This will add the tools to your $PATH.
Preparation
The first think you need to do is getting all your build logs in a directory.
The tools assume .log file extension, and they can be named
$ PACKAGE _*.log or just $ PACKAGE .log.
Creating a TODO file
cqa-scanlogs grep -v OK > todo
todo will contain one line for each log with a summary of the failure, if
it's able to find one. collab-qa-tools has a large set of regular expressions
for finding errors in the build logs
It's a good idea to split the TODO file in multiple ones. This can easily be
done with split(1), and can be used to delimit triaging sessions, and/or to
split the triaging between multiple people. For example this will create
todo into todo00, todo01, ..., each containing 30 lines:
split --lines=30 --numeric-suffixes todo todo
Triaging
You can now do the triaging. Let's say we split the TODO files, and will start
with todo01.
The first step is calling cqa-fetchbugs (it does what it says on the tin):
cqa-fetchbugs --TODO=todo01
Then, cqa-annotate will guide you through the logs and allow you to report
bugs:
cqa-annotate --TODO=todo01
I wrote myself a process.sh wrapper script for cqa-fetchbugs and
cqa-annotate that looks like this:
#!/bin/shset-eufor todo in$@;do# force downloading bugsawk' print(".bugs." $1) '"$ todo" xargs rm-f
cqa-fetchbugs --TODO="$ todo"
cqa-annotate \--template=template.txt.jinja2 \--TODO="$ todo"done
The --template option is a recent contribution of mine. This is a template
for the bug reports you will be sending. It uses
Liquid templates,
which is very similar to Jinja2 for Python. You will notice that I am even
pretending it is Jinja2 to trick vim into doing syntax highlighting
for me. The template I'm using looks like this:
From: fullname<email>
To: submit@bugs.debian.org
Subject: package: FTBFS with ruby3.0: summary
Source: package
Version: versionsplit:'+rebuild'first
Severity: serious
Justification: FTBFS
Tags: bookworm sid ftbfs
User: debian-ruby@lists.debian.org
Usertags: ruby3.0
Hi,
We are about to enable building against ruby3.0 on unstable. During a test
rebuild, package was found to fail to build in that situation.
To reproduce this locally, you need to install ruby-all-dev from experimental
on an unstable system or build chroot.
Relevant part (hopefully):
%forlineinextract% > line %endfor%
The full build log is available at
https://people.debian.org/~kanashiro/ruby3.0/round2/builds/3/package/filenamereplace:".log",".build.txt"
The cqa-annotate loop
cqa-annotate will parse each log file, display an extract of what it found as
possibly being the relevant part, and wait for your input:
######## ruby-cocaine_0.5.8-1.1+rebuild1633376733_amd64.log ########
--------- Error:
Failure/Error: undef_method :exitstatus
FrozenError:
can't modify frozen object: pid 2351759 exit 0
# ./spec/support/unsetting_exitstatus.rb:4:in undef_method'
# ./spec/support/unsetting_exitstatus.rb:4:in singleton class'
# ./spec/support/unsetting_exitstatus.rb:3:in assuming_no_processes_have_been_run'
# ./spec/cocaine/errors_spec.rb:55:in block (2 levels) in <top (required)>'
Deprecation Warnings:
Using should from rspec-expectations' old :should syntax without explicitly enabling the syntax is deprecated. Use the new :expect syntax or explicitly enable :should with config.expect_with(:rspec) c c.syntax = :should instead. Called from /<<PKGBUILDDIR>>/spec/cocaine/command_line/runners/backticks_runner_spec.rb:19:in block (2 levels) in <top (required)>'.
If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
config.raise_errors_for_deprecations! , and it will turn the
deprecation warnings into errors, giving you the full backtrace.
1 deprecation warning total
Finished in 6.87 seconds (files took 2.68 seconds to load)
67 examples, 1 failure
Failed examples:
rspec ./spec/cocaine/errors_spec.rb:54 # When an error happens does not blow up if running the command errored before execution
/usr/bin/ruby3.0 -I/usr/share/rubygems-integration/all/gems/rspec-support-3.9.3/lib:/usr/share/rubygems-integration/all/gems/rspec-core-3.9.2/lib /usr/share/rubygems-integration/all/gems/rspec-core-3.9.2/exe/rspec --pattern ./spec/\*\*/\*_spec.rb --format documentation failed
ERROR: Test "ruby3.0" failed:
----------------
ERROR: Test "ruby3.0" failed: Failure/Error: undef_method :exitstatus
----------------
package: ruby-cocaine
lines: 30
------------------------------------------------------------------------
s: skip
i: ignore this package permanently
r: report new bug
f: view full log
------------------------------------------------------------------------
Action [s i r f]:
You can then choose one of the options:
s - skip this package and do nothing. You can run cqa-annotate again
later and come back to it.
i - ignore this package completely. New runs of cqa-annotate won't ask
about it again.
This is useful if the package only fails in your rebuilds due to another
package, and would just work when that other package gets fixes. In the Ruby
transition this happens when A depends on B, while B builds a C extension and
failed to build against the new Ruby. So once B is fixed, A should
just work (in principle). But even if A would even have problems of its own,
we can't really know before B is fixed so we can retry A.
r - report a bug. cqa-annotate will expand the template with the data
from the current log, and feed it to mutt. This is currently a limitation:
you have to use mutt to report bugs.
After you report the bug, cqa-annotate will ask if it should edit the TODO
file. In my opinion it's best to not do this, and annotate the package with a
bug number when you have one (see below).
f - view the full log. This is useful when the extract displayed doesn't
have enough info, or you want to inspect something that happened earlier (or
later) during the build.
When there are existing bugs in the package, cqa-annotate will list them
among the options. If you choose a bug number, the TODO file will be annotated
with that bug number and new runs of cqa-annotate will not ask about that
package anymore. For example after I reported a bug for ruby-cocaine for the
issue listed above, I aborted with a ctrl-c, and when I run my process.sh
script again I then get this prompt:
----------------
ERROR: Test "ruby3.0" failed: Failure/Error: undef_method :exitstatus
----------------
package: ruby-cocaine
lines: 30
------------------------------------------------------------------------
s: skip
i: ignore this package permanently
1: 996206 serious ruby-cocaine: FTBFS with ruby3.0: ERROR: Test "ruby3.0" failed: Failure/Error: undef_method :exitstatus
r: report new bug
f: view full log
------------------------------------------------------------------------
Action [s i 1 r f]:
Chosing 1 will annotate the TODO file with the bug number, and I'm done with
this package. Only a few other hundreds to go.
Welcome to the October 2020 report from the Reproducible Builds project.
In our monthly reports, we outline the major things that we have been up to over the past month. As a brief reminder, the motivation behind the Reproducible Builds effort is to ensure flaws have not been introduced in the binaries we install on our systems. If you are interested in contributing to the project, please visit our main website.
The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.
GNU Mes rebuild is definitely an application of [Diverse Double-Compiling]. [..] This is an awesome application of DDC, and I believe it s the first publicly acknowledged use of DDC on a binary
Debian-related work
In August, Lucas Nussbaum performed an archive-wide rebuild of packages to test enabling the reproducible=+fixfilepath Debian build flag by default. Enabling this fixfilepath feature will likely fix reproducibility issues in an estimated 500-700 packages. However, this month Vagrant Cascadian posted to the debian-devel mailing list:
It would be great to see the reproducible=+fixfilepath feature enabled by default in dpkg-buildflags, and we would like to proceed forward with this soon unless we hear any major concerns or other outstanding issues. [ ] We would like to move forward with this change soon, so please raise any concerns or issues not covered already.
Debian Developer Stuart Prescott has been improving python-debian, a Python library that is used to parse Debian-specific files such as changelogs, .dscs, etc. In particular, Stuart is working on adding support for .buildinfo files used for recording reproducibility-related build metadata:
This can mostly be a very thin layer around the existing Deb822 types, using the existing Changes code for the file listings, the existing PkgRelations code for the package listing and gpg_* functions for signature handling.
Bernhard M. Wiedemann also reported three issues against bison, ibus and postgresql12.
Tools
diffoscope is our in-depth and content-aware diff utility. Not only could you locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds too. This month, Chris Lamb uploaded version 161 to Debian (later backported by Mattia Rizzolo), as well as made the following changes:
Move test_ocaml to the assert_diff helper. []
Update tests to support OCaml version 4.11.1. Thanks to Sebastian Ramacher for the report. (#972518)
Bump minimum version of the Black source code formatter to 20.8b1. (#972518)
In addition, Jean-Romain Garnier temporarily updated the dependency on radare2 to ensure our test pipelines continue to work [], and for the GNU Guix distribution Vagrant Cascadian diffoscope to version 161 [].
In related development, trydiffoscope is the web-based version of diffoscope. This month, Chris Lamb made the following changes:
Mark a --help-only test as being a superficial test. (#971506)
Add a real, albeit flaky, test that interacts with the try.diffoscope.org service. []
Bump debhelper compatibility level to 13 [] and bump Standards-Version to 4.5.0 [].
Lastly, disorderfs version 0.5.10-2 was uploaded to Debian unstable by Holger Levsen, which enabled security hardening via DEB_BUILD_MAINT_OPTIONS [] and dropped debian/disorderfs.lintian-overrides [].
Add a citation link to the academic article regarding dettrace [], and added yet another supply-chain security attack publication [].
Reformatted the Jekyll s Liquid templating language and CSS formatting to be consistent [] as well as expand a number of tab characters [].
Used relative_url to fix missing translation icon on various pages. []
Published two announcement blog posts regarding the restarting of our IRC meetings. [][]
Added an explicit note regarding the lack of an in-person summit in 2020 to our events page. []
Testing framework
The Reproducible Builds project operates a Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
Make a number of updates to reflect that our sponsor Profitbricks has renamed itself to IONOS. [][][][]
Run a F-Droid maintenance routine twice a month to utilise its cleanup features. []
Fix the target name in OpenWrt builds to ath79 from ath97. []
Add a missing Postfix configuration for a node. []
Temporarily disable Arch Linux builds until a core node is back. []
Make a number of changes to our thanks page. [][][]
Build node maintenance was performed by both Holger Levsen [][] and Vagrant Cascadian [][][], Vagrant Cascadian also updated the page listing the variations made when testing to reflect changes for in build paths [] and Hans-Christoph Steiner made a number of changes for F-Droid, the free software app repository for Android devices, including:
Do not fail reproducibility jobs when their cleanup tasks fail. []
Skip libvirt-related sudo command if we are not actually running libvirt. []
Use direct URLs in order to eliminate a useless HTTP redirect. []
If you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. However, you can also get in touch with us via:
Welcome to the September 2020 report from the Reproducible Builds project. In our monthly reports, we attempt to summarise the things that we have been up to over the past month, but if you are interested in contributing to the project, please visit our main website.
This month, the Reproducible Builds project was pleased to announce a donation from Amateur Radio Digital Communications (ARDC) in support of its goals. ARDC s contribution will propel the Reproducible Builds project s efforts in ensuring the future health, security and sustainability of our increasingly digital society. Amateur Radio Digital Communications (ARDC) is a non-profit which was formed to further research and experimentation with digital communications using radio, with a goal of advancing the state of the art of amateur radio and to educate radio operators in these techniques. You can view the full announcement as well as more information about ARDC on their website.
In August s report, we announced that Jennifer Helsby (redshiftzero) launched a new reproduciblewheels.com website to address the lack of reproducibility of Python wheels . This month, Kushal Das posted a brief follow-up to provide an update on reproducible sources as well.
The Threema privacy and security-oriented messaging application announced that within the next months , their apps will become fully open source, supporting reproducible builds :
This is to say that anyone will be able to independently review Threema s security and verify that the published source code corresponds to the downloaded app.
The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.
During the Reproducible Builds summit in Marrakesh, GNU Guix, NixOS and Debian were able to produce a bit-for-bit identical binary when building GNU Mes, despite using three different major versions of GCC. Since the summit, additional work resulted in a bit-for-bit identical Mes binary using tcc and this month, a fuller update was posted by the individuals involved.
diffoscopediffoscope is our in-depth and content-aware diff utility that can not only locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds too.
In September, Chris Lamb made the following changes to diffoscope, including preparing and uploading versions 159 and 160 to Debian:
New features:
Show ordering differences only in strings(1) output by applying the ordering check to all differences across the codebase. []
Bug fixes:
Mark some PGP tests that they require pgpdump, and check that the associated binary is actually installed before attempting to run it. (#969753)
Don t raise exceptions when cleaning up after guestfs cleanup failure. []
Ensure we check FALLBACK_FILE_EXTENSION_SUFFIX, otherwise we run pgpdump against all files that are recognised by file(1) as data. []
Codebase improvements:
Add some documentation for the EXTERNAL_TOOLS dictionary. []
Abstract out a variable we use a couple of times. []
Improve the documentation on how to signup to Salsa. []
Add some more links to academic papers. []
Also include the general news in our RSS feed [] and drop including weekly reports from the RSS feed (they are never shown now that we have over 10 items) [].
Update ordering and location of various news and links to tarballs, etc. [][][]
In addition, Holger Levsen re-added the documentation link to the top-level navigation [] and documented that the jekyll-polyglot package is required []. Lastly, diffoscope.org and reproducible-builds.org were transferred to Software Freedom Conservancy. Many thanks to Brett Smith from Conservancy, J r my Bobbio (lunar) and Holger Levsen for their help with transferring and to Mattia Rizzolo for initiating this.
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of these patches, including:
Testing framework
The Reproducible Builds project operates a Jenkins-based testing framework to power tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
Highlight important bad conditions in colour. [][]
Add support for detecting more problems, including Jenkins shutdown issues [], failure to upgrade Arch Linux packages [], kernels with wrong permissions [], etc.
Misc:
Delete old schroot sessions after 2 days, not 3. []
In addition, stefan0xC fixed a query for unknown results in the handling of Arch Linux packages [] and Mattia Rizzolo updated the template that notifies maintainers by email of their newly-unreproducible packages to ensure that it did not get caught in junk/spam folders []. Finally, build node maintenance was performed by Holger Levsen [][][][], Mattia Rizzolo [][] and Vagrant Cascadian [][][].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
As part of the LLVM release cycle, I am continuing rebuilding the Debian archive with clang instead of gcc to evaluate potential regressions.
Processed results are available on the website: https://clang.debian.net/status.php - Now includes some fancy graphs to show the evolution
Raw logs are published on github: https://github.com/opencollab/clang.debian.net/tree/master/logs
Since my last blog post on the subject (August 2017), Clang is more and more present in the tech ecosystem. It is now the compiler used to build Firefox and Chrome upstream binaries on all the supported architectures/operating systems. More architectures are supported, it has a new linker (lld), a new hybrid IR (MLIR), a lot of checkers in clang-tidy, cross-language linking with Rust, etc.
Results
Now, about Debian results, we rebuilt using 8.0.1, 9.0.1 and 10.0rc2. Results are pretty similar to what we had with previous versions: between 4 to 5% of packages are failing when gcc is replaced by clang.
Even if most of the software are still using gcc as compiler, we can see that clang has a positive effect on code quality. With many different kinds of errors and warnings found clang over the years, we noticed a steady decline of the number of errors. For example, the number of incorrect C/C++ main declarations has been decreasing years after years:
Errors found
The biggest offender is still the qmake changes which doesn't allow the used workaround (replacing /usr/bin/gcc by /usr/bin/clang) - about 250 errors. Most of these packages would probably compile fine with clang. More on the Qt bug tracker. The workaround proposed in the bug isn't applicable for us as we use the dropped-in replacement of the
The second error is still some differences in symbol generation. Unlike gcc, it seems that clang doesn't generate some symbols (or adds some). As a bunch of Debian packages are checking the list of symbols in the library (for ABI management), the build fails on purpose. For example, with libcec, the symbol _ZN10P8PLATFORM14CConditionImplD1Ev@Base 3.1.0 isn't generated anymore. I am not expecting this to be a big deal: the generated libraries probably works most of the time. More on C++ symbol management in Debian.
Current status
As previously said in a blog post, I don't think there is a strong intensive to go away from gcc for most of the Linux distributions. The big reason for BSD was the license (even if the move to the Apache 2 license wasn't received positively by some of them).
While the LLVM/clang ecosystem clearly won the tooling battle, as a C/C++ compiler, gcc is still an excellent compiler which supports more architecture and more languages.
In term of new warnings and checks, as the clang community moved the efforts in clang-tidy (which requires more complex tooling), out of the box, gcc provides a better experience (as example, see the Firefox meta bug to build with -Werror with the default warnings using gcc 9, gcc 10 and clang trunk for example).
Next steps
I see some potential next steps to decrease the number of failure:
Workaround the Qt/Qmake issue
Fix the objective-c header include issues (echo "#include <objc/objc.h>" > foo.m && clang -c foo.m is currently failing)
Identify why clang generates more/less symbols that gcc in the library and try to fix that
Rebuild the archive with clang-7 - Seems that I have some data problem
As part of the LLVM release cycle, I am continuing rebuilding the Debian archive with clang instead of gcc to evaluate potential regressions.
Processed results are available on the website: https://clang.debian.net/status.php - Now includes some fancy graphs to show the evolution
Raw logs are published on github: https://github.com/opencollab/clang.debian.net/tree/master/logs
Since my last blog post on the subject (August 2017), Clang is more and more present in the tech ecosystem. It is now the compiler used to build Firefox and Chrome upstream binaries on all the supported architectures/operating systems. More architectures are supported, it has a new linker (lld), a new hybrid IR (MLIR), a lot of checkers in clang-tidy, cross-language linking with Rust, etc.
Results
Now, about Debian results, we rebuilt using 8.0.1, 9.0.1 and 10.0rc2. Results are pretty similar to what we had with previous versions: between 4 to 5% of packages are failing when gcc is replaced by clang.
Even if most of the software are still using gcc as compiler, we can see that clang has a positive effect on code quality. With many different kinds of errors and warnings found clang over the years, we noticed a steady decline of the number of errors. For example, the number of incorrect C/C++ main declarations has been decreasing years after years:
Errors found
The biggest offender is still the qmake changes which doesn't allow the used workaround (replacing /usr/bin/gcc by /usr/bin/clang) - about 250 errors. Most of these packages would probably compile fine with clang. More on the Qt bug tracker. The workaround proposed in the bug isn't applicable for us as we use the dropped-in replacement of the compiler.
The second error is still some differences in symbol generation. Unlike gcc, it seems that clang doesn't generate some symbols (or adds some). As a bunch of Debian packages are checking the list of symbols in the library (for ABI management), the build fails on purpose. For example, with libcec, the symbol _ZN10P8PLATFORM14CConditionImplD1Ev@Base 3.1.0 isn't generated anymore. I am not expecting this to be a big deal: the generated libraries probably works most of the time. More on C++ symbol management in Debian.
I reported this bug upstream a while back: https://bugs.llvm.org/show_bug.cgi?id=30441
Current status
As previously said in a blog post, I don't think there is a strong intensive to go away from gcc for most of the Linux distributions. The big reason for BSD was the license (even if the move to the Apache 2 license wasn't received positively by some of them).
While the LLVM/clang ecosystem clearly won the tooling battle, as a C/C++ compiler, gcc is still an excellent compiler which supports more architecture and more languages.
In term of new warnings and checks, as the clang community moved the efforts in clang-tidy (which requires more complex tooling), out of the box, gcc provides a better experience (as example, see the Firefox meta bug to build with -Werror with the default warnings using gcc 9, gcc 10 and clang trunk for example).
Next steps
I see some potential next steps to decrease the number of failure:
Workaround the Qt/Qmake issue
Fix the objective-c header include issues (echo "#include <objc/objc.h>" > foo.m && clang -c foo.m is currently failing)
Identify why clang generates more/less symbols that gcc in the library and try to fix that
Rebuild the archive with clang-7 - Seems that I have some data problem
Here's what happened in the Reproducible Builds effort between Sunday October 29 and Saturday November 4 2017:
Past events
From October 31st November 2nd we held the 3rd Reproducible Builds summit in Berlin, Germany. A full, in-depth report will be posted in the next week or so.
Upcoming events
On November 8th Jonathan Bustillos Osornio (jathan) will present at CubaConf Havana.
On November 17th Chris Lamb will present at Open Compliance Summit, Yokohama, Japan on how reproducible builds ensures the long-term sustainability of technology infrastructure.
Reviews of unreproducible packages
7 package reviews have been added, 43 have been updated and 47 have been removed in this week,
adding to our knowledge about identified issues.
Weekly QA work
During our reproducibility testing, FTBFS bugs have been detected and reported by:
diffoscope development
Version 88 was uploaded to unstable by Mattia Rizzolo.
It included contributions
(already covered by posts of the previous weeks) from:
Mattia Rizzolo
tests/comparators/dtb: compatibility with version 1.4.5. (Closes: #880279)
Chris Lamb
comparators:
binwalk: improve names in output of "internal" members. #877525
Omit misleading "any of" prefix when only complaining about one module in ImportError messages.
Don't crash on malformed "md5sums" files. (Closes: #877473)
tests/comparators:
ps: ps2ascii > 9.21 now varies on timezone, so skip this test for now.
dtby: only parse the version number, not any "-dirty" suffix.
debian/watch: Use HTTPS URI.
Ximin Luo
comparators:
utils/file: Diff container metadata centrally. This fixes a last remaining bug in fuzzy-matching across containers. (Closes: #797759)
Fix all the affected comparators after the above change.
Holger Levsen
Bump Standards-Version to 4.1.1, no changes needed.
strip-nondeterminism development
Version 0.040-1 was uploaded to unstable by Mattia Rizzolo.
It included contributions
already covered by posts of the previous weeks, as well as new ones from:
Version 0.5.2-2 was uploaded to unstable by Holger Levsen.
It included contributions
already covered by posts of the previous weeks, as well as new ones from:
archlinux: try to re-enable one schroot creation job
lynxis
lede: replace TMPDIR -> RESULTSDIR
lede: openwrt_get_banner(): use locals instead of globals
lede: add newline to $CONFIG
lede: show git log -1 in jenkins log
Holger Levsen:
lede: add very simple landing page
Juliana Oliveira Rodrigues
archlinux: adds pacman-git dependencies
kpcyrd
archlinux: disable signature verification when running in the future
archlinux: use pacman-git until the next release
archlinux: make pacman fail less early
archlinux: use sudo to prepare chroot
archlinux: remove -rf for regular file
archlinux: avoid possible TOCTOU issue
archlinux: Try to fix tar extraction
archlinux: fix sha1sums parsing
Misc.
This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.
There has also been substantial activity regarding updating the Debian Policy regarding Reproducible Builds (#844431). This work is still ongoing and will be covered in future issues of this weekly blog.
Notes about reviews of unreproducible packages
13 package reviews have been added, 7 have been updated and 34 have been removed in this week,
adding to our knowledge about identified issues.
Packages reviewed and fixed, and reproducibility related bugs filed
Here's what happened in the Reproducible
Builds effort between Sunday July 30 and
Saturday August 5 2017:
Media coverage
We were mentioned on Late Night Linux Episode
17, around 29:30.
Packages reviewed and fixed, and bugs filed
Upstream packages:
Reviews of unreproducible packages
29 package reviews have been added, 72 have been updated and 151 have been
removed in this week, adding to our knowledge about identified
issues.
4 issue types have been updated:
Weekly QA work
During our reproducibility testing, FTBFS bugs have been detected and reported by:
Adrian Bunk (36)
Andreas Beckmann (2)
Daniel Schepler (2)
Logan Rosen (1)
Lucas Nussbaum (93)
diffoscope development
Version 85 was uploaded to unstable by Mattia Rizzolo. It included contributions from:
Mattia Rizzolo:
Add an explicit Recommends: on the defusedxml python package.
Various other code quality tweaks.
Juliana Oliveira Rodrigues:
Fix test_ico_image for ImageMagick identify >= 6.9.8.
Use the defusedxml XML library by default in the XML comparator, if it's
available. This protects against various XML parser DoS attacks and other
security holes, which other Python XML libraries are vulnerable to.
Ximin Luo:
Force a flush when writing output to diff. (Closes: #870049).
as well as previous weeks' contributions, summarised in the
changelog.
There were also further commits in git, which will be released in a later
version:
Guangyuan Yang:
tests/iso9660: support isoinfo's output coming from cdrtools' version
instead of genisoimage's
Mattia Rizzolo:
Code quality and test fixes.
Chris Lamb:
Code quality and test fixes.
Misc.
This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris
Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing
lists.
I ve been increasingly using systemd timers as a replacement for cron jobs. The fact that you get free logging is great, and also the fact that you don t have to care about multiple instances running simultaneously.
However, sometimes I would be interested in more complex scenarios, such as:
I d like to trigger a full run of the service unit: if the service is not running, it should be started immediately. If it s currently running, it should be started again when it terminates.
Same as the above, but with queue coalescing: If I do the above multiple times in a row, I only want the guarantee that there s one full run of the service after the last time I triggered it (typical scenario: each run processes all pending events, so there s no point in running multiple times).
Is this doable with systemd? If not, how do people do that outside of systemd?
Here's what happened in the Reproducible Builds effort between Sunday July 16 and Saturday July 22 2017:
Toolchain development
Bernhard M. Wiedemann wrote a
tool
to automatically run through different sources of non-determinism, and report
which of these caused irreproducibility.
Dan Kegel's patches to fpm were merged.
Bugs filed
Patches submitted upstream:
Reviews of unreproducible packages
73 package reviews have been added, 44 have been updated and 50 have been
removed in this week, adding to our knowledge about identified
issues.
No issue types were updated.
Weekly QA work
During our reproducibility testing, FTBFS bugs have been detected and reported by:
comparators.xml: Fix EPUB "missing file" tests; they ship a META-INF/container.xml file.
comparators.sqlite: Simplify file detection in Sqlite3Database.RE_FILE_TYPE
Style and attribution fixes to XML comparator and comparators.directory
Ximin Luo:
main, logging: restore old logger settings to avoid pytest vomiting in certain situations
comparators/directory: Fix #868534 by expecting less strict test output
reprotest development
Ximin Luo:
Use autopkgtest upstream paths, makes things easier to import
Add script for importing autopkgtest code, and import autopkgtest 4.4
Ximin also restarted the discussion with
autopkgtest-devel
about code reuse for reprotest.
Santiago Torres began a series of
patches
to make reprotest more distro-agnostic, with the aim of making it usable on
Arch Linux. Ximin reviewed these patches.
Misc.
This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris
Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing
lists.
Here's what happened in the Reproducible
Builds effort between Sunday July 2 and
Saturday July 8 2017:
Reproducible work in other projects
Ed Maste pointed to a thread on the LLVM developer mailing
list about
container iteration being the main source of non-determinism in LLVM, together
with discussion on how to solve this. Ignoring build path issues, container
iteration order was also the main issue with rustc, which was fixed by using a
fixed-order hash
map
for certain compiler structures. (It was unclear from the thread whether LLVM's
builds are truly path-independent or rather that they haven't done comparisons
between builds run under different paths.)
Bugs filed
Reviews of unreproducible packages
52 package reviews have been added, 62 have been updated and 20 have been
removed in this week, adding to our knowledge about identified
issues.
No issue types were updated or added this week.
Weekly QA work
During our reproducibility testing, FTBFS bugs have been detected and reported by:
Adrian Bunk (143)
Andreas Beckmann (1)
Dmitry Shachnev (1)
Lucas Nussbaum (3)
Niko Tyni (3)
Scott Kitterman (1)
Sean Whitton (1)
diffoscope development
Development continued in
git with
contributions from:
Ximin Luo:
Add a PartialString class to help with lazily-loaded output formats such as
html-dir.
html and html-dir output:
add a size-hint to the diff headers and lazy-load buttons
add new limit flags and deprecate old ones
html-dir output
split index pages up if they get too big
put css/icon data in separate files to avoid duplication
main: warn if loading a diff but also giving diff-calculation flags
Test fixes for Python 3.6 and CI environments without imagemagick (#865625).
Fix a performance regression (#865660) involving the Wagner-Fischer
algorithm for calculating levenshtein distance.
With these changes, we are able to generate a dynamically loaded HTML
diff
for GCC-6 that can be displayed in a normal web browser. For more details see
this mailing list
post.
Misc.
This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris
Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing
lists.
France passed a law about right to disconnect (more info here or here). The idea of not sending professional emails when people are not supposed to read them in order to protect their private lifes, is a pretty good one, especially when hierarchy is involved. However, I tend to do email at random times, and I would rather continue doing that, but just delay the actual sending of the email to the appropriate time (e.g., when I do email in the evening, it would actually be sent the following morning at 9am).
I wonder how I could make this fit into my email workflow. I write email using mutt on my laptop, then push it locally to nullmailer, that then relays it, over an SSH tunnel, to a remote server (running Exim4).
Of course the fallback solution would be to use mutt s postponing feature. Or to draft the email in a text editor. But that s not really nice, because it requires going back to the email at the appropriate time. I would like a solution where I would write the email, add a header (or maybe manually add a Date: header in all cases that header should reflect the time the mail was sent, not the time it was written), send the email, and have nullmailer or the remote server queue it until the appropriate time is reached (e.g., delaying while current_time < Date header in email ). I don t want to do that for all emails: e.g. personal emails can go out immediately.
Any ideas on how to implement that? I m attached to mutt and relaying using SSH, but not attached to nullmailer or exim4. Ideally the delaying would happen on my remote server, so that my laptop doesn t need to be online at the appropriate time.
Update: mutt does not allow to set the Date: field manually (if you enable the edit_headers option and edit it manually, its value gets overwritten). I did not find the relevant code yet, but that behaviour is mentioned in that bug.
Update 2: ah, it s this code in sendlib.c (and there s no way to configure that behaviour):
/* mutt_write_rfc822_header() only writes out a Date: header with
* mode == 0, i.e. _not_ postponment; so write out one ourself */
if (post)
fprintf (msg->fp, "%s", mutt_make_date (buf, sizeof (buf)));
Reproducible work in other projects
Reproducible Builds have been mentioned in the FSF high-priority project
list.
The F-Droid Verification Server has been
launched. It rebuilds apps from source that were built by
f-droid.org and checks that the results match.
Bernhard M. Wiedemann did some more work on
reproducibility for openSUSE.
Bootstrappable.org (unfortunately no HTTPS yet)
was launched after the initial work was started at our recent summit in Berlin.
This is another topic related to reproducible builds and both will be needed in
order to perform "Diverse Double Compilation" in practice in the future.
Toolchain development and fixes
Ximin Luo researched data formats for SOURCE_PREFIX_MAP and explored
different options for encoding a map data structure in a single environment
variable. He also continued to talk with the rustc
team on the topic.
Daniel Shahaf filed #851225 ('udd: patches: index by DEP-3 "Forwarded"
status') to make it easier to track our patches.
Chris Lamb forwarded #849972 upstream to
yard, a Ruby documentation
generator. Upstream has fixed the issue as of release
0.9.6.
Alexander Couzens (lynxis) has made mksquashfs
reproducible and is
looking for testers.
It compiles on BSD systems such as FreeBSD, OpenBSD and NetBSD.
Bugs filed
Chris Lamb:
Reviews of unreproducible packages
13 package reviews have been added and 13 have been removed in this week,
adding to our knowledge about identified
issues.
1 issue type has been added:
Weekly QA work
During our reproducibility testing, the following FTBFS bugs have been detected
and reported by:
Chris Lamb (3)
Lucas Nussbaum (11)
Nicola Corna (1)
diffoscope development
Many bugs were opened in
diffoscope during the
past few weeks, which probably is a good sign as it shows that diffoscope is
much more widely used than a year ago. We have been working hard to squash many
of them in time for Debian stable, though we will see how that goes in the end
Ximin Luo and Holger Levsen worked on stricter tests to check that /dev/shm
and /run/shm are both mounted with the correct permissions. Some of our
build machines currently still fail this test, and the problem is probably
the root cause of the
FTBFS
of some packages (which fails with issues regarding sem_open). The proper
fix is still being discussed in #851427.
Holger also further improved the daily mail notifications about problems.
Misc.
This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen and reviewed by a
bunch of Reproducible Builds folks on IRC & the mailing lists.