A brand new release 0.1.15 of the RcppSimdJson
package is now on CRAN.
RcppSimdJson
wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via
very clever algorithmic engineering to obtain largely branch-free code,
coupled with modern C++ and newer compiler instructions, it results in
parsing gigabytes of JSON parsed per second which is quite
mindboggling. The best-case performance is faster than CPU speed as
use of parallel SIMD instructions and careful branch avoidance can lead
to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire
at QCon.
This version updates to the current 4.2.4 upstream release. It also
updates the RcppExports.cpp file with glue between C++
and R. We want move away from using Rf_error() (as
Rcpp::stop() is generally preferable). Packages (such as
this one) that are declaring an interface have an actual
Rf_error() call generated in RcppExports.cpp
which can protect which is what current Rcpp code
generation does. Long story short, a minor internal reason.
The short NEWS entry for this release follows.
Changes in version 0.1.15
(2026-01-14)
simdjson was upgraded to version 4.2.4
(Dirk in #97
RcppExports.cpp was regenerated to aid a Rcpp transition
Standard maintenance updates for continuous integration and
URLs
A recent article in The Atlantic makes the case that
very large language models effectively contain much of the works they're trained on.
This article is an attempt to popularize the insights in the recent academic paper
Extracting books from production language models from Ahmed et al.
The authors of the paper demonstrate convincingly that
well-known copyrighted textual material can be extracted from
the chatbot interfaces of popular commercial LLM services.
The Atlantic article cites a podcast quote about the Stable Diffusion AI image-generator model, saying "We took 100,000 gigabytes of images and compressed it to a two-gigabyte file that can re-create any of those and iterations of those".
By analogy, this suggests we might think of LLMs (which work on text, not the images handled by Stable Diffusion) as a form of lossy textual compression.
The entire text of Moby Dick, the canonical Big American Novel is merely 1.2MiB uncompressed (and less than 0.4MiB losslessly compressed with bzip2 -9).
It's not surprising to imagine that a model
with hundreds of billions of parameters might contain copies of these works.
Warning: The next paragraph contains fuzzy math with no real concrete engineering practice behind it!
Consider a hypothetical model with 100 billion parameters,
where each parameter is stored as a 16-bit floating point value.
The model weights would take 200 GB of storage.
If you were to fill the parameter space only with losslessly compressed copies of books like Moby Dick,
you could still fit half a million books, more than anyone can read in a lifetime.
And lossy compression is typically orders of magnitude less in size than lossless compression,
so we're talking about millions of works effectively encoded, with the acceptance of some
artifacts being injected in the output.
I first encountered this "compression" view of AI nearly three years ago, in Ted Chiang's insightful
ChatGPT is a Blurry JPEG of the Web.
I was suprised that The Atlantic article didn't cite Chiang's piece.
If you haven't read Ted Chiang, i strongly recommend his work,
and this piece is a great place to start.
Chiang aside, the more recent writing that focuses on the idea of
compressed works being "contained" in the model weights
seems to be used by people interested in wielding
some sort of copyright claims against the AI companies that maintain or provide access to these models.
There are many many problems with AI today, but attacking AI companies based on copyright concerns
seems similar to going after Al Capone for tax evasion.
We should be much more concerned with the effect these projects have on
cultural homogeneity,
mentalhealth,
laborrights,
privacy, and
socialcontrol
than whether they're violating copyright in some specific instance.
A new maintenance release 0.4.25 of RProtoBuf
arrived on CRAN today. RProtoBuf
provides R with bindings for the
Google Protocol Buffers
( ProtoBuf ) data encoding and serialization library used and
released by Google, and deployed very widely in numerous projects as a
language and operating-system agnostic protocol.
This release brings an update to a header use force by R-devel, the
usual set of continunous integration updates, and a large overhaul of
URLs as CRAN is now running
more powerful checks. As a benefit the three vignettes have all been
refreshed. they are now also delivered via the new
Rcpp::asis() vignette builder that permits pre-made pdf
files to be used easily.
The following section from the NEWS.Rd file has full details.
Changes in
RProtoBuf version 0.4.25 (2026-01-11)
Several routine updates to continuous integration script
Include ObjectTable.h instead of
Callback.h to accommodate R 4.6.0
Switch vignettes to Rcpp::asis driver, update
references
Team Rcpp is thrilled to share that an exciting new version 1.1.1 of
Rcpp is now on CRAN (and also
uploaded to Debian and already
built for r2u).
Having switchted to C++11 as the minimum standard in the
previous 1.1.0 release, this version takes full advantage of it and
removes a lot of conditional code catering to older standards
that no longer need to be supported. Consequently, the source tarball
shrinks by 39% from 3.11 mb to 1.88 mb. That is a big deal. (Size peaked
with Rcpp 1.0.12 two years ago at 3.43 mb; relative to its size we are
down 45% !!) Removing unused code also makes maintenance easier, and
quickens both compilation and installation in general.
This release continues as usual with the six-months January-July
cycle started with release
1.0.5 in July 2020. Interim snapshots are always available via the
r-universe page and
repo. We continue to strongly encourage the use of these development
released and their testing we tend to run our systems with them too.
Rcpp has long established itself
as the most popular way of enhancing R with C or C++ code. Right now,
3020 packages on CRAN depend on
Rcpp for making analytical code go
faster and further. On CRAN, 13.1% of all packages depend (directly) on
Rcpp, and 60.9% of all compiled
packages do. From the cloud mirror of CRAN (which is but a subset of all
CRAN downloads), Rcpp has been
downloaded 109.8 million times. The two published papers (also included
in the package as preprint vignettes) have, respectively, 2151 (JSS, 2011) and 405 (TAS, 2018)
citations, while the the book (Springer useR!,
2013) has another 715.
This time, I am not attempting to summarize the different changes.
The full list follows below and details all these changes, their
respective PRs and, if applicable, issue tickets. Big thanks from all of
us to all contributors!
Changes in
Rcpp release version 1.1.1 (2026-01-08)
Changes in Rcpp API:
An unused old R function for a compiler version check has been
removed after checking no known package uses it (Dirk in #1395)
A narrowing warning is avoided via a cast (Dirk in #1398)
Demangling checks have been simplified (I aki in #1401 addressing
#1400)
The treatment of signed zeros is now improved in the Sugar code
(I aki in #1404)
Preparations for phasing out use of Rf_error have
been made (I aki in #1407)
The long-deprecated function loadRcppModules() has
been removed (Dirk in #1416 closing #1415)
Some non-API includes from R were refactored to accommodate
R-devel changes (I aki in #1418 addressing
#1417)
An accessor to Rf_rnbeta has been removed (Dirk in
#1419 also
addressing #1420)
Code accessing non-API Rf_findVarInFrame now uses
R_getVarEx (Dirk in #1423 fixing #1421)
Code conditional on the R version now expects at least R 3.5.0;
older code has been removed (Dirk in #1426 fixing #1425)
The non-API ATTRIB entry point to the R API is no
longer used (Dirk in #1430 addressing
#1429)
The unwind-protect mechanism is now used unconditionally (Dirk in
#1437 closing
#1436)
Changes in Rcpp Attributes:
The OpenMP plugin has been generalized for different macOS compiler
installations (Kevin in #1414)
Changes in Rcpp Documentation:
Vignettes are now processed via a new "asis" processor adopted
from R.rsp (Dirk in #1394 fixing #1393)
R is now cited via its DOI (Dirk)
A (very) stale help page has been removed (Dirk in #1428 fixing #1427)
The main README.md was updated emphasizing r-universe in favor of
the local drat repos (Dirk in #1431)
Changes in Rcpp Deployment:
A temporary change in R-devel concerning NA part in complex
variables was accommodated, and then reverted (Dirk in #1399 fixing #1397)
The macOS CI runners now use macos-14 (Dirk in #1405)
A message is shown if R.h is included before Rcpp
headers as this can lead to errors (Dirk in #1411 closing #1410)
Old helper functions use message() to signal they
are not used, deprecation and removal to follow (Dirk in #1413 closing #1412)
Three tests were being silenced following #1413 (Dirk in #1422)
The heuristic whether to run all available tests was refined
(Dirk in #1434
addressing #1433)
Coverage has been tweaked via additional #nocov tags
(Dirk in #1435)
Non-release Changes:
Two interim non-releases 1.1.0.8.1 and .2 were made in order to
unblock CRAN due to changes in R-devel rather than Rcpp
Welcome to the December 2025 from the Reproducible Builds project!
Our monthly reports outline what we ve been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
New orig-check service to validate Debian upstream tarballs
This month, Debian Developer Lucas Nussbaum announced the orig-check service, which attempts to automatically reproduce the generation upstream tarballs (ie. the original source component of a Debian source package), comparing that to the upstream tarball actually shipped with Debian.
As of the time of writing, it is possible for a Debian developer to upload a source archive that does not actually correspond to upstream s version. Whilst this is not inherently malicious (it typically indicates some tooling/process issue), the very possibility that a maintainer s version may differ potentially permits a maintainer to make (malicious) changes that would be misattributed to upstream.
This service therefore nicely complements the whatsrc.org service, which was reported in our reports for both April and August. The orig-check is dedicated to Lunar, who sadly passed away a year ago.
disorderfs updated to FUSE 3
disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into system calls to reliably flush out reproducibility issues.
This month, however, Roland Clobus upgraded disorderfs* from FUSE 2 to FUSE 3 after its package automatically got removed from Debian testing. Some tests in Debian currently require disorderfs to make the Debian live images reproducible, although disorderfs is not a Debian-specific tool.
Luca Di Maio announced stampdalf, a filesystem timestamp preservation tool that wraps arbitrary commands and ensures filesystem timestamp reproducibility :
stampdalf allows you to run any command that modifies files in a directory tree, then automatically resets all timestamps back to their original values. Any new files created during command execution are set to [the UNIX epoch] or a custom timestamp via SOURCE_DATE_EPOCH.
The project s GitHub page helpfully reveals that the project is pronounced: stamp-dalf (stamp like time-stamp, dalf like Gandalf the wizard) as it s a wizard of time and stamps .)
Lastly, Reproducible Builds developer cen1 posted to our list announcing that early/experimental/alpha support for FreeBSD was added to rebuilderd. In their post, cen1 reports that the initial builds are in progress and look quite decent . cen1 also interestingly notes that since the upstream is currently not technically reproducible I had to relax the bit-for-bit identical requirement of rebuilderd [ ] I consider the pkg to be reproducible if the tar is content-identical (via diffoscope), ignoring timestamps and some of the manifest files. .
Most package managers also generate a lockfile, which records the exact set of resolved dependency versions. Lockfiles are used to reduce build times; to verify the integrity of resolved packages; and to support build reproducibility across environments and time. Despite these beneficial features, developers often struggle with their maintenance, usage, and interpretation. In this study, we unveil the major challenges related to lockfiles, such that future researchers and engineers can address them. [ ]
While existing studies primarily focus on software supply chain attacks prevention and detection methods, there is a need for a broad overview of attacks and comprehensive risk assessment for software supply chain security. This study conducts a systematic literature review to fill this gap. By analyzing 96 papers published between 2015-2023, we identified 19 distinct SSC attacks, including 6 novel attacks highlighted in recent studies. Additionally, we developed 25 specific security controls and established a precisely mapped taxonomy that transparently links each control to one or more specific attacks. [ ]
[Achieving] reproducibility at scale remains difficult, especially in Java, due to a range of non-deterministic factors and caveats in the build process. In this work, we focus on reproducibility in Java-based software, archetypal of enterprise applications. We introduce a conceptual framework for reproducible builds, we analyze a large dataset from Reproducible Central, and we develop a novel taxonomy of six root causes of unreproducibility. [ ]
Robert Stupp made a number of useful changes, fixing and reorganising the Groovy / Kotlin pages [][][] as well adding a note about potential non-deterministic behaviour to the JVM page [].
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
Akvorado collects sFlow and IPFIX flows over UDP. Because UDP does
not retransmit lost packets, it needs to process them quickly. Akvorado runs
several workers listening to the same port. The kernel should load-balance
received packets fairly between these workers. However, this does not work as
expected. A couple of workers exhibit high packet loss:
Options for load-balancing
There are three methods to load-balance UDP packets across workers:
One worker receives the packets and dispatches them to the other workers.
All workers share the same socket.
Each worker has its own socket, listening to the same port, with the
SO_REUSEPORT socket option.
SO_REUSEPORT option
Tom Hebert added the SO_REUSEPORT socket option in Linux 3.9. The
cover letter for his patch series explains why this new option is better than
the two existing ones from a performance point of view:
SO_REUSEPORT allows multiple listener sockets to be bound to the same port.
[ ] Received packets are distributed to multiple sockets bound to the same
port using a 4-tuple hash.
The motivating case for SO_RESUSEPORT in TCP would be something like a web
server binding to port 80 running with multiple threads, where each thread
might have it s own listener socket. This could be done as an alternative to
other models:
have one listener thread which dispatches completed connections to workers, or
accept on a single listener socket from multiple threads.
In case #1, the listener thread can easily become the bottleneck with high
connection turn-over rate. In case #2, the proportion of connections accepted
per thread tends to be uneven under high connection load. [ ] We have seen the
disproportion to be as high as 3:1 ratio between thread accepting most
connections and the one accepting the fewest. With SO_REUSEPORT the
distribution is uniform.
The motivating case for SO_REUSEPORT in UDP would be something like a DNS
server. An alternative would be to receive on the same socket from multiple
threads. As in the case of TCP, the load across these threads tends to be
disproportionate and we also see a lot of contection on the socket lock.
Akvorado uses the SO_REUSEPORT option to dispatch the packets across the
workers. However, because the distribution uses a 4-tuple hash, a single socket
handles all the flows from one exporter.
SO_ATTACH_REUSEPORT_EBPF option
In Linux 4.5, Craig Gallek added the SO_ATTACH_REUSEPORT_EBPF
option to attach an eBPF program to select the target UDP socket. In Linux 4.6,
he extended it to support TCP. The socket(7) manual page
documents this mechanism:1
The BPF program must return an index between 0 and N-1 representing the socket
which should receive the packet (where N is the number of sockets in the
group). If the BPF program returns an invalid index, socket selection will
fall back to the plain SO_REUSEPORT mechanism.
In Linux 4.19, Martin KaFai Lau added the
BPF_PROG_TYPE_SK_REUSEPORT program type. Such an eBPF program
selects the socket from a BPF_MAP_TYPE_REUSEPORT_ARRAY map instead. This new
approach is more reliable when switching target sockets from one instance to
another for example, when upgrading, a new instance can add its sockets and
remove the old ones.
Load-balancing with eBPF and Go
Altering the load-balancing algorithm for a group of sockets requires two steps:
eBPF program in C
A simple load-balancing algorithm is to randomly choose the destination socket.
The kernel provides the bpf_get_prandom_u32() helper function to get a
pseudo-random number.
In , we declare a volatile constant for the number of sockets in the group. We
will initialize this constant before loading the eBPF program into the kernel.
In , we define the socket map. We will populate it with the socket file
descriptors. In , we randomly select the index of the target socket.3
In , we invoke the bpf_sk_select_reuseport() helper to record our decision.
Finally, in , we accept the packet.
Header files
If you compile the C source with clang, you get errors due to missing headers.
The recommended way to solve this is to generate a vmlinux.h file with
bpftool:
For my 6.17 kernel, the generated vmlinux.h is quite big: 2.7 MiB. Moreover,
bpf/bpf_helpers.h is shipped with libbpf. This adds another dependency for
users. As the eBPF program is quite small, I prefer to put the strict minimum in
vmlinux.h by cherry-picking the definitions I need.
Compilation
The eBPF Library for Go ships bpf2go, a tool to compile eBPF programs and
to generate some scaffolding code. We create a gen.go file with the following
content:
packagemain//go:generate go tool bpf2go -tags linux reuseport reuseport_kern.c
After running go generate ./..., we can inspect the resulting objects with
readelf and llvm-objdump:
$ readelf-Sreuseport_bpfeb.o
There are 14 section headers, starting at offset 0x840: [Nr] Name Type Address Offset[ ] [ 3] sk_reuseport PROGBITS 0000000000000000 00000040 [ 6] .maps PROGBITS 0000000000000000 000000c8 [ 7] license PROGBITS 0000000000000000 000000e8[ ]$ llvm-objdump-Sreuseport_bpfeb.o
reuseport_bpfeb.o: file format elf64-bpfDisassembly of section sk_reuseport:0000000000000000 <reuseport_balance_prog>:; 0: bf 61 00 00 00 00 00 00 r6 = r1; __u32 index = bpf_get_prandom_u32() % num_sockets; 1: 85 00 00 00 00 00 00 07 call 0x7[ ]
Usage from Go
Let s set up 10 workers listening to the same port.5 Each socket
enables the SO_REUSEPORT option before binding:6
The second step is to load the eBPF program, initialize the num_sockets
variable, populate the socket map, and attach the program to the first
socket.7
// Load the eBPF collection.spec,err:=loadReuseport()iferr!=nilt.Fatalf("loadVariables() error:\n%+v",err)// Set "num_sockets" global variable to the number of file descriptors we will registeriferr:=spec.Variables["num_sockets"].Set(uint32(len(fds)));err!=nilt.Fatalf("NumSockets.Set() error:\n%+v",err)// Load the map and the program into the kernel.varobjsreuseportObjectsiferr:=spec.LoadAndAssign(&objs,nil);err!=nilt.Fatalf("loadReuseportObjects() error:\n%+v",err)t.Cleanup(func()objs.Close() )// Assign the file descriptors to the socket map.forworker,fd:=rangefdsiferr:=objs.reuseportMaps.SocketMap.Put(uint32(worker),uint64(fd));err!=nilt.Fatalf("SocketMap.Put() error:\n%+v",err)// Attach the eBPF program to the first socket.socketFD:=int(fds[0])progFD:=objs.reuseportPrograms.ReuseportBalanceProg.FD()iferr:=unix.SetsockoptInt(socketFD,unix.SOL_SOCKET,unix.SO_ATTACH_REUSEPORT_EBPF,progFD);err!=nilt.Fatalf("SetsockoptInt() error:\n%+v",err)
We are now ready to process incoming packets. Each worker is a Go routine
incrementing a counter for each received packet:8
Graceful restart
You can also use SO_ATTACH_REUSEPORT_EBPF to gracefully restart an
application. A new instance of the application binds to the same address and
prepare its own version of the socket map. Once it attaches the eBPF program to
the first socket, the kernel steers incoming packets to this new instance. The
old instance needs to drain the already received packets before shutting down.
To check we are not losing any packet, we spawn a Go routine to send as many
packets as possible:
Then, while the Go routine runs, we start the second set of workers. Once they
are running, they start receiving packets. If we gracefully stop the initial set
of workers, not a single packet is lost!9
Unfortunately, gracefully shutting down a UDP socket is not trivial in Go.10
Previously, we were terminating workers by closing their sockets. However, if we
close them too soon, the application loses packets that were assigned to them
but not yet processed. Before stopping, a worker needs to call conn.Read()
until there are no more packets. A solution is to set a deadline for
conn.Read() and check if we should stop the Go routine when the deadline is
exceeded:
With TCP, this aspect is simpler: after enabling the net.ipv4.tcp_migrate_req
sysctl, the kernel automatically migrates waiting connections to a random socket
in the same group. Alternatively, eBPF can also control this
migration. Both features are available since Linux 5.14.
Addendum
After implementing this strategy in Akvorado, all workers now drop packets!
The root cause is the default limit of 32 records for Kafka batch sizes. This
limit is too low because the brokers have a large overhead when handling each
batch: they need to ensure they persist correctly before acknowledging
them. Increasing the limit to 4096 records fixes this issue.
While load-balancing incoming flows with eBPF remains useful, it did not solve
the main issue. At least the even distribution of dropped packets helped
identify the real bottleneck.
The current version of the manual page is incomplete and does not
cover the evolution introduced in Linux 4.19. There is a pending patch
about this.
Rust is another option. However, the program we use is so trivial that
it does not make sense to use Rust.
As bpf_get_prandom_u32() returns a pseudo-random 32-bit unsigned
value, this method exhibits a very slight bias towards the first indexes.
This is unlikely to be worth fixing.
Some examples include <linux/bpf.h> instead of "vmlinux.h". This
makes your eBPF program dependent on the installed kernel headers.
listenAddr is initially set to 127.0.0.1:0 to allocate a
random port. After the first iteration, it is updated with the allocated
port.
In C, we would poll() both the socket and a pipe used to signal for
shutdown. When the second condition is triggered, we drain the socket by
executing a series of non-blocking read() until we get EWOULDBLOCK.
Debusine is a tool designed for Debian developers and Operating
System developers in general. This post describes how Debusine stores
and manages files.
Debusine has been designed to run a network of workers that can
perform various tasks that consume and produce artifacts .
The artifact itself is a collection of files structured into an
ontology of artifact types.
This generic architecture should be suited to many sorts of build & CI
problems.
We have implemented artifacts to support building a Debian-like
distribution, but the foundations of Debusine aim to be more general
than that.
For example a package build task takes a debian:source-package as
input and produces some debian:binary-packages and a
debian:package-build-log as output.
This generalized approach is quite different to traditional Debian APT
archive implementations, which typically required having the archive
contents on the filesystem.
Traditionally, most Debian distribution management tasks happen within
bespoke applications that cannot share much common infrastructure.
File Stores
Debusine s files themselves are stored by the File
Store layer.
There can be multiple file stores configured, with different policies.
Local storage is useful as the initial destination for uploads to
Debusine, but it has to be backed up manually and might not scale to
sufficiently large volumes of data.
Remote storage such as S3 is also available.
It is possible to serve a file from any store, with policies for which
one to prefer for downloads and uploads.
Administrators can set policies for which file stores to use at the
scope level, as well as policies for populating and draining stores
of files.
Artifacts
As mentioned above, files are collected into Artifacts. They combine:
a set of files with names (including potentially parent directories)
key-value data in a schema specified by the category and stored as a
JSON-encoded dictionary.
Within the stores, files are content-addressed: a file with a given
SHA-256 digest is only stored once in any given store, and may be
retrieved by that digest.
When a new artifact is created, its files are uploaded to Debusine as
needed.
Some of the files may already be present in the Debusine instance.
In that case, if the file is already part of the artifact s workspace,
then the client will not need to re-upload the file.
But if not, it must be reuploaded to avoid users obtaining unauthorized
access to existing file contents in another private workspace or
multi-tenant scope.
Because the content-addressing makes storing duplicates cheap, it s
common to have artifacts that overlap files.
For example a debian:upload will contain some of the same files as
the related debian:source-package as well as the .changes file.
Looking at the debusine.debian.net instance that we run, we can see
a content-addressing savings of 629 GiB across our (currently) 2 TiB
file store.
This is somewhat inflated by the Debian Archive import, that did not
need to bother to share artifacts between suites.
But it still shows reasonable real-world savings.
APT Repository Representation
Unlike a traditional Debian APT repository management tool, the source
package and binary packages are not stored directly in the pool of an
APT repository on disk on the debusine server.
Instead we abstract the repository into a debian:suitecollection within the Debusine database.
The collection contains the artifacts that make up the APT repository.
To ensure that it can be safely represented as a valid URL structure (or
files on disk) the suite collection maintains an index of the pool
filenames of its artifacts.
Suite collections can combine into a debian:archive collection that
shares a common file pool.
Debusine collections can keep an historical record of when things were
added and removed. This, combined with the database-backed
collection-driven repository representation makes it very easy to
provide APT-consumable snapshot views to every point in a repository s
history.
Expiry
While a published distribution probably wants to keep the full history
of all its package builds, we don t need to retain all of the output of
all QA tasks that were run. Artifacts can have an expiration delay
or inherit one from their workspace.
Once this delay has expired, artifacts which are not being held in any
collection are eligible to be automatically cleaned up.
QA work that is done in a workspace that has automatic artifact expiry,
and isn t publishing the results to an APT suite, will safely
automatically expire.
Daily Vacuum
A daily vacuum task handles all of the file periodic maintenance for
file stores.
It does some cleanup of working areas, a scan for unreferenced & missing
files, and enforces file store policies.
The policy work could be copying files for backup or moving files
between stores to keep them within size limits (e.g. from a local upload
store into a general cloud store).
In Conclusion
Debusine provides abstractions for low-level file storage and object
collections.
This allows storage to be scalable beyond a single filesystem and highly
available.
Using content-addressed storage minimizes data duplication within a
Debusine instance.
For Debian distributions, storing the archive metadata entirely in a
database made providing built-in snapshot support easy in Debusine.
## 0.23 2025-12-20
commit be15aa25dea40aea66a8534143fb81b29d2e6c08
Author: C.J. Collier
Date: Sat Dec 20 22:40:44 2025 +0000
Fixes C-level test infrastructure and adds more test cases for upb_to_sv conversions.
- **Makefile.PL:**
- Allow extra_src in c_test_config.json to be an array.
- Add ASan flags to CCFLAGS and LDDLFLAGS for better debugging.
- Corrected echo newlines in test_c target.
- **c_test_config.json:**
- Added missing type test files to deps and extra_src for convert/sv_to_upb and convert/upb_to_sv test runners.
- **t/c/convert/upb_to_sv.c:**
- Fixed a double free of test_pool .
- Added missing includes for type test headers.
- Updated test plan counts.
- **t/c/convert/sv_to_upb.c:**
- Added missing includes for type test headers.
- Updated test plan counts.
- Corrected Perl interpreter initialization.
- **t/c/convert/types/**:
- Added missing test_util.h include in new type test headers.
- Completed the set of upb_to_sv test cases for all scalar types by adding optional and repeated tests for sfixed32 , sfixed64 , sint32 , and sint64 , and adding repeated tests to the remaining scalar type files.
- **Documentation:**
- Updated 01-xs-testing.md with more debugging tips, including ASan usage and checking for double frees and typos.
- Updated xs_learnings.md with details from the recent segfault.
- Updated llm-plan-execution-instructions.md to emphasize debugging steps.
## 0.22 2025-12-19
commit 2c171d9a5027e0150eae629729c9104e7f6b9d2b
Author: C.J. Collier
Date: Fri Dec 19 23:41:02 2025 +0000
feat(perl,testing): Initialize C test framework and build system
This commit sets up the foundation for the C-level tests and the build system for the Perl Protobuf module:
1. **Makefile.PL Enhancements:**
* Integrates Devel::PPPort to generate ppport.h for better portability.
* Object files now retain their path structure (e.g., xs/convert/sv_to_upb.o ) instead of being flattened, improving build clarity.
* The MY::postamble is significantly revamped to dynamically generate build rules for all C tests located in t/c/ based on the t/c/c_test_config.json file.
* C tests are linked against libprotobuf_common.a and use ExtUtils::Embed flags.
* Added JSON::MaybeXS to PREREQ_PM .
* The test target now also depends on the test_c target.
2. **C Test Infrastructure ( t/c/ ):
* Introduced t/c/c_test_config.json to configure individual C test builds, specifying dependencies and extra source files.
* Created t/c/convert/test_util.c and .h for shared test functions like loading descriptors.
* Initial t/c/convert/upb_to_sv.c and t/c/convert/sv_to_upb.c test runners.
* Basic t/c/integration/030_protobuf_coro.c for Coro safety testing on core utils using libcoro .
* Basic t/c/integration/035_croak_test.c for testing exception handling.
* Basic t/c/integration/050_convert.c for integration testing conversions.
3. **Test Proto:** Updated t/data/test.proto with more field types for conversion testing and regenerated test_descriptor.bin .
4. **XS Test Harness ( t/c/upb-perl-test.h ):** Added like_n macro for length-aware regex matching.
5. **Documentation:** Updated architecture and plan documents to reflect the C test structure.
6. **ERRSV Testing:** Note that the C tests ( t/c/ ) will primarily check *if* a croak occurs (i.e., that the exception path is taken), but will not assert on the string content of ERRSV . Reliably testing $@ content requires the full Perl test environment with Test::More , which will be done in the .t files when testing the Perl API.
This provides a solid base for developing and testing the XS and C components of the module.
## 0.21 2025-12-18
commit a8b6b6100b2cf29c6df1358adddb291537d979bc
Author: C.J. Collier
Date: Thu Dec 18 04:20:47 2025 +0000
test(C): Add integration tests for Milestone 2 components
- Created t/c/integration/030_protobuf.c to test interactions
between obj_cache, arena, and utils.
- Added this test to t/c/c_test_config.json.
- Verified that all C tests for Milestones 2 and 3 pass,
including the libcoro-based stress test.
## 0.20 2025-12-18
commit 0fcad68680b1f700a83972a7c1c48bf3a6958695
Author: C.J. Collier
Date: Thu Dec 18 04:14:04 2025 +0000
docs(plan): Add guideline review reminders to milestones
- Added a "[ ] REFRESH: Review all documents in @perl/doc/guidelines/**"
checklist item to the start of each component implementation
milestone (C and Perl layers).
- This excludes Integration Test milestones.
## 0.19 2025-12-18
commit 987126c4b09fcdf06967a98fa3adb63d7de59a34
Author: C.J. Collier
Date: Thu Dec 18 04:05:53 2025 +0000
docs(plan): Add C-level and Perl-level Coro tests to milestones
- Added checklist items for libcoro -based C tests
(e.g., t/c/integration/050_convert_coro.c ) to all C layer
integration milestones (050 through 220).
- Updated 030_Integration_Protobuf.md to standardise checklist
items for the existing 030_protobuf_coro.c test.
- Removed the single xt/author/coro-safe.t item from
010_Build.md .
- Added checklist items for Perl-level Coro tests
(e.g., xt/coro/240_arena.t ) to each Perl layer
integration milestone (240 through 400).
- Created perl/t/c/c_test_config.json to manage C test
configurations externally.
- Updated perl/doc/architecture/testing/01-xs-testing.md to describe
both C-level libcoro and Perl-level Coro testing strategies.
## 0.18 2025-12-18
commit 6095a5a610401a6035a81429d0ccb9884d53687b
Author: C.J. Collier
Date: Thu Dec 18 02:34:31 2025 +0000
added coro testing to c layer milestones
## 0.17 2025-12-18
commit cc0aae78b1f7f675fc8a1e99aa876c0764ea1cce
Author: C.J. Collier
Date: Thu Dec 18 02:26:59 2025 +0000
docs(plan): Refine test coverage checklist items for SMARTness
- Updated the "Tests provide full coverage" checklist items in
C layer plan files (020, 040, 060, 080, 100, 120, 140, 160, 180, 200)
to explicitly mention testing all public functions in the
corresponding header files.
- Expanded placeholder checklists in 140, 160, 180, 200.
- Updated the "Tests provide full coverage" and "Add coverage checks"
checklist items in Perl layer plan files (230, 250, 270, 290, 310, 330,
350, 370, 390) to be more specific about the scope of testing
and the use of Test::TestCoverage .
- Expanded Well-Known Types milestone (350) to detail each type.
## 0.16 2025-12-18
commit e4b601f14e3817a17b0f4a38698d981dd4cb2818
Author: C.J. Collier
Date: Thu Dec 18 02:07:35 2025 +0000
docs(plan): Full refactoring of C and Perl plan files
- Split both ProtobufPlan-C.md and ProtobufPlan-Perl.md into
per-milestone files under the perl/doc/plan/ directory.
- Introduced Integration Test milestones after each component
milestone in both C and Perl plans.
- Numbered milestone files sequentially (e.g., 010_Build.md,
230_Perl_Arena.md).
- Updated main ProtobufPlan-C.md and ProtobufPlan-Perl.md to
act as Tables of Contents.
- Ensured consistent naming for integration test files
(e.g., t/c/integration/030_protobuf.c , t/integration/260_descriptor_pool.t ).
- Added architecture review steps to the end of all milestones.
- Moved Coro safety test to C layer Milestone 1.
- Updated Makefile.PL to support new test structure and added Coro.
- Moved and split t/c/convert.c into t/c/convert/*.c.
- Moved other t/c/*.c tests into t/c/protobuf/*.c.
- Deleted old t/c/convert.c.
## 0.15 2025-12-17
commit 649cbacf03abb5e7293e3038bb451c0406e9d0ce
Author: C.J. Collier
Date: Wed Dec 17 23:51:22 2025 +0000
docs(plan): Refactor and reset ProtobufPlan.md
- Split the plan into ProtobufPlan-C.md and ProtobufPlan-Perl.md.
- Reorganized milestones to clearly separate C layer and Perl layer development.
- Added more granular checkboxes for each component:
- C Layer: Create test, Test coverage, Implement, Tests pass.
- Perl Layer: Create test, Test coverage, Implement Module/XS, Tests pass, C-Layer adjustments.
- Reset all checkboxes to [ ] to prepare for a full audit.
- Updated status in architecture/api and architecture/core documents to "Not Started".
feat(obj_cache): Add unregister function and enhance tests
- Added protobuf_unregister_object to xs/protobuf/obj_cache.c .
- Updated xs/protobuf/obj_cache.h with the new function declaration.
- Expanded tests in t/c/protobuf_obj_cache.c to cover unregistering,
overwriting keys, and unregistering non-existent keys.
- Corrected the test plan count in t/c/protobuf_obj_cache.c to 17.
## 0.14 2025-12-17
commit 40b6ad14ca32cf16958d490bb575962f88d868a1
Author: C.J. Collier
Date: Wed Dec 17 23:18:27 2025 +0000
feat(arena): Complete C layer for Arena wrapper
This commit finalizes the C-level implementation for the Protobuf::Arena wrapper.
- Adds PerlUpb_Arena_Destroy for proper cleanup from Perl's DEMOLISH.
- Enhances error checking in PerlUpb_Arena_Get .
- Expands C-level tests in t/c/protobuf_arena.c to cover memory allocation
on the arena and lifecycle through PerlUpb_Arena_Destroy .
- Corrects embedded Perl initialization in the C test.
docs(plan): Refactor ProtobufPlan.md
- Restructures the development plan to clearly separate "C Layer" and
"Perl Layer" tasks within each milestone.
- This aligns the plan with the "C-First Implementation Strategy" and improves progress tracking.
## 0.13 2025-12-17
commit c1e566c25f62d0ae9f195a6df43b895682652c71
Author: C.J. Collier
Date: Wed Dec 17 22:00:40 2025 +0000
refactor(perl): Rename C tests and enhance Makefile.PL
- Renamed test files in t/c/ to better match the xs module structure:
- 01-cache.c -> protobuf_obj_cache.c
- 02-arena.c -> protobuf_arena.c
- 03-utils.c -> protobuf_utils.c
- 04-convert.c -> convert.c
- load_test.c -> upb_descriptor_load.c
- Updated perl/Makefile.PL to reflect the new test names in MY::postamble 's $c_test_config .
- Refactored the $c_test_config generation in Makefile.PL to reduce repetition by using a default flags hash and common dependencies array.
- Added a fail() macro to perl/t/c/upb-perl-test.h for consistency.
- Modified t/c/upb_descriptor_load.c to use the t/c/upb-perl-test.h macros, making its output consistent with other C tests.
- Added a skeleton for t/c/convert.c to test the conversion functions.
- Updated documentation in ProtobufPlan.md and architecture/testing/01-xs-testing.md to reflect new test names.
## 0.12 2025-12-17
commit d8cb5dd415c6c129e71cd452f78e29de398a82c9
Author: C.J. Collier
Date: Wed Dec 17 20:47:38 2025 +0000
feat(perl): Refactor XS code into subdirectories
This commit reorganizes the C code in the perl/xs/ directory into subdirectories, mirroring the structure of the Python UPB extension. This enhances modularity and maintainability.
- Created subdirectories for each major component: convert , descriptor , descriptor_containers , descriptor_pool , extension_dict , map , message , protobuf , repeated , and unknown_fields .
- Created skeleton .h and .c files within each subdirectory to house the component-specific logic.
- Updated top-level component headers (e.g., perl/xs/descriptor.h ) to include the new sub-headers.
- Updated top-level component source files (e.g., perl/xs/descriptor.c ) to include their main header and added stub initialization functions (e.g., PerlUpb_InitDescriptor ).
- Moved code from the original perl/xs/protobuf.c to new files in perl/xs/protobuf/ (arena, obj_cache, utils).
- Moved code from the original perl/xs/convert.c to new files in perl/xs/convert/ (upb_to_sv, sv_to_upb).
- Updated perl/Makefile.PL to use a glob ( xs/*/*.c ) to find the new C source files in the subdirectories.
- Added perl/doc/architecture/core/07-xs-file-organization.md to document the new structure.
- Updated perl/doc/ProtobufPlan.md and other architecture documents to reference the new organization.
- Corrected self-referential includes in the newly created .c files.
This restructuring provides a solid foundation for further development and makes it easier to port logic from the Python implementation.
## 0.11 2025-12-17
commit cdedcd13ded4511b0464f5d3bdd72ce6d34e73fc
Author: C.J. Collier
Date: Wed Dec 17 19:57:52 2025 +0000
feat(perl): Implement C-first testing and core XS infrastructure
This commit introduces a significant refactoring of the Perl XS extension, adopting a C-first development approach to ensure a robust foundation.
Key changes include:
- **C-Level Testing Framework:** Established a C-level testing system in t/c/ with a dedicated Makefile, using an embedded Perl interpreter. Initial tests cover the object cache ( 01-cache.c ), arena wrapper ( 02-arena.c ), and utility functions ( 03-utils.c ).
- **Core XS Infrastructure:**
- Implemented a global object cache ( xs/protobuf.c ) to manage Perl wrappers for UPB objects, using weak references.
- Created an upb_Arena wrapper ( xs/protobuf.c ).
- Consolidated common XS helper functions into xs/protobuf.h and xs/protobuf.c .
- **Makefile.PL Enhancements:** Updated to support building and linking C tests, incorporating flags from ExtUtils::Embed , and handling both .c and .cc source files.
- **XS File Reorganization:** Restructured XS files to mirror the Python UPB extension's layout (e.g., message.c , descriptor.c ). Removed older, monolithic .xs files.
- **Typemap Expansion:** Added extensive typemap entries in perl/typemap to handle conversions between Perl objects and various const upb_*Def* pointers.
- **Descriptor Tests:** Added a new test suite t/02-descriptor.t to validate descriptor loading and accessor methods.
- **Documentation:** Updated development plans and guidelines ( ProtobufPlan.md , xs_learnings.md , etc.) to reflect the C-first strategy, new testing methods, and lessons learned.
- **Build Cleanup:** Removed ppport.h from .gitignore as it's no longer used, due to -DPERL_NO_PPPORT being set in Makefile.PL .
This C-first approach allows for more isolated and reliable testing of the core logic interacting with the UPB library before higher-level Perl APIs are built upon it.
## 0.10 2025-12-17
commit 1ef20ade24603573905cb0376670945f1ab5d829
Author: C.J. Collier
Date: Wed Dec 17 07:08:29 2025 +0000
feat(perl): Implement C-level tests and core XS utils
This commit introduces a C-level testing framework for the XS layer and implements key components:
1. **C-Level Tests ( t/c/ )**:
* Added t/c/Makefile to build standalone C tests.
* Created t/c/upb-perl-test.h with macros for TAP-compliant C tests ( plan , ok , is , is_string , diag ).
* Implemented t/c/01-cache.c to test the object cache.
* Implemented t/c/02-arena.c to test Protobuf::Arena wrappers.
* Implemented t/c/03-utils.c to test string utility functions.
* Corrected include paths and diagnostic messages in C tests.
2. **XS Object Cache ( xs/protobuf.c )**:
* Switched to using stringified pointers ( %p ) as hash keys for stability.
* Fixed a critical double-free bug in PerlUpb_ObjCache_Delete by removing an extra SvREFCNT_dec on the lookup key.
3. **XS Arena Wrapper ( xs/protobuf.c )**:
* Corrected PerlUpb_Arena_New to use newSVrv and PTR2IV for opaque object wrapping.
* Corrected PerlUpb_Arena_Get to safely unwrap the arena pointer.
4. **Makefile.PL ( perl/Makefile.PL )**:
* Added -Ixs to INC to allow C tests to find t/c/upb-perl-test.h and xs/protobuf.h .
* Added LIBS to link libprotobuf_common.a into the main Protobuf.so .
* Added C test targets 01-cache , 02-arena , 03-utils to the test config in MY::postamble .
5. **Protobuf.pm ( perl/lib/Protobuf.pm )**:
* Added use XSLoader; to load the compiled XS code.
6. **New files xs/util.h **:
* Added initial type conversion function.
These changes establish a foundation for testing the C-level interface with UPB and fix crucial bugs in the object cache implementation.
## 0.09 2025-12-17
commit 07d61652b032b32790ca2d3848243f9d75ea98f4
Author: C.J. Collier
Date: Wed Dec 17 04:53:34 2025 +0000
feat(perl): Build system and C cache test for Perl XS
This commit introduces the foundational pieces for the Perl XS implementation, focusing on the build system and a C-level test for the object cache.
- **Makefile.PL:**
- Refactored C test compilation rules in MY::postamble to use a hash ( $c_test_config ) for better organization and test-specific flags.
- Integrated ExtUtils::Embed to provide necessary compiler and linker flags for embedding the Perl interpreter, specifically for the t/c/01-cache.c test.
- Correctly constructs the path to the versioned Perl library ( libperl.so.X.Y.Z ) using $Config archlib and $Config libperl to ensure portability.
- Removed VERSION_FROM and ABSTRACT_FROM to avoid dependency on .pm files for now.
- **C Cache Test (t/c/01-cache.c):**
- Added a C test to exercise the object cache functions implemented in xs/protobuf.c .
- Includes tests for adding, getting, deleting, and weak reference behavior.
- **XS Cache Implementation (xs/protobuf.c, xs/protobuf.h):**
- Implemented PerlUpb_ObjCache_Init , PerlUpb_ObjCache_Add , PerlUpb_ObjCache_Get , PerlUpb_ObjCache_Delete , and PerlUpb_ObjCache_Destroy .
- Uses a Perl hash ( HV* ) for the cache.
- Keys are string representations of the C pointers, created using snprintf with "%llx" .
- Values are weak references ( sv_rvweaken ) to the Perl objects ( SV* ).
- PerlUpb_ObjCache_Get now correctly returns an incremented reference to the original SV, not a copy.
- PerlUpb_ObjCache_Destroy now clears the hash before decrementing its refcount.
- **t/c/upb-perl-test.h:**
- Updated is_sv to perform direct pointer comparison ( got == expected ).
- **Minor:** Added util.h (currently empty), updated typemap .
These changes establish a working C-level test environment for the XS components.
## 0.08 2025-12-17
commit d131fd22ea3ed8158acb9b0b1fe6efd856dc380e
Author: C.J. Collier
Date: Wed Dec 17 02:57:48 2025 +0000
feat(perl): Update docs and core XS files
- Explicitly add TDD cycle to ProtobufPlan.md.
- Clarify mirroring of Python implementation in upb-interfacing.md for both C and Perl layers.
- Branch and adapt python/protobuf.h and python/protobuf.c to perl/xs/protobuf.h and perl/xs/protobuf.c, including the object cache implementation. Removed old cache.* files.
- Create initial C test for the object cache in t/c/01-cache.c.
## 0.07 2025-12-17
commit 56fd6862732c423736a2f9a9fb1a2816fc59e9b0
Author: C.J. Collier
Date: Wed Dec 17 01:09:18 2025 +0000
feat(perl): Align Perl UPB architecture docs with Python
Updates the Perl Protobuf architecture documents to more closely align with the design and implementation strategies used in the Python UPB extension.
Key changes:
- **Object Caching:** Mandates a global, per-interpreter cache using weak references for all UPB-derived objects, mirroring Python's PyUpb_ObjCache .
- **Descriptor Containers:** Introduces a new document outlining the plan to use generic XS container types (Sequence, ByNameMap, ByNumberMap) with vtables to handle collections of descriptors, similar to Python's descriptor_containers.c .
- **Testing:** Adds a note to the testing strategy to port relevant test cases from the Python implementation to ensure feature parity.
## 0.06 2025-12-17
commit 6009ce6ab64eccce5c48729128e5adf3ef98e9ae
Author: C.J. Collier
Date: Wed Dec 17 00:28:20 2025 +0000
feat(perl): Implement object caching and fix build
This commit introduces several key improvements to the Perl XS build system and core functionality:
1. **Object Caching:**
* Introduces xs/protobuf.c and xs/protobuf.h to implement a caching mechanism ( protobuf_c_to_perl_obj ) for wrapping UPB C pointers into Perl objects. This uses a hash and weak references to ensure object identity and prevent memory leaks.
* Updates the typemap to use protobuf_c_to_perl_obj for upb_MessageDef * output, ensuring descriptor objects are cached.
* Corrected sv_weaken to the correct sv_rvweaken function.
2. **Makefile.PL Enhancements:**
* Switched to using the Bazel-generated UPB descriptor sources from bazel-bin/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/ .
* Updated INC paths to correctly locate the generated headers.
* Refactored MY::dynamic_lib to ensure the static library libprotobuf_common.a is correctly linked into each generated .so module, resolving undefined symbol errors.
* Overrode MY::test to use prove -b -j$(nproc) t/*.t xt/*.t for running tests.
* Cleaned up LIBS and LDDLFLAGS usage.
3. **Documentation:**
* Updated ProtobufPlan.md to reflect the current status and design decisions.
* Reorganized architecture documents into subdirectories.
* Added object-caching.md and c-perl-interface.md .
* Updated llm-guidance.md with notes on upb/upb.h and sv_rvweaken .
4. **Testing:**
* Fixed xt/03-moo_immutable.t to skip tests if no Moo modules are found.
This resolves the build issues and makes the core test suite pass.
## 0.05 2025-12-16
commit 177d2f3b2608b9d9c415994e076a77d8560423b8
Author: C.J. Collier
Date: Tue Dec 16 19:51:36 2025 +0000
Refactor: Rename namespace to Protobuf, build system and doc updates
This commit refactors the primary namespace from ProtoBuf to Protobuf
to align with the style guide. This involves renaming files, directories,
and updating package names within all Perl and XS files.
**Namespace Changes:**
* Renamed perl/lib/ProtoBuf to perl/lib/Protobuf .
* Moved and updated ProtoBuf.pm to Protobuf.pm .
* Moved and updated ProtoBuf::Descriptor to Protobuf::Descriptor (.pm & .xs).
* Removed other ProtoBuf::* stubs (Arena, DescriptorPool, Message).
* Updated MODULE and PACKAGE in Descriptor.xs .
* Updated NAME , *_FROM in perl/Makefile.PL .
* Replaced ProtoBuf with Protobuf throughout perl/typemap .
* Updated namespaces in test files t/01-load-protobuf-descriptor.t and t/02-descriptor.t .
* Updated namespaces in all documentation files under perl/doc/ .
* Updated paths in perl/.gitignore .
**Build System Enhancements (Makefile.PL):**
* Included xs/*.c files in the common object files list.
* Added -I. to the INC paths.
* Switched from MYEXTLIB to LIBS => ['-L$(CURDIR) -lprotobuf_common'] for linking.
* Removed custom keys passed to WriteMakefile for postamble.
* MY::postamble now sources variables directly from the main script scope.
* Added all :: $ common_lib dependency in MY::postamble .
* Added t/c/load_test.c compilation rule in MY::postamble .
* Updated clean target to include blib .
* Added more modules to TEST_REQUIRES .
* Removed the explicit PM and XS keys from WriteMakefile , relying on XSMULTI => 1 .
**New Files:**
* perl/lib/Protobuf.pm
* perl/lib/Protobuf/Descriptor.pm
* perl/lib/Protobuf/Descriptor.xs
* perl/t/01-load-protobuf-descriptor.t
* perl/t/02-descriptor.t
* perl/t/c/load_test.c : Standalone C test for UPB.
* perl/xs/types.c & perl/xs/types.h : For Perl/C type conversions.
* perl/doc/architecture/upb-interfacing.md
* perl/xt/03-moo_immutable.t : Test for Moo immutability.
**Deletions:**
* Old test files: t/00_load.t , t/01_basic.t , t/02_serialize.t , t/03_message.t , t/04_descriptor_pool.t , t/05_arena.t , t/05_message.t .
* Removed lib/ProtoBuf.xs as it's not needed with XSMULTI .
**Other:**
* Updated test_descriptor.bin (binary change).
* Significant content updates to markdown documentation files in perl/doc/architecture and perl/doc/internal reflecting the new architecture and learnings.
## 0.04 2025-12-14
commit 92de5d482c8deb9af228f4b5ce31715d3664d6ee
Author: C.J. Collier
Date: Sun Dec 14 21:28:19 2025 +0000
feat(perl): Implement Message object creation and fix lifecycles
This commit introduces the basic structure for ProtoBuf::Message object
creation, linking it with ProtoBuf::Descriptor and ProtoBuf::DescriptorPool ,
and crucially resolves a SEGV by fixing object lifecycle management.
Key Changes:
1. ** ProtoBuf::Descriptor :** Added _pool attribute to hold a strong
reference to the parent ProtoBuf::DescriptorPool . This is essential to
prevent the pool and its C upb_DefPool from being garbage collected
while a descriptor is still in use.
2. ** ProtoBuf::DescriptorPool :**
* find_message_by_name : Now passes the $self (the pool object) to the
ProtoBuf::Descriptor constructor to establish the lifecycle link.
* XSUB pb_dp_find_message_by_name : Updated to accept the pool SV* and
store it in the descriptor's _pool attribute.
* XSUB _load_serialized_descriptor_set : Renamed to avoid clashing with the
Perl method name. The Perl wrapper now correctly calls this internal XSUB.
* DEMOLISH : Made safer by checking for attribute existence.
3. ** ProtoBuf::Message :**
* Implemented using Moo with lazy builders for _upb_arena and
_upb_message .
* _descriptor is a required argument to new() .
* XS functions added for creating the arena ( pb_msg_create_arena ) and
the upb_Message ( pb_msg_create_upb_message ).
* pb_msg_create_upb_message now extracts the upb_MessageDef* from the
descriptor and uses upb_MessageDef_MiniTable() to get the minitable
for upb_Message_New() .
* DEMOLISH : Added to free the message's arena.
4. ** Makefile.PL :**
* Added -g to CCFLAGS for debugging symbols.
* Added Perl CORE include path to MY::postamble 's base_flags .
5. **Tests:**
* t/04_descriptor_pool.t : Updated to check the structure of the
returned ProtoBuf::Descriptor .
* t/05_message.t : Now uses a descriptor obtained from a real pool to
test ProtoBuf::Message->new() .
6. **Documentation:**
* Updated ProtobufPlan.md to reflect progress.
* Updated several files in doc/architecture/ to match the current
implementation details, especially regarding arena management and object
lifecycles.
* Added doc/internal/development_cycle.md and doc/internal/xs_learnings.md .
With these changes, the SEGV is resolved, and message objects can be successfully
created from descriptors.
## 0.03 2025-12-14
commit 6537ad23e93680c2385e1b571d84ed8dbe2f68e8
Author: C.J. Collier
Date: Sun Dec 14 20:23:41 2025 +0000
Refactor(perl): Object-Oriented DescriptorPool with Moo
This commit refactors the ProtoBuf::DescriptorPool to be fully object-oriented using Moo, and resolves several issues related to XS, typemaps, and test data.
Key Changes:
1. **Moo Object:** ProtoBuf::DescriptorPool.pm now uses Moo to define the class. The upb_DefPool pointer is stored as a lazy attribute _upb_defpool .
2. **XS Lifecycle:** DescriptorPool.xs now has pb_dp_create_pool called by the Moo builder and pb_dp_free_pool called from DEMOLISH to manage the upb_DefPool lifecycle per object.
3. **Typemap:** The perl/typemap file has been significantly updated to handle the conversion between the ProtoBuf::DescriptorPool Perl object and the upb_DefPool * C pointer. This includes:
* Mapping upb_DefPool * to T_PTR .
* An INPUT section for ProtoBuf::DescriptorPool to extract the pointer from the object's hash, triggering the lazy builder if needed via call_method .
* An OUTPUT section for upb_DefPool * to convert the pointer back to a Perl integer, used by the builder.
4. **Method Renaming:** add_file_descriptor_set_binary is now load_serialized_descriptor_set .
5. **Test Data:**
* Added perl/t/data/test.proto with a sample message and enum.
* Generated perl/t/data/test_descriptor.bin using protoc .
* Removed t/data/ from .gitignore to ensure test data is versioned.
6. **Test Update:** t/04_descriptor_pool.t is updated to use the new OO interface, load the generated descriptor set, and check for message definitions.
7. **Build Fixes:**
* Corrected #include paths in DescriptorPool.xs to be relative to the upb/ directory (e.g., upb/wire/decode.h ).
* Added -I../upb to CCFLAGS in Makefile.PL .
* Reordered INC paths in Makefile.PL to prioritize local headers.
**Note:** While tests now pass in some environments, a SEGV issue persists in make test runs, indicating a potential memory or lifecycle issue within the XS layer that needs further investigation.
## 0.02 2025-12-14
commit 6c9a6f1a5f774dae176beff02219f504ea3a6e07
Author: C.J. Collier
Date: Sun Dec 14 20:13:09 2025 +0000
Fix(perl): Correct UPB build integration and generated file handling
This commit resolves several issues to achieve a successful build of the Perl extension:
1. **Use Bazel Generated Files:** Switched from compiling UPB's stage0 descriptor.upb.c to using the Bazel-generated descriptor.upb.c and descriptor.upb_minitable.c located in bazel-bin/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/ .
2. **Updated Include Paths:** Added the bazel-bin path to INC in WriteMakefile and to base_flags in MY::postamble to ensure the generated headers are found during both XS and static library compilation.
3. **Removed Stage0:** Removed references to UPB_STAGE0_DIR and no longer include headers or source files from upb/reflection/stage0/ .
4. **-fPIC:** Explicitly added -fPIC to CCFLAGS in WriteMakefile and ensured $(CCFLAGS) is used in the custom compilation rules in MY::postamble . This guarantees all object files in the static library are compiled with position-independent code, resolving linker errors when creating the shared objects for the XS modules.
5. **Refined UPB Sources:** Used File::Find to recursively find UPB C sources, excluding /conformance/ and /reflection/stage0/ to avoid conflicts and unnecessary compilations.
6. **Arena Constructor:** Modified ProtoBuf::Arena::pb_arena_new XSUB to accept the class name argument passed from Perl, making it a proper constructor.
7. **.gitignore:** Added patterns to perl/.gitignore to ignore generated C files from XS ( lib/*.c , lib/ProtoBuf/*.c ), the copied src_google_protobuf_descriptor.pb.cc , and the t/data directory.
8. **Build Documentation:** Updated perl/doc/architecture/upb-build-integration.md to reflect the new build process, including the Bazel prerequisite, include paths, -fPIC usage, and File::Find .
Build Steps:
1. bazel build //src/google/protobuf:descriptor_upb_proto (from repo root)
2. cd perl
3. perl Makefile.PL
4. make
5. make test (Currently has expected failures due to missing test data implementation).
## 0.01 2025-12-14
commit 3e237e8a26442558c94075766e0d4456daaeb71d
Author: C.J. Collier
Date: Sun Dec 14 19:34:28 2025 +0000
feat(perl): Initialize Perl extension scaffold and build system
This commit introduces the perl/ directory, laying the groundwork for the Perl Protocol Buffers extension. It includes the essential build files, linters, formatter configurations, and a vendored Devel::PPPort for XS portability.
Key components added:
* ** Makefile.PL **: The core ExtUtils::MakeMaker build script. It's configured to:
* Build a static library ( libprotobuf_common.a ) from UPB, UTF8_Range, and generated protobuf C/C++ sources.
* Utilize XSMULTI => 1 to create separate shared objects for ProtoBuf , ProtoBuf::Arena , and ProtoBuf::DescriptorPool .
* Link each XS module against the common static library.
* Define custom compilation rules in MY::postamble to handle C vs. C++ flags and build the static library.
* Set up include paths for the project root, UPB, and other dependencies.
* **XS Stubs ( .xs files)**:
* lib/ProtoBuf.xs : Placeholder for the main module's XS functions.
* lib/ProtoBuf/Arena.xs : XS interface for upb_Arena management.
* lib/ProtoBuf/DescriptorPool.xs : XS interface for upb_DefPool management.
* **Perl Module Stubs ( .pm files)**:
* lib/ProtoBuf.pm : Main module, loads XS.
* lib/ProtoBuf/Arena.pm : Perl class for Arenas.
* lib/ProtoBuf/DescriptorPool.pm : Perl class for Descriptor Pools.
* lib/ProtoBuf/Message.pm : Base class for messages (TBD).
* **Configuration Files**:
* .gitignore : Ignores build artifacts, editor files, etc.
* .perlcriticrc : Configures Perl::Critic for static analysis.
* .perltidyrc : Configures perltidy for code formatting.
* ** Devel::PPPort **: Vendored version 3.72 to generate ppport.h for XS compatibility across different Perl versions.
* ** typemap **: Custom typemap for XS argument/result conversion.
* **Documentation ( doc/ )**: Initial architecture and plan documents.
This provides a solid foundation for developing the UPB-based Perl extension.
In January 2024 I wrote about the insanity of the Magnificent Seven dominating the MSCI World Index, and I wondered how long the number can continue to go up? It has continued to surge upward at an accelerating pace, which makes me worry that a crash is likely closer. As a software professional, I decided to analyze whether using stop-loss orders could reliably automate avoiding deep drawdowns.
As everyone with some savings in the stock market (hopefully) knows, the stock market eventually experiences crashes. It is just a matter of when and how deep the crash will be. Staying on the sidelines for years is not a good investment strategy, as inflation will erode the value of your savings. Assuming the current true inflation rate is around 7%, a restaurant dinner that costs 20 euros today will cost 24.50 euros in three years. Savings of 1000 euros today would drop in purchasing power from 50 dinners to only 40 dinners in three years.
Hence, if you intend to retain the value of your hard-earned savings, they need to be invested in something that grows in value. Most people try to beat inflation by buying shares in stable companies, directly or via broad market ETFs. These historically grow faster than inflation during normal years, but likely drop in value during recessions.
What is a trailing stop-loss order?
What if you could buy stocks to benefit from their value increasing without having to worry about a potential crash? All modern online stock brokers have a feature called stop-loss, where you can enter a price at which your stocks automatically get sold if they drop down to that price. A trailing stop-loss order is similar, but instead of a fixed price, you enter a margin (e.g. 10%). If the stock price rises, the stop-loss price will trail upwards by that margin.
For example, if you buy a share at 100 euros and it has risen to 110 euros, you can set a 10% trailing stop-loss order which automatically sells it if the price drops 10% from the peak of 110 euros, at 99 euros. Thus, no matter what happens, you only lost 1 euro. And if the stock price continues to rise to 150 euros, the trailing stop-loss would automatically readjust to 150 euros minus 10%, which is 135 euros (150-15=135). If the price dropped to 135 euros, you would lock in a gain of 35 euros, which is not the peak price of 150 euros, but still better than whatever the price fell down to as a result of a large crash.
In the simple case above, it obviously makes sense in theory, but it might not make sense in practice. Prices constantly oscillate, so you don t want a margin that is too small, otherwise you exit too early. Conversely, having a large margin may result in too large a drawdown before exiting. If markets crash rapidly, it might be that nobody buys your stocks at the stop-loss price, and shares have to be sold at an even lower price. Also, what will you do once the position is sold? The reason you invested in the stock market was to avoid holding cash, so would you buy the same stock back when the crash bottoms? But how will you know when the bottom has been reached?
Backtesting stock market strategies with Python, YFinance, Pandas and Lightweight Charts
I am not a professional investor, and nobody should take investment advice from me. However, I know what backtesting is and how to leverage open source software. So, I wrote a Python script to test if the trading strategy of using trailing stop-loss orders with specific margin values would have worked for a particular stock.
First you need to have data. YFinance is a handy Python library that can be used to download the historic price data for any stock ticker on Yahoo.com. Then you need to manipulate the data. Pandas is the Python data analysis library with advanced data structures for working with relational or labeled data. Finally, to visualize the results, I used Lightweight Charts, which is a fast, interactive library for rendering financial charts, allowing you to plot the stock price, the trailing stop-loss line, and the points where trades would have occurred. I really like how the zoom is implemented in Lightweight Charts, which makes drilling into the data points feel effortless.
The full solution is not polished enough to be published for others to use, but you can piece together your own by reusing some of the key snippets. To avoid re-downloading the same data repeatedly, I implemented a small caching wrapper that saves the data locally (as Parquet files):
pythonCACHE_DIR.mkdir(parents=True, exist_ok=True)
end_date = datetime.today().strftime("%Y-%m-%d")
cache_file = CACHE_DIR / f" TICKER - START_DATE -- end_date .parquet"
if cache_file.is_file():
dataframe = pandas.read_parquet(cache_file)
print(f"Loaded price data from cache: cache_file ")
else:
dataframe = yfinance.download(
TICKER,
start=START_DATE,
end=end_date,
progress=False,
auto_adjust=False
)
dataframe.to_parquet(cache_file)
print(f"Fetched new price data from Yahoo Finance and cached to: cache_file ")
CACHE_DIR.mkdir(parents=True, exist_ok=True)
end_date = datetime.today().strftime("%Y-%m-%d")
cache_file = CACHE_DIR /f"TICKER-START_DATE--end_date.parquet"if cache_file.is_file():
dataframe = pandas.read_parquet(cache_file)
print(f"Loaded price data from cache: cache_file")
else:
dataframe = yfinance.download(
TICKER,
start=START_DATE,
end=end_date,
progress=False,
auto_adjust=False )
dataframe.to_parquet(cache_file)
print(f"Fetched new price data from Yahoo Finance and cached to: cache_file")
The dataframe is a Pandas object with a powerful API. For example, to print a snippet from the beginning and the end of the dataframe to see what the data looks like, you can use:
pythonprint("First 5 rows of the raw data:")
print(df.head())
print("Last 5 rows of the raw data:")
print(df.tail())
print("First 5 rows of the raw data:")
print(df.head())
print("Last 5 rows of the raw data:")
print(df.tail())
Example output:
First 5 rows of the raw data
Price Adj Close Close High Low Open Volume
Ticker BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA
Date
2014-01-02 29.956285 55.540001 56.910000 55.349998 56.700001 316552
2014-01-03 30.031801 55.680000 55.990002 55.290001 55.580002 210044
2014-01-06 30.080338 55.770000 56.230000 55.529999 55.560001 185142
2014-01-07 30.943321 57.369999 57.619999 55.790001 55.880001 370397
2014-01-08 31.385597 58.189999 59.209999 57.750000 57.790001 489940
Last 5 rows of the raw data
Price Adj Close Close High Low Open Volume
Ticker BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA
Date
2025-12-11 78.669998 78.669998 78.919998 76.900002 76.919998 357918
2025-12-12 78.089996 78.089996 80.269997 78.089996 79.470001 280477
2025-12-15 79.080002 79.080002 79.449997 78.559998 78.559998 233852
2025-12-16 78.860001 78.860001 79.980003 78.809998 79.430000 283057
2025-12-17 80.080002 80.080002 80.150002 79.080002 79.199997 262818
First 5 rows of the raw data
Price Adj Close Close High Low Open Volume
Ticker BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA
Date
2014-01-02 29.956285 55.540001 56.910000 55.349998 56.700001 316552
2014-01-03 30.031801 55.680000 55.990002 55.290001 55.580002 210044
2014-01-06 30.080338 55.770000 56.230000 55.529999 55.560001 185142
2014-01-07 30.943321 57.369999 57.619999 55.790001 55.880001 370397
2014-01-08 31.385597 58.189999 59.209999 57.750000 57.790001 489940
Last 5 rows of the raw data
Price Adj Close Close High Low Open Volume
Ticker BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA
Date
2025-12-11 78.669998 78.669998 78.919998 76.900002 76.919998 357918
2025-12-12 78.089996 78.089996 80.269997 78.089996 79.470001 280477
2025-12-15 79.080002 79.080002 79.449997 78.559998 78.559998 233852
2025-12-16 78.860001 78.860001 79.980003 78.809998 79.430000 283057
2025-12-17 80.080002 80.080002 80.150002 79.080002 79.199997 262818
Adding new columns to the dataframe is easy. For example, I used a custom function to calculate the Relative Strength Index (RSI). To add a new column RSI with a value for every row based on the price from that row, only one line of code is needed, without custom loops:
After manipulating the data, the series can be converted into an array structure and printed as JSON into a placeholder in an HTML template:
python baseline_series = [
"time": ts, "value": val
for ts, val in df_plot[["timestamp", BASELINE_LABEL]].itertuples(index=False)
]
baseline_json = json.dumps(baseline_series)
template = jinja2.Template("template.html")
rendered_html = template.render(
title=title,
heading=heading,
description=description_html,
...
baseline_json=baseline_json,
...
)
with open("report.html", "w", encoding="utf-8") as f:
f.write(rendered_html)
print("Report generated!")
baseline_series = [
"time": ts, "value": val
for ts, val in df_plot[["timestamp", BASELINE_LABEL]].itertuples(index=False)
]
baseline_json = json.dumps(baseline_series)
template = jinja2.Template("template.html")
rendered_html = template.render(
title=title,
heading=heading,
description=description_html,
... baseline_json=baseline_json,
... )
with open("report.html", "w", encoding="utf-8") as f:
f.write(rendered_html)
print("Report generated!")
In the HTML template, the marker variable in Jinja syntax gets replaced with the actual JSON:
html<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title> title </title>
...
</head>
<body>
<h1> heading </h1>
<div id="chart"></div>
<script>
// Ensure the DOM is ready before we initialise the chart
document.addEventListener('DOMContentLoaded', () =>
// Parse the JSON data passed from Python
const baselineData = baseline_json safe ;
const strategyData = strategy_json safe ;
const markersData = markers_json safe ;
// Create the chart
const chart = LightweightCharts.createChart(document.getElementById('chart'),
width: document.getElementById('chart').clientWidth,
height: 500,
layout:
background: color: "#222" ,
textColor: "#ccc"
,
grid:
vertLines: color: "#555" ,
horzLines: color: "#555"
);
// Add baseline series
const baselineSeries = chart.addLineSeries(
title: ' baseline_label ',
lastValueVisible: false,
priceLineVisible: false,
priceLineWidth: 1
);
baselineSeries.setData(baselineData);
baselineSeries.priceScale().applyOptions(
entireTextOnly: true
);
// Add strategy series
const strategySeries = chart.addLineSeries(
title: ' strategy_label ',
lastValueVisible: false,
priceLineVisible: false,
color: '#FF6D00'
);
strategySeries.setData(strategyData);
// Add buy/sell markers to the strategy series
strategySeries.setMarkers(markersData);
// Fit the chart to show the full data range (full zoom)
chart.timeScale().fitContent();
)
</script>
</body>
</html>
<!DOCTYPE html><htmllang="en">
<head>
<metacharset="UTF-8">
<title> title </title>
...
</head>
<body>
<h1> heading </h1>
<divid="chart"></div>
<script>
// Ensure the DOM is ready before we initialise the chart
document.addEventListener('DOMContentLoaded', () =>
// Parse the JSON data passed from Python
constbaselineData=baseline_jsonsafe ;
conststrategyData=strategy_jsonsafe ;
constmarkersData=markers_jsonsafe ;
// Create the chart
constchart=LightweightCharts.createChart(document.getElementById('chart'),
width: document.getElementById('chart').clientWidth,
height:500,
layout:background:color:"#222" ,
textColor:"#ccc" ,
grid:vertLines:color:"#555" ,
horzLines:color:"#555" );
// Add baseline series
constbaselineSeries=chart.addLineSeries(
title:' baseline_label ',
lastValueVisible:false,
priceLineVisible:false,
priceLineWidth:1 );
baselineSeries.setData(baselineData);
baselineSeries.priceScale().applyOptions(
entireTextOnly:true );
// Add strategy series
conststrategySeries=chart.addLineSeries(
title:' strategy_label ',
lastValueVisible:false,
priceLineVisible:false,
color:'#FF6D00' );
strategySeries.setData(strategyData);
// Add buy/sell markers to the strategy series
strategySeries.setMarkers(markersData);
// Fit the chart to show the full data range (full zoom)
chart.timeScale().fitContent();
)
</script>
</body>
</html>
There are also Python libraries built specifically for backtesting investment strategies, such as Backtrader and Zipline, but they do not seem to be actively maintained, and probably have too many features and complexity compared to what I needed for doing this simple test.
The screenshot below shows an example of backtesting a strategy on the Waste Management Inc stock from January 2015 to December 2025. The baseline Buy and hold scenario is shown as the blue line and it fully tracks the stock price, while the orange line shows how the strategy would have performed, with markers for the sells and buys along the way.
Results
I experimented with multiple strategies and tested them with various parameters, but I don t think I found a strategy that was consistently and clearly better than just buy-and-hold.
It basically boils down to the fact that I was not able to find any way to calculate when the crash has bottomed based on historical data. You can only know in hindsight that the price has stopped dropping and is on a steady path to recovery, but at that point it is already too late to buy in. In my testing, most strategies underperformed buy-and-hold because they sold when the crash started, but bought back after it recovered at a slightly higher price.
In particular when using narrow margins and selling on a 3-6% drawdown the strategy performed very badly, as those small dips tend to recover in a few days. Essentially, the strategy was repeating the pattern of selling 100 stocks at a 6% discount, then being able to buy back only 94 shares the next day, then again selling 94 shares at a 6% discount, and only being able to buy back maybe 90 shares after recovery, and so forth, never catching up to the buy-and-hold.
The strategy worked better in large market crashes as they tended to last longer, and there were higher chances of buying back the shares while the price was still low. For example, in the 2020 crash selling at a 20% drawdown was a good strategy, as the stock I tested dropped nearly 50% and remained low for several weeks; thus, the strategy bought back the stocks while the price was still low and had not yet started to climb significantly. But that was just a lucky incident, as the delta between the trailing stop-loss margin of 20% and total crash of 50% was large enough. If the crash had been only 25%, the strategy would have missed the rebound and ended up buying back the stocks at a slightly higher price.
Also, note that the simulation assumes that the trade itself is too small to affect the price formation. We should keep in mind that in reality, if many people have stop-loss orders in place, a large price drop would trigger all of them, creating a flood of sell orders, which in turn would affect the price and drive it lower even faster and deeper. Luckily, it seems that stop-loss orders are generally not a good strategy, and we don t need to fear that too many people will be using them.
Conclusion
Even though using a trailing stop-loss strategy does not seem to help in getting consistently higher returns based on my backtesting, I would still say it is useful in protecting from the downside of stock investing. It can act as a kind of insurance policy to considerably decrease the chances of losing big while increasing the chances of losing a little bit. If you are risk-averse, which I think I probably am, this tradeoff can make sense. I d rather miss out on an initial 50% loss and an overall 3% gain on recovery than have to sit through weeks or months with a 50% loss before the price recovers to prior levels.
Most notably, the trailing stop-loss strategy works best if used only once. If it is repeated multiple times, the small losses in gains will compound into big losses overall.
Thus, I think I might actually put this automation in place at least on the stocks in my portfolio that have had the highest gains. If they keep going up, I will ride along, but once the crash happens, I will be out of those particular stocks permanently.
Do you have a favorite open source investment tool or are you aware of any strategy that actually works? Comment below!
We recently bought some Govee Glide Hexa Light Panels, because they have a local LAN API that is well integrated into Home Assistant.
Or so we thought.
Our network is not that complicated, but there is a dedicated VLAN for IOT devices.
Home Assistant runs in a container (with network=host) on a box in the basement, and that box has a NIC in the IOT VLAN so it can reach devices there easily.
So far, this has never been a problem.
Enter the Govee LAN API.
Or maybe its Python implementation.
Not exactly sure who's to blame here.
The API involves sending JSON over multicast, which the Govee device will answer to.
No devices found on the network
After turning logging for homeassistant.components.govee_light_local to 11, erm debug, we see:
DEBUG (MainThread) [homeassistant.components.govee_light_local.config_flow] Starting discovery with IP 192.168.42.2
DEBUG (MainThread) [homeassistant.components.govee_light_local.config_flow] No devices found with IP 192.168.42.2
That's not the IP address in the IOT VLAN!
Turns out the integration recently got support for multiple NICs,
but Home Assistant doesn't just use all the interfaces it sees by default.
You need to go to SettingsNetworkNetwork adapter and deselect "Autoconfigure",
which will allow your to select individual interfaces.
Once you've done that, you'll see Starting discovery with IP messages for all selected interfaces and adding of Govee Lights Local will work.
Go s embed feature lets you bundle static assets into an executable, but it
stores them uncompressed. This wastes space: a web interface with documentation
can bloat your binary by dozens of megabytes. A proposition to optionally
enable compression was declined because it is difficult to handle all use
cases. One solution? Put all the assets into a ZIP archive!
The automatic variable$@ is the rule target, while $^ expands to all
the dependencies, modified or not.
Space gain
Akvorado, a flow collector written in Go, embeds several static assets:
CSV files to translate port numbers, protocols or AS numbers, and
HTML, CSS, JS, and image files for the web interface, and
the documentation.
Breakdown of the space used by each component before (left) and after (right) the introduction of embed.zip.
Embedding these assets into a ZIP archive reduced the size of the Akvorado
executable by more than 4 MiB:
Performance loss
Reading from a compressed archive is not as fast as reading a flat file. A
simple benchmark shows it is more than 4 slower. It also allocates some
memory.2
Each access to an asset requires a decompression step, as seen in this flame
graph:
🖼 Flame graph when reading data from embed.zip compared to reading data directly
CPU flame graph comparing the time spent on CPU when reading data from embed.zip (left) versus reading data directly (right). Because the Go testing framework executes the benchmark for uncompressed data 4 times more often, it uses the same horizontal space as the benchmark for compressed data. The graph is interactive.
While a ZIP archive has an index to quickly find the requested file, seeking
inside a compressed file is currently not possible.3 Therefore, the files
from a compressed archive do not implement the io.ReaderAt or io.Seeker
interfaces, unlike directly embedded files. This prevents some features, like
serving partial files or detecting MIME types when serving files over HTTP.
For Akvorado, this is an acceptable compromise to save a few mebibytes from an
executable of almost 100 MiB. Next week, I will continue this futile adventure
by explaining how I prevented Go from disabling dead code elimination!
You can safely read multiple files concurrently. However, it does
not implement ReadDir() and ReadFile() methods.
You could keep frequently accessed assets in memory. This
reduces CPU usage and trades cached memory for resident memory.
SOZip is a profile that enables fast random access in a compressed
file. However, Go s archive/zip module does not support it.
It's done! It's over!
I've graduated, I have the scroll, I'm staring at
the eye-watering prices for the official photographer snap, I'm
adjusting to post-thesis life.
My PhD thesis revisions have been accepted and my thesis
is now available from Newcastle University Library's eThesis
repository.
As part of submitting my corrections, I wrote a brief report
detailing the changes I made from my thesis at the time of
the viva. I also produced a latexdiff
marked-up copy of the thesis to visualise the exact changes.
In order to shed some light on the post-viva corrections process, at least at
my institution, and in the hope that they are some use to someone, I'm sharing
those documents:
Welcome to the report for November 2025 from the Reproducible Builds project!
These monthly reports outline what we ve been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As always, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
In this report:
10 years of Reproducible Builds at SeaGL 2025
On Friday 8th November, Chris Lamb gave a talk called 10 years of Reproducible Builds at SeaGL in Seattle, WA.
Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture. Chris talk:
[ ] introduces the concept of reproducible builds, its technical underpinnings and its potentially transformative impact on software security and transparency. It is aimed at developers, security professionals and policy-makers who are concerned with enhancing trust and accountability in our software. It also provides a history of the Reproducible Builds project, which is approximately ten years old. How are we getting on? What have we got left to do? Aren t all the builds reproducible now?
Distribution work
In Debian this month, Jochen Sprickerhof created a merge request to replace the use of reprotest in Debian s Salsa Continuous Integration (CI) pipeline with debrebuild. Jochen cites the advantages as being threefold: firstly, that only one extra build needed ; it uses the same sbuild and ccache tooling as the normal build ; and works for any Debian release . The merge request was merged by Emmanuel Arias and is now active.
kpcyrd posted to our mailing list announcing the initial release of repro-threshold, which implements an APT transport that defines a threshold of at least X of my N trusted rebuilders need to confirm they reproduced the binary before installing Debian packages. Configuration can be done through a config file, or through a curses-like user interface.
Holger then merged two commits by Jochen Sprickerhof in order to address a fakeroot-related reproducibility issue in the debian-installer, and J rg Jaspert deployed a patch by Ivo De Decker for a bug originally filed by Holger in February 2025 related to some Debian packages not being archived on snapshot.debian.org.
Elsewhere, Roland Clobus performed some analysis on the live Debian trixie images, which he determined were not reproducible. However, in a follow-up post, Roland happily reports that the issues have been handled. In addition, 145 reviews of Debian packages were added, 12 were updated and 15 were removed this month adding to our knowledge about identified issues.
Lastly, Jochen Sprickerhof filed a bug announcing their intention to binary NMU a very large number of the R programming language after a reproducibility-related toolchain bug was fixed.
Bernhard M. Wiedemann posted another openSUSEmonthly update for their work there.
Julien Malka and Arnout Engelen launched the new hash collection
server for NixOS. Aside from improved reporting to help focus reproducible builds
efforts within NixOS, it collects build hashes as individually-signed attestations
from independent builders, laying the groundwork for further tooling.
Tool development
diffoscope version 307 was uploaded to Debian unstable (as well as version 309). These changes included further attempts to automatically attempt to deploy to PyPI by liaising with the PyPI developers/maintainers (with this experimental feature). [][][]
In addition, reprotest versions 0.7.31 and 0.7.32 were uploaded to Debian unstable by Holger Levsen, who also made the following changes:
Do not vary the architecture personality if the kernel is not varied. (Thanks to Ra l Cumplido). []
Drop the debian/watch file, as Lintian now flags this as error for native Debian packages. [][]
Bump Standards-Version to 4.7.2, with no changes needed. []
Drop the Rules-Requires-Root header as it is no longer required.. []
In addition, however, Vagrant Cascadian fixed a build failure by removing some extra whitespace from an older changelog entry. []
Website updates
Once again, there were a number of improvements made to our website this month including:
Bernhard M. Wiedemann updated the SOURCE_DATE_EPOCH page to fix the Lisp example syntax. []
Web3 applications, built on blockchain technology, manage billions of dollars in digital assets through decentralized applications (dApps) and smart contracts. These systems rely on complex, software supply chains that introduce significant security vulnerabilities. This paper examines the software supply chain security challenges unique to the Web3 ecosystem, where traditional Web2 software supply chain problems intersect with the immutable and high-stakes nature of blockchain technology. We analyze the threat landscape and propose mitigation strategies to strengthen the security posture of Web3 systems.
Their paper lists reproducible builds as one of the mitigating strategies. A PDF of the full text is available to download.
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
SLES 16 has been released. In the past, SUSE offered ready built vagrant
images. Unfortunately that s not the case anymore, as with more recent
SLES15 releases the official images were gone.
In the past, it was possible to clone existing projects on the opensuse build
service to build the images by yourself, but i couldn t find any templates
for SLES 16.
Naturally, there are several ways to build images, and the tooling around
involves kiwi-ng, opensuse build service, or packer recipes etc.. (existing
packer recipes wont work anymore, as Yast has been replaced by a new installer,
called agma). All pretty complicated,
So my current take on creating a vagrant image for SLE16 has been the
following:
Spin up an QEMU virtual machine
Manually install the system, all in default except for one special setting:
In the Network connection details, Edit Binding settings and set the
Interface to not bind a particular MAC address or interface. This will make
the system pick whatever network device naming scheme is applied during
boot.
After installation has finished, shutdown.
Two guestfs-tools that can now be used to modify the created qcow2 image:
run virt-sysrpep on the image to wipe settings that might cause troubles:
virt-sysprep -a sles16.qcow2
create a simple shellscript that setups all vagrant related settings:
#!/bin/bash
useradd vagrant
mkdir-p /home/vagrant/.ssh/
chmod 0700 /home/vagrant/.ssh/
echo"ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIF
o9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9W
hQ== vagrant insecure public key"> /home/vagrant/.ssh/authorized_keys
chmod 0600 /home/vagrant/.ssh/authorized_keys
chown-R vagrant:vagrant /home/vagrant/
# apply recommended ssh settings for vagrant boxesSSHD_CONFIG=/etc/ssh/sshd_config.d/99-vagrant.conf
if[[!-d"$(dirname$ SSHD_CONFIG)"]];then
SSHD_CONFIG=/etc/ssh/sshd_config
# prepend the settings, so that they take precedenceecho-e"UseDNS no\nGSSAPIAuthentication no\n$(cat$ SSHD_CONFIG)">$ SSHD_CONFIGelse
echo-e"UseDNS no\nGSSAPIAuthentication no">$ SSHD_CONFIGfi
SUDOERS_LINE="vagrant ALL=(ALL) NOPASSWD: ALL"if[-d /etc/sudoers.d ];then
echo"$SUDOERS_LINE"> /etc/sudoers.d/vagrant
visudo -cf /etc/sudoers.d/vagrant
chmod 0440 /etc/sudoers.d/vagrant
else
echo"$SUDOERS_LINE">> /etc/sudoers
visudo -cf /etc/sudoers
fi
mkdir-p /vagrant
chown-R vagrant:vagrant /vagrant
systemctl enable sshd
use virt-customize to upload the script into the qcow image:
virt-customize -a sle16.qcow2 --upload vagrant.sh:/tmp/vagrant.sh
execute the script via:
virt-customize -a sle16.qcow2 --run-command"/tmp/vagrant.sh"
After this, use the create-box.sh from the vagrant-libvirt project
to create an box image:
https://github.com/vagrant-libvirt/vagrant-libvirt/blob/main/tools/create_box.sh
and add the image to your environment:
Debian LTS
This was my hundred-thirty-sixth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:
[DLA 4316-1] open-vm-tools security update to fix one CVE related to a local privilege escalation.
[DLA 4329-1] libfcgi security update to fix one CVE related to a heap-based buffer overflow via crafted nameLen or valueLen values in data to the IPC socket.
[DLA 4337-1] svgpp security update to fix one CVE related to a nullpointer reference.
[DLA 4336-1] sysstat security update to fix two CVEs related to a size_t overflow and a multiplication integer overflow.
[DLA 4343-1] raptor2 security update to fix two CVEs related to a heap-based buffer over-read and an integer underflow.
[DLA 4349-1] request-tracker4 security update to fix one CVE related to CSV injection via ticket values with special characters. The patch was prepared by Andrew Ruthven
[DLA 4353-1] xorg-server security update to fix three CVES related to privilege escalation.
I also attended the monthly LTS/ELTS meeting.
Debian ELTS
This month was the eighty-seventh ELTS month. During my allocated time I uploaded or worked on:
[ELA-1538-1] libfcgi security update to fix one CVE in Buster and Stretch, related to a heap-based buffer overflow via crafted nameLen or valueLen values in data to the IPC socket.
[ELA-1551-1] raptor2 security update to fix two CVES in Buster and Stretch, related to a heap-based buffer over-read and an integer underflow.
[ELA-1555-1] request-tracker4 security update to fix one CVE in Buster, related to CSV injection via ticket values with special characters. The patch was prepared by Andrew Ruthven.
[ELA-1561-1] xorg-server security update to fix three CVEs in Buster and Stretch, related to privilege escalation.
I also attended the monthly LTS/ELTS meeting.
Debian Printing
This month I uploaded a new upstream version or a bugfix version of:
Debian IoT
Unfortunately I didn t found any time to work on this topic.
Debian Mobcom
This month I uploaded a new upstream version or a bugfix version of:
On my fight against outdated RFPs, I closed 31 of them in October. I could even close one RFP by uploading the new package gypsy. Meanwhile only 3373 are still open, so don t hesitate to help closing one or another.
FTP master
This month I accepted 420 and rejected 45 packages. The overall number of packages that got accepted was 423.
I would like to remind everybody that in case you don t agree with the removal of a package, please set the moreinfo tag on this bug. This is the only reliable way to prevent processing of that RM-bug. Well, there is a second way, of course you could also achieve this by closing the bug.
A couple of days ago in a short
post, I announced duckdb-mlpack
as ML quacks : combining the powerful C++
machine learning library mlpack with the amazing analytical database engine duckdb. See that
post for more background.
The duckdb-mlpack
package is now a community extension
joining an impressive list of existing
extensions. This means duckdb
builds and distributes duckdb-mlpack
for all supported platforms allowing users to just install the resulting
(signed) binary. (We currently only support Linux in both arm64 and
amd64, adding macOS should be straightforward once we sort one build
issue out. Windows and WASM should work too, with a little love and
polish, as both duckdb and mlpack support them.) Given the binary
build, a simple
INSTALL mlpack FROM community;LOAD mlpack;
installs and loads the package. By the duckdb convention the code is stored
per-user and per-version, so the first line needs to be executed only
once per duckdb release used. The
second line is then per session.
We also extended the capabilities of duckdb-mlpack.
While still a MVP stressing minimal viable product,
the two supported methods adaBoost and (regularized) linear regression
both serialize and store their model object permitting rapid prediction
on new data as shown in the adaBoost example:
-- Perform adaBoost (using weak learner 'Perceptron' by default)-- Read 'features' into 'X', 'labels' into 'Y', use optional parameters-- from 'Z', and prepare model storage in 'M'CREATETABLE X ASSELECT*FROM read_csv("https://eddelbuettel.github.io/duckdb-mlpack/data/iris.csv");CREATETABLE Y ASSELECT*FROM read_csv("https://eddelbuettel.github.io/duckdb-mlpack/data/iris_labels.csv");CREATETABLE Z (name VARCHAR, valueVARCHAR);INSERTINTO Z VALUES ('iterations', '50'), ('tolerance', '1e-7');CREATETABLE M (json VARCHAR);-- Train model for 'Y' on 'X' using parameters 'Z', store in 'M'CREATE TEMP TABLE A ASSELECT*FROM mlpack_adaboost("X", "Y", "Z", "M");-- Count by predicted groupSELECTCOUNT(*) as n, predicted FROM A GROUPBY predicted;-- Model 'M' can be used to predictCREATETABLE N (x1 DOUBLE, x2 DOUBLE, x3 DOUBLE, x4 DOUBLE);-- inserting approximate column mean valuesINSERTINTO N VALUES (5.843, 3.054, 3.759, 1.199);-- inserting approximate column mean values, min values, max valuesINSERTINTO N VALUES (5.843, 3.054, 3.759, 1.199), (4.3, 2.0, 1.0, 0.1), (7.9, 4.4, 6.9, 2.5);-- and this predict one element eachSELECT*FROM mlpack_adaboost_pred("N", "M");
Ryan and I have some ideas for where
to go from here, ideally towards autogenerating bindings for most (if
not all) methods as is done for the mlpack language bindings. Anybody
interested and willing to help should reach out to us.
A couple of weeks ago there was an article on the Freexian blog about Using
JavaScript in Debusine without depending on
JavaScript. It
describes how JavaScript is used in the Debusine Django app, namely for
progressive enhancement rather than core functionality .
This is an approach I also follow when implementing web interfaces and I think
developments in web technologies and standardization in recent years have made
this a lot easier.
One of the examples described in the post, the Bootstrap toast messages, was
something that I implemented myself recently, in a similar but slightly
different way.
In the main app I develop for my day job we also use the Bootstrap
framework. I have also used it for different
personal projects (for example the GSOC project I did for Debian in 2018, was
also a Django app that used
Bootstrap).
Bootstrap is still primarily a CSS framework, but it also comes with a
JavaScript library for some functionality. Previous versions of Bootstrap
depended on jQuery, but since version 5 of Bootstrap, you don t need jQuery
anymore. In my experience, two of the more commonly used JavaScript utilities
of Bootstrap are modals
(also called lightbox or popup, they are elements that are displayed above
the main content of a website) and
toasts (also called
alerts, they are little notification windows that often disappear after a
timeout). The thing is, Bootstrap 5 was released in 2021 and a lot has happened
since then regarding web technologies. I believe that both these UI components
can nowadays be implemented using standard HTML5 elements.
An eye opening talk I watched was Stop using JS for
that from last years JSConf(!).
In this talk the speaker argues that the Rule of least
power is one of the core
principles of web development, which means we should use HTML over CSS and CSS
over JavaScript. And the speaker also presents some CSS rules and HTML elements
that added recently and that help to make that happen, one of them being the
dialog
element:
The <dialog> HTML element represents a modal or non-modal dialog box or other
interactive component, such as a dismissible alert, inspector, or subwindow.
The Dialog element at MDN
The baseline for this element is widely available :
This feature is well established and works across many devices and browser
versions. It s been available across browsers since March 2022.
The Dialog element at MDN
This means there is an HTML element that does what a modal Bootstrap does!
Once I had watched that talk I removed all my Bootstrap modals and replaced
them with HTML <dialog> elements (JavaScript is still needed to .show() and
.close() the elements, though, but those are two methods instead of a full
library). This meant not only that I replaced code that depended on an external
library, I m now also a lot more flexible regarding the styling of the
elements.
When I started implementing notifications for our app, my first approach was to
use Bootstrap toasts, similar to how it is implemented in Debusine. But looking
at the amount of HTML code I had to write for a simple toast message, I thought
that it might be possible to also implement toasts with the <dialog> element.
I mean, basically it is the same, only the styling is a bit different. So what
I did was that I added a #snackbar area to the DOM of the app. This would be
the container for the toast messages. All the toast messages are simply
<dialog> elements with the open attribute, which means that they are
visible right away when the page loads.
<divid="snackbar">
% for message in messages %
<dialogclass="mytoast alert alert- message.tags "role="alert"open>
message
</dialog>
% endfor %
</div>
This looks a lot simpler than the Bootstrap toasts would have.
To make the <dialog> elements a little bit more fancy, I added some CSS to make
them fade in and out:
(If one would want to use the same HTML code for both script and noscript users,
then the CSS should probably adapted: it fades away and if there is no
JavaScript to close the element, it stays visible after the animation is over.
A solution would for example be to use a close button and for noscript users
simply let it stay visible - this is also what happens with the noscript
messages in Debusine).
So there are many new elements in HTML and a lot of new features of CSS. It
makes sense to sometimes ask ourselves if instead of the solutions we know (or
what a web search / some AI shows us as the most common solution) there might
be some newer solution that was not there when the first choice was
created. Using standardized solutions instead of custom libraries makes the
software more maintainable. In web development I also prefer standardized
elements over a third party library because they have usually better
accessibility and UX.
In How Functional Programming Shaped (and Twisted) Frontend
Development
the author writes:
Consider the humble modal dialog. The web has <dialog>, a native element with
built-in functionality: it manages focus trapping, handles Escape key
dismissal, provides a backdrop, controls scroll-locking on the body, and
integrates with the accessibility tree. It exists in the DOM but remains
hidden until opened. No JavaScript mounting required.
[ ]
you ve trained developers to not even look for native solutions. The platform
becomes invisible. When someone asks how do I build a modal? , the answer is
install a library or here s my custom hook, never use <dialog>.
Ahmad Alfy
The discovery of a backdoor in XZ Utils in the spring of 2024 shocked the open source community, raising critical questions about software supply chain security. This post explores whether better Debian packaging practices could have detected this threat, offering a guide to auditing packages and suggesting future improvements.
The XZ backdoor in versions 5.6.0/5.6.1 made its way briefly into many major Linux distributions such as Debian and Fedora, but luckily didn t reach that many actual users, as the backdoored releases were quickly removed thanks to the heroic diligence of Andres Freund. We are all extremely lucky that he detected a half a second performance regression in SSH, cared enough to trace it down, discovered malicious code in the XZ library loaded by SSH, and reported promtly to various security teams for quick coordinated actions.
This episode makes software engineers pondering the following questions:
Why didn t any Linux distro packagers notice anything odd when importing the new XZ version 5.6.0/5.6.1 from upstream?
Is the current software supply-chain in the most popular Linux distros easy to audit?
Could we have similar backdoors lurking that haven t been detected yet?
As a Debian Developer, I decided to audit the xz package in Debian, share my methodology and findings in this post, and also suggest some improvements on how the software supply-chain security could be tightened in Debian specifically.
Note that the scope here is only to inspect how Debian imports software from its upstreams, and how they are distributed to Debian s users. This excludes the whole story of how to assess if an upstream project is following software development security best practices. This post doesn t discuss how to operate an individual computer running Debian to ensure it remains untampered as there are plenty of guides on that already.
Downloading Debian and upstream source packages
Let s start by working backwards from what the Debian package repositories offer for download. As auditing binaries is extremely complicated, we skip that, and assume the Debian build hosts are trustworthy and reliably building binaries from the source packages, and the focus should be on auditing the source code packages.
As with everything in Debian, there are multiple tools and ways to do the same thing, but in this post only one (and hopefully the best) way to do something is presented for brevity.
The first step is to download the latest version and some past versions of the package from the Debian archive, which is easiest done with debsnap. The following command will download all Debian source packages of xz-utils from Debian release 5.2.4-1 onwards:
Verifying authenticity of upstream and Debian sources using OpenPGP signatures
As seen in the output of debsnap, it already automatically verifies that the downloaded files match the OpenPGP signatures. To have full clarity on what files were authenticated with what keys, we should verify the Debian packagers signature with:
$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc
gpg: Signature made Fri Oct 3 22:04:44 2025 UTC
gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B
gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org
gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Sebastian Andrzej Siewior" [unknown]
gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown]
gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1
Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B
$ gpg --verify --auto-key-retrieve --keyserver hkps://keyring.debian.org xz-utils_5.8.1-2.dsc
gpg: Signature made Fri Oct 3 22:04:44 2025 UTC
gpg: using RSA key 57892E705233051337F6FDD105641F175712FA5B
gpg: requesting key 05641F175712FA5B from hkps://keyring.debian.org
gpg: key 7B96E8162A8CF5D1: public key "Sebastian Andrzej Siewior" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Sebastian Andrzej Siewior" [unknown]
gpg: aka "Sebastian Andrzej Siewior <bigeasy@linutronix.de>" [unknown]
gpg: aka "Sebastian Andrzej Siewior <sebastian@breakpoint.cc>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1
Subkey fingerprint: 5789 2E70 5233 0513 37F6 FDD1 0564 1F17 5712 FA5B
The upstream tarball signature (if available) can be verified with:
$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc
gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz'
gpg: Signature made Thu Apr 3 11:38:23 2025 UTC
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620
$ gpg --verify --auto-key-retrieve xz-utils_5.8.1.orig.tar.xz.asc
gpg: assuming signed data in 'xz-utils_5.8.1.orig.tar.xz'
gpg: Signature made Thu Apr 3 11:38:23 2025 UTC
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: key 38EE757D69184620: public key "Lasse Collin <lasse.collin@tukaani.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3690 C240 CE51 B467 0D30 AD1C 38EE 757D 6918 4620
Note that this only proves that there is a key that created a valid signature for this content. The authenticity of the keys themselves need to be validated separately before trusting they in fact are the keys of these people. That can be done by checking e.g. the upstream website for what key fingerprints they published, or the Debian keyring for Debian Developers and Maintainers, or by relying on the OpenPGP web-of-trust .
Verifying authenticity of upstream sources by comparing checksums
In case the upstream in question does not publish release signatures, the second best way to verify the authenticity of the sources used in Debian is to download the sources directly from upstream and compare that the sha256 checksums match.
This should be done using the debian/watch file inside the Debian packaging, which defines where the upstream source is downloaded from. Continuing on the example situation above, we can unpack the latest Debian sources, enter and then run uscan to download:
$ tar xvf xz-utils_5.8.1-2.debian.tar.xz
...
debian/rules
debian/source/format
debian/source.lintian-overrides
debian/symbols
debian/tests/control
debian/tests/testsuite
debian/upstream/signing-key.asc
debian/watch
...
$ uscan --download-current-version --destdir /tmp
Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1
gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC
gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>"
Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz.
$ tar xvf xz-utils_5.8.1-2.debian.tar.xz
...
debian/rules
debian/source/format
debian/source.lintian-overrides
debian/symbols
debian/tests/control
debian/tests/testsuite
debian/upstream/signing-key.asc
debian/watch
...
$ uscan --download-current-version --destdir /tmp
Newest version of xz-utils on remote site is 5.8.1, specified download version is 5.8.1
gpgv: Signature made Thu Apr 3 11:38:23 2025 UTC
gpgv: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpgv: Good signature from "Lasse Collin <lasse.collin@tukaani.org>"
Successfully symlinked /tmp/xz-5.8.1.tar.xz to /tmp/xz-utils_5.8.1.orig.tar.xz.
The original files downloaded from upstream are now in /tmp along with the files renamed to follow Debian conventions. Using everything downloaded so far the sha256 checksums can be compared across the files and also to what the .dsc file advertised:
In the example above the checksum 0b54f79df85... is the same across the files, so it is a match.
Repackaged upstream sources can t be verified as easily
Note that uscan may in rare cases repackage some upstream sources, for example to exclude files that don t adhere to Debian s copyright and licensing requirements. Those files and paths would be listed under the Files-Excluded section in the debian/copyright file. There are also other situations where the file that represents the upstream sources in Debian isn t bit-by-bit the same as what upstream published. If checksums don t match, an experienced Debian Developer should review all package settings (e.g. debian/source/options) to see if there was a valid and intentional reason for divergence.
Reviewing changes between two source packages using diffoscope
Diffoscope is an incredibly capable and handy tool to compare arbitrary files. For example, to view a report in HTML format of the differences between two XZ releases, run:
If the changes are extensive, and you want to use a LLM to help spot potential security issues, generate the report of both the upstream and Debian packaging differences in Markdown with:
The Markdown files created above can then be passed to your favorite LLM, along with a prompt such as:
Based on the attached diffoscope output for a new Debian package version compared with the previous one, list all suspicious changes that might have introduced a backdoor, followed by other potential security issues. If there are none, list a short summary of changes as the conclusion.
Reviewing Debian source packages in version control
As of today only 93% of all Debian source packages are tracked in git on Debian s GitLab instance at salsa.debian.org. Some key packages such as Coreutils and Bash are not using version control at all, as their maintainers apparently don t see value in using git for Debian packaging, and the Debian Policy does not require it. Thus, the only reliable and consistent way to audit changes in Debian packages is to compare the full versions from the archive as shown above.
However, for packages that are hosted on Salsa, one can view the git history to gain additional insight into what exactly changed, when and why. For packages that are using version control, their location can be found in the Git-Vcs header in the debian/control file. For xz-utils the location is salsa.debian.org/debian/xz-utils.
Note that the Debian policy does not state anything about how Salsa should be used, or what git repository layout or development practices to follow. In practice most packages follow the DEP-14 proposal, and use git-buildpackage as the tool for managing changes and pushing and pulling them between upstream and salsa.debian.org.
To get the XZ Utils source, run:
$ gbp clone https://salsa.debian.org/debian/xz-utils.git
gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'
$ gbp clone https://salsa.debian.org/debian/xz-utils.git
gbp:info: Cloning from 'https://salsa.debian.org/debian/xz-utils.git'
At the time of writing this post the git history shows:
$ git log --graph --oneline
* bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2
* 4b769547 d: Remove the symlinks from -dev package.
* a39f3428 Correct the nocheck build profile
* 1b806b8d Import Debian changes 5.8.1-1.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h
* 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder
* 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding
* 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread
* d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
* c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state
* 831b55b9 liblzma: mt dec: Fix a comment
* b9d168ee liblzma: Add assertions to lzma_bufcpy()
* c8e0a489 DOS: Update Makefile to fix the build
* 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers
* 7ce38b31 Update THANKS
* 688e51bd Translations: Update the Croatian translation
* a6b54dde Prepare 5.8.0-1.
* 77d9470f Add 5.8 symbols.
* 9268eb66 Import 5.8.0
* 6f85ef4f Update upstream source from tag 'upstream/5.8.0'
\ \
* afba662b New upstream version 5.8.0
/
* 173fb5c6 doc/SHA256SUMS: Add 5.8.0
* db9258e8 Bump version and soname for 5.8.0
* bfb752a3 Add NEWS for 5.8.0
* 6ccbb904 Translations: Run "make -C po update-po"
* 891a5f05 Translations: Run po4a/update-po
* 4f52e738 Translations: Partially fix overtranslation in Serbian man pages
* ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage
* 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()
$ git log --graph --oneline
* bb787585 (HEAD -> debian/unstable, origin/debian/unstable, origin/HEAD) Prepare 5.8.1-2
* 4b769547 d: Remove the symlinks from -dev package.
* a39f3428 Correct the nocheck build profile
* 1b806b8d Import Debian changes 5.8.1-1.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (origin/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* 513cabcf Tests: Call lzma_code() in smaller chunks in fuzz_common.h
* 48440e24 Tests: Add a fuzzing target for the multithreaded .xz decoder
* 0c80045a liblzma: mt dec: Fix lack of parallelization in single-shot decoding
* 81880488 liblzma: mt dec: Don't modify thr->in_size in the worker thread
* d5a2ffe4 liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
* c0c83596 liblzma: mt dec: Simplify by removing the THR_STOP state
* 831b55b9 liblzma: mt dec: Fix a comment
* b9d168ee liblzma: Add assertions to lzma_bufcpy()
* c8e0a489 DOS: Update Makefile to fix the build
* 307c02ed sysdefs.h: Avoid <stdalign.h> even with C11 compilers
* 7ce38b31 Update THANKS
* 688e51bd Translations: Update the Croatian translation
* a6b54dde Prepare 5.8.0-1.
* 77d9470f Add 5.8 symbols.
* 9268eb66 Import 5.8.0
* 6f85ef4f Update upstream source from tag 'upstream/5.8.0'
\ \
* afba662b New upstream version 5.8.0
/
* 173fb5c6 doc/SHA256SUMS: Add 5.8.0
* db9258e8 Bump version and soname for 5.8.0
* bfb752a3 Add NEWS for 5.8.0
* 6ccbb904 Translations: Run "make -C po update-po"
* 891a5f05 Translations: Run po4a/update-po
* 4f52e738 Translations: Partially fix overtranslation in Serbian man pages
* ff5d9447 liblzma: Count the extra bytes in LZMA/LZMA2 decoder memory usage
* 943b012d liblzma: Use SSE2 intrinsics instead of memcpy() in dict_repeat()
This shows both the changes on the debian/unstable branch as well as the intermediate upstream import branch, and the actual real upstream development branch. See my Debian source packages in git explainer for details of what these branches are used for.
To only view changes on the Debian branch, run git log --graph --oneline --first-parent or git log --graph --oneline -- debian.
The Debian branch should only have changes inside the debian/ subdirectory, which is easy to check with:
If the upstream in question signs commits or tags, they can be verified with e.g.:
$ git verify-tag v5.6.2
gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: issuer "lasse.collin@tukaani.org"
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired]
gpg: Note: This key has expired!
$ git verify-tag v5.6.2
gpg: Signature made Wed 29 May 2024 09:39:42 AM PDT
gpg: using RSA key 3690C240CE51B4670D30AD1C38EE757D69184620
gpg: issuer "lasse.collin@tukaani.org"
gpg: Good signature from "Lasse Collin <lasse.collin@tukaani.org>" [expired]
gpg: Note: This key has expired!
The main benefit of reviewing changes in git is the ability to see detailed information about each individual change, instead of just staring at a massive list of changes without any explanations. In this example, to view all the upstream commits since the previous import to Debian, one would view the commit range from afba662b New upstream version 5.8.0 to fa1e8796 New upstream version 5.8.1 with git log --reverse -p afba662b...fa1e8796. However, a far superior way to review changes would be to browse this range using a visual git history viewer, such as gitk. Either way, looking at one code change at a time and reading the git commit message makes the review much easier.
Comparing Debian source packages to git contents
As stated in the beginning of the previous section, and worth repeating, there is no guarantee that the contents in the Debian packaging git repository matches what was actually uploaded to Debian. While the tag2upload project in Debian is getting more and more popular, Debian is still far from having any system to enforce that the git repository would be in sync with the Debian archive contents.
To detect such differences we can run diff across the Debian source packages downloaded with debsnap earlier (path source-xz-utils/xz-utils_5.8.1-2.debian) and the git repository cloned in the previous section (path xz-utils):
diff$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/
diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog
--- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700
+++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700
@@ -5,7 +5,7 @@
* Remove the symlinks from -dev, pointing to the lib package.
(Closes: #1109354)
- -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200
$ diff -u source-xz-utils/xz-utils_5.8.1-2.debian/ xz-utils/debian/
diff -u source-xz-utils/xz-utils_5.8.1-2.debian/changelog xz-utils/debian/changelog
--- debsnap/source-xz-utils/xz-utils_5.8.1-2.debian/changelog 2025-10-03 09:32:16.000000000 -0700
+++ xz-utils/debian/changelog 2025-10-12 12:18:04.623054758 -0700
@@ -5,7 +5,7 @@
* Remove the symlinks from -dev, pointing to the lib package.
(Closes: #1109354)
- -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:32:16 +0200
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Fri, 03 Oct 2025 18:36:59 +0200
In the case above diff revealed that the timestamp in the changelog in the version uploaded to Debian is different from what was committed to git. This is not malicious, just a mistake by the maintainer who probably didn t run gbp tag immediately after upload, but instead some dch command and ended up with having a different timestamps in the git compared to what was actually uploaded to Debian.
Creating syntetic Debian packaging git repositories
If no Debian packaging git repository exists, or if it is lagging behind what was uploaded to Debian s archive, one can use git-buildpackage s import-dscs feature to create synthetic git commits based on the files downloaded by debsnap, ensuring the git contents fully matches what was uploaded to the archive. To import a single version there is gbp import-dsc (no s at the end), of which an example invocation would be:
$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc
Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'
$ gbp import-dsc --verbose ../source-xz-utils/xz-utils_5.8.1-2.dsc
Version '5.8.1-2' imported under '/home/otto/debian/xz-utils-2025-09-29'
Example commit history from a repository with commits added with gbp import-dsc:
An online example repository with only a few missing uploads added using gbp import-dsc can be viewed at salsa.debian.org/otto/xz-utils-2025-09-29/-/network/debian%2Funstable
An example repository that was fully crafted using gbp import-dscs can be viewed at salsa.debian.org/otto/xz-utils-gbp-import-dscs-debsnap-generated/-/network/debian%2Flatest.
There exists also dgit, which in a similar way creates a synthetic git history to allow viewing the Debian archive contents via git tools. However, its focus is on producing new package versions, so fetching a package with dgit that has not had the history recorded in dgit earlier will only show the latest versions:
$ dgit clone xz-utils
canonical suite name for unstable is sid
starting new git history
last upload to archive: NO git hash
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz...
dpkg-source: info: extracting xz-utils in unpacked
dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz
dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz
synthesised git commit from .dsc 5.8.1-2
HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium
dgit ok: ready for work in xz-utils
$ dgit/sid git log --graph --oneline
* f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid)
\
* 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago
* 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago
$ dgit clone xz-utils
canonical suite name for unstable is sid
starting new git history
last upload to archive: NO git hash
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1.orig.tar.xz.asc...
downloading http://ftp.debian.org/debian//pool/main/x/xz-utils/xz-utils_5.8.1-2.debian.tar.xz...
dpkg-source: info: extracting xz-utils in unpacked
dpkg-source: info: unpacking xz-utils_5.8.1.orig.tar.xz
dpkg-source: info: unpacking xz-utils_5.8.1-2.debian.tar.xz
synthesised git commit from .dsc 5.8.1-2
HEAD is now at f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium
dgit ok: ready for work in xz-utils
$ dgit/sid git log --graph --oneline
* f9bcaf7 xz-utils (5.8.1-2) unstable; urgency=medium 9 days ago (HEAD -> dgit/sid, dgit/dgit/sid)
\
* 11d3a62 Import xz-utils_5.8.1-2.debian.tar.xz 9 days ago
* 15dcd95 Import xz-utils_5.8.1.orig.tar.xz 6 months ago
Unlike git-buildpackage managed git repositories, the dgit managed repositories cannot incorporate the upstream git history and are thus less useful for auditing the full software supply-chain in git.
Comparing upstream source packages to git contents
Equally important to the note in the beginning of the previous section, one must also keep in mind that the upstream release source packages, often called release tarballs, are not guaranteed to have the exact same contents as the upstream git repository. Projects might strip out test data or extra development files from their release tarballs to avoid shipping unnecessary files to users, or projects might add documentation files or versioning information into the tarball that isn t stored in git. While a small minority, there are also upstreams that don t use git at all, so the plain files in a release tarball is still the lowest common denominator for all open source software projects, and exporting and importing source code needs to interface with it.
In the case of XZ, the release tarball has additional version info and also a sizeable amount of pregenerated compiler configuration files. Detecting and comparing differences between git contents and tarballs can of course be done manually by running diff across an unpacked tarball and a checked out git repository. If using git-buildpackage, the difference between the git contents and tarball contents can be made visible directly in the import commit.
In this XZ example, consider this git history:
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
* b1cad34b Prepare 5.8.1-1
* a8646015 Import 5.8.1
* 2808ec2d Update upstream source from tag 'upstream/5.8.1'
\
* fa1e8796 (debian/upstream/v5.8, upstream/v5.8) New upstream version 5.8.1
* a522a226 (tag: v5.8.1) Bump version and soname for 5.8.1
* 1c462c2a Add NEWS for 5.8.1
The commit a522a226 was the upstream release commit, which upstream also tagged v5.8.1. The merge commit 2808ec2d applied the new upstream import branch contents on the Debian branch. Between these is the special commit fa1e8796 New upstream version 5.8.1 tagged upstream/v5.8. This commit and tag exists only in the Debian packaging repository, and they show what is the contents imported into Debian. This is generated automatically by git-buildpackage when running git import-orig --uscan for Debian packages with the correct settings in debian/gbp.conf. By viewing this commit one can see exactly how the upstream release tarball differs from the upstream git contents (if at all).
In the case of XZ, the difference is substantial, and shown below in full as it is very interesting:
To be able to easily inspect exactly what changed in the release tarball compared to git release tag contents, the best tool for the job is Meld, invoked via git difftool --dir-diff fa1e8796^..fa1e8796.
To compare changes across the new and old upstream tarball, one would need to compare commits afba662b New upstream version 5.8.0 and fa1e8796 New upstream version 5.8.1 by running git difftool --dir-diff afba662b..fa1e8796.
With all the above tips you can now go and try to audit your own favorite package in Debian and see if it is identical with upstream, and if not, how it differs.
Should the XZ backdoor have been detected using these tools?
The famous XZ Utils backdoor (CVE-2024-3094) consisted of two parts: the actual backdoor inside two binary blobs masqueraded as a test files (tests/files/bad-3-corrupt_lzma2.xz, tests/files/good-large_compressed.lzma), and a small modification in the build scripts (m4/build-to-host.m4) to extract the backdoor and plant it into the built binary. The build script was not tracked in version control, but generated with GNU Autotools at release time and only shipped as additional files in the release tarball.
The entire reason for me to write this post was to ponder if a diligent engineer using git-buildpackage best practices could have reasonably spotted this while importing the new upstream release into Debian. The short answer is no . The malicious actor here clearly anticipated all the typical ways anyone might inspect both git commits, and release tarball contents, and masqueraded the changes very well and over a long timespan.
First of all, XZ has for legitimate reasons for several carefully crafted .xz files as test data to help catch regressions in the decompression code path. The test files are shipped in the release so users can run the test suite and validate that the binary is built correctly and xz works properly. Debian famously runs massive amounts of testing in its CI and autopkgtest system across tens of thousands of packages to uphold high quality despite frequent upgrades of the build toolchain and while supporting more CPU architectures than any other distro. Test data is useful and should stay.
When git-buildpackage is used correctly, the upstream commits are visible in the Debian packaging for easy review, but the commit cf44e4b that introduced the test files does not deviate enough from regular sloppy coding practices to really stand out. It is unfortunately very common for git commit to lack a message body explaining why the change was done, and to not be properly atomic with test code and test data together in the same commit, and for commits to be pushed directly to mainline without using code reviews (the commit was not part of any PR in this case). Only another upstream developer could have spotted that this change is not on par to what the project expects, and that the test code was never added, only test data, and thus that this commit was not just a sloppy one but potentially malicious.
Secondly, the fact that a new Autotools file appeared (m4/build-to-host.m4) in the XZ Utils 5.6.0 is not suspicious. This is perfectly normal for Autotools. In fact, starting from XZ Utils version 5.8.1 it is now shipping a m4/build-to-host.m4 file that it actually uses now.
Spotting that there is anything fishy is practically impossible by simply reading the code, as Autotools files are full custom m4 syntax interwoven with shell script, and there are plenty of backticks () that spawn subshells and evals that execute variable contents further, which is just normal for Autotools. Russ Cox s XZ post explains how exactly the Autotools code fetched the actual backdoor from the test files and injected it into the build.
There is only one tiny thing that maybe a very experienced Autotools user could potentially have noticed: the serial 30 in the version header is way too high. In theory one could also have noticed this Autotools file deviates from what other packages in Debian ship with the same filename, such as e.g. the serial 3, serial 5a or 5b versions. That would however require and an insane amount extra checking work, and is not something we should plan to start doing. A much simpler solution would be to simply strongly recommend all open source projects to stop using Autotools to eventually get rid of it entirely.
Not detectable with reasonable effort
While planting backdoors is evil, it is hard not to feel some respect to the level of skill and dedication of the people behind this. I ve been involved in a bunch of security breach investigations during my IT career, and never have I seen anything this well executed.
If it hadn t slowed down SSH by ~500 milliseconds and been discovered due to that, it would most likely have stayed undetected for months or years. Hiding backdoors in closed source software is relatively trivial, but hiding backdoors in plain sight in a popular open source project requires some unusual amount of expertise and creativity as shown above.
Is the software supply-chain in Debian easy to audit?
While maintaining a Debian package source using git-buildpackage can make the package history a lot easier to inspect, most packages have incomplete configurations in their debian/gbp.conf, and thus their package development histories are not always correctly constructed or uniform and easy to compare. The Debian Policy does not mandate git usage at all, and there are many important packages that are not using git at all. Additionally the Debian Policy also allows for non-maintainers to upload new versions to Debian without committing anything in git even for packages where the original maintainer wanted to use git. Uploads that bypass git unfortunately happen surpisingly often.
Because of the situation, I am afraid that we could have multiple similar backdoors lurking that simply haven t been detected yet. More audits, that hopefully also get published openly, would be welcome! More people auditing the contents of the Debian archives would probably also help surface what tools and policies Debian might be missing to make the work easier, and thus help improve the security of Debian s users, and improve trust in Debian.
Is Debian currently missing some software that could help detect similar things?
To my knowledge there is currently no system in place as part of Debian s QA or security infrastructure to verify that the upstream source packages in Debian are actually from upstream. I ve come across a lot of packages where the debian/watch or other configs are incorrect and even cases where maintainers have manually created upstream tarballs as it was easier than configuring automation to work. It is obvious that for those packages the source tarball now in Debian is not at all the same as upstream. I am not aware of any malicious cases though (if I was, I would report them of course).
I am also aware of packages in the Debian repository that are misconfigured to be of type 1.0 (native) packages, mixing the upstream files and debian/ contents and having patches applied, while they actually should be configured as 3.0 (quilt), and not hide what is the true upstream sources. Debian should extend the QA tools to scan for such things. If I find a sponsor, I might build it myself as my next major contribution to Debian.
In addition to better tooling for finding mismatches in the source code, Debian could also have better tooling for tracking in built binaries what their source files were, but solutions like Fraunhofer-AISEC s supply-graph or Sony s ESSTRA are not practical yet. Julien Malka s post about NixOS discusses the role of reproducible builds, which may help in some cases across all distros.
Or, is Debian missing some policies or practices to mitigate this?
Perhaps more importantly than more security scanning, the Debian Developer community should switch the general mindset from anyone is free to do anything to valuing having more shared workflows. The ability to audit anything is severely hampered by the fact that there are so many ways to do the same thing, and distinguishing what is a normal deviation from a malicious deviation is too hard, as the normal can basically be almost anything.
Also, as there is no documented and recommended default workflow, both those who are old and new to Debian packaging might never learn any one optimal workflow, and end up doing many steps in the packaging process in a way that kind of works, but is actually wrong or unnecessary, causing process deviations that look malicious, but turn out to just be a result of not fully understanding what would have been the right way to do something.
In the long run, once individual developers workflows are more aligned, doing code reviews will become a lot easier and smoother as the excess noise of workflow differences diminishes and reviews will feel much more productive to all participants. Debian fostering a culture of code reviews would allow us to slowly move from the current practice of mainly solo packaging work towards true collaboration forming around those code reviews.
I have been promoting increased use of Merge Requests in Debian already for some time, for example by proposing DEP-18: Encourage Continuous Integration and Merge Request based Collaboration for Debian packages. If you are involved in Debian development, please give a thumbs up in dep-team/deps!21 if you want me to continue promoting it.
Can we trust open source software?
Yes and I would argue that we can only trust open source software. There is no way to audit closed source software, and anyone using e.g. Windows or MacOS just have to trust the vendor s word when they say they have no intentional or accidental backdoors in their software. Or, when the news gets out that the systems of a closed source vendor was compromised, like Crowdstrike some weeks ago, we can t audit anything, and time after time we simply need to take their word when they say they have properly cleaned up their code base.
In theory, a vendor could give some kind of contractual or financial guarantee to its customer that there are no preventable security issues, but in practice that never happens. I am not aware of a single case of e.g. Microsoft or Oracle would have paid damages to their customers after a security flaw was found in their software. In theory you could also pay a vendor more to have them focus more effort in security, but since there is no way to verify what they did, or to get compensation when they didn t, any increased fees are likely just pocketed as increased profit.
Open source is clearly better overall. You can, if you are an individual with the time and skills, audit every step in the supply-chain, or you could as an organization make investments in open source security improvements and actually verify what changes were made and how security improved.
If your organisation is using Debian (or derivatives, such as Ubuntu) and you are interested in sponsoring my work to improve Debian, please reach out.
Debian LTS
This was my hundred-thirty-fifth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:
[DLA 4168-2] openafs regression update to fix an incomplete patch in the previous upload.
[DSA 5998-1] cups security update to fix two CVEs related to a authentication bypass and a denial of service.
[DLA 4298-1] cups security update to fix two CVEs related to a authentication bypass and a denial of service.
[DLA 4304-1] cjson security update to fix one CVE related to an out-of-bounds memory access.
[DLA 4307-1] jq security update to fix one CVE related to a heap buffer overflow.
[DLA 4308-1] corosync security update to fix one CVE related to a stack-based buffer overflow.
An upload of spim was not needed, as the corresponding CVE could be marked as ignored.
I also started to work on an open-vm-tools and attended the monthly LTS/ELTS meeting.
Debian ELTS
This month was the eighty-sixth ELTS month. During my allocated time I uploaded or worked on:
[ELA-1512-1] cups security update to fix two CVEs in Buster and Stretch, related to a authentication bypass and a denial of service.
[ELA-1520-1] jq security update to fix one CVE in Buster and Stretch, related to a heap buffer overflow.
[ELA-1524-1] corosync security update to fix one CVE in Buster and Stretch, related to a stack-based buffer overflow.
[ELA-1527-1] mplayer security update to fix ten CVEs in Stretch, distributed all over the code.
The CVEs for open-vm-tools could be marked as not-affeceted as the corresponding plugin was not yet available. I also attended the monthly LTS/ELTS meeting.
Debian Printing
This month I uploaded a new upstream version or a bugfix version of:
misc
The main topic of this month has been gcc15 and cmake4, so my upload rate was extra high. This month I uploaded a new upstream version or a bugfix version of:
I wonder what MBF will happen next, I guess the /var/lock-issue will be a good candidate.
On my fight against outdated RFPs, I closed 30 of them in September. Meanwhile only 3397 are still open, so don t hesitate to help closing one or another.
FTP master
This month I accepted 294 and rejected 28 packages. The overall number of packages that got accepted was 294.
A brand new release 0.1.15 of the RcppSimdJson
package is now on CRAN.
RcppSimdJson
wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via
very clever algorithmic engineering to obtain largely branch-free code,
coupled with modern C++ and newer compiler instructions, it results in
parsing gigabytes of JSON parsed per second which is quite
mindboggling. The best-case performance is faster than CPU speed as
use of parallel SIMD instructions and careful branch avoidance can lead
to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire
at QCon.
This version updates to the current 4.2.4 upstream release. It also
updates the 
A recent article in The Atlantic makes the case that
Team Rcpp is thrilled to share that an exciting new version 1.1.1 of
Rcpp is now on 
In 
We recently bought some 
It's done! It's over!
I've graduated, I have the scroll, I'm staring at
the eye-watering prices for the official photographer snap, I'm
adjusting to post-thesis life.
My 
The discovery of a backdoor in XZ Utils in the spring of 2024 shocked the open source community, raising critical questions about software supply chain security. This post explores whether better Debian packaging practices could have detected this threat, offering a guide to auditing packages and suggesting future improvements.
The XZ backdoor in versions 5.6.0/5.6.1 made its way briefly into many major Linux distributions such as Debian and Fedora, but luckily didn t reach that many actual users, as the backdoored releases were quickly removed thanks to the heroic diligence of 
If the changes are extensive, and you want to use a LLM to help spot potential security issues, generate the report of both the upstream and Debian packaging differences in Markdown with:
To compare changes across the new and old upstream tarball, one would need to compare commits afba662b New upstream version 5.8.0 and fa1e8796 New upstream version 5.8.1 by running 
With all the above tips you can now go and try to audit your own favorite package in Debian and see if it is identical with upstream, and if not, how it differs.
There is only one tiny thing that maybe a very experienced Autotools user could potentially have noticed: the