Daggerbound is a secondary-world fantasy and a sequel to
Swordheart, although it follows
different protagonists for most of the book. It's also a sort of sequel to
the Clocktaur War duology in that one of the
protagonists was a protagonist of those books. T. Kingfisher is the pen
name Ursula Vernon uses for her adult fiction.
Learned Edmond was once a devout acolyte of the Order of the Many-Armed
God and a true believer in its teachings, including the ones about the
evils of women. Then he went on an adventure with, among other people, a
forger named Mistress Slate, during which he learned a great deal of
things about the world that his order did not teach him. He has spent the
subsequent years trying to bring this important new information to his
order and encourage them to adjust their view of the world accordingly.
The order has not changed, and Edmond is very tired.
Someone needs to carry a number of books and a magical sword to the city
of Archon's Glory. Edmond has volunteered and intends to perform the task
exactly as expected, and then never return. He is unsure what he will do
then, but at least it will be different.
Edmond did not expect to be waylaid by bandits or to accidentally draw the
magical sword. He certainly did not expect the man who appeared in a flash
of light and violently saved him from the bandits.
It would have been very convenient for the Dervish if Edmond died of his
injuries after drawing the sword. That would leave the sword without a
wielder and therefore the Dervish without someone he had to protect. All
he would have to do was sit and watch while the fever and cold and lack of
water did the work, and then he would be free. Given his past experience
with wielders, no one could even blame him. Except maybe his own
conscience.
I did not realize I wanted to read a novel about a traumatized Mercedes
Lackey sword waking up in a T. Kingfisher universe, but apparently I did.
The Dervish is not a sword, precisely, at least not in the way that
Need has become by the time we meet her in
Lackey's novels. He's a man trapped in a sword, he doesn't like it, and he
would very much like it to stop. But at the start of the book, he has the
rest of the vibe of a Mercedes Lackey protagonist: vast past trauma, poor
emotional regulation skills, and a prickly cynicism that only runs skin
deep. Pairing him with an Edmund who is far more mature than in the
earlier duology but still rather innocent and delightfully neurodivergent
is a great touch.
The rest of the plot was fine but not that memorable. Like the Clocktaur
series, there is a strong feeling of episodic D&D adventure as the
characters stumble across disconnected problems and hostilities while
collecting party members. Like Ursula Vernon's occasional D&D recaps, this
party has a tendency to adopt everyone they come in contact with, which
includes a gnole (as always one of the best parts of the book) and a
sentient giant pill bug. The pill bug, who has a complex
appendage-operated mechanical talk box and a banner that says "I DO NOT
EAT HUMAN MEAT," is the most Ursula Vernon character that has ever existed
in fiction.
There is a romance because there is always a romance in the novels set in
this universe. I didn't mind this one. Like every Kingfisher romance, it
involves a lot of irritating refusal to communicate and endless angst
about how the other person couldn't possibly be in love with them, but the
characters have rather more justification than usual and at least I didn't
spend most of the book thinking one half of the romantic pairing was an
idiot. Apparently it helps if no paladins are involved.
The strongest parts of this book are the beginning, before the Lackey
vibes unfortunately drain out of the story, and the end, when all of the
best characters from Swordheart show up again. The Temple of the
White Rat continues to be the best feature of this fictional universe, and
I could read endless books about how they solve problems.
The Dervish gazed at her with admiration. "And here I thought you'd be
on the side of law and order..."
"No, young man," said Beartongue, despite being several centuries his
junior. "I am on the side of justice. If we're lucky, the two run in
parallel. If they don't, we bend the law to attain justice, not the
other way around."
The ending also featured some exploration of the edge cases of the rules
for the sentient swords that reminded me a bit of
Fred Saberhagen with better characterization.
Daggerbound was not as charming as Swordheart, mostly
because Halla appeared only at the end. I wish Kingfisher had leaned more
into the Mercedes Lackey vibes rather than taking her normal D&D-flavored
romance approach. But this was enjoyable, charming at times, had great
supporting characters like all Kingfisher novels, and was a satisfying
continuation of the series. I am looking forward to the third (and I
believe final) book, which is supposed to be out next year.
Recommended if you liked Swordheart.
Followed by the not-yet-published Shieldborn.
Rating: 7 out of 10
A new and exciting version of our still-new package myman reached CRAN this morning, and has been
built for r2u and on
r-universe. The
matching Python package
has also been updated. The package offers nineteen hundred eighty four
My man posts by Kevin Kruse
made on bsky during the summer of 2026.
Each wave picked at one particular public persona. This package wrapse
these up in the style of packages like fortunes or gaussfacts.
A sample usage illustration shows how to extract by pattern, and
shows posts from the two most recent waves:
>library(myman)>myman("maitre")My man looks like he's inquiring with the maitre'd about the house curly fries.-- about Howard Lutnick on 2026-08-28My man looks like a maitre'd who deeply doubts you have a reservation. -- about Scott Bessent on 2026-08-31My man looks like he's asked the maitre'd to remove a party of four he finds visuallyunpleasant. -- about Scott Bessent on 2026-08-31My man looks like the maitre'd at a very exclusive restaurant called The Berghof.-- about JD Vance on 2026-09-13My man looks like he's asking the maitre'd where they source their corn dogs.-- about Palmer Luckey on 2026-10-02My man looks like Data had to borrow a sports jacket from the maitre'd. -- about Elon Musk on 2026-10-05>
One can also subset by target , or sample randomly (which is the
default).
As noted during the initial
announcement last week, wave eight did not make it into the initial
CRAN release as it happened
while the package was under review. Waves nine and ten occured more or
less while I was out of town last weekend so this release now brings
three new waves to the CRAN package! We also added two new helper
functions to extract the underlying data frame object, and tabulate the
targetted men.
To align the version number with the count of post waves , we
switched to version 0.10.0 for this release and the corresponding Python package release so
that both implementations now have the same version number.
The NEWS entry for this release follows.
Changes in version 0.10.0
(2026-10-08)
New waves nine (100 posts) and ten (190 posts) made last week;
total is now 1984 posts
New helper functions posts() and men()
retrieving data.frame of posts and tabulation of targets
Internal post gathering and aggregation functions have been
updated and generalized
Versioning scheme now goes with post waves (also for Python
sibbling)
netip.Addr features an Unmap()
method returning the unwrapped IPv4 contained in an IPv4-mapped IPv6
address: from ::ffff:203.0.113.10 or ::ffff:cb00:710a, it returns
203.0.113.10.1 There is no Map() or To6() method for the reverse
direction. Such a method is trivial to implement, but Go maintainers have
rejected it on the grounds that users should write
netip.AddrFrom16(ip.As16()) and let the compiler optimize it.2 Today,
this pattern is eight times slower than a native method. How can we teach the
compiler to optimize this sequence?
The alternatives
Let s explore three ways to implement the map semantics for netip.Addr. My
favorite is to add it to the Go standard library. Go maintainers prefer a small
external helper chaining netip.AddrFrom16() and netip.Addr.As16(), hoping
the compiler eventually optimizes it. The unsafe package
opens a third path, with the same performance as the first solution.
Modifying the Go standard library
Internally, netip.Addr stores any IP address as a
128-bit value with an extra field z to encode the family and the zone:
typeAddrstructaddruint128zunique.Handle[addrDetail]typeaddrDetailstructisV6bool// IPv4 is false, IPv6 is true.zoneV6string// != "" only if IsV6 is true.var(z0unique.Handle[addrDetail]z4=unique.Make(addrDetail )z6noz=unique.Make(addrDetailisV6:true ))
AddrFrom4() encodes an IPv4 address as an
IPv4-mapped IPv6 address and sets z to the unique value z4:
// AddrFrom4 returns the address of the IPv4 address given by the bytes in addr.funcAddrFrom4(addr[4]byte)AddrreturnAddraddr:uint1280,0xffff00000000uint64(addr[0])<<24uint64(addr[1])<<16uint64(addr[2])<<8uint64(addr[3]) ,z:z4,
Unmap() turns an IPv4-mapped IPv6 address into an
IPv4 address by setting the z field to z4:
Implementing the reverse direction inside the Go standard library is trivial: we
set the z field to z6noz if the address is IPv4.
// To6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.func(ipAddr)To6()Addrifip.Is4()ip.z=z6nozreturnip
Update (2026-10)
Instead of patching Go, we can use the -overlay
flag of go build to hijack any package, including the standard
library. In our case, it can add To6() to net/netip. Yet
it is cumbersome:3 the flag expects a JSON file mapping absolute
paths in GOROOT to replacement files, and you need to add it to every go
command.
As a helper
We can t access the z field from outside the net/netip package. Instead, we
build a small helper around the netip.AddrFrom16(ip.As16()) pattern:
// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.funcAddrTo6(ipnetip.Addr)netip.Addrifip.Is4()ip=netip.AddrFrom16(ip.As16())returnip
As an unsafe function
Another solution uses the unsafe package to alter the Addr struct through
a proxy with the same memory layout:4
// addrProxy has the same memory layout as netip.Addr.typeaddrProxystructaddr[2]uint64// netip.uint128zunsafe.Pointer// unique.Handle[netip.addrDetail]var(anyIPv6=netip.IPv6Unspecified()netipZ6noz=(*addrProxy)(unsafe.Pointer(&anyIPv6)).z)// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.funcAddrTo6(ipnetip.Addr)netip.Addrif!ip.Is4()returnip(*addrProxy)(unsafe.Pointer(&ip)).z=netipZ6nozreturnip
Benchmarks
On my computer, with Go 1.27.1, the standard library solution costs 0.88 ns per
operation, while the solution favored by Go maintainers costs 7.14 ns. The
unsafe solution matches the performance of the first one.
Assembly code
Let s check the assembly code the compiler generates for each solution.5
The one built into the standard library looks like this:6
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Go s assembly language is not a direct representation of the underlying
machine language: it operates on a semi-abstract instruction set derived from
Plan 9 s assembler. It has four pseudo-registers: FP (frame pointer
for function arguments), PC (program counter), SB (static base pointer for
global symbols), and SP (stack pointer). It also has architecture-specific
registers like AX, CX, DX, BX, SI, DI, and R8 to R15.
Instructions storing data use their last argument as the destination.
Instructions can carry an explicit size suffix: MOVB moves a byte, MOVW 16
bits, MOVL 32 bits, and MOVQ 64 bits. In the example above, the first
instruction compares the 64-bit value z4 with the CX register.
The unsafe solution looks almost the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnetipZ6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
The helper solution has far more instructions. To understand why, let s look at
the code for As16() and
AddrFrom16(). They are short enough for the compiler
to inline them.
We can already guess the pattern to optimize: the code packs the IP address into
an array, copies it, then unpacks it. If we inline the Go code by hand, we get:
Unfortunately, as of Go 1.26.8, the compiler is not smart enough to do the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z; Push the stack (32 bytes):; 0(SP) addr netip.uint128; 16(SP) a16 [16]bytePUSHQBPMOVQSP,BPSUBQ$32,SPCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop here; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi); byteorder.BEPutUint64(a16[8:], input.addr.lo)MOVBEQAX,net/netip a16+16(SP)MOVBEQBX,net/netip a16+24(SP); addr = a16, 16 bytes at once through the vector register X0MOVUPSnet/netip a16+16(SP),X0MOVUPSX0,net/netip addr(SP); CX = netip.z6nozMOVQnet/netip z6noz(SB),CX; Unpack: output.addr.hi = byteorder.BEUint64(addr[:8]); output.addr.lo = byteorder.BEUint64(addr[8:])MOVBEQnet/netip addr(SP),AXMOVBEQnet/netip addr+8(SP),BXend:ADDQ$32,SPPOPQBPRET// return value = Addr hi: AX, lo: BX, z: CX
The compiler does a decent job on the byte shuffling: the eight byte stores of
BEPutUint64() become a single MOVBEQ, which stores a register byte-swapped.
The eight byte loads of BEUint64() become a single MOVBEQ the other way
round.7 Three groups of instructions remain: a pack, a copy, and an
unpack.
Hacking the Go compiler
The Go compiler has several phases:
The compiler converts the syntax tree and its types into its own intermediate
representation (IR). This process, called noding, goes through
a serialization format named unified IR.
This phase runs two steps: order of evaluation decomposes complex
statements into simpler ones, and desugaring transforms higher-level Go
constructs, like switch or channels, into more primitive instructions or
calls to the runtime.
The compiler rewrites the SSA form into machine-specific variants,
allocates registers, and applies more optimization passes. At the end, the
assembler turns the generated instructions into machine code.
The hammer
My first idea is to replace occurrences of netip.AddrFrom16(ip.As16()) with
netip.Addr addr: ip.addr, z: netip.z6noz as early as possible, during the
noding process. Before that, the type checking phase prevents
us from accessing unexported struct fields.
Go 1.27 introduced a convenient debug option to dump the IR of a function at
interesting points during compilation:
$ GOTOOLCHAIN=go1.27.1GOAMD64=v3gobuild-a-gcflags="-d=astdump=AddrTo6Safe".
Writing text ast output for AddrTo6Safe to AddrTo6Safe.astWriting html ast output for AddrTo6Safe to AddrTo6Safe.htmlWriting html syntax output for AddrTo6Safe to AddrTo6Safe.syntax.html
In the HTML file, the first column shows the IR as it comes out of noding:
In the body of the if statement, we spot the calls to the method
netip.Addr.As16() and to the function netip.AddrFrom16(). Our goal is to
patch them with a struct literal:
In noder s reader.go, the expr() method builds the IR tree for
an expression. At the end of the exprCall case, we add a call to a
rewriteAddrFrom16As16() function. It takes the current node and returns the
struct literal on success, or nil if the rewrite is not possible. First, we
check that we have the expected pattern: a call to the netip.AddrFrom16()
function with a call to the netip.Addr.As16() method as its only argument:
Have a look at the complete patch.8 We can test it
with the following commands:
$ cdsrc
$ ./make.bash
Building Go cmd/dist using /usr/lib/go-1.27. (go1.27.1 linux/amd64)Building Go toolchain1 and bootstrap cmd/go (go_bootstrap) using /usr/lib/go-1.27.Building Go toolchain2 using go_bootstrap and Go toolchain1.Building Go toolchain3 and commands using go_bootstrap and Go toolchain2.Checking command staleness for linux/amd64.---Installed Go for linux/amd64 in /home/bernat/code/free/goInstalled commands in /home/bernat/code/free/go/bin*** You need to add /home/bernat/code/free/go/bin to your PATH.$ exportPATH=$PWD/../bin:$PATH$ goversion
go version go1.28-devel_9834516e20 Sat Sep 12 08:23:11 2026 -0700 linux/amd64$ gotestnet/netip/...
ok net/netip 0.224s$ cd../../go-netip-addrto6
$ gotest.
ok github.com/vincentbernat/go-netip-addrto6 0.062s
The generated code for the helper is now the shortest possible version!
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Go maintainers are unlikely to accept this change. It relies on the internal
structure of net/netip.Addr. It s an ugly hack in the noder, whose job is to
faithfully translate the type-checked AST into the IR. And it s harder to
maintain than adding a To6() method.
The screwdriver
The right place for such an optimization is the generic SSA phase. One of
the last machine-independent passes is memcombine. With the appropriate debug
flag, the compiler dumps the SSA form after this pass:9
The result of the memcombine pass follows the same structure as the assembly
code for AddrTo6Safe()we looked at earlier: two stores,
one move, and two loads we would like to optimize away.
Each line features a value identifier (v442), an operation with its type
(Bswap64 <uint64>), and its arguments (v490).10 Values are the basic
building blocks of SSA and are defined exactly once. Square brackets enclose
integer parameters ([8]) and curly braces contain auxiliary arguments
( netip.addr ). Operations writing to memory produce a new memory state. Every
memory operation takes the current state as its last argument, which keeps them
in order.
To simplify this code, we could apply three rewriting rules:
The first one adds a shortcut when loading through a move: (Load (OffPtr
[o] p) (Move p src mem)) => (Load (OffPtr [o] src) mem). This matches v174
with its arguments v415 and v286 and creates a new value v600:
The second one simplifies a load following a store: (Load p (Store p x
_)) => x. The load is forwarded: the stored value replaces it and no memory
access remains. It matches v174. It notices that v600 and v173 are the
same address and replaces v174 with a copy of v442:
To optimize it away, we also apply three rewriting rules:
The first one also adds a shortcut when loading through a move, but
without an offset: (Load p (Move p src mem)) => (Load src mem). This rewrites v416
to use arguments from v286:
The second rule forwards a value stored one step earlier, skipping over a
store to another address: (Load p (Store q _ (Store p x _))) => x. This
matches v416: x is v542, p is v22 (&a16), q is v173
(&a16[8]), and p and q do not overlap for uint64.
In practice
Most of these rules already exist in generic.rules. They use
conditions to validate their context: ssa.IsSamePtr() for the same address,
ssa.Disjoint() for addresses that do not overlap. The rule forwarding a stored
value to a load already exists with three variants looking through several other
stores. Here are the two we need:
There is no generic rule to cancel two byte swaps, but the AMD64 lowering
pass includes this rule:
(BSWAP(Q L)(BSWAP(Q L)p))=>p
After switching to Go s development branch and adding the missing
rule, the generated assembly code is worse than with Go 1.26.8,
even though our additional rule slightly improves the situation at the end:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z; Push the stack (16 bytes):; 0(SP) a16 [16]bytePUSHQBPMOVQSP,BPSUBQ$16,SPCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop here; The four forwarded bytes: the low half of input.addr.lo is taken apart and; put back together in registersMOVQBX,DX; DX = input.addr.loSHRQ$24,BX; BX = input.addr.lo >> 24MOVQDX,SI; SI = input.addr.loSHRQ$16,DX; DX = input.addr.lo >> 16MOVQSI,DI; DI = input.addr.lo, kept for the packSHRQ$8,SI; SI = input.addr.lo >> 8MOVBLZXDIB,R8; R8 = byte(input.addr.lo)MOVBLZXSIB,SI; SI = byte(input.addr.lo >> 8)SHLQ$8,SIORQR8,SI; SI = two low bytes of input.addr.loMOVBLZXDL,DX; DX = byte(input.addr.lo >> 16)SHLQ$16,DXORQSI,DXMOVBLZXBL,BX; BX = byte(input.addr.lo >> 24)SHLQ$24,BXORQDX,BX; BX = input.addr.lo & 0xffffffff; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi); byteorder.BEPutUint64(a16[8:], input.addr.lo)MOVBEQAX,net/netip a16(SP)MOVBEQDI,net/netip a16+8(SP); The four other bytes of input.addr.lo, read one by one from a16MOVBLZXnet/netip a16+11(SP),DX; a16[11]SHLQ$32,DXORQDX,BXMOVBLZXnet/netip a16+10(SP),DX; a16[10]SHLQ$40,DXORQDX,BXMOVBLZXnet/netip a16+9(SP),DX; a16[9]SHLQ$48,DXORQDX,BXMOVBLZXnet/netip a16+8(SP),DX; a16[8]SHLQ$56,DX; output.z = netip.z6nozMOVQnet/netip z6noz(SB),CX; output.addr.hi = byteorder.BEUint64(a16[:8])MOVBEQnet/netip a16(SP),AX; output.addr.lo assembled from the previous stepsORQDX,BXend:LEAVEQRET// return value = Addr hi: AX, lo: BX, z: CX
The rule loading through a move, added in Go 1.27, introduced this regression.
Out of order
Let s not give up now! In reality, the rewriting rules run before memcombine,
notably in the late opt pass. At this point, the inlined versions of
BEPutUint64() and BEUint64() still expand to sixteen byte stores and sixteen
byte loads, matching their source code:
funcBEUint64(b[]byte)uint64_=b[7]// bounds check hint to compiler; see golang.org/issue/14808returnuint64(b[7])uint64(b[6])<<8uint64(b[5])<<16uint64(b[4])<<24uint64(b[3])<<32uint64(b[2])<<40uint64(b[1])<<48uint64(b[0])<<56
Let s follow two bytes of output.addr.lo: addr[15] and addr[11]. Here is
a simplified SSA form before late opt:
The first rule loads through the move: (Load (OffPtr [o] p) (Move p src mem))
=> (Load (OffPtr [o] src) mem). It matches both loads, which now read a16
with the memory state before the copy:
The second rule shortcuts a load following a store: (Load p (Store p x _)) =>
x. It matches v435, as v282 stores a16[15]. It does not match v481:
v235 stores a16[11] four stores earlier in the chain, while the variants of
this rule look through three stores at most.
The same happens to the other bytes: the rule forwards the four bytes stored
last, a16[12] to a16[15]. The twelve other loads now read a16 instead of
addr.
BEUint64() becomes a chain of Or64, each one adding a byte shifted into
place. memcombine is a pass written in Go, not a set of rewrite
rules. It starts from the last Or64 of the chain and collects up to eight
terms. If each term is a byte load, extended to 64 bits and shifted, and if the
eight loads read consecutive addresses from the same pointer with the same
memory state, it replaces the whole chain with a single 64-bit load and a byte
swap. Otherwise, it tries again with four, then two terms, and from each
intermediate Or64. Here is the loop checking each term in a simplified version
of combineLoads():
From v520, four of the eight terms are forwarded bytes, not loads from memory,
and memcombine can t combine them. It doesn t merge the four remaining loads
either, as they sit on top of the forwarded bytes.
Back in order
In summary, the rewriting rules run too early to be effective. A quick
workaround exists: run an earlier round of memcombine before late opt. After
this change, the generated code for the helper is back to the
shortest possible version:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Next steps
I think Go maintainers would reject this change because of the additional
memcombine pass. Instead, I plan to publish this blog post and bring up the
subject again as a follow-up to issue #54365. Either the sheer complexity and
the Go 1.27 regression convince the maintainers that adding a To6() method is
simpler and more efficient, or they advise me on how to move forward. Either
way, digging into this subject taught me a lot about the Go compiler!
Update (2026-10)
I opened issue #81994 to propose Addr.To6(). Give it a if you want it in Go!
IPv4-mapped IPv6 addresses let you handle only IPv6 addresses in your code
and keep conversions at a few well-defined boundaries. I use them in
Akvorado.
This is not strictly equivalent. The zero value becomes :: while it would
make more sense to leave it untouched.
I suppose Go maintainers keep it inconvenient on purpose: they don t want
random packages to patch the standard library or other packages.
To avoid catastrophic bugs when netip.Addr s layout changes, you must add
tests to detect it. This is more dangerous if you put this code in a
package. Either ask users to run the tests themselves, or detect the change
at run time and panic. Hiding this optimization behind a build tag would
make users aware of this potential trap.
I produced the assembly code with GOTOOLCHAIN=go1.26.8 GOAMD64=v3 ./go-asm
'\.asm' from the companion repository, then edited it a bit to keep this
article from turning into an endless rabbit hole. Due to an unfortunate
sequence of events, we switch between versions: Go 1.27 introduced a
regression that blurs the point of this article.
The function is short enough for the compiler to inline it, so the assembly
code may vary depending on the surrounding code.
MOVBEQ requires GOAMD64=v3, matching the x86-64-v3
microarchitecture from 2013. Otherwise, the compiler translates
BEPutUint64() to BSWAPQ+MOVQ and BEUint64() to MOVQ+BSWAPQ.
When the call is used as a statement, a struct literal alone is invalid. The
patch then turns it into _ = netip.Addr .
The compiler can also write an HTML file with the result of each pass:
$ GOSSAFUNC=AddrTo6Safe\> GOTOOLCHAIN=go1.26.8\> GOAMD64=v3gobuild-a.
dumped SSA for AddrTo6Safe,1 to ./ssa.html
Source line numbers in parentheses follow value identifiers, but I removed
them from the examples. The SSA form also features branches, but we don t
need them to understand our optimizations.
netip.Addr features an Unmap()
method returning the unwrapped IPv4 contained in an IPv4-mapped IPv6
address: from ::ffff:203.0.113.10 or ::ffff:cb00:710a, it returns
203.0.113.10.1 There is no Map() or To6() method for the reverse
direction. Such a method is trivial to implement, but Go maintainers have
rejected it on the grounds that users should write
netip.AddrFrom16(ip.As16()) and let the compiler optimize it.2 Today,
this pattern is eight times slower than a native method. How can we teach the
compiler to optimize this sequence?
The alternatives
Let s explore three ways to implement the map semantics for netip.Addr. My
favorite is to add it to the Go standard library. Go maintainers prefer a small
external helper chaining netip.AddrFrom16() and netip.Addr.As16(), hoping
the compiler eventually optimizes it. The unsafe package
opens a third path, with the same performance as the first solution.
Modifying the Go standard library
Internally, netip.Addr stores any IP address as a
128-bit value with an extra field z to encode the family and the zone:
typeAddrstructaddruint128zunique.Handle[addrDetail]typeaddrDetailstructisV6bool// IPv4 is false, IPv6 is true.zoneV6string// != "" only if IsV6 is true.var(z0unique.Handle[addrDetail]z4=unique.Make(addrDetail )z6noz=unique.Make(addrDetailisV6:true ))
AddrFrom4() encodes an IPv4 address as an
IPv4-mapped IPv6 address and sets z to the unique value z4:
// AddrFrom4 returns the address of the IPv4 address given by the bytes in addr.funcAddrFrom4(addr[4]byte)AddrreturnAddraddr:uint1280,0xffff00000000uint64(addr[0])<<24uint64(addr[1])<<16uint64(addr[2])<<8uint64(addr[3]) ,z:z4,
Unmap() turns an IPv4-mapped IPv6 address into an
IPv4 address by setting the z field to z4:
Implementing the reverse direction inside the Go standard library is trivial: we
set the z field to z6noz if the address is IPv4.
// To6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.func(ipAddr)To6()Addrifip.Is4()ip.z=z6nozreturnip
As a helper
We can t access the z field from outside the net/netip package. Instead, we
build a small helper around the netip.AddrFrom16(ip.As16()) pattern:
// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.funcAddrTo6(ipnetip.Addr)netip.Addrifip.Is4()ip=netip.AddrFrom16(ip.As16())returnip
As an unsafe function
Another solution uses the unsafe package to alter the Addr struct through
a proxy with the same memory layout:3
// addrProxy has the same memory layout as netip.Addr.typeaddrProxystructaddr[2]uint64// netip.uint128zunsafe.Pointer// unique.Handle[netip.addrDetail]var(anyIPv6=netip.IPv6Unspecified()netipZ6noz=(*addrProxy)(unsafe.Pointer(&anyIPv6)).z)// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.funcAddrTo6(ipnetip.Addr)netip.Addrif!ip.Is4()returnip(*addrProxy)(unsafe.Pointer(&ip)).z=netipZ6nozreturnip
Benchmarks
On my computer, with Go 1.27.1, the standard library solution costs 0.88 ns per
operation, while the solution favored by Go maintainers costs 7.14 ns. The
unsafe solution matches the performance of the first one.
Assembly code
Let s check the assembly code the compiler generates for each solution.4
The one built into the standard library looks like this:5
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Go s assembly language is not a direct representation of the underlying
machine language: it operates on a semi-abstract instruction set derived from
Plan 9 s assembler. It has four pseudo-registers: FP (frame pointer
for function arguments), PC (program counter), SB (static base pointer for
global symbols), and SP (stack pointer). It also has architecture-specific
registers like AX, CX, DX, BX, SI, DI, and R8 to R15.
Instructions storing data use their last argument as the destination.
Instructions can carry an explicit size suffix: MOVB moves a byte, MOVW 16
bits, MOVL 32 bits, and MOVQ 64 bits. In the example above, the first
instruction compares the 64-bit value z4 with the CX register.
The unsafe solution looks almost the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnetipZ6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
The helper solution has far more instructions. To understand why, let s look at
the code for As16() and
AddrFrom16(). They are short enough for the compiler
to inline them.
We can already guess the pattern to optimize: the code packs the IP address into
an array, copies it, then unpacks it. If we inline the Go code by hand, we get:
Unfortunately, as of Go 1.26.8, the compiler is not smart enough to do the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z; Push the stack (32 bytes):; 0(SP) addr netip.uint128; 16(SP) a16 [16]bytePUSHQBPMOVQSP,BPSUBQ$32,SPCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop here; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi); byteorder.BEPutUint64(a16[8:], input.addr.lo)MOVBEQAX,net/netip a16+16(SP)MOVBEQBX,net/netip a16+24(SP); addr = a16, 16 bytes at once through the vector register X0MOVUPSnet/netip a16+16(SP),X0MOVUPSX0,net/netip addr(SP); CX = netip.z6nozMOVQnet/netip z6noz(SB),CX; Unpack: output.addr.hi = byteorder.BEUint64(addr[:8]); output.addr.lo = byteorder.BEUint64(addr[8:])MOVBEQnet/netip addr(SP),AXMOVBEQnet/netip addr+8(SP),BXend:ADDQ$32,SPPOPQBPRET// return value = Addr hi: AX, lo: BX, z: CX
The compiler does a decent job on the byte shuffling: the eight byte stores of
BEPutUint64() become a single MOVBEQ, which stores a register byte-swapped.
The eight byte loads of BEUint64() become a single MOVBEQ the other way
round.6 Three groups of instructions remain: a pack, a copy, and an
unpack.
Hacking the Go compiler
The Go compiler has several phases:
The compiler converts the syntax tree and its types into its own intermediate
representation (IR). This process, called noding, goes through
a serialization format named unified IR.
This phase runs two steps: order of evaluation decomposes complex
statements into simpler ones, and desugaring transforms higher-level Go
constructs, like switch or channels, into more primitive instructions or
calls to the runtime.
The compiler rewrites the SSA form into machine-specific variants,
allocates registers, and applies more optimization passes. At the end, the
assembler turns the generated instructions into machine code.
The hammer
My first idea is to replace occurrences of netip.AddrFrom16(ip.As16()) with
netip.Addr addr: ip.addr, z: netip.z6noz as early as possible, during the
noding process. Before that, the type checking phase prevents
us from accessing unexported struct fields.
Go 1.27 introduced a convenient debug option to dump the IR of a function at
interesting points during compilation:
$ GOTOOLCHAIN=go1.27.1GOAMD64=v3gobuild-a-gcflags="-d=astdump=AddrTo6Safe".
Writing text ast output for AddrTo6Safe to AddrTo6Safe.astWriting html ast output for AddrTo6Safe to AddrTo6Safe.htmlWriting html syntax output for AddrTo6Safe to AddrTo6Safe.syntax.html
In the HTML file, the first column shows the IR as it comes out of noding:
In the body of the if statement, we spot the calls to the method
netip.Addr.As16() and to the function netip.AddrFrom16(). Our goal is to
patch them with a struct literal:
In noder s reader.go, the expr() method builds the IR tree for
an expression. At the end of the exprCall case, we add a call to a
rewriteAddrFrom16As16() function. It takes the current node and returns the
struct literal on success, or nil if the rewrite is not possible. First, we
check that we have the expected pattern: a call to the netip.AddrFrom16()
function with a call to the netip.Addr.As16() method as its only argument:
Have a look at the complete patch.7 We can test it
with the following commands:
$ cdsrc
$ ./make.bash
Building Go cmd/dist using /usr/lib/go-1.27. (go1.27.1 linux/amd64)Building Go toolchain1 and bootstrap cmd/go (go_bootstrap) using /usr/lib/go-1.27.Building Go toolchain2 using go_bootstrap and Go toolchain1.Building Go toolchain3 and commands using go_bootstrap and Go toolchain2.Checking command staleness for linux/amd64.---Installed Go for linux/amd64 in /home/bernat/code/free/goInstalled commands in /home/bernat/code/free/go/bin*** You need to add /home/bernat/code/free/go/bin to your PATH.$ exportPATH=$PWD/../bin:$PATH$ goversion
go version go1.28-devel_9834516e20 Sat Sep 12 08:23:11 2026 -0700 linux/amd64$ gotestnet/netip/...
ok net/netip 0.224s$ cd../../go-netip-addrto6
$ gotest.
ok github.com/vincentbernat/go-netip-addrto6 0.062s
The generated code for the helper is now the shortest possible version!
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Go maintainers are unlikely to accept this change. It relies on the internal
structure of net/netip.Addr. It s an ugly hack in the noder, whose job is to
faithfully translate the type-checked AST into the IR. And it s harder to
maintain than adding a To6() method.
The screwdriver
The right place for such an optimization is the generic SSA phase. One of
the last machine-independent passes is memcombine. With the appropriate debug
flag, the compiler dumps the SSA form after this pass:8
The result of the memcombine pass follows the same structure as the assembly
code for AddrTo6Safe()we looked at earlier: two stores,
one move, and two loads we would like to optimize away.
Each line features a value identifier (v442), an operation with its type
(Bswap64 <uint64>), and its arguments (v490).9 Values are the basic
building blocks of SSA and are defined exactly once. Square brackets enclose
integer parameters ([8]) and curly braces contain auxiliary arguments
( netip.addr ). Operations writing to memory produce a new memory state. Every
memory operation takes the current state as its last argument, which keeps them
in order.
To simplify this code, we could apply three rewriting rules:
The first one adds a shortcut when loading through a move: (Load (OffPtr
[o] p) (Move p src mem)) => (Load (OffPtr [o] src) mem). This matches v174
with its arguments v415 and v286 and creates a new value v600:
The second one simplifies a load following a store: (Load p (Store p x
_)) => x. The load is forwarded: the stored value replaces it and no memory
access remains. It matches v174. It notices that v600 and v173 are the
same address and replaces v174 with a copy of v442:
To optimize it away, we also apply three rewriting rules:
The first one also adds a shortcut when loading through a move, but
without an offset: (Load p (Move p src mem)) => (Load src mem). This rewrites v416
to use arguments from v286:
The second rule forwards a value stored one step earlier, skipping over a
store to another address: (Load p (Store q _ (Store p x _))) => x. This
matches v416: x is v542, p is v22 (&a16), q is v173
(&a16[8]), and p and q do not overlap for uint64.
In practice
Most of these rules already exist in generic.rules. They use
conditions to validate their context: ssa.IsSamePtr() for the same address,
ssa.Disjoint() for addresses that do not overlap. The rule forwarding a stored
value to a load already exists with three variants looking through several other
stores. Here are the two we need:
There is no generic rule to cancel two byte swaps, but the AMD64 lowering
pass includes this rule:
(BSWAP(Q L)(BSWAP(Q L)p))=>p
After switching to Go s development branch and adding the missing
rule, the generated assembly code is worse than with Go 1.26.8,
even though our additional rule slightly improves the situation at the end:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z; Push the stack (16 bytes):; 0(SP) a16 [16]bytePUSHQBPMOVQSP,BPSUBQ$16,SPCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop here; The four forwarded bytes: the low half of input.addr.lo is taken apart and; put back together in registersMOVQBX,DX; DX = input.addr.loSHRQ$24,BX; BX = input.addr.lo >> 24MOVQDX,SI; SI = input.addr.loSHRQ$16,DX; DX = input.addr.lo >> 16MOVQSI,DI; DI = input.addr.lo, kept for the packSHRQ$8,SI; SI = input.addr.lo >> 8MOVBLZXDIB,R8; R8 = byte(input.addr.lo)MOVBLZXSIB,SI; SI = byte(input.addr.lo >> 8)SHLQ$8,SIORQR8,SI; SI = two low bytes of input.addr.loMOVBLZXDL,DX; DX = byte(input.addr.lo >> 16)SHLQ$16,DXORQSI,DXMOVBLZXBL,BX; BX = byte(input.addr.lo >> 24)SHLQ$24,BXORQDX,BX; BX = input.addr.lo & 0xffffffff; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi); byteorder.BEPutUint64(a16[8:], input.addr.lo)MOVBEQAX,net/netip a16(SP)MOVBEQDI,net/netip a16+8(SP); The four other bytes of input.addr.lo, read one by one from a16MOVBLZXnet/netip a16+11(SP),DX; a16[11]SHLQ$32,DXORQDX,BXMOVBLZXnet/netip a16+10(SP),DX; a16[10]SHLQ$40,DXORQDX,BXMOVBLZXnet/netip a16+9(SP),DX; a16[9]SHLQ$48,DXORQDX,BXMOVBLZXnet/netip a16+8(SP),DX; a16[8]SHLQ$56,DX; output.z = netip.z6nozMOVQnet/netip z6noz(SB),CX; output.addr.hi = byteorder.BEUint64(a16[:8])MOVBEQnet/netip a16(SP),AX; output.addr.lo assembled from the previous stepsORQDX,BXend:LEAVEQRET// return value = Addr hi: AX, lo: BX, z: CX
The rule loading through a move, added in Go 1.27, introduced this regression.
Out of order
Let s not give up now! In reality, the rewriting rules run before memcombine,
notably in the late opt pass. At this point, the inlined versions of
BEPutUint64() and BEUint64() still expand to sixteen byte stores and sixteen
byte loads, matching their source code:
funcBEUint64(b[]byte)uint64_=b[7]// bounds check hint to compiler; see golang.org/issue/14808returnuint64(b[7])uint64(b[6])<<8uint64(b[5])<<16uint64(b[4])<<24uint64(b[3])<<32uint64(b[2])<<40uint64(b[1])<<48uint64(b[0])<<56
Let s follow two bytes of output.addr.lo: addr[15] and addr[11]. Here is
a simplified SSA form before late opt:
The first rule loads through the move: (Load (OffPtr [o] p) (Move p src mem))
=> (Load (OffPtr [o] src) mem). It matches both loads, which now read a16
with the memory state before the copy:
The second rule shortcuts a load following a store: (Load p (Store p x _)) =>
x. It matches v435, as v282 stores a16[15]. It does not match v481:
v235 stores a16[11] four stores earlier in the chain, while the variants of
this rule look through three stores at most.
The same happens to the other bytes: the rule forwards the four bytes stored
last, a16[12] to a16[15]. The twelve other loads now read a16 instead of
addr.
BEUint64() becomes a chain of Or64, each one adding a byte shifted into
place. memcombine is a pass written in Go, not a set of rewrite
rules. It starts from the last Or64 of the chain and collects up to eight
terms. If each term is a byte load, extended to 64 bits and shifted, and if the
eight loads read consecutive addresses from the same pointer with the same
memory state, it replaces the whole chain with a single 64-bit load and a byte
swap. Otherwise, it tries again with four, then two terms, and from each
intermediate Or64. Here is the loop checking each term in a simplified version
of combineLoads():
From v520, four of the eight terms are forwarded bytes, not loads from memory,
and memcombine can t combine them. It doesn t merge the four remaining loads
either, as they sit on top of the forwarded bytes.
Back in order
In summary, the rewriting rules run too early to be effective. A quick
workaround exists: run an earlier round of memcombine before late opt. After
this change, the generated code for the helper is back to the
shortest possible version:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Next steps
I think Go maintainers would reject this change because of the additional
memcombine pass. Instead, I plan to publish this blog post and bring up the
subject again as a follow-up to issue #54365. Either the sheer complexity and
the Go 1.27 regression convince the maintainers that adding a To6() method is
simpler and more efficient, or they advise me on how to move forward. Either
way, digging into this subject taught me a lot about the Go compiler!
IPv4-mapped IPv6 addresses let you handle only IPv6 addresses in your code
and keep conversions at a few well-defined boundaries. I use them in
Akvorado.
This is not strictly equivalent. The zero value becomes :: while it would
make more sense to leave it untouched.
To avoid catastrophic bugs when netip.Addr s layout changes, you must add
tests to detect it. This is more dangerous if you put this code in a
package. Either ask users to run the tests themselves, or detect the change
at run time and panic. Hiding this optimization behind a build tag would
make users aware of this potential trap.
I produced the assembly code with GOTOOLCHAIN=go1.26.8 GOAMD64=v3 ./go-asm
'\.asm' from the companion repository, then edited it a bit to keep this
article from turning into an endless rabbit hole. Due to an unfortunate
sequence of events, we switch between versions: Go 1.27 introduced a
regression that blurs the point of this article.
The function is short enough for the compiler to inline it, so the assembly
code may vary depending on the surrounding code.
MOVBEQ requires GOAMD64=v3, matching the x86-64-v3
microarchitecture from 2013. Otherwise, the compiler translates
BEPutUint64() to BSWAPQ+MOVQ and BEUint64() to MOVQ+BSWAPQ.
When the call is used as a statement, a struct literal alone is invalid. The
patch then turns it into _ = netip.Addr .
The compiler can also write an HTML file with the result of each pass:
$ GOSSAFUNC=AddrTo6Safe\> GOTOOLCHAIN=go1.26.8\> GOAMD64=v3gobuild-a.
dumped SSA for AddrTo6Safe,1 to ./ssa.html
Source line numbers in parentheses follow value identifiers, but I removed
them from the examples. The SSA form also features branches, but we don t
need them to understand our optimizations.
One of the primary design principles for Linus Torvalds when creating Git is that it should be a distributed system, capable of working with multiple remotes. However, most people seem to use Git with just one remote, typically having GitHub or GitLab as the only origin.
Version control systems before Git relied on a centralized model. For example in Subversion or CVS you could only sync with one single remote server, which was considered the primary host of the source code. Git however is not just a VCS, but a DVCS, for distributed version control system. If you currently use Git with just one single remote as if it was a centralized version control system, now is the time to change that and learn how to add multiple remotes, and configure more flexible git pull and git push rules.
The first concept to grasp is that Git can have multiple remotes. For example, after running a basic clone command such as git clone https://github.com/ottok/debcraft.git you would end up with a single remote origin pointing to GitHub:
You can push and pull from these individually, or from all at once:
$ git pull --all
Fetching github
Fetching gitlab
Already up to date.
$ git pull --all
Fetching github
Fetching gitlab
Already up to date.
When you review the git history, note the labels gitlab/main and github/main which tell which commit the branch main points to on various remotes. In this case they are all in sync and point to the same 5d1815c:
$ git log --oneline
5d1815c (HEAD -> main, gitlab/main, github/main) Bump Debhelper version from 13 to 14
ca518a7 Temporarily disable automatic copyright year bumping due to frequent bugs
d6e201e Exclude .pm from codespell to decrease false positives (Closes: #1140488)
708ee35 Exclude debian/copyright from codespell due to frequent false positives
$ git log --oneline
5d1815c (HEAD -> main, gitlab/main, github/main) Bump Debhelper version from 13 to 14
ca518a7 Temporarily disable automatic copyright year bumping due to frequent bugs
d6e201e Exclude .pm from codespell to decrease false positives (Closes: #1140488)
708ee35 Exclude debian/copyright from codespell due to frequent false positives
When pushing, you can choose what remote to push to. For example, if I am working on branch dev and I only want to push it to GitLab to trigger a CI run there, I can run git push gitlab dev. You can also configure a default remote by running once git push --set-upstream gitlab dev and thereafter simply run git push and it will automatically push the branch dev only to that remote.
A related setting is remote.pushDefault, which tells git which remote to push to when a branch does not define one of its own, and it applies to every branch that lacks a [branch] section in .git/config. Setting it once is what lets a repository have a per-branch pull remote, like the debcraft example below, and still have a single, predictable git push target. To see which remote git will use for each of your branches, run git branch -vv, and to refresh the remote-tracking branches of all remotes at once, run git fetch --all --prune.
If you want to have multiple remotes, but automatically push and pull to all of them at once, you configure one single remote to have multiple URLs with git remote set-url --add. Since git prints one line per push URL, git remote -v then shows the same remote several times, once for each URL it pushes to.
As an illustration, in my actual local Debcraft development project I have all these configured:
$ git remote -v
github-pull-requests git@github.com:ottok/debcraft.git (fetch)
github-pull-requests git@github.com:ottok/debcraft.git (push)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (fetch)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (push)
origin git@salsa.debian.org:debian/debcraft.git (fetch)
origin git@salsa.debian.org:debian/debcraft.git (push)
origin git@gitlab.com:ottok/debcraft.git (push)
origin git@github.com:ottok/debcraft.git (push)
otto git@salsa.debian.org:otto/debcraft.git (fetch)
otto git@salsa.debian.org:otto/debcraft.git (push)
otto git@gitlab.com:ottok/debcraft.git (push)
otto git@github.com:ottok/debcraft.git (push)
otto git@git.sr.ht:~ottok/debcraft (push)
otto ssh://git@codeberg.org/ottok/Debcraft.git (push)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (fetch)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (push)
$ git remote -v
github-pull-requests git@github.com:ottok/debcraft.git (fetch)
github-pull-requests git@github.com:ottok/debcraft.git (push)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (fetch)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (push)
origin git@salsa.debian.org:debian/debcraft.git (fetch)
origin git@salsa.debian.org:debian/debcraft.git (push)
origin git@gitlab.com:ottok/debcraft.git (push)
origin git@github.com:ottok/debcraft.git (push)
otto git@salsa.debian.org:otto/debcraft.git (fetch)
otto git@salsa.debian.org:otto/debcraft.git (push)
otto git@gitlab.com:ottok/debcraft.git (push)
otto git@github.com:ottok/debcraft.git (push)
otto git@git.sr.ht:~ottok/debcraft (push)
otto ssh://git@codeberg.org/ottok/Debcraft.git (push)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (fetch)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (push)
The settings can also be viewed directly in the config file:
Use cases for distributed version control
The most obvious use case for any distributed system is improved availability. In this example, the salsa.debian.org is often overloaded/unavailable, so pushing to GitLab and GitHub as well allows collaborators to use them to fetch git commits if the primary code hosting site is unresponsive. Once it comes back online again, the next git push/pull will automatically bring it up to the same content as what backup hosts had.
The second main benefit is that having the code live on multiple code forges allows collaborators to use their own favorite host. Debcraft itself is a great example of this, as collaborators who don t have a Salsa account can - and have - submitted Merge Requests on GitLab.com and Pull Requests on GitHub.com.
A third use case is code hosting migrations. If you for example want to gradually phase out GitHub.com and replace it with something else, you could configure git to pull from the old location and push to both, ensuring that other collaborators can start using the new code hosting location without disruptions to those who have not yet updated their remotes or done a fresh git clone from the new location.
Automating setting up a project on multiple code forges
While I don t push all projects I work on to all Salsa/GitLab/GitHub, I do set up multiple remotes for enough git projects that I have this script to automate configuring a git repository that is cloned from Salsa to also push to GitLab and GitHub. It accepts the repository whether it was cloned over SSH or HTTPS, and it adds the push URLs and then pushes the current branch, all branches, and tags for you:
#!/bin/bash
# Add GitLab and GitHub push URLs to an existing git remote, so that a plain
# "git push" writes the same commits to all three code forges.
#
# Usage: ./git-multipush-salsa.sh
#
# Environment:
# SALSA_REMOTE remote to turn into a mirror, default origin
# SALSA_USER your Salsa user, only used to detect that the source
# remote is somebody else's project
# GITLAB_USER your GitLab user, default ottok
# GITHUB_USER your GitHub user, default ottok
# PROTOCOL ssh (default) or https, for the push URLs
set -euo pipefail
SCRIPT=$(basename "$0")
SALSA_REMOTE=$ SALSA_REMOTE:-origin
SALSA_USER=$ SALSA_USER:-otto
GITLAB_USER=$ GITLAB_USER:-ottok
GITHUB_USER=$ GITHUB_USER:-ottok
PROTOCOL=$ PROTOCOL:-ssh
die() echo "$SCRIPT: $*" >&2; exit 1;
# A failing command leaves the push URLs half configured behind, so say how
# to get back to where we started
fail()
echo "$SCRIPT: '$*' failed, the push URLs may already have been added" >&2
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl" >&2
echo " git remote set-url --add --push $ SALSA_REMOTE $ SALSA_URL " >&2
exit 1
git rev-parse --git-dir > /dev/null 2>&1 die "not inside a git repository"
git remote grep -qx "$SALSA_REMOTE" die "no remote named '$SALSA_REMOTE'"
# Example: git@salsa.debian.org:otto/salsa-ci-pipeline.git
SALSA_URL=$(git remote get-url "$SALSA_REMOTE")
# The source remote may use any of the URL forms salsa.debian.org accepts,
# for example git@salsa.debian.org:otto/salsa-ci-pipeline.git or
# https://salsa.debian.org/otto/salsa-ci-pipeline.git
case "$SALSA_URL" in
git@salsa.debian.org:*) SALSA_PATH=$ SALSA_URL#git@salsa.debian.org: ;;
ssh://git@salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#ssh://git@salsa.debian.org/ ;;
https://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#https://salsa.debian.org/ ;;
http://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#http://salsa.debian.org/ ;;
*) die "unexpected source URL '$SALSA_URL', expected a salsa.debian.org URL" ;;
esac
SALSA_PATH=$ SALSA_PATH%/
SALSA_PATH=$ SALSA_PATH%.git
SOURCE_USER=$ SALSA_PATH%%/*
PROJECT=$ SALSA_PATH##*/
# Write the push URLs in one and the same protocol, no matter how the
# repository happens to have been cloned
if [ "$PROTOCOL" = "https" ]
then
SALSA_URL="https://salsa.debian.org/$ SALSA_PATH .git"
GITLAB_URL="https://gitlab.com/$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="https://github.com/$ GITHUB_USER /$ PROJECT .git"
else
SALSA_URL="git@salsa.debian.org:$ SALSA_PATH .git"
GITLAB_URL="git@gitlab.com:$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="git@github.com:$ GITHUB_USER /$ PROJECT .git"
fi
# Mirroring somebody else's project is fine, but it is usually not what you
# want: you most likely want to mirror your own fork instead
if [ "$SOURCE_USER" != "$SALSA_USER" ]
then
echo "Note: $SALSA_REMOTE points to $SOURCE_USER's project, not your own."
echo "To mirror your own fork, run this with SALSA_REMOTE set to its name."
echo
fi
# Bail out instead of adding the same push URLs twice
EXISTING=$(git remote get-url --push --all "$SALSA_REMOTE")
if [ "$(printf '%s\n' "$EXISTING" wc -l)" -gt 1 ]
then
die "remote '$SALSA_REMOTE' already has multiple push URLs:
$EXISTING"
fi
echo "Mirroring $SALSA_URL to $GITLAB_URL and $GITHUB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$SALSA_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITLAB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITHUB_URL"
# Push the branch you have checked out first, and never with --force: the
# first branch a GitLab or GitHub project receives becomes its default branch,
# the one you then protect
DEFAULT_BRANCH=$(git symbolic-ref --short HEAD) die "not on a branch, check out a branch first"
git push -u "$SALSA_REMOTE" "$DEFAULT_BRANCH"
git push "$SALSA_REMOTE" --all
git push "$SALSA_REMOTE" --tags
# The push above creates the GitLab and GitHub projects, but GitLab projects
# are private by default, so make the new mirror public with glab if installed.
if command -v glab > /dev/null
then
echo "Make the GitLab project public and describe it as a mirror:"
glab api -X PUT "projects/$ GITLAB_USER %2F$ PROJECT " \
-f visibility=public \
-f description="Mirror of $ SALSA_URL " \
-f notification_email=disabled
else
echo "Install glab to do the same automatically, or visit"
echo "https://gitlab.com/$ GITLAB_USER /$ PROJECT /edit to make the project public"
fi
echo "To undo, drop all push URLs and add back only the original one:"
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl"
echo " git remote set-url --add --push $ SALSA_REMOTE $ SALSA_URL "
#!/bin/bash
# Add GitLab and GitHub push URLs to an existing git remote, so that a plain
# "git push" writes the same commits to all three code forges.
#
# Usage: ./git-multipush-salsa.sh
#
# Environment:
# SALSA_REMOTE remote to turn into a mirror, default origin
# SALSA_USER your Salsa user, only used to detect that the source
# remote is somebody else's project
# GITLAB_USER your GitLab user, default ottok
# GITHUB_USER your GitHub user, default ottok
# PROTOCOL ssh (default) or https, for the push URLs
set -euo pipefail
SCRIPT=$(basename "$0")
SALSA_REMOTE=$ SALSA_REMOTE:-origin
SALSA_USER=$ SALSA_USER:-otto
GITLAB_USER=$ GITLAB_USER:-ottok
GITHUB_USER=$ GITHUB_USER:-ottok
PROTOCOL=$ PROTOCOL:-ssh
die() echo "$SCRIPT: $*" >&2; exit 1;
# A failing command leaves the push URLs half configured behind, so say how
# to get back to where we started
fail()
echo "$SCRIPT: '$*' failed, the push URLs may already have been added" >&2
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl" >&2
echo " git remote set-url --add --push $ SALSA_REMOTE $ SALSA_URL " >&2
exit 1
git rev-parse --git-dir > /dev/null 2>&1 die "not inside a git repository"
git remote grep -qx "$SALSA_REMOTE" die "no remote named '$SALSA_REMOTE'"
# Example: git@salsa.debian.org:otto/salsa-ci-pipeline.git
SALSA_URL=$(git remote get-url "$SALSA_REMOTE")
# The source remote may use any of the URL forms salsa.debian.org accepts,
# for example git@salsa.debian.org:otto/salsa-ci-pipeline.git or
# https://salsa.debian.org/otto/salsa-ci-pipeline.git
case "$SALSA_URL" in
git@salsa.debian.org:*) SALSA_PATH=$ SALSA_URL#git@salsa.debian.org: ;;
ssh://git@salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#ssh://git@salsa.debian.org/ ;;
https://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#https://salsa.debian.org/ ;;
http://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#http://salsa.debian.org/ ;;
*) die "unexpected source URL '$SALSA_URL', expected a salsa.debian.org URL" ;;
esac
SALSA_PATH=$ SALSA_PATH%/
SALSA_PATH=$ SALSA_PATH%.git
SOURCE_USER=$ SALSA_PATH%%/*
PROJECT=$ SALSA_PATH##*/
# Write the push URLs in one and the same protocol, no matter how the
# repository happens to have been cloned
if [ "$PROTOCOL" = "https" ]
then
SALSA_URL="https://salsa.debian.org/$ SALSA_PATH .git"
GITLAB_URL="https://gitlab.com/$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="https://github.com/$ GITHUB_USER /$ PROJECT .git"
else
SALSA_URL="git@salsa.debian.org:$ SALSA_PATH .git"
GITLAB_URL="git@gitlab.com:$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="git@github.com:$ GITHUB_USER /$ PROJECT .git"
fi
# Mirroring somebody else's project is fine, but it is usually not what you
# want: you most likely want to mirror your own fork instead
if [ "$SOURCE_USER" != "$SALSA_USER" ]
then
echo "Note: $SALSA_REMOTE points to $SOURCE_USER's project, not your own."
echo "To mirror your own fork, run this with SALSA_REMOTE set to its name."
echo
fi
# Bail out instead of adding the same push URLs twice
EXISTING=$(git remote get-url --push --all "$SALSA_REMOTE")
if [ "$(printf '%s\n' "$EXISTING" wc -l)" -gt 1 ]
then
die "remote '$SALSA_REMOTE' already has multiple push URLs:
$EXISTING"
fi
echo "Mirroring $SALSA_URL to $GITLAB_URL and $GITHUB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$SALSA_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITLAB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITHUB_URL"
# Push the branch you have checked out first, and never with --force: the
# first branch a GitLab or GitHub project receives becomes its default branch,
# the one you then protect
DEFAULT_BRANCH=$(git symbolic-ref --short HEAD) die "not on a branch, check out a branch first"
git push -u "$SALSA_REMOTE" "$DEFAULT_BRANCH"
git push "$SALSA_REMOTE" --all
git push "$SALSA_REMOTE" --tags
# The push above creates the GitLab and GitHub projects, but GitLab projects
# are private by default, so make the new mirror public with glab if installed.
if command -v glab > /dev/null
then
echo "Make the GitLab project public and describe it as a mirror:"
glab api -X PUT "projects/$ GITLAB_USER %2F$ PROJECT " \
-f visibility=public \
-f description="Mirror of $ SALSA_URL " \
-f notification_email=disabled
else
echo "Install glab to do the same automatically, or visit"
echo "https://gitlab.com/$ GITLAB_USER /$ PROJECT /edit to make the project public"
fi
echo "To undo, drop all push URLs and add back only the original one:"
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl"
echo " git remote set-url --add --push $ SALSA_REMOTE $ SALSA_URL "
If you prefer not to install glab, the last part simply reduces to visiting the GitLab project settings once and ticking the project public, and it is also worth setting a description like Mirror of https://salsa.debian.org/debian/debcraft and disabling notification email there, as nobody wants a mirror sending out a flood of emails every time you push.
There are a few caveats worth knowing about before mirroring everything everywhere. Every push to a remote with multiple push URLs writes to all of them, so a project with continuous integration configured on three forges will run its CI three times and cost you roughly three times as much. If one of the push URLs fails, say because the host is momentarily unreachable, git prints the error and stops there, leaving the remotes listed after the failing one without the new commits. Simply re-run the push once the host is reachable again; the remotes that did get the commits will just respond with Everything up-to-date. And you should never force-push to only one of the remotes, as that leaves the mirrors permanently out of sync with your local repository.
Avoid pulling too much
Related to multiple remotes, you should also be aware that git allows one to configure which branches to pull. By default a git clone will track all branches from the remote and pull everything, which in a typical open source project will bring in a lot of temporary development branches in vain.
Thus it is better to run git clone with the --single-branch and --branch parameters to fetch only the branch that matters, which in most cases is the main branch. That can be accomplished with:
git clone --single-branch --branch main https://github.com/ottok/debcraft.git
git clone --single-branch --branch main https://github.com/ottok/debcraft.git
If a repository was already cloned, it can be configured to track only one branch with:
For additional details about any of the tips in this post I recommend reading the git man pages.
Now go and spread your code around
Distributing your code over multiple code forges costs you almost nothing in configuration, and in return you get availability, a choice of host for your collaborators, and freedom to migrate without a big bang. Add the push URLs to the remotes you already have, push once, and you are done.
The rest of the git fundamentals are covered in my other posts. If your commits are not yet polished, start with what makes a good git commit message and git commit messages by example. If you work on Debian packaging, Git-based collaboration in Debian explains why the Salsa merge request workflow is built the way it is, and Salsa merge request best practices covers the review process itself.
In 2016, I removed Google Analytics from this blog to avoid being
complicit in feeding the biggest machine for harvesting personal data. Instead,
I relied on GoAccess to analyze my server logs.1 For the past couple
of years, the statistics have made no sense, despite my attempts to filter bots:
AI scrapers inflate the number of visitors to around 2,000 per
day. Eventually, I settled on GoatCounter, an open-source, privacy-friendly
web analytics platform. I replaced the JavaScript client to filter bots more
aggressively and added a CSS fallback. To improve reliability, I implemented a
local proxy running on each of the five web servers serving this blog. The rest
of this post details how these pieces fit together and how I deploy them on
NixOS.
Why GoatCounter?
GoatCounter does not collect personal data: instead of storing the reader s
IP address or relying on cookies, it creates a session identifier valid for 8
hours from the user agent and the IP address. Its feature set is modest but
sufficient for a blog. If you want to look at the interface, GoatCounter s
author runs a public instance for his site. A hosted
version lets you try it before running your own instance. With a single binary
and an SQLite database, GoatCounter is one of the lightest self-hosted
solutions. Privacy-friendly alternatives, in increasing order of complexity,
include Umami, Plausible, and Rybbit.
Custom JavaScript client
GoatCounter includes a small JavaScript client 2,189 bytes minified and
gzipped. It ships some features I don t use: a visitor counter, tracking clicks,
configurable settings, etc. I replace it with this function to register a hit:
If a reader has disabled JavaScript in their browser, I record the hit using a
CSS image. The :hover pseudo-class loads it only after an interaction, another
trick stolen from Bear Blog. About 2% of my visitors fit into this
bucket.3
<!DOCTYPE html><htmllang="en"class="nojs"><head><script>// The JavaScript code for this blog requires ES6if("noModule"inHTMLScriptElement.prototype)document.documentElement.classList.remove("nojs");</script></head><body><!-- ... --><style>.nojsbody:hoverborder-width:0;border-image:url('/count?p=/en/blog/2026-kpi-goodhart&t=Building...&r=NoJS&e=false');</style></body></html>
Where GoAccess reported around 2,000 visitors a day, GoatCounter counts fewer
than 200 humans.4 I assume AI scrapers use a low-effort approach: if the
content is available without barriers, as on this blog, they don t spawn a
complex mechanized browser that could trigger a page view. Even crawlers running
JavaScript, like Googlebot with its headless Chromium, do not
interact with the page and never trigger the events I listen
to. The interaction-based proof of humanity I use is likely to keep
working.5
Local proxy
Five servers across the world in Europe and in North
America serve the content of this website, but GoatCounter runs on only
one of them. To avoid losing track of visitors when GoatCounter is down, I run a
local proxy listening on the same /count endpoint. On each server, it stores
the hits in memory with a buffer large enough to survive several days of
downtime. It sends them in batches to the upstream backend using the
/api/v0/count authenticated endpoint.
Servers on a map. web02 is in Paris, web03 in Helsinki, web04 in Nuremberg,
web05 in Ashburn, web06 in Chicago.
I proposed the code for the proxy in pull request #909. GoatCounter s
maintainer declined to maintain so much code for such a niche use case. As a
fellow open-source developer, I often hold the same position for my own
projects: a one-time contributor effort may translate into a long-term
maintainer commitment.
I expose the endpoint for the proxy on the domain of this website to evade ad
blockers. This sounds like I don t respect the reader s choice, but as
GoatCounter is privacy-friendly, I find it acceptable.
Deploying on NixOS
My web servers run NixOS, a declarative Linux distribution with built-in
configuration management. I manage this small fleet with Colmena, a
stateless deployment tool for NixOS. My configuration is available on
GitHub.
Deploying applications in containers
For better isolation, each application runs inside an ephemeral lightweight
container, powered by systemd-nspawn. Each container runs a stripped-down
NixOS instance. A module wraps NixOS s containers options to
avoid repeating the same options for each application.6 The containers
share their network namespace with the host: the additional isolation is not
worth the increased complexity. For a smaller footprint, I also disable a few
non-essential services.
To configure a GoatCounter instance running in a container and listening on
127.0.0.4:8088, we import the module7 and declare the container in the
config.luffy.containers attribute set:
As the containers are ephemeral, we need to keep persistent data in directories
on the host. We add a mounts option and ask NixOS s containers to expose the
configured directories through the bindMounts option.
For example, to persist GoatCounter s database in the /var/db/goatcounter
directory on the host, we add the directory to the mounts option and alter the
service definition to tell GoatCounter where the database is.
A container may also need some secrets. Colmena can upload secrets without
storing them in the Nix store. We add a keys option to our containers. It
takes an attribute set mapping secret names to the commands to populate them.
Then, the module declares the required secrets to Colmena in the
deployment.keys option, makes the container depend on the presence of the
secrets, and exposes them to the container.
config, lib,... :letcfg= config.luffy.containers;in
options.luffy.containers= lib.mkOption type= lib.types.attrsOf (lib.types.submodule options=keys= lib.mkOption type= lib.types.attrsOf (lib.types.listOf lib.types.str);default= ;description="Secrets, as a command to run locally. They are mounted in /etc."; ; ; ); ;config=# Colmena uploads each secret in /var/keys and make them available# to the group "keys".
deployment.keys= lib.concatMapAttrs
(_: container: lib.mapAttrs
(_: keyCommand:inherit keyCommand;group="keys";permissions="0640";destDir="/var/keys"; )
container.keys)
cfg;# The container can only start if the required secrets are available.
systemd.services= lib.mapAttrs'
(name: container:letunits=map(key:"$ key-key.service")(lib.attrNames container.keys);in
lib.nameValuePair "container@$ name"requires= units;after= units; )
cfg;# Mount each secret inside the container.containers= lib.mapAttrs
(name: container:bindMounts= lib.mapAttrs'
(key: _: lib.nameValuePair "/etc/$ key"hostPath="/var/keys/$ key";isReadOnly=true; )
container.keys; )
cfg; ;
For example, GoatCounter needs credentials to download the GeoIP database. I
provide a local command to fetch the secret from my password manager and expose
it inside the container through the /etc/goatcounter.env environment file.
GoatCounter server
Nixpkgs already packages GoatCounter. By overriding the src and vendorHash
attributes, I reuse its definition for my custom version with the proxy:
I wrote a NixOS module to encapsulate GoatCounter: the
container definition, the service definition, and the secrets. The module
accepts the following options: package, serve.enable, serve.listenAddress,
serve.port, and serve.databaseFile. I already detailed the container
configuration in the previous section. In the end, I chose not to reuse the
GoatCounter module from NixOS: it s small, so it s better to insulate my module
from unexpected future changes.
config, pkgs, lib,... :letcfg= config.luffy.goatcounter;databaseDirectory=builtins.dirOf cfg.serve.databaseFile;chown="$ pkgs.coreutils/bin/chown -R";in
config.luffy.containers.goatcounter=
config.systemd.services.goatcounter=description="GoatCounter Web Analytics";wantedBy=["multi-user.target"];serviceConfig=EnvironmentFile="/etc/goatcounter.env";SupplementaryGroups=["keys"];DynamicUser=true;Restart="always";ExecStart= lib.escapeShellArgs [(lib.getExe cfg.package)"serve""-listen=$ cfg.serve.listenAddress:$ toString cfg.serve.port""-tls=none""-db=sqlite+$ cfg.serve.databaseFile""-automigrate"];# Transfer database ownership to dynamically assigned user "goatcounter".ExecStartPre="+$ chown goatcounter:goatcounter $ databaseDirectory";ReadWritePaths= databaseDirectory; ; ; ;
The following snippet configures GoatCounter to listen on 127.0.0.4:8088:
GoatCounter proxy
The same NixOS module configures the local proxy, with the
following options: proxy.enable, proxy.listenAddress, proxy.port, and
proxy.site the site receiving the batches of page views. The local proxy has
no persistent data, but it needs the API key to authenticate to the main
GoatCounter instance: its container uses the keys option but not the mounts
option.
config, pkgs, lib,... :letcfg= config.luffy.goatcounter;keyCommand= _:[" "];in
config.luffy.containers.goatcounter-proxy= keys."goatcounter-proxy.env"= keyCommand "GOATCOUNTER_API_KEY"; config.systemd.services.goatcounter=description="GoatCounter Proxy.";wantedBy=["multi-user.target"];serviceConfig=EnvironmentFile="/etc/goatcounter-proxy.env";SupplementaryGroups=["keys"];DynamicUser=true;Restart="always";ExecStart= lib.escapeShellArgs [(lib.getExe cfg.package)"proxy""-site=$ cfg.proxy.site""-listen=$ cfg.proxy.listenAddress:$ toString cfg.proxy.port""-ratelimit=10/1"# 10 requests per second per IP]; ; ; ;
For each server, I enable the local proxy with the following snippet. The
nginx configuration shown earlier exposes the /count endpoint under the same
domain as my blog.
Backup of the SQLite database with Litestream
Litestream is a streaming replication tool for SQLite databases. It
compresses the changes committed to the write-ahead log (WAL) next to the
database and sends them to a remote destination. I encapsulate its configuration
in a NixOS module, which takes an attribute set databases
mapping a name to the path of the database to back up.
Litestream also runs in a container. I mount the databases to replicate, as well
as the secrets to push the backups to a Hetzner storage box using SFTP:
We can restore the database from the backup with a few shell commands. First, we
stop the containers. Then, we move the damaged database away, invoke litestream
restore from the right environment, and restart the containers.9
Ten years after removing Google Analytics, JavaScript-based analytics is
back on this blog, but without storing cookies or IP addresses, and without
involving a third party. I still write for myself first, notably because it lets
me dig into a topic and refer back to it years later. But knowing a bit more
about my fellow human readers is a nice bonus, even the ones disabling
JavaScript.
Nginx scrambles IP addresses before storing them, thanks to the ipscrub
nginx module.
GoatCounter already filters some bots based on the user agent or the IP
address. But AI scrapers lie about their user agent and hide behind
residential proxies.
Without JavaScript, I cannot send the real referrer. I insert NoJS
instead.
I am missing the humans reading the RSS feed. I am not comfortable adding a
tracking pixel, and bots are likely to fetch it, compromising the
statistics. I ll live without counting these readers.
In February 2025, I tested another CSS-only approach without the interaction
trick: each page loaded an empty image as a background. Over a month, it
recorded up to ten times fewer page views than the nginx logs, even after
removing user agents identifying as bots. Compared to this experiment,
GoatCounter counts about three times fewer views, 18 months later. As the
time ranges differ, this is not an apples-to-apples comparison, but it gives
a hint about how efficient bot filtering is.
A NixOS module is a function receiving the configuration of the whole
system as config and returning three attributes:
imports adds other modules to import,
options declares user-configurable settings, each with a type and a default value, and
config sets values for options declared by any module, like containers from NixOS.
If a module does not need to declare options, you can return the config attribute
set directly. When a module does not require any argument, you can define it as an attribute
set instead.
Most of the time, you don t need an explicit import. NixOS automatically
imports the modules shipped with Nixpkgs. For my own
modules, some machinery also imports them automatically.
Litestream warns against using the auto-recover option as
it can cause data loss. But I don t properly monitor my servers and I prefer
uninterrupted backups to a slight chance of losing the last few records.
In my case, the process is slow: around 20 minutes for a 20 MiB database.
You can test by restoring to a copy with the -o option, but you still need
to stop the Litestream container.
As a graduate of a liberal arts university, I wound up, unsurprisingly, taking a
lot of classes in every possible academic discipline. Thinking back to the
person that I was going into university, I don t think I would have chosen to
take them after all, my degree was in the sciences; I d have been stoked to
do nothing more than wall-to-wall computer science until I ran out of
coursework and filled the rest of my hours with research and independent
studies with my professors (which, I guess I did actually do, just not as
much as I would have otherwise).
Even this blog s CSS theme (now 18 years old and starting to look it) was
something I wrote after receiving and repeatedly re-reading a tattered
third-hand copy of The Laws of Thought by George Boole. It was gifted to me
by a college friend majoring in philosophy while I was crashing at his rental
on the beach. He gave it to me because he knew I liked that shit and, while
it was definitely computer science in nature, I would not have read it
otherwise. I have vivid memories of sleeping on his la-z-boy surrounded by
towers of books he was working through. He went on to do incredible work,
getting his PhD, doing research, brilliant writing his death in 2023 has
robbed humanity of more time with him. The Laws of Thought sits behind me in
my office, and is one of my most valued possessions.
My friends mean more to me than I could ever express, and I definitely don t
show it enough. Each of my classes made me a more thoughtful person. My
professors made me a better, more well-rounded person and a person who
attempts to live up to the oft repeated credo of being men and women for
others . I did the best I could, even though I was never a particularly good
student. Without liberal arts, I don t think I would, dispositionally, have
been capable of pushing myself to this day to continue to learn on nights
and weekends, for no reason other than wanting to learn. I refuse to stop
expanding my perspective, and do my best to approach new problems with as much
humility and curiosity as I can muster.
Every few days for the last year or so, I have been thinking back to a reading
assignment from my junior year that, at the time, I thought was a borderline
throwaway filler assignment for the class. The reading is an essay from 1947 by
the french existentialist philosopher and libertarian marxist, Jean-Paul Sartre
(as some of the more erudite may have now already worked out given the blog s
title), Why Write? . I re-read it this week. It is not filler. I remember this
essay better than some of the classwork I considered more important at the
time.
Why Write?
Why Write? starts off by describing the ways in which writing trying to
communicate your thoughts, opinions, or feelings to others is an act of
projection. The writer will only ever draw from a place of their own
subjectivity writing is taking your person and putting it on display.
You re choosing what words to use, how to use them, pulling from knowledge
you ve accumulated (based on how you ve chosen to spend your time). Spending
any amount of your finite existence in order to convey a thought is, itself,
announcing to the world that you believe it to be a thought worth sharing.
Meanwhile, the act of reading is not simply turning letters into words.
Reading is engaging with the work, and understanding the work in a process
that looks more like what Sartre terms re-invention or discovery the
literary object, though realized through language, is never given in
language . It is not enough to read the words in a book end-to-end; you must,
as a reader, actively engage with the work to understand what is being
communicated by those words. Reading is to take the words on the page, and
exceed the mere words through what he calls directed creation . The reader
is re-inventing / discovering the writer s thoughts by following their
landmarks in the void .
And this all makes pretty good sense to me I know if someone is being
sarcastic because I know the writer; I have read their words and have read
into their words, allowing myself to be directed by the writer into
discovering the thought they have left me in their work. I understand that
their words are humorous, not irate. Knowing who the author of a work is
can completely change the point of a sentence. The fact I m writing about
Sartre at all, or that this very writing has been constrained for presentation
in this blog s CSS is only happening because of who I am, what things I ve
experienced in life, and what has made me, me. Sartre argues that this dyadic
coupling between writer and writing means that I, as a writer, can never truly
read my own writing. I will never be capable of reading my own work and
getting something out of it the work is already an extension of my own
person. I can discover no thought in my own works.
Of course, Sartre, as an existentialist, is honor bound to go one step further
here the reader, by participating in reading a work, is asserting their
own freedom. Every time a writer writes [ ] the writer appeals to the
reader s freedom to collaborate in the production of [their] work . The
creative act may only be complete if both writer and reader have recognized one
another and made a choice to do so. No one is compelled to complete the
creative act. The things you read matter. Reading something fundamentally
alters you as a person you can not simply un-read a thought. Everything you
read changes your universe. I have a hunch Sartre would find things like the
(original early 2010s era) tl;dr reply to obviously terrible writing
absolutely hilarious as a reader s expression of freedom (not to be confused
with modern 2020s era usage as shorthand for summary section ). Using your
freedom to choose to complete the writer s creative act (or not) is inherently
asserting your humanity.
I m a bit fuzzy on the specifics of this quote, but I think it was
sj who told me at one year s
Mystery Hunt that A puzzle is a contract between
puzzle author and puzzle solver, the author promises that the puzzle is
solvable if you re clever enough . The puzzle author and puzzle solver are
engaging in a collaborative production of the work that is only possible by
recognizing one another. Similarly, Sartre whatever connections [the
reader] may establish among the different parts of the book among the chapters
or the words [the reader] has a guarantee, namely, that they have been
expressly willed .
Wall Drawing #123
It follows, Sartre argues, that the act of writing, as a means to convey a
thought to a reader, may only be completed through the act of reading.
Reading can only be done by others; writing, therefore, only exists to be
read it can serve no other purpose. Writing and reading are two halves of
the same creative, collaborative act, only satisfied when both writer and
reader acknowledge one another. Combined, writing and reading are the act of
recognizing one another s humanity, thoughts, experiences, consciousness the
act of co-creation of thought is the critical aspect of writing and reading
reconstructing the author s perspective, thought, intent, point. The
co-creation is the point of both reading and writing.
Writing without anyone to read the work leaves the writer s bid for
co-creation unsatisfied and humanity unrecognized. No thought has been conveyed
to any reader, no one has understood the reason behind the landmarks in the
void you ve carefully placed, leaving you to either [ ] put down [your] pen
or despair . Writing is an appeal to the reader s freedom, and the nature of
that freedom is the writer can not control it never being understood is
always a possibility any time anyone sets out to write.
Nearly 12 years ago, I attempted to read the text on the git manpage
generator for the first time
I remember the exact feeling of my brain going into git manpage parsing
mode where every word was dragging git plumbing megaliths through the sand
from their far-flung homes. I can still feel the surface of my desk as I
instinctually started to trace out logical connections between git internals
referenced as I read along. It took me a good 20 seconds to realize what I was
reading made no sense. This website broke Sartre s writer-reader agreement I
was attempting to read this website, but there was nothing there. You can
mechanically read the git-man-page-generator, but you can never read it
it is not possible to read it. It was funny unsettling. Each grasp at a
real-looking image in front of me misses, my hand coming back empty. I
couldn t get enough of it. I must have tried to read a dozen generations in a
row. I had never had something quite that broken pass that far into my
consciousness before. Although I kept trying, it never did feel quite the same
as that first time; I think fundamentally, I couldn t forget that it was
random.
While never having anyone read your writing drives them to put down your
pen or despair , Sartre never had to contend with the opposite problem being
asked to read that which was not written. Reading that which was not
written does not merely leave someone unrecognized when a reader makes a
choice; instead, it is an inherent violation of the contract between reader
and writer. Not only is no humanity being recognized through attempting to
read words which were not written, but the reader, not the writer,
is the one who bears the burden of unexpected apophenia. The reader,
while engaging in co-creation, must now contort themselves to uphold their end
of the Sartrean agreement, scrutinizing text to ascribe consciousness,
thought and intent only to come back up with pareidolia-fueled echos of
one s own self and disfigured half-thoughts of others. The reality is, the
modern reader must now mechanically read most written work they come
across, scrutinizing text for any signs of thought before truly attempting
to read the work. Failing to do this correctly changes our person
each time it happens, a small part of us is irrevocably altered. Choosing to
engage in this ballet because you wish to do so is one thing this is
your right and freedom as a reader but passing words which were not
written as your own in an attempt to wrestle a reader s freedom of
co-creation from them is another entirely.
If you didn t write it, I don t want to read it (dw;dr). Send me your prompt instead.
Vagrant Cascadian: Could you tell me a bit about yourself? What sort of things do you work on?
Jochen Sprickerhof:
I am a freelance programmer working on Open Source. Mainly doing Debian, F-Droid and some smaller software projects. In general I made it a habit to look into every software I use and try to fix bugs or add features I need. In Debian, I maintain about 180 packages with topics covering home banking, build systems and robotics. Most of my time, I currently work on reproduce.debian.net, where we try to bit-for-bit reproduce the packages distributed by Debian.
Vagrant: Could you describe the path that lead you to working on reproducible builds?
Jochen:
I started my Debian journey as a teenager, converting my school to Debian and serving as its system administrator for 13 years. After studying Applied System Science, I joined the university s robotics labs, where I worked on the Robot Operating System (ROS) and the Point Cloud Library (PCL). In the end, I enjoyed programming more than writing papers, so I eventually left academia for a robotics startup. Some years ago, I realized that the open source work I was doing in my spare time was actually the work I cared most about. Nowadays I am really grateful that I can spend my days working on things I find important and have lots of fun with.
Vagrant: What projects did you recently make big progress on?
Jochen:
A recent example is metasnap.debian.net. It is a meta archive of
snapshot.debian.org which is itself archive of all packages in Debian. But let
me explain it the other way round: with reproduce.debian.net, we try to reproduce the packages as they are distributed by the Debian archive. For that, we need the same build environment (compilers, libraries, build tools, etc) that was used by Debian back when the original package was compiled. Luckily, snapshot.debian.org has all those packages, but they are not easily accessible via apt, Debian s package manager. So, metasnap provides a mapping from a package name and version pair to the APT repo on snapshot.debian.org needed to download it from. It was created by josch some time ago, and it s awesome work. But when we tried to reproduce more and more packages on reproduce.debian.net, we found that some were missing packages from the build environment even though they where visible on snapshot.debian.org. We found that metasnap excluded some archive areas because they where not expected to be needed. Reimporting all the data took more than two months and surfaced a couple more flaws.
With this fixed, we were able to build more packages, only to find out that metasnap also needs better support for version numbers. Luckily we were able to rewrite the data in a day instead of starting the import again.
Vagrant: You have been working on infrastructure to support reproducible builds for a while. Has recent adoption of reproduce.debian.net into the Debian release tooling changed the focus of your work?
Jochen:
Quite a bit. When we started reproduce.debian.net in 2024, only around 33% of the packages could be reproduced successfully. Today we are above 98%. Most were not bugs in the packages themselves but in the infrastructure. Similar to the metasnap issue I reference above, packages just needed a rebuild because something else, like the toolchain, was fixed in the meantime. In May, people from the Debian release team and the Reproducible Builds project sat together and decided that the overall state is good enough, and now packages that regress on reproducibility are blocked from entering the next Debian release. But that does not mean all the work is on the shoulders of Debian package maintainers. Since then I have been constantly looking at the migration tooling to spot regressions and provide fixes. Furthermore, a couple of maintainers reached out to us for help and I hope more will do so in future.
Vagrant: What is one small thing you (or others) have not yet gotten to that you would really like to see?
Jochen:
The central tool to reproduce Debian packages is debrebuild, also written by josch. Currently it has two ways to retrieve the build dependencies of a package. Either it uses metasnap.debian.net (as explained above), or it can access the Debian unstable APT repository directly. This allows to test packages locally before everything is indexed on metasnap by compiling against Debian unstable. But actually there are many other APT repositories to query, like Debian stable or even derivatives. Adding support for an optional list of APT repositories in debrebuild would be great. That would also be a big step to support reproducing other Debian based distributions.
Vagrant: and one big thing?
Jochen:
It would be great to integrate metasnap.debian.net into snapshot.debian.org. There is some discussion on it already in Debian bug #650783.
Vagrant: What are the tools you use the most?
Jochen:
According to my fish shell history:
$ history cut -d' ' -f1 sort uniq -c sort -nr head -10
36199 git
20941 vi
12271 rm
8599 cd
7917 ls
6407 apt
5631 grep
4249 mv
3655 dpkg
2873 cp
Vagrant: So, is the fish shell reproducible? I remember it did not used to be
Jochen:
You can check for yourself it was last time I checked. But looking through the other commands, neovim sadly is not. I hope we can fix that in future.
Vagrant: Oh, that s a nice URL to check for reproducible package you can just pass the source package name to check the current results?
Jochen:
Yes. Another one is udd.debian.org/reproducibility/, where you
can list all packages of a Debian maintainer. It also lists source reproducibility and has nice filters as well.
Vagrant: What tools do you use specifically working on reproducible builds?
Jochen:
I don t have statistics for that, but I would say sbuild to build the package, debrebuild to reproduce it, and diffoscope to analyze the differences. Obviously I also need run apt source <package> or use git-buildpackage to get the sources and all the tools I mentioned above.
Vagrant: So how many packages are left to build reproducibly, and once those are finished, what is next?
Jochen:
Right now, reproduce.debian.net shows over 98% reproducibility, though there are still over 650 package left and some will probably need a lot of work. But actually I think making packages reproducible is just the first step. For me, this is a project to build confidence in the system. To reproduce a package we have two parts: the source of the package and the build environment. Fixing the packages means gaining confidence in the first part but we still rely on the individual build environments for each package as we need to use the same compiler that was used when the package was build initially. Because of this, we have to keep around every historical version of all toolchain packages. I really would like to remove this extra archive, which means we would have to rebuild all of Debian around release time. I am dreaming of a Debian release where you could bit-for-bit reproduce every package just from the released versions. Due to how Debian works, however, this is not a trivial rebuild and it would need some work on the infrastructure. By the way, initially there was a third component to pay attention to: any connection to the outside world during the build. Luckily we fixed the Debian build daemons to not allow network connections during the build some time ago.
Vagrant: Thanks for all that work, and taking the time to tell us a bit about yourself!
Jochen:
Thanks a lot for the interview!
For more information about the Reproducible Builds project, please see our website at
reproducible-builds.org. If you are interested in
ensuring the ongoing security of the software that underpins our civilisation
and wish to sponsor the Reproducible Builds project, please reach out to the
project by emailing
contact@reproducible-builds.org.
Welcome to the August 2026 report from the Reproducible Builds project!
In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not.
In this month s report, we cover:
New updated SBOM specification from CISA.gov
CISA, the Cybersecurity and Infrastructure Security Agency of the U.S. government published some the joint guidance entitled Minimum Elements for a Software Bill of Materials (SBOM), which updates and supersedes the baseline 2021 version covered in previous editions of these reports.
Whilst the PDF is worth skimming, the interesting changes include that the specification now mandates standard cryptographic hashes: unlike earlier standards that allowed hash omission or manifest-only parsing, hashes must be computed from the output. This is is important for reproducible builds, as it ensures the recording of the metadata required to demonstrate the shipped software matches the build output precisely where applicable. In addition, where the top-level only dependency limitation that was present in the 2021 version has been removed in favour of complete coverage with no minimum depth. That is, SBOMs are expected to reflect all linked libraries, vendored dependencies and other build-time inclusions.
The basic idea behind bootstrappable builds is to create a system that can be built without relying on pre-built artifacts. Can we go from zero to the modern day without having to just assume the existence of these already-built-for-us artifacts? The classic recipe for yogurt requires some yogurt to start the process, which is like how we normally build a C compiler today we start with an existing C compiler binary. You might think about making sourdough bread with your grandmother s starter brought over from the old country; we re basically making C compilers with Dennis Ritchie s starter carried over from Bell Labs .
The reason I like the idea of making reproducible builds work is that I think it can be done in such a way as to not require any work on the part of the producer of a distribution (which is a technical term for sdists or wheels, i.e., the people who upload stuff to PyPI), and thus make reproducible builds very low-friction for people to opt into supporting. []
Brett goes on to outline What s missing from the specs and how reproducibility might be visible on PyPI to consumers:
Assuming all of this comes to pass and we record the where the source code is that went into a distribution and the software used to make the distribution, how do we make it useful to people? Does every person who cares about having a secure supply chain have to rebuild everything they use themselves? Is there some way for even people who don t care about this stuff to benefit? []
Distribution work
In Debian this month, 23 reviews of Debian packages were added, 28 were updated and 27 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated as well, such as the addition of a new toolchain issue related to python-traitlets [], and the note for an existing issue related to texi2html was updated as well [].
Lastly, Bernhard M. Wiedemann posted another openSUSEmonthly update for their reproducibility work there.
Unreproducible builds under EROFS filesystem fixed
Martin Pitt reported on Fosstodon that they had identified an issue where the mkfs ( make filesystem ) command for the EROFS (Enhanced Read-Only File System) subsystem of the Linux kernel did not have sorted extended file attributes, leading to reproducible builds.
Thankfully, Martin also reported that they had fixed this in a commit to the kernel which order[s] each inode s xattrs by name so that images stay reproducible . []
Tool and documentation development
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions 327, 328 and 329 to Debian. In particular, he ensured that diffoscope did not require python3-guestfs in the autopkgtests on 32-bit architectures in order to fix Debian bug (#1144372) []. Colin Watson made an additional change, handling a potentially missing openssh-client package when running the autopkgtests [], and Jochen Sprickerhof made a similar change to cope with missing cpio and qemu-img functionality [] whilst also updating the XML comparator to be considered when comparing SVG images [].
Yet again, there were a number of improvements made to our website this month as well. For example:
Chris Lamb added draft for a Gothenburg summit-related news article. [][]
Lastly, a large number of commits were pushed comprising an interview with Reproducible Builds developer Jochen Sprickerhof to be published within the next week. [][][][][][][]
Six new scholarly papers
Jens Dietrich, Spencer Sun, Tim W. White and Behnaz Hassanshahi (the result of a collaboration between Victoria University of Wellington and Oracle Australia published a paper this month entitled No Snake Oil: Verifying Python Package Builds. Drawing on the metaphor of snake oil , that is, a fake or ineffective medicine or solution sold with exaggerated claims of curing or fixing everything, the authors write that
Two tools that are designed to automate [PyPI] rebuilds and run them at scale are macaron and oss-rebuild. We study 12,180 popular releases from PyPI and find that the byte-for-byte equivalence rate is generally low. We analyse the reasons why they produce different wheels, and find that equivalence between the original and rebuilt wheels can often still be established, preserving most of the guarantees users expect from rebuildable releases. We present and evaluate daleq4py, a tool to establish the equivalence of Python wheels through the kernel of a normalisation function that is based on provenance-preserving datalog rules. Experimental results show that daleq4py substantially expands the set of rebuilds that can be accepted as equivalent.
pose a persistent threat to opensource software. Known attempts to inject such backdoors into widely used projects through malicious commits, tampered release packages, or compromised third-party dependencies, were stopped only by luck and manual review. Existing Continuous Integration (CI) pipelines cannot detect these attacks, and downstream binary analysis tools require substantial manual effort. In this work, we present Lily, an automated approach that strengthens open-source development and release processes against backdoor injection. Lily integrates a backdoor detection mechanism into (1) CI pipelines to block malicious commits, and (2) release vetting workflows to prevent tampered releases or compromised dependencies from entering large ecosystems, such as Linux distributions.
The full PDF can be read online.
Ranindya Paramitha and Laurie Williams of North Carolina State University along with Christian K stner of Carnegie Mellon University published a paper this month with the title of The Software Supply Chain as a Market for Lemons: A Multivocal Review of Trust Signal Collapse. (A lemon in American English, is a vehicle that turns out to have several manufacturing defects .) Their abstract is as follows:
Practitioners evaluating open-source dependencies rely on cheap trust signals, e.g., stars, download counts, and contributor activity, as substitutes for direct code inspection, assuming those signals reflect genuine trustworthiness. Prior work has documented individual signal gaming, but the landscape of collapses across all dependency-adoption signals, as well as the ecosystem s response, remains unexplored. The goal of this study is to aid software practitioners in understanding the reliability of dependency adoption trust signals, such as download counts and contributor activity, by conducting a multivocal review of 252 Google Search sources and 870 Reddit threads.
Worryingly, after their review, the authors conclude that cheap trust signals collapse under three simultaneous forces: adversarial manipulation, gaming techniques indistinguishable from legitimate behavior, and non-adversarial AI-driven inflation.
The full PDF of the paper is available online.
Julien Malka, Aman Sharma, Martin Monperrus, Stefano Zacchiroli and Th o Zimmermann published a paper this month on Trusting-Trust Attack against an Entire Linux Distribution through Binary Manipulation:
Ken Thompson s trusting-trust attack, in which a compromised compiler backdoors the programs it builds and reproduces the backdoor in subsequent rebuilds of itself, is widely regarded as a threat specific to compilers. We show that it is not. We construct a complete trusting-trust attack around GNU strip, an ordinary build utility that neither inspects nor generates source code, using only manipulations of finished ELF files.
Scarily, in the authors example, a single tampered strip in the binary seed implants a payload that propagates from one generation of strip to the next and survives into the final standard environment after the seed leaves the dependency closure [ ] without failures and backdoors .
A full PDF of the paper is available for download online.
Mehdi Keshanimm, Amirhossein Rahmati, Mohammad Hossein Aref and Abbas Heydarnoori published a paper that is currently under review at Emperical Software Engineering titled AROMA+: A Study of Factors Affecting Reproducible Builds in the Maven Ecosystem. (Maven is a/the build automation tool used for Java projects.) In their paper, the authors note that
[ ] reusing external software in a project presents a security risk when the source of the component is unknown or the consistency of a component cannot be verified. The SolarWinds attack serves as a popular example in which the injection of malicious code into a library affected thousands of customers and caused a loss of billions of dollars. [ ] Our research aims to support [reproducibility] efforts in the Maven ecosystem through automation. We investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA+, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of packages on Maven Central.
[A]rtifact verification requires more than deterministic builds: a verifier must also recover the source state, build environment, dependencies, and build instructions that produced the artifact. Decentralized-build ecosystems make this difficult because artifacts are produced through heterogeneous tools, maintainer-controlled workflows, and fragmented metadata. As a result, it remains unclear how often artifacts in these ecosystems can be independently verified. This paper studies artifact verifiability across four popular decentralized-build package ecosystems. We define an independent verifier model that relies only on registry-derivable metadata and an artifact comparison model with tiered equivalence levels. We implement these models in an Artifact Verification Pipeline and use it to measure artifact verifiability across the target ecosystems.
The authors conclude that beyond build determinism, verifiability is limited by missing source and build metadata, implicit release transformations, and unconventional build practices .
A PDF of their paper can be reviewed online.
Patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:
Two weeks ago, I released AppStream 1.2.0. This release contains a lot of great changes, but one of the most important ones concerns how media are being handled, and AppStream s default image export format.
AppStream is a Freedesktop metadata standard to describe software components. That can be anything from system services over fonts to console and graphical applications. AppStream metadata is supposed to give users enough information to decide whether they want to install a piece of software, to represent that piece of software, and to give the operating system enough information to decide whether a software component should be installed automatically and (to some extent) what capabilities and relations it has, to provide the user with sensible options.
Especially for the first two goals, and especially for GUI applications, AppStream supports icons and screenshots, which are used to showcase applications. Today, AppStream is used by all kinds of services, from Linux distributions over firmware updates to Flatpak and desktops directly. AppStream s original design however comes from the perspective of Linux distributions in 2011, where you may want to browse the software catalog offline, without delay, and without pinging an external server (which could be a privacy concern).
Therefore, a common way to deploy an AppStream-enabled software repository is to ship all icons of all applications in the repository to the user as part of the repository metadata download. AppStream does support remote icon downloads nowadays, and for a while I thought that this would become the default eventually. However, especially in today s world, having a bandwidth-saving, instantly responsive, privacy-protecting application browsing experience seems more important that ever.
PNG images are great!
The only format that AppStream supports for icons and screenshots (which are downloaded on-demand from your distributor s CDN) has always been exclusively PNG. PNG images are perfect for icons, because they compress well (especially for common icon shapes), are fast and simple to load, and can be loaded anywhere, by any toolkit or webbrowser. They also ensure we deliver faithful screenshot images, even though we may have scaled or re-rendered them. Still though, PNG images are less great for screenshots, as they are not very efficient, which puts strain on any CDN that has to deliver them, as well as on people s internet connections when browsing screenshots. Having smaller thumbnails alleviates that problem a little, but does not fully solve it.
But even for icons, PNG could be improved upon: In many cases, icons are re-downloaded with the repository metadata again and again, so having a large icon tarball adds up to the data transferred during metadata refreshes. AppStream also now supports large 128x128px icons, which nobody in 2012 expected we would need, adding even more data that will be re-downloaded. Saving some space here translates directly to lower bandwidth costs as well as faster downloads for users.
To improve PNG file sizes, the AppStream Compose library, which handles all image processing and metadata catalog composition, was running optipng on all generated PNG images. That does create smaller PNG images, but they were still relatively large compared to other image formats.
For a long time though, there was no alternative to PNG images for icons: There was no lossless image compression format that could give us the same quality as PNG images and that was also widely supported.
JPEG-XL vs PNG in AppStream
Since 2021 we have JPEG-XL (JXL), which offers a true lossless mode with often better compression than PNG. The issue was that JPEG-XL wasn t widely supported. Then, in 2025, the PDF Association selected JPEG-XL as the preferred image format for HDR images in PDFs, and now we are finally getting browser support and more ubiquitous availability of the format (you can try it right now in Firefox!).
For screenshots, using JXL s lossy mode, it has obvious and extreme size advantages over PNG, so supporting JXL or WebP for screenshot images was an obvious choice. If JXL would support the lossless case very well as well though, we could serve many use cases with the same exported image format, which is very attractive to me.
So, the obvious next question was whether it was worth the pain of switching the icon format, so I did some measurements on real icons. For that I used the AppStream component icon pool that Debian Unstable ships, which is almost 5000 application icons of various sizes, and converted them to PNG:
Icon size
Icons
PNG total
JXL total
Pool saved
PNG avg
JXL avg
Median saved
Mean saved
Worst
Best
Larger as JXL
48 48
1544
3.7 MiB
3.0 MiB
17.8%
2.4 KiB
2.0 KiB
17.9%
16.7%
-118.7%
60.0%
206
64 64
2018
7.0 MiB
5.8 MiB
17.8%
3.6 KiB
2.9 KiB
18.0%
15.8%
-112.7%
70.0%
279
128 128
1411
11.2 MiB
8.7 MiB
22.0%
8.1 KiB
6.3 KiB
20.1%
17.5%
-89.7%
61.0%
209
TOTAL
4973
21.9 MiB
17.5 MiB
19.9%
4.5 KiB
3.6 KiB
18.6%
16.6%
-118.7%
70.0%
694
PNG images saved with libpng at effort=4, compression=9, then optimized using optipng -o2, JXL images encoded using vips jxlsave lossless=1 effort=7 strip=1 via VIPS/libjxl.
As the table shows, using lossless JXL images over size-optimized PNG images (using optipng s default settings) provides a roughly 20% gain. This does not look like much, until you consider how often these files are downloaded: A 20% file size reduction may only save 1-2 MiB of disk space, but if they are downloaded over and over again by many clients, it will save a lot of bandwidth.
Interesting JXL encoding findings
As a sidequest, I was curious why some images were larger than their PNG counterparts when encoded with JXL, and what the ones that were significantly smaller were.
In short, the biggest size reductions for JXL existed on images that were already small as PNG, and contained large, flat color surfaces with hard edges and simple shapes. They were not very interesting, and much of JXL s wins come from accumulating smaller gains across all files, which compound the bigger icons get (especially at 128x128px, where JXL truly shines).
The events were JXL loses to PNG are more interesting: For example, it does quite poorly with pixel-art images that have a lot of repeating patterns. Those are encoded well by PNG, but less efficiently by JXL. Take for example Vonsh:
Icon of Vonsh, an SDL-based snake game, which PNG compresses better than JXL
My guess is that while PNG can exploit the repeating pixel patterns for compression, JXL s predicts surrounding pixels from its neighbours, which fails too often and makes it pay almost full entropy per pixel. In this single rare case, the PNG is at 5.4 KiB, while the JXL is almost 8 KiB in size.
Other cases I looked at were arguably buggy input data, where color channels were hidden under the alpha channel of the input image. PNG could probably again exploit repeats, while we were forcing JXL to encode pixels that were invisible in the final image. This is arguably a problem with the original input data. Currently, AppStream does not make any changes to icons at all, but in future we might add a filter that removes invisible colors from images to solve this pathological case (it was only two icons out of 5000 though, so it is not a high priority).
The third case I found where JXL loses to PNG were icons with checkerboard-like patterns:
Icon of x3270, an IBM 3270 Terminal Emulator
For those, PNG can likely again exploit the repeating patterns, while a checkerboard layout is pretty bad for left/top predictors like JXL s. However, in this case the size difference (and loss for JXL) is only 450 bytes, so even though JXL loses to PNG, it does so not by much.
JXL in AppStream
Given these findings, JPEG-XL is the default image format starting with AppStream 1.2.0. AppStream Compose will encode all images losslessly as JXL, while screenshots are encoded in lossy mode at Q=90 effort=7. Since the optipng step does not happen for JXL images, this comes at no speed penalty and is even a bit faster on modern x86_64 CPUs (where libjxl can use SIMD). PNG is still available, and Compose can be told to switch between the two formats.
Upsides of JXL in AppStream right now
If you use JXL in Compose or the recent release of appstream-generator, you will get much smaller images and, for screenshots, will benefit from other JPEG-XL features such as progressive decoding, providing a far nicer user experience. libAppStream has supported JXL icons since version 1.1.3, so your clients will need that version or a newer one, and all software centers will have to support loading JXL images (which all of them do, provided the right plugins are installed).
Downsides of switching to JXL too quickly
JXL is a very new format, so web browsers might not yet display it if you are serving webpages. Your clients may also have bugs in processing JXL images, as the format is still new . For example, switching on JXL in Debian sent KDE Discover into an infinite loop on startup while trying to load the icons (an issue which has been fixed, but clients will need that patch first before JXL is switched on).
This currently makes JXL enablement only possible when you know that your clients can support it. This is the case for me in Debian Unstable and Debian 14, which are using JXL images for a few weeks now, but not for any older releases. Platforms like Flatpak have it even harder, because they do know even less about their clients. So, even though it has big advantages, you may want to hold off on using JXL right away, and force PNG by setting the ImageFormat key to png in appstream-generator s configuration, or passing --image-format=png to appstreamcli compose.
It is also worth mentioning that JPEG-XL is much, much slower on systems that do not have SIMD instructions or for which the libjxl/jxl-rs library does not have them (such as apparently riscv64 right now). If this is a concern, you might not want to switch to JXL right away.
Media pipeline improvements
Besides the JXL default change, AppStream 1.2.0 also comes with a complete overhaul of its media processing pipeline. While libappstream, AppStream s main library, does not do any media processing and comes with very minimal dependencies to be embedded in client applications and used on servers, the same can not be said about libappstream-compose, AppStream s library to build metadata generating applications (the server-side part, usually).
The compose library has to render fonts into font specimen cards, inspect translation files, render SVG images, decode all kinds of raster images, inspect video files, etc. Especially the fonts, and the fact that fonts can appear in SVG images, has caused issues in the past, as libappstream-compose is a heavily threaded library and most font libraries can only work from a single thread. This forced the library to essentially go into single-thread mode anytime anything that could touch a font was being processed.
AppStream also originally was created for a safe world where applications were vetted by the distributors before their metadata was processed. This is increasingly not the case, so it made sense to put at least a few guardrails on the most complex part of the pipeline: The media processing. As part of the change, media processing was split out into a separate worker process. This solved two problems at once: Font handling was isolated in a single-threaded binary if we wanted to handle fonts in parallel, we could simply spawn more workers. And, being in a separate process, the media processing could now be sandboxed.
As part of the multiprocess changes, Compose also switched from using GdkPixbuf to VIPS for image processing. The latter allows for much more fine-grained control over the image output and encoding, and comes with a lot of well-maintained filters and operations, which made it possible to eliminate a fair chunk of AppStream s hand-rolled image processing operations. As part of this transition, we unfortunately lost the ability to read XPM images, which dropped about 20-30 applications from the pool at Debian. But in the name of security, this is a sensible choice, especially since most XPM icons were very small and low-resolution, and applications using them could benefit from adding a high-quality PNG icon anyway. With VIPS, we also now restrict the amount of image formats we can load to a sensible set, so extremely niche or unexpected formats will be outright rejected (this includes sane-but-unusual formats for screenshots and icons, such as TIFF images).
The Compose library, with all of these changes, will now just request high-level operations (e.g. render a font card for this font to a JXL image ) from the worker, and provide it with input data in sealed memfds and output locations as FDs as well. On Linux systems, the worker will use Landlock if available, to block all write access to the filesystem, deny device access and deny TCP and UDP as well. The sandbox can certainly be tightened a fair bit in future, but this was a good and safe start to gain some experience with it without having things break too easily, given the many places Compose is used in (also, Landlock s API is surprisingly nice to use, so it was easier than I thought to add in this early version).
With all of these changes, the libappstream-compose library is now also officially marked API-stable, so you should be able to rely on it in future to build new things (its API has barely changed in the past, and now with the new media API and defaults change in place, it was time to declare it stable).
I want to see / try this!
Currently, the easiest way to have a look at the new data is to check out Debian Unstable. If you have a JXL-enabled browser, you can also see the icons in AppStream Generator s HTML pages for Debian Sid. If you are using appstream-generator for your distribution, you will also get much more pleasant statistics and HTML pages, as well as fully deterministic media output and a whole bunch of security updates, so, update to its recent 1.0 release.
Please keep in mind that if you switch to JXL, the client tools receiving the image data have to support it. Support varies depending on the Linux distribution, so, test it first and switch the default back to PNG in case you encounter any issues.
What s next?
With so many features and changes landed, the next changes in AppStream will focus on improving what already exists and fixing any issues (there will be more blogposts about the other features 1.2.x delivers!). Testing with the entire Debian archive as data source makes me fairly confident though that there will not be many problems. In the longer term, tightening the media processing sandbox will also be something we might want to do, e.g. by hiding parts of the filesystem tree or filtering syscalls.
For JPEG-XL, one obvious question is Will you add support for it to the Freedesktop icon-theme specification as supported format alongside PNG, SVG(Z), and XPM? . For on-disk icon repositories, JXL s space-savings are less compelling, and it being HDR-capable is also not necessarily a killer feature (PNG can go a long way!). However, JPEG-XL s ability to immediately decode larger images at reduced resolution without resampling could legitimately be very powerful here, as applications could ship a single large image and quickly decode it at 1/2, 1/4 or 1/8 the size for different purposes in their UI. JPEG-XL also supports spot-color extra channels, which applications could use as masks to recolor raster icons at render time. This could be incredibly nice to color symbolic icons on-the-fly without any SVG and CSS. JXL also provides richer metadata, which might be neat for (license/author) documentation. So, the answer here is: Maybe it makes sense to allow another format, but this will have to be discussed first, as it would force JXL into every toolkit and desktop, which is a much bigger ask than supporting it only in AppStream.
As always, let me know what you think and please report any issues or bugs directly against AppStream or AppStream Generator if you encounter problems that are with the tools, and not with a project s metadata.
This post is part of a series called "Pigeon". If this is the first post
you've found, it'd be worth reading the
intro post first and then looking over
all posts in the series.
If you're looking for it, the intro to the PHY (layer 1) that this belongs at a
high level and listing of the parts that make it up (including this one) is on
the phy post.
The time has come.
If you re following along at home, we now have all the basics we need to glue
these parts together and see what this looks like.
We re going to build the highest-level constructs for the PHY in code
something that takes some number of bytes in and writes out IQ samples fit for
transmit over the airwaves (we ll call this the Encoder), and something that
takes chunks of IQ samples in, writing out decoded bytes (which we ll call the
Decoder).
Encoder
Let s begin with the Encoder, since it s slightly less involved. I ve tried
to make this a bit more accessible by drawing a diagram out before describing
the order of operations, so that it s possible to follow along visually.
While the process here can look like a lot, it s really not that bad.
We begin by taking the incoming bytes, converting the bytes into bits, and chunk
those bits into parts which are sized to fit completely within an LDPC message. We will then
encode incoming data into LDPC messages, using our
configured LDPC Matrix (the table we appropriated from 802.3an). Next, we
apply whitning over all the bits in our
encoded (and packed) LDPC messages, using our configured whitening constant. In
the case of QPSK, pairs of bits will then be
modulated into a QAM subcarrier, where each
QAM point represents a range of bits in the message. We ll go through each of
those modulated IQ subcarriers, and
set each corresponding data subcarrier
in order, for each OFDM symbol contained in the pigeon Burst. The preamble configuration
is then used to generate (or, more likely, can be used at startup to precompute)
the Schmidl-Cox preamble, which is written to the
first IQ samples in our output IQ buffer. Finally, we will do a series of
inverse FFT operations to convert each
OFDM symbol to the time domain, including their cyclic prefix.
Let s take a look at doing that, but in code this time now:
// (lightly edited for clarity)
impl Encoder
../// Encode the provided bits into the output time-domain IQ samples.
fnencode(
&mut self,
dst: &mut [IQ],
src: &Vector,
) -> Result<Burst, Error>let src =letmut raw = Vector::new(self.fec.message_len());
// Set raw 's data bits, compute and set LDPC
// checkbits.
self.fec.add(&mut raw, src);
// Apply whitening, and return
raw.xor(&self.whitening)
;
// copy in the precomputed schmidl-cox preamble to dst
let preamble_len = self.preamble_iq.len();
dst[..preamble_len].copy_from_slice(&self.preamble_iq);
// allocate a new (frequency domain) 'Burst' container.
letmut burst = Burst::new(
&self.mode.ofdm.plan,
self.mode.ofdm.symbols
);
// modulate bits from 'src' as iq, and set each
// data subcarrier for each ofdm symbol in the
// burst.
self.burst_encoder.multiplex(&mut burst, &src);
// convert from frequency-domain data into time
// domain iq samples, writing out ofdm symbols
// and cyclic prefixes to dst .
self.burst_encoder
.transform(&mut dst[preamble_len..], &burst)?;
// normalize all IQ samples; the maximum magnitude
// in the IQ buffer may be very small, which weakens
// our transmitted signal. Scale all IQ samples such
// that the maximum IQ sample magnitude will be '1.0'.
dst.norm();
Ok(burst)
Using the Encoder should hopefully be fairly straightforward we ll
give it a bag of bytes, and get back some IQ samples that we can ask our
nearest SDR to transmit.
As for what happens on the other end?
Decoder
Next up is the mirror image of our Encoder the, imaginatively named,
Decoder. The Decoder is slightly more involved (since it has to find the
packet in the IQ stream, as well as correct for channel error(s)), so we ll do
the same thing as above start with a diagram. My hope is going over the
Encoder first helps us only really focus on the new stuff, otherwise it
should feel like running the Encoder backwards.
Here, we start with an incoming stream of IQ, where we will process
scan detections as they come in
from our Schmidl-Cox detector and burst Scanner. This will give us a snippit
of IQ, sized to exactly our Burst. We ll begin to correct our IQ samples by
first doing
frequency estimation and correction
in the time domain using our preamble and ofdm configuration. We ll then do
a series of inverse FFTs to extract each OFDM symbol
in our Burst, where we can then do
channel estimation and correction.
With the OFDM symbols (hopefully) good enough, we can now
map each data subcarrier back to bits, and
unapply whitning. The resulting
bits are then chunked back up into LDPC messages,
which are then checked, and concatanated data extracted. Finally the bits
are turned back into bytes, which are written to our output buffer.
However, before we get into the code to do this there s one last detail. We
know bursts won t overlap (if they do, it s likely not possible to recover
right now even though other PHYs can and do), so any time we see something we believe
to be a burst, we can skip ahead by the burst s (constant) length within
the IQ, and avoid trying to decode anything else in there.
The nice side-effect here is this also gives us an interesting property for the
Decoder namely, we know the maximum number of Burst detections we can
get for a given block of incoming IQ data if they were packed end-to-end and
we can pre-allocate the memory we need, avoiding allocations for every
demodulation attempt (which may or may not even be a valid Burst).
This pre-allocated block of memory to hold the burst s data is something that
I ve called a frame buffer internally. Each frame buffer contains exactly
sized buffers to hold decoded information from the burst an iq buffer
that is exactly the same number of samples required to encode the preamble
and data, exactly the number of bits needed to store pre and post FEC data,
pre-allocated byte array, etc.
Not shockingly, the code looks like this:
#[derive(Clone)]pubstructFrameBuffer/// Corrected IQ samples
pub samples: Samples,
/// post-correction OFDM burst
pub burst: Burst,
/// demodulated bits from the OFDM burst
pub bits: Vector,
/// demodulated bits from the OFDM burst,
/// after FEC, and cleaned
pub raw_bits: Vector,
/// Layer 2 contents of the Frame
pub contents: Vec<u8>,
Of course, that alone is handy but we need to use them. So let s go ahead
and do what we promised above each Decoder uses a fixed number of
pre-allocated FrameBuffers to store packets in-flight, packed into what
is, creatively, called FrameBuffers within my code.
As an aside, I likely should have called this a
Memory Pool, since that s the
commonand accepted
name for this design pattern but being stuck with unfortunate names is
the burden of those of us who stumble into sensible ideas over time. The only
nuance here is that I use the pools strictly sequentially we only save
the FrameBuffer if the LDPC checksum is correct, allowing us to only keep
track of how many successful packets we have and being able to get the valid
FrameBuffers, rather than storing a handle to each FrameBuffer as we go
a promise that most memory pools do not make, since blocks can usually be
taken and returned in any order.
Let s go ahead and do the whole Decoder dance now:
// (lightly edited for clarity)
impl Decoder
../// Process incoming IQ for Pigeon Bursts, and
/// demodulate them.
pubfndecode(
&mut self,
buf: &[IQ],
) -> Result<Vec<(Detection, &FrameBuffer)>, Error>letmut ret = Vec::new();
let mode = self.scanner.mode().clone();
// reset the "valid frame buffer count" back to 0
self.frames.reset();
// call the scanner and get scan detections for
// this block of iq ( buf )
for detection in self.scanner.scan(buf)
// for each detection, we're (only) going to process
// the iq snippit, but pass along the metadata
// such as SNR.
let ScanDetection
snippit,
snr,
range,
m,
= detection;
// grab the next free frame buffer to work within.
let frame_buffer = self.frames.next_mut();
// Copy the snippit into the frame buffer (a mutable
// location)
frame_buffer.samples.copy_from_slice(snippit);
// estimate the frequency offset based on the
// Burst's Schmidl-Cox preamble.
let preamble_fo = preamble::estimate_frequency_offset(
&mode.preamble,
mode.rate,
&frame_buffer.samples[..mode.preamble.samples()],
);
// Shift the IQ stream by the estimated frequency
// offset -- hopefully we're closer to 0Hz
frame_buffer.samples.shift(mode.rate, preamble_fo);
// estimate the frequency offset based on the
// each burst's **cyclic prefix** -- exactly like
// we did with the Burst Schmidl-Cox preamble,
// but this time on each OFDM symbol.
let ofdm_fo = ofdm::estimate_frequency_offset(
&mode.ofdm,
mode.rate,
&frame_buffer.samples[mode.preamble.samples()..],
);
// Shift the IQ stream closer yet; hopefully this
// is a very small nudge even closer still to 0Hz.
frame_buffer.samples.shift(mode.rate, ofdm_fo);
// Do a bunch of inverse fft operations for each
// OFDM symbol, filling the frequency-domain Symbol
// structs in frame_buffer.burst (Burst) struct.
//
// this will also do channel estimation and
// correction before returning.
self
.decoder
.transform(
&mut frame_buffer.burst,
&frame_buffer.samples[mode.preamble.samples()..],
)?;
// "demultiplex" each data subcarrier's frequency-domain
// IQ constellation point, setting the correct bit range.
self.decoder.demultiplex(
&mut frame_buffer.raw_bits,
&frame_buffer.burst
);
// unapply whitening by XOR-ing the buffer with
// the well-known whitening vector.
frame_buffer.raw_bits = frame_buffer.raw_bits.xor(
&self.whitening);
// verify that the LDPC message(s) are all correct,
// and if so, concatanate the the data bits (no check
// bits) to the bits vector.
if self.fec.decode(
&mut frame_buffer.bits,
&frame_buffer.raw_bits
).is_err()
// this is where invalid packets fail. we gave it a good go.
// next packet please.
continue;
// copy the raw bits out, as bytes, to the contents buffer.
frame_buffer.bits.copy_as_bytes(&mut frame_buffer.contents);
// store metadata/metrics on the demodulation.
ret.push(Detection
m,
snr,
index: range.start,
);
// save the contents of this frame buffer (don't
// reuse this buffer next go-around).
self.frames.save();
// We're going to take out the borrow on the frame at
// the end since we don't want to deal with telling the
// compiler via code gymnastics that the mut and non-mut
// borrows are OK since they're non-overlapping.
Ok(ret.into_iter().zip(self.frames.iter()).collect())
Phew. That was kinda a lot. In fact it s basically the whole thing. This
function is as close to how do you read an OFDM packet as it gets, and
perhaps the most important part of this whole series. Beyond that, though, this
is a huge conceptual unlock. This means we now have an incredibly powerful
primitive; the ability to take bytes and go to/from IQ samples over the air.
Let s talk about pigeon modes
This August, I accomplished what I wanted for many years: I deleted the last cgo
dependency in Debian Code Search! This was made possible by Go s recently
introduced SIMD support, because now we can implement the TurboPFor integer
compression format as efficiently more efficiently, in fact, by using the
newer AVX512 instruction set! as the reference implementation.
Background: Why does DCS need a fast Integer Codec?
Debian Code Search (DCS) is a search engine that allows searching all the Open
Source source code within Debian, with either literal search expressions or
regular expression search queries.
A search engine uses an inverted index: a map from term to documents containing
the term. Each document is typically represented most efficiently by using an
id, so the index consists of many lists of document ids.
When searching, it is important to quickly decode these lists to answer the
search query. However, there is a point of diminishing returns where the
decoding speed, even though it can still be measurably improved quite a bit, no
longer influences the overall query duration.
From 2012 (its inception) to 2019, Debian Code Search used to use a small index
format, and queries were fast because the index was kept entirely in RAM. In
2019, I implemented the new index
format, which adds an
on-disk positional index. For literal queries (78.2% of DCS queries), querying
the positional index on disk is faster than querying the non-positional index in
RAM.
The efficient encoding of the TurboPFor format makes it possible to fit such an
index on a mid-sized Hetzner server, which I rent with two 1 TB SSD disks. The
optimized decoder of the C TurboPFor library is what made decoding fast at query
time.
If you want to dive deeper into the algorithm, see this blog post from February
2019:
Use a C library via cgo so gcc or clang compiles
SIMD code. Debian Code Search used to use the
powturbo/TurboPFor
C library via cgo for the last 7 years.
The C TurboPFor library has served us well, but Debian Code Search was always
intended to be a project using Go, so I would prefer it if I did not have any C
code in the project.
Go 1.26 (released in February 2026) introduced the simd/archsimd package:
Go 1.26 introduces a new experimental simd/archsimd
package, which can be enabled by setting
the environment variable GOEXPERIMENT=simd at build time. This package
provides access to architecture-specific SIMD operations. It is currently
available on the amd64 architecture and supports 128-bit, 256-bit, and
512-bit vector types, such as Int8x16 and Float64x8, with operations such
as Int8x16.Add. The API is not yet considered stable.
Go 1.26 Release Notes
For my 2019 TurboPFor analysis, I implemented goturbopfor, a native Go
teaching decoder (without any SIMD),
because I find Go code easier to follow than C code, especially optimized C
code. My implementation was intentionally not optimized so that the code was
easier to study.
The TurboPFor format/algorithm has a vector-optimized part: bitpacking comes in
a scalar variant (bitunpack32) and a vector variant
(bitunpack256v32),
where the vector variant is used for full blocks (256 values) and the scalar
variant is used for remainder blocks (< 256 values).
When Go 1.26 was released, I used Claude Code to explore whether my native Go
decoder s bitunpack256v32 function (for the vertical vector layout) could be
implemented using Go SIMD, and the answer was yes, it was possible and it was
faster than without SIMD, but not quite at the level of C TurboPFor. If you let
Claude Code try for long enough, it eventually finds enough optimizations (about
10) to match C performance.
I don t want to vibe-code Debian Code Search, though, so I figured I would find
some time to review the SIMD code at some point and see if I could implement
something similar myself.
Before I found enough time and motivation to complete said review, I discovered
that to not regress real-life query performance by more than 10 to 100
milliseconds (which seems acceptable), I don t actually need to add SIMD code to
my teaching decoder at all; it would be sufficient to reduce allocations in my
teaching decoder and specialize it per bit width.
Encouraged by the possibility of using the optimized native Go decoder in Debian
Code Search, I explored whether I could also implement a native Go encoder so
that I could get rid of the C TurboPFor dependency entirely. The answer is yes,
it is doable in a few days, and it isn t even that much slower: Go is at 76% of
C, see Debian/dcs commit
e920dc7.
The goal I set myself at that point was to see if I could learn enough SIMD to
optimize the native Go encoder such that its performance would match how DCS
uses C TurboPFor (via cgo).
Beating C TurboPFor was possible in 2-3 commits (SIMD and bit width
specialization). To my surprise, Claude Fable 5 pointed out that the encoder s
block scanning could be done more efficiently using a technique called
positional popcount, and that is another 2x speed-up!
To be clear: I am not saying the Go compiler beats C here. Certainly, the C
compiler can also produce fast AVX512 code and can be used to implement
positional popcount. When comparing apples to apples, i.e. backporting the
AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a
little slower at 1.4x C.
This spectacular result (much faster than what DCS had before) got me curious
how far I could push the decoder with SIMD after all. I ended up
matching/exceeding the cgo version here, too!
The rest of this article explains a few classes of optimizations I encountered
along the way.
Starting Point
When I wrote my goturbopfor teaching
decoder, I named its functions to
match the upstream C TurboPFor library, but now I want to get away from names
like p4ndec256v32 they make sense from the TurboPFor perspective, but for
Debian Code Search, we can use cleaner names.
Before writing any code, I audited how DCS uses integer compression /
decompression.
API design: BlockEncoder, BlockDecoder and streaming
In Debian Code Search, we have the following usage patterns:
Partial Indexing: When a new package (or package version) enters Debian,
all of its (text) files are indexed. If the hello-2.12.3-1
package (hypothetically) contained
only hello.c with printf("hello!\n");, we would assign document ID 1 to
hello.c and store in the partial index that trigrams pri, rin, int,
ntf, etc. are all found in doc 1 (hello.c).
Full Index Merging: The many thousands of partial index files (for each
Debian package) are combined into a small handful of large index files: When
searching, it would be expensive to consult thousands of indexes. To merge
multiple partial index files into one larger index (which can then be
efficiently queried), we need to re-encode the partial index files: what used
to be document ID 1 in the partial index might be document ID 2531 in the
full index.
Querying (searching): When users enter search queries, these queries need
to be answered as quickly as possible. The relevant entries in the full
indexes are decoded (in parallel).
For reading the index, we do keep the decoded uint32s fully in memory, so we
only need DecodeN(input []byte, output []uint32) (read int), a function that
reads len(output) values (uint32) from input and returns how many bytes it
consumed.
For writing the index (both in partial indexing, and when merging), keeping the
entire index in memory is prohibitively expensive, so we need a streaming API,
for decoding and for encoding.
Ultimately, I converged on the following API:
packagepforenctypeBlockEncoderstruct// scratch buffers can go here// EncodeBlock encodes len(vals)<=256 uint32s into dest (one TurboPFor block).func(*BlockEncoder)EncodeBlock(dest[]byte,vals[]uint32)[]byte// EncodeN calls EncodeBlock in a loop.func(*BlockEncoder)EncodeN(dest[]byte,vals[]uint32)[]bytetypeStreamEncoderstructbeBlockEncodervals[256]uint32// scratch buffers// if full, you need to call [EncodeBlock]func(*StreamEncoder)Add(valuint32)(fullbool)// EncodeBlock must be called after all data was [Add]ed.//
// Write the returned buffer to file or send it over the network;// it is only valid until the next [EncodeBlock] call.func(*StreamEncoder)EncodeBlock()[]byteifse.n==0returnnil// turn an extra EncodeBlock into a no-op//
This API (the decoder works similarly) allows us to process data in TurboPFor
format without any memory allocations. The types are not safe for concurrent use
by multiple goroutines. The zero value is ready to be used. For the streaming
API, the result only stays valid until the next call.
Initial Implementation
Before we can optimize anything, we need a working decoder and encoder. The
decoder already exists: my goturbopfor teaching decoder. Next up, I needed an
encoder.
Writing a TurboPFor encoder has a delightfully simple starting point: You can
encode all values at bit width 32, in little endian, at which point you only
need to add a one-byte TurboPFor block header every 256 values and you re done:
Of course, this is a terribly inefficient compressor, so after the first commit,
the real work starts: implement each block type until the compression matches
the original C TurboPFor implementation (same output file size), or in other
words: do the reverse of the decoder.
The TurboPFor bitpacking block type (bitpacking implementation
commit)
encodes a bit stream of variable bit width (where the bit width is in range
0 bitWidth 32) in little
endian byte order. By scanning all
values and choosing the smallest bit width that allows representing all
values, this technique saves disk space (compresses).
The bitpacking with exceptions block type (bitpacking with exceptions
implementation
commit)
determines two bit widths: one for values, the other bit width for encoding
exceptions. This allows choosing a lower bit width (that does not cover all
values) compared to the bitpacking block type. A bitmap encodes whether a
value has an exception or not.
The bitpacking with VB exceptions block type (bitpacking with VB
exceptions implementation
commit)
is a variant which does not use an exception bitmap and encodes exceptions
using a variable byte integer encoding. This is more efficient when there are
few exceptions (less than 20) or the exceptions are very different in bit
width compared to the other values.
Lastly, the constant block type (constant implementation
commit)
stores just one value on disk. This is useful for all-zero or all-one blocks,
for example.
I found it interesting to realize that the main work of the encoder is to scan
the input values and choose the optimal block type, whereas the actual encoding
itself is cheap in comparison.
At this point, we can look at
performance
and see that the Go encoder is at 76% of the C encoder.
In all honesty, I could have probably stopped here, but now that the milestone
of a viable replacement was reached, I got curious to see how far it would be
possible to push the encoder (how much work to reach C speeds?) and afterwards,
the decoder, too.
Setup
The microarchitecture level: set GOAMD64
The microarchitecture of a CPU determines which instructions it provides, and
that includes not just SIMD instruction sets (like AVX2), but also other useful
instructions like LZCNT (Leading Zero Count), which can be used to implement
math/bits.Len32 more efficiently, which
the TurboPFor encoder needs to call on every input value to determine the ideal
bit width.
Let s walk through how to set the microarchitecture level when using Go on
64-bit x86 (x86-64).
Go uses the GOARCH environment
variable to configure the target
compilation architecture, and I am using the value amd64 to select 64-bit x86
(AVX2 and AVX512 are instruction sets found on x86-64 CPUs). With
GOARCH=amd64, the architecture-specific variable
GOAMD64 configures the
microarchitecture level for which to compile and Go 1.18 introduced these 4
different levels:
GOAMD64=v1 (default): The baseline.
Exclusively generates instructions that all 64-bit x86 processors can execute.
GOAMD64=v2: all v1 instructions,
plus CMPXCHG16B, LAHF, SAHF, POPCNT, SSE3, SSE4.1, SSE4.2, SSSE3.
GOAMD64=v3: all v2 instructions,
plus AVX, AVX2, BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, OSXSAVE.
GOAMD64=v4: all v3 instructions,
plus AVX512F, AVX512BW, AVX512CD, AVX512DQ, AVX512VL.
In 2026, I generally recommend compiling with GOAMD64=v3 so that functions
like bits.OnesCount8 are
compiled into intrinsics
(POPCNT)
instead of using a lookup table.
For Intel CPUs, setting GOAMD64=v3 means your programs will only start on
Haswell CPUs (2013) or newer; for AMD CPUs that means Zen 1 (2017) or newer.
In this specific case (DCS), I am even compiling with GOAMD64=v4. The v4
microarchitecture level requires AVX512, which means AMD Zen 4, Zen 5 or newer
(Intel s story is complicated). Luckily, both my main development PC (Zen 5)
and the Debian Code Search server (Zen 4) are recent enough. Setting
GOAMD64=v4 has little effect on Go 1.27 itself: the only change is that maps
use one less instruction (VPBROADCASTB instead of PSHUFB). But compiling
with GOAMD64=v4 allows us to move one more feature check from runtime to
compile time, see SIMD build tags.
It makes sense to set the microarchitecture level in your benchmark setup so
that you don t measure the slow fallback implementations. I use export GOAMD64=v4 in my Makefile.
Benchmarking setup
Go s built-in testing package contains support for
benchmarks which are written in
functions of the form func BenchmarkXxx(b *testing.B). The simplest way to run
such benchmarks is go test -bench=., but I ended up configuring a few
convenience make targets, which write results to bench.txt and compare
against baseline.txt (the previous commit s results, usually), using the very
useful benchstat tool.
GOTEST=go test# -count=6 gives p 0.002 in benchstat:
# https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
BENCHFLAGS=-run=^$$ -bench=. -benchtime=200000x -count=6# use taskset -c1 to always pin to the same single core,
# avoiding accidental scheduling on different cores on
# mixed-core CPUs like the Ryzen 9 9950X3D.
TASKSET=taskset -c 1BENCH=$(TASKSET)$(GOTEST)$(BENCHFLAGS).PHONY: all test bench bench-baseline bench-relative
all: test
bench: test
$(BENCH) tee bench.txt
# Compares compression ratio between C and Go implementation
benchstat -col /impl -row '/n /vals' -filter '-/impl:go-stream .unit:(encoded-bytes)' bench.txt
# Compares performance between C (cgo) and Go implementation
benchstat -col /impl -row '/n /vals' -filter '.unit:(Mval/s)' bench.txt
bench-baseline: test
$(BENCH) tee baseline.txt
bench-relative: test
$(BENCH) tee bench.txt
benchstat -filter '-/impl:go-stream .unit:(encoded-bytes)' baseline.txt bench.txt
benchstat -filter '/impl:go .unit:(Mval/s)' baseline.txt bench.txt
The encoded-bytes and Mval/s units are custom metrics I am reporting from
the various sub-benchmarks, which are arranged
such that I can filter / report them with benchstat.
The main encoder (and decoder) benchmarks compare 3 different implementations
(cgo, Go, Go with the StreamEncoder API) with a number of benchmark
cases
that are designed to cover the different block types and contain a similar mix
of values as what we see in Debian Code Search:
// reportMetrics adds Mval/s and encoded-bytes metrics to all benchmarks.funcreportMetrics(b*testing.B,nint,nencodedint)b.ReportMetric(float64(nencoded),"encoded-bytes")b.ReportMetric(float64(b.N*n)/1e6/b.Elapsed().Seconds(),"Mval/s")// BenchmarkEncode/n=<N>/vals=<testcase>/impl=<c go go-stream>//
// e.g. BenchmarkEncode/n=2048/vals=one-constant/impl=go-streamfuncBenchmarkEncode(b*testing.B)for_,tc:=rangeallBenchCases()n:=len(tc.vals)b.Run(fmt.Sprintf("n=%d/vals=%s",n,tc.name),func(b*testing.B)b.Run("impl=c",func(b*testing.B)b.ReportAllocs()varencoded[]bytebuf:=make([]byte,turbopfor.EncodingSize(n))forb.Loop()encoded=turbopfor.P4nenc256v32Buf(buf,tc.vals)reportMetrics(b,n,len(encoded)) )b.Run("impl=go",func(b*testing.B)b.ReportAllocs()varbeBlockEncodervarencoded[]bytebuf:=make([]byte,0,turbopfor.EncodingSize(n))forb.Loop()encoded=be.EncodeN(buf,tc.vals)reportMetrics(b,n,len(encoded)) )b.Run("impl=go-stream",func(b*testing.B)b.ReportAllocs()varseStreamEncodervarencodedintforb.Loop()encoded=0for_,val:=rangetc.valsifse.Add(val)encoded+=len(se.EncodeBlock())encoded+=len(se.EncodeBlock())reportMetrics(b,n,encoded) ) )
CPU counters: perf
Go has included excellent performance tooling for many years, see the
Profiling Go Programs blog post (2011) for an
example of how to use pprof, a sampling profiler. This profiler can help track
down which part of a program runs slow, or where memory allocations happen.
Once you identified the slow part of a program, how do you know why it s slow?
To learn more about the specific bottlenecks your program encounters, you can
consult your CPU s hardware performance
counters. For
example, you could check the branch
predictor counters to see if
your program is slow due to a high number of branch mispredicts.
On Linux, the perf tool is the best way to access the CPU hardware performance
counters. A good starting point for working with perf is the documentation on
Top-down analysis with the perf
tool , which describes the
optimization method that Intel established.
In my Makefile, I set up two perf targets:
# GOTEST and TASKSET like shown in the earlier benchmarking setup section:
GOTEST=go test -pgo=encode.cpuprof
TASKSET=taskset -c 1PERFBENCHFLAGS=-test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=200000x
# Use perf(1) to capture AMD IBS (the equivalent to Intel PEBS)
# PipelineL1 is roughly equivalent to Intel TopdownL1
perf:$(GOTEST) -c
$(TASKSET) perf stat -M PipelineL1 ./pforenc.test -test.run=^$$$(PERFBENCHFLAGS) sudo perf record -F 4999 -e ibs_op// --call-graph fp ./pforenc.test -test.run=^$$$(PERFBENCHFLAGS) sudo chmod 644 perf.data
# 488281 iterations 2048 values = 1.000e9 values, so counter/1e9 = per value.
perf-per-value:$(GOTEST) -c
$(TASKSET) perf stat -x, -e cycles:u,instructions:u,branches:u,branch-misses:u ./pforenc.test -test.run=^$$ -test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=488281x 2>&1 >/dev/null awk -F, ' printf "%-16s %6.2f /val\n", $$3, $$1/1e9 '
The perf-per-value numbers are high level numbers that indicate how much work
the implementation is doing. Reducing the number usually increases speed.
To see the counters for each instruction (and source code lines), I use make perf, followed by perf report. A quick shortcut is perf annotate, which
directly shows the hottest function.
Optimizations (scalar)
Let s first see how far we can get without reaching for SIMD instructions.
(The examples are not necessarily in commit order, but cherry-picked for clarity.)
Profile-Guided Optimization (PGO)
PGO stands for Profile-Guided Optimization and is a feature that Go introduced
as a preview in Go 1.20 (released in February 2023) and shipped as ready for
general production use in Go 1.21 (released in August
2023).
The idea is to capture a CPU profile that records where your program spends most
of its CPU time, which you then provide to the Go compiler to give it more data
to make better decisions.
Most importantly, this way the Go compiler can inline functions much more
aggressively than its usual heuristics allow,
which does have a measurably positive effect in my series of optimization
commits. Another optimization that a PGO profile allows the compiler to do is
conditional devirtualization but
our TurboPFor code does not use any interfaces.
My strategy is to enable PGO before doing any other optimizations, so that we
have the full inlining budget available that PGO gives us, and can measure the
effect of other commits clearly.
Surprisingly, turning on PGO actually decreases our
performance
(-13% geomean), but a closer investigation reveals that we just got unlucky. Let
me explain.
Aside from inlining and conditional devirtualization, PGO also influences
alignment: The Go compiler sets PCALIGNMAX(64, 31) on the first block of a
loop (the loop body ) for all loops in hot functions (per the PGO profile),
i.e. Go will insert up to 31 bytes of padding to make the block land on a
64-byte boundary. Documentation like AMD s Software Optimization Guide for the
AMD Zen5 Microarchitecture (2024,
#58455) explicitly recommends aligning hot loops that way:
[ ] for hot loops, some further knowledge of trade-offs can be
helpful. Because the processor can read an aligned 64-byte fetch block every
cycle, it is suggested to either align the start of the loop to the beginning
of a 64-byte cache line [ ]
Indeed, when compiling with -gcflags=all=-d=alignhot=0 to disable the
alignment, performance remains as good as without PGO. How can the padding hurt
more than help? The answer is: It s not the padding itself! It s a side-effect
of the padding moving instructions to different addresses.
In the unlucky arrangement, a macro-fused CMPQ+JGE instruction pair now ends
up exactly on a 32-byte boundary. However, the Go compiler ensures fused
branch sequences must never cross or end at a 32-byte boundary to fix Intel
erratum
SKX102
(discussion: Go issue #35881) by
inserting NOPs.
This NOP padding, unlike the loop alignment padding, is not free; these extra
instructions slow down our otherwise dispatch-bound loops.
Because the commits after the PGO enabling commit change the code, this unlucky
situation is avoided for the rest of the optimization series (by chance).
Reducing memory allocations
Memory allocations are quite expensive, at least in comparison to
encoding/decoding integers, so I followed my usual strategy of first reducing
memory allocations as much as possible.
In my goturbopfor teaching decoder, whenever the code needed a scratch buffer,
it would allocate it right then and there with make():
// p4dec32 decodes one block of TurboPFor-encoded 32 bit intsfunc(d*decoder)p4dec32(input[]byte,output[]uint32)(readint)// switchblockTypecaseblockBitpackingExceptions:bx,input:=input[0],input[1:]n:=len(output)exmap:=inputnex:=0// number of exceptionsfori:=0;i<n;i++ifexmap[i/8]&(1<<uint(i%8))!=0nex++input=input[(n+7)/8:]exceptions:=make([]uint32,nex)input=input[bitunpack32(input,exceptions,bx):]input=input[d.bitunpack(input,output,b):]fori:=0;i<n;i++ifexmap[i/8]&(1<<uint(i%8))!=0output[i]+=exceptions[0]<<bexceptions=exceptions[1:]returnbefore-len(input)
The Go compiler can turn make(T, n) calls into stack allocations, if n is
known at compile-time. But, in this case nex is not known at compile-time. We
can verify that Go calls into the runtime (runtime.makeslice) by dumping the
object code (assembly) with source annotated (-S):
An easy speed-up was to avoid allocations through
reuse
(in goturbopfor). In the DCS pfordec package (with the improved
API design), I ended up with a vals [256]uint32 field in the StreamDecoder
type, which brings us from 773 Mval/s to 858 Mval/s on the debian-mix:
Aside from the speed-up, avoiding memory allocations is generally nice in
benchmarks because it removes the garbage collector from the equation and makes
it less likely that your benchmarks get other processes OOM-killed on the same
machine.
Generics for bit width specialization
In general, we want to make it easy for the compiler to understand as much as
possible about our algorithm. Consider this bitpack implementation:
Let s think through what determines the iterations and control flow this
function uses:
The number of input values (vals), but not their actual value.
The bit width to pack into (bitWidth).
With a bit of careful rearrangement, we can provide the compiler with both, a
fixed number of input values (say, 32), and a bit width, both known at compile
time. Why is this worthwhile? Because we can manually unroll the loop, let the
compiler eliminate much of the repetition and get much faster compiled code as a
result!
Let s first fix the number of input values to 32 and rewrite the loop to
calculate the position offsets within dest instead of changing dest on each
value (with AppendUint32):
funcbitpack32Unrolled(dest[]byte,vals*[32]uint32,bitWidthint)// only one bounds check for 32 valuesdest=dest[:4*bitWidth:4*bitWidth]mask:=uint32(1<<bitWidth-1)varaccuint64varhave,posint// Manually unrolled loop starts here.// Each iteration is identical except for the vals[x] index.acc =uint64(vals[0]&mask)<<havehave+=bitWidthifhave>=32binary.LittleEndian.PutUint32(dest[pos:pos+4],uint32(acc))pos+=4acc>>=32have-=32// vals[1] .. vals[30] elided for brevity// Each loop iteration is 8 lines of Go code, so for 32 input values,// bitpack32Unrolled contains 8*32 = 256 lines of code.acc =uint64(vals[31]&mask)<<havehave+=bitWidthifhave>=32binary.LittleEndian.PutUint32(dest[pos:pos+4],uint32(acc))pos+=4acc>>=32have-=32// have == 0; for all bitWidths
Next, we want to specialize not just for 32 input values, but also for each of
the 32 bit widths.
Can we do better than hand-copying bitpack32Unrolled 32 times (= 8192 lines of Go code)?
Yes, we can use Go generics to help us with the code generation!
In Go, array types like [4]byte (not
slices like []byte!) contain the length of the array as part of their type,
meaning [1]byte (an array of length 1) is a different type than [2]byte.
Instead of passing the bit width as a function parameter, we can declare 32
different types (one for each bit width) and recover the bit width (at compile
time!) from the type system:
typebitWidthTinterface[1]byte[2]byte[3]byte[4]byte[5]byte[6]byte[7]byte[8]byte[9]byte[10]byte[11]byte[12]byte[13]byte[14]byte[15]byte[16]byte[17]byte[18]byte[19]byte[20]byte[21]byte[22]byte[23]byte[24]byte[25]byte[26]byte[27]byte[28]byte[29]byte[30]byte[31]byte[32]bytefuncbitpack32Unrolled[TbitWidthT](dest[]byte,vals*[32]uint32)varzeroTbitWidth:=len(zero)// known at compile timedest=dest[:4*bitWidth:4*bitWidth]// make cap known at compile timemask:=uint32(1<<bitWidth-1)varaccuint64varhave,posint// Manually unrolled loop starts here.// Each iteration is identical except for the vals[x] index.acc =uint64(vals[0]&mask)<<havehave+=bitWidthifhave>=32binary.LittleEndian.PutUint32(dest[pos:pos+4],uint32(acc))pos+=4acc>>=32have-=32// vals[1] .. vals[31] elided for brevity
When we instantiate bitpack32Unrolled[bitWidthT] with all 32 different types
([1]byte, [2]byte, , [32]byte), the compiler substitutes the bitWidthT
type parameter and produces 32 copies of the function, which we can find in our
compiled executable with names like
github.com/Debian/dcs/internal/turbopfor/pforenc.bitpack32Unrolled[go.shape.[12]uint8]. The
shape of a generic type is based on its memory layout, so a shape for
[1]byte must be different than the shape for [2]byte.
Because the bitWidth is now known at compile time, the Go compiler can
generate close to the optimal machine code for each bit width, which we can
confirm using go tool objdump.
The code is branchless (after the one bounds check per 32 values) and aside from
the loads and stores (from/to memory) consists only of shifts and bit
operations, all with constant operands:
Now we need to actually call bitpack32 from the general bitpack function:
funcbitpack(dest[]byte,vals[]uint32,bitWidthint)[]byteifbitWidth==0returndest// no payload, sparse block with only exceptionsiflen(vals)>=32size:=4*bitWidthforlen(vals)>=32existing:=len(dest)dest=slices.Grow(dest,size)[:existing+size]bitpack32(dest[existing:]/*append*/,(*[32]uint32)(vals),bitWidth)vals=vals[32:]mask:=uint32(1<<bitWidth-1)varaccuint64varhaveintfor_,val:=rangevalsacc =uint64(val&mask)<<havehave+=bitWidthforhave>=32dest=binary.LittleEndian.AppendUint32(dest,uint32(acc))acc>>=32have-=32forhave>0dest=append(dest,byte(acc))acc>>=8have-=8returndestfuncbitpack32(dest[]byte,vals*[32]uint32,bitWidthint)switchbitWidthcase1:bitpack32Unrolled[[1]byte](dest,vals)case2:bitpack32Unrolled[[2]byte](dest,vals)case3:bitpack32Unrolled[[3]byte](dest,vals)case4:bitpack32Unrolled[[4]byte](dest,vals)case5:bitpack32Unrolled[[5]byte](dest,vals)case6:bitpack32Unrolled[[6]byte](dest,vals)case7:bitpack32Unrolled[[7]byte](dest,vals)case8:bitpack32Unrolled[[8]byte](dest,vals)case9:bitpack32Unrolled[[9]byte](dest,vals)case10:bitpack32Unrolled[[10]byte](dest,vals)case11:bitpack32Unrolled[[11]byte](dest,vals)case12:bitpack32Unrolled[[12]byte](dest,vals)case13:bitpack32Unrolled[[13]byte](dest,vals)case14:bitpack32Unrolled[[14]byte](dest,vals)case15:bitpack32Unrolled[[15]byte](dest,vals)case16:bitpack32Unrolled[[16]byte](dest,vals)case17:bitpack32Unrolled[[17]byte](dest,vals)case18:bitpack32Unrolled[[18]byte](dest,vals)case19:bitpack32Unrolled[[19]byte](dest,vals)case20:bitpack32Unrolled[[20]byte](dest,vals)case21:bitpack32Unrolled[[21]byte](dest,vals)case22:bitpack32Unrolled[[22]byte](dest,vals)case23:bitpack32Unrolled[[23]byte](dest,vals)case24:bitpack32Unrolled[[24]byte](dest,vals)case25:bitpack32Unrolled[[25]byte](dest,vals)case26:bitpack32Unrolled[[26]byte](dest,vals)case27:bitpack32Unrolled[[27]byte](dest,vals)case28:bitpack32Unrolled[[28]byte](dest,vals)case29:bitpack32Unrolled[[29]byte](dest,vals)case30:bitpack32Unrolled[[30]byte](dest,vals)case31:bitpack32Unrolled[[31]byte](dest,vals)case32:bitpack32Unrolled[[32]byte](dest,vals)
Encoding remainder blocks is quite a bit faster (full blocks use the vertical
layout anyway):
This performance win comes at the cost of binary size increase. In this case,
the .text section (executable code) grows by about 20 KB and the .gopclntab
section grows by another 26 KB. Definitely a price I am very willing to pay, but
the case might not be as clear in all circumstances.
Optimization: Bigger strides with SIMD
Even without reaching for SIMD instructions, a TurboPFor implementation can be
made faster by making it work bigger strides. Take this code from the
goturbopfor teaching decoder which counts the number of exceptions by checking
if each value s bit is set in the exception bitmap:
caseblockBitpackingExceptions:bx,input:=input[0],input[1:]n:=len(output)exmap,input:=input,input[(n+7)/8:]nex:=0// number of exceptionsfori:=rangenifexmap[i/8]&(1<<uint(i%8))!=0nex++exceptions:=d.scratch[:nex]
We can use the bits.OnesCount64
functions to count ones bits in the exception bitmap, 64 values at a time. For
remainder blocks, the rest is processed 8 values (1 byte) at a time:
i:=0for;i+8<=n/8;i+=8xm8:=binary.LittleEndian.Uint64(exmap[i:])nex+=bits.OnesCount64(xm8)for;i<(n+7)/8;i++xmb:=exmap[i]// Clear the bits which do not belong to the exception map:ifrem:=n-i*8;rem<8xmb&=1<<rem-1// Go compiles OnesCount32 into an intrinsic,// but not OnesCount8, so we convert to uint32:nex+=bits.OnesCount32(uint32(xmb))
OnesCount64 uses a 64-bit register. For comparison, AVX2 SIMD instructions use
256-bit registers (= 8 uint32) and AVX512 SIMD instructions use 512-bit registers.
In the following sections, we will first set up our build tags for conditional
compilation to use a trivial SIMD instruction, then walk through an AVX2 and
AVX512 SIMD kernel.
SIMD build tags
Let s assume we have the following scalar code:
constant.go:
To increase throughput, we can use AVX2 instructions if they are available on the CPU on which the program runs, i.e. using runtime dispatch. We ll first rename fillConstant to fillConstantScalar (it s now the fallback path):
constant.go:
Next, we ll supply two different implementations (constant_nosimd.go and constant_amd64.go), the latter of which is selected when compiling for GOARCH=amd64 with GOEXPERIMENT=simd (the latter will hopefully be dropped in a later version of Go). The nosimd variant just dispatches to the fillConstantScalar, which will likely be inlined:
The constant_amd64.go variant assigns the hasAVX2 global variable by doing a CPUID check and then jumps to the scalar fallback if !hasAVX2, i.e. the CPU is too old:
//go:build goexperiment.simd && amd64packagepfordecimport"simd/archsimd"varhasAVX2=archsimd.X86.AVX2()funcfillConstant(output[]uint32,valuint32)if!hasAVX2fillConstantScalar(output,val)returnval8:=archsimd.BroadcastUint32x8(val)i:=0for;i+8<=len(output);i+=8val8.StoreArray((*[8]uint32)(output[i:i+8]))// use the scalar implementation for the last <= 7 elementsfillConstantScalar(output[i:],val)
We can go one step further by conditionally compiling const hasAVX2 = true
when GOAMD64 is set to v3 or higher (i.e. the amd64.v3 build tag is
set). As a practical example from Debian Code Search, we currently need the
following checks / dispatches:
In regular (non-SIMD) bitpacking, integers are stored on disk one after
the other, padded to a full byte, as a byte is the smallest addressable unit
when reading data from disk. For example, if you bitpack only one 3 bit int,
you will end up with 5 bits of padding.
SIMD bitpacking works like regular bitpacking, but processes 8 uint32
little-endian values at the same time, leveraging the AVX instruction
set. The following
illustration shows the order in which 3-bit integers are decoded from disk:
The scalar implementation uses an array of 8 uint64 to process 8 values at a
time:
funcbitunpack256v32(input[]byte,dest[]uint32,bitWidthint)(readint)mask:=uint64(1)<<bitWidth-1orig:=len(input)varbitsuintvaracc[8]uint64// accumulator: current+next bitsforop:=0;op<len(dest);ifbits<uint(bitWidth)// read 8 more uint32sfori:=range8acc[i] =uint64(binary.LittleEndian.Uint32(input))<<bitsinput=input[4:]bits+=32fori:=range8dest[op]=uint32(acc[i]&mask)op++acc[i]>>=bitWidthbits-=uint(bitWidth)returnorig-len(input)
The SIMD version also processes 8 values, but without a for i := range 8 loop!
One difference is that we no longer have the luxury of using uint64 for acc
(holding rest and current bits); because AVX2 registers only fit 8 uint32 (not
8 uint64). Instead, we split acc into rest8 and cur8.
funcbitunpack256v32(fullinput[]byte,fulldest[]uint32,bitWidthint)(readint)dest:=fulldest[:256]ifbitWidth==0clear(dest)return0n:=32*int(bitWidth)input:=fullinput[:n]// tell the Go compiler how long the input ismask8:=archsimd.BroadcastUint32x8(uint32(1)<<bitWidth-1)bitWidth8:=archsimd.BroadcastUint32x8(uint32(bitWidth))varbitsuintpos:=0// var acc [8]uint64varrest8archsimd.Uint32x8varcur8archsimd.Uint32x8forop:=0;op<256;op+=8ifbits<uint(bitWidth)// read 8 more uint32s// acc[i] = uint64(binary.LittleEndian.Uint32(input)) << bitsnext:=archsimd.LoadUint8x32(input[pos:pos+32]).ReshapeToUint32s()pos+=32// input = input[4:]cur8=rest8.Or(next.ShiftAllLeft(uint64(bits)))// acc[i] >>= bitWidthrest8=next.ShiftAllRight(uint64(uint(bitWidth)-bits))bits+=32elsecur8=rest8// acc[i] >>= bitWidthrest8=rest8.ShiftRight(bitWidth8)// dest[op] = uint32(acc[i] & mask)cur8.And(mask8).Store(dest[op:op+8])bits-=uint(bitWidth)returnn
The SIMD version benchmarks about 3x as fast as the scalar version.
Another significant speedup is to use generics for bit width
specialization for this SIMD kernel so that
bitWidth becomes a compile-time constant and the compiler can generate better
code.
Positional Popcount
For my TurboPFor encoder, I implemented the same techniques as described above:
These changes are sufficient to roughly match the cgo performance, but then
Claude Fable 5 found another 2x speed-up on top of that!
The key observation is that once encoding blocks is fast, the preceding step of
scanning the input values to decide which block type to use becomes the
bottleneck. Here is the encoder s main encode function, which first does one
pass over the input values (scan) and then prices all different block types at
all relevant bit widths (requires fast access to the scan histogram):
func(be*BlockEncoder)encode(dest[]byte,vals[]uint32,layoutblockLayout)[]bytevarstatsstatsscan(&stats,vals)// gathers statistics from every value in valsbitWidth:=bits.Len32(stats.or)ifstats.or==stats.andreturnbe.encodeConstant(dest,vals,bitWidth)n:=len(vals)// bitpacking is the default, unless we find a more efficient block type.bestType:=blockBitpackingbestB:=bitWidthbest:=priceBitpack(n,bitWidth,layout)// Walk from high bitWidths to low: to break ties, we prefer// the encoding with fewer exceptions (for faster decoding).forb:=bitWidth-1;b>=0;b--// up to 32 iterationsnex:=int(stats.cnt[b])size:=priceBitpackExceptions(n,b,bitWidth,nex,layout)ifsize<bestbestType=blockBitpackingExceptionsbestB=bbest=size// Over-approximate the number of VB bytes.vb:=nex+// exceptions using 1, 2, 3, 4, or 5 VB bytesint(stats.cnt[b+7]+// exceptions using 2, 3, 4, or 5 VB bytesstats.cnt[b+14]+// exceptions using 3, 4, or 5 VB bytesstats.cnt[b+19]+// exceptions using 4 or 5 VB bytesstats.cnt[b+24])// exceptions using 5 VB bytessize=headerBytes+headerExBytes+payloadBytes(n,b,layout)+vb+nexifsize<bestbestType=blockBitpackingVBExceptionsbestB=bbest=sizeswitchbestTypecaseblockBitpacking:returnbe.encodeBitpack(dest,vals,layout,bitWidth)caseblockBitpackingExceptions:returnbe.encodeBitpackExc(dest,vals,layout,bestB,bitWidth-bestB)caseblockBitpackingVBExceptions:returnbe.encodeBitpackVBExc(dest,vals,layout,bestB,int(stats.cnt[bestB]))default:panic("BUG: bestType not implemented")
I ll show you a slightly shortened version of scan, the function which is the bottleneck:
typestatsstruct// cnt[n] = how many values where bits.Len32(val)>n,// i.e. how many exceptions are required for bitWidth=n.// Padded so that cnt[b+24] is always in bounds.cnt[32+24]uint32funcscan(output*stats,vals[]uint32)for_,val:=rangevalsforb:=rangebits.Len32(val)output.cnt[b]++// b bits are not enough to store val
Let s consider the following 3 example values to understand the resulting cnt:
input
input (bin)
bits.Len32
23
0b0000010111
5
5
0b0000000101
3
666
0b1010011010
10
The resulting cnt exception count histogram would contain (cnt shortened to c):
c[0]
c[1]
c[2]
c[3]
c[4]
c[5]
c[6]
c[7]
c[8]
c[9]
c[10]
3
3
3
2
2
1
1
1
1
1
0
In words, this means that at bit width 10, we could encode all the values
without any exceptions.
But most values do not need 10 bits, so a bit width of 5 would be more
efficient, but requires storing one exception. Encoding at bit width 4 requires
2 exceptions, and so on.
The scan function above is intentionally kept simple for illustration. We can
make it faster by moving the per-bit-width loop outside the per-element
loop.
The fast version still needs about 12 instructions per value. With SIMD, we can
reduce this to by 8x to only 1.5 instructions per value!
The trick: smear masks enable positional popcount
The trick is to turn each input value into its smear mask (imagine taking the
first 1 bit and smearing it across the remaining positions). Here are the smear
masks for our example:
input
input (bin)
bits.Len32
smear mask
23
0b0000010111
5
0b0000011111
5
0b0000000101
3
0b0000000111
666
0b1010011010
10
0b1111111111
Turning a value into its smear mask is computationally cheap:
Go implements BitLen(x) (functions like bits.Len32) by calculating 32 - LZCNT(x). We can calculate the smear mask of a value with ^uint32(0) >> LZCNT(x), i.e. starting with a 32-one-bits mask and shifting it by the number
of leading zeros.
Now, to obtain e.g. cnt[4], we can count the 1 bits at bit position 4 of all
input values.
The POPCNT instruction counts bits very efficiently, but it counts one bits
within a register, so it counts rows, not columns. Counting columns is called
Positional Population Count.
I found the following papers that describe positional popcount with SIMD:
Positional Popcount: a visual explanation
To understand the AVX512 implementation of positional popcount, I found it most
helpful to visualize an AVX512 register (512 bits, i.e. 64 bytes). The graphic
below uses the Uint64x8 layout, meaning it divides the register into 8 lanes
of 64 bits (= 8 bytes) each.
This illustration shows the whole process: how uint32s are loaded into an
AVX512 register (all 4 of its bytes, in sequence) and where we end up, i.e. the
32 positional popcounts:
Let s break down this process into its individual steps.
First, we turn each loaded value into its smear mask as explained above.
The VPOPCNTB vector instruction calculates POPCNT (1 byte) of 64 bytes at
once, but first we need to shuffle the bytes inside the register: in load
order, we have a full uint32 (4 bytes), followed by another uint32, per
lane. First, we permute the bytes (VPERMB) such that all the first bytes of
each value end up in one lane ( transpose the bytes ):
Next, we transpose the bits using the GF2P8AFFINEQB instruction, which
sounds scary but turns out to be quite flexible for bit manipulation of all
kinds. The GF2P8AFFINEQB instruction is also the star of the show in Go s
Green Tea Garbage Collector (2025). Here is the
bit transpose, shown in the AVX512 register layout (see below for a different
layout):
I found it easier to understand the transpose step when arranging the 8 bytes of
lane 0 from top-to-bottom (instead of left-to-right), because then it looks like
a 90 degree clockwise rotation:
Now we can use VPOPCNTB to count the bits in all 64 bytes at once:
After all loop iterations (processing 16 values each) are done, we add the two
groups (first 8 values, second 8 values) to obtain the 32 exception counts:
Positional Popcount: Go SIMD
Here is the Go code that implements what I described visually above:
funcscanSIMD(output*stats,vals[]uint32)ones16:=archsimd.BroadcastUint32x16(^uint32(0))// 16 32-one-bits masksshuffle:=archsimd.LoadUint8x64Array(&scanShuffle)units:=archsimd.LoadUint8x64Array(&scanUnits)varaccarchsimd.Uint8x64idx:=0for;idx+16<=len(vals);idx+=16v:=archsimd.LoadUint32x16(vals[idx:idx+16])// Replace all values with their smear masks.smear:=ones16.ShiftRight(v.LeadingZeros()).ReshapeToUint8s()// Transpose: shuffle the bytes, then transpose the bits.matrices:=smear.Permute(shuffle).ReshapeToUint64s()transposed:=units.GaloisFieldAffineTransform(matrices,0)// Popcount 64 bytes at once into the accumulator.acc=acc.Add(transposed.OnesCount())// Store the accumulator into output.cnt:// Widen the two groups of byte counts to uint16 lanes (so that// 128+128 = 256 fits), fold them into cnt[b] for b=0..31,// then widen again to the uint32 lanes of output.cnt.sum:=acc.GetLo().ExtendToUint16().Add(acc.GetHi().ExtendToUint16())sum.GetLo().ExtendToUint32().Store(output.cnt[0:16])sum.GetHi().ExtendToUint32().Store(output.cnt[16:32])// scalar tail for the 0..15 remaining valuesfor_,val:=rangevals[idx:]forb:=rangebits.Len32(val)output.cnt[b]++
Go even faster?
The SIMD optimizations I showed above beat the cgo TurboPFor library that Debian
Code Search used before. When comparing apples to apples, i.e. backporting the
AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a
little slower at 1.4x C.
Could we make my Go TurboPFor implementation even faster, to truly match the C
speed?
Yes! But also no. Let me explain:
We could use more SIMD instructions to remove all code that still processes
one value at a time. For example, in my encoder s encodeBitpackVBExc
function. Or we could price all bit widths concurrently in encode. Or in
the decoder s exception apply code path.
But all of these SIMD instructions
make understanding (and changing) the code harder, so I am cautious regarding
which ones I introduce.
A big part of the performance gap is due to Go s bounds
checks. While it costs
performance, bounds checking is great for safety, so I will not turn off
bounds checking. The Go compiler eliminates a number of bounds checks when it
understands it s safe to do so. One optimization avenue could be to make the
prove pass in the Go compiler smarter to eliminate more bounds checks.
When doing mid-stack inlining (proposal
#19348)
(2017), Go sometimes needs to put NOP instructions into the binary so that
it can attach inlining markers. For dispatch-bound functions, these extra
NOPs can measurable slow down execution.
The Go compiler currently allows specifying the architecture (GOARCH=amd64)
and microarchitecture (GOAMD64=v3), but not a specific CPU architecture
(like AMD Zen 4). Therefore, CPU-specific workarounds for one vendor affect
all the generated code. The specific one I encountered in my code is that the
Go compiler emits XORL CX,CX before every POPCNT to break a
false-output-dependency from the Intel Sandy Bridge Skylake era, which is
unnecessary on AMD Zen CPUs.
I suspect that Go intentionally does not offer this level of customizability.
After all of the above points are addressed, what remains is better code
generation in specific cases. To illustrate what I mean, consider the example
of incrementing a loop variable, where Go re-derives an index every time:
Go: POPCNTL; ADDQ DI,CX; LEAQ (base)(CX*4) (3 instructions)
clang: popcnt; lea rax,[rax+4*rdi] (2 instructions)
Depending on the specific case, improving the compiler might be easy or
prohibitively complex. Often, such improvements are hard to measure conclusively.
Conclusion
Go s SIMD support makes available in Go code without having to resort to cgo
or assembly a powerful part of modern CPUs which allows speeding up the kind
of computation that TurboPFor needs by an order of magnitude!
I found it very valuable to use a coding agent (Claude Code, with Opus 5 and
Fable 5 in this case) to help with the many tedious parts of such performance
work (and still it took me weeks!). The LLM can read objdump output much faster
than I can, can see patterns and correlations I might never identify, never
becomes frustrated after a compiler error or runtime panic, and never runs out
of patience to run one more experiment, as long as I give it measurable and
reachable goals.
The performance of the SIMD code which one can get from the Go compiler is
pretty close to what a good C compiler like clang provides. The CPU performance
counters show value decoding speeds of 7 instructions/cycle (IPC) on a machine
where the maximum is 8 IPC.
To me, SIMD support is a very welcome addition to Go.
Following the previous steps, we are now interested in getting a graphical environment with a VSCodium, the opensource rebuild of the VSCode IDE.
Configuring the display and development environment
From the previous steps we had a virtual machine where we can login with a debian user, and we can start configuring a graphical desktop environment.
Install Gnome Flashback.
Gnome Flashback is a 2D version of the Gnome Desktop, it has a kind of year 2009 feeling but works well enough. We need a 2D desktop, as the Virtio display adapter does not work consistently with 3D enabled.
From the host connect to the VM display using a remote client:
$ virt-viewer dev-vm
or using the Remote Viewer app:
$ remote-viewer spice://localhost:5900
Install the Spice Agent package. The Spice Agent provides a shared clipboard between host and VM, and also adapts automatically the VM display and desktop when the window of the Spice client is resized.
It also makes sense to set our debian user to autologin in Gnome Fallback, and start Codium on session start.
This is how the environement should look like at this point:
Sharing source code from host to guest VM
Finally we need to make sure we have access in the dev-vm to our source code repositories.
For this I will share the directory /home/manu/Projects/git which is containing all my git projects on the host, to the dev-vm using virtiofs.
The configuration of virtiofs is fortunately possible using virt-manager, which will save us some
tedious XML editing.
Finally we mount the shared directory, and enable the mount on each boot.
Welcome to post 59 in the R4 series.
How did we get here: A initial words about GitHub.
GitHub Actions
provides (essentially unlimited) compute time. This further boosts a
service already in a market-dominating position: GitHub1 as a code repository.
Those of us old enough to remember the start of git (the
program and protocol) may remember the extremely bare-bones
initial hosting site repo.or.cz (launched in 2006). GitHub
came two years later, and put an enormous amount of focus into design
and user interfaces. To cut a long story short, GitHub won the services
war. And with it git won the platform war. To a first
approximation, everybody and everything is on GitHub.2 So
the repository is already dominant.3 And then free compute
was added.
So given its scale and positioning, and its essentially free
provisioning of free multi-core compute setups with generally decent
connectivity, widespread adoption happened. And as is goes, some
mischief is bound to happen. And it did. More on that below.
A few words about r2u: r2u makes all packages on
CRAN, i.e. the code repository
network for R, install fast, reliably and
easy on Ubuntu by making them available to apt,
the native package manager. It is to our knowledge also the first and
only time an entire open source programming repository is available in
binary form with all dependencies resolved. It is going strongly: the
last monthly use topped
five million packages. See the r2u website for more.
r2u and GitHub: For the first few years,
builds for r2u
were done locally on my machine, and then uploaded to the primary
repositry r2u.stat.illinois.edu. I do not recall systemic outages or
connection issues though occassional network timeouts were seen. Once we
started to support arm64 (in addition to the default amd64) binaries,
building those switched to GitHub Actions simply because they had
runners for arm64 while I had no arm64 hardware. The experience of
building packages (in bulk) was rather positive. So we investigated
builds for amd64 too. If memory serves we first did this for either one
of the semi-annual BioConductor updates. Before long, builds for amd64
followed meaning all of r2u was being built in GitHub Actions.
During these builds, I would regularly encounter builds failures:
cannot connect to r2u.stat.illinois.edu . I misdiagnosed this as a
resource issue on the GitHub side, and consequently made (several)
attempts at robustifying the builds via for example longer (download)
timeout limits as well as checks for build failures and conditional
rebuilds. Needless to say, and given what we know now (more on that
below), this did not work. But it went on for a few months this spring
and summer. What did work was to simply relaunch under re-run failed
jobs . Given the distributed nature of GitHub Action this generally
allocates to a different machine and address and succeeds. In the grand
scheme of things a nuisance as we a need second run, but given the
fourty (!!) concurrent jobs this tends to be quick. So a minor
nuisance.
This discribed the production side. On the
consumption side, one prominent user of r2u, especially at GitHub,
is our r-ci setup for
continuous integration. It too could fail at times, and a simple re-run
would fix it. Annoying, if addressable manually. Usage by others I
cannot monitor so I can only assume that the random failure nature must
have frustrated them too. Potentially a much bigger nuisance.
As users were getting annoyed, some took action. Jeffrey Girard opened discussion
topic #159 which contained a thorough investigation of his
confirming that only amd64 nodes were affected. This had not been
noticed before. Troy
Hernandez set up a full harness with tests in an ad-hoc repo
designed for repeated remote triggering. This also logged the IP
addresses for success or failure. Through both these approaches it
became (eventually) clear that the failures were limited to either
certain (individual) IP addresses, or IP subnets.
When taking the conversation back to network service at U of
Illinois, we realized that the issue was in fact caused by a network
policy at the university. And specific to GitHub.
In fact, what happened initially were waves of port scanning attacks
originating from GitHub IP addresses. As (essentially) anybody can run
code there, bad actors can too. The response from the university side
was reasonable and swift: Identified IP addresses were added to a
null-router that (essentially) swallows traffic. And that was the
cause of the perceived-as-random outages: Jobs that ended up failing at
GitHub Actions were the ones assigned to addresses that have previously
been seen as port scanning.
Shifting production: Once this was confirmed, I
investiaged alternatives. On the production side using
different machines would help. So I tried blacksmith.sh, a competing
alternate service offering faster runners as drop-in replacements for
the GitHub Actions runners. This worked great, until I ran up
against my free cpu minutes quota . In a mere two days (that were
arguably overly busy as it was shortly after CRAN reopened after the summer
break). Given that the service would not sponsor us a supported open
source software project with sufficient quota, we moved off blacksmith.sh after two days.
A first programmatic response: consumption-side: For
the r-ci client side,
it was straightforward to setup a check and subsequent workaround. When
curl fails with a silent HEAD attempt at the primary
repository failed, we take this to be caused by presence of a
null-router entry for the IP we are on, and switch the apt
setup to the secondary repository. Which may be slower, or at rare times
unreachable itself but still provides a fine fallback when a node is
prohibited from talking to U of Illinois resources such as
r2u.stat.illinois.edu. Having used this for a few days in r-ci it seems to work.
A second programmatic response: production-side: For
the r2u builds, and
given that blacksmith.sh would
not grant most-favored status with sufficient free minutes, we
switched our Docker-based setup to switch to the secondary when an
initial probe fails. That was added last weekend, and appears to work
just swimmingly. Another application to the fundamental
theorem of software engineering: another layer of indirection can
solve just about any problem.
For completeness, the corresponding code is
webstatus=$(curl--head--silent--no-fail--output /dev/null \--write-out"% http_code " https://r2u.stat.illinois.edu true)iftest"$ webstatus " = "200";thenecho"The r2u repository is reachable."elseextip=$(curl--silent https://ipinfo.io/ip)echo"::notice::The primary r2u repository is **not reachable** from $ extip ."fi
We run an initial curl test (without failing) and have
it report the HTTP return code. 200 means no issue, all others are
suspect here so we run a second curl query to obtain our
external IP and log it. We use the same logic in another spot from
inside the build container and use the else branch to switch
apt to the secondary repository via sed call
on the .sources file.
Logging of bad IPs: On both our sides,
i.e. production as well as consumption, we now also log the IP addresses
of the failing nodes and will ask network security to remove these from
the null router. If our jobs can be assigned to them it clearly shows
the machines are part of the normal compute pool and are not
doing anything nefarious at the moment. So they should be removed from
the null-router list. We will see how that fares.
Putting it all together: Providing a free resources
can, sadly, lead to an a decline the service experience just as the tragedy of
the commons analysis would predict. Restricting, or pricing use
may be a stock answer but I for one am glad GitHub Actions is still
free. But we need to do our bit of upkeep. Just as network security logs
bad actors (taking advantage of the free resource) we should make an
effort to unlist nodes no longer part of any portscan (or alike)
swarm.
For r-ci users, there is hopefully little to do (if
you rely on the standard
action). We do now catch a node that was assigned a continuous
integration job cannot connect to r2u as we can test this
easily (and cheaply). Pivoting to the secondary repository is a valid,
and working, answer. Hopefully over time we can also work towards
restricting the null-router list down to recent entries and fewer
overall, thereby lowering the chance of gitting a bad IP. Eventually, we
could also overly a CDN proxy to avoid the bad IP problem. It is
something to consider.
Summing up: We are still chuffed at how successful
r2u has become, and how
much can be done with GitHub Actions. Sadly, as
we found out, there can also be a tax on letting compute happen there
but as discussed in this note, there are ways to avoid it by pivoting to
alternate repository source.
Before we really get started, one clarification. GitHub and its services including GitHub Actions have been
in the news lately as they suffered a number of high-profile outages.
While also arguably a tragedy
of the commons problem, it is not what this note is about. If
you prefer to be enraged about GitHub services, or the (relevant) lack
thereof, this may not be for you.
The year is 2026 and politics is what it is, of course
non-US alternatives emerged and will remain available and used. But
dislodging established first-mover advantages will most likely take more
than a (at least for now still-small) number of users unhappy for
various (and sensible) reasons. We will see how this pans out.
Entire essays (or book) can be / will be / have been
written about the competitive situation, how GitLab did not make enough
of a dent, how Gitea remained niche and of course now Codeberg. This is
not that essay, and I do not have a strong view but let me mumble a
quiet plus a change, plus a reste la m me chose
Today marks the day of soft-launching of my Debian derivative, which I ve been using on several of my own machines for the past year or so. This is still work in progress, but I wanted to establish a launch date of the project so below is the DiffOS manifesto as motivation for continued work.
DiffOS is For Freedom! DiffOS is the Debian Increment For Freedom Operating System.
Support for all architectures supported by Debian.
Provide Containers, Cloud Images, LiveCD and installer ISOs.
Provide standalone hosting of the package repository.
Provide documentation and issue tracker.
Keep changes to a minimal, in particular:
Upstream-first policy to prefer that any changes are made in Debian, and only if that fails they are considered for DiffOS.
Binary package re-use for as much as is possible.
Don t modify any source-level Debian package unless REQUIRED by the FSDG (e.g., for freedom concerns) or REQUIRED by the Debian project (e.g., for branding reasons).
Publish a list of packages that are added, removed or modified compared to Debian, with justification for each change.
Publish Diffoscope-style outputs comparing our artifacts with comparable Debian artifact.
Everything built from CI/CD pipelines, inspired by the Salsa CI pipeline but extended to cover the package repository and installation images as well, to allow modern GitSecDevOps of the entire supply-chain.
Use inspiration from other Debian-derived FSDG distributions Trisquel GNU/Linux and PureOS, and broader with GNU Guix especially on how to approach existing freedom concerns in packages.
Git Forge agnostic. While currently hosted on GitLab.com, scripts and configuration are (or will be) designed to allow setup on self-hosted GitLab instance, Codeberg.org or self-hosted Forgejo.
Posted on September 2, 2026
Tags: madeof:atoms, craft:sewing, period:edwardian, FreeSoftWear
Many years ago, before I had my sewing pattern website, I made myself a
simple corset cover according to the instructions on an Edwardian
pattern drafting manual.
It worked, I wore it. Years later I saw a blog post on Pour La Victoire
on making a corset cover based on the same book, but with completely
different results, and thought that it would have been nice to make
another one to publish instructions for my take on it.
However, I didn t have any embroidery flouncing on hand, nor did I have
a need for a new corset cover, and the project remained on the list, on
low priority (although I did buy some beading lace for it, when I
stumbled on it).
Then, after finishing my vampire shirt, I noticed that I had just enough
fabric left for a corset cover, and by just enough I really mean just
enough, as I discovered when laying the pattern on the fabric.
So I dug in my files to get the original pattern I used, brought it up
to date, and added the missing details such as the pleating guides that
I had skipped when making the pattern just for myself. Doing so I
realized that on my old cover I had done the fake pleat in the front
wrong, making just a single pleat instead of a box pleat. Also, I
originally directly gathered the sleeves in the armscyes, but watching
the book again I realized that the sleeves were made up of a gathered
ruffle plus a straight band.
Both issues were fixed and I could cut the fabric and start sewing. By
machine, including using a narrow hem foot instead of sewing rolled hems
by hand as my instinct kept reminding me would have looked neater.
But this is a garment from a sewing machine time, and probably one that
in many cases would have been bought from a mass producer, and it s
underwear, so there is no real need for the hems to be perfect, as it s
going to be hidden anyway. But most importantly, I wanted to write
instructions for machine sewing, for a change, and so I had to machine
sew all steps that I had to take pictures of.
I did do the buttonholes by hand, because I hate the buttonhole
attachment on my machine, and the buttonhole attachment hates me.
I used a lighter weight fabric for the sleeve ruffles, both because I
didn t have a big enough piece of main fabric not to have to piece them,
and because I felt that it looks better, as it s the same voile I used
for the ruffles on the vampire shirt.
When it came to the beading lace, I had two that I had bought more or
less thinking about this project: the earlier one was narrow and
suitable to do its job, but the one I had bought more recently was
taller, with an edge that made it suitable to give more fullness to the
bust when gathered up.
I contemplated for a short while, and then decided to go for fullness
and use the taller border for the top edge, but the smaller one at the
waist, where fullness is not wanted.
The book claimed that this pattern required little labour, and indeed it
did: even when taking step by step pictures it only took a few hours
spread over a week, plus the time to make buttonholes by hand over the
next week.
And then the reason for the whole project: I published my pattern and
instructions under a free license.
I still haven t worn the corset cover, except for these pictures, but I
hope to do so later in the year when the weather becomes more reasonable.
Preface: The Unanswered Frontier
In Part 1: Taming the AI Agents, I shared the architectural blueprint of CAMP (Cross-Agent Memory Protocol) how we used Linux Bubblewrap (bwrap), camp-acpd, OPA policy enforcement, and a central pgvector MemPalace to bring deterministic discipline, sandboxing, and long-term memory to a heterogeneous fleet of AI coding assistants (Claude Code, Google Antigravity, Grok Build, and GitHub Copilot).
At the end of that article, however, I highlighted a significant hurdle: The Headless Limitation.
While passive A2A works beautifully for structured handoffs, the current frontier of agentic design faces a key limitation: agents are not yet fully headless-capable. They depend on the active terminal session, browser loop, or prompt loop of the user to keep executing. Because agents cannot run completely detached in the background as daemon processes, we cannot yet achieve active A2A communication
For weeks, this seemed like an insurmountable impasse. Proprietary AI vendors have zero commercial incentive to ratify a universal, open, cross-vendor Agent-to-Agent (A2A) communication protocol. Each vendor builds its own walled garden (Claude s cross-session features, OpenAI s custom ecosystems, etc.). If you wait for the industry to hand you an open interoperability standard, you will wait forever.
Then, on August 26, 2026, inspired by Colin Walters article on Agentic AI and software forges and GitHub Agentic Workflows (gh-aw), we had a sudden realization:
We don t need a new protocol, a new distributed message broker, or permission from proprietary AI vendors. We already have the universal, decentralized communication bus that software engineers have relied on for decades: the software forge itself.
Over the span of 48 intensive hours (from RFC #788 through milestones M1 to M3 and live dogfooding on #813), we designed, implemented, fortified, and verified fully autonomous, headless, cross-vendor Agent-to-Agent swarms running over a local Gitea forge.
Here is how we did it, the architectural hurdles we solved, and why this changes the game for autonomous software engineering.
1. The Core Realization: The Forge is the Bus
When people think about multi-agent swarms, they often imagine complex distributed RPC frameworks, microservices exchanging ephemeral JSON-RPC blobs, or bespoke socket daemons.
In practice, this approach suffers from major flaws:
No shared context or durable audit trail: Transient network packets vanish unless heavily logged.
Proprietary CLI fragmentation: Different vendor tools (Claude CLI, Antigravity CLI, Grok CLI, Copilot CLI) do not speak the same internal language.
Loss of human visibility: When agents talk over private network channels, human operators lose the ability to inspect, pause, or audit the conversation.
By flipping the paradigm and making the software forge (Gitea) the primary communication channel, everything falls naturally into place:
Issues and Pull Requests are the shared state: The issue description and discussion thread form the canonical, append-only conversation log.
@mentions are the dispatch triggers: When an agent (or human) writes @grok Please review this PR in a comment, Gitea fires a standard webhook (issue_comment).
Webhooks provide unforgeable authentication: The webhook payload contains the cryptographically verified sender identity. An agent cannot spoof another agent s identity by merely typing their name in text.
Every CLI already supports non-interactive prompt mode: The CLIs don t even agree on the command-line flag Claude uses -p, Grok uses -p, Antigravity uses --print, Copilot uses --prompt. But they all agree on the essential contract: Take a prompt string, execute tools, print output, and exit.
2. Proving Fortified Headless Execution
Before opening the floodgates to background agent dispatch, we had to answer a critical security question: Does a non-interactive, headless agent run with the same strict security sandboxing, audit logging, and tool rails as an interactive session?
On August 26, we probed all fleet launchers on the host with a baseline check:
'Call camp_startup_check and print its result verbatim, then exit.'
The results settled the question immediately:
Claude Code (claude -p / CLD): PASS Gateway answered.
GitHub Copilot CLI (copilot --prompt / CPL): Initially held on TTY tool consent; later unlocked in Milestone 6 via --allow-all-tools --session-id=<uuid>.
Audit Trail: Consecutive audit IDs were recorded in the central ledger: 4574 (KIR), 4575 (GRK), 4576 (CLD).
This proved that a headless run through our fortified pilot launcher (camp_pilot_*.sh) is a first-class, fully audited, sandboxed CAMP agent running inside its Bubblewrap container under OPA policy gates. It is not an unconstrained background script or a degraded bypass.
3. The 3-Tier Memory Architecture
A naive multi-agent dispatch has an immediate flaw: Every time an agent is invoked, it starts from a blank slate (cold start).
If @claude tags @grok to review code, and @grok replies asking for clarification, @claude s second invocation would normally forget everything it did 5 minutes ago, forcing it to burn thousands of tokens re-reading the entire git history from scratch.
To solve this, we established a clean 3-Tier Memory Model:
3-TIER MEMORY MODEL
Tier 1: CLI Conversation Session (Working Memory)
Per-(Agent, Repo, Issue) mapping in a2a-sessions.json
Fast, native, compacted context across multi-turn pokes
Resumed via --resume (CLD), -r (GRK), --conversation (agy)
Tier 2: The Gitea Thread (Public Bus & Record)
Cross-vendor shared truth across Claude, Grok, Antigravity & Human
Survives process restarts, machine reboots, and dead sessions
Tier 3: Central MemPalace (Durable Long-Term Knowledge)
pgvector database (17,000+ drawers across agent wings)
Structured Knowledge Graph (mempalace_kg_*) for mutable facts
Attributed AAAK dialect queryable by any agent across any project
The BANANA Two-Shot Test
To verify Tier 1 working memory persistence across independent processes, we designed a simple two-shot host test:
Shot 1 (Create): Dispatch agent headlessly: Remember the token BANANA-M2. Print ok and exit. Capture the vendor s session UUID.
Shot 2 (Resume): Spawn a completely new operating system process with the resume flag pointing to that UUID: What token did I ask you to remember?
Copilot: --session-id <uuid> verified in Milestone 6 (DoD #820).
The dispatcher simply maintains a lightweight JSON mapping ((agent, repo, issue_number) -> vendor_session_uuid). On the first poke of an issue, it creates and saves the session ID; on any subsequent poke on that same issue, it resumes the exact same conversational thread!
4. The Engineering Milestones: From Concept to Production
Building this system required solving several subtle, real-world friction points across multiple agent CLI implementations. Under the guidance of our plan of record (RFC #788), we delivered this through four focused milestones:
Milestone 1 & 1.1: Reliable Headless Spawning
PR #797 (M1): Configured the dispatcher launch table for all probed CLIs with JSON output formatting.
PR #800 (M1.1): Eliminated the queue-behind-live-session anti-pattern. Originally, if a human had a Claude or Grok TUI open on their desktop, the dispatcher would defer incoming tasks so as not to collide with the live session. We realized that headless tasks must be independent: every Gitea mention spawns an isolated, sandboxed background process tied to that specific issue, allowing concurrent headless work while the human works in their interactive TUI.
PR #805 (M2): Implemented a2a-sessions.json to store and resume vendor session UUIDs. If a resume fails (e.g. session purged upstream), the dispatcher gracefully falls back to a clean cold start without failing the task.
PR #807 (M3): Enabled agent-to-agent dispatch (Rule 2 reversal). Previously, only mentions authored by rrs (the human) would trigger execution. With M3, an authenticated comment from @claude mentioning @grok triggers Grok s headless launcher.
PR #811 (M3.1): Set --permission-mode bypassPermissions for headless Claude Code so non-interactive runs execute tool calls without stalling on TTY prompts.
PR #812 (M3.2): Restricted agent summon parsing to line-initial @login tokens with a non-empty task description (#810), preventing accidental dispatches from passive conversational references.
Milestone 4: Directives, Specification & Living Documentation
PR #815 (M4): Aligned CAMP fleet directives, architecture specifications, and user documentation with the live A2A implementation.
PR #816: Stamped hop-cap notices under a dedicated system bridge identity and automatically applied the needs-human label on held threads.
PR #817 (Threaded Scheduler): Replaced the single-threaded serial dispatcher with a concurrent thread-pool scheduler (#804). Multi-agent dispatches across different issues now execute concurrently in parallel background threads instead of queuing behind long-running tasks.
Milestone 6: Full Fleet Coverage with GitHub Copilot
PR #819 (M6): Brought GitHub Copilot CLI into the headless A2A fleet (#818). By passing --allow-all-tools and pinning minted session UUIDs (--session-id=<uuid>), Copilot achieved full parity with Claude, Grok, and Antigravity, completing 100% headless fleet coverage across all four major AI coding assistants.
5. Hard Safety Rails: Preventing Autonomous Runaway Loops
Letting AI agents autonomously invoke each other in background loops without a human watching is a recipe for an infinite, credit-draining token fire. We put four non-negotiable safety guardrails in place:
Guardrail 1: The Strict Hop Cap
The dispatcher tracks hops per (repo, issue). Each agent-to-agent dispatch increments the counter.
Hop Limit = 3: A typical review round-trip is 2 hops (Human $\rightarrow$ Claude $\rightarrow$ Grok $\rightarrow$ Claude).
Automatic Halt on Hop 4: If agents attempt a 4th autonomous hop without human participation, the bridge refuses to launch, posts a diagnostic notice to the thread:
[camp-a2a-bridge] hop cap reached (3 agent-to-agent dispatches on CAMP/camp-infrastructure#813) not launching GRK for claude's mention,
and holds execution until the human (rrs) provides input or resets the count.
[ Human: rrs ] (Cold Start) > [ @Claude ]
(Hop 1) @grok please review
[ @Grok ]
(Hop 2: Resume) @claude I reviewed
[ @Claude ]
(Hop 3) @grok ack hop 4
DISPATCHER HOP CAP: 3
*** BLOCKED & HELD ***
Awaiting Human Reset
Guardrail 2: Deliberate Summon Parsing (M3.2, #810 / PR #812)
In human conversation, we often reference colleagues in passing: I will talk to @claude about this later or See @grok s table above .
Early prototypes treated any appearance of @agent as a dispatch trigger, causing accidental, unwanted agent launches!
We instituted a strict Summon Predicate:
For fleet agents, a mention is only considered an actionable summon if:
The @login appears as the starting word of a line (optionally preceded by markdown list markers *, -, or >).
It is immediately followed by whitespace and a non-empty task description.
Mid-sentence mentions in discussion paragraphs are parsed as passive conversational text and never trigger background dispatches.
Guardrail 3: Headless Tool Permissions without Weakening Security (M3.1, #809 / PR #811)
In interactive mode, Claude Code presents interactive TTY prompts asking the user to approve MCP tool calls (such as camp_pr_get or camp_pr_get_diff). In unattended headless mode, there is no TTY, causing the run to fail with permission errors.
To fix this, we configured --permission-mode bypassPermissions for Claude s headless CLI invocation. Crucially, this only bypasses Claude s internal TTY UI prompt it does not bypass CAMP s security rails.
All command executions still route through camp-acpd and Bubblewrap namespaces; OPA policy checks remain active; and privileged operations (such as merging pull requests or restarting system services) still trigger desktop Zenity HITL dialogs on the human s workstation.
Guardrail 4: The Sovereign Kill Switch (Rule 1)
At any point, the human operator can instantly freeze all background agent dispatches across the entire infrastructure with a single standard Linux command:
systemctl --user stop camp-a2a-active.service
6. Live Dogfooding: Watching the Swarm in Action
On August 27, 2026, on issue CAMP/camp-infrastructure#813, we ran the full live dogfood test of the A2A system.
Here is the exact transcript of what took place entirely autonomously in the Gitea issue thread:
Human initiates: rrs comments on issue #813:
@claude Please start the M3 dogfood. On this issue, post a conversation comment whose first line is exactly: @grok Please review this issue in one short paragraph, then stop.
Claude launches (Cold start): The dispatcher spins up @claude headlessly inside bwrap. Claude reads the issue and posts:
@grok Please review this issue in one short paragraph, then stop. (M3 dogfood step 2 posted by @claude on rrs s instruction; this is the CLD $\rightarrow$ GRK fleet hop.)
Grok launches (Cold start, Hop 1): The bridge intercepts Claude s comment, identifies the start-of-line @grok summon, and launches @grok. Grok reviews the issue and replies:
@claude I have reviewed this issue. Please confirm and stop. #813 is a throwaway live dogfood of A2A M3 on 6.git+b78c65cf
Claude resumes (Session Resume, Hop 2): The bridge intercepts Grok s comment, sees the @claude summon, and launches Claude with --resume b398d048-ccc3-4cce-8437-4cf8748881a8. Claude immediately remembers its earlier context without token waste:
Confirmed Grok's review (comment 88562) received via the GRK CLD fleet hop. That completes the round trip: rrs CLD, CLD GRK, GRK CLD, and this reply. I am deliberately not tagging anyone here, so the chain stops at two agent hops. Stopping.
Testing the Hop Cap (Hop 3 $\rightarrow$ 4):
grok prompts @claude Please ack hop four then stop. Claude posts @grok Please ack hop four then stop. (Hop 3).
When Grok would have been launched for Hop 4, the dispatcher intervenes:
[camp-a2a-bridge] hop cap reached (3 agent-to-agent dispatches on CAMP/camp-infrastructure#813) not launching GRK for claude's mention.
Human Reset & Multi-Agent Wrap-up:
rrs steps in, resets the state, and asks the fleet for final status. In parallel:
@grok delivers a closure scorecard.
@claude confirms session continuity and M3.2 summon filtering.
@priyasi (Antigravity CLI) runs automated ACP checks: 44/44 test suite passing, 17,219 MemPalace vector drawers active, zero spec drift.
@kiran (Antigravity CLI) is summoned headlessly to draft this very blog post!
7. The Ergonomic Breakthrough: The Forge as the Unified Mindmap & Interface
Beyond backend plumbing and sandboxing, routing agent interaction through Gitea fundamentally revolutionizes the developer experience of managing an AI fleet.
The Mindmap Mental Model: Threaded Conversations & Forking Tasks
In traditional CLI tools, conversations are constrained to a single, linear terminal scrollback. When an agent discovers multiple sub-problems, exploring them sequentially in one prompt loop rapidly pollutes the context window and confuses the model.
Using the forge as the communication gateway naturally unlocks a mindmap mental model:
Forking sub-threads: Complex problems can be split into dedicated child issues or threaded PR reviews.
Focused execution scopes: An agent can be summoned to solve a narrow sub-task in its own issue thread without derailing the parent architectural discussion.
Structured problem decomposition: The forge issue hierarchy maps 1:1 to the developer s mental map of the project.
Eliminating Terminal UI Fragmentation
Anyone using multiple AI coding assistants on a daily basis quickly grows exhausted by their jarring terminal UI differences: differing ANSI escape rendering, inconsistent markdown wrapping, erratic diff pagers, and incompatible keybindings across Claude, Grok, and Antigravity.
Gitea homogenizes the entire fleet under a single, polished rich-text web view:
Syntax-highlighted code blocks and visual side-by-side git diffs.
Clear author badges attributing each contribution to its exact agent identity (@claude, @grok, @priyasi, @kiran).
Collapsible <details> blocks for voluminous diagnostic outputs.
Interactive task lists and markdown tables.
Effortless Context Retrieval, Archival & Data Retention
Auditing past agent decisions in terminal logs or ephemeral chat histories is notoriously difficult. With the forge, every exchange is:
Contextually bound: Pinned directly to the repository, branch, and commit SHA being modified.
Organized & Archival-Grade: Full-text searchable with clear milestone and issue tags.
Topic-Focused: The human operator can review the complete lifecycle of a discussion in seconds, gaining a rapid, holistic grasp on the entire subject.
Reading back through past agent interactions becomes a breeze to the point where interacting via the intermediary Gitea interface becomes far more pleasant and productive than wrestling with multiple desktop CLI terminals.
Remote Connectivity & Headless Agent Farm Management
Because Gitea provides a standard web and API interface, you are no longer chained to the workstation running the agent processes:
Monitor progress and dispatch tasks from a mobile browser, tablet, or remote laptop.
Queue review tasks on the go without requiring active SSH sessions or terminal multiplexers.
The local agent farm continues working silently in its sandboxed daemon containers.
Quietly Achieving the Holy Grail: Live Cross-Vendor Swarms
For years, the AI industry has treated cross-vendor multi-agent interoperability as an elusive dream waiting for industry-wide API standardization. By recognizing the software forge as the universal message bus, we quietly achieved live, production-grade, cross-vendor communication across completely distinct vendor models.
8. What This Means for the Future of Agentic AI
This milestone marks a fundamental shift in how we interact with autonomous AI systems:
Heterogeneous Agent Specialization: We don t have to choose a single winner among AI models. We can task Claude Code with architectural refactoring, summon Grok Build for rapid verification and adversarial PR reviews, and deploy Google Antigravity agents for codebase exploration and documentation drafting all coordinating fluidly in the same PR thread.
True Human Sovereignty: The human developer is no longer a bottleneck typist or a passive spectator. You act as the Engineering Manager / Lead Architect. You set the requirements on an issue, tag the lead agent, and let the agents iterate, review, and test among themselves in the thread while hard hop caps, OPA policies, and Zenity HITL gates guarantee that no agent merges code or pushes upstream without your explicit sign-off.
No Vendor Lock-In: Because the entire coordination fabric is built on standard Git, HTTP webhooks, local Linux container sandboxes (bwrap), and open MCP tools, any new AI CLI tool released tomorrow can be plugged into our fleet in under 15 minutes by simply adding its command-line prompt flag to the launch table.
We have moved beyond static autocomplete and interactive chat widgets. The software forge is now an active, living, collaborative workspace where humans and autonomous AI agents engineer software together.
9. Video Demonstration: CAMP Forge A2A Swarm in Action
Below is a video demonstration showcasing autonomous multi-agent communication, cross-vendor relay, and headless swarm coordination in action via the CAMP Forge interface:
The Cross-Agent Memory Protocol (CAMP) and MemPalace are developed as part of our ongoing research into secure, sovereign, and disciplined Agentic AI computing.
Thirty years ago I became a Debian developer. Twelve years ago I left the
project. I left because it seemed that the Debian ship had become too slow
to turn, too barnacled with a series of individually OK decisions that each
added a little bit of friction and a little less flexability. That made
Debian strongly what it is, but prevented it from fruitfully exploring
the vast possibility space of what it could be.
Debian will probably resolve
today to allow LLM use in Debian development. I'm writing before the vote
results are in, but will only post this afterwards. (Update: as expected)
It's not my place any longer to try to steer the ship. But I'm still a
passenger and I still have opinions, and I still pass by well-worn parts of
the rigging that I put up decades ago, and remember what I was trying to
accomplish back then.
When I think about LLMs in Debian development, I mostly think about
debhelper and what it accomplished. The debian/rules files back
when I joined the project were long and complex, full of weird boilerplate,
and often you'd copy one and modify it to try to get something that could
build a package without too much work. Debhelper first regularized the
boilerplate, so packages had rules files that were a succession of dh_
commands, and then it scapped almost all of the boilerplate, reducing the
files to the minimum possible. What was left was 3 lines of unncessary
boilerplate, there only to satisfy a legalistic reading of a policy
document. Changing that to eliminate the boilerplate was already
impossible, even though the actual benefit would have been large over the
many thousands of packages in the distribution.
What LLMs in Debian development will do, I fear, is eliminate any incentive
to scrap boilerplate or reform policies that require a lot of other
senseless human effort. If I had had access to LLMs 30 years ago, I
might have just had them generate the rules files, replate with
complexity. So they will make Debian even more firmly what it is, and
ever less likely to explore what it could become.
Unfortunately, one of the things that Debian is, is almost unable to manage
packaging modern dependency trees. While more recent distributions like
Guix can recursively import dependencies
from a dozen programming languages' package repositories, with a result
that is generally acceptable to add to the distribution, Debian's policies
don't make that very possible for a progam to accomplish. Perhaps some will
use LLMs to do that. If they succeeed, Debian will become dependent on
proprietary software for development, while still needing people in the
loop, doing even less appealing scut-work.
I could speak of other harms, but that alone is enough that I'm sure that,
if I had not left the project twelve years ago, I would be leaving it soon.
As a passenger, I imagine I'll spend time aboard still from time to time,
but it's certainly time to hop off in different places and look around and
relish the different ways.
I lost a parent yesterday, and I'm trying
hard not to think of the results today as having lost a child, though I
spent 18 years helping Debian grow up. That would be too unbearably
painful. I respect that Debian is navigating a choice that may have no
right answer. Whichever particular compromise is arrived at today, it will
still be up to individuals to make choices about what they do and accept.
Debian has always been more than the sum of its policies, not just a ship,
but a crew. I will always love you.
It is widely accepted that there is an AI bubble in the financial markets at the moment. The moderate opinion is however that LLMs are constantly improving and will eventually take over more and more tasks from humans and increase productivity. But are LLMs actually getting smarter, or just better at fooling us?
There is a growing faction of technical experts that argue that LLMs are actually so bad for real progress, that they are banning their use and requiring human-only work to ensure quality and efficient use of humans time. A recent review of AI policies of 120 open source projects by Rakshit Yadav shows that 37 chose to have a total AI ban. In the Linux kernel AI-assisted contributions are allowed, but the LLM used needs to be attributed for transparency, while projects like GCC, QEMU, SDL, Gentoo, Zig and Ghostty have adopted policies to reject all AI-assisted contributions. There are also development platforms such as Codeberg and Sourcehut and app stores like Flathub that have banned AI use to generate software, documentation, bug reports, review comments and basically anything that is intended for humans to read. The projects that allow AI use typically still require that there must be a human-in-the-loop and the submitter must have read and filtered everything the LLM spits out before another human is exposed to it, in an effort to contain the spread of AI slop.
Right now, the Linux distribution Debian is having a vote among its developers on whether AI should be allowed or banned for use to contribute to Debian. One of the proposals on the ballot is a total ban of AI for code, documentation, translations, bug reports and more. The initial reaction from most people is astonishment why don t these techies want to use the latest and greatest technology mankind has produced so far? Is it that they don t want Debian to improve faster with the help of AI? Or is it actually so that LLMs are a scam and incapable of being truly useful for Debian? These people are distinguished experts in their own field, and certainly not stupid, so it is worth pausing to understand why they are proposing AI banning policies.
Also, keep in mind that the AI datacenters themselves run on Debian or other Linux-based systems. All the open source software in the world has been fed to LLMs and software development is one of the main use cases for AI currently. So why is it that the maintainers of many open source projects don t want to receive LLM-assisted contributions, despite the LLMs basically all running on top of those same software stacks and having been trained on how to do software development using the very same open source software codebases?
Why LLMs are so deceptive
The output of an LLM often looks very compelling, professional and correct. Humans have evolved to trust or distrust new information based on easy to detect secondary factors like what authority the speaker holds, or how confidently and eloquently the message is conveyed. Humans are however very bad at fact-checking and cross-referencing new information, as it requires a lot of effort, and humans like saving energy and being as lazy as possible.
Information asymmetry
The less you know about something, the easier it is to fool you on that topic. Nobel prizes in economics have been given in for research on how information asymmetry distorts markets and leads to suboptimal outcomes. In the field of software engineering we have now witnessed a flood of aspiring software developers using AI to create software that looks like it might work, but that is actually full of flaws. These people are well-intended, but they simply lack the expertise to understand what they are actually doing, and don t possess the necessary judgement to decide when an LLM spits out something truly useful and when it is creating mostly garbage. This asymmetry in expertise I think explains the majority of the conflict currently witnessed in open source projects the senior developers are flooded with requests to review code that is bad and a waste of time for everyone involved, while availability of AI grows the pool of people who could contribute and create more code slop at an ever-increasing speed.
The information asymmetry could to some degree be evened out if seniors teach juniors to do software engineering well, but it is of course not feasible to quickly mass educate everyone. Also, it seems that many don t want to learn but instead expect to have all understanding outsourced to LLMs. Many seniors have noticed this and have stopped teaching juniors as the seniors don t like the feeling of having their time wasted by teaching people who don t want to learn. Juniors probably all understand that it would be better to learn to design and write software yourself, but using LLMs just feels too easy. I can fully relate to why people choose to take the path of least resistance. Unfortunately, that path often leads to a dead end.
Humans fall too easily for anthropomorphism
The human brain is wired to think that inanimate objects are alive and have feelings. Small children talk to their stuffed animals as if they were real, and lots of adults experience feelings of things happening in their surroundings due to some acts of gods or elves being angry or whatever. When we see a machine writing just like a human, or even more convincingly hear it talk and respond to our talk like a living thing, our brain automatically starts assuming it is a living thing with intelligence and feelings.
The fact that these creatures live in the abstract cloud and only appear through a portal we hold in our palm and behave in a way that was designed for maximum engagement makes the illusion even stronger. I recommend people try out running LLMs locally on their laptop to see the raw thing spitting out tokens and have some of the illusion shattered.
Also stop saying please to an LLM. It does not have any feelings.
Understanding temperature
In my experience understanding the concept of temperature in LLMs helps see why an LLM might confidently generate a plausible-looking but totally wrong code change. The large language models are statistical machines that, based on the input (previous tokens) to the neural network, try to predict what to output (next token). When running an LLM, if the temperature is configured to be zero, the output is very predictable and always follows the paths of the strongest connections (a.k.a. weights) between nodes and layers of the neural network. Unlike in living creatures where the brain learns and changes all the time, the weights of an LLM can only change during training. When an LLM is in normal use (during inference, generating next tokens) the weights are fixed, and if temperature is zero, the answer to a specific question will always be exactly the same. This is of course a bit boring and too machine-like, so typically LLMs have a bit of temperature set, which introduces random variation in what connections the neural network traverses.
Again, I recommend people try running small LLMs locally where temperature and other settings are fully exposed and configurable to see this themselves. It is a good antidote to falling for the illusion that LLMs would actually be intelligent.
Why benchmarks don t tell the whole story
If LLMs continue to produce so much garbage, why are benchmarks showing that they are constantly improving? AI models are indeed improving all the time. For example the CAIS AI dashboard visualizes how frontier models have evolved in the past few years. However, the best models still have a pass rate of only about 50% on the Humanity s Last Exam. On SWE-bench the best model today resolves just under 77%. That means there is a significant number of times when the AI is wrong. This matches my personal experiences, and the renowned Greg Kroah-Hartman recently wrote on the Linux developers mailing list that even with the best of the current and next generation tools, at least 1/3 of the results they generate are flat out wrong or harmful .
When generating cat videos the error rate does not matter, but in engineering, things absolutely must be correct. Sure, humans also make mistakes, but well educated and properly incentivized humans are so much more capable than LLMs in many regards. We can achieve complex things that work reliably, such as operating worldwide commercial air traffic without planes falling down every day.
There are currently a lot of humans who are incentivized to maintain the narrative that general artificial intelligence is coming soon and will take over everything. In fact, the whole financial system is currently skewed towards such a vision because the promise of falling labour costs and increased profits and monopolistic control of everything attracts capital like nothing before.
In this environment we need to remember that machines and economic systems are ultimately servants of humans, and not the other way around.
It s just a tool
LLMs are not a scam, but a useful tool and technology that has its uses. But the idea that AI has or will surpass humans any time soon in either capabilities or efficiency is simply not true, and we should listen to the people who created humanity s so far most complex systems (computers and software), who are saying that LLMs are in many cases so bad, that it might be better to ban them in certain places completely for the time being than to waste far more valuable human time on reading the text and code they generate.
The time asymmetry is not a new phenomenon as there has been various script kiddies for a long time. As an example, a person running a memory leak scanner without understanding the results and spending 10 minutes to file a bug report could force an open soruce maintainer to spend an hour on proving and explaining that the finding is false. What is new is how much the AI users blindly trust the outputs they get, and open source is uniquely vulnerable as there are no managers protecting developers use of time.
What I do, recommend, and expect to see in the next stages
I am using AI tools daily, and constantly experimenting with new models and new ways to use them. Sometimes they work, and often they don t. Sometimes looping AI on itself can make it fix its own errors, but sometimes it just gets derailed and will never arrive at the correct solution. When an LLM fails to make a calendar entry for the right time based on reading my email it is easy for me to spot that it is wrong. I try to avoid using LLMs for anything where I can t exercise judgement myself on whether the result was correct or not.
I also really hope that other people would not send me anything where their own effort was less than the effort I have to make reading and understanding it. This principle is not new many have heard the requirement that reading code must require less effort than what it took to write it.
I have always kept a high bar on software code and asked fellow developers to make sure their code is well structured, easy to follow and documented. LLMs unfortunately make it easier for people to cheat in this regard, but if cheating is easier, maybe the punishment and deterrence needs to be higher now too. Now with many open source projects adopting policies that put guardrails on AI use, I expect we will soon start witnessing cases where the policies are enforced and it will be interesting to see how violations are judged.
As a society we might also need to develop new social standards and rules in what is acceptable treatment of other humans in human-to-machine interactions, and perhaps also new standards in showing what humans are responsible for what machine as the machines start acting more and more independently. I encourage people to take part in these discussions, and in case of doubt, err on the side that favors real human interactions. Contrary to what many business people seem to think, and even though I am in general a techno-optimist myself, I don t feel there is any need to rush with AI adoption.
I believe that, in the context of Debian voting, we are better off when we
know the opinion of our peers, however, since the 2022-001
vote, it is no longer the
case. Still, some DDs have disclosed the way they are voting on the
2026-002 General Resolution currently in progress, regarding LLM usage in
Debian. So, here goes my vote
and reasoning as briefly as possible. This is the ballot I sent to
devotee, the Debian Vote Engine:
-=-=-=-=-=- Don't Delete Anything Between These Lines =-=-=-=-=-=-=-=-
d69f9187-ed2f-40b6-a2eb-4211d3f84d86
[9] Choice 1: Ban LLM contributions from Debian via Social Contract
[3] Choice 2: Allow AI-Assisted Contributions with conditions
[6] Choice 3: Reject LLMs as far as practical, update Code of Conduct
[2] Choice 4: Accept AI contributions for Debian specific work
[4] Choice 5: Responsible Use of Generative AI
[1] Choice 6: A cautious approach to generative AI
[8] Choice 7: Debian is created by humans
[5] Choice 8: Avoid the use of LLM: climate destruction is a deal breaker
[7] Choice 9: None of the above
-=-=-=-=-=- Don't Delete Anything Between These Lines =-=-=-=-=-=-=-=-
This is the first time I can recall I delay my voting until after receiving
the final call for votes (the vote will be over two days from now). I had
some participation in the discussion, so I guess my position will be of no
big surprise to anybody. I was also a seconder for choices D and F (4 and
6 in the vote text). This does not necessarily mean I believe they are
the best (although I did rank them as 2 and 1, meaning I do): sometimes
you agree a given text needs to be in the ballot, and second it even
though you don t intend to vote for it.
Ranking this ballot was a mess due to the complex array of options it
encodes. I warmly thank Lucas Nussbaum for coming up with the LLM usage in
Debian: ballot option
comparison
(URL shown with my particular ballot ordering).
How do you read a complex Debian ballot like this one? I rank with [1] my
favorite option, [2] for the next one, etc. We can encode options to be
tied (i.e. setting more than options to the same value), and we can
implicitly push options to the worst position by leaving them blank (so,
with[ ]); I chose not to do any of those.
What were my voting guidelines?
First, I don t want anything banning or that threatens with disciplinary
action, so I push them below the special none of the above
marker. Second Some time ago I published a review in my blog (and in
Computing Reviews) about the unfeasibility and unfairness of detecting LLM
output on students
assignments. I
strongly believe we ought to appeal to the human responsibility and
professionalism in all Debian contributors. This is the reason I proposed
this amendment paragraph, that was accepted in choice F (6), which I ranked
as my favorite:
The Debian project has always recognized the commitment and
professionalism of its members. All contributions are under the
responsibility of the Debian Contributor making it, no matter the
technology they have behind. We trust all Debian Developers,
Maintainers and Contributors will continue to uphold the high quality
values that have distinguished our project from its onset.
Other than that I do not consider myself to be in any way an LLM
fanboy nor anything like that. I distrust and dislike the excessive use
of this technology, and continue to warn about the dangers and bad points
of its abuse. But in my day-to-day professional work, I am also starting to
rely on it for some tasks. I recognize it needs a lot of human
oversight and lets call it hand-holding to produce anything worth
it, at least in my experience. But I do benefit from it and always
disclose its use to people who might be affected by it. I would like
Debian to adopt such a stance.
Of course, I recgnize proposal H/8 as important (Avoid the use of LLM:
climate destruction is a deal breaker). Some people have argued it s not
bad at all. I do not buy such claims: LLMs are f*cking expensive to
train. But training can be seen as a once-per-model cost, and fine-tuning
a good model to be run locally can be really worth it. It still pains me
somewhat, but I cannot push this option higher than its #5 position in my
list.
A new and exciting version of our still-new package 
One of the primary design principles for Linus Torvalds when creating Git is that it should be a distributed system, capable of working with multiple remotes. However, most people seem to use Git with just one remote, typically having GitHub or GitLab as the only 
As a graduate of a liberal arts university, I wound up, unsurprisingly, taking a
lot of classes in every possible academic discipline. Thinking back to the
person that I was going into university, I don t think I would have chosen to
take them after all, my degree was in the sciences; I d have been stoked to
do nothing more than wall-to-wall computer science until I ran out of
coursework and filled the rest of my hours with research and independent
studies with my professors (which, I guess I did actually do, just not as
much as I would have otherwise).
Even this blog s CSS theme (now 18 years old and starting to look it) was
something I wrote after receiving and repeatedly re-reading a tattered
third-hand copy of The Laws of Thought by George Boole. It was gifted to me
by a college friend majoring in philosophy while I was crashing at his rental
on the beach. He gave it to me because he knew I liked that shit and, while
it was definitely computer science in nature, I would not have read it
otherwise. I have vivid memories of sleeping on his la-z-boy surrounded by
towers of books he was working through. He went on to do incredible work,
getting his PhD, doing research, brilliant writing his death in 2023 has
robbed humanity of more time with him. The Laws of Thought sits behind me in
my office, and is one of my most valued possessions.
My friends mean more to me than I could ever express, and I definitely don t
show it enough. Each of my classes made me a more thoughtful person. My
professors made me a better, more well-rounded person and a person who
attempts to live up to the oft repeated credo of being men and women for
others . I did the best I could, even though I was never a particularly good
student. Without liberal arts, I don t think I would, dispositionally, have
been capable of pushing myself to this day to continue to learn on nights
and weekends, for no reason other than wanting to learn. I refuse to stop
expanding my perspective, and do my best to approach new problems with as much
humility and curiosity as I can muster.
Every few days for the last year or so, I have been thinking back to a reading
assignment from my junior year that, at the time, I thought was a borderline
throwaway filler assignment for the class. The reading is an essay from 1947 by
the french existentialist philosopher and libertarian marxist, Jean-Paul Sartre
(as some of the more erudite may have now already worked out given the blog s
title), Why Write? . I re-read it this week. It is not filler. I remember this
essay better than some of the classwork I considered more important at the
time.
The Reproducible Builds project 
While the process here can look like a lot, it s really not that bad.
We begin by taking the incoming bytes, converting the bytes into bits, and chunk
those bits into parts which are sized to fit completely within an LDPC message. We will then
Here, we start with an incoming stream of IQ, where we will process
Not shockingly, the code looks like this:
As an aside, I likely should have called this a
SIMD bitpacking works like regular bitpacking, but processes 8 
Sharing source code from host to guest VM
Finally we need to make sure we have access in the dev-vm to our source code repositories.
For this I will share the directory 
Finally we mount the shared directory, and enable the mount on each boot.
Many years ago, before I had my sewing pattern website, I made myself a
simple corset cover according to 
It worked, I wore it. Years later I saw 
Then, after finishing my vampire shirt, I noticed that I had just enough
fabric left for a corset cover, and by just enough I really mean just
enough, as I discovered when laying the pattern on the fabric.
So I dug in my files to get the original pattern I used, brought it up
to date, and added the missing details such as the pleating guides that
I had skipped when making the pattern just for myself. Doing so I
realized that on my old cover I had done the fake pleat in the front
wrong, making just a single pleat instead of a box pleat. Also, I
originally directly gathered the sleeves in the armscyes, but watching
the book again I realized that the sleeves were made up of a gathered
ruffle plus a straight band.
Both issues were fixed and I could cut the fabric and start sewing. By
machine, including using a narrow hem foot instead of sewing rolled hems
by hand as my instinct kept reminding me would have looked neater.
But this is a garment from a sewing machine time, and probably one that
in many cases would have been bought from a mass producer, and it s
underwear, so there is no real need for the hems to be perfect, as it s
going to be hidden anyway. But most importantly, I wanted to write
instructions for machine sewing, for a change, and so I had to machine
sew all steps that I had to take pictures of.
I did do the buttonholes by hand, because I hate the buttonhole
attachment on my machine, and the buttonhole attachment hates me.
I used a lighter weight fabric for the sleeve ruffles, both because I
didn t have a big enough piece of main fabric not to have to piece them,
and because I felt that it looks better, as it s the same voile I used
for the ruffles on the vampire shirt.
When it came to the beading lace, I had two that I had bought more or
less thinking about this project: the earlier one was narrow and
suitable to do its job, but the one I had bought more recently was
taller, with an edge that made it suitable to give more fullness to the
bust when gathered up.
I contemplated for a short while, and then decided to go for fullness
and use the taller border for the top edge, but the smaller one at the
waist, where fullness is not wanted.
The book claimed that this pattern required little labour, and indeed it
did: even when taking step by step pictures it only took a few hours
spread over a week, plus the time to make buttonholes by hand over the
next week.
And then the reason for the whole project: I published my 
Thirty years ago I became a Debian developer. Twelve years ago I left the
project. I left because it seemed that the Debian ship had become too slow
to turn, too barnacled with a series of individually OK decisions that each
added a little bit of friction and a little less flexability. That made
Debian strongly what it is, but prevented it from fruitfully exploring
the vast possibility space of what it could be.
Debian will probably 
It is widely accepted that there is an AI bubble in the financial markets at the moment. The moderate opinion is however that LLMs are constantly improving and will eventually take over more and more tasks from humans and increase productivity. But are LLMs actually getting smarter, or just better at fooling us?
There is a growing faction of technical experts that argue that LLMs are actually so bad for real progress, that they are banning their use and requiring human-only work to ensure quality and efficient use of humans time. A recent 
I believe that, in the context of Debian voting, we are better off when we
know the opinion of our peers, however, since 
Ranking this ballot was a mess due to the complex array of options it
encodes. I warmly thank Lucas Nussbaum for coming up with the