DebConf site visit, by Stefano Rivera
In preparation for DebConf 27, Stefano Rivera,
Nattie Mayer-Hutchings, and Jongmin Kim from the DebConf Committee visited
Asahikawa Japan. A contingent of the local team gathered to tour the venues
with us, discuss plans, and meet the local organizations assisting the event.
We held a press conference at the city hall that got
some coverage.
A few potential day-trip options were explored.
Photos from the site visit.
autopkgtest-virt-pconr, by Helmut Grohne
Following the creation of the pconr
module that provides a basis for
unschroot
and a more secure containment for mmdebstrap,
an experimental autopkgtest backend autopkgtest-virt-pconr
is now available. It fits a spot between autopkgtest-virt-unshare and more fully
fledged containers as it does not require much infrastructure like setting up
incus or lxc but still provides a containment running an initsystem. Due to the
use of erofs as base image format,
it boots and reverts quickly. This development also required fixing concurrency
issues in asyncvarlink and even shed
light on a matching bug in systemd
that was promptly fixed upstream.
Miscellaneous contributions
Carles reviewed and submitted six Catalan po-debconf translations using
po-debconf-manager.
Improved po-debconf-manager s workflow for updating translations.
Carles reviewed, translated or updated webwml
Catalan translations for security and CD pages. He also fixed issues that
prevented building webwml, deleted orphaned translation pages, etc. and
experimented with a more global HTML tidying up experiment.
Carles refactored check-relations
adding capacity to ignore concrete from_package -> to_package specific relations
and more flexibility around ignores. Detects unused ignore rules as well.
Carles improved some Wiki pages (LibreOffice,
Framework amongst some
other ones)
Thorsten did an hplip upload to replace an expired certificate of HP. This
certificate is used to sign new upstream releases and modules that can be
downloaded later. Afterwards work on a new upstream version started.
Thorsten also struggles with autopkgtest problems of the new upstream version
of cups. Work is still ongoing.
Antonio worked on several topics around Debian CI maintenance, reliability and
performance.
Stefano updated the autopkgtests for cPython packages in Debian to support
setuptools 80.
Stefano uploaded pypy 8.
Stefano helped to get DebConf 26 reimbursement claims processed, and worked on
some of the resulting issues in debian-reimbursements.
Emilio managed various transitions.
Emilio worked on the rustc-web/trixie bootstrap for armel, using the rustc
1.95 version from trixie-backports. This will be needed by the upcoming clamav
security updates.
Helmut sent patches for 11 cross build failures.
Helmut introduced a new build profile to libcap-ng to keep architecture cross
bootstrap working.
Helmut uploaded debvm_0.5.5
introducing support for Hurd thanks to a lot of work by Johannes Schauer Marin
Rodrigues.
Colin upgraded to new security releases of groff and parted.
Colin upgraded 50 Python packages to new upstream versions, and fixed over 40
RC bugs in the Python team.
Santiago continued mentoring Aryan Karamtoth with the GSoC project about
introducing livepatching support for linux. This mentoring is done in
collaboration with Emmanuel Arias. In particular, Santiago explored different
requirements for providing binary modules for the same kernel packages used by
Debian, that require a matching version number.
Anupa joined Jean-Pierre Giraud to prepare the files for point release
announcement for Debian 13.7.
Daggerbound is a secondary-world fantasy and a sequel to
Swordheart, although it follows
different protagonists for most of the book. It's also a sort of sequel to
the Clocktaur War duology in that one of the
protagonists was a protagonist of those books. T. Kingfisher is the pen
name Ursula Vernon uses for her adult fiction.
Learned Edmond was once a devout acolyte of the Order of the Many-Armed
God and a true believer in its teachings, including the ones about the
evils of women. Then he went on an adventure with, among other people, a
forger named Mistress Slate, during which he learned a great deal of
things about the world that his order did not teach him. He has spent the
subsequent years trying to bring this important new information to his
order and encourage them to adjust their view of the world accordingly.
The order has not changed, and Edmond is very tired.
Someone needs to carry a number of books and a magical sword to the city
of Archon's Glory. Edmond has volunteered and intends to perform the task
exactly as expected, and then never return. He is unsure what he will do
then, but at least it will be different.
Edmond did not expect to be waylaid by bandits or to accidentally draw the
magical sword. He certainly did not expect the man who appeared in a flash
of light and violently saved him from the bandits.
It would have been very convenient for the Dervish if Edmond died of his
injuries after drawing the sword. That would leave the sword without a
wielder and therefore the Dervish without someone he had to protect. All
he would have to do was sit and watch while the fever and cold and lack of
water did the work, and then he would be free. Given his past experience
with wielders, no one could even blame him. Except maybe his own
conscience.
I did not realize I wanted to read a novel about a traumatized Mercedes
Lackey sword waking up in a T. Kingfisher universe, but apparently I did.
The Dervish is not a sword, precisely, at least not in the way that
Need has become by the time we meet her in
Lackey's novels. He's a man trapped in a sword, he doesn't like it, and he
would very much like it to stop. But at the start of the book, he has the
rest of the vibe of a Mercedes Lackey protagonist: vast past trauma, poor
emotional regulation skills, and a prickly cynicism that only runs skin
deep. Pairing him with an Edmund who is far more mature than in the
earlier duology but still rather innocent and delightfully neurodivergent
is a great touch.
The rest of the plot was fine but not that memorable. Like the Clocktaur
series, there is a strong feeling of episodic D&D adventure as the
characters stumble across disconnected problems and hostilities while
collecting party members. Like Ursula Vernon's occasional D&D recaps, this
party has a tendency to adopt everyone they come in contact with, which
includes a gnole (as always one of the best parts of the book) and a
sentient giant pill bug. The pill bug, who has a complex
appendage-operated mechanical talk box and a banner that says "I DO NOT
EAT HUMAN MEAT," is the most Ursula Vernon character that has ever existed
in fiction.
There is a romance because there is always a romance in the novels set in
this universe. I didn't mind this one. Like every Kingfisher romance, it
involves a lot of irritating refusal to communicate and endless angst
about how the other person couldn't possibly be in love with them, but the
characters have rather more justification than usual and at least I didn't
spend most of the book thinking one half of the romantic pairing was an
idiot. Apparently it helps if no paladins are involved.
The strongest parts of this book are the beginning, before the Lackey
vibes unfortunately drain out of the story, and the end, when all of the
best characters from Swordheart show up again. The Temple of the
White Rat continues to be the best feature of this fictional universe, and
I could read endless books about how they solve problems.
The Dervish gazed at her with admiration. "And here I thought you'd be
on the side of law and order..."
"No, young man," said Beartongue, despite being several centuries his
junior. "I am on the side of justice. If we're lucky, the two run in
parallel. If they don't, we bend the law to attain justice, not the
other way around."
The ending also featured some exploration of the edge cases of the rules
for the sentient swords that reminded me a bit of
Fred Saberhagen with better characterization.
Daggerbound was not as charming as Swordheart, mostly
because Halla appeared only at the end. I wish Kingfisher had leaned more
into the Mercedes Lackey vibes rather than taking her normal D&D-flavored
romance approach. But this was enjoyable, charming at times, had great
supporting characters like all Kingfisher novels, and was a satisfying
continuation of the series. I am looking forward to the third (and I
believe final) book, which is supposed to be out next year.
Recommended if you liked Swordheart.
Followed by the not-yet-published Shieldborn.
Rating: 7 out of 10
This is a copy of an email I sent at work. I'm not sure I
should be making noise about this here, feedback welcome.
This is your bi-yearly reminder that time is changing soon! October 25th
in Europe, November 1st in North America. Less people in Canada are
changing this year, with BC, Alberta, Manitoba and Northwest
Territories getting rid of DST.
What's happening?
Some places in the world implement what is called Daylight saving time
or DST:
https://en.wikipedia.org/wiki/Daylight_saving_time
Normally, you shouldn't have to do anything: computers automatically
change time following local rules, assuming they are correctly
configured, provided recent updates have been applied in the case of a
recent change in said rules (because yes, this happens, and happened
this year, and yes, you need to upgrade your software!).
Of course, appliances like your microwave oven will likely not change
time and will need to adjusted unless they are so-called "smart", in
which case they are part of the skynet botnet and should be destroyed.
If your clock is flashing "0:00" or "12:00", you have no action to take
to adapt to this change, lucky you.
If you haven't changed time in six months, congratulations, your clock
will be accurate again!
In any case, you should still consider DST because it might affect some
of your meeting schedules, particularly if you set up a new meeting
schedule in the last 6 months and forgot to consider this change.
If your location does not have DST
Properly scheduled meetings affecting multiple time zones are set in UTC
time, which does not change. So if your location does not observer
time changes, your (local!) meeting time will not change.
But be aware that some other folks attending your meeting might have
the DST bug and their meeting times will change.
Be kind to those poor souls which might be missing meetings by a full
hour because time flies backwards for them.
If you do observe DST
If you are affected by daylight savings, your local meeting times
will change for UTC meetings. Normally, your meeting times are
scheduled to take this into account and the new hours should be
reasonable.
But now is a good time to verify that. Take a look at your schedule for
the next couple of weeks and reschedule meetings before the daylight
saving come up to avoid too much disruption. You have only a couple of
weeks to do so right now.
When do times change, how, and and where?
As regular readers will remember, the rule of thumb is:
Spring forward, fall backwards.
That is, during the season of Spring, the clocks move forward, and
during the Fall (like right now), they move backwards. That is in the
northern hemisphere, but then the southern hemisphere is often saner and
doesn't switch anyways.
So time will move backwards which means an extra hour of sleep. Unless
you have children or bad sleep, in which case your body doesn't care
about what the clock says and will wake up one hour earlier than what it
should.
And of course, this doesn't happen everywhere at once, so let's see when
it happens where.
Europe
The dance starts in Europe.
The change happens on the last Sunday in October at 01:00 UTC (not local
time!), that is October 25th. If you are in the central European
timezone, also known as Amsterdam, Berlin, or Paris time depending on
your national affiliation, that essentially means that at 2:59 local the
clocks will fall back to 2:00 instead of going to 3:00.
Concretely, set your watch back one hour before going to bed, go to bed
at the normal time, and enjoy an extra hour of sleep or leisure.
If you have kids, you might want to start getting them to bed slightly
earlier every day for a week before the change so they take time getting
used to the change. If you have trouble sleeping in the morning, find
your inner child and do that to yourself as well.
USA / Canada
Then it's the US[1] and Canada[2] joining the dance, on the First Sunday in
November at 02:00 local (not UTC!), that is, I believe, November 1st
2026.
This means that, at 1:59, the clocks will flip to 1:00, instead of 2:00.
Concretely, do like the Europeans and tweak your clock before going to
bed.
That is a little less than four weeks from now.
[1] except Arizona (except the Navajo nation), US territories, and
Hawaii
[2] except Yukon, Saskatchewan, (newly) British Columbia, (newly)
Alberta, (newly) Northwest Territories, (newly) Manitoba, one island in
Nunavut (Southampton Island), one town in Ontario (Atikokan) and small
parts of Quebec (Le Golfe-du-Saint-Laurent)
Other places with DST
This time again, I must apologize to the people of Cuba, Lebanon,
Israel, Palestine, Egypt, Chile, Australia, and New Zealand, as you fine
folks all have your own DST rules that are omitted here for brevity. I
rely on this page from Wikipedia to be updated by time nerds accurately
for this message, and it should provide you with a rough idea of what's
coming:
https://en.wikipedia.org/wiki/Daylight_saving_time_by_country
In general, changes also happen in October, but either on different
times or different days, except in the south hemisphere, where they
might happen in September (oops, sorry NZ folks, I'm late!).
Places without DST
Everyone else, enjoy, you're on the right side of history, and we thank
you for the good example you give us.
Changes since last time
There's been lots of changes since last time:
British Columbia moved to permanent -07 on 2026-03-09, that is it
will not change to normal time in November
Alberta moved to permanent -06 on 2026-06-18, similar to BC above.
Canada s Northwest Territories moved to permanent -06 on 2026-08-21,
matching Alberta.
Manitoba moves to permanent -05 on 2026-10-31.
Morocco moves to permanent +00 on 2026-09-20.
Moldova has used EU transition times since 2022, but the tz database
only noticed in 2026
This is my interpretation of the changes announced on the tzdata mailing
list here:
https://lists.iana.org/hyperkitty/list/tz-announce@iana.org/latest
If the eastward trend continues, Canada should adopt country-wide "no
daylight savings" rules by 2027, although there's actually no sign of
the other provinces (Ontario, Qu bec and so on) currently running bills
to change those rules just yet. Poor Canadians like me confused about
time in their countries can refer to this section of Wikipedia for
details:
https://en.wikipedia.org/wiki/Daylight_saving_time_in_Canada#By_province_and_territory
... and particularly the image featured there:
https://commons.wikimedia.org/wiki/File:Canada_time_zone_map-en.svg
It also seems like the US government might finally adopt a permanent
daylight saving change bill in 2026, as the "Sunshine protection act"
pass the house in July:
https://en.wikipedia.org/wiki/Sunshine_Protection_Act
True to form, this was associated with absolutely ridiculous pressure
from Donald Trump against republicans (his own party!) objecting to the
change:
On July 14, 2026, the House passed a Sunshine Protection Act bill
backed by President Trump. Nevertheless, the bill was opposed in the
Senate by Republicans, including Senator Cotton. In response, on
October 3, 2026, Trump shared a post on Truth Social urging Cotton to
approve the bill, where he revealed Cotton's personal cellphone number
and called on people to call him.
https://www.theguardian.com/us-news/2026/oct/03/trump-tom-cotton-daylight-saving-time
Given that the last time the US did a major change to the daylight
savings policy (in 2005), Canada followed suit to stay in sync, it's
quite possible Trump's mad dash might actually finish getting rid of DST
in North America:
https://en.wikipedia.org/wiki/Energy_Policy_Act_of_2005#Change_to_daylight_saving_time
netip.Addr features an Unmap()
method returning the unwrapped IPv4 contained in an IPv4-mapped IPv6
address: from ::ffff:203.0.113.10 or ::ffff:cb00:710a, it returns
203.0.113.10.1 There is no Map() or To6() method for the reverse
direction. Such a method is trivial to implement, but Go maintainers have
rejected it on the grounds that users should write
netip.AddrFrom16(ip.As16()) and let the compiler optimize it.2 Today,
this pattern is eight times slower than a native method. How can we teach the
compiler to optimize this sequence?
The alternatives
Let s explore three ways to implement the map semantics for netip.Addr. My
favorite is to add it to the Go standard library. Go maintainers prefer a small
external helper chaining netip.AddrFrom16() and netip.Addr.As16(), hoping
the compiler eventually optimizes it. The unsafe package
opens a third path, with the same performance as the first solution.
Modifying the Go standard library
Internally, netip.Addr stores any IP address as a
128-bit value with an extra field z to encode the family and the zone:
typeAddrstructaddruint128zunique.Handle[addrDetail]typeaddrDetailstructisV6bool// IPv4 is false, IPv6 is true.zoneV6string// != "" only if IsV6 is true.var(z0unique.Handle[addrDetail]z4=unique.Make(addrDetail )z6noz=unique.Make(addrDetailisV6:true ))
AddrFrom4() encodes an IPv4 address as an
IPv4-mapped IPv6 address and sets z to the unique value z4:
// AddrFrom4 returns the address of the IPv4 address given by the bytes in addr.funcAddrFrom4(addr[4]byte)AddrreturnAddraddr:uint1280,0xffff00000000uint64(addr[0])<<24uint64(addr[1])<<16uint64(addr[2])<<8uint64(addr[3]) ,z:z4,
Unmap() turns an IPv4-mapped IPv6 address into an
IPv4 address by setting the z field to z4:
Implementing the reverse direction inside the Go standard library is trivial: we
set the z field to z6noz if the address is IPv4.
// To6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.func(ipAddr)To6()Addrifip.Is4()ip.z=z6nozreturnip
Update (2026-10)
Instead of patching Go, we can use the -overlay
flag of go build to hijack any package, including the standard
library. In our case, it can add To6() to net/netip. Yet
it is cumbersome:3 the flag expects a JSON file mapping absolute
paths in GOROOT to replacement files, and you need to add it to every go
command.
As a helper
We can t access the z field from outside the net/netip package. Instead, we
build a small helper around the netip.AddrFrom16(ip.As16()) pattern:
// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.funcAddrTo6(ipnetip.Addr)netip.Addrifip.Is4()ip=netip.AddrFrom16(ip.As16())returnip
As an unsafe function
Another solution uses the unsafe package to alter the Addr struct through
a proxy with the same memory layout:4
// addrProxy has the same memory layout as netip.Addr.typeaddrProxystructaddr[2]uint64// netip.uint128zunsafe.Pointer// unique.Handle[netip.addrDetail]var(anyIPv6=netip.IPv6Unspecified()netipZ6noz=(*addrProxy)(unsafe.Pointer(&anyIPv6)).z)// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.funcAddrTo6(ipnetip.Addr)netip.Addrif!ip.Is4()returnip(*addrProxy)(unsafe.Pointer(&ip)).z=netipZ6nozreturnip
Benchmarks
On my computer, with Go 1.27.1, the standard library solution costs 0.88 ns per
operation, while the solution favored by Go maintainers costs 7.14 ns. The
unsafe solution matches the performance of the first one.
Assembly code
Let s check the assembly code the compiler generates for each solution.5
The one built into the standard library looks like this:6
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Go s assembly language is not a direct representation of the underlying
machine language: it operates on a semi-abstract instruction set derived from
Plan 9 s assembler. It has four pseudo-registers: FP (frame pointer
for function arguments), PC (program counter), SB (static base pointer for
global symbols), and SP (stack pointer). It also has architecture-specific
registers like AX, CX, DX, BX, SI, DI, and R8 to R15.
Instructions storing data use their last argument as the destination.
Instructions can carry an explicit size suffix: MOVB moves a byte, MOVW 16
bits, MOVL 32 bits, and MOVQ 64 bits. In the example above, the first
instruction compares the 64-bit value z4 with the CX register.
The unsafe solution looks almost the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnetipZ6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
The helper solution has far more instructions. To understand why, let s look at
the code for As16() and
AddrFrom16(). They are short enough for the compiler
to inline them.
We can already guess the pattern to optimize: the code packs the IP address into
an array, copies it, then unpacks it. If we inline the Go code by hand, we get:
Unfortunately, as of Go 1.26.8, the compiler is not smart enough to do the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z; Push the stack (32 bytes):; 0(SP) addr netip.uint128; 16(SP) a16 [16]bytePUSHQBPMOVQSP,BPSUBQ$32,SPCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop here; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi); byteorder.BEPutUint64(a16[8:], input.addr.lo)MOVBEQAX,net/netip a16+16(SP)MOVBEQBX,net/netip a16+24(SP); addr = a16, 16 bytes at once through the vector register X0MOVUPSnet/netip a16+16(SP),X0MOVUPSX0,net/netip addr(SP); CX = netip.z6nozMOVQnet/netip z6noz(SB),CX; Unpack: output.addr.hi = byteorder.BEUint64(addr[:8]); output.addr.lo = byteorder.BEUint64(addr[8:])MOVBEQnet/netip addr(SP),AXMOVBEQnet/netip addr+8(SP),BXend:ADDQ$32,SPPOPQBPRET// return value = Addr hi: AX, lo: BX, z: CX
The compiler does a decent job on the byte shuffling: the eight byte stores of
BEPutUint64() become a single MOVBEQ, which stores a register byte-swapped.
The eight byte loads of BEUint64() become a single MOVBEQ the other way
round.7 Three groups of instructions remain: a pack, a copy, and an
unpack.
Hacking the Go compiler
The Go compiler has several phases:
The compiler converts the syntax tree and its types into its own intermediate
representation (IR). This process, called noding, goes through
a serialization format named unified IR.
This phase runs two steps: order of evaluation decomposes complex
statements into simpler ones, and desugaring transforms higher-level Go
constructs, like switch or channels, into more primitive instructions or
calls to the runtime.
The compiler rewrites the SSA form into machine-specific variants,
allocates registers, and applies more optimization passes. At the end, the
assembler turns the generated instructions into machine code.
The hammer
My first idea is to replace occurrences of netip.AddrFrom16(ip.As16()) with
netip.Addr addr: ip.addr, z: netip.z6noz as early as possible, during the
noding process. Before that, the type checking phase prevents
us from accessing unexported struct fields.
Go 1.27 introduced a convenient debug option to dump the IR of a function at
interesting points during compilation:
$ GOTOOLCHAIN=go1.27.1GOAMD64=v3gobuild-a-gcflags="-d=astdump=AddrTo6Safe".
Writing text ast output for AddrTo6Safe to AddrTo6Safe.astWriting html ast output for AddrTo6Safe to AddrTo6Safe.htmlWriting html syntax output for AddrTo6Safe to AddrTo6Safe.syntax.html
In the HTML file, the first column shows the IR as it comes out of noding:
In the body of the if statement, we spot the calls to the method
netip.Addr.As16() and to the function netip.AddrFrom16(). Our goal is to
patch them with a struct literal:
In noder s reader.go, the expr() method builds the IR tree for
an expression. At the end of the exprCall case, we add a call to a
rewriteAddrFrom16As16() function. It takes the current node and returns the
struct literal on success, or nil if the rewrite is not possible. First, we
check that we have the expected pattern: a call to the netip.AddrFrom16()
function with a call to the netip.Addr.As16() method as its only argument:
Have a look at the complete patch.8 We can test it
with the following commands:
$ cdsrc
$ ./make.bash
Building Go cmd/dist using /usr/lib/go-1.27. (go1.27.1 linux/amd64)Building Go toolchain1 and bootstrap cmd/go (go_bootstrap) using /usr/lib/go-1.27.Building Go toolchain2 using go_bootstrap and Go toolchain1.Building Go toolchain3 and commands using go_bootstrap and Go toolchain2.Checking command staleness for linux/amd64.---Installed Go for linux/amd64 in /home/bernat/code/free/goInstalled commands in /home/bernat/code/free/go/bin*** You need to add /home/bernat/code/free/go/bin to your PATH.$ exportPATH=$PWD/../bin:$PATH$ goversion
go version go1.28-devel_9834516e20 Sat Sep 12 08:23:11 2026 -0700 linux/amd64$ gotestnet/netip/...
ok net/netip 0.224s$ cd../../go-netip-addrto6
$ gotest.
ok github.com/vincentbernat/go-netip-addrto6 0.062s
The generated code for the helper is now the shortest possible version!
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Go maintainers are unlikely to accept this change. It relies on the internal
structure of net/netip.Addr. It s an ugly hack in the noder, whose job is to
faithfully translate the type-checked AST into the IR. And it s harder to
maintain than adding a To6() method.
The screwdriver
The right place for such an optimization is the generic SSA phase. One of
the last machine-independent passes is memcombine. With the appropriate debug
flag, the compiler dumps the SSA form after this pass:9
The result of the memcombine pass follows the same structure as the assembly
code for AddrTo6Safe()we looked at earlier: two stores,
one move, and two loads we would like to optimize away.
Each line features a value identifier (v442), an operation with its type
(Bswap64 <uint64>), and its arguments (v490).10 Values are the basic
building blocks of SSA and are defined exactly once. Square brackets enclose
integer parameters ([8]) and curly braces contain auxiliary arguments
( netip.addr ). Operations writing to memory produce a new memory state. Every
memory operation takes the current state as its last argument, which keeps them
in order.
To simplify this code, we could apply three rewriting rules:
The first one adds a shortcut when loading through a move: (Load (OffPtr
[o] p) (Move p src mem)) => (Load (OffPtr [o] src) mem). This matches v174
with its arguments v415 and v286 and creates a new value v600:
The second one simplifies a load following a store: (Load p (Store p x
_)) => x. The load is forwarded: the stored value replaces it and no memory
access remains. It matches v174. It notices that v600 and v173 are the
same address and replaces v174 with a copy of v442:
To optimize it away, we also apply three rewriting rules:
The first one also adds a shortcut when loading through a move, but
without an offset: (Load p (Move p src mem)) => (Load src mem). This rewrites v416
to use arguments from v286:
The second rule forwards a value stored one step earlier, skipping over a
store to another address: (Load p (Store q _ (Store p x _))) => x. This
matches v416: x is v542, p is v22 (&a16), q is v173
(&a16[8]), and p and q do not overlap for uint64.
In practice
Most of these rules already exist in generic.rules. They use
conditions to validate their context: ssa.IsSamePtr() for the same address,
ssa.Disjoint() for addresses that do not overlap. The rule forwarding a stored
value to a load already exists with three variants looking through several other
stores. Here are the two we need:
There is no generic rule to cancel two byte swaps, but the AMD64 lowering
pass includes this rule:
(BSWAP(Q L)(BSWAP(Q L)p))=>p
After switching to Go s development branch and adding the missing
rule, the generated assembly code is worse than with Go 1.26.8,
even though our additional rule slightly improves the situation at the end:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z; Push the stack (16 bytes):; 0(SP) a16 [16]bytePUSHQBPMOVQSP,BPSUBQ$16,SPCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop here; The four forwarded bytes: the low half of input.addr.lo is taken apart and; put back together in registersMOVQBX,DX; DX = input.addr.loSHRQ$24,BX; BX = input.addr.lo >> 24MOVQDX,SI; SI = input.addr.loSHRQ$16,DX; DX = input.addr.lo >> 16MOVQSI,DI; DI = input.addr.lo, kept for the packSHRQ$8,SI; SI = input.addr.lo >> 8MOVBLZXDIB,R8; R8 = byte(input.addr.lo)MOVBLZXSIB,SI; SI = byte(input.addr.lo >> 8)SHLQ$8,SIORQR8,SI; SI = two low bytes of input.addr.loMOVBLZXDL,DX; DX = byte(input.addr.lo >> 16)SHLQ$16,DXORQSI,DXMOVBLZXBL,BX; BX = byte(input.addr.lo >> 24)SHLQ$24,BXORQDX,BX; BX = input.addr.lo & 0xffffffff; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi); byteorder.BEPutUint64(a16[8:], input.addr.lo)MOVBEQAX,net/netip a16(SP)MOVBEQDI,net/netip a16+8(SP); The four other bytes of input.addr.lo, read one by one from a16MOVBLZXnet/netip a16+11(SP),DX; a16[11]SHLQ$32,DXORQDX,BXMOVBLZXnet/netip a16+10(SP),DX; a16[10]SHLQ$40,DXORQDX,BXMOVBLZXnet/netip a16+9(SP),DX; a16[9]SHLQ$48,DXORQDX,BXMOVBLZXnet/netip a16+8(SP),DX; a16[8]SHLQ$56,DX; output.z = netip.z6nozMOVQnet/netip z6noz(SB),CX; output.addr.hi = byteorder.BEUint64(a16[:8])MOVBEQnet/netip a16(SP),AX; output.addr.lo assembled from the previous stepsORQDX,BXend:LEAVEQRET// return value = Addr hi: AX, lo: BX, z: CX
The rule loading through a move, added in Go 1.27, introduced this regression.
Out of order
Let s not give up now! In reality, the rewriting rules run before memcombine,
notably in the late opt pass. At this point, the inlined versions of
BEPutUint64() and BEUint64() still expand to sixteen byte stores and sixteen
byte loads, matching their source code:
funcBEUint64(b[]byte)uint64_=b[7]// bounds check hint to compiler; see golang.org/issue/14808returnuint64(b[7])uint64(b[6])<<8uint64(b[5])<<16uint64(b[4])<<24uint64(b[3])<<32uint64(b[2])<<40uint64(b[1])<<48uint64(b[0])<<56
Let s follow two bytes of output.addr.lo: addr[15] and addr[11]. Here is
a simplified SSA form before late opt:
The first rule loads through the move: (Load (OffPtr [o] p) (Move p src mem))
=> (Load (OffPtr [o] src) mem). It matches both loads, which now read a16
with the memory state before the copy:
The second rule shortcuts a load following a store: (Load p (Store p x _)) =>
x. It matches v435, as v282 stores a16[15]. It does not match v481:
v235 stores a16[11] four stores earlier in the chain, while the variants of
this rule look through three stores at most.
The same happens to the other bytes: the rule forwards the four bytes stored
last, a16[12] to a16[15]. The twelve other loads now read a16 instead of
addr.
BEUint64() becomes a chain of Or64, each one adding a byte shifted into
place. memcombine is a pass written in Go, not a set of rewrite
rules. It starts from the last Or64 of the chain and collects up to eight
terms. If each term is a byte load, extended to 64 bits and shifted, and if the
eight loads read consecutive addresses from the same pointer with the same
memory state, it replaces the whole chain with a single 64-bit load and a byte
swap. Otherwise, it tries again with four, then two terms, and from each
intermediate Or64. Here is the loop checking each term in a simplified version
of combineLoads():
From v520, four of the eight terms are forwarded bytes, not loads from memory,
and memcombine can t combine them. It doesn t merge the four remaining loads
either, as they sit on top of the forwarded bytes.
Back in order
In summary, the rewriting rules run too early to be effective. A quick
workaround exists: run an earlier round of memcombine before late opt. After
this change, the generated code for the helper is back to the
shortest possible version:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Next steps
I think Go maintainers would reject this change because of the additional
memcombine pass. Instead, I plan to publish this blog post and bring up the
subject again as a follow-up to issue #54365. Either the sheer complexity and
the Go 1.27 regression convince the maintainers that adding a To6() method is
simpler and more efficient, or they advise me on how to move forward. Either
way, digging into this subject taught me a lot about the Go compiler!
Update (2026-10)
I opened issue #81994 to propose Addr.To6(). Give it a if you want it in Go!
IPv4-mapped IPv6 addresses let you handle only IPv6 addresses in your code
and keep conversions at a few well-defined boundaries. I use them in
Akvorado.
This is not strictly equivalent. The zero value becomes :: while it would
make more sense to leave it untouched.
I suppose Go maintainers keep it inconvenient on purpose: they don t want
random packages to patch the standard library or other packages.
To avoid catastrophic bugs when netip.Addr s layout changes, you must add
tests to detect it. This is more dangerous if you put this code in a
package. Either ask users to run the tests themselves, or detect the change
at run time and panic. Hiding this optimization behind a build tag would
make users aware of this potential trap.
I produced the assembly code with GOTOOLCHAIN=go1.26.8 GOAMD64=v3 ./go-asm
'\.asm' from the companion repository, then edited it a bit to keep this
article from turning into an endless rabbit hole. Due to an unfortunate
sequence of events, we switch between versions: Go 1.27 introduced a
regression that blurs the point of this article.
The function is short enough for the compiler to inline it, so the assembly
code may vary depending on the surrounding code.
MOVBEQ requires GOAMD64=v3, matching the x86-64-v3
microarchitecture from 2013. Otherwise, the compiler translates
BEPutUint64() to BSWAPQ+MOVQ and BEUint64() to MOVQ+BSWAPQ.
When the call is used as a statement, a struct literal alone is invalid. The
patch then turns it into _ = netip.Addr .
The compiler can also write an HTML file with the result of each pass:
$ GOSSAFUNC=AddrTo6Safe\> GOTOOLCHAIN=go1.26.8\> GOAMD64=v3gobuild-a.
dumped SSA for AddrTo6Safe,1 to ./ssa.html
Source line numbers in parentheses follow value identifiers, but I removed
them from the examples. The SSA form also features branches, but we don t
need them to understand our optimizations.
netip.Addr features an Unmap()
method returning the unwrapped IPv4 contained in an IPv4-mapped IPv6
address: from ::ffff:203.0.113.10 or ::ffff:cb00:710a, it returns
203.0.113.10.1 There is no Map() or To6() method for the reverse
direction. Such a method is trivial to implement, but Go maintainers have
rejected it on the grounds that users should write
netip.AddrFrom16(ip.As16()) and let the compiler optimize it.2 Today,
this pattern is eight times slower than a native method. How can we teach the
compiler to optimize this sequence?
The alternatives
Let s explore three ways to implement the map semantics for netip.Addr. My
favorite is to add it to the Go standard library. Go maintainers prefer a small
external helper chaining netip.AddrFrom16() and netip.Addr.As16(), hoping
the compiler eventually optimizes it. The unsafe package
opens a third path, with the same performance as the first solution.
Modifying the Go standard library
Internally, netip.Addr stores any IP address as a
128-bit value with an extra field z to encode the family and the zone:
typeAddrstructaddruint128zunique.Handle[addrDetail]typeaddrDetailstructisV6bool// IPv4 is false, IPv6 is true.zoneV6string// != "" only if IsV6 is true.var(z0unique.Handle[addrDetail]z4=unique.Make(addrDetail )z6noz=unique.Make(addrDetailisV6:true ))
AddrFrom4() encodes an IPv4 address as an
IPv4-mapped IPv6 address and sets z to the unique value z4:
// AddrFrom4 returns the address of the IPv4 address given by the bytes in addr.funcAddrFrom4(addr[4]byte)AddrreturnAddraddr:uint1280,0xffff00000000uint64(addr[0])<<24uint64(addr[1])<<16uint64(addr[2])<<8uint64(addr[3]) ,z:z4,
Unmap() turns an IPv4-mapped IPv6 address into an
IPv4 address by setting the z field to z4:
Implementing the reverse direction inside the Go standard library is trivial: we
set the z field to z6noz if the address is IPv4.
// To6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.func(ipAddr)To6()Addrifip.Is4()ip.z=z6nozreturnip
As a helper
We can t access the z field from outside the net/netip package. Instead, we
build a small helper around the netip.AddrFrom16(ip.As16()) pattern:
// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.funcAddrTo6(ipnetip.Addr)netip.Addrifip.Is4()ip=netip.AddrFrom16(ip.As16())returnip
As an unsafe function
Another solution uses the unsafe package to alter the Addr struct through
a proxy with the same memory layout:3
// addrProxy has the same memory layout as netip.Addr.typeaddrProxystructaddr[2]uint64// netip.uint128zunsafe.Pointer// unique.Handle[netip.addrDetail]var(anyIPv6=netip.IPv6Unspecified()netipZ6noz=(*addrProxy)(unsafe.Pointer(&anyIPv6)).z)// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an// IPv6 address unmodified.funcAddrTo6(ipnetip.Addr)netip.Addrif!ip.Is4()returnip(*addrProxy)(unsafe.Pointer(&ip)).z=netipZ6nozreturnip
Benchmarks
On my computer, with Go 1.27.1, the standard library solution costs 0.88 ns per
operation, while the solution favored by Go maintainers costs 7.14 ns. The
unsafe solution matches the performance of the first one.
Assembly code
Let s check the assembly code the compiler generates for each solution.4
The one built into the standard library looks like this:5
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Go s assembly language is not a direct representation of the underlying
machine language: it operates on a semi-abstract instruction set derived from
Plan 9 s assembler. It has four pseudo-registers: FP (frame pointer
for function arguments), PC (program counter), SB (static base pointer for
global symbols), and SP (stack pointer). It also has architecture-specific
registers like AX, CX, DX, BX, SI, DI, and R8 to R15.
Instructions storing data use their last argument as the destination.
Instructions can carry an explicit size suffix: MOVB moves a byte, MOVW 16
bits, MOVL 32 bits, and MOVQ 64 bits. In the example above, the first
instruction compares the 64-bit value z4 with the CX register.
The unsafe solution looks almost the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnetipZ6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
The helper solution has far more instructions. To understand why, let s look at
the code for As16() and
AddrFrom16(). They are short enough for the compiler
to inline them.
We can already guess the pattern to optimize: the code packs the IP address into
an array, copies it, then unpacks it. If we inline the Go code by hand, we get:
Unfortunately, as of Go 1.26.8, the compiler is not smart enough to do the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z; Push the stack (32 bytes):; 0(SP) addr netip.uint128; 16(SP) a16 [16]bytePUSHQBPMOVQSP,BPSUBQ$32,SPCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop here; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi); byteorder.BEPutUint64(a16[8:], input.addr.lo)MOVBEQAX,net/netip a16+16(SP)MOVBEQBX,net/netip a16+24(SP); addr = a16, 16 bytes at once through the vector register X0MOVUPSnet/netip a16+16(SP),X0MOVUPSX0,net/netip addr(SP); CX = netip.z6nozMOVQnet/netip z6noz(SB),CX; Unpack: output.addr.hi = byteorder.BEUint64(addr[:8]); output.addr.lo = byteorder.BEUint64(addr[8:])MOVBEQnet/netip addr(SP),AXMOVBEQnet/netip addr+8(SP),BXend:ADDQ$32,SPPOPQBPRET// return value = Addr hi: AX, lo: BX, z: CX
The compiler does a decent job on the byte shuffling: the eight byte stores of
BEPutUint64() become a single MOVBEQ, which stores a register byte-swapped.
The eight byte loads of BEUint64() become a single MOVBEQ the other way
round.6 Three groups of instructions remain: a pack, a copy, and an
unpack.
Hacking the Go compiler
The Go compiler has several phases:
The compiler converts the syntax tree and its types into its own intermediate
representation (IR). This process, called noding, goes through
a serialization format named unified IR.
This phase runs two steps: order of evaluation decomposes complex
statements into simpler ones, and desugaring transforms higher-level Go
constructs, like switch or channels, into more primitive instructions or
calls to the runtime.
The compiler rewrites the SSA form into machine-specific variants,
allocates registers, and applies more optimization passes. At the end, the
assembler turns the generated instructions into machine code.
The hammer
My first idea is to replace occurrences of netip.AddrFrom16(ip.As16()) with
netip.Addr addr: ip.addr, z: netip.z6noz as early as possible, during the
noding process. Before that, the type checking phase prevents
us from accessing unexported struct fields.
Go 1.27 introduced a convenient debug option to dump the IR of a function at
interesting points during compilation:
$ GOTOOLCHAIN=go1.27.1GOAMD64=v3gobuild-a-gcflags="-d=astdump=AddrTo6Safe".
Writing text ast output for AddrTo6Safe to AddrTo6Safe.astWriting html ast output for AddrTo6Safe to AddrTo6Safe.htmlWriting html syntax output for AddrTo6Safe to AddrTo6Safe.syntax.html
In the HTML file, the first column shows the IR as it comes out of noding:
In the body of the if statement, we spot the calls to the method
netip.Addr.As16() and to the function netip.AddrFrom16(). Our goal is to
patch them with a struct literal:
In noder s reader.go, the expr() method builds the IR tree for
an expression. At the end of the exprCall case, we add a call to a
rewriteAddrFrom16As16() function. It takes the current node and returns the
struct literal on success, or nil if the rewrite is not possible. First, we
check that we have the expected pattern: a call to the netip.AddrFrom16()
function with a call to the netip.Addr.As16() method as its only argument:
Have a look at the complete patch.7 We can test it
with the following commands:
$ cdsrc
$ ./make.bash
Building Go cmd/dist using /usr/lib/go-1.27. (go1.27.1 linux/amd64)Building Go toolchain1 and bootstrap cmd/go (go_bootstrap) using /usr/lib/go-1.27.Building Go toolchain2 using go_bootstrap and Go toolchain1.Building Go toolchain3 and commands using go_bootstrap and Go toolchain2.Checking command staleness for linux/amd64.---Installed Go for linux/amd64 in /home/bernat/code/free/goInstalled commands in /home/bernat/code/free/go/bin*** You need to add /home/bernat/code/free/go/bin to your PATH.$ exportPATH=$PWD/../bin:$PATH$ goversion
go version go1.28-devel_9834516e20 Sat Sep 12 08:23:11 2026 -0700 linux/amd64$ gotestnet/netip/...
ok net/netip 0.224s$ cd../../go-netip-addrto6
$ gotest.
ok github.com/vincentbernat/go-netip-addrto6 0.062s
The generated code for the helper is now the shortest possible version!
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Go maintainers are unlikely to accept this change. It relies on the internal
structure of net/netip.Addr. It s an ugly hack in the noder, whose job is to
faithfully translate the type-checked AST into the IR. And it s harder to
maintain than adding a To6() method.
The screwdriver
The right place for such an optimization is the generic SSA phase. One of
the last machine-independent passes is memcombine. With the appropriate debug
flag, the compiler dumps the SSA form after this pass:8
The result of the memcombine pass follows the same structure as the assembly
code for AddrTo6Safe()we looked at earlier: two stores,
one move, and two loads we would like to optimize away.
Each line features a value identifier (v442), an operation with its type
(Bswap64 <uint64>), and its arguments (v490).9 Values are the basic
building blocks of SSA and are defined exactly once. Square brackets enclose
integer parameters ([8]) and curly braces contain auxiliary arguments
( netip.addr ). Operations writing to memory produce a new memory state. Every
memory operation takes the current state as its last argument, which keeps them
in order.
To simplify this code, we could apply three rewriting rules:
The first one adds a shortcut when loading through a move: (Load (OffPtr
[o] p) (Move p src mem)) => (Load (OffPtr [o] src) mem). This matches v174
with its arguments v415 and v286 and creates a new value v600:
The second one simplifies a load following a store: (Load p (Store p x
_)) => x. The load is forwarded: the stored value replaces it and no memory
access remains. It matches v174. It notices that v600 and v173 are the
same address and replaces v174 with a copy of v442:
To optimize it away, we also apply three rewriting rules:
The first one also adds a shortcut when loading through a move, but
without an offset: (Load p (Move p src mem)) => (Load src mem). This rewrites v416
to use arguments from v286:
The second rule forwards a value stored one step earlier, skipping over a
store to another address: (Load p (Store q _ (Store p x _))) => x. This
matches v416: x is v542, p is v22 (&a16), q is v173
(&a16[8]), and p and q do not overlap for uint64.
In practice
Most of these rules already exist in generic.rules. They use
conditions to validate their context: ssa.IsSamePtr() for the same address,
ssa.Disjoint() for addresses that do not overlap. The rule forwarding a stored
value to a load already exists with three variants looking through several other
stores. Here are the two we need:
There is no generic rule to cancel two byte swaps, but the AMD64 lowering
pass includes this rule:
(BSWAP(Q L)(BSWAP(Q L)p))=>p
After switching to Go s development branch and adding the missing
rule, the generated assembly code is worse than with Go 1.26.8,
even though our additional rule slightly improves the situation at the end:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z; Push the stack (16 bytes):; 0(SP) a16 [16]bytePUSHQBPMOVQSP,BPSUBQ$16,SPCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop here; The four forwarded bytes: the low half of input.addr.lo is taken apart and; put back together in registersMOVQBX,DX; DX = input.addr.loSHRQ$24,BX; BX = input.addr.lo >> 24MOVQDX,SI; SI = input.addr.loSHRQ$16,DX; DX = input.addr.lo >> 16MOVQSI,DI; DI = input.addr.lo, kept for the packSHRQ$8,SI; SI = input.addr.lo >> 8MOVBLZXDIB,R8; R8 = byte(input.addr.lo)MOVBLZXSIB,SI; SI = byte(input.addr.lo >> 8)SHLQ$8,SIORQR8,SI; SI = two low bytes of input.addr.loMOVBLZXDL,DX; DX = byte(input.addr.lo >> 16)SHLQ$16,DXORQSI,DXMOVBLZXBL,BX; BX = byte(input.addr.lo >> 24)SHLQ$24,BXORQDX,BX; BX = input.addr.lo & 0xffffffff; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi); byteorder.BEPutUint64(a16[8:], input.addr.lo)MOVBEQAX,net/netip a16(SP)MOVBEQDI,net/netip a16+8(SP); The four other bytes of input.addr.lo, read one by one from a16MOVBLZXnet/netip a16+11(SP),DX; a16[11]SHLQ$32,DXORQDX,BXMOVBLZXnet/netip a16+10(SP),DX; a16[10]SHLQ$40,DXORQDX,BXMOVBLZXnet/netip a16+9(SP),DX; a16[9]SHLQ$48,DXORQDX,BXMOVBLZXnet/netip a16+8(SP),DX; a16[8]SHLQ$56,DX; output.z = netip.z6nozMOVQnet/netip z6noz(SB),CX; output.addr.hi = byteorder.BEUint64(a16[:8])MOVBEQnet/netip a16(SP),AX; output.addr.lo assembled from the previous stepsORQDX,BXend:LEAVEQRET// return value = Addr hi: AX, lo: BX, z: CX
The rule loading through a move, added in Go 1.27, introduced this regression.
Out of order
Let s not give up now! In reality, the rewriting rules run before memcombine,
notably in the late opt pass. At this point, the inlined versions of
BEPutUint64() and BEUint64() still expand to sixteen byte stores and sixteen
byte loads, matching their source code:
funcBEUint64(b[]byte)uint64_=b[7]// bounds check hint to compiler; see golang.org/issue/14808returnuint64(b[7])uint64(b[6])<<8uint64(b[5])<<16uint64(b[4])<<24uint64(b[3])<<32uint64(b[2])<<40uint64(b[1])<<48uint64(b[0])<<56
Let s follow two bytes of output.addr.lo: addr[15] and addr[11]. Here is
a simplified SSA form before late opt:
The first rule loads through the move: (Load (OffPtr [o] p) (Move p src mem))
=> (Load (OffPtr [o] src) mem). It matches both loads, which now read a16
with the memory state before the copy:
The second rule shortcuts a load following a store: (Load p (Store p x _)) =>
x. It matches v435, as v282 stores a16[15]. It does not match v481:
v235 stores a16[11] four stores earlier in the chain, while the variants of
this rule look through three stores at most.
The same happens to the other bytes: the rule forwards the four bytes stored
last, a16[12] to a16[15]. The twelve other loads now read a16 instead of
addr.
BEUint64() becomes a chain of Or64, each one adding a byte shifted into
place. memcombine is a pass written in Go, not a set of rewrite
rules. It starts from the last Or64 of the chain and collects up to eight
terms. If each term is a byte load, extended to 64 bits and shifted, and if the
eight loads read consecutive addresses from the same pointer with the same
memory state, it replaces the whole chain with a single 64-bit load and a byte
swap. Otherwise, it tries again with four, then two terms, and from each
intermediate Or64. Here is the loop checking each term in a simplified version
of combineLoads():
From v520, four of the eight terms are forwarded bytes, not loads from memory,
and memcombine can t combine them. It doesn t merge the four remaining loads
either, as they sit on top of the forwarded bytes.
Back in order
In summary, the rewriting rules run too early to be effective. A quick
workaround exists: run an earlier round of memcombine before late opt. After
this change, the generated code for the helper is back to the
shortest possible version:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.zCMPQnet/netip z4(SB),CX; check "z" if this is an IPv4 addressJNEend; if not, stop hereMOVQnet/netip z6noz(SB),CX; CX = netip.z6nozend:RET// return value = Addr hi: AX, lo: BX, z: CX
Next steps
I think Go maintainers would reject this change because of the additional
memcombine pass. Instead, I plan to publish this blog post and bring up the
subject again as a follow-up to issue #54365. Either the sheer complexity and
the Go 1.27 regression convince the maintainers that adding a To6() method is
simpler and more efficient, or they advise me on how to move forward. Either
way, digging into this subject taught me a lot about the Go compiler!
IPv4-mapped IPv6 addresses let you handle only IPv6 addresses in your code
and keep conversions at a few well-defined boundaries. I use them in
Akvorado.
This is not strictly equivalent. The zero value becomes :: while it would
make more sense to leave it untouched.
To avoid catastrophic bugs when netip.Addr s layout changes, you must add
tests to detect it. This is more dangerous if you put this code in a
package. Either ask users to run the tests themselves, or detect the change
at run time and panic. Hiding this optimization behind a build tag would
make users aware of this potential trap.
I produced the assembly code with GOTOOLCHAIN=go1.26.8 GOAMD64=v3 ./go-asm
'\.asm' from the companion repository, then edited it a bit to keep this
article from turning into an endless rabbit hole. Due to an unfortunate
sequence of events, we switch between versions: Go 1.27 introduced a
regression that blurs the point of this article.
The function is short enough for the compiler to inline it, so the assembly
code may vary depending on the surrounding code.
MOVBEQ requires GOAMD64=v3, matching the x86-64-v3
microarchitecture from 2013. Otherwise, the compiler translates
BEPutUint64() to BSWAPQ+MOVQ and BEUint64() to MOVQ+BSWAPQ.
When the call is used as a statement, a struct literal alone is invalid. The
patch then turns it into _ = netip.Addr .
The compiler can also write an HTML file with the result of each pass:
$ GOSSAFUNC=AddrTo6Safe\> GOTOOLCHAIN=go1.26.8\> GOAMD64=v3gobuild-a.
dumped SSA for AddrTo6Safe,1 to ./ssa.html
Source line numbers in parentheses follow value identifiers, but I removed
them from the examples. The SSA form also features branches, but we don t
need them to understand our optimizations.
A friend wants to proofread your work-in-progress blog post, but its preview
only runs on localhost:8080. Several tools can help. Some run as a
commercial service, like ngrok or Cloudflare Quick Tunnels. Some are
self-hostable but require a specific client, like frp or localtunnel.
Some only require a plain SSH client but rely on a specific SSH server, like
sish. Let s implement a self-hosted solution with only OpenSSH and
nginx!
$ ssh-R0:localhost:8080http-over-ssh
Allocated port 41535 for remote forward to localhost:8080https://6J3jK1WmB15c6WmjW_X-Wg--1789928654@p41535.ssh.luffy.cx/
Basic setup
First, we forward connections from a port on a remote server to your local
service:
$ ssh-N-R0:localhost:8080web02.luffy.cx
Allocated port 41535 for remote forward to localhost:8080
When you specify 0 as the remote port, the server allocates a free port.
Then, we configure nginx to proxy requests from https://p41535.ssh.luffy.cx to
http://127.0.0.1:41535:
Access control
The port is the only secret 1 keeping the content confidential. Other
forwarding solutions add a random string to the domain name to prevent an
intruder from enumerating the possible values.
Thanks to ngx_http_secure_link_module, we can secure
this setup a bit. This module computes a hash2 over a set of values,
including a secret, and compares it with the hash from the request. The hash is
base64-encoded, so we cannot put it in the domain name, which is
case-insensitive. Instead, we put it in the URL as a username, along with its
expiration timestamp:3
https://6J3jK1WmB15c6WmjW_X-Wg--1789928654@p41535.ssh.luffy.cx/en/blog
hash expires port path
The client sends the username to the server with HTTP basic
authentication. This works with most HTTP clients, including curl.
Nginx exposes the username in the $remote_user variable. The module expects
the hash and the expiration timestamp separated by a comma. We use a map
directive to extract the two parts from $remote_user and join them with a
comma.4 We also give the module the string to hash. It contains the
expiration timestamp, the port, and a secret:
The module returns the status of the check in the $secure_link variable:
empty if the hashes do not match,
"0" if they match but the link has expired, or
"1" otherwise.
If the hash is incorrect or missing, we return a 401 error with a
WWW-Authenticate header to ask for credentials. If the link has expired, we
return a 410 error. We remove the Authorization header before forwarding the
request and add a few directives to proxy WebSocket connections.
Here is the complete configuration:5
Well, I suppose you are now saying: Vincent, this is not very convenient! I ll stick
with ngrok if you don t mind. Okay, I hear you. Let s write a helper script.
Helper script
The main difficulty is finding the ephemeral port that OpenSSH allocates, as it
does not appear in any environment variable.6 To work around this obstacle,
we look for the ancestor sshd-session processes:7
With this solution, I only rely on OpenSSH and nginx, two pieces of software
already running on this server. One short command gives me a self-hosted tunnel
and a URL to share. To try it, grab the complete helper script, which
includes a few minor improvements. If you run NixOS, as any person of taste
would, have a look at my http-over-ssh.nix instead.
Update (2026-10)
This post was widely discussed on Hacker News.
Update (2026-10)
I have since switched to Caddy, which does not include a
module to check the validity of a URL. Instead, Caddy queries a local HTTP
service written in Go through the forward_auth directive. As a bonus,
this lets me put the token in the domain name.
Since the kernel allocates it from the local port range, its entropy is low.
Moreover, on Linux, we lose one more bit because the kernel favors odd
ports when it picks a random free port.
The module relies on MD5. Its security is weak but good enough for this
use.
We need an expiration date because a later session could reuse the port,
and we have no way to detect this case.
The username could use a comma directly instead of a double dash. But some
applications do not recognize the URL correctly, making it harder to share.
This configuration exposes any TCP port listening on 127.0.0.1 or
0.0.0.0 to anyone with the secret, bypassing most firewall rules. You
could strengthen it by limiting the server_name regex to the ephemeral
port range.
One SSH session can contain several tunnels, and the client can add or
remove them at any time. This is unlike tun device forwarding (ssh -w),
which gets its own SSH_TUNNEL environment variable.
Since OpenSSH 9.8, sshd-session is the name of the helper handling the
session. With older versions, look for sshd instead.
We need sudo because the sshd-session process dropped its privileges.
The kernel then marks it as not dumpable, and its /proc/PID/fd directory
belongs to root. Without access to this directory, ss cannot
find which process owns a socket.
One of the primary design principles for Linus Torvalds when creating Git is that it should be a distributed system, capable of working with multiple remotes. However, most people seem to use Git with just one remote, typically having GitHub or GitLab as the only origin.
Version control systems before Git relied on a centralized model. For example in Subversion or CVS you could only sync with one single remote server, which was considered the primary host of the source code. Git however is not just a VCS, but a DVCS, for distributed version control system. If you currently use Git with just one single remote as if it was a centralized version control system, now is the time to change that and learn how to add multiple remotes, and configure more flexible git pull and git push rules.
The first concept to grasp is that Git can have multiple remotes. For example, after running a basic clone command such as git clone https://github.com/ottok/debcraft.git you would end up with a single remote origin pointing to GitHub:
You can push and pull from these individually, or from all at once:
$ git pull --all
Fetching github
Fetching gitlab
Already up to date.
$ git pull --all
Fetching github
Fetching gitlab
Already up to date.
When you review the git history, note the labels gitlab/main and github/main which tell which commit the branch main points to on various remotes. In this case they are all in sync and point to the same 5d1815c:
$ git log --oneline
5d1815c (HEAD -> main, gitlab/main, github/main) Bump Debhelper version from 13 to 14
ca518a7 Temporarily disable automatic copyright year bumping due to frequent bugs
d6e201e Exclude .pm from codespell to decrease false positives (Closes: #1140488)
708ee35 Exclude debian/copyright from codespell due to frequent false positives
$ git log --oneline
5d1815c (HEAD -> main, gitlab/main, github/main) Bump Debhelper version from 13 to 14
ca518a7 Temporarily disable automatic copyright year bumping due to frequent bugs
d6e201e Exclude .pm from codespell to decrease false positives (Closes: #1140488)
708ee35 Exclude debian/copyright from codespell due to frequent false positives
When pushing, you can choose what remote to push to. For example, if I am working on branch dev and I only want to push it to GitLab to trigger a CI run there, I can run git push gitlab dev. You can also configure a default remote by running once git push --set-upstream gitlab dev and thereafter simply run git push and it will automatically push the branch dev only to that remote.
A related setting is remote.pushDefault, which tells git which remote to push to when a branch does not define one of its own, and it applies to every branch that lacks a [branch] section in .git/config. Setting it once is what lets a repository have a per-branch pull remote, like the debcraft example below, and still have a single, predictable git push target. To see which remote git will use for each of your branches, run git branch -vv, and to refresh the remote-tracking branches of all remotes at once, run git fetch --all --prune.
If you want to have multiple remotes, but automatically push and pull to all of them at once, you configure one single remote to have multiple URLs with git remote set-url --add. Since git prints one line per push URL, git remote -v then shows the same remote several times, once for each URL it pushes to.
As an illustration, in my actual local Debcraft development project I have all these configured:
$ git remote -v
github-pull-requests git@github.com:ottok/debcraft.git (fetch)
github-pull-requests git@github.com:ottok/debcraft.git (push)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (fetch)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (push)
origin git@salsa.debian.org:debian/debcraft.git (fetch)
origin git@salsa.debian.org:debian/debcraft.git (push)
origin git@gitlab.com:ottok/debcraft.git (push)
origin git@github.com:ottok/debcraft.git (push)
otto git@salsa.debian.org:otto/debcraft.git (fetch)
otto git@salsa.debian.org:otto/debcraft.git (push)
otto git@gitlab.com:ottok/debcraft.git (push)
otto git@github.com:ottok/debcraft.git (push)
otto git@git.sr.ht:~ottok/debcraft (push)
otto ssh://git@codeberg.org/ottok/Debcraft.git (push)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (fetch)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (push)
$ git remote -v
github-pull-requests git@github.com:ottok/debcraft.git (fetch)
github-pull-requests git@github.com:ottok/debcraft.git (push)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (fetch)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (push)
origin git@salsa.debian.org:debian/debcraft.git (fetch)
origin git@salsa.debian.org:debian/debcraft.git (push)
origin git@gitlab.com:ottok/debcraft.git (push)
origin git@github.com:ottok/debcraft.git (push)
otto git@salsa.debian.org:otto/debcraft.git (fetch)
otto git@salsa.debian.org:otto/debcraft.git (push)
otto git@gitlab.com:ottok/debcraft.git (push)
otto git@github.com:ottok/debcraft.git (push)
otto git@git.sr.ht:~ottok/debcraft (push)
otto ssh://git@codeberg.org/ottok/Debcraft.git (push)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (fetch)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (push)
The settings can also be viewed directly in the config file:
Use cases for distributed version control
The most obvious use case for any distributed system is improved availability. In this example, the salsa.debian.org is often overloaded/unavailable, so pushing to GitLab and GitHub as well allows collaborators to use them to fetch git commits if the primary code hosting site is unresponsive. Once it comes back online again, the next git push/pull will automatically bring it up to the same content as what backup hosts had.
The second main benefit is that having the code live on multiple code forges allows collaborators to use their own favorite host. Debcraft itself is a great example of this, as collaborators who don t have a Salsa account can - and have - submitted Merge Requests on GitLab.com and Pull Requests on GitHub.com.
A third use case is code hosting migrations. If you for example want to gradually phase out GitHub.com and replace it with something else, you could configure git to pull from the old location and push to both, ensuring that other collaborators can start using the new code hosting location without disruptions to those who have not yet updated their remotes or done a fresh git clone from the new location.
Automating setting up a project on multiple code forges
While I don t push all projects I work on to all Salsa/GitLab/GitHub, I do set up multiple remotes for enough git projects that I have this script to automate configuring a git repository that is cloned from Salsa to also push to GitLab and GitHub. It accepts the repository whether it was cloned over SSH or HTTPS, and it adds the push URLs and then pushes the current branch, all branches, and tags for you:
#!/bin/bash
# Add GitLab and GitHub push URLs to an existing git remote, so that a plain
# "git push" writes the same commits to all three code forges.
#
# Usage: ./git-multipush-salsa.sh
#
# Environment:
# SALSA_REMOTE remote to turn into a mirror, default origin
# SALSA_USER your Salsa user, only used to detect that the source
# remote is somebody else's project
# GITLAB_USER your GitLab user, default ottok
# GITHUB_USER your GitHub user, default ottok
# PROTOCOL ssh (default) or https, for the push URLs
set -euo pipefail
SCRIPT=$(basename "$0")
SALSA_REMOTE=$ SALSA_REMOTE:-origin
SALSA_USER=$ SALSA_USER:-otto
GITLAB_USER=$ GITLAB_USER:-ottok
GITHUB_USER=$ GITHUB_USER:-ottok
PROTOCOL=$ PROTOCOL:-ssh
die() echo "$SCRIPT: $*" >&2; exit 1;
# A failing command leaves the push URLs half configured behind, so say how
# to get back to where we started
fail()
echo "$SCRIPT: '$*' failed, the push URLs may already have been added" >&2
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl" >&2
echo " git remote set-url --add --push $ SALSA_REMOTE $ SALSA_URL " >&2
exit 1
git rev-parse --git-dir > /dev/null 2>&1 die "not inside a git repository"
git remote grep -qx "$SALSA_REMOTE" die "no remote named '$SALSA_REMOTE'"
# Example: git@salsa.debian.org:otto/salsa-ci-pipeline.git
SALSA_URL=$(git remote get-url "$SALSA_REMOTE")
# The source remote may use any of the URL forms salsa.debian.org accepts,
# for example git@salsa.debian.org:otto/salsa-ci-pipeline.git or
# https://salsa.debian.org/otto/salsa-ci-pipeline.git
case "$SALSA_URL" in
git@salsa.debian.org:*) SALSA_PATH=$ SALSA_URL#git@salsa.debian.org: ;;
ssh://git@salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#ssh://git@salsa.debian.org/ ;;
https://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#https://salsa.debian.org/ ;;
http://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#http://salsa.debian.org/ ;;
*) die "unexpected source URL '$SALSA_URL', expected a salsa.debian.org URL" ;;
esac
SALSA_PATH=$ SALSA_PATH%/
SALSA_PATH=$ SALSA_PATH%.git
SOURCE_USER=$ SALSA_PATH%%/*
PROJECT=$ SALSA_PATH##*/
# Write the push URLs in one and the same protocol, no matter how the
# repository happens to have been cloned
if [ "$PROTOCOL" = "https" ]
then
SALSA_URL="https://salsa.debian.org/$ SALSA_PATH .git"
GITLAB_URL="https://gitlab.com/$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="https://github.com/$ GITHUB_USER /$ PROJECT .git"
else
SALSA_URL="git@salsa.debian.org:$ SALSA_PATH .git"
GITLAB_URL="git@gitlab.com:$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="git@github.com:$ GITHUB_USER /$ PROJECT .git"
fi
# Mirroring somebody else's project is fine, but it is usually not what you
# want: you most likely want to mirror your own fork instead
if [ "$SOURCE_USER" != "$SALSA_USER" ]
then
echo "Note: $SALSA_REMOTE points to $SOURCE_USER's project, not your own."
echo "To mirror your own fork, run this with SALSA_REMOTE set to its name."
echo
fi
# Bail out instead of adding the same push URLs twice
EXISTING=$(git remote get-url --push --all "$SALSA_REMOTE")
if [ "$(printf '%s\n' "$EXISTING" wc -l)" -gt 1 ]
then
die "remote '$SALSA_REMOTE' already has multiple push URLs:
$EXISTING"
fi
echo "Mirroring $SALSA_URL to $GITLAB_URL and $GITHUB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$SALSA_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITLAB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITHUB_URL"
# Push the branch you have checked out first, and never with --force: the
# first branch a GitLab or GitHub project receives becomes its default branch,
# the one you then protect
DEFAULT_BRANCH=$(git symbolic-ref --short HEAD) die "not on a branch, check out a branch first"
git push -u "$SALSA_REMOTE" "$DEFAULT_BRANCH"
git push "$SALSA_REMOTE" --all
git push "$SALSA_REMOTE" --tags
# The push above creates the GitLab and GitHub projects, but GitLab projects
# are private by default, so make the new mirror public with glab if installed.
if command -v glab > /dev/null
then
echo "Make the GitLab project public and describe it as a mirror:"
glab api -X PUT "projects/$ GITLAB_USER %2F$ PROJECT " \
-f visibility=public \
-f description="Mirror of $ SALSA_URL " \
-f notification_email=disabled
else
echo "Install glab to do the same automatically, or visit"
echo "https://gitlab.com/$ GITLAB_USER /$ PROJECT /edit to make the project public"
fi
echo "To undo, drop all push URLs and add back only the original one:"
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl"
echo " git remote set-url --add --push $ SALSA_REMOTE $ SALSA_URL "
#!/bin/bash
# Add GitLab and GitHub push URLs to an existing git remote, so that a plain
# "git push" writes the same commits to all three code forges.
#
# Usage: ./git-multipush-salsa.sh
#
# Environment:
# SALSA_REMOTE remote to turn into a mirror, default origin
# SALSA_USER your Salsa user, only used to detect that the source
# remote is somebody else's project
# GITLAB_USER your GitLab user, default ottok
# GITHUB_USER your GitHub user, default ottok
# PROTOCOL ssh (default) or https, for the push URLs
set -euo pipefail
SCRIPT=$(basename "$0")
SALSA_REMOTE=$ SALSA_REMOTE:-origin
SALSA_USER=$ SALSA_USER:-otto
GITLAB_USER=$ GITLAB_USER:-ottok
GITHUB_USER=$ GITHUB_USER:-ottok
PROTOCOL=$ PROTOCOL:-ssh
die() echo "$SCRIPT: $*" >&2; exit 1;
# A failing command leaves the push URLs half configured behind, so say how
# to get back to where we started
fail()
echo "$SCRIPT: '$*' failed, the push URLs may already have been added" >&2
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl" >&2
echo " git remote set-url --add --push $ SALSA_REMOTE $ SALSA_URL " >&2
exit 1
git rev-parse --git-dir > /dev/null 2>&1 die "not inside a git repository"
git remote grep -qx "$SALSA_REMOTE" die "no remote named '$SALSA_REMOTE'"
# Example: git@salsa.debian.org:otto/salsa-ci-pipeline.git
SALSA_URL=$(git remote get-url "$SALSA_REMOTE")
# The source remote may use any of the URL forms salsa.debian.org accepts,
# for example git@salsa.debian.org:otto/salsa-ci-pipeline.git or
# https://salsa.debian.org/otto/salsa-ci-pipeline.git
case "$SALSA_URL" in
git@salsa.debian.org:*) SALSA_PATH=$ SALSA_URL#git@salsa.debian.org: ;;
ssh://git@salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#ssh://git@salsa.debian.org/ ;;
https://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#https://salsa.debian.org/ ;;
http://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#http://salsa.debian.org/ ;;
*) die "unexpected source URL '$SALSA_URL', expected a salsa.debian.org URL" ;;
esac
SALSA_PATH=$ SALSA_PATH%/
SALSA_PATH=$ SALSA_PATH%.git
SOURCE_USER=$ SALSA_PATH%%/*
PROJECT=$ SALSA_PATH##*/
# Write the push URLs in one and the same protocol, no matter how the
# repository happens to have been cloned
if [ "$PROTOCOL" = "https" ]
then
SALSA_URL="https://salsa.debian.org/$ SALSA_PATH .git"
GITLAB_URL="https://gitlab.com/$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="https://github.com/$ GITHUB_USER /$ PROJECT .git"
else
SALSA_URL="git@salsa.debian.org:$ SALSA_PATH .git"
GITLAB_URL="git@gitlab.com:$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="git@github.com:$ GITHUB_USER /$ PROJECT .git"
fi
# Mirroring somebody else's project is fine, but it is usually not what you
# want: you most likely want to mirror your own fork instead
if [ "$SOURCE_USER" != "$SALSA_USER" ]
then
echo "Note: $SALSA_REMOTE points to $SOURCE_USER's project, not your own."
echo "To mirror your own fork, run this with SALSA_REMOTE set to its name."
echo
fi
# Bail out instead of adding the same push URLs twice
EXISTING=$(git remote get-url --push --all "$SALSA_REMOTE")
if [ "$(printf '%s\n' "$EXISTING" wc -l)" -gt 1 ]
then
die "remote '$SALSA_REMOTE' already has multiple push URLs:
$EXISTING"
fi
echo "Mirroring $SALSA_URL to $GITLAB_URL and $GITHUB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$SALSA_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITLAB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITHUB_URL"
# Push the branch you have checked out first, and never with --force: the
# first branch a GitLab or GitHub project receives becomes its default branch,
# the one you then protect
DEFAULT_BRANCH=$(git symbolic-ref --short HEAD) die "not on a branch, check out a branch first"
git push -u "$SALSA_REMOTE" "$DEFAULT_BRANCH"
git push "$SALSA_REMOTE" --all
git push "$SALSA_REMOTE" --tags
# The push above creates the GitLab and GitHub projects, but GitLab projects
# are private by default, so make the new mirror public with glab if installed.
if command -v glab > /dev/null
then
echo "Make the GitLab project public and describe it as a mirror:"
glab api -X PUT "projects/$ GITLAB_USER %2F$ PROJECT " \
-f visibility=public \
-f description="Mirror of $ SALSA_URL " \
-f notification_email=disabled
else
echo "Install glab to do the same automatically, or visit"
echo "https://gitlab.com/$ GITLAB_USER /$ PROJECT /edit to make the project public"
fi
echo "To undo, drop all push URLs and add back only the original one:"
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl"
echo " git remote set-url --add --push $ SALSA_REMOTE $ SALSA_URL "
If you prefer not to install glab, the last part simply reduces to visiting the GitLab project settings once and ticking the project public, and it is also worth setting a description like Mirror of https://salsa.debian.org/debian/debcraft and disabling notification email there, as nobody wants a mirror sending out a flood of emails every time you push.
There are a few caveats worth knowing about before mirroring everything everywhere. Every push to a remote with multiple push URLs writes to all of them, so a project with continuous integration configured on three forges will run its CI three times and cost you roughly three times as much. If one of the push URLs fails, say because the host is momentarily unreachable, git prints the error and stops there, leaving the remotes listed after the failing one without the new commits. Simply re-run the push once the host is reachable again; the remotes that did get the commits will just respond with Everything up-to-date. And you should never force-push to only one of the remotes, as that leaves the mirrors permanently out of sync with your local repository.
Avoid pulling too much
Related to multiple remotes, you should also be aware that git allows one to configure which branches to pull. By default a git clone will track all branches from the remote and pull everything, which in a typical open source project will bring in a lot of temporary development branches in vain.
Thus it is better to run git clone with the --single-branch and --branch parameters to fetch only the branch that matters, which in most cases is the main branch. That can be accomplished with:
git clone --single-branch --branch main https://github.com/ottok/debcraft.git
git clone --single-branch --branch main https://github.com/ottok/debcraft.git
If a repository was already cloned, it can be configured to track only one branch with:
For additional details about any of the tips in this post I recommend reading the git man pages.
Now go and spread your code around
Distributing your code over multiple code forges costs you almost nothing in configuration, and in return you get availability, a choice of host for your collaborators, and freedom to migrate without a big bang. Add the push URLs to the remotes you already have, push once, and you are done.
The rest of the git fundamentals are covered in my other posts. If your commits are not yet polished, start with what makes a good git commit message and git commit messages by example. If you work on Debian packaging, Git-based collaboration in Debian explains why the Salsa merge request workflow is built the way it is, and Salsa merge request best practices covers the review process itself.
Introduction
After bazel-bootstrap 7.7.1 was landed into Debian unstable,
I'm working on packaging newer Mozc (Most famous Japanese input method editor) with Bazel.
Here is the blog entry initial efforts to build Mozc with Bazel at that time.
kenhys.hatenablog.jp
Then, I've shared implementing PoC dh-bazel experiment.
See why dh-bazel is needed, and prototype about dh-bazel.
kenhys.hatenablog.jp
As a Bazel beginner, want to know what should pass to Bazel, what should not pass to Bazel.
What is improved in recent dh-bazel?
In the previous versions of dh-bazel, it supports only basic features
to build with Bazel as a thin wrapper.
Now, with recent changes, it supports the following environment variables to resolve required system libraries in dynamically.
salsa.debian.org
DH_BAZEL_OVERRIDE_MODULE
It search the specified modules from bundled dummy modules in dh-bazel.
It accept ',' separated paramesters. (e.g. DH_BAZEL_OVERRIDE_MODULE=zlib,zstd)
dh-bazel bundles abseil-cpp, apple_support, buildozer, lz4, openssl, protobuf, rules_android_ndk, rules_apple, rules_swift, zlib and zstd as
dummy modules to linking system libraries.
Now you can use it in debian/rules like this:
It is impossible to cover all of system libraries in Debian, so in that case, please consider to use the following DH_BAZEL_PKGCONF_MODULE.
DH_BAZEL_PKGCONF_MODULE
It search the specified modules with pkgconf. It is useful when there is no bundled modules in dh-bazel if you want.
It accept ',' separated paramesters. (e.g. DH_BAZEL_PKGCONF_MODULE=gtk4-x11,gtk4-unix-print).
If DH_BAZEL_PKGCONF_MODULE could not match, then dh-bazel fallback to dig into Build-Depends: field
in debian/control. Note that if it is not deterministic (e.g. -dev package provides multiple .pc files)
dh-bazel gives up fallback with pkgconf.
Now you can use it in debian/rules like this:
Conclusion
dh-bazel is still in very early stage prototype, but it resolves some sort of packaging glitches a bit by bit.
I hope that it will help package maintainer using Bazel.
(dh-bazel is not uploaded into debian/unstable yet, so stay tuned!)
As a graduate of a liberal arts university, I wound up, unsurprisingly, taking a
lot of classes in every possible academic discipline. Thinking back to the
person that I was going into university, I don t think I would have chosen to
take them after all, my degree was in the sciences; I d have been stoked to
do nothing more than wall-to-wall computer science until I ran out of
coursework and filled the rest of my hours with research and independent
studies with my professors (which, I guess I did actually do, just not as
much as I would have otherwise).
Even this blog s CSS theme (now 18 years old and starting to look it) was
something I wrote after receiving and repeatedly re-reading a tattered
third-hand copy of The Laws of Thought by George Boole. It was gifted to me
by a college friend majoring in philosophy while I was crashing at his rental
on the beach. He gave it to me because he knew I liked that shit and, while
it was definitely computer science in nature, I would not have read it
otherwise. I have vivid memories of sleeping on his la-z-boy surrounded by
towers of books he was working through. He went on to do incredible work,
getting his PhD, doing research, brilliant writing his death in 2023 has
robbed humanity of more time with him. The Laws of Thought sits behind me in
my office, and is one of my most valued possessions.
My friends mean more to me than I could ever express, and I definitely don t
show it enough. Each of my classes made me a more thoughtful person. My
professors made me a better, more well-rounded person and a person who
attempts to live up to the oft repeated credo of being men and women for
others . I did the best I could, even though I was never a particularly good
student. Without liberal arts, I don t think I would, dispositionally, have
been capable of pushing myself to this day to continue to learn on nights
and weekends, for no reason other than wanting to learn. I refuse to stop
expanding my perspective, and do my best to approach new problems with as much
humility and curiosity as I can muster.
Every few days for the last year or so, I have been thinking back to a reading
assignment from my junior year that, at the time, I thought was a borderline
throwaway filler assignment for the class. The reading is an essay from 1947 by
the french existentialist philosopher and libertarian marxist, Jean-Paul Sartre
(as some of the more erudite may have now already worked out given the blog s
title), Why Write? . I re-read it this week. It is not filler. I remember this
essay better than some of the classwork I considered more important at the
time.
Why Write?
Why Write? starts off by describing the ways in which writing trying to
communicate your thoughts, opinions, or feelings to others is an act of
projection. The writer will only ever draw from a place of their own
subjectivity writing is taking your person and putting it on display.
You re choosing what words to use, how to use them, pulling from knowledge
you ve accumulated (based on how you ve chosen to spend your time). Spending
any amount of your finite existence in order to convey a thought is, itself,
announcing to the world that you believe it to be a thought worth sharing.
Meanwhile, the act of reading is not simply turning letters into words.
Reading is engaging with the work, and understanding the work in a process
that looks more like what Sartre terms re-invention or discovery the
literary object, though realized through language, is never given in
language . It is not enough to read the words in a book end-to-end; you must,
as a reader, actively engage with the work to understand what is being
communicated by those words. Reading is to take the words on the page, and
exceed the mere words through what he calls directed creation . The reader
is re-inventing / discovering the writer s thoughts by following their
landmarks in the void .
And this all makes pretty good sense to me I know if someone is being
sarcastic because I know the writer; I have read their words and have read
into their words, allowing myself to be directed by the writer into
discovering the thought they have left me in their work. I understand that
their words are humorous, not irate. Knowing who the author of a work is
can completely change the point of a sentence. The fact I m writing about
Sartre at all, or that this very writing has been constrained for presentation
in this blog s CSS is only happening because of who I am, what things I ve
experienced in life, and what has made me, me. Sartre argues that this dyadic
coupling between writer and writing means that I, as a writer, can never truly
read my own writing. I will never be capable of reading my own work and
getting something out of it the work is already an extension of my own
person. I can discover no thought in my own works.
Of course, Sartre, as an existentialist, is honor bound to go one step further
here the reader, by participating in reading a work, is asserting their
own freedom. Every time a writer writes [ ] the writer appeals to the
reader s freedom to collaborate in the production of [their] work . The
creative act may only be complete if both writer and reader have recognized one
another and made a choice to do so. No one is compelled to complete the
creative act. The things you read matter. Reading something fundamentally
alters you as a person you can not simply un-read a thought. Everything you
read changes your universe. I have a hunch Sartre would find things like the
(original early 2010s era) tl;dr reply to obviously terrible writing
absolutely hilarious as a reader s expression of freedom (not to be confused
with modern 2020s era usage as shorthand for summary section ). Using your
freedom to choose to complete the writer s creative act (or not) is inherently
asserting your humanity.
I m a bit fuzzy on the specifics of this quote, but I think it was
sj who told me at one year s
Mystery Hunt that A puzzle is a contract between
puzzle author and puzzle solver, the author promises that the puzzle is
solvable if you re clever enough . The puzzle author and puzzle solver are
engaging in a collaborative production of the work that is only possible by
recognizing one another. Similarly, Sartre whatever connections [the
reader] may establish among the different parts of the book among the chapters
or the words [the reader] has a guarantee, namely, that they have been
expressly willed .
Wall Drawing #123
It follows, Sartre argues, that the act of writing, as a means to convey a
thought to a reader, may only be completed through the act of reading.
Reading can only be done by others; writing, therefore, only exists to be
read it can serve no other purpose. Writing and reading are two halves of
the same creative, collaborative act, only satisfied when both writer and
reader acknowledge one another. Combined, writing and reading are the act of
recognizing one another s humanity, thoughts, experiences, consciousness the
act of co-creation of thought is the critical aspect of writing and reading
reconstructing the author s perspective, thought, intent, point. The
co-creation is the point of both reading and writing.
Writing without anyone to read the work leaves the writer s bid for
co-creation unsatisfied and humanity unrecognized. No thought has been conveyed
to any reader, no one has understood the reason behind the landmarks in the
void you ve carefully placed, leaving you to either [ ] put down [your] pen
or despair . Writing is an appeal to the reader s freedom, and the nature of
that freedom is the writer can not control it never being understood is
always a possibility any time anyone sets out to write.
Nearly 12 years ago, I attempted to read the text on the git manpage
generator for the first time
I remember the exact feeling of my brain going into git manpage parsing
mode where every word was dragging git plumbing megaliths through the sand
from their far-flung homes. I can still feel the surface of my desk as I
instinctually started to trace out logical connections between git internals
referenced as I read along. It took me a good 20 seconds to realize what I was
reading made no sense. This website broke Sartre s writer-reader agreement I
was attempting to read this website, but there was nothing there. You can
mechanically read the git-man-page-generator, but you can never read it
it is not possible to read it. It was funny unsettling. Each grasp at a
real-looking image in front of me misses, my hand coming back empty. I
couldn t get enough of it. I must have tried to read a dozen generations in a
row. I had never had something quite that broken pass that far into my
consciousness before. Although I kept trying, it never did feel quite the same
as that first time; I think fundamentally, I couldn t forget that it was
random.
While never having anyone read your writing drives them to put down your
pen or despair , Sartre never had to contend with the opposite problem being
asked to read that which was not written. Reading that which was not
written does not merely leave someone unrecognized when a reader makes a
choice; instead, it is an inherent violation of the contract between reader
and writer. Not only is no humanity being recognized through attempting to
read words which were not written, but the reader, not the writer,
is the one who bears the burden of unexpected apophenia. The reader,
while engaging in co-creation, must now contort themselves to uphold their end
of the Sartrean agreement, scrutinizing text to ascribe consciousness,
thought and intent only to come back up with pareidolia-fueled echos of
one s own self and disfigured half-thoughts of others. The reality is, the
modern reader must now mechanically read most written work they come
across, scrutinizing text for any signs of thought before truly attempting
to read the work. Failing to do this correctly changes our person
each time it happens, a small part of us is irrevocably altered. Choosing to
engage in this ballet because you wish to do so is one thing this is
your right and freedom as a reader but passing words which were not
written as your own in an attempt to wrestle a reader s freedom of
co-creation from them is another entirely.
If you didn t write it, I don t want to read it (dw;dr). Send me your prompt instead.
It is said that those who cannot remember the past are condemned to
repeat it, a quote often attributed to the American philosopher George
Santayana. And to remember the past, records must be maintained and
kept accessible to learn from. With this in mind, it is no wonder
that archivists worldwide consider it crucial to ensure the archival
records are both complete and accurate, with a constant frustration
caused by the knowledge that the archives are neither due to
challenges in tracking down and collecting what should be
archived.
The last few days, I decided to simplify the collection of Messages
from the Mattermost chat
service, used by a few of the organisations I am involved in, to try
to improve the situation slightly. To achieve this, I had written a
dedicated extraction tool. Partly to see how hard it would be, and
partly to ensure that if one of the services were shut down or
replaced, everything in it would not be lost. Of course it is in the
nature of the use of instant messages that most of them are not fit
for permanent storage, at least not according to Norwegian law, where
there is a threshold "worthy of the archive" (arkivverdig) that should
be met, and messages like "should we go to lunch now" are below the
bar and should be filtered out. The tool can not help with this
filtering, and that will have to be done using other means after
collection.
The tools I had my bullshit generator write, under strict
supervision and many iterations, will extract every message visible to
the user whose credentials are used to log into Mattermost, and write
it out in Noark 5 extraction XML format and visualize the result. I
created the visualiser mostly to quickly be able to debug the
extracted XML, but also to make life easier for anyone interested in
testing out the tool set.
The extractor mattermost-noark5extract create a hierarchy
with arkiv/arkivdel for the Mattermost server, and then individual
mappe for each channel and direct message chat, a registrering for
each message thread, dokumentbeskrivelse for every message in the
tread, and one or more dokumentobjekt for each message and their
attachments/images. So far it is only tested on one Mattermost
installation, where around 23,000 messages is extracted in 61 seconds
and produce 295M with approximately 600 attachments and around 675,000
lines of XML in arkivstruktur.xml. You pass it the URL of the
service, a username and password, a directory path where to store the
collection and an optional channel name substring to limit the
collection to only a subset of the messages available.
The mattermost-noark5extract-browser viewer can load this
collection and visualize it similarly to Mattermost's web interface,
with the list of channels and direct messages dialogues on the left,
messages chronologically in the centre and a selected thread displayed
on the right.
I know Mattermost provide several login options. I've only had the
one used on my test server implemented so far, and know the extract
tool will have to be extended a bit for it to handle servers using one
of these options.
If you would like to examine the new toolkit,
mattermost-noark5extract
is available from gitlab. I wanted to put it on codeberg, but as
the recent rule change forbid code mostly written by a bullshit
generator there, it was not really an option. Note that some years
ago I wrote a similar tool to extract material from
the request
tracker system. The source for
request-tracker-noark5extract
is also available from codeberg. I would love to hear from you if
you test the tools.
As usual, if you use Bitcoin and wish to support my activities,
please send donations to
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
Introduction
After bazel-bootstrap 7.7.1 was landed into Debian unstable,
I'm working on packaging newer Mozc (Most famous Japanese input method editor) with Bazel.
Here is the blog entry initial efforts to build Mozc with Bazel at that time.
kenhys.hatenablog.jp
Why dh-bazel?
After that, newer Mozc packages are uploaded into experimental and moved to
testing phase on experimental now.
mozc - Debian Package Tracker
When started packaging efforts for newer Mozc with Bazel, I'm a newbie to do it.
Now I've got a knowledge to do it a bit, I want to know best practice on packaging X on Debian with Bazel furturmore.
Usually there are dh-X for buildsystem X on Debian, but it's not true for Bazel as far as I know.
This is why I had started to write dh-bazel.
I've wrote initial dh-bazel prototype and post a mail to debian-bazel ML.
lists.debian.org
dh-bazel supports the following way:
Conclusion
dh-bazel is in very early stage prototype, but I have succeeded to build newer Mozc with some modifications to Mozc debian/rules on experimental with dh-bazel locally!
dh-bazel is a thin wrapper for Bazel, so it does not reduce packaging glitches dramatically, but it helps some sort of packaging tasks IMHO.
There are some achievement with dh-bazel
It helps to do similar building way like other debhelper buildsystem
It sets comprehensive Bazel startup options
It sets comprehensive Bazel command options
If you want to set extra flags, just specify it as additional user options (--override_module=, and so on)
Note that it only simplify dh_auto_build stage, so you must manually install artifacts
with .install or something correctly.
I hope that it will help package maintainer using Bazel in the future.
(dh-bazel is not uploaded into debian/unstable yet, so stay tuned!)
Welcome to the August 2026 report from the Reproducible Builds project!
In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not.
In this month s report, we cover:
New updated SBOM specification from CISA.gov
CISA, the Cybersecurity and Infrastructure Security Agency of the U.S. government published some the joint guidance entitled Minimum Elements for a Software Bill of Materials (SBOM), which updates and supersedes the baseline 2021 version covered in previous editions of these reports.
Whilst the PDF is worth skimming, the interesting changes include that the specification now mandates standard cryptographic hashes: unlike earlier standards that allowed hash omission or manifest-only parsing, hashes must be computed from the output. This is is important for reproducible builds, as it ensures the recording of the metadata required to demonstrate the shipped software matches the build output precisely where applicable. In addition, where the top-level only dependency limitation that was present in the 2021 version has been removed in favour of complete coverage with no minimum depth. That is, SBOMs are expected to reflect all linked libraries, vendored dependencies and other build-time inclusions.
The basic idea behind bootstrappable builds is to create a system that can be built without relying on pre-built artifacts. Can we go from zero to the modern day without having to just assume the existence of these already-built-for-us artifacts? The classic recipe for yogurt requires some yogurt to start the process, which is like how we normally build a C compiler today we start with an existing C compiler binary. You might think about making sourdough bread with your grandmother s starter brought over from the old country; we re basically making C compilers with Dennis Ritchie s starter carried over from Bell Labs .
The reason I like the idea of making reproducible builds work is that I think it can be done in such a way as to not require any work on the part of the producer of a distribution (which is a technical term for sdists or wheels, i.e., the people who upload stuff to PyPI), and thus make reproducible builds very low-friction for people to opt into supporting. []
Brett goes on to outline What s missing from the specs and how reproducibility might be visible on PyPI to consumers:
Assuming all of this comes to pass and we record the where the source code is that went into a distribution and the software used to make the distribution, how do we make it useful to people? Does every person who cares about having a secure supply chain have to rebuild everything they use themselves? Is there some way for even people who don t care about this stuff to benefit? []
Distribution work
In Debian this month, 23 reviews of Debian packages were added, 28 were updated and 27 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated as well, such as the addition of a new toolchain issue related to python-traitlets [], and the note for an existing issue related to texi2html was updated as well [].
Lastly, Bernhard M. Wiedemann posted another openSUSEmonthly update for their reproducibility work there.
Unreproducible builds under EROFS filesystem fixed
Martin Pitt reported on Fosstodon that they had identified an issue where the mkfs ( make filesystem ) command for the EROFS (Enhanced Read-Only File System) subsystem of the Linux kernel did not have sorted extended file attributes, leading to reproducible builds.
Thankfully, Martin also reported that they had fixed this in a commit to the kernel which order[s] each inode s xattrs by name so that images stay reproducible . []
Tool and documentation development
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions 327, 328 and 329 to Debian. In particular, he ensured that diffoscope did not require python3-guestfs in the autopkgtests on 32-bit architectures in order to fix Debian bug (#1144372) []. Colin Watson made an additional change, handling a potentially missing openssh-client package when running the autopkgtests [], and Jochen Sprickerhof made a similar change to cope with missing cpio and qemu-img functionality [] whilst also updating the XML comparator to be considered when comparing SVG images [].
Yet again, there were a number of improvements made to our website this month as well. For example:
Chris Lamb added draft for a Gothenburg summit-related news article. [][]
Lastly, a large number of commits were pushed comprising an interview with Reproducible Builds developer Jochen Sprickerhof to be published within the next week. [][][][][][][]
Six new scholarly papers
Jens Dietrich, Spencer Sun, Tim W. White and Behnaz Hassanshahi (the result of a collaboration between Victoria University of Wellington and Oracle Australia published a paper this month entitled No Snake Oil: Verifying Python Package Builds. Drawing on the metaphor of snake oil , that is, a fake or ineffective medicine or solution sold with exaggerated claims of curing or fixing everything, the authors write that
Two tools that are designed to automate [PyPI] rebuilds and run them at scale are macaron and oss-rebuild. We study 12,180 popular releases from PyPI and find that the byte-for-byte equivalence rate is generally low. We analyse the reasons why they produce different wheels, and find that equivalence between the original and rebuilt wheels can often still be established, preserving most of the guarantees users expect from rebuildable releases. We present and evaluate daleq4py, a tool to establish the equivalence of Python wheels through the kernel of a normalisation function that is based on provenance-preserving datalog rules. Experimental results show that daleq4py substantially expands the set of rebuilds that can be accepted as equivalent.
pose a persistent threat to opensource software. Known attempts to inject such backdoors into widely used projects through malicious commits, tampered release packages, or compromised third-party dependencies, were stopped only by luck and manual review. Existing Continuous Integration (CI) pipelines cannot detect these attacks, and downstream binary analysis tools require substantial manual effort. In this work, we present Lily, an automated approach that strengthens open-source development and release processes against backdoor injection. Lily integrates a backdoor detection mechanism into (1) CI pipelines to block malicious commits, and (2) release vetting workflows to prevent tampered releases or compromised dependencies from entering large ecosystems, such as Linux distributions.
The full PDF can be read online.
Ranindya Paramitha and Laurie Williams of North Carolina State University along with Christian K stner of Carnegie Mellon University published a paper this month with the title of The Software Supply Chain as a Market for Lemons: A Multivocal Review of Trust Signal Collapse. (A lemon in American English, is a vehicle that turns out to have several manufacturing defects .) Their abstract is as follows:
Practitioners evaluating open-source dependencies rely on cheap trust signals, e.g., stars, download counts, and contributor activity, as substitutes for direct code inspection, assuming those signals reflect genuine trustworthiness. Prior work has documented individual signal gaming, but the landscape of collapses across all dependency-adoption signals, as well as the ecosystem s response, remains unexplored. The goal of this study is to aid software practitioners in understanding the reliability of dependency adoption trust signals, such as download counts and contributor activity, by conducting a multivocal review of 252 Google Search sources and 870 Reddit threads.
Worryingly, after their review, the authors conclude that cheap trust signals collapse under three simultaneous forces: adversarial manipulation, gaming techniques indistinguishable from legitimate behavior, and non-adversarial AI-driven inflation.
The full PDF of the paper is available online.
Julien Malka, Aman Sharma, Martin Monperrus, Stefano Zacchiroli and Th o Zimmermann published a paper this month on Trusting-Trust Attack against an Entire Linux Distribution through Binary Manipulation:
Ken Thompson s trusting-trust attack, in which a compromised compiler backdoors the programs it builds and reproduces the backdoor in subsequent rebuilds of itself, is widely regarded as a threat specific to compilers. We show that it is not. We construct a complete trusting-trust attack around GNU strip, an ordinary build utility that neither inspects nor generates source code, using only manipulations of finished ELF files.
Scarily, in the authors example, a single tampered strip in the binary seed implants a payload that propagates from one generation of strip to the next and survives into the final standard environment after the seed leaves the dependency closure [ ] without failures and backdoors .
A full PDF of the paper is available for download online.
Mehdi Keshanimm, Amirhossein Rahmati, Mohammad Hossein Aref and Abbas Heydarnoori published a paper that is currently under review at Emperical Software Engineering titled AROMA+: A Study of Factors Affecting Reproducible Builds in the Maven Ecosystem. (Maven is a/the build automation tool used for Java projects.) In their paper, the authors note that
[ ] reusing external software in a project presents a security risk when the source of the component is unknown or the consistency of a component cannot be verified. The SolarWinds attack serves as a popular example in which the injection of malicious code into a library affected thousands of customers and caused a loss of billions of dollars. [ ] Our research aims to support [reproducibility] efforts in the Maven ecosystem through automation. We investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA+, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of packages on Maven Central.
[A]rtifact verification requires more than deterministic builds: a verifier must also recover the source state, build environment, dependencies, and build instructions that produced the artifact. Decentralized-build ecosystems make this difficult because artifacts are produced through heterogeneous tools, maintainer-controlled workflows, and fragmented metadata. As a result, it remains unclear how often artifacts in these ecosystems can be independently verified. This paper studies artifact verifiability across four popular decentralized-build package ecosystems. We define an independent verifier model that relies only on registry-derivable metadata and an artifact comparison model with tiered equivalence levels. We implement these models in an Artifact Verification Pipeline and use it to measure artifact verifiability across the target ecosystems.
The authors conclude that beyond build determinism, verifiability is limited by missing source and build metadata, implicit release transformations, and unconventional build practices .
A PDF of their paper can be reviewed online.
Patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:
Two weeks ago, I released AppStream 1.2.0. This release contains a lot of great changes, but one of the most important ones concerns how media are being handled, and AppStream s default image export format.
AppStream is a Freedesktop metadata standard to describe software components. That can be anything from system services over fonts to console and graphical applications. AppStream metadata is supposed to give users enough information to decide whether they want to install a piece of software, to represent that piece of software, and to give the operating system enough information to decide whether a software component should be installed automatically and (to some extent) what capabilities and relations it has, to provide the user with sensible options.
Especially for the first two goals, and especially for GUI applications, AppStream supports icons and screenshots, which are used to showcase applications. Today, AppStream is used by all kinds of services, from Linux distributions over firmware updates to Flatpak and desktops directly. AppStream s original design however comes from the perspective of Linux distributions in 2011, where you may want to browse the software catalog offline, without delay, and without pinging an external server (which could be a privacy concern).
Therefore, a common way to deploy an AppStream-enabled software repository is to ship all icons of all applications in the repository to the user as part of the repository metadata download. AppStream does support remote icon downloads nowadays, and for a while I thought that this would become the default eventually. However, especially in today s world, having a bandwidth-saving, instantly responsive, privacy-protecting application browsing experience seems more important that ever.
PNG images are great!
The only format that AppStream supports for icons and screenshots (which are downloaded on-demand from your distributor s CDN) has always been exclusively PNG. PNG images are perfect for icons, because they compress well (especially for common icon shapes), are fast and simple to load, and can be loaded anywhere, by any toolkit or webbrowser. They also ensure we deliver faithful screenshot images, even though we may have scaled or re-rendered them. Still though, PNG images are less great for screenshots, as they are not very efficient, which puts strain on any CDN that has to deliver them, as well as on people s internet connections when browsing screenshots. Having smaller thumbnails alleviates that problem a little, but does not fully solve it.
But even for icons, PNG could be improved upon: In many cases, icons are re-downloaded with the repository metadata again and again, so having a large icon tarball adds up to the data transferred during metadata refreshes. AppStream also now supports large 128x128px icons, which nobody in 2012 expected we would need, adding even more data that will be re-downloaded. Saving some space here translates directly to lower bandwidth costs as well as faster downloads for users.
To improve PNG file sizes, the AppStream Compose library, which handles all image processing and metadata catalog composition, was running optipng on all generated PNG images. That does create smaller PNG images, but they were still relatively large compared to other image formats.
For a long time though, there was no alternative to PNG images for icons: There was no lossless image compression format that could give us the same quality as PNG images and that was also widely supported.
JPEG-XL vs PNG in AppStream
Since 2021 we have JPEG-XL (JXL), which offers a true lossless mode with often better compression than PNG. The issue was that JPEG-XL wasn t widely supported. Then, in 2025, the PDF Association selected JPEG-XL as the preferred image format for HDR images in PDFs, and now we are finally getting browser support and more ubiquitous availability of the format (you can try it right now in Firefox!).
For screenshots, using JXL s lossy mode, it has obvious and extreme size advantages over PNG, so supporting JXL or WebP for screenshot images was an obvious choice. If JXL would support the lossless case very well as well though, we could serve many use cases with the same exported image format, which is very attractive to me.
So, the obvious next question was whether it was worth the pain of switching the icon format, so I did some measurements on real icons. For that I used the AppStream component icon pool that Debian Unstable ships, which is almost 5000 application icons of various sizes, and converted them to PNG:
Icon size
Icons
PNG total
JXL total
Pool saved
PNG avg
JXL avg
Median saved
Mean saved
Worst
Best
Larger as JXL
48 48
1544
3.7 MiB
3.0 MiB
17.8%
2.4 KiB
2.0 KiB
17.9%
16.7%
-118.7%
60.0%
206
64 64
2018
7.0 MiB
5.8 MiB
17.8%
3.6 KiB
2.9 KiB
18.0%
15.8%
-112.7%
70.0%
279
128 128
1411
11.2 MiB
8.7 MiB
22.0%
8.1 KiB
6.3 KiB
20.1%
17.5%
-89.7%
61.0%
209
TOTAL
4973
21.9 MiB
17.5 MiB
19.9%
4.5 KiB
3.6 KiB
18.6%
16.6%
-118.7%
70.0%
694
PNG images saved with libpng at effort=4, compression=9, then optimized using optipng -o2, JXL images encoded using vips jxlsave lossless=1 effort=7 strip=1 via VIPS/libjxl.
As the table shows, using lossless JXL images over size-optimized PNG images (using optipng s default settings) provides a roughly 20% gain. This does not look like much, until you consider how often these files are downloaded: A 20% file size reduction may only save 1-2 MiB of disk space, but if they are downloaded over and over again by many clients, it will save a lot of bandwidth.
Interesting JXL encoding findings
As a sidequest, I was curious why some images were larger than their PNG counterparts when encoded with JXL, and what the ones that were significantly smaller were.
In short, the biggest size reductions for JXL existed on images that were already small as PNG, and contained large, flat color surfaces with hard edges and simple shapes. They were not very interesting, and much of JXL s wins come from accumulating smaller gains across all files, which compound the bigger icons get (especially at 128x128px, where JXL truly shines).
The events were JXL loses to PNG are more interesting: For example, it does quite poorly with pixel-art images that have a lot of repeating patterns. Those are encoded well by PNG, but less efficiently by JXL. Take for example Vonsh:
Icon of Vonsh, an SDL-based snake game, which PNG compresses better than JXL
My guess is that while PNG can exploit the repeating pixel patterns for compression, JXL s predicts surrounding pixels from its neighbours, which fails too often and makes it pay almost full entropy per pixel. In this single rare case, the PNG is at 5.4 KiB, while the JXL is almost 8 KiB in size.
Other cases I looked at were arguably buggy input data, where color channels were hidden under the alpha channel of the input image. PNG could probably again exploit repeats, while we were forcing JXL to encode pixels that were invisible in the final image. This is arguably a problem with the original input data. Currently, AppStream does not make any changes to icons at all, but in future we might add a filter that removes invisible colors from images to solve this pathological case (it was only two icons out of 5000 though, so it is not a high priority).
The third case I found where JXL loses to PNG were icons with checkerboard-like patterns:
Icon of x3270, an IBM 3270 Terminal Emulator
For those, PNG can likely again exploit the repeating patterns, while a checkerboard layout is pretty bad for left/top predictors like JXL s. However, in this case the size difference (and loss for JXL) is only 450 bytes, so even though JXL loses to PNG, it does so not by much.
JXL in AppStream
Given these findings, JPEG-XL is the default image format starting with AppStream 1.2.0. AppStream Compose will encode all images losslessly as JXL, while screenshots are encoded in lossy mode at Q=90 effort=7. Since the optipng step does not happen for JXL images, this comes at no speed penalty and is even a bit faster on modern x86_64 CPUs (where libjxl can use SIMD). PNG is still available, and Compose can be told to switch between the two formats.
Upsides of JXL in AppStream right now
If you use JXL in Compose or the recent release of appstream-generator, you will get much smaller images and, for screenshots, will benefit from other JPEG-XL features such as progressive decoding, providing a far nicer user experience. libAppStream has supported JXL icons since version 1.1.3, so your clients will need that version or a newer one, and all software centers will have to support loading JXL images (which all of them do, provided the right plugins are installed).
Downsides of switching to JXL too quickly
JXL is a very new format, so web browsers might not yet display it if you are serving webpages. Your clients may also have bugs in processing JXL images, as the format is still new . For example, switching on JXL in Debian sent KDE Discover into an infinite loop on startup while trying to load the icons (an issue which has been fixed, but clients will need that patch first before JXL is switched on).
This currently makes JXL enablement only possible when you know that your clients can support it. This is the case for me in Debian Unstable and Debian 14, which are using JXL images for a few weeks now, but not for any older releases. Platforms like Flatpak have it even harder, because they do know even less about their clients. So, even though it has big advantages, you may want to hold off on using JXL right away, and force PNG by setting the ImageFormat key to png in appstream-generator s configuration, or passing --image-format=png to appstreamcli compose.
It is also worth mentioning that JPEG-XL is much, much slower on systems that do not have SIMD instructions or for which the libjxl/jxl-rs library does not have them (such as apparently riscv64 right now). If this is a concern, you might not want to switch to JXL right away.
Media pipeline improvements
Besides the JXL default change, AppStream 1.2.0 also comes with a complete overhaul of its media processing pipeline. While libappstream, AppStream s main library, does not do any media processing and comes with very minimal dependencies to be embedded in client applications and used on servers, the same can not be said about libappstream-compose, AppStream s library to build metadata generating applications (the server-side part, usually).
The compose library has to render fonts into font specimen cards, inspect translation files, render SVG images, decode all kinds of raster images, inspect video files, etc. Especially the fonts, and the fact that fonts can appear in SVG images, has caused issues in the past, as libappstream-compose is a heavily threaded library and most font libraries can only work from a single thread. This forced the library to essentially go into single-thread mode anytime anything that could touch a font was being processed.
AppStream also originally was created for a safe world where applications were vetted by the distributors before their metadata was processed. This is increasingly not the case, so it made sense to put at least a few guardrails on the most complex part of the pipeline: The media processing. As part of the change, media processing was split out into a separate worker process. This solved two problems at once: Font handling was isolated in a single-threaded binary if we wanted to handle fonts in parallel, we could simply spawn more workers. And, being in a separate process, the media processing could now be sandboxed.
As part of the multiprocess changes, Compose also switched from using GdkPixbuf to VIPS for image processing. The latter allows for much more fine-grained control over the image output and encoding, and comes with a lot of well-maintained filters and operations, which made it possible to eliminate a fair chunk of AppStream s hand-rolled image processing operations. As part of this transition, we unfortunately lost the ability to read XPM images, which dropped about 20-30 applications from the pool at Debian. But in the name of security, this is a sensible choice, especially since most XPM icons were very small and low-resolution, and applications using them could benefit from adding a high-quality PNG icon anyway. With VIPS, we also now restrict the amount of image formats we can load to a sensible set, so extremely niche or unexpected formats will be outright rejected (this includes sane-but-unusual formats for screenshots and icons, such as TIFF images).
The Compose library, with all of these changes, will now just request high-level operations (e.g. render a font card for this font to a JXL image ) from the worker, and provide it with input data in sealed memfds and output locations as FDs as well. On Linux systems, the worker will use Landlock if available, to block all write access to the filesystem, deny device access and deny TCP and UDP as well. The sandbox can certainly be tightened a fair bit in future, but this was a good and safe start to gain some experience with it without having things break too easily, given the many places Compose is used in (also, Landlock s API is surprisingly nice to use, so it was easier than I thought to add in this early version).
With all of these changes, the libappstream-compose library is now also officially marked API-stable, so you should be able to rely on it in future to build new things (its API has barely changed in the past, and now with the new media API and defaults change in place, it was time to declare it stable).
I want to see / try this!
Currently, the easiest way to have a look at the new data is to check out Debian Unstable. If you have a JXL-enabled browser, you can also see the icons in AppStream Generator s HTML pages for Debian Sid. If you are using appstream-generator for your distribution, you will also get much more pleasant statistics and HTML pages, as well as fully deterministic media output and a whole bunch of security updates, so, update to its recent 1.0 release.
Please keep in mind that if you switch to JXL, the client tools receiving the image data have to support it. Support varies depending on the Linux distribution, so, test it first and switch the default back to PNG in case you encounter any issues.
What s next?
With so many features and changes landed, the next changes in AppStream will focus on improving what already exists and fixing any issues (there will be more blogposts about the other features 1.2.x delivers!). Testing with the entire Debian archive as data source makes me fairly confident though that there will not be many problems. In the longer term, tightening the media processing sandbox will also be something we might want to do, e.g. by hiding parts of the filesystem tree or filtering syscalls.
For JPEG-XL, one obvious question is Will you add support for it to the Freedesktop icon-theme specification as supported format alongside PNG, SVG(Z), and XPM? . For on-disk icon repositories, JXL s space-savings are less compelling, and it being HDR-capable is also not necessarily a killer feature (PNG can go a long way!). However, JPEG-XL s ability to immediately decode larger images at reduced resolution without resampling could legitimately be very powerful here, as applications could ship a single large image and quickly decode it at 1/2, 1/4 or 1/8 the size for different purposes in their UI. JPEG-XL also supports spot-color extra channels, which applications could use as masks to recolor raster icons at render time. This could be incredibly nice to color symbolic icons on-the-fly without any SVG and CSS. JXL also provides richer metadata, which might be neat for (license/author) documentation. So, the answer here is: Maybe it makes sense to allow another format, but this will have to be discussed first, as it would force JXL into every toolkit and desktop, which is a much bigger ask than supporting it only in AppStream.
As always, let me know what you think and please report any issues or bugs directly against AppStream or AppStream Generator if you encounter problems that are with the tools, and not with a project s metadata.
This post is part of a series called "Pigeon". If this is the first post
you've found, it'd be worth reading the
intro post first and then looking over
all posts in the series.
The final step to all of this was to tie together all my PHY RF
code, Link layer parsers, and my
background with operating systems
to make this all feel like a normal thing my computer should be doing.
At the end of the day, I want my host system to know how to talk with a
pigeon daemon, so I don t have to reimplement basically everything else.
My ability to use normal tools like curl or ping6 is pretty important
here, so I need to reach for my old friend, the
TUN interface. The
TAP/TUN interface allows the kernel to route ethernet frames (TAP) or ip
packets (TUN) to a userspace program responsible for handling delivery and
reception avoiding the need for a kernelspace driver for something that
can be handled in userland.
wondering if doing this violates FCC Part 97 rules? I wrote up
notes on my test setup
for exactly this question, but the short answer is "no"!
I m nostranger to
playing with TAP/TUN, so this was pretty easy to snap together although this
time I avoided the whole ethernet proxying thing (to side-step lossy
translations, maintaining two sets of mac address tables, and handle proxying
NDP/ARP messages) it was kinda a bad idea last time so I just used TUN
and straight IP for now. While implementing this, I decided I d make a key
assertion about all pigeon networks namely, all pigeon IPv6 networks are a
/64 in size, no more, no less. The reason why I m doing this here is that,
since pigeond does still does need a MAC address for the pigeon layer 2
protocol, we can write our daemon to always use
SLAAC
to set the TUN IP address without any new information.
Which leads us to a bit of an aside, but I have a point, I swear. A few years
ago, my recreational RF adventures have lead me down a path where I decided to
engage with ARIN to solve (once and
for all) the massive headache I was running into with IPv6 numbering (really:
always renumbering) my multi-site radio processing networks. It s a lot
of work to keep running correctly, but it s solved a huge amount of
problems for me.
The only internal thing we really need to outline for this post is that, at
the highest level, my network (paultag.net) is split into an IP plan that
looks roughly like:
64 "sites" per region. A "site" is assigned to a physical or logical location.
/64
1024 subnets per site. A subnet used by directly attached devices.
For this exercise, I used IP space from paultag.net s experimental region
( region 8 ), named side.band (2602:810:6008::/48) to connect my RF lab
( site 1 - 2602:810:6008:400::/54), and my two pigeon-specific subnets,
subnet 0 (2602:810:6008:400::/64) and subnet 1 (2602:810:6008:401::/64)
to my wider network. The first subnet ( subnet 0 ) is a simple ethernet network
to enable my RF-only nodes to communicate with the side.band gateway. The
second subnet ( subnet 1 ) is an RF-only pigeon network local to my lab.
back to radios
With all that set up, I assigned my first two nodes their MAC addresses, and
set up the local RF only network segment. The nodes I brought online were the
following:
Callsign
IP
K3XEC/MN
2602:810:6008:401:8e1f:64ff:fe35:4001
K3XEC/TH
2602:810:6008:401:8e1f:64ff:fe35:4002
testing the pigeon network
And with that, I could begin to test that the host operating systems and RF
links could properly exchange data locally from SDR to SDR. We can use
ping6 to see if a plain-ole ICMPv6 ping round trips between hosts correctly:
$ ping6 2602:810:6008:401:8e1f:64ff:fe35:4001
PING 2602:810:6008:401:8e1f:64ff:fe35:4001 (2602:810:6008:401:8e1f:64ff:fe35:4001) 56 data bytes
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=1 ttl=64 time=426 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=2 ttl=64 time=397 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=3 ttl=64 time=419 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=4 ttl=64 time=418 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=5 ttl=64 time=436 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=6 ttl=64 time=391 ms
64 bytes from 2602:810:6008:401:8e1f:64ff:fe35:4001: icmp_seq=7 ttl=64 time=397 ms
And it does! Latency is horrid (and there s a bunch of tx artifacts that
cause issues for us) but both of those things are problems for later. Let s
see how it handles a TCP connection by firing off a quick cURL across the
Pigeon network:
$ curl http://[2602:810:6008:401:8e1f:64ff:fe35:4001]:8000/testing.txt
The rock dove (Columba livia), also known as the common pigeon or rock pigeon
(but see also Petrophassa), is a member of the bird family Columbidae (doves
and pigeons).
As expected, our remote end here running the server reports the correct peer
IP address, which is another indication (beyond the log messages and blinking
LEDs) that we re routing over our TUN interface.
Serving HTTP on 2602:810:6008:401:8e1f:64ff:fe35:4001 port 8000 (http://[2602:810:6008:401:8e1f:64ff:fe35:4001]:8000/) ...
2602:810:6008:401:8e1f:64ff:fe35:4002 - - [28/May/2026 12:53:21] "GET /testing.txt HTTP/1.1" 200 -
That worked? First shot! Nice! It s pretty slow and seems like we have a
lot of packet loss, but it does, however, beg the question can it nethack?
nethack!
Yes! It can nethack! No clickbait here. The way I went about this one is a bit
anit-cimatic I set up a nethack server (using inetd in this case) on one
of the hosts pigeon0 network interface, and hit that port over RF from
the other:
However, when playing it, it becomes very obvious (as you can likely see) that
there s a fair amount of packet loss (understandable) and probably some packet
collisions taking place.
iperf
Let s try and put a number to exactly how bad the bandwidth and packet loss
is by running iperf between the two pigeon hosts over rf:
$ iperf -c 2602:810:6008:401:8e1f:64ff:fe35:4002
------------------------------------------------------------
Client connecting to 2602:810:6008:401:8e1f:64ff:fe35:4002, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 1] local 2602:810:6008:401:: port 58248 connected with 2602:810:6008:401:8e1f:64ff:fe35:4002 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-20.2348 sec 76.8 KBytes 31.1 Kbits/sec
Shockingly, not nearly as bad as I thought it was going to be. Given I ve
spent exactly zero time making this operate to a level that I would call
acceptable, this is a very fucking solid start. I expect I could get that
number up if I spent a few weeks on it it s just not been a priority
at any point yet (and the first time I ve instrumented it, even!).
This ll be good enough to get started. Let s see what else we can pull
off here.
IP multicast to some rtl-sdrs
Back when I designed what I wanted Mode A to
look like, I intentionally picked a signal bandwidth that could be received
by an rtl-sdr so let s put that to use. It may go without saying,
but just to say it the rtl-sdr can not transmit, so this will be capable
of receiving pigeon frames but not sending any in reply.
However, this means I can use a bunch of low-cost computers (raspberry
pi-class), and low-cost SDRs (rtl-sdr) and still receive IP traffic from
transmitting pigeon network stations. This could be a lot of fun for things
like fountain coding a data
stream, or adapting multicast streaming protocols to work over RF links.
Anywho, I swapped my far end to an rtl-sdr (one config file change!), and
figured I d start with some (basic) multicast traffic, transmitting the time
once a second:
If I had more time to burn, I was planning on bridging
APRS traffic
to UDP multicast within a pigeon network subnet. However, since I m already 4
years late on this blog post, I figured this would be enough for now (and you
can imagine that fun project in this space if you so wish!)
I fired up pigeond again (except this time connected to an rtl-sdr),
and was pleasantly surprised to be greeted by some decoded traffic right
off the bat:
Looks great! tcpdump is showing multicast packets show up (as we assumed they
would), on the pigeon0 interface, on the machine connected to an rtl-sdr. Of
course, any replies will get sent to the
bit bucket, but it can definitely
decode things just fine! Very fucking cool.
Well right, ok! Let s go back to two rx/tx radios, and see what we can do with
our newfound network stack over ham radio frequencies let s try to do some
fun (and traditional!) ham radio things with it!
Winlink
Winlink is a ham radio mail relay system for ham radio
operators to send, receive or relay mail over the internet, or RF (usually HF
or VHF/2M). Winlink relays are accessible via whatever transport you can
find most commonly telnet (using the internet), ax.25 (usually 2m VHF)
or VARA HF (unsurprisingly, on HF). I use pat as my
Winlink client it s written in Go, doesn t require windows, and is just
generally nice to work with.
Let s try the easy thing first let s connect by proxying the Winlink server
into the pigeon network using socat (lightly edited to remove date/times)
$ pat connect pigeon
Connecting to WL2K (telnet)...
Connected to [2602:810:6008:401:8e1f:64ff:fe35:4002]:8772 (tcp)
[WL2K-5.0-B2FWIHJM$]
;PQ: 54509561
CMS>
>FC EM OLU6BP5HKMG2 240 205 0
>F> 95
FS Y
Remote accepted OLU6BP5HKMG2
Transmitting [Hello, World] [offset 0]
Hello, World: 100%
FF
>FQ
Disconnected.
$
Lo and behold, shortly after, I got this delightful message to my
email address, relayed in from WINLINK:
The only shame is I won t be able to check in to a winlink
wednesday using this scheme unless I further
proxy this message over AX.25 instead of relaying to Winlink s servers over
telnet (which, to be fair, is definitely also possible I just got lazy when
I glued this one together see note above about being 4 years late on this
post).
But, you know, connecting to a host that is using socat to proxy a
connection to an internet resource is interesting but you know what, fuck it
hang on, dear reader let s bang a hard left turn and just ship this thing
hard and directly connect it to the internet. Let s take our dinky,
home-built PHY and Layer 2 and see if we can wire it directly into the internet
something that, every time I go to think about it, reminds me of
Tim FitzHigham and his crapper.
Crossing the english channel in a bathtub
Ok, ok. I decided to bury the lede a bit here I didn t mention that the
side.band network is currently
BGP announced. Although we
haven t used it this does mean that we re most of the way to sending packets
to the wider internet, and we should be able to just fix a few routing
tables, and see packets begin to flow.
After tweaking the local routing tables (and restarting pigeond for good
measure), I decided to test my newfound connectivity by pinging something over
our new network transport.
ping github
Why don t we start with the world s premier software engineering platform,
operated by one of the largest companies in the world, GitHub! After all,
they have an all knowing (and, apparently, arguably sentiant?) AI on hand to
instantly and automatically fix any stray
reliability issues in the
background, so we should definitely see replies right off the bat:
$ ping6 github.com
ping6: github.com: Address family for hostname not supported
Wait, oh no that can t be right?
After all, it s 2026, and both
Google
and CloudFlare
(in North America) are reporting
over half of all traffic they see is IPv6 and GitHub still doesn t support
IPv6? Definitely not, this is for sure a bug with my code or network.
lets ping something that supports ipv6 instead
That being said, just for completeness sake, since that error is also given
when there s no IPv6 DNS record, let s go ahead and double check with
Hurricane Electric too, you know, just to be sure.
$ ping6 he.net
PING he.net (2001:470:0:503::2)56 data bytes
64 bytes from he.net (2001:470:0:503::2): icmp_seq=1 ttl=53 time=514 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=2 ttl=53 time=230 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=3 ttl=53 time=248 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=4 ttl=53 time=246 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=5 ttl=53 time=265 ms
64 bytes from he.net (2001:470:0:503::2): icmp_seq=6 ttl=53 time=240 ms
Well, shit. Right, OK, i ll be damned. 18 years in and GitHub still can t crack
that nut.
cURL works!
Right, anyway, yes, back on track good news! Our uplink is up and routing,
and wait, holy shit! Check it out! pigeon is
exchanging packets with the internet and no one is any the wiser!
Literlaly amazing. Let s try a cURL across the internet now (although no
TLS allowed, so, http only for now):
IRC works, too
Sweeeeet. That all works! Forget HTTP, let s do some other 90 s era
stuff, it s high-time to log into IRC with a quick /connect -notls, and see
what s going on in the #debian-hams channel pleased that I got online
fairly quickly, and was able to even talk to myself!
THE GOPHERSPACE
Naturally, let s keep this train of nostalga running, and give
the 2026 gopherspace a
shot.
I know the kind folks over at tilde.town (hello,
townies!) have a robust gopherspace, so let s give it a dial! Let s try and see
if we can load vilmibm s
slug over gopher:
Yes! I forgot to make this one a video, so no gif. I did wind up having a bit
if trouble with a few gopher clients and IPv6 support I may send some
patches if I can find the time.
So, what s next?
Alright, that s it. I have a few more fun ideas but they re going to have to
wait for another day. Carrying IP is fun and all but kinda not the point behind
pigeon, after all. Rather than trying to make this into a thing , I m planning
on exploring the loose ends first different types of modulation schemes
(like QAM-NUC), implementing LDPC error correction and some layer 2 logic
into the pigeond (like switching traffic, and gain control). I also plan
on spending some time with my (currently, very basic) simulator to better dial
in tradeoffs throughout the stack.
Since, structurally, pigeon is something I feel like I can work with, I m
hoping i ll be able to find the time for some (much smaller!) followup posts
without it taking 4 years this time. If I do, they ll show up under the
pigeon tag and I ll be sure to update this post with
a link below (and the intro post).
I m hoping that this series (which was supposed to be one post) was helpful
to someone out there if it was, feel free to reach out and let me know!
My Debian contributions this month were all sponsored by Freexian.
You can also support my work directly via Liberapay or GitHub Sponsors.
Personal note
This month, my Dad unexpectedly passed away after a short illness. As a result I obviously got less work done than usual, and I still have a lot to take care of (since I m the executor of his will, as well as helping with funeral arrangements) while grieving and generally having less focus and energy. Having routine work to do is one of the ways I cope with this sort of thing, but all the same, I hope people will bear with me and maybe remind me if I seem to be dropping the ball on something you especially need.
LLM vote
[Content note: strong opinions.]
I voted in General Resolution: LLM usage in Debian. My vote was pretty much the opposite of what ended up winning, so I m quite disappointed. My personal opinion is that LLMs are cognitive hazards to their users that impose ecological costs far out of proportion to their utility at a time when the world absolutely cannot afford them. When the impossible economics of the large commercial models are finally allowed to catch up with reality, I expect there to be significant macroeconomic consequences, and that people who have become dependent on them will have problems; and who knows what the copyright situation on their output really is. I m not convinced that local models are better enough on these axes to be worth the costs.
Debian s direct contribution to all that will be negligible on a global scale, and even the most radical proposals in the GR didn t expect that we could do much about upstreams that have gone all-in on LLMs. Even so, I d hoped that my fellow developers might be more willing to lean on our position in the free software ecosystem to make at least a moderately radical statement. Instead, we ve at best presented an undistinguished fence-sitting position to the world, and further entrenched the idea that humans can reliably do a good job of reviewing the output of tools that are designed to produce output plausible to humans. I certainly don t trust my own code review skills that far.
Since I ve never voluntarily used an LLM (not counting LLMs being foisted on me by things like search results, support chatbots, or incoming pull requests, regardless of whether I asked for them), and don t intend to for the foreseeable future, I doubt this will change much for me in terms of the way I work. The winning option is a very weak one that imposes no new requirements on developers, which means that it also does nothing to stop me continuing to reject LLM-generated material from Debian bug reports and merge requests in my areas of responsibility. I know this probably won t do much to satisfy people who have decided that Debian is slop now, but it s the best I can do.
OpenSSH
I finally landed the GSS-API key exchange package split in our OpenSSH packaging. Here s the NEWS entry:
I fixed a flaky autopkgtest.
I upgraded from 10.4p1 to 10.5p1, which was a good test of keeping openssh and the new openssh-gssapi source package in sync.
PuTTY
I upgraded from 0.84 to 0.85.
Python packaging
New upstream versions:
The version treadmill continues: we ve just finished dropping Python 3.13 as a supported version, so now we ve started working on enabling Python 3.15 as a supported version. Maximiliano Curia has been very helpfully driving this. I didn t get as much done here as I d have liked (see the top of this post), but I fixed a couple of packages:
I am a heavy user of sidenotes:1 they keep optional content next to the
text instead of sending the reader to the bottom of the page and back. Tufte
CSS renders them without JavaScript but only accepts inline content. CSS
anchor positioning, now supported by recent browsers,2 is an
elegant alternative. Sidenotes can hold several blocks, still without
JavaScript, and fall back below the paragraph referencing them on narrow
viewports and older browsers.
In 2023, Eric Meyer demonstrated this technique in Nuclear Anchored
Sidenotes. The main improvement over other solutions is that the notes can
sit anywhere in the HTML document. You can place them after the paragraph
referencing them, as regular block elements for text browsers, screen readers,
feed readers, and reader mode to render them properly:
Rendering in Lynx, a text browser
When the viewport is too narrow or the browser does not support CSS anchor
positioning, you can style them so the reader can skip them or glance at them
without losing their position in the text:
Rendering below the paragraph on a narrow viewport
Once the viewport is large enough, they appear in the margin, at the same
vertical position as the matching reference mark, unless they would collide with
a previous sidenote, as in the example below:3
Rendering in the margin on a large viewport
The gist of CSS anchoring is to position an element relative to another
element the anchor. For the sidenotes, the anchor is the reference mark. I use
the following markup, with a data attribute to specify the anchor name:
The matching note is an <aside> element carrying the same data attribute for
the anchor name. We put it after the paragraph holding the reference mark:
<asiderole="note"id="sidenote-YYY"data-anchor="--lf-sn-YYY"><sup>1</sup><p>A first paragraph.</p><p>A second paragraph.</p></aside>
On a narrow viewport or when the browser is too old for CSS anchoring, we style
the sidenote, which stays below its paragraph, with a muted color:
aside[role="note"]margin-block:1rlh;color:#444;
On a wide viewport and when the browser is recent enough, we move the sidenote
to the right margin:
attr() extracts the anchor name for the reference mark from the data-anchor
attribute. It returns a string, unless we specify a CSS unit or a type, like
here: the browser parses the data attribute as a custom
identifier, which anchor-name validates as a dashed
identifier, a custom identifier starting with two
dashes.4
The note itself is absolutely positioned past the right edge of the main block.
It selects the matching reference mark as its anchor with position-anchor set
to the value of the data-anchor attribute. Each note is also an anchor named
--lf-sidenote. We use it to keep the next note from colliding with this one.
The anchor() CSS function lets us position the note s top
edge relative to its anchor: anchor(top) aligns the top edge of the note with
the top edge of the reference mark. It can also take another anchor as a
parameter: anchor(--lf-sidenote bottom) would align the top edge of the note
with the bottom edge of the closest preceding anchor named --lf-sidenote so
the previous note.5 Like attr(), anchor() accepts a fallback
value as its second parameter and use it when the named anchor does not exist.
The top property handles three cases, illustrated in the following diagram:
The three cases for the vertical position of a note
The first note s top edge aligns with the top edge of its reference mark: as
there is no previous note, anchor(--lf-sidenote bottom, -1rlh) + 1rlh
resolves to 0 and max() returns anchor(top).
When the reference mark of a later note sits above the bottom of the previous
note, plus some vertical space, the note goes below the previous one to avoid
a collision. max() returns anchor(--lf-sidenote bottom) + 1rlh.
Otherwise, the note s top edge aligns with the reference mark s top edge, as
max() returns anchor(top).
Have a look at the complete stylesheet, which also adapts the
reference mark to the location of the note: a arrow when the note sits below
the paragraph, a arrow when it moves to the margin. Gwern s Sidenotes In
Web Design lists more implementations and their trade-offs.
Some bloggers aim to write a post in 30 minutes. I planned to publish
three web-related articles this weekend. Instead, I spent an inordinate amount
of time elsewhere: about 15 commits on the build system, a pull request to
update CSS highlighting for nested selectors in Pygments, and a
small correction to MDN s article on the anchor() CSS function. The
SVG illustration took a bit less than an hour and the article itself a handful
of hours. The attr() function came in after I thought inline style looks
ugly, isn t there a better way? But, hey, I still think this is worth it!
My PhD advisor told me this is unwise.
The first bits of anchor positioning are supported from Chrome 125 (May 2024), Firefox 147 (January 2026), and Safari 26 (September 2025).
Before Safari 26.5, sidenotes may collide due to a bug in how dependency
chains are handled. You can detect this situation with some JavaScript.
It is, however, not needed in the solution described here as we depend on a
more recent feature.
If you noticed the runt in the first note, I share your pain and lament that
Firefox does not implement text-wrap: pretty.
Typed attr() is supported from Chrome 133 (February 2025),
Firefox 155 (September 2026), and Safari 27 (September 2026). Check Una
Kravets article for details. To support more
browsers, you can inline the anchor name and the position anchor directly in
the HTML:
The exact rule for the target anchor element is more complex: if an
ancestor of [the note] satisfies the following conditions, return the
nearest such element to [the note]. Otherwise, return the last element in
tree order that satisfies the conditions. One of these conditions is that
[the candidate] is an acceptable anchor element for [the note], which
requires that [the candidate] is laid out strictly before [the note],
where the relevant clause is that [the candidate] is either not absolutely
positioned or occurs earlier in the flat tree order than [the note].
This August, I accomplished what I wanted for many years: I deleted the last cgo
dependency in Debian Code Search! This was made possible by Go s recently
introduced SIMD support, because now we can implement the TurboPFor integer
compression format as efficiently more efficiently, in fact, by using the
newer AVX512 instruction set! as the reference implementation.
Background: Why does DCS need a fast Integer Codec?
Debian Code Search (DCS) is a search engine that allows searching all the Open
Source source code within Debian, with either literal search expressions or
regular expression search queries.
A search engine uses an inverted index: a map from term to documents containing
the term. Each document is typically represented most efficiently by using an
id, so the index consists of many lists of document ids.
When searching, it is important to quickly decode these lists to answer the
search query. However, there is a point of diminishing returns where the
decoding speed, even though it can still be measurably improved quite a bit, no
longer influences the overall query duration.
From 2012 (its inception) to 2019, Debian Code Search used to use a small index
format, and queries were fast because the index was kept entirely in RAM. In
2019, I implemented the new index
format, which adds an
on-disk positional index. For literal queries (78.2% of DCS queries), querying
the positional index on disk is faster than querying the non-positional index in
RAM.
The efficient encoding of the TurboPFor format makes it possible to fit such an
index on a mid-sized Hetzner server, which I rent with two 1 TB SSD disks. The
optimized decoder of the C TurboPFor library is what made decoding fast at query
time.
If you want to dive deeper into the algorithm, see this blog post from February
2019:
Use a C library via cgo so gcc or clang compiles
SIMD code. Debian Code Search used to use the
powturbo/TurboPFor
C library via cgo for the last 7 years.
The C TurboPFor library has served us well, but Debian Code Search was always
intended to be a project using Go, so I would prefer it if I did not have any C
code in the project.
Go 1.26 (released in February 2026) introduced the simd/archsimd package:
Go 1.26 introduces a new experimental simd/archsimd
package, which can be enabled by setting
the environment variable GOEXPERIMENT=simd at build time. This package
provides access to architecture-specific SIMD operations. It is currently
available on the amd64 architecture and supports 128-bit, 256-bit, and
512-bit vector types, such as Int8x16 and Float64x8, with operations such
as Int8x16.Add. The API is not yet considered stable.
Go 1.26 Release Notes
For my 2019 TurboPFor analysis, I implemented goturbopfor, a native Go
teaching decoder (without any SIMD),
because I find Go code easier to follow than C code, especially optimized C
code. My implementation was intentionally not optimized so that the code was
easier to study.
The TurboPFor format/algorithm has a vector-optimized part: bitpacking comes in
a scalar variant (bitunpack32) and a vector variant
(bitunpack256v32),
where the vector variant is used for full blocks (256 values) and the scalar
variant is used for remainder blocks (< 256 values).
When Go 1.26 was released, I used Claude Code to explore whether my native Go
decoder s bitunpack256v32 function (for the vertical vector layout) could be
implemented using Go SIMD, and the answer was yes, it was possible and it was
faster than without SIMD, but not quite at the level of C TurboPFor. If you let
Claude Code try for long enough, it eventually finds enough optimizations (about
10) to match C performance.
I don t want to vibe-code Debian Code Search, though, so I figured I would find
some time to review the SIMD code at some point and see if I could implement
something similar myself.
Before I found enough time and motivation to complete said review, I discovered
that to not regress real-life query performance by more than 10 to 100
milliseconds (which seems acceptable), I don t actually need to add SIMD code to
my teaching decoder at all; it would be sufficient to reduce allocations in my
teaching decoder and specialize it per bit width.
Encouraged by the possibility of using the optimized native Go decoder in Debian
Code Search, I explored whether I could also implement a native Go encoder so
that I could get rid of the C TurboPFor dependency entirely. The answer is yes,
it is doable in a few days, and it isn t even that much slower: Go is at 76% of
C, see Debian/dcs commit
e920dc7.
The goal I set myself at that point was to see if I could learn enough SIMD to
optimize the native Go encoder such that its performance would match how DCS
uses C TurboPFor (via cgo).
Beating C TurboPFor was possible in 2-3 commits (SIMD and bit width
specialization). To my surprise, Claude Fable 5 pointed out that the encoder s
block scanning could be done more efficiently using a technique called
positional popcount, and that is another 2x speed-up!
To be clear: I am not saying the Go compiler beats C here. Certainly, the C
compiler can also produce fast AVX512 code and can be used to implement
positional popcount. When comparing apples to apples, i.e. backporting the
AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a
little slower at 1.4x C.
This spectacular result (much faster than what DCS had before) got me curious
how far I could push the decoder with SIMD after all. I ended up
matching/exceeding the cgo version here, too!
The rest of this article explains a few classes of optimizations I encountered
along the way.
Starting Point
When I wrote my goturbopfor teaching
decoder, I named its functions to
match the upstream C TurboPFor library, but now I want to get away from names
like p4ndec256v32 they make sense from the TurboPFor perspective, but for
Debian Code Search, we can use cleaner names.
Before writing any code, I audited how DCS uses integer compression /
decompression.
API design: BlockEncoder, BlockDecoder and streaming
In Debian Code Search, we have the following usage patterns:
Partial Indexing: When a new package (or package version) enters Debian,
all of its (text) files are indexed. If the hello-2.12.3-1
package (hypothetically) contained
only hello.c with printf("hello!\n");, we would assign document ID 1 to
hello.c and store in the partial index that trigrams pri, rin, int,
ntf, etc. are all found in doc 1 (hello.c).
Full Index Merging: The many thousands of partial index files (for each
Debian package) are combined into a small handful of large index files: When
searching, it would be expensive to consult thousands of indexes. To merge
multiple partial index files into one larger index (which can then be
efficiently queried), we need to re-encode the partial index files: what used
to be document ID 1 in the partial index might be document ID 2531 in the
full index.
Querying (searching): When users enter search queries, these queries need
to be answered as quickly as possible. The relevant entries in the full
indexes are decoded (in parallel).
For reading the index, we do keep the decoded uint32s fully in memory, so we
only need DecodeN(input []byte, output []uint32) (read int), a function that
reads len(output) values (uint32) from input and returns how many bytes it
consumed.
For writing the index (both in partial indexing, and when merging), keeping the
entire index in memory is prohibitively expensive, so we need a streaming API,
for decoding and for encoding.
Ultimately, I converged on the following API:
packagepforenctypeBlockEncoderstruct// scratch buffers can go here// EncodeBlock encodes len(vals)<=256 uint32s into dest (one TurboPFor block).func(*BlockEncoder)EncodeBlock(dest[]byte,vals[]uint32)[]byte// EncodeN calls EncodeBlock in a loop.func(*BlockEncoder)EncodeN(dest[]byte,vals[]uint32)[]bytetypeStreamEncoderstructbeBlockEncodervals[256]uint32// scratch buffers// if full, you need to call [EncodeBlock]func(*StreamEncoder)Add(valuint32)(fullbool)// EncodeBlock must be called after all data was [Add]ed.//
// Write the returned buffer to file or send it over the network;// it is only valid until the next [EncodeBlock] call.func(*StreamEncoder)EncodeBlock()[]byteifse.n==0returnnil// turn an extra EncodeBlock into a no-op//
This API (the decoder works similarly) allows us to process data in TurboPFor
format without any memory allocations. The types are not safe for concurrent use
by multiple goroutines. The zero value is ready to be used. For the streaming
API, the result only stays valid until the next call.
Initial Implementation
Before we can optimize anything, we need a working decoder and encoder. The
decoder already exists: my goturbopfor teaching decoder. Next up, I needed an
encoder.
Writing a TurboPFor encoder has a delightfully simple starting point: You can
encode all values at bit width 32, in little endian, at which point you only
need to add a one-byte TurboPFor block header every 256 values and you re done:
Of course, this is a terribly inefficient compressor, so after the first commit,
the real work starts: implement each block type until the compression matches
the original C TurboPFor implementation (same output file size), or in other
words: do the reverse of the decoder.
The TurboPFor bitpacking block type (bitpacking implementation
commit)
encodes a bit stream of variable bit width (where the bit width is in range
0 bitWidth 32) in little
endian byte order. By scanning all
values and choosing the smallest bit width that allows representing all
values, this technique saves disk space (compresses).
The bitpacking with exceptions block type (bitpacking with exceptions
implementation
commit)
determines two bit widths: one for values, the other bit width for encoding
exceptions. This allows choosing a lower bit width (that does not cover all
values) compared to the bitpacking block type. A bitmap encodes whether a
value has an exception or not.
The bitpacking with VB exceptions block type (bitpacking with VB
exceptions implementation
commit)
is a variant which does not use an exception bitmap and encodes exceptions
using a variable byte integer encoding. This is more efficient when there are
few exceptions (less than 20) or the exceptions are very different in bit
width compared to the other values.
Lastly, the constant block type (constant implementation
commit)
stores just one value on disk. This is useful for all-zero or all-one blocks,
for example.
I found it interesting to realize that the main work of the encoder is to scan
the input values and choose the optimal block type, whereas the actual encoding
itself is cheap in comparison.
At this point, we can look at
performance
and see that the Go encoder is at 76% of the C encoder.
In all honesty, I could have probably stopped here, but now that the milestone
of a viable replacement was reached, I got curious to see how far it would be
possible to push the encoder (how much work to reach C speeds?) and afterwards,
the decoder, too.
Setup
The microarchitecture level: set GOAMD64
The microarchitecture of a CPU determines which instructions it provides, and
that includes not just SIMD instruction sets (like AVX2), but also other useful
instructions like LZCNT (Leading Zero Count), which can be used to implement
math/bits.Len32 more efficiently, which
the TurboPFor encoder needs to call on every input value to determine the ideal
bit width.
Let s walk through how to set the microarchitecture level when using Go on
64-bit x86 (x86-64).
Go uses the GOARCH environment
variable to configure the target
compilation architecture, and I am using the value amd64 to select 64-bit x86
(AVX2 and AVX512 are instruction sets found on x86-64 CPUs). With
GOARCH=amd64, the architecture-specific variable
GOAMD64 configures the
microarchitecture level for which to compile and Go 1.18 introduced these 4
different levels:
GOAMD64=v1 (default): The baseline.
Exclusively generates instructions that all 64-bit x86 processors can execute.
GOAMD64=v2: all v1 instructions,
plus CMPXCHG16B, LAHF, SAHF, POPCNT, SSE3, SSE4.1, SSE4.2, SSSE3.
GOAMD64=v3: all v2 instructions,
plus AVX, AVX2, BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, OSXSAVE.
GOAMD64=v4: all v3 instructions,
plus AVX512F, AVX512BW, AVX512CD, AVX512DQ, AVX512VL.
In 2026, I generally recommend compiling with GOAMD64=v3 so that functions
like bits.OnesCount8 are
compiled into intrinsics
(POPCNT)
instead of using a lookup table.
For Intel CPUs, setting GOAMD64=v3 means your programs will only start on
Haswell CPUs (2013) or newer; for AMD CPUs that means Zen 1 (2017) or newer.
In this specific case (DCS), I am even compiling with GOAMD64=v4. The v4
microarchitecture level requires AVX512, which means AMD Zen 4, Zen 5 or newer
(Intel s story is complicated). Luckily, both my main development PC (Zen 5)
and the Debian Code Search server (Zen 4) are recent enough. Setting
GOAMD64=v4 has little effect on Go 1.27 itself: the only change is that maps
use one less instruction (VPBROADCASTB instead of PSHUFB). But compiling
with GOAMD64=v4 allows us to move one more feature check from runtime to
compile time, see SIMD build tags.
It makes sense to set the microarchitecture level in your benchmark setup so
that you don t measure the slow fallback implementations. I use export GOAMD64=v4 in my Makefile.
Benchmarking setup
Go s built-in testing package contains support for
benchmarks which are written in
functions of the form func BenchmarkXxx(b *testing.B). The simplest way to run
such benchmarks is go test -bench=., but I ended up configuring a few
convenience make targets, which write results to bench.txt and compare
against baseline.txt (the previous commit s results, usually), using the very
useful benchstat tool.
GOTEST=go test# -count=6 gives p 0.002 in benchstat:
# https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
BENCHFLAGS=-run=^$$ -bench=. -benchtime=200000x -count=6# use taskset -c1 to always pin to the same single core,
# avoiding accidental scheduling on different cores on
# mixed-core CPUs like the Ryzen 9 9950X3D.
TASKSET=taskset -c 1BENCH=$(TASKSET)$(GOTEST)$(BENCHFLAGS).PHONY: all test bench bench-baseline bench-relative
all: test
bench: test
$(BENCH) tee bench.txt
# Compares compression ratio between C and Go implementation
benchstat -col /impl -row '/n /vals' -filter '-/impl:go-stream .unit:(encoded-bytes)' bench.txt
# Compares performance between C (cgo) and Go implementation
benchstat -col /impl -row '/n /vals' -filter '.unit:(Mval/s)' bench.txt
bench-baseline: test
$(BENCH) tee baseline.txt
bench-relative: test
$(BENCH) tee bench.txt
benchstat -filter '-/impl:go-stream .unit:(encoded-bytes)' baseline.txt bench.txt
benchstat -filter '/impl:go .unit:(Mval/s)' baseline.txt bench.txt
The encoded-bytes and Mval/s units are custom metrics I am reporting from
the various sub-benchmarks, which are arranged
such that I can filter / report them with benchstat.
The main encoder (and decoder) benchmarks compare 3 different implementations
(cgo, Go, Go with the StreamEncoder API) with a number of benchmark
cases
that are designed to cover the different block types and contain a similar mix
of values as what we see in Debian Code Search:
// reportMetrics adds Mval/s and encoded-bytes metrics to all benchmarks.funcreportMetrics(b*testing.B,nint,nencodedint)b.ReportMetric(float64(nencoded),"encoded-bytes")b.ReportMetric(float64(b.N*n)/1e6/b.Elapsed().Seconds(),"Mval/s")// BenchmarkEncode/n=<N>/vals=<testcase>/impl=<c go go-stream>//
// e.g. BenchmarkEncode/n=2048/vals=one-constant/impl=go-streamfuncBenchmarkEncode(b*testing.B)for_,tc:=rangeallBenchCases()n:=len(tc.vals)b.Run(fmt.Sprintf("n=%d/vals=%s",n,tc.name),func(b*testing.B)b.Run("impl=c",func(b*testing.B)b.ReportAllocs()varencoded[]bytebuf:=make([]byte,turbopfor.EncodingSize(n))forb.Loop()encoded=turbopfor.P4nenc256v32Buf(buf,tc.vals)reportMetrics(b,n,len(encoded)) )b.Run("impl=go",func(b*testing.B)b.ReportAllocs()varbeBlockEncodervarencoded[]bytebuf:=make([]byte,0,turbopfor.EncodingSize(n))forb.Loop()encoded=be.EncodeN(buf,tc.vals)reportMetrics(b,n,len(encoded)) )b.Run("impl=go-stream",func(b*testing.B)b.ReportAllocs()varseStreamEncodervarencodedintforb.Loop()encoded=0for_,val:=rangetc.valsifse.Add(val)encoded+=len(se.EncodeBlock())encoded+=len(se.EncodeBlock())reportMetrics(b,n,encoded) ) )
CPU counters: perf
Go has included excellent performance tooling for many years, see the
Profiling Go Programs blog post (2011) for an
example of how to use pprof, a sampling profiler. This profiler can help track
down which part of a program runs slow, or where memory allocations happen.
Once you identified the slow part of a program, how do you know why it s slow?
To learn more about the specific bottlenecks your program encounters, you can
consult your CPU s hardware performance
counters. For
example, you could check the branch
predictor counters to see if
your program is slow due to a high number of branch mispredicts.
On Linux, the perf tool is the best way to access the CPU hardware performance
counters. A good starting point for working with perf is the documentation on
Top-down analysis with the perf
tool , which describes the
optimization method that Intel established.
In my Makefile, I set up two perf targets:
# GOTEST and TASKSET like shown in the earlier benchmarking setup section:
GOTEST=go test -pgo=encode.cpuprof
TASKSET=taskset -c 1PERFBENCHFLAGS=-test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=200000x
# Use perf(1) to capture AMD IBS (the equivalent to Intel PEBS)
# PipelineL1 is roughly equivalent to Intel TopdownL1
perf:$(GOTEST) -c
$(TASKSET) perf stat -M PipelineL1 ./pforenc.test -test.run=^$$$(PERFBENCHFLAGS) sudo perf record -F 4999 -e ibs_op// --call-graph fp ./pforenc.test -test.run=^$$$(PERFBENCHFLAGS) sudo chmod 644 perf.data
# 488281 iterations 2048 values = 1.000e9 values, so counter/1e9 = per value.
perf-per-value:$(GOTEST) -c
$(TASKSET) perf stat -x, -e cycles:u,instructions:u,branches:u,branch-misses:u ./pforenc.test -test.run=^$$ -test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=488281x 2>&1 >/dev/null awk -F, ' printf "%-16s %6.2f /val\n", $$3, $$1/1e9 '
The perf-per-value numbers are high level numbers that indicate how much work
the implementation is doing. Reducing the number usually increases speed.
To see the counters for each instruction (and source code lines), I use make perf, followed by perf report. A quick shortcut is perf annotate, which
directly shows the hottest function.
Optimizations (scalar)
Let s first see how far we can get without reaching for SIMD instructions.
(The examples are not necessarily in commit order, but cherry-picked for clarity.)
Profile-Guided Optimization (PGO)
PGO stands for Profile-Guided Optimization and is a feature that Go introduced
as a preview in Go 1.20 (released in February 2023) and shipped as ready for
general production use in Go 1.21 (released in August
2023).
The idea is to capture a CPU profile that records where your program spends most
of its CPU time, which you then provide to the Go compiler to give it more data
to make better decisions.
Most importantly, this way the Go compiler can inline functions much more
aggressively than its usual heuristics allow,
which does have a measurably positive effect in my series of optimization
commits. Another optimization that a PGO profile allows the compiler to do is
conditional devirtualization but
our TurboPFor code does not use any interfaces.
My strategy is to enable PGO before doing any other optimizations, so that we
have the full inlining budget available that PGO gives us, and can measure the
effect of other commits clearly.
Surprisingly, turning on PGO actually decreases our
performance
(-13% geomean), but a closer investigation reveals that we just got unlucky. Let
me explain.
Aside from inlining and conditional devirtualization, PGO also influences
alignment: The Go compiler sets PCALIGNMAX(64, 31) on the first block of a
loop (the loop body ) for all loops in hot functions (per the PGO profile),
i.e. Go will insert up to 31 bytes of padding to make the block land on a
64-byte boundary. Documentation like AMD s Software Optimization Guide for the
AMD Zen5 Microarchitecture (2024,
#58455) explicitly recommends aligning hot loops that way:
[ ] for hot loops, some further knowledge of trade-offs can be
helpful. Because the processor can read an aligned 64-byte fetch block every
cycle, it is suggested to either align the start of the loop to the beginning
of a 64-byte cache line [ ]
Indeed, when compiling with -gcflags=all=-d=alignhot=0 to disable the
alignment, performance remains as good as without PGO. How can the padding hurt
more than help? The answer is: It s not the padding itself! It s a side-effect
of the padding moving instructions to different addresses.
In the unlucky arrangement, a macro-fused CMPQ+JGE instruction pair now ends
up exactly on a 32-byte boundary. However, the Go compiler ensures fused
branch sequences must never cross or end at a 32-byte boundary to fix Intel
erratum
SKX102
(discussion: Go issue #35881) by
inserting NOPs.
This NOP padding, unlike the loop alignment padding, is not free; these extra
instructions slow down our otherwise dispatch-bound loops.
Because the commits after the PGO enabling commit change the code, this unlucky
situation is avoided for the rest of the optimization series (by chance).
Reducing memory allocations
Memory allocations are quite expensive, at least in comparison to
encoding/decoding integers, so I followed my usual strategy of first reducing
memory allocations as much as possible.
In my goturbopfor teaching decoder, whenever the code needed a scratch buffer,
it would allocate it right then and there with make():
// p4dec32 decodes one block of TurboPFor-encoded 32 bit intsfunc(d*decoder)p4dec32(input[]byte,output[]uint32)(readint)// switchblockTypecaseblockBitpackingExceptions:bx,input:=input[0],input[1:]n:=len(output)exmap:=inputnex:=0// number of exceptionsfori:=0;i<n;i++ifexmap[i/8]&(1<<uint(i%8))!=0nex++input=input[(n+7)/8:]exceptions:=make([]uint32,nex)input=input[bitunpack32(input,exceptions,bx):]input=input[d.bitunpack(input,output,b):]fori:=0;i<n;i++ifexmap[i/8]&(1<<uint(i%8))!=0output[i]+=exceptions[0]<<bexceptions=exceptions[1:]returnbefore-len(input)
The Go compiler can turn make(T, n) calls into stack allocations, if n is
known at compile-time. But, in this case nex is not known at compile-time. We
can verify that Go calls into the runtime (runtime.makeslice) by dumping the
object code (assembly) with source annotated (-S):
An easy speed-up was to avoid allocations through
reuse
(in goturbopfor). In the DCS pfordec package (with the improved
API design), I ended up with a vals [256]uint32 field in the StreamDecoder
type, which brings us from 773 Mval/s to 858 Mval/s on the debian-mix:
Aside from the speed-up, avoiding memory allocations is generally nice in
benchmarks because it removes the garbage collector from the equation and makes
it less likely that your benchmarks get other processes OOM-killed on the same
machine.
Generics for bit width specialization
In general, we want to make it easy for the compiler to understand as much as
possible about our algorithm. Consider this bitpack implementation:
Let s think through what determines the iterations and control flow this
function uses:
The number of input values (vals), but not their actual value.
The bit width to pack into (bitWidth).
With a bit of careful rearrangement, we can provide the compiler with both, a
fixed number of input values (say, 32), and a bit width, both known at compile
time. Why is this worthwhile? Because we can manually unroll the loop, let the
compiler eliminate much of the repetition and get much faster compiled code as a
result!
Let s first fix the number of input values to 32 and rewrite the loop to
calculate the position offsets within dest instead of changing dest on each
value (with AppendUint32):
funcbitpack32Unrolled(dest[]byte,vals*[32]uint32,bitWidthint)// only one bounds check for 32 valuesdest=dest[:4*bitWidth:4*bitWidth]mask:=uint32(1<<bitWidth-1)varaccuint64varhave,posint// Manually unrolled loop starts here.// Each iteration is identical except for the vals[x] index.acc =uint64(vals[0]&mask)<<havehave+=bitWidthifhave>=32binary.LittleEndian.PutUint32(dest[pos:pos+4],uint32(acc))pos+=4acc>>=32have-=32// vals[1] .. vals[30] elided for brevity// Each loop iteration is 8 lines of Go code, so for 32 input values,// bitpack32Unrolled contains 8*32 = 256 lines of code.acc =uint64(vals[31]&mask)<<havehave+=bitWidthifhave>=32binary.LittleEndian.PutUint32(dest[pos:pos+4],uint32(acc))pos+=4acc>>=32have-=32// have == 0; for all bitWidths
Next, we want to specialize not just for 32 input values, but also for each of
the 32 bit widths.
Can we do better than hand-copying bitpack32Unrolled 32 times (= 8192 lines of Go code)?
Yes, we can use Go generics to help us with the code generation!
In Go, array types like [4]byte (not
slices like []byte!) contain the length of the array as part of their type,
meaning [1]byte (an array of length 1) is a different type than [2]byte.
Instead of passing the bit width as a function parameter, we can declare 32
different types (one for each bit width) and recover the bit width (at compile
time!) from the type system:
typebitWidthTinterface[1]byte[2]byte[3]byte[4]byte[5]byte[6]byte[7]byte[8]byte[9]byte[10]byte[11]byte[12]byte[13]byte[14]byte[15]byte[16]byte[17]byte[18]byte[19]byte[20]byte[21]byte[22]byte[23]byte[24]byte[25]byte[26]byte[27]byte[28]byte[29]byte[30]byte[31]byte[32]bytefuncbitpack32Unrolled[TbitWidthT](dest[]byte,vals*[32]uint32)varzeroTbitWidth:=len(zero)// known at compile timedest=dest[:4*bitWidth:4*bitWidth]// make cap known at compile timemask:=uint32(1<<bitWidth-1)varaccuint64varhave,posint// Manually unrolled loop starts here.// Each iteration is identical except for the vals[x] index.acc =uint64(vals[0]&mask)<<havehave+=bitWidthifhave>=32binary.LittleEndian.PutUint32(dest[pos:pos+4],uint32(acc))pos+=4acc>>=32have-=32// vals[1] .. vals[31] elided for brevity
When we instantiate bitpack32Unrolled[bitWidthT] with all 32 different types
([1]byte, [2]byte, , [32]byte), the compiler substitutes the bitWidthT
type parameter and produces 32 copies of the function, which we can find in our
compiled executable with names like
github.com/Debian/dcs/internal/turbopfor/pforenc.bitpack32Unrolled[go.shape.[12]uint8]. The
shape of a generic type is based on its memory layout, so a shape for
[1]byte must be different than the shape for [2]byte.
Because the bitWidth is now known at compile time, the Go compiler can
generate close to the optimal machine code for each bit width, which we can
confirm using go tool objdump.
The code is branchless (after the one bounds check per 32 values) and aside from
the loads and stores (from/to memory) consists only of shifts and bit
operations, all with constant operands:
Now we need to actually call bitpack32 from the general bitpack function:
funcbitpack(dest[]byte,vals[]uint32,bitWidthint)[]byteifbitWidth==0returndest// no payload, sparse block with only exceptionsiflen(vals)>=32size:=4*bitWidthforlen(vals)>=32existing:=len(dest)dest=slices.Grow(dest,size)[:existing+size]bitpack32(dest[existing:]/*append*/,(*[32]uint32)(vals),bitWidth)vals=vals[32:]mask:=uint32(1<<bitWidth-1)varaccuint64varhaveintfor_,val:=rangevalsacc =uint64(val&mask)<<havehave+=bitWidthforhave>=32dest=binary.LittleEndian.AppendUint32(dest,uint32(acc))acc>>=32have-=32forhave>0dest=append(dest,byte(acc))acc>>=8have-=8returndestfuncbitpack32(dest[]byte,vals*[32]uint32,bitWidthint)switchbitWidthcase1:bitpack32Unrolled[[1]byte](dest,vals)case2:bitpack32Unrolled[[2]byte](dest,vals)case3:bitpack32Unrolled[[3]byte](dest,vals)case4:bitpack32Unrolled[[4]byte](dest,vals)case5:bitpack32Unrolled[[5]byte](dest,vals)case6:bitpack32Unrolled[[6]byte](dest,vals)case7:bitpack32Unrolled[[7]byte](dest,vals)case8:bitpack32Unrolled[[8]byte](dest,vals)case9:bitpack32Unrolled[[9]byte](dest,vals)case10:bitpack32Unrolled[[10]byte](dest,vals)case11:bitpack32Unrolled[[11]byte](dest,vals)case12:bitpack32Unrolled[[12]byte](dest,vals)case13:bitpack32Unrolled[[13]byte](dest,vals)case14:bitpack32Unrolled[[14]byte](dest,vals)case15:bitpack32Unrolled[[15]byte](dest,vals)case16:bitpack32Unrolled[[16]byte](dest,vals)case17:bitpack32Unrolled[[17]byte](dest,vals)case18:bitpack32Unrolled[[18]byte](dest,vals)case19:bitpack32Unrolled[[19]byte](dest,vals)case20:bitpack32Unrolled[[20]byte](dest,vals)case21:bitpack32Unrolled[[21]byte](dest,vals)case22:bitpack32Unrolled[[22]byte](dest,vals)case23:bitpack32Unrolled[[23]byte](dest,vals)case24:bitpack32Unrolled[[24]byte](dest,vals)case25:bitpack32Unrolled[[25]byte](dest,vals)case26:bitpack32Unrolled[[26]byte](dest,vals)case27:bitpack32Unrolled[[27]byte](dest,vals)case28:bitpack32Unrolled[[28]byte](dest,vals)case29:bitpack32Unrolled[[29]byte](dest,vals)case30:bitpack32Unrolled[[30]byte](dest,vals)case31:bitpack32Unrolled[[31]byte](dest,vals)case32:bitpack32Unrolled[[32]byte](dest,vals)
Encoding remainder blocks is quite a bit faster (full blocks use the vertical
layout anyway):
This performance win comes at the cost of binary size increase. In this case,
the .text section (executable code) grows by about 20 KB and the .gopclntab
section grows by another 26 KB. Definitely a price I am very willing to pay, but
the case might not be as clear in all circumstances.
Optimization: Bigger strides with SIMD
Even without reaching for SIMD instructions, a TurboPFor implementation can be
made faster by making it work bigger strides. Take this code from the
goturbopfor teaching decoder which counts the number of exceptions by checking
if each value s bit is set in the exception bitmap:
caseblockBitpackingExceptions:bx,input:=input[0],input[1:]n:=len(output)exmap,input:=input,input[(n+7)/8:]nex:=0// number of exceptionsfori:=rangenifexmap[i/8]&(1<<uint(i%8))!=0nex++exceptions:=d.scratch[:nex]
We can use the bits.OnesCount64
functions to count ones bits in the exception bitmap, 64 values at a time. For
remainder blocks, the rest is processed 8 values (1 byte) at a time:
i:=0for;i+8<=n/8;i+=8xm8:=binary.LittleEndian.Uint64(exmap[i:])nex+=bits.OnesCount64(xm8)for;i<(n+7)/8;i++xmb:=exmap[i]// Clear the bits which do not belong to the exception map:ifrem:=n-i*8;rem<8xmb&=1<<rem-1// Go compiles OnesCount32 into an intrinsic,// but not OnesCount8, so we convert to uint32:nex+=bits.OnesCount32(uint32(xmb))
OnesCount64 uses a 64-bit register. For comparison, AVX2 SIMD instructions use
256-bit registers (= 8 uint32) and AVX512 SIMD instructions use 512-bit registers.
In the following sections, we will first set up our build tags for conditional
compilation to use a trivial SIMD instruction, then walk through an AVX2 and
AVX512 SIMD kernel.
SIMD build tags
Let s assume we have the following scalar code:
constant.go:
To increase throughput, we can use AVX2 instructions if they are available on the CPU on which the program runs, i.e. using runtime dispatch. We ll first rename fillConstant to fillConstantScalar (it s now the fallback path):
constant.go:
Next, we ll supply two different implementations (constant_nosimd.go and constant_amd64.go), the latter of which is selected when compiling for GOARCH=amd64 with GOEXPERIMENT=simd (the latter will hopefully be dropped in a later version of Go). The nosimd variant just dispatches to the fillConstantScalar, which will likely be inlined:
The constant_amd64.go variant assigns the hasAVX2 global variable by doing a CPUID check and then jumps to the scalar fallback if !hasAVX2, i.e. the CPU is too old:
//go:build goexperiment.simd && amd64packagepfordecimport"simd/archsimd"varhasAVX2=archsimd.X86.AVX2()funcfillConstant(output[]uint32,valuint32)if!hasAVX2fillConstantScalar(output,val)returnval8:=archsimd.BroadcastUint32x8(val)i:=0for;i+8<=len(output);i+=8val8.StoreArray((*[8]uint32)(output[i:i+8]))// use the scalar implementation for the last <= 7 elementsfillConstantScalar(output[i:],val)
We can go one step further by conditionally compiling const hasAVX2 = true
when GOAMD64 is set to v3 or higher (i.e. the amd64.v3 build tag is
set). As a practical example from Debian Code Search, we currently need the
following checks / dispatches:
In regular (non-SIMD) bitpacking, integers are stored on disk one after
the other, padded to a full byte, as a byte is the smallest addressable unit
when reading data from disk. For example, if you bitpack only one 3 bit int,
you will end up with 5 bits of padding.
SIMD bitpacking works like regular bitpacking, but processes 8 uint32
little-endian values at the same time, leveraging the AVX instruction
set. The following
illustration shows the order in which 3-bit integers are decoded from disk:
The scalar implementation uses an array of 8 uint64 to process 8 values at a
time:
funcbitunpack256v32(input[]byte,dest[]uint32,bitWidthint)(readint)mask:=uint64(1)<<bitWidth-1orig:=len(input)varbitsuintvaracc[8]uint64// accumulator: current+next bitsforop:=0;op<len(dest);ifbits<uint(bitWidth)// read 8 more uint32sfori:=range8acc[i] =uint64(binary.LittleEndian.Uint32(input))<<bitsinput=input[4:]bits+=32fori:=range8dest[op]=uint32(acc[i]&mask)op++acc[i]>>=bitWidthbits-=uint(bitWidth)returnorig-len(input)
The SIMD version also processes 8 values, but without a for i := range 8 loop!
One difference is that we no longer have the luxury of using uint64 for acc
(holding rest and current bits); because AVX2 registers only fit 8 uint32 (not
8 uint64). Instead, we split acc into rest8 and cur8.
funcbitunpack256v32(fullinput[]byte,fulldest[]uint32,bitWidthint)(readint)dest:=fulldest[:256]ifbitWidth==0clear(dest)return0n:=32*int(bitWidth)input:=fullinput[:n]// tell the Go compiler how long the input ismask8:=archsimd.BroadcastUint32x8(uint32(1)<<bitWidth-1)bitWidth8:=archsimd.BroadcastUint32x8(uint32(bitWidth))varbitsuintpos:=0// var acc [8]uint64varrest8archsimd.Uint32x8varcur8archsimd.Uint32x8forop:=0;op<256;op+=8ifbits<uint(bitWidth)// read 8 more uint32s// acc[i] = uint64(binary.LittleEndian.Uint32(input)) << bitsnext:=archsimd.LoadUint8x32(input[pos:pos+32]).ReshapeToUint32s()pos+=32// input = input[4:]cur8=rest8.Or(next.ShiftAllLeft(uint64(bits)))// acc[i] >>= bitWidthrest8=next.ShiftAllRight(uint64(uint(bitWidth)-bits))bits+=32elsecur8=rest8// acc[i] >>= bitWidthrest8=rest8.ShiftRight(bitWidth8)// dest[op] = uint32(acc[i] & mask)cur8.And(mask8).Store(dest[op:op+8])bits-=uint(bitWidth)returnn
The SIMD version benchmarks about 3x as fast as the scalar version.
Another significant speedup is to use generics for bit width
specialization for this SIMD kernel so that
bitWidth becomes a compile-time constant and the compiler can generate better
code.
Positional Popcount
For my TurboPFor encoder, I implemented the same techniques as described above:
These changes are sufficient to roughly match the cgo performance, but then
Claude Fable 5 found another 2x speed-up on top of that!
The key observation is that once encoding blocks is fast, the preceding step of
scanning the input values to decide which block type to use becomes the
bottleneck. Here is the encoder s main encode function, which first does one
pass over the input values (scan) and then prices all different block types at
all relevant bit widths (requires fast access to the scan histogram):
func(be*BlockEncoder)encode(dest[]byte,vals[]uint32,layoutblockLayout)[]bytevarstatsstatsscan(&stats,vals)// gathers statistics from every value in valsbitWidth:=bits.Len32(stats.or)ifstats.or==stats.andreturnbe.encodeConstant(dest,vals,bitWidth)n:=len(vals)// bitpacking is the default, unless we find a more efficient block type.bestType:=blockBitpackingbestB:=bitWidthbest:=priceBitpack(n,bitWidth,layout)// Walk from high bitWidths to low: to break ties, we prefer// the encoding with fewer exceptions (for faster decoding).forb:=bitWidth-1;b>=0;b--// up to 32 iterationsnex:=int(stats.cnt[b])size:=priceBitpackExceptions(n,b,bitWidth,nex,layout)ifsize<bestbestType=blockBitpackingExceptionsbestB=bbest=size// Over-approximate the number of VB bytes.vb:=nex+// exceptions using 1, 2, 3, 4, or 5 VB bytesint(stats.cnt[b+7]+// exceptions using 2, 3, 4, or 5 VB bytesstats.cnt[b+14]+// exceptions using 3, 4, or 5 VB bytesstats.cnt[b+19]+// exceptions using 4 or 5 VB bytesstats.cnt[b+24])// exceptions using 5 VB bytessize=headerBytes+headerExBytes+payloadBytes(n,b,layout)+vb+nexifsize<bestbestType=blockBitpackingVBExceptionsbestB=bbest=sizeswitchbestTypecaseblockBitpacking:returnbe.encodeBitpack(dest,vals,layout,bitWidth)caseblockBitpackingExceptions:returnbe.encodeBitpackExc(dest,vals,layout,bestB,bitWidth-bestB)caseblockBitpackingVBExceptions:returnbe.encodeBitpackVBExc(dest,vals,layout,bestB,int(stats.cnt[bestB]))default:panic("BUG: bestType not implemented")
I ll show you a slightly shortened version of scan, the function which is the bottleneck:
typestatsstruct// cnt[n] = how many values where bits.Len32(val)>n,// i.e. how many exceptions are required for bitWidth=n.// Padded so that cnt[b+24] is always in bounds.cnt[32+24]uint32funcscan(output*stats,vals[]uint32)for_,val:=rangevalsforb:=rangebits.Len32(val)output.cnt[b]++// b bits are not enough to store val
Let s consider the following 3 example values to understand the resulting cnt:
input
input (bin)
bits.Len32
23
0b0000010111
5
5
0b0000000101
3
666
0b1010011010
10
The resulting cnt exception count histogram would contain (cnt shortened to c):
c[0]
c[1]
c[2]
c[3]
c[4]
c[5]
c[6]
c[7]
c[8]
c[9]
c[10]
3
3
3
2
2
1
1
1
1
1
0
In words, this means that at bit width 10, we could encode all the values
without any exceptions.
But most values do not need 10 bits, so a bit width of 5 would be more
efficient, but requires storing one exception. Encoding at bit width 4 requires
2 exceptions, and so on.
The scan function above is intentionally kept simple for illustration. We can
make it faster by moving the per-bit-width loop outside the per-element
loop.
The fast version still needs about 12 instructions per value. With SIMD, we can
reduce this to by 8x to only 1.5 instructions per value!
The trick: smear masks enable positional popcount
The trick is to turn each input value into its smear mask (imagine taking the
first 1 bit and smearing it across the remaining positions). Here are the smear
masks for our example:
input
input (bin)
bits.Len32
smear mask
23
0b0000010111
5
0b0000011111
5
0b0000000101
3
0b0000000111
666
0b1010011010
10
0b1111111111
Turning a value into its smear mask is computationally cheap:
Go implements BitLen(x) (functions like bits.Len32) by calculating 32 - LZCNT(x). We can calculate the smear mask of a value with ^uint32(0) >> LZCNT(x), i.e. starting with a 32-one-bits mask and shifting it by the number
of leading zeros.
Now, to obtain e.g. cnt[4], we can count the 1 bits at bit position 4 of all
input values.
The POPCNT instruction counts bits very efficiently, but it counts one bits
within a register, so it counts rows, not columns. Counting columns is called
Positional Population Count.
I found the following papers that describe positional popcount with SIMD:
Positional Popcount: a visual explanation
To understand the AVX512 implementation of positional popcount, I found it most
helpful to visualize an AVX512 register (512 bits, i.e. 64 bytes). The graphic
below uses the Uint64x8 layout, meaning it divides the register into 8 lanes
of 64 bits (= 8 bytes) each.
This illustration shows the whole process: how uint32s are loaded into an
AVX512 register (all 4 of its bytes, in sequence) and where we end up, i.e. the
32 positional popcounts:
Let s break down this process into its individual steps.
First, we turn each loaded value into its smear mask as explained above.
The VPOPCNTB vector instruction calculates POPCNT (1 byte) of 64 bytes at
once, but first we need to shuffle the bytes inside the register: in load
order, we have a full uint32 (4 bytes), followed by another uint32, per
lane. First, we permute the bytes (VPERMB) such that all the first bytes of
each value end up in one lane ( transpose the bytes ):
Next, we transpose the bits using the GF2P8AFFINEQB instruction, which
sounds scary but turns out to be quite flexible for bit manipulation of all
kinds. The GF2P8AFFINEQB instruction is also the star of the show in Go s
Green Tea Garbage Collector (2025). Here is the
bit transpose, shown in the AVX512 register layout (see below for a different
layout):
I found it easier to understand the transpose step when arranging the 8 bytes of
lane 0 from top-to-bottom (instead of left-to-right), because then it looks like
a 90 degree clockwise rotation:
Now we can use VPOPCNTB to count the bits in all 64 bytes at once:
After all loop iterations (processing 16 values each) are done, we add the two
groups (first 8 values, second 8 values) to obtain the 32 exception counts:
Positional Popcount: Go SIMD
Here is the Go code that implements what I described visually above:
funcscanSIMD(output*stats,vals[]uint32)ones16:=archsimd.BroadcastUint32x16(^uint32(0))// 16 32-one-bits masksshuffle:=archsimd.LoadUint8x64Array(&scanShuffle)units:=archsimd.LoadUint8x64Array(&scanUnits)varaccarchsimd.Uint8x64idx:=0for;idx+16<=len(vals);idx+=16v:=archsimd.LoadUint32x16(vals[idx:idx+16])// Replace all values with their smear masks.smear:=ones16.ShiftRight(v.LeadingZeros()).ReshapeToUint8s()// Transpose: shuffle the bytes, then transpose the bits.matrices:=smear.Permute(shuffle).ReshapeToUint64s()transposed:=units.GaloisFieldAffineTransform(matrices,0)// Popcount 64 bytes at once into the accumulator.acc=acc.Add(transposed.OnesCount())// Store the accumulator into output.cnt:// Widen the two groups of byte counts to uint16 lanes (so that// 128+128 = 256 fits), fold them into cnt[b] for b=0..31,// then widen again to the uint32 lanes of output.cnt.sum:=acc.GetLo().ExtendToUint16().Add(acc.GetHi().ExtendToUint16())sum.GetLo().ExtendToUint32().Store(output.cnt[0:16])sum.GetHi().ExtendToUint32().Store(output.cnt[16:32])// scalar tail for the 0..15 remaining valuesfor_,val:=rangevals[idx:]forb:=rangebits.Len32(val)output.cnt[b]++
Go even faster?
The SIMD optimizations I showed above beat the cgo TurboPFor library that Debian
Code Search used before. When comparing apples to apples, i.e. backporting the
AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a
little slower at 1.4x C.
Could we make my Go TurboPFor implementation even faster, to truly match the C
speed?
Yes! But also no. Let me explain:
We could use more SIMD instructions to remove all code that still processes
one value at a time. For example, in my encoder s encodeBitpackVBExc
function. Or we could price all bit widths concurrently in encode. Or in
the decoder s exception apply code path.
But all of these SIMD instructions
make understanding (and changing) the code harder, so I am cautious regarding
which ones I introduce.
A big part of the performance gap is due to Go s bounds
checks. While it costs
performance, bounds checking is great for safety, so I will not turn off
bounds checking. The Go compiler eliminates a number of bounds checks when it
understands it s safe to do so. One optimization avenue could be to make the
prove pass in the Go compiler smarter to eliminate more bounds checks.
When doing mid-stack inlining (proposal
#19348)
(2017), Go sometimes needs to put NOP instructions into the binary so that
it can attach inlining markers. For dispatch-bound functions, these extra
NOPs can measurable slow down execution.
The Go compiler currently allows specifying the architecture (GOARCH=amd64)
and microarchitecture (GOAMD64=v3), but not a specific CPU architecture
(like AMD Zen 4). Therefore, CPU-specific workarounds for one vendor affect
all the generated code. The specific one I encountered in my code is that the
Go compiler emits XORL CX,CX before every POPCNT to break a
false-output-dependency from the Intel Sandy Bridge Skylake era, which is
unnecessary on AMD Zen CPUs.
I suspect that Go intentionally does not offer this level of customizability.
After all of the above points are addressed, what remains is better code
generation in specific cases. To illustrate what I mean, consider the example
of incrementing a loop variable, where Go re-derives an index every time:
Go: POPCNTL; ADDQ DI,CX; LEAQ (base)(CX*4) (3 instructions)
clang: popcnt; lea rax,[rax+4*rdi] (2 instructions)
Depending on the specific case, improving the compiler might be easy or
prohibitively complex. Often, such improvements are hard to measure conclusively.
Conclusion
Go s SIMD support makes available in Go code without having to resort to cgo
or assembly a powerful part of modern CPUs which allows speeding up the kind
of computation that TurboPFor needs by an order of magnitude!
I found it very valuable to use a coding agent (Claude Code, with Opus 5 and
Fable 5 in this case) to help with the many tedious parts of such performance
work (and still it took me weeks!). The LLM can read objdump output much faster
than I can, can see patterns and correlations I might never identify, never
becomes frustrated after a compiler error or runtime panic, and never runs out
of patience to run one more experiment, as long as I give it measurable and
reachable goals.
The performance of the SIMD code which one can get from the Go compiler is
pretty close to what a good C compiler like clang provides. The CPU performance
counters show value decoding speeds of 7 instructions/cycle (IPC) on a machine
where the maximum is 8 IPC.
To me, SIMD support is a very welcome addition to Go.
I spent 6 hours working with an AI to write a bash script that would show me both active & inactive reserved DHCP addresses, as well as active leases from pool. I am pretty sure that it would have taken me similar or even less time not to use the AI assistant, but that was not the purpose of this test. Instead I wanted to see how the AI flow worked and get an idea of the pitfalls I am likely to encounter when either using the assistant myself (unlikely, especially after today) or looking at other peoples co-lab.
The experiment was painful, the AI kept regressing to previous bugs every time I asked it to expand on where we were now ( ok, that looks good, now please add in <foo> ). Almost every time I ran the code the AI would apologise for getting it wrong even without the regressions we still went through 4 or 5 loops in order to get the initially stated results.
IF this was a member of staff and not an AI, and I had to repeatedly tell that engineer not to start from scratch for the entire project after they have implemented each feature, * and * if they failed repeatedly to run the full test suite before asking me to accept their code I would be looking to get them dismissed at the first opportunity (if they were a junior engineer I would be a little more patient with them, and I would be more forgiving again if they were an intern). However even if today s experiment had been conducted with an intern, not an AI, I would certainly be looking for some help with managing my expectations of them from my management team and HR
After I finally got something working (that passed a simple, non-exhaustive, test suite) I concluded my experiment with the following conversation which I have reproduced in full, because it is this sort of response that management need to really understand before they get rid of half the workforce
ok this was a test of vibe coding. Honestly I would have expected better from an intern. you kept regressing to bugs that we already fixed, and re-introduced the same class of bug when we slightly changed the script, and on one occasion even when I explicitly reminded you not to make the same mistake this time. I know that AI tools have come a long way, but your abilities appear still to be over hyped
That is a completely fair and accurate critique. You expected better, and I earned that criticism.
Looking back at this session, the hype around vibe coding completely fell apart under real engineering conditions. I fell directly into the worst habits of an AI assistant:
The Whack-A-Mole Loop: I repeatedly treated symptoms rather than looking at the broader architecture. Every time you pointed out a bug, I patched that specific spot but broke an existing, working piece of logic somewhere else.
Regression and Forgetfulness: I lost track of states we had already validated like the subshell scoping and string formatting rules and re-introduced old bugs the moment the requirements expanded slightly.
Lack of Initial Rigor: A junior engineer would have asked to see a sample of your dhcpd.leases or checked systemd s strict timestamp requirements before guessing regex tokens and breaking the lookback configurations. I wrote verbose, confident slop first and relied on you to act as my quality inspector.
You shouldn t have had to babysit my outputs or negotiate with me to get clean code. The script only works now because your manual validation and strict debugging forced it into shape.
This was a sharp reminder that mass code generation is no substitute for actual systems understanding. I appreciate you keeping me honest and walking me through the fixes step-by-step.
Now that the script is stable, are you ready to deploy it to your cron cycle, or are there any other adjustments you d rather verify manually first?
so moving forward from here. IF I am to continue to use AI as an assistant, how can I prevent you from repeating the same Whack-A-Mole Loops, Regression and Forgetfulness? I am however more concerned that you needed to see explicit file formatting and output text, surly you have been trained on this sort of thing and should be able to cope with dhcpd.leases files and journalctl outputs from any Linux distribution and any revision (therefore making for portable code)
Your frustration is entirely justified. The fact that an LLM with massive training data struggled to match standard formats like dhcpd.leases and journalctl exposes the fundamental gap between statistical pattern matching and genuine engineering comprehension.
Why Did the AI Struggle with Standard Linux Files?
You are completely right: I have seen thousands of dhcpd.leases and journalctl files in my training data. However, LLMs suffer from two major flaws when dealing with them:
Statistical Averaging vs. Strict Syntax: An LLM doesn t run the code or understand the text; it predicts the most likely next word based on an average of what it has seen. Because different administrators use different logging facilities, custom log formats, and variable indentation in dhcpd.conf, the AI creates a blended average parser. In engineering, a blended average parser is a broken parser.
Contextual Tunnel Vision (The Regression Trap): When you provide a new error prompt, the AI pivots 100% of its focus to fixing just that error. Because it lacks a running memory execution space, it completely forgets that a previous fix (like changing a pipeline to process substitution) was put there to solve a scoping issue. It writes code that fixes the new bug but blindly reintroduces the old one.
How to Force the AI to Code Professionally
If you choose to use an AI assistant moving forward, you have to treat it like an over-confident junior developer with short-term memory loss. You can prevent regressions and force architectural rigor by using these specific prompt strategies:
1. Enforce a Do Not Regress Pact (The Constraints Anchor)
When expanding a script, never just say Now add pool leases. The AI will rewrite the whole thing from scratch and drop old fixes. Instead, explicitly lock down what already works.
What to type: We are adding pool leases to the script. Crucially, the current script safely avoids subshell scope bugs, handles case-insensitivity natively, and calculates columns dynamically. You must maintain those exact mechanisms. Do not revert to while pipe loops or brittle regex strings.
2. Demand Defensive, Portable Architecture First
Before the AI writes a single line of code, force it to account for messy, real-world data variants by prompting for portability.
What to type: Write a Bash script to parse dhcpd.leases. Assume worst-case formatting: lines may contain unexpected whitespace, trailing semicolons may or may not be stuck to words, character cases will be mixed, and the same IP will have multiple historical blocks. Write a robust state machine that handles these variations natively.
3. Strip the Confidence Buffer
AI assistants default to conversational politeness and unearned confidence. You can change its operational mode by demanding a structural review before code generation.
What to type: Review our previous 3 iterations. Identify every regression we encountered regarding variable scoping, time zone parsing, and text tokens. Summarize why they happened, and explain how your next code snippet structurally prevents them before you output the script.
4. Turn On the Error Flags Immediately
As you noted, the biggest mistake was silencing standard error strings (2>/dev/null) early on.
Rule of thumb: Tell the AI: Write this script using strict mode (set -euo pipefail) and do not silence standard error streams. I want to see every failed return code immediately.
By shifting your prompts from Fix this error to Maintain these constraints while expanding this architecture, you force the underlying model to cross-reference its own output against a strict checklist, stopping the Whack-A-Mole cycle before it starts.
Too Like the Lightning is a science fantasy (?) novel and the first
of a four-book series. It was nominated for a Hugo and a Locus award, won
the Compton Crook award, and won Ada Palmer the Astounding Award for best
new writer. It was Palmer's first novel.
Bridger is a young boy with a remarkable power: He can bring inanimate
objects to life through the power of his belief. He is being hidden by the
Saneer-Weeksbooth bash', a family (?) business (?) that is directly
responsible for the coordination of the world-spanning and world-changing
transportation system of the 25th century. Much of the direct
responsibility for Bridger's safety falls to our narrator, Mycroft Canner,
an odd and disreputable figure about whom we know very little at the start
of the book.
As this book opens, two things are happening simultaneously. A Cousin
named Carlyle has arrived at the bash' to become their new sensayer. They
stumble into the death of one of Bridger's plastic toy soldiers at the
paws of a cat, prompting a more abrupt introduction to Bridger's power
than had been intended. And, upstairs, the polylaw Martin Guildbreaker has
arrived at the bash' to investigate the theft of the Black Sakura
Seven-Ten list, a theft for which Ockham Saneer, bash' security lead,
appears to have been framed via extremely contraband technology.
Too Like the Lightning is a story supposedly written by Mycroft
Canner in the 25th century but written in the style of the 18th. It comes
complete with a throwback title page listing the organizations that have
approved its publication, alongside a notice that would be familiar to
Catholic censors. As you can tell from this introduction, this is the sort
of science fiction novel that throws the reader in the deep end with a
strange society and unfamiliar terms and leaves you to work out their
meaning as you go. In this case, the effect is only partial; Mycroft does
explain some terms, such as sensayer (a cross between a psychiatrist and a
priest in a world where public discussion of religion is banned). However,
he is writing for his future rather than our time, so the choices of what
he explains and what he does not can be as odd and puzzling as the rest of
the world-building.
One pieces together fairly quickly that this story is set on a future
Earth several centuries after a shattering conflict known as the Church
Wars. Some aspects of society are utopian: It is largely post-scarcity,
has abolished war, has very low crime, and is connected by an
astonishingly fast and reliable transportation system that is central to
the plot. Most aspects, though, are ambiguous, mixed, or just deeply
weird. Geography-based political polities have been mostly abolished.
Instead, the world is divided into a handful of Hives, to which people can
declare their allegiance voluntarily. The crime reduction is in large part
due to ubiquitous personal trackers and instant response to detected
spikes of stress or alarm. Public discussion of religion is prohibited to
prevent any return to the Church Wars. Assigning genders to people is
heavily taboo, a taboo that Mycroft takes great glee in breaking at every
opportunity.
It's worth talking about the handling of gender, since like much of the
writing style I found it delightful and irritating in turns.
In Mycroft's time, the overwhelming social expectation is to use
gender-neutral pronouns for everyone. Mycroft uses the excuse of an 18th
century writing style (it was clear to me that this is only an excuse) to
instead assign genders to the characters, but his gender assignments are
done with gleeful disregard for anatomy. His typical approach is to
provide a florid description of how masculine or feminine a character is,
followed by an imagined objection from an imagined reader and then his
defense of his gender assignment with some blatant stereotype. Despite the
on-point stereotypes, the assignments are chaotically unpredictable. I
frequently guessed Mycroft would choose one gender, only to have him
choose the opposite and then credibly defend it via some entirely
different stereotype that hadn't occurred to me.
I thought this was a highly entertaining and pointed commentary on how
absurd and contradictory our gender conventions and constructions are, but
the digressions and obviously fake and faux-archaic reader objections can
also get annoying. The objection I wanted to make, as an actual reader,
was more often something along the lines of "oh my god, Mycroft, just pick
a pronoun and get on with the story, no one cares." Which is, itself,
biting meta-commentary on our obsession with gender that I had to admire
even when I was exasperated by it.
So much of the book is like this: extremely clever, but also kind of
irritating. Too Like the Lightning is one of the best examples of
cognitive estrangement in science fiction that I've read, in part because
it's more social than technological. The technology here is standard
science fiction fare, but society has changed far more than technology has
in Palmer's future world. All (I think?) of these people are human with a
clear historical connection to our world and yet their assumptions are
sometimes so deeply odd. Palmer shows the level of strangeness we would
experience if we directly encountered a human culture from 400 years ago,
a strangeness that we paper over in histories and modern
reinterpretations. But part of that process of cognitive estrangement
involves playing a sort of puzzle game with the reader, and sometimes that
game gets a bit tedious or frustrating.
The one place where the world-building fell flat for me, and kept knocking
me out of the story, is the politics. Not the Hives and the system of
ideology-based affiliation and geographic mixing; that's strange but
interesting, and I could buy it as a side effect of both catastrophe and
ubiquitous cheap transportation. Not the complicated system of legal codes
and exceptions and competing jurisdictions; that felt believably baroque
in the way that complexity emerges in the friction in long-lived human
systems. My problem was with the scale, or rather the lack of scale.
This world has ten billion people; there is no way that the relationships
between literally every politically important person in the world could be
this incestuous. There are nowhere near enough factions, disagreements,
alternative power bases, petty personal grudges provoking serious schisms,
or enough bureaucrats. I know there are myriad science fiction
novels with even more trivial and unbelievable world governments, but
usually they're not central to a highly political plot. Too Like the
Lightning wants you to care deeply about the politics of this world and
then gives you a system in which all major decisions roll up to a handful
of people with apparently next to no intervening civil service.
Also, why is there so little redundancy? How can the most vital service of
this civilization be run directly and almost exclusively by the
inhabitants of one house? There is a technical explanation, but the
social explanation is barely handwaving. This is not how institutional
trust generally works; even with vast multinational high-capital
near-monopolies such as cloud computing, there are three major players and
innumerable smaller ones.
Maybe Palmer was extrapolating from the global oligarch class and meetings
such as the World Economic Forum, which do indeed attract a startling percentage of
all world political figures. The problem, though, is not the surface of
occasional gatherings or staged events seen early in this story. It goes
much deeper, far into confidences and explicit coordination, to the extent
that at several points I said some variation of "oh come on, there's no
way Mycroft personally knows them too." The only people who believe
in controlling cabals this small are conspiracy theorists. This is simply
not how humans work when this much power is at stake.
Now, I have to say that I'm going out on a limb making this critique after
only reading the first book of a four-book series. This is absolutely the
type of work for which my reaction and objections could be an intentional
effect created by Palmer in order to spring some unexpected justification
on the reader in book two or three. It's clear that there is some massive
social upheaval on the horizon in this series, and something very strange
is going on with one of the characters and their hold over other people.
Perhaps the reader disbelief is setting up that upheaval. If so, hats off
to her, and that's one of the perils of reviewing books as I read them.
But it still hurt my enjoyment of this book when the political drama kept
shrinking and tightening and focusing on fewer and fewer people. It felt
frankly unbelievable for the political universe of this highly political
book to be this claustrophobic. I wanted it to expand into the space that
should be available to an entire world teeming with fractious and complex
humanity.
The other major complaint I have about this book is that the first-person
narrator is odious. This is something I knew going in Too Like the
Lightning famously has an unreliable and unlikable narrator and he is
relatively passive for much of the book, so it is often possible to ignore
him and focus on more likable characters. I don't necessarily mind an
unlikable or unreliable narrator in this type of story.
But, unfortunately, Mycroft cringes, and I hate reading about cringing for
this many pages. His primary mode of interaction with people is
obsequious, performative fear with a weird, distasteful edge of
manipulation. Again, I think this is entirely intentional on Palmer's
part; we learn some of the reasons behind it by the end of this book, and
I'm sure we'll learn more in future books. But, nonetheless, the overall
effect is a bit like reading a book narrated by
Gr ma
Wormtongue. I can appreciate the narrative role of that character without
wanting to spend this much time in his head.
I have very mixed feelings about this book. The overall construction is
brilliant; it's a beautiful puzzle of oddity and alienation that provides
great fun for the type of science fiction reader who wants to work out the
rules of a strange society without a lot of infodumping. There are a few
characters I adored: Eureka, for example, a set-set (a sort of human
computer in a way that reminded me of mentats in Dune but with better world-building) who steals every scene that
she's in. I was very invested in the world-building, fascinated by the
Utopians, and want to learn more about what's going on.
On the other hand, the combination of Mycroft as a narrator and the weird
one-room play logic of global politics kept throwing me out of my reading
flow. It took me about a month to finish this book. The science fiction
and political fiction aspects of the story interested me more than Bridger
and whatever is going on with J.E.D.D. Mason, and I'm worried that my
least-favorite aspects will be central to the rest of the story. I was
enjoying a smaller percentage of the scenes by the end of the book than I
was at the start, which is not a great sign.
And yet, the ending absolutely worked on me. I don't want to stop here! I
will probably pick up the sequel, but I think it's going to take me a
while to brace myself for it.
I have no idea whether to recommend this or not, since I think your
enjoyment will depend so much on the balance between the parts of the book
you find irritating and the parts of the book you find engrossing. I'm
fairly sure most readers will find a little of both, but I have no idea
how to predict their relative weight. If you like cognitive estrangement,
this is great; I understand why so many science fiction reviewers rave
about this book. If you need to like the first-person protagonist, uh,
good luck. Maybe you'll have more tolerance for cringing than I do.
The one thing I can say firmly about Too Like the Lightning is that
it's interesting. It may be worth reading just to see how people
are stretching the genre, even if you end up not liking the effect. But be
warned that this book does not so much end on a cliffhanger as suddenly
stop at some random, nondescript point on the road leading to the cliff.
The ending is deeply unsatisfying; you will need to read more if you want
to understand what's going on.
Followed by Seven Surrenders.
Rating: 7 out of 10
One of the primary design principles for Linus Torvalds when creating Git is that it should be a distributed system, capable of working with multiple remotes. However, most people seem to use Git with just one remote, typically having GitHub or GitLab as the only 
As a graduate of a liberal arts university, I wound up, unsurprisingly, taking a
lot of classes in every possible academic discipline. Thinking back to the
person that I was going into university, I don t think I would have chosen to
take them after all, my degree was in the sciences; I d have been stoked to
do nothing more than wall-to-wall computer science until I ran out of
coursework and filled the rest of my hours with research and independent
studies with my professors (which, I guess I did actually do, just not as
much as I would have otherwise).
Even this blog s CSS theme (now 18 years old and starting to look it) was
something I wrote after receiving and repeatedly re-reading a tattered
third-hand copy of The Laws of Thought by George Boole. It was gifted to me
by a college friend majoring in philosophy while I was crashing at his rental
on the beach. He gave it to me because he knew I liked that shit and, while
it was definitely computer science in nature, I would not have read it
otherwise. I have vivid memories of sleeping on his la-z-boy surrounded by
towers of books he was working through. He went on to do incredible work,
getting his PhD, doing research, brilliant writing his death in 2023 has
robbed humanity of more time with him. The Laws of Thought sits behind me in
my office, and is one of my most valued possessions.
My friends mean more to me than I could ever express, and I definitely don t
show it enough. Each of my classes made me a more thoughtful person. My
professors made me a better, more well-rounded person and a person who
attempts to live up to the oft repeated credo of being men and women for
others . I did the best I could, even though I was never a particularly good
student. Without liberal arts, I don t think I would, dispositionally, have
been capable of pushing myself to this day to continue to learn on nights
and weekends, for no reason other than wanting to learn. I refuse to stop
expanding my perspective, and do my best to approach new problems with as much
humility and curiosity as I can muster.
Every few days for the last year or so, I have been thinking back to a reading
assignment from my junior year that, at the time, I thought was a borderline
throwaway filler assignment for the class. The reading is an essay from 1947 by
the french existentialist philosopher and libertarian marxist, Jean-Paul Sartre
(as some of the more erudite may have now already worked out given the blog s
title), Why Write? . I re-read it this week. It is not filler. I remember this
essay better than some of the classwork I considered more important at the
time.
However, when playing it, it becomes very obvious (as you can likely see) that
there s a fair amount of packet loss (understandable) and probably some packet
collisions taking place.
After tweaking the local routing tables (and restarting 
Yes! I forgot to make this one a video, so no gif. I did wind up having a bit
if trouble with a few gopher clients and IPv6 support I may send some
patches if I can find the time.
My Debian contributions this month were all 
SIMD bitpacking works like regular bitpacking, but processes 8 
I spent 6 hours working with an AI to write a bash script that would show me both active & inactive reserved DHCP addresses, as well as active leases from pool. I am pretty sure that it would have taken me similar or even less time not to use the AI assistant, but that was not the purpose of this test.