About the project and me Your program crashes. You open gdb and get ?? instead of a stack trace. So you go find the right -dbgsym package, for the right version, for the right architecture, install it, and start again. Debuginfod removes that entire detour: gdb asks a server for symbols by the build-ID baked into the binary. Debusine already built packages, already produced -dbgsym files, and already hosted the archives; it just couldn t answer the question. This summer I made it answer. My project was to add debuginfod server functionality to Debusine so that it not only hosts -dbgsym packages, but also serves their debug symbols over the debuginfod(8) protocol. Debian developers can then debug binaries by setting a single URL that gdb uses to fetch the matching debug symbols. This project took me through design, backend work, an extraction pipeline on the worker, HTTP serving, documentation, and testing from the first blueprint all the way to a live demo on debusine.debian.net.

Initial planning and design changes A design first, in !3030. The proposal submitted for GSoC 2026 was just an overview of how things will work, but in reality there were a lot of design questions which needed to be answered before starting with contribution. Debusine keeps development blueprints in its docs tree, reviewed like code, it s basically a blueprint of what feature or new changes are we going to make. I was assigned the work item #957, which was basically about how the idea of implementing a debuginfod server functionality inside Debusine was initially proposed by a fellow member which later became a project idea under GSoC 2026. My developer blueprint pinned down the four decisions everything else depends on: extraction happens on the worker after the build, symbols are stored as artifacts keyed by build-ID, they re published into suites alongside their binaries, and they re served from the archive root rather than per-suite. Settling that up front meant the design discussions happened in a document instead of across three merged branches. One of those arguments became its own fix. My wording implied symbols were unpacked inside the isolated sbuild environment (the consequence was I was handed a bug to be solved in the first week of contribution period), when they re actually extracted afterwards on the worker, where the build output already sits, a distinction that matters, because doing work inside the unshare environment means extra tooling in the chroot and more ways to affect the build. !3119 corrected it before the wrong model spread into the code.

A new artifact type Artifacts are a major concept in Debusine overall, so as per the developer blueprint we introduced a new artifact which was debian:debug-symbols. It holds every .debug file from one -dbgsym package. Its data is a validated list of lowercase 40-character build-IDs, and each file is stored under its build-ID as the path, so answering what are the symbols for this ID? is a direct lookup, with no path translation in the request handler. One artifact per package rather than per file: a util-linux build would otherwise spray hundreds of artifacts, collection items and relations across the database for no benefit. For implementing debian:debug-symbols artifact, I changed the main models.py file, along with that since it s a norm to write unit tests, all mentioned under !3088.

Publishing workflow and solving a bug Extracting symbols is only useful if they reach the archive people actually install from, so !3180 taught package_publish to follow the relates-to relation: copying binaries into a suite now brings their debug symbols along automatically, with nothing extra for the publisher to configure. Each build-ID becomes its own collection item, for example debugsym:hello_2.10-5_amd64_fcc9064 each carrying the package name, version and architecture copied from the binary, so the item is meaningful on its own without dereferencing anything. Uniqueness is enforced at both the suite and archive level, because the serving URLs are archive-wide and two suites must never disagree about what a build-ID means: republishing an identical file is accepted quietly, while two different files claiming the same ID is an error worth failing on. A partial index on the build-ID keeps the eventual HTTP lookup fast. That looked finished until symbols started arriving in target suites disconnected from their binaries published, but unfindable, because copying items between collections silently dropped their artifact relations, and that relation is the only thing tying the two together. The fix sat one level above my feature, in the generic CopyCollectionItems task that does the copying, and since it was reusable infrastructure rather than anything debuginfod-specific, Colin implemented it himself in !3228. My project needed it to work at all; every other Debusine feature that copies items now gets it for free.

Endpoint and CI tests With symbols in the archive, !3212 added the part users actually touch: GET / scope / workspace /buildid/<build-id>/debuginfo looks the ID up across every suite in that workspace s archive, streams the file, and sets the X-DEBUGINFOD-FILE and X-DEBUGINFOD-SIZE headers the protocol expects. It also handles the two things gdb actually does: a HEAD probe before committing to a download, and ranged requests to pull individual ELF sections instead of the whole file. Scoping it to the archive rather than the suite is what lets one URL cover a whole workspace, so the developer never has to know which suite their binary came from. Every merge request above landed with unit tests, but those only tell you that the pieces behave correctly. What Colin and I wanted was a real gdb fetching real symbols from a real instance, so !3261 adds an autopkgtest that builds a package, publishes it, checks the HTTP headers, then sets DEBUGINFOD_URLS and makes gdb go and get the symbols, wired into the CI integration tests so it runs on every change. It took me a day to learn that skipping the signing worker doesn t simplify that test, it just hangs until the 30-minute timeout, because update_suites needs signing to produce a usable repository. The last piece, !3301 covers the new artifact, the suite and archive changes, the new archive URL, and a how-to for using it. My first how-to draft explained how everything worked and offered four ways to set DEBUGINFOD_URLS; the version that shipped gives one recommended setup and gets out of the way. The same pass trimmed the blueprint down to only what s still unimplemented, since a design document describing merged code is just an obstacle for the next reader.

What s left Only one item on my original plan didn t land: an archive-level build_debug_symbols switch, modelled on Launchpad s equivalent, letting an archive skip building -dbgsym packages entirely by passing DEB_BUILD_OPTIONS=noautodbgsym to sbuild. It was always the stretch goal rather than core scope, landing the extract-publish-serve path solidly mattered more than landing it broadly. The design is written up in the blueprint, and I intend to implement it myself. The other gaps were deliberately out of scope from the start, and the blueprint says so. DWZ supplement files aren t ingested, so packages using compressed debug info may render without the alternate strings table; debugging still works, it s just less complete. Source-file serving runs into the same Debian packaging limits that constrain debuginfod.debian.net today, making it a design question rather than a coding one. Executable serving, the metrics and metadata endpoints, and federation to upstream debuginfod servers were excluded for similar reasons, none of them are needed for Debusine s core use case, and each would have crowded out the parts that are. One open bug is left too. On the last day of the coding period, Stefano Rivera found that publishing ledger and linux was failing, because I had told the database that a build-ID identifies one exact debug file which isn t true in Debian, since dh_dwz runs once per binary package, so when one object ships in two binary packages their .debug files differ while describing identical code. How to fix it is still an open discussion #1582, though it may not land before the formal end of the project. None of that is a handoff. GSoC s timeline is ending, my involvement isn t, I m carrying on with Debusine until both the build_debug_symbols switch and DWZ supplement support are merged, and I expect to keep contributing beyond that. This project got me familiar with a codebase I enjoy working in, and the remaining pieces are mine to finish.

Thanks! The biggest thanks go to my mentor, Colin Watson, whose reviews consistently found the thing I hadn t thought about. He also gave me room to get things wrong first and understand why, which taught me more than being handed the answer would have. Thanks as well to Rapha l Hertzog, Enrico Zini, Stefano Rivera, Carles Pina i Estany and Helmut Grohne and everyone else around Debusine and Freexian for reviews, comments and patience with my questions. Special thanks to Freexian for developing Debusine in the open and for giving me access to test on debusine.debian.net. Finally, thanks to the wider Debian community, whose build-ID and -dbgsym conventions did most of the hard work before I arrived and to Google Summer of Code for providing a platform and the time to do this properly.

Background: Why does DCS need a fast Integer Codec? Debian Code Search (DCS) is a search engine that allows searching all the Open Source source code within Debian, with either literal search expressions or regular expression search queries. A search engine uses an inverted index: a map from term to documents containing the term. Each document is typically represented most efficiently by using an id, so the index consists of many lists of document ids. When searching, it is important to quickly decode these lists to answer the search query. However, there is a point of diminishing returns where the decoding speed, even though it can still be measurably improved quite a bit, no longer influences the overall query duration. From 2012 (its inception) to 2019, Debian Code Search used to use a small index format, and queries were fast because the index was kept entirely in RAM. In 2019, I implemented the new index format, which adds an on-disk positional index. For literal queries (78.2% of DCS queries), querying the positional index on disk is faster than querying the non-positional index in RAM. The efficient encoding of the TurboPFor format makes it possible to fit such an index on a mid-sized Hetzner server, which I rent with two 1 TB SSD disks. The optimized decoder of the C TurboPFor library is what made decoding fast at query time. If you want to dive deeper into the algorithm, see this blog post from February 2019:

TurboPFor: an analysis

Motivation I have recently been looking into speeding up Debian Code Search. As a quick reminder, search engines answer queries by consulting an inverted index: a map from term to documents containing that term (called a posting list ). See the Debian Code Search Bachelor Thesis (PDF) for a lot more details. Read more

If you want to learn more about the positional index, see this blog post from September 2019:

Debian Code Search: positional index, TurboPFor-compressed

Over the last few months, I have been developing a new index format for Debian Code Search. This required a lot of careful refactoring, re-implementation, debug tool creation and debugging. Read more

SIMD in Go For many years, you had the following options for using SIMD instructions in Go:

1. Hand-writing Go assembler code. This is only doable for small functions, for example bytes.IndexByte is implemented with hand-written Go assembly (including AVX2).
2. Generating Go assembler code with tools like Michael McLoughlin s Avo . This is how crypto/internal/fips140/sha256 uses AVX2. While Avo generator code definitely is higher-level than hand-written assembly, it is still too close to assembly for my taste.
3. Use a C library via cgo so gcc or clang compiles SIMD code. Debian Code Search used to use the powturbo/TurboPFor C library via cgo for the last 7 years.

The C TurboPFor library has served us well, but Debian Code Search was always intended to be a project using Go, so I would prefer it if I did not have any C code in the project. Go 1.26 (released in February 2026) introduced the simd/archsimd package:

Go 1.26 introduces a new experimental simd/archsimd package, which can be enabled by setting the environment variable GOEXPERIMENT=simd at build time. This package provides access to architecture-specific SIMD operations. It is currently available on the amd64 architecture and supports 128-bit, 256-bit, and 512-bit vector types, such as Int8x16 and Float64x8, with operations such as Int8x16.Add. The API is not yet considered stable. Go 1.26 Release Notes

For my 2019 TurboPFor analysis, I implemented goturbopfor, a native Go teaching decoder (without any SIMD), because I find Go code easier to follow than C code, especially optimized C code. My implementation was intentionally not optimized so that the code was easier to study. The TurboPFor format/algorithm has a vector-optimized part: bitpacking comes in a scalar variant (bitunpack32) and a vector variant (bitunpack256v32), where the vector variant is used for full blocks (256 values) and the scalar variant is used for remainder blocks (< 256 values). When Go 1.26 was released, I used Claude Code to explore whether my native Go decoder s bitunpack256v32 function (for the vertical vector layout) could be implemented using Go SIMD, and the answer was yes, it was possible and it was faster than without SIMD, but not quite at the level of C TurboPFor. If you let Claude Code try for long enough, it eventually finds enough optimizations (about 10) to match C performance. I don t want to vibe-code Debian Code Search, though, so I figured I would find some time to review the SIMD code at some point and see if I could implement something similar myself. Before I found enough time and motivation to complete said review, I discovered that to not regress real-life query performance by more than 10 to 100 milliseconds (which seems acceptable), I don t actually need to add SIMD code to my teaching decoder at all; it would be sufficient to reduce allocations in my teaching decoder and specialize it per bit width. Encouraged by the possibility of using the optimized native Go decoder in Debian Code Search, I explored whether I could also implement a native Go encoder so that I could get rid of the C TurboPFor dependency entirely. The answer is yes, it is doable in a few days, and it isn t even that much slower: Go is at 76% of C, see Debian/dcs commit e920dc7. The goal I set myself at that point was to see if I could learn enough SIMD to optimize the native Go encoder such that its performance would match how DCS uses C TurboPFor (via cgo). Beating C TurboPFor was possible in 2-3 commits (SIMD and bit width specialization). To my surprise, Claude Fable 5 pointed out that the encoder s block scanning could be done more efficiently using a technique called positional popcount, and that is another 2x speed-up! To be clear: I am not saying the Go compiler beats C here. Certainly, the C compiler can also produce fast AVX512 code and can be used to implement positional popcount. When comparing apples to apples, i.e. backporting the AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a little slower at 1.4x C. This spectacular result (much faster than what DCS had before) got me curious how far I could push the decoder with SIMD after all. I ended up matching/exceeding the cgo version here, too! The rest of this article explains a few classes of optimizations I encountered along the way.

Starting Point When I wrote my goturbopfor teaching decoder, I named its functions to match the upstream C TurboPFor library, but now I want to get away from names like p4ndec256v32 they make sense from the TurboPFor perspective, but for Debian Code Search, we can use cleaner names. Before writing any code, I audited how DCS uses integer compression / decompression.

API design: BlockEncoder, BlockDecoder and streaming In Debian Code Search, we have the following usage patterns:

• Partial Indexing: When a new package (or package version) enters Debian, all of its (text) files are indexed. If the hello-2.12.3-1 package (hypothetically) contained only hello.c with printf("hello!\n");, we would assign document ID 1 to hello.c and store in the partial index that trigrams pri, rin, int, ntf, etc. are all found in doc 1 (hello.c).
• Full Index Merging: The many thousands of partial index files (for each Debian package) are combined into a small handful of large index files: When searching, it would be expensive to consult thousands of indexes. To merge multiple partial index files into one larger index (which can then be efficiently queried), we need to re-encode the partial index files: what used to be document ID 1 in the partial index might be document ID 2531 in the full index.
• Querying (searching): When users enter search queries, these queries need to be answered as quickly as possible. The relevant entries in the full indexes are decoded (in parallel).

For reading the index, we do keep the decoded uint32s fully in memory, so we only need DecodeN(input []byte, output []uint32) (read int), a function that reads len(output) values (uint32) from input and returns how many bytes it consumed. For writing the index (both in partial indexing, and when merging), keeping the entire index in memory is prohibitively expensive, so we need a streaming API, for decoding and for encoding. Ultimately, I converged on the following API:

package pforenc

type BlockEncoder struct  
    // scratch buffers can go here
 

// EncodeBlock encodes len(vals)<=256 uint32s into dest (one TurboPFor block).
func (*BlockEncoder) EncodeBlock(dest []byte, vals []uint32) []byte  

// EncodeN calls EncodeBlock in a loop.
func (*BlockEncoder) EncodeN(dest []byte, vals []uint32) []byte  

type StreamEncoder struct  
  be   BlockEncoder
  vals [256]uint32
  // scratch buffers
 

// if full, you need to call [EncodeBlock]
func (*StreamEncoder) Add(val uint32) (full bool)

// EncodeBlock must be called after all data was [Add]ed.
//
// Write the returned buffer to file or send it over the network;
// it is only valid until the next [EncodeBlock] call.
func (*StreamEncoder) EncodeBlock() []byte  
  if se.n == 0   return nil   // turn an extra EncodeBlock into a no-op
  //  
 

This API (the decoder works similarly) allows us to process data in TurboPFor format without any memory allocations. The types are not safe for concurrent use by multiple goroutines. The zero value is ready to be used. For the streaming API, the result only stays valid until the next call.

Initial Implementation Before we can optimize anything, we need a working decoder and encoder. The decoder already exists: my goturbopfor teaching decoder. Next up, I needed an encoder. Writing a TurboPFor encoder has a delightfully simple starting point: You can encode all values at bit width 32, in little endian, at which point you only need to add a one-byte TurboPFor block header every 256 values and you re done:

func (be *BlockEncoder) EncodeN(dest []byte, vals []uint32) []byte  
  for len(vals) > 0  
    chunk := min(len(vals), 256)
    dest = be.EncodeBlock(dest, vals[:chunk])
    vals = vals[chunk:]
   
  return dest
 

func (be *BlockEncoder) EncodeBlock(dest []byte, vals []uint32) []byte  
  const bitWidth = 32
  dest = append(dest, bitWidth)
  for _, val := range vals  
    dest = binary.LittleEndian.AppendUint32(dest, val)
   
  return dest
 

Of course, this is a terribly inefficient compressor, so after the first commit, the real work starts: implement each block type until the compression matches the original C TurboPFor implementation (same output file size), or in other words: do the reverse of the decoder.

1. The TurboPFor bitpacking block type (bitpacking implementation commit) encodes a bit stream of variable bit width (where the bit width is in range 0 bitWidth 32) in little endian byte order. By scanning all values and choosing the smallest bit width that allows representing all values, this technique saves disk space (compresses).
2. The bitpacking with exceptions block type (bitpacking with exceptions implementation commit) determines two bit widths: one for values, the other bit width for encoding exceptions. This allows choosing a lower bit width (that does not cover all values) compared to the bitpacking block type. A bitmap encodes whether a value has an exception or not.
3. The bitpacking with VB exceptions block type (bitpacking with VB exceptions implementation commit) is a variant which does not use an exception bitmap and encodes exceptions using a variable byte integer encoding. This is more efficient when there are few exceptions (less than 20) or the exceptions are very different in bit width compared to the other values.
4. Lastly, the constant block type (constant implementation commit) stores just one value on disk. This is useful for all-zero or all-one blocks, for example.

I found it interesting to realize that the main work of the encoder is to scan the input values and choose the optimal block type, whereas the actual encoding itself is cheap in comparison. At this point, we can look at performance and see that the Go encoder is at 76% of the C encoder. In all honesty, I could have probably stopped here, but now that the milestone of a viable replacement was reached, I got curious to see how far it would be possible to push the encoder (how much work to reach C speeds?) and afterwards, the decoder, too.

Setup

The microarchitecture level: set GOAMD64 The microarchitecture of a CPU determines which instructions it provides, and that includes not just SIMD instruction sets (like AVX2), but also other useful instructions like LZCNT (Leading Zero Count), which can be used to implement math/bits.Len32 more efficiently, which the TurboPFor encoder needs to call on every input value to determine the ideal bit width. Let s walk through how to set the microarchitecture level when using Go on 64-bit x86 (x86-64). Go uses the GOARCH environment variable to configure the target compilation architecture, and I am using the value amd64 to select 64-bit x86 (AVX2 and AVX512 are instruction sets found on x86-64 CPUs). With GOARCH=amd64, the architecture-specific variable GOAMD64 configures the microarchitecture level for which to compile and Go 1.18 introduced these 4 different levels:

GOAMD64=v1 (default): The baseline.
Exclusively generates instructions that all 64-bit x86 processors can execute. GOAMD64=v2: all v1 instructions,
plus CMPXCHG16B, LAHF, SAHF, POPCNT, SSE3, SSE4.1, SSE4.2, SSSE3. GOAMD64=v3: all v2 instructions,
plus AVX, AVX2, BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, OSXSAVE. GOAMD64=v4: all v3 instructions,
plus AVX512F, AVX512BW, AVX512CD, AVX512DQ, AVX512VL.

In 2026, I generally recommend compiling with GOAMD64=v3 so that functions like bits.OnesCount8 are compiled into intrinsics (POPCNT) instead of using a lookup table. For Intel CPUs, setting GOAMD64=v3 means your programs will only start on Haswell CPUs (2013) or newer; for AMD CPUs that means Zen 1 (2017) or newer. In this specific case (DCS), I am even compiling with GOAMD64=v4. The v4 microarchitecture level requires AVX512, which means AMD Zen 4, Zen 5 or newer (Intel s story is complicated). Luckily, both my main development PC (Zen 5) and the Debian Code Search server (Zen 4) are recent enough. Setting GOAMD64=v4 has little effect on Go 1.27 itself: the only change is that maps use one less instruction (VPBROADCASTB instead of PSHUFB). But compiling with GOAMD64=v4 allows us to move one more feature check from runtime to compile time, see SIMD build tags. It makes sense to set the microarchitecture level in your benchmark setup so that you don t measure the slow fallback implementations. I use export GOAMD64=v4 in my Makefile.

Benchmarking setup Go s built-in testing package contains support for benchmarks which are written in functions of the form func BenchmarkXxx(b *testing.B). The simplest way to run such benchmarks is go test -bench=., but I ended up configuring a few convenience make targets, which write results to bench.txt and compare against baseline.txt (the previous commit s results, usually), using the very useful benchstat tool.

GOTEST=go test

# -count=6 gives p 0.002 in benchstat:
# https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
BENCHFLAGS=-run=^$$ -bench=. -benchtime=200000x -count=6

# use taskset -c1 to always pin to the same single core,
# avoiding accidental scheduling on different cores on
# mixed-core CPUs like the Ryzen 9 9950X3D.
TASKSET=taskset -c 1
BENCH=$(TASKSET) $(GOTEST) $(BENCHFLAGS)

.PHONY: all test bench bench-baseline bench-relative

all: test

bench: test
	$(BENCH)   tee bench.txt
# Compares compression ratio between C and Go implementation
	benchstat -col /impl -row '/n /vals' -filter '-/impl:go-stream .unit:(encoded-bytes)' bench.txt
# Compares performance between C (cgo) and Go implementation
	benchstat -col /impl -row '/n /vals' -filter '.unit:(Mval/s)' bench.txt

bench-baseline: test
	$(BENCH)   tee baseline.txt

bench-relative: test
	$(BENCH)   tee bench.txt
	benchstat -filter '-/impl:go-stream .unit:(encoded-bytes)' baseline.txt bench.txt
	benchstat -filter '/impl:go .unit:(Mval/s)' baseline.txt bench.txt

The encoded-bytes and Mval/s units are custom metrics I am reporting from the various sub-benchmarks, which are arranged such that I can filter / report them with benchstat. The main encoder (and decoder) benchmarks compare 3 different implementations (cgo, Go, Go with the StreamEncoder API) with a number of benchmark cases that are designed to cover the different block types and contain a similar mix of values as what we see in Debian Code Search:

// reportMetrics adds Mval/s and encoded-bytes metrics to all benchmarks.
func reportMetrics(b *testing.B, n int, nencoded int)  
   b.ReportMetric(float64(nencoded), "encoded-bytes")
   b.ReportMetric(float64(b.N*n)/1e6/b.Elapsed().Seconds(), "Mval/s")
 

// BenchmarkEncode/n=<N>/vals=<testcase>/impl=<c go go-stream>
//
// e.g. BenchmarkEncode/n=2048/vals=one-constant/impl=go-stream
func BenchmarkEncode(b *testing.B)  
   for _, tc := range allBenchCases()  
     n := len(tc.vals)
     b.Run(fmt.Sprintf("n=%d/vals=%s", n, tc.name), func(b *testing.B)  
       b.Run("impl=c", func(b *testing.B)  
         b.ReportAllocs()
         var encoded []byte
         buf := make([]byte, turbopfor.EncodingSize(n))
         for b.Loop()  
           encoded = turbopfor.P4nenc256v32Buf(buf, tc.vals)
          
         reportMetrics(b, n, len(encoded))
        )
       b.Run("impl=go", func(b *testing.B)  
         b.ReportAllocs()
         var be BlockEncoder
         var encoded []byte
         buf := make([]byte, 0, turbopfor.EncodingSize(n))
         for b.Loop()  
           encoded = be.EncodeN(buf, tc.vals)
          
         reportMetrics(b, n, len(encoded))
        )
       b.Run("impl=go-stream", func(b *testing.B)  
         b.ReportAllocs()
         var se StreamEncoder
         var encoded int
         for b.Loop()  
           encoded = 0
           for _, val := range tc.vals  
             if se.Add(val)  
               encoded += len(se.EncodeBlock())
              
            
           encoded += len(se.EncodeBlock())
          
         reportMetrics(b, n, encoded)
        )
      )
    
 

CPU counters: perf Go has included excellent performance tooling for many years, see the Profiling Go Programs blog post (2011) for an example of how to use pprof, a sampling profiler. This profiler can help track down which part of a program runs slow, or where memory allocations happen. Once you identified the slow part of a program, how do you know why it s slow? To learn more about the specific bottlenecks your program encounters, you can consult your CPU s hardware performance counters. For example, you could check the branch predictor counters to see if your program is slow due to a high number of branch mispredicts. On Linux, the perf tool is the best way to access the CPU hardware performance counters. A good starting point for working with perf is the documentation on Top-down analysis with the perf tool , which describes the optimization method that Intel established. In my Makefile, I set up two perf targets:

# GOTEST and TASKSET like shown in the earlier benchmarking setup section:
GOTEST=go test -pgo=encode.cpuprof
TASKSET=taskset -c 1
PERFBENCHFLAGS=-test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=200000x

# Use perf(1) to capture AMD IBS (the equivalent to Intel PEBS)
# PipelineL1 is roughly equivalent to Intel TopdownL1
perf:
	$(GOTEST) -c
	$(TASKSET) perf stat -M PipelineL1 ./pforenc.test -test.run=^$$ $(PERFBENCHFLAGS)
	sudo perf record -F 4999 -e ibs_op// --call-graph fp ./pforenc.test -test.run=^$$ $(PERFBENCHFLAGS)
	sudo chmod 644 perf.data

# 488281 iterations   2048 values = 1.000e9 values, so counter/1e9 = per value.
perf-per-value:
	$(GOTEST) -c
	$(TASKSET) perf stat -x, -e cycles:u,instructions:u,branches:u,branch-misses:u ./pforenc.test -test.run=^$$ -test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=488281x 2>&1 >/dev/null   awk -F, ' printf "%-16s %6.2f /val\n", $$3, $$1/1e9 '

The perf-per-value numbers are high level numbers that indicate how much work the implementation is doing. Reducing the number usually increases speed. To see the counters for each instruction (and source code lines), I use make perf, followed by perf report. A quick shortcut is perf annotate, which directly shows the hottest function.

Optimizations (scalar) Let s first see how far we can get without reaching for SIMD instructions. (The examples are not necessarily in commit order, but cherry-picked for clarity.)

Profile-Guided Optimization (PGO) PGO stands for Profile-Guided Optimization and is a feature that Go introduced as a preview in Go 1.20 (released in February 2023) and shipped as ready for general production use in Go 1.21 (released in August 2023). The idea is to capture a CPU profile that records where your program spends most of its CPU time, which you then provide to the Go compiler to give it more data to make better decisions. Most importantly, this way the Go compiler can inline functions much more aggressively than its usual heuristics allow, which does have a measurably positive effect in my series of optimization commits. Another optimization that a PGO profile allows the compiler to do is conditional devirtualization but our TurboPFor code does not use any interfaces. My strategy is to enable PGO before doing any other optimizations, so that we have the full inlining budget available that PGO gives us, and can measure the effect of other commits clearly. Surprisingly, turning on PGO actually decreases our performance (-13% geomean), but a closer investigation reveals that we just got unlucky. Let me explain. Aside from inlining and conditional devirtualization, PGO also influences alignment: The Go compiler sets PCALIGNMAX(64, 31) on the first block of a loop (the loop body ) for all loops in hot functions (per the PGO profile), i.e. Go will insert up to 31 bytes of padding to make the block land on a 64-byte boundary. Documentation like AMD s Software Optimization Guide for the AMD Zen5 Microarchitecture (2024, #58455) explicitly recommends aligning hot loops that way:

[ ] for hot loops, some further knowledge of trade-offs can be helpful. Because the processor can read an aligned 64-byte fetch block every cycle, it is suggested to either align the start of the loop to the beginning of a 64-byte cache line [ ]

Indeed, when compiling with -gcflags=all=-d=alignhot=0 to disable the alignment, performance remains as good as without PGO. How can the padding hurt more than help? The answer is: It s not the padding itself! It s a side-effect of the padding moving instructions to different addresses. In the unlucky arrangement, a macro-fused CMPQ+JGE instruction pair now ends up exactly on a 32-byte boundary. However, the Go compiler ensures fused branch sequences must never cross or end at a 32-byte boundary to fix Intel erratum SKX102 (discussion: Go issue #35881) by inserting NOPs. This NOP padding, unlike the loop alignment padding, is not free; these extra instructions slow down our otherwise dispatch-bound loops. Because the commits after the PGO enabling commit change the code, this unlucky situation is avoided for the rest of the optimization series (by chance).

Reducing memory allocations Memory allocations are quite expensive, at least in comparison to encoding/decoding integers, so I followed my usual strategy of first reducing memory allocations as much as possible. In my goturbopfor teaching decoder, whenever the code needed a scratch buffer, it would allocate it right then and there with make():

// p4dec32 decodes one block of TurboPFor-encoded 32 bit ints
func (d *decoder) p4dec32(input []byte, output []uint32) (read int)  
    //  
  switch blockType  
  case blockBitpackingExceptions:
    bx, input := input[0], input[1:]
    n := len(output)

    exmap := input
    nex := 0 // number of exceptions
    for i := 0; i < n; i++  
      if exmap[i/8]&(1<<uint(i%8)) != 0  
        nex++
       
     
    input = input[(n+7)/8:]

    exceptions := make([]uint32, nex)
    input = input[bitunpack32(input, exceptions, bx):]
    input = input[d.bitunpack(input, output, b):]

    for i := 0; i < n; i++  
      if exmap[i/8]&(1<<uint(i%8)) != 0  
        output[i] += exceptions[0] << b
        exceptions = exceptions[1:]
       
     

    return before - len(input)
   
 

The Go compiler can turn make(T, n) calls into stack allocations, if n is known at compile-time. But, in this case nex is not known at compile-time. We can verify that Go calls into the runtime (runtime.makeslice) by dumping the object code (assembly) with source annotated (-S):

% cd ~/go/src/github.com/stapelberg/goturbopfor
% git reset --hard 49b7c05cc61e77f0257568eb73833467714d2b4a
% go test -c  # go1.27.0
% go tool objdump -S goturbopfor.test   perl -nlE 'say if /p4dec32/ .. /^$/'
TEXT github.com/stapelberg/goturbopfor.(*decoder).p4dec32(SB) /home/michael/go/src/github.com/stapelberg/goturbopfor/goturbopfor.go
func (d *decoder) p4dec32(input []byte, output []uint32) (read int)  
  0x549f60		4c8da42460ffffff	LEAQ 0xffffff60(SP), R12
  0x549f68		4d3b6610		CMPQ R12, 0x10(R14)
  0x549f6c		0f86d9070000		JBE 0x54a74b
  0x549f72		55			PUSHQ BP
  0x549f73		4889e5			MOVQ SP, BP
  0x549f76		4881ec18010000		SUBQ $0x118, SP
  0x549f7d		48899c2430010000	MOVQ BX, 0x130(SP)
  0x549f85		4889b42448010000	MOVQ SI, 0x148(SP)
	if len(output) == 0  
  0x549f8d		4d85c0			TESTQ R8, R8
  0x549f90		0f84a7030000		JE 0x54a33d
  0x549f96		660f1f840000000000	NOPW 0(AX)(AX*1)
  0x549f9f		90			NOPL
[ ]
		exceptions := make([]uint32, nex)
  0x54a4be		488d057bec1700		LEAQ 0x17ec7b(IP), AX
  0x54a4c5		4c89fb			MOVQ R15, BX
  0x54a4c8		4889d9			MOVQ BX, CX
  0x54a4cb		e8f0ddf3ff		CALL runtime.makeslice(SB)
[ ]

An easy speed-up was to avoid allocations through reuse (in goturbopfor). In the DCS pfordec package (with the improved API design), I ended up with a vals [256]uint32 field in the StreamDecoder type, which brings us from 773 Mval/s to 858 Mval/s on the debian-mix:

% benchstat -filter '/impl:go /vals:debian-mix .unit:(Mval/s)' \
  baseline.txt bench.txt
goos: linux
goarch: amd64
pkg: github.com/Debian/dcs/internal/turbopfor/pfordec
cpu: AMD Ryzen 9 9950X3D 16-Core Processor
             baseline.txt               bench.txt               
                Mval/s        Mval/s     vs base                
n=2048        1.089k   1%   1.175k   0%   +7.85% (p=0.002 n=6)
n=2039         974.7   0%   1046.0   0%   +7.32% (p=0.002 n=6)
n=160          434.9   1%    513.6   5%  +18.11% (p=0.002 n=6)
geomean        772.9         857.7       +10.98%

Aside from the speed-up, avoiding memory allocations is generally nice in benchmarks because it removes the garbage collector from the equation and makes it less likely that your benchmarks get other processes OOM-killed on the same machine.

Generics for bit width specialization In general, we want to make it easy for the compiler to understand as much as possible about our algorithm. Consider this bitpack implementation:

func bitpack(dest []byte, vals []uint32, bitWidth int) []byte  
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have int
  for _, val := range vals  
    acc  = uint64(val&mask) << have
    have += bitWidth
    for have >= 32  
      dest = binary.LittleEndian.AppendUint32(dest, uint32(acc))
      acc >>= 32
      have -= 32
     
   
  for have > 0  
    dest = append(dest, byte(acc))
    acc >>= 8
    have -= 8
   
  return dest
 

Let s think through what determines the iterations and control flow this function uses:

1. The number of input values (vals), but not their actual value.
2. The bit width to pack into (bitWidth).

With a bit of careful rearrangement, we can provide the compiler with both, a fixed number of input values (say, 32), and a bit width, both known at compile time. Why is this worthwhile? Because we can manually unroll the loop, let the compiler eliminate much of the repetition and get much faster compiled code as a result! Let s first fix the number of input values to 32 and rewrite the loop to calculate the position offsets within dest instead of changing dest on each value (with AppendUint32):

func bitpack32Unrolled(dest []byte, vals *[32]uint32, bitWidth int)  
  // only one bounds check for 32 values
  dest = dest[: 4*bitWidth : 4*bitWidth]
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have, pos int
  // Manually unrolled loop starts here.
  // Each iteration is identical except for the vals[x] index.
  acc  = uint64(vals[0]&mask) << have
  have += bitWidth
  if have >= 32  
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
   

  // vals[1] .. vals[30] elided for brevity

  // Each loop iteration is 8 lines of Go code, so for 32 input values,
  // bitpack32Unrolled contains 8*32 = 256 lines of code.

  acc  = uint64(vals[31]&mask) << have
  have += bitWidth
  if have >= 32  
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
   

  // have == 0; for all bitWidths
 

Next, we want to specialize not just for 32 input values, but also for each of the 32 bit widths. Can we do better than hand-copying bitpack32Unrolled 32 times (= 8192 lines of Go code)? Yes, we can use Go generics to help us with the code generation! In Go, array types like [4]byte (not slices like []byte!) contain the length of the array as part of their type, meaning [1]byte (an array of length 1) is a different type than [2]byte. Instead of passing the bit width as a function parameter, we can declare 32 different types (one for each bit width) and recover the bit width (at compile time!) from the type system:

type bitWidthT interface  
  [1]byte   [2]byte   [3]byte   [4]byte   [5]byte  
  [6]byte   [7]byte   [8]byte   [9]byte   [10]byte  
  [11]byte   [12]byte   [13]byte   [14]byte   [15]byte  
  [16]byte   [17]byte   [18]byte   [19]byte   [20]byte  
  [21]byte   [22]byte   [23]byte   [24]byte   [25]byte  
  [26]byte   [27]byte   [28]byte   [29]byte   [30]byte  
  [31]byte   [32]byte
 

func bitpack32Unrolled[T bitWidthT](dest []byte, vals *[32]uint32)  
  var zero T
  bitWidth := len(zero)                  // known at compile time
  dest = dest[: 4*bitWidth : 4*bitWidth] // make cap known at compile time
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have, pos int
  // Manually unrolled loop starts here.
  // Each iteration is identical except for the vals[x] index.
  acc  = uint64(vals[0]&mask) << have
  have += bitWidth
  if have >= 32  
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
   

  // vals[1] .. vals[31] elided for brevity
 

When we instantiate bitpack32Unrolled[bitWidthT] with all 32 different types ([1]byte, [2]byte, , [32]byte), the compiler substitutes the bitWidthT type parameter and produces 32 copies of the function, which we can find in our compiled executable with names like github.com/Debian/dcs/internal/turbopfor/pforenc.bitpack32Unrolled[go.shape.[12]uint8]. The shape of a generic type is based on its memory layout, so a shape for [1]byte must be different than the shape for [2]byte. Because the bitWidth is now known at compile time, the Go compiler can generate close to the optimal machine code for each bit width, which we can confirm using go tool objdump. The code is branchless (after the one bounds check per 32 values) and aside from the loads and stores (from/to memory) consists only of shifts and bit operations, all with constant operands:

% go test -c && go tool objdump -S pforenc.test
[ ]
TEXT github.com/Debian/dcs/internal/turbopfor/pforenc.bitpack32Unrolled[go.shape.[28]uint8](SB) /home/michael/dcs/internal/turbopfor/pforenc/bitpackunroll.go
func bitpack32Unrolled[T bitWidthT](dest []byte, vals *[32]uint32)  
  0x660580              55                      PUSHQ BP
  0x660581              4889e5                  MOVQ SP, BP
  0x660584              48895c2418              MOVQ BX, 0x18(SP)
        dest = dest[: 4*bitWidth : 4*bitWidth] // make cap known at compile time
  0x660589              4883ff70                CMPQ DI, $0x70
  0x66058d              0f820b030000            JB 0x66089e
        acc  = uint64(vals[0]&mask) << have
  0x660593              8b06                    MOVL 0(SI), AX
  0x660595              25ffffff0f              ANDL $0xfffffff, AX
        acc  = uint64(vals[1]&mask) << have
  0x66059a              8b4e04                  MOVL 0x4(SI), CX
  0x66059d              81e1ffffff0f            ANDL $0xfffffff, CX
  0x6605a3              48c1e11c                SHLQ $0x1c, CX
  0x6605a7              4809c8                  ORQ CX, AX
                acc >>= 32
  0x6605aa              4889c1                  MOVQ AX, CX
  0x6605ad              48c1e820                SHRQ $0x20, AX
                binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
  0x6605b1              90                      NOPL
        b[0] = byte(v)
  0x6605b2              890b                    MOVL CX, 0(BX)
        acc  = uint64(vals[2]&mask) << have
  0x6605b4              8b4e08                  MOVL 0x8(SI), CX
  0x6605b7              81e1ffffff0f            ANDL $0xfffffff, CX
  0x6605bd              48c1e118                SHLQ $0x18, CX
  0x6605c1              4809c1                  ORQ AX, CX
                acc >>= 32
  0x6605c4              4889c8                  MOVQ CX, AX
  0x6605c7              48c1e920                SHRQ $0x20, CX
                binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
  0x6605cb              90                      NOPL
        b[0] = byte(v)
  0x6605cc              894304                  MOVL AX, 0x4(BX)

Now we need to actually call bitpack32 from the general bitpack function:

func bitpack(dest []byte, vals []uint32, bitWidth int) []byte  
  if bitWidth == 0  
    return dest // no payload, sparse block with only exceptions
   
  if len(vals) >= 32  
    size := 4 * bitWidth
    for len(vals) >= 32  
      existing := len(dest)
      dest = slices.Grow(dest, size)[:existing+size]
      bitpack32(dest[existing:] /*append*/, (*[32]uint32)(vals), bitWidth)
      vals = vals[32:]
     
   
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have int
  for _, val := range vals  
    acc  = uint64(val&mask) << have
    have += bitWidth
    for have >= 32  
      dest = binary.LittleEndian.AppendUint32(dest, uint32(acc))
      acc >>= 32
      have -= 32
     
   
  for have > 0  
    dest = append(dest, byte(acc))
    acc >>= 8
    have -= 8
   
  return dest
 

func bitpack32(dest []byte, vals *[32]uint32, bitWidth int)  
  switch bitWidth  
  case 1: bitpack32Unrolled[[1]byte](dest, vals)
  case 2: bitpack32Unrolled[[2]byte](dest, vals)
  case 3: bitpack32Unrolled[[3]byte](dest, vals)
  case 4: bitpack32Unrolled[[4]byte](dest, vals)
  case 5: bitpack32Unrolled[[5]byte](dest, vals)
  case 6: bitpack32Unrolled[[6]byte](dest, vals)
  case 7: bitpack32Unrolled[[7]byte](dest, vals)
  case 8: bitpack32Unrolled[[8]byte](dest, vals)
  case 9: bitpack32Unrolled[[9]byte](dest, vals)
  case 10: bitpack32Unrolled[[10]byte](dest, vals)
  case 11: bitpack32Unrolled[[11]byte](dest, vals)
  case 12: bitpack32Unrolled[[12]byte](dest, vals)
  case 13: bitpack32Unrolled[[13]byte](dest, vals)
  case 14: bitpack32Unrolled[[14]byte](dest, vals)
  case 15: bitpack32Unrolled[[15]byte](dest, vals)
  case 16: bitpack32Unrolled[[16]byte](dest, vals)
  case 17: bitpack32Unrolled[[17]byte](dest, vals)
  case 18: bitpack32Unrolled[[18]byte](dest, vals)
  case 19: bitpack32Unrolled[[19]byte](dest, vals)
  case 20: bitpack32Unrolled[[20]byte](dest, vals)
  case 21: bitpack32Unrolled[[21]byte](dest, vals)
  case 22: bitpack32Unrolled[[22]byte](dest, vals)
  case 23: bitpack32Unrolled[[23]byte](dest, vals)
  case 24: bitpack32Unrolled[[24]byte](dest, vals)
  case 25: bitpack32Unrolled[[25]byte](dest, vals)
  case 26: bitpack32Unrolled[[26]byte](dest, vals)
  case 27: bitpack32Unrolled[[27]byte](dest, vals)
  case 28: bitpack32Unrolled[[28]byte](dest, vals)
  case 29: bitpack32Unrolled[[29]byte](dest, vals)
  case 30: bitpack32Unrolled[[30]byte](dest, vals)
  case 31: bitpack32Unrolled[[31]byte](dest, vals)
  case 32: bitpack32Unrolled[[32]byte](dest, vals)
   
 

Encoding remainder blocks is quite a bit faster (full blocks use the vertical layout anyway):

% benchstat -filter '/impl:go /n:160 .unit:(Mval/s)' baseline.txt bench.txt
goos: linux
goarch: amd64
pkg: github.com/Debian/dcs/internal/turbopfor/pforenc
cpu: AMD Ryzen 9 9950X3D 16-Core Processor
                           baseline.txt               bench.txt               
                              Mval/s        Mval/s     vs base                
vals=bitpacking-bw1          751.2   3%   1120.5   0%  +49.15% (p=0.002 n=6)
vals=bitpacking-bw2          716.8   2%   1176.0   0%  +64.07% (p=0.002 n=6)
vals=bitpacking-bw7          700.0   1%   1078.5   0%  +54.08% (p=0.002 n=6)
vals=bitpacking-bw1-exc      524.8   1%    736.8   0%  +40.40% (p=0.002 n=6)
vals=bitpacking-bw2-exc      543.7   1%    758.2   0%  +39.46% (p=0.002 n=6)
vals=bitpacking-bw7-exc      566.7   1%    787.7   0%  +38.99% (p=0.002 n=6)
vals=bitpacking-vb-exc       442.6   1%    616.5   0%  +39.29% (p=0.002 n=6)
vals=sparse-exc              532.4   0%    787.8   0%  +47.97% (p=0.002 n=6)
vals=sparse-vb-exc           408.9   1%    597.8   0%  +46.20% (p=0.002 n=6)
vals=debian-mix              559.5   0%    783.8   9%  +40.09% (p=0.002 n=6)

This performance win comes at the cost of binary size increase. In this case, the .text section (executable code) grows by about 20 KB and the .gopclntab section grows by another 26 KB. Definitely a price I am very willing to pay, but the case might not be as clear in all circumstances.

Optimization: Bigger strides with SIMD Even without reaching for SIMD instructions, a TurboPFor implementation can be made faster by making it work bigger strides. Take this code from the goturbopfor teaching decoder which counts the number of exceptions by checking if each value s bit is set in the exception bitmap:

case blockBitpackingExceptions:
  bx, input := input[0], input[1:]
  n := len(output)

  exmap, input := input, input[(n+7)/8:]
  nex := 0 // number of exceptions
  for i := range n  
    if exmap[i/8]&(1<<uint(i%8)) != 0  
      nex++
     
   
  exceptions := d.scratch[:nex]

We can use the bits.OnesCount64 functions to count ones bits in the exception bitmap, 64 values at a time. For remainder blocks, the rest is processed 8 values (1 byte) at a time:

i := 0
for ; i+8 <= n/8; i += 8  
  xm8 := binary.LittleEndian.Uint64(exmap[i:])
  nex += bits.OnesCount64(xm8)
 
for ; i < (n+7)/8; i++  
  xmb := exmap[i]
  // Clear the bits which do not belong to the exception map:
  if rem := n - i*8; rem < 8  
    xmb &= 1<<rem - 1
   
  // Go compiles OnesCount32 into an intrinsic,
  // but not OnesCount8, so we convert to uint32:
  nex += bits.OnesCount32(uint32(xmb))
 

OnesCount64 uses a 64-bit register. For comparison, AVX2 SIMD instructions use 256-bit registers (= 8 uint32) and AVX512 SIMD instructions use 512-bit registers. In the following sections, we will first set up our build tags for conditional compilation to use a trivial SIMD instruction, then walk through an AVX2 and AVX512 SIMD kernel.

SIMD build tags Let s assume we have the following scalar code: constant.go:

package pfordec

func fillConstant(output []uint32, val uint32)  
  for i := range output  
    output[i] = val
   
 

To increase throughput, we can use AVX2 instructions if they are available on the CPU on which the program runs, i.e. using runtime dispatch. We ll first rename fillConstant to fillConstantScalar (it s now the fallback path): constant.go:

package pfordec

func fillConstantScalar(output []uint32, val uint32)  
  for i := range output  
    output[i] = val
   
 

Next, we ll supply two different implementations (constant_nosimd.go and constant_amd64.go), the latter of which is selected when compiling for GOARCH=amd64 with GOEXPERIMENT=simd (the latter will hopefully be dropped in a later version of Go). The nosimd variant just dispatches to the fillConstantScalar, which will likely be inlined:

//go:build !goexperiment.simd   !amd64

package pfordec

func fillConstant(output []uint32, val uint32)  
  fillConstantScalar(output, val)
 

The constant_amd64.go variant assigns the hasAVX2 global variable by doing a CPUID check and then jumps to the scalar fallback if !hasAVX2, i.e. the CPU is too old:

//go:build goexperiment.simd && amd64

package pfordec

import "simd/archsimd"

var hasAVX2 = archsimd.X86.AVX2()

func fillConstant(output []uint32, val uint32)  
  if !hasAVX2  
    fillConstantScalar(output, val)
    return
   
  val8 := archsimd.BroadcastUint32x8(val)
  i := 0
  for ; i+8 <= len(output); i += 8  
    val8.StoreArray((*[8]uint32)(output[i : i+8]))
   
  // use the scalar implementation for the last <= 7 elements
  fillConstantScalar(output[i:], val)
 

We can go one step further by conditionally compiling const hasAVX2 = true when GOAMD64 is set to v3 or higher (i.e. the amd64.v3 build tag is set). As a practical example from Debian Code Search, we currently need the following checks / dispatches:

code	function	vector instruction set	GOAMD64
encoder	bitpack256v	AVX2	GOAMD64=v3
encoder	exbitmap	AVX512	GOAMD64=v4
encoder	scan	AVX512+VBMI+GFNI+BITALG	n/a
decoder	bitunpack	AVX2	GOAMD64=v3
decoder	bitunpack256v32	AVX2	GOAMD64=v3
decoder	bitunpack256v32Ex	AVX512	GOAMD64=v4

In DCS, the effect is measurably positive, but small.

The 256 uint32 vertical layout First, here is the layout explanation from my 2019 TurboPFor analysis blog post:

In regular (non-SIMD) bitpacking, integers are stored on disk one after the other, padded to a full byte, as a byte is the smallest addressable unit when reading data from disk. For example, if you bitpack only one 3 bit int, you will end up with 5 bits of padding. SIMD bitpacking works like regular bitpacking, but processes 8 uint32 little-endian values at the same time, leveraging the AVX instruction set. The following illustration shows the order in which 3-bit integers are decoded from disk:

The scalar implementation uses an array of 8 uint64 to process 8 values at a time:

func bitunpack256v32(input []byte, dest []uint32, bitWidth int) (read int)  
  mask := uint64(1)<<bitWidth - 1
  orig := len(input)
  var bits uint
  var acc [8]uint64 // accumulator: current+next bits
  for op := 0; op < len(dest);  
    if bits < uint(bitWidth)  
      // read 8 more uint32s
      for i := range 8  
        acc[i]  = uint64(binary.LittleEndian.Uint32(input)) << bits
        input = input[4:]
       
      bits += 32
     
    for i := range 8  
      dest[op] = uint32(acc[i] & mask)
      op++
      acc[i] >>= bitWidth
     
    bits -= uint(bitWidth)
   
  return orig - len(input)
 

The SIMD version also processes 8 values, but without a for i := range 8 loop! One difference is that we no longer have the luxury of using uint64 for acc (holding rest and current bits); because AVX2 registers only fit 8 uint32 (not 8 uint64). Instead, we split acc into rest8 and cur8.

func bitunpack256v32(fullinput []byte, fulldest []uint32, bitWidth int) (read int)  
  dest := fulldest[:256]
  if bitWidth == 0  
    clear(dest)
    return 0
   
  n := 32 * int(bitWidth)
  input := fullinput[:n] // tell the Go compiler how long the input is
  mask8 := archsimd.BroadcastUint32x8(uint32(1)<<bitWidth - 1)
  bitWidth8 := archsimd.BroadcastUint32x8(uint32(bitWidth))
  var bits uint
  pos := 0
  // var acc [8]uint64
  var rest8 archsimd.Uint32x8
  var cur8 archsimd.Uint32x8
  for op := 0; op < 256; op += 8  
    if bits < uint(bitWidth)  
      // read 8 more uint32s
      // acc[i]  = uint64(binary.LittleEndian.Uint32(input)) << bits
      next := archsimd.LoadUint8x32(input[pos : pos+32]).ReshapeToUint32s()
      pos += 32  // input = input[4:]
      cur8 = rest8.Or(next.ShiftAllLeft(uint64(bits)))
      // acc[i] >>= bitWidth
      rest8 = next.ShiftAllRight(uint64(uint(bitWidth) - bits))
      bits += 32
      else  
      cur8 = rest8
      // acc[i] >>= bitWidth
      rest8 = rest8.ShiftRight(bitWidth8)
     
    // dest[op] = uint32(acc[i] & mask)
    cur8.And(mask8).Store(dest[op : op+8])
    bits -= uint(bitWidth)
   
  return n
 

The SIMD version benchmarks about 3x as fast as the scalar version. Another significant speedup is to use generics for bit width specialization for this SIMD kernel so that bitWidth becomes a compile-time constant and the compiler can generate better code.

Positional Popcount For my TurboPFor encoder, I implemented the same techniques as described above:

1. Bitpack full blocks with SIMD (AVX2)
2. Gather exceptions using SIMD (AVX512)
3. Use generics to specialize per bit width

These changes are sufficient to roughly match the cgo performance, but then Claude Fable 5 found another 2x speed-up on top of that! The key observation is that once encoding blocks is fast, the preceding step of scanning the input values to decide which block type to use becomes the bottleneck. Here is the encoder s main encode function, which first does one pass over the input values (scan) and then prices all different block types at all relevant bit widths (requires fast access to the scan histogram):

func (be *BlockEncoder) encode(dest []byte, vals []uint32, layout blockLayout) []byte  
  var stats stats
  scan(&stats, vals) // gathers statistics from every value in vals
  bitWidth := bits.Len32(stats.or)
  if stats.or == stats.and  
    return be.encodeConstant(dest, vals, bitWidth)
   
  n := len(vals)
  // bitpacking is the default, unless we find a more efficient block type.
  bestType := blockBitpacking
  bestB := bitWidth
  best := priceBitpack(n, bitWidth, layout)

  // Walk from high bitWidths to low: to break ties, we prefer
  // the encoding with fewer exceptions (for faster decoding).
  for b := bitWidth - 1; b >= 0; b--   // up to 32 iterations
    nex := int(stats.cnt[b])
    size := priceBitpackExceptions(n, b, bitWidth, nex, layout)
    if size < best  
      bestType = blockBitpackingExceptions
      bestB = b
      best = size
     
    // Over-approximate the number of VB bytes.
    vb := nex + // exceptions using 1, 2, 3, 4, or 5 VB bytes
      int(stats.cnt[b+7]+ // exceptions using 2, 3, 4, or 5 VB bytes
        stats.cnt[b+14]+ // exceptions using 3, 4, or 5 VB bytes
        stats.cnt[b+19]+ // exceptions using 4 or 5 VB bytes
        stats.cnt[b+24]) // exceptions using 5 VB bytes
    size = headerBytes + headerExBytes + payloadBytes(n, b, layout) + vb + nex
    if size < best  
      bestType = blockBitpackingVBExceptions
      bestB = b
      best = size
     
   
  switch bestType  
  case blockBitpacking:
    return be.encodeBitpack(dest, vals, layout, bitWidth)
  case blockBitpackingExceptions:
    return be.encodeBitpackExc(dest, vals, layout, bestB, bitWidth-bestB)
  case blockBitpackingVBExceptions:
    return be.encodeBitpackVBExc(dest, vals, layout, bestB, int(stats.cnt[bestB]))
  default:
    panic("BUG: bestType not implemented")
   
 

I ll show you a slightly shortened version of scan, the function which is the bottleneck:

type stats struct  
  // cnt[n] = how many values where bits.Len32(val)>n,
  // i.e. how many exceptions are required for bitWidth=n.
  // Padded so that cnt[b+24] is always in bounds.
  cnt [32 + 24]uint32
 

func scan(output *stats, vals []uint32)  
  for _, val := range vals  
    for b := range bits.Len32(val)  
      output.cnt[b]++ // b bits are not enough to store val
     
   
 

Let s consider the following 3 example values to understand the resulting cnt:

input	input (bin)	bits.Len32
23	0b0000010111	5
5	0b0000000101	3
666	0b1010011010	10

The resulting cnt exception count histogram would contain (cnt shortened to c):

c[0]	c[1]	c[2]	c[3]	c[4]	c[5]	c[6]	c[7]	c[8]	c[9]	c[10]
3	3	3	2	2	1	1	1	1	1	0

In words, this means that at bit width 10, we could encode all the values without any exceptions. But most values do not need 10 bits, so a bit width of 5 would be more efficient, but requires storing one exception. Encoding at bit width 4 requires 2 exceptions, and so on. The scan function above is intentionally kept simple for illustration. We can make it faster by moving the per-bit-width loop outside the per-element loop. The fast version still needs about 12 instructions per value. With SIMD, we can reduce this to by 8x to only 1.5 instructions per value!

The trick: smear masks enable positional popcount The trick is to turn each input value into its smear mask (imagine taking the first 1 bit and smearing it across the remaining positions). Here are the smear masks for our example:

input	input (bin)	bits.Len32	smear mask
23	0b0000010111	5	0b0000011111
5	0b0000000101	3	0b0000000111
666	0b1010011010	10	0b1111111111

Turning a value into its smear mask is computationally cheap: Go implements BitLen(x) (functions like bits.Len32) by calculating 32 - LZCNT(x). We can calculate the smear mask of a value with ^uint32(0) >> LZCNT(x), i.e. starting with a 32-one-bits mask and shifting it by the number of leading zeros. Now, to obtain e.g. cnt[4], we can count the 1 bits at bit position 4 of all input values. The POPCNT instruction counts bits very efficiently, but it counts one bits within a register, so it counts rows, not columns. Counting columns is called Positional Population Count. I found the following papers that describe positional popcount with SIMD:

• 2019: Efficient Computation of Positional Population Counts Using SIMD Instructions (Klarqvist, Mu a, Lemire) introduces an AVX512 implementation using Carry-Save Adder (CSA) networks, but in my testing, that approach is slower for TurboPFor encoding.
• 2024: Histogramming bytes with positional popcount (GF2P8AFFINEQB edition) (Harold Aptroot) shared a C++ implementation illustrating the technique I am using. If you are not familiar with how C++ vector intrinsics look, take a look at this example.
• 2025: Faster Positional-Population Counts for AVX2, AVX-512, and ASIMD (Clausecker, Lemire, Schintke) improves performance over the 2019 paper, but cites the Aptroot blog post as Future Work , promising even faster processing for smaller inputs.

Positional Popcount: a visual explanation To understand the AVX512 implementation of positional popcount, I found it most helpful to visualize an AVX512 register (512 bits, i.e. 64 bytes). The graphic below uses the Uint64x8 layout, meaning it divides the register into 8 lanes of 64 bits (= 8 bytes) each. This illustration shows the whole process: how uint32s are loaded into an AVX512 register (all 4 of its bytes, in sequence) and where we end up, i.e. the 32 positional popcounts: Let s break down this process into its individual steps. First, we turn each loaded value into its smear mask as explained above. The VPOPCNTB vector instruction calculates POPCNT (1 byte) of 64 bytes at once, but first we need to shuffle the bytes inside the register: in load order, we have a full uint32 (4 bytes), followed by another uint32, per lane. First, we permute the bytes (VPERMB) such that all the first bytes of each value end up in one lane ( transpose the bytes ): Next, we transpose the bits using the GF2P8AFFINEQB instruction, which sounds scary but turns out to be quite flexible for bit manipulation of all kinds. The GF2P8AFFINEQB instruction is also the star of the show in Go s Green Tea Garbage Collector (2025). Here is the bit transpose, shown in the AVX512 register layout (see below for a different layout): I found it easier to understand the transpose step when arranging the 8 bytes of lane 0 from top-to-bottom (instead of left-to-right), because then it looks like a 90 degree clockwise rotation: Now we can use VPOPCNTB to count the bits in all 64 bytes at once: After all loop iterations (processing 16 values each) are done, we add the two groups (first 8 values, second 8 values) to obtain the 32 exception counts:

Positional Popcount: Go SIMD Here is the Go code that implements what I described visually above:

func scanSIMD(output *stats, vals []uint32)  
  ones16 := archsimd.BroadcastUint32x16(^uint32(0)) // 16 32-one-bits masks
  shuffle := archsimd.LoadUint8x64Array(&scanShuffle)
  units := archsimd.LoadUint8x64Array(&scanUnits)
  var acc archsimd.Uint8x64
  idx := 0
  for ; idx+16 <= len(vals); idx += 16  
    v := archsimd.LoadUint32x16(vals[idx : idx+16])
    // Replace all values with their smear masks.
    smear := ones16.ShiftRight(v.LeadingZeros()).ReshapeToUint8s()
    // Transpose: shuffle the bytes, then transpose the bits.
    matrices := smear.Permute(shuffle).ReshapeToUint64s()
    transposed := units.GaloisFieldAffineTransform(matrices, 0)
    // Popcount 64 bytes at once into the accumulator.
    acc = acc.Add(transposed.OnesCount())
   
  // Store the accumulator into output.cnt:
  // Widen the two groups of byte counts to uint16 lanes (so that
  // 128+128 = 256 fits), fold them into cnt[b] for b=0..31,
  // then widen again to the uint32 lanes of output.cnt.
  sum := acc.GetLo().ExtendToUint16().Add(acc.GetHi().ExtendToUint16())
  sum.GetLo().ExtendToUint32().Store(output.cnt[0:16])
  sum.GetHi().ExtendToUint32().Store(output.cnt[16:32])
  // scalar tail for the 0..15 remaining values
  for _, val := range vals[idx:]  
    for b := range bits.Len32(val)  
      output.cnt[b]++
     
   
 

Have a look at the commit introducing positional popcount to DCS for the full code (including shuffle tables and ISA checks) as well as the detailed benchmark results.

Go even faster? The SIMD optimizations I showed above beat the cgo TurboPFor library that Debian Code Search used before. When comparing apples to apples, i.e. backporting the AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a little slower at 1.4x C. Could we make my Go TurboPFor implementation even faster, to truly match the C speed? Yes! But also no. Let me explain:

1. We could use more SIMD instructions to remove all code that still processes one value at a time. For example, in my encoder s encodeBitpackVBExc function. Or we could price all bit widths concurrently in encode. Or in the decoder s exception apply code path.
   But all of these SIMD instructions make understanding (and changing) the code harder, so I am cautious regarding which ones I introduce.
2. A big part of the performance gap is due to Go s bounds checks. While it costs performance, bounds checking is great for safety, so I will not turn off bounds checking. The Go compiler eliminates a number of bounds checks when it understands it s safe to do so. One optimization avenue could be to make the prove pass in the Go compiler smarter to eliminate more bounds checks.
3. When doing mid-stack inlining (proposal #19348) (2017), Go sometimes needs to put NOP instructions into the binary so that it can attach inlining markers. For dispatch-bound functions, these extra NOPs can measurable slow down execution.
4. The Go compiler currently allows specifying the architecture (GOARCH=amd64) and microarchitecture (GOAMD64=v3), but not a specific CPU architecture (like AMD Zen 4). Therefore, CPU-specific workarounds for one vendor affect all the generated code. The specific one I encountered in my code is that the Go compiler emits XORL CX,CX before every POPCNT to break a false-output-dependency from the Intel Sandy Bridge Skylake era, which is unnecessary on AMD Zen CPUs.
   I suspect that Go intentionally does not offer this level of customizability.
5. After all of the above points are addressed, what remains is better code generation in specific cases. To illustrate what I mean, consider the example of incrementing a loop variable, where Go re-derives an index every time:
   Go: POPCNTL; ADDQ DI,CX; LEAQ (base)(CX*4) (3 instructions)
   clang: popcnt; lea rax,[rax+4*rdi] (2 instructions)
   Depending on the specific case, improving the compiler might be easy or prohibitively complex. Often, such improvements are hard to measure conclusively.

Conclusion Go s SIMD support makes available in Go code without having to resort to cgo or assembly a powerful part of modern CPUs which allows speeding up the kind of computation that TurboPFor needs by an order of magnitude! I found it very valuable to use a coding agent (Claude Code, with Opus 5 and Fable 5 in this case) to help with the many tedious parts of such performance work (and still it took me weeks!). The LLM can read objdump output much faster than I can, can see patterns and correlations I might never identify, never becomes frustrated after a compiler error or runtime panic, and never runs out of patience to run one more experiment, as long as I give it measurable and reachable goals. The performance of the SIMD code which one can get from the Go compiler is pretty close to what a good C compiler like clang provides. The CPU performance counters show value decoding speeds of 7 instructions/cycle (IPC) on a machine where the maximum is 8 IPC. To me, SIMD support is a very welcome addition to Go.

Why LLMs are so deceptive The output of an LLM often looks very compelling, professional and correct. Humans have evolved to trust or distrust new information based on easy to detect secondary factors like what authority the speaker holds, or how confidently and eloquently the message is conveyed. Humans are however very bad at fact-checking and cross-referencing new information, as it requires a lot of effort, and humans like saving energy and being as lazy as possible.

Information asymmetry The less you know about something, the easier it is to fool you on that topic. Nobel prizes in economics have been given in for research on how information asymmetry distorts markets and leads to suboptimal outcomes. In the field of software engineering we have now witnessed a flood of aspiring software developers using AI to create software that looks like it might work, but that is actually full of flaws. These people are well-intended, but they simply lack the expertise to understand what they are actually doing, and don t possess the necessary judgement to decide when an LLM spits out something truly useful and when it is creating mostly garbage. This asymmetry in expertise I think explains the majority of the conflict currently witnessed in open source projects the senior developers are flooded with requests to review code that is bad and a waste of time for everyone involved, while availability of AI grows the pool of people who could contribute and create more code slop at an ever-increasing speed. The information asymmetry could to some degree be evened out if seniors teach juniors to do software engineering well, but it is of course not feasible to quickly mass educate everyone. Also, it seems that many don t want to learn but instead expect to have all understanding outsourced to LLMs. Many seniors have noticed this and have stopped teaching juniors as the seniors don t like the feeling of having their time wasted by teaching people who don t want to learn. Juniors probably all understand that it would be better to learn to design and write software yourself, but using LLMs just feels too easy. I can fully relate to why people choose to take the path of least resistance. Unfortunately, that path often leads to a dead end.

Humans fall too easily for anthropomorphism The human brain is wired to think that inanimate objects are alive and have feelings. Small children talk to their stuffed animals as if they were real, and lots of adults experience feelings of things happening in their surroundings due to some acts of gods or elves being angry or whatever. When we see a machine writing just like a human, or even more convincingly hear it talk and respond to our talk like a living thing, our brain automatically starts assuming it is a living thing with intelligence and feelings. The fact that these creatures live in the abstract cloud and only appear through a portal we hold in our palm and behave in a way that was designed for maximum engagement makes the illusion even stronger. I recommend people try out running LLMs locally on their laptop to see the raw thing spitting out tokens and have some of the illusion shattered. Also stop saying please to an LLM. It does not have any feelings.

Understanding temperature In my experience understanding the concept of temperature in LLMs helps see why an LLM might confidently generate a plausible-looking but totally wrong code change. The large language models are statistical machines that, based on the input (previous tokens) to the neural network, try to predict what to output (next token). When running an LLM, if the temperature is configured to be zero, the output is very predictable and always follows the paths of the strongest connections (a.k.a. weights) between nodes and layers of the neural network. Unlike in living creatures where the brain learns and changes all the time, the weights of an LLM can only change during training. When an LLM is in normal use (during inference, generating next tokens) the weights are fixed, and if temperature is zero, the answer to a specific question will always be exactly the same. This is of course a bit boring and too machine-like, so typically LLMs have a bit of temperature set, which introduces random variation in what connections the neural network traverses. Again, I recommend people try running small LLMs locally where temperature and other settings are fully exposed and configurable to see this themselves. It is a good antidote to falling for the illusion that LLMs would actually be intelligent.

Why benchmarks don t tell the whole story If LLMs continue to produce so much garbage, why are benchmarks showing that they are constantly improving? AI models are indeed improving all the time. For example the CAIS AI dashboard visualizes how frontier models have evolved in the past few years. However, the best models still have a pass rate of only about 50% on the Humanity s Last Exam. On SWE-bench the best model today resolves just under 77%. That means there is a significant number of times when the AI is wrong. This matches my personal experiences, and the renowned Greg Kroah-Hartman recently wrote on the Linux developers mailing list that even with the best of the current and next generation tools, at least 1/3 of the results they generate are flat out wrong or harmful . When generating cat videos the error rate does not matter, but in engineering, things absolutely must be correct. Sure, humans also make mistakes, but well educated and properly incentivized humans are so much more capable than LLMs in many regards. We can achieve complex things that work reliably, such as operating worldwide commercial air traffic without planes falling down every day. There are currently a lot of humans who are incentivized to maintain the narrative that general artificial intelligence is coming soon and will take over everything. In fact, the whole financial system is currently skewed towards such a vision because the promise of falling labour costs and increased profits and monopolistic control of everything attracts capital like nothing before. In this environment we need to remember that machines and economic systems are ultimately servants of humans, and not the other way around.

It s just a tool LLMs are not a scam, but a useful tool and technology that has its uses. But the idea that AI has or will surpass humans any time soon in either capabilities or efficiency is simply not true, and we should listen to the people who created humanity s so far most complex systems (computers and software), who are saying that LLMs are in many cases so bad, that it might be better to ban them in certain places completely for the time being than to waste far more valuable human time on reading the text and code they generate. The time asymmetry is not a new phenomenon as there has been various script kiddies for a long time. As an example, a person running a memory leak scanner without understanding the results and spending 10 minutes to file a bug report could force an open soruce maintainer to spend an hour on proving and explaining that the finding is false. What is new is how much the AI users blindly trust the outputs they get, and open source is uniquely vulnerable as there are no managers protecting developers use of time.

What I do, recommend, and expect to see in the next stages I am using AI tools daily, and constantly experimenting with new models and new ways to use them. Sometimes they work, and often they don t. Sometimes looping AI on itself can make it fix its own errors, but sometimes it just gets derailed and will never arrive at the correct solution. When an LLM fails to make a calendar entry for the right time based on reading my email it is easy for me to spot that it is wrong. I try to avoid using LLMs for anything where I can t exercise judgement myself on whether the result was correct or not. I also really hope that other people would not send me anything where their own effort was less than the effort I have to make reading and understanding it. This principle is not new many have heard the requirement that reading code must require less effort than what it took to write it. I have always kept a high bar on software code and asked fellow developers to make sure their code is well structured, easy to follow and documented. LLMs unfortunately make it easier for people to cheat in this regard, but if cheating is easier, maybe the punishment and deterrence needs to be higher now too. Now with many open source projects adopting policies that put guardrails on AI use, I expect we will soon start witnessing cases where the policies are enforced and it will be interesting to see how violations are judged. As a society we might also need to develop new social standards and rules in what is acceptable treatment of other humans in human-to-machine interactions, and perhaps also new standards in showing what humans are responsible for what machine as the machines start acting more and more independently. I encourage people to take part in these discussions, and in case of doubt, err on the side that favors real human interactions. Contrary to what many business people seem to think, and even though I am in general a techno-optimist myself, I don t feel there is any need to rush with AI adoption.

As a Debian developer, I have had to cast a vote for the General Resolution named LLM usage in Debian (progress report here). This was not an easy task for me

It s a good thing that the vote is secret so that people are not scared of voting according to their own beliefs. I have Debian friends on the whole spectrum of opinions that are represented here, and I hesitated twice on sharing my own thoughts for fear of alienating my relationship with them. But in the end, we all make efforts to respect the opinions of those who are not thinking like us, and it s precisely that willingness to work together towards a solution that is acceptable by the majority that makes Debian so strong. So here s the train of thoughts that I followed to cast my vote.

The difficulty for me was to reconcile the political statement that I want to make and my desire for this vote to not be (too) divisive for the Debian community, and to make sure we are not putting off newcomers with choices that might be hard to stand by in the long term.

So let s be clear : if I had a magical wand to make AI and LLM disappear, I would use it for that purpose, since at this point in time I don t believe that the benefits outweigh the costs that the AI race is inflicting on us. If I were a political decision-maker, I would forbid the construction of new data centers unless they also build renewable energy infrastructure to cover for their additional energy consumption. I would also legislate so that AI companies have to document what material they used to train their models, and I would forbid scraping for that purpose, and build ways for those companies to buy copies of properly-sourced training data. That is to say, I don t like the way LLM are built by the players in that market, I m pretty scared of the ecological impact of what those players are doing, and I m certainly worried about the long term effect that LLM will have on society as a whole.

Nevertheless what brought me to Debian is the ability to experiment and contribute to something useful with cool technologies, and as a computer scientist, the potential of LLM done right is hard to ignore. Given what we have seen already, I expect that LLM will empower (a part of) the next generation to learn IT, computing and even Debian packaging. Completely refusing the use of LLM is likely to make it harder for us to attract new contributors. In fact, we have already seen people inside Debian that would likely stop contributing if they are now forbidden to use LLM. I know there are likely others that will quit Debian if we accept it too, but I hope we can find a middle-ground where such persons can decide that LLM are not welcome in the small corner of Debian that they are in charge of

In the end, I decided that answering clearly the question Shall we accept LLM contributions ? was more important than making the political statement about the current state of affairs in the AI landscape, both because I believe that Debian statements have a negligible impact on policy-makers, and because historically Debian has grown by staying close to technical excellence and relatively far from politics, except when it comes to the way we handle people. And as much as I care about climate change, I don t see how bringing this up in the context of a Debian statement is helping its cause.

More concretely, it gives the following ranking (in decreasing order of importance):

• B, D: those two choices are the clearest to express Yes we should accept LLM contributions and still acknowledge concerns about the way AI is built today
• F, H: those two choices do not forbid LLM usage but discourage their use and clearly voice the concerns
• E: this choice is basically the statu-quo and fails to acknowledge the concerns, but it does not forbid LLM usage
• None of the above
• G, A, C: those choices forbid LLM usage in various ways

I don t know what option will win, but assuming that LLM-assisted contributions are allowed, I believe that it would be helpful to have further statements to clarify a few things:

• Even if Debian as a whole doesn t want to ban LLM-assisted contributions, each maintainer or each team shall be free to forbid LLM assisted contributions in the parts of Debian that they are maintaining
• We should discourage usage of LLM provided by players with unethical behaviors (not sure if there are good players but well )
  

• Understanding and decision advantage
• Electromagnetic and information advantage
• Planning and automation
• Autonomous systems

Also in this series:

• The Four Horsemen of the LLM Apocalypse
• The people vs the AI overlords

The Debian vote context This is not a coming out. I am not using LLMs on a daily basis, and this blog is, again, written out of my cold dead hands in a dying world, with over-engineered hardware and (to a certain extent, hi Emacs!) software, powered by 100% green energy built on stolen land. There is a vote going on in Debian. If you're unfamiliar with it, you can catch up at LWN. So far I've essentially said "LLM is bad" which is not a very balanced or useful opinion. Obviously, people are using LLMs, sometimes unknowing or unwillingly, and we need to take that into account. Furthermore, there has been many different blog posts on Debian planet about this. Some that I found balanced, good summaries, even if I didn't fully agree with them, at least some did the basic civil service of being short. But others were just not only Wrong but also so long that I couldn't finish that I just had to write something.¹ This is not an explanation of the ballots, nor how I will vote. This vote is Debian's failure of framing that debate in a reasonable way: we have 8 options on the ballot with many duplicates. We have failed to do the hard work of summarizing and aggregating options into a meaningful set. I doubt the final vote will represent a readable position we can rally around. I have not read the two months of debates on the topic either. Normally, before voting, I take a cursory look at the debate to see points of view I might have missed. But in this case, it will just make me sad, add noise, and I'm already pretty sure on where I stand on this. So let me describe how I use LLMs and how I think they fit in our work, as computer engineers and hobbyists.

My LLM use

Debian Packaging An astute reader has pointed out that I maintain a package in Debian made to use Anthropic. It's actually multiple packages:

• llm: a CLI utility and Python library for interacting with Large Language Models, with OpenAI as its default API backend
• llm-anthropic: a plugin for llm which allows me to talk to Anthropic's API instead of OpenAI
• anthropic-sdk-python: the SDK llm-anthropic requires to do its work

As I previously explained in response, I am not entirely comfortable with this work: it's a compromise. In fact, I first uploaded llm to the contrib section of Debian, where we keep software that depends on other non-free software, but I was told that, since yt-dlp was in main, llm belonged there as well. So I moved it to main, alongside similarly controversial tools like llama.cpp or the python-openai library.

OpenAI and Anthropic usage An important part of my work is technology watch. I keep tabs on thousands of (new and old) software projects, follow news, and generally try to keep my skills up to date. It's a pretty impossible race, especially as I grow older, but I still think I'm doing the right choices in my job. Testing large language models is part of that work. At first, I was using ChatGPT's web interface, but it was annoying to copy-paste things into a browser, so I looked for different interfaces. For a while I tried gptel, a "simple, extensible LLM client for Emacs" but I found it kind of terrifying. Giving a LLM control over an Emacs buffer seems like a security nightmare, so I stopped doing that. So I use the llm command-line tool to talk to Anthropic's API. I started that in the summer of 2025, when I bought 20$USD of API credits. Before that, I paid for a ChatGPT subscription and then OpenAI credits, which expired and sent me over to Anthropic, which seemed then to have better ethics. As it turns out, Anthropic is also happy to work for the US military (which is a big red line for me). Anthropic also won't let you talk about the genocide in Gaza, it is destroying physical books, and is blackmailing us to use their product for security coverage. Needless to say, Anthropic and "Claude" are not my friends, but they seem like the lesser evil in current "frontier models". So I have renewed, a couple of weeks ago, another 20$USD of API credits with Anthropic.

Actual prompts and responses So what does 20$ give you at Anthropic anyways? What am I using LLMs for and how? The neat thing with llm is that everything is logged in a sqlite database, so there are some answers that are easy to get:

> llm logs status
Logging is ON for all prompts
Found log database at /home/anarcat/.config/io.datasette.llm/logs.db
Number of threads logged:   7
Number of turns logged:     12
Number of legacy conversations: 543
Number of legacy responses: 970
Database file size:         9.61MB

That is 10MB of logs, with about a thousand prompts. My logs go back to 2024-03-07, a little over two years ago, and include a mix of Anthropic and OpenAI responses. I used it more in 2024 than 2025, and if the trend continues, I will have used it less in 2026 again:

> llm logs list -n 0  --json   jq -r .[].datetime_utc   sed 's/-.*//'   sort   uniq -c 
    527 2024
    357 2025
     98 2026

It looks like about 10 prompts per month right now, down from a peak of about 60 per month in 2024. It's pretty difficult to analyze those actual logs to get more patterns and I won't run the prompts through a model again to process them.

How I'm using models now At first, I was using it partly for benchmarking model's capabilities, like Simon Willison does with his pelicans, clearly not trusting its output. But I was impressed by the capacities of the Claude Opus 4.5 model when it wrote this script in January. Impressed, but also scared: it's the first time I felt I could delegate the entirety of my programming to a model. Just run the code, if it works, it works, right? So what do I use it now? As an example, here are the 10 last prompts in my history:

1. there is now Claude 5, and a fable model, maybe you know about it?
2. impress me
3. not impressive, i already know all of this
4. chat
5. in postfix, i have a 300k mailing that happens regularly here. normally, it delivers within about...
6. is there a way i could have drained the maildrop queue faster without removing the milter?
7. the problem was that rspamd was timing out on the FUZZY_CALLBACK check. how do i disable that?
8. how do i disable all spam checks? i just want rspamd to add dkim signatures
9. how do the default_destination_concurrency_limit and initial_destination_concurrency settings int...
10. mic check

The first one was me trying to confirm which model I am using, which is not always obvious when going through the whole llm stack I've been using. The following two are an attempt at seeing what the model is capable of and I was "not impressed", to which Claude answered that I have a "high bar", which, fair enough. The chat is me failing to use a command line, which shows that perhaps I need to readjust that "high bar", again. The next five are a rather embarrassing debacle in a large Postfix mailing that went sideways, and where I couldn't find an actual Postfix expert of my level to help. The fabled Claude Fable 5 answered rather correctly, but dangerously, that I could empty the queue by disabling the non_smtpd_milters. What Fable (and myself) did not realize is that the milter was also adding DKIM signatures, so while the mailing was expedited, it was done without those precious signatures, which got us promptly blocked at Gmail. We have recovered since, and, thanks to the model and reading the Postfix manual for the hundredth time, that pickup(8) is single-threaded and that we needed to review the architecture of that mailing (and our spam filters) a bit. Many tickets ensued. The last one is a test I did to make sure my last uploads of llm-anthropic and its dependency worked correctly.

Note that the above excludes 5 questions I asked Anthropic while writing this article, where I asked for synonyms and "what nanometer scale are arduino processors built from? how is an arduino CPU printed?", a question which Wikipedia furiously evades providing a good answer.

Those prompts are pretty typical of my LLM use: I'm testing the models to see if they work at all, but also, out of desperation, I fire off a prompt after I fire off questions to colleagues or search engines (in that order). It's often weird edge cases like the Prometheus query language, Python's matplotlib, LaTeX, Elisp, optimizations, and so on. I use models for translation a lot. Being fully bilingual, it is common for me to think of a word in French or English and fail to find exactly the right word for that in the other language. Models help with that, and are also useful to find synonyms. Those are low-token uses that seem pretty innocuous to me, but I realize the irony of this after writing about the tower of Babel.

What I am not using models for I am not using models to write prose. I am not using models to read prose. If it's generated with LLMs, I stop reading. I am not using models to write code, with the exception of that single Python script above. I am generally not using models to review code, with exceptions. If I get stuck on a hard problem, I might feed a piece of code to the model. I repeatedly fed asncounter into Claude to try to fix a performance regression I had introduced. It found micro-optimizations that taught me a thing or two about Python's internal implementations, but overall, it was mostly a waste of time. This was in June 2025, so perhaps now models would fare better. I have not tried again. I am not using LLMs to do Debian packaging. When I can, I manually review the diffs of packages I upload into Debian, still, by hand. I do this for the reasons outlined in The Four Horsemen of the LLM Apocalypse, because I refuse to be complicit in the:

1. aggressive and illegal scraping of the servers I steward
2. world-wide computer hardware shortage (making it, by the way, nearly impossible to run presumably clean local models) and the attack on our job conditions (also discussed in The people vs the AI overlords)
3. death of copyright and free software
4. complication and enshifitication of everything, and the destruction of our communities
5. the imperialist Nerd Reich that wants to take over the world

Like I reluctantly use Intel computers, I do fire off a prompt. But I still hold on to the dream that we can build communities of practice that hold human knowledge collectively and not offload that as a utility to some megalomaniac billionaire.

Their LLM use I am forced into So that's me. Clearly, I'm going against the grain here. Everywhere I look, I see LLM-generated code and projects. Slop and botnets have flooded the web. I use Wadamesh, clearly vibe-coded, because it's the best graphical interface for MeshCore that runs on portable devices. I wish it was made by a human, in a community I could participate in, but it isn't, and I don't. I package the above llm toolset, which is more and more vibe-coded, but I still review the diffs. And I have to say: I trust Simon here. The code is verbose as hell, feels overengineered, and llm feels slow, but it generally works, and Simon is still at the gate. The Anthropic SDK is another thing entirely. The 0.91.0 to 0.120 upload, for example, was nuts:

 806 files changed, 72281 insertions(+), 1478 deletions(-)

I explicitly did not review that entire diff. It feels like there's a lot of garbage there to just have a shim between a proprietary API and Python. But this is the hand I've been dealt.

Larger projects LLM use LLMs are being used in the Linux kernel, Firefox, rsync, Rust, and other places. I don't feel good about this, particularly in Rust, but they at least made a decent policy. I am glad GCC made a policy against LLM contributions and I support the human Emacs project. We need to have a set of foundational tools that are "clean" in the sense that they are built upon a community of people that understand how they are built. Maybe that's naive or even impossible. The Linux kernel and GCC, in particular, are massive projects that have long grown past the scale of a single person's understanding. But the theory was that a community of humans can understand collectively. Now we seem to be throwing up our hands and giving up on that community. That LLMs will just fix the problem, whatever it is. But we're all just one rug pull away from being completely incapable of managing those projects. The argument there is that we'll just switch to local models, but no one is actually doing that. All I see is people use local models as a corner case (for privacy) or as in theory, but in reality, everyone uses the centralized frontier models right now. We just can't fallback. We're in the same situation we were, a decade or two ago, when Microsoft decided it would kill free office alternatives by making Office free for non-profits. It worked: thousands, if not millions of schools, community groups and individuals stopped looking for alternatives (including free software but also "piracy") for Office and embraced what seemed like a generous offer. Now Microsoft pulled the plug and Over 170,000 Nonprofits Lost All Their Data. I'm afraid the rug pull on LLMs will be much worse: never mind that Linus won't be able to use his tireless helper to fix obscure kernel bugs; we're looking at a collapse of the economy so large that we are already talking about bailing out the companies responsible. In a sense, the most striking thing about the Debian vote is it has actually no option to completely refuse upstream LLM contributions. It seems the community has taken it for granted that it's now impossible to build Debian entirely without LLMs. We lost the battle even without a fight, it seems.

A plea for small If it has really become impossible for us to manage the complexity we have built, maybe it's time to stop and think about what we're doing in the first place. We're struggling to even bootstrap our current toolchain! This is one of the things I like the most about working on the mesh: it's low tech, small Arduino devices that is built with decades-old semiconductor processes that is understandable by human beings. Maybe the answer lies more in single-purpose devices like those communicators and simpler multi-purpose computers than what we have now, which is what the permacomputing movement is about. Small is beautiful, let's scale it down.

---

1. and yes, I'm sorry this has gotten this long, I hope you will forgive those 3000 words.

In 2025 I was honored to be selected for the first cohort of Sovereign Tech Fellows, a program by Germany s Sovereign Tech Agency to improve the resilience of the open source ecosystem by supporting maintainers directly (complementing their existing support for larger FOSS organizations). Back in 2025, I was only working very limited hours however, this has changed in 2026.

For the second half of 2026, I am working again as a Sovereign Tech Fellow, but this time with significantly increased hours. After finishing my PhD, I do have time now for new tasks (and new jobs!), and the fellowship presents an amazing opportunity to really advance projects that I maintain or am part of. This also has a very nice effect on contributors and bug reporters, as their feedback gets addressed a lot faster. With some luck, this ultimately will help finding new (co)maintainers for projects as well (although in the age of AI, a lot of how open source used to work is much more uncertain, but that is a matter for a different blog post).

The fellowship is time-limited, so I am intending to make the time I currently have count!

So, what s planned?

I am involved in many projects, but three of them will be getting attention as part of the fellowship. I know I am notoriously slow at blogging, but expect more details on each of them very soon. Here s an overview:

Freedesktop.org, Specifications and Organization

I maintain the Freedesktop Specifications, which is an area of Freedesktop that has traditionally been a bit chaotic. This worked in the past, because Freedesktop was never intended to be a formal standards body, but more a shared space where people could throw a lot of code and ideas over the wall and see what sticks and what people can collaborate on.

While I very much love the spirit of this and want to keep it in some form, we definitely would benefit not just from more formalization and better procedures, but also from better organization of the specifications in general. A lot of conflicts can be avoided by that. I will work on improving procedures, crunching through the (lots!) of pending bug reports and MRs, and to make the specifications site better searchable and accessible (similar to how Mozilla s MDN presents information, but I am not sure if we will get quite that far). I also intent to add a compatibility matrix for specifications, so if a desktop opts out of any one of them (or does not implement them yet) that fact is documented and authors of applications know what they can expect. This will allow us to move a lot faster and avoid a lot of conflict, because there is no implicit assumption that everybody will implement everything anymore (which has never been quite true anyway).

Hopefully, this will ultimately result in a Freedesktop that is both a lot more useful for application authors who want to bring their project to Linux, as well as developers of desktop environments who need to see which specifications are available and which ones are current.

In addition to that, I have also worked on a Freedesktop.org website refresh, which is pretty much done in its first iteration (pending sysadmin action). The aim there is to have a more official website, separate from user-contributed wiki content, that showcases what Freedesktop is and which projects are using it for hosting. Once the new website is live, I will also review every page again, archive dead projects in their own section and reorganize the software and specifications directory. Those sections are severely outdated and are missing recent efforts from the community, while still containing long-dead old projects (remember HAL? ).

AppStream

A lot of extra maintenance work will be (has been!) done on it. This includes things such as JPEG-XL support (blog post soon), sandboxed media processing, support for newer specification additions, better OARS integration (and potentially migrating it to fd.o infrastructure), improvements and API stabilization for libappstream-compose and a lot of bugfixing and resolution of issues found by AI code review.

AppStream was originally designed to parse only trusted data from vetted Linux distribution sources this is no longer the case in today s world and in the way Flatpak uses it, so we need to increase resilience of the project.

I am also exploring a project that could vastly improve search accuracy for AppStream. Stay tuned for that.

PackageKit & System Upgrades

Many years ago, people thought we would all migrate to atomic Linux distributions and slowly not need PackageKit anymore. This has not turned out to be the case, and there are still plenty of reasons to use a package-based OS, especially in development environments. At the same time, PackageKit has been basically the same for years, and its older architecture is beginning to show. It being a daemon who s literal job it is to modify the entire system also makes it one of the most security-sensitive components that a Linux system can have, while simultaneously making it near-impossible to sandbox.

My plan is to create PackageKit 2.0 by building on the great foundation of PackageKit 1.0, but modernizing it. This will include simplifying its code and removing a bunch of features that have no more use in modern desktops, while also adding some features that PackageKit never had but that would be useful to expose to frontends (still no to interactivity an terminal-progress forwarding though!). PK 2.0 will also allow me to solve a few design issues that have been worked around in the past, by replacing them with better solutions. This will be a painful transition, as PackageKit 2.0 will break all interfaces PackageKit has and those interfaces have been frozen for more than a decade. However, I do fully expect this change to be worth the effort.

In addition to that, I intend to look into the offline-update procedure again and improve it. The current multi-reboot operation comes with downsides, that newer systemd features such as soft-reboot can alleviate. The end result should be a much smoother, less annoying offline-update experience for users (I especially want to get rid of updates running on system startup, which I consider quite bad from a usability perspective). The new behavior is in the early drafting stages and may need direct support from systemd. I will share more about it once I can.

That s a lot of tasks!

Yes! I will see how far I get. I am moving project-by-project though, to allow me to focus on one project at a time, rather than scattering my attention continuously. Amazingly, this means that the major tasks for AppStream are already almost done, and we are nearing the 1.2.0 release. AppStream got priority, because the new Freedesktop Flatpak runtime will be released soon, and because I want FlatHub/Flatpak to have access to the new AppStream release sooner. Freedesktop and PackageKit are next on the task list.

Either way, a lot of progress is coming if you have any feedback or want to help out, please don t hesitate to reach out! All work is happening fully in the open, so you can also chime in on the respective GitHub/GitLab tasks .

You can also expect blog posts about key features or interesting changes, so stay tuned!

:demo
A1 @0,0 prio=4096
A2 @0,1
A3 @0,2
A4 @0,3
B1 @1,0 prio=8192
B2 @1,1
B3 @1,2
B4 @1,3
C1 @2,0 prio=8192
C2 @2,1
C3 @2,2
C4 @2,3
A1 -- A2 hazard=0
A2 -- A3 hazard=0
A3 -- A4 hazard=0
B1 -- B2
B2 -- B3
B3 -- B4
C1 -- C2 hazard=0
C2 -- C3 hazard=0
C3 -- C4 hazard=0
A1 -- B1 cost=10
B1 -- C1 cost=10
A4 -- B4 cost=20
B4 -- C4 cost=20
Leo @-0.3,0.7 proto=none icon= 
Mia @-0.3,1.3 proto=none icon= 
Joy @0.3,0.7  proto=none icon= 
Roy @0.3,1.3  proto=none icon= 
A2 -- Leo hazard=0 A2:edge
A2 -- Mia hazard=0 A2:edge
A2 -- Joy hazard=0 A2:edge
A2 -- Roy hazard=0 A2:edge
Max @-0.3,1.7 proto=none icon= 
Zoe @-0.3,2.3 proto=none icon= 
Ada @0.3,1.7  proto=none icon= 
Amy @0.3,2.3  proto=none icon= 
A3 -- Max hazard=0 A3:edge
A3 -- Zoe hazard=0 A3:edge
A3 -- Ada hazard=0 A3:edge
A3 -- Amy hazard=0 A3:edge
Eli @0.7,0.7 proto=none icon= 
Jay @0.7,1.3 proto=none icon= 
Kai @1.3,0.7  proto=none icon= 
Ben @1.3,1.3  proto=none icon= 
B2 -- Eli hazard=0.2 B2:edge
B2 -- Jay hazard=0.2 B2:edge
B2 -- Kai hazard=0.2 B2:edge
B2 -- Ben hazard=0.2 B2:edge
Ava @0.7,1.7 proto=none icon= 
Lea @0.7,2.3 proto=none icon= 
Ivy @1.3,1.7  proto=none icon= 
Rex @1.3,2.3  proto=none icon= 
B3 -- Ava hazard=0.2 B3:edge
B3 -- Lea hazard=0.2 B3:edge
B3 -- Ivy hazard=0.2 B3:edge
B3 -- Rex hazard=0.2 B3:edge
Ana @1.7,0.7 proto=none icon= 
Eve @1.7,1.3 proto=none icon= 
Abe @2.3,0.7  proto=none icon= 
Ian @2.3,1.3  proto=none icon= 
C2 -- Ana hazard=0 C2:edge
C2 -- Eve hazard=0 C2:edge
C2 -- Abe hazard=0 C2:edge
C2 -- Ian hazard=0 C2:edge
Ned @1.7,1.7 proto=none icon= 
Lou @1.7,2.3 proto=none icon= 
Fay @2.3,1.7  proto=none icon= 
Sue @2.3,2.3  proto=none icon= 
C3 -- Ned hazard=0 C3:edge
C3 -- Lou hazard=0 C3:edge
C3 -- Fay hazard=0 C3:edge
C3 -- Sue hazard=0 C3:edge

The basics Designed in the 80s, the spanning tree protocol has evolved into a rapid flavor (RSTP) and a VLAN-aware variation (MSTP).² Any sound-minded network engineer knows there are better alternatives, like BGP EVPN VXLAN. Yet, because any switch speaks it, the venerable spanning tree protocol still fills a niche. We focus on RSTP: it replaced the original protocol in 2004. To eliminate network loops, RSTP implements a complex state machine. Timers, link state changes, and the link-local control frames a bridge receives from its neighbors drive its transitions. These Ethernet frames are the Bridge Protocol Data Units (BPDUs). You can watch them in action below: hit the Start button.

:protocol rstp
:tx-hold 10
A1 @0,1
C11 @1,0 prio=4096 icon= 
C12 @1,2 prio=4096 icon= 
C21 @2,0 prio=4096 icon= 
C22 @2,2 prio=4096 icon= 
A2 @3,1
H1 @0,0.2 proto=none icon= 
H2 @0,1.8 proto=none icon= 
H3 @3,0.2 proto=none icon= 
H4 @3,1.8 proto=none icon= 
A1 -- C11
A1 -- C12
A2 -- C21
A2 -- C22
C11 -- C12
C11 -- C21
C11 -- C21
C11 -- C22
C12 -- C21
C12 -- C22
C21 -- C22
A1 -- H1 A1:edge
A1 -- H2 A1:edge
A2 -- H3 A2:edge
A2 -- H4 A2:edge

After some time, the topology converges to a tree: from the root C11, there is a path to each bridge³ and no loop. In the upper right corner, the interface displays a tree icon followed by the time it took to reach this state. Cut a link and see how the protocol finds an alternate path to reach C12 in less than a second. You can stop the simulation, move it forward step by step, reset it to its initial state, or slow it down with the snail mode . Don t worry about all the displayed information: I explain it later. All examples run in your browser, powered by MSTPD an open-source user-space⁴ implementation of RSTP.⁵

Historical interlude Radia Perlman, an inductee of the Internet Hall of Fame in 2014, summarized the ancestor of STP she invented at DEC with this poem, later included in a US patent:

I think that I shall never see
A graph more lovely than a tree.
A tree whose crucial property
Is loop-free connectivity.
A tree which must be sure to span
So packets can reach every LAN.
First, the root must be selected.
By ID, it is elected.
Least cost paths from root are traced.
In the tree, these paths are placed.
A mesh is made by folks like me,
Then bridges find a spanning tree. Radia Perlman, Algorhyme.

Electing the root bridge To build a tree, RSTP first elects the bridge with the lowest bridge identifier as the root bridge. The bridge identifier combines the priority and the MAC address: 8192.6e:2b:10:a0:5f:29. In the example below, S1 and S2 have priorities of 4,096 and 8,192: S1 becomes root. S4 has a priority of 12,288, while S3 keeps the default priority of 32,768:⁶ S4 becomes root. S5 and S6 don t have a specific priority, so the lowest MAC address wins and S5 becomes root.

:protocol rstp
S1 @0,0 prio=4096
S2 @0,1 prio=8192
S1 -- S2
S3 @1,0
S4 @1,1 prio=12288
S3 -- S4
S5 @2,0
S6 @2,1
S5 -- S6

Initially, each bridge advertises itself as root:⁷

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    Root Identifier: 8192.02:00:00:01:00:01
    Bridge Identifier: 8192.02:00:00:01:00:01

Once a bridge receives a BPDU advertising a better root bridge, it propagates this new information to its neighbors.

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    Root Identifier: 4096.02:00:00:00:00:00
    Bridge Identifier: 8192.02:00:00:00:00:01

Assigning roles to ports The second step is to assign a role to each port. RSTP defines five roles, each denoted by a letter:

• root (R),
• designated (D),
• alternate (A),
• disabled (X), or
• backup (B).⁸

Each non-root bridge chooses its root port, the one with the lowest-cost path to the root. Unless you override it, each bridge derives the link cost from the speed: 20,000 for 1 Gbps. In case of equality, the lowest port identifier wins. Each remaining port becomes a designated port if the BPDU it sends is better than the BPDU it receives. Otherwise, it becomes an alternate port. Later, if the root port goes down, the best alternate port becomes the new root port. The tiebreakers for the best BPDU are:

1. the lowest root bridge identifier,
2. the lowest accumulated cost to the root,
3. the lowest bridge identifier, and
4. the lowest port identifier.

:protocol rstp
S1 @1,0  prio=4096 icon= 
S2 @0,1
S3 @2,1
S1 -- S2
S1 -- S3
S1 -- S3
S2 -- S3

In the example above, after convergence, S1 is the root bridge because it has a priority of 4,096, while the other bridges have a priority of 32,768. All its ports are designated ports because the accumulated cost to the root is 0. S2 s port facing S1 becomes a root port because it has the lowest accumulated cost to the root 20,000 vs 40,000. S3 has two ports facing S1, and the one with the lowest port identifier becomes the root port 0x8000 vs 0x8001. The other candidate is an alternate port because the remote port on the link sends a better BPDU, with an accumulated cost of 0. On the segment between S2 and S3, S2 s port wins: while both bridges have the same accumulated cost to the root (20,000), S2 s bridge identifier is smaller 32768.02:00:00:00:00:01 vs 32768.02:00:00:00:00:02.

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 20000
    Bridge Identifier: 32768.02:00:00:00:00:01
    Port identifier: 0x8002

If you cut the active link between S1 and S3, S3 promotes the best alternate port to root port. If you also disable the second link, S3 chooses the remaining alternate port as a root port. But if you disable the link between S1 and S2, S2 needs a bit more work to elect a new root port because it does not have an alternate port. Unless a specific event happens, designated ports send BPDUs every 2 seconds.⁹ If a bridge does not receive BPDUs from its neighbor for 3 consecutive hello periods, it considers the neighbor dead and removes the port information.

Port state transition Each port can have one of three states. The diagram displays a background color for each state:

• discarding (red),
• learning (yellow), or
• forwarding (green).

A root port transitions automatically to the forwarding state. An alternate port stays in the discarding state. A designated port has two options to transition from the discarding state to the forwarding state:

• If the port is an edge port, either through configuration or because the remote device does not speak any flavor of STP, the bridge assumes the device won t participate in the protocol and cannot create a loop. In this case, the designated port immediately transitions to the forwarding state.
• Otherwise, it sends a proposal to its downstream neighbor. If the remote bridge agrees that the received BPDU is better than any other BPDU stored for other ports, it elects the receiving port as its root port and starts the synchronization process: it transitions all non-edge non-synced designated ports to the discarding state to avoid a loop. Then, it sends back an agreement. Upon receiving the agreement, the peer designated port transitions to the forwarding state.¹⁰

:protocol rstp
S1 @1,0 prio=4096 icon= 
S2 @1,1
S3 @0,2
S4 @2,2
S5 @0,3 prio=8192 icon= 
S6 @2,3
H1 @0,1.2   proto=none icon= 
H2 @2,1.2   proto=none icon= 
H3 @2.5,1.3 proto=none icon= 
H4 @2.5,2.3 proto=none icon= 
S1 -- S2
S2 -- S3
S2 -- S4
S3 -- S5
S4 -- S6
S4 -- S3
S5 -- S6
S3 -- H1 S3:edge
S4 -- H2 S4:edge
S4 -- H3 S4:edge
S6 -- H4 S6:edge

In the topology above, H1, H2, H3, and H4 are end devices not participating in the protocol. We configure the ports they connect to as edge ports, so these ports immediately move to the forwarding state. Use the step button to move the simulation forward. The clock moves to 1 second. Step again and S1 and S2 send a proposal to each other. Here is the proposal from S2:

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x4e, Agreement, Port Role: Designated, Proposal
        0... .... = Topology Change Acknowledgment: No
        .1.. .... = Agreement: Yes
        ..0. .... = Forwarding: No
        ...0 .... = Learning: No
        .... 11.. = Port Role: Designated (3)
        .... ..1. = Proposal: Yes
        .... ...0 = Topology Change: No
    Root Identifier: 32768.02:00:00:00:00:01
    Root Path Cost: 0
    Bridge Identifier: 32768.02:00:00:00:00:01
    Port identifier: 0x8001

S1 ignores it: its own root identifier is lower. When S2 receives a similar proposal from S1, it accepts S1 as its root bridge. It also elects the port to S1 as the root port and starts the synchronization process. The two designated ports are already discarding, so no change here. Step again and S2 sends two BPDUs to S1. In one of them, the agreement bit is 1 and the proposal bit is 0. It also shows that S2 accepted S1 as the root bridge and its root port is now in the forwarding state. When receiving this BPDU, S1 transitions its own designated port to the forwarding state. From this point, the link between S1 and S2 forwards user traffic.

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x79, Agreement, Forwarding, Learning, Port Role: Root, Topology Change
        0... .... = Topology Change Acknowledgment: No
        .1.. .... = Agreement: Yes
        ..1. .... = Forwarding: Yes
        ...1 .... = Learning: Yes
        .... 10.. = Port Role: Root (2)
        .... ..0. = Proposal: No
        .... ...1 = Topology Change: Yes
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 20000
    Bridge Identifier: 32768.02:00:00:00:00:01
    Port identifier: 0x8001

Let s look at what happened to S5. Reset the simulation and step twice. S5 exchanges BPDUs with both S3 and S6. Since S5 has a lower root identifier than S3 and S6, it stays the root bridge, while S3 and S6 accept the proposal and elect their root ports. S3 and S6 start the synchronization process. S6 s port to H4 keeps forwarding because it is an edge port. Move one step. Both S3 and S6 send an agreement back to S5, which transitions both designated ports to the forwarding state. Yet, the link between S5 and S3 keeps discarding user traffic! If you look carefully, S3 s port toward S5 is now a designated port, not a root port. During the same step, S3 also receives a better BPDU from S2 with S1 as the root bridge. It elects its port to S2 as the root port and downgrades the port to S5 to a designated port, which stays in the discarding state. On the next step, things get a bit tricky. S3 sends a proposal to S5:¹¹

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x4f, Agreement, Port Role: Designated, Proposal, Topology Change
        0... .... = Topology Change Acknowledgment: No
        .1.. .... = Agreement: Yes
        ..0. .... = Forwarding: No
        ...0 .... = Learning: No
        .... 11.. = Port Role: Designated (3)
        .... ..1. = Proposal: Yes
        .... ...1 = Topology Change: Yes
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 40000
    Bridge Identifier: 32768.02:00:00:00:00:02
    Port identifier: 0x8002

S5 elects S1 as its root bridge and the port toward S3 as its root port. It starts its synchronization process, but the designated port to S6 does not move into the discarding state. Why? That port stays a designated port and its neighbor S6 has already sent an agreement on the link, so the port keeps its synced status. Now, let s step back to look at what happens to S6. At this point, S6 believes S5 is the root bridge. Step once and S4 sends a new proposal to S6. S6 accepts the proposal, elects S1 as the root bridge and the port to S4 as its root port. The role of the port facing S5 changes: from a root port, it becomes a designated port. Because its peer keeps advertising an inferior BPDU on the link, this port becomes disputed and moves to the discarding state. The root port transitions to the forwarding state and the link starts forwarding immediately because S4 s designated port is already in the forwarding state. If we step one more time, S5 and S6 exchange two BPDUs. The one from S5 is better because of its lower bridge identifier. S5 s port stays a designated port, while S6 downgrades its own port to an alternate port. Let s rewind to the start one last time: cut the link between S1 and S2, run the simulation until the topology is stable, stop the simulation, and restore the link between S1 and S2. During the first step, S1 and S2 exchange proposals. S2 elects S1 as the root bridge instead of S5 and the port to S1 as the root port. It downgrades the previous root port to a designated port and moves it into the discarding state. The other designated port stays synced and keeps its forwarding state. At the next step, S2 sends an agreement to S1 and the link between them starts forwarding user traffic. It also sends a proposal to S3, but not to S4. Instead, it sends a regular BPDU to S4. S4 still elects S1 as its root bridge and the port to S2 as its root port. It demotes its previous root port, the one to S3, to a designated port, which transitions to the discarding state because of the root port change. The other alternate port, to S6, also becomes a designated port and stays in the discarding state. The new root port moves to the forwarding state. On the next step, S4 s port to S3 settles as an alternate port after receiving a better BPDU from S3. RSTP is a giant state machine split into smaller ones: bridge detection, port information, port protocol migration, port role selection, port role transitions, port receive, port state transitions, port timers, port transmit, and topology change. Some of them are per bridge, some per port. Each bridge runs an instance. Time, operational port state changes, and the BPDUs it receives from other instances drive the transitions. Being event-driven makes RSTP more efficient but also more difficult to understand.

Placeholder for the Port Information state machine extracted from IEEE 802.1Q-2005, page 182. Pending IEEE authorization for reproduction, this is the blueprint for the Western Australian Government Railways class Msa Garratt articulated steam locomotive.

Topology change notification A bridge populates a MAC address table: it associates each source MAC address with the port that last received it. When forwarding an Ethernet frame, it looks up this table to choose the right port.¹² When a link fails, a connected fridge reachable through one port may become reachable through another one. The affected bridges should flush the MAC addresses they learned, because these entries may now be wrong. For this purpose, RSTP implements topology change notifications using a flooding mechanism. When a non-edge port transitions to the forwarding state, a bridge generates BPDUs with the topology change (TC) bit set. It sends them to all the non-edge designated ports and to the root port. It also flushes the MAC address table on these ports. When a bridge receives such a BPDU, it propagates the notification to all non-edge designated ports and the root port, except the one the notification came from. It also flushes the MAC address table on these ports. In the examples, the BPDUs with the TC bit set to 1 have a red circle.

:protocol rstp
S1 @1,0 prio=4096 icon= 
S2 @0,1
S3 @1,1
S4 @2,1
S5 @1,2
LPT @0.1,2 proto=none icon= 
S1 -- S2
S1 -- S3
S1 -- S4
S2 -- S3
S2 -- S5
S4 -- S5
S5 -- LPT S5:edge

Start the simulation and wait a few seconds for the topology to settle. Stop the simulation and disable the link between S2 and S5. S5 elects the port facing S4 as the root port, which transitions immediately to the forwarding state. Step once and S5 emits a BPDU with the TC bit set to 1:

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x79, Agreement, Forwarding, Learning, Port Role: Root, Topology Change
        0... .... = Topology Change Acknowledgment: No
        .1.. .... = Agreement: Yes
        ..1. .... = Forwarding: Yes
        ...1 .... = Learning: Yes
        .... 10.. = Port Role: Root (2)
        .... ..0. = Proposal: No
        .... ...1 = Topology Change: Yes
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 40000
    Bridge Identifier: 32768.02:00:00:00:00:04
    Port identifier: 0x8002

S4 receives this BPDU. It flushes the MAC address table on the port facing S1: while LPT was previously reachable through this port, it is now reachable through S5. Step once. S4 sends S1 a BPDU with the TC bit set to 1. When S1 receives this BPDU, it flushes the MAC address table on the ports facing S2 and S3. Step once and S1 sends a notification to S2 and S3. Step once again and S2 sends a notification to S3, while S3 does nothing because the port toward S2 is an alternate port. S3 does not flush any MAC address table: LPT is still reachable through its port to S1. If you step a bit more, you will see that some of the periodic BPDUs keep the TC bit set to 1. Each port runs a timer equal to the hello timer plus one second.¹³ The timer starts when the port emits a notification. Until it expires, the port sets the TC bit to 1 in every BPDU it sends. You can also see some periodic BPDUs without the TC bit: they originate from a port that only received a notification and therefore did not arm its timer.

Security RSTP is weak against configuration errors and malicious actors. A bridge not talking RSTP can create a loop. An attacker can insert themselves into the topology to disrupt the service, spy on the traffic, or alter it. To mitigate such problems, you need to identify the edge ports. An edge port connects to an end device, like a PC or a printer. Such devices do not generate BPDUs and cannot create a loop. RSTP defines two related flags:

• When true, AdminEdge initializes a port as an edge port. It defaults to false.
• When true, AutoEdge lets a port become an edge port when it does not receive BPDUs for 3 seconds. It defaults to true.

If an edge port receives a BPDU, regardless of the values of these two flags, it reverts to a non-edge port.

R0 @1.5,1.5 prio=8192
# AutoEdge=true, AdminEdge=false, bridge
S1 @3,1.58
R0 -- S1
# AutoEdge=true, AdminEdge=false, end device
H1 @2.84,2.18 icon=  proto=none
R0 -- H1
# AutoEdge=true, AdminEdge=true, bridge
S2 @2.18,2.84
R0 -- S2 R0:edge
# AutoEdge=true, AdminEdge=true, end device
H2 @1.58,3 icon=  proto=none
R0 -- H2 R0:edge
# AutoEdge=false, AdminEdge=true, bridge
S3 @0.68,2.76
R0 -- S3 R0:edge R0:no-auto-edge
# AutoEdge=false, AdminEdge=true, end device
H3 @0.24,2.32 icon=  proto=none
R0 -- H3 R0:edge R0:no-auto-edge
# AutoEdge=false, AdminEdge=false, bridge
S4 @0,1.42
R0 -- S4 R0:no-auto-edge
# AutoEdge=false, AdminEdge=false, end device
H4 @0.16,0.82 icon=  proto=none
R0 -- H4 R0:no-auto-edge
# Network port, bridge
S5 @0.82,0.16
R0 -- S5 R0:network S5:network
# Network port, end device
H5 @1.42,0 icon=  proto=none
R0 -- H5 R0:network
# AdminEdge=true, bpdu-guard=true, bridge
S6 @2.32,0.24
R0 -- S6 R0:bpdu-guard R0:edge
# AdminEdge=true, bpdu-guard=true, end device
H6 @2.76,0.68 icon=  proto=none
R0 -- H6 R0:bpdu-guard R0:edge

In the topology above, S1, S2, S3, S4, S5, and S6 act as bridges, while H1, H2, H3, H4, H5, and H6 act as end devices:

• S1 and H1 are on a port without a specific configuration: AutoEdge is true, AdminEdge is false,
• S2 and H2 are on a port where AdminEdge is true,
• S3 and H3 are on a port where AutoEdge is false and AdminEdge is true,
• S4 and H4 are on a port where AutoEdge is false.

If you start the simulation and wait about 20 seconds, links to S1, S2, S3, S4, H1, H2, H3, and H4 eventually forward user traffic: none of the flags matter. But what about the two remaining pairs? S5 and H5 connect to a network port. Such a port enables a non-standard feature: bridge assurance. The port transmits BPDUs regardless of its role. If it does not receive BPDUs for 3 consecutive hello periods, it transitions to the discarding state. On the link between R0 and S5, you can see BPDUs traveling in both directions, unlike the other links, where only designated ports send BPDUs. S6 and H6 connect to a port where AdminEdge is true and BPDU guard is enabled. This is another non-standard feature that shuts down a port if it receives a BPDU. In summary, if you expect a port to be an edge port, you should set AdminEdge to true and enable BPDU guard. Otherwise, declare it as a network port.

Why RSTP today? A compelling use case for RSTP today is an out-of-band network for a datacenter, since you can tolerate an outage of a few seconds. The configuration is minimal and you can use cheap switches, like a Cisco 2960X.¹⁴ You need two switches acting as root bridges, and you build several loops to connect OOB switches in each cabinet. This simple design survives one failure on each loop.¹⁵

:protocol rstp
:tx-hold 10
# Root bridges
R1 @0,1 prio=0
R2 @0,2 prio=4096
R1 -- R2 cost=200 R1:network R2:network
R1 -- R2 cost=200 R1:network R2:network
# First loop
C1  @1,0 icon= 
C4  @2,0 icon= 
C7  @3,0 icon= 
C10 @4,0 icon= 
C12 @5,0 icon= 
C13 @5,3 icon= 
C15 @4,3 icon= 
C18 @3,3 icon= 
C21 @2,3 icon= 
C24 @1,3 icon= 
R1  -- C1  R1:network C1:network
C1  -- C4  C1:network C4:network
C4  -- C7  C4:network C7:network
C7  -- C10 C7:network C10:network
C10 -- C12 C10:network C12:network
C12 -- C13 C12:network C13:network
C13 -- C15 C13:network C15:network
C15 -- C18 C15:network C18:network
C18 -- C21 C18:network C21:network
C21 -- C24 C21:network C24:network
C24 -- R2  C24:network R2:network
# Second loop
C2  @1,0.5 icon= 
C5  @2,0.5 icon= 
C8  @3,0.5 icon= 
C11 @4,0.5 icon= 
C14 @4,2.5 icon= 
C17 @3,2.5 icon= 
C20 @2,2.5 icon= 
C23 @1,2.5 icon= 
R1  -- C2  R1:network C2:network
C2  -- C5  C2:network C5:network
C5  -- C8  C5:network C8:network
C8  -- C11 C8:network C11:network
C11 -- C14 C11:network C14:network
C14 -- C17 C14:network C17:network
C17 -- C20 C17:network C20:network
C20 -- C23 C20:network C23:network
C23 -- R2  C23:network R2:network
# Third loop
C3  @1,1 icon= 
C6  @2,1 icon= 
C9  @3,1 icon= 
C16 @3,2 icon= 
C19 @2,2 icon= 
C22 @1,2 icon= 
R1  -- C3  R1:network C3:network
C3  -- C6  C3:network C6:network
C6  -- C9  C6:network C9:network
C9  -- C16 C9:network C16:network
C16 -- C19 C16:network C19:network
C19 -- C22 C19:network C22:network
C22 -- R2  C22:network R2:network

This topology converges in about 6 seconds. Each loop should stay small (around 16 bridges) to reduce the probability of a double failure and to avoid sharing too much bandwidth. The design can evolve a bit without adding too much complexity: one VLAN per loop or one bridge domain per loop.

How large can a network be? The maximum age, whose default value is 20, governs the maximum distance of a bridge from the root. The topology below is too big for BPDUs from R1 to reach beyond S20.¹⁶

:protocol rstp
:tx-hold 10
:max-age 20
R1 @0,0 prio=4096 icon= 
R2 @0,5 prio=4096 icon= 
S1  @1,0
S2  @2,0
S3  @3,0
S4  @4,0
S5  @5,0
S6  @6,0
S7  @6,1
S8  @5,1
S9  @4,1
S10 @3,1
S11 @2,1
S12 @1,1
S13 @1,2
S14 @2,2
S15 @3,2
S16 @4,2
S17 @5,2
S18 @6,2
S19 @6,3
S20 @5,3
S21 @4,3
S22 @3,3
S23 @2,3
S24 @1,3
S25 @1,4
S26 @2,4
S27 @3,4
S28 @4,4
S29 @5,4
S30 @6,4
S31 @6,5
S32 @5,5
S33 @4,5
S34 @3,5
S35 @2,5
S36 @1,5
R1  -- S1
S1  -- S2
S2  -- S3
S3  -- S4
S4  -- S5
S5  -- S6
S6  -- S7
S7  -- S8
S8  -- S9
S9  -- S10
S10 -- S11
S11 -- S12
S12 -- S13
S13 -- S14
S14 -- S15
S15 -- S16
S16 -- S17
S17 -- S18
S18 -- S19
S19 -- S20
S20 -- S21
S21 -- S22
S22 -- S23
S23 -- S24
S24 -- S25
S25 -- S26
S26 -- S27
S27 -- S28
S28 -- S29
S29 -- S30
S30 -- S31
S31 -- S32
S32 -- S33
S33 -- S34
S34 -- S35
S35 -- S36
S36 -- R2
R1  -- R2 cost=200 down

Once the topology settles, part of the network considers R1 the root, while the other votes for R2. At the boundary, S20 tries to start a synchronization with S21 to move its designated port to the forwarding state. The BPDU looks like this:

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x4e, Agreement, Port Role: Designated, Proposal
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 400000
    Bridge Identifier: 32768.02:00:00:00:00:15
    Port identifier: 0x8002
    Message Age: 20
    Max Age: 20

S21 rejects it because the message age equals the maximum age. On the other hand, the BPDU S21 sends to S20 looks like this:

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x7c, Agreement, Forwarding, Learning, Port Role: Designated
    Root Identifier: 4096.02:00:00:00:00:01
    Root Path Cost: 320000
    Bridge Identifier: 32768.02:00:00:00:00:16
    Port identifier: 0x8001
    Message Age: 16
    Max Age: 20

This is not enough to change S20 s root port because S20 has a lower root identifier 4096.02:00:00:00:00:00 vs 4096.02:00:00:00:00:01. Fixing the link between R1 and R2 resolves the issue. The maximum message age any packet carries is now 18, below the configured maximum age. But it only works until another link breaks. A plausible fix is to increase the maximum age to 40.¹⁷

How fast is RSTP? RSTP usually converges in a couple of seconds at startup. It often repairs a tree in less than a second. Even the 38-bridge topology takes less than 10 seconds to converge.¹⁸ Some topologies can take a bit more time to recover when the root bridge becomes unavailable.¹⁹

:protocol rstp
R0 @1,0 prio=0
S1 @1,1 prio=4096
S2 @0,2 prio=8192
S3 @2,2
R0 -- S1
S1 -- S2
S2 -- S3
S3 -- S1

In the topology above, start the simulation, wait for convergence, hit stop, and cut the link between R0 and S1. The topology is already optimal, but RSTP has a hard time converging again. First, S1 loses its root port. It has no more information about R0 and elects itself as the root bridge. It keeps its ports to S2 and S3 as designated ports in the forwarding state. Step once and it sends a BPDU to both S2 and S3 to let them know about the root change. When receiving it, S2 accepts S1 as its root because it does not have a better root on another port. It elects the port to S1 as its root port. The other port stays a designated port. Both ports keep forwarding. When receiving the BPDU from S1, S3 behaves differently: it knows R0 as a better root than S1 through its alternate port to S2. It promotes this port to a root port and demotes the port facing S1 to a designated port, which requires a new agreement. Step once and S3 sends a proposal to S1 with R0 as the root bridge. S1 elects R0 as the root bridge and promotes its port to S3 as a root port. During the same step, S3 also receives a BPDU from S2 stating that S1 is the root bridge. Therefore, S3 has no port left with R0 as the root bridge: it elects S1 as the root bridge and its port to S2 as the root port. Step once and its next BPDU to S1 includes this information: S1 elects itself again as the root bridge. But during the same wave, S1 sends a proposal to S2 with R0 as the root bridge. While S1 and S3 agree that S1 is the root bridge, S2 now believes this is R0! In turn, S2 again convinces S3 that R0 is the root bridge, S3 convinces S1, S1 convinces S2, and S2 convinces S3. This could go on forever, but it does not. The BPDUs saying R0 is root eventually age out when the message age goes past the maximum age. In the example above, at the eleventh second, S2 sends a BPDU to S3 with R0 as root, but S3 drops it because its message age reached the maximum. With some luck, the topology can also converge faster if a port stops transmitting new BPDUs after tripping the transmit hold count, whose default value is 6 per second.

About MSTP MSTP is the VLAN-aware version of RSTP: it runs several instances of RSTP and lets the administrator map each VLAN to a specific instance. For example, you can map VLANs 100 to 200 to a first instance, and 300 to 400 to a second instance. The remaining VLANs map to a special instance named the Internal Spanning Tree (IST). MSTP adds its own complexity, but the gist is that you have several logical topologies acting independently. If you want to dig deeper, have a look at MSTP Tutorial Part I: Inside a Region.

About the interactive examples The interactive examples run MSTPD directly in your browser, compiled to WebAssembly with emscripten. A C API replaces the code talking to the Linux kernel: it manages bridges and ports, exports state as JSON, and drives time deterministically. A JavaScript wrapper makes it more user-friendly:

import   loadMSTPD   from "./dist/mstpd.mjs";
const mstp = await loadMSTPD();
// Create 3 bridges
const a = mstp.createBridge("A",   priority: 4096  );
const b = mstp.createBridge("B",   priority: 8192  );
const c = mstp.createBridge("C");
// Each bridge has two ports
const a1 = a.addPort("a-b",   portno: 1  );
const a2 = a.addPort("a-c",   portno: 2  );
const b1 = b.addPort("b-a",   portno: 1  );
const b2 = b.addPort("b-c",   portno: 2  );
const c1 = c.addPort("c-a",   portno: 1  );
const c2 = c.addPort("c-b",   portno: 2  );
// Build a triangle topology
mstp.link(a1, b1);
mstp.link(a2, c1);
mstp.link(b2, c2);
// Enable all bridges and ports
for (const br of [a, b, c]) br.enable();
for (const p of [a1, a2, b1, b2, c1, c2]) p.enable();
// Execute 40 seconds' worth of wall clock and display the topology
mstp.step(40);
console.log("Topology:", mstp.topology());

Several dozen unit tests explore the features of MSTPD and check that they work correctly in this environment:

$ node --test *.test.mjs
  two bridges: lower priority becomes root (41.657342ms)
  triangle loop: exactly one port blocks and all agree on the root (5.832ms)
  breaking the active link reconverges and restoring recovers (18.730753ms)
[ ]
  tests 40
  pass 40
  fail 0
[ ]
  duration_ms 396.190897

Additional JavaScript code looks for specific <pre> blocks containing a topology definition and turns them into interactive widgets. You can inspect and modify the definition by hitting the edit button. There is also a cool trick to tell whether the topology has converged. After each step, we save a snapshot of the simulation memory, play 50 seconds worth of simulation to check if the topology is stable, and travel back in time by restoring that snapshot. The complete code lives on GitHub. I am happy with the result. It can be difficult to follow everything happening during a single step, but stepping forward and backward helps. I plan to use the same approach in future blog posts about networking features.

Note Michael Lynch reviewed a first draft of this article. He authored Refactoring English, a book to sharpen your writing for blog posts, documentation, commit messages, and tutorials. Any errors are still mine!

---

1. The sprites for Stan and Blobby come from Craftpix, the coffee cups from Yanin.
2. STP was introduced in IEEE 802.1D-1990. It is still present in IEEE 802.1D-1998 but was withdrawn in IEEE 802.1D-2004 in favor of RSTP, introduced in IEEE 802.1w-2001. MSTP was introduced in IEEE 802.1s-2002 and merged into IEEE 802.1Q-2003. Both of them are part of IEEE 802.1Q-2022 along with SPB a protocol I had never heard of until writing this article.
3. From here, I use bridge instead of the more common word switch.
4. The Linux kernel only runs STP. It delegates the other protocols to user space.
5. MSTPD implements the state machine from IEEE 802.1Q-2005, but on Linux it runs RSTP only. Linux 5.18 added support for forwarding multiple spanning tree, but MSTPD does not use it yet. See PR #150 for progress on this front.
6. The priority is a multiple of 4,096: with MSTP, the lower 12 bits of the bridge priority encode the MST instance identifier, leaving only the upper 4 bits for the configured priority.
7. To inspect the BPDUs crossing a link, select it, click the Download packets button, and open the file with Wireshark.
8. A backup port only exists if the bridge has several ports on the same collision domain. This should not happen in a switched network.
9. This is the value of the hello timer. It used to be configurable, but IEEE 802.1Q-2005 pins it to 2. MSTPD does not allow another value.
10. If the peer port does not receive an agreement after the hello timer elapses or the maximum age if the port has just come up it falls back to the timer-based method for compatibility with STP: it transitions to the learning state, waits again for the hello timer to expire, and transitions to the forwarding state.
11. As in many proposals, S3 also sets the agreement bit to 1. The proposal bit says I am the designated port on this link and I want to transition to the forwarding state. The agreement bit says I am already in sync with the rest of my bridge on this root information. Both can be true.
12. If it finds no entry, the bridge duplicates the Ethernet frame on all ports, except the incoming one. The same happens if the destination MAC address is the broadcast one (ff:ff:ff:ff:ff:ff). This behavior bootstraps the learning process.
13. This timer makes RSTP resistant to packet loss.
14. You can get them for less than US$100 through a broker. All the ports run PVST+ by default and automatically fall back to plain RSTP.
15. An alternative would be Ethernet Ring Protection Switching (ERPS) another protocol I had never heard of until researching this article.
16. If you look closely at what happens at t=2s, you can see that R2 is gaining popularity as root: S17 to S36 believe R2 is the root bridge. S16 does not follow because we hit the maximum age. Later, S17 to S20 reverse their position. I ll let you explore the state of the various bridges to understand the root cause.
17. When increasing the maximum age to 40, you also need to increase the forward delay to 21 (:forward-delay 21), as the standard enforces this condition: 2 (Forward Delay 1) Max Age. For this specific topology, you could also increase the maximum age to 37 and forward-delay to 20.
18. The simulation may seem slow, but it does not run in real time. Look at the current timestamp in the upper right corner to know the wall clock, e.g. t=8s. Once the topology stabilizes, the same corner shows the convergence time, e.g. 2s.
19. Khaled Elmeleegy, Alan Cox, and Eugene Ng formalized this phenomenon in On Count-to-Infinity Induced Forwarding Loops in Ethernet Networks and later in Understanding and Mitigating the Effects of Count to Infinity in Ethernet Networks. They propose a fix that did not find its way into a standard.

Activity summary During the month of July, 23 contributors have been paid to work on Debian LTS (links to individual contributor reports are located below). The team released 52 DLAs fixing 2159 CVEs. In July, the Debian Stable Release Managers published the last point release of Debian 12 ( bookworm ), after which the Debian LTS team took full responsibility of Debian 12. This completes the handover from the Security Team, that took place in June. This also marks the second month in a row where the Debian LTS has been focusing on two simultaneous Debian releases. Other than Debian 12, the team is maintaining Debian 11 ( bullseye ), which will reach the end of its Long Term Support on 31 August 2026. After that date, Freexian will continue the security support under the Extended LTS offer. The team published several notable updates:

• jq (DLA 4662-1 and DLA 4661-1) prepared by Andreas Henriksson in collaboration with Jochen Sprickerhof, addressing multiple vulnerabilities.
• Several updates for the different linux supported versions prepared by Ben Hutchings, in collaboration with Emilio Pozuelo Monfort. Other than the regular security advisories: DLA 4664-1, DLA 4665-1, DLA 4671-1, DLA 4688-1, and DLA 4700-1, Ben started preparing packages of 6.12 via bookworm-backports.
• nginx (DLA 4667-1), updated for bookworm by Carlos Henrique Lima Melara, as a follow up of the bullseye update (DLA 4660-1), that was prepared in June.
• grub2/bullseye (DLA 4685-1), prepared by Emilio. Other than addressing several security issues, this DLA was needed for being able to update the shim boot loader.
• samba (DLA 4692-1), uploaded by Markus Koschany, to fix several security flaws in bullseye, including issues that could yield to remote code execution.
• imagemagick (DLA 4680-1 and DLA 4696-1), prepared by Bastien Roucari s, addressing several issues that could lead to denial of service, information disclosure or potentially arbitrary code execution in some scenarios.
• poppler (DLA 4709-1), by Guilhem Moulin, fixing several vulnerabilities.
• nss (DLA-4694-1), by Jochen, fixing flaws that may result in or denial of service or potentially the execution of arbitrary code.

Contributions from outside the LTS Team:

• Guillem Jover prepared the dpkg update, released as DLA-4673-1 by Arnaud.
• Nicholas Guriev released a rlottie security update (DLA 4675-1)
• Bernhard Schmidt prepared the openvpn updates for both bookworm and trixie, released as DLA 4666-1.

The LTS Team has also contributed with updates to the latest Debian releases:

• Bastien also proposed two updates for imagemagick. The first one released as DSA 6383-1, and the second as a trixie point update proposal (#1142554).
• python-httplib2 by Emmanuel Arias, and released by the security team as DSA 6441-1 in August.
• hplip (DSA 6402-1), prepared by Thorsten Alteholz, to address privilege escalation and arbitrary code execution related flaws.
• libnfs trixie update (#1142351), by Thorsten
• patool update for trixie #1141607, by Abhijith PA

Other contributions: Besides the work on security updates, different documentation and tooling changes were needed, especially in the context of the Debian 12 handover. This work was mainly done by Sylvain Beucler.

Individual Debian LTS contributor reports

• Abhijith PA
• Andreas Henriksson
• Andrej Shadura
• Arnaud Rebillout
• Bastien Roucari s
• Ben Hutchings
• Carlos Henrique Lima Melara
• Chris Lamb
• Daniel Leidert
• Emmanuel Arias
• Emilio Pozuelo Monfort
• Guilhem Moulin
• Jochen Sprickerhof
• Lee Garrett
• Lucas Kanashiro
• Lukas M rdian
• Markus Koschany
• Paride Legovini
• Santiago Ruano Rinc n
• Sylvain Beucler
• Thorsten Alteholz
• Tobias Frost
• Utkarsh Gupta

Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • Toshiba Corporation (for 130 months)
  • Civil Infrastructure Platform (CIP) (for 98 months)
  • VyOS Inc (for 63 months)
• Gold sponsors:
  • F. Hoffmann-La Roche AG (for 141 months)
  • CONET Deutschland GmbH (for 124 months)
  • University of Oxford (for 81 months)
  • EDF SA (for 52 months)
  • Dataport A R (for 27 months)
  • CERN (for 25 months)
• Silver sponsors:
  • Domeneshop AS (for 145 months)
  • Nantes M tropole (for 139 months)
  • Akamai - Linode (for 135 months)
  • Univention GmbH (for 131 months)
  • Universit Jean Monnet de St Etienne (for 131 months)
  • Ribbon Communications, Inc. (for 125 months)
  • Exonet B.V. (for 115 months)
  • Leibniz Rechenzentrum (for 109 months)
  • Minist re de l Europe et des Affaires trang res (for 93 months)
  • Dinahosting SL (for 80 months)
  • Upsun Formerly Platform.sh (for 75 months)
  • Moxa Inc. (for 69 months)
  • sipgate GmbH (for 66 months)
  • OVH US LLC (for 64 months)
  • Tilburg University (for 64 months)
  • GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 56 months)
  • THINline s.r.o. (for 28 months)
  • Copenhagen Airports A/S (for 22 months)
  • Conseil D partemental de l Is re (for 8 months)
  • CEOS GmbH
• Bronze sponsors:
  • Seznam.cz, a.s. (for 146 months)
  • Evolix (for 145 months)
  • Linuxhotel GmbH (for 143 months)
  • Intevation GmbH (for 142 months)
  • WAAYS (for 141 months)
  • Megaspace Internet Services GmbH (for 140 months)
  • Greenbone AG (for 139 months)
  • NUMLOG (for 139 months)
  • WinGo AG (for 138 months)
  • Entr ouvert (for 130 months)
  • Adfinis AG (for 127 months)
  • Plat Home (for 124 months)
  • Laboratoire LEGI - UMR 5519 / CNRS (for 122 months)
  • Tesorion (for 122 months)
  • Bearstech (for 114 months)
  • LiHAS (for 114 months)
  • Catalyst IT Ltd (for 108 months)
  • Demarcq SAS (for 102 months)
  • Universit Grenoble Alpes (for 88 months)
  • TouchWeb SAS (for 80 months)
  • SPiN AG (for 77 months)
  • CoreFiling (for 73 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 65 months)
  • Tem Innovations GmbH (for 59 months)
  • WordFinder.pro (for 59 months)
  • CNRS DT INSU R sif (for 58 months)
  • Soliton Systems K.K. (for 53 months)
  • Alter Way (for 51 months)
  • SOBIS Software GmbH (for 25 months)
  • Tuxera Inc. (for 17 months)
  • OPM-OP AS (for 8 months)
  • LU-CIX Management G.I.E.

Programming language generations When I was young, I learned about a model of classifying programming language: the system of programming language generations. In this model, first generation programming languages are, basically, where you program the computer in the language that is defined by its architecture. On a Von Neumann machine, with its load-and-store architecture, you do that by inputting a string of numbers. The first programmer in human history -- her name was Ada Lovelace -- wrote in a first-generation language. 1GLs aren't so much invented as they are a byproduct of the computers for which they're created. Second-generation languages are the assembler languages. Because humans are not computers, and because decoding long lines of numbers to understand what the computer is doing, when programming became a full-time job, the programmers that did it decided that doing all this assembling manually is too complicated, so they quickly wrote assemblers to automate the process for them. They still could understand the 1GL output of the 2GL assembler, but most of them quickly forgot how to write software in a first-generation language. Not that anyone cared, as the translation from a 2GL to a 1GL is lossless and you can just revert it. Third-generation languages are higher-level languages. When the first 3GLs were invented (such as COBOL and, more famously, FORTRAN) in the late 1950s and early 1960s, it was believed by some that the work of programming a computer so accessible to non-programmers that the job of programmer would eventually cease to exist, and people would just ask the computer what they needed by entering COBOL instructions. This of course was ridiculous and incorrect, because converting algorithms to computer instructions, whether at the 2GL or 3GL level, is a specialized skill that some automation can perhaps make simpler but never completely take away the need for. At the time, some people also felt to some extent that using 3GL wasn't the same thing as actually programming 3GLs, but eventually the world moved on and embraced things. The invention of 3GL environments reduced, but did not completely take away, the need for people to understand 2GLs, as compiler and operating system authors still need to understand them, and some highly optimized code still continues to be written in 2GLs to this day. Fourth-generation languages abstract away some or all of the process of programming. For instance, a database-related 4GL will hide away the complexities of storing data in particular locations, how to fetch that data, how to index it such that you can fetch it efficiently, how to loop over the data to get you a summary of that data, and instead allows you to express the required information in an abstract way, expecing the computer to fill in the blanks. When SQL, an early 4GL, was invented, some people believed that the language made accessing databases so simple that the requirement to implement database applications would eventually cease to exist and we would just hand SQL prompts to users who need to access data. This of course was ridiculous and incorrect, because understanding data schemas and using that understanding to query data from a database is a specialized skill that perhaps a higher abstraction can help you make simpler, but that in the longer run it can never completely take away the need for. The invention of 4GLs also reduced, but did not completely take away, the need for people to understand how to do the things that the 4GLs automate for you manually, as the people who do write those things still need to understand them, and there are also environments where these particular 4GLs are rather not appropriate or just very slow. The first definition of programming language generations that I read about in the 1980s simply stated that fifth-generation languages did not yet exist, but that they would in the future, and that in those, you would "tell the computer what to do, and it would then do that". Now that we have a way of doing so, it could be said that by some definition, we now actually do have a number of 5GLs. The existence of these LLM systems has caused some, especially the people who build and exploit these systems, to exclaim that programming as we know it today is going to cease to exist, and everyone will just ask an LLM to generate a program, which will then do so. That is of course ridiculous and incorrect, as no automaton can generate software from nothing; input is still required for the model to be able to produce something that approaches usability, and being able to word that input in a correct and productive fashion will be a skill that future programmers can benefit from. I ran some experiments a while back, and from that concluded that, if we look only at the technical side, LLM use can, in some niches, increase productivity for a programmer. There are certainly things that you shouldn't use an LLM for, but equally there can be cases where use of an LLM to perform some task that traditionally would have been done by a programmer would be a net positive. But LLMs, as they exist today, are highly problematic. They require vast amounts of data to build the model. The companies that build these models are disrespectful of people who run web services, and as a result, everyone now has to implement various types of application firewalls just to not make systems fall over from the overwhelming requests for data. They are also disregarding the licenses that are attached to these vast amounts of data, which makes me, as a person who believes in the tenets of free software, sad. They require vast amounts of energy, causing an already-critical global warming crisis to, well, not improve. They require vast amounts of coolant to dissipate the energy concentrated in their data centers, causing further environmental effects. In this, they are problematic and to be avoided. But these are side states of the current state of affairs; I do not believe that they are inherently implied to be able to build and operate an LLM -- any LLM. I guess it's fair to say that my feelings towards LLM usage are complex and many-faceted. I haven't been involved in many debates about the subject, debates that to me seem to be mostly focused on "LLM good" vs "LLM bad" arguments that aren't as nuanced as the position that I would believe is more accurate. This is not because I don't care, but partially because I've been busy in my personal life recently and partially because the whole thing seems somewhat disheartening. But then Debian popped up GR 2026-002, meaning, I now have to come up with an opinion about various candidate statements in the context of the above, which is... not easy. But I did it anyway. There are 8 choices on the ballot, and they all have some truth and some falsehood to them. My position about LLMs can be summarized as:

• The current state of affairs wrt LLMs is disastrous and we should not encourage them
• However, there's no technical reason why this must remain true for all time
• And so any statement should keep in mind what might happen in the future and that the current disastrousness of the whole thing isn't guaranteed to continue to exist for all eternity.

With that, let's go over them.

GR vote options

Proposal A Its summary, from the GR text:

This proposal aims to expressly forbid any contributions to Debian written with the use or assistance of large language models (LLMs) or other generative AI tools.

This falls squarely in the "LLM bad" camp, outlawing all generative-AI contributions, disregarding potential future ones where the problematic situations that exist today are not present. It makes a change to the social contract, which is especially difficult to reverse (on purpose), and which therefore also will require a 3:1 supermajority, but if we want to ban LLM-assisted contributions, this is probably the best way to do it.

Proposal B This one tries to allow AI-assisted contributions under certain conditions. It's mostly an "LLM good" proposal, with some caveats that can be discribed as "make sure you know what you're doing".

Proposal C This proposal is both a weaker (in some places) and stronger (in other places) version of Proposal A. It makes changes to the code of conduct instead of to the social contract, and it also wants to, at least, suggest policy to parties beyond the Debian project. By not changing the social contract, however, it is more likely to reach its simple majority requirement than proposal A. I don't think the language that it wants to add to the code of conduct is particularly well phrased, however.

Proposal D This is a weaker form of proposal B. The language is more compact and there are a few requirements that are spelled out in proposal B that are not spelled out in proposal D, but if you read between the lines you'll see that the requirement is still there really and I don't understand why proposals B and D were not merged into one.

Proposal E This proposal tries to hold a middle ground between "LLM good" and "LLM bad". It appreciates that things are quite muddled at the present time, and that perhaps the situation might might change in the future. It acknowledges that certain questions remain unanswered and that perhaps future considerations might therefore be different. But it essentially refuses to take a stance on whether LLMs should be accepted by the project or not.

Proposal F Similar to proposal E, this proposal tries to discourage Debian contributors from using LLMs, while still allowing people to use it should they want to, but with some requests and requirements to mark LLM-assisted contributions to account for those people who don't want to interact with LLM-generated software. As such, it is a proposal similar to proposal E that leans closer to the "LLM bad" camp.

Proposal G

This proposal aims to ensure that contributions directly to Debian are created by humans, while at the same time avoiding restrictions on the tools those humans may choose to use when contributing

Another "LLM bad" proposal, it however restricts the "bad" bits to only the direct output of the LLM. If you use an LLM to do something and then clean-room re-implement the same thing yourself, that's apparently fine.

Proposal H This proposal condemns the use of LLM for its environmental and moral problems, but explicitly not for its technical considerations. I feel that it is closest to my position as explained above.

Voting Expressing a vote on a ballot so convoluted and complicated like this one takes time. I have to read and understand every ballot option, and formulate an order of them. And I shouldn't just state which option has my preference; Debian's voting process allows a rich expression of opinion on ballot options. Anyway, I eventually ended up voting in a way that I think is consistent with my opinion. But it wasn't easy.

• Proposal A(1) - Action: ban all AI-assisted contributions via Social Contract amendment, except from upstreams (so not rolling back the Linux kernel and other software to "pure", pre-AI state). Claims that copyright/licensing status is unclear, quality is bad, community is being destroyed, web resources see extra load and that training consumes "staggering" resources. Needs 2/3rd majority to pass. - IMHO worst and most inconsistent. If copyright and licensing of AI products is unclear, then be consistent - ban ALL software with AI contributions, fork Linux kernel and other software from pre-AI versions, reject all security fixes of issues found with AI. Quality section lists problems that have not existed in the real world since at least a year of rapid AI coding development. Community section assumes that now all Debian contributions will be drive-by AI slop and no one will learn anything anymore. Ethics section mixes up effects of badly configured systems (AI web load is no different from load from a badly configured Perl script) with claimed "resource" usage without any context, taking on trust project ambitions of startups and assuming exponential growth. And then concludes that delivering less is in the interest of our users somehow.
• Proposal B(2) - Action: allow AI-assisted contributions, with conditions of: legality, accountability, disclosure, no uncoordinated bulk actions, privacy. Concerns on quality and legal status as well as environmental impact and scraper load are noted, but not really addressed beyond labelling them as concerns. - IMHO it is an ok starting position as it establishes that each contributing person must still be fully responsible for their contribution (both legally and technically) and for that has to also understand (and review) what they submit. Disclosure lets others know to watch out for other classes of problems when code was changed with AI assistance. Prior discussion for bulk changes just says that the (already established) practice should not be neglected just because now large changes are easier to do. And the privacy part warns against accidentally sending private or confidential data (like a not yet published security bug) to a public service where it could become public. Personally I would have liked a stronger statement to encourage use of environmentally responsible AI services and local AI tools. Possibly a preference for open-weight models with a clear path forward to preferring truly free AI models, when such a category of products could be clearly delineated and established.
• Proposal C(3) - Action: reject AI-assisted contributions at Code of Conduct level. Claims all the world's evils come from LLMs and that "Ethical and safe use of this technology is almost impossible". Goes as far as banning any use of LLMs even in Debian mailing list emails and Debian Planet blog posts - if you do, it's a CoC violation and may result in exclusion from the project. Additionally mandates the disclosure of the usage ... presumably to ban you more efficiently for it. - IMHO truly a dictatorial nightmare option. Zero actual reasoning or basis for such a decision. Zero sources. Nothing claimed in this option's rationale is even close to reality and nothing claimed there is in any way related to the actual technology being discussed. Like, an "LLM" does not automagically commit "fraud" when you use it, like this proposal claims, as if that was a well-known fact. LLMs are not all "owned by horrible people and companies". Even if some include a (prominent Debian user, long-time supporter and sponsor) Google into "horrible companies" (which is what this proposal implies!), there are plenty of LLMs owned by all kinds of companies all over the world and there are plenty of open-weight LLMs that are not really owned by anyone. Most invasive and dishonest option on the ballot.
• Proposal D(4) - Action: allow AI-assisted contributions, with conditions of: legality, accountability, disclosure, privacy. IMHO same as B, just shorter. Adds a "we don't recommend" towards others developing software with AI assistance. Seems pretty weird to add that and then immediately accept Debian contributors doing so. Assumes that the bulk change bit of B is implied as AI is just tooling, so bulk changes should be pre-discussed just like today - so no change and thus no point in mentioning that. Fair. D is a bit more explicit on expected technical details - like that the "person" submitting the change is supposed to sign it, not AI. Notable is the complete absence of resource usage or the environment from concerns. IMHO it would be better to have that and also recommendations on how to avoid causing environmental damage when using AI.
• Proposal E(5) - Action: no action as such - AI-assisted contributions must follow the same rules as all other contributions and those rules are sufficient. IMHO despite its length this is a very well-worded position statement that describes how and why AI-assisted contributions already work perfectly fine in the Debian context when all the same rules that apply to all contributions are also consistently applied to AI-assisted contributions. It describes how the same legality, accountability, no bulk change and privacy requirements are already in place and still apply and how AI-assisted contributions can and must still satisfy them. I could add again that some guidance would be nice here for both legal and environmental decisions when using AI, but in this case it does not really belong in this proposal itself. We as Debian do not have a document that requires that our non-AI-assisted contributions be made with only sustainably sourced electricity, for example. So why should AI be special one way or another? IMHO Debian should have a datacenter sustainability policy, regardless of the AI discussion.
• Proposal F(6) - Action: discourage AI, but allow it based on existing processes (similar idea to E). Dances a bit around the question of disclosure of AI use (as a courtesy) and accepting that some people may still ban all contributions where any AI was involved in any way. Which in turn discourages disclosure to avoid pointless rejection of valuable contributions (like security patches). IMHO this option is ok, but so watered down that it is bound to bring up further discussions and conflicts on details.
• Proposal G(7) - Action: ban non-humans from directly contributing to Debian. IMHO - another bizarre and self-contradictory option. It bans all Debian interactions with AI assistance, including email messages to Debian mailing lists and (supposedly) blog posts on Planet Debian. It "reminds" people who "use such tools assistively" of the DFSG and Social Contract - isn't that a threat of a ban and expulsion similar to C? The proposal does take pains to delineate where a contribution comes from AI as output (bad) vs when you are assisted by AI in the process of exploring, researching or maybe even reviewing the code, but you actually type all the code yourself and use the AI just as a taskmaster with a whip (good). And just like A or C it completely ignores how this inherently evil and unstable AI-generated code becomes perfectly fine and good as soon as someone develops that outside of the Debian project. Even if the same person then packages it for Debian the next day. It is hypocritical, unsustainable and ignores the needs of our users. Just like C it also bans someone writing an email or bug report in their native language and using a modern translation tool or service (that uses LLMs nowadays for better grammatical clarity) to translate that to English before sending it to a Debian mailing list or BTS. Heavy-handed and invasive. And the only reasoning provided for this is some unnamed "concerns" of "extra work" being borne by "other people"? Kind of does not feel right to bear such draconian restrictions for some unspecified concerns.
• Proposal H(8) - Action: condemn usage, but not actually ban anything. And then it goes on to claim (without any evidence or elaboration) that LLM usage accelerates the destruction of "planet earth" (sic). IMHO this proposal is at the same time the loudest ("The planet is burning") and also the one that demands the least action. It dances a really twisty line between raising "significant" concerns in all areas and even claiming that use of LLMs destroys the planet, flies by explicit condemnation of LLM usage and then suddenly collapses with not condemning LLM users and swinging to lamentations that it is actually impossible to impose policies on LLM usage or even detect when an LLM was used (which kind of directly contradicts bad quality claims from A, C and G) and lands on "encouraging" contributors not to use LLMs (where practical) and otherwise do nothing else. It's like this is a 5th draft that started off with the rationale and total ban like in C, but then got defanged so far that its action side no longer matches the rationale stated.

• Proposal E(5) - solid hack of integrating AI into already existing Debian rules and conventions
• Proposal B(2) - explicit and detailed
• Proposal D(4) - lower because of discouragement to others on what we agreed to do ourselves
• Proposal F(6) - I am not a fan of dancing around with disclosures
• Further discussion(9) - I do not want any option below this to succeed as they would do more harm than good
• Proposal H(8) - loud, but not doing anything actually
• Proposal A(1) - at least this one does not set rules for emails
• Proposal G(7) - at least this one allows an AI overseer to tell you what to write with your own fingers
• Proposal C(3) - the most draconic and invasive one that explicitly wants to kick people out of the project

• information on which AI services have Terms and Conditions that make them problematic for free software development, legally speaking
• information on which AI services do (or do not) achieve a sufficient level of sustainability to be worth recommending (and then do the same for other data centers we already use)
• information on which local AI models were trained in sustainable ways
• base-level prompts to set technical expectations on various types of contributions, like bug reports or patches to packaging or translations
• default configuration for AI-assisted code reviews on Salsa that projects could enable and supplement with their own instructions on top

• encourage them to improve sustainability (where needed)
• investigate and fix problems causing excessive scraping load on systems
• provide AI resources for Debian usage, for example in CI infrastructure or to provide equal development support opportunities for Debian developers who cannot afford paid AI services
• improve coding outputs of their models in the Debian context if/when systematic deficiencies in the output are found by us

• [1] https://www.youtube.com/watch?v=TCn7bA1eg_0
• [2] https://www.youtube.com/watch?v=ZJ8KThKAfbs
• [3] https://www.youtube.com/watch?v=gpSghW4J3ws
• [4] https://tinyurl.com/28b6jclu
• [5] https://zanestjohn.com/blog/reing-with-claude-code
• [6] https://tinyurl.com/2y2b6a58
• [7] https://tinyurl.com/2awjppn5
• [8] https://tinyurl.com/2y89gftt
• [9] https://tinyurl.com/2b27vo8o
• [10] https://365tomorrows.com/2024/03/27/cold-war-2/
• [11] https://tinyurl.com/2aqkn4z6
• [12] https://tinyurl.com/2dy5vy8g
• [13] https://tinyurl.com/25rld7v3
• [14] https://doctoreww.github.io/EvilFontTool/
• [15] https://tinyurl.com/2c2guu89
• [16] https://tinyurl.com/27kq4qqu
• [17] https://tinyurl.com/2blp477w
• [18] https://tinyurl.com/27aybgge
• [19] https://tinyurl.com/29covbrp
• [20] https://tinyurl.com/27qdj6co
• [21] https://tinyurl.com/22pxub8v
• [22] https://tinyurl.com/25zf4hnp

For human developers, using coding LLMs magnifies their difference in skill levels.

• Packaging upstream software for Debian (by far the largest task)
• Writing Debian-specific code (eg, apt or the Debian installer)
• Maintaining Debian infrastructure (build systems, for instance)
• Writing documentation and translations

• Proposal A: seeks to amend the social contract, which I am opposed to for reasons already laid out above. It names some real concerns about AI that I agree with, but implies that all LLM uses and models are guilty of the problems, which is not the case with all of the claims. It also sets us behind the curve on security and stability by forbidding the use or assistance of those tools, even if run by others. It requires us to ignore reports of actual security bugs, or correct fixes, if those reports were generated with the assistance of an LLM, which I find to be absolutely untenable.
• Proposal B: This is the AI with accountability approach. It notes the real concerns with LLMs without painting with an overbroad brush. It strikes me as level-headed and sensible.
• Proposal C: It paints with an over-broad brush and makes some non-binding requests. Then it winds up largely like proposal B, though while it is worded more strongly, has fewer binding requirements (for instance, it lacks proposal B s prohibition on transmitting sensitive information to untrusted providers)
• Proposal D: Seems broadly similar to proposal B, an AI with accountability approach. I m not really clear why we need both.
• Proposal E: Largely the status quo. It is like proposals B and D in that it says humans are accountable for their contributions. It encourages disclosure of LLM use, but does not mandate it. Like proposal B, it prohibits disclosing sensitive information to third-party AI services. Note that both proposals B and D have an appropriate nuance: a local model is fine, a third-party one is not.
• Proposal F: This seems really similar to proposal E. I m not sure why we have these two.
• Proposal G: Disallows the output of generative AI as direct contributions to Debian. This is something of a weakened proposal A; it doesn t seek to amend the social contract, nor does it ban all use; it simply bans the use as a direct contribution.
• Proposal H: Ban due to climate impacts. How is this even an argument is disrespectful to reasoned conversation. I have already noted that LLMs can be and are used in ways that are not climate-harming. It explicitly contains no binding requirements at all, and is effectively a rant. While I agree with the sentiment that climate change is an urgent problem, and that some LLMs are exacerbating it, I disagree with that all LLM usage does so and therefore disagree with the conclusion.

Not a screenshot of despair. But only because it s not a screenshot.

This event is happening soon see below for registration instructions!

We are extremely pleased to announce the upcoming Reproducible Builds summit, which will take place from September 22nd 24th 2026 in the city of Gothenburg, Sweden. This year, we are thrilled to host the tenth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Vienna (2025), Hamburg (2023 2024), Venice (2022), Marrakesh (2019), Paris (2018), Berlin (2017), Berlin (2016) and Athens (2015). If you re excited about joining us this year, please make sure to read the event page which has more details about the event and location. As in previous years, we will be sending invitations to all those who attended our previous summit events or expressed interest to do so. However, even if you do not receive a personal invitation, please do email the organizers and we will find a way to accommodate you.

About the event The Reproducible Builds Summit is a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.

Schedule Although the exact content of the meeting will be shaped by the participants, the main goals will include:

• Update & exchange about the status of reproducible builds in various projects.
• Improve collaboration both between and inside projects.
• Expand the scope and reach of reproducible builds to more projects.
• Work together and hack on solutions.
• Establish space for more strategic and long-term thinking than is possible in virtual channels.
• Brainstorm designs on tools enabling users to get the most benefits from reproducible builds.
• Discuss how reproducible builds will be usable and meaningful to users and developers alike.

Logs and minutes will be published after the meeting.

Location & date

• Clarion Hotel Draken, Olof Palmes Plats 2, 413 30 G teborg, Sweden. (OpenStreetMap, Google Maps)
• September 22nd to September 24th 2026.

Registration instructions Please reach out if you d like to participate in hopefully interesting, inspiring and intense technical sessions about reproducible builds and beyond! We look forward to what we anticipate to be yet another extraordinary event!

Welcome to the July 2026 report from the Reproducible Builds project! In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not. If you are interested in contributing to the project, please visit the Contribute page on our website. In this month s report, we cover:

1. Tool development
2. Distribution work
3. Three new scholarly papers
4. Patches
5. Misc news

Tool development diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 324, 325 and 326 to Debian:

• Fix tests to work with zipdetails 4.0008. (#1141359)
• Bump debhelper compatibility level to 13. [ ]
• Update copyright years. [ ]

In addition, Paul Spooren made changes to allow trailing garbage in Gzip files [ ] and Vagrant Cascadian added an external tool reference for the pedump binary to use the mono package under GNU Guix. [ ]
disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into system calls to reliably flush out reproducibility issues. This month, Christelle Gloor added the option to sort by ctime as returned by the lstat(2) syscall. [ ], which Chris Lamb uploaded whilst bumping the Standards-Version to version 4.7.4 [ ]. Bernhard Wiedemann also updated disorderfs to version 0.7.0 in openSUSE.
Yet again, there were a number of improvements made to our website this month as well. For example, Chris Lamb, by request of Digital Ocean, changed the target of a referral link so that they can manage incoming referrers [ ] and pushed a number of changes to the Tools page [ ].

Distribution work In Debian this month, 32 reviews of Debian packages were added, 26 were updated and a total of 21 were removed this month, adding to our extensive knowledge about identified issues. A number of issue types were added by Chris Lamb, including:

• python_towncrier_build_date [ ][ ]
• log_files_installed_in_package [ ]
• fontforge_varies_by_timezone [ ][ ]

Chris also added a further note for the build_date_in_manpage_generated_by_spf13_cobra issue. [ ]
In addition, there is a new page showing verification rebuilds of OpenWrt APK packages and firmware images, powered by rebuilderd:

Three new scholarly papers Yan Li, Nan Jiang, Qihang Zhou, Shaowen Xu, Yamin Xie and Xiaoqi Jia of the Chinese Academy of Sciences published a paper titled VCAligner: Aligning Source Distribution Versions with Upstream Git Commits to Secure Supply Chain:

We present VCAligner, a content-based alignment methodology that constructs inverted indexes over VCS histories to precisely map released artifacts to their originating commits, independent of fragile version tags. We evaluated VCAligner on a dataset of 2,984 verifiable PyPI packages derived from the 4,000 most-downloaded projects linked to public GitHub upstreams. Our results reveal a critical weakness in conventional tag-based heuristics: while they appear effective on 85% of the dataset, the residual 15% failure rate generates a catastrophic downstream audit workload of over 10.3 million commits. In contrast, VCAligner reduces this burden by two orders of magnitude ( 158 ), bounding the total workload to under 65,000 commits. Furthermore, we provide the large-scale characterization of Packaging Noise, classifying artifact divergence into structural additions (Path Phantoms) and content mutations (Blob Phantoms), thereby isolating the distinct attack surfaces of malicious injection and code tampering.

Jens Dietrich and Spencer Sun from the Victoria University of Wellington together with Tim W. White and Behnaz Hassanshahi from Oracle Inc pre-published their paper No Snake Oil: Verifying Python Package Builds (PDF):

Python has become the default language for interacting with AI, with packages being distributed through registries like the Python Package Index (PyPI). This creates a need to analyse supply chains comprising such packages. One such analysis is to rebuild packages in order to identify compromised builds injecting malware. Independent rebuilds in hardened environments have the added advantage that they can generate and record provenance in order to increase the trustworthiness of packages. Two tools that are designed to automate such rebuilds and run them at scale are macaron and oss-rebuild. We study 12,180 popular releases from PyPI and find that the byte-for-byte equivalence rate is generally low. We analyse the reasons why they produce different wheels, and find that equivalence between the original and rebuilt wheels can often still be established, preserving most of the guarantees users expect from rebuildable releases. We present and evaluate daleq4py, a tool to establish the equivalence of Python wheels through the kernel of a normalisation function that is based on provenance-preserving datalog rules. Experimental results show that daleq4py substantially expands the set of rebuilds that can be accepted as equivalent. Although only 15.4% of macaron rebuilds and 19.1% of oss-rebuild rebuilds are byte-for-byte identical to the published PyPI wheels, daleq4py establishes wheel equivalence for 60.2% and 78.9% of source-equivalent rebuilds, respectively.

Denise Nanni, Julien Malka, Stefano Zacchiroli and Th o Zimmermann from T l com Paris together with Gabriele D Angelo from the University of Bologna pre-published their paper Understanding Build Reproducibility in the F-Droid Ecosystem (PDF), which was accepted at the 2026 ACM Conference on Reproducibility and Replicability:

The security of open source applications benefits considerably from the possibility of rebuilding their source and verifying the output. F-Droid, a prominent distribution for open source Android applications, systematically rebuilds them from source and tests their bitwise reproducibility at app publishing time. However, F-Droid offers no guarantee that app reproducibility will continue to hold in the future. As software ecosystems evolve, reproducibility may degrade, with potential negative consequences for software preservation and security. We present the first empirical study of build reproducibility in the F-Droid app ecosystem. Analyzing historical reproducibility logs, we find that the overall bitwise reproducibility rate has been steadily increasing over time (as new versions of apps are published). We then evaluate how reproducibility holds in time for fixed app versions, by attempting to rebuild 18 904 app versions that F-Droid had previously confirmed bitwise reproducible, published between September 2018 and February 2026, achieving an 83% rebuild success rate, and identify missing dependencies as the dominant cause of failure, accounting for 76% of non-rebuildable cases. Among successfully rebuilt apps, 94% are also bitwise reproducible-i.e., they still yield bitwise identical artifacts upon rebuild. Together, these results show that while bitwise reproducibility largely holds for apps that can be rebuilt, rebuildability itself is highly sensitive to temporal decay.

Patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:

• openSUSE monthly
• Arnout Engelen:
  • ivy
  • pekko-grpc
  • sbt-ivy
  • scala
• Bernhard M. Wiedemann:
  • angelfish
  • cpython
  • efl
  • erlang+ex_doc
  • eww/glib-macros
  • intel-graphics-compiler
  • java-11-openjdk
  • java-17-openjdk
  • java-1_8_0-openjdk
  • java-21-openjdk
  • mono-core
  • nasm/rav1e
  • python-langsmith
  • zathura*
  • zig0.15
• Chris Lamb:
  • #1141505 filed against golang-github-tidwall-wal.
  • #1141506 filed against lightproof.
  • #1141582 filed against libslow5lib.
  • #1141687 filed against siso.
  • #1141841 filed against grout.
  • #1142126 filed against libpsl.
  • #1142492 filed against node-grunt-contrib-internal.
  • #1142495 filed against fontforge.
  • #1142496 filed against spopt.
  • #1142887 filed against go-dlib.
  • #1143147 filed against towncrier.
• Jochen Sprickerhof:
  • #1141412 filed against python-sphinx-chango.
  • #1141553 filed against gasnet.
  • #1141577 filed against yubihsm-connector.
  • #1141663 filed against dh-fortran.
  • #1142000 filed against watcher.
  • #1142001 filed against rakudo.
  • #1142128 filed against kf6-breeze-icons.
  • #1142141 filed against oxygen-icons.
  • #1142256 filed against gnu-apl.
  • #1142513 filed against barvinok.
  • #1143169 filed against ecbuild.

Misc news On our mailing list this month, Colin Winter of Markovian Protocol wrote to our mailing list on the topic of Reproducible verification for retained logs:

Reproducible builds remove trust in the builder: anyone re-derives the same artifact from the same source, byte for byte. The same shape applies one layer over, to a retained record. Most record-keeping regimes (the EU AI Act s Article 12 logging is the current example) require that events be recorded and logs retained, but not that a retained log be verifiable, by a party who was not present, as unaltered and existing when claimed. That leaves an integrity obligation resting on trusting the party being audited.

(Full thread)

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

• [1] https://etbe.coker.com.au/2008/09/15/the-problem-is-too-many-remote-controls/
• [2] https://en.wikipedia.org/wiki/VHS
• [3] https://etbe.coker.com.au/2025/11/25/edid-and-my-8k-tv/