Warning
This post contains interactive examples. To visualize and interact
with them, you need to leave your RSS reader.
Imagine you rent office space for a three-day event. You quickly set up a few
Ethernet switches and tape some cables on the floor to get everyone online.
Unfortunately, Stan, your clumsiest coworker, kicks out a cable every time he
gets up for coffee. You could add extra cables, but then you d get a broadcast
storm: Ethernet packets that loop and multiply until nothing else gets through.
That s where the spanning tree protocol (STP) comes in. STP blocks just enough
of your spare cables to leave a loop-free tree. When Stan strikes again, it
rebuilds the tree in a second, leaving some time for Blobby, your one-person
support crew, to reconnect the cable.1 See for yourself: the diagram
below runs a real STP implementation in your browser!
:demo
A1 @0,0 prio=4096
A2 @0,1
A3 @0,2
A4 @0,3
B1 @1,0 prio=8192
B2 @1,1
B3 @1,2
B4 @1,3
C1 @2,0 prio=8192
C2 @2,1
C3 @2,2
C4 @2,3
A1 -- A2 hazard=0
A2 -- A3 hazard=0
A3 -- A4 hazard=0
B1 -- B2
B2 -- B3
B3 -- B4
C1 -- C2 hazard=0
C2 -- C3 hazard=0
C3 -- C4 hazard=0
A1 -- B1 cost=10
B1 -- C1 cost=10
A4 -- B4 cost=20
B4 -- C4 cost=20
Leo @-0.3,0.7 proto=none icon=
Mia @-0.3,1.3 proto=none icon=
Joy @0.3,0.7 proto=none icon=
Roy @0.3,1.3 proto=none icon=
A2 -- Leo hazard=0 A2:edge
A2 -- Mia hazard=0 A2:edge
A2 -- Joy hazard=0 A2:edge
A2 -- Roy hazard=0 A2:edge
Max @-0.3,1.7 proto=none icon=
Zoe @-0.3,2.3 proto=none icon=
Ada @0.3,1.7 proto=none icon=
Amy @0.3,2.3 proto=none icon=
A3 -- Max hazard=0 A3:edge
A3 -- Zoe hazard=0 A3:edge
A3 -- Ada hazard=0 A3:edge
A3 -- Amy hazard=0 A3:edge
Eli @0.7,0.7 proto=none icon=
Jay @0.7,1.3 proto=none icon=
Kai @1.3,0.7 proto=none icon=
Ben @1.3,1.3 proto=none icon=
B2 -- Eli hazard=0.2 B2:edge
B2 -- Jay hazard=0.2 B2:edge
B2 -- Kai hazard=0.2 B2:edge
B2 -- Ben hazard=0.2 B2:edge
Ava @0.7,1.7 proto=none icon=
Lea @0.7,2.3 proto=none icon=
Ivy @1.3,1.7 proto=none icon=
Rex @1.3,2.3 proto=none icon=
B3 -- Ava hazard=0.2 B3:edge
B3 -- Lea hazard=0.2 B3:edge
B3 -- Ivy hazard=0.2 B3:edge
B3 -- Rex hazard=0.2 B3:edge
Ana @1.7,0.7 proto=none icon=
Eve @1.7,1.3 proto=none icon=
Abe @2.3,0.7 proto=none icon=
Ian @2.3,1.3 proto=none icon=
C2 -- Ana hazard=0 C2:edge
C2 -- Eve hazard=0 C2:edge
C2 -- Abe hazard=0 C2:edge
C2 -- Ian hazard=0 C2:edge
Ned @1.7,1.7 proto=none icon=
Lou @1.7,2.3 proto=none icon=
Fay @2.3,1.7 proto=none icon=
Sue @2.3,2.3 proto=none icon=
C3 -- Ned hazard=0 C3:edge
C3 -- Lou hazard=0 C3:edge
C3 -- Fay hazard=0 C3:edge
C3 -- Sue hazard=0 C3:edge
Note
This article is also available as a video, but I advise you to
keep reading here to try the interactive demonstrations.
The basics
Designed in the 80s, the spanning tree protocol has evolved into a rapid
flavor (RSTP) and a VLAN-aware variation (MSTP).2 Any sound-minded
network engineer knows there are better alternatives, like BGP EVPN VXLAN.
Yet, because any switch speaks it, the venerable spanning tree protocol still
fills a niche.
We focus on RSTP: it replaced the original protocol in 2004. To eliminate
network loops, RSTP implements a complex state machine. Timers, link state
changes, and the link-local control frames a bridge receives from its neighbors
drive its transitions. These Ethernet frames are the Bridge Protocol Data
Units (BPDUs). You can watch them in action below: hit the Start button.
After some time, the topology converges to a tree: from the root
C11, there is a path to each bridge3 and no loop. In the upper right
corner, the interface displays a tree icon followed by the time it took to
reach this state. Cut a link and see how the protocol
finds an alternate path to reach C12 in less than a second. You can stop the
simulation, move it forward step by step, reset it to its initial state, or slow
it down with the snail mode . Don t worry about all the displayed
information: I explain it later.
Note
While you scroll, the current simulation stays in view so you can
look at it while reading. Uncheck this box to disable this behavior: .
All examples run in your browser, powered by MSTPD an open-source
user-space4 implementation of RSTP.5
Historical interlude
Radia Perlman, an inductee of the Internet Hall of Fame in 2014,
summarized the ancestor of STP she invented at DEC with this poem, later
included in a US patent:
I think that I shall never see
A graph more lovely than a tree.
A tree whose crucial property
Is loop-free connectivity.
A tree which must be sure to span
So packets can reach every LAN.
First, the root must be selected.
By ID, it is elected.
Least cost paths from root are traced.
In the tree, these paths are placed.
A mesh is made by folks like me,
Then bridges find a spanning tree.
Radia Perlman, Algorhyme.
Electing the root bridge
To build a tree, RSTP first elects the bridge with the lowest bridge
identifier as the root bridge. The bridge identifier combines the priority
and the MAC address: 8192.6e:2b:10:a0:5f:29.
In the example below, S1 and S2 have priorities of 4,096 and 8,192: S1 becomes
root. S4 has a priority of 12,288, while S3 keeps the default priority of
32,768:6 S4 becomes root. S5 and S6 don t have a specific priority, so
the lowest MAC address wins and S5 becomes root.
Each non-root bridge chooses its root port, the one with the lowest-cost
path to the root. Unless you override it, each bridge derives the link cost
from the speed: 20,000 for 1 Gbps. In case of equality, the lowest port
identifier wins.
Each remaining port becomes a designated port if the BPDU it sends is
better than the BPDU it receives. Otherwise, it becomes an alternate port.
Later, if the root port goes down, the best alternate port becomes the new
root port. The tiebreakers for the best BPDU are:
In the example above, after convergence, S1 is the root bridge
because it has a priority of 4,096, while the other bridges have a priority of
32,768. All its ports are designated ports because the accumulated cost to the
root is 0.
S2 s port facing S1 becomes a root port because it has the lowest accumulated
cost to the root 20,000 vs 40,000. S3 has two ports facing S1, and the one with
the lowest port identifier becomes the root port 0x8000 vs 0x8001. The other
candidate is an alternate port because the remote port on the link sends a
better BPDU, with an accumulated cost of 0. On the segment between S2 and S3,
S2 s port wins: while both bridges have the same accumulated cost to the root
(20,000), S2 s bridge identifier is smaller 32768.02:00:00:00:00:01 vs
32768.02:00:00:00:00:02.
Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Rapid Spanning Tree (2) BPDU Type: Rapid/Multiple Spanning Tree (0x02) Root Identifier: 4096.02:00:00:00:00:00
Root Path Cost: 20000
Bridge Identifier: 32768.02:00:00:00:00:01
Port identifier: 0x8002
If you cut the active link between S1 and S3, S3 promotes
the best alternate port to root port. If you also disable the second
link, S3 chooses the remaining alternate port as a
root port. But if you disable the link between S1 and
S2, S2 needs a bit more work to elect a new root port
because it does not have an alternate port.
Unless a specific event happens, designated ports send BPDUs every 2
seconds.9 If a bridge does not
receive BPDUs from its neighbor for 3 consecutive hello periods, it considers
the neighbor dead and removes the port information.
Port state transition
Each port can have one of three states. The diagram displays a background color
for each state:
discarding (red),
learning (yellow), or
forwarding (green).
A root port transitions automatically to the forwarding state. An alternate
port stays in the discarding state. A designated port has two options to
transition from the discarding state to the forwarding state:
If the port is an edge port, either through configuration or because the
remote device does not speak any flavor of STP, the bridge assumes the device
won t participate in the protocol and cannot create a loop. In this case, the
designated port immediately transitions to the forwarding state.
Otherwise, it sends a proposal to its downstream neighbor. If the remote
bridge agrees that the received BPDU is better than any other BPDU stored
for other ports, it elects the receiving port as its root port and starts the
synchronization process: it transitions all non-edge non-synced designated
ports to the discarding state to avoid a loop. Then, it sends back an
agreement. Upon receiving the agreement, the peer designated port
transitions to the forwarding state.10
In the topology above, H1, H2, H3, and H4 are end devices not participating in
the protocol. We configure the ports they connect to as edge ports, so these
ports immediately move to the forwarding state.
Use the step button to move the simulation forward. The clock moves to 1
second. Step again and S1 and S2 send a proposal to
each other. Here is the proposal from S2:
Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Rapid Spanning Tree (2) BPDU Type: Rapid/Multiple Spanning Tree (0x02) BPDU flags: 0x4e, Agreement, Port Role: Designated, Proposal
0... .... = Topology Change Acknowledgment: No
.1.. .... = Agreement: Yes
..0. .... = Forwarding: No
...0 .... = Learning: No
.... 11.. = Port Role: Designated (3).... ..1. = Proposal: Yes
.... ...0 = Topology Change: No
Root Identifier: 32768.02:00:00:00:00:01
Root Path Cost: 0
Bridge Identifier: 32768.02:00:00:00:00:01
Port identifier: 0x8001
S1 ignores it: its own root identifier is lower. When S2 receives a similar
proposal from S1, it accepts S1 as its root bridge. It also elects the port to
S1 as the root port and starts the synchronization process. The two designated
ports are already discarding, so no change here. Step
again and S2 sends two BPDUs to S1. In one of them, the
agreement bit is 1 and the proposal bit is 0. It also shows that S2 accepted S1
as the root bridge and its root port is now in the forwarding state. When
receiving this BPDU, S1 transitions its own designated port to the forwarding
state. From this point, the link between S1 and S2 forwards user traffic.
Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Rapid Spanning Tree (2) BPDU Type: Rapid/Multiple Spanning Tree (0x02) BPDU flags: 0x79, Agreement, Forwarding, Learning, Port Role: Root, Topology Change
0... .... = Topology Change Acknowledgment: No
.1.. .... = Agreement: Yes
..1. .... = Forwarding: Yes
...1 .... = Learning: Yes
.... 10.. = Port Role: Root (2).... ..0. = Proposal: No
.... ...1 = Topology Change: Yes
Root Identifier: 4096.02:00:00:00:00:00
Root Path Cost: 20000
Bridge Identifier: 32768.02:00:00:00:00:01
Port identifier: 0x8001
Let s look at what happened to S5. Reset the simulation and step
twice. S5 exchanges BPDUs with both S3
and S6. Since S5 has a lower root identifier than S3 and S6, it stays the root
bridge, while S3 and S6 accept the proposal and elect their root ports. S3 and
S6 start the synchronization process. S6 s port to H4 keeps forwarding because
it is an edge port. Move one step. Both S3 and S6 send
an agreement back to S5, which transitions both designated ports to the
forwarding state. Yet, the link between S5 and S3 keeps discarding user traffic!
If you look carefully, S3 s port toward S5 is now a designated port, not a root
port. During the same step, S3 also receives a better BPDU
from S2 with S1 as the root bridge. It elects its port to S2 as the root port
and downgrades the port to S5 to a designated port, which stays in the
discarding state.
On the next step, things get a bit tricky. S3 sends a
proposal to S5:11
Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Rapid Spanning Tree (2) BPDU Type: Rapid/Multiple Spanning Tree (0x02) BPDU flags: 0x4f, Agreement, Port Role: Designated, Proposal, Topology Change
0... .... = Topology Change Acknowledgment: No
.1.. .... = Agreement: Yes
..0. .... = Forwarding: No
...0 .... = Learning: No
.... 11.. = Port Role: Designated (3).... ..1. = Proposal: Yes
.... ...1 = Topology Change: Yes
Root Identifier: 4096.02:00:00:00:00:00
Root Path Cost: 40000
Bridge Identifier: 32768.02:00:00:00:00:02
Port identifier: 0x8002
S5 elects S1 as its root bridge and the port toward S3 as its root port. It
starts its synchronization process, but the designated port to S6 does not
move into the discarding state. Why? That port stays a designated port and its
neighbor S6 has already sent an agreement on the link, so the port keeps its
synced status.
Now, let s step back to look at what happens to S6. At this point,
S6 believes S5 is the root bridge. Step once and S4 sends
a new proposal to S6. S6 accepts the proposal, elects S1 as the root bridge and
the port to S4 as its root port. The role of the port facing S5 changes: from a
root port, it becomes a designated port. Because its peer keeps advertising an
inferior BPDU on the link, this port becomes disputed and moves to the
discarding state. The root port transitions to the forwarding state and the link
starts forwarding immediately because S4 s designated port is already in the
forwarding state. If we step one more time, S5 and S6
exchange two BPDUs. The one from S5 is better because of its lower bridge
identifier. S5 s port stays a designated port, while S6 downgrades its own port
to an alternate port.
Let s rewind to the start one last time: cut the link between S1 and S2, run
the simulation until the topology is stable, stop the
simulation, and restore the link between S1 and S2. During the first
step, S1 and S2 exchange proposals. S2
elects S1 as the root bridge instead of S5 and the port to S1 as the root port.
It downgrades the previous root port to a designated port and moves it into the
discarding state. The other designated port stays synced and keeps its
forwarding state. At the next step, S2 sends
an agreement to S1 and the link between them starts forwarding user traffic. It
also sends a proposal to S3, but not to S4.
Instead, it sends a regular BPDU to S4. S4
still elects S1 as its root bridge and the port to S2 as its root port. It
demotes its previous root port, the one to S3, to a designated port, which
transitions to the discarding state because of the root port change. The other
alternate port, to S6, also becomes a designated port and stays in the
discarding state. The new root port moves to the forwarding state. On the next
step, S4 s port to S3 settles as an
alternate port after receiving a better BPDU from S3.
RSTP is a giant state machine split into smaller ones: bridge detection, port
information, port protocol migration, port role selection, port role
transitions, port receive, port state transitions, port timers, port transmit,
and topology change. Some of them are per bridge, some per port. Each bridge
runs an instance. Time, operational port state changes, and the BPDUs it
receives from other instances drive the transitions. Being event-driven makes
RSTP more efficient but also more difficult to understand.
Placeholder for the Port Information state machine extracted from IEEE 802.1Q-2005, page 182. Pending IEEE authorization for reproduction, this is the blueprint for the Western Australian Government Railways class Msa Garratt articulated steam locomotive.
Topology change notification
A bridge populates a MAC address table: it associates each source MAC address
with the port that last received it. When forwarding an Ethernet frame, it looks
up this table to choose the right port.12 When a link fails, a connected
fridge reachable through one port may become reachable through another one. The
affected bridges should flush the MAC addresses they learned, because these
entries may now be wrong.
For this purpose, RSTP implements topology change notifications using a
flooding mechanism. When a non-edge port transitions to the forwarding state, a
bridge generates BPDUs with the topology change (TC) bit set. It sends them to
all the non-edge designated ports and to the root port. It also flushes the MAC
address table on these ports. When a bridge receives such a BPDU, it propagates
the notification to all non-edge designated ports and the root port, except the
one the notification came from. It also flushes the MAC address table on these
ports. In the examples, the BPDUs with the TC bit set to 1 have a red circle.
Start the simulation and wait a few seconds for the topology to
settle. Stop the simulation and disable the link between S2 and
S5. S5 elects the port facing S4 as the root port, which
transitions immediately to the forwarding state. Step
once and S5 emits a BPDU with the TC bit set to 1:
Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Rapid Spanning Tree (2) BPDU Type: Rapid/Multiple Spanning Tree (0x02) BPDU flags: 0x79, Agreement, Forwarding, Learning, Port Role: Root, Topology Change
0... .... = Topology Change Acknowledgment: No
.1.. .... = Agreement: Yes
..1. .... = Forwarding: Yes
...1 .... = Learning: Yes
.... 10.. = Port Role: Root (2).... ..0. = Proposal: No
.... ...1 = Topology Change: Yes
Root Identifier: 4096.02:00:00:00:00:00
Root Path Cost: 40000
Bridge Identifier: 32768.02:00:00:00:00:04
Port identifier: 0x8002
S4 receives this BPDU. It flushes the MAC address table on the port facing S1:
while LPT was previously reachable through this port, it is now reachable
through S5. Step once. S4 sends S1 a BPDU
with the TC bit set to 1. When S1 receives this BPDU, it flushes the MAC address
table on the ports facing S2 and S3. Step
once and S1 sends a notification to S2 and
S3. Step once again and S2 sends a notification to
S3, while S3 does nothing because the port toward S2 is an alternate port. S3
does not flush any MAC address table: LPT is still reachable through its port to
S1.
If you step a bit more, you will see that some of the
periodic BPDUs keep the TC bit set to 1. Each port runs a timer equal to the
hello timer plus one second.13 The timer starts when the port emits a
notification. Until it expires, the port sets the TC bit to 1 in every BPDU it
sends. You can also see some periodic BPDUs
without the TC bit: they originate from a port that only received a notification
and therefore did not arm its timer.
Security
RSTP is weak against configuration errors and malicious actors. A bridge not
talking RSTP can create a loop. An attacker can insert themselves into the
topology to disrupt the service, spy on the traffic, or alter it.
To mitigate such problems, you need to identify the edge ports. An edge port
connects to an end device, like a PC or a printer. Such devices do not generate
BPDUs and cannot create a loop. RSTP defines two related flags:
When true, AdminEdge initializes a port as an edge port. It defaults to
false.
When true, AutoEdge lets a port become an edge port when it does not
receive BPDUs for 3 seconds. It defaults to true.
If an edge port receives a BPDU, regardless of the values of these two flags, it
reverts to a non-edge port.
In the topology above, S1, S2, S3, S4, S5, and S6 act as bridges, while H1, H2,
H3, H4, H5, and H6 act as end devices:
S1 and H1 are on a port without a specific configuration: AutoEdge is true,
AdminEdge is false,
S2 and H2 are on a port where AdminEdge is true,
S3 and H3 are on a port where AutoEdge is false and AdminEdge is true,
S4 and H4 are on a port where AutoEdge is false.
If you start the simulation and wait about 20 seconds, links to S1,
S2, S3, S4, H1, H2, H3, and H4 eventually forward user traffic: none of the
flags matter.
But what about the two remaining pairs? S5 and H5 connect to a network port.
Such a port enables a non-standard feature: bridge assurance. The port
transmits BPDUs regardless of its role. If it does not receive BPDUs for 3
consecutive hello periods, it transitions to the discarding state. On the link
between R0 and S5, you can see BPDUs traveling in both
directions, unlike the other links, where only
designated ports send BPDUs.
S6 and H6 connect to a port where AdminEdge is true and BPDU guard is
enabled. This is another non-standard feature that shuts down a port if it
receives a BPDU.
In summary, if you expect a port to be an edge port, you should set AdminEdge
to true and enable BPDU guard. Otherwise, declare it as a network port.
Why RSTP today?
A compelling use case for RSTP today is an out-of-band network for a datacenter,
since you can tolerate an outage of a few seconds. The configuration is minimal
and you can use cheap switches, like a Cisco 2960X.14 You need two
switches acting as root bridges, and you build several loops to connect OOB
switches in each cabinet. This simple design survives one failure on each
loop.15
This topology converges in about 6 seconds. Each loop should stay
small (around 16 bridges) to reduce the probability of a double failure and to
avoid sharing too much bandwidth. The design can evolve a bit without adding too
much complexity: one VLAN per loop or one bridge domain per loop.
How large can a network be?
The maximum age, whose default value is 20, governs the maximum distance of a
bridge from the root. The topology below is too big for BPDUs from R1 to reach
beyond S20.16
Once the topology settles, part of the network considers R1 the
root, while the other votes for R2. At the boundary, S20 tries to start a
synchronization with S21 to move its designated port to the forwarding state.
The BPDU looks like this:
Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Rapid Spanning Tree (2) BPDU Type: Rapid/Multiple Spanning Tree (0x02) BPDU flags: 0x4e, Agreement, Port Role: Designated, Proposal
Root Identifier: 4096.02:00:00:00:00:00
Root Path Cost: 400000
Bridge Identifier: 32768.02:00:00:00:00:15
Port identifier: 0x8002
Message Age: 20
Max Age: 20
S21 rejects it because the message age equals the maximum age. On the other
hand, the BPDU S21 sends to S20 looks like this:
Spanning Tree Protocol Protocol Identifier: Spanning Tree Protocol (0x0000) Protocol Version Identifier: Rapid Spanning Tree (2) BPDU Type: Rapid/Multiple Spanning Tree (0x02) BPDU flags: 0x7c, Agreement, Forwarding, Learning, Port Role: Designated
Root Identifier: 4096.02:00:00:00:00:01
Root Path Cost: 320000
Bridge Identifier: 32768.02:00:00:00:00:16
Port identifier: 0x8001
Message Age: 16
Max Age: 20
This is not enough to change S20 s root port because S20 has a lower root
identifier4096.02:00:00:00:00:00 vs 4096.02:00:00:00:00:01.
Fixing the link between R1 and R2 resolves the issue. The
maximum message age any packet carries is now 18, below the configured maximum
age. But it only works until another link breaks. A plausible fix is to increase
the maximum age to 40.17
How fast is RSTP?
RSTP usually converges in a couple of seconds at startup. It often repairs a
tree in less than a second. Even the 38-bridge topology takes less than 10
seconds to converge.18 Some topologies can take a bit more time to recover
when the root bridge becomes unavailable.19
In the topology above, start the simulation, wait for convergence,
hit stop, and cut the link between R0 and S1. The topology is
already optimal, but RSTP has a hard time converging again.
First, S1 loses its root port. It has no more information about R0 and elects
itself as the root bridge. It keeps its ports to S2 and S3 as designated ports
in the forwarding state. Step once and it
sends a BPDU to both S2 and S3 to let them know about the root change. When
receiving it, S2 accepts S1 as its root because it does not have a better root
on another port. It elects the port to S1 as its root port. The other port stays
a designated port. Both ports keep forwarding.
When receiving the BPDU from S1, S3 behaves differently: it knows R0 as a better
root than S1 through its alternate port to S2. It promotes this port to a root
port and demotes the port facing S1 to a designated port, which requires a new
agreement. Step once and S3 sends a proposal to
S1 with R0 as the root bridge. S1 elects R0 as the root bridge and promotes its
port to S3 as a root port.
During the same step, S3 also receives a BPDU from
S2 stating that S1 is the root bridge. Therefore, S3 has no port left with R0 as
the root bridge: it elects S1 as the root bridge and its port to S2 as the root
port. Step once and its next BPDU to S1 includes
this information: S1 elects itself again as the root bridge. But during the
same wave, S1 sends a proposal to S2 with R0 as
the root bridge. While S1 and S3 agree that S1 is the root bridge, S2 now
believes this is R0! In turn, S2 again convinces S3
that R0 is the root bridge, S3 convinces S1, S1 convinces S2, and S2 convinces
S3.
This could go on forever, but it does not. The BPDUs saying R0 is root
eventually age out when the message age goes past the maximum age. In the
example above, at the eleventh second, S2 sends a
BPDU to S3 with R0 as root, but S3 drops it because its message age reached the
maximum. With some luck, the topology can also converge faster if a port stops
transmitting new BPDUs after tripping the transmit hold count, whose default
value is 6 per second.
About MSTP
MSTP is the VLAN-aware version of RSTP: it runs several instances of RSTP and
lets the administrator map each VLAN to a specific instance. For example, you
can map VLANs 100 to 200 to a first instance, and 300 to 400 to a second
instance. The remaining VLANs map to a special instance named the Internal
Spanning Tree (IST). MSTP adds its own complexity, but the gist is that you have
several logical topologies acting independently. If you want to dig deeper, have
a look at MSTP Tutorial Part I: Inside a Region.
About the interactive examples
The interactive examples run MSTPD directly in your browser, compiled to
WebAssembly with emscripten. A C API replaces the code talking to the
Linux kernel: it manages bridges and ports, exports state as JSON, and drives
time deterministically. A JavaScript wrapper makes it more user-friendly:
importloadMSTPDfrom"./dist/mstpd.mjs";constmstp=awaitloadMSTPD();// Create 3 bridgesconsta=mstp.createBridge("A",priority:4096 );constb=mstp.createBridge("B",priority:8192 );constc=mstp.createBridge("C");// Each bridge has two portsconsta1=a.addPort("a-b",portno:1 );consta2=a.addPort("a-c",portno:2 );constb1=b.addPort("b-a",portno:1 );constb2=b.addPort("b-c",portno:2 );constc1=c.addPort("c-a",portno:1 );constc2=c.addPort("c-b",portno:2 );// Build a triangle topologymstp.link(a1,b1);mstp.link(a2,c1);mstp.link(b2,c2);// Enable all bridges and portsfor(constbrof[a,b,c])br.enable();for(constpof[a1,a2,b1,b2,c1,c2])p.enable();// Execute 40 seconds' worth of wall clock and display the topologymstp.step(40);console.log("Topology:",mstp.topology());
Several dozen unit tests explore the features of MSTPD and check that they work
correctly in this environment:
$ node--test*.test.mjs
two bridges: lower priority becomes root (41.657342ms) triangle loop: exactly one port blocks and all agree on the root (5.832ms) breaking the active link reconverges and restoring recovers (18.730753ms)[ ] tests 40 pass 40 fail 0[ ] duration_ms 396.190897
Additional JavaScript code looks for specific <pre> blocks containing a
topology definition and turns them into interactive widgets. You can inspect
and modify the definition by hitting the edit button.
There is also a cool trick to tell whether the topology has converged. After
each step, we save a snapshot of the simulation memory, play 50 seconds worth
of simulation to check if the topology is stable, and travel back in time by
restoring that snapshot.
The complete code lives on GitHub. I am happy with the result. It can be
difficult to follow everything happening during a single step, but stepping
forward and backward helps. I plan to use the same approach in future blog posts
about networking features.
Note
Michael Lynch reviewed a first draft of this article. He authored
Refactoring English, a book to sharpen your writing for blog posts,
documentation, commit messages, and tutorials. Any errors are still mine!
The sprites for Stan and Blobby come from Craftpix, the coffee
cups from Yanin.
The priority is a multiple of 4,096: with MSTP, the lower 12 bits
of the bridge priority encode the MST instance identifier, leaving only the
upper 4 bits for the configured priority.
To inspect the BPDUs crossing a link, select it, click the
Download packets button, and open the file with Wireshark.
A backup port only exists if the bridge has several ports on the same
collision domain. This should not happen in a switched network.
This is the value of the hello timer. It used to be configurable,
but IEEE 802.1Q-2005 pins it to 2. MSTPD does not allow another value.
If the peer port does not receive an agreement after the hello
timer elapses or the maximum age if the port has just come up it falls back
to the timer-based method for compatibility with STP: it transitions to
the learning state, waits again for the hello timer to expire, and
transitions to the forwarding state.
As in many proposals, S3 also sets the agreement bit to 1. The
proposal bit says I am the designated port on this link and I want to
transition to the forwarding state. The agreement bit says I am already in
sync with the rest of my bridge on this root information. Both can be true.
If it finds no entry, the bridge duplicates the Ethernet frame on all
ports, except the incoming one. The same happens if the destination MAC
address is the broadcast one (ff:ff:ff:ff:ff:ff). This behavior bootstraps
the learning process.
This timer makes RSTP resistant to packet loss.
You can get them for less than US$100 through a broker. All the ports
run PVST+ by default and automatically fall back to plain RSTP.
If you look closely at what happens at t=2s, you can see that R2 is
gaining popularity as root: S17 to S36 believe R2 is the root bridge. S16
does not follow because we hit the maximum age. Later, S17 to S20 reverse
their position. I ll let you explore the state of the various bridges to
understand the root cause.
When increasing the maximum age to 40, you also need to
increase the forward delay to 21 (:forward-delay 21), as the standard
enforces this condition: 2 (Forward Delay 1) Max Age. For this
specific topology, you could also increase the maximum age to 37 and
forward-delay to 20.
The simulation may seem slow, but it does not run in real time. Look at
the current timestamp in the upper right corner to know the wall clock, e.g.
t=8s. Once the topology stabilizes, the same corner shows the convergence
time, e.g. 2s.
A new minor release 0.4.28 of RProtoBuf
arrived on CRAN today. RProtoBuf
provides R with bindings to the
Google Protocol Buffers
( ProtoBuf ) data encoding and serialization library used and
released by Google, and deployed very widely in numerous projects as a
language and operating-system agnostic protocol. The new release is also
already as a binary via r2u.
This release corrects a really old bug. Troy found, when working on
gRPC based extensions, which is in and by
itself exciting, that a small part of our interface surface (for service
descriptors) was just wrong confusing single and double underscores.
adjusts to a change upstream. This has been corrected. I updated a few
of the usual continuous integration parts, updated a help page for a
newly-added nag by CRAN, and
also got a last-minute round of noodling in as the JSS paper vignette
was still referencing OmegaHat which the CRAN URL checker objected to. I
created a quick one-off repo to serve pdf files should the need arise
again, and rebuilt the vignette linking to it. No other changes.
The following section from the NEWS.Rd file has all details and
links.
Changes in
RProtoBuf version 0.4.28 (2026-08-21)
Standard maintenance of continuous integration
The type help page has received a usage section
Cleanup of several methods for ServiceDescriptor, correct several
other declaration (Troy Hernandez in #117
fixing #116)
Adjusted vignette reference to Omegahat paper to alternate
location
Another maintenance release of RcppMsgPack
got onto CRAN today. MessagePack itself is an efficient
binary serialization format. It lets you exchange data among multiple
languages like JSON. But it is faster and smaller. Small integers are
encoded into a single byte, and typical short strings require only one
extra byte in addition to the strings themselves. RcppMsgPack
brings both the C++ headers of MessagePack as well as clever code (in
both R and C++) Travers wrote to access MsgPack-encoded objects directly
from R.
This release is once again chiefly maintenance. Besides standard
upkeep to the README.md and continuous integration setup we had to add
one #include. The clang++-23 compiler, when
also running with its own library, now now needs the
type_traits.h header file (in the upstream MessagePack code) so we added that. No
other changes, so no user-facing changes. Details follow from the NEWS
file.
Changes in version 0.2.5
(2026-08-19)
Explicitly include header "type_traits.h" to appease
clang++-23
Standard maintenance updating continuous integration, adding
minor helper script, and updating README.md
While many Western democracies seem to be in a state of decay and are no longer the safe, civilized and prosperous countries they once were, there are still some European countries that are governed well. One of those that stand out is Estonia.
Estonia is probably most well known for multiple software companies that originated from there, such as Wise, Bolt, Pipedrive and Skype. The government itself is also famous for being early in issuing government IDs with an embedded smart chip for online authentication already in the 1990s. Via the national portal at eesti.ee all residents can access extensive eServices ranging from viewing their health benefits to filing taxes.
Estonia has also been running an e-Residency program since 2014, where they issue digital ID cards to foreigners, making it easy for them to remotely log into the government portals and for example, establish businesses, file annual reports and so forth. Note that the e-Residency is not a path to physical residency. Estonia does, however, have a separate Digital Nomad visa program that makes it easy for non-EU citizens to also physically establish themselves in Estonia, assuming, of course, you meet the criteria, which includes, among others, a minimum monthly income of 3960 from outside Estonia. EU citizens naturally have free mobility inside the EU and can simply get an apartment and register as a resident in Estonia if they so choose. And there are plenty of reasons to do so.
Estonians value health, education and entrepreneurship
I moved to Estonia about one and a half years ago. In my observations Estonia strikes me as a country that values health, education (in particular programming and natural sciences) and entrepreneurship highly.
I don t know how Estonians achieve it, but they look pretty fit and rarely obese. Estonia has rye bread and sauna in their culture just like Finland, and in addition the flat terrain and well-planned bike routes and extensive network of parks (and pull-up bars everywhere) seem to create an environment where it is easy to live in a healthier way. The consumption of processed foods and candy also seems relatively low among Estonians.
The gym chain MyFitness also seems to be present everywhere. Even the Tallinn airport has a calisthenics workout station right at the departure gates, which anyone is free to use while waiting for their flight to take off. One of the top longevity influencers in Europe, Siim Land, is Estonian.
Estonians also seem to value the school system highly. The government has been actively raising teacher pay and has a stated goal of reaching 120% of the national average by 2027. The students are also expected to value the education and respect their teachers. According to the TALIS 2024 survey (OECD s international teacher survey), Estonian teachers spend significantly more time on actual teaching and learning and waste less time on interruptions or keeping classroom order compared to the OECD average. This is among the highest rates internationally, meaning they spend relatively little time on maintaining order or dealing with disruptions. While the international education benchmark PISA scores have been dropping globally, Estonia has consistently been climbing the ranks in past decades. In the latest PISA study (from 2022), Estonia ranked number one in Europe for reading, mathematics and science. In the overall results, Estonia ranked seventh globally, only behind countries such as Japan, Korea and Singapore.
Valuing entrepreneurship is evident in how the taxation system is set up in Estonia. Famously, in Estonia, companies can defer taxes on annual earnings and reinvest all of their profit in growing the company. Taxes are due only later, when paid out from the company, for example as dividends. For individuals, receiving dividends from any Estonian or foreign company is tax-free as long as the company paying dividends already paid corporate tax on the same income.
The tax system is also very simple. For all individuals, all types of income, including salary and capital gains, are always taxed at a flat rate of 22%. This removes the incentive for anyone to try to convert income into different types, setting up holding company structures and other optimizations as there is no gain. All entrepreneurs can simply focus on growing their business and forget extra bureaucracy. There is also no marginal tax rate cliff to stop working at everyone is encouraged to always try to produce as much value as they can. A simple tax system is also reflected in the fact that anyone can easily read all tax rules that apply to individuals in plain English on the Estonian tax authority website, and one does not need to hire any accountants simply to file taxes.
Estonia also has a very straightforward investment account system: any person can freely open a self-directed investment account at any brokerage at no extra cost and report it as such to the tax authority, and then use it to save for an apartment, retirement, or other purposes, and defer all income taxes until withdrawal. There are no caps or time limits, and all residents are encouraged to save and invest as much as they can and thus take responsibility for their own wealth accumulation.
It seems that culturally Estonians respect people who are active and progress in their careers and businesses more than in other countries. Unlike in Finland, successful people are admired and living on government welfare is not romanticized. At the same time, the government benefits are less generous so people can t live comfortably on them and, for example, many of the asylum seekers Estonia accepted have since left on their own initiative to other European countries in search of better benefits.
Low crime, high trust
Another thing that strikes me when walking the streets of Tallinn is that there are no drug addicts, beggars, thugs or the like. The difference compared to, for example, Vancouver (where I lived previously) is stark. In public buildings, people leave their coats and bags hanging in the lobby while visiting. Private houses and apartment block yards are not fenced. In my building, I noticed people even leave their bikes unlocked in the bike shed. I have also seen the staff of a coffee stall in a shopping mall going for a break and leaving everything unattended, not worrying that anyone would take anything while the staff is away.
This is not just my personal experience. According to the Numbeo crime index, Estonia has one of the lowest crime rates in the world. Also, comparing drug and property crime stats, for example, Finland has twice as much crime per capita, and places like Vancouver in Canada almost five times more.
I don t have any clear explanation for why crime is so much lower in Estonia, but some suggest that higher social cohesion and lower levels of welfare contribute to people standing to lose more if they behave antisocially. Compared to Finland, Estonia also more readily jails repeat offenders, and those who are put on trial will experience a swifter court process thanks to simplified legal processes and more efficient governance.
Moving to Estonia: quick checklist
Moving to Estonia is very easy for any EU citizen, in particular if your work is not location-dependent (e.g., remote work or an online business) and you are simply looking for the cleanest and safest environment to live in.
First, check into a hotel in Tallinn, check out various neighborhoods to figure out what area you like (my favorites are Kalaranna, Kalamaja, Noblessner and Volta) and start browsing available apartments in English at kv.ee. Most professionals speak fluent English, so there should not be any difficulty in reaching out to people by email or phone.
The next step is to buy a local prepaid SIM card at e.g., an R-Kiosk or a convenience store as signing up for other matters later on will require an Estonian phone number. Once you have an apartment and e.g., signed a rental agreement, you can register in the population registry online.
After that, you have proof you are a local resident with an address and telephone number in Estonia. With local resident status, you can go to the police station to get a local ID card. Don t bother scheduling an appointment, just go to the Tammesaare police station in Tallinn, take a queue number and wait. Once it is your turn, they will guide you on how to use the photo booth, fingerprint registration device, and file your application. A few days later you will receive an email confirming whether your application was accepted, and after a few more days, you will get another email notifying you that your ID card has been printed and is available for pick-up at the same location. This will also be your first practical experience of how fast and efficient the government in Estonia is.
Once you have the local ID card, you can use the smart card feature to log into all the government eServices and sort out the rest of the relocation process, such as registering tax residency and getting a family doctor.
How did Estonia evolve to be like this?
After Estonia regained its independence after the fall of the Soviet Union in 1991, the first elected government in 1992 was led by a very progressive 31-year-old Prime Minister Mart Laar, who managed to set up some very good policies and laid the foundation of a society that has evolved well in the decades since. Estonia was very lucky to have people in power in the 1990s who didn t simply copy what other Western countries were doing, but who tried to think about things from first principles and create Estonia s own model for efficient and fair governance. As a post-Soviet country, the population had also been vaccinated against overly socialist and unrealistic ideals, and everyone had a healthy distrust of the government s ability to solve problems and emphasis was placed on people s liberty to work for themselves as they best see fit. The improvement in living standards over the past 35+ years has also been witnessed by the population, and voting behavior supports keeping the country on the same trajectory.
General living standards still continue to improve as the nominal wage growth sits at around 6%, clearly above the annual inflation rate of about 3%. In 2026, the Estonian economy is expected to grow about 2.4%, which is faster than the EU average. The growth in Estonia is not the result of any accounting tricks - Estonia is part of the euro and can t print its own currency, nor has it been funding the public sector with excessive debt. With a 24% debt-to-GDP ratio, Estonia consistently ranks as one of the most responsibly managed countries among advanced Western economies.
As wages in Estonia soon catch up with the EU average, and higher defence spending has forced the government to raise taxes in recent years, the economic growth that Estonia has enjoyed for 35+ years since it exited the Soviet Union might slow down a bit in future years. The policies that fueled this growth in living standards, however, are likely to stay.
The tiger leap
There is one specific government policy in Estonia s history that I think should be highlighted in particular. Estonia was very progressive by announcing the Tiigrih pe (Tiger Leap) program in 1996 with the goal of equipping all schools with computers and teaching all students the basics of programming. This surely had a large influence on why Estonia has so many successful software companies, why the government eServices are so mature that even neighboring countries like Finland are striving to copy the Estonian government s IT architecture called X-road.
The Tiigrih pe project was originally suggested in the mid-1990s by Toomas Hendrik Ilves, then ambassador of Estonia to the United States, Canada and Mexico, and later President of Estonia in 2006 2016. While he was a psychologist by education, he was also a self-taught amateur programmer and used his political influence to promote sensible adoption of information technology in both Estonia and the EU.
The history of Estonia has several prominent figures who were not lawyers by profession but engineers, scientists and historians who were very practical in their political decisions, which I think is now reflected in how government processes were formed.
The video below shows how the Estonian government advertises itself and what values they choose to highlight:
Should other countries adopt policies from Estonia?
Of course, not everything is perfect in Estonia either. The fertility rate of 1.16 in Estonia is very low. This trend is present globally, but in Estonia it is way below the EU average. Also, the service culture is something that needs to improve in Estonia. While services are in general fast and efficient, the attitude of people working in cafes and stores does not reflect a willingness to fill in gaps if the standard process falls short, nor are visitors actively made to feel welcome as individual humans, but are treated as mere process inputs.
However, many of the things listed earlier I think should be studied by policymakers elsewhere. Societies are complex systems and there is of course no guarantee that copying a single policy to another country with different ethnicities, history and ingrained culture would lead to the same policy outcomes. But considering that Estonia is a small country without favorable geography and no natural resources, and that it started out from a place of total chaos, low economic activity and high crime in 1992 to rise to what it is now in 2026, the success it has seen is surely largely a result of good policies, governance and culture that other countries can and should mimic.
Following up on my previous post, I am releasing
debvulns-exporter, a Prometheus exporter
for tracking Debian system vulnerabilities. The underlying vulnerability
analysis logic remains identical to the previously released MCP server and CLI
utility.
Why an Exporter?
In my engineering workflows, I frequently deal with Debian and vulnerability
management. Most enterprise environments rely on commercial, paid vulnerability
platforms like Tenable or Rapid7. While these platforms provide extensive
feature sets, I noticed a distinct lack of open-source tools tailored for this
specific pipeline. While debsecan exists, it lacks a structured, parseable
format suitable for building dashboards aimed at management consumption. What
started as an experimental MCP server for learning purposes evolved into a
practical question: why not convert it into a Prometheus exporter? Given that
Prometheus is the de facto standard metrics platform across the industry, this
architecture was the logical next step.
Design and Exported Metrics
The exporter is implemented as a native Prometheus exporter utilizing the
prometheus-client library. It operates using two threads: one handles fetching
the vulnerability data, parsing EPSS feeds, and cross-referencing installed
packages to identify local vulnerabilities; the second handles serving the
metrics endpoint. The full architecture details and metrics specifications can
be found in the design doc.
The specification was drafted during a technical brainstorming session with
Claude 4.6 Sonnet on Antigravity prior to writing the implementation.
Testing and Dashboarding
To validate the exporter, I spun up older Debian 11 and Debian 12 cloud
images sourced from the Debian Cloud team. The older image was intentionally
selected to guarantee a standard baseline of unpatched vulnerabilities for
testing. The local evaluation topology is structured as shown below:
Rather than constructing the Grafana dashboard from scratch, I used Claude 4.6
Sonnet via Antigravity to generate the layout configuration. The generated
dashboard for the local testbed functions effectively:
The complete, ready-to-import Grafana dashboard configuration is included
directly in the debvulns source code.
Renaming the project
To prevent namespace conflicts and confusion with the native debsecan utility
in Debian, I have unified the ecosystem under the debvulns moniker. The core
CLI is named debvulns, the exporter is debvulns-exporter, and the MCP
component is debvulns-mcp. The migration release has been published to PyPI, and the new consolidated repository
is active at debvulns.
Conclusion
While this began as a personal utility to fill a niche tool gap, I expect it
will be useful for others managing Debian infrastructure at scale. My next
objective is to formalize Debian packaging for both the CLI and the exporter.
The MCP component will likely remain available as an independent artifact. Until
then, happy hacking.
Note: As a core design choice, debvulns still uses native debsecan as its
ground-truth standard. The tool continuously cross-verifies its output against
debsecan to ensure perfect functional parity and data consistency.
Nextcloud PHP debugging with Xdebug in Zed editor
I started to switch from PhpStorm to Zed as IDE recently as Zed is open source
and has a much smaller footprint and is more slick than PhpStorm.
One thing that I didn't get running immediately was Xdebug integration, so I did
a bit of research and asked Claude for help. Here's a quick writeup of how to
get it running.
I have Zed installed as Flatpak on a Debian Trixie host system.
The PHP process runs in a nextcloud-docker-dev
Docker container.
Install Zed and configure debugging there
Install Zed: flatpak install flathub dev.zed.Zed
In Zed: open the Extensions view and install PHP.
Configure the debugger:
Create ~/.var/app/dev.zed.Zed/config/zed/debug.json:
Add one entry per bind-mounted app directory.
After creating the file, restart Zed.
Inside Zed, select "debugger: start" from command palette and then "PHP: Listen to Xdebug".
Verify Zed is listening. Running ss -tlnp grep 9003 on the host should show *:9003 with Zed as the process.
Configure Xdebug inside the container
/usr/local/etc/php/conf.d/xdebug.ini:
xdebug.mode = debug
xdebug.idekey = PHPSTORM
xdebug.trace_output_name=trace.%R.%u
xdebug.profiler_output_name=profile.%R.%u
xdebug.output_dir=/shared/xdebug
xdebug.log = /var/log/xdebug.log
xdebug.log_level = 3
; Try to discover the client host, otherwise fall back to the docker host
xdebug.discover_client_host=true
xdebug.client_host=host.docker.internal
; When you cannot specify a trigger, use "xdebug.start_with_request = yes" to autostart debugging for all requests
; https://xdebug.org/docs/all_settings#start_with_request
xdebug.start_with_request = trigger
; Set xdebug.mode trace to use this
; More details at https://derickrethans.nl/flamboyant-flamegraphs.html
xdebug.trace_format=3
xdebug.trace_output_name=xdebug.%R.%u
Apply changes by restarting apache in the container: apache2ctl -k graceful
Notes:
host.docker.internal resolves on Linux Docker only if the container was started with --add-host=host.docker.internal:host-gateway (nextcloud-docker-dev already does this).
discover_client_host = true makes xdebug follow X-Forwarded-For - useful behind Nextcloud's dev reverse proxy.
Test xdebug with a PHP command inside the container
Run XDEBUG_SESSION=PHPSTORM php occ status inside the container and check /var/log/xdebug.log.
Install the browser extension
Install Xdebug Helper (Firefox/Chrome). In its preferences, set the IDE Key to PhpStorm. It will set the XDEBUG_SESSION cookie when toggled to Debug.
Click the Xdebug Helper icon in the browser and set it to Debug.
Test Xdebug with browser extension
Load the URL that exercises the code path with the breakpoint. Zed should stop the code exection at the breakpoint.
Welcome to the June 2026 report from the Reproducible Builds project!
In these reports, we outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not.
If you are interested in contributing to the project, please visit our Contribute page on our website.
In this month s report, we cover:
Only installing reproducible packages with repro-threshold
A very interesting demonstration is now available showing how you might configure your Debian system to only install packages that have been reproduced by m/n rebuilders.
This is implemented via a reproduced+https://APT transport ( a mechanism for communicating between the APT client and its repository source commonly HTTP):
Every package download is intercepted by repro-threshold, which queries two independent rebuilders for a signed attestation before allowing installation to proceed. [It] is important to note that [an] install will only succeed if all package dependencies are also reproducible.
The demo gives examples of how to quickly experiment with this using a Docker container.
Various OpenJDK packages were also uploaded to Debian, including the fix for JDK-8385738 ( Javadoc does not produce reproducible output ) (for example). [][][]
The reason pages on reproduce.debian.net, such as the one for ppc64el, now feature links labeled with the bug emoji (i.e. ) which links to the categorized issues packages have been tagged with in the reproducible-notes.git repo.
Indeed, 25 reviews of Debian packages were added, 31 were updated and 33 were removed this month adding to our extensive knowledge about identified issues. Two issue types were updated as well. [][]
The IzzyOnDroid Android APK repository reached its next milestone this month, now covering 2 out of every 3 apps (66.7%) with reproducible builds. Their documentation for debugging and fixing failed builds has steadily grown as well. More clients have picked up showing reproducibility results (e.g. Droid-ify), and Neo Store now can be configured to stick to only reproducible applications. Further, an independent builder has been added to the build farm, increasing the trust level even more as APK builds can have multiple confirmations now.
At the same time, IzzyOnDroid s rbtlog got several new features. The most outstanding is caching for frequently used resources such as reproducible-apk-tools, command-line tools and NodeJS in order to counter ongoing issues with GitHub availability, while at the same time saving bandwidth and build time. This change also enables some other some smaller enhancements such as being able to configure build timeouts per recipe for those builds running longer than the average, release pattern filtering for update checks or having a field for maintainer notes to shortly summing up e.g. why a reproducible build failed.
Lastly, Bernhard M. Wiedemann posted another openSUSEmonthly update for their reproducibility work there.
diffoscope development
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 319, 320, 321, 322 and 323 to Debian:
Debian adds an extra Flags: line in the output of ocamlobjinfo, so adjust the test for cross-distribution compatibility. []
Bump debhelper compatibility level to 14. []
Fix compatibility with Ocaml 5.4.1. []
Use --long-form-style arguments when calling apktool in order to support apktool version 3. []
Support Androguard version 4 and previous versions at the same time. []
Update copyright years. []
In addition, Jochen Sprickerhof added better header detection for the Sphinx documentation system [], Michael Daniels fixed the tests when run with zipdetails version 4.006 [] and Zbigniew J drzejewski-Szmek added a version of the deprecated os.path.commonprefix method [].
In addition, Vagrant Cascadian updated diffoscope in GNU Guix to version 321 and 323.
Chris Lamb also made the following changes to strip-nondeterminism, our tool to remove specific non-deterministic results from a completed build:
Skip symlinks when manually called via /usr/bin/strip-nondeterminism. (#1139000)
Update debian/watch format. []
Drop Rules-Requires-Root: no and Priority: optional fields. []
kpcyrd posted to our mailing list regarding the waves of malware uploads to aur.archlinux.org . Curiously, every incident I looked at used npmjs.com for malware delivery , specifically where the npm package includes an (automatically executed) preinstall script that is an ELF binary.
kpcyrd also announced the release of debian-repro-status version 0.4.0, a tool written to give you an approximate idea of how viable it would be to enforce a reproducible packages only update policy for the computer system you ve built :
The change updates dependencies to the latest versions, and adds support for multiple -H options, to query results from multiple rebuilderd instances. The results are also now fetched concurrently.
kpcyrd also reported that, whilst taking a screenshot for the above release, they noticed that the debian:sidcontainer now is 100% reproducible.
Documentation updates
Yet again, there were a number of improvements made to our website this month including:
Chris Lamb added a reminder re. using the UTC variants of the Javascript Date methods. []
Mattia Rizzolo moved OTF to the old sponsors list. Thank you for your support!. []
kpcyrd updated the Rust documentation to recommend using the --release argument for consistency. []
Patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:
Four new scholarly papers
Kenichiro Muto and Kuniyasu Suzaki of the Institute of Information Security in Yokohama, Japan published an interesting paper this month titled Attestable Build Chain: Enabling Trust in Reproducible Builds (PDF). Their abstract is as follows:
Ensuring trust in software supply chains requires verifying not only artifacts but also the processes that produce them. Although Reproducible Builds (R-B) require rebuilding to validate artifacts, they cannot verify whether the build was executed with the intended toolchain and inputs and may reproduce unintended or compromised builds without detection. We present Attestable Build Chain, a framework for externally verifying build-time execution without rebuilding. Rather than preventing compromise, it provides verifiable, tamper-evident evidence of actual build-time execution, enabling verification of build process integrity from observed file accesses during the build. []
Julien Malka, Stefano Zacchiroli and Th o Zimmermann published a 50-page report detailing A Decade of Software Reproducibility in the Nix Package Ecosystem:
We find that functional package management enables extremely high rebuildability over time (near-universal ability to reconstitute historical build environments and rebuild software packages), while bitwise reproducibility has steadily improved and reaches a high point in recent years (up to 93% in 2024). Early years show substantially lower bitwise reproducibility, indicating that functional package management alone does not guarantee bitwise-identical outputs, and that the observed high level of bitwise reproducibility is not solely due to the package management approach. Common causes of unreproducibility, both in the rebuildability and bitwise reproducibility dimensions, include management of dates in build and test processes; we quantify their prevalence and other common causes using manual analysis of logs of rebuild failures and automated analysis of diffoscope.
This paper presents how the DevGuard project rebuilt its OCI container pipeline around reproducible Nix builds and independent dual-platform digest verification. DevGuard images are built hermetically from pinned source revisions, signed with Sigstore/Cosign, and verified through digest comparison across GitHub Actions and sovereign GitLab infrastructure hosted on container.gov.de. We describe the practical integration of reproducible OCI image builds into existing CI/CD workflows and argue that independently reproducible container digests provide a stronger integrity guarantee against build tampering than provenance alone. The paper further discusses remaining trust assumptions and the relevance of sovereign build infrastructure for government and regulated environments.
Software bills of materials (SBOMs) support supply chain transparency, but they do not prove that a delivered SBOM reproducibly corresponds to its software artifact. Existing signing and provenance mechanisms protect integrity and traceability, yet lack consumer-side reproducible verification. We propose an SBOM integrity verification framework combining procedure disclosure, consumer-side reproduction, authority-generated reference evidence, and digest comparison. A trusted authority records a reference digest, and consumers compare it with locally reproduced and delivered SBOM digests. Experiments on 100 real-world container images show detection of artifact tampering, SBOM substitution, distribution modification, and adaptive tampering beyond signature-based approaches
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
Team Rcpp is excited to share that an brandnew new version 1.1.2 of
Rcpp is now on CRAN, has also
been uploaded to Debian, and has
already built for r2u
and r-universe;
Windows etc builds at CRAN
should follow in due course.
Rcpp has long established itself
as the most popular way of enhancing R with C or C++ code. Right now,
3236 packages on CRAN depend on
Rcpp for making analytical code go
faster and further. On CRAN, 13.4% of all packages depend (directly) on
Rcpp, and 61.4% of all compiled
packages do. From the cloud mirror of CRAN (which is but a subset of all
CRAN downloads), Rcpp has been
downloaded 121.6 million times. The two published papers (also included
in the package as preprint vignettes) have, respectively, 2263 (JSS, 2011) and 471 (TAS, 2018)
citations, while the the book (Springer useR!,
2013) has another 742.
The is the second update in the 1.1.* series which had, among other
changes, switched to C++11 as
the minimum standard. This release continues as usual with the
six-months January-July cycle started with release
1.0.5 in July 2020. Interim snapshots are always available via the
r-universe page and
repo. We continue to strongly encourage the use of these development
released and their testing we tend to run our systems with them too.
Having said that, we would like to reiterate that we strongly object
to the upstream R release and change management which in this 4.6.*
cycle made several abrupt changes forcing packages which
consume header files to make very abrupt change. Rcpp, just like numerous other CRAN packages demonstrates that
API changes can be undertaken responsibly in a managed manner which
allows for transition periods followed by possible warning periods,
deprecations periods and finally (but only at long last) errors. What
happened here is a speed run to the final stage of forced errors. Uncool
and irritating for something as widely used as R. This forced us to make
an interim release 1.1.1-1.1 even though we have of course had a policy
of always keeping properly tested, installable, and error-free
releases candidate version in the main repository branch and hence
available via R-universe tested
packages for all relevant platforms, and even via binaries for most
(including Ubuntu LTS). It would be nice if R Core found a way to take
advantage of this. Maybe development cycles, running apart for a year as
they do for R, should also include selected packages.
Once again I am not attempting to summarize the different changes.
The full list follows below and details all these changes, their
respective PRs and, if applicable, issue tickets. Big thanks from all of
us to all contributors!
Changes in
Rcpp release version 1.1.2 (2026-07-01)
Changes in Rcpp API:
Use of execinfo.h is again conditional to avoid
build complexity (Dirk in #1445 addressing
#1442)
An internal state component for Datetime is now
int (Dirk in #1448 and #1449 fixing #1447)
Three new (in R 4.6.0) attribute accessors are used conditionally
(Dirk in #1450
closing #1432)
An UBSAN error in the Sugar-based NA comparison has been
corrected (I aki in #1453 fixing #1452)
Treatment of Inf outside of integer range in Sugar function has
been corrected (I aki in #1458 fixing #1455)
Integer overflow protection has been added for sugar functions
(I aki in #1457
fixing #1454)
The parent environment is now accessed via
R_ParentEnv (Dirk in #1460 fixing #1459)
Change to returning dataptr again for better
handling of empty vectors (I aki in #1462 fixing #1461)
Undefined behavior errors in use of ListOf proxies
have been addressed (I aki in #1464 fixing #1463)
Under newer R version, R_UnboundValue is no longer
used (I aki in #1466 fixing #1465)
New R API access point R_getRegisteredNamespace() is
used with current R versions (Dirk in #1469 fixing #1468)
The Nullable::as() exporter now uses an explicit
cast to the templated type (Dirk in #1471 fixing #1470)
A memory leak in the variadic Rcpp::warning()
template has been fixed (Kevin in #1475 fixing #1474)
The Nullable::operatorT() has been added as a
'opt-out' (Dirk in #1477 with
coordination in #1472)
Add templated integer-index overload for operator[]
on small systems such as WASM (Jeroen Ooms in #1482)
The attribute accessors in AttributeProxyPolicy no
longer rely on get__() (Kevin in #1484 fixing #1483)
Changes in Rcpp Documentation:
Reference in the bibliography used by the package vignettes have
been updated.
Changes in Rcpp Deployment:
Excute permissions are set consistently on scripts with shebangs
(Mattias Ellert in #1467)
R 4.5.* has been added to the CI matrix (Dirk in #1476)
Three nag messages issued when obsolete build flag accessors are
used now show Rcpp::: (Dirk in #1480 fixing #1456)
Reference GitHub Actions have been updated to their current
versions (Dirk in #1481)
Non-release Changes:
A non-release hotfix 1.1.1-1 used by CRAN accommodates breaking
changes to the API in R 4.6.0. It would be nice to have the same level
of release management in R itself that CRAN expects from us.
Thanks to my CRANberries, you
can also look at a diff
to the previous interim release along with pre-releases 1.1.1-1
and 1.1.1-1.1
that were needed because R-devel once again sudden decided to move fast
and break things. Not our doing. And there also should not have been a
need to two such uploads but it was amateur hour all around.
Questions, comments etc should go to the GitHub discussion or issue section, or the
Rcpp
list. Bugs reports are welcome at the GitHub issue tracker as
well. GitHub offers decent search for issue, pull requests and
discussions; as many topics have been covered it is worth checking as
well.
Uploaded labwc 0.20.0-1 and 0.20.1-1 to unstable; these releases come with
support for wlroots-0.20, which made labwc reenter testing
Uploaded swaylock 1.8.5-2 to unstable to make it use the common-auth
directive of pam (seeh #1140096)
Uploaded swayimg 5.4-1 to unstable
Uploaded wayback 0.3-2 to unstable, which was waiting in experimental
for a reupload and I had forgotten about it; also fixed a
typo
in wayback upstream
Uploaded xdg-desktop-portal-wlr 0.8.3-1 to unstable
DH Related Work
The search app I was working on last month was still a focus in June. I refactored
the data model a bit and made it simpler. I stumbled over the Python
Koans and Koan 15: The Invisible
Ink gave me the
idea of using unicode normalization when indexing the items.
I released a couple of bug fix releases for the APIS framework, namely 0.64.2,
0.64.3 and 0.64.4. I also release 0.65.0 which is one step further in dropping
support for the legacy apis_entities app. When the search module is merged
it will give way for removing the last bits of the old cruft to be removed.
During a regular dependency update session I looked at the changes in the dal
dependency. After a
long time with no commits, the project suddenly had a lot of commits
co-authored by Claude and then released a new major version with a regression.
Given the state of the project, we decided to keep using the previous release
for now and look into replacing the dependency with an HTMX based solution. I
implemented a POC for one of the plugins we develop and it was actually pretty
easy. I also managed to combine the autocomplete approach with a multi-select
form field, based on this blog
post.
In the
PFP
project I finally merged the stats endpoint which give statistics about the
named graphs that are used as data sources.
Other
I attended BSidesVienna 0x7EA but it was on one of
the hottest days this year so far so I left after a couple of talks.
The seL4 organisation on GitHub uses
git-repo to manage
multiple source repositories, and so there are a large number of
projects to get your head around when figuring out the ecosystem.
As an experiment, I have taken the various manifest files across the
org, and constructed a graph based on how frequently each pair of
repositories is mentioned in a manifest together. See below:
[This may render badly when syndicated outside of my blog; and also
on small screens. And probably large screens. I ve attempted to make
sure there s a non-JS fallback
on my site with JS enabled, if you hover over a node, it should
highlight connected nodes.]
The colouring of the nodes is mostly manual; I experimented with graph
clustering algorithms but have not found a satisfactory result so far.
Still, some clusters are obvious:
Kernel the seL4 microkernel proper. This often but not
always co-exists with the main cluster of core libraries, but it
is pulled away slightly by the verification and microkit
manifests.
Verification the verification repositories (l4v, HOL,
graph-refine, polyml, isabelle) form a very distinct group.
These are connected only to the seL4 microkernel itself, which is
the only component formally verified.
Microkit microkit is a newer operating system framework
that does not use CAmkES, so stands apart from the rest of the
pack. I chose to scope this work to the seL4 org, so the LionsOS
ecosystem and sDDF which are maintained by Trustworthy Systems are
not shown. Also not linked is rust-sel4, because this modern
world isn t using git-repo in the main to manage its repositories.
RefOS I d not come across refos before, but it appears to
be an example OS from 2021 built on the seL4 kernel.
It s quite hard to pull apart the CAmkES framework and the core
libraries; there are definitely some which are more associated with VM
management, but the overall shape of this co-occurence data is a messy
ball in the middle with some outliers in orbit. One observation is
that camkes is correctly identified as more peripheral than
camkes-tool, which contains the actual core CAmkES code.
Reflecting on this approach, in hindsight I m surprised that using
co-occurences worked as well as it did there was no attempt to
actually inspect the code and find direct mentions of other code
e.g. library header dependencies. As the newer microkit effort
largely eschews git-repo, better results might be found by actually
taking that more detailed approach, so that graph edges could
represent real dependencies between two packages. Additionally, this
could allow diving into the various libraries held in the different
libs repos, to get a more granular graph of relationships between
them.
However, I think I spent more time on making it possible to render
graphviz graphs easily on my blog than actually gaining any insight
into the codebase!
Following up on my previous post, I have released the
debvulns CLI. This utility uses
the same parsing logic as the debsecan-mcp server but exposes the
functionality directly via the command line.
Why a new CLI?
While Debian's native debsecan utility exists, it lacks modern output formats
like JSON and CSV, and fails to expose a significant amount of metadata
available in the Debian Security Team's daily snapshot.
Additionally, running a persistent Model Context Protocol (MCP) server
introduces context window overhead. The manifests and tool descriptions required
by the protocol consume tokens even when idle. For debsecan-mcp, the MCP
Inspector utility shows an
overhead of roughly 150 tokens.
By contrast, an LLM can parse a standard CLI help menu on-demand without
permanently draining the context window. Integrating the CLI into a persistent
agent workflow can be achieved via a skill file, allowing the LLM to leverage the
tool without repeated discovery overhead.
What else is NEW?
During testing, I observed discrepancies between the output of debsecan-mcp/debvulns
and native debsecan. Debugging with an LLM revealed a bug in the version
comparison logic
that caused debvulns to underreport vulnerabilities. This has been resolved.
The current interface supports structured formatting and customizable data backends:
By allowing users to override data sources with local snapshots of the Debian
Security Tracker and EPSS feeds, debvulns can run natively in airgapped
environments.
What Next?
The next step is building a Prometheus exporter for this vulnerability data to
streamline scanning and monitoring across data center infrastructure. Stay tuned.
For almost two decades, the PackageKit package management abstraction layer has shipped with pkcon as its command-line client. pkcon does its job, but it was always kind of a testing front-end for the PackageKit daemon rather than a tool designed for everyday use. The focus has instead been on the GUI tools, automatic system updates, GUI application managers and other front-ends. Its command names mirror the D-Bus API almost one-to-one (get-details, get-updates, get-depends), output is very plain, and there is no machine-readable mode for scripting. Most importantly though, there has been no development on it at all for almost a decade, so pkcon was stuck in its rudimentary state from that era.
Since a lot of changes will be coming to PackageKit, and testing the daemon and working with it from the command-line was not very pleasant anymore in 2025/2026, I decided to modernize the tool as part of my work as fellow for the Sovereign Tech Agency last year. pkgcli is the new command-line client for PackageKit. It is built from the ground up to be pleasant to use interactively and easy to drive from scripts.
Why a new tool?
Of course, instead of introducing a new tool, I could have just expanded pkcon instead. The problem with that approach is that the pkcon utility has been around for so long and its command-line API had ossified so much, that rather than changing it and potentially breaking a lot of scripts relying on its quirks, I decided to introduce a new tool instead. pkcon can still be optionally compiled for people who need it in their scripts and workflows.
The goals for pkgcli, and the features it now has are:
Human-friendly command names. Verbs that read the way you d describe the task, instead of mirroring the D-Bus API 1:1: show, search, list-updates, what-provides, instead of get-details and friends.
Readable, colored output by default (still respecting NO_COLOR and degrading gracefully).
A real scripting mode. A global --json flag emits JSONL instead of fully human-readable output when possible, to make it easier to use the tool for scripting purposes.
Sensible defaults. A few defaults have been changed, such as the metadata cache-age, or automatic cleanup of unused dependencies being enabled by default. This is more in line with current defaults by other tools and frontends. We also print package information in a slightly different, more readable way.
Better handling of internationalized text. Text should now align properly in the terminal window, and we should no longer have completely chaotic text output on non-English locales (especially Chinese/Japanese).
Why not pkgctl?
Originally, this tool was called pkgctl, to match other common cross-distro tool names. However, that name was already taken by an Arch-specific distro development tool. When this issue was raised, we decided to just rename our tool to pkgcli with the next release, to avoid the name clash on Arch Linux.
Examples!
Here are some examples on how to use the new tool (some of which include the abridged output pkgcli prints).
Search for anything containing the string editor in name or description, then look at the details of one result:
$ pkgcli search editor
Querying [ ] 100%
ace-of-penguins1.5~rc2-7.amd64 [debian-testing-main]
acorn-fdisk3.0.6-14.amd64 [debian-testing-main]
ardour1:9.2.0+ds-1.amd64 [debian-testing-main]
audacity3.7.7+dfsg-1.amd64 [manual:debian-testing-main]
audacity-data3.7.7+dfsg-1.all [auto:debian-testing-main]
augeas-tools1.14.1-1.1.amd64 [debian-testing-main]
emacs1:30.2+1-3.all [debian-testing-main]
gedit48.1-9+b1.amd64 [debian-testing-main]
gedit-common48.1-9.all [debian-testing-main]
gedit-dev48.1-9+b1.amd64 [debian-testing-main]
[...]
$ pkgcli show nano
Package: nano
Version: 9.0-1
Summary: small, friendly text editor inspired by Pico
Description: GNU nano is an easy-to-use text editor originally designed as
a replacement for Pico, the ncurses-based editor from the non-free mailer
package Pine.
[...]
URL: https://www.nano-editor.org/
Group: publishing
Installed Size: 2.9 MB
Download Size: 646.0 KB
Search only within package names rather than descriptions:
$ pkgcli search name python3
Check for updates. refresh updates the metadata, then list-updates reports what s available:
You can also have JSON output for most commands! Attach --json to any query and pipe the result straight into jq. Each line is a self-contained JSON object:
pkgcli is built by default alongside the rest of PackageKit since PackageKit 1.3.4. If your distribution ships a recent enough PackageKit, it should already be on your PATH. You can read its man page man pkgcli for more information. Feedback, bug reports, and patches are very welcome.
My Debian contributions this month were all sponsored by Freexian.
You can also support my work directly via Liberapay or GitHub Sponsors.
OpenSSH
I backported various security fixes from 10.3 to trixie, bookworm, bullseye, buster, and stretch. For trixie, I also backported several IPQoS fixes to line up with upstream s traffic management settings and drop a rather hacky Debian-specific patch; this needed a quick follow-up fix.
I upgraded trixie-backports to 10.3.
I fixed openssh uses pidof but does not depend on procps.
PuTTY
I upgraded from 0.83 to 0.84.
Python packaging
New upstream versions:
AI slop is invading the web. A recent story about disallowing LLM-generated
submissions on Lobsters triggered a lot of debate. My personal worst
offenders are LinkedIn articles with AI-generated images and uninspired
articles filled with emojis from people trying to masquerade as experts on a
subject they don t care enough to write themselves. While I am unhappy about
this situation, I rely on LLMs for grammar, copyediting, and
translation. I don t see this as a contradiction.
I am a native French speaker, but I blog in both English and French. When I
started writing this blog in 2011, I was composing in French and translating
to English, but I found it was better to work in the reverse order to
avoid unnatural and non-idiomatic constructions. One of my goals is to write
good English but I never felt it was my strong point.1 For example, verb
tenses are often an issue, even if I mostly stick with the present tense. I
learn the rules and forget them right away. I also don t feel like hiring an
editor for something I see as a hobby.
As an example, I have kept the history of the successive iterations when writing
Scaling Akvorado BMP RIB with sharding :
the first draft, authored with the help of a thesaurus,2
I know that LLMs may alter the author s voice when editing, but the
corrections in the second step are minor. The prompt asks to apply light
stylistic edits, with some guidance around avoiding passive voice, long
sentences, bland verbs, and filler words. It also defines the target audience:
technical with a B2 level in English.
In the following excerpt, I used long time instead of long-standing. The
former is missing a hyphen and applies to people a long-time friend, while
the later relates to a situation a long-standing agreement. I had a hard
time understanding the reason of the second change: the LLM prefers a
defining relative clause to provide the definition of RIB sharding.
As the Internet routing table contains more than 1 million routes, Akvorado
needs to scale to tens of millions of routes. This has been a long
time long-standing challenge, but I expect this issue is now
fixed by using RIB sharding, a method to split that
splits the routing database into several parts to enable concurrent
updates.
In the next modification, the LLM puts device instead of equipment. This is
correct as equipment is an uncountable noun. I know that, but I still fall
into this trap.
When Akvorado does not find a route from a specific device, it falls back to a
route sent by another equipment device.
I ask the LLM to use descriptive verbs and it complies by replacing a
multi-word predicate with a lexically rich verb:
The benchmarks demonstrate it has better performance than
outperforms other packages, both packages for
lookups, insertions, and memory usage.
It also fixes grammar errors. In the next excerpt, a list of routes is a
singular expression. Moreover, stored is a state and I should not use into
as it expresses a change.
The list of routes for each prefix are is not stored
directly into in the prefix tree.
As a last example, consider the following snippet. The require verb
accepts a noun or an object followed by a to-infinitive. I can t use it with
just a to-infinitive.
An alternative would be to have one prefix tree for each peer but it would
require to configure configuring all routers to export
their routes.
As someone who didn t grow up speaking English, I struggle with these grammar
rules despite reading a lot of English material.3 French is more
complex to get started but more systematic. English is full of irregularities.
On each page, I disclose in the footer whether an AI modified the content. There
are three levels:
: no AI or almost no AI (e.g., grammar corrections)
: enhanced (e.g., copyediting)
: generated (e.g., translated from another language, even if human-edited)
Hover or tap the icon to reveal the AI s name and its role in the document.
Example of AI usage disclosure: Claude Sonnet 4.5 edited this article.
The graph below shows which tool altered each post, year by year. Recently, I
applied the grammar skill to past articles. Since 2018,
French articles have been translated with the help of DeepL first, then of
an LLM. Since 2024, English articles are copyedited.
🖼 Graph showing the AI usage over the years. Each level get its own
color.
AI usage over the years. Hover or tap a band for the details.
If you are strongly against any usage of LLMs specifically for writing, I hope
you accept my more nuanced position on the usage of these tools as a trade-off
to provide clearer and more engaging articles. Years of literature on improving
English told us it is important to choose the right word to keep the reader
engaged.
[ ] Good writing consists of mastering the fundamentals (vocabulary,
grammar, the elements of style) and then filling the third level of your
toolbox with the right instruments.
Stephen King, On Writing
Note
Unlike other recent articles, I did not use an LLM to edit this post:
an unnamed person kindly accepted to proofread it. I translated it to French
without using an LLM either.
I am quite happy with the writing tools provided by Kagi. Both the
translate tool and the dictionary are a valuable help to find
different wordings. I also lean on Kagi s research assistant when
researching a topic.
When I was ten, I played Monkey Island 2 in English without having
taken any classes. I used a dictionary to translate word by word and I found
the irregular verbs confusing and not in the dictionary.
This was a particularly busy month for me in terms of Debian
contributions.
It started with a week in Hamburg for the MiniDebConf. I talked to
many colleagues face-to-face and worked on various bugs and
maintenance tasks. I m pleased to have finally found the time to
reproduce and fix the boot-time crashes in the parallel port
subsystem that have been reported
many times recently.
A series of easily exploited kernel LPE (local privilege execution)
issues were published this month, mostly with very little coordination
with distributions. Salvatore and I had to upload fixes for these at
roughly weekly intervals. All of these fixes needed to be applied to
4 different upstream branches (currently 5.10, 6.1, 6.12, and 7.0) and
7 Debian branches (including backports).
This was a particularly busy month for me in terms of Debian
contributions.
It started with a week in Hamburg for the MiniDebConf. I talked to
many colleagues face-to-face and worked on various bugs and
maintenance tasks. I m pleased to have finally found the time to
reproduce and fix the boot-time crashes in the parallel port
subsystem that have been reported
many times recently.
A series of easily exploited kernel LPE (local privilege execution)
issues were published this month, mostly with very little coordination
with distributions. Salvatore and I had to upload fixes for these at
roughly weekly intervals. All of these fixes needed to be applied to
4 different upstream branches (currently 5.10, 6.1, 6.12, and 7.0) and
7 Debian branches (including backports).
A while back, I came across
the AI Fabric
system created by Daniel Miessler. I liked its approach of
providing command-line tools for filtering text using artificial
idiocy services, allowing stepwise operations to be applied to a piece
of text. The output of one operation can then serve as the input for
another in other words, Unix pipeline processing powered by large
language models. I do no longer remember exactly how I discovered it,
but suspect it was via Matthew Berman's video
"How To Install
Fabric - Open-Source AI Framework That Can Automate Your Life".
While the idea and concept behind AI Fabric appealed to me, its
implementation has continued to rub me the wrong way. It started off
as a Python project that I could only get running by downloading
random programs from the internet using Poetry. I tried to assess how
much work it would take to package all its missing dependencies for
Debian. However, before I got very far, the project shifted away from
Python and over to Go. This new implementation also relied on a build
system that seemed to encourage users to run arbitrary code downloaded
from the internet to get software working, and further moved to a
language I do not master as well as Python. The change bothered me
enough that I set my effort to set up a working command line LLM tool
in Debian aside for several months.
By chance, I came across a simple Python recipe in January
demonstrating how to communicate with a
llama.cpp API
server. I had already been working on packaging llama.cpp for Debian
together with the rest of Debian's AI team, and was fortunate enough
to own a working instance with a 24 GiB VRAM GPU from AMD, allowing me
to run useful models. Until that point, I had only used the basic web
client provided by the Debian package, lacking the spare time to
explore what else could be done. Then, I found this simple 50 line
Python script demonstrating how to interact with llama.cpp's
OpenAI-compatible API. I decided to revive the AI Fabric concept, and
implement the Unix pipeline filter tool with as few dependencies as
possible. It is now operational and working very well, relying solely
on standard Python features. The tool include a copy of the LLM
recipes from the AI Fabric project (called "patterns"), enabling easy
access to request summaries, translations, code review and other
useful tasks. Several hundred patterns are included, though I have
only tested about ten so far.
The LLM API server can be specified in
~/.config/hraesvelgr/config.ini like this:
With this configuration in place (you can also specify these values
directly on the command line), you can specify a pattern and a file to
process like this:
% bin/hraesvelgr --pattern explain_code bin/hraesvelgr
EXPLANATION:
This Python script is a client tool for interacting with an AI
service (likely a local LLM server) to process text using prompts
defined in the "AI Fabric" repository. It reads system and user
prompts from markdown files, sends them along with input text to a
chat completion API endpoint, and prints the generated response.
Key components:
1. It uses argparse for command-line argument parsing
2. The send_chat_completion_request function formats messages
(system, user, query) into JSON and sends them via HTTP POST to
an AI service endpoint
3. read_file function reads markdown files, replacing placeholders
like lang_code with actual values from arguments
4. In main():
- Parses command-line arguments for input file, API base URL,
pattern type, language code, and debug flag
- Ensures the base URL ends with a slash
- Reads system prompt from data/patterns/ pattern /system.md
- Optionally reads user prompt from data/patterns/ pattern /user.md
- Reads input text either from stdin (when "-" is passed) or a file
- Handles encoding fallback to ISO-8859-1 if UTF-8 fails
- Sends the formatted request to the AI service and prints the response
The script assumes it's running in a directory containing a git
clone of https://github.com/danielmiessler/fabric/, which contains
the necessary prompt files.
This tool is designed to interface with local LLM servers that
support OpenAI-compatible chat completion APIs.
%
The list of available patterns can be viewed by running
bin/hraesvelgr --list-patterns. I have found the
summarize, translate,
improve_writing, review_code, and
explain_terms_and_conditions patterns particularly
useful. For example using the latter combined with a text based web
browser capable of dumping a page as plain text, can be done like this
(originally formatted in markdown, I converted to HTML using pandoc
for easier readability):
% w3m -dump https://runbox.com/about/terms-service/ \
hraesvelgr --pattern explain_terms_and_conditions
Executive Summary
This is a transparent, privacy-focused contract from
a Norwegian provider that generally respects user data rights and
operates under strict EU/EEA standards. However, it carries
strict liability limitations and an aggressive
data-deletion policy upon cancellation. The vibe is Professional &
Privacy-First, but you must manage your own backups and understand that
the company heavily shields itself from financial responsibility during
technical failures.
Key Takeaways
Your Data Stays Yours: Section 10.2 explicitly
states Runbox will never use your transmitted or stored data for
commercial purposes. This is a major privacy win.
[... trimmed output, as it is not the focus of this blog post ...]
If you sign:
Set up automated backups
immediately. Use IMAP sync to a local drive or a secondary
email provider before storing any critical documents or emails.
Do not rely on Runbox as your only archive.
Mark your
calendar for the 30-day trial end date. Miss the payment
window, and access closes instantly with no recovery period.
Monitor price changes at renewal. Since they can adjust
fees anytime, check their pricing page a few days before your
subscription renews to avoid unexpected charges.
NO FORCED ARBITRATION CLAUSE FOUND. REFUND POLICY IS STRICTLY CONDITIONAL (see Sections 4.2 4.5).
As you might have already noticed, I name my project
after the Norse God of Wind. I found a nice description of the
origin of the name on
Wikipedia:
In Vaf r nism l (The Lay of Vaf r nir), Odin questions the wise
j tunn Vaf r nir about the origin of the wind, and the j tunn
answers:
He is called Hr svelg,
who sits at heaven s end,
a giant, in the shape of an eagle;
from his wings
they say the wind comes over all people.
(translated by John Lindow in Norse Mythology: A Guide to Gods,
Heroes, Rituals, and Beliefs 2002)
The latest version of the code can be found at
https://codeberg.org/pere/hraesvelgr/. Perhaps you will find it as useful as I did?
As usual, if you use Bitcoin and wish to show your support of my
activities, please send Bitcoin donations to my address
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
I've heard "containers are not a security boundary" enough times that it's started to feel like received wisdom, and my honest read (after 13+ years) is that it's technically defensible but practically sloppy and the sloppiness matters.
The part that's true: containers share a kernel, and a kernel exploit crosses the container boundary where a VM would not. That difference is real and non-trivial, and the CVE history backs it up CVE-2019-5736, CVE-2022-0492, and CVE-2024-21626 all happened in "correctly configured" production containers.
The part I'd push back on is that the comparison point is almost never stated. "Containers aren't a security boundary" is being used as shorthand for "containers aren't a VM boundary" but the conclusion people seem to draw from that is "therefore don't bother", which doesn't actually follow. The more honest version is that default Docker doesn't provide strong isolation between mutually untrusting parties, but a hardened configuration does.
What ships by default in Moby is actually a pretty reasonable foundation: seccomp is enabled (with a builtin profile blocking ~50 syscalls credit where it's due: this is mostly @jessfraz's work; she even ran contained.af as a public CTF for years daring people to escape a container under her seccomp profile, and to my knowledge it was never claimed), AppArmor is enabled (the docker-default profile), and several sensitive /proc paths are masked. What's not on by default: no-new-privileges (setuid binaries inside can escalate), CAP_NET_RAW is still granted to every container (even though the kernel has supported unprivileged ICMP sockets for over a decade, meaning most modern distributions no longer need CAP_NET_RAW for ping), and user namespace remapping though user namespaces aren't quite the silver bullet they might sound like; Debian left them disabled by default for years because the kernel attack surface they exposed hadn't been hardened against unprivileged callers.
The boundary isn't absent it doesn't come completely pre-assembled. With VMs, the hypervisor is there whether you asked for it or not; with containers, assembling the boundary is left as an exercise for the operator. That's a much more solvable problem than "the technology is incapable", but it does mean the work falls to whoever's running the containers.
So, some things you can do today without waiting for defaults to change:
--user (or USER in your Dockerfile) is worth calling out specifically, because I think it's arguably stronger than user namespace remapping in one important way and partly for the same reason Debian was hesitant about user namespaces in the first place. User namespace remapping protects the host from a root-in-container escape: if you do escape, you land as an unprivileged user on the host. But you were still root inside the container the whole time. Running as a non-root user means you were never root anywhere. The blast radius of a compromised process is limited whether or not it escapes, including for things like reading secrets, modifying container contents, or lateral movement within the container itself. Most application containers have no legitimate reason to be root.
Beyond that, a short list of things that are easy to enable and hard to justify leaving off:
--security-opt no-new-privileges prevents setuid binaries from escalating; can also be set daemon-wide in daemon.json with "no-new-privileges": true
--read-only a read-only root filesystem means a compromised process can't easily persist tooling or modify the container (pair with a writable tmpfs mount for /tmp etc as needed)
--cap-drop NET_RAW or --cap-drop ALL and add back only what you actually need; CAP_NET_RAW is almost never legitimately needed by application containers
never --privileged if something seems to require it, the right answer is almost always a more targeted capability grant or bind mount, not the nuclear option
docker run \--user 1234:5678 \--security-opt no-new-privileges \--read-only\--tmpfs /tmp \--cap-drop ALL \
acme/untrusted-workload:latest
None of these require a daemon restart or infrastructure changes, and stacked together they go a long way toward actually building the boundary that the defaults leave unbuilt.
(this post was written with the assistance of "claude my eyes right out" but all thoughts and understanding are Tianon's)
Undeclared file conflicts, by Helmut Grohne
The duplication checker, the
Multi-Arch hinter, and the
/usr-move analyzer share significant
parts of their code. While the /usr-move transition is
complete, the other tools needed a
bit of love. Helmut added Python type annotations, slightly improved the
performance of the duplication website and shared more code between these tools.
Building upon this Helmut looked into file conflicts of various kinds such as
unrelated packages installing overlapping files, file type conflicts,
mismatching directory metadata and shared files of Multi-Arch: same packages
with varying content. Implementing reliable detection proved to be difficult due
to the amount of corner cases. So Helmut semi-manually
filedbugs.
In that process, it became apparent that binNMUs
do not reproduce SOURCE_DATE_EPOCH across architectures and therefore some
shared files embedding the build date would vary in content. Additionally, a
significant number of reports required further correspondence.
contributors.debian.org mini-sprint, by Enrico Zini
Enrico Zini met with Mattia Rizzolo to continue the work started at DebConf 25
on crediting contributions done via salsa, and to catch up with accumulated site
issues.
Building on the same kind of infrastructure used to notify tag2upload,
salsa.debian.org triggers a webping on pushes and merge
request activity, which causes a small JSON payload to be queued in a private
directory on contributors.debian.org.
We worked on processing, filtering and aggregating the files in the queue into a
private, staging database table. When configuring a
data source on the site, it is now
possible to configure automated submission of contributions from information in
the staging table. This makes it significantly simpler to credit contributors
for all teams that use Salsa as their code repository and coordination tool, as
the site can take care of the data mining for you.
See more details in the sprint report
posted to debian-devel-announce.
MiniDebConf Campinas, by Lucas Kanashiro, Santiago Ruano Rinc n and Antonio Terceiro
MiniDebConf Campinas was held between April 23rd and 25th, at the State
University of Campinas, and was preceded by a MiniDebcamp between April 20th
and 22nd. Freexian was Gold sponsor for the event, and Freexian collaborators
were active contributors to the conference success.
Lucas and Santiago delivered a talk about Debian LTS
during MiniDebConf Campinas 2026, where they described how the LTS project
benefits Debian users and developers, while strengthening Debian itself.
Lucas and Antonio delivered a talk about internship programs
in Debian during MiniDebConf Campinas 2026, with the goal of getting students
interested in working in and with Debian.
Lucas took part in the MiniDebConf Campinas content team, reviewing/accepting
talks and building the schedule.
Antonio led a session where he invited the audience to weigh in on current
controversies in Debian. The session presented playful elements as colored signs
to denote agree/disagree, and was not recorded, to help people feel more
comfortable about speaking up. He might be convinced to lead a similar session
at the next DebConf.
Antonio also organized a debate to discuss the consequences of new Brazilian
regulation for the protection of children and adolescents in digital spaces for
Debian and other free operating systems, but also for the free software
community in general. This session was very fruitful and will lead into further
actions, as one of the main outcomes was the realization that the free software
community must follow the discussion leading up to similar regulations more
closely to avoid being caught by surprise when they come into effect.
security-tracker performance, by Helmut Grohne and Emilio Pozuelo Monfort
Prompted by spontaneous influx of web requests on Freexian s security-tracker
back in February, we considered the options for managing that demand. One of our
mitigations was making it faster. To that end, Helmut sent
twoMRs
towards improving the situation. There are four notable improvements. The use of
Python s str.translate
generally speeds up rendering of larger templates.
Indexing
the CVE names avoids a costly sequential table scan.
Avoiding FFI calls while sorting
and reducing the queryset
speeds up the source package view. Emilio reviewed and deployed the changes on
to the Debian instance. Together these changes provide a twofold speedup on both
Freexian s and
Debian s instance on average.
dput-ng data loss bug, by Colin Watson
Ian Jackson (not affiliated with Freexian) reported that dput-ng could
lose data when using the local install
method, which could cause misleading results in tests of other packages; they
also filed an initial merge request
to fix it. Colin improved this to isolate its tests properly, and uploaded it.
Miscellaneous contributions
Lucas coordinated the src:valkey update to version 9 in unstable with a
potential co-maintainer.
Lucas provided a security update for src:valkey targeting trixie .
Thorsten did two uploads of foo2zjs, one to fix a bug and one to improve
packaging. As there have been several CVEs published for cups he also did an
upload of a new upstream version. Unfortunately this introduces a regression
and another upload was needed to take care of a crash. The patch for one CVE
also broke a test script, which is used by lots of printing packages in Debian.
As a result some autopkgtest runs failed. This could be fixed as well and the
only remaining issue that needs some more investigation is related to
cups-pdf. It is also worth mentioning that some issues related to the apparmor
configuration of cups could be resolved.
Helmut sent patches for 11 cross build failures.
Helmut sent a MR for enabling the new mainline YT6801 ethernet Linux driver
and it is now working fine with Debian s 7.x kernels.
Carles did further code developments for check-relations:
steps towards making it production ready when the initial round of reports are
analyzed. New show-package (information) command, improvements for
report_missing cases, added support for ignoring packages for specific
reasons, added unit tests, added CI. Used it to open 39 new bugs. Also followed
up different open bugs
Rapha l completed the French translation of Zulip for the
release of version 12.0.
Zulip is a nice 100% free software threaded communication platform for
distributed teams.
Stefano did routine uploads of python-pipx, python-mitogen,
platformdirs, python-authlib, python-discovery, distro-info-data,
python-virtualenv, python-certifi, python-wheel, pypy3.
Stefano uploaded distro-info-data updates to stable and oldstable proposed
updates, with the latest Ubuntu release.
Stefano took part in DebConf 26 preparation meetings.
Stefano prepared DebConf s online video streaming infrastructure for
MiniDebConf Campinas, and configured the
Debian reimbursement system to handle their travel bursary claims.
Stefano helped MiniDebConf Hamburg prepare their website for 2027.
Stefano did some sysadmin work on debian.social infrastructure.
Stefano reviewed Matthias python3.15 packaging and rebased his work on top
of it.
Antonio debugged a general slowness in salsa, caused by loss of IPv6
connectivity between the salsa host and the remote object storage in the cloud ,
which is a problem due to an open upstream bug in gitlab.
Santiago reviewed different changes to the Salsa CI pipeline, including the
new uscan test job,
prepared by Tha s Rebou as Araujo, and the final review to introduce
faketime testing, made by quila Macedo.
Santiago continued helping the DebConf 26 local team to prepare the conference.
Emilio updated libxpm to address a security issue.
Colin finished upgrading groff to 1.24.1; 1.24.0 and 1.24.1 were the first
upstream releases since 2023 and had extensive changes, so this took some time
to get right.
Colin released bookworm and trixie fixes for CVE-2026-3497
in openssh, and issued the corresponding BSA-130
for trixie-backports.
Colin upgraded openssh to 10.3p1.
Anupa worked on the accounting tasks for MiniDebConf Kanpur and prepared and
submitted a report to the fiscal host.
Debian LTS/ELTS
This was my hundred-forty-second month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.
During my allocated time I uploaded or worked on:
[DLA 4530-1] gst-plugins-bad1.0 security update to fix two CVEs related to denial of service or execution of arbitrary code if a malformed media file is opened.
[DLA 4544-1] ntfs-3g to fix one CVE related to local root privilege escalation.
[DLA 4545-1] packagekit security update to fix one CVE related to local privilege escalation.
[DLA 4547-1] gimp security update to fix three CVEs related to denial of service or execution of arbitrary code if a malformed PSP, JPEG 2000 or PSD file is opened.
[ELA-1682-1] gst-plugins-bad1.0 security update to fix two CVEs in Buster and Stretch related to denial of service or execution of arbitrary code.
[ELA-1689-1] ntfs-3g security update to fix one CVE in Buster and Stretch related to local root privilege escalation..
[ELA-1693-1] pakagekit security update to fix one CVE in Buster and Stretch related to local privilege escalation.
[libcoap3] upload to sid to fix two CVEs related to out-of-bounds read and stacked based buffer overflow.
[#1134340] trixie-pu bug for libcoap3 to fix two CVEs in Trixie.
[cups] upload to sid to fix six CVEs.
I also did a week of front desk duties and started to work on backports of the cups CVEs.
Debian Printing
This month I uploaded a new upstream versions:
Unfortunately the first upload of cups introduces a regression and another upload was needed to take care of a crash. The patch for one CVE also broke a test script, which is used by lots of printing packages in Debian. As a result some autopkgtest runs failed. This could be fixed as well and the only remaining issue that needs some more investigation is related to cups-pdf.
This work is generously funded by Freexian!
Debian Lomiri
This month I continued to work on unifying packaging on Debian and Ubuntu. This makes it easier to work on those packages independent of the used platform.
I also started working on two new packages: lomiri-radio-app and lomiri-fretboardtrainer-app
This work is generously funded by Fre(i)e Software GmbH!
Debian Astro
This month I uploaded a new upstream version or a bugfix version of: