This is a copy of an email I sent at work. I'm not sure I should be making noise about this here, feedback welcome.

What's happening? Some places in the world implement what is called Daylight saving time or DST: https://en.wikipedia.org/wiki/Daylight_saving_time Normally, you shouldn't have to do anything: computers automatically change time following local rules, assuming they are correctly configured, provided recent updates have been applied in the case of a recent change in said rules (because yes, this happens, and happened this year, and yes, you need to upgrade your software!). Of course, appliances like your microwave oven will likely not change time and will need to adjusted unless they are so-called "smart", in which case they are part of the skynet botnet and should be destroyed. If your clock is flashing "0:00" or "12:00", you have no action to take to adapt to this change, lucky you. If you haven't changed time in six months, congratulations, your clock will be accurate again! In any case, you should still consider DST because it might affect some of your meeting schedules, particularly if you set up a new meeting schedule in the last 6 months and forgot to consider this change.

If your location does not have DST Properly scheduled meetings affecting multiple time zones are set in UTC time, which does not change. So if your location does not observer time changes, your (local!) meeting time will not change. But be aware that some other folks attending your meeting might have the DST bug and their meeting times will change. Be kind to those poor souls which might be missing meetings by a full hour because time flies backwards for them.

If you do observe DST If you are affected by daylight savings, your local meeting times will change for UTC meetings. Normally, your meeting times are scheduled to take this into account and the new hours should be reasonable. But now is a good time to verify that. Take a look at your schedule for the next couple of weeks and reschedule meetings before the daylight saving come up to avoid too much disruption. You have only a couple of weeks to do so right now.

When do times change, how, and and where? As regular readers will remember, the rule of thumb is:

Spring forward, fall backwards.

That is, during the season of Spring, the clocks move forward, and during the Fall (like right now), they move backwards. That is in the northern hemisphere, but then the southern hemisphere is often saner and doesn't switch anyways. So time will move backwards which means an extra hour of sleep. Unless you have children or bad sleep, in which case your body doesn't care about what the clock says and will wake up one hour earlier than what it should. And of course, this doesn't happen everywhere at once, so let's see when it happens where.

Europe The dance starts in Europe. The change happens on the last Sunday in October at 01:00 UTC (not local time!), that is October 25th. If you are in the central European timezone, also known as Amsterdam, Berlin, or Paris time depending on your national affiliation, that essentially means that at 2:59 local the clocks will fall back to 2:00 instead of going to 3:00. Concretely, set your watch back one hour before going to bed, go to bed at the normal time, and enjoy an extra hour of sleep or leisure. If you have kids, you might want to start getting them to bed slightly earlier every day for a week before the change so they take time getting used to the change. If you have trouble sleeping in the morning, find your inner child and do that to yourself as well.

USA / Canada Then it's the US[1] and Canada[2] joining the dance, on the First Sunday in November at 02:00 local (not UTC!), that is, I believe, November 1st 2026. This means that, at 1:59, the clocks will flip to 1:00, instead of 2:00. Concretely, do like the Europeans and tweak your clock before going to bed. That is a little less than four weeks from now. [1] except Arizona (except the Navajo nation), US territories, and Hawaii [2] except Yukon, Saskatchewan, (newly) British Columbia, (newly) Alberta, (newly) Northwest Territories, (newly) Manitoba, one island in Nunavut (Southampton Island), one town in Ontario (Atikokan) and small parts of Quebec (Le Golfe-du-Saint-Laurent)

Other places with DST This time again, I must apologize to the people of Cuba, Lebanon, Israel, Palestine, Egypt, Chile, Australia, and New Zealand, as you fine folks all have your own DST rules that are omitted here for brevity. I rely on this page from Wikipedia to be updated by time nerds accurately for this message, and it should provide you with a rough idea of what's coming: https://en.wikipedia.org/wiki/Daylight_saving_time_by_country In general, changes also happen in October, but either on different times or different days, except in the south hemisphere, where they might happen in September (oops, sorry NZ folks, I'm late!).

Places without DST Everyone else, enjoy, you're on the right side of history, and we thank you for the good example you give us.

Changes since last time There's been lots of changes since last time:

• British Columbia moved to permanent -07 on 2026-03-09, that is it will not change to normal time in November
• Alberta moved to permanent -06 on 2026-06-18, similar to BC above.
• Canada s Northwest Territories moved to permanent -06 on 2026-08-21, matching Alberta.
• Manitoba moves to permanent -05 on 2026-10-31.
• Morocco moves to permanent +00 on 2026-09-20.
• Moldova has used EU transition times since 2022, but the tz database only noticed in 2026

This is my interpretation of the changes announced on the tzdata mailing list here: https://lists.iana.org/hyperkitty/list/tz-announce@iana.org/latest If the eastward trend continues, Canada should adopt country-wide "no daylight savings" rules by 2027, although there's actually no sign of the other provinces (Ontario, Qu bec and so on) currently running bills to change those rules just yet. Poor Canadians like me confused about time in their countries can refer to this section of Wikipedia for details: https://en.wikipedia.org/wiki/Daylight_saving_time_in_Canada#By_province_and_territory ... and particularly the image featured there: https://commons.wikimedia.org/wiki/File:Canada_time_zone_map-en.svg It also seems like the US government might finally adopt a permanent daylight saving change bill in 2026, as the "Sunshine protection act" pass the house in July: https://en.wikipedia.org/wiki/Sunshine_Protection_Act True to form, this was associated with absolutely ridiculous pressure from Donald Trump against republicans (his own party!) objecting to the change:

On July 14, 2026, the House passed a Sunshine Protection Act bill backed by President Trump. Nevertheless, the bill was opposed in the Senate by Republicans, including Senator Cotton. In response, on October 3, 2026, Trump shared a post on Truth Social urging Cotton to approve the bill, where he revealed Cotton's personal cellphone number and called on people to call him.

https://www.theguardian.com/us-news/2026/oct/03/trump-tom-cotton-daylight-saving-time Given that the last time the US did a major change to the daylight savings policy (in 2005), Canada followed suit to stay in sync, it's quite possible Trump's mad dash might actually finish getting rid of DST in North America: https://en.wikipedia.org/wiki/Energy_Policy_Act_of_2005#Change_to_daylight_saving_time

Also in this series:

• The Four Horsemen of the LLM Apocalypse
• A more nuanced view of LLMs

How people use models People using LLMs in their workflow have radically changed how programming works, even for people who claim to avoid vibe-coding. And I'm sorry to single out one poor maintainer here: it's not you, Brian, you're just one example among many. But this is typical use of those models nowadays:

Once it s done, I ll use /code-review and let Claude spawn sub-agents to do a full review of the new code. This usually finds some problems, even problems that the main Claude instance didn t find during its validation. I usually keep running /code-review again and again after finding and fixing issues, until there aren t any left.

Think about what that means for a minute. This is automation built to fire up dozens of agents crunching at a problem for minutes if not hours of GPU compute time, in parallel. This is essentially a couple of shelves in a datacenter rack, totally maxed out on power and cooling, abstracted behind a cute little /code-review command. The author, here, is rightly concerned that "Anthropic could pull the rug out and require API pricing", which is perhaps a code word for "charging something closer to actual costs". Brian also pays lip service to environmental and societal costs but those are largely abstracted away, so let's keep that conversation aside here as well, as we have discussed it before anyways. But clearly, this way of working has an (externalized) cost, to say the least.

Paying for non-free tools For decades my work has been focused on free and open source software. I've long stopped using proprietary operating systems like Windows or Mac, and even before that switch, I was mostly using free software on those platforms, partly out of principle, but also because I was too poor. So the tools of my trade are free, and I build free tools with them. It feels like we're going backwards: when I was in school, a millennia ago, my classmates didn't have access to a compiler and were wondering how they would scrape the money to buy a compiler like Borland's or Microsoft's. I had a compiler built into my operating system (FreeBSD at the time), so that wasn't a problem for me. For them, it was a significant expense, but at least those expenses (or more shady sourcing of programs) were a one-shot deal. Fast forward 30 years, and software is rented: you pay monthly for Adobe's Photoshop and Microsoft's office suite just like you pay for Netflix, Disney+ or Spotify². And now you need to add dozens (if not hundreds of dollars) of monthly credits to access LLMs on top of that. So, now we have to pay to get anything done? This is peak enshitification of our job: first they steal our work to train their models, and then they sell it back to us at a profit.

Attacking the engineers AI is coming for our jobs, as engineers, if not everyone, according to the narrative. For a while now, our job market has deteriorated: less jobs, for less pay. Lots of skilled engineers looking for work and finding crap jobs then still looking while working. This is not by accident.³ We engineers have a lot of power, it is not organized, but that's just a couple of unions away (easy!). Tech overlords know this, so they are attacking our profession, directly, by forcing us to train and use models that they can control. Even in environments where programmers are not forced to use LLMs, the mere pressure of other people's LLM-generated work is huge. One can be forced to review LLM outputs, or just peer pressured you into producing more. We're now supposed to accelerate delivery, because models can presumably do things so much better and faster. With supply chain security becoming such a large vector that we now have worms crawling around developers accounts on NPM, increasing the delivery cadence seems like a really bad idea.⁴ The LLM hype is part of the larger wave of cyberwar against workers, against water, against the Earth, against all the people. This is not a matter of individually "adapting to the reality" or personal choice, but a political, social, hard problem we need to address collectively.

---

1. I again prefer the term LLM to "AI" because models do not possess intelligence. I did use it in the title because click baiting is apparently important, but I stopped short of calling this one "Rage Against the Machines" because that would be the title of every blog post I have ever made.
2. Yes, I know that Visual Studio is kind of free now, but I wouldn't be surprised if they turn that into a rental as well, because why not.
3. Beyond sabotaging the job market, Sam Altman event wants to sell "intelligence as a utility" something that is just a really bad idea but especially shows how megalomaniac those people are.
4. This brings back memories of another era, walking us back decades in terms of computer security.

• hardware
• scouting
• organization
• preparation
• execution
• processing
• publishing

• good quality vector graphics of the current Debconf logo
• good quality vector graphics of the next years Debconf logo (even if preliminary)
• motto of the conference

0.6.5 - 27th June 2026

• Lots of fixes/improvements around length checking
• Added extra basic tests for maxpaths/sixdegrees/CGI
• Correctly end transactions in the stacked backend
• Ensure the file backend avoids stale key data on updates
• Fix decoding of v2/3 signature creation times
• Fix EdDSA signature parsing when r < 249 bits long
• Fix migration of bools from old to new config style
• Fix parsing of new config details for DB parameters
• Fix problems with linking + dynamic backends
• Fix RSA-SHA2-384 signature checking
• Fix sixdegrees parsing of keyids with high bit set
• Handle failures in maxpath more gracefully
• Make new style config path match old path

• the RISC-V CPU architecture is open source,
• the price point is within reach of home and small business users and
• the overall feature set makes it an ideal platform to build local and offline AI systems.

Comparing the resellers SpacemiT does not sell anything directly to consumers. Instead you need to buy a board that includes the K3 chip from an integrator. Currently the main resellers are:

• Milk-V
• Sipeed
• Banana Pi
• Firefly
• DeepComputing

All of the above are Chinese companies that ship to customers both inside and outside China. DeepComputing stands out as the only one that actually has done real integration and ships the K3 on a custom board, while the others simply resell the SpacemiT-produced K3 pico-ITX and K3 CoM260 Kit.

Milk-V Milk-V is a RISC-V specialized integrator, as the name already implies. They sell the K3 under the name Jupiter2. Of all the K3 pico-ITX reseller product pages, the Jupiter2 presentation is the nicest and most detailed. Unfortunately their order page at arace.tech only states that it is a pre-order with no information about shipping schedule, taxes, or other details like what SSD is included (if any). Based on the pictures it does ship with a Milk-V branded case. The 32 GB RAM lists at 504 EUR, which is a very reasonable price. The @MilkV_Official account on X recently promoted the K3.

Documentation and support As of this writing, the Milk-V Jupiter2 documentation site is just a stub and has no actual content, and only two links to the SpacemiT K3 documentation site. For support there is a web forum with a dedicated Jupiter2 section. There is also a Matrix space, but unlike their other products, there is no dedicated Jupiter (neither v1 nor v2) channel.

Community size and open source involvement At least one prior Milk-V product was certified by Canonical, which indicates there is some collaboration in progress. Canonical also lists the Milk-V Titan on its official Ubuntu for RISC-V partner-built hardware page.

Sipeed The Sipeed K3 announcement is well written (in English) with all the relevant details and links to additional PDF manuals. However, their main page at sipeed.com says nothing about the K3, so one must know the subpage URL to access it. They offer both the K3 CoM260 kit compatible with Jetson Orin Nano carrier boards, and the stand-alone K3 pico-ITX-sized motherboard. The CoM260 kit is only 10 USD cheaper than the full pico-ITX motherboard, so choosing the latter is a no-brainer if starting from scratch. The pico-ITX model with 32 GB DDR5 RAM sells for 639 USD. The product page does not mention anything about hard disk size, so you don t really know exactly what you will be getting if placing an order. There is no indication about case, Wi-Fi antennas or power supply either, so most likely they are not included. Their store.sipeed.com website does not work at all, and their Taobao and AliExpress stores are not public and only accessible to registered users. The order page also says nothing about shipping time, delivery time, or taxes. The X account @SipeedIO is active and recently posted pictures of shipments in progress.

Documentation and support The main documentation wiki does not yet have any K3 content at the time of writing. There is a Discord channel for general RISC-V discussion, and their MaixHub also has a discussion board, but I didn t find anything K3-specific.

Community size and open source involvement Sipeed has had at least one of their previous devices certified by Canonical, which indicates they are active in the community. Note that the other RISC-V company SiFive that also has had hardware certified and officially supported by Canonical is a different company, despite the very similar name.

Banana Pi Banana Pi announced that they offer both the K3 CoM260 kit and the K3 pico-ITX motherboard version. Their product page for the K3 confusingly shows a MediaTek product in the page banner rather than the SpacemiT K3. Based on the product description and the fact they renamed the product as BPI-SM10, it seems to ship with some carrier board. The product pictures look identical to the SpacemiT documentation and there is no picture of the carrier board, and details are very sparse. The pico-ITX version with 8 GB RAM and 128 GB SSD sells for 293 USD and the CoM260 developer kit with the same specs sells for 287 USD and the 32 GB RAM with 128 GB SSD model sells for 595 USD. The shop page shows only five orders so far and items are currently out of stock. As there was no 32 GB RAM version of the pico-ITX available at all, this isn t an option for me as I want to run 30B parameter models that need the larger memory version. Of all of these resellers, the Banana Pi website seems the most outdated. It does not have a search feature, it is not mobile-friendly, pictures can t be pinched to zoom in and so forth. Product names are also almost all identical, and as the product listings only show the beginning of the product name, figuring out what product is what requires extra effort that just makes the online purchase experience plain bad.

Documentation and support I was only able to find the documentation page for the CoM260 kit, but none for the pico-ITX version. For support there is a forum, but the category list does not show any section for K3, and the forum search prohibits using the search term k3 as too short.

Community size and open source involvement Banana Pi has a long history in the ARM single-board computer market, but their presence in the RISC-V ecosystem is still growing. Their X account @sinovoip has posted only once about the K3 and otherwise promotes their ARM boards. However, their community culture page does express a commitment to open hardware in general, but there is no visible K3-specific community activity.

Firefly Firefly s K3 product page is comprehensive. Based on the details, they do not offer the K3 pico-ITX variant at all, but only the K3 CoM260 board inside the AIBOX-K3 Firefly RISC-V Edge Mini PC product. This is a feature-complete offering with a Jetson Orin Nano carrier board and case. The AIBOX-K3 with 32 GB RAM and 128 GB SSD in a case sells for 689 USD in their own Firefly.store. Unfortunately it only has HDMI and there is no USB-C with DisplayPort support, which is a deal-breaker for me personally. Interestingly, Firefly also offers rack-mounted servers with K3 as the CPU.

Documentation and support The wiki link on the product page is broken. The Firefly wiki does have a section for the AIBOX-K3, but it too has a broken link. It seems that as of the time of writing, there is no wiki section for this product yet. For support there is a web forum, which does have at least one K3 thread covering guides such as Hermes Agent installation, though broader K3-specific sections are still sparse.

Community size and open source involvement Firefly s X account @TeeFirefly has had no posts since 2024, and their GitLab/T-Firefly shows mostly 2024 activity, with only one repository updated in 2025 and nothing in 2026. Historically they have built a moderate community around their ARM-based Rockchip boards, with active forums and wiki contributions for those product lines. Their RISC-V K3 offerings are newer, and likely need a lot more polish to be attractive products overall.

DeepComputing Last, but certainly not least, is the laptop manufacturer DeepComputing that offers a Framework laptop compatible motherboard with the SpacemiT K3 chip. They also sell the plain motherboard, or with the Cooler Master case, which allows one to easily connect it to an external monitor and keyboard and use it as a desktop computer. The plain board with 32 GB RAM and no SSD sells for about 882 EUR. Shipping of the first batch is expected to start by end of June 2026. Their X account @DeepComputingio promotes this DC-ROMA RISC-V Mainboard III as their flagship product, so they seem to put a lot of effort into it. The overall product design and packaging seems good. Of all the K3 resellers and integrators that I was able to find, DeepComputing is the only one that actually designs their own boards with the K3 processor, while all the other vendors above are simply reselling the vanilla K3 boards with or without a case. After reviewing all these options I decided to buy the DC-ROMA RISC-V Mainboard III for Framework Laptop 13 with 32 GB RAM, 1 TB SSD and the Cooler Master case, totalling about 1100 EUR.

Documentation and support DeepComputing maintains product information for their RISC-V hardware at github.com/DC-DeepComputing/Framework, with documentation of the newest Mainboard III (FML13V05) still being finalized ahead of the first batch shipment. They provide community support through Discord and web forum, although the latter has very little activity.

Community size and open source involvement DeepComputing has established itself as a pioneer in RISC-V laptops, beginning with the DC-ROMA. I have seen their stand at FOSDEM, which shows they are genuinely active in the open source community. Canonical lists DeepComputing s first mainboard / FML13V01 on its official Ubuntu for RISC-V partner-built hardware page, and it seems likely that they will continue to collaborate with Canonical with the new model once it ships. While the underlying Linux enablement depends on SpacemiT s upstream efforts, DeepComputing s involvement helps bridge the gap between reference hardware and consumer-ready products.

Conclusion After weighing all the options, I ended up placing an order with DeepComputing for their custom K3 board with the Cooler Master case. Despite the premium price, the active community support and the properly documented promise of a complete, working system made it easy to place an order with confidence. The SpacemiT K3 is poised to be one of the most significant RISC-V chips for local AI workloads, thanks to its RVA23 compliance and high tokens per second potential. Yet the buying experience in mid-2026 remains fragmented and incomplete. Hopefully this is just because the product is new, and they will get the purchase experience polished soon. What struck me most during this process was how poor the customer experience is across nearly all of these vendor websites: broken links, missing search functions, outdated product banners, pages that show the wrong product entirely, and no information about shipping times, stock levels, taxes, and so on. One wonders why these companies don t fully invest in their web presence. Personally I would assume they likely have enough customers already, primarily through domestic channels like Taobao and JD.com, that they do not feel any pressure to improve their international-facing sites. However, I did also review what was offered on Taobao, and the product details were very incomplete there too. Taobao, however, has a built-in live chat with almost all sellers, which can be used to ask questions and thus compensate for missing product details. I don t fully understand why the sales process seems unpolished. The websites feel almost like an afterthought a checkbox to claim global reach while the real business apparently happens elsewhere via closed platforms or via inaccessible reseller channels. It is a frustrating reminder that in the RISC-V hardware world, the technology may be open and global, but the purchase experience is less so.

• [DLA 4580-1] exim4 security update to fix one CVE related to remote code execution.
• [DLA 4591-1] rsync security update to fix five CVEs related to local root privilege escalation.
• [#1134340] trixie-pu bug for libcoap3 to fix two CVEs in Trixie; the debdiff was confirmed and the upload was accepted to the proposed update queue.
• [#1126167] bookworm-pu upload of zvbi has been flagged for acceptance
• [#1126273] bookworm-pu upload of taglib has been flagged for acceptance
• [#1126370] bookworm-pu upload of libuev has been flagged for acceptance
• [hplip] upload to sid to fix two CVEs.

• lprng to unstable.
• epson-inkjet-printer-escpr to unstable.
• hplip to unstable.

• supernovas to unstable (sponsored upload).
• virtualgps to unstable.
• nautic to unstable.

• pyicloud to unstable.

• visam to unstable.
• tntdb to unstable.
• ae56 to unstable.
• texify to unstable.
• chktex to unstable.
• ta-lib to unstable.

[zram0]
compression-algorithm=zstd(level=10)

zswap.enabled=1 zswap.compressor=zstd zswap.max_pool_percent=20 zswap.shrinker_enabled=1

echo zstd > /sys/module/zswap/parameters/compressor

echo 1 > /sys/module/zswap/parameters/enabled
echo zstd > /sys/module/zswap/parameters/compressor
echo 1 > /sys/module/zswap/parameters/shrinker_enabled

# grep -r . /sys/kernel/debug/zswap/
/sys/kernel/debug/zswap/stored_pages:262541
/sys/kernel/debug/zswap/pool_total_size:455266304
/sys/kernel/debug/zswap/written_back_pages:384
/sys/kernel/debug/zswap/reject_compress_poor:0
/sys/kernel/debug/zswap/reject_compress_fail:160911
/sys/kernel/debug/zswap/reject_kmemcache_fail:0
/sys/kernel/debug/zswap/reject_alloc_fail:0
/sys/kernel/debug/zswap/reject_reclaim_fail:0
/sys/kernel/debug/zswap/pool_limit_hit:0

echo "scale=2; " $(</sys/kernel/debug/zswap/stored_pages) " * $(getconf PAGESIZE) /" $(</sys/kernel/debug/zswap/pool_total_size)   bc

# zramctl 
NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          7.7G  2.1G  375M  386M       4 [SWAP]

#!/bin/bash
if [ ! -f /sys/kernel/debug/zswap/stored_pages ]; then
  echo "ZSwap not enabled"
  exit 0
fi
PAGES=$(</sys/kernel/debug/zswap/stored_pages)
PAGESIZE=$(getconf PAGESIZE)
RAM=$(echo "$PAGESIZE * " $(getconf _PHYS_PAGES)   bc)
POOL=$(</sys/kernel/debug/zswap/pool_total_size)
if [ "$POOL" == "0" ]; then
  echo "ZSwap not used yet"
  exit 0
fi
COMP=$(</sys/module/zswap/parameters/compressor)
echo -n "$COMP compression ratio: "
echo "scale=2; $PAGES * $PAGESIZE / $POOL"   bc
echo -n "RAM%: "
echo "100 * $POOL / $RAM"   bc

• [1] https://etbe.coker.com.au/2025/08/27/zram-vms/
• [2] https://tinyurl.com/2bhknn7e
• [3] https://tinyurl.com/2bxvj3pq
• [4] https://wiki.debian.org/Zswap

Previous implementation Akvorado connects 2 elements to build its RIB:

1. a prefix tree, and
2. a list of routes attached to each prefix.

Akvorado BMP RIB implementation without sharding. One single read/write lock.

In the diagram above, the RIB stores five IPv4 prefixes and two IPv6 prefixes. One of them, 2001:db8:1::/48, contains three routes:

• from peer 3, next hop 2001:db8::3:1, AS 65402, AS path 65402, community 65402:31,
• from peer 4, next hop 2001:db8::4:1, same ASN, AS path, and community,
• from peer 5, next hop 2001:db8::5:1, AS 65402, AS path 65401 65402, community 65402:31.

The rib structure is defined in Go as follows:

type rib struct  
    tree          *bart.Table[prefixIndex]
    routes        map[routeKey]route
    nlris         *intern.Pool[nlri]
    nextHops      *intern.Pool[nextHop]
    rtas          *intern.Pool[routeAttributes]
    nextPrefixID  prefixIndex
    freePrefixIDs []prefixIndex
 

The prefix tree uses the bart package, an adaptation of Donald Knuth s ART algorithm. The benchmarks demonstrate it outperforms other packages for lookups, insertions, and memory usage.³ Plus, the author is quite helpful.

Storing routes in a map The list of routes for each prefix is not stored directly in the prefix tree: it would put too much pressure on the garbage collector by allocating per-prefix arrays. Instead, the RIB assigns a unique 32-bit prefix identifier for each prefix, either by picking the last available prefix identifier from the freePrefixIDs array if any, or using the nextPrefixID value before incrementing it. Then, the routes are stored in the routes map, leveraging the optimized Swiss table in Go. To retrieve routes attached to a prefix, we look them up one by one in the routes map with a 64-bit key combining the 32-bit prefix index with a 32-bit route index matching the position of the route in the list. Akvorado scans routes from the first to the last to find the best one.⁴ It knows there is no more route if the route key returns no result.

type prefixIndex uint32
type routeIndex uint32
type routeKey uint64

Interning routes A route contains a BGP peer identifier, a partial NLRI⁵, the next hop, and the attributes.

type route struct  
    peer       uint32
    nlri       intern.Reference[nlri]
    nextHop    intern.Reference[nextHop]
    attributes intern.Reference[routeAttributes]
    prefixLen  uint8
 
type nlri struct  
    family bgp.Family
    path   uint32
    rd     RD
 
type nextHop netip.Addr
type routeAttributes struct  
    asn              uint32
    asPath           []uint32
    communities      []uint32
    largeCommunities []bgp.LargeCommunity
 

To save memory and allocations, NLRI, next hops, and route attributes are interned : a 32-bit integer replaces the real value. The mechanism predates the unique package introduced in Go 1.23. We keep it because it has different trade-offs:

• It uses explicit reference counting instead of relying on weak pointers.
• It works with non-comparable values implementing Hash() and Equal() methods.⁶
• It uses explicit pool instances. This will be useful for sharding.
• It has better performance. See for example this benchmark.
• It consumes half the memory thanks to unsigned 32-bit references instead of pointers.
• But it is not safe for concurrent use.

Why does it not scale?

Note At AS 12322, we don t use BMP yet.⁷ But Gerhard Bogner had the patience, availability, and technical skills to help me debug this issue.

The global read/write lock is a bottleneck in this implementation. But how? There are several users of the RIB, each with its own set of constraints:

• The Kafka workers look up the RIB to enrich flows with routing information. They are bound by the number of Kafka partitions.⁸ Akvorado also adjusts their number to ensure efficient batching to ClickHouse. On our setup, the number of workers oscillates between 8 and 16. As we want to observe the latest data, we cannot afford for the Kafka workers to lag too much.
• The monitored routers send route updates through the BMP protocol. When connecting, they can send millions of routes.⁹ After the initial synchronization, updates are sent continuously and may spike from time to time. The router detects a stuck BMP station when its TCP window is full and resets the session in this case. While Akvorado implements a large incoming buffer, it still needs to update the received routes with the write lock held fast enough to avoid being detected as stuck.
• When a remote BGP peer goes down, Akvorado flushes the associated routes by walking the RIB with the write lock held. When a monitored router goes down, Akvorado waits a bit but eventually flushes all the associated routes.

In short: on a busy setup, lock contention is high for both readers and writers, and neither can lag too much behind.

RIB sharding

First step: basic sharding To remove the global lock, the RIB is split into several shards, each one handling a subset of the prefixes:

Akvorado BMP RIB implementation with sharding.

The prefix tree stays global and is protected by a single lock. Each shard gets its read/write lock, its route map, and its intern pools to store NLRIs, next hops, and route attributes, which would not have been possible with Go s unique package. The prefix indexes are also sharded: the 8 most significant bits are the shard index and the 24 remaining bits are the local prefix index. Gerhard confirmed that after this blind change, the BMP receiver chugged steadily. Later, I wrote a concurrent benchmark over half a million synthetic but plausible routes¹⁰ partitioned over 0 to 8 writers, churning routes as fast as possible, while 1 to 16 readers continuously look up a set of 10,000 routes. I don t know if this benchmark is realistic, but it confirms the improvements for both read and write latencies:

Read and write latency performance improvement after sharding.

It also shows that a high number of writers degrades read latency.

Second step: lock-free reads The single read/write lock protecting the prefix tree is the next target. The bart package provides alternative mutation methods returning an updated tree using copy-on-write. Readers don t need the global lock any more, leaving it only to synchronize writers. The prefix tree is boxed in an atomic pointer.

Akvorado BMP RIB implementation with sharding and lock-free reads.

Without a lock, readers can now fetch a stale prefix index when walking their copy of the tree if a concurrent writer removes the last route attached to this prefix index and recycles it for another prefix. To avoid this issue, we combine the prefix index with a generation number and store them in the tree:

type generation uint32
type prefixRef struct  
    idx prefixIndex
    gen generation
 
type rib struct  
    mu     sync.Mutex
    tree   atomic.Pointer[bart.Table[prefixRef]]
    shards []*ribShard
 

Each shard stores the generation number for each local prefix index. The generation number increases by one if the associated prefix index is freed. When looking up the routes attached to a prefix index, the reader checks if the generation number matches. Otherwise, it assumes the index was recycled and the list of routes is empty.¹¹ You can see this case in the diagram above for prefix index 5, stored with a generation index of 3, while the current value in the []generations array is 4. The generation number could overflow, but it is not a problem as lookups are quick. Running the concurrent benchmark against this new implementation shows the improvements for the read latency as soon as the cost of the copy-on-write prefix tree is amortized.

Read and write latency performance improvement after lock-free reads. The middle column shows the cumulative improvements of both steps.

---

Among the multiple attempts to optimize the BMP component, RIB sharding is one of the more satisfying. Akvorado 2.2 implements the first step. PR #2433, drafted while writing this blog post, implements the second step and was released with Akvorado 2.4.

---

1. Each router exporting flows doesn t need to send its routes. When Akvorado does not find a route from a specific device, it falls back to a route sent by another device. It is up to the operator to decide if this is a good enough approximation.
2. I made many attempts to scale the BMP component. See for example PR #254, PR #255, PR #278, PR #2244, and PR #2245. Despite these efforts, this component remained problematic for some users. See discussion #2287 as the latest example.
3. It keeps improving: bart 0.28.0 features a new implementation that trades a bit of memory for greater lookup performance. I did not test it yet, as I have been preparing this blog post for a couple of months already.
4. Akvorado prefers the route matching the exact next hop. Otherwise, it falls back to any other route. This is an approximation. An alternative would be to have one prefix tree for each BGP peer but it would require configuring all routers to export their routes. pmacct s BMP daemon implements this approach.
5. If we consider the BGP RIB as a database, the Network Layer Reachability Information (NLRI) is the primary key. Its content depends on the BGP family. With IPv4 or IPv6 unicast, this is the prefix. For VPNv4 and VPNv6 families, it includes the route distinguisher. If you enable the ADD-PATH extension, the NLRI also contains a path identifier. In our implementation, we don t store the prefix as we get it from the looked-up IP address using the prefix length stored separately.
6. The Hash() methods rely on the hash/maphash package and on the unsafe package to avoid memory copies. See for example the Hash() function for the nlri structure.
7. Despite being an author or co-author of the first BMP-related RFCs since 2016 (RFC 7854, RFC 8671, RFC 9069), Cisco did not implement it in a usable way in IOS XR until version 24.2.1. We still need to upgrade a few routers to enable this feature.
8. KIP-932 introduces, in Kafka 4.2, the concept of share groups to enable cooperative consumption on the same partition. This is not supported in Akvorado yet.
9. You can configure BMP to send routes for each BGP peer before or after applying the incoming policies. In this case, you can get more than one million routes for each transit peer. You can also tell BMP to send the local RIB, which only contains the best path for each prefix.
10. The prefixes are random, but the prefix size distribution and the AS path length distribution follow the data provided by Geoff Huston.
11. Alternatively, we could retry the lookup, but it would be pointless: the RIB is an eventually consistent database, and an empty list was a correct answer at some point in the recent past.

What is persistence actually? To think clearly about this topic, one must first grasp the concept of grit and what it looks like in practice. Research by psychologist Angela Duckworth on grit shows that sustained effort in the face of setbacks separates high achievers from those who quit too soon. Entrepreneurs who iterated through dozens of failed prototypes or writers who revised manuscripts for years understand this truth. Persistence builds resilience, deep expertise, and the kind of compounding results that shortcuts cannot deliver. It also protects against the distraction of shiny new ideas that pull focus from what actually works. Persistence is about:

1. Believing in an outcome and working towards it despite people around you not sharing the belief, and despite your own work and experiments not being successful.
2. Continuing to hold the belief and sticking to the decision despite other ideas, solutions and competing alternatives surfacing.
3. The more time passes, the firmer the conviction becomes. Time, money, and emotional energy invested in a failing direction create psychological pressure to continue (sunk-cost fallacy).

Simply following through on a plan or upholding a contract is not true persistence. Grit is a personal trait one can cultivate to actually become more energized to do something precisely because it turns out to be harder than expected.

Pivoting: a calculated choice The opposite of being persistent is giving up. Pivoting is not about giving up, but about redirecting the energy and momentum towards a new goal. Pivoting requires coming to the realization that you were wrong, and going through the painful process of discovering a new truth. Ideas tend to be abundant, and doing something new isn t hard as such. The hard part is to abandon a previously held belief and adopt a new one with equal conviction. To have that conviction you need to have data and metrics. This is also the key to how to decide between persisting vs pivoting at any moment in time.

Key metrics of success Any decision is only as good as the information available at the time it was made. To be set up for success one needs to start by deciding on what the actual goal is, what one values and how progress is measured. Key metrics are usually easiest to discover by working backwards from the goal. If you want to build an electric car, you might decide that the goal is to have a car that costs 30,000 euros and can drive 300 km on one charge. From that goal you can break down what the cost structure should be, what volume of production is needed to break even, what raw materials are needed and what the battery chemistry needs to achieve to meet the goal. That can further be broken down into a rate of progress. Suppose the plan requires battery energy density to reach 150 Wh/kg to be viable. If the state of the art starts at 100 Wh/kg and funding lasts a maximum of five years, the team needs at least an 8% improvement every year (1.08^5 100 Wh/kg 150 Wh/kg). This can then be used as a guideline. Sometimes progress is not steady, but happens in jumps. Even in those cases there should be a trajectory to benchmark the jumps against. In an online business, the key metric could, for example, be one of these:

• 7- or 30-day retention rate: Do new users who try the service actually like it?
• Weekly or monthly active users: Is usage trending up?
• Feature adoption rate: In an existing service, how many users are using the new feature?
• Product-Market Fit Score (from Sean Ellis test): Percentage of users who say they would be very disappointed if the product disappeared. Above 40% is a strong early indicator. A number below that (after multiple iterations) is a good data point to pivot.
• Revenue run rate or burn rate: The most generic metric everything eventually boils down to. Healthy markets reward good products.

Weekly metrics are better than monthly, as they make the feedback loop faster and allow you to get validation quickly and do minor course corrections along the way. A complete pivot should, however, be based on long-term data, driven by the key metric and supported by additional data points. Metrics are also needed because they can t be bribed or convinced to be anything other than what they are. Listening to other people is good, but just relying on the opinion of others is extremely dangerous because people are biased either for you or against you depending on whether they see you as a trusted leader or an outcast. Key metrics are of course domain-specific and everyone needs to come up with their own. However, you must have some key metric. You can t have the excuse that what you are doing can t be measured. If you are part of a larger organization and you need to advocate for a difficult decision for example, to kill your darlings when facing a pivot you need to have the metrics to back up your views, and those metrics need to have been established way before as something the organization values, and not cherry-picked just for this one decision. It does not matter if you are on a personal improvement journey, running a political campaign, inventing a new product, or growing a business you need to have some metric you can check at any given time to see if things are improving fast enough to predict success. Metrics can and should also be used in daily work to validate that you are on the correct path, and to optimize execution.

Famous examples of persistence and pivoting that led to breakthroughs In all of the cases below it is of course in hindsight easy to say they made the right decision. However, take a minute to try to imagine yourself in their shoes at the time of the decision. What metrics might they have had available to support their decision? What would you have wanted to measure or find out if you were in the same situation?

• Frustrated that his vacuum lost suction, James Dyson spent five years and built thousands of failed prototypes in a backyard shed. He remortgaged his home, lived on savings, and faced rejection from every major manufacturer who wanted to protect their bag-replacement business. The 5,127th prototype based on an idea from a sawmill with a cyclone finally worked. Launched in 1993, the Dyson DC01 became Britain s best-selling vacuum within two years.
• As a single mother on welfare in the mid-1990s, J.K. Rowling finished her manuscript for Harry Potter and the Philosopher s Stone while battling depression and poverty. She hand-typed copies and mailed them to publishers. Twelve rejected it outright, with comments like children s books about magic don t sell. She nearly quit multiple times but kept revising and submitting. Bloomsbury finally accepted it after the CEO s eight-year-old daughter read the first chapter and demanded the rest. The series has since sold hundreds of millions of copies worldwide.
• Founded in 1997 as a mail-order DVD rental service, Netflix added unlimited subscriptions in 1999 to compete with Blockbuster. By 2007, broadband growth and declining DVD sales signaled a shift. CEO Reed Hastings pivoted aggressively toward streaming, investing in bandwidth deals and original content while de-emphasizing physical media. The move faced skepticism, but eventually changed the whole culture of how entertainment is consumed.
• YouTube launched in 2005 as a video-dating site. Founders offered money to women who uploaded dating videos, but almost no one did. Meanwhile, users uploaded random clips. The team recognized the mismatch and pivoted within months to a general-purpose video-sharing platform with easy uploading. Google bought it just 18 months later.
• Instagram began in 2010 as Burbn, a location-based check-in app that let users post plans, earn points, and share photos. Co-founders Kevin Systrom and Mike Krieger quickly noticed users ignored most features and mainly used it for photo-sharing. They made the tough call: scrap everything else. Within weeks, they rebuilt the app around clean, simple photography with filters. The pivot launched as Instagram in October 2010. It gained 1 million users in two months and was acquired by Facebook just 18 months later.

Insanity or conviction? English has several proverbs that warn against excessive persistence, such as banging your head against the wall . Insanity is commonly defined as Doing the same thing over and over again and expecting different results. In Finland, the national identity is practically built on the concept of sisu . It means much more than just grit . The word is derived from the word for inside or guts and represents an unexplained, almost superhuman force that makes one stoically take action despite seemingly impossible odds and somehow succeed anyway. It became a defining national mythos during the Winter War (1939 1940), where a force 10 times larger than the Finnish army tried to invade the country but was stopped and Finland just barely managed to keep its independence. The word sisu transitioned from a character trait to a pillar of national survival. I think Finns survived because the more you believe in persistence, the more likely you are to persist. I view persistence as a religion that requires faith, while pivoting is a science where you derive the truth from the numbers. When in doubt, I would always choose persistence over pivoting. Perhaps it is because of my genetic tendency towards having sisu , but I would also rather keep on going a bit more and try one more time before giving up and pivoting in order to get more data, so that when I pivot, I know it is absolutely the right thing to do at that point. Depending on the situation, the costs of postponing the pivot vary. Of course, if the main metric is the burn rate and a company is running out of money, a pivot must be done early enough that the remaining runway is enough to execute the pivot, and then some more. In some situations a business idea might simply be ahead of its time. If that is the conviction and the key metrics support it, the best way to navigate the situation is to cut down on costs and wait for competitors to appear, help build general awareness, and then ramp up again to ride the wave. Remember that success does not come from grit alone there is always an element of timing and luck as well. But if you are not persistent and stop showing up every day, you won t be able to seize the opportunities if and when they arise.

Failure is the likely outcome you have to avoid it at any cost One must also realize that most attempts end in failure. Failure is the baseline, and success is the exception. To reach a breakthrough, one must be stubbornly persistent. In particular, if you are a leader, you need to be so high in conviction that it almost becomes an aura that radiates to those around you. Postponing the decision to pivot allows you to get a bit more data for the decision, so that once you pivot, you have full belief in the new direction. Once you pivot, there is no looking back, otherwise you will undermine morale and most certainly fail with the new thing as people will execute it with hesitation. Failure is statistically always the more likely outcome. Most things end in failure and we never hear about them. If someone on your team does not believe in what you are doing, it is very easy for them to prove that something is a failure by spreading negativity, putting in less effort (perhaps unconsciously due to lack of conviction) and thus actually contributing to a self-fulfilling failure. In most areas of life, ideas are cheap and the only thing that matters is execution. To be good at executing, you need to be good at making decisions. When drafting plans it is good to have alternatives and a lot of consideration. However, when execution starts, there is no room for doubt, otherwise the chances of success decrease. Therefore, the best way of balancing persistence vs pivoting is to

1. plan well ahead,
2. establish the key metrics,
3. have thresholds established for what would trigger a pivot, and
4. do everything you can to move the metrics in the direction you want them to go.

Finally, if you decide to pivot, you must do so only with very high conviction, as you can t undo a pivot, and you should not be doing multiple pivots in a row either. If you are fully convinced yourself about the pivot, you will also be able to convince others about it, and carry the momentum.

• Activist Checklist: Install the latest software updates
• Consumer Reports: Update your Mac, Windows PC, Chromebook, Android phone, iOS device
• Apple: Obsolete products (obsolete devices do not receive security updates)
• Google: How long you'll get Pixel updates
• Samsung: Devices that receive security updates
• Other brands: check the manufacturer's website

• Activist Checklist: Use Signal
• Activist Checklist: Turn on disappearing messages
• Electronic Frontier Foundation: How to use Signal
• Consumer Reports: Communicate privately with Signal
• Signal: Enabling Incognito Keyboard (Android) to provide slightly more privacy

• Electronic Frontier Foundation: Creating strong passwords
• Electronic Frontier Foundation: Remove fingerprint or face unlock
• How-To Geek: Temporarily disable biometric unlock on Android and iOS
• Electronic Frontier Foundation: How to encrypt your Windows, Mac, or Linux computer, your iPhone, Android Privacy and Security settings
• Consumer Reports: Protect your Mac, Windows PC, Android phone, iPhone devices with encryption

• Consumer Reports: Add an ad blocker
• uBlock Origin, a highly recommended browser-based ad blocker (Firefox, Chrome)
• AdGuard Ad Blocker is multi-platform and also supports Mac/iOS

• Consumer Reports: Set Up HTTPS-Only Mode on your browser

1. They generate secure passwords with ease. You don't need to worry about getting your digits and special characters just right; the app will do it for you, and generate long, secure passwords.
2. They remember all your passwords for you, and you just need to remember one password to access all of them. The most common reason people's accounts get hacked online is because they used the same password across multiple websites, and one of the websites had all their passwords leaked. When you use a unique password on every website, it doesn't matter if your password gets leaked!
3. They autofill passwords based on the website you're visiting. This is important because it helps prevent you from getting phished. If you're tricked into visiting an evil lookalike site, your password manager will refuse to fill the password.

• Activist Checklist: Use a password manager
• Electronic Frontier Foundation: An animated overview of password managers
• Electronic Frontier Foundation: Choosing a password manager
• Consumer Reports: Get a password manager

• Activist Checklist: Enable two-factor authentication
• Electronic Frontier Foundation: How to enable two-factor authentication
• Consumer Reports: Set up multifactor authentication
• Apps: There are many different choices available. The linked guides recommend the open source app Ente. Other options include Google and Microsoft Authenticator, Duo, etc.
• Hardware tokens: Common choices include Yubikey and Google's Titan Security Key

• Consumer Reports: Remove your contact information from people-search sites
• Electronic Frontier Foundation: Manage your digital footprint
• Big Ass Data Broker Opt Out List (BADBOOL), maintained by Yael Grauer
• State-specific tools for residents of: California, Oregon

• Threat modelling, which you can get started with by reading the EFF's or VCW's guides
• Browser addons for privacy, which Consumer Reports has a tip for
• Secure DNS, which you can read more about here

$ git diff --shortstat v1.11.5
 493 files changed, 25015 insertions(+), 21135 deletions(-)

New outlet service The major change in Akvorado 2.0 is splitting the inlet service into two parts: the inlet and the outlet. Previously, the inlet handled all flow processing: receiving, decoding, and enrichment. Flows were then sent to Kafka for storage in ClickHouse:

Akvorado flow processing before the introduction of the outlet service

Network flows reach the inlet service using UDP, an unreliable protocol. The inlet must process them fast enough to avoid losing packets. To handle a high number of flows, the inlet spawns several sets of workers to receive flows, fetch metadata, and assemble enriched flows for Kafka. Many configuration options existed for scaling, which increased complexity for users. The code needed to avoid blocking at any cost, making the processing pipeline complex and sometimes unreliable, particularly the BMP receiver.¹ Adding new features became difficult without making the problem worse.² In Akvorado 2.0, the inlet receives flows and pushes them to Kafka without decoding them. The new outlet service handles the remaining tasks:

Akvorado flow processing after the introduction of the outlet service

This change goes beyond a simple split:³ the outlet now reads flows from Kafka and pushes them to ClickHouse, two tasks that Akvorado did not handle before. Flows are heavily batched to increase efficiency and reduce the load on ClickHouse using ch-go, a low-level Go client for ClickHouse. When batches are too small, asynchronous inserts are used (e20645). The number of outlet workers scales dynamically (e5a625) based on the target batch size and latency (50,000 flows and 5 seconds by default). This new architecture also allows us to simplify and optimize the code. The outlet fetches metadata synchronously (e20645). The BMP component becomes simpler by removing cooperative multitasking (3b9486). Reusing the same RawFlow object to decode protobuf-encoded flows from Kafka reduces pressure on the garbage collector (8b580f). The effect on Akvorado s overall performance was somewhat uncertain, but a user reported 35% lower CPU usage after migrating from the previous version, plus resolution of the long-standing BMP component issue.

Other changes This new version includes many miscellaneous changes, such as completion for source and destination ports (f92d2e), and automatic restart of the orchestrator service (0f72ff) when configuration changes to avoid a common pitfall for newcomers. Let s focus on some key areas for this release: observability, documentation, CI, Docker, Go, and JavaScript.

Observability Akvorado exposes metrics to provide visibility into the processing pipeline and help troubleshoot issues. These are available through Prometheus HTTP metrics endpoints, such as /api/v0/inlet/metrics. With the introduction of the outlet, many metrics moved. Some were also renamed (4c0b15) to match Prometheus best practices. Kafka consumer lag was added as a new metric (e3a778). If you do not have your own observability stack, the Docker Compose setup shipped with Akvorado provides one. You can enable it by activating the profiles introduced for this purpose (529a8f). The prometheus profile ships Prometheus to store metrics and Alloy to collect them (2b3c46, f81299, and 8eb7cd). Redis and Kafka metrics are collected through the exporter bundled with Alloy (560113). Other metrics are exposed using Prometheus metrics endpoints and are automatically fetched by Alloy with the help of some Docker labels, similar to what is done to configure Traefik. cAdvisor was also added (83d855) to provide some container-related metrics. The loki profile ships Loki to store logs (45c684). While Alloy can collect and ship logs to Loki, its parsing abilities are limited: I could not find a way to preserve all metadata associated with structured logs produced by many applications, including Akvorado. Vector replaces Alloy (95e201) and features a domain-specific language, VRL, to transform logs. Annoyingly, Vector currently cannot retrieve Docker logs from before it was started. Finally, the grafana profile ships Grafana, but the shipped dashboards are broken. This is planned for a future version.

Documentation The Docker Compose setup provided by Akvorado makes it easy to get the web interface up and running quickly. However, Akvorado requires a few mandatory steps to be functional. It ships with comprehensive documentation, including a chapter about troubleshooting problems. I hoped this documentation would reduce the support burden. It is difficult to know if it works. Happy users rarely report their success, while some users open discussions asking for help without reading much of the documentation. In this release, the documentation was significantly improved.

$ git diff --shortstat v1.11.5 -- console/data/docs
 10 files changed, 1873 insertions(+), 1203 deletions(-)

The documentation was updated (fc1028) to match Akvorado s new architecture. The troubleshooting section was rewritten (17a272). Instructions on how to improve ClickHouse performance when upgrading from versions earlier than 1.10.0 was added (5f1e9a). An LLM proofread the entire content (06e3f3). Developer-focused documentation was also improved (548bbb, e41bae, and 871fc5). From a usability perspective, table of content sections are now collapsable (c142e5). Admonitions help draw user attention to important points (8ac894).

Example of use of admonitions in Akvorado's documentation

Continuous integration This release includes efforts to speed up continuous integration on GitHub. Coverage and race tests run in parallel (6af216 and fa9e48). The Docker image builds during the tests but gets tagged only after they succeed (8b0dce).

GitHub workflow to test and build Akvorado

End-to-end tests (883e19) ensure the shipped Docker Compose setup works as expected. Hurl runs tests on various HTTP endpoints, particularly to verify metrics (42679b and 169fa9). For example:

## Test inlet has received NetFlow flows
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_flow_input_udp_packets_total job="akvorado-inlet",listener=":2055" )
HTTP 200
[Captures]
inlet_receivedflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_receivedflows" > 10
## Test inlet has sent them to Kafka
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_kafka_sent_messages_total job="akvorado-inlet" )
HTTP 200
[Captures]
inlet_sentflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_sentflows" >=   inlet_receivedflows  

Docker Akvorado ships with a comprehensive Docker Compose setup to help users get started quickly. It ensures a consistent deployment, eliminating many configuration-related issues. It also serves as a living documentation of the complete architecture. This release brings some small enhancements around Docker:

• an example to configure Traefik for TLS with Let s Encrypt (1a27bb),
• HTTP compression (bee9a5), and
• support of IPv6 (a74a41).

Previously, many Docker images were pulled from the Bitnami Containers library. However, VMWare acquired Bitnami in 2019 and Broadcom acquired VMWare in 2023. As a result, Bitnami images were deprecated in less than a month. This was not really a surprise⁴. Previous versions of Akvorado had already started moving away from them. In this release, the Apache project s Kafka image replaces the Bitnami one (1eb382). Thanks to the switch to KRaft mode, Zookeeper is no longer needed (0a2ea1, 8a49ca, and f65d20). Akvorado s Docker images were previously compiled with Nix. However, building AArch64 images on x86-64 is slow because it relies on QEMU userland emulation. The updated Dockerfile uses multi-stage and multi-platform builds: one stage builds the JavaScript part on the host platform, one stage builds the Go part cross-compiled on the host platform, and the final stage assembles the image on top of a slim distroless image (268e95 and d526ca).

# This is a simplified version
FROM --platform=$BUILDPLATFORM node:20-alpine AS build-js
RUN apk add --no-cache make
WORKDIR /build
COPY console/frontend console/frontend
COPY Makefile .
RUN make console/data/frontend
FROM --platform=$BUILDPLATFORM golang:alpine AS build-go
RUN apk add --no-cache make curl zip
WORKDIR /build
COPY . .
COPY --from=build-js /build/console/data/frontend console/data/frontend
RUN go mod download
RUN make all-indep
ARG TARGETOS TARGETARCH TARGETVARIANT VERSION
RUN make
FROM gcr.io/distroless/static:latest
COPY --from=build-go /build/bin/akvorado /usr/local/bin/akvorado
ENTRYPOINT [ "/usr/local/bin/akvorado" ]

When building for multiple platforms with --platform linux/amd64,linux/arm64,linux/arm/v7, the build steps until the highlighted line execute only once for all platforms. This significantly speeds up the build. Akvorado now ships Docker images for these platforms: linux/amd64, linux/amd64/v3, linux/arm64, and linux/arm/v7. When requesting ghcr.io/akvorado/akvorado, Docker selects the best image for the current CPU. On x86-64, there are two choices. If your CPU is recent enough, Docker downloads linux/amd64/v3. This version contains additional optimizations and should run faster than the linux/amd64 version. It would be interesting to ship an image for linux/arm64/v8.2, but Docker does not support the same mechanism for AArch64 yet (792808).

Go This release includes many changes related to Go but not visible to the users.

Toolchain In the past, Akvorado supported the two latest Go versions, preventing immediate use of the latest enhancements. The goal was to allow users of stable distributions to use Go versions shipped with their distribution to compile Akvorado. However, this became frustrating when interesting features, like go tool, were released. Akvorado 2.0 requires Go 1.25 (77306d) but can be compiled with older toolchains by automatically downloading a newer one (94fb1c).⁵ Users can still override GOTOOLCHAIN to revert this decision. The recommended toolchain updates weekly through CI to ensure we get the latest minor release (5b11ec). This change also simplifies updates to newer versions: only go.mod needs updating. Thanks to this change, Akvorado now uses wg.Go() (77306d) and I have started converting some unit tests to the new test/synctest package (bd787e, 7016d8, and 159085).

Testing When testing equality, I use a helper function Diff() to display the differences when it fails:

got := input.Keys()
expected := []int 1, 2, 3 
if diff := helpers.Diff(got, expected); diff != ""  
    t.Fatalf("Keys() (-got, +want):\n%s", diff)
 

This function uses kylelemons/godebug. This package is no longer maintained and has some shortcomings: for example, by default, it does not compare struct private fields, which may cause unexpectedly successful tests. I replaced it with google/go-cmp, which is stricter and has better output (e2f1df).

Another package for Kafka Another change is the switch from Sarama to franz-go to interact with Kafka (756e4a and 2d26c5). The main motivation for this change is to get a better concurrency model. Sarama heavily relies on channels and it is difficult to understand the lifecycle of an object handed to this package. franz-go uses a more modern approach with callbacks⁶ that is both more performant and easier to understand. It also ships with a package to spawn fake Kafka broker clusters, which is more convenient than the mocking functions provided by Sarama.

Improved routing table for BMP To store its routing table, the BMP component used kentik/patricia, an implementation of a patricia tree focused on reducing garbage collection pressure. gaissmai/bart is a more recent alternative using an adaptation of [Donald Knuth s ART algorithm][] that promises better performance and delivers it: 90% faster lookups and 27% faster insertions (92ee2e and fdb65c). Unlike kentik/patricia, gaissmai/bart does not help efficiently store values attached to each prefix. I adapted the same approach as kentik/patricia to store route lists for each prefix: store a 32-bit index for each prefix, and use it to build a 64-bit index for looking up routes in a map. This leverages Go s efficient map structure. gaissmai/bart also supports a lockless routing table version, but this is not simple because we would need to extend this to the map storing the routes and to the interning mechanism. I also attempted to use Go s new unique package to replace the intern package included in Akvorado, but performance was worse.⁷

Miscellaneous Previous versions of Akvorado were using a custom Protobuf encoder for performance and flexibility. With the introduction of the outlet service, Akvorado only needs a simple static schema, so this code was removed. However, it is possible to enhance performance with planetscale/vtprotobuf (e49a74, and 8b580f). Moreover, the dependency on protoc, a C++ program, was somewhat annoying. Therefore, Akvorado now uses buf, written in Go, to convert a Protobuf schema into Go code (f4c879). Another small optimization to reduce the size of the Akvorado binary by 10 MB was to compress the static assets embedded in Akvorado in a ZIP file. It includes the ASN database, as well as the SVG images for the documentation. A small layer of code makes this change transparent (b1d638 and e69b91).

JavaScript Recently, two large supply-chain attacks hit the JavaScript ecosystem: one affecting the popular packages chalk and debug and another impacting the popular package @ctrl/tinycolor. These attacks also exist in other ecosystems, but JavaScript is a prime target due to heavy use of small third-party dependencies. The previous version of Akvorado relied on 653 dependencies. npm-run-all was removed (3424e8, 132 dependencies). patch-package was removed (625805 and e85ff0, 69 dependencies) by moving missing TypeScript definitions to env.d.ts. eslint was replaced with oxlint, a linter written in Rust (97fd8c, 125 dependencies, including the plugins). I switched from npm to Pnpm, an alternative package manager (fce383). Pnpm does not run install scripts by default⁸ and prevents installing packages that are too recent. It is also significantly faster.⁹ Node.js does not ship Pnpm but it ships Corepack, which allows us to use Pnpm without installing it. Pnpm can also list licenses used by each dependency, removing the need for license-compliance (a35ca8, 42 dependencies). For additional speed improvements, beyond switching to Pnpm and Oxlint, Vite was replaced with its faster Rolldown version (463827). After these changes, Akvorado only pulls 225 dependencies.

Next steps I would like to land three features in the next version of Akvorado:

• Add Grafana dashboards to complete the observability stack. See issue #1906 for details.
• Integrate OVH s Grafana plugin by providing a stable API for such integrations. Akvorado s web console would still be useful for browsing results, but if you want to build and share dashboards, you should switch to Grafana. See issue #1895.
• Move some work currently done in ClickHouse (custom dictionaries, GeoIP and IP enrichment) back into the outlet service. This should give more flexibility for adding features like the one requested in issue #1030. See issue #2006.

---

I started working on splitting the inlet into two parts more than one year ago. I found more motivation in recent months, partly thanks to Claude Code, which I used as a rubber duck. Almost none of the produced code was kept:¹⁰ it is like an intern who does not learn.

---

1. Many attempts were made to make the BMP component both performant and not blocking. See for example PR #254, PR #255, and PR #278. Despite these efforts, this component remained problematic for most users. See issue #1461 as an example.
2. Some features have been pushed to ClickHouse to avoid the processing cost in the inlet. See for example PR #1059.
3. This is the biggest commit:

   $ git show --shortstat ac68c5970e2c   tail -1
   231 files changed, 6474 insertions(+), 3877 deletions(-)
   

4. Broadcom is known for its user-hostile moves. Look at what happened with VMWare.
5. As a Debian developer, I dislike these mechanisms that circumvent the distribution package manager. The final straw came when Go 1.25 spent one month in the Debian NEW queue, an arbitrary mechanism I don t like at all.
6. In the early years of Go, channels were heavily promoted. Sarama was designed during this period. A few years later, a more nuanced approach emerged. See notably Go channels are bad and you should feel bad.
7. This should be investigated further, but my theory is that the intern package uses 32-bit integers, while unique uses 64-bit pointers. See commit 74e5ac.
8. This is also possible with npm. See commit dab2f7.
9. An even faster alternative is Bun, but it is less available.
10. The exceptions are part of the code for the admonition blocks, the code for collapsing the table of content, and part of the documentation.

Distance yourself from the digital distractions The key to avoiding distractions and procrastination is to make it inconvenient enough that you rarely do it. If continuing to do work is easier than switching to procrastination, work is more likely to continue. Tips to minimize digital distractions, listed in order of importance:

1. Put your phone away. Just like when you go to a movie and turn off your phone for two hours, you can put the phone away completely when starting to work. Put the phone in a different room to ensure there is enough physical distance between you and the distraction, so it is impossible for you to just take a quick peek .
2. Turn off notifications from apps. Don t let the apps call you like sirens luring Odysseus. You don t need to have all the notifications. You will see what the apps have when you eventually open them at a time you choose to use them.
3. Remove or disable social media apps, games and the like from your phone and your computer. You can install them back when you have vacation. You can probably live without them for some time. If you can t remove them, explore your phone s screen time restriction features to limit your own access to apps that most often waste your time. These features are sometimes listed in the phone settings under digital health .
4. Have a separate work computer and work phone. Having dedicated ones just for work that are void of all unnecessary temptations helps keep distance from the devices that could derail your focus.
5. Listen to music. If you feel your brain needs a dose of dopamine to get you going, listening to music helps satisfy your brain s cravings while still being able to simultaneously keep working.

Doing a full digital detox is probably not practical, or not sustainable for an extended time. One needs apps to stay in touch with friends and family, and staying current in software development probably requires spending some time reading news online and such. However the tips above can help contain the distractions and minimize the spontaneous attention the distractions get. Some of the distractions may ironically be from the work itself, for example Slack notifications or new email notifications. I recommend turning them off for a couple of hours every day to have some distraction free time. It should be enough to check work mail a couple times a day. Checking them every hour probably does not add much overall value for the company unless your work is in sales or support where the main task itself is responding to emails.

Distraction free work environment Following the same principle of distancing yourself from distractions, try to use a dedicated physical space for working. If you don t have a spare room to dedicate to work, use a neighborhood caf or sign up for a local co-working space or start commuting to the company office to find a space to be focused on work in.

Break down tasks into smaller steps Sometimes people postpone tasks because they feel intimidated by the size or complexity of a task. In particular in software engineering problems may be vague and appear large until one reaches the breakthrough that brings the vision of how to tackle it. Breaking down problems into smaller more manageable pieces has many advantages in software engineering. Not only can it help with task-avoidance, but it can also make the problem easier to analyze, suggest solutions and test them and build a solid foundation to expand upon to ultimately later reach a full solution on the entire larger problem. Working on big problems as a chain of smaller tasks may also offer more opportunities to celebrate success on completing each subtask and help getting in a suitable cadence of solving a single thing, taking a break and then tackling the next issue. Breaking down a task into concrete steps may also help with getting more realistic time estimations. Sometimes procrastination isn t real someone could just be overly ambitious and feel bad about themselves for not doing an unrealistic amount of work.

Intrinsic motivation Of course, you should follow your passion when possible. Strive to pick a career that you enjoy, and thus maximize the intrinsic motivation you experience. However, even a dream job is still a job. Nobody is ever paid to do whatever they want. Any work will include at least some tasks that feel like a chore or otherwise like something you would not do unless paid to. Some would say that the definition of work itself is having to do things one would otherwise not do. You can only fully do whatever you want while on vacation or when you choose to not have a job at all. But if you have a job, you simply need to find the intrinsic motivation to do it. Simply put, some tasks are just unpleasant or boring. Our natural inclination is to avoid them in favor of more enjoyable activities. For these situations we just have to find the discipline to force ourselves to do the tasks and figuratively speaking whip ourselves into being motivated to complete the tasks.

Extrinsic motivation As the name implies, this is something people external to you need to provide, such as your employer or manager. If you have challenges in managing yourself and delivering results on a regular basis, somebody else needs to set goals and deadlines and keep you accountable for them. At the end of the day this means that eventually you will stop receiving salary or other payments unless you did your job. Forcing people to do something isn t nice, but eventually it needs to be done. It would not be fair for an employer to pay those who did their work the same salary as those who procrastinated and fell short on their tasks. If you work solo, you can also simulate the extrinsic motivation by publicly announcing milestones and deadlines to build up pressure for yourself to meet them and avoid publicly humiliation. It is a well-studied and scientifically proven phenomenon that most university students procrastinate at the start of assignments, and truly start working on them only once the deadline is imminent.

External help for addictions If procrastination is mainly due to a single distraction that is always on your mind, it may be a sign of an addiction. For example, constantly thinking about a computer game or staying up late playing a computer game, to the extent that it seriously affects your ability to work, may be a symptom of an addiction, and getting out of it may be easier with external help.

Discipline and structure Most of the time procrastination is not due to an addiction, but simply due to lack of self-discipline and structure. The good thing is that those things can be learned. It is mostly a matter of getting into new habits, which most young software engineers pick up more or less automatically while working along the more senior ones. Hopefully these tips can help you stay on track and ensure you do everything you are expected to do with clear focus, and on time!

tl;dr: there is an attack in the wild which is triggering dangerous-but-seemingly-intended behaviour in the Oj JSON parser when used in the default and recommended manner, which can lead to everyone s favourite kind of security problem: object deserialization bugs! If you have the oj gem anywhere in your Gemfile.lock, the quickest mitigation is to make sure you have Oj.default_options = mode: :strict somewhere, and that no library is overwriting that setting to something else.

Prologue As a sensible sysadmin, all the sites I run send me a notification if any unhandled exception gets raised. Mostly, what I get sent is error-handling corner cases I missed, but now and then things get more interesting. In this case, it was a PG::UndefinedColumn exception, which looked something like this:

PG::UndefinedColumn: ERROR:  column "xyzzydeadbeef" does not exist

This is weird on two fronts: firstly, this application has been running for a while, and if there was a schema problem, I d expect it to have made itself apparent long before now. And secondly, while I don t profess to perfection in my programming, I m usually better at naming my database columns than that. Something is definitely hinky here, so let s jump into the mystery mobile!

The column name is coming from outside the building! The exception notifications I get sent include a whole lot of information about the request that caused the exception, including the request body. In this case, the request body was JSON, and looked like this:

 "name":":xyzzydeadbeef", ... 

The leading colon looks an awful lot like the syntax for a Ruby symbol, but it s in a JSON string. Surely there s no way a JSON parser would be turning that into a symbol, right? Right?!? Immediately, I thought that that possibly was what was happening, because I use Sequel for my SQL database access needs, and Sequel treats symbols as database column names. It seemed like too much of a coincidence that a vaguely symbol-shaped string was being sent in, and the exact same name was showing up as a column name. But how the flying fudgepickles was a JSON string being turned into a Ruby symbol, anyway? Enter Oj.

Oj? I barely know aj A long, long time ago, the standard Ruby JSON library had a reputation for being slow. Thus did many competitors flourish, claiming more features and better performance. Strong amongst the contenders was oj (for Optimized JSON ), touted as The fastest JSON parser and object serializer . Given the history, it s not surprising that people who wanted the best possible performance turned to Oj, leading to it being found in a great many projects, often as a sub-dependency of a dependency of a dependency (which is how it ended up in my project). You might have noticed in Oj s description that, in addition to claiming fastest , it also describes itself as an object serializer . Anyone who has kept an eye on the security bug landscape will recall that object deserialization is a rich vein of vulnerabilities to mine. Libraries that do object deserialization, especially ones with a history that goes back to before the vulnerability class was well-understood, are likely to be trouble magnets. And thus, it turns out to be with Oj. By default, Oj will happily turn any string that starts with a colon into a symbol:

>> require "oj"
>> Oj.load(' "name":":xyzzydeadbeef","username":"bob","answer":42 ')
=>  "name"=>:xyzzydeadbeef, "username"=>"bob", "answer"=>42 

How that gets exploited is only limited by the creativity of an attacker. Which I ll talk about more shortly but first, a word from my rant cortex.

Insecure By Default is a Cancer While the object of my ire today is Oj and its fast-and-loose approach to deserialization, it is just one example of a pervasive problem in software: insecurity by default. Whether it s a database listening on 0.0.0.0 with no password as soon as its installed, or a library whose default behaviour is to permit arbitrary code execution, it all contributes to a software ecosystem that is an appalling security nightmare. When a user (in this case, a developer who wants to parse JSON) comes across a new piece of software, they have by definition no idea what they re doing with that software. They re going to use the defaults, and follow the most easily-available documentation, to achieve their goal. It is unrealistic to assume that a new user of a piece of software is going to do things the right way , unless that right way is the only way, or at least the by-far-the-easiest way. Conversely, the developer(s) of the software is/are the domain experts. They have knowledge of the problem domain, through their exploration while building the software, and unrivalled expertise in the codebase. Given this disparity in knowledge, it is tantamount to malpractice for the experts the developer(s) to off-load the responsibility for the safe and secure use of the software to the party that has the least knowledge of how to do that (the new user). To apply this general principle to the specific case, take the Using section of the Oj README. The example code there calls Oj.load, with no indication that this code will, in fact, parse specially-crafted JSON documents into Ruby objects. The brand-user user of the library, no doubt being under pressure to Get Things Done, is almost certainly going to look at this Using example, get the apparent result they were after (a parsed JSON document), and call it a day. It is unlikely that a brand-new user will, for instance, scroll down to the Further Reading section, find the second last (of ten) listed documents, Security.md , and carefully peruse it. If they do, they ll find an oblique suggestion that parsing untrusted input is never a good idea . While that s true, it s also rather unhelpful, because I d wager that by far the majority of JSON parsed in the world is untrusted , in one way or another, given the predominance of JSON as a format for serializing data passing over the Internet. This guidance is roughly akin to putting a label on a car s airbags that driving at speed can be hazardous to your health : true, but unhelpful under the circumstances. The solution is for default behaviours to be secure, and any deviation from that default that has the potential to degrade security must, at the very least, be clearly labelled as such. For example, the Oj.load function should be named Oj.unsafe_load, and the Oj.load function should behave as the Oj.safe_load function does presently. By naming the unsafe function as explicitly unsafe, developers (and reviewers) have at least a fighting chance of recognising they re doing something risky. We put warning labels on just about everything in the real world; the same should be true of dangerous function calls. OK, rant over. Back to the story.

But how is this exploitable? So far, I ve hopefully made it clear that Oj does some Weird Stuff with parsing certain JSON strings. It caused an unhandled exception in a web application I run, which isn t cool, but apart from bombing me with exception notifications, what s the harm? For starters, let s look at our original example: when presented with a symbol, Sequel will interpret that as a column name, rather than a string value. Thus, if our save an update to the user code looked like this:

# request_body has the JSON representation of the form being submitted
body = Oj.load(request_body)
DB[:users].where(id: user_id).update(name: body["name"])

In normal operation, this will issue an SQL query along the lines of UPDATE users SET name='Jaime' WHERE id=42. If the name given is Jaime O Dowd , all is still good, because Sequel quotes string values, etc etc. All s well so far. But, imagine there is a column in the users table that normally users cannot read, perhaps admin_notes. Or perhaps an attacker has gotten temporary access to an account, and wants to dump the user s password hash for offline cracking. So, they send an update claiming that their name is :admin_notes (or :password_hash). In JSON, that ll look like "name":":admin_notes" , and Oj.load will happily turn that into a Ruby object of "name"=>:admin_notes . When run through the above update the user code fragment, it ll produce the SQL UPDATE users SET name=admin_notes WHERE id=42. In other words, it ll copy the contents of the admin_notes column into the name column which the attacker can then read out just by refreshing their profile page.

But Wait, There s More! That an attacker can read other fields in the same table isn t great, but that s barely scratching the surface. Remember before I said that Oj does object serialization ? That means that, in general, you can create arbitrary Ruby objects from JSON. Since objects contain code, it s entirely possible to trigger arbitrary code execution by instantiating an appropriate Ruby object. I m not going to go into details about how to do this, because it s not really my area of expertise, and many others have covered it in detail. But rest assured, if an attacker can feed input of their choosing into a default call to Oj.load, they ve been handed remote code execution on a platter.

Mitigations As Oj s object deserialization is intended and documented behaviour, don t expect a future release to make any of this any safer. Instead, we need to mitigate the risks. Here are my recommended steps:

1. Look in your Gemfile.lock (or SBOM, if that s your thing) to see if the oj gem is anywhere in your codebase. Remember that even if you don t use it directly, it s popular enough that it is used in a lot of places. If you find it in your transitive dependency tree anywhere, there s a chance you re vulnerable, limited only by the ingenuity of attackers to feed crafted JSON into a deeply-hidden Oj.load call.
2. If you depend on oj directly and use it in your project, consider not doing that. The json gem is acceptably fast, and JSON.parse won t create arbitrary Ruby objects.
3. If you really, really need to squeeze the last erg of performance out of your JSON parsing, and decide to use oj to do so, find all calls to Oj.load in your code and switch them to call Oj.safe_load.
4. It is a really, really bad idea to ever use Oj to deserialize JSON into objects, as it lacks the safety features needed to mitigate the worst of the risks of doing so (for example, restricting which classes can be instantiated, as is provided by the permitted_classes argument to Psych.load). I d make it a priority to move away from using Oj for that, and switch to something somewhat safer (such as the aforementioned Psych). At the very least, audit and comment heavily to minimise the risk of user-provided input sneaking into those calls somehow, and pass mode: :object as the second argument to Oj.load, to make it explicit that you are opting-in to this far more dangerous behaviour only when it s absolutely necessary.
5. To secure any unsafe uses of Oj.load in your dependencies, consider setting the default Oj parsing mode to :strict, by putting Oj.default_options = mode: :strict somewhere in your initialization code (and make sure no dependencies are setting it to something else later!). There is a small chance that this change of default might break something, if a dependency is using Oj to deliberately create Ruby objects from JSON, but the overwhelming likelihood is that Oj s just being used to parse ordinary JSON, and these calls are just RCE vulnerabilities waiting to give you a bad time.

Is Your Bacon Saved? If I ve helped you identify and fix potential RCE vulnerabilities in your software, or even just opened your eyes to the risks of object deserialization, please help me out by buying me a refreshing beverage. I would really appreciate any support you can give. Alternately, if you d like my help in fixing these (and many other) sorts of problems, I m looking for work, so email me.

This post is based on presentation given at the Validos annual members meeting on June 25th, 2025.

Maintain your SBOMs For every piece of software, regardless of whether the code was done in-house, from an open source project, or a combination of these, every company needs to produce a Software Bill of Materials (SBOM). The SBOMs provide a standardized and interoperable way to track what software and which versions are used where, what software licenses apply, who holds the copyright of which component, which security fixes have been applied and so forth. A catalog of SBOMs, or equivalent, forms the backbone of software supply-chain management in corporations.

Identify your strategic upstream vendors The SBOMs are likely to reveal that for any piece of non-trivial software, there are hundreds or thousands of upstream open source projects in use. Few organizations have resources to contribute to all of their upstreams. If your organization is just starting to organize upstream contribution activities, identify the key projects that have the largest impact on your business and prioritize forming a relationship with them first. Organizations with a mature contribution process will be collaborating with tens or hundreds of upstreams.

Create a written policy with input from business owners, legal and marketing An upstream contribution policy typically covers things such as who decides what can be contributed upstream from a business point of view, what licenses are allowed or to avoid, how to document copyright, how to deal with projects that require signing copyright assignments (e.g. contributor license agreements), other potential legal guidelines to follow. Additionally, the technical steps on how to prepare a contribution should be outlined, including how to internally review and re-review them, who the technical approvers are to ensure high quality and good reputation and so on. The policy does not have to be static or difficult to produce. Start with a small policy and a few trusted senior developers following it, and update its contents as you run into new situations that need internal company alignment. For example, don t require staff to create new GitHub accounts merely for the purpose of doing one open source contribution. Initially, do things with minimal overhead and add requirements to the policy only if they have clear and strong benefits. The purpose of a policy should be to make it obvious and easy for employees to do the right thing, not to add obstacles and stop progress or encourage people to break the policy.

Appoint an internal coordinator and champions Having a written policy on how to contribute upstream will help ensure a consistent process and avoid common pitfalls. However, a written policy alone does not automatically translate into a well-running process. It is highly recommended to appoint at least one internal coordinator who is knowledgeable about how open source communities work, how software licensing and patents work, and is senior enough to have a good sense of what business priorities to optimize for. In small organizations it can be a single person, while larger organizations typically have a full Open Source Programs Office. This coordinator should oversee the contribution process, track all contributions made across the organization, and further optimize the process by working with stakeholders across the business, including legal experts, business owners and CTOs. The marketing and recruiting folks should also be involved, as upstream contributions will have a reputation-building aspect as well, which can be enhanced with systematic tracking and publishing of activities. Additionally, at least in the beginning, the organization should also appoint key staff members as open source champions. Implementing a new process always includes some obstacles and occasional setbacks, which may discourage employees from putting in the extra effort to reap the full long-term benefits for the company. Having named champions will empower them to make the first few contributions themselves, setting a good example and encouraging and mentoring others to contribute upstream as well.

Avoid excessive approvals To maintain a high quality bar, it is always good to have all outgoing submissions reviewed by at least one or two people. Two or three pairs of eyeballs are significantly more likely to catch issues that might slip by someone working alone. The review also slows down the process by a day or two, which gives the author time to sleep on it , which usually helps to ensure the final submission is well-thought-out by the author. Do not require more than one or two reviewers. The marginal utility goes quickly to zero beyond a few reviewers, and at around four or five people the effect becomes negative, as the weight of each approval decreases and the reviewers begin to take less personal responsibility. Having too many people in the loop also makes each feedback round slow and expensive, to the extent that the author will hesitate to make updates and ask for re-reviews due to the costs involved. If the organization experiences setbacks due to mistakes slipping through the review process, do not respond by adding more reviewers, as it will just grind the contribution process to a halt. If there are quality concerns, invest in training for engineers, CI systems and perhaps an internal certification program for those making public upstream code submissions. A typical software engineer is more likely to seriously try to become proficient at their job and put effort into a one-off certification exam and then make multiple high-quality contributions, than it is for a low-skilled engineer to improve and even want to continue doing more upstream contributions if they are burdened by heavy review processes every time they try to submit an upstream contribution.

Don t expect upstream to accept all code contributions Sure, identifying the root cause of and fixing a tricky bug or writing a new feature requires significant effort. While an open source project will certainly appreciate the effort invested, it doesn t mean it will always welcome all contributions with open arms. Occasionally, the project won t agree that the code is correct or the feature is useful, and some contributions are bound to be rejected. You can minimize the chance of experiencing rejections by having a solid internal review process that includes assessing how the upstream community is likely to understand the proposal. Sometimes how things are communicated is more important than how they are coded. Polishing inline comments and git commit messages help ensure high-quality communication, along with a commitment to respond quickly to review feedback and conducting regular follow-ups until a contribution is finalized and accepted.

Start small to grow expertise and reputation In addition to keeping the open source contribution policy lean and nimble, it is also good to start practical contributions with small issues. Don t aim to contribute massive features until you have a track record of being able to make multiple small contributions. Keep in mind that not all open source projects are equal. Each has its own culture, written and unwritten rules, development process, documented requirements (which may be outdated) and more. Starting with a tiny contribution, even just a typo fix, is a good way to validate how code submissions, reviews and approvals work in a particular project. Once you have staff who have successfully landed smaller contributions, you can start planning larger proposals. The exact same proposal might be unsuccessful when proposed by a new person, and successful when proposed by a person who already has a reputation for prior high-quality work.

Embrace all and any publicity you get Some companies have concerns about their employees working in the open. Indeed, every email and code patch an employee submits, and all related discussions become public. This may initially sound scary, but is actually a potential source of good publicity. Employees need to be trained on how to conduct themselves publicly, and the discussions about code should contain only information strictly related to the code, without any references to actual production environments or other sensitive information. In the long run most employees contributing have a positive impact and the company should reap the benefits of positive publicity. If there are quality issues or employee judgment issues, hiding the activity or forcing employees to contribute with pseudonyms is not a proper solution. Instead, the problems should be addressed at the root, and bad behavior addressed rather than tolerated. When people are working publicly, there tends to also be some degree of additional pride involved, which motivates people to try their best. Contributions need to be public for the sponsoring corporation to later be able to claim copyright or licenses. Considering that thousands of companies participate in open source every day, the prevalence of bad publicity is quite low, and the benefits far exceed the risks.

Scratch your own itch When choosing what to contribute, select things that benefit your own company. This is not purely about being selfish - often people working on resolving a problem they suffer from are the same people with the best expertise of what the problem is and what kind of solution is optimal. Also, the issues that are most pressing to your company are more likely to be universally useful to solve than any random bug or feature request in the upstream project s issue tracker.

Remember there are many ways to help upstream While submitting code is often considered the primary way to contribute, please keep in mind there are also other highly impactful ways to contribute. Submitting high-quality bug reports will help developers quickly identify and prioritize issues to fix. Providing good research, benchmarks, statistics or feedback helps guide development and the project make better design decisions. Documentation, translations, organizing events and providing marketing support can help increase adoption and strengthen long-term viability for the project. In some of the largest open source projects there are already far more pending contributions than the core maintainers can process. Therefore, developers who contribute code should also get into the habit of contributing reviews. As Linus law states, given enough eyeballs, all bugs are shallow. Reviewing other contributors submissions will help improve quality, and also alleviate the pressure on core maintainers who are the only ones providing feedback. Reviewing code submitted by others is also a great learning opportunity for the reviewer. The reviewer does not need to be better than the submitter - any feedback is useful; merely posting review feedback is not the same thing as making an approval decision. Many projects are also happy to accept monetary support and sponsorships. Some offer specific perks in return. By human nature, the largest sponsors always get their voice heard in important decisions, as no open source project wants to take actions that scare away major financial contributors.

Starting is the hardest part Long-term success in open source comes from a positive feedback loop of an ever-increasing number of users and collaborators. As seen in the examples of countless corporations contributing open source, the benefits are concrete, and the process usually runs well after the initial ramp-up and organizational learning phase has passed. In open source ecosystems, contributing upstream should be as natural as paying vendors in any business. If you are using open source and not contributing at all, you likely have latent business risks without realizing it. You don t want to wake up one morning to learn that your top talent left because they were forbidden from participating in open source for the company s benefit, or that you were fined due to CRA violations and mismanagement in sharing security fixes with the correct parties. The faster you start with the process, the less likely those risks will materialize.