Dragon Pearl is a middle-grade space fantasy based on Korean
mythology and the first book of a series.
Min is a fourteen-year-old girl living on the barely-terraformed world of
Jinju with her extended family. Her older brother Jun passed the entrance
exams for the Academy and left to join the Thousand Worlds Space Forces,
and Min is counting the years until she can do the same. Those plans are
thrown into turmoil when an official investigator appears at their door
claiming that Jun deserted to search for the Dragon Pearl. A series of
impulsive fourteen-year-old decisions lead to Min heading for a spaceport
alone, determined to find her brother and prove his innocence.
This would be a rather improbable quest for a young girl, but Min is a
gumiho, one of the supernaturals who live in the Thousand Worlds alongside
non-magical humans. Unlike the more respectable dragons, tigers, goblins,
and shamans, gumiho are viewed with suspicion and distrust because their
powers are useful for deception. They are natural shapeshifters who can
copy the shapes of others, and their Charm ability lets them influence
people's thoughts and create temporary illusions of objects such as ID
cards. It will take all of Min's powers, and some rather lucky
coincidences, to infiltrate the Space Forces and determine what happened
to her brother.
It's common for reviews of this book to open with a caution that this is a
middle-grade adventure novel and you should not expect a story like
Ninefox Gambit. I will be boring and
repeat that caution. Dragon Pearl has a single first-person
viewpoint and a very linear and straightforward plot. Adult readers are
unlikely to be surprised by plot twists; the fun is the world-building and
seeing how Min manages to work around plot obstacles.
The world-building is enjoyable but not very rigorous. Min uses and abuses
Charm with the creative intensity of a Dungeons & Dragons
min-maxer. Each individual event makes sense given the implication that
Min is unusually powerful, but I'm dubious about the surrounding society
and lack of protections against Charm given what Min is able to do. Min
does say that gumiho are rare and many people think they're extinct, which
is a bit of a fig leaf, but you'll need to bring your urban fantasy
suspension of disbelief skills to this one.
I did like that the world-building conceit went more than skin deep and
influenced every part of the world. There are ghosts who are critical to
the plot. Terraforming is done through magic, hence the quest for the
Dragon Pearl and the miserable state of Min's home planet due to its loss.
Medical treatment involves the body's meridians, as does engineering: The
starships have meridians similar to those of humans, and engineers partly
merge with those meridians to adjust them. This is not the sort of book
that tries to build rigorous scientific theories or explain them to the
reader, and I'm not sure everything would hang together if you poked at it
too hard, but Min isn't interested in doing that poking and the story
doesn't try to justify itself. It's mostly a vibe, but it's a vibe that I
enjoyed and that is rather different than other space fantasy I've read.
The characters were okay but never quite clicked for me, in part because
proper character exploration would have required Min take a detour from
her quest to find her brother and that was not going to happen. The reader
gets occasional glimpses of a military SF cadet story and a friendship on
false premises story, but neither have time to breathe because Min drops
any entanglement that gets in the way of her quest. She's almost amoral in
a way that I found believable but not quite aligned with my reading mood.
I also felt a bit wrong-footed by how her friendships developed; saying
too much more would be a spoiler, but I was expecting more human
connection than I got.
I think my primary disappointment with this book was something I knew
going in, not in any way its fault, and part of the reason why I'd put off
reading it: This is pitched at young teenagers and didn't have quite
enough plot and characterization complexity to satisfy me. It's a linear,
somewhat episodic adventure story with some neat world-building, and it
therefore glides over the spots where an adult novel would have added
political and factional complexity. That is exactly as advertised, so it's
up to you whether that's the book you're in the mood for.
One warning: The text of this book opens with an introduction by Rick
Riordan that is just fluff marketing and that spoils the first few
chapters of the book. It is unmarked as such at the beginning and tricked
me into thinking it was the start of the book proper, and then deeply
annoyed me. If you do read this book, I recommend skipping the utterly
pointless introduction and going straight to chapter one.
Followed by Tiger Honor.
Rating: 6 out of 10
2026 already! The winter weather here has really been beautiful and I always enjoy
this time of year. Writing this yearly musical retrospective has now become a
beloved tradition of mine1 and I enjoy retracing the year's various
events through albums I listened to and concerts I went to.
Albums
In 2025, I added 141 new albums to my collection, around 60% more than last
year's haul. I think this might have been too much? I feel like I didn't have
time to properly enjoy all of them and as such, I decided to slow down my
acquisition spree sometimes in early December, around the time I normally do the
complete opposite.
This year again, I bought the vast majority of my music on Bandcamp. Most of
the other albums I bought as CDs and ripped them.
Concerts
In 2025, I went to the following 25 (!!) concerts:
January 17th: Uzu, Young Blades, She came to quit, Fever Visions
February 1st: Over the Hill, Jail, Mortier, Ain't Right
February 7th: B ton Arm , Mulchulation II, Ooz
February 15th: The Prowlers, Ultra Razzia, Sistema de Muerte,
Trauma Bond
February 28th: Francb tards
March 28th: Conflit Majeur, to Even Exist, Crachat
April 12th: Jetsam, Mortier, NIIVI, Canette
April 26th-27th (Montreal Oi! Fest 2025): The
Buzzers, Bad Terms, Sons of Pride, Liberty and Justice, Flafoot 56, The
Beltones, Mortier, Street Code, The Stress, Alternate Action
May 1st: Bauxite, Atomic threat, the 351's
May 30th: Uzu, Tenaz, Extra a Humana, Sistema de muerte
June 7th: Ordures Ioniques, Tulaviok, Fucking Raymonds, Voyou
June 18th: Tiken Jah Fakoly
June 21st: Sa an Supa Celebration
June 26th: Taxi Girls, Death Proof, Laura Krieg
July 4th: Frente Cumbiero
July 12th: Montreal's Big Fiesta DJ Set
August 16th: Guerilla Poubelle
September 11th: No Suicide Act, Mortier
September 20th: Hors Contr le, Union Thugs, Barricade Mentale
October 20th: Ezra Furman, The Golden Dregs
October 24th: Overbass, Hommage B rurier Noir, Self Control,
Vermin Kaos
November 6th: B ton Arm , Faze, Slash Need, Chain Block
November 28th (Blood Moon Ritual 2025): Bhatt, Channeler,
Pyrocene Death Cult, Masse d'Armes
December 13th (Stomp Records' 30th Anniversary Bash):
The Planet Smashers, The Flatliners, Wine Lips, The Anti-Queens, Crash ton
rock
Although I haven't touched metalfinder's code in a good while, my instance
still works very well and I get the occasional match when a big-name artist in
my collection comes in town. Most the venues that advertise on Bandsintown are
tied to Ticketmaster though, which means most underground artists (i.e. most of
the music I listen to) end up playing elsewhere.
As such, shout out again to the Gancio project and to the folks running the
Montreal instance. It continues to be a smash hit and most of
the interesting concerts end up being advertised there.
See you all in 2026!
Same as last year, this is a summary of what I ve been up to throughout the year.
See also the recap/retrospection published by my friends (antiz, jvoisin, orhun).
Uploaded 467 packages to Arch Linux
Most of them being reproducible, meaning I provably didn t abuse my position of compiling the binaries
35 of them are signal-desktop
29 of them are metasploit
Made 53 uploads to Debian
All of them being related to my work in the debian-rust team, that I ve been a part of since 2018
Also applied for Debian Developer status (with 4 Debian Developers advocating for me)
Made 14 commits in Alpine Linux aports
13 of them being package releases
Made 2 commits in NixOS nixpkgs
Also joined their Github org
Made 4 commits in homebrew-core
With special focus on polishing the Rust development experience for the RP2040 microcontroller
Lost Onion, my cat of 13 years, to inoperable cancer. He has been with me throughout my entire open source journey (sometimes being credited as co-author) and who looked after me for my entire adult life. You won t be forgotten.
Developed 6 hand-held games with embedded Rust, most of them being birthday gifts for people close to me
embedded-graphics-colorcast a library I developed so I can keep using embedded-mono-img on ST7789/ILI9486 screens - I used tinybmp in one project but it was fairly slow
ch32v003-demo to demo and document the lowend ch32v003 RISC-V microcontroller, with devboards that are commonly sold for 0.50-0.70 on AliExpress (it s cute but lacks the required 5.1k resistors on the USB-C configuration pins that tell the host to provide 5V, so it won t work with many USB-C chargers, which is quite annoying)
embedded-savegame an atomic/transactional savegame library, with powerfail-safety and wear-leveling, optimized for flash and EEPROM storage
djb2 a very lightweight non-cryptographic checksum algorithm that replaced my use of CRC32 in the embedded-savegame library, to make it more suitable for the ch32v003
We re happy to announce that
Debusine can now be used
to maintain APT-compatible add-on package repositories for Debian. This
facility is available in public beta to Debian developers and maintainers.
Why?
Debian developers typically put most of their effort towards maintaining the
main Debian archive. However, it s often useful to have other places to
work, for various reasons:
Developers working on a set of packages might need to check that changes
to several of them all work properly together on a real system.
Somebody fixing a bug might need to ask affected users to test the fix
before uploading it to Debian.
Some projects are difficult to package in a way that meets Debian policy,
or are too niche to include in Debian, but it s still useful to distribute
them in a packaged form.
For some packages, it s useful to provide multiple upstream versions for
multiple Debian releases, even though Debian itself would normally want to
keep that to a minimum.
The Ubuntu ecosystem has had PPAs for a long time to meet these sorts of needs, but
people working directly on Debian have had to make do with putting things
together themselves using something like
reprepro or
aptly. Discussions about this have been
happening for long enough that people started referring to PPAs for Debian
as bikesheds , and users often find themselves trying to use Ubuntu PPAs on
Debian systems and hoping that dependencies will be compatible enough for
things to more or less work. This clearly isn t ideal, and solving it is
one of Freexian s objectives for Debusine.
Developers publishing packages to Debusine repositories can take advantage
of all Debusine s existing facilities, including a battery of QA tests and
regression tracking (coming soon). Repositories are signed using
per-repository keys held in Debusine s signing service, and uploads to
repositories are built against the current contents of that repository as
well as the corresponding base Debian release. All repositories include
automatic built-in snapshot capabilities.
Who can use this service?
We ve set up debusine.debian.net to allow
using repositories. All Debian
Developers and Debian
Maintainers can log in there and
publish packages to it. The resulting repositories are public by default.
debusine.debian.net only allows packages with licences that allow
distribution by Debian, and it is intended primarily for work that could
reasonably end up in Debian; Freexian reserves the right to remove
repositories from it.
How can I use it?
If you are a Debian contributor, we d be very excited to have you try this
out, especially if you give us
feedback.
We have published instructions for
developers on using
this. Since this is a beta service, you can expect things to change, but
we ll maintain compatibility where we can.
If you re interested in using this in a commercial setting, please contact
Freexian to discuss what we can do for you.
Posted on November 17, 2025
Tags: madeof:atoms, craft:sewing
After cartridge pleating and honeycombing, I was still somewhat in the
mood for that kind of fabric manipulation, and directing my internet
searches in that vague direction, and I stumbled on this:
https://katafalk.wordpress.com/2012/06/26/patternmaking-for-the-kampfrau-hemd-chemise/
Now, do I want to ever make myself a 16th century German costume,
especially a kampfrau one? No! I m from lake Como! Those are the
enemies who come down the Alps pillaging and bringing the Black Death
with them!
Although I have to admit that at times during my day job I have found the
idea of leaving everything to go march with the J germonsters
attractive. You know, the exciting prospective of long days of march
spent knitting sturdy socks, punctuated by the excitement of settling
down in camp and having a chance of doing lots of laundry. Or something.
Sometimes being a programmer will make you think odd things.
Anyway, going back to the topic, no, I didn t need an historically
accurate hemd. But I did need a couple more shirts for daily wear, I did
want to try my hand at smocking, and this looked nice, and I was
intrigued by the way the shaping of the neck and shoulder worked, and
wondered how comfortable it would be.
And so, it had to be done.
I didn t have any suitable linen, but I did have quite a bit of cotton
voile, and since I wasn t aiming at historical accuracy it looked like a
good option for something where a lot of fabric had to go in a small
space.
At first I considered making it with a bit less fabric than the one in
the blog, but then the voile was quite thin, so I kept the original
measurement as is, only adapting the sleeve / sides seams to my size.
With the pieces being rectangles the width of the fabric, I was able to
have at least one side of selvedge on all seams, and took advantage of it
by finishing the seams by simply folding the allowances to one sides so
that the selvedge was on top, and hemstitching them down as I would have
done with a folded edge when felling.
Also, at first I wanted to make the smocking in white on white, but then
I thought about a few hanks of electric blue floss I had in my stash,
and decided to just go with it.
The initial seams were quickly made, then I started the smocking at the
neck, and at that time the project went on hold while I got ready to go
to DebConf. Then I came back and took some time to get back into a
sewing mood, but finally the smocking on the next was finished, and I
could go on with the main sewing, which, as I expected, went decently
fast for a handsewing project.
While doing the diagonal smocking on the collar I counted the stitches
to make each side the same length, which didn t completely work because
the gathers weren t that regular to start with, and started each line
from the two front opening going towards the center back, leaving a
triangle with a different size right in the middle. I think overall it
worked well enough.
Then there were a few more interruptions, but at last it was ready! just
as the weather turned cold-ish and puffy shirts were no longer in
season, but it will be there for me next spring.
I did manage to wear it a few times and I have to say that the neck
shaping is quite comfortable indeed: it doesn t pull in odd ways like
the classical historically accurate pirate shirt sometimes does, and the
heavy gathering at the neck makes it feel padded and soft.
I m not as happy with the cuffs: the way I did them with just
honeycombing means that they don t need a closure, and after washing and
a bit of steaming they lie nicely, but then they tend to relax in a
wider shape. The next time I think I ll leave a slit in the sleeves,
possibly make a different type of smocking (depending on whether I have
enough fabric) and then line them like the neck so that they are stable.
Because, yes, I think that there will be another time: I have a few more
project before that, and I want to spend maybe another year working from
my stash, but then I think I ll buy some soft linen and make at least
another one, maybe with white-on-white smocking so that it will be
easier to match with different garments.
The Raven Scholar is an epic fantasy and the first book of a
projected trilogy. It is Antonia Hodgson's first published fantasy novel;
her previous published novels are historical mystery. I would classify
this as adult fantasy the main character is thirty-four with a stable
court position but it has strong YA vibes because of the generational
turnover feel of the main plot.
Eight years before the start of this book, Andren Valit attempted to
assassinate the emperor and failed. Since then, his widow and three
children twins Yana and Ruko and infant Nisthala have been living in
disgrace in a cramped apartment, subject to constant inspections and
suspicion. As the story opens, they have been summoned to appear before
the emperor, escorted by a young and earnest Hound (essentially the state
security services) named Shal Worthy. The resulting interrogation is full
of dangerous traps. Not all of them will be avoided.
The formalization of the consequences of that imperial summons falls to an
unpopular Junior Archivist (Third Class) whose one notable skill is her
penmanship. A meeting that was disasterous for the Valits becomes
unexpectedly fortunate for the archivist, albeit with a poisonous core.
Eight years later, Neema Kraa is High Scholar, and Emperor Bersun's
twenty-four years of permitted reign is coming to an end. The Festival is
about to begin. One representative from each of the empire's eight anats
(religious schools) will compete in seven days of Trials, save for the
Dragons who do not want the throne and will send a proxy. The victor
according to the Trials scoring system will become emperor and reign
unquestioned for twenty-four years or until resignation. This is the
system that put an end to the era of chaos and has been in place for over
a thousand years.
On the eve of the Trials, the Raven contender is found murdered. Neema is
immediately a suspect; she even has reasons to suspect herself. She
volunteers to lead the investigation because she has to know what
happened. She is also volunteered to be the replacement Raven contender.
There is no chance that she will become emperor; she doesn't even know how
to fight. But agnostic Neema has a rather unexpected ally.
As the last chime fades we drop neatly on to the balcony's rusting
hand rail, folding our wings with a soft shuffle. Noon, on the ninth
day of the eighth month, 1531. Neema Kraa's lodgings. We are here,
exactly where we should be, at exactly the right moment, because we
are the Raven, and we are magnificent.
The Raven Scholar is a rather good epic fantasy, with some caveats
that I'll get to in a moment, but I found it even more fascinating as a
genre artifact.
I've read my share of epic fantasy over the years, although most of my
familiarity of the current wave of new adult fairy epics comes from
reviews rather than personal experience. The Raven Scholar is epic
fantasy, through and through. There is court intrigue, a main character
who is a court functionary unexpectedly thrown into the middle of some
problem, civilization-wide stakes, dramatic political alliances, detailed
magic and mythological systems, and gods. There were moments that reminded
me of a Guy Gavriel Kay novel, although Hodgson's characters tend more
towards disarming moments of humanization instead of Kay's operatic scenes
of emotional intensity.
But The Raven Scholar is also a murder mystery, complete with a
crime scene, clues, suspects, evidence, an investigation, a possibly
compromised detective, and a morass of possible motives and red herrings.
I'm not much of a mystery reader, but this didn't feel like sort of
ancillary mystery that might crop up in the course of a typical epic
fantasy. It felt like a full-fledged investigation with an amateur
detective; one can tell that Hodgson's previous four books were historical
mysteries.
And then there's the Trials, which are the centerpiece of the book.
This book helped me notice that people (okay, me, I'm the people) have
been sleeping on the influence of The
Hunger Games, Battle Royale, and reality TV (specifically
Survivor) on genre fiction, possibly because the more obvious riffs
on the idea (Powerless, The Selection) have been young adult
or new adult. Once I started looking, I realized this idea is everywhere
now: Throne of Glass, Fourth Wing, even The Night
Circus to some extent. Competitions with consequences are having a
moment.
I suspect having a competition to decide the next emperor is going to
strike some traditional fantasy readers as sufficiently absurd and
unbelievable that it will kick them out of the book. I had a moment of
"okay, this is weird, why would anyone stick with this system for so long"
myself. But I would encourage such readers to interrogate whether that's
only a response from unfamiliarity; after all, strange women lying in
ponds distributing swords is no basis for a system of government either.
This is hardly the most unrealistic epic fantasy trope, and it has the
advantage of being a hell of a plot generator when handled well.
Hodgson handles it well. Society in this novel is structured around the
anats and the eight Guardians, gods who, according to myth, had returned
seven times previously to save the world, but who will destroy the world
when they return again. Each Guardian represents a group of
characteristics and useful societal functions: the Ox is trustworthy,
competent and hard-working; the Fox is a trickster and a rule-bender; the
Raven is shrewd and careful and is the Guardian of scholars and lawyers.
Each Trial is organized by one of the anats and tests the contenders for
the skills most valued by that Guardian, often in subtle and rather
ingenious ways. There are flaws here that you could poke at if you wanted
to, but I was charmed and thoroughly entertained by how well Hodgson
weaves the story around the Trials and uses the conflicting values to
create character conflict, unexpected alliances, and engrossing plot.
Most importantly for a book of this sort, I liked Neema. She has a
charming combination of competence, quirks (she is almost physically
unable to not correct people's factual errors), insecurity, imposter
syndrome, and determination. She is way out of her depth and knows it, but
she has an ethical core and an insatiable curiosity that won't let her
leave the central mysteries of the book alone. And the character dynamics
are great; there are a lot of characters, including the competition
problem of having to juggle eight contenders and give them all sufficient
characterization to be meaningful, but this book uses its length to give
each character some room to breathe. This is a long book, well over 600
pages, but it felt packed with events and plot twists. After every chapter
I had to fight the urge to read just one more.
The biggest drawback of this book is that it is very much the first book
of a trilogy, none of the other volumes are out yet, and the ending is
rather nasty. This is the sort of trilogy that opens with a whole lot of
bad things happening, and while I am thoroughly hooked and will purchase
the next volume as soon as it's available, I wish Hodgson had found a way
to end the book on a somewhat more positive or hopeful note. The middle of
the book was great; the end was a bit of an emotional slog, alas. The
writing is good enough here that I'm fairly sure the depression will be
worth it, but if you need your endings to be triumphant (and who could
blame you in this moment in history), you may want to wait on this one
until more volumes are out.
Apart from that, though, this was a lot of fun. The Guardians felt like
they came from a different strand of fantasy than you usually see in epic,
more of a traditional folk tale vibe, which adds an intriguing twist to
the epic fantasy setting. The characters all work, and Hodgson even pulls
off some Game of Thrones style twists that make you sympathetic to
characters you previously hated. The magic system apart from the Guardians
felt underbaked, but the politics had more depth than a lot of fantasy
novels. If you want the truly complex and twisty politics you would get
from one of Guy Gavriel Kay's historical rewrites, you will come away
disappointed, but it was good enough for me. And I did enjoy the Raven.
Respect, that's all we demand. Recognition of our magnificence.
Offerings. Love. Fear. Trembling awe. Worship. Shiny things. Blood
sacrifice, some of us very much enjoy blood sacrifice. Truly, we ask
for so little.
Followed by an as-yet untitled sequel that I hope will materialize.
Rating: 7 out of 10
Avoiding
5XX errors by adjusting Load Balancer Idle Timeout
Recently I faced a problem in production where a client was running a
RabbitMQ server behind the Load Balancers we provisioned and the TCP
connections were closed every minute.
My team is responsible for the LBaaS (Load Balancer as a Service)
product and this Load Balancer was an Envoy proxy provisioned by our
control plane.
The error was similar to this:
At first glance, the issue is simple: the Load Balancer's idle
timeout is shorter than the RabbitMQ heartbeat interval.
The idle timeout is the time at which a downstream or upstream
connection will be terminated if there are no active streams. Heartbeats
generate periodic network traffic to prevent idle TCP connections from
closing prematurely.
Adjusting these timeout settings to align properly solved the
issue.
However, what I want to explore in this post are other similar
scenarios where it's not so obvious that the idle timeout is the
problem. Introducing an extra network layer, such as an Envoy proxy, can
introduce unpredictable behavior across your services, like intermittent
5XX errors.
To make this issue more concrete, let's look at a minimal,
reproducible setup that demonstrates how adding an Envoy proxy can lead
to sporadic errors.
Reproducible setup
I'll be using the following tools:
I'll be running experiments with two different
envoy.yaml configurations: one that uses Envoy's TCP proxy,
and another that uses Envoy's HTTP connection manager.
Here's the simplest Envoy TCP proxy setup: a listener on port 8000
forwarding traffic to a backend running on port 8080.
The default idle timeout if not otherwise specified is 1 hour, which
is the case here.
The backend setup is simple as well:
package mainimport("fmt""net/http""time")func helloHandler(w http.ResponseWriter, r *http.Request) w.Write([]byte("Hello from Go!"))func main() http.HandleFunc("/", helloHandler) server := http.Server Addr:":8080", IdleTimeout:3* time.Second, fmt.Println("Starting server on :8080")panic(server.ListenAndServe())
The IdleTimeout is set to 3 seconds to make it easier to test.
Now, oha is the perfect tool to generate the HTTP
requests for this test. The Load test is not meant to stress this setup,
the idea is to wait long enough so that some requests are closed. The
burst-delay feature will help with that:
I'm running the Load test for 30 seconds, sending 100 requests at
three-second intervals. I also use the -w option to wait
for ongoing requests when the duration is reached. The result looks like
this:
We had 886 responses with status code 200 and 64 connections closed.
The backend terminated 64 connections while the load balancer still had
active requests directed to it.
Let's change the Load Balancer idle_timeout to two
seconds.
filter_chains:-filters:-name: envoy.filters.network.tcp_proxytyped_config:"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxystat_prefix: go_server_tcpcluster: go_server_clusteridle_timeout: 2s # <--- NEW LINE
Run the same test again.
Great! Now all the requests worked.
This is a common issue, not specific to Envoy Proxy or the setup
shown earlier. Major cloud providers have all documented it.
AWS
troubleshoot guide for Application Load Balancers says this:
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.Google
troubleshoot guide for Application Load Balancers mention this as
well:
Verify that the keepalive configuration parameter for the HTTP server software running on the backend instance is not less than the keepalive timeout of the load balancer, whose value is fixed at 10 minutes (600 seconds) and is not configurable.The load balancer generates an HTTP 5XX response code when the connection to the backend has unexpectedly closed while sending the HTTP request or before the complete HTTP response has been received. This can happen because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed keepalive timeout of the load balancer. Ensure that the keepalive timeout configuration for HTTP server software on each backend is set to slightly greater than 10 minutes (the recommended value is 620 seconds).RabbitMQ
docs also warn about this:
Certain networking tools (HAproxy, AWS ELB) and equipment (hardware load balancers) may terminate "idle" TCP connections when there is no activity on them for a certain period of time. Most of the time it is not desirable.
Most of them are talking about Application Load Balancers and the
test I did was using a Network Load Balancer. For the sake of
completeness, I will do the same test but using Envoy's HTTP connection
manager.
The updated envoy.yaml:
The yaml above is an example of a service proxying HTTP from
0.0.0.0:8000 to 0.0.0.0:8080. The only difference from a minimal
configuration is that I enabled access logs.
Let's run the same tests with oha.
Even thought the success rate is 100%, the status code distribution
show some responses with status code 503. This is the case where is not
that obvious that the problem is related to idle timeout.
However, it's clear when we look the Envoy access logs:
UC is the short name for
UpstreamConnectionTermination. This means the upstream,
which is the golang server, terminated the connection.
To fix this once again, the Load Balancer idle timeout needs to
change:
clusters:-name: go_server_clustertype: STATICtyped_extension_protocol_options: # <--- NEW BLOCKenvoy.extensions.upstreams.http.v3.HttpProtocolOptions:"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptionscommon_http_protocol_options:idle_timeout: 2s # <--- NEW VALUEexplicit_http_config:http_protocol_options:
Finally, the sporadic 503 errors are over:
To Sum Up
Here's an example of the values my team recommends to our
clients:
Key Takeaways:
The Load Balancer idle timeout should be less than the backend
(upstream) idle/keepalive timeout.
When we are working with long lived connections, the client
(downstream) should use a keepalive smaller than the LB idle
timeout.
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1273 other packages on CRAN, downloaded 41.8 million
times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 651 times according
to Google Scholar.
Armadillo 15 brought changes. We mentioned these in the 15.0.2-1
and 15.0.2-1
release blog posts:
Minimum C++ standard of C++14
No more suppression of deprecation notes
(The second point is a consequence of the first. Prior to C++14,
deprecation notes were issue via a macro, and the macro was set up by Conrad in the common way of
allowing an override, which we took advantage of in RcppArmadillo
effectively shielding downstream package. In C++14 this is now an
attribute, and those cannot be suppressed.)
We tested this then-upcoming change extensively: Thirteen reverse
dependency runs expoloring different settings and leading to the current
package setup where an automatic fallback to the last Armadillo 14
release offers fallback for hardwired C++11 use and Armadillo 15 others.
Given the 1200+ reverse deoendencies, this took considerable time. All
this was also quite extensively discussed with CRAN (especially Kurt
Hornik) and documented / controlled via a series of issue tickets
starting with overall issue #475
covering the subissues:
open issue
#475 describes the version selection between Armadillo 14 and 15 via
#define
open issue
#476 illustrates how package without deprecation notes are already
suitable for Armadillo 15 and C++14
open issue
#477 demonstrates how a package with a simple deprecation note can
be adjusted for Armadillo 15 and C++14
closed issue
#479 documents a small bug we created in the initial transition
package RcppArmadillo 15.0.1-1 and fixed in the 15.0.2-1
closed issue
#481 discusses removal of the check for insufficient LAPACK routines
which has been removed given that R 4.5.0 or later has sufficient code
in its fallback LAPACK (used e.g. on Windows)
open issue
#484 offering help to the (then 226) packages needing help
transitioning from (enforced) C++11
open issue
#485 offering help to the (then 135) packages needing help with
deprecations
open issue
#489 coordinating pull requests and patches to 35 packages for the
C++11 transition
open issue
#491 coordinating pull requests and patches to 25 packages for
deprecation transition
The sixty pull requests (or emailed patches) followed a suggestion by
CRAN to rank-order packages affected by their reverse dependencies
sorted in descending package count. Now, while this change from
Armadillo 14 to 15 was happening, CRAN also tightened the C++11
requirement for packages and imposed a deadline for changes. In
discussion, CRAN also convinced me that a deadline for the deprecation
warning, now unmasked, was viable (and is in fairness commensurate with
similar, earlier changes triggered by changes in the behaviour of either
gcc/g++ or clang/clang++). So we now have two larger deadline campaigns
affecting the package (and as always there are some others).
These deadlines are coming close: October 17 for the C++11
transition, and October 23 for the deprecation warning. Now, as became
clear preparing the sixty pull requests and patches, these changes are
often relatively straightforward. For the former, remove the C++11
enforcement and the package will likely build without changes. For the
latter, make the often simple (e.g. swith from
arma::is_finite to std::isfinite) change. I
did not encounter anything much more complicated yet.
The number of affected packages approximated by looking at all
packages with a reverse dependency on RcppArmadillo and having
a deadline can be computed as
and has been declining steadily from over 350 to now under 200. For
that a big and heartfelt Thank You! to all the
maintainers who already addressed their package and uploaded updated
packages to CRAN. That rocks, and is truly appreciated.
Yet the number is still large. And while issues #489 and
#491
show a number of pending packages that have merged but not uploaded
(yet?) there are also all the other packages I have not been able to
look at in detail. While preparing sixty PRs / patches was viable over a
period of a good week, I cannot create these for all packages. So with
that said, here is a different suggestion for help: All of next week, I
will be holding open door open source office hours online two times
each day (11:00h to 13:00h Central, 16:00h to 18:00h Central) which can
be booked via this booking
link for Monday to Friday next week in either fifteen or thirty
minutes slots you can book. This should offer Google Meet video
conferencing (with jitsi as an
alternate, you should be able to control that) which should allow for
screen sharing. (I cannot hookup Zoom as my default account has
organization settings with a different calendar integration.)
If you are reading this and have a package that still needs helps, I
hope to see you in the Open Source Office Hours to aid in the
RcppArmadillo package updates for your package. Please book a slot!
In December 2024, I went on a trip through four countries - Singapore, Malaysia, Brunei, and Vietnam - with my friend Badri. This post covers our experiences in Singapore.
I took an IndiGo flight from Delhi to Singapore, with a layover in Chennai. At the Chennai airport, I was joined by Badri. We had an early morning flight from Chennai that would land in Singapore in the afternoon. Within 48 hours of our scheduled arrival in Singapore, we submitted an arrival card online. At immigration, we simply needed to scan our passports at the gates, which opened automatically to let us through, and then give our address to an official nearby. The process was quick and smooth, but it unfortunately meant that we didn t get our passports stamped by Singapore.
Before I left the airport, I wanted to visit the nature-themed park with a fountain I saw in pictures online. It is called Jewel Changi, and it took quite some walking to get there. After reaching the park, we saw a fountain that could be seen from all the levels. We roamed around for a couple of hours, then proceeded to the airport metro station to get to our hotel.
A shot of Jewel Changi. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
There were four ATMs on the way to the metro station, but none of them provided us with any cash. This was the first country (outside India, of course!) where my card didn t work at ATMs.
To use the metro, one can tap the EZ-Link card or bank cards at the AFC gates to get in. You cannot buy tickets using cash. Before boarding the metro, I used my credit card to get Badri an EZ-Link card from a vending machine. It was 10 Singapore dollars ( 630) - 5 for the card, and 5 for the balance. I had planned to use my Visa credit card to pay for my own fare. I was relieved to see that my card worked, and I passed through the AFC gates.
We had booked our stay at a hostel named Campbell s Inn, which was the cheapest we could find in Singapore. It was 1500 per night for dorm beds. The hostel was located in Little India. While Little India has an eponymous metro station, the one closest to our hostel was Rochor.
On the way to the hostel, we found out that our booking had been canceled.
We had booked from the Hostelworld website, opting to pay the deposit in advance and to pay the balance amount in person upon reaching. However, Hostelworld still tried to charge Badri s card again before our arrival. When the unauthorized charge failed, they sent an automatic message saying we tried to charge and to contact them soon to avoid cancellation, which we couldn t do as we were in the plane.
Despite this, we went to the hostel to check the status of our booking.
The trip from the airport to Rochor required a couple of transfers. It was 2 Singapore dollars (approx. 130) and took approximately an hour.
Upon reaching the hostel, we were informed that our booking had indeed been canceled, and were not given any reason for the cancelation. Furthermore, no beds were available at the hostel for us to book on the spot.
We decided to roam around and look for accommodation at other hostels in the area. Soon, we found a hostel by the name of Snooze Inn, which had two beds available. It was 36 Singapore dollars per person (around 2300) for a dormitory bed. Snooze Inn advertised supporting RuPay cards and UPI. Some other places in that area did the same. We paid using my card. We checked in and slept for a couple of hours after taking a shower.
By the time we woke up, it was dark. We met Praveen s friend Sabeel to get my FLX1 phone. We also went to Mustafa Center nearby to exchange Indian rupees for Singapore dollars. Mustafa Center also had a shopping center with shops selling electronic items and souvenirs, among other things. When we were dropping off Sabeel at a bus stop, we discovered that the bus stops in Singapore had a digital board mentioning the bus routes for the stop and the number of minutes each bus was going to take.
In addition to an organized bus system, Singapore had good pedestrian infrastructure. There were traffic lights and zebra crossings for pedestrians to cross the roads. Unlike in Indian cities, rules were being followed. Cars would stop for pedestrians at unmanaged zebra crossings; pedestrians would in turn wait for their crossing signal to turn green before attempting to walk across. Therefore, walking in Singapore was easy.
Traffic rules were taken so seriously in Singapore I (as a pedestrian) was afraid of unintentionally breaking them, which could get me in trouble, as breaking rules is dealt with heavy fines in the country. For example, crossing roads without using a marked crossing (while being within 50 meters of it) - also known as jaywalking - is an offence in Singapore.
Moreover, the streets were litter-free, and cleanliness seemed like an obsession.
After exploring Mustafa Center, we went to a nearby 7-Eleven to top up Badri s EZ-Link card. He gave 20 Singapore dollars for the recharge, which credited the card by 19.40 Singapore dollars (0.6 dollars being the recharge fee).
When I was planning this trip, I discovered that the World Chess Championship match was being held in Singapore. I seized the opportunity and bought a ticket in advance. The next day - the 5th of December - I went to watch the 9th game between Gukesh Dommaraju of India and Ding Liren of China. The venue was a hotel on Sentosa Island, and the ticket was 70 Singapore dollars, which was around 4000 at the time.
We checked out from our hostel in the morning, as we were planning to stay with Badri s aunt that night. We had breakfast at a place in Little India. Then we took a couple of buses, followed by a walk to Sentosa Island. Paying the fare for the buses was similar to the metro - I tapped my credit card in the bus, while Badri tapped his EZ-Link card. We also had to tap it while getting off.
If you are tapping your credit card to use public transport in Singapore, keep in mind that the total amount of all the trips taken on a day is deducted at the end. This makes it hard to determine the cost of individual trips. For example, I could take a bus and get off after tapping my card, but I would have no way to determine how much this journey cost.
When you tap in, the maximum fare amount gets deducted. When you tap out, the balance amount gets refunded (if it s a shorter journey than the maximum fare one). So, there is incentive for passengers not to get off without tapping out. Going by your card statement, it looks like all that happens virtually, and only one statement comes in at the end. Maybe this combining only happens for international cards.
We got off the bus a kilometer away from Sentosa Island and walked the rest of the way. We went on the Sentosa Boardwalk, which is itself a tourist attraction. I was using Organic Maps to navigate to the hotel Resorts World Sentosa, but Organic Maps route led us through an amusement park. I tried asking the locals (people working in shops) for directions, but it was a Chinese-speaking region, and they didn t understand English. Fortunately, we managed to find a local who helped us with the directions.
A shot of Sentosa Boardwalk. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Following the directions, we somehow ended up having to walk on a road which did not have pedestrian paths. Singapore is a country with strict laws, so we did not want to walk on that road. Avoiding that road led us to the Michael Hotel. There was a person standing at the entrance, and I asked him for directions to Resorts World Sentosa. The person told me that the bus (which was standing at the entrance) would drop me there! The bus was a free service for getting to Resorts World Sentosa. Here I parted ways with Badri, who went to his aunt s place.
I got to the Resorts Sentosa and showed my ticket to get in. There were two zones inside - the first was a room with a glass wall separating the audience and the players. This was the room to watch the game physically, and resembled a zoo or an aquarium. :) The room was also a silent room, which means talking or making noise was prohibited. Audiences were only allowed to have mobile phones for the first 30 minutes of the game - since I arrived late, I could not bring my phone inside that room.
The other zone was outside this room. It had a big TV on which the game was being broadcast along with commentary by David Howell and Jovanka Houska - the official FIDE commentators for the event. If you don t already know, FIDE is the authoritative international chess body.
I spent most of the time outside that silent room, giving me an opportunity to socialize. A lot of people were from Singapore. I saw there were many Indians there as well. Moreover, I had a good time with Vasudevan, a journalist from Tamil Nadu who was covering the match. He also asked questions to Gukesh during the post-match conference. His questions were in Tamil to lift Gukesh s spirits, as Gukesh is a Tamil speaker.
Tea and coffee were free for the audience. I also bought a T-shirt from their stall as a souvenir.
After the game, I took a shuttle bus from Resorts World Sentosa to a metro station, then travelled to Pasir Ris by metro, where Badri was staying with his aunt. I thought of getting something to eat, but could not find any caf s or restaurants while I was walking from the Pasir Ris metro station to my destination, and was positively starving when I got there.
Badri s aunt s place was an apartment in a gated community. On the gate was a security guard who asked me the address of the apartment. Upon entering, there were many buildings. To enter the building, you need to dial the number of the apartment you want to go to and speak to them. I had seen that in the TV show Seinfeld, where Jerry s friends used to dial Jerry to get into his building.
I was afraid they might not have anything to eat because I told them I was planning to get something on the way. This was fortunately not the case, and I was relieved to not have to sleep with an empty stomach.
Badri s uncle gave us an idea of how safe Singapore is. He said that even if you forget your laptop in a public space, you can go back the next day to find it right there in the same spot. I also learned that owning cars was discouraged in Singapore - the government imposes a high registration fee on them, while also making public transport easy to use and affordable. I also found out that 7-Eleven was not that popular among residents in Singapore, unlike in Malaysia or Thailand.
The next day was our third and final day in Singapore. We had a bus in the evening to Johor Bahru in Malaysia. We got up early, had breakfast, and checked out from Badri s aunt s home. A store by the name of Cat Socrates was our first stop for the day, as Badri wanted to buy some stationery. The plan was to take the metro, followed by the bus. So we got to Pasir Ris metro station. Next to the metro station was a mall. In the mall, Badri found an ATM where our cards worked, and we got some Singapore dollars.
It was noon when we reached the stationery shop mentioned above. We had to walk a kilometer from the place where the bus dropped us. It was a hot, sunny day in Singapore, so walking was not comfortable. We had to go through residential areas in Singapore. We saw some non-touristy parts of Singapore.
After we were done with the stationery shop, we went to a hawker center to get lunch. Hawker centers are unique to Singapore. They have a lot of shops that sell local food at cheap prices. It is similar to a food court. However, unlike the food courts in malls, hawker centers are open-air and can get quite hot.
This is the hawker center we went to. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
To have something, you just need to buy it from one of the shops and find a table. After you are done, you need to put your tray in the tray-collecting spots. I had a kaya toast with chai, since there weren t many vegetarian options. I also bought a persimmon from a nearby fruit vendor. On the other hand, Badri sampled some local non-vegetarian dishes.
Table littering at the hawker center was prohibited by law. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Next, we took a metro to Raffles Place, as we wanted to visit Merlion, the icon of Singapore. It is a statue having the head of a lion and the body of a fish. While getting through the AFC gates, my card was declined. Therefore, I had to buy an EZ-Link card, which I had been avoiding because the card itself costs 5 Singapore dollars.
From the Raffles Place metro station, we walked to Merlion. The place also gave a nice view of Marina Bay Sands. It was filled with tourists clicking pictures, and we also did the same.
Merlion from behind, giving a good view of Marina Bay Sands. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
After this, we went to the bus stop to catch our bus to the border city of Johor Bahru, Malaysia. The bus was more than an hour late, and we worried that we had missed the bus. I asked an Indian woman at the stop who also planned to take the same bus, and she told us that the bus was late. Finally, our bus arrived, and we set off for Johor Bahru.
Before I finish, let me give you an idea of my expenditure. Singapore is an expensive country, and I realized that expenses could go up pretty quickly. Overall, my stay in Singapore for 3 days and 2 nights was approx. 5500 rupees. That too, when we stayed one night at Badri s aunt s place (so we didn t have to pay for accomodation for one of the nights) and didn t have to pay for a couple of meals. This amount doesn t include the ticket for the chess game, but includes the costs of getting there. If you are in Singapore, it is likely you will pay a visit to Sentosa Island anyway.
Stay tuned for our experiences in Malaysia!
Credits: Thanks to Dione, Sahil, Badri and Contrapunctus for reviewing the draft. Thanks to Bhe for spotting a duplicate sentence.
If you're still using Vagrant (I am) and try to boot a box that uses UEFI (like boxen/debian-13),
a simple vagrant init boxen/debian-13 and vagrant up will entertain you with a nice traceback:
% vagrantup
Bringing machine 'default' up with 'libvirt' provider...==> default: Checking if box 'boxen/debian-13' version '2025.08.20.12' is up to date...==> default: Creating image (snapshot of base box volume).==> default: Creating domain with the following settings...==> default: -- Name: tmp.JV8X48n30U_default==> default: -- Description: Source: /tmp/tmp.JV8X48n30U/Vagrantfile==> default: -- Domain type: kvm==> default: -- Cpus: 1==> default: -- Feature: acpi==> default: -- Feature: apic==> default: -- Feature: pae==> default: -- Clock offset: utc==> default: -- Memory: 2048M==> default: -- Loader: /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd==> default: -- Nvram: /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/efivars.fd==> default: -- Base box: boxen/debian-13==> default: -- Storage pool: default==> default: -- Image(vda): /home/evgeni/.local/share/libvirt/images/tmp.JV8X48n30U_default.img, virtio, 20G==> default: -- Disk driver opts: cache='default'==> default: -- Graphics Type: vnc==> default: -- Video Type: cirrus==> default: -- Video VRAM: 16384==> default: -- Video 3D accel: false==> default: -- Keymap: en-us==> default: -- TPM Backend: passthrough==> default: -- INPUT: type=mouse, bus=ps2==> default: -- CHANNEL: type=unix, mode===> default: -- CHANNEL: target_type=virtio, target_name=org.qemu.guest_agent.0==> default: Creating shared folders metadata...==> default: Starting domain.==> default: Removing domain...==> default: Deleting the machine folder/usr/share/gems/gems/fog-libvirt-0.13.1/lib/fog/libvirt/requests/compute/vm_action.rb:7:in 'Libvirt::Domain#create': Call to virDomainCreate failed: internal error: process exited while connecting to monitor: 2025-09-22T10:07:55.081081Z qemu-system-x86_64: -blockdev "driver":"file","filename":"/home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap" : Could not open '/home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd': Permission denied (Libvirt::Error) from /usr/share/gems/gems/fog-libvirt-0.13.1/lib/fog/libvirt/requests/compute/vm_action.rb:7:in 'Fog::Libvirt::Compute::Shared#vm_action' from /usr/share/gems/gems/fog-libvirt-0.13.1/lib/fog/libvirt/models/compute/server.rb:81:in 'Fog::Libvirt::Compute::Server#start' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/start_domain.rb:546:in 'VagrantPlugins::ProviderLibvirt::Action::StartDomain#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/set_boot_order.rb:22:in 'VagrantPlugins::ProviderLibvirt::Action::SetBootOrder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/share_folders.rb:22:in 'VagrantPlugins::ProviderLibvirt::Action::ShareFolders#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/prepare_nfs_settings.rb:21:in 'VagrantPlugins::ProviderLibvirt::Action::PrepareNFSSettings#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/synced_folders.rb:87:in 'Vagrant::Action::Builtin::SyncedFolders#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/delayed.rb:19:in 'Vagrant::Action::Builtin::Delayed#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/synced_folder_cleanup.rb:28:in 'Vagrant::Action::Builtin::SyncedFolderCleanup#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/plugins/synced_folders/nfs/action_cleanup.rb:25:in 'VagrantPlugins::SyncedFolderNFS::ActionCleanup#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/prepare_nfs_valid_ids.rb:14:in 'VagrantPlugins::ProviderLibvirt::Action::PrepareNFSValidIds#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:127:in 'block in Vagrant::Action::Warden#finalize_action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builder.rb:180:in 'Vagrant::Action::Builder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'block in Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/util/busy.rb:19:in 'Vagrant::Util::Busy.busy' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/call.rb:53:in 'Vagrant::Action::Builtin::Call#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:127:in 'block in Vagrant::Action::Warden#finalize_action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builder.rb:180:in 'Vagrant::Action::Builder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'block in Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/util/busy.rb:19:in 'Vagrant::Util::Busy.busy' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/call.rb:53:in 'Vagrant::Action::Builtin::Call#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/create_network_interfaces.rb:197:in 'VagrantPlugins::ProviderLibvirt::Action::CreateNetworkInterfaces#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/create_networks.rb:40:in 'VagrantPlugins::ProviderLibvirt::Action::CreateNetworks#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/create_domain.rb:452:in 'VagrantPlugins::ProviderLibvirt::Action::CreateDomain#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/resolve_disk_settings.rb:143:in 'VagrantPlugins::ProviderLibvirt::Action::ResolveDiskSettings#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/create_domain_volume.rb:97:in 'VagrantPlugins::ProviderLibvirt::Action::CreateDomainVolume#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/handle_box_image.rb:127:in 'VagrantPlugins::ProviderLibvirt::Action::HandleBoxImage#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/handle_box.rb:56:in 'Vagrant::Action::Builtin::HandleBox#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/handle_storage_pool.rb:63:in 'VagrantPlugins::ProviderLibvirt::Action::HandleStoragePool#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/set_name_of_domain.rb:34:in 'VagrantPlugins::ProviderLibvirt::Action::SetNameOfDomain#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/provision.rb:80:in 'Vagrant::Action::Builtin::Provision#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/cleanup_on_failure.rb:21:in 'VagrantPlugins::ProviderLibvirt::Action::CleanupOnFailure#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:127:in 'block in Vagrant::Action::Warden#finalize_action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builder.rb:180:in 'Vagrant::Action::Builder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'block in Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/util/busy.rb:19:in 'Vagrant::Util::Busy.busy' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/call.rb:53:in 'Vagrant::Action::Builtin::Call#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/box_check_outdated.rb:93:in 'Vagrant::Action::Builtin::BoxCheckOutdated#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/config_validate.rb:25:in 'Vagrant::Action::Builtin::ConfigValidate#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builder.rb:180:in 'Vagrant::Action::Builder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'block in Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/util/busy.rb:19:in 'Vagrant::Util::Busy.busy' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/machine.rb:248:in 'Vagrant::Machine#action_raw' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/machine.rb:217:in 'block in Vagrant::Machine#action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/environment.rb:631:in 'Vagrant::Environment#lock' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/machine.rb:203:in 'Method#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/machine.rb:203:in 'Vagrant::Machine#action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/batch_action.rb:86:in 'block (2 levels) in Vagrant::BatchAction#run'
The important part here is
Call to virDomainCreate failed: internal error: process exited while connecting to monitor:
2025-09-22T10:07:55.081081Z qemu-system-x86_64: -blockdev "driver":"file","filename":"/home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap" :
Could not open '/home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd': Permission denied (Libvirt::Error)
Of course we checked that the file permissions on this file are correct (I'll save you the ls output), so what's next?
Yes, of course, SELinux!
A process in the svirt_t domain tries to access something in the user_home_t domain and is denied by the kernel.
So far, SELinux is both working as designed and preventing us from doing our work, nice.
For "normal" (non-UEFI) boxes, Vagrant uploads the image to libvirt, which stores it in ~/.local/share/libvirt/images/ and boots fine from there.
For UEFI boxen, one also needs loader and nvram files, which Vagrant keeps in ~/.vagrant.d/boxes/<box_name> and that's what explodes in our face here.
As ~/.local/share/libvirt/images/ works well, and is labeled svirt_home_t let's see what other folders use that label:
# semanagefcontext-lgrepsvirt_home_t
/home/[^/]+/\.cache/libvirt/qemu(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.config/libvirt/qemu(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.libvirt/qemu(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.local/share/gnome-boxes/images(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.local/share/libvirt/boot(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.local/share/libvirt/images(/.*)? all files unconfined_u:object_r:svirt_home_t:s0
Okay, that all makes sense, and it's just missing the Vagrant-specific folders!
% restorecon-rv~/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13
Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13 from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/metadata_url from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12 from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/box_0.img from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/metadata.json from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/Vagrantfile from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_VARS.fd from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/box_update_check from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/efivars.fd from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0
And it works!
% vagrantup
Bringing machine 'default' up with 'libvirt' provider...==> default: Checking if box 'boxen/debian-13' version '2025.08.20.12' is up to date...==> default: Creating image (snapshot of base box volume).==> default: Creating domain with the following settings...==> default: -- Name: tmp.JV8X48n30U_default==> default: -- Description: Source: /tmp/tmp.JV8X48n30U/Vagrantfile==> default: -- Domain type: kvm==> default: -- Cpus: 1==> default: -- Feature: acpi==> default: -- Feature: apic==> default: -- Feature: pae==> default: -- Clock offset: utc==> default: -- Memory: 2048M==> default: -- Loader: /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd==> default: -- Nvram: /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/efivars.fd==> default: -- Base box: boxen/debian-13==> default: -- Storage pool: default==> default: -- Image(vda): /home/evgeni/.local/share/libvirt/images/tmp.JV8X48n30U_default.img, virtio, 20G==> default: -- Disk driver opts: cache='default'==> default: -- Graphics Type: vnc==> default: -- Video Type: cirrus==> default: -- Video VRAM: 16384==> default: -- Video 3D accel: false==> default: -- Keymap: en-us==> default: -- TPM Backend: passthrough==> default: -- INPUT: type=mouse, bus=ps2==> default: -- CHANNEL: type=unix, mode===> default: -- CHANNEL: target_type=virtio, target_name=org.qemu.guest_agent.0==> default: Creating shared folders metadata...==> default: Starting domain.==> default: Domain launching with graphics connection settings...==> default: -- Graphics Port: 5900==> default: -- Graphics IP: 127.0.0.1==> default: -- Graphics Password: Not defined==> default: -- Graphics Websocket: 5700==> default: Waiting for domain to get an IP address...==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 192.168.124.157:22 default: SSH username: vagrant default: SSH auth method: private key default: default: Vagrant insecure key detected. Vagrant will automatically replace default: this with a newly generated keypair for better security. default: default: Inserting generated public key within guest... default: Removing insecure key from the guest if it's present... default: Key inserted! Disconnecting and reconnecting using new SSH key...==> default: Machine booted and ready!
Akvorado 2.0 was released today! Akvorado collects network flows with
IPFIX and sFlow. It enriches flows and stores them in a ClickHouse database.
Users can browse the data through a web console. This release introduces an
important architectural change and other smaller improvements. Let s dive in!
New outlet service
The major change in Akvorado 2.0 is splitting the inlet service into two
parts: the inlet and the outlet. Previously, the inlet handled all flow
processing: receiving, decoding, and enrichment. Flows were then sent to Kafka
for storage in ClickHouse:
Akvorado flow processing before the introduction of the outlet service
Network flows reach the inlet service using UDP, an unreliable protocol. The
inlet must process them fast enough to avoid losing packets. To handle a
high number of flows, the inlet spawns several sets of workers to receive flows,
fetch metadata, and assemble enriched flows for Kafka. Many configuration
options existed for scaling, which increased complexity for users. The code
needed to avoid blocking at any cost, making the processing pipeline complex
and sometimes unreliable, particularly the BMP receiver.1 Adding new
features became difficult without making the problem worse.2
In Akvorado 2.0, the inlet receives flows and pushes them to Kafka without
decoding them. The new outlet service handles the remaining tasks:
Akvorado flow processing after the introduction of the outlet service
This change goes beyond a simple split:3 the outlet now reads flows from
Kafka and pushes them to ClickHouse, two tasks that Akvorado did not handle
before. Flows are heavily batched to increase efficiency and reduce the load
on ClickHouse using ch-go, a low-level Go client for ClickHouse. When
batches are too small, asynchronous inserts are used (e20645). The number of
outlet workers scales dynamically (e5a625) based on the target batch
size and latency (50,000 flows and 5 seconds by default).
This new architecture also allows us to simplify and optimize the code. The
outlet fetches metadata synchronously (e20645). The BMP component becomes
simpler by removing cooperative multitasking (3b9486). Reusing the same
RawFlow object to decode protobuf-encoded flows from Kafka reduces pressure on
the garbage collector (8b580f).
The effect on Akvorado s overall performance was somewhat uncertain, but a
user reported 35% lower CPU usage after migrating from the previous
version, plus resolution of the long-standing BMP component issue.
Other changes
This new version includes many miscellaneous changes, such as completion for
source and destination ports (f92d2e), and automatic restart of the
orchestrator service (0f72ff) when configuration changes to avoid a common
pitfall for newcomers.
Let s focus on some key areas for this release: observability,
documentation, CI,
Docker, Go, and JavaScript.
Observability
Akvorado exposes metrics to provide visibility into the processing pipeline and
help troubleshoot issues. These are available through Prometheus HTTP metrics
endpoints, such as /api/v0/inlet/metrics. With the introduction
of the outlet, many metrics moved. Some were also renamed (4c0b15) to match
Prometheus best practices. Kafka consumer lag was added as a new metric
(e3a778).
If you do not have your own observability stack, the Docker Compose setup
shipped with Akvorado provides one. You can enable it by activating the profiles
introduced for this purpose (529a8f).
The prometheus profile ships Prometheus to store metrics and Alloy
to collect them (2b3c46, f81299, and 8eb7cd). Redis and Kafka
metrics are collected through the exporter bundled with Alloy (560113).
Other metrics are exposed using Prometheus metrics endpoints and are
automatically fetched by Alloy with the help of some Docker labels, similar to
what is done to configure Traefik. cAdvisor was also added (83d855) to
provide some container-related metrics.
The loki profile ships Loki to store logs (45c684). While Alloy
can collect and ship logs to Loki, its parsing abilities are limited: I could
not find a way to preserve all metadata associated with structured logs produced
by many applications, including Akvorado. Vector replaces Alloy (95e201)
and features a domain-specific language, VRL, to transform logs. Annoyingly,
Vector currently cannot retrieve Docker logs from before it was
started.
Finally, the grafana profile ships Grafana, but the shipped dashboards are
broken. This is planned for a future version.
Documentation
The Docker Compose setup provided by Akvorado makes it easy to get the web
interface up and running quickly. However, Akvorado requires a few mandatory
steps to be functional. It ships with comprehensive documentation, including
a chapter about troubleshooting problems. I hoped this documentation would
reduce the support burden. It is difficult to know if it works. Happy users
rarely report their success, while some users open discussions asking for help
without reading much of the documentation.
In this release, the documentation was significantly improved.
The documentation was updated (fc1028) to match Akvorado s new architecture.
The troubleshooting section was rewritten (17a272). Instructions on how to
improve ClickHouse performance when upgrading from versions earlier than 1.10.0
was added (5f1e9a). An LLM proofread the entire content (06e3f3).
Developer-focused documentation was also improved (548bbb, e41bae, and
871fc5).
From a usability perspective, table of content sections are now collapsable
(c142e5). Admonitions help draw user attention to important points
(8ac894).
Example of use of admonitions in Akvorado's documentation
Continuous integration
This release includes efforts to speed up continuous integration on GitHub.
Coverage and race tests run in parallel (6af216 and fa9e48). The Docker
image builds during the tests but gets tagged only after they succeed
(8b0dce).
GitHub workflow to test and build Akvorado
End-to-end tests (883e19) ensure the shipped Docker Compose setup works as
expected. Hurl runs tests on various HTTP endpoints, particularly to verify
metrics (42679b and 169fa9). For example:
## Test inlet has received NetFlow flowsGEThttp://127.0.0.1:8080/prometheus/api/v1/query[Query]query:sum(akvorado_inlet_flow_input_udp_packets_total job="akvorado-inlet",listener=":2055" )
HTTP200[Captures]inlet_receivedflows:jsonpath "$.data.result[0].value[1]" toInt
[Asserts]variable"inlet_receivedflows">10## Test inlet has sent them to KafkaGEThttp://127.0.0.1:8080/prometheus/api/v1/query[Query]query:sum(akvorado_inlet_kafka_sent_messages_total job="akvorado-inlet" )
HTTP200[Captures]inlet_sentflows:jsonpath "$.data.result[0].value[1]" toInt
[Asserts]variable"inlet_sentflows">=inlet_receivedflows
Docker
Akvorado ships with a comprehensive Docker Compose setup to help users get
started quickly. It ensures a consistent deployment, eliminating many
configuration-related issues. It also serves as a living documentation of the
complete architecture.
This release brings some small enhancements around Docker:
Previously, many Docker images were pulled from the Bitnami Containers
library. However, VMWare acquired Bitnami in 2019 and Broadcom acquired
VMWare in 2023. As a result, Bitnami images were deprecated in less than a
month. This was not really a surprise4. Previous versions of Akvorado
had already started moving away from them. In this release, the Apache project s
Kafka image replaces the Bitnami one (1eb382). Thanks to the switch to KRaft
mode, Zookeeper is no longer needed (0a2ea1, 8a49ca, and f65d20).
Akvorado s Docker images were previously compiled with Nix. However, building
AArch64 images on x86-64 is slow because it relies on QEMU userland emulation.
The updated Dockerfile uses multi-stage and multi-platform builds: one
stage builds the JavaScript part on the host platform, one stage builds the Go
part cross-compiled on the host platform, and the final stage assembles the
image on top of a slim distroless image (268e95 and d526ca).
# This is a simplified versionFROM--platform=$BUILDPLATFORMnode:20-alpineASbuild-js
RUNapkadd--no-cachemake
WORKDIR/buildCOPYconsole/frontendconsole/frontend
COPYMakefile.
RUNmakeconsole/data/frontend
FROM--platform=$BUILDPLATFORMgolang:alpineASbuild-go
RUNapkadd--no-cachemakecurlzip
WORKDIR/buildCOPY..
COPY--from=build-js/build/console/data/frontendconsole/data/frontend
RUNgomoddownload
RUNmakeall-indep
ARGTARGETOSTARGETARCHTARGETVARIANTVERSION
RUNmake
FROMgcr.io/distroless/static:latestCOPY--from=build-go/build/bin/akvorado/usr/local/bin/akvorado
ENTRYPOINT["/usr/local/bin/akvorado"]
When building for multiple platforms with --platform
linux/amd64,linux/arm64,linux/arm/v7, the build steps until the highlighted
line execute only once for all platforms. This significantly speeds up the
build.
Akvorado now ships Docker images for these platforms: linux/amd64,
linux/amd64/v3, linux/arm64, and linux/arm/v7. When requesting
ghcr.io/akvorado/akvorado, Docker selects the best image for the current CPU.
On x86-64, there are two choices. If your CPU is recent enough, Docker
downloads linux/amd64/v3. This version contains additional optimizations and
should run faster than the linux/amd64 version. It would be interesting to
ship an image for linux/arm64/v8.2, but Docker does not support the same
mechanism for AArch64 yet (792808).
Go
This release includes many changes related to Go but not visible to the users.
Toolchain
In the past, Akvorado supported the two latest Go versions, preventing immediate
use of the latest enhancements. The goal was to allow users of stable
distributions to use Go versions shipped with their distribution to compile
Akvorado. However, this became frustrating when interesting features, like go
tool, were released. Akvorado 2.0 requires Go 1.25 (77306d) but can be
compiled with older toolchains by automatically downloading a newer one
(94fb1c).5 Users can still override GOTOOLCHAIN to revert this
decision. The recommended toolchain updates weekly through CI to ensure we get
the latest minor release (5b11ec). This change also simplifies updates to
newer versions: only go.mod needs updating.
Thanks to this change, Akvorado now uses wg.Go() (77306d) and I have
started converting some unit tests to the new test/synctest package
(bd787e, 7016d8, and 159085).
Testing
When testing equality, I use a helper function Diff() to display the
differences when it fails:
This function uses kylelemons/godebug. This package is
no longer maintained and has some shortcomings: for example, by default, it does
not compare struct private fields, which may cause unexpectedly successful
tests. I replaced it with google/go-cmp, which is stricter
and has better output (e2f1df).
Another package for Kafka
Another change is the switch from Sarama to franz-go to interact with
Kafka (756e4a and 2d26c5). The main motivation for this change is to
get a better concurrency model. Sarama heavily relies on channels and it is
difficult to understand the lifecycle of an object handed to this package.
franz-go uses a more modern approach with callbacks6 that is both more
performant and easier to understand. It also ships with a package to spawn fake
Kafka broker clusters, which is more convenient than the mocking functions
provided by Sarama.
Improved routing table for BMP
To store its routing table, the BMP component used
kentik/patricia, an implementation of a patricia tree
focused on reducing garbage collection pressure.
gaissmai/bart is a more recent alternative using an
adaptation of [Donald Knuth s ART algorithm][] that promises better
performance and delivers it: 90% faster lookups and 27% faster
insertions (92ee2e and fdb65c).
Unlike kentik/patricia, gaissmai/bart does not help efficiently store values
attached to each prefix. I adapted the same approach as kentik/patricia to
store route lists for each prefix: store a 32-bit index for each prefix, and use
it to build a 64-bit index for looking up routes in a map. This leverages Go s
efficient map structure.
gaissmai/bart also supports a lockless routing table version, but this is not
simple because we would need to extend this to the map storing the routes and to
the interning mechanism. I also attempted to use Go s new unique package to
replace the intern package included in Akvorado, but performance was
worse.7
Miscellaneous
Previous versions of Akvorado were using a custom Protobuf encoder for
performance and flexibility. With the introduction of the outlet service,
Akvorado only needs a simple static schema, so this code was removed. However,
it is possible to enhance performance with
planetscale/vtprotobuf (e49a74, and 8b580f).
Moreover, the dependency on protoc, a C++ program, was somewhat annoying.
Therefore, Akvorado now uses buf, written in Go, to convert a Protobuf
schema into Go code (f4c879).
Another small optimization to reduce the size of the Akvorado binary by
10 MB was to compress the static assets embedded in Akvorado in a ZIP file. It
includes the ASN database, as well as the SVG images for the documentation. A
small layer of code makes this change transparent (b1d638 and e69b91).
JavaScript
Recently, two large supply-chain attacks hit the JavaScript ecosystem: one
affecting the popular packages chalk and debug and another
impacting the popular package @ctrl/tinycolor. These attacks also
exist in other ecosystems, but JavaScript is a prime target due to heavy use of
small third-party dependencies. The previous version of Akvorado relied on 653
dependencies.
npm-run-all was removed (3424e8, 132 dependencies). patch-package was
removed (625805 and e85ff0, 69 dependencies) by moving missing
TypeScript definitions to env.d.ts. eslint was replaced with oxlint, a
linter written in Rust (97fd8c, 125 dependencies, including the plugins).
I switched from npm to Pnpm, an alternative package manager (fce383).
Pnpm does not run install scripts by default8 and prevents installing
packages that are too recent. It is also significantly faster.9 Node.js
does not ship Pnpm but it ships Corepack, which allows us to use Pnpm
without installing it. Pnpm can also list licenses used by each dependency,
removing the need for license-compliance (a35ca8, 42 dependencies).
For additional speed improvements, beyond switching to Pnpm and Oxlint, Vite
was replaced with its faster Rolldown version (463827).
After these changes, Akvorado only pulls 225 dependencies.
Next steps
I would like to land three features in the next version of Akvorado:
Add Grafana dashboards to complete the observability stack. See issue
#1906 for details.
Integrate OVH s Grafana plugin by providing a stable API for such
integrations. Akvorado s web console would still be useful for browsing
results, but if you want to build and share dashboards, you should switch to
Grafana. See issue #1895.
Move some work currently done in ClickHouse (custom dictionaries, GeoIP and IP
enrichment) back into the outlet service. This should give more flexibility
for adding features like the one requested in issue #1030. See issue #2006.
I started working on splitting the inlet into two parts more than one year ago.
I found more motivation in recent months, partly thanks to Claude Code,
which I used as a rubber duck. Almost none of the produced code was
kept:10 it is like an intern who does not learn.
Many attempts were made to make the BMP component both performant and
not blocking. See for example PR #254, PR #255, and PR #278.
Despite these efforts, this component remained problematic for most users.
See issue #1461 as an example.
Some features have been pushed to ClickHouse to avoid the
processing cost in the inlet. See for example PR #1059.
Broadcom is known for its user-hostile moves. Look at what happened
with VMWare.
As a Debian developer, I dislike these mechanisms that circumvent
the distribution package manager. The final straw came when Go 1.25 spent one month in the Debian NEW queue, an arbitrary mechanism I
don t like at all.
In the early years of Go, channels were heavily promoted. Sarama
was designed during this period. A few years later, a more nuanced approach
emerged. See notably Go channels are bad and you should feel bad.
This should be investigated further, but my theory is that the
intern package uses 32-bit integers, while unique uses 64-bit pointers.
See commit 74e5ac.
This is also possible with npm. See commit dab2f7.
An even faster alternative is Bun, but it is less available.
The exceptions are part of the code for the admonition blocks,
the code for collapsing the table of content, and part of the documentation.
Here, in classic Goerzen deep dive fashion, is more information than you knew you wanted about a topic you ve probably never thought of. I found it pretty interesting, because it took me down a rabbit hole of subsystems I ve never worked with much and a mishmash of 1980s and 2020s tech.
I had previously tried and failed to get an actual 80x25 Linux console, but I ve since figured it out!
This post is about the Linux text console not X or Wayland. We re going to get the console right without using those systems. These instructions are for Debian trixie, but should be broadly applicable elsewhere also. The end result can look like this:
(That s a Wifi Retromodem that I got at VCFMW last year in the Hayes modem case)
What s a pixel?
How would you define a pixel these days? Probably something like a uniquely-addressable square dot in a two-dimensional grid .
In the world of VGA and CRTs, that was just a logical abstraction. We got an API centered around that because it was convenient. But, down the VGA cable and on the device, that s not what a pixel was.
A pixel, back then, was a time interval. On a multisync monitor, which were common except in the very early days of VGA, the timings could be adjusted which produced logical pixels of different sizes. Those screens often had a maximum resolution but not necessarily a native resolution in the sense that an LCD panel does. Different timings produced different-sized pixels with equal clarity (or, on cheaper monitors, equal fuzziness).
A side effect of this was that pixels need not be square. And, in fact, in the standard DOS VGA 80x25 text mode, they weren t.
You might be seeing why DVI, DisplayPort, and HDMI replaced VGA for LCD monitors: with a VGA cable, you did a pixel-to-analog-timings conversion, then the display did a timings-to-pixels conversion, and this process could be a bit lossy. (Hence why you sometimes needed to fill the screen with an image and push the center button on those older LCD screens)
(Note to the pedantically-inclined: yes I am aware that I have simplified several things here; for instance, a color LCD pixel is made up of approximately 3 sub-dots of varying colors, and that things like color eInk displays have two pixel grids with different sizes of pixels layered atop each other, and printers are another confusing thing altogether, and and and . MOST PEOPLE THINK OF A PIXEL AS A DOT THESE DAYS, OK?)
What was DOS text mode?
We think of this as the standard display: 80 columns wide and 25 rows tall. 80x25. By the time Linux came along, the standard Linux console was VGA text mode something like the 4th incarnation of text modes on PCs (after CGA, MDA, and EGA). VGA also supported certain other sizes of characters giving certain other text dimensions, but if I cover all of those, this will explode into a ridiculously more massive page than it already is.
So to display text on an 80x25 DOS VGA system, ultimately characters and attributes were written into the text buffer in memory. The VGA system then rendered it to the display as a 720x400 image (at 70Hz) with non-square pixels such that the result was approximately a 4:3 aspect ratio.
The font used for this rendering was a bitmapped one using 8x16 cells. You might do some math here and point out that 8 * 80 is only 640, and you d be correct. The fonts were 8x16 but the rendered cells were 9x16. The extra pixel was normally used for spacing between characters. However, in line graphics mode, characters 0xC0 through 0xDF repeated the 8th column in the position of the 9th, allowing the continuous line-drawing characters we re used to from TUIs.
Problems rendering DOS fonts on modern systems
By now, you re probably seeing some of the issues we have rendering DOS screens on more modern systems. These aren t new at all; I remember some of these from back in the days when I ran OS/2, and I think also saw them on various terminals and consoles in OS/2 and Windows.
Some issues you d encounter would be:
Incorrect aspect ratio caused by using the original font and rendering it using 1:1 square pixels (resulting in a squashed appearance)
Incorrect aspect ratio for ANOTHER reason, caused by failing to render column 9, resulting in text that is overall too narrow
Characters appearing to be touching each other when they shouldn t (failing to render column 9; looking at you, dosbox)
Gaps between line drawing characters that should be continuous, caused by rendering column 9 as empty space in all cases
Character set issues
DOS was around long before Unicode was. In the DOS world, there were codepages that selected the glyphs for roughly the high half of the 256 possible characters. CP437 was the standard for the USA; others existed for other locations that needed different characters. On Unix, the USA pre-Unicode standard was Latin-1. Same concept, but with different character mappings.
Nowadays, just about everything is based on UTF-8. So, we need some way to map our CP437 glyphs into Unicode space. If we are displaying DOS-based content, we ll also need a way to map CP437 characters to Unicode for display later, and we need these maps to match so that everything comes out right. Whew.
So, let s get on with setting this up!
Selecting the proper video mode
As explained in my previous post, proper hardware support for DOS text mode is limited to x86 machines that do not use UEFI. Non-x86 machines, or x86 machines with UEFI, simply do not contain the necessary support for it. As these are now standard, most of the time, the text console you see on Linux is actually the kernel driving the video hardware in graphics mode, and doing the text rendering in software.
That s all well and good, but it makes it quite difficult to actually get an 80x25 console.
First, we need to be running at 720x400. This is where I ran into difficulty last time. I realized that my laptop s LCD didn t advertise any video modes other than its own native resolution. However, almost all external monitors will, and 720x400@70 is a standard VGA mode from way back, so it should be well-supported.
You need to find the Linux device name for your device. You can look at the possible devices with ls -l /sys/class/drm. If you also have a GUI, xrandr may help too. But in any case, each directory under /sys/class/drm has a file named modes, and if you cat them all, you will eventually come across one with a bunch of modes defined. Drop the leading card0 or whatever from the directory name, and that s your device. (Verify that 720x400 is in modes while you re at it.)
Now, you re going to edit /etc/default/grub and add something like this to GRUB_CMDLINE_LINUX_DEFAULT:
video=DP-1:720x400@70
Of course, replace DP-1 with whatever your device is.
Now you can run update-grub and reboot. You should have a 720x400 display.
At first, I thought I had succeeded by using Linux s built-in VGA font with that mode. But it looked too tall. After noticing that repeated 0s were touching, I got suspicious about the missing 9th column in the cells. stty -a showed that my screen was 90x25, which is exactly what it would show if I was using 8x16 instead of 9x16 cells. Sooo . I need to prepare a 9x16 font.
Building it yourself
First, install some necessary software: apt-get install fontforge bdf2psf
Start by going to the Oldschool PC Font Pack Download page. Download oldschool_pc_font_pack_v2.2_FULL.zip and unpack it.
The file we re interested in is otb - Bm (linux bitmap)/Bm437_IBM_VGA_9x16.otb. Open it in fontforge by running fontforge BmPlus_IBM_VGA_9x16.otb. When it asks if you will load the bitmap fonts, hit select all, then yes. Go to File -> generate fonts. Save in a BDF, no need for outlines, and use guess for resolution.
Now you have a file such as Bm437_IBM_VGA_9x16-16.bdf. Excellent.
Now we need to generate a Unicode map file. We will make sure this matches the system s by enumerating every character from 0x00 to 0xFF, converting it from CP437 to Unicode, and writing the appropriate map.
Here s a Python script to do that:
By convention, we normally store these files gzipped, so gzip CP437-VGA.psf.
You can test it on the console with setfont CP437-VGA.psf.gz.
Now copy this file into /usr/local/etc.
Activating the font
Now, edit /etc/default/console-setup. It should look like this:
# CONFIGURATION FILE FOR SETUPCON
# Consult the console-setup(5) manual page.
ACTIVE_CONSOLES="/dev/tty[1-6]"
CHARMAP="UTF-8"
CODESET="Lat15"
FONTFACE="VGA"
FONTSIZE="8x16"
FONT=/usr/local/etc/CP437-VGA.psf.gz
VIDEOMODE=
# The following is an example how to use a braille font
# FONT='lat9w-08.psf.gz brl-8x8.psf'
At this point, you should be able to reboot. You should have a proper 80x25 display! Log in and run stty -a to verify it is indeed 80x25.
Using and testing CP437
Part of the point of CP437 is to be able to access BBSs, ANSI art, and similar.
Now, remember, the Linux console is still in UTF-8 mode, so we have to translate CP437 to UTF-8, then let our font map translate it back to CP437. A weird trip, but it works.
Let s test it using the Textfiles ANSI art collection. In the artworks section, I randomly grabbed a file near the top: borgman.ans. Download that, and display with:
clear; iconv -f CP437 -t UTF-8 < borgman.ans
You should see something similar to but actually more accurate than the textfiles PNG rendering of it, which you ll note has an incorrect aspect ratio and some rendering issues. I spot-checked with a few others and they seemed to look good. belinda.ans in particular tries quite a few characters and should give you a good sense if it is working.
Use with interactive programs
That s all well and good, but you re probably going to want to actually use this with some interactive program that expects CP437. Maybe Minicom, Kermit, or even just telnet?
For this, you ll want to apt-get install luit. luit maps CP437 (or any other encoding) to UTF-8 for display, and then of course the Linux console maps UTF-8 back to the CP437 font.
Here s a way you can repeat the earlier experiment using luit to run the cat program:
clear; luit -encoding CP437 cat borgman.ans
You can run any command under luit. You can even run luit -encoding CP437 bash if you like. If you do this, it is probably a good idea to follow my instructions on generating locales on my post on serial terminals, and then within luit, set LANG=en_us.IBM437. But note especially that you can run programs like minicom and others for accessing BBSs under luit.
Final words
This gave you a nice DOS-type console. Although it doesn t have glyphs for many codepoints, it does run in UTF-8 mode and therefore is compatible with modern software.
You can achieve greater compatibility with more UTF-8 codepoints with the DOS font, at the expense of accuracy of character rendering (especially for the double-line drawing characters) by using /usr/share/bdf2psf/standard.equivalents instead of /dev/null in the bdf2psf command.
Or you could go for another challenge, such as using the DEC vt-series fonts for coverage of ISO-8859-1. But just using fonts extracted from DEC ROM won t work properly, because DEC terminals had even more strangeness going on than DOS fonts.
My trip to pgday.at started Wednesday at the airport in D sseldorf. I was there on time, and the plane started with an estimated flight time of about 90 minutes. About half an hour into the flight, the captain announced that we would be landing in 30 minutes - in D sseldorf, because of some unspecified technical problems. Three hours after the original departure time, the plane made another attempt, and we made it to Vienna.
On the plane I had already met Dirk Krautschick who had the great honor of bringing Slonik (in the form of a big extra bag) to the conference, and we took a taxi to the hotel. On the taxi, the next surprise happened: Hans-J rgen Sch nig unfortunately couldn't make it to the conference, and his talks had to be replaced. I had submitted a talk to the conference, but it was not accepted, and neither queued on the reserve list. But two speakers on the reserve list had cancelled, and another was already giving a talk in parallel to the slot that had to be filled, so Pavlo messaged me if I could hold the talk - well of course I could. Before, I didn't have any specific plans for the evening yet, but suddenly I was a speaker, so I joined the folks going to the speakers dinner at the Wiener Grill Haus two corners from the hotel. It was a very nice evening, chatting with a lot of folks from the PostgreSQL community that I had not seen for a while.
Thursday was the conference day. The hotel was a short walk from the venue, the Apothekertrakt in Vienna's Schloss Sch nbrunn. The courtyard was already filled with visitors registering for the conference. Since I originally didn't have a talk scheduled, I had signed up to volunteer for a shift as room host. We got our badge and swag bag, and I changed into the "crew" T-shirt.
The opening and sponsor keynotes took place in the main room, the Orangerie. We were over 100 people in the room, but apparently still not enough to really fill it, so the acoustics with some echo made it a bit difficult to understand everything. I hope that part can be improved for next time (which is planned to happen!).
I was host for the Maximilian room, where the sponsor sessions were scheduled in the morning. The first talk was by our Peter Hofer, also replacing the absent Hans. He had only joined the company at the beginning of the same week, and was already tasked to give Hans' talk on PostgreSQL as Open Source. Of course he did well.
Next was Tanmay Sinha from Readyset. They are building a system that caches expensive SQL queries and selectively invalidates the cache whenever any data used by these queries changes. Whenever actually fixing the application isn't feasible, that system looks like an interesting alternative to manually maintaining materialized views, or perhaps using pg_ivm.
After lunch, I went to Federico Campoli's Mastering Index Performance, but really spent the time polishing the slides for my talk. I had given the original version at pgconf.de in Berlin in May, and the slides were still in German, so I had to do some translating. Luckily, most slides are just git commit messages, so the effort was manageable.
The next slot was mine, talking about Modern VACUUM. I started with a recap of MVCC, vacuum and freezing in PostgreSQL, and then showed how over the past years, the system was updated to be more focused (the PostgreSQL 8.4 visibility map tells vacuum which pages to visit), faster (12 made autovacuum run 10 times faster by default), less scary (14 has an emergency mode where freezing switches to maximum speed if it runs out of time; 16 makes freezing create much less WAL) and more performant (17 makes vacuum use much less memory). In summary, there is still room for the DBA to tune some knobs (for example, the default autovacuum_max_workers=3 isn't much), but the vacuum default settings are pretty much okay these days for average workloads. Specific workloads still have a whopping 31 postgresql.conf settings at their disposal just for vacuum.
Right after my talk, there was another vacuum talk: When Autovacuum Met FinOps by Mayuresh Bagayatkar. He added practical advice on tuning the performance in cloud environments. Luckily, our contents did not overlap.
After the coffee break, I was again room host, now for Floor Drees and Contributing to Postgres beyond code. She presented the various ways in which PostgreSQL is more than just the code in the Git repository: translators, web site, system administration, conference organizers, speakers, bloggers, advocates. As a member of the PostgreSQL Contributors Committee, I could only approve and we should closer cooperate in the future to make people's contributions to PostgreSQL more visible and give them the recognition they deserve.
That was already the end of the main talks and everyone rushed to the Orangerie for the lightning talks. My highlight was the Sheldrick Wildlife Trust. Tickets for the conference had included the option to donate for the elephants in Kenya, and the talk presented the trust's work in the elephant orphanage there.
After the conference had officially closed, there was a bonus track: the Celebrity DB Deathmatch, aptly presented by Boriss Mejias. PostgreSQL, MongoDB, CloudDB and Oracle were competing for the grace of a developer. MongoDB couldn't stand the JSON workload, CloudDB was dismissed for handing out new invoices all the time, and Oracle had even brought a lawyer to the stage, but then lost control over a literally 10 meter long contract with too much fine print. In the end, PostgreSQL (played by Floor) won the love of the developer (played by our Svitlana Lytvynenko).
The day closed with a gathering at the Brandauer Schlossbr u - just at the other end of the castle ground, but still a 15min walk away. We enjoyed good beer and Kaiserschmarrn. I went back to the hotel a bit before midnight, but some extended that time quite some bit more.
On Friday, my flight back was only in the afternoon, so I spent some time in morning in the Technikmuseum just next to the hotel, enjoying some old steam engines and a live demonstration of Tesla coils. This time, the flight actually went to the destination, and I was back in D sseldorf in the late afternoon.
In summary, pgday.at was a very nice event in a classy location. Thanks to the organizers for putting in all the work - and next year, Hans will hopefully be present in person!
The post A Trip To Vienna With Surprises appeared first on CYBERTEC PostgreSQL Services & Support.
About 95% of my Debian contributions this month were
sponsored by Freexian.
You can also support my work directly via
Liberapay or GitHub
Sponsors.
Python team
forky is
open!
As a result I m starting to think about the upcoming Python
3.14. At some point we ll doubtless do
a full test rebuild, but in advance of that I concluded that one of the most
useful things I could do would be to work on our very long list of packages
with new upstream
versions.
Of course there s no real chance of this ever becoming empty since upstream
maintainers aren t going to stop work for that long, but there are a lot of
packages there where we re quite a long way out of date, and many of those
include fixes that we ll need for 3.14, either directly or by fixing
interactions with new versions of other packages that in turn will need to
be fixed. We can backport changes when we need to, but more often than not
the most efficient way to do things is just to keep up to date.
So, I upgraded these packages to new upstream versions (deep breath):
That s only about 10% of the backlog, but of course others are working on
this too. If we can keep this up for a while then it should help.
I packaged pytest-run-parallel,
pytest-unmagic (still in NEW), and
python-forbiddenfruit (still in NEW),
all needed as new dependencies of various other packages.
setuptools upstream will be removing the setup.py install
command on 31
October. While this may not trickle down immediately into Debian, it does
mean that in the near future nearly all Python packages will have to use
pybuild-plugin-pyproject (note that this does not mean that they
necessarily have to use pyproject.toml; this is just a question of how the
packaging runs the build system). We talked about this a bit at DebConf,
and I said that I d noticed a number of packages where this isn t
straightforward and promised to write up some notes. I wrote the
Python/PybuildPluginPyproject
wiki page for this; I expect to add more bits and pieces to it as I find them.
On that note, I converted several packages to pybuild-plugin-pyproject:
I reviewed Debian defaults: nftables as banaction and systemd as
backend,
but it looked as though nothing actually needed to be changed so we closed
this with no action.
Rust team
Upgrading Pydantic was complicated, and required a rust-pyo3 transition
(which Jelmer Vernoo started and Peter Michael Green has mostly been
driving, thankfully), packaging rust-malloc-size-of (including an upstream
portability fix), and
upgrading several packages to new upstream versions:
Skimming through the latest libvirt releases, to my surprise, i found that
latest versions (>= v10.10.0) have added support for the QCOW data-file
setting.
Usually the incremental backup feature using bitmaps was limited to qcow2 based
images, as there was no way to store the bitmaps persistently within raw
devices. This basically ruled out proper incremental backups for direct
attached luns, etc.
In the
past,
there were some discussions how to implement this, mostly by using a separate
metadata qcow image, holding the bitmap information persistently.
These approaches have been discussed again lately and required features were
implemented
In order to be able to use the feature, you need to configure the virtual
machines and its disks in a special way:
Lets assume you have a virtual machine that uses a raw device
/tmp/datafile.raw
1) Create an qcow image (same size as the raw image):
# point the data-file to a temporary file, as create will overwrite whatever it finds here
qemu-img create -f qcow2 /tmp/metadata.qcow2 -odata_file=/tmp/TEMPFILE,data_file_raw=true ..
rm-f /tmp/TEMPFILE
2) Now use the amend option to point the qcow image to the right raw device
using the data-file option:
Here s my 70th monthly but brief update about the activities I ve done in the F/L/OSS world.
Debian
This was my 79th month of actively contributing to Debian.
I became a DM in late March 2019 and a DD on Christmas 19! \o/
Debian was in freeze throughout so whilst I didn t do many uploads, there s a bunch of other things I did:
Attended DebConf25 in Brest, France.
Lead the bursary BOF and discussions.
Participated in other sessions, especially around the FTP masters.
I ve started to look at things with my trainee hat on.
Participated in the Debian Security Tracker sprints during DebCamp. More on that below.
Mentoring for newcomers.
Moderation of -project mailing list.
Ubuntu
This was my 54th month of actively contributing to Ubuntu.
I joined Canonical to work on Ubuntu full-time back in February 2021.
Whilst I can t give a full, detailed list of things I did (there s so much and some of it might not be public yet!), here s a quick TL;DR of what I did:
Got a recognition award for leading 24.04.2 LTS release and leading the Release Management team.
Preparing for the 24.04.3 LTS release early next month.
Debian (E)LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.
And Debian Extended LTS (ELTS) is its sister project, extending support to the stretch and jessie release (+2 years after LTS support).
This was my 70th month as a Debian LTS and 57th month as a Debian ELTS paid contributor.
I only worked for 15.00 hours for LTS and 5.00 hours for ELTS and did the following things:
Coordinated with upstream due to lack of clarity on 1.11.4 being affected & not having a clear reproducer.
As 1.11.4 was still partially vulnerable and the backport was non-trivial, it was probably conveinent to bump the upstream version to 1.11.12 instead, fixing:
Helped Bastien with ca-certificates bit & coordinating with fellow Debian contributors.
Also triaged some newly supported packages for ELTS.
[LTS] Attended the monthly LTS meeting on IRC. Summary here.
Debian Security Tracker sprint 2025
Thanks to the LTS team for also organizing a security tracker sprint during DebCamp25. I attended the sprint and spent 10 hours working on the following tasks:
This includes some coordination with the web team and some back and forth with them to ensure both the Web team and the Security team would be happy with the fix. It also fixes more issues as mentioned in the description of the MR.
Show packages from next-point-release.txt in source package overview
I have been using Disroot as my primary email ever since
openmailbox.org stopped. I am very grateful for Disroot s service and
I occasionally donate to them.
Recently, my Disroot inbox has been flooded with spam. On an average
day, I used to receive around 90% spams on entire email
count. However, the situation has improved since then. I contacted the
Disroot team, and they informed me that they are aware of the
situation and planning to migrate to Rspamd from Spamassassin.
I don t know whether they deployed Rspamd, even if so that only going
to process incoming mails, I am looking for a way to identify spams
and purge that are already entered my Imap folders.
Later I found this script nh2/rspamd-move[1], which seems fit my need.
I made couple of trivial changes in the script for my use case. I
wasn t sure of directly running this on my Mail/ dir, so I cloned my
entire local mail directory to another directory and made available to
podman container where I script and rspamd instance exist. I trained
rspamd from the /Spam. Later, I manually moved couple of mails to
/spam folder/. I requested friends to share their spam folder in the
#debian-in channel, but that didn t happen :P
$podman run -it --mount
type=bind,source=/home/abhijith/$MAILS/,target=/container-mail-clone
id:latest
$script.py
(It took some time since I have around 10000+ emails)
Wow, it was quite a successful attempt, I was able to catch most of it
and move to spam/ and couple of false positive in a different
folder. Now I want to do the same in the actual maildir yet very
skeptical. While going through the cloned folder with mutt -f I
remembered that the mails are already indexed by notmuch.
So all I need to do is operate tagging and deletion with notmuch and
it will be synced back to the original mail dir. Ta-da. I cleaned by
Inbox.
[1] - https://github.com/nh2/rspamd-move
Update on 2025-07-28: added note about Debian 13/trixie support for OpenVox (thanks, Ben Ford!)
Debian v13 with codename trixie is scheduled to be published as new stable release on 9th of August 2025.
I was the driving force at several of my customers to be well prepared for the upcoming stable release (my efforts for trixie started in August 2024). On the one hand, to make sure packages we care about are available and actually make it into the release. On the other hand, to ensure there are no severe issues that make it into the release and to get proper and working upgrades. So far everything is looking pretty well and working fine, the efforts seemed to have payed off. :)
As usual with major upgrades, there are some things to be aware of, and hereby I m starting my public notes on trixie that might be worth for other folks. My focus is primarily on server systems and looking at things from a sysadmin perspective.
Further readings
As usual start at the official Debian release notes, make sure to especially go through What s new in Debian 13 + issues to be aware of for trixie (strongly recommended read!).
Package versions
As a starting point, let s look at some selected packages and their versions in bookworm vs. trixie as of 2025-07-20 (mainly having amd64 in mind):
Package
bookworm/v12
trixie/v13
ansible
2.14.3
2.19.0
apache
2.4.62
2.4.64
apt
2.6.1
3.0.3
bash
5.2.15
5.2.37
ceph
16.2.11
18.2.7
docker
20.10.24
26.1.5
dovecot
2.3.19
2.4.1
dpkg
1.21.22
1.22.21
emacs
28.2
30.1
gcc
12.2.0
14.2.0
git
2.39.5
2.47.2
golang
1.19
1.24
libc
2.36
2.41
linux kernel
6.1
6.12
llvm
14.0
19.0
lxc
5.0.2
6.0.4
mariadb
10.11
11.8
nginx
1.22.1
1.26.3
nodejs
18.13
20.19
openjdk
17.0
21.0
openssh
9.2p1
10.0p1
openssl
3.0
3.5
perl
5.36.0
5.40.1
php
8.2+93
8.4+96
podman
4.3.1
5.4.2
postfix
3.7.11
3.10.3
postgres
15
17
puppet
7.23.0
8.10.0
python3
3.11.2
3.13.5
qemu/kvm
7.2
10.0
rsync
3.2.7
3.4.1
ruby
3.1
3.3
rust
1.63.0
1.85.0
samba
4.17.12
4.22.3
systemd
252.36
257.7-1
unattended-upgrades
2.9.1
2.12
util-linux
2.38.1
2.41
vagrant
2.3.4
2.3.7
vim
9.0.1378
9.1.1230
zsh
5.9
5.9
Misc unsorted
The asterisk package once again didn t make it into trixie (see #1031046)
The new debian-repro-status package provides the identically named command-line tool debian-repro-status to query the reproducibility status of your installed Debian packages
The Grml live system project provided further of their packages into Debian. Available as of trixie are now also grml-keyring (OpenPGP certificates used for signing the Grml repositories), grml-hwinfo (a tool which collects information of the hardware ) + grml-paste (command line interface for paste.debian.net)
If you use pacemaker, be aware that its fence-agents package is now a transitional package. All the fence-agents got split into separate packages (fence-agents-$whatever). If you want to have all the fence-agents available, make sure to install the fence-agents-all package. If you have Recommends disabled, you definitely should be aware of this.
support for colors (f.e. green for installs/upgrades, yellow for downgrades, red for removals, can be disabled via &dash&dashno-color, APT_NO_COLOR=1 or NO_COLOR=1 and customized via e.g. APT::Color::Action::Install cyan )
organizes output in more readable sections and shows removals more prominently
the new apt modernize-sources command converts /etc/apt/sources.list.d/*.list files into the new .sources format (AKA DEB822)
the new apt distclean command removes all files under $statedir/lists except Release, Release.gpg, and InRelease (it can be used for example, when finalizing images distributed to users)
new configuration option APT::NeverAutoRemove::KernelCount for keeping a configurable amount of kernels, f.e. setting APT::NeverAutoRemove::KernelCount 3 will keep 3 kernels (including the running, and most recent)
new command line option &dash&dashsnapshot, and configuration option APT::Snapshot, controlling the snapshot chosen for archives with Snapshot: enable
new command line option &dash&dashupdate to run the update command before the specified command, like apt &dash&dashupdate install zsh, apt &dash&dashupdate remove foobar or apt &dash&dashupdate safe-upgrade
apt-key is gone, and there s no replacement for it available (if you need an interface for listing present keys)
systemd
systemd got upgraded from v252.36-1~deb12u1 to 257.7-1 and there are lots of changes.
Be aware that systemd v257 has a new net.naming_scheme, v257 being PCI slot number is now read from firmware_node/sun sysfs file. The naming scheme based on devicetree aliases was extended to support aliases for individual interfaces of controllers with multiple ports. This might affect you, see e.g. #1092176 and #1107187, the Debian Wiki provides further useful information.
There are new systemd tools available:
run0: temporarily and interactively acquire elevated or different privileges (serves a similar purpose as sudo)
systemd-ac-power: Report whether we are connected to an external power source
systemd-vpick: Resolve paths to .v/ versioned directories
varlinkctl: Introspect with and invoke Varlink services
The tools provided by systemd gained several new options:
busctl: new option &dash&dashlimit&dashmessages=NUMBER (Stop monitoring after receiving the specified number of message)
hostnamectl: new option &dashj (same as &dash&dashjson=pretty on tty, &dash&dashjson=short otherwise)
journalctl: new options &dash&dashimage&dashpolicy=POLICY (Specify disk image dissection policy), &dash&dashinvocation=ID (Show logs from the matching invocation ID), &dashI (Show logs from the latest invocation of unit), &dash&dashexclude-identifier=STRING (Hide entries with the specified syslog identifier),&dash&dashtruncate-newline (Truncate entries by first newline character), &dash&dashlist-invocations (Show invocation IDs of specified unit), &dash&dashlist-namespaces (Show list of journal namespaces)
localectl: new option &dash&dashfull (Do not ellipsize output)
loginctl: new options &dash&dashjson=MODE (Generate JSON output for list-sessions/users/seats) + &dashj (Same as &dash&dashjson=pretty on tty, &dash&dashjson=short otherwise)
networkctl: new commands edit FILES DEVICES (Edit network configuration files), cat [FILES DEVICES ] (Show network configuration files), mask FILES (Mask network configuration files) + unmask FILES (Unmask network configuration files) + persistent-storage BOOL (Notify systemd-networkd if persistent storage is ready), and new options &dash&dashno-ask-password (Do not prompt for password), &dash&dashno-reload (Do not reload systemd-networkd or systemd-udevd after editing network config), &dash&dashdrop-in=NAME (Edit specified drop-in instead of main config file), &dash&dashruntime (Edit runtime config files) + &dash&dashstdin (Read new contents of edited file from stdin)
systemctl new commands list-paths [PATTERN] (List path units currently in memory, ordered by path), whoami [PID ] (Return unit caller or specified PIDs are part of), soft-reboot (Shut down and reboot userspace) + sleep (Put the system to sleep), and new options &dash&dashcapsule=NAME (Connect to service manager of specified capsule), &dash&dashbefore (Show units ordered before with list-dependencies ), &dash&dashafter (Show units ordered after with list-dependencies ), &dash&dashkill-value=INT (Signal value to enqueue), &dash&dashno-warn (Suppress several warnings shown by default), &dash&dashmessage=MESSAGE (Specify human readable reason for system shutdown), &dash&dashimage&dashpolicy=POLICY (Specify disk image dissection policy), &dash&dashreboot&dashargument=ARG (Specify argument string to pass to reboot()), &dash&dashdrop-in=NAME (Edit unit files using the specified drop-in file name), &dash&dashwhen=TIME (Schedule halt/power-off/reboot/kexec action after a certain timestamp) + &dash&dashstdin (Read
new contents of edited file from stdin)
systemd-analyze new commands architectures [NAME ] (List known architectures), smbios11 (List strings passed via SMBIOS Type #11), image-policy POLICY (Analyze image policy string), fdstore SERVICE (Show file descriptor store contents of service), malloc [D-BUS SERVICE ] (Dump malloc stats of a D-Bus service), has-tpm2 (Report whether TPM2 support is available), pcrs [PCR ] (Show TPM2 PCRs and their names) + srk [>FILE] (Write TPM2 SRK (to FILE)) and new options &dash&dashno-legend (Disable column headers and hints in plot with either &dash&dashtable or &dash&dashjson=), &dash&dashinstance=NAME (Specify fallback instance name for template units), &dash&dashunit=UNIT (Evaluate conditions and asserts of unit), &dash&dashtable (Output plot s raw time data as a table), &dash&dashscale-svg=FACTOR (Stretch x-axis of plot by FACTOR (default: 1.0)), &dash&dashdetailed (Add more details to SVG plot), &dash&dashtldr (Skip comments and empty lines), &dash&dashimage
-policy=POLICY (Specify disk image dissection policy) + &dash&dashmask (Parse parameter as numeric capability mask)
systemd-ask-password: new options &dash&dashuser (Ask only our own user s agents) + &dash&dashsystem (Ask agents of the system and of all users)
systemd-cat: new option &dash&dashnamespace=NAMESPACE (Connect to specified journal namespace)
systemd-creds: new options &dash&dashuser (Select user-scoped credential encryption), &dash&dashuid=UID (Select user for scoped credentials) + &dash&dashallow-null (Allow decrypting credentials with empty key)
systemd-detect-virt: new options &dash&dashcvm (Only detect whether we are run in a confidential VM) + &dash&dashlist-cvm (List all known and detectable types of confidential virtualization)
systemd-firstboot: new options &dash&dashimage-policy=POLICY (Specify disk image dissection policy), &dash&dashkernel-command-line=CMDLINE (Set kernel command line) + &dash&dashreset (Remove existing files)
systemd-id128: new commands var-partition-uuid (Print the UUID for the /var/ partition) + show [NAME UUID] (Print one or more UUIDs), and new options &dash&dashno-pager (Do not pipe output into a pager), &dash&dashno-legend (Do not show the headers and footers), &dash&dashjson=FORMAT (Output inspection data in JSON), &dashj (Equivalent to &dash&dashjson=pretty (on TTY) or &dash&dashjson=short (otherwise)) + &dashP &dash&dashvalue (Only print the value)
systemd-inhibit: new option &dash&dashno-ask-password (Do not attempt interactive authorization)
systemd-machine-id-setup: new option &dash&dashimage-policy=POLICY (Specify disk image dissection policy)
systemd-mount: new options &dash&dashjson=pretty short off (Generate JSON output) + &dash&dashtmpfs (Create a new tmpfs on the mount point)
systemd-notify: new options &dash&dashreloading (Inform the service manager about configuration reloading), &dash&dashstopping (Inform the service manager about service shutdown), &dash&dashexec (Execute command line separated by ; once done), &dash&dashfd=FD (Pass specified file descriptor with along with message) + &dash&dashfdname=NAME (Name to assign to passed file descriptor(s))
systemd-path: new option &dash&dashno-pager (Do not pipe output into a pager)
systemd-run: new options &dash&dashexpand-environment=BOOL (Control expansion of environment variables), &dash&dashjson=pretty short off (Print unit name and invocation id as JSON), &dash&dashignore-failure (Ignore the exit status of the invoked process) + &dash&dashbackground=COLOR (Set ANSI color for background)
systemd-sysext: new options &dash&dashmutable=yes no auto import ephemeral ephemeral-import (Specify a mutability mode of the merged hierarchy), &dash&dashno-reload (Do not reload the service manager), &dash&dashimage-policy=POLICY (Specify disk image dissection policy) + &dash&dashnoexec=BOOL (Whether to mount extension overlay with noexec)
systemd-sysusers: new options &dash&dashtldr (Show non-comment parts of configuration) + &dash&dashimage-policy=POLICY (Specify disk image dissection policy)
systemd-tmpfiles: new command &dash&dashpurge(Delete files and directories marked for creation in specified configuration files (careful!)), and new options &dash&dashuser (Execute user configuration), &dash&dashtldr (Show non-comment parts of configuration files), &dash&dashgraceful (Quietly ignore unknown users or groups), &dash&dashimage-policy=POLICY (Specify disk image dissection policy) + &dash&dashdry-run (Just print what would be done)
systemd-umount: new options &dash&dashjson=pretty short off (Generate JSON output) + &dash&dashtmpfs (Create a new tmpfs on the mount point)
timedatectl: new commands ntp-servers INTERFACE SERVER (Set the interface specific NTP servers) + revert INTERFACE (Revert the interface specific NTP servers) and new option &dashP NAME (Equivalent to &dash&dashvalue &dash&dashproperty=NAME)
Debian s systemd ships new binary packages:
systemd-boot-efi-amd64-signed (Tools to manage UEFI firmware updates (signed))
systemd-cryptsetup (Provides cryptsetup, integritysetup and veritysetup utilities)
systemd-netlogd (journal message forwarder)
systemd-repart (Provides the systemd-repart and systemd-sbsign utilities)
systemd-standalone-shutdown (standalone shutdown binary for use in exitrds)
systemd-ukify (tool to build Unified Kernel Images)
Linux Kernel
The trixie release ships a Linux kernel based on latest longterm version 6.12. As usual there are lots of changes in the kernel area, including better hardware support, and this might warrant a separate blog entry. To highlight some changes with Debian trixie:
New Debian package linux-bpf-dev, providing the header file for BPF CO-RE builds
New Debian package intel-sdsi, Intel On Demand (SDSi) provisioning tool
New Debian package virtme-ng, providing helper scripts to easily test a Linux kernel on a QEMU VM
See Kernelnewbies.org for further changes between kernel versions.
Configuration management
For puppet users, Debian provides the puppet-agent (v8.10.0), puppetserver (v8.7.0) and puppetdb (v8.4.1) packages. Puppet s upstream does not provide packages for trixie, yet. Given how long it took them for Debian bookworm, and with their recent Plans for Open Source Puppet in 2025, it s unclear when (and whether at all) we might get something. As a result of upstream behavior, also the OpenVox project evolved, and they already provide Debian 13/trixie support (https://apt.voxpupuli.org/openvox8-release-debian13.deb). FYI: the AIO puppet-agent package for bookworm (v7.34.0-1bookworm) so far works fine for me on Debian/trixie. Be aware that due to the apt-key removal you need a recent version of the puppetlabs-apt for usage with trixie. The puppetlabs-ntp module isn t yet ready for trixie (regarding ntp/ntpsec), if you should depend on that.
ansible is available and made it with version 2.19 into trixie.
Prometheus stack
Prometheus server was updated from v2.42.0 to v2.53, and all the exporters that got shipped with bookworm are still around (in more recent versions of course). Trixie gained some new exporters:
Virtualization
docker (v26.1.5), ganeti (v3.1.0), libvirt (v11.3.0, be aware of significant changes to libvirt packaging), lxc (v6.0.4), podman (v5.4.2), openstack (see openstack-team on Salsa), qemu/kvm (v10.0.2), xen (v4.20.0) are all still around.
Proxmox already announced their PVE 9.0 BETA, being based on trixie and providing 6.14.8-1 kernel, QEMU 10.0.2, LXC 6.0.4, OpenZFS 2.3.3.
Vagrant is available in version 2.3.7, but Vagrant upstream does not provide packages for trixie yet. Given that HashiCorp adopted the BSL, the future of vagrant in Debian is unclear.
If you re relying on VirtualBox, be aware that upstream doesn t provide packages for trixie, yet. VirtualBox is available from Debian/unstable (version 7.1.12-dfsg-1 as of 2025-07-20), but not shipped with stable release since quite some time (due to lack of cooperation from upstream on security support for older releases, see #794466). Be aware that starting with Linux kernel 6.12, KVM initializes virtualization on module loading by default. This prevents VirtualBox VMs from starting. In order to avoid this, either add kvm.enable_virt_at_load=0 parameter into kernel command line or unload the corresponding kvm_intel / kvm_amd module.
If you want to use Vagrant with VirtualBox on trixie, be aware that Debian s vagrant package as present in trixie doesn t support the VirtualBox package version 7.1 as present in Debian/unstable (manually patching vagrant s meta.rb and rebuilding the package without Breaks: virtualbox (>= 7.1) is known to be working).
util-linux
The are plenty of new options available in the tools provided by util-linux:
blkdiscard: new option &dash&dashquiet (suppress warning messages)
blockdev: new options &dash&dashgetdiskseq (get disk sequence number) + &dash&dashgetzonesz (get zone size)
dmesg: new option &dash&dashkmsg-file (use the file in kmsg format), new &dash&dashtime-format argument raw
findmnt: new options &dash&dashlist-columns (list the available columns), &dash&dashdfi (imitate the output of df(1) with -i option), &dash&dashid (filter by mount node ID), &dash&dashfilter (apply display filter) + &dash&dashuniq-id (filter by
mount node 64-bit ID)
fstrim: new option -types . (limit the set of filesystem types)
hardlink: new options &dash&dashrespect-dir (directory names have to be identical), &dash&dashexclude-subtree (regular expression to exclude directories), &dash&dashprioritize-trees (files found in the earliest specified top-level directory have higher priority), &dash&dashlist-duplicates (print every group of duplicate files), &dash&dashmount (stay within the same filesystem) + &dash&dashzero (delimit output with NULs instead of newlines)
ipcmk: new options &dash&dashposix-shmem (create POSIX shared memory segment of size), &dash&dashposix-semaphore (create POSIX semaphore), &dash&dashposix-mqueue (create POSIX message queue) + &dash&dashname (name of the POSIX resource)
ipcrm: new options &dash&dashposix-shmem (remove POSIX shared memory segment by name), &dash&dashposix-mqueue (remove POSIX message queue by name), &dash&dashposix-semaphore (remove POSIX semaphore by name) + &dash&dashall= (remove all in specified category)
lsblk: new options &dash&dashct-filter (restrict the next counter), &dash&dashct (define a custom counter), &dash&dashhighlight (colorize lines matching the expression), &dash&dashlist-columns (list the available columns), &dash&dashnvme (output info about NVMe devices), &dash&dashproperties-by (methods used to gather data), &dash&dashfilter (print only lines matching the expression), &dash&dashvirtio (output info about virtio devices)
lscpu: new options &dash&dashraw (use raw output format (for -e, -p and -C)) + &dash&dashhierarchic= (use subsections in summary (auto, never, always))
lslocks: new option &dash&dashlist-columns (list the available columns)
lslogins: new option &dash&dashlastlog2 (set an alternate path for lastlog2)
lsns: new options &dash&dashpersistent (namespaces without processes), &dash&dashfilter (apply display filter) + &dash&dashlist-columns (list the available columns)
mkswap: new options &dash&dashendianness= (specify the endianness to use (native, little or big)), &dash&dashoffset (specify the offset in the device), &dash&dashsize (specify the size of a swap file in bytes) + &dash&dashfile (create a swap file)
namei: &dash&dashcontext (print any security context of each file)
nsenter: new options &dash&dashnet-socket (enter socket s network namespace), &dash&dashuser-parent (enter parent user namespace), &dash&dashkeep-caps (retain capabilities granted in user namespaces), &dash&dashenv (inherit environment variables from tar get process) + &dash&dashjoin-cgroup (join the cgroup of the target process)
runuser: new option &dash&dashno-pty (do not create a new pseudo-terminal)
setarch: new option &dash&dashshow= (show current or specific personality and exit)
setpriv: new options &dash&dashptracer (allow ptracing from the given process), &dash&dashlandlock-access (add Landlock access), &dash&dashlandlock-rule (add Landlock rule) + &dash&dashseccomp-filter (load seccomp filter from file)
su: new option &dash&dashno-pty (do not create a new pseudo-terminal)
unshare: new option &dash&dashload-interp ( load binfmt definition in the namespace)
whereis: new option -g (interpret name as glob (pathnames pattern))
wipefs: new argument option feature for &dash&dashbackup= option to specify directory (instead of default $HOME)
zramctl: new option &dash&dashalgorithm-params (algorithm parameters to use)
Now no longer present in util-linux as of trixie:
addpart (tell the kernel about the existence of a specified partition): use partx instead
delpart (tell the kernel to forget about a specified partition): use partx instead
last (show a listing of last logged in users, binary got moved to wtmpdb), lastb (show a listing of last logged in users), mesg (control write access of other users to your terminal), utmpdump (dump UTMP and WTMP files in raw format): see Debian release notes for details
coresched: manage core scheduling cookies for tasks
enosys: utility to make syscalls fail with ENOSYS
exch: atomically exchanges paths between two files
fadvise: utility to use the posix_fadvise system call
pipesz: set or examine pipe buffer sizes and optionally execute command.
waitpid: utility to wait for arbitrary processes
OpenSSH
OpenSSH was updated from v9.2p1 to 10.0p1-5, so if you re interested in all the changes, check out the release notes between those versions (9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9 + 10.0).
Let s highlight some notable behavior changes in Debian:
add support for reading ED25519 private keys in PEM PKCS8 format. Previously only the OpenSSH private key format was supported.
the new hybrid post-quantum algorithm mlkem768x25519-sha256 (based on the FIPS 203 Module-Lattice Key Encapsulation mechanism (ML-KEM) combined with X25519 ECDH) is now used by default for key agreement. This algorithm is considered to be safe against attack by quantum computers, is guaranteed to be no less strong than the popular curve25519-sha256 algorithm, has been standardised by NIST and is considerably faster than the previous default.
the ssh-agent will now delete all loaded keys when signaled with SIGUSR1. This allows deletion of keys without having access to $SSH_AUTH_SOCK.
support systemd-style socket activation in ssh-agent using the LISTEN_PID/LISTEN_FDS mechanism. Activated when these environment variables are set, the agent is started with the -d or -D option and no socket path is set.
add a sshd -G option that parses and prints the effective configuration without attempting to load private keys and perform other checks. (This allows usage of the option before keys have been generated and for configuration evaluation and verification by unprivileged users.)
add support for configuration tags to ssh(1). This adds a ssh_config(5) Tag directive and corresponding Match tag predicate that may be used to select blocks of configuration.
add a match localnetwork predicate. This allows matching on the addresses of available network interfaces and may be used to vary the effective client configuration based on network location.
add a %j token that expands to the configured ProxyJump hostname
add support for Match sessiontype to ssh_config. Allows matching on the type of session initially requested, either shell for interactive sessions, exec for command execution sessions, subsystem for subsystem requests, such as sftp, or none for transport/forwarding-only sessions.
allow glob(3) patterns to be used in sshd_config AuthorizedKeysFile and AuthorizedPrincipalsFile directives.
Thanks to everyone involved in the release, looking forward to trixie + and happy upgrading!
Let s continue with working towards Debian/forky. :)
122 steps later this yields 9dff55ebaef7 (bisect would be better if
we knew a "good" commit).
$ git branch -D tmp ; git switch -c tmp 9dff55ebaef7
$ git am v3_20250418_shawn_lin_pci_dw_rockchip_add_system_pm_support.mbx
$ git rebase -i rockchip-devel
This fails with 3 conflicts. The first is easy, as the one non-comment
line just moves around. The other two involve a new function
rockchip_pcie_unmask_dll_indicator used to reduce code duplication, and in all
3 cases I just took the version of the code from Shawn's patch.
EDIT This rebase turns out to miss (at least) changes in the names
of the PCI* constants. By amusing(?) coincidence, as I was
discovering that, the patch was being rebased by someone more
competent at collabora, and is now in the rockchip-devel branch.
previous episodenext episode
For some time I ve been noticing news reports about PFAs [1]. I hadn t thought much about that issue, I grew up when leaded petrol was standard, when almost all thermometers had mercury, when all small batteries had mercury, and I had generally considered that I had already had so many nasty chemicals in my body that as long as I don t eat bottom feeding seafood often I didn t have much to worry about. I already had a higher risk of a large number of medical issues than I d like due to decisions made before I was born and there s not much to do about it given that there are regulations restricting the emissions of lead, mercury etc.
I just watched a Veritasium video about Teflon and the PFA poisoning related to it s production [2]. This made me realise that it s more of a problem than I realised and it s a problem that s getting worse. PFA levels in the parts-per-trillion range in the environment can cause parts-per-billion in the body which increases the risks of several cancers and causes other health problems. Fortunately there is some work being done on water filtering, you can get filters for a home level now and they are working on filters that can work at a sufficient scale for a city water plant.
There is a map showing PFAs in the environment in Australia which shows some sites with concerning levels that are near residential areas [3]. One of the major causes for that in Australia is fire retardant foam Australia has never had much if any Teflon manufacturing AFAIK.
Also they noted that donating blood regularly can decrease levels of PFAs in the bloodstream. So presumably people who have medical conditions that require receiving donated blood regularly will have really high levels.
2026 already! The winter weather here has really been beautiful and I always enjoy
this time of year. Writing this yearly musical retrospective has now become a
beloved tradition of mine
We re happy to announce that
After cartridge pleating and honeycombing, I was still somewhat in the
mood for that kind of fabric manipulation, and directing my internet
searches in that vague direction, and I stumbled on this:
With the pieces being rectangles the width of the fabric, I was able to
have at least one side of selvedge on all seams, and took advantage of it
by finishing the seams by simply folding the allowances to one sides so
that the selvedge was on top, and hemstitching them down as I would have
done with a folded edge when felling.
Also, at first I wanted to make the smocking in white on white, but then
I thought about a few hanks of electric blue floss I had in my stash,
and decided to just go with it.
The initial seams were quickly made, then I started the smocking at the
neck, and at that time the project went on hold while I got ready to go
to DebConf. Then I came back and took some time to get back into a
sewing mood, but finally the smocking on the next was finished, and I
could go on with the main sewing, which, as I expected, went decently
fast for a handsewing project.
While doing the diagonal smocking on the collar I counted the stitches
to make each side the same length, which didn t completely work because
the gathers weren t that regular to start with, and started each line
from the two front opening going towards the center back, leaving a
triangle with a different size right in the middle. I think overall it
worked well enough.
Then there were a few more interruptions, but at last it was ready! just
as the weather turned cold-ish and puffy shirts were no longer in
season, but it will be there for me next spring.
I did manage to wear it a few times and I have to say that the neck
shaping is quite comfortable indeed: it doesn t pull in odd ways like
the classical historically accurate pirate shirt sometimes does, and the
heavy gathering at the neck makes it feel padded and soft.
I m not as happy with the cuffs: the way I did them with just
honeycombing means that they don t need a closure, and after washing and
a bit of steaming they lie nicely, but then they tend to relax in a
wider shape. The next time I think I ll leave a slit in the sleeves,
possibly make a different type of smocking (depending on whether I have
enough fabric) and then line them like the neck so that they are stable.
Because, yes, I think that there will be another time: I have a few more
project before that, and I want to spend maybe another year working from
my stash, but then I think I ll buy some soft linen and make at least
another one, maybe with white-on-white smocking so that it will be
easier to match with different garments.
We had 886 responses with status code 200 and 64 connections closed.
The backend terminated 64 connections while the load balancer still had
active requests directed to it.
Let's change the Load Balancer 
Great! Now all the requests worked.
This is a common issue, not specific to Envoy Proxy or the setup
shown earlier. Major cloud providers have all documented it.
Even thought the success rate is 100%, the status code distribution
show some responses with status code 503. This is the case where is not
that obvious that the problem is related to idle timeout.
However, it's clear when we look the Envoy access logs:
Key Takeaways:
A shot of Jewel Changi. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
A shot of Sentosa Boardwalk. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
This is the hawker center we went to. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Table littering at the hawker center was prohibited by law. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Merlion from behind, giving a good view of Marina Bay Sands. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
If you're still using Vagrant (I am) and try to boot a box that uses UEFI (like 
My trip to pgday.at started Wednesday at the airport in D sseldorf. I was there on time, and the plane started with an estimated flight time of about 90 minutes. About half an hour into the flight, the captain announced that we would be landing in 30 minutes - in D sseldorf, because of some unspecified technical problems. Three hours after the original departure time, the plane made another attempt, and we made it to Vienna.
About 95% of my Debian contributions this month were
