At May First we have been carefully planning our migration of about 1200 lists from mailman2 to mailman3 for almost six months now. We did a lot of user communications, had several months of beta testing with a handful of lists ported over, and everything was looking good. So we kicked off the migration! But, about 15% of the way through I started seeing sqlite lock errors. Wait, what? I carefully re-configured mailman3 to use postgres, not sqlite. Well, yes, but apparently that was for the database managing the email list configuration, not the database powering the django web app, which, incidentally, also includes hundresds of gigabytes of archives. In other words, the one we really need in postgres, not sqlite.

Moving from sqlite to postgres Well that sucks. We immediately stopped the migration to deal with this. I noticed that the web is full of useful django instructions on how to migrate your database from one database to antoher. However, if you read the fine print, those convenient looking dumpdata loaddata workflows are designed to move the table definitions and a small amount of data. In our case, even after just 15% of our lists moved, our sqlite database was about 30GB. I considered some of the hacks to manage memory and try to run this via django, but eventually decided that pgloader was a more robust option. This option also allowed me to more easily test things out on a copy of our sqlite database (made while mailman was turned off). This way I could migrate and re-migrate the sqlite database over and over without impacting our live installation until I was satisfied it was all working. My first decision was to opt out of pgloader s schema creation. I used django s schema creation tool by:

• Turning off mailman3 and mailman3-web and changing the mailman web configuration to use the new postgresql database.
• Running mailman-web migrate
• Changing the mailman web configuration back to sqlite and starting everything again.

Note: I tried just adding new database settings in the mailman web configuration indexed to new - django has the ability to define different databases by name, then you can run mailman-web migrate --database new. But, during the migration, I caught django querying the sqlite database for some migrations that required referencing existing fields (specifically hyperkitty s 0003_thread_starting_email). I didn t want any of these steps to touch the live database so I opted for the cleaner approach. Once I had a clean postgres schema, I dumped it so I could easily return to this spot. Next I started working on our pgloader load file. After a lot of trial and error, I ended with:

LOAD DATABASE
    FROM sqlite:///var/lib/mailman3/sqlite-postgres-migration/mailman3web.clean.backup.db
    INTO postgresql://mailmanweb:xxxxxxxxxxx@localhost:5432/mailmanweb

WITH data only,
    reset sequences,
    include no drop,
    disable triggers,
    create no tables,
    batch size = 5MB,
    batch rows = 500,
    prefetch rows = 50,
    workers = 2,
    concurrency = 1

SET work_mem to '64MB',
    maintenance_work_mem to '512MB'

CAST type datetime to timestamptz drop default drop not null,
    type date to date drop default drop not null,
    type int when (= precision 1) to boolean using tinyint-to-boolean,
    type text to varchar using remove-null-characters;

The batch, prefetch, workers and concurreny settings are all there to ensure memory doesn t blow up. I also discovered that I had to make some changes to the schema before loading data. Mostly truncating tables that the django migrate command populated to avoid duplicate key errors:

TRUNCATE TABLE django_migrations CASCADE;
TRUNCATE TABLE django_content_type CASCADE;
TRUNCATE TABLE auth_permission CASCADE;
TRUNCATE TABLE django_site CASCADE;

And also, I had to change a column type. Apparently the mailman import process allowed an attachment file name that exceeds the limit for postgres, but was allowed into sqlite:

ALTER TABLE hyperkitty_attachment ALTER COLUMN name TYPE text

When pgloader runs, we still get a lot of warnings from pgloader, which wants to cast columns differently than django does. These are harmless (I was able to import the data without a problem). And there are still a lot of warnings along the lines of:

2026-03-30T14:08:01.691990Z WARNING PostgreSQL warning: constraint hyperkitty_vote_email_id_73a50f4d_fk_hyperkitty_email_id of relation hyperkitty_vote does not exist, skipping

These are harmless as well. They appear because disable triggers disables foreign key constraints. Without it, we wouldn t be able to load tables that require values in tables that have not yet been populated. After all the tweaking, the import of our 30GB sqlite database took about 40 minutes.

Final Steps I think the reset sequences from pgloader should take care of this, but just in case:

mailman-web sqlsequencereset hyperkitty mailman_django auth   mailman-web dbshell

And, just to ensure postgres is optimized, run this in the psql shell:

ANALYZE VERBOSE;

Last thoughts I understand very well all the decisions the mailman3 devs made in designing the next version of mailman, and if I was in the same place I may have made them the same ones. For example, separating the code running the mailing list from the code managing the archives and the web interface makes perfectly good sense - many people might want to run just the mailing list part without a web interface. And building the web interface in django makes a lot of sense as well - why re-invent the wheel? I m sure a lot of time and effort was saved by simply using the built in features you get for free with django. But the unfortunate consequence of these decisions is that sys admins have a much harder time. Almost everyone wants the email lists along with the web interface and the archives. But nobody wants two different configuration files with different syntaxes and logic, not to mention two different command lines to use for maintenance and configuration with completely different APIs. Trying to understand how to change a default template or set list defaults requires a lot of research and usually you have to write a python script to do it. I have finally come to the conclusion that mailman2 is designed for sys admins, while mailman3 is designed for developers. Despite these short comings, I am impressed with the community and their quick and friendly responses to the questions of a confused sys admin. That might be more valuable than anything else.

It s well known and universally agreed that radios are cool. Among the contested field of coolest radios, Software Defined Radios (SDRs) are definitely the most interesting to me. Out of all of my (entirely too many) SDRs I own, the rtlsdr is still my #1. It s just good. It s a great price, extremely capable, reliable, well-supported, and compact. Why bother with anything else? Sure, it can t transmit, uses a (fairly weird) 8 bit unsigned integer IQ representation, limited sampling rate, limited frequency range but even with all that, it s still the radio I will pack first. Don t get me wrong, I love my Ettus radios, PlutoSDRs, HackRFs, my AirspyHF+ - they re great! I just always find myself falling back to an rtl-sdr, every time. Perhaps the best reason to use an rtlsdr is the absolutely mind-boggling amount of cool stuff people have written for it. The rtlsdr API is super easy to use, widely supported if you re building on top of existing radio processing frameworks it s still a shock to me when something omits rtlsdr support.

sparky Over the last 7 years, I ve been learning about radios I got my ham radio license (de K3XEC), hacked on some cool stuff where I ve learned how radios work by doing , and even was lucky enough to give my first rf-centric talk at districtcon. Embarrassingly, I still haven t gotten around to learning how the fancy stuff like GNU Radio works. I m sure I m going to love it when I do. As part of this, I ve also cooked up some very unprofessional formats and protocols I use for convenience. Locally, all my on-disk captures are stored in rfcap or more recently arf, while direct SDR access at my house is almost entirely a mix of the widely used rtl-tcp protocol, and my riq protocol (post on this coming soon). Both rtl-tcp and riq operate over the network, so I don t have to bother with plugging things into USB ports, and I can share my radios with my friends. All of that work sits in my current generation of radio processing code, sparky (a reference to spark-gap transmitters), which is a heap of Rust, supporting everything from no_std for embedded experiments, conditional support for interfacing with all the radios I own, and tokio-based async support in addition to blocking i/o for highly concurrent daemons. This quickly advanced beyond my old Go-based code (hz.tools/go-sdr), which I archived so I can focus on learning. I still think Go is a great language to write RF code in but I can t focus on that tech tree anymore. Of course, this now poses a new problem no one supports my format(s) or radio protocol(s), since, well, I m the only one using them. I ve committed a fair amount of my hardware to this setup, and yanking it from the rack to try something out does pose a bit of a pickle. This isn t a huge deal for learning, but it does make it tedious to try out something from the internets.

librtlsdr.so Thankfully, Rust has robust support for wrap[ping itself] in a grotesque simulacra of C s skin and mak[ing its] flesh undulate, which is an attractive nuisance if i ve ever seen one. Naturally, my ability to restrain myself from engaging in ill-advised rf adventures is basically zero, so it s time to do the thing any similarly situated person would do reimplement the API and ABI of librtlsdr.so, backed with sparky instead. Since enumeration of devices is going to be annoying (specifically, they re over the network), I decided early-on to rely on an explicit list of devices via a configuration file. I d rather only load that once so programs don t get confused, so I opted to use a CTOR to run a stub when the ELF is linked at runtime.

// lightly edited for clarity

#[used]
#[expect(unused)]
#[unsafe(link_section = ".init_array")]
pub static INITIALIZE: extern "C" fn() = sparky_rtlsdr_ctor;

#[unsafe(no_mangle)]
pub extern "C" fn sparky_rtlsdr_ctor()  
 let config: Config =  
 if let Ok(config_bytes) = std::fs::read("/etc/sparky-rtlsdr.toml")  
 toml::from_slice(&config_bytes).unwrap()
   else  
 Config   device: vec![]  
  
  ;
 CONFIG.set(config);
 

Next, it s time to start with the basics. Opening and closing a handle using rtlsdr_open and rtlsdr_close. Given we don t control the runtime, and the rtl-sdr device handle is opaque (for good reason!), I opted to smuggle a rust Box<Device> non-FFI safe heap-allocated struct through the device handle pointer, and let C take ownership of the Box. No one should be looking in there anyway.

// lightly edited for clarity

#[unsafe(no_mangle)]
pub unsafe extern "C" fn rtlsdr_open(dev: *mut *mut Handle, index: u32) -> int  
 let config = &CONFIG.device[index as usize];
 let sdr = match config.load()  
 Ok(v) => v,
 Err(err) =>  
 return -1;
  
  ;
 let handle = Box::new(Handle   config, sdr  );
 unsafe   *dev = Box::into_raw(handle)  ;
 0
 

#[unsafe(no_mangle)]
pub unsafe extern "C" fn rtlsdr_close(dev: *mut Handle) -> int  
 let dev = unsafe   Box::from_raw(dev)  ;
 drop(dev);
 0
 

With that in place, we can chip away at the API surface, translating calls as best as we can. I won t bother listing it all, since it s not very interesting but here s an example implementation of rtlsdr_set_sample_rate and rtlsdr_get_sample_rate. These calls are translating from an rtl-sdr frequency (which is a u32 containing the value as Hz) into a sparky Frequency type, and invoking get_sample_rate or set_sample_rate on the device s rust handle. Since each device implements the sparky Sdr trait, the actual underlying device doesn t matter much here.

#[unsafe(no_mangle)]
pub unsafe extern "C" fn rtlsdr_set_sample_rate(dev: *mut Handle, rate: u32) -> int  
 let dev = unsafe   &mut *dev  ;
 let rate = Frequency::from_hz(rate as i64);
 if let Err(err) = dev.sdr.set_sample_rate(dev.channel, rate)  
 return -1;
  
 0
 

#[unsafe(no_mangle)]
pub unsafe extern "C" fn rtlsdr_get_sample_rate(dev: *mut Handle) -> u32  
 let dev = unsafe   &mut *dev  ;
 let freq = match dev.sdr.get_sample_rate(dev.channel)  
 Ok(freq) => freq,
 Err(err) =>  
 return 0;
  
  ;
 freq.as_hz() as u32
 

After repeating this process for the rest of the stubs I could (and otherwise setting error conditions if the functionality is not supported), I was ready to try it out. Within sparky, I patched my MockSDR (basically a Sdr traited Mock type) to implement the same testmode IQ protocol that the RTL-SDR has, and decided to see if rtl_test from apt without any changes could be fooled.

$ rtl_test
No supported devices found.

Great, cool. No devices plugged in. Looks great. Let s try it with my librtlsdr.so LD_PRELOAD-ed into the binary first:

$ LD_PRELOAD=target/release/librtlsdr.so rtl_test
Found 1 device(s):
 0: hz.tools, mock sdr, SN: totally legit no tricks

Using device 0: sparky mock sdr
Supported gain values (0):
Sampling at 2048000 S/s.

Info: This tool will continuously read from the device, and report if
samples get lost. If you observe no further output, everything is fine.

Reading samples in async mode...
^CSignal caught, exiting!

User cancel, exiting...
Samples per million lost (minimum): 0
$

Outstanding. Even more outstandingly, if I change my testmode implementation to skip samples, rtl_test correctly reports the errors I think it s showing promise! On to try the real endgame here let s have our new librtlsdr.so connect to an rtl-tcp endpoint and see if rtl_fm works:

LD_PRELOAD=target/release/librtlsdr.so \
 rtl_fm -d 1 -s 120k -E deemp -M fm -f 90.9M   \
 ffplay -f s16le -ar 120k -i -
Found 2 device(s):
 0: hz.tools, mock sdr, SN: totally legit no tricks
 1: hz.tools, rtl-tcp, SN: node2.rf.lan:1202

Using device 1: sparky rtltcp node2
Tuner gain set to automatic.
Tuned to 91170000 Hz.
Oversampling input by: 9x.
Oversampling output by: 1x.
Buffer size: 7.59ms
Sampling at 1080000 S/s.
Output at 120000 Hz.

And there it was! Not the best audio quality (mostly due to my inability to correctly read the rtl_fm manpage to tune the filter and downsample/oversampling rates to audio), but it s definitely passable. I figured I d try something that was a bit more interesting next gqrx, since it s super handy, I use it a ton, and will definitely amuse me to no end. To my surprise and delight, LD_PRELOAD=target/release/librtlsdr.so gqrx wound up running, and I saw my devices pop right up in the setting menu: Huge. Huge. Amazing. It did crash as soon as I tried to actually use the radio, but after fixing a few dangling bugs in the API surface (and some assumptions I think some underlying gnuradio driver may be making that I need to double check in the code), I was able to get a super solid stream of broadcast fm radio, with gqrx being none the wiser. It thought it was just talking to the device it knows as rtl=1. Nice. I can t wait to try this with the rest of the rtl-sdr based tools I like having around using my riq protocol next. I don t think that ll be worth a post, but hopefully I ll get around to publishing details on that stack next.

epilogue Well. That s it. End of story. A bit anti-climatic, sure. While this new shim will provide me endless minutes of mild amusement, I could see using this to expose my sparky testing utilities via librtlsdr.so my mock sdr driver allows for replaying captures off disk, which could be interesting to make sure that signals are still properly decoded after changes, or instrument performance changes (via SNR, BER, packets observed, etc) on reference samples I have on my NAS. Maybe that ll come in handy one day! Truth be told, I m not sure I actually want to encourage anyone to do this for real (although I think I ll definitely be using it on my LAN to see what happens). I also don t have a repo to share I don t particularly feel with dealing with the secondary effects of publishing sparky (and sparky-rtlsdr) yet, since i m still getting my feet under me on the radio aspect of all this. I ll be sure to post updates if anything changes with this here (tagged sparky) and at @paul@soylent.green. I can t wait to post more about some of the odd sidequests (like this one!) i ve completed over the last few years I ve been waiting to feel confident that my work has matured and was withstood the new problems i ve thrown at it, and it largely has. It s my hope that these projects (and this project in particular) has provided a glimpse into the world of software defined radio for my systems friends, and a bit about systems for my radio friends. It s not all magic, and I hope someone out there feels inclined to have some fun with radios themselves!

AI sure is a hot topic right now, and I see a lot of people arguing about it. To a lot of people around here, I m the computer person they know and I get asked a lot about AI. I m going to suggest a lot of things can be true at once. For instance:

• LLMs are changing how we work and will continue to do so.
• LLMs are vastly over-hyped by vested interests, and may be in a bubble.

Or how about:

• Huge investment in GenAI is having many negative consequences, ranging from environmental to causing affordability problems in many industries that use hardware (ie, everywhere)
• Useful results can be had from models that run on local hardware, even battery-powered hardware, which may have negligible harm or even some benefit

And:

• GenAI is further concentrating wealth and power in megacorps, with the effect of squeezing out the smaller players even more.
• GenAI is lowering the cost of entry for people without a lot of resources already.

I have sympathy for the naysayers; those that say it s nothing but a stochastic parrot. But I don t have a lot of sympathy for the naysayers that deny ever using it; you can t form a credible argument against something without having an understanding of it informed by experience. I also have sympathy for the cheerleaders. I have seen some impressive things from AI; for instance, a story from an engineer who has a child with a rare disease without a credible cure. The engineer did a lot of research on it, started feeding research papers into AI to analyze, and the AI started finding correlations between different areas of research that humans hadn t yet found leading to a positive result for the child. To be fair, I have rarely seen an AI deliver a 100% correct answer on anything with any real level of complexity. I have seen it both waste more time than it saves, and save a ton of time. My point here is: It is neither always fantastic nor always terrible. Let me talk you through an example. I am a fan of inbox zero for email. That is, the inbox should be empty. Unfortunately, mine has 8000 messages in it. According to the oldest messages in my inbox, I last had inbox zero 8 years ago. But really, only a handful are older than 2020. I guess something must have happened that year I ve been chipping away at this for quite some time now. The problem is, there are certain emails in there that really do still need some action maybe it s photos to save off into our photo collection, for instance. But when looking at things sorted by date or thread, there are old shipping confirmations next to phishing attempts and family photos. One can t just scan down the list. I ve tried all the usual tricks, most of which involve selecting groups of message that are easy to bulk erase, or at least easy to scan visually for the occasional thing worth saving. Sort by sender or subject line, for instance. Then I can, for instance, delete all the old messages from the shopping sites I commonly use all at once. But then they start using different senders and different subject lines and that doesn t get all of them. I ve tried keyword searches for this sort of thing too. Still, that got me down to about 8000 messages. So I thought: why not see if an LLM could help me classify these? Maybe it could categorize them, and then I could look at emails grouped by category. I have one machine with a discrete GPU, an Nvidia RTX 4070. It s a desktop machine I don t use all that often. But I set up Ollama on it, running in a Docker container. Ollama runs models locally. I should also mention at this point that we are solar-powered, and this time of year is a time of peak production of excess solar, because it is sunny and not much heat or AC is required. So that machine is solar-powered and isn t causing environmental harm. In any case, charging the EV uses much more power than that GPU. I figured I would do this in two passes. First, ask the LLM to classify each message (or a sampling of them would probably work too), letting it pick its own categories for each. Then, look at the patterns that emerge and give it a single, much smaller, set of broad categories to use and rerun it over that. Then I can easily select messages from my Maildirs by category and process them in bulk. I used open-interpreter pointing to that GPU on my network to help me write the scripts for this. It didn t get things right on its own; for instance, it didn t call the Ollama API correctly, and insisted on appending /cur to the path to the Maildir (which was not going to fly with Python s maildir module). It took roughly an hour to classify those 8000 messages (or, as I had it do, the first 2000 characters of them), and then the same to do it a second time. I had it output lines in the form of filename\tcategory and hand-wrote the shell script that processed those. In the end, was it useful? Yes, quite. Its classifications weren t perfect (and it didn t even follow my prompt perfectly; sometimes it would give me a long discussion on why it picked a certain category rather than just that category, and occasionally it picked categories not on the list). But then, neither were my manual keyword searches. So far I ve gotten rid of nearly 1000 more messages. Several categories were a visual scan for sanity and then delete all sort of thing. My emails never left my network. I didn t rely on a cloud AI to process them. I didn t contribute to global warming (this may have even been a case of saving energy, since it no doubt will offset quite a bit of manual time that would keep screens and room lights energized and so forth). I used about as much energy as watching a movie on a TV. Did it complete the task for me entirely autonomously? Also no. AI isn t a mind reader and it can t possibly evaluate exactly what my thought process would be for a given task. But it can do a decent enough job to save me some time. Still, this didn t require hyperscaler datacenters. AI even runs on-phone (Google Translate being one of the most useful AI-driven apps I ve ever seen, and it can run on-device).

Note: I have not published blog posts about my academic papers over the past few years. To ensure that my blog contains a more comprehensive record of my published papers and to surface them for folks who missed them, I will periodically (re) publish blog posts about some older published projects. This post draws material from a previously published post by Kaylea Champion on the Community Data Science Blog.

Taboo subjects such as sexuality and mental health are as important to discuss as they are difficult to raise in conversation. Although many people turn to online resources for information on taboo subjects, censorship and low-quality information are common in search results. In two papers I recently published at CSCW both led by Kaylea Champion we presented a series of analyses showing how taboo shapes the process of collaborative knowledge building on English Wikipedia.

The first study is a quantitative analysis showing that articles on taboo subjects are much more popular and are the subject of more vandalism than articles on non-taboo topics. In surprising news, we also found that they were edited more often and were of higher quality!

Short video of Kaylea s presentation of the work given at Wikimania in August 2023.

The first challenge we faced in conducting this work was identifying taboo articles. Kaylea had a brilliant idea for a new computational approach to doing so without relying on our individual intuitions about what qualifies as taboo (something we understood would be highly specific to our own culture, class, etc). Her approach was to make use of an insight from linguistics: people develop euphemisms as ways to talk about taboos (i.e., think about all the euphemisms we ve devised for death, or sex, or menstruation, or mental health).

We used this insight to build a new machine-learning classifier based on English Wiktionary definitions. If a sense of a word was tagged as euphemistic, we treated the words in the definition as indicators of taboo. The end result was a series of words and phrases that most powerfully differentiate taboo from non-taboo. We then did a simple match between those words and phrases and the titles of Wikipedia articles. The topics were taboo enough that we were a little uncomfortable discussing them in our meetings! We built a comparison sample of articles whose titles are words that, like our taboo articles, appear in Wiktionary definitions. In the first paper, we used this new dataset to test a series of hypotheses about how taboo shapes collaborative production in Wikipedia. Our initial hypotheses were based on the idea that taboo information is often in high demand but that Wikipedians might be reluctant to associate their names (or usernames) with taboo topics. The result, we argued, would be articles that were in high demand but of low quality. We found that taboo articles are thriving on Wikipedia! In summary, we found that in comparison to non-taboo articles:

• Taboo articles are more popular (as expected).
• Taboo articles receive more contributions (contrary to expectations).
• Taboo articles receive more low-quality contributions (as expected).
• Taboo articles are higher quality (contrary to expectations).
• Taboo article contributors are more likely to contribute without an account (as expected), and have less experience (as expected), but that accountholders are more likely to make themselves more identifiable by having a user page, disclosing their gender, and making themselves emailable (all three of these are contrary to expectation!).

Image of the estimated qualiy of articles of the four articles in the second mixed-methods paper. Extreme dips reflect periods of frequent vandalism.

Kaylea attempted to understand these somewhat confusing results by designing a fantastic mixed-methods analysis that sought to unpack some of the nuance missing in the quantitative analysis by delving deep into the life histories of four articles on English Wikipedia: two on taboo topics related to women s anatomy (Clitoris and Menstration) and two nontaboo articles chosen for comparison (Cell membrance and Philip Pullman). Although the findings from the analysis can be difficult to summarize succinctly (as with many qualitative studies), we showed how the taboo example articles success was hard-won amid real challenges and attacks. The paper describes how challenges were overcome through resilient leadership, often provided by a single dedicated individual. The paper provides a template for how taboo can be and frequently is overcome by dedicated Wikipedians in ways that provide useful knowledge resources in real demand. For more details, visualizations, statistics, and more, we hope you ll take a look at our papers, both linked below.

---

The full citation for the papers are: (1) Champion, Kaylea, and Benjamin Mako Hill. 2023. Taboo and Collaborative Knowledge Production: Evidence from Wikipedia. Proceedings of the ACM on Human-Computer Interaction 7 (CSCW2): 299:1-299:25. https://doi.org/10.1145/3610090. (2) Champion, Kaylea, and Benjamin Mako Hill. 2024. Life Histories of Taboo Knowledge Artifacts. Proceedings of the ACM: Human-Computer Interaction 8 (CSCW2): 505:1-505:32. https://doi.org/10.1145/3687044.

We have also released replication materials for the paper, including all the data and code used to conduct the analyses.

This blog post and the paper it describes are collaborative work by Kaylea Champion and Benjamin Mako Hill.

I often need a quick calculation or a unit conversion. Rather than reaching for a separate tool, a few lines of Zsh configuration turn = into a calculator. Typing = 660km / (2/3)c * 2 -> ms gives me 6.60457 ms¹ without leaving my terminal, thanks to the Zsh line editor.

The equal alias The main idea looks simple: define = as an alias to a calculator command. I prefer Numbat, a scientific calculator that supports unit conversions. Qalculate is a close second.² If neither is available, we fall back to Zsh s built-in zcalc module. As the alias built-in uses = as a separator for name and value, we need to alter the aliases associative array:

if (( $+commands[numbat] )); then
  aliases[=]='numbat -e'
elif (( $+commands[qalc] )); then
  aliases[=]='qalc'
else
  autoload -Uz zcalc
  aliases[=]='zcalc -f -e'
fi

With this in place, = 847/11 becomes numbat -e 847/11.

The quoting problem The first problem surfaces quickly. Typing = 5 * 3 fails: Zsh expands the * character as a glob pattern before passing it to the calculator. The same issue applies to other characters that Zsh treats specially, such as > or . You must quote the expression:

$ = '5 * 3'
15

We fix this by hooking into the Zsh line editor to quote the expression before executing it.

Automatic quoting with ZLE Zsh calls the line-finish widget before submitting a command. We hook a function that detects the = prefix and quotes the expression:

_vbe_calc_quote()  
  case $BUFFER in
    "="*)
      typeset -g _vbe_calc_expr=$BUFFER # not used yet
      BUFFER="= $ (q-)$ $ BUFFER#= #  "
      ;;
  esac
 
add-zle-hook-widget line-finish _vbe_calc_quote

When you type = 5 * 3 and press , _vbe_calc_quote strips the = prefix, quotes the remainder with the (q-) parameter expansion flag, and rewrites the buffer to = '5 * 3' before Zsh submits the command. As a bonus, you can save a few keystrokes with =5*3! You can now compute math expressions and convert units directly from your shell. Zsh automatically quotes your expressions:

$ = '1 + 2'
3
$ = 'pi/3 + pi  > cos'
-0.5
$ = '17 USD -> EUR'
14.7122  
$ = '180*500mg -> g'
90 g
$ = '5 gigabytes / (2 minutes + 17 seconds) -> megabits/s'
291.971 Mbit/s
$ = 'now() -> tz("Asia/Tokyo")'
2026-03-22 22:00:03 JST (UTC +09), Asia/Tokyo
$ = '1 / (40 rods / hogshead) -> L / 100km'
118548   0.01 l/km

The metric system is the tool of the devil! My car gets forty rods to the hogshead, and that's the way I like it! Grampa Simpson, A Star Is Burns

Storing unquoted history As is, Zsh records the quoted expression in history. You must unquote it before submitting it again. Otherwise, the ZLE widget quotes it a second time. Bart Schaefer provided a solution to store the original version:

_vbe_calc_history()  
  return $ +_vbe_calc_expr 
 
add-zsh-hook zshaddhistory _vbe_calc_history
_vbe_calc_preexec()  
  (( $ +_vbe_calc_expr  )) && print -s $_vbe_calc_expr
  unset _vbe_calc_expr
  return 0
 
add-zsh-hook preexec _vbe_calc_preexec

The zshaddhistory hook returns 1 if we are evaluating an expression, telling Zsh not to record the command. The preexec hook then adds the original, unquoted command with print -s.

---

The complete code is available in my zshrc. A common alternative is the noglob precommand modifier. If you stick with to instead of -> for unit conversion, it covers 90% of use cases. For a related Zsh line editor trick, see how I use auto-expanding aliases to fix common typos.

---

1. This is the fastest a packet can travel back and forth between Paris and Marseille over optical fiber.
2. Qalculate is less understanding with units. For example, it parses Mbps as megabarn per picosecond:

   $ numbat -e '5 MB/s -> Mbps'
   40 Mbps
   $ qalc 5 MB/s to Mbps
   5 megabytes/second = 0.000005 B/ps
   

The
WWW::Mechanize::Chrome Saga: A Comprehensive Narrative of PR #104 This document synthesizes the extensive work performed from March
13th to March 20th, 2026, to harden, stabilize, and refactor the
WWW::Mechanize::Chrome library and its test suite. This
effort involved deep dives into asynchronous programming,
platform-specific bug hunting, and strategic architectural
decisions.

---

Part I:
The Quest for Cross-Platform Stability (March 13 16) The initial phase of work focused on achieving a green test suite
across a variety of Linux distributions and preparing for a new release.
This involved significant hardening of the library to account for
different browser versions, OS-level security restrictions, and
filesystem differences.

Key Milestones &
Engineering Decisions:

• Fedora & RHEL-family Success: A major effort
  was undertaken to achieve a 100% pass rate on modern Fedora 43 and
  CentOS Stream 10. This required several key engineering decisions to
  handle modern browser behavior:
  • Decision: Implement Asynchronous DOM Serialization
    Fallback. Synchronous fallbacks in an async context are
    dangerous. To prevent Resource was not cached errors during
    saveResources, we implemented a fully asynchronous fallback
    in _saveResourceTree. By chaining
    _cached_document with DOM.getOuterHTML
    messages, we can reconstruct document content without blocking the event
    loop, even if Chromium has evicted the resource from its cache. This
    also proved resilient against Fedora s security policies, which often
    block file:// access.
  • Decision: Truncate Filenames for Cross-Platform
    Safety. To avoid File name too long errors,
    especially on Windows where the MAX_PATH limit is 260
    characters, filenameFromUrl was hardened. The filename
    truncation was reduced to a more conservative 150
    characters, leaving ample headroom for deeply nested CI
    temporary directories. Logic was also added to preserve file extensions
    during truncation and to sanitize backslashes from URI paths.
  • Decision: Expand Browser Discovery Paths. To
    support RHEL-based systems out-of-the-box, the
    default_executable_names was expanded to include
    headless_shell and search paths were updated to include
    /usr/lib64/chromium-browser/.
  • Decision: Mitigate Race Conditions with Stabilization Waits
    and Resilient Fetching. On fast systems,
    DOM.documentUpdated events could invalidate
    nodeIds immediately after navigation, causing XPath queries
    to fail with Could not find node with given id . A small stabilization
    sleep(0.25s) was added after page loads to ensure the DOM
    is settled. Furthermore, the asynchronous DOM fetching loop was hardened
    to gracefully handle these errors by catching protocol errors and
    returning an empty string for any node that was invalidated during
    serialization, ensuring the overall process could complete.
• Windows Hardening:
  • Decision: Adopt Platform-Aware Watchdogs. The test
    suite s reliance on ualarm was a blocker for Windows, where
    it is not implemented. The t::helper::set_watchdog function
    was refactored to use standard alarm() (seconds) on Windows
    and ualarm (microseconds) on Unix-like systems, enabling
    consistent test-level timeout enforcement.
• Version 0.77 Release:
  • Decision: Adopt SOP for Version Synchronization.
    The project maintains duplicate version strings across 24+ files. A
    Standard Operating Procedure was adopted to use a batch-replacement tool
    to update all sub-modules in lib/ and to always run
    make clean and perl Makefile.PL to ensure
    META.json and META.yml reflect the new
    version. After achieving stability on Linux, the project version was
    bumped to 0.77.
• Infrastructure & Strategic Work:
  • The ad2 Windows Server 2025 instance was restored and
    optimized, with Active Directory demoted and disk I/O performance
    improved.
  • A strategic proposal for the Heterogeneous Directory
    Replication Protocol (HDRP) was drafted and published.

---

Part II: The
Great Async Refactor (March 17 18) Despite success on Linux, tests on the slow ad2 Windows
host were still plagued by intermittent, indefinite hangs. This
triggered a fundamental architectural shift to move the library s core
from a mix of synchronous and asynchronous code to a fully non-blocking
internal API.

Key Milestones &
Engineering Decisions:

• Decision: Expose a _future API.
  Instead of hardcoding timeouts in the library, the core strategy was to
  refactor all blocking methods (xpath, field,
  get, etc.) into thin wrappers around new non-blocking
  ..._future counterparts. This moved timeout management to
  the test harness, allowing for flexible and explicit handling of
  stalls.

  # Example library implementation
  sub xpath($self, $query, %options)  
      return $self->xpath_future($query, %options)->get;
   
  
  sub xpath_future($self, $query, %options)  
      # Async implementation using $self->target->send_message(...)
   

• Decision: Centralize Test Hardening in a Helper.
  A dedicated test library, t/lib/t/helper.pm, was created to
  contain all stabilization logic. Safe wrappers (safe_get,
  safe_xpath) were implemented there, using
  Future->wait_any to race asynchronous operations against
  a timeout, preventing tests from hanging.

  # Example test helper implementation
  sub safe_xpath  
      my ($mech, $query, %options) = @_;
      my $timeout = delete $options timeout    5;
      my $call_f = $mech->xpath_future($query, %options);
      my $timeout_f = $mech->sleep_future($timeout)->then(sub   Future->fail("Timeout")  );
      return Future->wait_any($call_f, $timeout_f)->get;
   

• Decision: Refactor Node Attribute Cache.
  Investigations into flaky checkbox tests (t/50-tick.t)
  revealed that WWW::Mechanize::Chrome::Node was storing
  attributes as a flat list ([key, val, key, val]), which was
  inefficient for lookups and individual updates. The cache was refactored
  to definitively use a HashRef, providing O(1) lookups
  and enabling atomic dual-updates where both the browser property (via
  JS) and the internal library attribute are synchronized
  simultaneously.
• Decision: Implement Self-Cancelling Socket
  Watchdog. On Windows, traditional watchdog processes often
  failed to detect parent termination, leading to 60-second hangs after
  successful tests. We implemented a new socket-based watchdog in
  t::helper that listens on an ephemeral port; the background
  process terminates immediately when the parent socket closes,
  eliminating these cumulative delays.
• Decision: Deep Recursive Refactoring & Form
  Selection. To make the API truly non-blocking, the entire
  internal call stack had to be refactored. For example, making
  get_set_value_future non-blocking required first making its
  dependency, _field_by_name, asynchronous. This culminated
  in refactoring the entire form selection API (form_name,
  form_id, etc.) to use the new asynchronous
  _future lookups, which was a key step in mitigating the
  Windows deadlocks.
• Decision: Fix Critical Regressions & Memory
  Cycles.
  • Evaluation Normalization: Implemented a
    _process_eval_result helper to centralize the parsing of
    results from Runtime.evaluate. This ensures consistent
    handling of return values and exceptions between synchronous
    (eval_in_page) and asynchronous (eval_future)
    calls.
  • Memory Cycle Mitigation: A significant memory
    leak was discovered where closures attached to CDP event futures (like
    for asynchronous body retrieval) would capture strong references to
    $self and the $response object, creating a
    circular reference. The established rule is to now always use
    Scalar::Util::weaken on both $self and any
    other relevant objects before they are used inside a
    ->then block that is stored on an object.
  • Context Propagation (wantarray): A
    major regression was discovered where Perl s wantarray
    context, which distinguishes between scalar and list context, was lost
    inside asynchronous Future->then blocks. This caused
    methods like xpath to return incorrect results (e.g., a
    count instead of a list of nodes). The solution was to adopt the Async
    Context Pattern : capture wantarray in the synchronous
    wrapper, pass it as an option to the _future method, and
    then use that captured value inside the future s final resolution
    block.

    # Synchronous Wrapper
    sub xpath($self, $query, %options)  
        $options  wantarray   = wantarray; # 1. Capture
        return $self->xpath_future($query, %options)->get; # 2. Pass
     
    
    # Asynchronous Implementation
    sub xpath_future($self, $query, %options)  
        my $wantarray = delete $options  wantarray  ; # 3. Retrieve
        # ... async logic ...
        return $doc->then(sub  
            if ($wantarray)   # 4. Respect
                return Future->done(@results);
              else  
                return Future->done($results[0]);
             
         );
     

  • Asynchronous Body Retrieval & Robust Content
    Fallbacks: Fixed a bug where decoded_content()
    would return empty strings by ensuring it awaited a
    __body_future. This was implemented by storing the
    retrieval future directly on the response object
    ($response-> __body_future ). To make this more robust,
    a tiered strategy was implemented: first try to get the content from the
    network response, but if that fails (e.g., for about:blank
    or due to cache eviction), fall back to a JavaScript
    XMLSerializer to get the live DOM content.
  • Signature Hardening: Fixed Too few arguments
    errors when using modern Perl signatures with
    Future->then. Callbacks were updated to use optional
    parameters (sub($result = undef) ... ) to gracefully
    handle futures that resolve with no value.
  • XHTML Split-Brain Bug: Resolved a
    long-standing Chromium bug (40130141) where content provided via
    setDocumentContent is parsed differently than content
    loaded from a URL. A workaround was implemented: for XHTML documents,
    WMC now uses a JavaScript-based XPath evaluation
    (document.evaluate) against the live DOM, bypassing the
    broken CDP search mechanism.

Derived Architectural Rules
& SOPs:

• Rule: Always provide _future variants.
  Every library method that interacts with the browser via CDP must have a
  non-blocking asynchronous counterpart.
• Rule: Centralize stabilization in the test layer.
  All timeout and retry logic should reside in the test harness
  (t/lib/t/helper.pm), not in the core library.
• Rule: Explicitly propagate wantarray
  context. Synchronous wrappers must capture the caller s context
  and pass it down the Future chain to ensure correct
  scalar/list behavior.
• Rule: The entire call chain must be asynchronous.
  To enable non-blocking timeouts, even a single hidden blocking call in
  an otherwise asynchronous method will cause a stall.
• SOP: Reduce Library Noise. Diagnostic messages
  (warn, note, diag) should be
  removed from library code before commits. All such messages should be
  converted to use the internal $self->log('debug', ...)
  mechanism, ensuring a clean TAP output for CI systems.

---

Part III: The
MutationObserver Saga (March 19) With most of the library refactored to be asynchronous, one stubborn
test, t/65-is_visible.t, continued to fail with timeouts.
This led to an ambitious, but ultimately unsuccessful, attempt to
replace the wait_until_visible polling logic with a more
modern MutationObserver.

Key Milestones & Challenges:

• The Theory: The goal was to replace an inefficient
  repeat sleep loop with an event-driven
  MutationObserver in JavaScript that would notify Perl
  immediately when an element s visibility changed.
• Implementation & Cascade Failure: The
  implementation proved incredibly difficult and introduced a series of
  new, hard-to-diagnose bugs:
  1. An incorrect function signature for
     callFunctionOn_future.
  2. A critical unit mismatch, passing seconds from Perl to JavaScript s
     setTimeout, which expected milliseconds.
  3. A fundamental hang where the MutationObserver s
     JavaScript Promise would never resolve, even after the
     underlying DOM element changed.
• Debugging Maze: Multiple attempts to fix the
  checkVisibility JavaScript logic inside the observer
  callback, including making it more robust by adding DOM tree traversal
  and extensive console.log tracing, failed to resolve the
  hang. This highlighted the opacity and difficulty of debugging complex,
  cross-language asynchronous interactions, especially when dealing with
  low-level browser APIs.

Procedural Learning:
Granular Edits The effort was plagued by procedural missteps in using automated
file-editing tools. Initial attempts to replace large code blocks in a
single operation led to accidental code loss and match failures.

• Decision: Adopt Delete, then Add Workflow.
  Following forceful user correction, a new SOP was established for all
  future modifications:
  1. Isolate: Break the file into small, manageable
     chunks (e.g., 250 lines).
  2. Delete: Perform a delete operation by replacing
     the old code block with an empty string.
  3. Add: Perform an add operation by inserting the
     new code into the empty space.
  4. Verify: Verifying each atomic step before
     proceeding. This granular process, while slower, ensured surgical
     precision and regained technical control over the large
     Chrome.pm module.

The consistent failure of the MutationObserver approach
eventually led to the decision to abandon it in favor of stabilizing the
original, more transparent implementation.

---

Part IV:
Reversion and Final Stabilization (March 20) After exhausting all reasonable attempts to fix the
MutationObserver, a strategic decision was made to revert
to the simpler, more transparent polling implementation and fix it
correctly. This proved to be the correct path to a stable solution.

Key Milestones &
Engineering Decisions:

• Decision: Perform Strategic Reversion. The
  MutationObserver implementation, when integrated via
  callFunctionOn_future with awaitPromise,
  proved fundamentally unstable. Its JavaScript promise would consistently
  fail to resolve, causing indefinite hangs. A decision was made to
  revert all MutationObserver code from
  WWW::Mechanize::Chrome.pm and restore the original
  repeat sleep polling mechanism. A stable,
  understandable solution was prioritized over an elegant but broken
  one.
• Decision: Correct Timeout Delegation in the
  Harness. The root cause of the original timeout failure was
  identified as a race condition in the t/lib/t/helper.pm
  test harness. The safe_wait_until_* wrappers were
  implementing their own timeout (via wait_any and
  sleep_future) that raced against the underlying polling
  function s internal timeout. This led to intermittent failures on slow
  machines. The helpers were refactored to delegate all timeout
  management to the library s polling functions, ensuring a
  single, authoritative timer controlled the operation.
• Decision: Optimize Polling Performance. At the
  user s request, the polling interval was reduced from 300ms to
  150ms. This modest performance improvement reduced the
  test suite s wallclock execution time by over a second while maintaining
  stability.
• Decision: Tune Test Watchdogs. The global watchdog
  timeout was adjusted to 12 seconds, specifically calculated as 1.5x the
  observed real execution time of the optimized test. This provides a
  data-driven safety margin for CI.

---

Part
V: The Last Bug A Platform-Specific Memory Leak (March 20) With all other tests passing, a single memory leak failure in
t/78-memleak.t persisted, but only on the Windows
ad2 environment. This required a different approach than
the timeout fixes.

Key Milestones:

• The Bug: A strong reference cycle involving the
  on_dialog event listener was not being broken on Windows,
  despite multiple attempts to fix it. Fixes that worked on Linux (such as
  calling on_dialog(undef) in DESTROY) were not
  sufficient on the Windows host.
• The Diagnosis: The issue was determined to be a
  deep, platform-specific interaction between Perl s garbage collector,
  the IO::Async event loop implementation on Windows, and the
  Test::Memory::Cycle module. The cycle report was identical
  on both platforms, but the cleanup behavior was different.
• Failed Attempts: A series of increasingly
  aggressive fixes were attempted to break the cycle, including:
  1. Moving the on_dialog(undef) call from
     close() to DESTROY().
  2. Explicitly deleteing the listener and callback
     properties from the object hash in DESTROY.
  3. Swapping between $self->remove_listener and
     $self->target->unlisten in a mistaken attempt to find
     the correct un-registration method.
• Pragmatic Solution: After exhausting all reasonable
  code-level fixes without a resolution on Windows, the user opted to mark
  the failing test as a known issue for that specific platform.
• Final Fix: The single failing test in
  t/78-memleak.t was wrapped in a conditional
  TODO block that only executes on Windows
  (if ($^O =~ /MSWin32/i)), formally acknowledging the bug
  without blocking the build. This allows the test suite to pass in CI
  environments while flagging the issue for future, deeper
  investigation.

---

Part VI: CI Hardening (March
20) A final failure in the GitHub Actions CI environment revealed one
last configuration flaw.

Key Milestones:

• The Bug: The CI was running
  prove --nocount --jobs 3 -I local/ -bl xt t directly. This
  command was missing the crucial -It/lib include path, which
  is necessary for test files to locate the t::helper module.
  This resulted in nearly all tests failing with
  Can't locate t/helper.pm in @INC.
• The Investigation: An analysis of
  Makefile.PL revealed a custom MY::test block
  specifically designed to inject the -It/lib flag into the
  make test command. This confirmed that
  make test is the correct, canonical way to run the test
  suite for this project.
• The Fix: The
  .github/workflows/linux.yml file was modified to replace
  the direct prove call with make test in the
  Run Tests step. This ensures the CI environment runs the
  tests in the exact same way as a local developer, with all necessary
  include paths correctly configured by the project s build system.

Final Outcome After this long and arduous journey, the
WWW::Mechanize::Chrome test suite is now stable and
passing on all targeted platforms, with known
platform-specific issues clearly documented in the code. The project is
in a vastly more robust and reliable state.

Tweet

Linux kernel security modules provide a good additional layer of security around individual programs by restricting what they are allowed to do, and at best block and detect zero-day security vulnerabilities as soon as anyone tries to exploit them, long before they are widely known and reported. However, the challenge is how to create these security profiles without accidentally also blocking legitimate actions. For MariaDB in Debian and Ubuntu, a new AppArmor profile was recently created by leveraging the extensive test suite with 7000+ tests, giving good confidence that AppArmor is unlikely to yield false positive alerts with it. AppArmor is a Mandatory Access Control (MAC) system, meaning that each process controlled by AppArmor has a sort of an allowlist called profile that defines all capabilities and file paths a program can access. If a program tries to do something not covered by the rules in its AppArmor profile, the action will be denied on the Linux kernel level and a warning logged in the system journal. This additional security layer is valuable because even if a malicious user found a security vulnerability some day in the future, the AppArmor profile severely restricts the ability to exploit it and gain access to the operating system. AppArmor was originally developed by Novell for use in SUSE Linux, but nowadays the main driver is Canonical and AppArmor is extensively used in Ubuntu and Debian, and many of their derivatives (e.g. Linux Mint, Pop!_OS, Zorin OS) and in Arch. AppArmor s benefit compared to the main alternative SELinux (used mainly in the RedHat/Fedora ecosystem) is that AppArmor is easier to manage. AppArmor continues to be actively developed, with new major version 5.0 expected to arrive soon. I also have some personal history contributing some notification handler scripts in Python and I also created the website that AppArmor.net still runs.

Regular review of denials in the system log required Any system administrator using Debian/Ubuntu needs to know how to check for AppArmor denials. The point of using AppArmor is kind of moot if nobody is checking the denials. When AppArmor blocks an action, it logs the event to the system audit or kernel logs. Understanding these logs is crucial for troubleshooting custom configurations or identifying potential security incidents. To view recent denials, check /var/log/audit/audit.log or run journalctl -ke --grep=apparmor. A typical denial entry for MariaDB will look like this (split across multiple lines for legibility):

Copy

msg=audit(1700000000.123:456): apparmor="DENIED" operation="open" profile="/usr/sbin/mariadbd" name="/custom/data/path/test.ibd" pid=1234 comm="mariadbd" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

msg=audit(1700000000.123:456): apparmor="DENIED" operation="open"
profile="/usr/sbin/mariadbd" name="/custom/data/path/test.ibd" pid=1234
comm="mariadbd" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

How to interpret this output:

• msg=audit( ): The audit timestamp and event serial number.
• apparmor= DENIED : Indicates AppArmor blocked the action.
• operation: The action being attempted (e.g., open, mknod, file_mmap, file_perm).
• profile: The specific AppArmor profile that triggered the denial (in this case the /usr/sbin/mariadbd profile).
• name: The file path or resource that was blocked. In the example above, a custom data path was denied access because it wasn t defined in the profile s allowed abstractions.
• comm: The command name that triggered the denial (here mariadbd).
• requested_mask / denied_mask: Shows the permissions requested (e.g., r for read, w for write).
• pid: The process ID.
• fsuid: The user ID of the process attempting the action.
• ouid: The owner user ID of the target file.

If an action seems legit and should not be denied, the sysadmin needs to update the existing rules at /etc/apparmor.d/ or drop a local customization file in at /etc/apparmor.d/local/. If the denied action looks malicious, the sysadmin should start a security investigation and if needed report a suspected zero-day vulnerability to the upstream software vendor (e.g. Ubuntu customers to Canonical, or MariaDB customers to MariaDB).

AppArmor in MariaDB - not a novel thing, and not easy to implement well Based on old bug reports, there was an AppArmor profile already back in 2011, but it was removed in MariaDB 5.1.56 due to backlash from users running into various issues. A new profile was created in 2015, but kept opt-in only due to the risk of side effects. It likely had very few users and saw minimal maintenance, getting only a handful of updates in the past 10 years. The primary challenge in using mandatory access control systems with MariaDB lies in the sheer breadth of MariaDB s operational footprint with diverse storage engines and plugins. Also the code base in MariaDB assumes that system calls to Linux always work which they do under normal circumstances and do not handle errors well if AppArmor suddenly denies a system call. MariaDB is also a large and complex piece of software to run and operate, and it can be very challenging for system administrators to root-cause that a misbehavior in their system was due to AppArmor blocking a single syscall. Ironically, AppArmor is most beneficial exactly due to the same reasons for MariaDB. The larger and more complex a software is, the larger are the odds of a security vulnerability arising between the various components. And AppArmor profile helps reduce this complexity down to a single access list. Over the years there has been users requesting to get the AppArmor profile back, such as Debian Bug#875890 since 2017. The need was raised recently again by the Ubuntu security team during the MariaDB Ubuntu main inclusion review in 2025, which prompted a renewed effort by Debian/Ubuntu developers, mainly myself and Aquila Macedo, with upstream MariaDB assistance from Daniel Black.

A fresh approach: leverage the MariaDB test suite for automated testing and the open source community for reviews The key to creating a robust AppArmor profile is the ability to know in detail what is expected and normal behavior of the system. One could in theory read all of the source code in MariaDB, but with over two million lines, it is of course not feasible in practice. However, MariaDB does have a very extensive 7000+ test suite, and running it should trigger most code paths in MariaDB. Utilizing the test suite was key in creating the new AppArmor profile for MariaDB: we installed MariaDB on a Ubuntu system, enabled AppArmor in complain mode and iterated on the allowlist by running the full mariadb-test-run with all MariaDB plugins and features enabled until we had a comprehensive yet clean list of rules. To be extra diligent, we also reworked the autopkgtest for MariaDB in Debian and Ubuntu CI systems to run with the AppArmor profile enabled and to print all AppArmor notices at the end of the run, making it easy to detect now and in the future if the MariaDB test suite triggers any AppArmor denials. If any test fails, the release would not get promoted further, protecting users from regressions. While developing and triggering manual test runs we used the maximal achievable test suite with 7177 tests. The test is however so extensive it takes over two hours to run, and it also has some brittle tests, so the standard test run in Debian and Ubuntu autopkgtest is limited just to MariaDB s main suite with about 1000 tests. Having some tests fail while testing the AppArmor profile was not a problem, because we didn t need all the tests to pass we merely needed them to run as many code paths as possible to see if they run any system calls not accounted for in the AppArmor profile. Note that extending the profile was not just mechanical copying of log messages to the profile. For example, even though a couple of tests involve running the dash shell, we decided to not allow it, as it opens too much of a path for a potential exploit to access the operating system. The result of this effort is a modernized, robust profile that is now production-ready. Those interested in the exact technical details can read the Debian Bug#1130272 and the Merge Request discussions at salsa.debian.org, which hosts the Debian packaging source code.

Now available in Debian unstable, soon Ubuntu feedback welcome! Even though the file is just 200 lines long, the work to craft it spanned several weeks. To minimize risk we also did a gradual rollout by releasing the first new profile version in complain mode, so AppArmor only logs would-be-denials without blocking anything. The AppArmor profile was switched to enforce mode only in the very latest MariaDB revision 1:11.8.6-4 in Debian, and a NEWS item issued to help increase user awareness of this change. It is also slated for the upcoming Ubuntu 26.04 Resolute Raccoon release next month, providing out-of-the-box hardening for the wider ecosystem. While automated testing is extensive, it cannot simulate everything. Most notably various complicated replication topologies and all Galera setups are likely not covered. Thus, I am calling on the community to deploy this profile and monitor for any audit denials in the kernel logs. If you encounter unexpected behavior or legitimate denials, please submit a bug report via the Debian Bug Tracking System. To ensure you are running the latest MariaDB version, run apt install --update --yes mariadb-server. To view the latest profile rules, run cat /etc/apparmor.d/mariadbd and to see if it is enforced review the output of aa-status. To quickly check if there were any AppArmor denials, simply run journalctl -k grep -i apparmor grep -i mariadb.

Systemd hardening also adopted as security features keep evolving For those interested in MariaDB security hardening, note that also new systemd hardening options were rolled out in Debian/Ubuntu recently. Note that Debian and Ubuntu are mainly volunteer-driven open source developer communities, and if you find this topic interesting and you think you have the necessary skills, feel free to submit your improvement ideas as Merge Requests at salsa.debian.org/mariadb-team. If your improvement suggestions are not Debian/Ubuntu specific, please submit them directly to upstream at GitHub.com/MariaDB.

And yet another maintenance release of the tidyCpp package arrived on CRAN this morning, just days after previous release which itself came a mere week and a half after its predecessor. It has been built for r2u as well. The package offers a clean C++ layer (as well as one small C++ helper class) on top of the C API for R which aims to make use of this robust (if awkward) C API a little easier and more consistent. See the vignette for motivating examples. This release restores the small CSS file used by the vignette which we, in a last-second decision, omitted from the previous release. Oddly, it only failed under the oldrel i.e. the R from now nearly two years ago. But it was still an unenforced error, and this upload corrects it. Changes are summarized in the NEWS entry that follows.

Changes in tidyCpp version 0.0.11 (2026-03-17)

• Keep a CSS file in the package to allow vignette build on r-oldrel too

Thanks to my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Another minor maintenance release version 0.1.4 of package RcppClassicExamples arrived earlier today on CRAN, and has been built for r2u. This package illustrates usage of the old and otherwise deprecated initial Rcpp API which no new projects should use as the normal and current Rcpp API is so much better. This release, the first in two and half years, mostly aids Rcpp in moving from Rf_error() to Rcpp::stop() for better behaviour under error conditions or excections. A few other things were updated in the interim such as standard upgrade to continuous integration, use of Authors@R, and switch to static linking and an improved build to support multiple macOS architectures. No new code or features. Full details below. And as a reminder, don t use the old RcppClassic use Rcpp instead.

Changes in version 0.1.4 (2026-03-16)

• Continuous integration has been updated several times
• DESCRIPTION now uses Authors@R
• Static linking is enforced, RcppClassic (>= 0.9.14) required
• Calls to Rf_error() have been replaced with Rcpp::stop()
• Updated versioned dependencies

Thanks to CRANberries, you can also look at a diff to the previous release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Review: The Martian Contingency, by Mary Robinette Kowal

Series:	Lady Astronaut #4
Publisher:	Tor
Copyright:	2025
ISBN:	1-250-23703-3
Format:	Kindle
Pages:	390

The Martian Contingency is the fourth book of the mostly-realistic science fiction alternate history series that began with the novelette "The Lady Astronaut of Mars" and the novel The Calculating Stars. It returns to Elma York as the main character, covering her second trip to Mars after the events of The Fated Sky. It's helpful to remember the events of the previous two books to follow some of the plot. Elma is back on Mars, this time as second in command. The immediate goal of the second Mars mission is to open more domes and land additional crew currently in orbit, creating the first permanent human settlement on Mars. The long-term goal is to set up Mars as a refuge in case the greenhouse effect caused by the meteor strike in The Calculating Stars continues to spiral out of control. Elma is anxious and not looking forward to being partly in charge, particularly since her position is partly due to her fame with the public (and connection with the American president). She'd rather just be a pilot. But she'll do what the mission needs from her, and at least this time her husband is with her on Mars. As one might expect from earlier installments of this series, The Martian Contingency starts with the details and rhythms of life in a dangerous, highly technical, and mission-driven scientific environment: hard science fiction of the type most closely modeled on NASA and real space missions. Given that this is aimed at permanent Mars colonies that would theoretically have to be independent of Earth, it requires a huge amount of suspension of disbelief for the premise, but Kowal at least tries for verisimilitude in the small details. I am not an expert in early space program technology (Kowal's alternate history diverges into a greatly accelerated space program in the 1950s and, for example, uses female mathematicians for most calculations), so I don't know how successful this is, but it feels crunchy and believable. As with the previous books, though, this is not just a day in the life of an astronaut. There's something wrong, something that happened during the first Mars expedition while Elma was in orbit and left odd physical clues, and no one is willing to talk about it. Elma is just starting to poke around before the politics at home go off the rails (again), exacerbated by a cringe-worthy social error by Elma herself, and she once again has to navigate egregious sexism and political meddling in a highly dangerous environment a long way from home. It is a little surprising that I like this series as much as I do. I don't particularly care for pseudo-realistic science fiction, although I admit there is something deeply satisfying about reading about people following checklists properly. The idea of permanent Mars colonies as an escape from a doomed Earth is unbelievable and deeply silly, but Kowal locked herself into that alternate future with "The Lady Astronaut of Mars," which is still set in the future of all of the books so far. A primary conflict in each of the books comes from the egregious sexism and racism of a culture based on 1950s American attitudes towards both, and the amount of progress Elma can make against either is limited, contingent, and constantly compromised. And yet. At its best, this series is excellent competence porn, both in the spirit of the Apollo 13 movie and for the navigation of social and political obstacles and idiocy. Elma is highly competent in a believable and sympathetic way, with strengths, weaknesses, and an ongoing struggle with anxiety. There is something rewarding in watching people solve problems and eventually triumph by being professional, careful, principled, and creative. It's enough to make a good book, even if I am not that interested in the setting and technology. As with the rest of the series, this will not be for everyone. You have to be up for reading about a lot of truly awful sexism and racism without the payoff of a complete triumph. This is a system that Elma navigates, not overthrows, and that's not going to be enough for some readers. You also have to accept the premise of a Mars colony, which in an otherwise hard science fiction novel is a bit much despite Kowal's attempts to acknowledge some of the difficulties. But if you don't mind that drawbacks, this series continues to be an opportunity to read about people being quietly and professionally competent. This is not my favorite entry, mostly because Elma makes a rather humiliating mistake that's central to the plot and has a lot of after-effects (and therefore a lot of time in the spotlight), and because there is rather a lot of discussion of sexuality that felt childish to me. The intent was to try to capture the way people in the 1950s talked about sex, and perhaps Kowal was successful in that, but I didn't enjoy the experience. But I still found myself pulled into the plot and happily rooting for the characters, even though a reader of "The Lady Astronaut of Mars" has a pretty good idea of how everything will turn out. If you liked the series so far, recommended, although I doubt it will be the favorite entry for most readers. If you did not like the earlier books of the series, this one will not change your mind. Content notes: Way, way too much detailed discussion of an injury to a fingernail than I wanted to read, as well as some other rather explicit description of physical injury. Reproductive health care through the lens of the 1950s, so, uh, yeah. A whole lot of sexism, racism, and other forms of discrimination that is mostly worked around rather than confronted. Rating: 7 out of 10

On Friday May the 13th OpenSSL project has published advisory details for CVE-2026-2673. The CVE is treated as non-important by the project. The patches are only provided as commits on the stable branches. No git tag, no precise fixed version, and no source tarballs provided.The patches that were merged to openssl-3.5 and openssl-3.6 branches were not based on top of the last stable point release and did not split code changes & documentation updates. It means that cherry-picking the commits referenced in the advisory will always lead to conflicts requiring manual resolution. It is not clear if support is provided for snapshot builds off the openssl-3.5 and openssl-3.6 branches. As the builds from the stable branches declare themselves as dev builds of the next unreleased point release. For example, in contrast to projects such as vim and glibc, with every commit to stable branches explicitly recommended for distributors to ship and is supported.I have requested OpenSSL upstream in the past for the security fixes to branch off the last point release, commit code changes separate from the NEWS.md / CHANGES.md updates, and then merge that into the stable branches. This way the advisory that recommends cherry-picking individual commits, would actually apply conflict free - at no additional maintenance burden to the OpenSSL project and everyone who has to cherry-pick these updates. There is a wide support voiced for such strategy by the OpenSSL distributors and the OpenSSL Corporation. But this is not something that OpenSSL Project is yet choosing to provide.To avoid duplication of work, I am starting to provide stable OpenSSL re-releases of the last upstream tagged stable point release with security only patches split into code-change only; documentation update; version update to create security only source tarball releases that are easy to build; easy to identify by the security scanners; and which cherry-pick changes without conflicts. The first two releases are published on GitHub as immutable releases with attestations:

• OpenSSL 3.6.1+1 - OpenSSL 3.6.1 with fix for CVE-2026-2673
• OpenSSL 3.5.5+1 - OpenSSL 3.5.5 with fix for CVE-2026-2673

Regression Tracking Debusine is a tool designed for Debian developers and Operating System developers in general. Debusine can run QA pipelines to check that Debian packages are ready to upload. This blog post describes the regression tracking mechanism that s recently become available in Debusine QA pipelines. The debian_pipeline workflow in Debusine can build, test, and upload a package to the Debian archive (or any other repository, such as a native Debusine APT repository). The QA tests involve running the standard Debian QA utilities (lintian, autopkgtest, piuparts, blhc) on the built artifacts. In addition we can run the autopkgtests of every other package in the archive that depends on the built package, like britney does for testing migration in Debian. Some of these other packages may have currently-failing autopkgtests that have nothing to do with the changes in the upload under test. For example: Figuring out which of these failures are new (and thus worth investigating) has been a manual process in Debusine until now. We have just completed the basic functionality of the regression_tracking=true feature, and have enabled it in the upload-to-* workflows on debusine.debian.net. With this enabled, you ll get a new QA tab on your debian_pipeline workflows that shows the trend of each test: This is determined by looking at recent task history for each task in the debian:qa-results collection. If there is no recent result for a given <package, version, architecture>, then tasks are queued under the reference tests qa workflow tree on the pipeline. These reference tests are run by using the same tasks as the main qa workflow, but without the addition of the package under test. In fact, it uses the same qa workflow that we use to check the package, but with a few different parameters to populate the regression tracking results collection. The debian:qa-results collection used for analyzing regressions is specified to the debian_pipeline with the regression_tracking_qa_results lookup parameter. On debusine.debian.net we have configured a debian:qa-results collection for sid that can be referenced and added to by tasks in any workspace. Regressions can be more subtle than a simple Success Failure. If the number of autopkgtests that fail increases, or the number of lintian tags emitted increases, those are also considered a regressions.

Using regression tracking now It s enabled by default on most of the upload-to-* workflows on debusine.debian.net. To disable, pass -O debusine_workflow_data.enable_regression_tracking=false when you dput an upload to debusine. To use the regression-tracking in your own workflows, use a debian_pipeline workflow that is configured with enable_regression_tracking=true. This will require a qa_suite to be specified, pointing to the baseline suite. We hope this will make it easier to check QA results for packages tested on debusine.debian.net.

Yet another maintenance release of the tidyCpp package arrived on CRAN this morning, a mere week and a half after the previous release. It has been built for r2u as well. The package offers a clean C++ layer (as well as one small C++ helper class) on top of the C API for R which aims to make use of this robust (if awkward) C API a little easier and more consistent. See the vignette for motivating examples. This release, just like the preceding one less than two weeks ago, had its hand forced by an overnight change in R-devel. The breakage this created has since been reverted in R-devel but as the writing is on the wall are now removing the definition involving these accessors preemptively. We were also missing version checks for two newly added alternates. Changes are summarized in the NEWS entry that follows.

Changes in tidyCpp version 0.0.10 (2026-03-15)

• Hide five accessors as underlying macros removed from Rinternals
• Preemptively hide another accessor
• Ensure two definitions are conditional on R 4.5.0 or later

Thanks to my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Recently I found myself with a few hours to kill, but with the only available connectivity provided by an annoying firewall which would normally allow requests only to a few very specific web sites. This post shows how to work around this kind of restrictions by hiding SSH in an HTTPS connection, which then can be used as a SOCKS proxy to allow general connectivity. socat does all the hard work. First, create two self-signed RSA keys pairs, one for the client (bongo) and one for the server (attila):

domain=bongo.example.net
openssl req -x509 -newkey rsa:2048 -days 7300 \
  -subj /CN=$domain -addext "subjectAltName = DNS:$domain" \
  -keyout socat.key -nodes \
  -out socat.pem

Then, concatenate the public and private keys to create the file provided to the cert option, and use the public key as the file for the cafile option on the other side. On the client side, if you normally would connect to attila.example.net then you can add something like this to ~/.ssh/config:

Host httpstunnel-attila.example.net
 ProxyCommand socat --statistics STDIO OPENSSL:attila.example.net:443, 
   cert=$HOME/.ssh/socat-bongo.pem,cafile=$HOME/.ssh/socat-attila.pem, 
   snihost=$ SOCAT_SNI:-x.com 
 DynamicForward 1080
 Compression yes
 HostKeyAlias attila.example.net
 ControlMaster yes
 ControlPath ~/.ssh/.control_attila.example.net_22_%r

The ProxyCommand directive uses socat to provide the connectivity which ssh will use over stdio instead of connecting to port 22 of the server. The snihost option is enough to make many firewalls believe that this is an authorized HTTPS request. On the server side we use a simple systemd unit to start a forking instance of socat, which will accept and process requests from the client (and from random crawlers on the Internet: expect a lot of cruft in that log...):

[Unit]
Description=socat tunnel
After=network.target
[Service]
Type=exec
ExecStart=socat -ly OPENSSL-LISTEN:443,fork,reuseaddr, 
  cert=%d/tlskey,cafile=%d/tlsca TCP:localhost:22
SuccessExitStatus=143
LoadCredential=tlskey:/etc/ssh/socat-attila.pem
LoadCredential=tlsca:/etc/ssh/socat-bongo.pem
Restart=on-abnormal
RestartSec=5s
DynamicUser=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectClock=yes
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=invisible
ProtectSystem=strict
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
SystemCallArchitectures=native
SystemCallErrorNumber=EPERM
SystemCallFilter=@system-service
SystemCallFilter=~@resources
SystemCallFilter=~@privileged
[Install]
WantedBy=multi-user.target

Strong sandboxing is enabled, so the socat instance is confined with very limited privileges. An interesting point is the use of systemd credentials to provide the cryptographic keys, since it allows to store them in a part of the file system which would not be accessible to the program. Advanced users can use this method to provide the keys from secure storage.

Introduction With various forms of IM becoming so prevalent and a lot of communication that used to be via email happening via IM I ve been thinking about the differences between Email and IM. I think it s worth comparing them not for the purpose of convincing people to use one or the other (most people will use whatever is necessary to communicate with the people who are important to them) but for the purpose of considering ways to improve them and use them more effectively. Also I don t think that users of various electronic communications systems have had a free choice in what to use for at least 25 years and possibly much longer depending on how you define a free choice. What you use is determined by who you want to communicate with and by what systems are available in your region. So there s no possibility of an analysis of this issue giving a result of let s all change what we use as almost everyone lacks the ability to make a choice. What the Difference is Not The name Instant Messaging implies that it is fast, and probably faster than other options. This isn t necessarily the case, when using a federated IM system such as Matrix or Jabber there can be delays while the servers communicate with each other. Email used to be a slow communication method, in the times of UUCP and Fidonet email there could be multiple days of delay in sending email. In recent times it s expected that email is quite fast, many web sites have options for authenticating an email address which have to be done within 5 minutes so the common expectation seems to be that all email is delivered to the end user in less than 5 minutes. When an organisation has a mail server on site (which is a common configuration choice for a small company) the mail delivery can be faster than common IM implementations. The Wikipedia page about Instant Messaging [1] links to the Wikipedia page about Real Time Computing [2] which is incorrect. Most IM systems are obviously designed for minimum average delays at best. For most software it s not a bad thing to design for the highest performance on average and just let users exercise patience when they get an unusual corner case that takes much longer than expected. If an IM message takes a few minutes to arrive then that s life on the Internet which was the catchphrase of an Australian Internet entrepreneur in the 90s that infuriated some of his customers. Protocol and Data Format Differences Data Formats Email data contains the sender, one or more recipients, some other metadata (time, subject, etc), and the message body. The recipients are typically an arbitrary list of addresses which can only be validated by the destination mail servers. The sender addresses weren t validated in any way and are now only minimally validated as part of anti-spam measures. IM data is sent through predefined connections called rooms or channels. When an IM message is sent to a room it can tag one or more members of the room to indicate that they may receive a special notification of the message. In many implementations it s possible to tag a user who isn t in the room which may result in them being invited to the room. But in IM there is no possibility to add a user to the CC list for part of a discussion and then just stop CCing messages to them later on in the discussion. Protocols Internet email is a well established system with an extensive user base. Adding new mandatory features to the protocols isn t viable because many old systems won t be updated any time soon. So while it is possible to send mail that s SSL encrypted and has a variety of authentication mechanisms that isn t something that can be mandatory for all email. Most mail servers are configured to use the SSL option if it s available but send in cleartext otherwise, so a hostile party could launch a Man In the Middle (MITM) attack and pretend to be the mail server in question but without SSL support. Modern IM protocols tend to be based on encryption, even XMPP (Jabber) which is quite an old IM protocol can easily be configured to only support encrypted messaging and it s reasonable to expect that all other servers that will talk to you will at least support SSL. Even for an IM system that is run by a single company the fact that communication with the servers is encrypted by SSL makes it safer than most email. A security model of this can only be read by you, me, and the staff at an American corporation isn t the worst type of Internet security. The Internet mail infrastructure makes no attempt to send mail in order and the design of the Simple Mail Transfer Protocol (SMTP) means that a network problem after a message has been sent but before the recipient has confirmed receipt will mean that the message is duplicated and this is not considered to be a problem. The IM protocols are designed to support reliable ordered transfer of messages and Matrix (the most recently designed IM protocol) has cryptographic connections between users. Forgery For most email systems there is no common implementation that prevents forging email. For Internet email transferred via SMTP it s possible to use technologies like SPF and DKIM/DMARC to make recipients aware of attempts at forgery, but many recipient systems will still allow email that fails such checks to be delivered. The default configuration tends to be permitting everything and all of the measures to prevent forgery require extra configuration work and often trade-offs as some users desire features that go against security. The default configuration of most mail servers doesn t even prevent trivial forgeries of email from the domain(s) owned by that server. For evidence check the SPF records of some domains that you communicate with and see if they end with -all (to block email from bad sources), ~all (to allow email from bad sources through after possibly logging an error), ?all (to be neutral on mail from unknown sources, or just lack a SPF record entirely. The below shows that of the the top four mail servers in the world only outlook.com has a policy to reject mail from bad sources.

# dig -t txt _spf.google.com grep spf1
_spf.google.com.	300	IN	TXT	"v=spf1 ip4:74.125.0.0/16 ip4:209.85.128.0/17 ip6:2001:4860:4864::/56 ip6:2404:6800:4864::/56 ip6:2607:f8b0:4000::/36 ip6:2800:3f0:4000::/36 ip6:2a00:1450:4000::/36 ip6:2c0f:fb50:4000::/36 ~all"
# dig -t txt outlook.com grep spf1
outlook.com.		126	IN	TXT	"v=spf1 include:spf2.outlook.com -all"
# dig -t txt _spf.mail.yahoo.com grep spf1
_spf.mail.yahoo.com.	1800	IN	TXT	"v=spf1 ptr:yahoo.com ptr:yahoo.net ip4:34.2.71.64/26 ip4:34.2.75.0/26 ip4:34.2.84.64/26 ip4:34.2.85.64/26 ip4:34.2.64.0/22 ip4:34.2.68.0/23 ip4:34.2.70.0/23 ip4:34.2.72.0/22 ip4:34.2.78.0/23 ip4:34.2.80.0/23 ip4:34.2.82.0/23 ip4:34.2.84.0/24 ip4:34.2.86.0" "/23 ip4:34.2.88.0/23 ip4:34.2.90.0/23 ip4:34.2.92.0/23 ip4:34.2.85.0/24 ip4:34.2.94.0/23 ?all"
# dig -t txt icloud.com grep spf1
icloud.com.		3586	IN	TXT	"v=spf1 ip4:17.41.0.0/16 ip4:17.58.0.0/16 ip4:17.142.0.0/15 ip4:17.57.155.0/24 ip4:17.57.156.0/24 ip4:144.178.36.0/24 ip4:144.178.38.0/24 ip4:112.19.199.64/29 ip4:112.19.242.64/29 ip4:222.73.195.64/29 ip4:157.255.1.64/29" " ip4:106.39.212.64/29 ip4:123.126.78.64/29 ip4:183.240.219.64/29 ip4:39.156.163.64/29 ip4:57.103.64.0/18" " ip6:2a01:b747:3000:200::/56 ip6:2a01:b747:3001:200::/56 ip6:2a01:b747:3002:200::/56 ip6:2a01:b747:3003:200::/56 ip6:2a01:b747:3004:200::/56 ip6:2a01:b747:3005:200::/56 ip6:2a01:b747:3006:200::/56 ~all"

In most IM systems there is a strong connection between people who communicate. If I send you two direct messages they will appear in the same room, and if someone else tries forging messages from me (EG by replacing the c and e letters in my address with Cyrillic letters that look like them or by mis-spelling my name) a separate room will be created and it will be obvious that something unexpected is happening. Protecting against the same attacks in email requires the user carefully reading the message, given that it s not uncommon for someone to start a message to me with Hi Russel (being unable to correctly copy my name from the To: field of the message they are writing) it s obvious that any security measure relying on such careful reading will fail. The IM protections against casual forgery also apply to rooms with multiple users, a new user can join a room for the purpose of spamming but they can t send a casual message impersonating a member of the room. A user can join a Matrix room I m in with the name Russell from another server but the potential for confusion will be minimised by a message notifying everyone that another Russell has joined the room and the list of users will show two Russells. For email the protections against forgery when sending to a list server are no different than those when sending to an individual directly which means very weak protections. Authenticating the conversation context once as done with IM is easier and more reliable than authenticating each message independently. Is Email Sucking the Main Technical Difference? It seems that the problems with forgery, spam, and general confusion when using email are a large part of the difference between email and IM. But in terms of technical issues the fact that email has significantly more users (if only because you need an email account to sign up for an IM system) is a major difference. Internet email is currently a universal system (apart from when it breaks from spam) and it has historically been used to gateway to other email systems like Fidonet, Uucp, and others. The lack of tight connection between parties that exchange messages in email makes it easier to bridge between protocols but harder to authenticate communication. Most of the problems with Internet email are not problems for everyone at all times, they are technical trade-offs that work well for some situations and for some times. Unfortunately many of those trade-offs are for things that worked well 25+ years ago. The GUI From a user perspective there doesn t have to be a great difference between email and IM. Email is usually delivered quickly enough to be in the same range as IM. The differences in layout between IM client software and email client software is cosmetic, someone could write an email client that organises messages in the same way as Slack or another popular IM system such that the less technical users wouldn t necessarily know the difference. The significant difference in the GUI for email and IM software was a design choice. Conversation Organisation The most significant difference in the operation of email and IM at the transport level is the establishment of connections in IM. Another difference is the fact that there are no standards implemented for the common IM implementations to interoperate which is an issue of big corporations creating IM systems and deliberately making them incompatible. The methods for managing email need to be improved. Having an inbox that s an unsorted mess of mail isn t useful if you want to track one discussion, breaking it out into different sub folders for common senders (similar to IM folders for DMs) as a standard feature without having to setup rules for each sender would be nice. Someone could design an email program with multiple layouts, one being the traditional form (which seems to be copied from Eudora [3]) and one with the inbox (or other folders) split up into conversations. There are email clients that support managing email threads which can be handy in some situations but often isn t the best option for quickly responding to messages that arrived recently. Archiving Most IM systems have no method for selectively archiving messages, there s a request open for a bookmark function in Matrix and there s nothing stopping a user from manually copying a message. But there s nothing like the convenient ability to move email to an archive folder in most IM systems. Without good archiving IM is a transient medium. This is OK for conversations but not good for determining the solutions to technical problems unless there is a Wiki or other result which can be used without relying on archives. Composing Messages In a modern email client when sending a message it prompts you for things that it considers complete, so if you don t enter a Subject or have the word attached in the message body but no file is attached to the message then it will prompt you to confirm that you aren t making a mistake. In an IM client the default is usually that pressing ENTER sends the message so every paragraph is a new message. IM clients are programmed to encourage lots of short messages while email clients are programmed to encourage more complete messages. Social Issues Quality The way people think about IM and email is very different, as one example there was never a need for a site like nohello.net for email. The idea that it s acceptable to use even lower quality writing in IM than people tend to use in email seems to be a major difference between the communication systems. It can be a good thing to have a chatty environment with messages that are regarded as transient for socialising, but that doesn t seem ideal for business use. Ownership Email is generally regarded as being comparable to physical letters. It is illegal and widely considered to be socially wrong to steal a letter from someone s letterbox if you regret sending it. In email the only unsend function I m aware of is that in Microsoft software which is documented to only work within the same organisation, and that only works if the recipient hasn t read the message. The message is considered to be owned by the recipient. But for IM it s a widely supported and socially acceptable function to delete or edit messages that have been sent. The message is regarded as permanently the property of the sender. What Should We Do? Community Creators When creating a community (and I use this in the broadest sense including companies) you should consider what types of communication will work well. When I started the Flounder group [4] I made a deliberate decision that non-free communication systems go against the aim of the group, I started it with a mailing list and then created a Matrix room which became very popular. Now the list hardly gets any use. It seems that most of the communication in the group is fairly informal and works better with IM. Does it make sense to use both? Should IM systems be supplemented with other systems that facilitate more detail such as a Wiki or a Lemmy room/instance [5] to cover the lack of long form communication? I have created a Lemmy room for Flounder but it hasn t got much interest so far. It seems that almost no-one makes a strategic decision about such issues. Software Developers It would be good to have the same options for archiving IM as there are for email. Also some options to encourage quality in IM communication similar to the way email clients want confirmation before sending messages without a subject or that might be missing an attachment. It would also be good to have better options for managing conversations in email. The Inbox as currently used is good for some things but a button to switch between that and a conversation view would be good. There are email clients that allow selecting message sort order and aggregation (kmail has a good selection of options) but they are designed for choosing a single setup that you like not between multiple views based on the task you are doing. It would be good to have links between different communication systems, if users had the option of putting their email address in their IM profile it would make things much easier. Having entirely separate systems for email and IM isn t good for users. Users The overall communications infrastructure could be improved if more people made tactical decisions about where and how to communicate. Keep the long messages to email and the chatty things to IM. Also for IM just do the communication not start with hello . To discourage wasting time I generally don t reply to messages that just say hello unless it s the first ever IM from someone. Conclusion A large part of the inefficiencies in electronic communication are due to platforms and usage patterns evolving with little strategic thought. The only apparent strategic thought is coming from corporations that provide IM services and have customer lock in at the core of their strategies. Free software developers have done great work in developing software to solve tactical problems but the strategies of large scale communications aren t being addressed. Email is loosely coupled and universal while IM is tightly coupled, authenticated, and often siloed. This makes email a good option for initial contact but a risk for ongoing discussions. There is no great solution to these issues as they are largely a problem due to the installed user base. But I think we can mitigate things with some GUI design changes and strategic planning of communication.

• [1] https://en.wikipedia.org/wiki/Instant_messaging
• [2] https://en.wikipedia.org/wiki/Real-time_computing
• [3] https://en.wikipedia.org/wiki/Eudora_(email_client)
• [4] https://flounder.linux.org.au/
• [5] https://en.wikipedia.org/wiki/Lemmy_(social_network)

So you want to join Outreachy but you don t understand it, you re scared, or you don t know what open source is about.

What is FOSS anyway?

Free and Open Source Software (FOSS) refers to software that anyone can use, modify, and share freely. Think of it as a community garden; instead of one company owning the food, people from all over the world contribute, improve, and maintain it so everyone can benefit for free. You can read more here on what it means to contribute to open source.

Outreachy provides paid internships to anyone from any background who faces underrepresentation, systemic bias, or discrimination in the technical industry where they live. Their goal is to increase diversity in open source. Read their website for more. I spent a good amount of time reading all the guides listed, including the applicant guide and the how-to-apply guide.

The Secret to Applying (Spoiler: It s not a secret)

I know newcomers are scared or unsure and would prefer answers from previous participants, but the Outreachy website is actually a goldmine, almost every question you have is already answered there if you look closely. I used to hate reading documentation, but I ve learned to love it. Documentation is the Source of Truth.

• My Advice: Read every single guide on their site. The applicant guide is your roadmap. Embracing documentation now will make you a much better contributor later.

The AI Trap: Be Yourself

Now for the part most newcomers have asked about is the initial essay. I know it s tempting to use AI, but I really encourage you to skip it for this. Your own story is much more powerful than a generated one. Outreachy and its mentoring organizations value your unique story. They are strongly against fabricated or AI-exaggerated essays.

For example, when I contributed to Debian using openQA, the information wasn t well established on the web. When I tried to use AI, it suggested imaginary ideas. The project maintainers had a particular style of contributing, so I had to follow the instructions carefully, observe the codebase, and read the provided documentation. With that information, I always wrote a solution first before consulting AI, and mine was always better. AI can only be intelligent in the context of what you give it; if it doesn t have your answer, it will look for the most similar solution (hallucinate). We do not want to increase the burden on reviewers their time is important because they are volunteers, too. This is crucial when you qualify for the contribution phase.

The Application Process

There are two main stages:

• The initial application: Here you fill in basic details, time availability, and essay questions (you can find these on the Outreachy website).
• The contribution phase: This is where you show you have the skills to work on the projects. Every project will list the skills needed and the level of proficiency.

When you qualify for the contribution phase:

• A lot of people will try to create buzz or even panic; you just have to focus. Once you ve gotten the hang of the project, remember to help others along the way.
• You can start contributions with spelling corrections, move to medium tasks (do multiple of these), then a hard task if possible. You don t need to be a guru on day one.
• It s all about community building. Do your part to help others understand the project too; this is also a form of contribution.
• Lastly, every project mentor has a way of evaluating candidates. My summary is: be confident, demonstrate your skills, and learn where you are lacking. Start small and work your way up, you don t have to prove yourself as a guru.

Tips

• Watch this: This step-by-step video is a great walkthrough of the initial application process.
• Sign up for the email list to get updates: https://lists.outreachy.org/cgi-bin/mailman/listinfo/announce
• Be fast: Complete your initial application in the first 3 days, as there are a lot of applicants.
• Back it up: In your essay about systemic bias, include some statistics to back it up.
• Learn Git: Even if you don t have programming skills, contributions are pushed to GitHub or GitLab. Practice some commands and contribute to a first open issue to understand the flow: https://github.com/firstcontributions/first-contributions

The most important tip? Apply anyway. Even if you feel underqualified, the process itself is a massive learning experience.

Another minor update 0.3.13 for our nanotime package is now on CRAN, and has been uploaded to Debian and compiled for r2u. nanotime relies on the RcppCCTZ package (as well as the RcppDate package for additional C++ operations) and offers efficient high(er) resolution time parsing and formatting up to nanosecond resolution, using the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from a rigorous refactoring by Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations. This release, the first in eleven months, rounds out a few internal corners and helps Rcpp with the transition away from Rf_error to only using Rcpp::stop which deals more gracefully with error conditions and unwinding. We also updated how the vignette is made, its references, updated the continuous integration as one does, altered how the documentation site is built, gladly took a PR from Michael polishing another small aspect, and tweaked how the compilation standard is set. The NEWS snippet below has the fuller details.

Changes in version 0.3.13 (2026-03-08)

• The methods package is now a Depends as WRE recommends (Michael Chirico in #141 based on a suggestion by Dirk in #140)
• The mkdocs-material documentation site is now generated via altdoc
• Continuous Integration scripts have been updated
• Replace Rf_error with Rcpp::stop, turn remaining one into (Rf_error) (Dirk in #143)
• Vignette now uses the Rcpp::asis builder for pre-made pdfs (Dirk in #146 fixing #144)
• The C++ compilation standard is explicitly set to C++17 if an R version older than 4.3.0 is used (Dirk in #148 fixing #147)
• The vignette references have been updated

Thanks to my CRANberries, there is a diffstat report for this release. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository and all documentation is provided at the nanotime documentation site.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

---

Edited 2026-03-05 18:48 UTC to add a missing --noop-ok to the Salsa CI runes. Thanks to Charlemagne Lasse for the report. Apologies if this causes Debian Planet to re-post this article as if it were new.

comments

A new release 0.3.14 of RcppGSL is now on CRAN. The RcppGSL package provides an interface from R to the GNU GSL by relying on the Rcpp package. It has already been uploaded to Debian, and is also already available as a binary via r2u. This release, the first in over three years, contains mostly maintenance changes. We polished the fastLm example implementation a little more, updated continunous integration as one does over such a long period, adopted the Authors@R convention, switched the (pre-made) pdf vignette to a new driver now provided by Rcpp, updated vignette references and URLs, and updated one call to Rf_error to aid in a Rcpp transition towards using only Rcpp::stop which unwinds error conditions better. (Technically this was a false positive on Rf_error but on the margin worth tickling this release after all this time.) The NEWS entry follows:

Changes in version 0.3.14 (2026-03-05)

• Updated some internals of fastLm example, and regenerated RcppExports.* files
• Several updates for continuous integration
• Switched to using Authors@R
• Replace ::Rf_error with (Rf_error) in old example to aid Rcpp transition to Rcpp::stop (or this pass-through)
• Vignette now uses the Rcpp::asis builder for pre-made pdfs
• Vignette references have been updated, URLs prefer https and DOIs

Thanks to my CRANberries, there is also a diffstat report for this release. More information is on the RcppGSL page. Questions, comments etc should go to the issue tickets at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Akvorado, a network flow collector, relies on Traefik, a reverse HTTP proxy, to expose HTTP endpoints for its Docker Compose services. Docker labels attached to each service define the routing rules. Traefik picks them up automatically when a container starts. Instead of maintaining a static configuration file to collect Prometheus metrics, we apply the same approach with Grafana Alloy.

Traefik & Docker Traefik listens for events on the Docker socket. Each service advertises its configuration through labels. For example, here is the Loki service in Akvorado:

services:
  loki:
    #  
    expose:
      - 3100/tcp
    labels:
      - traefik.enable=true
      - traefik.http.routers.loki.rule=PathPrefix( /loki )

Once the container is healthy, Traefik creates a router forwarding requests matching /loki to its first exposed port. Colocating Traefik configuration with the service definition is attractive. How do we achieve the same for Prometheus metrics?

Metrics discovery with Alloy Grafana Alloy, a metrics collector that scrapes Prometheus endpoints, includes a discovery.docker component. Just like Traefik, it connects to the Docker socket.¹ With a few relabeling rules, we teach it to use Docker labels to locate and scrape metrics. We define three labels on each service:

• metrics.enable set to true enables metrics collection,
• metrics.port specifies the port exposing the Prometheus endpoint, and
• metrics.path specifies the path to the metrics endpoint.

If a service exposes more than one port, metrics.port is mandatory. Otherwise, it defaults to the only exposed port. The default value for metrics.path is /metrics. The Loki service from earlier becomes:

services:
  loki:
    #  
    expose:
      - 3100/tcp
    labels:
      - traefik.enable=true
      - traefik.http.routers.loki.rule=PathPrefix( /loki )
      - metrics.enable=true
      - metrics.path=/loki/metrics

Alloy s configuration is split into four parts:

1. discover containers through the Docker socket,
2. filter and relabel targets using Docker labels,
3. scrape the matching endpoints, and
4. forward the metrics to Prometheus.

Discovering Docker containers The first building block discovers running containers:

discovery.docker "docker"  
  host             = "unix:///var/run/docker.sock"
  refresh_interval = "30s"
  filter  
    name   = "label"
    values = ["com.docker.compose.project=akvorado"]
   
 

This connects to the Docker socket and lists containers every 30 seconds.² The filter block restricts discovery to containers belonging to the akvorado project, avoiding interference with unrelated containers on the same host. For each discovered container, Alloy produces a target with labels such as __meta_docker_container_label_metrics_port for the metrics.port Docker label.

Relabeling targets The relabeling step filters and transforms raw targets from Docker discovery into scrape targets. The first stage keeps only targets with metrics.enable set to true:

discovery.relabel "prometheus"  
  targets = discovery.docker.docker.targets
  // Keep only targets with metrics.enable=true
  rule  
    source_labels = ["__meta_docker_container_label_metrics_enable"]
    regex         =  true 
    action        = "keep"
   
  //  
 

The second stage overrides the discovered port when the service defines metrics.port:

// When metrics.port is set, override __address__.
rule  
  source_labels = ["__address__", "__meta_docker_container_label_metrics_port"]
  regex         =  (.+):\d+;(.+) 
  target_label  = "__address__"
  replacement   = "$1:$2"
 

Next, we handle containers in host network mode. When __meta_docker_network_name equals host, Alloy rewrites the address to host.docker.internal instead of localhost:³

// When host networking, override __address__ to host.docker.internal.
rule  
  source_labels = ["__meta_docker_container_label_metrics_port", "__meta_docker_network_name"]
  regex         =  (.+);host 
  target_label  = "__address__"
  replacement   = "host.docker.internal:$1"
 

The next stage derives the job name from the service name, stripping any numbered suffix. The instance label is the address without the port:

rule  
  source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
  regex         =  (.+)(?:-\d+)? 
  target_label  = "job"
 
rule  
  source_labels = ["__address__"]
  regex         =  (.+):\d+ 
  target_label  = "instance"
 

If a container defines metrics.path, Alloy uses it. Otherwise, it defaults to /metrics:

rule  
  source_labels = ["__meta_docker_container_label_metrics_path"]
  regex         =  (.+) 
  target_label  = "__metrics_path__"
 
rule  
  source_labels = ["__metrics_path__"]
  regex         = ""
  target_label  = "__metrics_path__"
  replacement   = "/metrics"
 

Scraping and forwarding With the targets properly relabeled, scraping and forwarding are straightforward:

prometheus.scrape "docker"  
  targets         = discovery.relabel.prometheus.output
  forward_to      = [prometheus.remote_write.default.receiver]
  scrape_interval = "30s"
 
prometheus.remote_write "default"  
  endpoint  
    url = "http://prometheus:9090/api/v1/write"
   
 

prometheus.scrape periodically fetches metrics from the discovered targets. prometheus.remote_write sends them to Prometheus.

Built-in exporters Some services do not expose a Prometheus endpoint. Redis and Kafka are common examples. Alloy ships built-in Prometheus exporters that query these services and expose metrics on their behalf.

prometheus.exporter.redis "docker"  
  redis_addr = "redis:6379"
 
discovery.relabel "redis"  
  targets = prometheus.exporter.redis.docker.targets
  rule  
    target_label = "job"
    replacement  = "redis"
   
 
prometheus.scrape "redis"  
  targets         = discovery.relabel.redis.output
  forward_to      = [prometheus.remote_write.default.receiver]
  scrape_interval = "30s"
 

The same pattern applies to Kafka:

prometheus.exporter.kafka "docker"  
  kafka_uris = ["kafka:9092"]
 
discovery.relabel "kafka"  
  targets = prometheus.exporter.kafka.docker.targets
  rule  
    target_label = "job"
    replacement  = "kafka"
   
 
prometheus.scrape "kafka"  
  targets         = discovery.relabel.kafka.output
  forward_to      = [prometheus.remote_write.default.receiver]
  scrape_interval = "30s"
 

Each exporter is a separate component with its own relabeling and scrape configuration. We set the job label explicitly since no Docker metadata can provide it.

---

With this setup, adding metrics to a new service with a Prometheus endpoint requires only a few labels in docker-compose.yml, just like adding a Traefik route. Alloy picks it up automatically. You can apply the same pattern with another discovery method, like discovery.kubernetes, discovery.scaleway, or discovery.http.

---

1. Both Traefik and Alloy require access to the Docker socket, which grants root-level access to the host. A Docker socket proxy mitigates this by exposing only the read-only API endpoints needed for discovery.
2. Unlike Traefik, which watches for events, Grafana Alloy polls the container list at regular intervals a behavior inherited from Prometheus.
3. The Alloy service needs extra_hosts: ["host.docker.internal:host-gateway"] in its definition.