A colleague asked me if we should move all our money to our pillow cases after reading the latest AI editorial from Thomas Friedman. The article reads like a press release from Anthropic, repeating the claim that their latest AI model is so good at finding software vulnerabilities that it is a danger to the world. I think I now know what it s like to be a doctor who is forced to watch Gray s Anatomy. By now every journalist should be able to recognize the AI publicity playbook: Step 1: Start with a wildly unsubstantiated claim about how dangerous your product is: AI will cause human extinction before we have a chance to colonize mars (remember that one? Even Kim Stanley Robinson, author of perhaps the most compelling science fiction on colonizing mars calls bull shit on it). AI will eliminate all of our jobs (this one was extremely effective at providing cover for software companies laying off staff but it has quickly dawned on people that the companies that did this are living in chaos not humming along happily with functional robots) AI will discover massive software vulnerabilities allowing bad actors to hack pretty much every major software system in the world . (Did Friedman pull that directly from Anthropic s press release or was that his contribution?) Step 2: To help stave off human collapse, only release the new version to a vetted group of software companies and developers, preferably ones with big social media followings Step 3: Wait for the limited release developers to spew unbridled enthusiasm and shocking examples that seem to suggest this new AI produce is truly unbelievable Step 4: Watch stock prices and valuations soar Step 5: Release to the world, and experience a steady stream of mockery as people discover how wrong you are Step 6: Start over Even if Friedman missed the text book example of the playbook, I have to ask: if you think bad actors compromising software resulting in massive loss of private data, major outages and wasted resources needs to be reported on, then where have you been for the last 10 years? This literally happens on a daily basis due to the fundamentally flawed way capitalism has been writing software even before the invention of AI. A small part of me wonders - maybe AI writing software is not so bad, because how could it be any worse than it is now? Also, let s keep in mind that AI s super ability at finding vulnerable software depends on having access to the software s source code, which most companies keep locked up tight. That means the owners of the software can use AI to find vulnerabilities and fix them but bad actors can t. Oh, but wait, what if a company is so incompetent that they accidentally release their proprietary software to the Internet? Surely that would allow AI bots to discover their vulnerabilities and destroy the company right? I m not sure if anyone has discovered world ending vulnerabilities in Anthropic s Claude code since it was accidentally released, but it is fun to watch people mock software that is clearly written by AI (and spoiler alert, it seems way worse that software written now). Well we probably should all be keeping our money in a pillow case anyway.

It s been about a month since I wrapped up my Outreachy internship, but my journey with Debian is far from over. I planned to keep contributing and exploring the community, and these past few weeks have been busy

Testing Locales and Solving Bug #1111214

For the openQA project, we decided to explore how accurate local language installations are and see if we can improve the translations. While exploring this, I started working on automating a test for a specific bug report: Debian Bug #1111214

This is a test I had started by writing a detailed description of the installation process to confirm that selecting the Spanish_panama locale works accurately. I spent time studying previous language installation tests, and I learned that I needed to add a specific tag (LANGUAGE-) to the needles (visual test markers).

Since the installation wasn t in English anymore, taking the correct screenshots and defining the areas took quite some time. I used the following command on the CLI to run the test:

openqa-cli api -X POST isos ISO=debian-live-testing-amd64-gnome.iso DISTRI=debian-live VERSION=forky FLAVOR=gnome LANGUAGE=spanish_panama ARCH=x86_64 BUILD=1311 CHECKSUM=unknown

While working on this, I got stuck at the complete_installation step. Because the keyboard layout had changed to Spanish, the commands required to confirm a successful install weren t working as expected. Specifically, we had an issue typing the greater than sign (>).

My mentor, Roland Clobus, worked on a clever maneuver for the keys (AltGr-Shift-X), which was actually submitted upstream to openSUSE.

In this step, I also had to confirm that the locale was correctly set to LANG= es_PA.UTF-8 . I had to dig into the scripts and Linux commands to make this work. It was a bit intimidating at first, but it turned out to be a great learning experience. You can follow my progress on this Merge Request here. I m currently debugging a small issue where the home key seems to click twice in the final step, and after that, the test would be complete .

Community & Connections

Beyond the code, I ve been getting more involved in the social side of Debian:

• Debian Women: I attended the monthly meeting and met Sruthi Chandran. I ve always seen her name as an Outreachy organizer, so it was great to meet her! She is currently running for Debian Project Leader (DPL). We also discussed starting technical sessions to introduce members to packaging, which I am very excited to learn.
• DebConf Preparation: I am officially preparing for my first DebConf! My mentors, Tassia and Roland, along with my fellow intern Hellen, have been incredibly supportive in guiding me through the application and presentation process.

On June 19 and 20, I will cycle a little over 100 miles from downtown Chicago and its wonderful Millenium Park to New Buffalo, Michigan, as part of the Tour de Shore 2026. The ride passes through northwest Indiana and the extended Indiana Dunes National Park ending the next morning in the southwestern Michigan town of New Buffalo. I rode Tour de Shore once before in 2024 and had a generally wonderful time (even considering some soreness after a century of miles over 1 1/2 days). Tour de Shore is riding in support of Maywood Fine Arts Center, a local arts and sports center in Maywood, Illinois, a suburb one over from where I live and hence just a few good miles west of downtown. Maywood, Illinois is home to legends such as the late John Prine as well as several NBA players such as player and coach Doc Rivers. But Maywood, Illinois is also little less well off than other western suburbs. The Maywood Fine Arts Center is simply legendary is what they do for this community (and surrounding communities), and especially the youth support. They can use a dollar a two. Their story about Tour de Shore is worth a read too for background and motivation. I have bootstrapped my donation page page with a dollar for each mile to be cycled. It would be simply terrific if you could join me. A nickel, a dime, or a quarter per mile cycled would help. Multiples of that help too: More is of course still always better. Anything you can afford will go a long way towards a worthy goal in a community that could use the help. Of and if you are local to the area, I believe you can still register for Tour de Shore 2026. So see you out there in June? And if not, maybe help with a dollar or two?

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog.

Haven t written here about it, but last March we finally started on our journey to get our own house build, so we can move out of the rented flat here. That will be a big step, both the actual building, but also the moving - I am living at this one single place for 36 years now. If you can read german there is a dedicated webpage where I sometimes write about the process. Will have much more details (and way more ramblings) than the following part. If you can t read german, a somewhat short summary follows. Yes, still a lot of text, but shortened, still.

What? Why now? Current flat has 83m - which simply isn t enough space. And the number of rooms also doesn t fit anymore. But it is hard to find a place that fits our requirements (which do include location). Moving to a different rented place would also mean changed amount of rent. And nowadays that would be huge increase (my current rent is still the price from about 30 years ago!). So if we go and pay more - we could adjust and pay for something we own instead. And both, my wife and I had changes in our jobs that made it possible for us now, so we started looking.

Market Brrrr, looking is good, actually finding something that fits - not so. We never found an offer that fit. Space wise, sure. But then location was off, or price was idiotically high. Location fit, but then size was a joke, and guess about the price Who needs 200 square meters with 3 rooms? Entirely stupid design choices there. Or how about 40 square meters of hallway - with 50m of tiny rooms around. What are they smoking? Oh, there, useful size, good rooms - but now you want more money than a kidney is worth, or something. Thanks, no.

New place In February 2025 we finally got lucky and found a (newly opened) area with a large number of places to build a house on. Had multiple talks with someone from on of the companies developing that area (there are two you can select from), then talked with banks and signed a contract in March 2025. We got promised that actual house construction would be first quarter of 2026, finished in second quarter.

House type There are basically 2 ways of building a new house (that matter here). First is called Massivhaus , second is called Fertighaus in german, roughly translating to solid and prefabricated. The latter commonly a wood based construction, though it doesn t need to be. The important part of it is the prefabrication, walls and stuff get assembled in a factory somewhere and then transported to your place, where they play big kid lego for a day and suddenly a house is there. A common thought is prefabricated is faster, but that is only a half true. Sure, the actual work on side is way shorter - usually one or two days and the house is done - while a massive construction usually takes weeks to build up. But that is only a tiny part of the time needed, the major part goes of into planning and waiting and in there it doesn t matter what material you end up with.

Money fun Last year already wasn t the best time to start a huge loan - but isn t it always a few years ago would have been better ? So we had multiple talks with different banks and specialised consultants until we found something that we thought is good for us. Thinking about it now - we should have put even more money on top as reserve , but who could have thought that 2026 turns into such a shitshow? Does not help at all, quite the contrary. And that damn lotto game always ends up with the wrong numbers, meh.

Plans and plans and more plans - and rules For whichever reason you can not just go and put something on your ground and be happy. At least not if you are part of the normal people and not enormously rich. There is a large set of rules to follow. Usually that is a good thing, even though some rules are sometimes hard to understand. In Germany, besides the usual laws, we have something that is called Bebauungsplan , which translates to development plan (don t know if that carries the right meaning, it s a plan on what and how may be build, which can have really detailed specifications in). It basically tells you every aspect on top of the normal law that you have to keep in mind. In our case we have the requirement of 2 full floors and CAN have a third smaller on top, it limits how high the house can be and also how high our ground floor may be compared to the street. It regulates where on the property we may build and how much ground we may cover with the house, it gives a set of colors we are allowed to use, it demands a flat roof that we must have as a green roof and has a number of things more that aren t important enough to list here. If you do want to see the full list, my german post on it has all the details that matter to us. With all that stuff in mind - off to plans. Wouldn t have believed how many details there are to take in. Room sizes are simple, but how to arrange them for ideal usage of the sun, useful ways inside the house, but also keeping in mind that water needs to flow through and out. Putting a bath room right atop a living room means a water pipe needs to go down there. Switch the bath room side in the house, and it suddenly is above the kitchen - means you can connect the pipes from it to the ones from kitchen, which is much preferred than going through the living room. And lots more such things. It took us until nearly end of October to finalize the plans! And we learned a whole load from it. We started with a lot of wishes. The planner tried to make them work. Then we changed our minds. Plans changed. Minds changed again. Comparing the end result with the first draft we changed most of the ground floor around, with only the stairs and the entrance door at the same position. Less changes for the upper floor, but still enough.

Side quests The whole year was riddled with something my son named side quests. We visited a construction exhibition near us, we went to the house builders factory and took a look on how they work. We went to many different other companies that do SOME type of work which we need soon, say inside floors, painters, kitchen and more stuff. Of course the most important side quest was a visit to the notary to finalize the contracts, especially for the plot of land (in Germany you must have a notary for that to get entered into the governments books). Creates lots of fees, of course, for the notary and also the government (both fees and taxes here).

Building permit We had been lucky and only needed a small change to the plans to get the building permit - and the second part, the wastewater permit (yes, you need a separate one for this) also got through without trouble.

Choices, so many of them So in January we finally had an appointment for something that s called Bemusterung which badly translates to Sampling . Basically two days at the house builders factory to select all of what s needed for the house that you don t do in the plans. Doors, inside and out and their type and color and handles. Same things for the windows and the blinds and the protection level you want the windows to have. Decide about stairs, design for the sanitary installations - and also the height of the toilet! - and the tiles to put into the bathrooms. Decisions on all the tech needed (heating system, ventilation and whatnot. Two days, busy ones - and you can easily spend a lot of extra money here if you aren t careful. We managed to get out of it with only about 4000 extra, so pretty good.

Electro and automation Now, here I am special. Back when I was young the job I learned is electrician. So here I have very detailed wishes. I am also running lots of automatism in my current flat - obviously the new house should be better than that. So I have a lot of ideas and thoughts on it, so this is entirely extra and certainly out of the ordinary the house builder usually see. Which means I do all of that on my own. Well, the planning and some of the work, I must have a company at hand for certain tasks, it is required by some rules. But they will do what I planned, as long as I don t violate regulations. Which means the whole electrical installation is different. Entirely planned for automatisms and using KNX for it. I am so happy to ditch Homeassistant and the load of Homematic, Zigbee and ZWave based wireless things. Ok, Homeassistant is a nice thing - it can do a lot. And it can bridge between about any system you can find. But it is a central single point of failure. And it is a system that needs constant maintenance. Not touched for a while? Plan for a few hours playing update whack-a-mole. And often enough a component here or there breaks with an update. Can be fixed, but takes another hour or two. So I change. Away from wireless based stuff. To wires. To a system thats a standard for decades already. And works entirely without a SPOF. (Yes, you can add one here too). And, most important, should I ever die - can easily be maintained by anyone out there dealing with KNX, which is a large number of people and companies. Without digging through dozens of specialised integrations and whatnot. I may even end up with Homeassistant again - but that will entirely be as a client. It won t drive automations. It won t be the central point to do anything for the house. It will be a logging and data collecting thing that enables me to put up easy visualizations. It may be an easy interface for smartphones or tablets to control parts of the house, for those parts where one wants this to happen. Not the usual day-to-day stuff, extras on top.

Actual work happening Since march there finally is action visible. The base of the house is getting build. Wednesday the 1st April we finally got the base slab poured on the construction site and in another 10 days the house is getting delivered and build up. A 40ton mobile crane will be there.

tl;dr Starting with curl 8.17.0-1.amzn2023.0.2 in Amazon Linux 2023, you can now use HTTP/3.

dnf swap -y libcurl-minimal libcurl-full
dnf swap -y curl-minimal curl-full
curl --http3-only https://example.com

(HTTP/3 is only enabled in the curl -full builds) Or, if you would like to try it out in a container:

podman run amazonlinux:2023 /bin/sh -c 'dnf upgrade -y --releasever=latest && dnf swap -y libcurl-minimal libcurl-full && dnf swap -y curl-minimal curl-full && curl --http3-only https://example.com'

For a list of test endpoints, you can refer to https://bagder.github.io/HTTP3-test/

The Upgrade I Didn't Have to Make My teammate Steve Zarkos, who previously worked on upgrading OpenSSL in Amazon Linux from 3.0 to 3.2, spent the last few months on the complex task of bumping OpenSSL again, this time to 3.5. A bump like this only happens after extensive code analysis and testing, something that I didn't foresee happening when AL2023 was released but that was a notable request from users. Having enabled HTTP/3 on Debian, I was always keeping an eye on when I would get to do the same for Amazon Linux (mind you, I work at AWS, in the Amazon Linux org). The bump to OpenSSL 3.5 was the perfect opportunity to do that, for the first time Amazon Linux is shipping an OpenSSL version that is supported by ngtcp2 for HTTP/3 support.

Non-Intrusive Change In order to avoid any intrusive changes to existing users of AL2023, I've only enabled HTTP/3 in the full build of curl, not in the minimal one, this means there is no change for the minimal images. The way curl handles HTTP/3 today also does not lead to any behavior changes for those who have the full variants of curl installed, this is due to the fact that HTTP/3 is only used if the user explicitly asks for it with the flags --http3 or --http3-only.

Side Quests Supporting HTTP/3 on curl also requires building it with ngtcp2 and nghttp3, two packages which were not shipped in Amazon Linux, besides, my team doesn't even own the curl package, we are a security team so our packages are the security related stuff such as OpenSSL and GnuTLS. Our main focus is the services behind Amazon Linux's vulnerability handling, not package maintenance. I worked with the owners of the curl package and got approvals on a plan to introduce the two new dependencies under their ownership and to enable the feature on curl, I appreciate their responsiveness. Amazon Linux 2023 is forked from Fedora, so while introducing ngtcp2, I also sent a couple of Pull Requests upstream to keep things in sync: [ngtcp2] package latest release 1.21.0 [ngtcp2] do not skip tests While building the curl package in Amazon Linux, I've noticed the build was taking 1 hour from start to end, and the culprit was something well known to me; tests. The curl test suite is quite extensive, with more than 1600 tests, all of that running without parallelization, running two times for each build of the package; once for the minimal build and again for the full build. I had previously enabled parallel tests in Debian back in 2024 but never got around to submit the same improvements to Amazon Linux or Fedora, this is now fixed. The build times for Amazon Linux came down to 10 minutes under the same host (previously 1 hour), and Fedora promptly merged my PR to do the same there: [curl] run tests in parallel All of this uncovered a test which is timing-dependent, meaning it's not supposed to be run with high levels of parallelism, so there goes another PR, this time to curl: Flag test 766 as timing-dependent#21155 What started as enabling a single feature turned into improvements that landed in curl, Fedora, and Amazon Linux alike. I did this in a mix of work and volunteer time, mostly during work hours (work email address used when this was the case), but I'm glad I put in the extra time for the sake of improving curl for everyone.

Release Notes Amazon Linux 2023 release notes for 2023.10.20260330

Review: Code Blue Emergency, by James White

Series:	Sector General #7
Publisher:	Orb
Copyright:	1987
Printing:	May 2003
ISBN:	0-7653-0663-8
Format:	Trade paperback
Pages:	252

Code Blue Emergency (annoying em-dash in original title) is the seventh book of James White's Sector General science fiction series about a vast multi-species hospital station. While there are some references to (and spoilers for) earlier books in the series, you don't have to remember the previous books to read this one. I had no trouble despite a nine-year gap. I read this as part of the Orb General Practice omnibus, which collects this novel and The Genocidal Healer. Cha Thrat is a Sommaradvan warrior-surgeon, member of a newly-discovered species that is beginning the process of contact with the Federation. She saved a Monitor corps human after an accident on her world, performing some some highly competent surgery on a species she had never seen before. That plus her somewhat outcast status on her own world due to her very traditional attitude towards medical ethics led Sector General to extend an offer of medical internship, and led her to leap into the unknown by accepting. This may have been a mistake; there is a great deal that Sector General does not understand about Sommaradvan medical ethics. This series entry is another proper (if somewhat episodic) novel and the first book of the series that doesn't primarily focus on Conway. He makes an appearance in his new role as Diagnostician, but only as a supporting character. Code Blue Emergency is told in the tight third-person perspective of Cha Thrat, an alien who finds many things about Sector General baffling, confusing, and ethically troubling (and who therefore provides a good reader surrogate for reintroducing the basics of how the hospital works). Using an alien viewpoint is a more sophisticated narrative technique than White has used previously. I'm glad he tried it, and it mostly works, although I have some complaints. Cha Thrat comes from the middle caste of a strictly hierarchical society of three castes, but is also immensely stubborn and used to a medical system in which doctors take sole responsibility for their patients. This creates a lot of cultural conflicts, and I do enjoy science fiction where the human attitudes are portrayed as the strange ones, but the cultural analysis offered by this novel is not very deep. The pattern of this book is for Cha Thrat to stumble into a successful approach to a problem while being either oblivious to or hostile to the normal hierarchical structure expected of medical trainees. This is believable as far as it goes. She is a skilled and intelligent doctor with some good instincts and a strong commitment to patient care, but is also culturally inclined to not ask for help. It makes sense for that to be a serious problem in a hospital. Unfortunately, no one says this directly. Sector General staff get quite upset in ways that seem more territorial than oriented towards patient safety, no one directly explains to Cha Thrat why following a process is important or shows examples of what could go wrong, and plot armor means that her mistakes usually have positive outcomes. One can extrapolate the reasons why she is not a good medical student, but the reader is forced to do the extrapolation. This is the sort of book where the narration makes clear there are unresolved cultural clashes that are going to cause problems but hides the details. To Cha Thrat, her perspective is so obvious she never bothers to explain it to the reader, so the specifics come as a surprise. As with the alien perspective, I've seen this technique used with more subtlety and sophistication in other books, but White's version mostly works. Cha Thrat is a sympathetic protagonist because she is truly trying to take the most ethical and empathetic action in every situation and is clearly competent. Most of my frustration as a reader, ironically, lands on the other Sector General doctors who seem to make little to no effort to understand her perspective when she fails to conform to their expectations. This is believable in the abstract, but the whole point of Sector General is that they're supposed to be wiser about interspecies difference than this. Also, sometimes their reactions just seem petty. Cha Thrat has a very hierarchical concept of medicine that matches the social classes of her culture. For her, the highest tier of doctor are wizards who treat rulers, because the work of rulers is mostly mental and intellectual and therefore the diseases of rulers are treated with magic spells performed with words to reshape their thinking rather than surgery on their bodies. O'Mara and the other Sector General psychologists take great offense at this, muttering about being called witch doctors, which I found completely absurd. This is a comprehensible, if odd, description of psychology from a wholly alien species. Surely one's first reaction should be that words like "wizard" or "magic" are translation errors. Don't get offended; look to see if the underlying substance matches, which it clearly does. Apart from cultural and psychological clashes, Code Blue Emergency has the standard episodic Sector General structure of interesting medical mysteries that require lateral thinking. I find this sort of puzzle story satisfying, particularly given the firm belief of every character in an essentially pacifist and empathetic approach to even the most alien of creatures. This determined non-violence is one of the more interesting things about this series, and it continues here. White does tend towards both biological and gender essentialism for everyone other than the protagonist and main supporting characters, but he seemed to be walking back some of the more outrageous limitations on women that appeared in previous books. There is still some nonsense in here about how females of any species can't be Diagnosticians, but then Cha Thrat, who is female, seems to violate the justification for that rule over the course of this novel (sadly without comment). Perhaps he's setting up for proving Sector General wrong about this prejudice. I picked this up after reading Elizabeth Bear's Machine, which is essentially a (better written) Sector General novel that got me in the mood for reading more. I wouldn't give Code Blue Emergency any awards, but it delivered exactly what I was looking for. This series is not as deep or well-written as some more recent SF, but it is reliably itself and reliably entertaining. There are worse things in a series. Recommended if you're in the mood for alien ER in space. The omnibus edition that I read has an introduction to both novels by John Clute. It does add some interesting insights, but (as is somewhat typical for Clute) it also spoils parts of both books. You may want to read it after you read the novels. Followed by The Genocidal Healer. Rating: 7 out of 10

At May First we have been carefully planning our migration of about 1200 lists from mailman2 to mailman3 for almost six months now. We did a lot of user communications, had several months of beta testing with a handful of lists ported over, and everything was looking good. So we kicked off the migration! But, about 15% of the way through I started seeing sqlite lock errors. Wait, what? I carefully re-configured mailman3 to use postgres, not sqlite. Well, yes, but apparently that was for the database managing the email list configuration, not the database powering the django web app, which, incidentally, also includes hundresds of gigabytes of archives. In other words, the one we really need in postgres, not sqlite.

Moving from sqlite to postgres Well that sucks. We immediately stopped the migration to deal with this. I noticed that the web is full of useful django instructions on how to migrate your database from one database to antoher. However, if you read the fine print, those convenient looking dumpdata loaddata workflows are designed to move the table definitions and a small amount of data. In our case, even after just 15% of our lists moved, our sqlite database was about 30GB. I considered some of the hacks to manage memory and try to run this via django, but eventually decided that pgloader was a more robust option. This option also allowed me to more easily test things out on a copy of our sqlite database (made while mailman was turned off). This way I could migrate and re-migrate the sqlite database over and over without impacting our live installation until I was satisfied it was all working. My first decision was to opt out of pgloader s schema creation. I used django s schema creation tool by:

• Turning off mailman3 and mailman3-web and changing the mailman web configuration to use the new postgresql database.
• Running mailman-web migrate
• Changing the mailman web configuration back to sqlite and starting everything again.

Note: I tried just adding new database settings in the mailman web configuration indexed to new - django has the ability to define different databases by name, then you can run mailman-web migrate --database new. But, during the migration, I caught django querying the sqlite database for some migrations that required referencing existing fields (specifically hyperkitty s 0003_thread_starting_email). I didn t want any of these steps to touch the live database so I opted for the cleaner approach. Once I had a clean postgres schema, I dumped it so I could easily return to this spot. Next I started working on our pgloader load file. After a lot of trial and error, I ended with:

LOAD DATABASE
    FROM sqlite:///var/lib/mailman3/sqlite-postgres-migration/mailman3web.clean.backup.db
    INTO postgresql://mailmanweb:xxxxxxxxxxx@localhost:5432/mailmanweb

WITH data only,
    reset sequences,
    include no drop,
    disable triggers,
    create no tables,
    batch size = 5MB,
    batch rows = 500,
    prefetch rows = 50,
    workers = 2,
    concurrency = 1

SET work_mem to '64MB',
    maintenance_work_mem to '512MB'

CAST type datetime to timestamptz drop default drop not null,
    type date to date drop default drop not null,
    type int when (= precision 1) to boolean using tinyint-to-boolean,
    type text to varchar using remove-null-characters;

The batch, prefetch, workers and concurreny settings are all there to ensure memory doesn t blow up. I also discovered that I had to make some changes to the schema before loading data. Mostly truncating tables that the django migrate command populated to avoid duplicate key errors:

TRUNCATE TABLE django_migrations CASCADE;
TRUNCATE TABLE django_content_type CASCADE;
TRUNCATE TABLE auth_permission CASCADE;
TRUNCATE TABLE django_site CASCADE;

And also, I had to change a column type. Apparently the mailman import process allowed an attachment file name that exceeds the limit for postgres, but was allowed into sqlite:

ALTER TABLE hyperkitty_attachment ALTER COLUMN name TYPE text

When pgloader runs, we still get a lot of warnings from pgloader, which wants to cast columns differently than django does. These are harmless (I was able to import the data without a problem). And there are still a lot of warnings along the lines of:

2026-03-30T14:08:01.691990Z WARNING PostgreSQL warning: constraint hyperkitty_vote_email_id_73a50f4d_fk_hyperkitty_email_id of relation hyperkitty_vote does not exist, skipping

These are harmless as well. They appear because disable triggers disables foreign key constraints. Without it, we wouldn t be able to load tables that require values in tables that have not yet been populated. After all the tweaking, the import of our 30GB sqlite database took about 40 minutes.

Final Steps I think the reset sequences from pgloader should take care of this, but just in case:

mailman-web sqlsequencereset hyperkitty mailman_django auth   mailman-web dbshell

And, just to ensure postgres is optimized, run this in the psql shell:

ANALYZE VERBOSE;

Last thoughts I understand very well all the decisions the mailman3 devs made in designing the next version of mailman, and if I was in the same place I may have made them the same ones. For example, separating the code running the mailing list from the code managing the archives and the web interface makes perfectly good sense - many people might want to run just the mailing list part without a web interface. And building the web interface in django makes a lot of sense as well - why re-invent the wheel? I m sure a lot of time and effort was saved by simply using the built in features you get for free with django. But the unfortunate consequence of these decisions is that sys admins have a much harder time. Almost everyone wants the email lists along with the web interface and the archives. But nobody wants two different configuration files with different syntaxes and logic, not to mention two different command lines to use for maintenance and configuration with completely different APIs. Trying to understand how to change a default template or set list defaults requires a lot of research and usually you have to write a python script to do it. I have finally come to the conclusion that mailman2 is designed for sys admins, while mailman3 is designed for developers. Despite these short comings, I am impressed with the community and their quick and friendly responses to the questions of a confused sys admin. That might be more valuable than anything else.

MiniDebConf Kanpur 2026 was held on 14th and 15th March 2026 at the Indian Institute of Technology Kanpur. Having a Debian conference in the North was something many folks wanted. Ravi started the discussion (with IIT Kanpur folks) almost 7 months before the conference. Lots of folks from Debian India joined in organizing the conference, which was nice. All the meeting notes and discussions were posted on the Debian India mailing list, a first. Despite all the efforts, the conference start was delayed due to logistical issues. Things went fine post Day 1 lunch. We had two days of almost full schedules. disaster s Decentralising Indian Communication was an interesting talk, diving into decentralised communication. IIT Kanpur is a huge campus with nice footpaths and greenery. We got the opportunity to explore their HPC at the Computer Center post conference. Work has already started for MiniDebCamp Kochi. More details can be found on the wiki. Working to make this conference happen was different with all the challenges involved, but overall, everyone was happy with the outcome.

Review: The Sovereign, by C.L. Clark

Series:	Magic of the Lost #3
Publisher:	Orbit
Copyright:	September 2025
ISBN:	0-316-54286-5
Format:	Kindle
Pages:	575

The Sovereign is the third and concluding book of C.L. Clark's Magic of the Lost high fantasy trilogy. I recommend reading the books of this series close together, since there are a lot of characters and a lot of continuity between books that is helpful to remember, but it was not quite as difficult this time to remember where the story left off. At the end of The Faithless, the political situation in Balladaire (not-France) was more stable, but the threat of a plague lay on the horizon. That threat arrives in earnest in this book, along with new threats from both Balladaire's former colonial conscript soldiers and from neighboring Taargen (not-Germany, sort of, although the parallel isn't as close). Luca and Touraine have finally admitted that they're deeply in love, but they are still very different people with different goals and ethics. Luca is determined to do anything necessary to save her kingdom, but her definition of her kingdom is sharp and brittle. Touraine is torn between far too many loyalties, plus the lingering worry that her morals and Luca's may not be compatible. I think the hardest part of this sort of series is finding an ending the reader will find satisfying. This one, unfortunately, did not work for me, but that may be more due to personal preference than objective flaws. There have been two threads through this series: an improbable romance embedded in a network of complex personal relationships, and a political commentary on colonialism and post-colonial wars. I was enjoying the former, but it was the latter that felt fresh and interesting to me. The plot threads in The Faithless outside of Balladaire expanded that complexity, and I was hoping the final volume would continue in that direction. How could a colonial power atone for its history? How does the former colony establish its own governance? Is there a path to freedom without violence? Are attempts to chart a more moral course doomed to open lines of attack for one's other enemies? It's clear that Clark was thinking about similar themes, but The Sovereign narrows the field instead of widens it, restricts the political options, and then resolves most questions in a massive war. This is not that surprising of a conclusion, but it's one that I found unsatisfying and, honestly, a little boring. Yes, one way to resolve all the competing tensions is for everyone to try to kill each other and whoever survives wins, and historically that's one of the more likely outcomes, but that ending doesn't wrestle with the politics as much as it collapses them. Clark instead focuses this concluding volume on the romance, which becomes even more fraught, tragic, and dramatic than it was in previous books (and that's saying something). The hard questions of divided loyalties and moral conflicts are mostly framed by questions about Touraine's loyalty to Luca and Luca's trust of Touraine. This is all very Shakespearean, full of hard choices, sudden reversals, miscommunication, and a very deep conflict between Luca's realpolitik and Touraine's stubborn personal morality. If this is what you were reading the series for, if you were hoping for a maximum-drama sapphic relationship, you may thoroughly enjoy this. I thought it had its moments, but I wish they had been balanced by more moments of cool-headed practicality and creative political ingenuity. My biggest frustration with this ending is that the characters largely stop doing politics. The political complexity was the strength of both The Unbroken and The Faithless: People who intensely dislike each other negotiate because there is something larger to be gained, personal decisions made without considering the political ramifications have costs, and multiple characters are trying hard to find a way to turn a nasty, exploitative world into something better without simply killing everyone who disagrees. Many of the characters were objectively bad at politics, inexperienced and immature, but they stumbled or dragged or fought their way into political solutions anyway. I thought Clark moved too far away from that in The Sovereign. Everyone goes deep into their own emotions and desire for vengeance or conquest or revolution and stops compromising. To a depressingly large extent, the story is resolved by killing everyone who disagrees. I think the story is poorer for it. One of the other threads of the series is Balladairan magic, or rather its odd absence. Luca has one understanding of it, the rebels introduced in The Faithless have a different understanding of it, and its pursuit is set up as critical to resolving the threat of a plague. We do get an explanation of sorts, but it's not as complete or as satisfying as I was hoping, and the symbolism of Balladaire's missing magic is left frustratingly murky. For me, this has some of the same problems as the political conclusion: I wanted an intellectual catharsis alongside the emotional catharsis, but that was not the direction Clark was taking the story. I like reading about these characters. All of Luca, Touraine, and Pruett are complex, comprehensible, flawed, and often intriguing. But my favorite character in the story, the person I latched on to as an emotional path through the story, was Sabine. Her refreshingly straightforward loyalty and lack of drama was a breath of fresh air. She has some great moments in this book, but there too I got wrong-footed by the direction Clark went with her arc and found its conclusion deeply unsatisfying. I'm not sure how many of these complaints are because of missed opportunities in the novel, how many were due to a mismatch of taste, and how many were due to not being in the right mood to read this conclusion. I'm sure that it didn't help that I read this simultaneous with another novel in which the characters were always miserable, or that I read it in early 2026 with, uh, all that entails. I suspect that if you came away from the first two books invested in the messy romance and wanting MOAR DRAMA, you may get exactly what you were hoping for. That, sadly, was not what I was hoping for. I can't really recommend this. I thought it dragged in places and didn't deliver the ending I wanted. But it has some great moments, it does wrap up the threads of the trilogy as advertised, and at least the romance gets a dramatic climax worthy of the tension that has been built through the previous books. If that matches what you were enjoying in the previous books, you may well enjoy this more than I did. Rating: 5 out of 10

tl;dr This is not an official package, it's good enough for me and it might be good enough for you, confirmed as working in Debian Testing but I don't have a Stable machine to test there. You can use my custom repo to install the latest NVIDIA drivers on Debian Stable, Testing or Unstable (install from Sid repository): https://deb.debusine.debian.net/debian/r-samueloph-nvidia-ai/ The page above contains the APT sources you need, just add the one for your release to /etc/apt/sources.list.d/r-samueloph-nvidia-ai.sources, run sudo apt update and install the packages, you might need to disable Secure Boot.

This is not about AI Discussions about AI are quite divisive in the Free Software communities, and there's so much to be said about it that I'm not willing to go into in this blog post. This is rather just me telling people that if they need up-to-date NVIDIA packages for Debian, they could check if my custom repository gets the job done. The AI part is a means to an end, I've been careful to note in the repository names that the packages were produced with AI to respect people who do not want to run it for any reason.

RTX 5000 series support Back in May 2025 I opened a bug report asking for the NVIDIA drivers on Debian to be updated to support the RTX 5000 series. The Nouveau drivers might be good enough for some people, but I need the NVIDIA drivers because I want to play games and do experiments with open weight models. Opening a bug report doesn't guarantee anything, at the end of the day Debian Developers are volunteers, so if I really wanted the newer drivers, I would have to do something about it, ideally submitting a merge request. I briefly looked into the NVIDIA packaging, which involves 3 source packages (and one extra git repo for tarballs), unfortunately this was going to take more time and effort than what I was willing to spend.

What I Did After a few weeks of lamenting that I wasn't running the NVIDIA drivers, I figured I was willing to put in more effort than I originally thought, just enough to instruct the Claude Code agent to package the latest releases. I'm skilled enough with agentic tools that I knew how to use it to save time; providing a clear instruction on how to build the package and explaining the packaging layout, then letting the agent iterate until it gets a working build. The agent was running inside a VM that didn't have any of my credentials. After a little bit of back and forth, where I was reviewing the changes guiding the agent into how to fix certain issues, I ended up with a working set of packages. Once I installed it on my machine and confirmed they worked, I set up a debusine repository to make it easier to install future updates, and let others test it out. Debusine is analogous to Ubuntu's famous PPA, or Fedora's EPEL, it's a relatively new project but it has been working fine for this. Matheus Polkorny helped me test the packages and did spot a few issues which are fixed now. The Debusine developers were also always quick to respond to my questions and bug reports.

How Good Is It? Short answer: good enough for daily use, but not a substitute for an official Debian package. The whole point of doing this is because I don't have enough free time to maintain the package myself. All of this work was done as a volunteer, on my personal time. This means I'm trusting the agent to some degree; I review its commits but I don't go too deep into it, the quality will be dictated by the fact that I'm a Debian Developer and so by how easily I can spot issues without double checking everything. I only have a single machine with an NVIDIA GPU, this machine runs Debian Testing and so I don't have a way to test the Stable packages. I can do my best to address problems but at this point there is a risk that new updates break something. Installing NVIDIA drivers has always been a bit risky regardless, if you're comfortable with reverting updates and handling a system without a graphical interface (in case you end up in a tty), you will be fine. You will likely need to disable Secure Boot in order to use them, or set up your BIOS so that a MOK can be used to sign the DKMS modules. When choosing the version strings for the packages, I was careful enough to pick something that would sort lower than an official Debian package, meaning that whenever that same version is packaged in Debian, your system will see it as an upgrade. If you have any other methods of installing the NVIDIA drivers on your Debian system that is working for you, you should likely stick to that. I have a strong preference for installing them through .deb packages, making the package sort out configuration changes and dependency updates, besides handling the DKMS modules. Ultimately I'm not happy with the amount of difficulty that Debian users have in installing up-to-date NVIDIA drivers, and I hope this makes it easier for some.

How To Install Head over to the Debusine page that contains both repos for Trixie (Debian Stable) and Sid (for Debian Testing and Unstable): https://deb.debusine.debian.net/debian/r-samueloph-nvidia-ai/ If you are running Debian Testing, then pick the Sid repository. That page contains the contents of the apt .sources file you need, create the file /etc/apt/sources.list.d/r-samueloph-nvidia-ai.sources with the sources for your release. Run sudo apt update and install the packages you need, if you already have a previous version installed, sudo apt upgrade --update would update them. If there are no upgrades, meaning you don't have a previous version installed, then you need to explicitly install them.

sudo apt install nvidia-open-kernel-dkms nvidia-driver

If you run into issues in Debian Stable, consider using the Linux kernel package from the backports repository, if you need an up-to-date NVIDIA driver, you likely should also be running the backports kernel package (if you can't upgrade to Debian Testing).

Future Plans I currently have no means of measuring how many people are using the debusine repositories, so if you do end up using it feel free to let me know somehow. I don't know for how long I will keep managing this repository, and how much effort I will spend, but my machine needs it and for now I will keep it up-to-date with the latest production-grade NVIDIA drivers.

Sources The sources of the packages are available under a namespace in Salsa (Debian's GitLab instance): https://salsa.debian.org/samueloph-forks-team/nvidia-drivers-forks-with-ai You can also get the exact sources used in the repositories from debusine: https://debusine.debian.net/debian/r-samueloph-nvidia-ai/collection/debian:suite/sid-nvidia-ai/search/?category=debian:source-package https://debusine.debian.net/debian/r-samueloph-nvidia-ai/collection/debian:suite/trixie-nvidia-ai/search/?category=debian:source-package

We seem to be entering an AI apocalypse of sorts, they aren t going to kill us or even take our jobs. What they are doing is destroying the Internet commons by filling it with rubbish. This isn t even real AI, just pattern matching and prediction systems, mostly LLMs. The Problem Scott Shambaugh s saga of being attacked and defamed by an OpenClaw AI bot is interesting and raises some disturbing possibilities for future online discussion [1]. Imagine what it would be like if everyone who was in any way notable for free software work had 100 such bots going after them. Dania Dumas wrote an insightful blog post about why OpenClaw is impossible to secure and why it won t go away [2]. Bruce Schneier and Nathan E. Sanders wrote an insightful article about the AI generated text arms race [3] primarily concentrating on situations in which text that was assumed to be written by humans but was actually written in bulk by bots was performing a DOS attack on people who were reviewing it. There are many situations such as book publishing and publishing letters to the editor of newspapers where getting new material from unknown people is an important part of the job but where there are also people making low quality submissions that are almost a DOS attack at the best of times. Currently the email spam problem continues to get worse and when LLM use increases it will get significantly worse. Email encryption isn t viable [4]. The PGP web of trust never really worked well as it s too difficult for most users. The amount of AI generated content that s being recommended to users on platforms like YouTube and Facebook is steadily increasing and the amount of LLM generated commentary that purports to be from real people on Twitter and Facebook is also increasing. Here s an informative blog post by Erich Schubert about this [5]. Potential Solutions Surrender? One option and possibly the default option is to surrender to this and just let everything we built on the Internet over decades get destroyed. Whether to surrender is a decision that can be made on a per-service basis. Twitter is pretty much useless anyway, I quit Twitter because Elon deliberately made it suck [6]. In my opinion this is not surrendering to what s being done there, I m just stopping wasting time on it and using better options. I used to have about 300 followers on Twitter and I don t think that many of them would ever choose to stop following me, so I presume that about 1/3 of the people following me have decided to totally quit Twitter and delete their accounts. I also presume that some of the remainder have done the same as me and just kept a mostly inactive account. If Elon suddenly stopped being a stupid asshole it probably wouldn t change anything as the value of the system was connections to others. Some people will consider my abandonment of Twitter as surrender and I accept that it s not an unreasonable opinion. I think that the possibly 100 Twitter followers of mine who deleted their accounts surrendered. Facebook has been becoming a worse service, it s business model is becoming increasingly exploitative and it s interface is designed to be addictive. It s probably best avoided unless you really need it. The only good thing about Facebook at the moment is that Facebook Marketplace doesn t take a cut on sales and there are some really good deals on computers if you know what to look for. Unfortunately Facebook has a large number of users who are from marginalised communities and have no other alternatives for communication. It would be good to get them migrated to other platforms. We could just give up on a lot of general communications services and have everyone accept that good content is drowned out by rubbish and have the Internet become divided between people who accept the rubbish and those who cease using large portions of the Internet environment to avoid it. Using Non Commercial Services Lemmy is a good FOSS federated alternative to Reddit which also covers some of the uses of Facebook. It needs more users to get critical mass but is still quite usable. A post that might get a dozen comments on Reddit may get 1 comment on Lemmy but that one comment will be a good one. Reddit doesn t appear to be attacked much by LLM generated content at least not yet. Even if the Reddit model proves to be resilient to LLM attack the Lemmy software can be used to replace some things that are done on Facebook, Mastodon is a good FOSS federated replacement for Twitter, it has a decent user-base including some VIPs. While it is aimed at the Twitter use case it can also cover a significant part of the Facebook use case. There are some other FOSS social media programs which could take over other parts of the commercial social media environment. Generally commercially run Internet services will have a financial incentive to allow the problems to get worse so we need to rely on FOSS software, non-commercial implementations, and government services. Web Search For a long time Google has had a monopoly on web search, but now they default to including an AI Overview at the start of the results which is sometimes useful but also sometimes very wrong. You can use the search URL https://www.google.com/search?q=%s&udm=web to get google results without rubbish. But I presume that they will break that if it gets too popular. Searxng is a AGPL licensed metasearch engine that aggregates results from other engines, here s the Searxng source [7] and here s a list of Searxng instances if you want to try one [8]. Even using meta search engines like Searxng won t help if the original data is overloaded with spam, but alleviating the problem is a good temporary measure. Web of Trust for the Web? I ve idly considered the possibility of having some sort of rating system for web pages that uses a web of trust so that you can securely use trust ratings of friends of friends etc. But given all the difficulties in using a web of trust for signing GPG key for software developers (the demographic that is most skilled at doing such things) it doesn t seem viable. Should we surrender the idea of having a usable public web? In the early days of the web (before Google) it was standard practice to rely on recommendations from other people or from trusted sites to find other sites, that could be considered to be an informal web of trust. We could go back to that sort of usage pattern if Google and many of the big sites get overwhelmed by LLM generated spam. Wikipedia I believe that Wikipedia will be at the front lines of this battle. It s model has always included anonymous contributions. Benjamin Mako Hill wrote an interesting blog post about research he did with Kaylea Champion into Wikipedia pages on taboo topics which have a larger portion of contributors choosing to be anonymous than non-taboo pages [9]. Wikipedia also has a long history of being abused for various reasons, one that I witnessed was someone putting false content into Wikipedia pages to immediately cite them in support of their facebook arguments. That sort of thing can be dealt with at human scale but a large scale attack by bots is a different problem to solve. Also with the recent developments in AI developing multiple web sites entirely populated for the purpose of supporting one fake entry in Wikipedia is plausible. The upside of these attacks that I predict is that they will attract the attention of all the people who have skills related to developing counter-measures. While LLM bots are filling the inboxes of publishers with rubbish and messing up the stackoverflow comments section not a lot of people are bothered, but once the attacks on Wikipedia get serious everyone will take notice. National AI Bruce Schneier and Nathan E. Sanders wrote an interesting blog post about nationalised public AI [10]. While that won t directly address this issue it will get the right technology in the hands of people who can use it in the right way. Conclusion This is going to be a difficult problem to solve, more difficult than the email spam problem we have been unable to solve after 30 years of working on it. This is also a very important problem, we are currently in an age where we have access to information that most people couldn t even dream of 30 years ago. We also have disinformation that combines some of the worst aspects of authoritarian regimes throughout history combined with the worst aspects of cult brainwashing. If we lose access to the information but the disinformation remains (or get worse) then the result will be terrible. I don t have great ideas for solving this. I have outlined some small ideas to mitigate things and I hope that others can expand on them. Please write comments with any good ideas you have, or even ideas that don t totally suck. A problem this difficult is not going to be solved in a blog comment, but a blog comment might point in the right direction.

• [1] https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/
• [2] https://tinyurl.com/26wm43e2
• [3] https://tinyurl.com/22ghka6s
• [4] https://tinyurl.com/29to4cw5
• [5] https://www.vitavonni.de/blog/202602/20260213dogfood-the-AI.html
• [6] https://etbe.coker.com.au/2026/03/25/death-of-twitter/
• [7] https://github.com/searxng/searxng
• [8] https://searx.space/
• [9] https://tinyurl.com/26m98gca
• [10] https://tinyurl.com/24xt9gst

I was reading a post on Alex Chan's website¹ that referenced the concept of digital gardens, a concept/analogy for organising information which dates back to the 90s. This old concept is getting new traction today by contrasting the approach with "endless stream" as used and abused by social media, but also how blogs are typically presented. This site, my homepage, has a blog, and that's the bit that most people who interact with the site will experience. Partly, because it's the bit that gets syndicated out: via feeds; on Planet Debian and downstream from it; once upon a time on Twitter; nowadays on the Fediverse. However there's more to my homepage than that. The rest of it may be of little interest to anyone beside me, but it's useful to me, at least. So I may switch focus a little bit from mainly writing blog posts, and tend to the rest of the garden a bit more. Some recent seeding and pruning: Recently my guest status at Newcastle University came up for renewal, so I wrote down my goals in the Historic Computing Committee for the next year or so, and put them here: nuhcc. I've also been pondering what I'm up to in Debian at the moment, so took some time to add my current projects to that page.

At the end of last year I uninstalled the Twitter app on my phone. In the past Twitter used to be very useful for providing feedback to large organisations. I had responses from supermarkets, chain restaurants, online stores, major computer companies, and even the IT department of a court. In recent times I have had less responses from corporations which significantly reduces the value of Twitter to me and to many other users. It seems that Elon s management style has discouraged not only advertising but also all forms of corporate interaction. Messing up the check mark on accounts to make it harder to work out which is a real corporate Since Elon bought it Twitter has been increasingly pushing conservative Tweets and has done little to stop bot accounts. The incidence of useful discussions has steadily decreased. I know people who have quit Twitter entirely due to opposition to Elon, I am not doing that. I finally decided to stop using Twitter in any serious way when the notifications on my phone about popular Tweets started only being about Tweets from conservative influencers and Elon. This was obviously not any algorithm based on Tweets I was liking, it was based on political decisions. I didn t uninstall the app due to political disagreement, I uninstalled it because it was through deliberate design promoting material that any algorithm would know was something I wouldn t either like or like . I still announce new blog posts on Twitter for my 198 followers at the same time as announcing them on Mastodon and Facebook. I get the most reactions to such announcements on Mastodon, the second most on Facebook, and hardly any on Twitter. I m wondering how long it will be worth announcing blog posts on Twitter or whether I should stop now. I am sure that many other people are making similar decisions and this is going to affect Twitter overall. The web site www.russellcoker.com has information on all the ways of following me.

AI sure is a hot topic right now, and I see a lot of people arguing about it. To a lot of people around here, I m the computer person they know and I get asked a lot about AI. I m going to suggest a lot of things can be true at once. For instance:

• LLMs are changing how we work and will continue to do so.
• LLMs are vastly over-hyped by vested interests, and may be in a bubble.

Or how about:

• Huge investment in GenAI is having many negative consequences, ranging from environmental to causing affordability problems in many industries that use hardware (ie, everywhere)
• Useful results can be had from models that run on local hardware, even battery-powered hardware, which may have negligible harm or even some benefit

And:

• GenAI is further concentrating wealth and power in megacorps, with the effect of squeezing out the smaller players even more.
• GenAI is lowering the cost of entry for people without a lot of resources already.

I have sympathy for the naysayers; those that say it s nothing but a stochastic parrot. But I don t have a lot of sympathy for the naysayers that deny ever using it; you can t form a credible argument against something without having an understanding of it informed by experience. I also have sympathy for the cheerleaders. I have seen some impressive things from AI; for instance, a story from an engineer who has a child with a rare disease without a credible cure. The engineer did a lot of research on it, started feeding research papers into AI to analyze, and the AI started finding correlations between different areas of research that humans hadn t yet found leading to a positive result for the child. To be fair, I have rarely seen an AI deliver a 100% correct answer on anything with any real level of complexity. I have seen it both waste more time than it saves, and save a ton of time. My point here is: It is neither always fantastic nor always terrible. Let me talk you through an example. I am a fan of inbox zero for email. That is, the inbox should be empty. Unfortunately, mine has 8000 messages in it. According to the oldest messages in my inbox, I last had inbox zero 8 years ago. But really, only a handful are older than 2020. I guess something must have happened that year I ve been chipping away at this for quite some time now. The problem is, there are certain emails in there that really do still need some action maybe it s photos to save off into our photo collection, for instance. But when looking at things sorted by date or thread, there are old shipping confirmations next to phishing attempts and family photos. One can t just scan down the list. I ve tried all the usual tricks, most of which involve selecting groups of message that are easy to bulk erase, or at least easy to scan visually for the occasional thing worth saving. Sort by sender or subject line, for instance. Then I can, for instance, delete all the old messages from the shopping sites I commonly use all at once. But then they start using different senders and different subject lines and that doesn t get all of them. I ve tried keyword searches for this sort of thing too. Still, that got me down to about 8000 messages. So I thought: why not see if an LLM could help me classify these? Maybe it could categorize them, and then I could look at emails grouped by category. I have one machine with a discrete GPU, an Nvidia RTX 4070. It s a desktop machine I don t use all that often. But I set up Ollama on it, running in a Docker container. Ollama runs models locally. I should also mention at this point that we are solar-powered, and this time of year is a time of peak production of excess solar, because it is sunny and not much heat or AC is required. So that machine is solar-powered and isn t causing environmental harm. In any case, charging the EV uses much more power than that GPU. I figured I would do this in two passes. First, ask the LLM to classify each message (or a sampling of them would probably work too), letting it pick its own categories for each. Then, look at the patterns that emerge and give it a single, much smaller, set of broad categories to use and rerun it over that. Then I can easily select messages from my Maildirs by category and process them in bulk. I used open-interpreter pointing to that GPU on my network to help me write the scripts for this. It didn t get things right on its own; for instance, it didn t call the Ollama API correctly, and insisted on appending /cur to the path to the Maildir (which was not going to fly with Python s maildir module). It took roughly an hour to classify those 8000 messages (or, as I had it do, the first 2000 characters of them), and then the same to do it a second time. I had it output lines in the form of filename\tcategory and hand-wrote the shell script that processed those. In the end, was it useful? Yes, quite. Its classifications weren t perfect (and it didn t even follow my prompt perfectly; sometimes it would give me a long discussion on why it picked a certain category rather than just that category, and occasionally it picked categories not on the list). But then, neither were my manual keyword searches. So far I ve gotten rid of nearly 1000 more messages. Several categories were a visual scan for sanity and then delete all sort of thing. My emails never left my network. I didn t rely on a cloud AI to process them. I didn t contribute to global warming (this may have even been a case of saving energy, since it no doubt will offset quite a bit of manual time that would keep screens and room lights energized and so forth). I used about as much energy as watching a movie on a TV. Did it complete the task for me entirely autonomously? Also no. AI isn t a mind reader and it can t possibly evaluate exactly what my thought process would be for a given task. But it can do a decent enough job to save me some time. Still, this didn t require hyperscaler datacenters. AI even runs on-phone (Google Translate being one of the most useful AI-driven apps I ve ever seen, and it can run on-device).

Review: A Shadow in Summer, by Daniel Abraham

Series:	Long Price Quartet #1
Publisher:	Tor
Copyright:	March 2006
ISBN:	0-7653-1340-5
Format:	Hardcover
Pages:	331

A Shadow in Summer is a high fantasy novel, the first of (as the name implies) a completed four-book series. Daniel Abraham is perhaps better known as half of the writing pair behind James S.A. Corey, author of the Expanse series. This was his first novel. Otah was the sixth son of a Khai, sent like many of the unwanted later children of the powerful to learn the secrets of the andat and be trained as a poet. He learned his lessons well enough to reject the school and its teachings and walk away. Amat Kyaan has worked her way up from nothing to become the senior overseer of the foreign Galtic House Wilsin in the sun-drenched port city of Saraykeht. Liat is her apprentice, distracted by young love. Maati is a new apprentice poet, having endured his training and sent to learn from Heshai how to eventually hold the andat Removing-The-Part-That-Continues, better known as Seedless. None of them know they will find themselves entangled in a plot to destroy the poet of Saraykeht and, through him, the city's most potent economic tool. A poet in this world is not what we would think of a poet. They are, in essence, magical slave-drivers who capture the essence of an andat, a spirit embodying an idea that is coerced into the prison of volition and obedience by the poet. The andat Seedless, the embodiment of the concept of removing the spark of life, is central to the economic wealth of Saraykeht in a way that is startling in its simplicity: Seedless can remove the seeds from a warehouse full of cotton at a thought. This gives Saraykeht a massive productivity advantage in the cotton trade. Seedless is also a powerful potential weapon. What he can do to cotton, he could as easily do to any other crop, or to people. The Galts are not fond of the independence and power of Saraykeht, but as long as the city controls a powerful andat, they do not dare to attack it directly. Indirectly, though... that's another matter. This is one of those fantasy novels with meticulous and thoughtful world-building, careful and evocative prose, and a complex ensemble cast of interesting characters that the novel then attempts to make utterly miserable and complicit in their own misery. There should be a name for this style of writing. It's not tragedy because the ending is not tragic, precisely. It's not magic realism; the andats are openly magical, which makes this clearly high fantasy. But Abraham approaches the story from the type of realist frame that considers the pain and desperation of the characters to be more interesting than their ability to overcome challenges. Amat starts the story as an admirable, sharp-witted expert manager, so her life is destroyed and she's subjected to sexual violence. Heshai loathes himself and veers between a tragic figure and a wastrel as the story systematically undermines opportunities for redemption. Maati is young and idealistic, so of course every character in the book sets out to crush his idealism under the weight of unforeseen consequences. There is a sad and depressing love triangle, because this is exactly the sort of book that has a sad and depressing love triangle. At the end of the novel, everyone who survives is older and wiser in the sense that some stories seem to think wisdom comes from the accumulation of trauma. I find books like this so immensely frustrating because their merits are so clear. The world-building is careful and detailed in a way that includes economic systems, unlike so much fantasy. It is full of small, intriguing touches, such as the use of posture and gesture to communicate the emotional valence of one's words. Abraham understands the moral implications of poets and andats and the story tackles them head-on. The writing flows beautifully and gave me a strong sense of the city. I wanted to like this book for the obvious skill that went into it, and sometimes I even managed. And yet, it's taken me three months to finish A Shadow in Summer because I simply do not want to spend this much time around miserable people. I would get through one or two chapters in a night and then wanted to read something happy or defiant or heroic, rather than watching slow-motion train wrecks intermixed with desperate attempts to navigate stifling layers of immoral systems. It's not that the story lacks a moral compass. The characters are sincerely trying to make the world a better place, with some success. It even delivers a happy ending of sorts. But so much of the journey was watching the lives of the characters fall apart. I am completely unsurprised that some people loved this book. I'm still intrigued enough by the world-building that I'm half-tempted to try to read the sequel even after having to drag myself through this one. I had a similar reaction to Abraham's The Dragon's Path, though, so I think Abraham is just not for me. I may get back to the Expanse at some point, but having to drag myself through both of his solo novels I've tried, in two different series, probably indicates an incompatibility between author and reader. That's a shame, given the quality of the writing. Followed by A Betrayal in Winter. Content notes: Sexual and reproductive violence as significant plot elements. Rating: 6 out of 10

This needs to be clear: systemd is under attack by a trolling campaign orchestrated by fascist elements. Nobody is forced to like or use systemd, but anybody who wants to pick a side should know the facts. Recently, the free software Nazi bar crowd styling themselves as "concerned citizens" has tried to start a moral panic by saying that systemd is implementing age verification checks or that somehow it will require providing personally identifiable information. This is a lie: the facts are simply that the systemd users database has gained an optional "date of birth" field, which the desktop environments may use or not as they deem appropriate. Of course there is no "identity verification" or requirements to provide any data, which in any case would not be shared beyond authorized local applications. While the multiple recent bills proposing that general purpose operating systems implement age verification mechanisms are often concerning, both from a social and technical point of view, this is not the topic being discussed here. They are often suboptimal, but for a long time I have been opposing attempts to implement parental control at the network level and argued that it should be managed locally, by parents on their own machines: I cannot see why I should outright reject an attempt to implement the infrastructure to do that. If we want to keep age-appropriate controls out of the hands of centralized authorities, the alternative is giving families the means to manage it themselves: this is what this field enables. Whether desktop environments use it for parental controls, for birthday reminders, or for nothing at all, is their users' decision. By the way, the original UNIX users database has allowed storing PII in the GECOS field since it was invented in the '70s. Similar fields are also specified by many popular LDAP schemes: adding such an optional field is consistent with the UNIX tradition. And while we are at it, let's also refute the other smear campaign started by the same people: the systemd project is not accepting "AI slop". What happened is that a documentation file for the benefit of coding agents was added to the repository. To be clear: agents still cannot submit merge requests. The file itself remarks that all contributions must be reviewed in detail by humans, and this is basically the same policy used by the Linux kernel.

Note: I have not published blog posts about my academic papers over the past few years. To ensure that my blog contains a more comprehensive record of my published papers and to surface them for folks who missed them, I will periodically (re) publish blog posts about some older published projects. This post draws material from a previously published post by Kaylea Champion on the Community Data Science Blog.

Taboo subjects such as sexuality and mental health are as important to discuss as they are difficult to raise in conversation. Although many people turn to online resources for information on taboo subjects, censorship and low-quality information are common in search results. In two papers I recently published at CSCW both led by Kaylea Champion we presented a series of analyses showing how taboo shapes the process of collaborative knowledge building on English Wikipedia.

The first study is a quantitative analysis showing that articles on taboo subjects are much more popular and are the subject of more vandalism than articles on non-taboo topics. In surprising news, we also found that they were edited more often and were of higher quality!

Short video of Kaylea s presentation of the work given at Wikimania in August 2023.

The first challenge we faced in conducting this work was identifying taboo articles. Kaylea had a brilliant idea for a new computational approach to doing so without relying on our individual intuitions about what qualifies as taboo (something we understood would be highly specific to our own culture, class, etc). Her approach was to make use of an insight from linguistics: people develop euphemisms as ways to talk about taboos (i.e., think about all the euphemisms we ve devised for death, or sex, or menstruation, or mental health).

We used this insight to build a new machine-learning classifier based on English Wiktionary definitions. If a sense of a word was tagged as euphemistic, we treated the words in the definition as indicators of taboo. The end result was a series of words and phrases that most powerfully differentiate taboo from non-taboo. We then did a simple match between those words and phrases and the titles of Wikipedia articles. The topics were taboo enough that we were a little uncomfortable discussing them in our meetings! We built a comparison sample of articles whose titles are words that, like our taboo articles, appear in Wiktionary definitions. In the first paper, we used this new dataset to test a series of hypotheses about how taboo shapes collaborative production in Wikipedia. Our initial hypotheses were based on the idea that taboo information is often in high demand but that Wikipedians might be reluctant to associate their names (or usernames) with taboo topics. The result, we argued, would be articles that were in high demand but of low quality. We found that taboo articles are thriving on Wikipedia! In summary, we found that in comparison to non-taboo articles:

• Taboo articles are more popular (as expected).
• Taboo articles receive more contributions (contrary to expectations).
• Taboo articles receive more low-quality contributions (as expected).
• Taboo articles are higher quality (contrary to expectations).
• Taboo article contributors are more likely to contribute without an account (as expected), and have less experience (as expected), but that accountholders are more likely to make themselves more identifiable by having a user page, disclosing their gender, and making themselves emailable (all three of these are contrary to expectation!).

Image of the estimated qualiy of articles of the four articles in the second mixed-methods paper. Extreme dips reflect periods of frequent vandalism.

Kaylea attempted to understand these somewhat confusing results by designing a fantastic mixed-methods analysis that sought to unpack some of the nuance missing in the quantitative analysis by delving deep into the life histories of four articles on English Wikipedia: two on taboo topics related to women s anatomy (Clitoris and Menstration) and two nontaboo articles chosen for comparison (Cell membrance and Philip Pullman). Although the findings from the analysis can be difficult to summarize succinctly (as with many qualitative studies), we showed how the taboo example articles success was hard-won amid real challenges and attacks. The paper describes how challenges were overcome through resilient leadership, often provided by a single dedicated individual. The paper provides a template for how taboo can be and frequently is overcome by dedicated Wikipedians in ways that provide useful knowledge resources in real demand. For more details, visualizations, statistics, and more, we hope you ll take a look at our papers, both linked below.

---

The full citation for the papers are: (1) Champion, Kaylea, and Benjamin Mako Hill. 2023. Taboo and Collaborative Knowledge Production: Evidence from Wikipedia. Proceedings of the ACM on Human-Computer Interaction 7 (CSCW2): 299:1-299:25. https://doi.org/10.1145/3610090. (2) Champion, Kaylea, and Benjamin Mako Hill. 2024. Life Histories of Taboo Knowledge Artifacts. Proceedings of the ACM: Human-Computer Interaction 8 (CSCW2): 505:1-505:32. https://doi.org/10.1145/3687044.

We have also released replication materials for the paper, including all the data and code used to conduct the analyses.

This blog post and the paper it describes are collaborative work by Kaylea Champion and Benjamin Mako Hill.

Before reaching Vietnam Continuing from the last post, Badri and I took a flight from the Brunei International Airport to Kuala Lumpur on the 12th of December 2024. We reached Kuala Lumpur in the evening. After arriving at the airport, we went through immigration. In a previous post, I mentioned that we had put our stuff in lockers at the TBS bus terminal in Kuala Lumpur. Therefore, we had to go there. The locker was automated and required us to enter the PIN we had set. Upon entering the PIN, the locker wasn t getting unlocked. After trying this for 10-15 minutes without any luck, we tried getting some help as there the lockers weren t under supervision. So, I roamed around and found a staff member, reporting that our lockers weren t getting unlocked. They called the person who was in-charge of the lockers. He came to us in a few minutes and used their admin access to open the locker. We were supposed to pay for using the lockers by putting the banknotes inside through a slot. However, as the machine wasn t working, we gave the amount for the use of our locker service to that person instead. We soon went back to the KL airport to catch our morning flight to Ho Chi Minh City in Vietnam. At the flight counter, we were afraid we would have to pay extra as our luggage surpassed the allowed weight limit. This one was also a budget airline AirAsia and our tickets didn t include a check-in bag. Generally, passengers from countries requiring a visa to visit Vietnam (such as India) require going to the airline and showing their visa to get the boarding pass. However, when we went to the AirAsia counter at the Kuala Lumpur airport, they didn t weigh our bags and asked us to get our boarding passes from an automated kiosk. So, we got our boarding passes printed and proceeded to the airport security. While clearing the airport security, a lotion I bought from Singapore was confiscated because it was 200 mL, exceeding the limit of 100 mL per bottle. Had that 200 mL liquid been in two different bottles of 100 mL each, I would have been allowed to take it in my carry-on bag, but a single 200 mL bottle wasn t! I was allowed to keep it in the check-in bag, but I didn t have it included in my ticket. Huh, airports and their weird rules :( The lotion was an expensive one, so having it thrown away did ruin my mood.

Overview We started our Vietnam trip from Ho Chi Minh City in the south on the 13th of December 2024 and finished it in Hanoi in the north on the 20th of December. We traveled from Ho Chi Minh City to Hanoi mostly by train, except for a hundred or so kilometers by bus, in chunks. On the way, we visited Nha Trang, Hoi An, and Hue. The distance between Ho Chi Minh City and Hanoi is 1700 km. For your reference, here are those places labeled on Vietnam s map.

A map of Vietnam with points of places we went to labeled. CARTO MAPTILER OPENSTREETMAP

Ho Chi Minh City We landed in Ho Chi Minh City early morning on the 13th of December 2024. I was tired and sleepy as I hadn t gotten a good night s sleep. After going through immigration, we went to a currency exchange counter to get Vietnamese Dong. Unlike other countries on this trip, money exchange counters in Vietnam didn t accept Indian rupees. Therefore, we exchanged euros to get Vietnamese dong at the airport. After getting out of the airport, we took a bus to the city center. It was 15,000 dongs approximately 50 Indian rupees. Our plan was to meet Badri s friend and stay the night at his apartment. So we went to a caf nearby and bought a coffee for each of us for 75,000 dongs. We went upstairs and sat for a while. The Wi-Fi password was mentioned on our bill. During the trip, I found out about the caf culture of Vietnam. They have their own coffee brands (such as Highlands Coffee), and you can sit down at any of the caf s for work or wait for the rain to stop. It rained a lot while we were there, so we did use these caf s for that purpose. Badri s friend met us there, and we roamed around the area a bit, which included roaming inside a beautiful park. Then Badri s friend took us to a restaurant. Because I do not eat meat, he took us to a vegan restaurant. Having been to four Southeast Asian countries at this point (excluding Vietnam), I was under the impression that there wouldn t be a lot of things for my diet in Vietnam.

A picture of the park we roamed around in Ho Chi Minh City. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

However, I was pleasantly surprised at the restaurant. I found all the dishes to be tasty, especially their signature noodles called Pho. I liked another dish so much that I tracked down the restaurant again with Badri using the geotagged image of the bill I had taken earler to have it again. As a tip for vegans coming to Vietnam, the places having the letters Chay (without any accented letters) in their name are vegan only.

This is the restaurant Badri s friend took us to. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

One of the dishes we had in the restaurant. This one was especially tasty. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

One of the dishes we had in the restaurant. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

These noodles are called Pho and are very popular in Vietnam. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

In the night, we went to a supermarket where I got myself some oranges and guavas. Then, we went to a Japanese restaurant where I didn t have anything, as there was no vegetarian option available for me. Then we took a free bus to the place to Badri s friend s apartment. The construction company that built the apartment also runs this free bus service from their residential area to different parts of the city as a way of promoting their apartments. Anyone can take the bus, not just residents. The next day, we took the free bus back to the city center and checked in to a hostel for a night. We took two beds in dormitories, which were 88,000 dongs (270 rupees) for each bed for a night. In Vietnam, if you can spend around 300 rupees per night, you can get a bed in a decent hostel.

Train from Ho Chi Minh City to Nha Trang On the night of the 15th of December 2024, we boarded a train from Ho Chi Minh City to Nha Trang. The ticket for each of us was 519,000 dongs (1600 Indian rupees). The train name was SNT2. When we reached the Ho Chi Minh City train station, we noticed that the station was rather small by Indian standards. After entering the train station, we went inside to the first platform, where the tickets were checked by a staff member. Ho Chi Minh City was the originating station for our train, so our train was already standing at the station. We had to cross the railway tracks on foot to reach the platform our train was on. Then we located our coach, where a ticket inspector was standing at the gate. He let us in after checking our tickets. In all these instances, we just had to show our digital boarding pass which we had received by email. Unlike Indian trains, the train didn t have side berths. Additionally, I liked the fact that it had a dedicated space to put our bags in, which was very convenient. The train departed from Ho Chi Minh City at 21:05 and arrived in Nha Trang at 05:30 in the morning.

Interior of our train coach. Trains in Vietnam don t have side berths, unlike India. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

A picture of the berths from our coach. It had three tiers, similar to a 3 AC coach in Indian trains. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

The train had a cabin to put the bags in. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Nha Trang train station. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Nha Trang Nha Trang is a coastal place, and we planned to go to a beach. We figured out that the bus to the airport goes can drop us near the beach. Therefore, we went to the bus station to get to the airport bus. The bus station was walking distance from the railway station. So, we decided to walk. On the way, we stopped at a small shop for a coffee. The shop also gave a complimentary cup of green tea along with the coffee. I found out later that it is common for local shops to give a cup of complimentary green tea in Vietnam.

I got a complimentary cup of green tea along with coffee in Nha Trang. In this trip, Badri and I found out that this is customary at local places in Vietnam. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Soon we reached the bus station and took a bus to the beach. It was 65,000 dongs ( 200). After getting down from the bus, I had coconut water and some eggs at a small local place.

Eggs being cooked on a pan for my order. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Then we went to the beach, but nobody else was there. We spent some time there and went back to the place where the bus dropped us as it started raining. We couldn t find a bus for some time. A taxi driver approached us and agreed to take us to the city center for 200,000 dongs ( 650). For reference, the place where he dropped us was 35 km from the place we took the taxi. Taxi fares in Vietnam were also cheap!

The beach we went to in Nha Trang. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Nha Trang was a beautiful place, and so we roamed around for a while. Then we stopped at a Highlands Coffee branch for a while. Since Christmas was coming up, the caf had a Christmas tree, and I liked the Christmas vibes. They were playing Mariah Carey s All I Want for Christmas Is You.

This one was shot in the city center. In this trip, Badri and I found out that this is customary at local places in Vietnam. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Inside a Highlands Coffee cafe in Nha Trang. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

A coffee I got from Highlands Coffee in Nha Trang. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

During the evening, we went to a local place to eat. The place mentioned Chay in its name, and you know what it means it was a vegan place. There was a man there and no other customers. I don t remember the names of the dishes we ordered, but it was a bowl of soupy noodles and a bowl of dry noodles. They were very tasty. To top that off, the meal was a total of 55,000 dongs ( 180) for both of us. The host was welcoming and friendly. We had a nice conversation with the host. In Vietnam, restaurants give chopsticks to eat noodles. While Badri was good at using them, I wasn t. So, the host of this restaurant helped me in using chopsticks. Although my technique was not perfect and I take a bit of time, I could now eat solely with chopsticks.

The restaurant we went to in Nha Trang. The word Chay in the name means it was a vegan restaurant. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Soupy noodles we got at that restaurant. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Dry noodles we got at that restaurant. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Our plan was to take a night bus to Hoi An, and we were hoping to find a bus stand. However, we couldn t find one. Asking around about the pickup location of the Hoi An bus led us to many different locations. Finally, we ended up at a bus booking agency s office where we found out that there were no tickets available for Hoi An. At this point, we gave up on booking the bus and searched for trains instead. As we didn t have a local SIM, we asked the agency to let us connect to their Wi-Fi so that we could look for trains. They were kind enough to let us do that. It also seemed like they were going to close the office in like 10 minutes. Unfortunately, all the sleeper berths were booked from Nha Trang till Hoi An on the next train with only seating berths being available. It takes around 10 hours, so I wasn t comfortable traveling on seating berths. Here I came up with the idea to look for sleeper berths from an intermediate stop. Fortunately, there were sleeper berths available from the next stop, Ninh H a. Therefore, we booked a seating berth from Nha Trang to Ninh H a and a sleeper berth from Ninh H a to Tr Ki u (the nearest railway station from Hoi An). The train name was SE6, and it was a total of 500,000 dongs per person ( 1600 per person). So, we went to the Nha Trang railway station and boarded the train. We had to spend 40 minutes seated for the train to reach the next stop before we could go to our sleeper berths. Badri had some friendly co-passengers on that trip who gave him Saigon beer and some crispy papad-like thing. They offered me as well, but I thought it was non-veg, so I declined it.

Hoi An On the morning of 17th December 2024, we got down at the Tr Ki u station at around 09:30. Our hostel was in Hoi An, which was around 22 km from the station. There was no public transport to get there. Instead, there was a taxi driver at the train platform. We told him the name of our hostel, and he quoted 270,000 dongs (around 850). We said it was too expensive for us, so he agreed to bargain at 250,000 dongs. At this point, we told him that we could give him no more than 200,000 dongs, but he didn t agree. Badri tried a trick. He asked the driver to show us prices in the Grab app (a popular taxi booking app in Southeast Asia). Unfortunately, the Grab app showed 258,000 dongs, which was more than the fare the driver agreed to. So we walked away as if we had so many options (we didn t!) to reach the hostel. We got out of the station and stopped at a small shop outside to have some coffee. As is customary in Vietnam, we got a complimentary green tea here as well.

This was the place we had our coffee in Tra Kieu. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

That taxi driver also joined us and sat in that shop. He started talking with the locals in the shop in the local language. The taxi driver was insistent on taking us to Hoi An for 250,000 dongs. At this point, Badri told the taxi driver (by the use of translation software) that we usually use public transport during our trips, and we aren t used to paying high prices to get around. So, he can drop us somewhere in Hoi An for 200,000 dongs as we don t mind walking a bit to reach our hotel. After reading this, the taxi driver agreed to take us to our hostel for 200,000 dongs ( 660). He also had me take a picture with Badri after this. I think such a bargain tactic would not work in India.

Photo of Badri with taxi driver. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

The nice thing we noticed in Vietnam is, once bargaining is done and the deal is settled, people don t try to bargain more or keep on talking about the subject. Before the deal, the driver was being somewhat insistent and argumentative, but after the deal was done, it was as if no argument had happened at all.

A picture of Tra Kieu area near the train station we got down at. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

We were treated to some beautiful scenery on the way to our hostel. Soon we reached our place and completed all the formalities for checking-in. During the time our room was being prepared for check-in, we had an egg sandwich with coffee in the hotel. I found the egg sandwich very tasty. The bread looked like the French baguette. The hostel was 240 per night for each of us. The name of the hostel was Bana Spa. We liked staying here and we can recommend it if you find yourself there. It is operated by a family.

Our breakfast in Hoi An. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

A photo of the hostel we stayed in Hoi An. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

We also rented a bicycle for each of us 25,000 dongs per day ( 80) and explored the old town during the evening. Hoi An is popular for Vietnamese silk. Tourists come here to buy fabric and get it done by the tailor. The buildings here looked old, and they were painted in yellow with a gabled roof.

Typical yellow house with gabled roof in Hoi An old town. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Here, I also had egg coffee for the first time, and I liked it. Egg coffee is a delicacy of Hanoi, but you can get it in other parts of Vietnam. If you find yourself in Vietnam, then I recommend you try egg coffee. We also bought some cool T-shirts and other souvenirs, such as a Vietnamese hat, from here.

Egg coffee I had in Hoi An. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Hue The next day the 18th of December 2024 we went to Hue by bus. As we could not take a bus on our own in Nha Trang, we asked the hostel to book it for us this time. We booked it a day before, and they told us to be ready by 07:00 in the morning. At 07:00, a minibus arrived, which took us to a bus agency s office. There we waited for a few minutes and got into the bus to Hue. The bus had sleeper seats, so I took the opportunity to catch some sleep. The ride was comfortable, so I am assuming the roads were good. In a couple of hours, we reached Hue. Again, we went to Highlands Coffee to have some coffee, charge our phones, and use the internet, not to mention using the bathrooms. During the afternoon, we went to a local restaurant named Qu n Chay Thanh Li u. It was a vegan restaurant (remember the thing I mentioned earlier about Chay being in the name?). On the way, we had a steamed dumpling shaped like a momo called banh bao from a street vendor. It wasn t very good, but I found it worthwhile.

Bahn Bao in Hue. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

At the restaurant, we ordered a hot pot. First, they brought noodles and a gas stove. Then came the stock and our gas stove was turned on. The stock was kept simmering on the stove. Then, we had it bit by bit with the noodles. A big hot pot at this place costs 50,000 dongs ( 170). Then we had b nh cu n. These were steamed rolls made of rice flour for 10,000 dongs ( 33).

Hot Pot. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Added soup to the noodles. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Steamed rolls made of rice flour. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Restaurants in Vietnam usually add photos of the meals in their menu or write a description in English. So, even though the dish names were Vietnamese, we had no problems in ordering food there. In addition, all the places we went to provided free Wi-Fi. They either mention the Wi-Fi password on the bill, on the menu or paste it on the wall. This made our trip smoother without getting a local SIM.

Menu from a restaurant in Ho Chi Minh City with detailed description of the food. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Then we slowly walked towards the railway station, as we had a night train to Hanoi. We had egg coffee in a cafe. Near the railway station, we had a b nh m (egg sandwich). As for sightseeing, we had plans to visit a couple of places in Hue, but we ended up spending all our time inside sheltered spaces due to heavy rain. We had booked the train SE20 for Hanoi, which had a departure time of 20:41 from Hue. This one was 948,000 dongs ( 3100) for myself and 870,000 dongs ( 2900) for Badri. My ticket was pricier than Badri s because I got a lower berth. Our train was late by half an hour, so we waited in the common area of the station. After the train arrived, we got inside and took our seats. The cabin had four berths two upper and two lower, similar to India s First AC class. The ticket inspector came to us and offered us the whole cabin (two additional berths) for 300,000 dongs ( 1,000), which we declined. However, this hinted at the other two seats not being reserved. Eventually, we had the whole cabin to ourselves, as nobody else showed up for the other two berths. It was a 14-hour journey, and I got a good sleep.

Our berths in the train. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Hanoi On the morning of the 19th of December 2024, we reached Vietnam s capital, Hanoi. We had booked a private hotel room for 800. It was 1 km from the Hanoi Airport. However, it was pretty far from the railway station. So, we roamed around in the city and went to the hotel in the evening. First, we walked to a place and had egg coffee with egg sandwiches. Then we went to Hanoi Train Street, which was walking distance from the train station. After clicking some pictures at the train street, we went to a museum nearby. Upon reaching there, we found out that it was closed.

Egg coffee in Hanoi. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Hanoi train street is a tourist attraction in Hanoi. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Then we went shopping for jackets, as Hanoi was cold compared to other parts of Vietnam we had been to, and since many of them are manufactured in Vietnam, we thought they would be cheaper. I liked some jackets, but they were not my size. Eventually, we didn t buy anything at the clothes shop. In the evening, I bought a Vietnamese-styled phin coffee filter and coffee powder from Highlands Coffee. We spent a lot of time in their cafes, so it made sense to buy some souvenirs from there. Badri bought a few coffee filters for his family at Trung Nguyen, where I also bought another filter. We had dinner at a local place where we had pho and banh it. Bahn it was served packed in banana leaves and it was made of sticky rice.

A picture of pho we had in Hanoi. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Bahn it is served packed in banana leaves. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Bahn it. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

Next, we went to Hanoi railway station to catch a bus to the airport since our hotel was 1 km from the airport. The locals there helped us take the bus. It took like an hour to get to the airport. We saw on OpenStreetMap that we can take a bus from there to the hotel, but we could not find it. So we walked to our hotel instead. It was a decent hotel room for 800 for a night. We went outside to explore the area and had egg sandwiches and egg coffee at a local place. Again, we were given a complimentary green tea. We went to this place like three times. We had practically become regulars by the time we left. The next day 20th of December 2024 we took a bus to the airport and boarded our flight to Delhi. Credits: Thanks Badri, Kishy and Richard for proofreading.

The
WWW::Mechanize::Chrome Saga: A Comprehensive Narrative of PR #104 This document synthesizes the extensive work performed from March
13th to March 20th, 2026, to harden, stabilize, and refactor the
WWW::Mechanize::Chrome library and its test suite. This
effort involved deep dives into asynchronous programming,
platform-specific bug hunting, and strategic architectural
decisions.

---

Part I:
The Quest for Cross-Platform Stability (March 13 16) The initial phase of work focused on achieving a green test suite
across a variety of Linux distributions and preparing for a new release.
This involved significant hardening of the library to account for
different browser versions, OS-level security restrictions, and
filesystem differences.

Key Milestones &
Engineering Decisions:

• Fedora & RHEL-family Success: A major effort
  was undertaken to achieve a 100% pass rate on modern Fedora 43 and
  CentOS Stream 10. This required several key engineering decisions to
  handle modern browser behavior:
  • Decision: Implement Asynchronous DOM Serialization
    Fallback. Synchronous fallbacks in an async context are
    dangerous. To prevent Resource was not cached errors during
    saveResources, we implemented a fully asynchronous fallback
    in _saveResourceTree. By chaining
    _cached_document with DOM.getOuterHTML
    messages, we can reconstruct document content without blocking the event
    loop, even if Chromium has evicted the resource from its cache. This
    also proved resilient against Fedora s security policies, which often
    block file:// access.
  • Decision: Truncate Filenames for Cross-Platform
    Safety. To avoid File name too long errors,
    especially on Windows where the MAX_PATH limit is 260
    characters, filenameFromUrl was hardened. The filename
    truncation was reduced to a more conservative 150
    characters, leaving ample headroom for deeply nested CI
    temporary directories. Logic was also added to preserve file extensions
    during truncation and to sanitize backslashes from URI paths.
  • Decision: Expand Browser Discovery Paths. To
    support RHEL-based systems out-of-the-box, the
    default_executable_names was expanded to include
    headless_shell and search paths were updated to include
    /usr/lib64/chromium-browser/.
  • Decision: Mitigate Race Conditions with Stabilization Waits
    and Resilient Fetching. On fast systems,
    DOM.documentUpdated events could invalidate
    nodeIds immediately after navigation, causing XPath queries
    to fail with Could not find node with given id . A small stabilization
    sleep(0.25s) was added after page loads to ensure the DOM
    is settled. Furthermore, the asynchronous DOM fetching loop was hardened
    to gracefully handle these errors by catching protocol errors and
    returning an empty string for any node that was invalidated during
    serialization, ensuring the overall process could complete.
• Windows Hardening:
  • Decision: Adopt Platform-Aware Watchdogs. The test
    suite s reliance on ualarm was a blocker for Windows, where
    it is not implemented. The t::helper::set_watchdog function
    was refactored to use standard alarm() (seconds) on Windows
    and ualarm (microseconds) on Unix-like systems, enabling
    consistent test-level timeout enforcement.
• Version 0.77 Release:
  • Decision: Adopt SOP for Version Synchronization.
    The project maintains duplicate version strings across 24+ files. A
    Standard Operating Procedure was adopted to use a batch-replacement tool
    to update all sub-modules in lib/ and to always run
    make clean and perl Makefile.PL to ensure
    META.json and META.yml reflect the new
    version. After achieving stability on Linux, the project version was
    bumped to 0.77.
• Infrastructure & Strategic Work:
  • The ad2 Windows Server 2025 instance was restored and
    optimized, with Active Directory demoted and disk I/O performance
    improved.
  • A strategic proposal for the Heterogeneous Directory
    Replication Protocol (HDRP) was drafted and published.

---

Part II: The
Great Async Refactor (March 17 18) Despite success on Linux, tests on the slow ad2 Windows
host were still plagued by intermittent, indefinite hangs. This
triggered a fundamental architectural shift to move the library s core
from a mix of synchronous and asynchronous code to a fully non-blocking
internal API.

Key Milestones &
Engineering Decisions:

• Decision: Expose a _future API.
  Instead of hardcoding timeouts in the library, the core strategy was to
  refactor all blocking methods (xpath, field,
  get, etc.) into thin wrappers around new non-blocking
  ..._future counterparts. This moved timeout management to
  the test harness, allowing for flexible and explicit handling of
  stalls.

  # Example library implementation
  sub xpath($self, $query, %options)  
      return $self->xpath_future($query, %options)->get;
   
  
  sub xpath_future($self, $query, %options)  
      # Async implementation using $self->target->send_message(...)
   

• Decision: Centralize Test Hardening in a Helper.
  A dedicated test library, t/lib/t/helper.pm, was created to
  contain all stabilization logic. Safe wrappers (safe_get,
  safe_xpath) were implemented there, using
  Future->wait_any to race asynchronous operations against
  a timeout, preventing tests from hanging.

  # Example test helper implementation
  sub safe_xpath  
      my ($mech, $query, %options) = @_;
      my $timeout = delete $options timeout    5;
      my $call_f = $mech->xpath_future($query, %options);
      my $timeout_f = $mech->sleep_future($timeout)->then(sub   Future->fail("Timeout")  );
      return Future->wait_any($call_f, $timeout_f)->get;
   

• Decision: Refactor Node Attribute Cache.
  Investigations into flaky checkbox tests (t/50-tick.t)
  revealed that WWW::Mechanize::Chrome::Node was storing
  attributes as a flat list ([key, val, key, val]), which was
  inefficient for lookups and individual updates. The cache was refactored
  to definitively use a HashRef, providing O(1) lookups
  and enabling atomic dual-updates where both the browser property (via
  JS) and the internal library attribute are synchronized
  simultaneously.
• Decision: Implement Self-Cancelling Socket
  Watchdog. On Windows, traditional watchdog processes often
  failed to detect parent termination, leading to 60-second hangs after
  successful tests. We implemented a new socket-based watchdog in
  t::helper that listens on an ephemeral port; the background
  process terminates immediately when the parent socket closes,
  eliminating these cumulative delays.
• Decision: Deep Recursive Refactoring & Form
  Selection. To make the API truly non-blocking, the entire
  internal call stack had to be refactored. For example, making
  get_set_value_future non-blocking required first making its
  dependency, _field_by_name, asynchronous. This culminated
  in refactoring the entire form selection API (form_name,
  form_id, etc.) to use the new asynchronous
  _future lookups, which was a key step in mitigating the
  Windows deadlocks.
• Decision: Fix Critical Regressions & Memory
  Cycles.
  • Evaluation Normalization: Implemented a
    _process_eval_result helper to centralize the parsing of
    results from Runtime.evaluate. This ensures consistent
    handling of return values and exceptions between synchronous
    (eval_in_page) and asynchronous (eval_future)
    calls.
  • Memory Cycle Mitigation: A significant memory
    leak was discovered where closures attached to CDP event futures (like
    for asynchronous body retrieval) would capture strong references to
    $self and the $response object, creating a
    circular reference. The established rule is to now always use
    Scalar::Util::weaken on both $self and any
    other relevant objects before they are used inside a
    ->then block that is stored on an object.
  • Context Propagation (wantarray): A
    major regression was discovered where Perl s wantarray
    context, which distinguishes between scalar and list context, was lost
    inside asynchronous Future->then blocks. This caused
    methods like xpath to return incorrect results (e.g., a
    count instead of a list of nodes). The solution was to adopt the Async
    Context Pattern : capture wantarray in the synchronous
    wrapper, pass it as an option to the _future method, and
    then use that captured value inside the future s final resolution
    block.

    # Synchronous Wrapper
    sub xpath($self, $query, %options)  
        $options  wantarray   = wantarray; # 1. Capture
        return $self->xpath_future($query, %options)->get; # 2. Pass
     
    
    # Asynchronous Implementation
    sub xpath_future($self, $query, %options)  
        my $wantarray = delete $options  wantarray  ; # 3. Retrieve
        # ... async logic ...
        return $doc->then(sub  
            if ($wantarray)   # 4. Respect
                return Future->done(@results);
              else  
                return Future->done($results[0]);
             
         );
     

  • Asynchronous Body Retrieval & Robust Content
    Fallbacks: Fixed a bug where decoded_content()
    would return empty strings by ensuring it awaited a
    __body_future. This was implemented by storing the
    retrieval future directly on the response object
    ($response-> __body_future ). To make this more robust,
    a tiered strategy was implemented: first try to get the content from the
    network response, but if that fails (e.g., for about:blank
    or due to cache eviction), fall back to a JavaScript
    XMLSerializer to get the live DOM content.
  • Signature Hardening: Fixed Too few arguments
    errors when using modern Perl signatures with
    Future->then. Callbacks were updated to use optional
    parameters (sub($result = undef) ... ) to gracefully
    handle futures that resolve with no value.
  • XHTML Split-Brain Bug: Resolved a
    long-standing Chromium bug (40130141) where content provided via
    setDocumentContent is parsed differently than content
    loaded from a URL. A workaround was implemented: for XHTML documents,
    WMC now uses a JavaScript-based XPath evaluation
    (document.evaluate) against the live DOM, bypassing the
    broken CDP search mechanism.

Derived Architectural Rules
& SOPs:

• Rule: Always provide _future variants.
  Every library method that interacts with the browser via CDP must have a
  non-blocking asynchronous counterpart.
• Rule: Centralize stabilization in the test layer.
  All timeout and retry logic should reside in the test harness
  (t/lib/t/helper.pm), not in the core library.
• Rule: Explicitly propagate wantarray
  context. Synchronous wrappers must capture the caller s context
  and pass it down the Future chain to ensure correct
  scalar/list behavior.
• Rule: The entire call chain must be asynchronous.
  To enable non-blocking timeouts, even a single hidden blocking call in
  an otherwise asynchronous method will cause a stall.
• SOP: Reduce Library Noise. Diagnostic messages
  (warn, note, diag) should be
  removed from library code before commits. All such messages should be
  converted to use the internal $self->log('debug', ...)
  mechanism, ensuring a clean TAP output for CI systems.

---

Part III: The
MutationObserver Saga (March 19) With most of the library refactored to be asynchronous, one stubborn
test, t/65-is_visible.t, continued to fail with timeouts.
This led to an ambitious, but ultimately unsuccessful, attempt to
replace the wait_until_visible polling logic with a more
modern MutationObserver.

Key Milestones & Challenges:

• The Theory: The goal was to replace an inefficient
  repeat sleep loop with an event-driven
  MutationObserver in JavaScript that would notify Perl
  immediately when an element s visibility changed.
• Implementation & Cascade Failure: The
  implementation proved incredibly difficult and introduced a series of
  new, hard-to-diagnose bugs:
  1. An incorrect function signature for
     callFunctionOn_future.
  2. A critical unit mismatch, passing seconds from Perl to JavaScript s
     setTimeout, which expected milliseconds.
  3. A fundamental hang where the MutationObserver s
     JavaScript Promise would never resolve, even after the
     underlying DOM element changed.
• Debugging Maze: Multiple attempts to fix the
  checkVisibility JavaScript logic inside the observer
  callback, including making it more robust by adding DOM tree traversal
  and extensive console.log tracing, failed to resolve the
  hang. This highlighted the opacity and difficulty of debugging complex,
  cross-language asynchronous interactions, especially when dealing with
  low-level browser APIs.

Procedural Learning:
Granular Edits The effort was plagued by procedural missteps in using automated
file-editing tools. Initial attempts to replace large code blocks in a
single operation led to accidental code loss and match failures.

• Decision: Adopt Delete, then Add Workflow.
  Following forceful user correction, a new SOP was established for all
  future modifications:
  1. Isolate: Break the file into small, manageable
     chunks (e.g., 250 lines).
  2. Delete: Perform a delete operation by replacing
     the old code block with an empty string.
  3. Add: Perform an add operation by inserting the
     new code into the empty space.
  4. Verify: Verifying each atomic step before
     proceeding. This granular process, while slower, ensured surgical
     precision and regained technical control over the large
     Chrome.pm module.

The consistent failure of the MutationObserver approach
eventually led to the decision to abandon it in favor of stabilizing the
original, more transparent implementation.

---

Part IV:
Reversion and Final Stabilization (March 20) After exhausting all reasonable attempts to fix the
MutationObserver, a strategic decision was made to revert
to the simpler, more transparent polling implementation and fix it
correctly. This proved to be the correct path to a stable solution.

Key Milestones &
Engineering Decisions:

• Decision: Perform Strategic Reversion. The
  MutationObserver implementation, when integrated via
  callFunctionOn_future with awaitPromise,
  proved fundamentally unstable. Its JavaScript promise would consistently
  fail to resolve, causing indefinite hangs. A decision was made to
  revert all MutationObserver code from
  WWW::Mechanize::Chrome.pm and restore the original
  repeat sleep polling mechanism. A stable,
  understandable solution was prioritized over an elegant but broken
  one.
• Decision: Correct Timeout Delegation in the
  Harness. The root cause of the original timeout failure was
  identified as a race condition in the t/lib/t/helper.pm
  test harness. The safe_wait_until_* wrappers were
  implementing their own timeout (via wait_any and
  sleep_future) that raced against the underlying polling
  function s internal timeout. This led to intermittent failures on slow
  machines. The helpers were refactored to delegate all timeout
  management to the library s polling functions, ensuring a
  single, authoritative timer controlled the operation.
• Decision: Optimize Polling Performance. At the
  user s request, the polling interval was reduced from 300ms to
  150ms. This modest performance improvement reduced the
  test suite s wallclock execution time by over a second while maintaining
  stability.
• Decision: Tune Test Watchdogs. The global watchdog
  timeout was adjusted to 12 seconds, specifically calculated as 1.5x the
  observed real execution time of the optimized test. This provides a
  data-driven safety margin for CI.

---

Part
V: The Last Bug A Platform-Specific Memory Leak (March 20) With all other tests passing, a single memory leak failure in
t/78-memleak.t persisted, but only on the Windows
ad2 environment. This required a different approach than
the timeout fixes.

Key Milestones:

• The Bug: A strong reference cycle involving the
  on_dialog event listener was not being broken on Windows,
  despite multiple attempts to fix it. Fixes that worked on Linux (such as
  calling on_dialog(undef) in DESTROY) were not
  sufficient on the Windows host.
• The Diagnosis: The issue was determined to be a
  deep, platform-specific interaction between Perl s garbage collector,
  the IO::Async event loop implementation on Windows, and the
  Test::Memory::Cycle module. The cycle report was identical
  on both platforms, but the cleanup behavior was different.
• Failed Attempts: A series of increasingly
  aggressive fixes were attempted to break the cycle, including:
  1. Moving the on_dialog(undef) call from
     close() to DESTROY().
  2. Explicitly deleteing the listener and callback
     properties from the object hash in DESTROY.
  3. Swapping between $self->remove_listener and
     $self->target->unlisten in a mistaken attempt to find
     the correct un-registration method.
• Pragmatic Solution: After exhausting all reasonable
  code-level fixes without a resolution on Windows, the user opted to mark
  the failing test as a known issue for that specific platform.
• Final Fix: The single failing test in
  t/78-memleak.t was wrapped in a conditional
  TODO block that only executes on Windows
  (if ($^O =~ /MSWin32/i)), formally acknowledging the bug
  without blocking the build. This allows the test suite to pass in CI
  environments while flagging the issue for future, deeper
  investigation.

---

Part VI: CI Hardening (March
20) A final failure in the GitHub Actions CI environment revealed one
last configuration flaw.

Key Milestones:

• The Bug: The CI was running
  prove --nocount --jobs 3 -I local/ -bl xt t directly. This
  command was missing the crucial -It/lib include path, which
  is necessary for test files to locate the t::helper module.
  This resulted in nearly all tests failing with
  Can't locate t/helper.pm in @INC.
• The Investigation: An analysis of
  Makefile.PL revealed a custom MY::test block
  specifically designed to inject the -It/lib flag into the
  make test command. This confirmed that
  make test is the correct, canonical way to run the test
  suite for this project.
• The Fix: The
  .github/workflows/linux.yml file was modified to replace
  the direct prove call with make test in the
  Run Tests step. This ensures the CI environment runs the
  tests in the exact same way as a local developer, with all necessary
  include paths correctly configured by the project s build system.

Final Outcome After this long and arduous journey, the
WWW::Mechanize::Chrome test suite is now stable and
passing on all targeted platforms, with known
platform-specific issues clearly documented in the code. The project is
in a vastly more robust and reliable state.

Tweet

Linux kernel security modules provide a good additional layer of security around individual programs by restricting what they are allowed to do, and at best block and detect zero-day security vulnerabilities as soon as anyone tries to exploit them, long before they are widely known and reported. However, the challenge is how to create these security profiles without accidentally also blocking legitimate actions. For MariaDB in Debian and Ubuntu, a new AppArmor profile was recently created by leveraging the extensive test suite with 7000+ tests, giving good confidence that AppArmor is unlikely to yield false positive alerts with it. AppArmor is a Mandatory Access Control (MAC) system, meaning that each process controlled by AppArmor has a sort of an allowlist called profile that defines all capabilities and file paths a program can access. If a program tries to do something not covered by the rules in its AppArmor profile, the action will be denied on the Linux kernel level and a warning logged in the system journal. This additional security layer is valuable because even if a malicious user found a security vulnerability some day in the future, the AppArmor profile severely restricts the ability to exploit it and gain access to the operating system. AppArmor was originally developed by Novell for use in SUSE Linux, but nowadays the main driver is Canonical and AppArmor is extensively used in Ubuntu and Debian, and many of their derivatives (e.g. Linux Mint, Pop!_OS, Zorin OS) and in Arch. AppArmor s benefit compared to the main alternative SELinux (used mainly in the RedHat/Fedora ecosystem) is that AppArmor is easier to manage. AppArmor continues to be actively developed, with new major version 5.0 expected to arrive soon. I also have some personal history contributing some notification handler scripts in Python and I also created the website that AppArmor.net still runs.

Regular review of denials in the system log required Any system administrator using Debian/Ubuntu needs to know how to check for AppArmor denials. The point of using AppArmor is kind of moot if nobody is checking the denials. When AppArmor blocks an action, it logs the event to the system audit or kernel logs. Understanding these logs is crucial for troubleshooting custom configurations or identifying potential security incidents. To view recent denials, check /var/log/audit/audit.log or run journalctl -ke --grep=apparmor. A typical denial entry for MariaDB will look like this (split across multiple lines for legibility):

Copy

msg=audit(1700000000.123:456): apparmor="DENIED" operation="open" profile="/usr/sbin/mariadbd" name="/custom/data/path/test.ibd" pid=1234 comm="mariadbd" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

msg=audit(1700000000.123:456): apparmor="DENIED" operation="open"
profile="/usr/sbin/mariadbd" name="/custom/data/path/test.ibd" pid=1234
comm="mariadbd" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

How to interpret this output:

• msg=audit( ): The audit timestamp and event serial number.
• apparmor= DENIED : Indicates AppArmor blocked the action.
• operation: The action being attempted (e.g., open, mknod, file_mmap, file_perm).
• profile: The specific AppArmor profile that triggered the denial (in this case the /usr/sbin/mariadbd profile).
• name: The file path or resource that was blocked. In the example above, a custom data path was denied access because it wasn t defined in the profile s allowed abstractions.
• comm: The command name that triggered the denial (here mariadbd).
• requested_mask / denied_mask: Shows the permissions requested (e.g., r for read, w for write).
• pid: The process ID.
• fsuid: The user ID of the process attempting the action.
• ouid: The owner user ID of the target file.

If an action seems legit and should not be denied, the sysadmin needs to update the existing rules at /etc/apparmor.d/ or drop a local customization file in at /etc/apparmor.d/local/. If the denied action looks malicious, the sysadmin should start a security investigation and if needed report a suspected zero-day vulnerability to the upstream software vendor (e.g. Ubuntu customers to Canonical, or MariaDB customers to MariaDB).

AppArmor in MariaDB - not a novel thing, and not easy to implement well Based on old bug reports, there was an AppArmor profile already back in 2011, but it was removed in MariaDB 5.1.56 due to backlash from users running into various issues. A new profile was created in 2015, but kept opt-in only due to the risk of side effects. It likely had very few users and saw minimal maintenance, getting only a handful of updates in the past 10 years. The primary challenge in using mandatory access control systems with MariaDB lies in the sheer breadth of MariaDB s operational footprint with diverse storage engines and plugins. Also the code base in MariaDB assumes that system calls to Linux always work which they do under normal circumstances and do not handle errors well if AppArmor suddenly denies a system call. MariaDB is also a large and complex piece of software to run and operate, and it can be very challenging for system administrators to root-cause that a misbehavior in their system was due to AppArmor blocking a single syscall. Ironically, AppArmor is most beneficial exactly due to the same reasons for MariaDB. The larger and more complex a software is, the larger are the odds of a security vulnerability arising between the various components. And AppArmor profile helps reduce this complexity down to a single access list. Over the years there has been users requesting to get the AppArmor profile back, such as Debian Bug#875890 since 2017. The need was raised recently again by the Ubuntu security team during the MariaDB Ubuntu main inclusion review in 2025, which prompted a renewed effort by Debian/Ubuntu developers, mainly myself and Aquila Macedo, with upstream MariaDB assistance from Daniel Black.

A fresh approach: leverage the MariaDB test suite for automated testing and the open source community for reviews The key to creating a robust AppArmor profile is the ability to know in detail what is expected and normal behavior of the system. One could in theory read all of the source code in MariaDB, but with over two million lines, it is of course not feasible in practice. However, MariaDB does have a very extensive 7000+ test suite, and running it should trigger most code paths in MariaDB. Utilizing the test suite was key in creating the new AppArmor profile for MariaDB: we installed MariaDB on a Ubuntu system, enabled AppArmor in complain mode and iterated on the allowlist by running the full mariadb-test-run with all MariaDB plugins and features enabled until we had a comprehensive yet clean list of rules. To be extra diligent, we also reworked the autopkgtest for MariaDB in Debian and Ubuntu CI systems to run with the AppArmor profile enabled and to print all AppArmor notices at the end of the run, making it easy to detect now and in the future if the MariaDB test suite triggers any AppArmor denials. If any test fails, the release would not get promoted further, protecting users from regressions. While developing and triggering manual test runs we used the maximal achievable test suite with 7177 tests. The test is however so extensive it takes over two hours to run, and it also has some brittle tests, so the standard test run in Debian and Ubuntu autopkgtest is limited just to MariaDB s main suite with about 1000 tests. Having some tests fail while testing the AppArmor profile was not a problem, because we didn t need all the tests to pass we merely needed them to run as many code paths as possible to see if they run any system calls not accounted for in the AppArmor profile. Note that extending the profile was not just mechanical copying of log messages to the profile. For example, even though a couple of tests involve running the dash shell, we decided to not allow it, as it opens too much of a path for a potential exploit to access the operating system. The result of this effort is a modernized, robust profile that is now production-ready. Those interested in the exact technical details can read the Debian Bug#1130272 and the Merge Request discussions at salsa.debian.org, which hosts the Debian packaging source code.

Now available in Debian unstable, soon Ubuntu feedback welcome! Even though the file is just 200 lines long, the work to craft it spanned several weeks. To minimize risk we also did a gradual rollout by releasing the first new profile version in complain mode, so AppArmor only logs would-be-denials without blocking anything. The AppArmor profile was switched to enforce mode only in the very latest MariaDB revision 1:11.8.6-4 in Debian, and a NEWS item issued to help increase user awareness of this change. It is also slated for the upcoming Ubuntu 26.04 Resolute Raccoon release next month, providing out-of-the-box hardening for the wider ecosystem. While automated testing is extensive, it cannot simulate everything. Most notably various complicated replication topologies and all Galera setups are likely not covered. Thus, I am calling on the community to deploy this profile and monitor for any audit denials in the kernel logs. If you encounter unexpected behavior or legitimate denials, please submit a bug report via the Debian Bug Tracking System. To ensure you are running the latest MariaDB version, run apt install --update --yes mariadb-server. To view the latest profile rules, run cat /etc/apparmor.d/mariadbd and to see if it is enforced review the output of aa-status. To quickly check if there were any AppArmor denials, simply run journalctl -k grep -i apparmor grep -i mariadb.

Systemd hardening also adopted as security features keep evolving For those interested in MariaDB security hardening, note that also new systemd hardening options were rolled out in Debian/Ubuntu recently. Note that Debian and Ubuntu are mainly volunteer-driven open source developer communities, and if you find this topic interesting and you think you have the necessary skills, feel free to submit your improvement ideas as Merge Requests at salsa.debian.org/mariadb-team. If your improvement suggestions are not Debian/Ubuntu specific, please submit them directly to upstream at GitHub.com/MariaDB.