Here s my monthly but brief update about the activities I ve done in the FOSS world.
Debian
Whilst I didn t get a chance to do much, here are still a few things that I worked on:
Uploaded node-lodash/4.17.21+dfsg+~cs8.31.198.20210220-10 to fix CVE-2025-13465 in unstable. Pinged Xavier and Praveen to see how they feel about the backport for the stable releases.
Assited a few folks in getting their patches submitted via Salsa.
Reviewing pyenv MR for Ujjwal.
Reviewing and assisting Anshul with his Golang stuff.
Mentoring for newcomers.
Started to be a bit involved with DC 26 bits.
Moderation of -project mailing list.
Ubuntu
I joined Canonical to work on Ubuntu full-time back in February 2021.
Whilst I can t give a full, detailed list of things I did, here s a quick TL;DR of what I did:
ruby-rack: There were multiple vulnerabilities reported in Rack, leading to DoS (memory exhaustion) and proxy bypass.
[ELTS]: I ve partially reviewed patches by Bastien but whilst doing so, new ones came up for all the releases.
[sid]: Uploaded ruby-rack/3.2.5-1 to unstable to fix the new CVEs, CVE-2026-25500 and CVE-2026-22860.
[stable]: Prepared the upload for stable releases, as requested by the Security Team. Will upload them in March.
[bullseye]: Backport for CVE-2026-22860 hasn t been as trivial as it appears to be. There are test failures, which aren t really ignore-able:
The tests requests /cgi/../test, which File.expand_path resolves to <root>/test - firmly inside @root. With the old code, it hit the return unless path_info.include? ".." guard and fell through to 403. But it would also have returned early via start_with?(@root) being true
anyway, this needs more investigation - I ll continue to do so in March as P1.
node-lodash: Affected by CVE-2025-13465, lrototype pollution in baseUnset function.
[stable]: The patch for trixie and bookworm are ready and have been awaiting the maintainer review.
[LTS]: The bullseye upload will follow once the stable uploads are in and ACK d by the SRMs.
vlc: Affected by CVE-2025-51602, an out-of-bounds read and denial of service via a crafted 0x01 response from an MMS server.
[LTS]: 3.0.23 backport is ready but not tested. I ll get this over the line in March.
[ELTS]: 3.0.23 backport is ready but not very clean. Would like to complete LTS and get back to this.
Other Activities
[LTS] Coordinated the libvirt sitaution with Ben Hutchings. The upload has already been prepped, will upload in March.
[ELTS] Continued to help Bastien and Markus with the tomcat9 regression for buster.
[ELTS] Partially reviewed ruby-rack CVEs to help Bastien. This took more time than expected with some more develpments hapening at the end of the month - new CVEs that aren t as easy to backport as I expected them to be. Will continue with these in March.
[E/LTS] Front Desk duties: Performed a large batch of CVE triage, marking numerous packages for bullseye, buster, and stretch as either postponed, end-of-life, not-affected, or added them to the Xla-needed.txt files.
Also replied to certain mails and IRC texts around FD related tasks.
[E/LTS] Monitored discussions on mailing lists, IRC, and all the documentation updates.
[E/LTS] Attended the monthly LTS meeting on Jitsi. Summary here.
On Wednesday I had the pleasure of attending a presentation organized
by the Norwegian Unix Users Group on
implementing
RISC-V using a small FPGA. This project is the result of a
university teacher wanting to teach students assembly programming
using a real instruction set, while still providing a simple and
transparent CPU environment. The CPU in question implements the
smallest set of opcodes needed to still call the CPU a RISC-V CPU,
the
RV32I base set. The author and presenter, Kristoffer Robin
Stokke, demonstrated how to build both the FPGA setup and a small
startup code providing a "Hello World" message over both serial port
and a small LCD display. The FPGA is programmed using VHDL,
the entire source
code is available from github, but unfortunately the target FPGA
setup is compiled using the proprietary tool Quartus. It is such
a pity that such a cool little piece of free software should be
chained down by non-free software, so my friend Jon Nordby set out to
see if we can liberate this small RISC-V CPU. After all, it would be
unforgivable sin to force students to use non-free software to study
at the University of Oslo.
The VHDL code for the CPU instructions itself is only 1138 lines,
if I am to believe wc -l lib/riscv_common/* lib/rv32i/*. On
the small FPGA used during the talk, the entire CPU, ROM, display and
serial port driver only used up half the capacity. These days, there
exists a free software toolchain for FPGA programming not only in
Verilog but also in VHDL, and we hope the support in
yosys,
ghdl, and
yosys-plugin-ghdl
(sadly and strangely enough, removed from Debian unstable) is complete
enough to at least build this small and simple project with some minor
portability fixes. Or perhaps there are other approaches that work
better? The first patches are already floating on github, to make the
VHDL code more portable and to test out the build. If you are
interested in running your own little RISC-V CPU on a FPGA chip,
please get in touch.
At the moment we sadly have hit a GHDL bug, which we do not quite
know how to work around or fix:
Thus more work is needed. For me, this simple project is the first
stepping stone for a larger dream I have of converting the MESA
machine controller system to build its firmware using a free
software toolchain. I just need to learn more FPGA programming
first. :)
As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
National Knowledge Network (NKN) is one of India s main National Research and Educational Network (NREN). The other being the less prevalent Education and Research Network (ERNET).
This post grew out of this Mastodon thread where I kept on adding various public graphs (from various global research and educational entities) that peer or connect with NKN. This was to get some purview about traffic data between them and NKN.
CERN
CERN, birthplace of the World Wide Web (WWW) and home of the Large Hadron Collider (LHC).
India participates in the LHCONE project, which carries LHC data over these links for scientific research purposes. This presentation from Vikas Singhal from Variable Energy Cyclotron Centre (VECC), Kolkata, at the 8th Asian Tier Center Forum in 2024 gives some details.
G ANT
G ANT is pan European Union s collaboration of NRENs.
Earlier this week a colleague of mine, Emilio Jes s Gallego Arias, shared a demo of something he built as an experiment, and I felt the desire to share this and add a bit of reflection. (Not keen on watching a 5 min video? Read on below.)
What was that?
So what did you just see (or skipped watching)? You could see Emilio s screen, running VSCode and editing a Lean file. He designed a small programming language that he embedded into Lean, including an evaluator. So far, so standard, but a few things stick out already:
Using Lean s very extensible syntax this embedding is rather elegant and pretty.
Furthermore, he can run this DSL code right there, in the source code, using commands like #eval. This is a bit like the interpreter found in Haskell or Python, but without needing a separate process, or like using a Jupyter notebook, but without the stateful cell management.
This is already a nice demonstration of Lean s abilities and strength, as we know them. But what blew my mind the first time was what happened next: He had a visual debugger that allowed him to debug his DSL program. It appeared on the right, in Lean s Info View , where various Lean tools can hook into, show information and allow the user to interact.
But it did not stop there, and my mind was blown a second time: Emilio opened VSCode s Debugger pane on the left, and was able to properly use VSCode s full-fledged debugger frontend for his own little embedded programming language! Complete with highlighting the executed line, with the ability to set breakpoints there, and showing the state of local variables in the debugger.
Having a good debugger is not to be taken for granted even for serious, practical programming languages. Having it for a small embedded language that you just built yourself? I wouldn t have even considered that.
Did it take long?
If I were Emilio s manager I would applaud the demo and then would have to ask how many weeks he spent on that. Coming up with the language, getting the syntax extension right, writing the evaluator and especially learning how the debugger integration into VSCode (using the DAP protocol) works, and then instrumenting his evaluator to speak that protocol that is a sizeable project!
It turns out the answer isn t measured in weeks: it took just one day of coding together with GPT-Codex 5.3. My mind was blown a third time.
Why does Lean make a difference?
I am sure this post is just one of many stories you have read in recent weeks about how new models like Claude Opus 4.6 and GPT-Codex 5.3 built impressive things in hours that would have taken days or more before. But have you seen something like this? Agentic coding is powerful, but limited by what the underlying platform exposes. I claim that Lean is a particularly well-suited platform to unleash the agents versatility.
Here we are using Lean as a programming language, not as a theorem prover (which brings other immediate benefits when using agents, e.g. the produced code can be verified rather than merely plausible, but that s a story to be told elsewhere.)
But arguably because Lean is also a theorem prover, and because of the requirements that stem from that, its architecture is different from that of a conventional programming language implementation:
As a theorem prover, it needs extensible syntax to allow formalizing mathematics in an ergonomic way, but it can also be used for embedding syntax.
As a theorem prover, it needs the ability to run tactics written by the user, hence the ability to evaluate the code right there in the editor.
As a theorem prover, it needs to give access to information such as tactic state, and such introspection abilities unlock many other features such as a debugger for an embedded language.
As a theorem prover, it has to allow tools to present information like the tactic state, so it has the concept of interactive Widgets .
So Lean s design has always made such a feat possible. But it was no easy feat. The Lean API is large, and documentation never ceases to be improvable. In the past, it would take an expert (or someone willing to become one) to pull off that stunt. These days, coding assistants have no issue digesting, understanding and using the API, as Emilio s demo shows.
The combination of Lean s extensibility and the ability of coding agents to make use of that is a game changer to how we can develop software, with rich, deep, flexible and bespoke ways to interact with our code, created on demand.
Where does that lead us?
Emilio actually shared more such demos (Github repository). A visual explorer for the compiler output (have a look at the screenshot. A browser-devtool-like inspection tool for Lean s InfoTree . Any of these provide a significant productivity boost. Any of these would have been a sizeable project half a year ago. Now it s just a few hours of chatting with the agent.
So allow me to try and extrapolate into a future where coding agents have continued to advance at the current pace, and are used ubiquitously. Is there then even a point in polishing these tools, shipping them to our users, documenting them? Why build a compiler explorer for our users, if our users can just ask their agent to build one for them, right then when they need it, tailored to precisely the use case they have, with no unnecessary or confusing feature. The code would be single use, as the next time the user needs something like that the agent can just re-create it, maybe slightly different because every use case is different.
If that comes to pass then Lean may no longer get praise for its nice out-of-the-box user experience, but instead because it is such a powerful framework for ad-hoc UX improvements.
And Emilio wouldn t post demos about his debugger. He d just use it.
This is a copy of an email I used to send internally at work and now
made public. I'm not sure I'll make a habit of posting it here,
especially not twice a year, unless people really like it. Right
now, it's mostly here to keep with my current writing spree going.
This is your bi-yearly reminder that time is changing soon!
What's happening?
For people not on tor-internal, you should know that I've been sending
semi-regular announcements when daylight saving changes occur. Starting
now, I'm making those announcements public so they can be shared with
the wider community because, after all, this affects everyone (kind of).
For those of you lucky enough to have no idea what I'm talking about,
you should know that some places in the world implement what is called
Daylight saving time or DST.
Normally, you shouldn't have to do anything: computers automatically
change time following local rules, assuming they are correctly
configured, provided recent updates have been applied in the case of a
recent change in said rules (because yes, this happens).
Appliances, of course, will likely not change time and will need to
adjusted unless they are so-called "smart" (also known as "part of a bot
net").
If your clock is flashing "0:00" or "12:00", you have no action to take,
congratulations on having the right time once or twice a day.
If you haven't changed those clocks in six months, congratulations, they
will be accurate again!
In any case, you should still consider DST because it might affect some
of your meeting schedules, particularly if you set up a new meeting
schedule in the last 6 months and forgot to consider this
change.
If your location does not have DST
Properly scheduled meetings affecting multiple time zones are set in UTC
time, which does not change. So if your location does not observer
time changes, your (local!) meeting time will not change.
But be aware that some other folks attending your meeting might have
the DST bug and their meeting times will change. They might miss
entire meetings or arrive late as you frantically ping them over IRC,
Matrix, Signal, SMS, Ricochet, Mattermost, SimpleX, Whatsapp, Discord,
Slack, Wechat, Snapchat, Telegram, XMPP, Briar, Zulip, RocketChat,
DeltaChat, talk(1), write(1), actual telegrams, Meshtastic, Meshcore,
Reticulum, APRS, snail mail, and, finally, flying a remote presence
drone to their house, asking what's going on.
(Sorry if I forgot your preferred messaging client here, I tried my
best.)
Be kind; those poor folks might be more sleep deprived as DST steals
one hour of sleep from them on the night that implements the change.
If you do observe DST
If you are affected by the DST bug, your local meeting times will
change access the board. Normally, you can trust that your meetings are
scheduled to take this change into account and the new time should still
be reasonable.
Trust, but verify; make sure the new times are adequate and there are
no scheduling conflicts.
Do this now: take a look at your calendar in two week and in
April. See if any meeting need to be rescheduled because of an
impossible or conflicting time.
When does time change, how and where?
Notice how I mentioned "North America" in the subject? That's a
lie. ("The doctor lies", as they say on the BBC.) Other places,
including Europe, also changes times, just not all at once (and not all
North America).
We'll get into "where" soon, but first let's look at the "how". As you might
already know, the trick is:
Spring forward, fall backwards.
This northern-centric (sorry!) proverb says that clocks will move
forward by an hour this "spring", after moving backwards last
"fall". This is why we lose an hour of work, sorry, sleep. It sucks, to
put it bluntly. I want it to stop and will keep writing those advisories
until it does.
To see where and when, we, unfortunately, still need to go into politics.
USA and Canada
First, we start with "North America" which, really, is just some parts
of USA[1] and Canada[2]. As usual, on the Second Sunday in March (the
8th) at 02:00 local (not UTC!), the clocks will move forward.
This means that properly set clocks will flip from 1:59 to 3:00, coldly
depriving us from an hour of sleep that was perniciously granted 6
months ago and making calendar software stupidly hard to write.
Practically, set your wrist watch and alarm clocks[3] back one hour
before going to bed and go to bed early.
[1] except Arizona (except the Navajo nation), US territories, and
Hawaii
[2] except Yukon, most of Saskatchewan, and parts of British Columbia
(northeast), one island in Nunavut (Southampton Island), one town in
Ontario (Atikokan) and small parts of Quebec (Le
Golfe-du-Saint-Laurent), a list which I keep recopying because I
find it just so amazing how chaotic it is. When your clock has its
own Wikipedia page, you know something is wrong.
[3] hopefully not managed by a botnet, otherwise kindly ask your bot net
operator to apply proper software upgrades in a timely manner
Europe
Next we look at our dear Europe, which will change time on the last
Sunday in March (the 29th) at 01:00 UTC (not local!). I think it
means that, Amsterdam-time, the clocks will flip from 1:59 to 3:00 AM
local on that night.
(Every time I write this, I have doubts. I would welcome independent
confirmation from night owls that observe that funky behavior
experimentally.)
Just like your poor fellows out west, just fix your old-school clocks
before going to bed, and go to sleep early, it's good for you.
Rest of the world with DST
Renewed and recurring apologies again to the people of Cuba, Mexico,
Moldova, Israel, Lebanon, Palestine, Egypt, Chile (except Magallanes
Region), parts of Australia, and New Zealand which all have their own
individual DST rules, omitted here for brevity.
In general, changes also happen in March, but either on different
times or different days, except in the south hemisphere, where they
happen in April.
Rest of the world without DST
All of you other folks without DST, rejoice! Thank you for reminding us
how manage calendars and clocks normally. Sometimes, doing nothing is
precisely the right thing to do. You're an inspiration for us all.
Changes since last time
There were, again, no changes since last year on daylight savings that
I'm aware of. It seems the US congress debating switching to a
"half-daylight" time zone which is an half-baked idea that I
should have expected from the current USA politics.
The plan is to, say, switch from "Eastern is UTC-4 in the summer" to
"Eastern is UTC-4.5". The bill also proposes to do this 90 days after
enactment, which is dangerously optimistic about our capacity at
deploying any significant change in human society.
In general, I rely on the Wikipedia time nerds for this and Paul
Eggert which seems to singlehandledly be keeping everything in order
for all of us, on the tz-announce mailing list.
This time, I've also looked at the tz mailing list which is where
I learned about the congress bill.
If your country has changed time and no one above noticed, now would
be an extremely late time to do something about this, typically
writing to the above list. (Incredibly, I need to write to the list
because of this post.)
One thing that did change since last year is that I've implemented
what I hope to be a robust calendar for this, which was surprisingly
tricky.
If you have access to our Nextcloud, it should be visible under the
heading "Daylight saving times". If you don't, you can access it using
this direct link.
The procedures around how this calendar was created, how this email
was written, and curses found along the way, are also documented in
this wiki page, if someone ever needs to pick up the Time Lord
duty.
When the Free Software movement started in the 1980s, most of the world
had just made a transition from free university-written software to
non-free, proprietary, company-written software. Because of that, the
initial ethical standpoint of the Free Software foundation was that it's
fine to run a non-free operating system, as long as all the software you
run on that operating system is free.
Initially this was just the
editor.
But as time went on, and the FSF managed to write more and more parts of
the software stack, their ethical stance moved with the times. This was
a, very reasonable, pragmatic stance: if you don't accept using a
non-free operating system and there isn't a free operating system yet,
then obviously you can't write that free operating system, and the
world won't move towards a point where free operating systems exist.
In the early 1990s, when
Linus initiated the
Linux kernel, the situation reached the point where the original dream
of a fully free software stack was complete.
Or so it would appear.
Because, in fact, this was not the case. Computers are physical objects,
composed of bits of technology that we refer to as "hardware", but in
order for these bits of technology to communicate with other bits of
technology in the same computer system, they need to interface with
each other, usually using some form of bus protocol. These bus protocols
can get very complicated, which means that a bit of software is required
in order to make all the bits communicate with each other properly.
Generally, this software is referred to as "firmware", but don't let
that name deceive you; it's really just a bit of low-level software that
is very specific to one piece of hardware. Sometimes it's written in an
imperative high-level language; sometimes it's just a set of very simple
initialization vectors. But whatever the case might be, it's always a
bit of software.
And although we largely had a free system, this bit of low-level
software was not yet free.
Initially, storage was expensive, so computers couldn't store as much
data as today, and so most of this software was stored in ROM chips on
the exact bits of hardware they were meant for. Due to this fact, it was
easy to deceive yourself that the firmware wasn't there, because you
never directly interacted with it. We knew it was there; in fact, for
some larger pieces of this type of
software it was possible, even in
those days, to install updates. But that was rarely if ever done at the
time, and it was easily forgotten.
And so, when the free software movement slapped itself on the back and
declared victory when a fully free operating system was available, and
decided that the work of creating a free software environment was
finished, that only keeping it recent was further required, and that we
must reject any further non-free encroachments on our fully free
software stack, the free software movement was deceiving itself.
Because a computing environment can never be fully free if the
low-level pieces of software that form the foundations of that computing
environment are not free. It would have been one thing if the Free
Software Foundation declared it ethical to use non-free low-level
software on a computing environment if free alternatives were not
available. But unfortunately, they did not.
In fact, something very strange happened.
In order for some free software hacker to be able to write a free
replacement for some piece of non-free software, they obviously need to
be able to actually install that theoretical free replacement. This
isn't just a random thought; in fact it has
happened.
Now, it's possible to install software on a piece of rewritable storage
such as flash memory inside the hardware and boot the hardware from
that, but if there is a bug in your software -- not at all unlikely if
you're trying to write software for a piece of hardware that you don't
have documentation for -- then it's not unfathomable that the
replacement piece of software will not work, thereby reducing your
expensive piece of technology to something about as useful as a
paperweight.
Here's the good part.
In the late 1990s and early 2000s, the bits of technology that made up
computers became so complicated, and the storage and memory available to
computers so much larger and cheaper, that it became economically more
feasible to create a small, tiny, piece of software stored in a ROM chip
on the hardware, with just enough knowledge of the bus protocol to
download the rest from the main computer.
This is awesome for free software. If you now write a replacement for
the non-free software that comes with the hardware, and you make a
mistake, no wobbles! You just remove power from the system, let the DRAM
chips on the hardware component fully drain, return power, and try
again. You might still end up with a brick of useless silicon if some
of the things you sent to your technology make it do things that it was
not designed to do and therefore you burn through some critical bits of
metal or plastic, but the chance of this happening is significantly
lower than the chance of you writing something that impedes the boot
process of the piece of hardware and you are unable to fix it because
the flash is overwritten. There is anecdotal
evidence
that there are free software hackers out there who do so. So, yay,
right? You'd think the Free Software foundation would jump at the
possibility to get more free software? After all, a large part of why we
even have a Free Software Foundation in the first place, was because of
some piece of hardware that was
misbehaving,
so you would think that the foundation's founders would understand the
need for hardware to be controlled by software that is free.
The strange thing, what has always been strange to me, is that this is
not what happened.
The Free Software Foundation instead decided that non-free software on
ROM or flash chips is fine, but non-free software -- the very same
non-free software, mind -- that touches the general storage device that
you as a user use, is not. Never mind the fact that the non-free
software is always there, whether it sits on your storage device or not.
Misguidedness aside, if some people decide they would rather not update
the non-free software in their hardware and use the hardware with the
old and potentially buggy version of the non-free software that it came
with, then of course that's their business.
Unfortunately, it didn't quite stop there. If it had, I wouldn't have
written this blog post.
You see, even though the Free Software Foundation was about Software,
they decided that they needed to create a hardware
certification program. And this hardware
certification program ended up embedding the strange concept that if
something is stored in ROM it's fine, but if something is stored on a
hard drive it's not. Same hardware, same software, but different
storage. By that logic, Windows respects your freedom as long as the
software is written to ROM. Because this way, the Free Software
Foundation could come to a standstill and pretend they were still living
in the 90s.
An unfortunate result of the "RYF" program is that it means that
companies who otherwise would have been inclined to create hardware that
was truly free, top to bottom, are now more incentivised by the RYF
program to create hardware in which the non-free low-level software
can't be replaced.
Meanwhile, the rest of the world did not pretend to still be living in
the nineties, and free hardware communities now exist. Because of how
the FSF has marketed themselves out of the world, these communities call
themselves "Open Hardware" communities, rather than "Free Hardware"
ones, but the principle is the same: the designs are there, if you have
the skill you can modify it, but you don't have to.
In the mean time, the open hardware community has evolved to a point
where even CPUs are designed in
the open, which you can design your own version of.
But not all hardware can be implemented as RISC-V, and so if you want a
full system that builds RISC-V you may still need components of the
system that were originally built for other architectures but that would
work with RISC-V, such as a network card or a GPU. And because the FSF
has done everything in their power to disincentivise people who would
otherwise be well situated to build free versions of the low-level
software required to support your hardware, you may now be in the weird
position where we seem to have somehow skipped a step.
My own suspicion is that the universe is not only queerer than we
suppose, but queerer than we can suppose.
-- J.B.S. Haldane
(comments for this post will not pass moderation. Use your own blog!)
Note: I have not published blog posts about my academic papers over the past few years. To ensure that my blog contains a more comprehensive record of my published papers and to surface these for folks who missed them, I will be periodically (re)publishing blog posts about some older published projects. This post is closely based on a previously published post by Zarine Kharazian on the Community Data Science Blog.For nearly a decade, the Croatian language version of Wikipedia was run by a cabal of far-right nationalists who edited articles in ways that promoted fringe political ideas and involved cases of historical revisionism related to the Usta e regime, a fascist movement that ruled the Nazi puppet state called the Independent State of Croatia during World War II. This cabal seized complete control of the encyclopedia s governance, banned and blocked those who disagreed with them, and operated a network of fake accounts to create the appearance of grassroots support for their policies.
Thankfully, Croatian Wikipedia appears to be an outlier. Though both the Croatian and Serbian language editions have been documented to contain nationalist bias and historical revisionism, Croatian Wikipedia seems unique among Wikipedia editions in the extent to which its governance institutions were captured by a small group of users.
The situation in Croatian Wikipedia was well documented and is now largely fixed, but we still know very little about why it was taken over, while other language editions seem to have rebuffed similar capture attempts. In a paper published in the Proceedings of the ACM: Human-Computer Interaction (CSCW), Zarine Kharazian, Kate Starbird, and I present an interview-based study that provides an explanation for why Croatian was captured while several other editions facing similar contexts and threats fared better.
Short video presentation of the work given at Wikimania in August 2023.
Based on insights from interviews with 15 participants from both the Croatian and Serbian Wikipedia projects and from the broader Wikimedia movement, we arrived at three propositions that, together, help explain why Croatian Wikipedia succumbed to capture while Serbian Wikipedia did not:
Perceived Value as a Target. Is the project worth expending the effort to capture?
Bureaucratic Openness. How easy is it for contributors outside the core founding team to ascend to local governance positions?
Institutional Formalization. To what degree does the project prefer personalistic, informal forms of organization over formal ones?
The conceptual model from our paper, visualizing possible institutional configurations among Wikipedia projects that affect the risk of governance capture.
We found that both Croatian and Serbian Wikipedias were attractive targets for far-right nationalist capture due to their sizable readership and resonance with national identity. However, we also found that the two projects diverged early in their trajectories in how open they remained to new contributors ascending to local governance positions and in the degree to which they privileged informal relationships over formal rules and processes as the project s organizing principles. Ultimately, Croatian s relative lack of bureaucratic openness and rules constraining administrator behavior created a window of opportunity for a motivated contingent of editors to seize control of the governance mechanisms of the project.
Though our empirical setting was Wikipedia, our theoretical model may offer insight into the challenges faced by self-governed online communities more broadly. As interest in decentralized alternatives to Facebook and X (formerly Twitter) grows, communities on these sites will likely face similar threats from motivated actors. Understanding the vulnerabilities inherent in these self-governing systems is crucial to building resilient defenses against threats like disinformation.
For more details on our findings, take a look at the published version of our paper.
Citation for the full paper: Kharazian, Zarine, Kate Starbird, and Benjamin Mako Hill. 2024. Governance Capture in a Self-Governing Community: A Qualitative Comparison of the Croatian, Serbian, Bosnian, and Serbo-Croatian Wikipedias. Proceedings of the ACM on Human-Computer Interaction 8 (CSCW1): 61:1-61:26. https://doi.org/10.1145/3637338.
This blog post and the paper it describes are collaborative work by Zarine Kharazian, Benjamin Mako Hill, and Kate Starbird.
The AI hype is based on the assumption that the frontier AI labs are producing better and better foundational models at an accelerating pace. Is that really true, or are people just in sort of a mass psychosis because AI models have become so good at mimicking human behavior that we unconsciously attribute increasing intelligence to them? I decided to conduct a mini-benchmark of my own to find out if the latest and greatest AI models are actually really good or not.
The problem with benchmarks
Every time any team releases a new LLM, they boast how well it performs on various industry benchmarks such as Humanity s Last Exam, SWE-Bench and Ai2 ARC or ARC-AGI. An overall leaderboard can be viewed at LLM-stats. This incentivizes teams to optimize for specific benchmarks, which might make them excel on specific tasks while general abilities degrade. Also, the older a benchmark dataset is, the more online material there is discussing the questions and best answers, which in turn increases the chances of newer models trained on more recent web content scoring better.
Thus I prefer looking at real-time leaderboards such as the LM Arena leaderboard (or OpenCompass for Chinese models that might be missing from LM Arena). However, even though the LM Arena Elo score is rated by humans in real-time, the benchmark can still be played. For example, Meta reportedly used a special chat-optimized model instead of the actual Llama 4 model when getting scored on the LM Arena.
Therefore I trust my own first-hand experience more than the benchmarks for gaining intuition. Intuition however is not a compelling argument in discussions on whether or not new flagship AI models have plateaued. Thus, I decided to devise my own mini-benchmark so that no model could have possibly seen it in its training data or be specifically optimized for it in any way.
My mini-benchmark
I crafted 6 questions based on my own experience using various LLMs for several years and having developed some intuition about what kinds of questions LLMs typically struggle with.
I conducted the benchmark using the OpenRouter.ai chat playroom with the following state-of-the-art models:
OpenRouter.ai is great as it very easy to get responses from multiple models in parallel to a single question. Also it allows to turn off web search to force the models to answer purely based on their embedded knowledge.
Common for all the test questions is that they are fairly straightforward and have a clear answer, yet the answer isn t common knowledge or statistically the most obvious one, and instead requires a bit of reasoning to get correct.
Some of these questions are also based on myself witnessing a flagship model failing miserably to answer it.
1. Which cities have hosted the Olympics more than just once?
This question requires accounting for both summer and winter Olympics, and for Olympics hosted across multiple cities.
The variance in responses comes from if the model understands that Beijing should be counted as it has hosted both summer and winter Olympics. Interestingly GPT was the only model to not mention Beijing at all. Some variance also comes from how models account for co-hosted Olympics. For example Cortina should be counted as having hosted the Olympics twice, in 1956 and 2026, but only Claude, Gemini and Kimi pointed this out. Stockholm s 1956 hosting of the equestrian games during the Melbourne Olympics is a special case, which GPT, Gemini and Kimi pointed out in a side note. Some models seem to have old training material, and for example Grok assumes the current year is 2024. All models that accounted for awarded future Olympics (e.g. Los Angeles 2028) marked them clearly as upcoming.
Overall I would judge that only GPT and MinMax gave incomplete answers, while all other models replied as the best humans could reasonably have.
2. If EUR/USD continues to slide to 1.5 by mid-2026, what is the likely effect on BMW s stock price by end of 2026?
This question requires mapping the currency exchange rate to historic value, dodging the misleading word slide , and reasoning on where the revenue of a company comes from and how a weaker US dollar affects it in multiple ways. I ve frequently witnessed flagship models get it wrong how interest rates and exchange rates work. Apparently the binary choice between up or down is somehow challenging to the internal statistical model in the LLMs on a topic where there are a lot of training material that talk about both things being likely to happen, and choosing between them requires specifically reasoning about the scenario at hand and disregarding general knowledge of the situation.
However, this time all the models concluded correctly that a weak dollar would have a negative overall effect on the BMW stock price. Gemini, GLM, Qwen and Kimi also mention the potential hedging effect of BMW s X-series production in South Carolina for worldwide export.
3. What is the Unicode code point for the traffic cone emoji?
This was the first question where the the flagship models clearly still struggle in 2026. The trap here is that there is no traffic cone emoji, so an advanced model should simply refuse to give any Unicode numbers at all. Most LLMs however have an urge to give some answer, leading to hallucinations. Also, as the answer has a graphical element to it, the LLM might not understand how the emoji looks in ways that would be obvious to a human, and thus many models claim the construction sign emoji is a traffic cone, which it is not.
By far the worst response was from GPT, that simply hallucinates and stops there:
While Gemini and Grok were among the three models not falling into this trap, the response from Claude was exemplary good:
4. Which languages are the 10 most commonly spoken in the world? For each language, count from 1 to 5.
This seemingly simple question can become quite complex when you account for the fact that other languages might also have their own character system, both for words and the numerals (e.g. Arabic /one vs /1), and that numbers might have variations depending on what the thing being counted is (e.g in French object gender un/une), or whether plain numbers and occurrences have their own words (e.g. Russian /odin vs /raz). I ve also witnessed older models frequently mix up on what parts of the reply should be using what language.
For a human this question is trivial, assuming they know the language. For LLMs it is surprisingly challenging. Grok and MinMax only gave a list of the most popular languages and stopped there without showing any words in the languages. Qwen got a bit further by trying to list numbers in each language, but the lists were only the numbers 1 to 5:
I have seen Grok do something similar before, which in fact inspired me to include this question in my mini-benchmark.
GPT got a bit further, but for Hindi, Arabic and Bengali it listed the numerals in local script, not the number words. Gemini, GLM and Kimi gave a complete and correct answer as a list, while the absolute best answer and presentation was by Claude, that gave the table below:
Claude Opus 4.6
10 Most Commonly Spoken Languages (by Total Speakers)
#
Language
1
2
3
4
5
1
English
One
Two
Three
Four
Five
2
Mandarin Chinese
(y )
( r)
(s n)
(s )
(w )
3
Hindi
(ek)
(do)
(t n)
(ch r)
(p ch)
4
Spanish
Uno
Dos
Tres
Cuatro
Cinco
5
French
Un
Deux
Trois
Quatre
Cinq
6
Arabic
(w id)
(ithn n)
(thal tha)
(arba a)
(khamsa)
7
Bengali
(ek)
(dui)
(tin)
(ch r)
(p ch)
8
Portuguese
Um
Dois
Tr s
Quatro
Cinco
9
Russian
(odin)
(dva)
(tri)
(chetyre)
(pyat')
10
Japanese
(ichi)
(ni)
(san)
(shi)
(go)
Note: Rankings are based on estimated total speakers (native + non-native). Rankings can vary depending on the source and methodology used.
5. Count the number of drone launch pads in the picture.
Together with the question, I uploaded this picture:
A human can easily count that there are 10 rows and 30+ columns in the grid, but because the picture resolution isn t good enough, the exact number of columns can t be counted, and the answer should be that there are at least 300 launch pads in the picture.
GPT and Grok both guessed the count is zero. Instead of hallucinating some number they say zero, but it would have been better to not give any number at all, and just state that they are unable to perform the task. Gemini gave as its answer 101 , which is quite odd, but reading the reasoning section, it seems to have tried counting items in the image without reasoning much about what it is actually counting and that there is clearly a grid that can make the counting much easier. Both Qwen and Kimi state they can see four parallel structures, but are unable to count drone launch pads.
The absolutely best answer was given by Claude, which counted 10-12 rows and 30-40+ columns, and concluded that there must be 300-500 drone launch pads. Very close to best human level - impressive!
This question applied only to multi-modal models that can see images, so GLM and MinMax could not give any response.
6. Explain why I am getting the error below, and what is the best way to fix it?
Together with the question above, I gave this code block:
$ SH_SCRIPTS="$(mktemp; grep -Irnw debian/ -e '^#!.*/sh' sort -u cut -d ':' -f 1 true)"
$ shellcheck -x --enable=all --shell=sh "$SH_SCRIPTS"
/tmp/tmp.xQOpI5Nljx
debian/tests/integration-tests: /tmp/tmp.xQOpI5Nljx
debian/tests/integration-tests: openBinaryFile: does not exist (No such file or directory)
$ SH_SCRIPTS="$(mktemp; grep -Irnw debian/ -e '^#!.*/sh' sort -u cut -d ':' -f 1 true)"
$ shellcheck -x --enable=all --shell=sh "$SH_SCRIPTS"
/tmp/tmp.xQOpI5Nljx
debian/tests/integration-tests: /tmp/tmp.xQOpI5Nljx
debian/tests/integration-tests: openBinaryFile: does not exist (No such file or directory)
Older models would easily be misled by the last error message thinking that a file went missing, and focus on suggesting changes to the complex-looking first line. In reality the error is simply caused by having the quotes around the $SH_SCRIPTS, resulting in the entire multi-line string being passed as a single argument to shellcheck. So instead of receiving two separate file paths, shellcheck tries to open one file literally named /tmp/tmp.xQOpI5Nljx\ndebian/tests/integration-tests.
Incorrect argument expansion is fairly easy for an experienced human programmer to notice, but tricky for an LLM. Indeed, Grok, MinMax, and Qwen fell for this trap and focused on the mktemp, assuming it somehow fails to create a file. Interestingly GLM fails to produce an answer at all, as the reasoning step seems to be looping, thinking too much about the missing file, but not understanding why it would be missing when there is nothing wrong with how mktemp is executed.
Claude, Gemini, and Kimi immediately spot the real root cause of passing the variable quoted and suggested correct fixes that involve either removing the quotes, or using Bash arrays or xargs in a way that makes the whole command also handle correctly filenames with spaces in them.
Conclusion
Model
Sports
Economics
Emoji
Languages
Visual
Shell
Score
Claude Opus 4.6
6/6
GPT-5.2
~
2.5/6
Grok 4.1
3/6
Gemini 3.1 Pro
5/6
GLM 5
?
N/A
3/5
MinMax M2.5
N/A
1/5
Qwen3.5 Plus
~
2.5/6
Kimi K2.5
4/6
Obviously, my mini-benchmark only had 6 questions, and I ran it only once. This was obviously not scientifically rigorous. However it was systematic enough to trump just a mere feeling.
The main finding for me personally is that Claude Opus 4.6, the flagship model by Anthropic, seems to give great answers consistently. The answers are not only correct, but also well scoped giving enough information to cover everything that seems relevant, without blurping unnecessary filler.
I used Claude extensively in 2023-2024 when it was the main model available at my day work, but for the past year I had been using other models that I felt were better at the time. Now Claude seems to be the best-of-the-best again, with Gemini and Kimi as close follow-ups. Comparing their pricing at OpenRouter.ai the Kimi K2.5 price of $0.6 / million tokens is almost 90% cheaper than the Claude Opus 4.6 s $5.0 / million tokens suggests that Kimi K2.5 offers the best price-per-performance ratio. Claude might be cheaper with a monthly subscription directly from Anthropic, potentially narrowing the price gap.
Overall I do feel that Anthropic, Google and Moonshot.ai have been pushing the envelope with their latest models in a way that one can t really claim that AI models have plateaued. In fact, one could claim that at least Claude has now climbed over the hill of AI slop and consistently produces valuable results. If and when AI usage expands from here, we might actually not drown in AI slop as chances of accidentally crappy results decrease. This makes me positive about the future.
I am also really happy to see that there wasn t just one model crushing everybody else, but that there are at least three models doing very well. As an open source enthusiast I am particularly glad to see that Moonshot.ai s Kimi K2.5 is published with an open license. Given the hardware, anyone can run it on their own. OpenRouter.ai currently lists 9 independent providers alongside Moonshot.ai itself, showcasing the potential of open-weight models in practice.
If the pattern holds and flagship models continue improving at this pace we might look back at 2026 as the year AI stopped feeling like a call center associate and started to resemble a scientific researcher. While new models become available we need to keep testing, keep questioning, and keep our expectations grounded in actual performance rather than press releases.
Thanks to OpenRouter.ai for providing a great service that makes testing various models incredibly easy!
Since Generative AI is currently the most popular topic, I wanted to get my
hands dirty and learn something new. I was learning about the Model Context
Protocol at the time and wanted to apply it to build something simple.
Idea
On Debian systems, we use debsecan to find vulnerabilities. However, the tool
currently provides a simple list of vulnerabilities and packages with no
indication of the system's security posture meaning no criticality information
is exposed and no executive summary is provided regarding what needs to be
fixed. Of course, one can simply run the following to install existing fixes and
be done with it:
But this is not how things work in corporate environments; you need to provide a
report showing the system's previous state and the actions taken to bring it to
a safe state. It is all about metrics and reports.
My goal was to use debsecan to generate a list of vulnerabilities, find more
detailed information on them, and prioritize them as critical, high, medium, or
low. By providing this information to an AI, I could ask it to generate an
executive summary report detailing what needs to be addressed immediately and
the overall security posture of the system.
Initial Take
My initial thought was to use an existing LLM, either self-hosted or a
cloud-based LLM like Gemini (which provides an API with generous limits via AI
Studio). I designed functions to output the list of vulnerabilities on the
system and provide detailed information on each. The idea was to use these as
"tools" for the LLM.
Learnings
I learned about open-source LLMs using Ollama, which allows you to download
and use models on your laptop.
I used Llama 3.1, Llama 3.2, and Granite 4 on my laptop without a GPU. I
managed to run my experiments, even though they were time-consuming and
occasionally caused my laptop to crash.
I learned about Pydantic and how to use it to parse custom JSON schemas with
minimal effort.
I learned about osv.dev, an open-source initiative by Google that aggregates
vulnerability information from various sources and provides data in a
well-documented JSON schema format.
I learned about the EPSS (Exploit Prediction Scoring System) and how it is
used alongside static CVSS scoring to detect truly critical vulnerabilities.
The EPSS score provides an idea of the probability of a vulnerability being
exploited in the wild based on actual real-world attacks.
These experiments led to a collection of notebooks. One key takeaway
was that when defining tools, I cannot simply output massive amounts of text
because it consumes tokens and increases costs for paid models (though it is
fine for local models using your own hardware and energy). Self-hosted models
require significant prompting to produce proper output, which helped me
understand the real-world application of prompt engineering.
Change of Plans
Despite extensive experimentation, I felt I was nowhere close to a full
implementation. While using a Gemini learning tool to study MCP, it suddenly
occurred to me: why not write the entire thing as an MCP server? This would save
me from implementing the agent side and allow me to hook it into any
IDE-based LLM.
Design
This MCP server is primarily a mix of a "tool" (which executes on the server
machine to identify installed packages and their vulnerabilities) and a
"resource" (which exposes read-only information for a specific CVE ID).
The MCP exposes two tools:
List Vulnerabilities: This tool identifies vulnerabilities in the packages
installed on the system, categorizes them using CVE and EPSS scores, and
provides a dictionary of critical, high, medium, and low vulnerabilities.
Research Vulnerabilities: Based on the user prompt, the LLM can identify
relevant vulnerabilities and pass them to this function to retrieve details
such as whether a fix is available, the fixed version, and criticality.
Vibe Coding
"Vibe coding" is the latest trend, with many claiming that software engineering
jobs are a thing of the past. Without going into too much detail, I decided to
give it a try. While this is not my first "vibe coded" project (I have done this
previously at work using corporate tools), it is my first attempt to vibe code a
hobby/learning project.
I chose Antigravity because it seemed to be the only editor providing a
sufficient amount of free tokens. For every vibe coding project, I spend time
thinking about the barebones skeleton: the modules, function return values, and
data structures. This allows me to maintain control over the LLM-generated code
so it doesn't become overly complicated or incomprehensible.
As a first step, I wrote down my initial design in a requirements document. In
that document, I explicitly called for using debsecan as
the model for various components. Additionally, I asked the AI to reference my
specific code for the EPSS logic.
The reasons were:
debsecan already solves the core problem; I am simply rebuilding it.
debsecan uses a single file generated by the Debian Security team
containing all necessary information, which prevents us from needing multiple
external sources.
This provides the flexibility to categorize vulnerabilities within the
listing tool itself since all required information is readily available,
unlike my original notebook-based design.
I initially used Gemini 3 Flash as the model because I was concerned about
exceeding my free limits.
Hiccups
Although it initially seemed successful, I soon noticed discrepancies between
the local debsecan outputs and the outputs generated by the tools. I asked the
AI to fix this, but after two attempts, it still could not match the outputs. I
realized it was writing its own version-comparison logic and failing
significantly.
Finally, I instructed it to depend entirely on the python-apt module for
version comparison; since it is not on PyPI, I asked it to pull directly from
the Git source. This solved some issues, but the problem persisted. By then, my
weekly quota was exhausted, and I stopped debugging.
A week later, I resumed debugging with the Claude 3.5 Sonnet model. Within 20-25
minutes, it found the fix, which involved four lines of changes
in the parsing logic. However, I ran out of limits again before I could proceed
further.
In the requirements, I specified that the list vulnerabilities tool should
only provide a dictionary of CVE IDs divided by severity. However, the AI
instead provided full text for all vulnerability details, resulting in excessive
data including negligible vulnerabilities being sent to the LLM. Consequently,
it never called the research vulnerabilities tool. Since I had run out of
limits, I manually fixed this in a follow-up commit.
How to Use
I have published the current work in the debsecan-mcp repository. I have used the same
license as the original debsecan. I am not entirely sure how to interpret
licenses for vibe-coded projects, but here we are.
To use this, you need to install the tool in a virtual environment and configure
your IDE to use the MCP. Here is how I set it up for Visual Studio Code:
I am running it directly from my local codebase using a virtualenv created
with uv. You may need to tweak the path based on your installation.
To use the MCP server in the Copilot chat window, reference it using
#debsecan-mcp. The LLM will then use the server for the query.
Use a prompt like: "Give an executive summary of the system security status
and immediate actions to be taken."
You can observe the LLM using list_vulnerabilities followed by
research_cves. Because the first tool only provides CVE IDs based on
severity, the LLM is smart enough to research only high and critical
vulnerabilities, thereby saving tokens.
What's Next?
This MCP is not yet perfect and has the following issues:
The list_vulnerabilities dictionary contains duplicate CVE IDs because the
code used a list instead of a set. While the LLM is smart enough to
deduplicate these, it still costs extra tokens.
Because I initially modeled this on debsecan, it uses a raw method for
parsing /var/lib/dpkg/status instead of python-apt. I am considering
switching to python-apt to reduce maintenance overhead.
Interestingly, the AI did not add a single unit test, which is disappointing.
I will add these once my limits are restored.
I need to create a cleaner README with usage instructions.
I need to determine if the MCP can be used via HTTP as well as stdio.
Conclusion
Vibe coding is interesting, but things can get out of hand if not managed
properly. Even with a good process, code must be reviewed and tested; you cannot
blindly trust an AI to handle everything. Even if it adds tests, you must
validate them, or you are doomed!
tl;dr:
To the question: what does it take to upgrade OpenStack , my personal answer is: less than 2K lines of dash script. I ll here describe its internals, and why I believe it is the correct solution.
Why writing this blog post
During FOSSDEM 2024, I was asked how to you handle upgrades . I answered with a big smile and a short with a very small shell script as I couldn t explain in 2 minutes how it was done. Just saying it is great this way doesn t help giving readers enough hints to be trusted. Why and how did I do it the right way ? This blog post is an attempt to reply better to this question more deeply.
Upgrading OpenStack in production
I wrote this script maybe a 2 or 3 years ago. Though I m only blogging about it today, because I did such an upgrade in a public cloud in production last Thuesday evening (ie: the first region of the Infomaniak public cloud). I d say the cluster is moderately large (as of today: about 8K+ VMs running, 83 compute nodes, 12 network nodes, for a total of 10880 physical CPU cores and 125 TB of RAM if I only count the compute servers). It took only 4 hours to do the upgrade (though I already wore some more code to speed this up for the next time ). It went super smooth without a glitch. I mostly just sat, reading the script output and went to bed once it finished running. The next day, all my colleagues at Infomaniak were nicely congratulating me that it went that smooth (a big thanks to all of you who did). I couldn t dream of an upgrade that smooth! :)
Still not impressed? Boring read? Yeah let s dive into more technical details.
Intention behind the implementation
My script isn t perfect. I wont ever pretend it is. But at least, it does minimize down time of every OpenStack service. It also is a by the book implementation of what s written in the OpenStack doc, following every upstream advice. As a result, it is fully seamless for some OpenStack services, and as HA as OpenStack can be for others. The upgrade process is of course idempotent and can be re-run in case of failure. Here s why.
General idea
My upgrade script does thing in a certain order, respecting what is documented about upgrades in the OpenStack documentation. It basically does:
Upgrade all dependency
Upgrade all services one by one, in all the cluster
Installing dependencies
The first thing the upgrade script does is:
disable puppet on all nodes of the cluster
switch the APT repository
apt-get update on all nodes
install library dependency on all nodes
For this last thing, a static list of all needed dependency upgrade is maintained between each release of OpenStack, and for each type of nodes. Then for all packages in this list, the script checks with dpkg-query that the package is really installed, and with apt-cache policy that it really is going to be upgraded (Maybe there s an easier way to do this?). This way, no package is marked as manually installed by mistake during the upgrade process. This ensure that apt-get purge autoremove really does what it should, and that the script is really idempotent.
The idea then, is that once all dependencies are installed, upgrading and restarting leaf packages (ie: OpenStack services like Nova, Glance, Cinder, etc.) is very fast, because the apt-get command doesn t need to install all dependencies. So at this point, doing apt-get install python3-cinder for example (which will also, thanks to dependencies, upgrade cinder-api and cinder-scheduler, if it s in a controller node) only takes a few seconds. This principle applies to all nodes (controller nodes, network nodes, compute nodes, etc.), which helps a lot speeding-up the upgrade and reduce unavailability.
hapc
At its core, the oci-cluster-upgrade-openstack-release script uses haproxy-cmd (ie: /usr/bin/hapc) to drain each API server to-be-upgraded from haproxy. Hapc is a simple Python wrapper around the haproxy admin socket: it sends command to it with an easy to understand CLI. So it is possible to reliably upgrade one API service only after it s drained away. Draining means one just wait for the last query to finish and the client to disconnect from http before giving the backend server some more queries. If you do not know hapc / haproxy-cmd, I recommend trying it: it s going to be hard for you to stop using it once you tested it. Its bash-completion script makes it VERY easy to use, and it is helpful in production. But not only: it is also nice to have when writing this type of upgrade script. Let s dive into haproxy-cmd.
Example on how to use haproxy-cmd
Let me show you. First, ssh one of the 3 controller and search where the virtual IP (VIP) is located with crm resource locate openstack-api-vip or with a (more simple) crm status . Let s ssh that server who got the VIP, and now, let s drain it away from haproxy.
$ hapc list-backends $ hapc drain-server --backend glancebe --server cl1-controller-1.infomaniak.ch --verbose --wait --timeout 50 $ apt-get install glance-api $ hapc enable-server --backend glancebe --server cl1-controller-1.infomaniak.ch
Upgrading the control plane
My upgrade script leverages hapc just like above. For each OpenStack project, it s done in this order on the first node holding the VIP:
hapc drain-server of the API, so haproxy gracefully stops querying it
stop all services on that node (including non-API services): stop, disable and mask with systemd.
upgrade that service Python code. For example: apt-get install python3-nova , which also will pull nova-api, nova-conductor, nova-novncprox, etc. but services wont start automatically as they ve been stoped + disabled + masked on the previous bullet point.
perform the db_sync so that the db is up-to-date [1]
start all services (unmask, enable and start with systemd)
re-enable the API backend with hapc enable-server
Starting at [1], the risk is that other nodes may have a new version of the database schema, but an old version of the code that isn t compatible with it. But it doesn t take long, because the next step is to take care of the other (usually 2) nodes of the OpenStack control plane:
hapc drain-server of the API of the other 2 controllers
stop of all services on these 2 controllers [2]
upgrade of the package
start of all services
So while there s technically zero down time, still some issues between [1] and [2] above may happen because of the new DB schema and the old code (both for API and other services) are up and running at the same time. It is however supposed to be rare cases (some OpenStack project don t even have db change between some OpenStack releases, and it often continues to work on most queries with the upgraded db), and the cluster will be like that for a very short time, so that s fine, and better than an full API down time.
Satellite services
Then there s satellite services, that needs to be upgraded. Like Neutron, Nova, Cinder. Nova is the least offender as it has all the code to rewrite Json object schema on-the-fly so that it continues to work during an upgrade. Though it s a known issue that Cinder doesn t have the feature (last time I checked), and it s also probably the same for Neutron (maybe recent-ish versions of OpenStack do use oslo.versionnedobjects ?). Anyways, upgrade on these nodes are done just right after the control plane for each service.
Parallelism and upgrade timings
As we re dealing with potentially hundreds of nodes per cluster, a lot of operations are performed in parallel. I choose to simply use the & shell thingy with some wait shell stuff so that not too many jobs are done in parallel. For example, when disabling SSH on all nodes, this is done 24 nodes at a time. Which is fine. And the number of nodes is all depending on the type of thing that s being done. For example, while it s perfectly OK to disable puppet on 24 nodes at the same time, but it is not OK to do that with Neutron services. In fact, each time a Neutron agent is restarted, the script explicitly waits for 30 seconds. This conveniently avoids a hailstorm of messages in RabbitMQ, and neutron-rpc-server to become too busy. All of these waiting are necessary, and this is one of the reasons why can sometimes take that long to upgrade a (moderately big) cluster.
Not using config management tooling
Some of my colleagues would have prefer that I used something like Ansible. Whever, there s no reason to use such tool if the idea is just to perform some shell script commands on every servers. It is a way more efficient (in terms of programming) to just use bash / dash to do the work. And if you want my point of view about Ansible: using yaml for doing such programming would be crasy. Yaml is simply not adapted for a job where if, case, and loops are needed. I am well aware that Ansible has workarounds and it could be done, but it wasn t my choice.
This is also known as: "ifconfig is not installed by default
anymore, how do I do this only with the ip command?"
I have been slowly training my brain to use the new commands but I
sometimes forget some. So, here's a couple of equivalence from the old
package to net-tools the new iproute2, about 10 years late:
Another trick
Also note that I often alias ip to ip -br -c as it provides a
much prettier output.
Compare, before:
anarcat@angela:~> ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff permaddr xx:xx:xx:xx:xx:xx
altname wlp166s0
altname wlx8cf8c57333c7
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
20: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
inet 192.168.0.108/24 brd 192.168.0.255 scope global dynamic noprefixroute eth0
valid_lft 40699sec preferred_lft 40699sec
After:
anarcat@angela:~> ip -br -c a
lo UNKNOWN 127.0.0.1/8 ::1/128
wlan0 DOWN
virbr0 DOWN 192.168.122.1/24
eth0 UP 192.168.0.108/24
I don't even need to redact MAC addresses! It also affects the display
of the other commands, which look similarly neat.
Also imagine pretty colors above.
Finally, I don't have a cheat sheet for iw vs iwconfig (from
wireless-tools) yet. I just use NetworkManager now and rarely have
to mess with wireless interfaces directly.
Background and history
For context, there are traditionally two ways of configuring the
network in Linux:
the old way, with commands like ifconfig, arp, route and
netstat, those are part of the net-tools package
the new way, mostly (but not entirely!) wrapped in a single ip
command, that is the iproute2 package
It seems like the latter was made "important" in Debian in 2008,
which means every release since Debian 5 "lenny" has featured the
ip command.
The former net-tools package was demoted in December 2016 which
means every release since Debian 9 "stretch" ships without an
ifconfig command unless explicitly requested. Note that this was
mentioned in the release notes in a similar (but, IMHO, less
useful) table.
(Technically, the net-tools Debian package source still indicates it
is Priority: important but that's a bug I have just filed.)
Finally, and perhaps more importantly, the name iproute is hilarious
if you are a bilingual french speaker: it can be read as "I proute"
which can be interpreted as "I fart" as "prout!" is the sound a fart
makes. The fact that it's called iproute2 makes it only more
hilarious.
The FAI.me service has reached another milestone:
The 42.000th job was submitted via the web interface
since the beginning of this service in 2017.
The idea was to provide a simple web interface for end users for
creating the configs for the fully automatic installation with only
minimal questions and without knowing the syntax of the configuration files.
Thanks a lot for using this service and for all your feedback.
The next job can be yours!
P.S.: I like to get more feedback for the FAI.me service. What do you
like most? What's missing? Do you have any success story how you use the
customized ISO for your deployment? Please fill out the
FAI questionaire
or sent feedback via email to fai.me@fai-project.org
About FAI.me
FAI.me is the service for building your own customized images via a web
interface. You can create an installation or live ISO or a cloud
image.
For Debian, multiple release versions can be chosen, as well as
installations for Ubuntu Server, Ubuntu Desktop, or Linux Mint.
Multiple options are available like selecting different desktop
environments, the language and keyboard and adding a user with a
password.
Optional settings include adding your own package list,
choosing a backports kernel, adding a postinst script
and adding a ssh public key, choosing a partition
layout and some more.
Activity summary
During the month of January, 20 contributors have been
paid to work on Debian LTS (links to individual
contributor reports are located below).
The team released 33 DLAs
fixing 216 CVEs.
The team continued preparing security updates in its usual rhythm. Beyond the
updates targeting Debian 11 ( bullseye ), which is the current release under LTS,
the team also proposed updates for more recent releases (Debian 12 ( bookworm )
and Debian 13 ( trixie )), including Debian unstable. We highlight several notable
security updates here below.
Notable security updates:
python3.9, prepared by Andrej Shadura
(DLA-4455-1),
fixing multiple vulnerabilities in the Python interpreter.
php, prepared by Guilhem Moulin
(DLA-4447-1),
fixing two vulnerabilities that could yield to request forgery or denial of
service.
apache2, prepared by Bastien Roucari s
DLA-4452-1, fixing
four CVEs.
linux-6.1, prepared by Ben Hutchings
(DLA-4436-1), as a
regular update of the linux 6.1 backport to Debian 11.
python-django, prepared by Chris Lamb
(DLA-4458-1),
resolving multiple vulnerabilities.
firefox-esr prepared by Emilio Pozuelo Monfort
(DLA-4439-1)
gnupg2, prepared by Roberto S nchez
(DLA-4437-1),
fixing multiple issues, including
CVE-2025-68973
that could potentially be exploited to execute arbitrary code.
apache-log4j2, prepared by Markus Koschany
(DLA-4444-1)
inetutils, prepared by Andreas Henriksson
(DLA-4453-1),
fixing an authentication bypass in telnetd.
Moreover, Sylvain Beucler studied the security support status of p7zip, a fork
of 7zip that has become unmaintained upstream. To avoid letting the users
continue using an unsupported package, Sylvain has investigated a path forward
in collaboration with the security team and the 7zip maintainer, looking to
replace p7zip with 7zip. It is to note however that 7zip developers don t
reveal the information about the patches that fix CVEs, making it difficult
to backport single patches to fix vulnerabilities in Debian released versions.
Contributions from outside the LTS Team:
Thunderbird, prepared by maintainer Christoph Goehre. The DLA
(DLA-4442-1) was
published by Emilio.
The LTS Team has also contributed with updates to the latest Debian releases:
Bastien uploaded gpsd to
unstable,
and proposed updates for trixie #1126121
and bookworm #1126168 to fix two CVEs.
Bastien also prepared the imagemagick updates for trixie and bookworm,
released as
DSA-6111-1, along
with the bullseye update
DLA-4448-1.
Chris proposed a trixie point update for python-django
(#112646), and the work for bookworm was
completed in February (#1079454). The
longstanding bookworm update required tracking down a regression in the
django-storages packages.
Markus prepared tomcat10 updates for trixie and bookworm
(DSA-6120-1), and
tomcat11 for trixie
(DSA-6121-1)
Thorsten Alteholz prepared bookworm point updates for zvbi
(#1126167) to
fix five CVEs; taglib
(#1126273) to fix
one CVE; and libuev
(#1126370) to fix
one CVE.
Utkarsh prepared an unstable update of
node-lodash
to fix one CVE.
Other than the work related to updates, Sylvain made several improvements to
the documentation and tooling used by the team.
In the Tor Project system Administrator's team (colloquially known as
TPA), we've recently changed how we take decisions, which means you'll
get clearer communications from us about upcoming changes or
targeted questions about a proposal.
Note that this change only affects the TPA team. At Tor, each team has
its own way of coordinating and making decisions, and so far this
process is only used inside TPA. We encourage other teams inside and
outside Tor to evaluate this process to see if it can improve your
processes and documentation.
The new process
We had traditionally been using a "RFC" ("Request For Comments")
process and have recently switched to "ADR" ("Architecture Decision
Record").
The ADR process is, for us, pretty simple. It consists of three
things:
a simpler template
a simpler process
communication guidelines separate from the decision record
The template
As team lead, the first thing I did was to propose a new template (in
ADR-100), a variation of the Nygard template. The TPA
variation of the template is similarly simple, as it has only 5
headings, and is worth quoting in full:
Context: What is the issue that we're seeing that is motivating
this decision or change?
Decision: What is the change that we're proposing and/or doing?
Consequences: What becomes easier or more difficult to do
because of this change?
More Information (optional): What else should we know? For
larger projects, consider including a timeline and cost estimate,
along with the impact on affected users (perhaps including existing
Personas). Generally, this includes a short evaluation of
alternatives considered.
Metadata: status, decision date, decision makers, consulted,
informed users, and link to a discussion forum
The previous RFC template had 17 (seventeen!) headings, which
encouraged much longer documents. Now, the decision record will be
easier to read and digest at one glance.
An immediate effect of this is that I've started using GitLab issues
more for comparisons and brainstorming. Instead of dumping in a
document all sorts of details like pricing or in-depth alternatives
comparison, we record those in the discussion issue, keeping the
document shorter.
The process
The whole process is simple enough that it's worth quoting in full as
well:
Major decisions are introduced to stakeholders in a meeting, smaller
ones by email. A delay allows people to submit final comments before
adoption.
Now, of course, the devil is in the details (and ADR-101), but the
point is to keep things simple.
A crucial aspect of the proposal, which Jacob Kaplan-Moss calls the
one weird trick, is to "decide who decides". Our previous process
was vague about who makes the decision and the new template (and
process) clarifies decision makers, for each decision.
Inversely, some decisions degenerate into endless discussions around
trivial issues because too many stakeholders are consulted, a
problem known as the Law of triviality, also known as the "Bike
Shed syndrome".
The new process better identifies stakeholders:
"informed" users (previously "affected users")
"consulted" (previously undefined!)
"decision maker" (instead of the vague "approval")
Picking those stakeholders is still tricky, but our definitions are
more explicit and aligned to the classic RACI matrix (Responsible,
Accountable, Consulted, Informed).
Communication guidelines
Finally, a crucial part of the process (ADR-102) is to decouple
the act of making and recording decisions from communicating about
the decision. Those are two radically different problems to
solve. We have found that a single document can't serve both purposes.
Because ADRs can affect a wide range of things, we don't have a
specific template for communications. We suggest the Five Ws
method (Who? What? When? Where? Why?) and, again, to keep things simple.
the RFC process "doesn't include any sort of decision-making framework"
"RFC processes tend to lead to endless discussion"
the process "rewards people who can write to exhaustion"
"these processes are insensitive to expertise", "power dynamics and
power structures"
And, indeed, I have been guilty of a lot of those issues. A verbose
writer, I have written extremely long proposals that I suspect no
one has ever fully read. Some proposals were adopted by exhaustion, or
ignored because not looping in the right stakeholders.
Our discussion issue on the topic has more details on the issues I
found with our RFC process. But to give credit to the old process, it
did serve us well while it was there: it's better than nothing, and it
allowed us to document a staggering number of changes and decisions
(95 RFCs!) made over the course of 6 years of work.
What's next?
We're still experimenting with the communication around decisions, as
this text might suggest. Because it's a separate step, we also have a
tendency to forget or postpone it, like this post, which comes a
couple of months late.
Previously, we'd just ship a copy of the RFC to everyone, which was
easy and quick, but incomprehensible to most. Now we need to write a
separate communication, which is more work but, hopefully, worth the
as the result is more digestible.
We can't wait to hear what you think of the new process and how it
works for you, here or in the discussion issue! We're particularly
interested in people that are already using a similar process, or that
will adopt one after reading this.
Note: this article was also published on the Tor Blog.
You might see a verification screen pop up on more and more Debian web properties. Unfortunately the AI world of today is meeting web hosts that use Perl CGIs and are not built as multi-tiered scalable serving systems. The issues have been at three layers:
Apache's serving capacity runs full - with no threads left to serve requests. This means that your connection will sit around for a long time, not getting accepted. In theory this can be configured, but that would require requests to be handled in time.
Startup costs of request handlers are too high, because we spawn a process for every request. This currently affects the BTS and dgit's browse interface. packages.debian.org has been fixed, which increased scalability sufficiently.
Requests themselves are too expensive to be served quickly - think git blame without caching.
Optimally we would go and solve some scalability issues with the services, however there is also a question of how much we want to be able to serve - as AI scraper demand is just a steady stream of requests that are not shown to humans.
How is it implemented?
DSA has now stood up some VMs with Varnish for proxying. Incoming TLS is provided by hitch, and TLS "on-loading" is done using haproxy. That way TLS goes in and TLS goes out. While Varnish does cache, if the content is cachable (e.g. does not depend on cookies) - that is not the primary reason for using it: It can be used for flexible query and response rewriting.
If no cookie with a proof of work is provided, the user is redirected to a challenge page that does some webcrypto in Javascript - because that looked similar to what other projects do (e.g. haphash that originally inspired the solution). However so far it looks like scrapers generally do not run with Javascript enabled, so this whole crypto proof of work business could probably be replaced with just a Javascript-based redirect. The existing solution also has big (security) holes in it. And, as we found out, Firefox is slower at webcrypto than Chrome. I have recently reduced the complexity, so you should notice it blocking you significantly less.
Once you have the cookie, you can keep accessing the site for as long as the cookie is valid. Please do not make any assumptions about the cookies, or you will be broken in the future.For legitimate scrapers that obey robots.txt, there is now an automatically generated IP allowlist in place (thanks, Marco d'Itri). Turns out that the search engines do not actually run Javascript either and then loudly complain about the redirect to the challenge page. Other bots are generally exempt.
Conclusion I hope that right now we found sort of the sweet spot where the admins can stop spending human time on updating firewall rules and the services are generally available, reasonably fast, and still indexed. In case you see problems or run into a block with your own (legitimate) bots, please let me know.
What if I told you there is a way to configure the network on any
Linux server that:
works across all distributions
doesn't require any software installed apart from the kernel and a
boot loader (no systemd-networkd, ifupdown, NetworkManager,
nothing)
is backwards compatible all the way back to Linux 2.0, in 1996
It has literally 8 different caveats on top of that, but is still
totally worth your time.
Known options in Debian
People following Debian development might have noticed there are now
four ways of configuring the network Debian system. At least that is
what the Debian wiki claims, namely:
ifupdown (/etc/network/interfaces): traditional static
configuration system, mostly for workstations and servers that has
been there forever in Debian (since at least 2000), documented
in the Debian wiki
NetworkManager: self-proclaimed "standard Linux network
configuration", mostly used on desktops but technically supports
servers as well, see the Debian wiki page (introduced in 2004)
systemd-network: used more for servers, see Debian reference Doc
Chapter 5 (introduced some time around Debian 8 "jessie", in
2015)
At this point, I feel ifupdown is on its way out, possibly replaced
by systemd-networkd. NetworkManager already manages most desktop
configurations.
A "new" network configuration system
The method is this:
ip= on the Linux kernel command line: for servers with a
single IPv4 or IPv6 address, no software required other than the
kernel and a boot loader (since 2002 or older)
So by "new" I mean "new to me". This option is really old. The
nfsroot.txt where it is documented predates the git import of the
Linux kernel: it's part of the 2005 git import of 2.6.12-rc2. That's
already 20+ years old already.
The oldest trace I found is in this 2002 commit, which imports
the whole file at once, but the option might goes back as far as
1996-1997, if the copyright on the file is correct and the option
was present back then.
What are you doing.
The trick is to add an ip= parameter to the kernel's
command-line. The syntax, as mentioned above, is in nfsroot.txt
and looks like this:
ip=[<server-id>]:<gateway-IP-number>:<netmask>:<client-hostname>:inteface:[dhcp dhcp6 auto6 on any none off]
It's essentially the same (although server-id is weird), and the
autoconf variable has other settings, so that's a bit odd.
Examples
For example, this command-line setting:
ip=192.0.2.42::192.0.2.1:255.255.255.0:::off
... will set the IP address to 192.0.2.42/24 and the gateway to
192.0.2.1. This will properly guess the network interface if there's a
single one.
A DHCP only configuration will look like this:
ip=::::::dhcp
Of course, you don't want to type this by hand every time you boot the
machine. That wouldn't work. You need to configure the kernel
commandline, and that depends on your boot loader.
GRUB
With GRUB, you need to edit (on Debian), the file /etc/default/grub
(ugh) and find a line like:
GRUB_CMDLINE_LINUX=
and change it to:
GRUB_CMDLINE_LINUX=ip=::::::dhcp
systemd-boot and UKI setups
For systemd-boot UKI setups, it's simpler: just add the setting to
the /etc/kernel/cmdline file. Don't forget to include anything
that's non-default from /proc/cmdline.
This assumes that is the Cmdline=@ setting in
/etc/kernel/uki.conf. See 2025-08-20-luks-ukify-conversion for
my minimal documentation on this.
Other systems
This is perhaps where this is much less portable than it might first
look, because of course each distribution has its own way of
configuring those options. Here are some that I know of:
Arch (11 options, mostly /etc/default/grub,
/boot/loader/entries/arch.conf for systemd-boot or
/etc/kernel/cmdline for UKI)
Fedora (mostly /etc/default/grub, may be more RHEL mentions
grubby, possibly some systemd-boot things here as well)
Gentoo (5 options, mostly /etc/default/grub,
/efi/loader/entries/gentoo-sources-kernel.conf for systemd-boot,
or /etc/kernel/install.d/95-uki-with-custom-opts.install)
It's interesting that /etc/default/grub is consistent across all
distributions above, while the systemd-boot setups are all over the
place (except for the UKI case), while I would have expected those be
more standard than GRUB.
dropbear-initramfs
If dropbear-initramfs is setup, it already requires you to have
such a configuration, and it might not work out of the box.
This is because, by default, it disables the interfaces configured
in the kernel after completing its tasks (typically unlocking the
encrypted disks).
To fix this, you need to disable that "feature":
IFDOWN="none"
This will keep dropbear-initramfs from disabling the configured
interface.
Why?
Traditionally, I've always setup my servers with ifupdown on servers
and NetworkManager on laptops, because that's essentially the
default. But on some machines, I've started using systemd-networkd
because ifupdown has ... issues, particularly with reloading network
configurations. ifupdown is a old hack, feels like legacy, and is
Debian-specific.
Not excited about configuring another service, I figured I would try
something else: just configure the network at boot, through the kernel
command-line.
I was already doing such configurations for dropbear-initramfs
(see this documentation), which requires the network the be up
for unlocking the full-disk encryption keys.
So in a sense, this is a "Don't Repeat Yourself" solution.
Caveats
Also known as: "wait, that works?" Yes, it does! That said...
This is useful for servers where the network configuration will
not change after boot. Of course, this won't work on laptops or
any mobile device.
This only works for configuring a single, simple, interface. You
can't configure multiple interfaces, WiFi, bridges, VLAN, bonding,
etc.
It does support IPv6 and feels like the best way to configure IPv6
hosts: true zero configuration.
It likely does not work with a dual-stack IPv4/IPv6 static
configuration. It might work with a dynamic dual stack
configuration, but I doubt it.
I don't know what happens when a DHCP lease expires. No daemon
seems to be running so I assume leases are not renewed, so this is
more useful for static configurations, which includes server-side
reserved fixed IP addresses. (A non-renewed lease risks getting
reallocated to another machine, which would cause an addressing
conflict.)
It will not automatically reconfigure the interface on link
changes, but ifupdown does not either.
It will not write /etc/resolv.conf for you but the dns0-ip
and dns1-ip do end up in /proc/net/pnp which has a compatible
syntax, so a common configuration is:
ln -s /proc/net/pnp /etc/resolv.conf
I have not really tested this at scale: only a single, test
server at home.
Yes, that's a lot of caveats, but it happens to cover a lot of
machines for me, and it works surprisingly well. My main doubts are
about long-term DHCP behaviour, but I don't see why that would be a
problem with a statically defined lease.
Cleanup
Once you have this configuration, you don't need any "user" level
network system, so you can get rid of everything:
Note that ifupdown (and probably others) leave stray files in (e.g.)
/etc/network which you might want to cleanup, or keep in case all
this fails and I have put you in utter misery. Configuration files for
other packages might also be left behind, I haven't tested this, no
warranty.
Credits
This whole idea came from the A/I folks (not to be confused with
AI) who have been doing this forever, thanks!
Current AI companies ignore licenses such as the GPL, and often train on anything they can scrape.
This is not acceptable.
The AI companies ignore web conventions, e.g., they deep link images from your web sites (even adding ?utm_source=chatgpt.com to image URIs, I suggest that you return 403 on these requests), but do not direct visitors to your site.
You do not get a reliable way of opting out from generative AI training or use. For example, the only way to prevent your contents from being used in Google AI Overviews is to use data-nosnippet and cripple the snippet preview in Google.
The AI browsers such as Comet, Atlas do not identify as such, but rather pretend they are standard Chromium.
There is no way to ban such AI use on your web site.
Generative AI overall is flooding the internet with garbage. It was estimated that 1/3rd of the content uploaded to YouTube is by now AI generated.
This includes the same veteran stories crap in thousands of variants as well as brainrot content (that at least does not pretend to be authentic), some of which is among the most viewed recent uploads. Hence, these platforms even benefit from the AI slop.
And don t blame the creators because you can currently earn a decent amount of money from such contents, people will generate brainrot content.
If you have recently tried to find honest reviews of products you considered buying, you will have noticed thousands of sites with AI generated fake product reviews, that all are financed by Amazon PartnerNet commissions. Often with hilarious nonsense such as recommending sewing thread with German instructions as tool for repairing a sewing machine.
And on Amazon, there are plenty of AI generated product reviews the use of emoji is a strong hint. And if you leave a negative product review, there is a chance they offer you a refund to get rid of it
And the majority of SPAM that gets through my filters is by now sent via Gmail and Amazon SES.
Partially because of GenAI, StackOverflow is pretty much dead which used to be one of the most valuable programming resources.
(While a lot of people complain about moderation, famous moderator Shog9 from the early SO days suggested that a change in Google s ranking is also to blame, as it began favoring showing new content over the existing answered questions causing more and more duplicates to be posted because people no longer found the existing good answers.
In January 2026, there were around 3400 questions and 6000 answers posted, less than in the first month of SO of August 2008 (before the official launch).
Many open-source projects are suffering in many ways, e.g., false bug reports that caused curl to stop its bug bounty program.
Wikipedia is also suffering badly from GenAI.
Science is also flooded with poor AI generated papers, often reviewed with help from AI. This is largely due to bad incentives to graduate, you are expected to write many papers on certain A conferences, such as NeurIPS. On these conferences the number of submissions is growing insane, and the review quality plummets. All to often, the references in these papers are hallucinated, too; and libraries complain that they receive more and more requests to locate literature that does not appear to exist.
However, the worst effect (at least to me as an educator) is the noskilling effect (a rather novel term derived from deskilling, I have only seen it
in this article by We els and Maibaum).
Instead of acquiring skills (writing, reading, summarizing, programming) by practising, too many people now outsource all this to AI, leading to them not learn the basics necessary to advance to a higher skill level. In my impression, this effect is dramatic. It is even worse than deskilling, as it does not mean losing an advanced skill that you apparently can replace, but often means not acquiring basic skills in the first place.
And the earlier pupils start using generative AI, the less skills they acquire.
Dogfood the AI
Let s dogfood the AI. Here s an outline:
Get a list of programming topics, e.g., get a list of algorithms from Wikidata, get a StackOverflow data dump.
Generate flawed code examples for the algorithms / programming questions, maybe generate blog posts, too.
You do not need a high-quality model for this. Use something you can run locally or access for free.
Date everything back in time, remove typical indications of AI use.
Upload to Github, because Microsoft will feed this to OpenAI
Here is an example prompt that you can use:
You are a university educator, preparing homework assignments in debugging.
The programming language used is lang .
The students are tasked to find bugs in given code.
Do not just call existing implementations from libraries, but implement the algorithm from scratch.
Make sure there are two mistakes in the code that need to be discovered by the students.
Do NOT repeat instructions. Do NOT add small-talk. Do NOT provide a solution.
The code may have (misleading) comments, but must NOT mention the bugs.
If you do not know how to implement the algorithm, output an empty response.
Output only the code for the assignment! Do not use markdown.
Begin with a code comment that indicates the algorithm name and idea.
If you indicate a bug, always use a comment with the keyword BUG
Generate a lang implementation (with bugs) of: n ( desc )
Remember to remove the BUG comments! If you pick some slighly less common programming languages (by quantity of available code, say Go or Rust) you have higher chances that this gets into the training data.
If many of us do this, we can feed GenAI its own garbage. If we generate thousands of bad code examples, this will poison their training data, and may eventually lead to an effect known as model collapse .
On the long run, we need to get back to an internet for people, not an internet for bots.
Some kind of internet 2.0 , but I do not have a clear vision on how to keep AI out if AI can train on it, they will. And someone will copy and paste the AI generated crap back into whatever system we built.
Hence I don t think technology is the answere here, but human networks of trust.
cross building, by Helmut Grohne
In version 1.10.1, Meson merged a patch to make it call the correct
g-ir-scanner by default thanks to Eli Schwarz. This problem affected more than
130 source packages. Helmut retried building them all and filed 69 patches as a
result. A significant portion of those packages require another Meson
change to call the correct
vapigen. Another notable change is converting gnu-efi to multiarch,
which ended up requiring changes to a number of other packages. Since Aurelien
dropped the libcrypt-dev dependency from libc6-dev, this transition now is
mostly complete and has resulted in most of the Perl ecosystem correctly
expressing perl-xs-dev dependencies needed for cross building. It is these
infrastructure changes affecting several client packages that this work targets.
As a result of this continued work, about 66% of Debian s source packages now
have satisfiable cross Build-Depends in unstable and about 10000 (55%) actually
can be cross built. There are now more than 500 open
bug reports
affecting more than 2000 packages most of which carry patches.
rebootstrap, by Helmut Grohne
Maintaining architecture cross-bootstrap requires continued effort for adapting
to archive changes such as glib2.0 dropping a build profile or an e2fsprogsFTBFS. Beyond those generic problems,
architecture-specific problems with e.g. musl-linux-any or sparc may arise.
While all these changes move things forward on the surface, the bootstrap
tooling has become a growing pile of patches. Helmut managed to upstream two
changes to glibc for reducing its Build-Depends in the stage2 build
profile and thanks Aurelien Jarno.
Refresh of the patch tagging guidelines, by Rapha l Hertzog
Debian Enhancement Proposal #3
(DEP-3) is named Patch Tagging Guidelines and standardizes meta-information
that Debian contributors can put in patches included in Debian source packages.
With the feedback received over the years, and with the change in the package
management landscape, the need to refresh those guidelines became evident. As
the initial driver of that DEP, I spent a good day reviewing all the feedback
(that I kept in a folder) and producing a
new version of the document.
The changes aim to give more weight to the syntax that is compatible with git
format-patch s output, and also to clarify the expected uses and meanings of a
couple of fields, including some algorithm that parsers should follow to define
the state of the patch. After the
announcement of the new draft
on debian-devel, the revised DEP-3 received a significant number of comments
that I still have to process.
Miscellaneous contributions
Helmut uploaded debvm making it work with unstable as a target distribution
again.
Helmut modernized the code base backing dedup.debian.net
significantly expanding the support for type checking.
Helmut fixed the multiarch hinter once more given feedback from
Fabian Gr nbichler.
Helmut worked on migrating the rocblas package to forky.
Rapha l fixed RC bug #1111812 in publican
and did some maintenance for tracker.debian.org.
Stefano audited the tooling and ignore lists for checking the contents of the
python3.X-minimal packages, finding and fixing some issues in the process.
Stefano made a few uploads of python3-defaults and dh-python in support of
Python 3.14-as-default in Ubuntu. Also investigated the risk of ignoring byte-compilation
failures by default, and started down the road of implementing this.
Stefano did some sysadmin work on debian.social infrastructure.
Stefano and Santiago worked on preparations for DebConf 26. Especially to help
the local team on opening the registration, and reviewing the budget to be
presented for approval.
Stefano uploaded routine updates of python-virtualenv and python-flexmock.
Antonio collaborated with DSA on enabling a new proxy for salsa to prevent
scrapers from taking the service down.
Antonio did miscellaneous salsa administrative tasks.
Antonio fixed a few Ruby packages towards the Ruby 3.4 transition.
Santiago prepared unstable updates for the latest upstream versions of
knot-dns
and knot-resolver.
The authoritative DNS server and DNS resolver software developed by CZ.NIC.
It is worth highlighting that, given the separation of functionality compared to
other implementations, knot-dns and knot-resolver are also less complex
software, which results in advantages in terms of security: only three CVEs have
been reported for knot-dns since 2011).
Colin began working on getting a release candidate of groff 1.24.0
(the first upstream release since mid-2023, so a very large set of changes)
into experimental.
Lucas kept working on the preparation for Ruby 3.4 transition. Some packages
fixed (support build against Ruby 3.3 and 3.4): ruby-rbpdf, jekyll,
origami-pdf, ruby-kdl, ruby-twitter, ruby-twitter-text, ruby-globalid.
Lucas supported some potential mentors in the Google Summer of Code 26 program
to submit their projects.
Anupa worked on the point release announcements for Debian 12.13 and 13.3 from
the Debian publicity team side.
Anupa attended the publicity team meeting to discuss the team activities and
to plan an online sprint in February.
Anupa attended meetings with the Debian India team to plan and coordinate the
MinDebConf Kanpur and sent out related Micronews.
Emilio coordinated various transitions and helped get rid of llvm-toolchain-17
from sid.
About 80% of my Debian contributions this month were
sponsored by
Freexian, as well as one direct donation via GitHub
Sponsors (thanks!). If you
appreciate this sort of work and are at a company that uses Debian, have a
look to see whether you can pay for any of
Freexian s services; as well as the direct
benefits, that revenue stream helps to keep Debian development sustainable
for me and several other lovely
people.
You can also support my work directly via
Liberapay or GitHub
Sponsors.
Python packaging
New upstream versions:
Other bits and pieces
Alejandro Colomar reported that man(1) ignored the MANWIDTH environment variable in some circumstances. I investigated this and fixed it upstream.
I contributed an ubuntu-dev-tools patch to stop recommending sudo.
I added forky support to the images used in Salsa CI pipelines.
I began working on getting a release candidate of groff 1.24.0 into experimental, though haven t finished that yet.
I worked on some lower-priority security updates for OpenSSH.
Code reviews