Here s my 66th monthly but brief update about the activities I ve done in the F/L/OSS world.
Debian
This was my 75th month of actively contributing to Debian.
I became a DM in late March 2019 and a DD on Christmas 19! \o/
There s a bunch of things I do, both, technical and non-technical. Here s what I did:
Updating Rails to v7.2.2.1 for Trixie.
Updating Redmine to v6.0.4 for Trixie.
Kickstarting the bursary team for DC25.
Mentoring for newcomers.
Moderation of -project mailing list.
Ubuntu
This was my 50th month of actively contributing to Ubuntu.
Now that I joined Canonical to work on Ubuntu full-time, there s a bunch of things I do! \o/
I mostly worked on different things, I guess.
I was too lazy to maintain a list of things I worked on so there s
no concrete list atm. Maybe I ll get back to this section later or
will start to list stuff from the fall, as I was doing before. :D
Debian (E)LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.
And Debian Extended LTS (ELTS) is its sister project, extending support to the stretch and jessie release (+2 years after LTS support).
This was my 66th month as a Debian LTS and 53rd month as a Debian ELTS paid contributor.
I worked for 15.00 hours for LTS and 7.50 hours for ELTS.
I did the following things:
Cascade Failure is a far-future science fiction adventure with a
small helping of cyberpunk vibes. It is the first of a (so far) two-book
series, and was the author's first novel.
The Ambit is an old and small Guild ship, not much to look at, but
it holds a couple of surprises. One is its captain, Eoan, who is an AI
with a deep and insatiable curiosity that has driven them and their ship
farther and farther out into the Spiral. The other is its surprisingly
competent crew: a battle-scarred veteran named Saint who handles the
fighting, and a talented engineer named Nash who does literally everything
else. The novel opens with them taking on supplies at Aron Outpost. A
supposed Guild deserter named Jalsen wanders into the ship looking for
work.
An AI ship with a found-family crew is normally my catnip, so I wanted to
love this book. Alas, I did not.
There were parts I liked. Nash is great: snarky, competent, and direct.
Eoan is a bit distant and slightly more simplistic of a character than I
was expecting, but I appreciated the way Sagas put them firmly in charge
of the ship and departed from the conventional AI character presentation.
Once the plot starts in earnest (more on that in a moment), we meet Anke,
the computer hacker, whose charming anxiety reaction is a complete
inability to stop talking and who adds some needed depth to the character
interactions. There's plenty of action, a plot that makes at least some
sense, and a few moments that almost achieved the emotional payoff the
author was attempting.
Unfortunately, most of the story focuses on Saint and Jal, and both of
them are irritatingly dense cliches.
The moment Jal wanders onto the Ambit in the first chapter, the
reader is informed that Jal, Saint, and Eoan have a history. The crew of
the Ambit spent a year looking for Jal and aren't letting go of him now
that they've found him. Jal, on the other hand, clearly blames Saint for
something and is not inclined to trust him. Okay, fine, a bit generic of a
setup but the writing moved right along and I was curious enough.
It then takes a full 180 pages before the reader finds out what the hell
is going on with Saint and Jal. Predictably, it's a stupid
misunderstanding that could have been cleared up with one conversation in
the second chapter.
Cascade Failure does not contain a romance (and to the extent that
it hints at one, it's a sapphic romance), but I swear Saint and Jal are
both the male protagonist from a certain type of stereotypical
heterosexual romance novel. They're both the brooding man with the past,
who is too hurt to trust anyone and assumes the worst because he's unable
to use his words or ask an open question and then listen to the answer.
The first half of this book is them being sullen at each other at great
length while both of them feel miserable. Jal keeps doing weird and
suspicious things to resolve a problem that would have been far more
easily resolved by the rest of the crew if he would offer any explanation
at all. It's not even suspenseful; we've read about this character enough
times to know that he'll turn out to have a heart of gold and everything
will be a misunderstanding. I found it tedious. Maybe people who like slow
burn romances with this character type will have a less negative reaction.
The real plot starts at about the time Saint and Jal finally get their
shit sorted out. It turns out to have almost nothing to do with either of
them. The environmental control systems of worlds are suddenly failing
(hence the book title), and Anke, the late-arriving computer programmer
and terraforming specialist, has a rather wild theory about what's
happening. This leads to a lot of action, some decent twists, and a plot
that felt very cyberpunk to me, although unfortunately it culminates in an
absurdly-cliched action climax.
This book is an action movie that desperately wants to make you feel all
the feels, and it worked about as well as that typically works in action
movies for me. Jaded cynicism and an inability to communicate are not the
ways to get me to have an emotional reaction to a book, and Jal (once he
finally starts talking) is so ridiculously earnest that it's like reading
the adventures of a Labrador puppy. There was enough going on that it kept
me reading, but not enough for the story to feel satisfying. I needed a
twist, some depth, way more Nash and Anke and way less of the men,
something.
Everyone is going to compare this book to Firefly, but
Firefly had better banter, created more complex character
interactions due to the larger and more varied crew, and played the
cynical mercenary for laughs instead of straight, all of which suited me
better. This is not a bad book, particularly once it gets past the halfway
point, but it's not that memorable either, at least for me. If you're
looking for a space adventure with heavy action hero and military SF vibes
that wants to be about Big Feelings but gets there in mostly obvious ways,
you could do worse. If you're looking for a found-family starship crew
story more like Becky Chambers, I think you'll find this one a bit too
shallow and obvious.
Not really recommended, although there's nothing that wrong with it and
I'm sure other people's experience will differ.
Followed by Gravity Lost, which I'm unlikely to read.
Rating: 6 out of 10
Rust, and resistance to it in some parts of the Linux community, has been in my feed recently. One undercurrent seems to be the notion that Rust is woke (and should therefore be rejected as part of culture wars).
I m going to argue that Rust, the language, is woke. So the opponents are right, in that sense. Of course, as ever, dissing something for being woke is nasty and fascist-adjacent.
Community
The obvious way that Rust may seem woke is that it has the trappings, and many of the attitudes and outcomes, of a modern, nice, FLOSS community. Rust certainly does better than toxic environments like the Linux kernel, or Debian. This is reflected in a higher proportion of contributors from various kinds of minoritised groups. But Rust is not outstanding in this respect. It certainly has its problems. Many other projects do as well or better.
And this is well-trodden ground. I have something more interesting to say:
Technological values - particularly, compared to C/C++
Rust is woke technology that embodies a woke understanding of what it means to be a programming language.
Ostensible values
Let s start with Rust s strapline:
A language empowering everyone to build reliable and efficient software.
Surprisingly, this motto is not mere marketing puff. For Rustaceans, it is a key goal which strongly influences day-to-day decisions (big and small).
Empowering everyone is a key aspect of this, which aligns with my own personal values. In the Rust community, we care about empowerment. We are trying to help liberate our users. And we want to empower everyone because everyone is entitled to technological autonomy. (For a programming language, empowering individuals means empowering their communities, of course.)
This is all very airy-fairy, but it has concrete consequences:
Attitude to the programmer s mistakes
In Rust we consider it a key part of our job to help the programmer avoid mistakes; to limit the consequences of mistakes; and to guide programmers in useful directions.
If you write a bug in your Rust program, Rust doesn t blame you. Rust asks how could the compiler have spotted that bug .
This is in sharp contrast to C (and C++). C nowadays is an insanely hostile programming environment. A C compiler relentlessly scours your program for any place where you may have violated C s almost incomprehensible rules, so that it can compile your apparently-correct program into a buggy executable. And then the bug is considered your fault.
These aren t just attitudes implicitly embodied in the software. They are concrete opinions expressed by compiler authors, and also by language proponents. In other words:
Rust sees programmers writing bugs as a systemic problem, which must be addressed by improvements to the environment and the system. The toxic parts of the C and C++ community see bugs as moral failings by individual programmers.
Sound familiar?
The ideology of the hardcore programmer
Programming has long suffered from the myth of the rockstar . Silicon Valley techbro culture loves this notion.
In reality, though, modern information systems are far too complicated for a single person. Developing systems is a team sport. Nontechnical, and technical-adjacent, skills are vital: clear but friendly communication; obtaining and incorporating the insights of every member of your team; willingness to be challenged. Community building. Collaboration. Governance.
The hardcore C community embraces the rockstar myth: they imagine that a few super-programmers (or super-reviewers) are able to spot bugs, just by being so brilliant. Of course this doesn t actually work at all, as we can see from the atrocious bugfest that is the Linux kernel.
These rockstars want us to believe that there is a steep hierarchy in programmming; that they are at the top of this hierarchy; and that being nice isn t important.
Sound familiar?
Memory safety as a power struggle
Much of the modern crisis of software reliability arises from memory-unsafe programming languages, mostly C and C++.
Addressing this is a big job, requiring many changes. This threatens powerful interests; notably, corporations who want to keep shipping junk. (See also, conniptions over the EU Product Liability Directive.)
The harms of this serious problem mostly fall on society at large, but the convenience of carrying on as before benefits existing powerful interests.
Sound familiar?
Memory safety via Rust as a power struggle
Addressing this problem via Rust is a direct threat to the power of established C programmers such as gatekeepers in the Linux kernel. Supplanting C means they will have to learn new things, and jostle for status against better Rustaceans, or be replaced. More broadly, Rust shows that it is practical to write fast, reliable, software, and that this does not need (mythical) rockstars .
So established C programmer experts are existing vested interests, whose power is undermined by (this approach to) tackling this serious problem.
Sound familiar?
NotesThis is not a RIIR manifesto
I m not saying we should rewrite all the world s C in Rust. We should not try to do that.
Rust is often a good choice for new code, or when a rewrite or substantial overhaul is needed anyway. But we re going to need other techniques to deal with all of our existing C. CHERI is a very promising approach. Sandboxing, emulation and automatic translation are other possibilities. The problem is a big one and we need a toolkit, not a magic bullet.
But as for Linux: it is a scandal that substantial new drivers and subsystems are still being written in C. We could have been using Rust for new code throughout Linux years ago, and avoided very many bugs. Those bugs are doing real harm. This is not OK.
Disclosure
I first learned C from K&R I in 1989. I spent the first three decades of my life as a working programmer writing lots and lots of C. I ve written C++ too. I used to consider myself an expert C programmer, but nowadays my C is a bit rusty and out of date. Why is my C rusty? Because I found Rust, and immediately liked and adopted it (despite its many faults).
I like Rust because I care that the software I write actually works: I care that my code doesn t do harm in the world.
On the meaning of woke
The original meaning of woke is something much more specific, to do with racism. For the avoidance of doubt, I don t think Rust is particularly antiracist.
I m using woke (like Rust s opponents are) in the much broader, and now much more prevalent, culture wars sense.
Pithy conclusion
If you re a senior developer who knows only C/C++, doesn t want their authority challenged, and doesn t want to have to learn how to write better software, you should hate Rust.
Also you should be fired.
Edited 2025-03-28 17:10 UTC to fix minor problems and add a new note about the meaning of the word "woke".
As I write this in March 2025, there is a lot of confusion about Signal messenger due to the recent news of people using Signal in government, and subsequent leaks.
The short version is: there was no problem with Signal here. People were using it because they understood it to be secure, not the other way around.
Both the government and the Electronic Frontier Foundation recommend people use Signal. This is an unusual alliance, and in the case of the government, was prompted because it understood other countries had a persistent attack against American telephone companies and SMS traffic.
So let s dive in. I ll cover some basics of what security is, what happened in this situation, and why Signal is a good idea.
This post isn t for programmers that work with cryptography every day. Rather, I hope it can make some of these concepts accessible to everyone else.
What makes communications secure?
When most people are talking about secure communications, they mean some combination of these properties:
Privacy - nobody except the intended recipient can decode a message.
Authentication - guarantees that the person you are chatting with really is the intended recipient.
Ephemerality - preventing a record of the communication from being stored. That is, making it more like a conversation around the table than a written email.
Anonymity - keeping your set of contacts to yourself and even obfuscating the fact that communications are occurring.
If you think about it, most people care the most about the first two. In fact, authentication is a key part of privacy. There is an attack known as man in the middle in which somebody pretends to be the intended recipient. The interceptor reads the messages, and then passes them on to the real intended recipient. So we can t really have privacy without authentication.
I ll have more to say about these later. For now, let s discuss attack scenarios.
What compromises security?
There are a number of ways that security can be compromised. Let s think through some of them:
Communications infrastructure snooping
Let s say you used no encryption at all, and connected to public WiFi in a coffee shop to send your message. Who all could potentially see it?
The owner of the coffee shop s WiFi
The coffee shop s Internet provider
The recipient s Internet provider
Any Internet providers along the network between the sender and the recipient
Any government or institution that can compel any of the above to hand over copies of the traffic
Any hackers that compromise any of the above systems
Back in the early days of the Internet, most traffic had no encryption. People were careful about putting their credit cards into webpages and emails because they knew it was easy to intercept them. We have been on a decades-long evolution towards more pervasive encryption, which is a good thing.
Text messages (SMS) follow a similar path to the above scenario, and are unencrypted. We know that all of the above are ways people s texts can be compromised; for instance, governments can issue search warrants to obtain copies of texts, and China is believed to have a persistent hack into western telcos. SMS fails all four of our attributes of secure communication above (privacy, authentication, ephemerality, and anonymity).
Also, think about what information is collected from SMS and by who. Texts you send could be retained in your phone, the recipient s phone, your phone company, their phone company, and so forth. They might also live in cloud backups of your devices. You only have control over your own phone s retention.
So defenses against this involve things like:
Strong end-to-end encryption, so no intermediate party even the people that make the app can snoop on it.
Using strong authentication of your peers
Taking steps to prevent even app developers from being able to see your contact list or communication history
You may see some other apps saying they use strong encryption or use the Signal protocol. But while they may do that for some or all of your message content, they may still upload your contact list, history, location, etc. to a central location where it is still vulnerable to these kinds of attacks.
When you think about anonymity, think about it like this: if you send a letter to a friend every week, every postal carrier that transports it even if they never open it or attempt to peak inside will be able to read the envelope and know that you communicate on a certain schedule with that friend. The same can be said of SMS, email, or most encrypted chat operators. Signal s design prevents it from retaining even this information, though nation-states or ISPs might still be able to notice patterns (every time you send something via Signal, your contact receives something from Signal a few milliseconds later). It is very difficult to provide perfect anonymity from well-funded adversaries, even if you can provide very good privacy.
Device compromise
Let s say you use an app with strong end-to-end encryption. This takes away some of the easiest ways someone could get to your messages. But it doesn t take away all of them.
What if somebody stole your phone? Perhaps the phone has a password, but if an attacker pulled out the storage unit, could they access your messages without a password? Or maybe they somehow trick or compel you into revealing your password. Now what?
An even simpler attack doesn t require them to steal your device at all. All they need is a few minutes with it to steal your SIM card. Now they can receive any texts sent to your number - whether from your bank or your friend. Yikes, right?
Signal stores your data in an encrypted form on your device. It can protect it in various ways. One of the most important protections is ephemerality - it can automatically delete your old texts. A text that is securely erased can never fall into the wrong hands if the device is compromised later.
An actively-compromised phone, though, could still give up secrets. For instance, what if a malicious keyboard app sent every keypress to an adversary? Signal is only as secure as the phone it runs on but still, it protects against a wide variety of attacks.
Untrustworthy communication partner
Perhaps you are sending sensitive information to a contact, but that person doesn t want to keep it in confidence. There is very little you can do about that technologically; with pretty much any tool out there, nothing stops them from taking a picture of your messages and handing the picture off.
Environmental compromise
Perhaps your device is secure, but a hidden camera still captures what s on your screen. You can take some steps against things like this, of course.
Human error
Sometimes humans make mistakes. For instance, the reason a reporter got copies of messages recently was because a participant in a group chat accidentally added him (presumably that participant meant to add someone else and just selected the wrong name). Phishing attacks can trick people into revealing passwords or other sensitive data. Humans are, quite often, the weakest link in the chain.
Protecting yourself
So how can you protect yourself against these attacks? Let s consider:
Use a secure app like Signal that uses strong end-to-end encryption where even the provider can t access your messages
Keep your software and phone up-to-date
Be careful about phishing attacks and who you add to chat rooms
Be aware of your surroundings; don t send sensitive messages where people might be looking over your shoulder with their eyes or cameras
There are other methods besides Signal. For instance, you could install GnuPG (GPG) on a laptop that has no WiFi card or any other way to connect it to the Internet. You could always type your messages on that laptop, encrypt them, copy the encrypted text to a floppy disk (or USB device), take that USB drive to your Internet computer, and send the encrypted message by email or something. It would be exceptionally difficult to break the privacy of messages in that case (though anonymity would be mostly lost). Even if someone got the password to your secure laptop, it wouldn t do them any good unless they physically broke into your house or something. In some ways, it is probably safer than Signal. (For more on this, see my article How gapped is your air?)
But, that approach is hard to use. Many people aren t familiar with GnuPG. You don t have the convenience of sending a quick text message from anywhere. Security that is hard to use most often simply isn t used. That is, you and your friends will probably just revert back to using insecure SMS instead of this GnuPG approach because SMS is so much easier.
Signal strikes a unique balance of providing very good security while also being practical, easy, and useful. For most people, it is the most secure option available.
Signal is also open source; you don t have to trust that it is as secure as it says, because you can inspect it for yourself. Also, while it s not federated, I previously addressed that.
Government use
If you are a government, particularly one that is highly consequential to the world, you can imagine that you are a huge target. Other nations are likely spending billions of dollars to compromise your communications. Signal itself might be secure, but if some other government can add spyware to your phones, or conduct a successful phishing attack, you can still have your communications compromised.
I have no direct knowledge, but I think it is generally understood that the US government maintains communications networks that are entirely separate from the Internet and can only be accessed from secure physical locations and secure rooms. These can be even more secure than the average person using Signal because they can protect against things like environmental compromise, human error, and so forth. The scandal in March of 2025 happened because government employees were using Signal rather than official government tools for sensitive information, had taken advantage of Signal s ephemerality (laws require records to be kept), and through apparent human error had directly shared this information with a reporter. Presumably a reporter would have lacked access to the restricted communications networks in the first place, so that wouldn t have been possible.
This doesn t mean that Signal is bad. It just means that somebody that can spend billions of dollars on security can be more secure than you. Signal is still a great tool for people, and in many cases defeats even those that can spend lots of dollars trying to defeat it.
And remember - to use those restricted networks, you have to go to specific rooms in specific buildings. They are still not as convenient as what you carry around in your pocket.
Conclusion
Signal is practical security. Do you want phone companies reading your messages? How about Facebook or X? Have those companies demonstrated that they are completely trustworthy throughout their entire history?
I say no. So, go install Signal. It s the best, most practical tool we have.
This post is also available on my website, where it may be periodically updated.
Kubuntu work:
Fixed an issue in apparmor preventing QT6 webengine applications from starting.
Beta testing!
KDE Snaps:
Updated Qt6 to 6.8.2
Updated Kf6 6.11.0
Rolling out 25.04 RC applications! You can find them in the candidate channel!
Life:
I have decided to strike out on my own. I can t take any more rejections! Honestly, I don t blame them, I wouldn t want a one armed engineer either. However, I have persevered and accomplished quite a bit with my one arm! So I have decided to take a leap of faith and with your support for open source work and a resurrected side gig of web development I will survive. If you can help sponsor my work, anything at all, even a dollar! I would be eternally grateful. I have several methods to do so:
If you want your cool application packaged in a variety of formats please contact me!
If you want focused help with an annoying bug, please contact me!
Contact me for any and all kinds of help, if I can t do it, I will say so.
Do you need web work? Someone to maintain your website? I can do that too!
Portfolio
Thank you all for your support in this new adventure!
We are pleased to announce that Viridien
has committed to sponsor DebConf25 as a
Platinum Sponsor.
Viridien is an advanced technology, digital
and Earth data company that pushes the boundaries of science for a more
prosperous and sustainable future.
Viridien has been using Debian-based systems to power most of its HPC
infrastructure and its cloud platform since 2009 and currently employs two
active Debian Project Members.
As a Platinum Sponsor, Viridien is contributing to the Debian annual
Developers' conference, directly supporting the progress of Debian and Free
Software. Viridien contributes to strengthen the community that collaborates
on the Debian project from all around the world throughout all of the year.
Thank you very much, Viridien, for your support of DebConf25!
Become a sponsor too!
DebConf25 will take place from 14 to 20
July 2025 in Brest, France, and will be preceded by DebCamp, from 7 to 13
July 2025.
DebConf25 is accepting sponsors! Interested companies and organizations may
contact the DebConf team through
sponsors@debconf.org, and visit the DebConf25
website at
https://debconf25.debconf.org/sponsors /become-a-sponsor/.
Are you a student aspiring to participate in the Google Summer of Code 2025? Would you like to improve the continuous integration pipeline used at salsa.debian.org, the Debian GitLab instance, to help improve the quality of tens of thousands of software packages in Debian?
This summer 2025, I and Emmanuel Arias will be participating as mentors in the GSoC program. We are available to mentor students who propose and develop improvements to the Salsa CI pipeline, as we are members of the Debian team that maintains it.
A post by Santiago Ruano Rinc n in the GitLab blog explains what Salsa CI is and its short history since inception in 2018. At the time of the article in fall 2023 there were 9000+ source packages in Debian using Salsa CI. Now in 2025 there are over 27,000 source packages in Debian using it, and since summer 2024 some Ubuntu developers have started using it for enhanced quality assurance of packaging changes before uploading new package revisions to Ubuntu. Personally, I have been using Salsa CI since its inception, and contributing as a team member since 2019. See my blog post about GitLab CI for MariaDB in Debian for a description of an advanced and extensive use case.
Helping Salsa CI is a great way to make a global impact, as it will help avoid regressions and improve the quality of Debian packages. The benefits reach far beyond just Debian, as it will also help hundreds of Debian derivatives, such as Ubuntu, Linux Mint, Tails, Purism PureOS, Pop!_OS, Zorin OS, Raspberry Pi OS, a large portion of Docker containers, and even the Windows Subsystem for Linux.
Improving Salsa CI: more features, robustness, speed
While Salsa CI with contributions from 71 people is already quite mature and capable, there are many ideas floating around about how it could be further extended. For example, Salsa CI issue #147 describes various static analyzers and linters that may be generally useful. Issue #411 proposes using libfaketime to run autopkgtest on arbitrary future dates to test for failures caused by date assumptions, such as the Y2038 issue.
There are also ideas about making Salsa CI more robust and code easier to reuse by refactoring some of the yaml scripts into independent scripts in #230, which could make it easier to run Salsa CI locally as suggested in #169. There are also ideas about improving the Salsa CI s own CI to avoid regressions from pipeline changes in #318.
The CI system is also better when it s faster, and some speed improvement ideas have been noted in #412.
Improvements don t have to be limited to changes in the pipeline itself. A useful project would also be to update more Debian packages to use Salsa CI, and ensure they adopt it in an optimal way as noted in #416. It would also be nice to have a dashboard with statistics about all public Salsa CI pipeline runs as suggested in #413.
These and more ideas can be found in the issue list by filtering for tags Newcomer, Nice-To-Have or Accepting MRs. A Google Summer of Code proposal does not have to be limited to these existing ideas. Participants are also welcome to propose completely novel ideas!
Good time to also learn Debian packaging
Anyone working with Debian team should also take the opportunity to learn Debian packaging, and contribute to the packaging or maintenance of 1-2 packages in parallel to improving the Salsa CI. All Salsa CI team members are also Debian Developers who can mentor and sponsor uploads to Debian.
Maintaining a few packages is a great way to eat your own cooking and experience Salsa CI from the user perspective, and likely to make you better at Salsa CI development.
Apply now!
The contributor applications opened yesterday on March 24, so to participate act now! If you are an eligible student and want to attend, head over to summerofcode.withgoogle.com to learn more.
There are over a thousand participating organizations, with Debian, GitLab and MariaDB being some examples. Within these organizations there may be multiple subteams and projects to choose from. The full list of participating Debian projects can be found in the Debian wiki.
If you are interested in GSoC for Salsa CI specifically, feel free to
Reach out to me and Emmanuel by email at otto@ and eamanu@ (debian.org).
Sign up at salsa.debian.org for an account (note it takes a few days due to manual vetting and approval process)
Note that you don t have to wait for GSoC to officially start to contribute. In fact, it may be useful to start immediately by submitting a Merge Request to do some small contribution, just to learn the process and to get more familiar with how everything works, and the team maintaining Salsa CI. Looking forward to seeing new contributors!
I am told, by friends who have spent time at Google, about the reason Google Reader finally disappeared. Apparently it had become a 20% Project for those who still cared about it internally, and there was some major change happening to one of it upstream dependencies that was either going to cause a significant amount of work rearchitecting Reader to cope, or create additional ongoing maintenance burden. It was no longer viable to support it as a side project, so it had to go. This was a consequence of an internal culture at Google where service owners are able to make changes that can break downstream users, and the downstream users are the ones who have to adapt.
My experience at Meta goes the other way. If you own a service or other dependency and you want to make a change that will break things for the users, it s on you to do the migration, or at the very least provide significant assistance to those who own the code. You don t just get to drop your new release and expect others to clean up; doing that tends to lead to changes being reverted. The culture flows the other way; if you break it, you fix it (nothing is someone else s problem).
There are pluses and minuses to both approaches. Users having to drive the changes to things they own stops them from blocking progress. Service/code owners having to drive the changes avoids the situation where a wildly used component drops a new release that causes a lot of high priority work for folk in order to adapt.
I started thinking about this in the context of Debian a while back, and a few incidents since have resulted in my feeling that we re closer to the Google model than the Meta model. Anyone can upload a new version of their package to unstable, and that might end up breaking all the users of it. It s not quite as extreme as rolling out a new service, because it s unstable that gets affected (the clue is in the name, I really wish more people would realise that), but it can still result in release critical bugs for lots other Debian contributors.
A good example of this are toolchain changes. Major updates to GCC and friends regularly result in FTBFS issues in lots of packages. Now in this instance the maintainer is usually diligent about a heads up before the default changes, but it s still a whole bunch of work for other maintainers to adapt (see the list of FTBFS bugs for GCC 15 for instance - these are important, but not serious yet). Worse is when a dependency changes and hasn t managed to catch everyone who might be affected, so by the time it s discovered it s release critical, because at least one package no longer builds in unstable.
Commercial organisations try to avoid this with a decent CI/CD setup that either vendors all dependencies, or tracks changes to them and tries rebuilds before allowing things to land. This is one of the instances where a monorepo can really shine; if everything you need is in there, it s easier to track the interconnections between different components. Debian doesn t have a CI/CD system that runs for every upload, allowing us to track exact causes of regressions. Instead we have Lucas, who does a tremendous job of running archive wide rebuilds to make sure we can still build everything. Unfortunately that means I am often unfairly grumpy at him; my heart sinks when I see a bug come in with his name attached, because it often means one of my packages has a new RC bug where I m going to have to figure out what changed elsewhere to cause it. However he s just (very usefully) surfacing an issue someone else created, rather than actually being the cause of the problem.
I don t know if I have a point to this post. I think it s probably that I wish folk in Free Software would try and be mindful of the incompatible changes they might introducing, and the toil they create for other volunteer developers, often not directly visible to the person making the change. The approach done by the Debian toolchain maintainers strikes me as a good balance; they do a bunch of work up front to try and flag all the places that might need to make changes, far enough in advance of the breaking change actually landing. However they don t then allow a tardy developer to block progress.
Around a year ago I discussed two concerns with software release archives (tarball artifacts) that could be improved to increase confidence in the supply-chain security of software releases. Repeating the goals for simplicity:
Release artifacts should be built in a way that can be reproduced by others
It should be possible to build a project from source tarball that doesn t contain any generated or vendor files (e.g., in the style of git-archive).
While implementing these ideas for a small project was accomplished within weeks see my announcement of Libntlm version 1.8 adressing this in complex projects uncovered concerns with tools that had to be addressed, and things stalled for many months pending that work.
I had the notion that these two goals were easy and shouldn t be hard to accomplish. I still believe that, but have had to realize that improving tooling to support these goals takes time. It seems clear that these concepts are not universally agreed on and implemented generally.
I m now happy to recap some of the work that led to releases of libtasn1 v4.20.0, inetutils v2.6, libidn2 v2.3.8, libidn v1.43. These releases all achieve these goals. I am working on a bunch of more projects to support these ideas too.
What have the obstacles so far been to make this happen? It may help others who are in the same process of addressing these concerns to have a high-level introduction to the issues I encountered. Source code for projects above are available and anyone can look at the solutions to learn how the problems are addressed.
First let s look at the problems we need to solve to make git-archive style tarballs usable:
Version Handling
To build usable binaries from a minimal tarballs, it need to know which version number it is. Traditionally this information was stored inside configure.ac in git. However I use gnulib s git-version-gen to infer the version number from the git tag or git commit instead. The git tag information is not available in a git-archive tarball. My solution to this was to make use of the export-subst feature of the .gitattributes file. I store the file .tarball-version-git in git containing the magic cookie like this:
$Format:%(describe)$
With this, git-archive will replace with a useful version identifier on export, see the libtasn1 patch to achieve this. To make use of this information, the git-version-gen script was enhanced to read this information, see the gnulib patch. This is invoked by ./configure to figure out which version number the package is for.
Translations
We want translations to be included in the minimal source tarball for it to be buildable. Traditionally these files are retrieved by the maintainer from the Translation project when running ./bootstrap, however there are two problems with this. The first one is that there is no strong authentication or versioning information on this data, the tools just download and place whatever wget downloaded into your source tree (printf-style injection attack anyone?). We could improve this (e.g., publish GnuPG signed translations messages with clear versioning), however I did not work on that further. The reason is that I want to support offline builds of packages. Downloading random things from the Internet during builds does not work when building a Debian package, for example. The translation project could solve this by making a monthly tarball with their translations available, for distributors to pick up and provide as a separate package that could be used as a build dependency. However that is not how these tools and projects are designed. Instead I reverted back to storing translations in git, something that I did for most projects back when I was using CVS 20 years ago. Hooking this into ./bootstrap and gettext workflow can be tricky (ideas for improvement most welcome!), but I used a simple approach to store all directly downloaded po/*.po files directly as po/*.po.in and make the ./bootstrap tool move them in place, see the libidn2 commit followed by the actual make update-po commit with all the translations where one essential step is:
# Prime po/*.po from fall-back copy stored in git.
for poin in po/*.po.in; do
po=$(echo $poin sed 's/.in//')
test -f $po cp -v $poin $po
done
ls po/*.po sed 's .*/ ; s \.po$ ' > po/LINGUAS
Fetching vendor files like gnulib
Most build dependencies are in the shape of You need a C compiler . However some come in the shape of source-code files intended to be vendored , and gnulib is a huge repository of such files. The latter is a problem when building from a minimal git archive. It is possible to consider translation files as a class of vendor files, since they need to be copied verbatim into the project build directory for things to work. The same goes for *.m4 macros from the GNU Autoconf Archive. However I m not confident that the solution for all vendor files must be the same. For translation files and for Autoconf Archive macros, I have decided to put these files into git and merge them manually occasionally. For gnulib files, in some projects like OATH Toolkit I also store all gnulib files in git which effectively resolve this concern. (Incidentally, the reason for doing so was originally that running ./bootstrap took forever since there is five gnulib instances used, which is no longer the case since gnulib-tool was rewritten in Python.) For most projects, however, I rely on ./bootstrap to fetch a gnulib git clone when building. I like this model, however it doesn t work offline. One way to resolve this is to make the gnulib git repository available for offline use, and I ve made some effort to make this happen via a Gnulib Git Bundle and have explained how to implement this approach for Debian packaging. I don t think that is sufficient as a generic solution though, it is mostly applicable to building old releases that uses old gnulib files. It won t work when building from CI/CD pipelines, for example, where I have settled to use a crude way of fetching and unpacking a particular gnulib snapshot, see this Libntlm patch. This is much faster than working with git submodules and cloning gnulib during ./bootstrap. Essentially this is doing:
Test the git-archive tarball
This goes without saying, but if you don t test that building from a git-archive style tarball works, you are likely to regress at some point. Use CI/CD techniques to continuously test that a minimal git-archive tarball leads to a usable build.
Mission Accomplished
So that wasn t hard, was it? You should now be able to publish a minimal git-archive tarball and users should be able to build your project from it.
I recommend naming these archives as PROJECT-vX.Y.Z-src.tar.gz replacing PROJECT with your project name and X.Y.Z with your version number. The archive should have only one sub-directory named PROJECT-vX.Y.Z/ containing all the source-code files. This differentiate it against traditional PROJECT-X.Y.Z.tar.gz tarballs in that it embeds the git tag (which typically starts with v) and contains a wildcard-friendly -src substring. Alas there is no consistency around this naming pattern, and GitLab, GitHub, Codeberg etc all seem to use their own slightly incompatible variant.
Let s go on to see what is needed to achieve reproducible make dist source tarballs. This is the release artifact that most users use, and they often contain lots of generated files and vendor files. These files are included to make it easy to build for the user. What are the challenges to make these reproducible?
Build dependencies causing different generated content
The first part is to realize that if you use tool X with version A to generate a file that goes into the tarball, version B of that tool may produce different outputs. This is a generic concern and it cannot be solved. We want our build tools to evolve and produce better outputs over time. What can be addressed is to avoid needless differences. For example, many tools store timestamps and versioning information in the generated files. This causes needless differences, which makes audits harder. I have worked on some of these, like Autoconf Archive timestamps but solving all of these examples will take a long time, and some upstream are reluctant to incorporate these changes. My approach meanwhile is to build things using similar environments, and compare the outputs for differences. I ve found that the various closely related forks of GNU/Linux distributions are useful for this. Trisquel 11 is based on Ubuntu 22.04, and building my projects using both and comparing the differences only give me the relevant differences to improve. This can be extended to compare AlmaLinux with RockyLinux (for both versions 8 and 9), Devuan 5 against Debian 12, PureOS 10 with Debian 11, and so on.
Timestamps
Sometimes tools store timestamps in files in a way that is harder to fix. Two notable examples of this are *.po translation files and Texinfo manuals. For translation files, I have resolved this by making sure the files use a predictable POT-Creation-Date timestamp, and I set it to the modification timestamps of the NEWS file in the repository (which I set to the git commit of the latest commit elsewhere) like this:
dist-hook: po-CreationDate-to-mtime-NEWS
.PHONY: po-CreationDate-to-mtime-NEWS
po-CreationDate-to-mtime-NEWS: mtime-NEWS-to-git-HEAD
$(AM_V_GEN)for p in $(distdir)/po/*.po $(distdir)/po/$(PACKAGE).pot; do \
if test -f "$$p"; then \
$(SED) -e 's,POT-Creation-Date: .*\\n",POT-Creation-Date: '"$$(env LC_ALL=C TZ=UTC0 stat --format=%y $(srcdir)/NEWS cut -c1-16,31-)"'\\n",' < $$p > $$p.tmp && \
if cmp $$p $$p.tmp > /dev/null; then \
rm -f $$p.tmp; \
else \
mv $$p.tmp $$p; \
fi \
fi \
done
Similarily, I set a predictable modification time of the texinfo source file like this:
dist-hook: mtime-NEWS-to-git-HEAD
.PHONY: mtime-NEWS-to-git-HEAD
mtime-NEWS-to-git-HEAD:
$(AM_V_GEN)if test -e $(srcdir)/.git \
&& command -v git > /dev/null; then \
touch -m -t "$$(git log -1 --format=%cd \
--date=format-local:%Y%m%d%H%M.%S)" $(srcdir)/NEWS; \
fi
However I ve realized that this needs to happen earlier and probably has to be run during ./configure time, because the doc/version.texi file is generated on first build before running make dist and for some reason the file is not rebuilt at release time. The Automake texinfo integration is a bit inflexible about providing hooks to extend the dependency tracking.
The method to address these differences isn t really important, and they change over time depending on preferences. What is important is that the differences are eliminated.
ChangeLog
Traditionally ChangeLog files were manually prepared, and still is for some projects. I maintain git2cl but recently I ve settled with gnulib s gitlog-to-changelog because doing so avoids another build dependency (although the output formatting is different and arguable worse for my git commit style). So the ChangeLog files are generated from git history. This means a shallow clone will not produce the same ChangeLog file depending on how deep it was cloned. For Libntlm I simply disabled use of generated ChangeLog because I wanted to support an even more extreme form of reproducibility: I wanted to be able to reproduce the full make dist source archives from a minimal git-archive source archive. However for other projects I ve settled with a middle ground. I realized that for git describe to produce reproducible outputs, the shallow clone needs to include the last release tag. So it felt acceptable to assume that the clone is not minimal, but instead has some but not all of the history. I settled with the following recipe to produce ChangeLog's covering all changes since the last release.
dist-hook: gen-ChangeLog
.PHONY: gen-ChangeLog
gen-ChangeLog:
$(AM_V_GEN)if test -e $(srcdir)/.git; then \
LC_ALL=en_US.UTF-8 TZ=UTC0 \
$(top_srcdir)/build-aux/gitlog-to-changelog \
--srcdir=$(srcdir) -- \
v$(PREV_VERSION)~.. > $(distdir)/cl-t && \
printf '\n\nSee the source repo for older entries\n' \
>> $(distdir)/cl-t && \
rm -f $(distdir)/ChangeLog && \
mv $(distdir)/cl-t $(distdir)/ChangeLog; \
fi
I m undecided about the usefulness of generated ChangeLog files within make dist archives. Before we have stable and secure archival of git repositories widely implemented, I can see some utility of this in case we lose all copies of the upstream git repositories. I can sympathize with the concept of ChangeLog files died when we started to generate them from git logs: the files no longer serve any purpose, and we can ask people to go look at the git log instead of reading these generated non-source files.
Long-term reproducible trusted build environment
Distributions comes and goes, and old releases of them goes out of support and often stops working. Which build environment should I chose to build the official release archives? To my knowledge only Guix offers a reliable way to re-create an older build environment (guix gime-machine) that have bootstrappable properties for additional confidence. However I had two difficult problems here. The first one was that I needed Guix container images that were usable in GitLab CI/CD Pipelines, and this side-tracked me for a while. The second one delayed my effort for many months, and I was inclined to give up. Libidn distribute a C# implementation. Some of the C# source code files included in the release tarball are generated. By what? You guess it, by a C# program, with the source code included in the distribution. This means nobody could reproduce the source tarball of Libidn without trusting someone elses C# compiler binaries, which were built from binaries of earlier releases, chaining back into something that nobody ever attempts to build any more and likely fail to build due to bit-rot. I had two basic choices, either remove the C# implementation from Libidn (which may be a good idea for other reasons, since the C and C# are unrelated implementations) or build the source tarball on some binary-only distribution like Trisquel. Neither felt appealing to me, but a late christmas gift of a reproducible Mono came to Guix that resolve this.
Embedded images in Texinfo manual
For Libidn one section of the manual has an image illustrating some concepts. The PNG, PDF and EPS outputs were generated via fig2dev from a *.fig file (hello 1985!) that I had stored in git. Over time, I had also started to store the generated outputs because of build issues. At some point, it was possible to post-process the PDF outputs with grep to remove some timestamps, however with compression this is no longer possible and actually the grep command I used resulted in a 0-byte output file. So my embedded binaries in git was no longer reproducible. I first set out to fix this by post-processing things properly, however I then realized that the *.fig file is not really easy to work with in a modern world. I wanted to create an image from some text-file description of the image. Eventually, via the Guix manual on guix graph, I came to re-discover the graphviz language and tool called dot (hello 1993!). All well then? Oh no, the PDF output embeds timestamps. Binary editing of PDF s no longer work through simple grep, remember? I was back where I started, and after some (soul- and web-) searching I discovered that Ghostscript (hello 1988!) pdfmarks could be used to modify things here. Cooperating with automake s texinfo rules related to make dist proved once again a worthy challenge, and eventually I ended up with a Makefile.am snippet to build images that could be condensed into:
info_TEXINFOS = libidn.texi
libidn_TEXINFOS += libidn-components.png
imagesdir = $(infodir)
images_DATA = libidn-components.png
EXTRA_DIST += components.dot
DISTCLEANFILES = \
libidn-components.eps libidn-components.png libidn-components.pdf
libidn-components.eps: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Teps < $< > $@.tmp
$(AM_V_at)! grep %%CreationDate $@.tmp
$(AM_V_at)mv $@.tmp $@
libidn-components.pdf: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Tpdf < $< > $@.tmp
# A simple sed on CreationDate is no longer possible due to compression.
# 'exiftool -CreateDate' is alternative to 'gs', but adds ~4kb to file.
# Ghostscript add <1kb. Why can't 'dot' avoid setting CreationDate?
$(AM_V_at)printf '[ /ModDate ()\n /CreationDate ()\n /DOCINFO pdfmark\n' > pdfmarks
$(AM_V_at)$(GS) -q -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile=$@.tmp2 $@.tmp pdfmarks
$(AM_V_at)rm -f $@.tmp pdfmarks
$(AM_V_at)mv $@.tmp2 $@
libidn-components.png: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Tpng < $< > $@.tmp
$(AM_V_at)mv $@.tmp $@
pdf-recursive: libidn-components.pdf
dvi-recursive: libidn-components.eps
ps-recursive: libidn-components.eps
info-recursive: $(top_srcdir)/.version libidn-components.png
Surely this can be improved, but I m not yet certain in what way is the best one forward. I like having a text representation as the source of the image. I m sad that the new image size is ~48kb compared to the old image size of ~1kb. I tried using exiftool -CreateDate as an alternative to GhostScript, but using it to remove the timestamp added ~4kb to the file size and naturally I was appalled by this ignorance of impending doom.
Test reproducibility of tarball
Again, you need to continuously test the properties you desire. This means building your project twice using different environments and comparing the results. I ve settled with a small GitLab CI/CD pipeline job that perform bit-by-bit comparison of generated make dist archives. It also perform bit-by-bit comparison of generated git-archive artifacts. See the Libidn2 .gitlab-ci.yml 0-compare job which essentially is:
Notice that I discovered that git archive outputs differ over time too, which is natural but a bit of a nuisance. The output of the job is illuminating in the way that all SHA256 checksums of generated tarballs are included, for example the libidn2 v2.3.8 job log:
I m sure I have forgotten or suppressed some challenges (sprinkling LANG=C TZ=UTC0 helps) related to these goals, but my hope is that this discussion of solutions will inspire you to implement these concepts for your software project too. Please share your thoughts and additional insights in a comment below. Enjoy Happy Hacking in the course of practicing this!
This morning, internet was down at home. The last time I had such an
issue was in February 2023, when my
provider was Oricom. Now I'm with a business service at Teksavvy
Internet (TSI), in which I pay 100$ per month for a 250/50 mbps
business package, with a static IP address, on which I run, well,
everything: email services, this website, etc.
Mitigation
Email
The main problem when the service goes down like this for prolonged
outages is email. Mail is pretty resilient to failures like this but
after some delay (which varies according to the other end), mail
starts to drop. I am actually not sure what the various settings are
among different providers, but I would assume mail is typically kept
for about 24h, so that's our mark.
Last time, I setup VMs at Linode and Digital Ocean to deal better with
this. I have actually kept those VMs running as DNS servers until now,
so that part is already done.
I had fantasized about Puppetizing the mail server configuration so
that I could quickly spin up mail exchangers on those machines. But
now I am realizing that my Puppet server is one of the service that's
down, so this would not work, at least not unless the manifests can be
applied without a Puppet server (say with puppet apply).
Thankfully, my colleague groente did amazing work to refactor our
Postfix configuration in Puppet at Tor, and that gave me the
motivation to reproduce the setup in the lab. So I have finally
Puppetized part of my mail setup at home. That used to be hand-crafted
experimental stuff documented in a couple of pages in this wiki, but
is now being deployed by Puppet.
It's not complete yet: spam filtering (including DKIM checks and
graylisting) are not implemented yet, but that's the next step,
presumably to do during the next outage. The setup should be
deployable with puppet apply, however, and I have refined that
mechanism a little bit, with the run script.
Heck, it's not even deployed yet. But the hard part / grunt work is
done.
Other
The outage was "short" enough (5 hours) that I didn't take time to
deploy the other mitigations I had deployed in the previous incident.
But I'm starting to seriously consider deploying a web (and caching)
reverse proxy so that I endure such problems more gracefully.
Side note on proper servics
Typically, I tend to think of a properly functioning service as having
four things:
backups
documentation
monitoring
automation
high availability
Yes, I miscounted. This is why you have high availability.
Backups
Duh. If data is maliciously or accidentally destroyed, you need a copy
somewhere. Preferably in a way that malicious joe can't get to.
This is harder than you think.
You probably know this is hard, and this is why you're not doing
it. Do it anyways, you'll think it sucks, but you'll be really
grateful for whatever scraps you wrote when you're in trouble.
Monitoring
If you don't have monitoring, you'll know it fails too late, and you
won't know it recovers. Consider high availability, work hard to
reduce noise, and don't have machine wake people up, that's literally
torture and is against the Geneva convention.
Consider predictive algorithm to prevent failures, like "add storage
within 2 weeks before this disk fills up".
This is harder than you think.
Automation
Make it easy to redeploy the service elsewhere.
Yes, I know you have backups. That is not enough: that typically
restores data and while it can also include configuration, you're
going to need to change things when you restore, which is what
automation (or call it "configuration management" if you will) will do
for you anyways.
This also means you can do unit tests on your configuration, otherwise
you're building legacy.
This is probably as hard as you think.
High availability
Make it not fail when one part goes down.
Eliminate single points of failures.
This is easier than you think, except for storage and DNS (which, I
guess, means it's harder than you think too).
Assessment
In the above 5 items, I check two:
backups
documentation
And barely: I'm not happy about the offsite backups, and my
documentation is much better at work than at home (and even there, I
have a 15 year backlog to catchup on).
I barely have monitoring: Prometheus is scraping parts of the infra,
but I don't have any sort of alerting -- by which I don't mean
"electrocute myself when something goes wrong", I mean "there's a set
of thresholds and conditions that define an outage and I can look at
it".
Automation is wildly incomplete. My home server is a random collection
of old experiments and technologies, ranging from Apache with Perl and
CGI scripts to Docker containers running Golang applications. Most of
it is not Puppetized (but the ratio is growing). Puppet itself
introduces a huge attack vector with kind of catastrophic lateral
movement if the Puppet server gets compromised.
And, fundamentally, I am not sure I can provide high availability in
the lab. I'm just this one guy running my home network, and I'm
growing older. I'm thinking more about winding things down than
building things now, and that's just really sad, because I feel
we're losing (well that escalated
quickly).
Resolution
In the end, I didn't need any mitigation and the problem fixed
itself. I did do quite a bit of cleanup so that feels somewhat good,
although I despaired quite a bit at the amount of technical debt I've
accumulated in the lab.
Timeline
Times are in UTC-4.
6:52: IRC bouncer goes offline
9:20: called TSI support, waited on the line 15 minutes then was
told I'd get a call back
9:54: outage apparently detected by TSI
11:00: no response, tried calling back support again
11:10: confirmed bonding router outage, no official ETA but "today",
source of the 9:54 timestamp above
12:08: TPA monitoring notices service restored
12:34: call back from TSI; service restored, problem was with the
"bonder" configuration on their end, which was "fighting between
Montr al and Toronto"
Warning: this is a long ramble I wrote after an outage of my home
internet. You'll get your regular scheduled programming shortly.
I didn't realize this until relatively recently, but we're at war.
Fascists and capitalists are trying to take over the world, and it's
bringing utter chaos.
We're more numerous than them, of course: this is only a handful of
people screwing everyone else over, but they've accumulated so much
wealth and media control that it's getting really, really hard to move
around.
Everything is surveilled: people are carrying tracking and recording
devices in their pockets at all time, or they drive around in
surveillance machines. Payments are all turning digital. There's
cameras everywhere, including in cars. Personal data leaks are so
common people kind of assume their personal address, email address,
and other personal information has already been leaked.
The internet itself is collapsing: most people are using the network
only as a channel to reach a "small" set of "hyperscalers":
mind-boggingly large datacenters that don't really operate like the
old internet. Once you reach the local endpoint, you're not on the
internet anymore. Netflix, Google, Facebook (Instagram, Whatsapp,
Messenger), Apple, Amazon, Microsoft (Outlook, Hotmail, etc), all
those things are not really the internet anymore.
Those companies operate over the "internet" (as in the
TCP/IP network), but they are not an "interconnected network" as
much as their own, gigantic silos so much bigger than everything else
that they essentially dictate how the network operates, regardless of
standards. You access it over "the web" (as in "HTTP") but
the fabric is not made of interconnected links that cross sites: all
those sites are trying really hard to keep you captive on their
platforms.
Besides, you think you're writing an email to the state department,
for example, but you're really writing to Microsoft Outlook. That app
your university or border agency tells you to install, the backend is
not hosted by those institutions, it's on Amazon. Heck, even Netflix
is on Amazon.
Meanwhile I've been operating my own mail server first under my bed
(yes, really) and then in a cupboard or the basement for almost three
decades now. And what for?
So I can tell people I can? Maybe!
I guess the reason I'm doing this is the same reason people are
suddenly asking me about the (dead) mesh again. People are
worried and scared that the world has been taken over, and they're
right: we have gotten seriously screwed.
It's the same reason I keep doing radio, minimally know how to grow
food, ride a bike, build a shed, paddle a canoe, archive and document
things, talk with people, host an assembly. Because, when push comes
to shove, there's no one else who's going to do it for you, at least
not the way that benefits the people.
The Internet is one of humanity's greatest accomplishments. Obviously,
oligarchs and fascists are trying to destroy it. I just didn't expect
the tech bros to be flipping to that side so easily. I thought we were
friends, but I guess we are, after all, enemies.
That said, that old internet is still around. It's getting harder to
host your own stuff at home, but it's not impossible. Mail is tricky
because of reputation, but it's also tricky in the cloud (don't get
fooled!), so it's not that much easier (or cheaper) there.
So there's things you can do, if you're into tech.
Share your wifi with your neighbours.
Build a LAN. Throw a wire over to your neighbour too, it works better
than wireless.
Use Tor. Run a relay, a snowflake, a webtunnel.
Host a web server. Build a site with a static site generator and throw
it in the wind.
Download and share torrents, and why not a tracker.
Run an IRC server (or Matrix, if
you want to federate and lose high availability).
At least use Signal, not Whatsapp or Messenger.
And yes, why not, run a mail server, join a mesh.
Don't write new software, there's plenty of that around already.
(Just kidding, you can write code, cypherpunk.)
You can do many of those things just by setting up a FreedomBox.
That is, after all, the internet: people doing their own thing for
their own people.
Otherwise, it's just like sitting in front of the television and
watching the ads. Opium of the people, like the good old time.
Let a billion droplets build the biggest multitude of clouds that will
storm over this world and rip apart this fascist conspiracy.
Disobey. Revolt. Build.
We are more than them.
Two years after OpenAI launched ChatGPT 3.5, humanity is not on the cusp of
extinction and Elon Musk seems more responsible for job loss than any AI agent.
However, ask any web administrator and you will learn that large language
models are having a significant impact on the world wide
web
(or, for a less technical account, see Forbes articles on
bots).
At May First, a membership organization that has been
supporting thousands of web site for over 20 years, we have never seen anything
like this before.
Wow, my site is getting really popular
It started in 2023. Web sites that performed quite well with a steady
viewership started having traffic spikes. These were relatively easy to
diagnose, since most of the spikes came from visitors that properly identified
themselves as bots, allowing us to see that the big players - OpenAI, Bing,
Google, Facebook - were increasing their efforts to scrape as much content from
web sites as possible.
Small brochure sites were mostly unaffected because they could be scraped in a
matter of minutes. But large sites with an archive of high quality human
written content were getting hammered. Any web site with a search feature or a
calendar or any interface that generated exponential hits that could be
followed were particularly vulnerable.
But hey, that s what robots.txt is for, right? To
tell robots to back off if you don t want them scraping your site?
Eventually, the cracks began to show. Bots were ignoring robots.txt (did they
ever pay that much attention to it in the first place?). Furthermore, rate
limiting requests by user agent also began to fail. When you post a link on
Facebook, a bot identifying itself as facebooketernalhit is invoked to
preview the page so it can show a picture and other meta data. We don t want to
rate limit that bot, right? Except, Facebook is also using this bot to scrape
your site, often bringing your site to its knees. And don t get me started on
TwitterBot.
Eventually, it became clear that the majority of the armies of bots scraping
our sites have completely given up on identifying themselves as bots and are
instead using user agents indistinguishable from regular browsers. By using
thousands of different IP addresses, it has become really hard to separate the
real humans from the bots.
Now what?
So, no, unfortunately, your web site is not suddenly getting really popular.
And, you are blessed with a whole new set of strategic decisions.
Fortunately, May First has undergone a major infrastructure transition,
resulting in centralized logging of all web sites and a fleet of web proxy
servers that intercept all web traffic. Centralized logging means we can
analyze traffic and identify bots more easily, and a web proxy fleet allows us
to more easily implement rules across all web sites.
However, even with all of our latest changes and hours upon hours of work to
keep out the bots, our members are facing some hard decisions about maintaining
an open web.
One member of May First provides Google translations of their web site to every
language available. But wow, that is now a disaster because instead of having
every bot under the sun scrapping all 843 (a made up number) pieces of unique
content on their site, the same bots are scraping 843 * (number of available
languages) pieces of content on their site. Should they stop providing this
translation service in order to ensure people can access their site in the
site s primary language?
Should web sites turn off their search features that include drop down options
of categories to prevent bots from systematically refreshing the search page
with every possible combination of search terms?
Do we need to alter our calendar software to avoid providing endless links into
the future (ok, that is an easy one)?
What s next?
Something has to change.
Lock down web 2.0. Web 2.0 brought us wonderful dynamic web sites, which
Drupal and WordPress and many other pieces of amazing software have
supported for over a decade. This is the software that is getting bogged
down by bots. Maybe we need to figure out a way to lock down the dynamic
aspects of this software to logged in users and provide static content for
everyone else?
Paywalls and accounts everywhere. There s always been an amazing
non-financial reward to providing a web site with high quality movement
oriented content for free. It populates the search engines, provides links
to inspiring and useful content in moments of crises, and can galvanize
movements. But these moments of triumph happen between long periods of hard
labor that now seems to mostly feed capitalist AI scumbags. If we add a new
set of expenses and labor to keep the sites running for this purpose, how
sustainable is that? Will our treasure of free movement content have to move
behind paywalls or logins? If we provide logins, will that keep the bots out
or just create a small hurdle for them to automate the account creation
process? What happens when we can t search for this kind of content via
search engines?
Cutting deals. What if our movement content providers are forced to cut
deals with the AI entrepreneurs to allow the paying scumbags to fund the content
creation. Eww. Enough said.
Bot detection. Maybe we just need to get better at bot detection? This
will surely be an arms race, but would have some good benefits. Bots have
also been filling out our forms and populating our databases with spam,
testing credit cards against our donation pages, conducting denial of
service attacks and all kinds of other irritating acts of vandalism. If we
were better at stopping bots automatically it would have a lot of benefits.
But what impact would it have on our web sites and the experience of using
them? What about good bots (RSS feed readers, payment processors,
web hooks, uptime detectors)? Will we cut the legs off any developer trying
to automate something?
I m not really sure where this is going, but it seems that the world wide web is
about to head in a new direction.
Time flies! 15 years ago, on 2010-03-18, my first upload to the Debian
archive was accepted.
Debian had replaced Windows as my primary OS in 2005, but it was only when I
saw that package
zd1211-firmware had been
orphaned that I thought of becoming a contributor. I owned a Zyxel G-202 USB
WiFi fob that needed said firmware, and as is so often is with open-source
software, I was going to scratch my own itch. Bart Martens thankfully helped
me adopt the package, and sponsored my upload.
I then joined Javier Fern ndez-Sanguino Pe a as a cron maintainer and upstream,
and also worked within the Debian Python Applications, Debian Python Modules,
and Debian Science Teams, where Jakub Wilk and Yaroslav Halchenko were kind
enough to mentor me and eventually support my application to become a
Debian Maintainer.
Life intervened, and I was mostly inactive in Debian for the next two years.
Upon my return in 2014, I had Vincent Cheng to thank for sponsoring most of
my newer work, and for eventually supporting my application to become a
Debian Developer. It was around that time that I also attended my first
DebConf, in Portland, which remains one
of my fondest memories. I had never been to an open-source software conference
before, and DebConf14 really knocked it out of the park in so many ways.
After another break, I returned in 2019 to work mostly on Python and machine
learning libraries. In 2020, I finally completed a process that I had first
started in 2012 but had never managed to finish before: converting cron from
source format 1.0
(one big diff) to
source format 3.0 (quilt)
(a series of patches). This was a process where I converted 25 years worth of
organic growth into a minimal series of logically grouped changes (more
here).
This was my white whale.
In early 2023, shortly after the launch of ChatGPT which triggered an
unprecedented AI boom, I started contributing to the Debian ROCm Team, where
over the following year, I bootstrapped our CI at
ci.rocm.debian.net. Debian's current tooling
lack a way to express dependencies on specific hardware other than CPU ISA,
nor does it have the means to run autopkgtests using such hardware. To get
autopkgtests to make use of AMD GPUs in QEMU VMs and in containers, I had to
fork autopkgtest, debci, and a few other components, as well as create a fair
share of new tooling for ourselves. This worked out pretty well, and the CI
has grown to support 17 different AMD GPU architectures. I will share more
on this in upcoming posts.
I have mentioned a few contributors by name, but I have countless others to
thank for collaborations over the years. It has been a wonderful experience,
and I look forward to many years more.
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1234 other packages on CRAN, downloaded 38.8 million
times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 617 times according
to Google Scholar.
Conrad released a new
minor version 14.4.0 last month. That was preceding by several extensive
rounds of reverse-dependency checks covering the 1200+ packages at CRAN.
We eventually narrowed the impact down to just eight packages, and I
opened issue
#462 to manage the transition along with GitHub-only release
14.4.0-0 of RcppArmadillo.
Several maintainers responded very promptly and updated within days
this is truly appreciated. Yesterday the last package updated at CRAN coinciding nicely with our
planned / intended upload to CRAN one month after the release.
So this new release, at version -1, is now on CRAN. It brings the usual number
of small improvements to Armadillo itself as well as
updates to packaging.
The changes since the last CRAN release are summarised
below.
Changes in
RcppArmadillo version 14.4.0-1 (2025-03-17)
CRAN release having given a few packages time to catch-up to
small upstream change as discussed and managed in #462
Updated bibliography, and small edits to sparse matrix
vignette
Switched continuous integration action to r-ci with implicit
bootstrap
Changes
in RcppArmadillo version 14.4.0-0 (2025-02-17) (GitHub Only)
Upgraded to Armadillo release 14.4.0 (Filtered Espresso)
Faster handling of pow() and square()
within accu() and sum() expressions
Faster sort() and sort_index() for
complex matrices
Expanded the field class with .reshape() and
.resize() member functions
More efficient handling of compound expressions by
sum(), reshape(),
trans()
Better detection of vector expressions by pow(),
imag(), conj()
The package generator helper function now supports additional
DESCRIPTIONs
This release revealed a need for very minor changes for a handful
reverse-dependency packages which will be organized via GitHub issue
tracking
As promised on my previous post, on this entry I ll explain how I ve set up forgejo
actions on the source repository of this site to build it using a runner instead of doing it on the public server using
a webhook to trigger the operation.
Setting up the systemThe first thing I ve done is to disable the forgejo webhook call that was used to publish the site, as I don t want to
run it anymore.
After that I added a new workflow to the repository that does the following things:
push the result to a branch that contains the generated site (we do this because the server is already configured
to work with the git repository and we can use force pushes to keep only the last version of the site, removing the
need of extra code to manage package uploads and removals).
uses curl to send a notification to an instance of the webhook server installed on the
remote server that triggers a script that updates the site using the git branch.
Setting up the webhook serviceOn the server machine we have installed and configured the webhook service to run a script that updates the site.
To install the application and setup the configuration we have used the following script:
#!/bin/shset-e# ---------# VARIABLES# ---------ARCH="$(dpkg --print-architecture)"WEBHOOK_VERSION="2.8.2"DOWNLOAD_URL="https://github.com/adnanh/webhook/releases/download"WEBHOOK_TGZ_URL="$DOWNLOAD_URL/$WEBHOOK_VERSION/webhook-linux-$ARCH.tar.gz"WEBHOOK_SERVICE_NAME="webhook"# FilesWEBHOOK_SERVICE_FILE="/etc/systemd/system/$WEBHOOK_SERVICE_NAME.service"WEBHOOK_SOCKET_FILE="/etc/systemd/system/$WEBHOOK_SERVICE_NAME.socket"WEBHOOK_TML_TEMPLATE="/srv/blogops/action/webhook.yml.envsubst"WEBHOOK_YML="/etc/webhook.yml"# Config file valuesWEBHOOK_USER="$(id-u)"WEBHOOK_GROUP="$(id-g)"WEBHOOK_LISTEN_STREAM="172.31.31.1:4444"# ----# MAIN# ----# Install binary from releases (on Debian only version 2.8.0 is available, but# I need the 2.8.2 version to support the systemd activation mode).
curl -fsSL-o"/tmp/webhook.tgz""$WEBHOOK_TGZ_URL"tar-C /tmp -xzf /tmp/webhook.tgz
sudo install-m 755 "/tmp/webhook-linux-$ARCH/webhook" /usr/local/bin/webhook
rm-rf"/tmp/webhook-linux-$ARCH" /tmp/webhook.tgz
# Service filesudo sh -c"cat >'$WEBHOOK_SERVICE_FILE'"<<EOF
[Unit]
Description=Webhook server
[Service]
Type=exec
ExecStart=webhook -nopanic -hooks $WEBHOOK_YML
User=$WEBHOOK_USER
Group=$WEBHOOK_GROUPEOF
# Socket configsudo sh -c"cat >'$WEBHOOK_SOCKET_FILE'"<<EOF
[Unit]
Description=Webhook server socket
[Socket]
# Set FreeBind to listen on missing addresses (the VPN can be down sometimes)
FreeBind=true
# Set ListenStream to the IP and port you want to listen on
ListenStream=$WEBHOOK_LISTEN_STREAM
[Install]
WantedBy=multi-user.target
EOF
# Config fileBLOGOPS_TOKEN="$(uuid)"\
envsubst <"$WEBHOOK_TML_TEMPLATE"sudo sh -c"cat >$WEBHOOK_YML"chmod 0640 "$WEBHOOK_YML"
chwon "$WEBHOOK_USER:$WEBHOOK_GROUP""$WEBHOOK_YML"# Restart and enable servicesudo systemctl daemon-reload
sudo systemctl stop "$WEBHOOK_SERVICE_NAME.socket"sudo systemctl start "$WEBHOOK_SERVICE_NAME.socket"sudo systemctl enable"$WEBHOOK_SERVICE_NAME.socket"# ----# vim: ts=2:sw=2:et:ai:sts=2
As seen on the code, we ve installed the application using a binary from the project repository instead of a package
because we needed the latest version of the application to use systemd with socket activation.
The configuration file template is the following one:
The version on /etc/webhook.yml has the BLOGOPS_TOKEN adjusted to a random value that has to exported as a secret on
the forgejo project (see later).
Once the service is started each time the action is executed the webhook daemon will get a notification and will run
the following update-blogops.sh script to publish the updated version of the site:
#!/bin/shset-e# ---------# VARIABLES# ---------# ValuesREPO_URL="ssh://git@forgejo.mixinet.net/mixinet/blogops.git"REPO_BRANCH="html"REPO_DIR="public"MAIL_PREFIX="[BLOGOPS-UPDATE-ACTION] "# Address that gets all messages, leave it empty if not wantedMAIL_TO_ADDR="blogops@mixinet.net"# DirectoriesBASE_DIR="/srv/blogops"PUBLIC_DIR="$BASE_DIR/$REPO_DIR"NGINX_BASE_DIR="$BASE_DIR/nginx"PUBLIC_HTML_DIR="$NGINX_BASE_DIR/public_html"ACTION_BASE_DIR="$BASE_DIR/action"ACTION_LOG_DIR="$ACTION_BASE_DIR/log"# FilesOUTPUT_BASENAME="$(date +%Y%m%d-%H%M%S.%N)"ACTION_LOGFILE_PATH="$ACTION_LOG_DIR/$OUTPUT_BASENAME.log"# ---------# Functions# ---------
action_log()echo"$(date-R)$*">>"$ACTION_LOGFILE_PATH"
action_check_directories()for _d in"$ACTION_BASE_DIR""$ACTION_LOG_DIR";do[-d"$_d"]mkdir"$_d"done
action_clean_directories()# Try to remove empty dirsfor _d in"$ACTION_LOG_DIR""$ACTION_BASE_DIR";do
if[-d"$_d"];then
rmdir"$_d" 2>/dev/null true
fi
done
mail_success()to_addr="$MAIL_TO_ADDR"if["$to_addr"];then
subject="OK - updated blogops site"
mail -s"$ MAIL_PREFIX $ subject""$to_addr" <"$ACTION_LOGFILE_PATH"fi
mail_failure()to_addr="$MAIL_TO_ADDR"if["$to_addr"];then
subject="KO - failed to update blogops site"
mail -s"$ MAIL_PREFIX $ subject""$to_addr" <"$ACTION_LOGFILE_PATH"fi
exit 1
# ----# MAIN# ----ret="0"# Check directories
action_check_directories
# Go to the base directorycd"$BASE_DIR"# Remove the old build dir if presentif[-d"$PUBLIC_DIR"];then
rm-rf"$PUBLIC_DIR"fi# Update the repository checkout
action_log "Updating the repository checkout"
git fetch --all>>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"if["$ret"-ne"0"];then
action_log "Failed to update the repository checkout"
mail_failure
fi# Get it from the repo branch & extract it
action_log "Downloading and extracting last site version using 'git archive'"
git archive --remote="$REPO_URL""$REPO_BRANCH""$REPO_DIR"\tar xf - >>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"# Fail if public dir was missingif["$ret"-ne"0"][!-d"$PUBLIC_DIR"];then
action_log "Failed to download or extract site"
mail_failure
fi# Remove old public_html copies
action_log 'Removing old site versions, if present'
find $NGINX_BASE_DIR-mindepth 1 -maxdepth 1 -name'public_html-*'-type d \-execrm-rf\;>>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"if["$ret"-ne"0"];then
action_log "Removal of old site versions failed"
mail_failure
fi# Switch site directoryTS="$(date +%Y%m%d-%H%M%S)"if[-d"$PUBLIC_HTML_DIR"];then
action_log "Moving '$PUBLIC_HTML_DIR' to '$PUBLIC_HTML_DIR-$TS'"mv"$PUBLIC_HTML_DIR""$PUBLIC_HTML_DIR-$TS">>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"fi
if["$ret"-eq"0"];then
action_log "Moving '$PUBLIC_DIR' to '$PUBLIC_HTML_DIR'"mv"$PUBLIC_DIR""$PUBLIC_HTML_DIR">>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"fi
if["$ret"-ne"0"];then
action_log "Site switch failed"
mail_failure
else
action_log "Site updated successfully"
mail_success
fi# ----# vim: ts=2:sw=2:et:ai:sts=2
The hugo-adoc workflowThe workflow is defined in the .forgejo/workflows/hugo-adoc.yml file and looks like this:
name:hugo-adoc# Run this job on push events to the main branchon:push:branches:-'main'jobs:build-and-push:if:$ vars.BLOGOPS_WEBHOOK_URL != '' && secrets.BLOGOPS_TOKEN != '' runs-on:dockercontainer:image:forgejo.mixinet.net/oci/hugo-adoc:latest# Allow the job to write to the repository (not really needed on forgejo)permissions:contents:writesteps:-name:Checkout the repouses:actions/checkout@v4with:submodules:'true'-name:Build the siteshell:shrun:rm -rf publichugo-name:Push compiled site to html branchshell:shrun:# Set the git usergit config --global user.email "blogops@mixinet.net"git config --global user.name "BlogOps"# Create a new orphan branch called html (it was not pulled by the# checkout step)git switch --orphan html# Add the public directory to the branchgit add public# Commit the changesgit commit --quiet -m "Updated site @ $(date -R)" public# Push the changes to the html branchgit push origin html --force# Switch back to the main branchgit switch main-name:Call the blogops update webhook endpointshell:shrun:HEADER="X-Blogops-Token: $ secrets.BLOGOPS_TOKEN "curl --fail -k -H "$HEADER" $ vars.BLOGOPS_WEBHOOK_URL
The only relevant thing is that we have to add the BLOGOPS_TOKEN variable to the project secrets (its value is the one
included on the /etc/webhook.yml file created when installing the webhook service) and the BLOGOPS_WEBHOOK_URL
project variable (its value is the URL of the webhook server, in my case
http://172.31.31.1:4444/hooks/update-blogops); note that the job includes the -k flag on the curl command just in
case I end up using TLS on the webhook server in the future, as discussed previously.
ConclusionNow that I have forgejo actions on my server I no longer need to build the site on the public server as I did initially,
a good thing when the server is a small OVH VPS that only runs a couple of containers and a web server directly on the
host.
I m still using a notification system to make the server run a script to update the site because that way the forgejo
server does not need access to the remote machine shell, only the webhook server which, IMHO, is a more secure setup.
Last week I decided I wanted to try out forgejo actions to build this blog instead of using
webhooks, so I looked the documentation and started playing with it until I had it working as I wanted.
This post is to describe how I ve installed and configured a forgejo runner, how I ve added an
oci organization to my instance to build, publish and mirror container images and added a couple of
additional organizations (actions and docker for now) to mirror interesting
actions.
The changes made to build the site using actions will be documented on a separate post, as I ll be using this entry to
test the new setup on the blog project.
Installing the runnerThe first thing I ve done is to install a runner on my server, I decided to use the
OCI image installation method, as it seemed to be the easiest and fastest
one.
The commands I ve used to setup the runner are the following:
$ cd /srv
$ git clone https://forgejo.mixinet.net/blogops/forgejo-runner.git
$ cd forgejo-runner
$ sh ./bin/setup-runner.sh
The setup-runner.sh script does multiple things:
create a forgejo-runner user and group
create the necessary directories for the runner
create a .runner file with a predefined secret and the docker label
The setup-runner.sh code is available here.
After running the script the runner has to be registered with the forgejo server, it can be done using the following
command:
The RUNNER_NAME variable is defined on the setup-runner.sh script and the FORGEJO_SECRET must match the value used
on the .runner file.
Starting it with docker-composeTo launch the runner I m going to use a docker-compose.yml file that starts two containers, a docker in docker
service to run the containers used by the workflow jobs and another one that runs the forgejo-runner itself.
The initial version used a TCP port to communicate with the dockerd server from the runner, but when I tried to build
images from a workflow I noticed that the containers launched by the runner were not going to be able to execute
another dockerd inside the dind one and, even if they were, it was going to be expensive computationally.
To avoid the issue I modified the dind service to use a unix socket on a shared volume that can be used by the
runner service to communicate with the daemon and also re-shared with the job containers so the dockerd server can
be used from them to build images.
Warning: The use of the same docker server that runs the jobs from them has security implications, but this instance is
for a home server where I am the only user, so I am not worried about it and this way I can save some resources (in
fact, I could use the host docker server directly instead of using a dind service, but just in case I want to run
other containers on the host I prefer to keep the one used for the runner isolated from it).
For those concerned about sharing the same server an alternative would be to launch a second dockerd only for the jobs
(i.e. actions-dind) using the same approach (the volume with its socket will have to be shared with the runner
service so it can be re-shared, but the runner does not need to use it).
There are multiple things to comment about this file:
The dockerd server is started with the -H unix:///dind/docker.sock flag to use the unix socket to communicate
with the daemon instead of using a TCP port (as said, it is faster and allows us to share the socket with the
containers started by the runner).
We are running the dockerd daemon with the RUNNER_GID group so the runner can communicate with it (the socket
gets that group which is the same used by the runner).
The runner container mounts three volumes: the data directory, the dind folder where docker creates the unix
socket and a config.yaml file used by us to change the default runner configuration.
The config.yaml file was originally created using the forgejo-runner:
$ docker run --rm data.forgejo.org/forgejo/runner:6.2.2 \
forgejo-runner generate-config > config.yaml
The changes to it are minimal, the runner capacity has been increased to 2 (that allows it to run two jobs at the
same time) and the /dind/docker.sock value has been added to the valid_volumes key to allow the containers launched
by the runner to mount it when needed; the diff against the default version is as follows:
@@ -13,7 +13,8 @@
# Where to store the registration result.
file: .runner
# Execute how many tasks concurrently at the same time.
- capacity: 1
+ # STO: Allow 2 concurrent tasks
+ capacity: 2
# Extra environment variables to run jobs.
envs:
A_TEST_ENV_NAME_1: a_test_env_value_1
@@ -87,7 +88,9 @@
# If you want to allow any volume, please use the following configuration:
# valid_volumes:
# - '**'
- valid_volumes: []
+ # STO: Allow to mount the /dind/docker.sock on the containers
+ valid_volumes:
+ - /dind/docker.sock
# overrides the docker client host with the specified one.
# If "-" or "", an available docker host will automatically be found.
# If "automount", an available docker host will automatically be found and ...
To start the runner we export the RUNNER_UID and RUNNER_GID variables and call docker-compose up to start the
containers on the background:
$ RUNNER_UID="$(id-u forgejo-runner)"RUNNER_GID="$(id-g forgejo-runner)"\
docker compose up -d
If the server was configured right we are now able to start using actions with this runner.
Preparing the system to run things locallyTo avoid unnecessary network traffic we are going to create a multiple organizations in our forgejo instance to maintain
our own actions and container images and mirror remote ones.
The rationale behind the mirror use is that we reduce a lot the need to connect to remote servers to download the
actions and images, which is good for performance and security reasons.
In fact, we are going to build our own images for some things to install the tools we want without needing to do it over
and over again on the workflow jobs.
Mirrored actionsThe actions we are mirroring are on the actions and docker organizations, we have
created the following ones for now (the mirrors were created using the forgejo web interface and we have disabled
manually all the forgejo modules except the code one for them):
To use our actions by default (i.e., without needing to add the server URL on the uses keyword) we have added the
following section to the app.ini file of our forgejo server:
Setting up credentials to push imagesTo be able to push images to the oci organization I ve created a token with package:write permission for my own
user because I m a member of the organization and I m authorized to publish packages on it (a different user could be
created, but as I said this is for personal use, so there is no need to complicate things for now).
To allow the use of those credentials on the actions I have added a secret (REGISTRY_PASS) and a variable
(REGISTRY_USER) to the oci organization to allow the actions to use them.
I ve also logged myself on my local docker client to be able to push images to the oci group by hand, as I it is
needed for bootstrapping the system (as I m using local images on the worflows I need to push them to the server before
running the ones that are used to build the images).
Local and mirrored imagesOur images will be stored on the packages section of a new organization called oci, inside it we have
created two projects that use forgejo actions to keep things in shape:
images: contains the source files used to generate our own images and the actions to build, tag and
push them to the oci organization group.
mirrors: contains a configuration file for the regsync tool to mirror containers and an
action to run it.
On the next sections we are going to describe the actions and images we have created and mirrored from those projects.
The oci/images projectThe images project is a monorepo that contains the source files for the images we are going to build and a couple of
actions.
The image sources are on sub directories of the repository, to be considered an image the folder has to contain a
Dockerfile that will be used to build the image.
The repository has two workflows:
build-image-from-tag: Workflow to build, tag and push an image to the oci organization
multi-semantic-release: Workflow to create tags for the images using the multi-semantic-release tool.
As the workflows are already configured to use some of our images we pushed some of them from a checkout of the
repository using the following commands:
On the next sub sections we will describe what the workflows do and will show their source code.
build-image-from-tag workflowThis workflow uses a docker client to build an image from a tag on the repository with the format
image-name-v[0-9].[0-9].[0-9]+.
As the runner is executed on a container (instead of using lxc) it seemed unreasonable to run another dind
container from that one, that is why, after some tests, I decided to share the dind service server socket with the
runner container and enabled the option to mount it also on the containers launched by the runner when needed (I only
do it on the build-image-from-tag action for now).
The action was configured to run using a trigger or when new tags with the right format were created, but when the tag
is created by multi-semantic-release the trigger does not work for some reason, so now it only runs the job on
triggers and checks if it is launched for a tag with the right format on the job itself.
The source code of the action is as follows:
name:build-image-from-tagon:workflow_dispatch:jobs:build:# Don't build the image if the registry credentials are not set, the ref is not a tag or it doesn't contain '-v'if:$ vars.REGISTRY_USER != '' && secrets.REGISTRY_PASS != '' && startsWith(github.ref, 'refs/tags/') && contains(github.ref, '-v') runs-on:dockercontainer:image:forgejo.mixinet.net/oci/node-mixinet:latest# Mount the dind socket on the container at the default locationoptions:-v /dind/docker.sock:/var/run/docker.socksteps:-name:Extract image name and tag from git and get registry name from envid:job_datarun:echo "::set-output name=img_name::$ GITHUB_REF_NAME%%-v* "echo "::set-output name=img_tag::$ GITHUB_REF_NAME##*-v "echo "::set-output name=registry::$(echo "$ github.server_url " sed -e 's%https://%%')"echo "::set-output name=oci_registry_prefix::$(echo "$ github.server_url /oci" sed -e 's%https://%%')"-name:Checkout the repouses:actions/checkout@v4-name:Export build dir and Dockerfileid:build_datarun:img="$ steps.job_data.outputs.img_name "build_dir="$(pwd)/$ img "dockerfile="$ build_dir /Dockerfile"if [ -f "$dockerfile" ]; thenecho "::set-output name=build_dir::$build_dir"echo "::set-output name=dockerfile::$dockerfile"elseecho "Couldn't find the Dockerfile for the '$img' image"exit 1fi-name:Login to the Container Registryuses:docker/login-action@v3with:registry:$ steps.job_data.outputs.registry username:$ vars.REGISTRY_USER password:$ secrets.REGISTRY_PASS -name:Set up Docker Buildxuses:docker/setup-buildx-action@v3-name:Build and Pushuses:docker/build-push-action@v6with:push:truetags:$ steps.job_data.outputs.oci_registry_prefix /$ steps.job_data.outputs.img_name :$ steps.job_data.outputs.img_tag $ steps.job_data.outputs.oci_registry_prefix /$ steps.job_data.outputs.img_name :latestcontext:$ steps.build_data.outputs.build_dir file:$ steps.build_data.outputs.dockerfile build-args:OCI_REGISTRY_PREFIX=$ steps.job_data.outputs.oci_registry_prefix /
Some notes about this code:
The if condition of the build job is not perfect, but it is good enough to avoid wrong uses as long as nobody
uses manual tags with the wrong format and expects things to work (it checks if the REGISTRY_USER and
REGISTRY_PASS variables are set, if the ref is a tag and if it contains the -v string).
To be able to access the dind socket we mount it on the container using the options key on the container section
of the job (this only works if supported by the runner configuration as explained before).
We use the job_data step to get information about the image from the tag and the registry URL from the environment
variables, it is executed first because all the information is available without checking out the repository.
We use the job_data step to get the build dir and Dockerfile paths from the repository (right now we are
assuming fixed paths and checking if the Dockerfile exists, but in the future we could use a configuration file to
get them, if needed).
As we are using a docker daemon that is already running there is no need to use the
docker/setup-docker-action to install it.
On the build and push step we pass the OCI_REGISTRY_PREFIX build argument to the Dockerfile to be able to use it
on the FROM instruction (we are using it in our images).
multi-semantic-release workflowThis workflow is used to run the multi-semantic-release tool on pushes to the main branch.
It is configured to create the configuration files on the fly (it prepares things to tag the folders that contain a
Dockerfile using a couple of template files available on the repository s .forgejo directory) and run the
multi-semantic-release tool to create tags and push them to the repository if new versions are to be built.
Initially we assumed that the tag creation pushed by multi-semantic-release would be enough to run the
build-tagged-image-task action, but as it didn t work we removed the rule to run the action on tag creation and added
code to trigger the action using an api call for the newly created tags (we get them from the output of the
multi-semantic-release execution).
The source code of the action is as follows:
name:multi-semantic-releaseon:push:branches:-'main'jobs:multi-semantic-release:runs-on:dockercontainer:image:forgejo.mixinet.net/oci/multi-semantic-release:lateststeps:-name:Checkout the repouses:actions/checkout@v4-name:Generate multi-semantic-release configurationshell:shrun:# Get the list of images to work with (the folders that have a Dockerfile)images="$(for img in */Dockerfile; do dirname "$img"; done)"# Generate a values.yaml file for the main packages.json filepackage_json_values_yaml=".package.json-values.yaml"echo "images:" >"$package_json_values_yaml"for img in $images; doecho " - $img" >>"$package_json_values_yaml"doneecho "::group::Generated values.yaml for the project"cat "$package_json_values_yaml"echo "::endgroup::"# Generate the package.json file validating that is a good json file with jqtmpl -f "$package_json_values_yaml" ".forgejo/package.json.tmpl" jq . > "package.json"echo "::group::Generated package.json for the project"cat "package.json"echo "::endgroup::"# Remove the temporary values filerm -f "$package_json_values_yaml"# Generate the package.json file for each imagefor img in $images; dotmpl -v "img_name=$img" -v "img_path=$img" ".forgejo/ws-package.json.tmpl" jq . > "$img/package.json"echo "::group::Generated package.json for the '$img' image"cat "$img/package.json"echo "::endgroup::"done-name:Run multi-semantic-releaseshell:shrun:multi-semantic-release tee .multi-semantic-release.log-name:Trigger buildsshell:shrun:# Get the list of tags published on the previous stepstags="$(sed -n -e 's/^\[.*\] \[\(.*\)\] .* Published release \([0-9]\+\.[0-9]\+\.[0-9]\+\) on .*$/\1-v\2/p' \.multi-semantic-release.log)"rm -f .multi-semantic-release.logif [ "$tags" ]; then# Prepare the url for building the imagesworkflow="build-image-from-tag.yaml"dispatch_url="$ github.api_url /repos/$ github.repository /actions/workflows/$workflow/dispatches"echo "$tags" while read -r tag; doecho "Triggering build for tag '$tag'"curl \-H "Content-Type:application/json" \-H "Authorization: token $ secrets.GITHUB_TOKEN " \-d " \"ref\":\"$tag\" " "$dispatch_url"donefi
Notes about this code:
The use of the tmpl tool to process the multi-semantic-release configuration templates comes from previous uses,
but on this case we could use a different approach (i.e. envsubst could be used) but we left it because it keeps
things simple and can be useful in the future if we want to do more complex things with the template files.
We use tee to show and dump to a file the output of the multi-semantic-release execution.
We get the list of pushed tags using sed against the output of the multi-semantic-release execution and for
each one found we use curl to call the forgejo API to trigger the build job; as the call is against the same
project we can use the GITHUB_TOKEN generated for the workflow to do it, without creating a user token that has to
be shared as a secret.
The .forgejo/package.json.tmpl file is the following one:
"name":"multi-semantic-release","version":"0.0.0-semantically-released","private":true,"multi-release":"tagFormat":"$ name -v$ version " ,"workspaces":.imagestoJson
As can be seen it only needs a list of paths to the images as argument (the file we generate contains the names and
paths, but it could be simplified).
And the .forgejo/ws-package.json.tmpl file is the following one:
The oci/mirrors projectThe repository contains a template for the configuration file we are going to use with regsync
(regsync.envsubst.yml) to mirror images from remote registries using a workflow that generates a configuration file
from the template and runs the tool.
The initial version of the regsync.envsubst.yml file is prepared to mirror alpine containers from version 3.21 to
3.29 (we explicitly remove version 3.20) and needs the forgejo.mixinet.net/oci/node-mixinet:latest image to run
(as explained before it was pushed manually to the server):
mirror workflowThe mirror workflow creates a configuration file replacing the value of the REGISTRY environment variable (computed
by removing the protocol from the server_url), the REGISTRY_USER organization value and the REGISTRY_PASS secret
using the envsubst command and running the regsync tool to mirror the images using the configuration file.
The action is configured to run daily, on push events when the regsync.envsubst.yml file is modified on the main
branch and can also be triggered manually.
The source code of the action is as follows:
.forgejo/workflows/mirror.yaml
ConclusionWe have installed a forgejo-runner and configured it to run actions for our own server and things are working fine.
This approach allows us to have a powerful CI/CD system on a modest home server, something very useful for maintaining
personal projects and playing with things without needing SaaS platforms like github or
gitlab.
FreedomBox is a Debian blend that makes it easier to run your own server. Approximately every two years, there is a new stable release of Debian. This year s release will be called Debian 13 "trixie".
This post will provide an overview of changes between FreedomBox 23.6 (the version that shipped in Debian 12 "bookworm") and 25.5 (the latest release). Note: Debian 13 "trixie" is not yet released, so things may still change, be added or removed, before the official release.
General
A number of translations were updated, including Albanian, Arabic, Belarusian, Bulgarian, Chinese (Simplified Han script), Chinese (Traditional Han script), Czech, Dutch, French, German, Hindi, Japanese, Norwegian Bokm l, Polish, Portuguese, Russian, Spanish, Swedish, Telugu, Turkish, and Ukrainian.
Fix cases where a package or service is used by multiple apps, so that disabling or uninstalling one app does not affect the other app.
When uninstalling an app, purge the packages, to remove all data and configuration.
For configuration files that need to be placed into folders owned by other packages, we now install these files under /usr/share/freedombox/etc/, and create a symbolic link to the other package s configuration folder. This prevents the files being lost when other packages are purged.
Add an action to re-run the setup process for an app. This can fix many of the possible issues that occur.
Various improvements related to the "force upgrade" feature, which handles upgrading packages with conffile prompts.
Fix install/uninstall issues for apps that use MySQL database (WordPress, Zoph).
Improve handling of file uploads (Backups, Feather Wiki, Kiwix).
Switch to Bootstrap 5 front-end framework.
Removed I2P app, since the i2p package was removed from Debian.
Various user interface changes, including:
Add tags for apps, replacing short descriptions. When a tag is clicked, search and filter for one or multiple tags.
Organize the System page into sections.
Add breadcrumbs for page hierarchy navigation.
Add next steps page after initial FreedomBox setup.
Diagnostics
Add diagnostic checks to detect common errors.
Add diagnostics daily run, with notifications about failures.
Add Repair action for failed diagnostics, and option for automatic repairs.
Name Services
Move hostname and domain name configuration to Names page.
Support multiple static and/or dynamic domains.
Use systemd-resolved for DNS resolution.
Add options for setting global DNS-over-TLS and DNSSEC preferences.
Networks
Add more options for IPv6 configuration method.
Overhaul Wi-Fi networks scan page.
Privacy
Add option to disable fallback DNS servers.
Add option to set the lookup URL to get the public IP address of the FreedomBox.
Users and Groups
Delete or move home folder when user is deleted or renamed.
When a user is inactivated, also inactivate the user in LDAP.
Deluge
This BitTorrent client app should be available once again in Debian 13 "trixie".
Ejabberd
Turn on Message Archive Management setting by default, to help various XMPP clients use it.
Feather Wiki
Add new app for note taking.
This app lives in a single HTML file, which is downloaded from the FreedomBox website.
GitWeb
Disable snapshot feature, due to high resource use.
Various fixes for repository operations.
GNOME
Add new app to provide a graphical desktop environment.
Requires a monitor, keyboard, and mouse to be physically connected to the FreedomBox.
Not suitable for low-end hardware.
ikiwiki
Disable discussion pages by default for new wiki/blog, to avoid spam.
Kiwix
Add new app for offline reader of Wikipedia and other sites.
Matrix Synapse
Add an option for token-based registration verification, so that users signing up for new accounts will need to provide a token during account registration.
MediaWiki
Allow setting the site language code.
Increase PHP maximum execution time to 100 seconds.
MiniDLNA
Add media directory selection form.
Miniflux
Add new app for reading news from RSS/ATOM feeds.
Nextcloud
Add new app for file sync and collaboration.
Uses a Docker container maintained by the Nextcloud community. The container is downloaded from FreedomBox container registry.
OpenVPN
Renew server/client certificates, and set expiry to 10 years.
Postfix/Dovecot
Fix DKIM signing.
Show DNS entries for all domains.
Shadowsocks Server
Add new app for censorship resistance, separate from Shadowsocks Client app.
SOGo
Add new app for groupware (webmail, calendar, tasks, and contacts).
Works with Postfix/Dovecot email server app.
TiddlyWiki
Add new app for note taking.
This app lives in a single HTML file, which is downloaded from the FreedomBox website.
Tor Proxy
Add new app for Tor SOCKS proxy, separate from Tor app.
Transmission
Allow remote user interfaces to connect.
Conclusion
Over the past two years, FreedomBox has been increasing the number of features and applications available to its users. We have also focused on improving the reliability of the system, detecting unexpected situations, and providing means to return to a known good state. With these improvements, FreedomBox has become a good solution for people with limited time or energy to set up and start running a personal server, at home or in the cloud.
Looking forward, we would like to focus on making more powerful hardware available with FreedomBox pre-installed and ready to be used. This hardware would also support larger storage devices, making it suitable as a NAS or media server. We are also very interested in exploring new features such as atomic updates, which will further enhance the reliability of the system.
An offline PKI enhances security by physically isolating the certificate
authority from network threats. A YubiKey is a low-cost solution to store a
root certificate. You also need an air-gapped environment to operate the root
CA.
Offline PKI backed up by 3 YubiKeys
This post describes an offline PKI system using the following components:
2 YubiKeys for the root CA (with a 20-year validity),
1 YubiKey for the intermediate CA (with a 5-year validity), and
It is possible to add more YubiKeys as a backup of the root CA if needed. This
is not needed for the intermediate CA as you can generate a new one if
the current one gets destroyed.
The software part
offline-pki is a small Python application to manage an offline PKI.
It relies on yubikey-manager to manage YubiKeys and cryptography for
cryptographic operations not executed on the YubiKeys. The application has some
opinionated design choices. Notably, the cryptography is hard-coded to use NIST
P-384 elliptic curve.
The first step is to reset all your YubiKeys:
$ offline-pkiyubikeyreset
This will reset the connected YubiKey. Are you sure? [y/N]: yNew PIN code:Repeat for confirmation:New PUK code:Repeat for confirmation:New management key ('.' to generate a random one):WARNING[pki-yubikey] Using random management key: e8ffdce07a4e3bd5c0d803aa3948a9c36cfb86ed5a2d5cf533e97b088ae9e629INFO[pki-yubikey] 0: Yubico YubiKey OTP+FIDO+CCID 00 00INFO[pki-yubikey] SN: 23854514INFO[yubikit.management] Device config writtenINFO[yubikit.piv] PIV application data reset performedINFO[yubikit.piv] Management key setINFO[yubikit.piv] New PUK setINFO[yubikit.piv] New PIN setINFO[pki-yubikey] YubiKey reset successful!
Then, generate the root CA and create as many copies as you want:
$ offline-pkicertificateroot--permittedexample.com
Management key for Root X:Plug YubiKey "Root X"...INFO[pki-yubikey] 0: Yubico YubiKey CCID 00 00INFO[pki-yubikey] SN: 23854514INFO[yubikit.piv] Data written to object slot 0x5fc10aINFO[yubikit.piv] Certificate written to slot 9C (SIGNATURE), compression=TrueINFO[yubikit.piv] Private key imported in slot 9C (SIGNATURE) of type ECCP384Copy root certificate to another YubiKey? [y/N]: yPlug YubiKey "Root X"...INFO[pki-yubikey] 0: Yubico YubiKey CCID 00 00INFO[pki-yubikey] SN: 23854514INFO[yubikit.piv] Data written to object slot 0x5fc10aINFO[yubikit.piv] Certificate written to slot 9C (SIGNATURE), compression=TrueINFO[yubikit.piv] Private key imported in slot 9C (SIGNATURE) of type ECCP384Copy root certificate to another YubiKey? [y/N]: n
Then, you can create an intermediate certificate with offline-pki yubikey
intermediate and use it to sign certificates by providing a CSR to offline-pki
certificate sign. Be careful and inspect the CSR before signing it, as only the
subject name can be overridden. Check the documentation for more details.
Get the available options using the --help flag.
The hardware part
To ensure the operations on the root and intermediate CAs are air-gapped,
a cost-efficient solution is to use an ARM64 single board computer. The Libre
Computer Sweet Potato SBC is a more open alternative to the well-known
Raspberry Pi.1
Libre Computer Sweet Potato SBC, powered by the AML-S905X SOC
I interact with it through an USB to TTL UART converter:
$ tio/dev/ttyUSB0
[16:40:44.546] tio v3.7[16:40:44.546] Press ctrl-t q to quit[16:40:44.555] Connected to /dev/ttyUSB0GXL:BL1:9ac50e:bb16dc;FEAT:ADFC318C:0;POC:1;RCY:0;SPI:0;0.0;CHK:0;TE: 36574BL2 Built : 15:21:18, Aug 28 2019. gxl g1bf2b53 - luan.yuan@droid15-szset vcck to 1120 mvset vddee to 1000 mvBoard ID = 4CPU clk: 1200MHz[ ]
The Nix glue
To bring everything together, I am using Nix with a Flake providing:
a package for the offline-pki application, with shell completion,
a development shell, including an editable version of the offline-pki application,
a NixOS module to setup the offline PKI, resetting the system at each boot,
a QEMU image for testing, and
an SD card image to be used on the Sweet Potato or another ARM64 SBC.
# Execute the application locally
nixrungithub:vincentbernat/offline-pki----help
# Run the application inside a QEMU VM
nixrungithub:vincentbernat/offline-pki\#qemu
# Build a SD card for the Sweet Potato or for the Raspberry Pi
nixbuild--systemaarch64-linuxgithub:vincentbernat/offline-pki\#sdcard.potato
nixbuild--systemaarch64-linuxgithub:vincentbernat/offline-pki\#sdcard.generic
# Get a development shell with the application
nixdevelopgithub:vincentbernat/offline-pki
The key for the root CA is not generated by the YubiKey. Using an
air-gapped computer is all the more important. Put it in a safe with the
YubiKeys when done!
Thank you everyone for keeping the lights on for a bit longer. KDE snaps have been restored. I also released 24.12.3! In addition, I have moved most snaps to core24. The remaining snaps need newer qt6/kf6, which is a WIP. The Bad luck girl has been hit once again with another loss, so with that, I will be reducing my hours on snaps while I consider my options for my future. I am still around, just a bit less.
Thanks again everyone, if you can get me through one more ( lingering broken arm ) surgery I would be forever grateful! https://gofund.me/d5d59582
A few months ago I explained that one reason why this blog has become more quiet is that all my work on Lean is covered elsewhere.
This post is an exception, because it is an observation that is (arguably) interesting, but does not lead anywhere, so where else to put it than my own blog
Want to share your thoughts about this? Please join the discussion on the Lean community zulip!
Background
When defining a function recursively in Lean that has nested recursion, e.g. a recusive call that is in the argument to a higher-order function like List.map, then extra attention used to be necessary so that Lean can see that xs.map applies its argument only elements of the list xs. The usual idiom is to write xs.attach.map instead, where List.attach attaches to the list elements a proof that they are in that list. You can read more about this my Lean blog post on recursive definitions and our new shiny reference manual, look for Example Nested Recursion in Higher-order Functions .
To make this step less tedious I taught Lean to automatically rewrite xs.map to xs.attach.map (where suitable) within the construction of well-founded recursion, so that nested recursion just works (issue #5471). We already do such a rewriting to change if c then else to the dependent if h : c then else , but the attach-introduction is much more ambitious (the rewrites are not definitionally equal, there are higher-order arguments etc.) Rewriting the terms in a way that we can still prove the connection later when creating the equational lemmas is hairy at best. Also, we want the whole machinery to be extensible by the user, setting up their own higher order functions to add more facts to the context of the termination proof.
I implemented it like this (PR #6744) and it ships with 4.18.0, but in the course of this work I thought about a quite different and maybe better way to do this, and well-founded recursion in general:
WellFounded.fix : (hwf : WellFounded r) (F : (x : ) ((y : ) r y x C y) C x) (x : ) : C x
we have to rewrite the functorial of the recursive function, which naturally has type
F : ((y : ) C y) ((x : ) C x)
to the one above, where all recursive calls take the termination proof r y x. This is a fairly hairy operation, mangling the type of matcher s motives and whatnot.
Things are simpler for recursive definitions using the new partial_fixpoint machinery, where we use Lean.Order.fix
so the functorial s type is unmodified (here will be ((x : ) C x)), and everything else is in the propositional side-condition montone F. For this predicate we have a syntax-guided compositional tactic, and it s easily extensible, e.g. by
theorem monotone_mapM (f : m ) (xs : List ) (hmono : monotone f) :
monotone (fun x => xs.mapM (f x))
Once given, we don t care about the content of that proof. In particular proving the unfolding theorem only deals with the unmodified F that closely matches the function definition as written by the user. Much simpler!
Isabelle has it easier
Isabelle also supports well-founded recursion, and has great support for nested recursion. And it s much simpler!
There, all you have to do to make nested recursion work is to define a congruence lemma of the form, for List.map something like our List.map_congr_left
List.map_congr_left : (h : a l, f a = g a) :
List.map f l = List.map g l
This is because in Isabelle, too, the termination proofs is a side-condition that essentially states the functorial F calls its argument f only on smaller arguments .
Can we have it easy, too?
I had wished we could do the same in Lean for a while, but that form of congruence lemma just isn t strong enough for us.
But maybe there is a way to do it, using an existential to give a witness that F can alternatively implemented using the more restrictive argument. The following callsOn P F predicate can express that F calls its higher-order argument only on arguments that satisfy the predicate P:
section setup
variable : Sort u
variable : Sort v
variable : Sort w
def callsOn (P : Prop) (F : ( y, y) ) :=
(F': ( y, P y y) ), f, F' (fun y _ => f y) = F f
variable (R : Prop)
variable (F : ( y, y) ( x, x))
local infix:50 " " => R
def recursesVia : Prop := x, callsOn ( x) (fun f => F f x)
noncomputable def fix (wf : WellFounded R) (h : recursesVia R F) : ( x, x) :=
wf.fix (fun x => (h x).choose)
def fix_eq (wf : WellFounded R) h x :
fix R F wf h x = F (fix R F wf h) x := by
unfold fix
rw [wf.fix_eq]
apply (h x).choose_spec
This allows nice compositional lemmas to discharge callsOn predicates:
theorem callsOn_base (y : ) (hy : P y) :
callsOn P (fun (f : x, x) => f y) := by
exists fun f => f y hy
intros; rfl
@[simp]
theorem callsOn_const (x : ) :
callsOn P (fun (_ : x, x) => x) :=
fun _ => x, fun _ => rfl
theorem callsOn_app
: Sort uu : Sort ww
(F : ( y, y) ) -- can this also support dependent types?
(F : ( y, y) )
(h : callsOn P F )
(h : callsOn P F ) :
callsOn P (fun f => F f (F f)) := by
obtain F ', h := h
obtain F ', h := h
exists (fun f => F ' f (F ' f))
intros; simp_all
theorem callsOn_lam
: Sort uu
(F : ( y, y) ) -- can this also support dependent types?
(h : x, callsOn P (F x)) :
callsOn P (fun f x => F x f) := by
exists (fun f x => (h x).choose f)
intro f
ext x
apply (h x).choose_spec
theorem callsOn_app2
: Sort uu : Sort ww
(g : )
(F : ( y, y) ) -- can this also support dependent types?
(F : ( y, y) )
(h : callsOn P F )
(h : callsOn P F ) :
callsOn P (fun f => g (F f) (F f)) := by
apply_rules [callsOn_app, callsOn_const]
With this setup, we can have the following, possibly user-defined, lemma expressing that List.map calls its arguments only on elements of the list:
theorem callsOn_map ( : Type uu) ( : Type ww)
(P : Prop) (F : ( y, y) ) (xs : List )
(h : x, x xs callsOn P (fun f => F f x)) :
callsOn P (fun f => xs.map (fun x => F f x)) := by
suffices callsOn P (fun f => xs.attach.map (fun x, h => F f x)) by
simpa
apply callsOn_app
apply callsOn_app
apply callsOn_const
apply callsOn_lam
intro x', hx'
dsimp
exact (h x' hx')
apply callsOn_const
end setup
So here is the (manual) construction of a nested map for trees:
section examples
structure Tree ( : Type u) where
val :
cs : List (Tree )
-- essentially
-- def Tree.map (f : ) : Tree Tree :=
-- fun t => f t.val, t.cs.map Tree.map )
noncomputable def Tree.map (f : ) : Tree Tree :=
fix (sizeOf < sizeOf ) (fun map t => f t.val, t.cs.map map )
(InvImage.wf (sizeOf ) WellFoundedRelation.wf) < by
intro v, cs
dsimp only
apply callsOn_app2
apply callsOn_const
apply callsOn_map
intro t' ht'
apply callsOn_base
-- ht' : t' cs -- !
-- sizeOf t' < sizeOf val := v, cs := cs
decreasing_trivial
end examples
This makes me happy!
All details of the construction are now contained in a proof that can proceed by a syntax-driven tactic and that s easily and (likely robustly) extensible by the user. It also means that we can share a lot of code paths (e.g. everything related to equational theorems) between well-founded recursion and partial_fixpoint.
I wonder if this construction is really as powerful as our current one, or if there are certain (likely dependently typed) functions where this doesn t fit, but the above is dependent, so it looks good.
With this construction, functions defined by well-founded recursion will reduce even worse in the kernel, I assume. This may be a good thing.
The cake is a lie
What unfortunately kills this idea, though, is the generation of the functional induction principles, which I believe is not (easily) possible with this construction: The functional induction principle is proved by massaging F to return a proof, but since the extra assumptions (e.g. for ite or List.map) only exist in the termination proof, they are not available in F.
Oh wey, how anticlimactic.
PS: Path dependencies
Curiously, if we didn t have functional induction at this point yet, then very likely I d change Lean to use this construction, and then we d either not get functional induction, or it would be implemented very differently, maybe a more syntactic approach that would re-prove termination. I guess that s called path dependence.
Are you a student aspiring to participate in the 
I am told, by friends who have spent time at Google, about the reason 
As promised on my previous 
KDE Mascot
A few months ago