Search Results: "zed"

4 January 2026

Matthew Garrett: What is a PC compatible?

Wikipedia says An IBM PC compatible is any personal computer that is hardware- and software-compatible with the IBM Personal Computer (IBM PC) and its subsequent models . But what does this actually mean? The obvious literal interpretation is for a device to be PC compatible, all software originally written for the IBM 5150 must run on it. Is this a reasonable definition? Is it one that any modern hardware can meet? Before we dig into that, let s go back to the early days of the x86 industry. IBM had launched the PC built almost entirely around off-the-shelf Intel components, and shipped full schematics in the IBM PC Technical Reference Manual. Anyone could buy the same parts from Intel and build a compatible board. They d still need an operating system, but Microsoft was happy to sell MS-DOS to anyone who d turn up with money. The only thing stopping people from cloning the entire board was the BIOS, the component that sat between the raw hardware and much of the software running on it. The concept of a BIOS originated in CP/M, an operating system originally written in the 70s for systems based on the Intel 8080. At that point in time there was no meaningful standardisation - systems might use the same CPU but otherwise have entirely different hardware, and any software that made assumptions about the underlying hardware wouldn t run elsewhere. CP/M s BIOS was effectively an abstraction layer, a set of code that could be modified to suit the specific underlying hardware without needing to modify the rest of the OS. As long as applications only called BIOS functions, they didn t need to care about the underlying hardware and would run on all systems that had a working CP/M port. By 1979, boards based on the 8086, Intel s successor to the 8080, were hitting the market. The 8086 wasn t machine code compatible with the 8080, but 8080 assembly code could be assembled to 8086 instructions to simplify porting old code. Despite this, the 8086 version of CP/M was taking some time to appear, and a company called Seattle Computer Products started producing a new OS closely modelled on CP/M and using the same BIOS abstraction layer concept. When IBM started looking for an OS for their upcoming 8088 (an 8086 with an 8-bit data bus rather than a 16-bit one) based PC, a complicated chain of events resulted in Microsoft paying a one-off fee to Seattle Computer Products, porting their OS to IBM s hardware, and the rest is history. But one key part of this was that despite what was now MS-DOS existing only to support IBM s hardware, the BIOS abstraction remained, and the BIOS was owned by the hardware vendor - in this case, IBM. One key difference, though, was that while CP/M systems typically included the BIOS on boot media, IBM integrated it into ROM. This meant that MS-DOS floppies didn t include all the code needed to run on a PC - you needed IBM s BIOS. To begin with this wasn t obviously a problem in the US market since, in a way that seems extremely odd from where we are now in history, it wasn t clear that machine code was actually copyrightable. In 1982 Williams v. Artic determined that it could be even if fixed in ROM - this ended up having broader industry impact in Apple v. Franklin and it became clear that clone machines making use of the original vendor s ROM code wasn t going to fly. Anyone wanting to make hardware compatible with the PC was going to have to find another way. And here s where things diverge somewhat. Compaq famously performed clean-room reverse engineering of the IBM BIOS to produce a functionally equivalent implementation without violating copyright. Other vendors, well, were less fastidious - they came up with BIOS implementations that either implemented a subset of IBM s functionality, or didn t implement all the same behavioural quirks, and compatibility was restricted. In this era several vendors shipped customised versions of MS-DOS that supported different hardware (which you d think wouldn t be necessary given that s what the BIOS was for, but still), and the set of PC software that would run on their hardware varied wildly. This was the era where vendors even shipped systems based on the Intel 80186, an improved 8086 that was both faster than the 8086 at the same clock speed and was also available at higher clock speeds. Clone vendors saw an opportunity to ship hardware that outperformed the PC, and some of them went for it. You d think that IBM would have immediately jumped on this as well, but no - the 80186 integrated many components that were separate chips on 8086 (and 8088) based platforms, but crucially didn t maintain compatibility. As long as everything went via the BIOS this shouldn t have mattered, but there were many cases where going via the BIOS introduced performance overhead or simply didn t offer the functionality that people wanted, and since this was the era of single-user operating systems with no memory protection, there was nothing stopping developers from just hitting the hardware directly to get what they wanted. Changing the underlying hardware would break them. And that s what happened. IBM was the biggest player, so people targeted IBM s platform. When BIOS interfaces weren t sufficient they hit the hardware directly - and even if they weren t doing that, they d end up depending on behavioural quirks of IBM s BIOS implementation. The market for DOS-compatible but not PC-compatible mostly vanished, although there were notable exceptions - in Japan the PC-98 platform achieved significant success, largely as a result of the Japanese market being pretty distinct from the rest of the world at that point in time, but also because it actually handled Japanese at a point where the PC platform was basically restricted to ASCII or minor variants thereof. So, things remained fairly stable for some time. Underlying hardware changed - the 80286 introduced the ability to access more than a megabyte of address space and would promptly have broken a bunch of things except IBM came up with an utterly terrifying hack that bit me back in 2009, and which ended up sufficiently codified into Intel design that it was one mechanism for breaking the original XBox security. The first 286 PC even introduced a new keyboard controller that supported better keyboards but which remained backwards compatible with the original PC to avoid breaking software. Even when IBM launched the PS/2, the first significant rearchitecture of the PC platform with a brand new expansion bus and associated patents to prevent people cloning it without paying off IBM, they made sure that all the hardware was backwards compatible. For decades, PC compatibility meant not only supporting the officially supported interfaces, it meant supporting the underlying hardware. This is what made it possible to ship install media that was expected to work on any PC, even if you d need some additional media for hardware-specific drivers. It s something that still distinguishes the PC market from the ARM desktop market. But it s not as true as it used to be, and it s interesting to think about whether it ever was as true as people thought. Let s take an extreme case. If I buy a modern laptop, can I run 1981-era DOS on it? The answer is clearly no. First, modern systems largely don t implement the legacy BIOS. The entire abstraction layer that DOS relies on isn t there, having been replaced with UEFI. When UEFI first appeared it generally shipped with a Compatibility Services Module, a layer that would translate BIOS interrupts into UEFI calls, allowing vendors to ship hardware with more modern firmware and drivers without having to duplicate them to support older operating systems¹. Is this system PC compatible? By the strictest of definitions, no. Ok. But the hardware is broadly the same, right? There s projects like CSMWrap that allow a CSM to be implemented on top of stock UEFI, so everything that hits BIOS should work just fine. And well yes, assuming they implement the BIOS interfaces fully, anything using the BIOS interfaces will be happy. But what about stuff that doesn t? Old software is going to expect that my Sound Blaster is going to be on a limited set of IRQs and is going to assume that it s going to be able to install its own interrupt handler and ACK those on the interrupt controller itself and that s really not going to work when you have a PCI card that s been mapped onto some APIC vector, and also if your keyboard is attached via USB or SPI then reading it via the CSM will work (because it s calling into UEFI to get the actual data) but trying to read the keyboard controller directly won t², so you re still actually relying on the firmware to do the right thing but it s not, because the average person who wants to run DOS on a modern computer owns three fursuits and some knee length socks and while you are important and vital and I love you all you re not enough to actually convince a transglobal megacorp to flip the bit in the chipset that makes all this old stuff work. But imagine you are, or imagine you re the sort of person who (like me) thinks writing their own firmware for their weird Chinese Thinkpad knockoff motherboard is a good and sensible use of their time - can you make this work fully? Haha no of course not. Yes, you can probably make sure that the PCI Sound Blaster that s plugged into a Thunderbolt dock has interrupt routing to something that is absolutely no longer an 8259 but is pretending to be so you can just handle IRQ 5 yourself, and you can probably still even write some SMM code that will make your keyboard work, but what about the corner cases? What if you re trying to run something built with IBM Pascal 1.0? There s a risk that it ll assume that trying to access an address just over 1MB will give it the data stored just above 0, and now it ll break. It d work fine on an actual PC, and it won t work here, so are we PC compatible? That s a very interesting abstract question and I m going to entirely ignore it. Let s talk about PC graphics³. The original PC shipped with two different optional graphics cards - the Monochrome Display Adapter and the Color Graphics Adapter. If you wanted to run games you were doing it on CGA, because MDA had no mechanism to address individual pixels so you could only render full characters. So, even on the original PC, there was software that would run on some hardware but not on other hardware. Things got worse from there. CGA was, to put it mildly, shit. Even IBM knew this - in 1984 they launched the PCjr, intended to make the PC platform more attractive to home users. As well as maybe the worst keyboard ever to be associated with the IBM brand, IBM added some new video modes that allowed displaying more than 4 colours on screen at once⁴, and software that depended on that wouldn t display correctly on an original PC. Of course, because the PCjr was a complete commercial failure, it wouldn t display correctly on any future PCs either. This is going to become a theme. There s never been a properly specified PC graphics platform. BIOS support for advanced graphics modes⁵ ended up specified by VESA rather than IBM, and even then getting good performance involved hitting hardware directly. It wasn t until Microsoft specced DirectX that anything was broadly usable even if you limited yourself to Microsoft platforms, and this was an OS-level API rather than a hardware one. If you stick to BIOS interfaces then CGA-era code will work fine on graphics hardware produced up until the 20-teens, but if you were trying to hit CGA hardware registers directly then you re going to have a bad time. This isn t even a new thing - even if we restrict ourselves to the authentic IBM PC range (and ignore the PCjr), by the time we get to the Enhanced Graphics Adapter we re not entirely CGA compatible. Is an IBM PC/AT with EGA PC compatible? You d likely say yes , but there s software written for the original PC that won t work there. And, well, let s go even more basic. The original PC had a well defined CPU frequency and a well defined CPU that would take a well defined number of cycles to execute any given instruction. People could write software that depended on that. When CPUs got faster, some software broke. This resulted in systems with a Turbo Button - a button that would drop the clock rate to something approximating the original PC so stuff would stop breaking. It s fine, we d later end up with Windows crashing on fast machines because hardware details will absolutely bleed through. So, what s a PC compatible? No modern PC will run the DOS that the original PC ran. If you try hard enough you can get it into a state where it ll run most old software, as long as it doesn t have assumptions about memory segmentation or your CPU or want to talk to your GPU directly. And even then it ll potentially be unusable or crash because time is hard. The truth is that there s no way we can technically describe a PC Compatible now - or, honestly, ever. If you sent a modern PC back to 1981 the media would be amazed and also point out that it didn t run Flight Simulator. PC Compatible is a socially defined construct, just like Woman . We can get hung up on the details or we can just chill.

Windows 7 is entirely happy to boot on UEFI systems except that it relies on being able to use a BIOS call to set the video mode during boot, which has resulted in things like UEFISeven to make that work on modern systems that don t provide BIOS compatibility
Back in the 90s and early 2000s operating systems didn t necessarily have native drivers for USB input devices, so there was hardware support for trapping OS accesses to the keyboard controller and redirecting that into System Management Mode where some software that was invisible to the OS would speak to the USB controller and then fake a response anyway that s how I made a laptop that could boot unmodified MacOS X
(my name will not be Wolfwings Shadowflight)
Yes yes ok 8088 MPH demonstrates that if you really want to you can do better than that on CGA
and by advanced we re still talking about the 90s, don t get excited

Valhalla's Things: And now for something completely different

Warning

mention of bodies being bodies and minds being minds, and not in the perfectly working sense.

One side of Porto Ceresio along the lake: there is a small strip of houses, and the hills behind them are covered in woods. A boat is parked around the middle of the picture.

A lot of the youtube channels I follow tend to involve somebody making things, so of course one of the videos my SO and I watched a few days ago was about walking around San Francisco Bay, and that recalled my desire to go to places by foot. Now, for health-related reasons doing it properly would be problematic, and thus I ve never trained for that, but during this Christmas holiday-heavy time I suggested my very patient SO the next best thing: instead of our usual 1.5 hours uphill walk in the woods, a 2 hours and a bit mostly flat walk on paved streets, plus some train, to a nearby town: Porto Ceresio, on the Italian side of Lake Lugano. I started to prepare for it on the day before, by deciding it was a good time to upgrade my PinePhone, and wait, I m still on Trixie? I could try Forky, what could possibly go wrong? And well, the phone was no longer able to boot, and reinstalling from the latest weekly had a system where the on-screen keyboard didn t appear, and I didn t want to bother finding out why, so re-installed another time from the 13.0 image, and between that, and distracting myself with widelands while waiting for the downloads and uploads and reboots etc., well, all of the afternoon and the best part of the evening disappeared. So, in a hurry, between the evening and the next morning I prepared a nice healthy lunch, full of all the important nutrients such as sugar, salt, mercury and arsenic. Tuna (mercury) soboro (sugar and salt) on rice and since I was in a hurry I didn t prepare any vegetables, but used pickles (more salt) and shio kombu (arsenic and various heavy metals, sugar and salt). Plus a green tea mochi for dessert, in case we felt low on sugar. :D Then on the day of the walk we woke up a bit later than usual, and then my body decided it was a good day for my belly to not exactly hurt, but not not-hurt either, and there I took an executive decision to wear a corset, because if something feels like it wants to burst open, wrapping it in a steel reinforced cage will make it stop. (I m not joking. It does. At least in those specific circumstances.) This was followed by hurrying through the things I had to do before leaving the house, having a brief anxiety attack and feeling feverish (it wasn t fever), and finally being able to leave the house just half an hour late. A stream running on rocks with the woods to both sides.

A stream running on rocks with the woods to both sides.

And then, 10 minutes after we had left, realizing that I had written down the password for the train website, since it was no longer saved on the phone, but i had forgotten the bit of paper at home. We could have gone back to take it, but decided not to bother, as we could also hopefully buy paper-ish tickets at the train station (we could). Later on, I also realized I had also forgotten my GPS tracker, so I have no record of where we went exactly (but it s not hard to recognize it on a map) nor on what the temperature was. It s a shame, but by that point it was way too late to go back. Anyway, that probably was when Murphy felt we had paid our respects, and from then on everything went lovingly well! Routing had been done on the OpenStreetMap website, with OSRM, and it looked pretty easy to follow, but we also had access to an Android phone, so we used OSMAnd to check that we were still on track. It tried to lead us to the Statale (i.e. most important and most trafficked road) a few times, but we ignored it, and after a few turns and a few changes of the precise destination point we managed to get it to cooperate. At one point a helpful person asked us if we needed help, having seen us looking at the phone, and gave us indication for the next fork (that way to Cuasso al Piano, that way to Porto Ceresio), but it was pretty easy, since the way was clearly marked also for cars. Then we started to notice red and white markings on poles and other places, and on the next fork there was a signpost for hiking routes with our destination and we decided to follow it instead of the sign for cars. I knew that from our starting point to or destination there was also a hiking route, uphill both ways :) , through the hills, about 5 or 6 hours instead of two, but the sign was pointing downhill and we were past the point where we would expect too long of a detour. A wide and flat unpaved track passing through a flat grassy area with trees to the sides and rolling hills in the background.

A wide and flat unpaved track passing through a flat grassy area with trees to the sides and rolling hills in the background.

And indeed, after a short while the paved road ended, but the path continued on a wide and flat track, and was a welcome detour through what looked like water works to prevent flood damage from a stream. In a warmer season, with longer grass and ticks maybe the fact that I was wearing a long skirt may have been an issue, but in winter it was just fine. And soon afterwards, we were in Porto Ceresio. I think I have been there as a child, but I had no memory of it. On the other hand, it was about as I expected: a tiny town with a lakeside street full of houses built in the early 1900s when the area was an important tourism destination, with older buildings a bit higher up on the hills (because streams in this area will flood). And of course, getting there by foot rather than by train we also saw the parts where real people live (but not work: that s cross-border commuters country). Dried winter grass with two strips of frost, exactly under the shade of a fence.

Dried winter grass with two strips of frost, exactly under the shade of a fence.

Soon after arriving in Porto Ceresio we stopped to eat our lunch on a bench at the lakeside; up to then we had been pretty comfortable in the clothing we had decided to wear: there was plenty of frost on the ground, in the shade, but the sun was warm and the temperatures were cleanly above freezing. Removing the gloves to eat, however, resulted in quite cold hands, and we didn t want to stay still for longer than strictly necessary. So we spent another hour and a bit walking around Porto Ceresio like proper tourists and taking pictures. There was an exhibition of nativity scenes all around the streets, but to get a map one had to go to either facebook or instagram, or wait for the opening hours of an office that were later than the train we planned to get to go back home, so we only saw maybe half of them, as we walked around: some were quite nice, some were nativity scenes, and some showed that the school children must have had some fun making them. three gnome adjacent creatures made of branches of evergreen trees, with a pointy hat made of moss, big white moustaches and red Christmas tree balls at the end of the hat and as a nose. They are wrapped in LED strings, and the lake can be seen in the background.

three gnome adjacent creatures made of branches of evergreen trees, with a pointy hat made of moss, big white moustaches and red Christmas tree balls at the end of the hat and as a nose. They are wrapped in LED strings, and the lake can be seen in the background.

Another Christmas decoration were groups of creatures made of evergreen branches that dotted the sidewalks around the lake: I took pictures of the first couple of groups, and then after seeing a few more something clicked in my brain, and I noticed that they were wrapped in green LED strings, like chains, and they had a red ball that was supposed to be the nose, but could just be around the mouth area, and suddenly I felt the need to play a certain chord to release them, but sadly I didn t have a weaponized guitar on me :D A bench in the shape of an open book, half of the pages folded in a reversed U to make the seat and half of the pages standing straight to form the backrest. It has the title page and beginning of the Constitution of the Italian Republic.

A bench in the shape of an open book, half of the pages folded in a reversed U to make the seat and half of the pages standing straight to form the backrest. It has the title page and beginning of the Constitution of the Italian Republic.

Another thing that we noticed were some benches in the shape of books, with book quotations on them; most were on reading-related topics, but the one with the Constitution felt worth taking a picture of, especially these days. And then, our train was waiting at the station, and we had to go back home for the afternoon; it was a nice outing, if a bit brief, and we agreed to do it again, possibly with a bit of a detour to make the walk a bit longer. And then maybe one day we ll train to do the whole 5-6 hour thing through the hills.

3 January 2026

Benjamin Mako Hill: Effects of Algorithmic Flagging on Fairness: Quasi-experimental Evidence from Wikipedia

Note: I have not published blog posts about my academic papers over the past few years. To ensure that my blog contains a more comprehensive record of my published papers and to surface these for folks who missed them, I will be periodically (re)publishing blog posts about some older published projects. This particular post is closely based on a previously published post by Nate TeBlunthuis from the Community Data Science Blog. Many online platforms are adopting AI and machine learning as a tool to maintain order and high-quality information in the face of massive influxes of user-generated content. Of course, AI algorithms can be inaccurate, biased, or unfair. How do signals from AI predictions shape the fairness of online content moderation? How can we measure an algorithmic flagging system s effects? In our paper published at CSCW, Nate TeBlunthuis, together with myself and Aaron Halfaker, analyzed the RCFilters system: an add-on to Wikipedia that highlights and filters edits that a machine learning algorithm called ORES identifies as likely to be damaging to Wikipedia. This system has been deployed on large Wikipedia language editions and is similar to other algorithmic flagging systems that are becoming increasingly widespread. Our work measures the causal effect of being flagged in the RCFilters user interface.

Our work takes advantage of the fact that RCFilters, like many algorithmic flagging systems, create discontinuities in the relationship between the probability that a moderator should take action and whether a moderator actually does. This happens because the output of machine learning systems like ORES is typically a continuous score (in RCFilters, an estimated probability that a Wikipedia edit is damaging), while the flags (in RCFilters, the yellow, orange, or red highlights) are either on or off and are triggered when the score crosses some arbitrary threshold. As a result, edits slightly above the threshold are both more visible to moderators and appear more likely to be damaging than edits slightly below. Even though edits on either side of the threshold have virtually the same likelihood of truly being damaging, the flagged edits are substantially more likely to be reverted. This fact lets us use a method called regression discontinuity to make causal estimates of the effect of being flagged in RCFilters.

To understand how this system may affect the fairness of Wikipedia moderation, we estimate the effects of flagging on edits on different groups of editors. Comparing the magnitude of these estimates lets us measure how flagging is associated with several different definitions of fairness. Surprisingly, we found evidence that these flags improved fairness for categories of editors that have been widely perceived as troublesome particularly unregistered (anonymous) editors. This occurred because flagging has a much stronger effect on edits by the registered than on edits by the unregistered. We believe that our results are driven by the fact that algorithmic flags are especially helpful for finding damage that can t be easily detected otherwise. Wikipedia moderators can see the editor s registration status in the recent changes, watchlists, and edit history. Because unregistered editors are often troublesome, Wikipedia moderators attention is often focused on their contributions, with or without algorithmic flags. Algorithmic flags make damage by registered editors (in addition to unregistered editors) much more detectable to moderators and so help moderators focus on damage overall, not just damage by suspicious editors. As a result, the algorithmic flagging system decreases the bias that moderators have against unregistered editors. This finding is particularly surprising because the ORES algorithm we analyzed was itself demonstrably biased against unregistered editors (i.e., the algorithm tended to greatly overestimate the probability that edits by these editors were damaging). Despite the fact that the algorithms were biased, their introduction could still lead to less biased outcomes overall. Our work shows that although it is important to design predictive algorithms to avoid such biases, it is equally important to study fairness at the level of the broader sociotechnical system. Since we first published a preprint of our paper, a follow-up piece by Leijie Wang and Haiyi Zhu replicated much of our work and showed that differences between different Wikipedia communities may be another important factor driving the effect of the system. Overall, this work suggests that social signals and social context can interact with algorithmic signals, and together these can influence behavior in important and unexpected ways.

The full citation for the paper is: TeBlunthuis, Nathan, Benjamin Mako Hill, and Aaron Halfaker. 2021. Effects of Algorithmic Flagging on Fairness: Quasi-Experimental Evidence from Wikipedia. Proceedings of the ACM on Human-Computer Interaction 5 (CSCW): 56:1-56:27. https://doi.org/10.1145/3449130.

We have also released replication materials for the paper, including all the data and code used to conduct the analysis and compile the paper itself.

Russ Allbery: Review: Challenges of the Deeps

Review: Challenges of the Deeps, by Ryk E. Spoor

Series:	Arenaverse #3
Publisher:	Baen
Copyright:	March 2017
ISBN:	1-62579-564-5
Format:	Kindle
Pages:	438

Challenges of the Deeps is the third book in the throwback space opera Arenaverse series. It is a direct sequel to Spheres of Influence, but Spoor provides a substantial recap of the previous volumes for those who did not read the series in close succession (thank you!). Ariane has stabilized humanity's position in the Arena with yet another improbable victory. (If this is a spoiler for previous volumes, so was telling you the genre of the book.) Now is a good opportunity to fulfill the promise humanity made to their ally Orphan: accompaniment on a journey into the uncharted deeps of the Arena for reasons that Orphan refuses to explain in advance. Her experienced crew provide multiple options to serve as acting Leader of Humanity until she gets back. What can go wrong? The conceit of this series is that as soon as a species achieves warp drive technology, their ships are instead transported into the vast extradimensional structure of the Arena where a godlike entity controls the laws of nature and enforces a formal conflict resolution process that looks alternatingly like a sporting event, a dueling code, and technology-capped total war. Each inhabitable system in the real universe seems to correspond to an Arena sphere, but the space between them is breathable atmosphere filled with often-massive storms. In other words, this is an airship adventure as written by E.E. "Doc" Smith. Sort of. There is an adventure, and there are a lot of airships (although they fight mostly like spaceships), but much of the action involves tense mental and physical sparring with a previously unknown Arena power with unclear motives. My general experience with this series is that I find the Arena concept fascinating and want to read more about it, Spoor finds his much-less-original Hyperion Project in the backstory of the characters more fascinating and wants to write about that, and we reach a sort of indirect, grumbling (on my part) truce where I eagerly wait for more revelations about the Arena and roll my eyes at the Hyperion stuff. Talking about Hyperion in detail is probably a spoiler for at least the first book, but I will say that it's an excuse to embed versions of literary characters into the story and works about as well as most such excuses (not very). The characters in question are an E.E. "Doc" Smith mash-up, a Monkey King mash-up, and a number of other characters that are obviously references to something but for whom I lack enough hints to place (which is frustrating). Thankfully we get far less human politics and a decent amount of Arena world-building in this installment. Hyperion plays a role, but mostly as foreshadowing for the next volume and the cause of a surprising interaction with Arena rules. One of the interesting wrinkles of this series is that humanity have an odd edge against the other civilizations in part because we're borderline insane sociopaths from the perspective of the established powers. That's an old science fiction trope, but I prefer it to the Campbell-style belief in inherent human superiority. Old science fiction tropes are what you need to be in the mood for to enjoy this series. This is an unapologetic and intentional throwback to early pulp: individuals who can be trusted with the entire future of humanity because they're just that moral, super-science, psychic warfare, and even coruscating beams that would make E.E. "Doc" Smith proud. It's an occasionally glorious but mostly silly pile of technobabble, but Spoor takes advantage of the weird, constructed nature of the Arena to provide more complex rules than competitive superlatives. The trick is that while this is certainly science fiction pulp, it's also a sort of isekai novel. There's a lot of anime and manga influence just beneath the surface. I'm not sure why it never occurred to me before reading this series that melodramatic anime and old SF pulps have substantial aesthetic overlap, but of course they do. I loved the Star Blazers translated anime that I watched as a kid precisely because it had the sort of dramatic set pieces that make the Lensman novels so much fun. There is a bit too much Wu Kong in this book for me (although the character is growing on me a little), and some of the maneuvering around the mysterious new Arena actor drags on longer than was ideal, but the climax is great stuff if you're in the mood for dramatic pulp adventure. The politics do not bear close examination and the writing is serviceable at best, but something about this series is just fun. I liked this book much better than Spheres of Influence, although I wish Spoor would stop being so coy about the nature of the Arena and give us more substantial revelations. I'm also now tempted to re-read Lensman, which is probably a horrible idea. (Spoor leaves the sexism out of his modern pulp.) If you got through Spheres of Influence with your curiosity about the Arena intact, consider this one when you're in the mood for modern pulp, although don't expect any huge revelations. It's not the best-written book, but it sits squarely in the center of a genre and mood that's otherwise a bit hard to find. Followed by the Kickstarter-funded Shadows of Hyperion, which sadly looks like it's going to concentrate on the Hyperion Project again. I will probably pick that up... eventually. Rating: 6 out of 10

1 January 2026

Russ Allbery: Review: This Brutal Moon

Review: This Brutal Moon, by Bethany Jacobs

Series:	Kindom Trilogy #3
Publisher:	Orbit
Copyright:	December 2025
ISBN:	0-316-46373-6
Format:	Kindle
Pages:	497

This Brutal Moon is a science fiction thriller with bits of cyberpunk and space opera. It concludes the trilogy begun with These Burning Stars. The three books tell one story in three volumes, and ideally you would read all three in close succession. There is a massive twist in the first book that I am still not trying to spoil, so please forgive some vague description. At the conclusion of These Burning Stars, Jacobs had moved a lot of pieces into position, but it was not yet clear to me where the plot was going, or even if it would come to a solid ending in three volumes as promised by the series title. It does. This Brutal Moon opens with some of the political maneuvering that characterized These Burning Stars, but once things start happening, the reader gets all of the action they could wish for and then some. I am pleased to report that, at least as far as I'm concerned, Jacobs nails the ending. Not only is it deeply satisfying, the characterization in this book is so good, and adds so smoothly to the characterization of the previous books, that I saw the whole series in a new light. I thought this was one of the best science fiction series finales I've ever read. Take that with a grain of salt, since some of those reasons are specific to me and the mood I was in when I read it, but this is fantastic stuff. There is a lot of action at the climax of this book, split across at least four vantage points and linked in a grand strategy with chaotic surprises. I kept all of the pieces straight and understood how they were linked thanks to Jacobs's clear narration, which is impressive given the number of pieces in motion. That's not the heart of this book, though. The action climax is payoff for the readers who want to see some ass-kicking, and it does contain some moving and memorable moments, but it relies on some questionable villain behavior and a convenient plot device introduced only in this volume. The action-thriller payoff is competent but not, I think, outstanding. What put this book into a category of its own were the characters, and specifically how Jacobs assembles sweeping political consequences from characters who, each alone, would never have brought about such a thing, and in some cases had little desire for it. Looking back on the trilogy, I think Jacobs has captured, among all of the violence and action-movie combat and space-opera politics, the understanding that political upheaval is a relay race. The people who have the personalities to start it don't have the personality required to nurture it or supply it, and those who can end it are yet again different. This series is a fascinating catalog of political actors the instigator, the idealist, the pragmatist, the soldier, the one who supports her friends, and several varieties and intensities of leaders and it respects all of them without anointing any of them as the One True Revolutionary. The characters are larger than life, yes, and this series isn't going to win awards for gritty realism, but it's saying something satisfyingly complex about where we find courage and how a cause is pushed forward by different people with different skills and emotions at different points in time. Sometimes accidentally, and often in entirely unexpected ways. As before, the main story is interwoven with flashbacks. This time, we finally see the full story of the destruction of the moon of Jeve. The reader has known about this since the first volume, but Jacobs has a few more secrets to show (including, I will admit, setting up a plot device) and some pointed commentary on resource extraction economies. I think this part of the book was a bit obviously constructed, although the characterization was great and the visible junction points of the plot didn't stop me from enjoying the thrill when the pieces came together. But the best part of this book was the fact there was 10% of it left after the climax. Jacobs wrote an actual denouement, and it was everything I wanted and then some. We get proper story conclusions for each of the characters, several powerful emotional gut punches, some remarkably subtle and thoughtful discussion of political construction for a series that tended more towards space-opera action, and a conclusion for the primary series relationship that may not be to every reader's taste but was utterly, perfectly, beautifully correct for mine. I spent a whole lot of the last fifty pages of this book trying not to cry, in the best way. The character evolution over the course of this series is simply superb. Each character ages like fine wine, developing more depth, more nuance, but without merging. They become more themselves, which is an impressive feat across at least four very different major characters. You can see the vulnerabilities and know what put them there, you can see the strengths they developed to compensate, and you can see why they need the support the other characters provide. And each of them is so delightfully different. This was so good. This was so precisely the type of story that I was in the mood for, with just the type of tenderness for its characters that I wanted, that I am certain I am not objective about it. It will be one of those books where other people will complain about flaws that I didn't see or didn't care about because it was doing the things I wanted from it so perfectly. It's so good that it elevated the entire trilogy; the journey was so worth the ending. I'm afraid this review will be less than helpful because it's mostly nonspecific raving. This series is such a spoiler minefield that I'd need a full spoiler review to be specific, but my reaction is so driven by emotion that I'm not sure that would help if the characters didn't strike you the way that they struck me. I think the best advice I can offer is to say that if you liked the emotional tone of the end of These Burning Stars (not the big plot twist, the character reaction to the political goal that you learn drove the plot), stick with the series, because that's a sign of the questions Jacobs is asking. If you didn't like the characters at the end (not the middle) of the first novel, bail out, because you're going to get a lot more of that. Highly, highly recommended, and the best thing I've read all year, with the caveats that you should read the content notes, and that some people are going to bounce off this series because it's too intense and melodramatic. That intensity will not let up, so if that's not what you're in the mood for, wait on this trilogy until you are. Content notes: Graphic violence, torture, mentions of off-screen child sexual assault, a graphic corpse, and a whole lot of trauma. One somewhat grumbly postscript: This is the sort of book where I need to not read other people's reviews because I'll get too defensive of it (it's just a book I liked!). But there is one bit of review commentary I've seen about the trilogy that annoys me enough I have to mention it. Other reviewers seem to be latching on to the Jeveni (an ethnic group in the trilogy) as Space Jews and then having various feelings about that. I can see some parallels, I'm not going to say that it's completely wrong, but I also beg people to read about a fictional oppressed ethnic and religious minority and not immediately think "oh, they must be stand-ins for Jews." That's kind of weird? And people from the US, in particular, perhaps should not read a story about an ethnic group enslaved due to their productive skill and economic value and think "they must be analogous to Jews, there are no other possible parallels here." There are a lot of other comparisons that can be made, including to the commonalities between the methods many different oppressed minorities have used to survive and preserve their culture. Rating: 10 out of 10

Daniel Baumann: P r ch storage (in Western Europe)

The majority of P r ch ( ) is stored in Asia (predominantly China, some in Malaysia and Taiwan) because of its climatic conditions:

warm (~30 C)

humid (~75% RH)

stable (comparatively small day to day, day to night, and seasonal differences)

These are ideal for ageing and allow efficient storage in simple storehouses, usually without the need for any air conditioning. The climate in Western European countries is significantly colder, drier, and more variable. However, residential houses offers throughout the year a warm (~20 C), humid (~50% RH), and stable baseline to start with. High quality, long term storage and ageing is possible too with some of the Asian procedures slightly adjusted for the local conditions. Nevertheless, fast accelerated ageing still doesn t work here (even with massive climate control). Personally I prefer the balanced, natural storage over the classical wet or dry , or the mixed traditional storage (of course all storage types are not that meaningful as they are all relative terms anyway). Also I don t like strong fermentation either (P r is not my favourite tea, I only drink it in the 3 months of winter). Therefore, my intention is primarily to preserve the tea while it continues to age normally, keep it in optimal drinking condition and don t rush its fermentation. The value of correct storage is of great importance and has a big effect on P r, and is often overlooked and underrated. Here s a short summary on how to store P r ch (in Western Europe).

1. Location

1.1 No light Choose a location with neither direct nor indirect sunlight (= ultraviolet radiation) exposure:

direct sunlight damages organic material ( bleeching )

indirect sunlight by heating up the containers inbalances the microclimate ( suppression )

1.2 No odors Choose a location with neither direct nor indirect odor exposure:

direct odor immissions (like incense, food, air polution, etc.) makes the tea undrinkable

indirect odor immissions (especially other teas stored next to it) taint the taste and dilute the flavours

Always use individual containers for each tea, store only identical tea of the same year and batch in the same container. Idealy never store one cake, brick or tuo on its own, buy multiples and store them together for better ageing.

2. Climate

2.1 Consistent temperature Use a location with no interference from any devices or structures next to it:

use an regular indoor location for mild temperature curves (i.e. no attics with large day/night temperature differences)

aim for >= 20 C average temperature for natural storage (i.e. no cold basements)

don t use a place next to any heat dissipating devices (like radiators, computers, etc.)

don t use a place next to an outside facing wall

always leave 5 to 10cm distance to a wall for air to circulate (generally prevents mold on the wall but also isolates the containers further from possible immissions)

As consistent temperature as possible allows even and steady fermentation. However, neither air conditioning or filtering is needed. Regular day to day, day to night, and season to season fluctuations are balanced out fine with otherwise correct indoor storage. Also humidity control is much more important for storage and ageing, and much less forgiving than temperature control.

2.2 Consistent humidity Use humidity control packs to ensure a stable humidity level:

aim for ~65% RH

Lower than 60% RH will (completely) dry out the tea, higher than 70% RH increases chances for mold.

3. Equipment

3.1 Proper containers Use containers that are long term available and that are easily stackable, both in form and dimensions as well as load-bearing capacity. They should be inexpensive, otherwise it s most likely scam (there are people selling snake-oil containers specifically for tea storage). For long term storage and ageing:

never use plastic: they leak chemicals over time (no tupperdors , mylar and zip lock bags, etc.)

never use cardboard or bamboo: they let to much air in, absorb too much humity and dry to slow

never use wood: they emit odors (no humidors)

never use clay: they absorb all humidity and dry out the tea (glazed or porcelain urns are acceptable for short term storage)

Always use sealed but not air tight, cleaned steal cans (i.e. with no oil residues from manufacturing; such as these).

3.2 Proper humidity control Use two-way humidity control packs to either absorb or add moisture depending on the needs:

use humidity control pack(s) in every container

use one 60g pack (such as these) per 500g of tea

3.3 Proper labels Put labels on the outside of the containers to not need to open them to know what is inside and disturbe the ageing:

write at least manufacturer and name, pressing year, and batch number on the label (such as these)

if desired and not otherwise kept track already elsewhere, add additional information such as the amount of items, weights, previous storage location/type, date of purchase, vendor, or price, etc.

3.4 Proper monitoring Measuring and checking temperature and humidity regularly prevents storage parameters turning bad going unnoticed:

put temperature and humidity measurement sensors (such as these) inside some of the containers (e.g. at least one in every different size/type of container)

keep at least one temperature and humidity measurement sensor next to the containers on the outside to monitor the storage location

4. Storage

4.1 Continued maintenance Before beginning to initially store a new tea, let it acclimatize for 2h after unpacking from transport (or longer if temperature differences between indoors and outdoors are high, i.e. in winter). Then continuesly:

once a month briefly air all containers for a minute or two

once a month check for mold, just to be safe

every 3 to 6 months check all humidity control packs for need of replacement

monitor battery life of temperature and humidity measurement sensors

Humidity control packs can be recharged (usually voiding any warranty) by submerging them for 2 days in distilled water and 4 hours drying on paper towl afterwards. Check with a weight scale after recharging, they should regain their original weight (e.g. 60g plus ~2 to 5g for the packaging).

Finally With a correct P r ch storage:

beginn to drink Sh ng P r ( ) roughly after about 10-15 years, or later

beginn to drink Sh u P r ( ) roughly after about 3-5 years, or later

Prepare the tea by breaking it into, or breaking off from it, ~5 to 10g pieces about 1 to 3 months ahead of drinking and consider increase humidity to 70% RH during that time. 2026

31 December 2025

Sergio Cipriano: Zero-Code Instrumentation of an Envoy TCP Proxy using eBPF

Zero-Code Instrumentation of an Envoy TCP Proxy using eBPF I recently had to debug an Envoy Network Load Balancer, and the options Envoy provides just weren't enough. We were seeing a small number of HTTP 499 errors caused by latency somewhere in our cloud, but it wasn't clear what the bottleneck was. As a result, each team had to set up additional instrumentation to catch latency spikes and figure out what was going on. My team is responsible for the LBaaS product (Load Balancer as a Service) and, of course, we are the first suspects when this kind of problem appear. Before going for the current solution, I read a lot of Envoy's documentation. It is possible to enable access logs for Envoy, but they don't provide the information required for this debug. This is an example of the output:

[2025-12-08T20:44:49.918Z] "- - -" 0 - 78 223 1 - "-" "-" "-" "-" "172.18.0.2:8080"

I won't go into detail about the line above, since it's not possible to trace the request using access logs alone. Envoy also has OpenTelemetry tracing, which is perfect for understanding sources of latency. Unfortunatly, it is only available for Application Load Balancers. Most of the HTTP 499 were happening every 10 minutes, so we managed to get some of the requests with tcpdump, Wireshark and using http headers to filter the requests. This approach helped us reproduce and track down the problem, but it wasn't a great solution. We clearly needed better tools to catch this kind of issue the next time it happened. Therefore, I decided to try out OpenTelemetry eBPF Instrumentation, also referred to as OBI. I saw the announcement of Grafana Beyla before it was renamed to OBI, but I didn't have the time or a strong reason to try it out until now. Even then, I really liked the idea, and the possibility of using eBPF to solve this instrumentation problem had been in the back of my mind. OBI promises zero-code automatic instrumentation for Linux services using eBPF, so I put together a minimal setup to see how well it works.

Reproducible setup I used the following tools:

Setting up a TCP Proxy with Envoy was straightforward:

static_resources:
  listeners:
  - name: go_server_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: go_server_tcp
          cluster: go_server_cluster
  clusters:
  - name: go_server_cluster
    connect_timeout: 1s
    type: LOGICAL_DNS
    load_assignment:
      cluster_name: go_server_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: target-backend
                port_value: 8080

This is the simplest Envoy TCP proxy configuration: a listener on port 8000 forwarding traffic to a backend running on port 8080. For the backend, I used a basic Go HTTP server:

package main

import (
    "fmt"
    "net/http"
)

func main()  
    http.Handle("/", http.FileServer(http.Dir(".")))

    server := http.Server Addr: ":8080" 

    fmt.Println("Starting server on :8080")
    panic(server.ListenAndServe())

Finally, I wrapped everything together with Docker Compose:

services:
  autoinstrumenter:
    image: otel/ebpf-instrument:main
    pid: "service:envoy"
    privileged: true
    environment:
      OTEL_EBPF_TRACE_PRINTER: text
      OTEL_EBPF_OPEN_PORT: 8000

  envoy:
    image: envoyproxy/envoy:v1.33-latest
    ports:
      - 8000:8000
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml 
    depends_on:
      - target-backend
  
  target-backend:
    image: golang:1.22-alpine
    command: go run /app/backend.go
    volumes:
      - ./backend.go:/app/backend.go:ro
    expose:
      - 8080

OBI should output traces to the standard output similar to this when a HTTP request is made to Envoy:

2025-12-08 20:44:49.12884449 (305.572 s[305.572 s]) HTTPClient 200 GET /(/) [172.18.0.3 as envoy:36832]->[172.18.0.2 as localhost:8080] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-529458a2be271956134872668dc5ee47-6dba451ec8935e3e[06c7f817e6a5dae2]-01]
2025-12-08 20:44:49.12884449 (1.260901ms[366.65 s]) HTTP 200 GET /(/) [172.18.0.1 as 172.18.0.1:36282]->[172.18.0.3 as envoy:8000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-529458a2be271956134872668dc5ee47-06c7f817e6a5dae2[0000000000000000]-01]

This is exactly what we needed, with zero-code. The above trace shows:

2025-12-08 20:44:49.12884449: time of the trace.
(1.260901ms[366.65 s]): total response time for the request, with the actual internal execution time of the request (not counting the request enqueuing time).
HTTP 200 GET /: protocol, response code, HTTP method, and URL path.
[172.18.0.1 as 172.18.0.1:36282]->[172.18.0.3 as envoy:8000]: source and destination host:port. The initial request originates from my machine through the gateway (172.18.0.1), hits the Envoy (172.23.0.3), the proxy then forwards it to the backend application (172.23.0.2).
contentLen:78B: HTTP Content-Length. I used curl and the default request size for it is 78B.
responseLen:223B: Size of the response body.
svc=[envoy generic]: traced service.
traceparent: ids to trace the parent request. We can see that the Envoy makes a request to the target and this request has the other one as parent.

Let's add one more Envoy to show that it's also possible to track multiple services.

  envoy1:
    image: envoyproxy/envoy:v1.33-latest
    ports:
      - 9000:9000
    volumes:
      - ./envoy1.yaml:/etc/envoy/envoy.yaml
    depends_on:
      - envoy

The new Envoy will listen on port 9000 and forward the request to the other Envoy listening on port 8000. Now we just need to change OBI open port variable to look at a range:

OTEL_EBPF_OPEN_PORT: 8000-9000

And change the pid field of the autoinstrumenter service to use the host's PID namespace inside the container:

pid: host

This is the output I got after one curl:

2025-12-09 12:28:05.12912285 (2.202041ms[1.524713ms]) HTTP 200 GET /(/) [172.19.0.1 as 172.19.0.1:59030]->[172.19.0.5 as envoy:9000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-856c9f700e73bf0d[0000000000000000]-01]
2025-12-09 12:28:05.12912285 (1.389336ms[1.389336ms]) HTTPClient 200 GET /(/) [172.19.0.5 as envoy:59806]->[172.19.0.4 as localhost:8000] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-caa7f1ad1c68fa77[856c9f700e73bf0d]-01]
2025-12-09 12:28:05.12912285 (1.5431ms[848.574 s]) HTTP 200 GET /(/) [172.19.0.5 as 172.19.0.5:59806]->[172.19.0.4 as envoy:8000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-cbca9d64d3d26b40[caa7f1ad1c68fa77]-01]
2025-12-09 12:28:05.12912285 (690.217 s[690.217 s]) HTTPClient 200 GET /(/) [172.19.0.4 as envoy:34256]->[172.19.0.3 as localhost:8080] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-5502f7760ed77b5b[cbca9d64d3d26b40]-01]
2025-12-09 12:28:05.12912285 (267.9 s[238.737 s]) HTTP 200 GET /(/) [172.19.0.4 as 172.19.0.4:34256]->[172.19.0.3 as backend:8080] contentLen:0B responseLen:0B svc=[backend go] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-ac05c7ebe26f2530[5502f7760ed77b5b]-01]

Each log line represents a span belonging to the same trace (69977bee0c2964b8fe53cdd16f8a9d19). For readability, I ordered the spans by their traceparent relationship, showing the request's path as it moves through the system: from the client-facing Envoy, through the internal Envoy hop, and finally to the Go backend. You can see both server-side (HTTP) and client-side (HTTPClient) spans at each hop, along with per-span latency, source and destination addresses, and response sizes, making it easy to pinpoint where time is spent along the request chain. The log lines are helpful, but we need better ways to visualize the traces and the metrics generated by OBI. I'll share another setup that more closely reflects what we actually use.

Production setup I'll be using the following tools this time:

The goal of this setup is to mirror an environment similar to what I used in production. This time, I've omitted the load balancer and shifted the emphasis to observability instead. setup diagram

I will run three HTTP servers on port 8080: two inside Incus containers and one on the host machine. The OBI process will export metrics and traces to an OpenTelemetry Collector, which will forward traces to Jaeger and expose a metrics endpoint for Prometheus to scrape. Grafana will also be added to visualize the collected metrics using dashboards. The aim of this approach is to instrument only one of the HTTP servers while ignoring the others. This simulates an environment with hundreds of Incus containers, where the objective is to debug a single container without being overwhelmed by excessive and irrelevant telemetry data from the rest of the system. OBI can filter metrics and traces based on attribute values, but I was not able to filter by process PID. This is where the OBI Collector comes into play, it allows me to use a processor to filter telemetry data by the PID of the process being instrumented. These are the steps to reproduce this setup:

Create the incus containers.

$ incus launch images:debian/trixie server01
Launching server01
$ incus launch images:debian/trixie server02
Launching server02

Start the HTTP server on each container.

$ apt install python3 --update -y
$ tee /etc/systemd/system/server.service > /dev/null <<'EOF'
[Unit]
Description=Python HTTP server
After=network.target
[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/bin/python3 -m http.server 8080
Restart=always
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
EOF
$ systemctl start server.service

Start the HTTP server on the host.

$ python3 -m http.server 8080

Start the Docker compose.

services:
  autoinstrumenter:
    image: otel/ebpf-instrument:main
    pid: host
    privileged: true
    environment:
      OTEL_EBPF_CONFIG_PATH: /etc/obi/obi.yml 
    volumes:
      - ./obi.yml:/etc/obi/obi.yml

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.98.0
    command: ["--config=/etc/otel-collector-config.yml"]
    volumes:
      - ./otel-collector-config.yml:/etc/otel-collector-config.yml
    ports:
      - "4318:4318" # Otel Receiver
      - "8889:8889" # Prometheus Scrape
    depends_on:
      - autoinstrumenter
      - jaeger
      - prometheus

  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090" # Prometheus UI

  grafana:
    image: grafana/grafana
    restart: always
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=RandomString123!
    volumes:
      - ./grafana-ds.yml:/etc/grafana/provisioning/datasources/datasource.yml
    ports:
      - "3000:3000" # Grafana UI

  jaeger:
    image: jaegertracing/all-in-one
    container_name: jaeger
    ports:
      - "16686:16686" # Jaeger UI
      - "4317:4317"  # Jaeger OTLP/gRPC Collector

Here's what the configuration files look like:

obi.yml:

log_level: INFO
trace_printer: text

discovery:
  instrument:
    - open_ports: 8080

otel_metrics_export:
  endpoint: http://otel-collector:4318
otel_traces_export:
  endpoint: http://otel-collector:4318

prometheus.yml:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

grafana-ds.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

otel-collector-config.yml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: otel-collector:4318

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: default

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/jaeger]

    metrics:
      receivers: [otlp] 
      exporters: [prometheus]

We're almost there, the OpenTelemetry Collector is just missing a processor. To create the processor filter, we can look at the OBI logs to find the PID of the HTTP server being instrumented:

autoinstrumenter-1    time=2025-12-30T19:57:17.593Z level=INFO msg="instrumenting process" component=discover.traceAttacher cmd=/usr/bin/python3.13 pid=297514 ino=460310 type=python service=""
autoinstrumenter-1    time=2025-12-30T19:57:18.320Z level=INFO msg="instrumenting process" component=discover.traceAttacher cmd=/usr/bin/python3.13 pid=310288 ino=722998 type=python service=""
autoinstrumenter-1    time=2025-12-30T19:57:18.512Z level=INFO msg="instrumenting process" component=discover.traceAttacher cmd=/usr/bin/python3.13 pid=315183 ino=2888480 type=python service=""

Which can also be obtained using standard GNU/Linux utilities:

$ cat /sys/fs/cgroup/lxc.payload.server01/system.slice/server.service/cgroup.procs 
297514
$ cat /sys/fs/cgroup/lxc.payload.server02/system.slice/server.service/cgroup.procs 
310288
$ ps -aux   grep http.server
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1000000   297514  0.0  0.1  32120 14856 ?        Ss   16:03   0:00 /usr/bin/python3 -m http.server 8080
1000000   310288  0.0  0.1  32120 10616 ?        Ss   16:09   0:00 /usr/bin/python3 -m http.server 8080
cipriano  315183  0.0  0.1 103476 11480 pts/3    S+   16:17   0:00 python -m http.server 8080

If we search for the PID in the OpenTelemetry Collector endpoint where Prometheus metrics are exposed, we can find the attribute values to filter on.

$ curl http://localhost:8889/metrics   rg 297514
default_target_info host_id="148f400ad3ea",host_name="148f400ad3ea",instance="148f400ad3ea:297514",job="python3.13",os_type="linux",service_instance_id="148f400ad3ea:297514",service_name="python3.13",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry-ebpf-instrumentation",telemetry_sdk_version="main"  1

Now we just need to add the processor to the collector configuration:

processors: # <--- NEW BLOCK
  filter/host_id:
    traces:
      span:
        - 'resource.attributes["service.instance.id"] == "148f400ad3ea:297514"'

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/host_id] # <--- NEW LINE
      exporters: [otlp/jaeger]

    metrics:
      receivers: [otlp] 
      processors:  # <--- NEW BLOCK
        - filter/host_id
      exporters: [prometheus]

That's it! The processor will handle the filtering for us, and we'll only see traces and metrics from the HTTP server running in the server01 container. Below are some screenshots from Jaeger and Grafana: jaeger search will all traces

Closing Notes I am still amazed at how powerful OBI can be. For those curious about the debug, we found out that a service responsible for the network orchestration of the Envoy containers was running `netplan apply` every 10 minutes because of a bug. Netplan apply causes interfaces to go down temporarily and this made the latency go above 500ms which caused the 499s.

Ravi Dwivedi: Transit through Kuala Lumpur

In my last post, Badri and I reached Kuala Lumpur - the capital of Malaysia - on the 7th of December 2024. We stayed in Bukit Bintang, the entertainment district of the city. Our accommodation was pre-booked at Manor by Mingle , a hostel where I had stayed for a couple of nights in a dormitory room earlier in February 2024. We paid 4937 rupees (the payment was online, so we paid in Indian rupees) for 3 nights for a private room. From the Terminal Bersepadu Selatan (TBS) bus station, we took the metro to the Plaza Rakyat LRT station, which was around 500 meters from the hostel. Upon arriving at the hostel, we presented our passports at their request, followed by a 20 ringgit (400 rupee) deposit which would be refunded once we returned the room keys at checkout.

Outside view of the hostel Manor by Mingle

Our room was upstairs and it had a bunk bed. I had seen bunk beds in dormitories before, but this was my first time seeing a bunk bed in a private room. The room did not have any toilets, so we had to use shared toilets. Unusually, the hostel was equipped with a pool. It also had a washing machine with dryers - this was one of the reasons we chose this hostel, because we were traveling light and hadn t packed too many clothes. The machine and dryer cost 10 ringgits (200 rupees) per use, and we only used it once. The hostel provided complimentary breakfast, which included coffee. Outside of breakfast hours, there was also a paid coffee machine. During our stay, we visited a gurdwara - a place of worship for Sikhs - which was within walking distance from our hostel. The name of the gurdwara was Gurdwara Sahib Mainduab. However, it wasn t as lively as I had thought. The gurdwara was locked from the inside, and we had to knock on the gate and call for someone to open it. A man opened the gate and invited us in. The gurdwara was small, and there was only one other visitor - a man worshipping upstairs. We went upstairs briefly, then settled down on the first floor. We had some conversations with the person downstairs who kindly made chai for us. They mentioned that the langar (community meal) is organized on every Friday, which was unlike the gurdwaras I have been to where the langar is served every day. We were there for an hour before we left. We also went to Adyar Ananda Bhavan (a restaurant chain) near our hostel to try the chain in Malaysia. The chain is famous in Southern India and also known by its short name A2B. We ordered

an onion dosa for 10 ringgits (200 rupees),
1 masala tea for 6 ringgits (120 rupees),
2 pooris for 8 ringgits (160 rupees) and
1 plate potato bajji for 7 ringgits (140 rupees).

All this came down to around 33 ringgits (including taxes), i.e. around 660 rupees. We also purchased some snacks such as murukku from there for our trip. We had planned a day trip to Malacca, but had to cancel it due to rain. We didn t do a lot in Kuala Lumpur, and it ended up acting as a transit point for us to other destinations: flights from Kuala Lumpur were cheaper than Singapore, and in one case a flight via Kuala Lumpur was even cheaper than a direct flight! We paid 15,000 rupees in total for the following three flights:

Kuala Lumpur to Brunei,
Brunei to Kuala Lumpur, and
Kuala Lumpur to Ho Chi Minh City (Vietnam).

These were all AirAsia flights. The cheap tickets, however, did not include any checked-in luggage, and the cabin luggage weight limit was 7 kg. We also bought quite some stuff in Kuala Lumpur and Singapore, leading to an increase in the weight of our luggage. We estimated that it would be cheaper for us to take only essential items such as clothes, cameras, and laptops, and to leave behind souvenirs and other non-essentials in lockers at the TBS bus stand in Kuala Lumpur, than to pay more for check-in luggage. It would take 140 ringgits for us to add a checked-in bag from Kuala Lumpur to Bandar Seri Begawan and back, while the cost for lockers was 55 ringgits at the rate of 5 ringgits every six hours. We had seen these lockers when we alighted at the bus stand while coming from Johor Bahru. There might have been lockers in the airport itself as well, which would have been more convenient as we were planning to fly back in soon, but we weren t sure about finding lockers at the airport and we didn t want to waste time looking. We had an early morning flight for Brunei on the 10th of December. We checked out from our hostel on the night of the 9th of December, and left for TBS to take a bus to the airport. We took a metro from the nearest metro station to TBS. Upon reaching there, we put our luggage in the lockers. The lockers were automated and there was no staff there to guide us.

We bought a ticket for the airport bus from a counter at TBS for 26 ringgits for both of us. In order to give us tickets, the person at the counter asked for our passports, and we handed it over to them promptly. Since paying in cash did not provide any extra anonymity, I would advise others to book these buses online. In Malaysia, you also need a boarding pass for buses. The bus terminal had kiosks for getting these printed, but they were broken and we had to go to a counter to obtain them. The boarding pass mentioned our gate number and other details such as our names and departure time of the bus. The company was Jet Bus.

My boarding pass for the bus to the airport in Kuala Lumpur

To go to our boarding gate, we had to scan our boarding pass to let the AFC gates open. Then we went downstairs, leading into the waiting area. It had departure boards listing the bus timings and their respective gates. We boarded our bus around 10 minutes before the departure time - 00:00 hours. It departed at its scheduled time and took 45 minutes to reach KL Airport Terminal 2, where we alighted. We reached 6 hours before our flight s departure time of 06:30. We stopped at a convenience store at the airport to have some snacks. Then we weighed our bags at a weighing machine to check whether we were within the weight limit. It turned out that we were. We went to an AirAsia counter to get our boarding passes. The lady at our counter checked our Brunei visas carefully and looked for any Brunei stamps on the passports to verify whether we had used that visa in the past. However, she didn t weigh our bags to check whether they were within the limit, and gave us our boarding passes. We had more than 4 hours to go before our flight. This was the downside of booking an early morning flight - we weren t able to get a full night s sleep. A couple of hours before our flight time, we were hanging around our boarding gate. The place was crowded, so there were no seats available. There were no charging points. There was a Burger King outlet there which had some seating space and charging points. As we were hungry, we ordered two cups of cappuccino coffee (15.9 ringgits) and one large french fries (8.9 ringgits) from Burger King. The total amount was 24 ringgits. When it was time to board the flight, we went to the waiting area for our boarding gates. Soon, we boarded the plane. It took 2.5 hours to reach the Brunei International Airport in the capital city of Bandar Seri Begawan.

Stay tuned for our experiences in Brunei! Credits: Thanks to Badri, Benson and Contrapunctus for reviewing the draft.

Freexian Collaborators: How files are stored by Debusine (by Stefano Rivera)

Debusine is a tool designed for Debian developers and Operating System developers in general. This post describes how Debusine stores and manages files. Debusine has been designed to run a network of workers that can perform various tasks that consume and produce artifacts . The artifact itself is a collection of files structured into an ontology of artifact types. This generic architecture should be suited to many sorts of build & CI problems. We have implemented artifacts to support building a Debian-like distribution, but the foundations of Debusine aim to be more general than that. For example a package build task takes a debian:source-package as input and produces some debian:binary-packages and a debian:package-build-log as output. This generalized approach is quite different to traditional Debian APT archive implementations, which typically required having the archive contents on the filesystem. Traditionally, most Debian distribution management tasks happen within bespoke applications that cannot share much common infrastructure.

File Stores Debusine s files themselves are stored by the File Store layer. There can be multiple file stores configured, with different policies. Local storage is useful as the initial destination for uploads to Debusine, but it has to be backed up manually and might not scale to sufficiently large volumes of data. Remote storage such as S3 is also available. It is possible to serve a file from any store, with policies for which one to prefer for downloads and uploads. Administrators can set policies for which file stores to use at the scope level, as well as policies for populating and draining stores of files.

Artifacts As mentioned above, files are collected into Artifacts. They combine:

a set of files with names (including potentially parent directories)

a category, e.g. debian:source-package

key-value data in a schema specified by the category and stored as a JSON-encoded dictionary.

Within the stores, files are content-addressed: a file with a given SHA-256 digest is only stored once in any given store, and may be retrieved by that digest. When a new artifact is created, its files are uploaded to Debusine as needed. Some of the files may already be present in the Debusine instance. In that case, if the file is already part of the artifact s workspace, then the client will not need to re-upload the file. But if not, it must be reuploaded to avoid users obtaining unauthorized access to existing file contents in another private workspace or multi-tenant scope. Because the content-addressing makes storing duplicates cheap, it s common to have artifacts that overlap files. For example a debian:upload will contain some of the same files as the related debian:source-package as well as the `.changes` file. Looking at the debusine.debian.net instance that we run, we can see a content-addressing savings of 629 GiB across our (currently) 2 TiB file store. This is somewhat inflated by the Debian Archive import, that did not need to bother to share artifacts between suites. But it still shows reasonable real-world savings.

APT Repository Representation Unlike a traditional Debian APT repository management tool, the source package and binary packages are not stored directly in the pool of an APT repository on disk on the debusine server. Instead we abstract the repository into a debian:suite collection within the Debusine database. The collection contains the artifacts that make up the APT repository. To ensure that it can be safely represented as a valid URL structure (or files on disk) the suite collection maintains an index of the pool filenames of its artifacts. Suite collections can combine into a debian:archive collection that shares a common file pool. Debusine collections can keep an historical record of when things were added and removed. This, combined with the database-backed collection-driven repository representation makes it very easy to provide APT-consumable snapshot views to every point in a repository s history.

Expiry While a published distribution probably wants to keep the full history of all its package builds, we don t need to retain all of the output of all QA tasks that were run. Artifacts can have an expiration delay or inherit one from their workspace. Once this delay has expired, artifacts which are not being held in any collection are eligible to be automatically cleaned up. QA work that is done in a workspace that has automatic artifact expiry, and isn t publishing the results to an APT suite, will safely automatically expire.

Daily Vacuum A daily vacuum task handles all of the file periodic maintenance for file stores. It does some cleanup of working areas, a scan for unreferenced & missing files, and enforces file store policies. The policy work could be copying files for backup or moving files between stores to keep them within size limits (e.g. from a local upload store into a general cloud store).

In Conclusion Debusine provides abstractions for low-level file storage and object collections. This allows storage to be scalable beyond a single filesystem and highly available. Using content-addressed storage minimizes data duplication within a Debusine instance. For Debian distributions, storing the archive metadata entirely in a database made providing built-in snapshot support easy in Debusine.

kpcyrd: 2025 wrapped

Same as last year, this is a summary of what I ve been up to throughout the year. See also the recap/retrospection published by my friends (antiz, jvoisin, orhun).

Uploaded 467 packages to Arch Linux
- Most of them being reproducible, meaning I provably didn t abuse my position of compiling the binaries
- 35 of them are signal-desktop
- 29 of them are metasploit
Made 53 uploads to Debian
- All of them being related to my work in the debian-rust team, that I ve been a part of since 2018
- Also applied for Debian Developer status (with 4 Debian Developers advocating for me)
Made 14 commits in Alpine Linux aports
- 13 of them being package releases
Made 2 commits in NixOS nixpkgs
- Also joined their Github org
Made 4 commits in homebrew-core
- With special focus on polishing the Rust development experience for the RP2040 microcontroller
Lost Onion, my cat of 13 years, to inoperable cancer. He has been with me throughout my entire open source journey (sometimes being credited as co-author) and who looked after me for my entire adult life. You won t be forgotten.
Developed 6 hand-held games with embedded Rust, most of them being birthday gifts for people close to me
- game-taco-burglar
  - A motorcycling lockpicker
- game-antifa-syndikitty
  - A nurse with a secret double life
  - At that point the longest and most in-depth game I built throughout my life
- game-chop-chop
  - A French tetris-spinoff, this was one of my Fusion projects this year
  - The hardware was specifically designed to be easy to solder/make from readily available parts (~ 5 per unit)
  - I gave away a few devices I made, some people successfully built one on their own
- game-ratatat
  - A space-invader like game about a very enthusiastic seamster
- game-octo-space-irs
  - As an employee of the intergalactic revenue service, you tax the rich through reversing and cracking computer programs
  - I gifted another copy to a Tor directory authority operator I m friends with, who was very excited about the concept and levels I designed
- game-the-curse-of-the-headless-goose
  - A turn-based game about an underground kickboxing club
  - This one was meant to be a rogue-lite (which I needed the savegame library for), but only managed to build the introduction/tutorial unfortunately
Picked up work on apt-swarm again
- Replaced the old database code with a custom engine, reducing RAM usage from multiple gigabytes down to ~9MB
- Ran a small p2p network all over the world, with ~10-15 locations/countries on average
- As part of this, found a bug in tokio that could lead to silent data loss in some cases
Had 2 of my projects explicitly mentioned in the Debian release notes in their What s new in Debian 13 summary
Was mentioned in multiple academic papers on arxiv.org:
- Reproducible Builds and Insights from an Independent Verifier for Arch Linux (explicitly in the Acknowledgments section)
- Beneath the Mask: Can Contribution Data Unveil Malicious Personas in Open-Source Projects?
- Wolves in the Repository: A Software Engineering Analysis of the XZ Utils Supply Chain Attack
- Causes and Canonicalization of Unreproducible Builds in Java
- Reproducible Builds for Quantum Computing (mentions rebuilderd)
Was referenced twice on LWN:
- Hash-based module integrity checking (mentions me directly)
- Fedora change aims for 99% package reproducibility (doesn t mention me, but rebuilderd 10x)
Published a draft version of PlatypOS, an Experimental toy unix-like userspace operating system with strong preference towards Rust . As part of this:
- Developed custom pacman-database tooling in Rust instead of bash
- During this project I found and reported issues in uutils install (uutils/coreutils#8033) and mv (uutils/coreutils#8044) (both fixed shortly after)
- The project stalled because it s too big to side-quest
Had the first ever CVE issued for software I wrote: CVE-2025-52926
- Found, reported and fixed by a c t Open Source editor
Published 9 repositories related to my embedded Rust work
- embedded-mono-img for all the graphics in my games
- rp2040-psp-joystick to demo use of an analog joystick input
- rp2040-demo-st7789 to demo a higher resolution screen I experimented with
- rp2040-demo-w25qxx to demo how to store data in NOR flash
- rp2040-demo-at24cxx to demo how to store data in an EEPROM
- embedded-graphics-colorcast a library I developed so I can keep using embedded-mono-img on ST7789/ILI9486 screens - I used tinybmp in one project but it was fairly slow
- ch32v003-demo to demo and document the lowend ch32v003 RISC-V microcontroller, with devboards that are commonly sold for 0.50-0.70 on AliExpress (it s cute but lacks the required 5.1k resistors on the USB-C configuration pins that tell the host to provide 5V, so it won t work with many USB-C chargers, which is quite annoying)
- embedded-savegame an atomic/transactional savegame library, with powerfail-safety and wear-leveling, optimized for flash and EEPROM storage
- djb2 a very lightweight non-cryptographic checksum algorithm that replaced my use of CRC32 in the embedded-savegame library, to make it more suitable for the ch32v003
Contributed to the Reproducible Builds mailing list 30 times
Developed repro-threshold, an integration for apt to act as a rebuilderd client, enforcing a reproducible builds trust policy of your choice
- The feature was suggested/requested by a CCC member during MiniDebConf Hamburg 2025
Collaborated with an openSUSE engineer I ve known for several years to debug and fix an issue in gtk-rs that caused indeterministic build output for many desktop programs
Volunteered at a soldering workshop for beginners for the 4th year in a row
Completed the first year of volunteering in an awareness team
Wrote 1 blog post (besides this one)
Attended FOSDEM, MiniDebConf, Fusion, the Reproducible Builds summit, the Arch Summit and 39c3
- Hosted sessions at both FOSDEM (1st time) and Fusion (2nd time)
Grew and harvested 2 plants
Traveled to
- Denmark
- Sweden
- Turkey, visiting a good friend
- Belgium
- Austria
Made and printed
- 2 new sticker designs
- 2 new hoodie designs
Changed my medication plan
Got 4 new tattoos

Thanks to everybody who has been part of my human experience, past or present. Especially those who ve been closest.

30 December 2025

Utkarsh Gupta: FOSS Activites in December 2025

Here s my monthly but brief update about the activities I ve done in the FOSS world.

Debian

Whilst I didn t get a chance to do much, here are still a few things that I worked on:

Prepared security update for wordpress for trixie and bookworm.

A few discussions with the new DFSG team, et al.

Assited a few folks in getting their patches submitted via Salsa.

Mentoring for newcomers.

Moderation of -project mailing list.

Ubuntu

I joined Canonical to work on Ubuntu full-time back in February 2021. Whilst I can t give a full, detailed list of things I did, here s a quick TL;DR of what I did:

Successfully released Resolute Snapshot 2!

This one was also done without the ISO tracker and cdimage access.

I think this one went rather smooth. Let s see what we re able to do for snapshot 3.

Worked on removing GPG keys from the cdimage instance. That took a while, whew!

Assisted a bunch of folks with my Archive Admin and Release team hats to:

review NEW packages for Ubuntu Studio.

remove old binaries that are stalling transition and/or migration.

LTS requalification of Ubuntu flavours.

bootstrapping dotnet-10 packages for Stable Release Updates.

With that, we ve entered the EOY break. :)

I was anyway on vacation for majority of this month. ;)

Debian (E)LTS

This month I have worked 72 hours on Debian Long Term Support (LTS) and on its sister Extended LTS project and did the following things:

Released Security Updates

ruby-git: Multiple vulnerabilities leading to command line injection and improper path escaping.

[LTS]: Fixed CVE-2022-25648, CVE-2022-46648, and CVE-2022-47318 via 1.7.0-1+deb11u1 for bullseye. This has been released as DLA 4406-1.

ruby-sidekiq: Multiple vulnerabilities leading to Cross-site Scripting (XSS) and Denial of Service in Web UI.

[LTS]: Fixed CVE-2021-30151 and CVE-2022-23837 via 6.0.4+dfsg-2+deb11u1 for bullseye. This has been released as DLA 4407-1.

python-apt: Vulnerability leading to crash via invalid nullptr dereference in TagSection.keys().

[LTS]: Fixed CVE-2025-6966 via 2.2.1.1 for bullseye. This has been released as DLA 4408-1.

[ELTS]: Fixed CVE-2025-6966 via 1.8.4.4 for buster and 1.4.4 for stretch. This has been released as ELA 1596-1.

All of this was coordinated b/w the Security team and Julian Andres Klode. Julian will take care of the stable uploads.

node-url-parse: Vulnerability allowing authorization bypass through specially crafted URL with empty userinfo and no host.

[LTS]: Fixed CVE-2022-0639 via 1.5.3-1+deb11u3 for bullseye. This has been released as DLA 4413-1.

wordpress: Multiple vulnerabilities in WordPress core, leading to Sent Data & Cross-site Scripting.

[stable]: Fixed CVE-2025-58674 and CVE-2025-58246 via 6.8.3+dfsg1-0+deb13u1 for trixie. This has been released as DSA 6091-1.

usbmuxd: Privilege escalation vulnerability via path traversal in SavePairRecord command.

[LTS]: Fixed CVE-2025-66004 via 1.1.1-2+deb11u1 for bullseye. This has been released as DLA 4417-1.

[ELTS]: Fixed CVE-2025-66004 via 1.1.1~git20181007.f838cf6-1+deb10u1 for buster and 1.1.0-2+deb9u1 for stretch. This has been released as ELA 1599-1.

All of this was coordinated b/w the Security team and Yves-Alexis Perez. Yves will take care of the stable uploads.

gst-plugins-good1.0: Multiple vulnerabilities in isomp4 plugin leading to potential out-of-bounds reads and information disclosure.

[LTS]: Fixed CVE-2025-47219 and CVE-2025-47183 via 1.18.4-2+deb11u4 for bullseye. This has been released as DLA 4419-1.

postgresql-13: Multiple vulnerabilities including unauthorized schema statistics creation and integer overflow in libpq allocation calculations.

[LTS]: Fixed CVE-2025-12817 and CVE-2025-12818 via 13.23-0+deb11u1 for bullseye. This update has been prepared by the maintainer, Christoph Berg, and released as DLA 4420-1.

gst-plugins-base1.0: Multiple vulnerabilities in SubRip subtitle parsing leading to potential crashes and buffer issues.

[ELTS]: Fixed CVE-2025-47806, CVE-2025-47807, and CVE-2025-47808 via 1.14.4-2+deb10u5 for buster and 1.10.4-1+deb9u6 for stretch. This has been released as ELA 1600-1.

Work in Progress

ceph: Affected by CVE-2024-47866, using the argument `x-amz-copy-source` to put an object and specifying an empty string as its content leads to the RGW daemon crashing, resulting in a DoS attack.

[LTS]: Whilst the patch is straightforward, backports are a bit tricky. Also trying to backport the fix for CVE-2022-0670. The update has been prepared as of December 16th and awaiting Thomas Goirand s review.

Git repository for bullseye: https://salsa.debian.org/lts-team/packages/ceph/-/tree/debian/bullseye.

[ELTS]: Same as LTS - I ve tested the update but waiting for Thomas to ack the LTS upload and so I can proceed with ELTS uploads.

Git repository for buster: http://salsa.debian.org/lts-team/packages/ceph/-/commits/debian/buster

Git repository for stretch: http://salsa.debian.org/lts-team/packages/ceph/-/commits/debian/stretch

knot-resolver: Affected by CVE-2023-26249, CVE-2023-46317, and CVE-2022-40188, leading to Denial of Service.

[LTS]: The patches aren t very trivial, with CVE-2023-46317 reverting much of the patch for CVE-2023-26249. And then it had to be re-adjusted a bit for v5.3.1. The backports are ready as of December 23rd and awaiting Jakub or Santiago s review.

Git repository for bullseye: https://salsa.debian.org/lts-team/packages/knot-resolver/-/tree/debian/bullseye

adminer: Affected by CVE-2023-45195 and CVE-2023-45196, leading to SSRF and DoS, respectively.

[ELTS]: I picked it up again and the patches were already ready in Git. I ve reached out to Alexandre for a quick review and smoke test before we can release it for buster.

Git repository for buster: https://salsa.debian.org/lts-team/packages/adminer/-/tree/debian/buster

u-boot: Affected by CVE-2025-24857, where boot code access control flaw in U-Boot allowing arbitrary code execution via physical access.

[ELTS]: As it s only affected the version in stretch, I ve started the work to find the fixing commits and prepare a backport. Not much progress there, I ll roll it over to January.

ruby-rack: There were multiple vulnerabilities reported in Rack, leading to DoS (memory exhaustion) and proxy bypass.

[ELTS]: After completing the work for LTS myself, Bastien picked it up for ELTS and reached out about an upstream regression and we ve been doing some exchanges. Bastien has done most of the work backporting the patches but needs a review and help backporting CVE-2025-61771.

Other Activities

Frontdesk from 01-12-2025 to 07-12-2025.
- Auto EOL d a bunch of packages.
- Marked CVE-2025-12084/python2.7 as end-of-life for bullseye, buster, and stretch.
- Marked CVE-2025-12084/jython as end-of-life for bullseye.
- Marked CVE-2025-13992/chromium as end-of-life for bullseye.
- Marked apache2 CVEs as postponed for bullseye, buster, and stretch.
- Marked CVE-2025-13654/duc as postponed for bullseye and buster.
- Marked CVE-2025-32900/kdeconnect as ignored for bullseye.
- Marked CVE-2025-12084/pypy3 as postponed for bullseye.
- Marked CVE-2025-14104/util-linux as postponed for bullseye, buster, and stretch.
- Marked several CVEs for fastdds as postponed for bullseye.
- Marked several CVEs for pytorch as postponed for bullseye.
- Marked CVE-2025-2486/edk2 as postponed for bullseye.
- Marked CVE-2025-6172 7,9 /golang-1.15 as postponed for bullseye.
- Marked CVE-2025-65637/golang-logrus as postponed for bullseye.
- Marked CVE-2025-12385/qtdeclarative-opensource-src ,gles as postponed for bullseye, buster, and stretch.
- Marked TEMP-0000000-D08402/rust-maxminddb as postponed for bullseye.
- Added the following packages to d,e la-needed.txt:
  - liblivemedia, sogo.
- During my triage, I had to make the bin/elts-eol script robust to determine the lts_admin repository - did a back and forth with Emilio about this on the list.
- I sent a gentle reminder to the LTS team about the issues fixed in bullseye but not in bookworm via mailing list: https://lists.debian.org/debian-lts/2025/12/msg00013.html.
I claimed php-horde-css-parser to work on CVE-2020-13756 for buster and did almost all the work only to realize that the patch already existed in buster and the changelog confirmed that it was intentionally fixed.
- After speaking with Andreas Henriksson, we figured that the CVE ID was missed when the ELA was generated and so I fixed that via 87afaaf19ce56123bc9508d9c6cd5360b18114ef and 5621431e84818b4e650ffdce4c456daec0ee4d51 in the ELTS security tracker to reflect the situation.
Participated in a thread which I started last month around using Salsa CI for E/LTS packages and if we plan to sunset it in favor of using Debusine. The plan for now is to keep it around as it s still beneficial and Debusine is still in its early phase.
Did a lot of back and forth with Helmut about debusine uploads on #debian-elts.
- While debugging a failure in dcut uploads, I ran into an SSH compatibility issue on deb-master.freexian.com that could be fixed on the server-side. I shared all my findings to Freexian s sysadmin team.
- A minimal fix on the server side would be one of:
```
PubkeyAcceptedAlgorithms -ssh-dss
```
  or explicitly restricting to modern algorithms, e.g.:
```
PubkeyAcceptedAlgorithms
ssh-ed25519,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256
```
Jelly on #debian-lts reported that all my DLA mails had broken GMail s DKIM signature. So I set up sending replies from @debian.org and that seems to have fixed that! \o/
[LTS] Attended a rather short monthly LTS meeting on Jitsi. Summary here.
[E/LTS] Monitored discussions on mailing lists, IRC, and all the documentation updates.

Until next time.
:wq for today.

23 December 2025

Daniel Kahn Gillmor: AI and Secure Messaging Don't Mix

AI and Secure Messaging Don't Mix Over on the ACLU's Free Future blog, I just published an article titled AI and Secure Messaging Don't Mix. The blogpost assumes for the sake of the argument that people might actually want to have an AI involved in their personal conversations, and explores why Meta's Private Processing doesn't offer the level of assurance that they want it to offer. In short, the promises of "confidential cloud computing" are built on shaky foundations, especially against adversaries as powerful as Meta themselves. If you really want AI in your chat, the baseline step for privacy preservation is to include it in your local compute base, not to use a network service! But these operators clearly don't value private communication as much as they value binding you to their services. But let's imagine some secure messenger that actually does put message confidentiality first -- and imagine they had integrated some sort of AI capability into the messenger. That at least bypasses the privacy questions around AI use. Would you really want to talk with your friends, as augmented by their local AI, though? Would you want an AI, even one running locally with perfect privacy, intervening in your social connections? What if it summarized your friend's messages to you in a way that led you to misunderstand (or ignore) an important point your friend had made? What if it encouraged you to make an edgy joke that comes across wrong? Or to say something that seriously upsets a friend? How would you respond? How would you even know that it had happened? My handle is `dkg`. More times than i can count, i've had someone address me in a chat as "dog" and then cringe and apologize and blame their spellchecker/autocorrect. I can laugh these off because the failure mode is so obvious and transparent -- and repeatable. (also, dogs are awesome, so i don't really mind!) But when our attention (and our responses!) are being shaped and molded by these plausibility engines, how will we even know that mistakes are being made? What if the plausibility engine you've hooked into your messenger embeds subtle (or unsubtle!) bias? Don't we owe it to each other to engage with actual human attention?

22 December 2025

Isoken Ibizugbe: Everybody Struggles

That s right: everyone struggles. You could be working on a project only to find a mountain of new things to learn, or your code might keep failing until you start to doubt yourself. I feel like that sometimes, wondering if I m good enough. But in those moments, I whisper to myself: You don t know it yet; once you do, it will get easy. While contributing to the Debian openQA project, there was so much to learn, from understanding what Debian actually is to learning the various installation methods and image types. I then had to tackle the installation and usage of openQA itself. I am incredibly grateful for the installation guideprovided by Roland Clobus and the documentation on writing code for openQA.

Overcoming Technical Hurdles Even with amazing guides, I hit major roadblocks. Initially, I was using Windows with VirtualBox, but openQA couldn t seem to run the tests properly. Despite my mentors (Roland and Phil) suggesting alternatives, the issues persisted. I actually installed openQA twice on VirtualBox and realized that if you miss even one small step in the installation, it becomes very difficult to move forward. Eventually, I took the big step and dual-booted my machine to Linux. Even then, the challenges didn t stop. My openQA Virtual Machine (VM) ran out of allocated space and froze, halting my testing. I reached out on the IRC chat and received the help I needed to get back on track.

My Research Line-up When I m struggling for information, I follow my go-to first step for research, followed by other alternatives:

Google: This is my first stop. It helped me navigate the switch to a Linux OS and troubleshoot KVM connection issues for the VM. Whether it s an Ubuntu forum or a technical blog, I skim through until I find what can help.

The Upstream Documentation: If Google doesn t have the answer, I go straight to the official openQA documentation. This is a goldmine. It explains functions, how to use them, and lists usable test variables.

The Debian openQA UI: While working on the apps_startstop tests, I look at previous similar tests on openqa.debian.net/tests. I checked the Settings tab to see exactly what variables were used and how the test was configured.

Salsa (Debian s GitLab): I reference the Salsa Debian openQA README and the developer guides sometimes; Getting started, Developer docs on how to write tests

I also had to learn the basics of the Perl programming language during the four-week contribution stage. While we don t need to be Perl experts, I found it essential to understand the logic so I can explain my work to others. I ve spent a lot of time studying the codebase, which is time-consuming but incredibly valuable. For example, my apps_startstop test command originally used a long list of applications via ENTRYPOINT. I began to wonder if there was a more efficient way. With Roland s guidance, I explored the main.pm file. This helped me understand how the apps_startstop function works and how it calls variables. I also noticed there are utility functions that are called in tests. I also check them and try to understand their function, so I know if I need them or not. I know I still have a lot to learn, and yes, the doubt still creeps in sometimes. But I am encouraged by the support of my mentors and the fact that they believe in my ability to contribute to this project.
If you re struggling too, just remember: you don t know it yet; once you do, it will get easy.

Emmanuel Kasper: Configuring a mail transfer agent to interact with the Debian bug tracker

Email interface of the Debian bug tracker The main interface of the Debian bug tracker, at http://bugs.debian.org, is e-mail, and modifications are made to existing bugs by sending an email to an address like 873518@bugs.Debian.org. The web interface allows to browse bugs, but any addition to the bug itself will require an email client. This sounds a bit weird in 2025, as http REST clients with Oauth access tokens for command line tools interacting with online resources are today the norm. However we should remember the Debian project goes back to 1993 and the bug tracker software debugs, was released in 1994. REST itself was first introduced in 2000, six years later. In any case, using an email client to create or modify bug reports is not a bad idea per se:

the internet mail protocol, SMTP, is a well known and standardized protocol defined in an IETF RFC.
no need for account creation and authentication, you just need an email address to interact. There is a risk of spam, but in my experience this has been very low. When authentication is needed, Debian Developpers sign their work with their private GPG key.
you can use the bug tracker using the interface of your choice: webmail, graphical mail clients like Thunderbird or Evolution, text clients like Mutt or Pine, or command line tools like bts.

A system wide minimal Mail Transfer Agent to send mail We can configure bts as a SMTP client, with username and password. In SMTP client mode, we would need to enter the SMTP settings from our mail service provider. The other option is to configure a Mail Transfer Agent (MTA) which provides a system wide sendmail interface, that all command line and automation tools can use send email. For instance reportbug and git send-email are able to use the sendmail interface. Why a sendmail interface ? Because sendmail used to be the default MTA of Unix back in the days, thus many programs sending mails expect something which looks like sendmail locally. A popular, maintained and packaged minimal MTA is msmtp, we are going to use it. msmtp installation and configuration Installation is just an apt away:

# apt install msmtp msmtp-mta
# msmtp --version
msmtp version 1.8.23

You can follow this blog post to configure msmtp, including saving your mail account credentials in the Gnome keyring. Once installed, you can verify that msmtp-mta created a sendmail symlink.

$ ls -l /usr/sbin/sendmail 
lrwxrwxrwx 1 root root 12 16 avril  2025 /usr/sbin/sendmail -&gt; ../bin/msmtp

bts, git-send-email and reportbug will pipe their output to /usr/sbin/sendmail and msmtp will send the email in the background. Testing with with a simple mail client Debian comes out of the box with a primitive mail client, bsd-mailx that you can use to test your MTA set up. If you have configured msmtp correctly you send an email to yourself using

$ echo "hello world"   mail -s "my mail subject" user@domain.org

Now you can open bugs for Debian with reportbug, tag them with bts and send git formated patches from the command line with git send-email.

C.J. Collier: I m learning about perlguts today.

## 0.23	2025-12-20
commit be15aa25dea40aea66a8534143fb81b29d2e6c08
Author: C.J. Collier 
Date:   Sat Dec 20 22:40:44 2025 +0000
    Fixes C-level test infrastructure and adds more test cases for upb_to_sv conversions.
    
    - **Makefile.PL:**
        - Allow  extra_src  in  c_test_config.json  to be an array.
        - Add ASan flags to CCFLAGS and LDDLFLAGS for better debugging.
        - Corrected echo newlines in  test_c  target.
    - **c_test_config.json:**
        - Added missing type test files to  deps  and  extra_src  for  convert/sv_to_upb  and  convert/upb_to_sv  test runners.
    - **t/c/convert/upb_to_sv.c:**
        - Fixed a double free of  test_pool .
        - Added missing includes for type test headers.
        - Updated test plan counts.
    - **t/c/convert/sv_to_upb.c:**
        - Added missing includes for type test headers.
        - Updated test plan counts.
        - Corrected Perl interpreter initialization.
    - **t/c/convert/types/**:
        - Added missing  test_util.h  include in new type test headers.
        - Completed the set of  upb_to_sv  test cases for all scalar types by adding optional and repeated tests for  sfixed32 ,  sfixed64 ,  sint32 , and  sint64 , and adding repeated tests to the remaining scalar type files.
    - **Documentation:**
        - Updated  01-xs-testing.md  with more debugging tips, including ASan usage and checking for double frees and typos.
        - Updated  xs_learnings.md  with details from the recent segfault.
        - Updated  llm-plan-execution-instructions.md  to emphasize debugging steps.
## 0.22	2025-12-19
commit 2c171d9a5027e0150eae629729c9104e7f6b9d2b
Author: C.J. Collier 
Date:   Fri Dec 19 23:41:02 2025 +0000
    feat(perl,testing): Initialize C test framework and build system
    
    This commit sets up the foundation for the C-level tests and the build system for the Perl Protobuf module:
    
    1.  **Makefile.PL Enhancements:**
        *   Integrates  Devel::PPPort  to generate  ppport.h  for better portability.
        *   Object files now retain their path structure (e.g.,  xs/convert/sv_to_upb.o ) instead of being flattened, improving build clarity.
        *   The  MY::postamble  is significantly revamped to dynamically generate build rules for all C tests located in  t/c/  based on the  t/c/c_test_config.json  file.
        *   C tests are linked against  libprotobuf_common.a  and use  ExtUtils::Embed  flags.
        *   Added  JSON::MaybeXS  to  PREREQ_PM .
        *   The  test  target now also depends on the  test_c  target.
    
    2.  **C Test Infrastructure ( t/c/ ):
        *   Introduced  t/c/c_test_config.json  to configure individual C test builds, specifying dependencies and extra source files.
        *   Created  t/c/convert/test_util.c  and  .h  for shared test functions like loading descriptors.
        *   Initial  t/c/convert/upb_to_sv.c  and  t/c/convert/sv_to_upb.c  test runners.
        *   Basic  t/c/integration/030_protobuf_coro.c  for Coro safety testing on core utils using  libcoro .
        *   Basic  t/c/integration/035_croak_test.c  for testing exception handling.
        *   Basic  t/c/integration/050_convert.c  for integration testing conversions.
    
    3.  **Test Proto:** Updated  t/data/test.proto  with more field types for conversion testing and regenerated  test_descriptor.bin .
    
    4.  **XS Test Harness ( t/c/upb-perl-test.h ):** Added  like_n  macro for length-aware regex matching.
    
    5.  **Documentation:** Updated architecture and plan documents to reflect the C test structure.
    6.  **ERRSV Testing:** Note that the C tests ( t/c/ ) will primarily check *if* a  croak  occurs (i.e., that the exception path is taken), but will not assert on the string content of  ERRSV . Reliably testing  $@  content requires the full Perl test environment with  Test::More , which will be done in the  .t  files when testing the Perl API.
    
    This provides a solid base for developing and testing the XS and C components of the module.
## 0.21	2025-12-18
commit a8b6b6100b2cf29c6df1358adddb291537d979bc
Author: C.J. Collier 
Date:   Thu Dec 18 04:20:47 2025 +0000
    test(C): Add integration tests for Milestone 2 components
    
    - Created t/c/integration/030_protobuf.c to test interactions
      between obj_cache, arena, and utils.
    - Added this test to t/c/c_test_config.json.
    - Verified that all C tests for Milestones 2 and 3 pass,
      including the libcoro-based stress test.
## 0.20	2025-12-18
commit 0fcad68680b1f700a83972a7c1c48bf3a6958695
Author: C.J. Collier 
Date:   Thu Dec 18 04:14:04 2025 +0000
    docs(plan): Add guideline review reminders to milestones
    
    - Added a "[ ] REFRESH: Review all documents in @perl/doc/guidelines/**"
      checklist item to the start of each component implementation
      milestone (C and Perl layers).
    - This excludes Integration Test milestones.
## 0.19	2025-12-18
commit 987126c4b09fcdf06967a98fa3adb63d7de59a34
Author: C.J. Collier 
Date:   Thu Dec 18 04:05:53 2025 +0000
    docs(plan): Add C-level and Perl-level Coro tests to milestones
    
    - Added checklist items for  libcoro -based C tests
      (e.g.,  t/c/integration/050_convert_coro.c ) to all C layer
      integration milestones (050 through 220).
    - Updated  030_Integration_Protobuf.md  to standardise checklist
      items for the existing  030_protobuf_coro.c  test.
    - Removed the single  xt/author/coro-safe.t  item from
       010_Build.md .
    - Added checklist items for Perl-level  Coro  tests
      (e.g.,  xt/coro/240_arena.t ) to each Perl layer
      integration milestone (240 through 400).
    - Created  perl/t/c/c_test_config.json  to manage C test
      configurations externally.
    - Updated  perl/doc/architecture/testing/01-xs-testing.md  to describe
      both C-level  libcoro  and Perl-level  Coro  testing strategies.
## 0.18	2025-12-18
commit 6095a5a610401a6035a81429d0ccb9884d53687b
Author: C.J. Collier 
Date:   Thu Dec 18 02:34:31 2025 +0000
    added coro testing to c layer milestones
## 0.17	2025-12-18
commit cc0aae78b1f7f675fc8a1e99aa876c0764ea1cce
Author: C.J. Collier 
Date:   Thu Dec 18 02:26:59 2025 +0000
    docs(plan): Refine test coverage checklist items for SMARTness
    
    - Updated the "Tests provide full coverage" checklist items in
      C layer plan files (020, 040, 060, 080, 100, 120, 140, 160, 180, 200)
      to explicitly mention testing all public functions in the
      corresponding header files.
    - Expanded placeholder checklists in 140, 160, 180, 200.
    - Updated the "Tests provide full coverage" and "Add coverage checks"
      checklist items in Perl layer plan files (230, 250, 270, 290, 310, 330,
      350, 370, 390) to be more specific about the scope of testing
      and the use of  Test::TestCoverage .
    - Expanded Well-Known Types milestone (350) to detail each type.
## 0.16	2025-12-18
commit e4b601f14e3817a17b0f4a38698d981dd4cb2818
Author: C.J. Collier 
Date:   Thu Dec 18 02:07:35 2025 +0000
    docs(plan): Full refactoring of C and Perl plan files
    
    - Split both ProtobufPlan-C.md and ProtobufPlan-Perl.md into
      per-milestone files under the  perl/doc/plan/  directory.
    - Introduced Integration Test milestones after each component
      milestone in both C and Perl plans.
    - Numbered milestone files sequentially (e.g., 010_Build.md,
      230_Perl_Arena.md).
    - Updated main ProtobufPlan-C.md and ProtobufPlan-Perl.md to
      act as Tables of Contents.
    - Ensured consistent naming for integration test files
      (e.g.,  t/c/integration/030_protobuf.c ,  t/integration/260_descriptor_pool.t ).
    - Added architecture review steps to the end of all milestones.
    - Moved Coro safety test to C layer Milestone 1.
    - Updated Makefile.PL to support new test structure and added Coro.
    - Moved and split t/c/convert.c into t/c/convert/*.c.
    - Moved other t/c/*.c tests into t/c/protobuf/*.c.
    - Deleted old t/c/convert.c.
## 0.15	2025-12-17
commit 649cbacf03abb5e7293e3038bb451c0406e9d0ce
Author: C.J. Collier 
Date:   Wed Dec 17 23:51:22 2025 +0000
    docs(plan): Refactor and reset ProtobufPlan.md
    
    - Split the plan into ProtobufPlan-C.md and ProtobufPlan-Perl.md.
    - Reorganized milestones to clearly separate C layer and Perl layer development.
    - Added more granular checkboxes for each component:
      - C Layer: Create test, Test coverage, Implement, Tests pass.
      - Perl Layer: Create test, Test coverage, Implement Module/XS, Tests pass, C-Layer adjustments.
    - Reset all checkboxes to  [ ]  to prepare for a full audit.
    - Updated status in architecture/api and architecture/core documents to "Not Started".
    
    feat(obj_cache): Add unregister function and enhance tests
    
    - Added  protobuf_unregister_object  to  xs/protobuf/obj_cache.c .
    - Updated  xs/protobuf/obj_cache.h  with the new function declaration.
    - Expanded tests in  t/c/protobuf_obj_cache.c  to cover unregistering,
      overwriting keys, and unregistering non-existent keys.
    - Corrected the test plan count in  t/c/protobuf_obj_cache.c  to 17.
## 0.14	2025-12-17
commit 40b6ad14ca32cf16958d490bb575962f88d868a1
Author: C.J. Collier 
Date:   Wed Dec 17 23:18:27 2025 +0000
    feat(arena): Complete C layer for Arena wrapper
    
    This commit finalizes the C-level implementation for the Protobuf::Arena wrapper.
    
    - Adds  PerlUpb_Arena_Destroy  for proper cleanup from Perl's DEMOLISH.
    - Enhances error checking in  PerlUpb_Arena_Get .
    - Expands C-level tests in  t/c/protobuf_arena.c  to cover memory allocation
      on the arena and lifecycle through  PerlUpb_Arena_Destroy .
    - Corrects embedded Perl initialization in the C test.
    
    docs(plan): Refactor ProtobufPlan.md
    
    - Restructures the development plan to clearly separate "C Layer" and
      "Perl Layer" tasks within each milestone.
    - This aligns the plan with the "C-First Implementation Strategy" and improves progress tracking.
## 0.13	2025-12-17
commit c1e566c25f62d0ae9f195a6df43b895682652c71
Author: C.J. Collier 
Date:   Wed Dec 17 22:00:40 2025 +0000
    refactor(perl): Rename C tests and enhance Makefile.PL
    
    - Renamed test files in  t/c/  to better match the  xs  module structure:
        -  01-cache.c  ->  protobuf_obj_cache.c 
        -  02-arena.c  ->  protobuf_arena.c 
        -  03-utils.c  ->  protobuf_utils.c 
        -  04-convert.c  ->  convert.c 
        -  load_test.c  ->  upb_descriptor_load.c 
    - Updated  perl/Makefile.PL  to reflect the new test names in  MY::postamble 's  $c_test_config .
    - Refactored the  $c_test_config  generation in  Makefile.PL  to reduce repetition by using a default flags hash and common dependencies array.
    - Added a  fail()  macro to  perl/t/c/upb-perl-test.h  for consistency.
    - Modified  t/c/upb_descriptor_load.c  to use the  t/c/upb-perl-test.h  macros, making its output consistent with other C tests.
    - Added a skeleton for  t/c/convert.c  to test the conversion functions.
    - Updated documentation in  ProtobufPlan.md  and  architecture/testing/01-xs-testing.md  to reflect new test names.
## 0.12	2025-12-17
commit d8cb5dd415c6c129e71cd452f78e29de398a82c9
Author: C.J. Collier 
Date:   Wed Dec 17 20:47:38 2025 +0000
    feat(perl): Refactor XS code into subdirectories
    
    This commit reorganizes the C code in the  perl/xs/  directory into subdirectories, mirroring the structure of the Python UPB extension. This enhances modularity and maintainability.
    
    - Created subdirectories for each major component:  convert ,  descriptor ,  descriptor_containers ,  descriptor_pool ,  extension_dict ,  map ,  message ,  protobuf ,  repeated , and  unknown_fields .
    - Created skeleton  .h  and  .c  files within each subdirectory to house the component-specific logic.
    - Updated top-level component headers (e.g.,  perl/xs/descriptor.h ) to include the new sub-headers.
    - Updated top-level component source files (e.g.,  perl/xs/descriptor.c ) to include their main header and added stub initialization functions (e.g.,  PerlUpb_InitDescriptor ).
    - Moved code from the original  perl/xs/protobuf.c  to new files in  perl/xs/protobuf/  (arena, obj_cache, utils).
    - Moved code from the original  perl/xs/convert.c  to new files in  perl/xs/convert/  (upb_to_sv, sv_to_upb).
    - Updated  perl/Makefile.PL  to use a glob ( xs/*/*.c ) to find the new C source files in the subdirectories.
    - Added  perl/doc/architecture/core/07-xs-file-organization.md  to document the new structure.
    - Updated  perl/doc/ProtobufPlan.md  and other architecture documents to reference the new organization.
    - Corrected self-referential includes in the newly created .c files.
    
    This restructuring provides a solid foundation for further development and makes it easier to port logic from the Python implementation.
## 0.11	2025-12-17
commit cdedcd13ded4511b0464f5d3bdd72ce6d34e73fc
Author: C.J. Collier 
Date:   Wed Dec 17 19:57:52 2025 +0000
    feat(perl): Implement C-first testing and core XS infrastructure
    
    This commit introduces a significant refactoring of the Perl XS extension, adopting a C-first development approach to ensure a robust foundation.
    
    Key changes include:
    
    -   **C-Level Testing Framework:** Established a C-level testing system in  t/c/  with a dedicated Makefile, using an embedded Perl interpreter. Initial tests cover the object cache ( 01-cache.c ), arena wrapper ( 02-arena.c ), and utility functions ( 03-utils.c ).
    -   **Core XS Infrastructure:**
        -   Implemented a global object cache ( xs/protobuf.c ) to manage Perl wrappers for UPB objects, using weak references.
        -   Created an  upb_Arena  wrapper ( xs/protobuf.c ).
        -   Consolidated common XS helper functions into  xs/protobuf.h  and  xs/protobuf.c .
    -   **Makefile.PL Enhancements:** Updated to support building and linking C tests, incorporating flags from  ExtUtils::Embed , and handling both  .c  and  .cc  source files.
    -   **XS File Reorganization:** Restructured XS files to mirror the Python UPB extension's layout (e.g.,  message.c ,  descriptor.c ). Removed older, monolithic  .xs  files.
    -   **Typemap Expansion:** Added extensive typemap entries in  perl/typemap  to handle conversions between Perl objects and various  const upb_*Def*  pointers.
    -   **Descriptor Tests:** Added a new test suite  t/02-descriptor.t  to validate descriptor loading and accessor methods.
    -   **Documentation:** Updated development plans and guidelines ( ProtobufPlan.md ,  xs_learnings.md , etc.) to reflect the C-first strategy, new testing methods, and lessons learned.
    -   **Build Cleanup:** Removed  ppport.h  from  .gitignore  as it's no longer used, due to  -DPERL_NO_PPPORT  being set in  Makefile.PL .
    
    This C-first approach allows for more isolated and reliable testing of the core logic interacting with the UPB library before higher-level Perl APIs are built upon it.
## 0.10	2025-12-17
commit 1ef20ade24603573905cb0376670945f1ab5d829
Author: C.J. Collier 
Date:   Wed Dec 17 07:08:29 2025 +0000
    feat(perl): Implement C-level tests and core XS utils
    
    This commit introduces a C-level testing framework for the XS layer and implements key components:
    
    1.  **C-Level Tests ( t/c/ )**:
        *   Added  t/c/Makefile  to build standalone C tests.
        *   Created  t/c/upb-perl-test.h  with macros for TAP-compliant C tests ( plan ,  ok ,  is ,  is_string ,  diag ).
        *   Implemented  t/c/01-cache.c  to test the object cache.
        *   Implemented  t/c/02-arena.c  to test  Protobuf::Arena  wrappers.
        *   Implemented  t/c/03-utils.c  to test string utility functions.
        *   Corrected include paths and diagnostic messages in C tests.
    
    2.  **XS Object Cache ( xs/protobuf.c )**:
        *   Switched to using stringified pointers ( %p ) as hash keys for stability.
        *   Fixed a critical double-free bug in  PerlUpb_ObjCache_Delete  by removing an extra  SvREFCNT_dec  on the lookup key.
    
    3.  **XS Arena Wrapper ( xs/protobuf.c )**:
        *   Corrected  PerlUpb_Arena_New  to use  newSVrv  and  PTR2IV  for opaque object wrapping.
        *   Corrected  PerlUpb_Arena_Get  to safely unwrap the arena pointer.
    
    4.  **Makefile.PL ( perl/Makefile.PL )**:
        *   Added  -Ixs  to  INC  to allow C tests to find  t/c/upb-perl-test.h  and  xs/protobuf.h .
        *   Added  LIBS  to link  libprotobuf_common.a  into the main  Protobuf.so .
        *   Added C test targets  01-cache ,  02-arena ,  03-utils  to the test config in  MY::postamble .
    
    5.  **Protobuf.pm ( perl/lib/Protobuf.pm )**:
        *   Added  use XSLoader;  to load the compiled XS code.
    
    6.  **New files  xs/util.h **:
        *   Added initial type conversion function.
    
    These changes establish a foundation for testing the C-level interface with UPB and fix crucial bugs in the object cache implementation.
## 0.09	2025-12-17
commit 07d61652b032b32790ca2d3848243f9d75ea98f4
Author: C.J. Collier 
Date:   Wed Dec 17 04:53:34 2025 +0000
    feat(perl): Build system and C cache test for Perl XS
    
    This commit introduces the foundational pieces for the Perl XS implementation, focusing on the build system and a C-level test for the object cache.
    
    -   **Makefile.PL:**
        -   Refactored C test compilation rules in  MY::postamble  to use a hash ( $c_test_config ) for better organization and test-specific flags.
        -   Integrated  ExtUtils::Embed  to provide necessary compiler and linker flags for embedding the Perl interpreter, specifically for the  t/c/01-cache.c  test.
        -   Correctly constructs the path to the versioned Perl library ( libperl.so.X.Y.Z ) using  $Config archlib  and  $Config libperl  to ensure portability.
        -   Removed  VERSION_FROM  and  ABSTRACT_FROM  to avoid dependency on  .pm  files for now.
    
    -   **C Cache Test (t/c/01-cache.c):**
        -   Added a C test to exercise the object cache functions implemented in  xs/protobuf.c .
        -   Includes tests for adding, getting, deleting, and weak reference behavior.
    
    -   **XS Cache Implementation (xs/protobuf.c, xs/protobuf.h):**
        -   Implemented  PerlUpb_ObjCache_Init ,  PerlUpb_ObjCache_Add ,  PerlUpb_ObjCache_Get ,  PerlUpb_ObjCache_Delete , and  PerlUpb_ObjCache_Destroy .
        -   Uses a Perl hash ( HV* ) for the cache.
        -   Keys are string representations of the C pointers, created using  snprintf  with  "%llx" .
        -   Values are weak references ( sv_rvweaken ) to the Perl objects ( SV* ).
        -    PerlUpb_ObjCache_Get  now correctly returns an incremented reference to the original SV, not a copy.
        -    PerlUpb_ObjCache_Destroy  now clears the hash before decrementing its refcount.
    
    -   **t/c/upb-perl-test.h:**
        -   Updated  is_sv  to perform direct pointer comparison ( got == expected ).
    
    -   **Minor:** Added  util.h  (currently empty), updated  typemap .
    
    These changes establish a working C-level test environment for the XS components.
## 0.08	2025-12-17
commit d131fd22ea3ed8158acb9b0b1fe6efd856dc380e
Author: C.J. Collier 
Date:   Wed Dec 17 02:57:48 2025 +0000
    feat(perl): Update docs and core XS files
    
    - Explicitly add TDD cycle to ProtobufPlan.md.
    - Clarify mirroring of Python implementation in upb-interfacing.md for both C and Perl layers.
    - Branch and adapt python/protobuf.h and python/protobuf.c to perl/xs/protobuf.h and perl/xs/protobuf.c, including the object cache implementation. Removed old cache.* files.
    - Create initial C test for the object cache in t/c/01-cache.c.
## 0.07	2025-12-17
commit 56fd6862732c423736a2f9a9fb1a2816fc59e9b0
Author: C.J. Collier 
Date:   Wed Dec 17 01:09:18 2025 +0000
    feat(perl): Align Perl UPB architecture docs with Python
    
    Updates the Perl Protobuf architecture documents to more closely align with the design and implementation strategies used in the Python UPB extension.
    
    Key changes:
    
    -   **Object Caching:** Mandates a global, per-interpreter cache using weak references for all UPB-derived objects, mirroring Python's  PyUpb_ObjCache .
    -   **Descriptor Containers:** Introduces a new document outlining the plan to use generic XS container types (Sequence, ByNameMap, ByNumberMap) with vtables to handle collections of descriptors, similar to Python's  descriptor_containers.c .
    -   **Testing:** Adds a note to the testing strategy to port relevant test cases from the Python implementation to ensure feature parity.
## 0.06	2025-12-17
commit 6009ce6ab64eccce5c48729128e5adf3ef98e9ae
Author: C.J. Collier 
Date:   Wed Dec 17 00:28:20 2025 +0000
    feat(perl): Implement object caching and fix build
    
    This commit introduces several key improvements to the Perl XS build system and core functionality:
    
    1.  **Object Caching:**
        *   Introduces  xs/protobuf.c  and  xs/protobuf.h  to implement a caching mechanism ( protobuf_c_to_perl_obj ) for wrapping UPB C pointers into Perl objects. This uses a hash and weak references to ensure object identity and prevent memory leaks.
        *   Updates the  typemap  to use  protobuf_c_to_perl_obj  for  upb_MessageDef *  output, ensuring descriptor objects are cached.
        *   Corrected  sv_weaken  to the correct  sv_rvweaken  function.
    
    2.  **Makefile.PL Enhancements:**
        *   Switched to using the Bazel-generated UPB descriptor sources from  bazel-bin/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/ .
        *   Updated  INC  paths to correctly locate the generated headers.
        *   Refactored  MY::dynamic_lib  to ensure the static library  libprotobuf_common.a  is correctly linked into each generated  .so  module, resolving undefined symbol errors.
        *   Overrode  MY::test  to use  prove -b -j$(nproc) t/*.t xt/*.t  for running tests.
        *   Cleaned up  LIBS  and  LDDLFLAGS  usage.
    
    3.  **Documentation:**
        *   Updated  ProtobufPlan.md  to reflect the current status and design decisions.
        *   Reorganized architecture documents into subdirectories.
        *   Added  object-caching.md  and  c-perl-interface.md .
        *   Updated  llm-guidance.md  with notes on  upb/upb.h  and  sv_rvweaken .
    
    4.  **Testing:**
        *   Fixed  xt/03-moo_immutable.t  to skip tests if no Moo modules are found.
    
    This resolves the build issues and makes the core test suite pass.
## 0.05	2025-12-16
commit 177d2f3b2608b9d9c415994e076a77d8560423b8
Author: C.J. Collier 
Date:   Tue Dec 16 19:51:36 2025 +0000
    Refactor: Rename namespace to Protobuf, build system and doc updates
    
    This commit refactors the primary namespace from  ProtoBuf  to  Protobuf 
    to align with the style guide. This involves renaming files, directories,
    and updating package names within all Perl and XS files.
    
    **Namespace Changes:**
    
    *   Renamed  perl/lib/ProtoBuf  to  perl/lib/Protobuf .
    *   Moved and updated  ProtoBuf.pm  to  Protobuf.pm .
    *   Moved and updated  ProtoBuf::Descriptor  to  Protobuf::Descriptor  (.pm & .xs).
    *   Removed other  ProtoBuf::*  stubs (Arena, DescriptorPool, Message).
    *   Updated  MODULE  and  PACKAGE  in  Descriptor.xs .
    *   Updated  NAME ,  *_FROM  in  perl/Makefile.PL .
    *   Replaced  ProtoBuf  with  Protobuf  throughout  perl/typemap .
    *   Updated namespaces in test files  t/01-load-protobuf-descriptor.t  and  t/02-descriptor.t .
    *   Updated namespaces in all documentation files under  perl/doc/ .
    *   Updated paths in  perl/.gitignore .
    
    **Build System Enhancements (Makefile.PL):**
    
    *   Included  xs/*.c  files in the common object files list.
    *   Added  -I.  to the  INC  paths.
    *   Switched from  MYEXTLIB  to  LIBS => ['-L$(CURDIR) -lprotobuf_common']  for linking.
    *   Removed custom keys passed to  WriteMakefile  for postamble.
    *    MY::postamble  now sources variables directly from the main script scope.
    *   Added  all :: $ common_lib  dependency in  MY::postamble .
    *   Added  t/c/load_test.c  compilation rule in  MY::postamble .
    *   Updated  clean  target to include  blib .
    *   Added more modules to  TEST_REQUIRES .
    *   Removed the explicit  PM  and  XS  keys from  WriteMakefile , relying on  XSMULTI => 1 .
    
    **New Files:**
    
    *    perl/lib/Protobuf.pm 
    *    perl/lib/Protobuf/Descriptor.pm 
    *    perl/lib/Protobuf/Descriptor.xs 
    *    perl/t/01-load-protobuf-descriptor.t 
    *    perl/t/02-descriptor.t 
    *    perl/t/c/load_test.c : Standalone C test for UPB.
    *    perl/xs/types.c  &  perl/xs/types.h : For Perl/C type conversions.
    *    perl/doc/architecture/upb-interfacing.md 
    *    perl/xt/03-moo_immutable.t : Test for Moo immutability.
    
    **Deletions:**
    
    *   Old test files:  t/00_load.t ,  t/01_basic.t ,  t/02_serialize.t ,  t/03_message.t ,  t/04_descriptor_pool.t ,  t/05_arena.t ,  t/05_message.t .
    *   Removed  lib/ProtoBuf.xs  as it's not needed with  XSMULTI .
    
    **Other:**
    
    *   Updated  test_descriptor.bin  (binary change).
    *   Significant content updates to markdown documentation files in  perl/doc/architecture  and  perl/doc/internal  reflecting the new architecture and learnings.
## 0.04	2025-12-14
commit 92de5d482c8deb9af228f4b5ce31715d3664d6ee
Author: C.J. Collier 
Date:   Sun Dec 14 21:28:19 2025 +0000
    feat(perl): Implement Message object creation and fix lifecycles
    
    This commit introduces the basic structure for  ProtoBuf::Message  object
    creation, linking it with  ProtoBuf::Descriptor  and  ProtoBuf::DescriptorPool ,
    and crucially resolves a SEGV by fixing object lifecycle management.
    
    Key Changes:
    
    1.  ** ProtoBuf::Descriptor :** Added  _pool  attribute to hold a strong
        reference to the parent  ProtoBuf::DescriptorPool . This is essential to
        prevent the pool and its C  upb_DefPool  from being garbage collected
        while a descriptor is still in use.
    
    2.  ** ProtoBuf::DescriptorPool :**
        *    find_message_by_name : Now passes the  $self  (the pool object) to the
             ProtoBuf::Descriptor  constructor to establish the lifecycle link.
        *   XSUB  pb_dp_find_message_by_name : Updated to accept the pool  SV*  and
            store it in the descriptor's  _pool  attribute.
        *   XSUB  _load_serialized_descriptor_set : Renamed to avoid clashing with the
            Perl method name. The Perl wrapper now correctly calls this internal XSUB.
        *    DEMOLISH : Made safer by checking for attribute existence.
    
    3.  ** ProtoBuf::Message :**
        *   Implemented using Moo with lazy builders for  _upb_arena  and
             _upb_message .
        *    _descriptor  is a required argument to  new() .
        *   XS functions added for creating the arena ( pb_msg_create_arena ) and
            the  upb_Message  ( pb_msg_create_upb_message ).
        *    pb_msg_create_upb_message  now extracts the  upb_MessageDef*  from the
            descriptor and uses  upb_MessageDef_MiniTable()  to get the minitable
            for  upb_Message_New() .
        *    DEMOLISH : Added to free the message's arena.
    
    4.  ** Makefile.PL :**
        *   Added  -g  to  CCFLAGS  for debugging symbols.
        *   Added Perl CORE include path to  MY::postamble 's  base_flags .
    
    5.  **Tests:**
        *    t/04_descriptor_pool.t : Updated to check the structure of the
            returned  ProtoBuf::Descriptor .
        *    t/05_message.t : Now uses a descriptor obtained from a real pool to
            test  ProtoBuf::Message->new() .
    
    6.  **Documentation:**
        *   Updated  ProtobufPlan.md  to reflect progress.
        *   Updated several files in  doc/architecture/  to match the current
            implementation details, especially regarding arena management and object
            lifecycles.
        *   Added  doc/internal/development_cycle.md  and  doc/internal/xs_learnings.md .
    
    With these changes, the SEGV is resolved, and message objects can be successfully
    created from descriptors.
## 0.03	2025-12-14
commit 6537ad23e93680c2385e1b571d84ed8dbe2f68e8
Author: C.J. Collier 
Date:   Sun Dec 14 20:23:41 2025 +0000
    Refactor(perl): Object-Oriented DescriptorPool with Moo
    
    This commit refactors the  ProtoBuf::DescriptorPool  to be fully object-oriented using Moo, and resolves several issues related to XS, typemaps, and test data.
    
    Key Changes:
    
    1.  **Moo Object:**  ProtoBuf::DescriptorPool.pm  now uses  Moo  to define the class. The  upb_DefPool  pointer is stored as a lazy attribute  _upb_defpool .
    2.  **XS Lifecycle:**  DescriptorPool.xs  now has  pb_dp_create_pool  called by the Moo builder and  pb_dp_free_pool  called from  DEMOLISH  to manage the  upb_DefPool  lifecycle per object.
    3.  **Typemap:** The  perl/typemap  file has been significantly updated to handle the conversion between the  ProtoBuf::DescriptorPool  Perl object and the  upb_DefPool *  C pointer. This includes:
        *   Mapping  upb_DefPool *  to  T_PTR .
        *   An  INPUT  section for  ProtoBuf::DescriptorPool  to extract the pointer from the object's hash, triggering the lazy builder if needed via  call_method .
        *   An  OUTPUT  section for  upb_DefPool *  to convert the pointer back to a Perl integer, used by the builder.
    4.  **Method Renaming:**  add_file_descriptor_set_binary  is now  load_serialized_descriptor_set .
    5.  **Test Data:**
        *   Added  perl/t/data/test.proto  with a sample message and enum.
        *   Generated  perl/t/data/test_descriptor.bin  using  protoc .
        *   Removed  t/data/  from  .gitignore  to ensure test data is versioned.
    6.  **Test Update:**  t/04_descriptor_pool.t  is updated to use the new OO interface, load the generated descriptor set, and check for message definitions.
    7.  **Build Fixes:**
        *   Corrected  #include  paths in  DescriptorPool.xs  to be relative to the  upb/  directory (e.g.,  upb/wire/decode.h ).
        *   Added  -I../upb  to  CCFLAGS  in  Makefile.PL .
        *   Reordered  INC  paths in  Makefile.PL  to prioritize local headers.
    
    **Note:** While tests now pass in some environments, a SEGV issue persists in  make test  runs, indicating a potential memory or lifecycle issue within the XS layer that needs further investigation.
## 0.02	2025-12-14
commit 6c9a6f1a5f774dae176beff02219f504ea3a6e07
Author: C.J. Collier 
Date:   Sun Dec 14 20:13:09 2025 +0000
    Fix(perl): Correct UPB build integration and generated file handling
    
    This commit resolves several issues to achieve a successful build of the Perl extension:
    
    1.  **Use Bazel Generated Files:** Switched from compiling UPB's stage0 descriptor.upb.c to using the Bazel-generated  descriptor.upb.c  and  descriptor.upb_minitable.c  located in  bazel-bin/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/ .
    2.  **Updated Include Paths:** Added the  bazel-bin  path to  INC  in  WriteMakefile  and to  base_flags  in  MY::postamble  to ensure the generated headers are found during both XS and static library compilation.
    3.  **Removed Stage0:** Removed references to  UPB_STAGE0_DIR  and no longer include headers or source files from  upb/reflection/stage0/ .
    4.  **-fPIC:** Explicitly added  -fPIC  to  CCFLAGS  in  WriteMakefile  and ensured  $(CCFLAGS)  is used in the custom compilation rules in  MY::postamble . This guarantees all object files in the static library are compiled with position-independent code, resolving linker errors when creating the shared objects for the XS modules.
    5.  **Refined UPB Sources:** Used  File::Find  to recursively find UPB C sources, excluding  /conformance/  and  /reflection/stage0/  to avoid conflicts and unnecessary compilations.
    6.  **Arena Constructor:** Modified  ProtoBuf::Arena::pb_arena_new  XSUB to accept the class name argument passed from Perl, making it a proper constructor.
    7.  **.gitignore:** Added patterns to  perl/.gitignore  to ignore generated C files from XS ( lib/*.c ,  lib/ProtoBuf/*.c ), the copied  src_google_protobuf_descriptor.pb.cc , and the  t/data  directory.
    8.  **Build Documentation:** Updated  perl/doc/architecture/upb-build-integration.md  to reflect the new build process, including the Bazel prerequisite, include paths,  -fPIC  usage, and  File::Find .
    
    Build Steps:
    1.   bazel build //src/google/protobuf:descriptor_upb_proto  (from repo root)
    2.   cd perl 
    3.   perl Makefile.PL 
    4.   make 
    5.   make test  (Currently has expected failures due to missing test data implementation).
## 0.01	2025-12-14
commit 3e237e8a26442558c94075766e0d4456daaeb71d
Author: C.J. Collier 
Date:   Sun Dec 14 19:34:28 2025 +0000
    feat(perl): Initialize Perl extension scaffold and build system
    
    This commit introduces the  perl/  directory, laying the groundwork for the Perl Protocol Buffers extension. It includes the essential build files, linters, formatter configurations, and a vendored Devel::PPPort for XS portability.
    
    Key components added:
    
    *   ** Makefile.PL **: The core  ExtUtils::MakeMaker  build script. It's configured to:
        *   Build a static library ( libprotobuf_common.a ) from UPB, UTF8_Range, and generated protobuf C/C++ sources.
        *   Utilize  XSMULTI => 1  to create separate shared objects for  ProtoBuf ,  ProtoBuf::Arena , and  ProtoBuf::DescriptorPool .
        *   Link each XS module against the common static library.
        *   Define custom compilation rules in  MY::postamble  to handle C vs. C++ flags and build the static library.
        *   Set up include paths for the project root, UPB, and other dependencies.
    
    *   **XS Stubs ( .xs  files)**:
        *    lib/ProtoBuf.xs : Placeholder for the main module's XS functions.
        *    lib/ProtoBuf/Arena.xs : XS interface for  upb_Arena  management.
        *    lib/ProtoBuf/DescriptorPool.xs : XS interface for  upb_DefPool  management.
    
    *   **Perl Module Stubs ( .pm  files)**:
        *    lib/ProtoBuf.pm : Main module, loads XS.
        *    lib/ProtoBuf/Arena.pm : Perl class for Arenas.
        *    lib/ProtoBuf/DescriptorPool.pm : Perl class for Descriptor Pools.
        *    lib/ProtoBuf/Message.pm : Base class for messages (TBD).
    
    *   **Configuration Files**:
        *    .gitignore : Ignores build artifacts, editor files, etc.
        *    .perlcriticrc : Configures Perl::Critic for static analysis.
        *    .perltidyrc : Configures perltidy for code formatting.
    
    *   ** Devel::PPPort **: Vendored version 3.72 to generate  ppport.h  for XS compatibility across different Perl versions.
    
    *   ** typemap **: Custom typemap for XS argument/result conversion.
    
    *   **Documentation ( doc/ )**: Initial architecture and plan documents.
    
    This provides a solid foundation for developing the UPB-based Perl extension.

19 December 2025

Otto Kek l inen: Backtesting trailing stop-loss strategies with Python and market data

Featured image of post Backtesting trailing stop-loss strategies with Python and market data

In January 2024 I wrote about the insanity of the Magnificent Seven dominating the MSCI World Index, and I wondered how long the number can continue to go up? It has continued to surge upward at an accelerating pace, which makes me worry that a crash is likely closer. As a software professional, I decided to analyze whether using stop-loss orders could reliably automate avoiding deep drawdowns. As everyone with some savings in the stock market (hopefully) knows, the stock market eventually experiences crashes. It is just a matter of when and how deep the crash will be. Staying on the sidelines for years is not a good investment strategy, as inflation will erode the value of your savings. Assuming the current true inflation rate is around 7%, a restaurant dinner that costs 20 euros today will cost 24.50 euros in three years. Savings of 1000 euros today would drop in purchasing power from 50 dinners to only 40 dinners in three years. Hence, if you intend to retain the value of your hard-earned savings, they need to be invested in something that grows in value. Most people try to beat inflation by buying shares in stable companies, directly or via broad market ETFs. These historically grow faster than inflation during normal years, but likely drop in value during recessions.

What is a trailing stop-loss order? What if you could buy stocks to benefit from their value increasing without having to worry about a potential crash? All modern online stock brokers have a feature called stop-loss, where you can enter a price at which your stocks automatically get sold if they drop down to that price. A trailing stop-loss order is similar, but instead of a fixed price, you enter a margin (e.g. 10%). If the stock price rises, the stop-loss price will trail upwards by that margin. For example, if you buy a share at 100 euros and it has risen to 110 euros, you can set a 10% trailing stop-loss order which automatically sells it if the price drops 10% from the peak of 110 euros, at 99 euros. Thus, no matter what happens, you only lost 1 euro. And if the stock price continues to rise to 150 euros, the trailing stop-loss would automatically readjust to 150 euros minus 10%, which is 135 euros (150-15=135). If the price dropped to 135 euros, you would lock in a gain of 35 euros, which is not the peak price of 150 euros, but still better than whatever the price fell down to as a result of a large crash. In the simple case above, it obviously makes sense in theory, but it might not make sense in practice. Prices constantly oscillate, so you don t want a margin that is too small, otherwise you exit too early. Conversely, having a large margin may result in too large a drawdown before exiting. If markets crash rapidly, it might be that nobody buys your stocks at the stop-loss price, and shares have to be sold at an even lower price. Also, what will you do once the position is sold? The reason you invested in the stock market was to avoid holding cash, so would you buy the same stock back when the crash bottoms? But how will you know when the bottom has been reached?

Backtesting stock market strategies with Python, YFinance, Pandas and Lightweight Charts I am not a professional investor, and nobody should take investment advice from me. However, I know what backtesting is and how to leverage open source software. So, I wrote a Python script to test if the trading strategy of using trailing stop-loss orders with specific margin values would have worked for a particular stock. First you need to have data. YFinance is a handy Python library that can be used to download the historic price data for any stock ticker on Yahoo.com. Then you need to manipulate the data. Pandas is the Python data analysis library with advanced data structures for working with relational or labeled data. Finally, to visualize the results, I used Lightweight Charts, which is a fast, interactive library for rendering financial charts, allowing you to plot the stock price, the trailing stop-loss line, and the points where trades would have occurred. I really like how the zoom is implemented in Lightweight Charts, which makes drilling into the data points feel effortless. The full solution is not polished enough to be published for others to use, but you can piece together your own by reusing some of the key snippets. To avoid re-downloading the same data repeatedly, I implemented a small caching wrapper that saves the data locally (as Parquet files):

python

CACHE_DIR.mkdir(parents=True, exist_ok=True)
end_date = datetime.today().strftime("%Y-%m-%d")
cache_file = CACHE_DIR / f" TICKER - START_DATE -- end_date .parquet"

if cache_file.is_file():
 dataframe = pandas.read_parquet(cache_file)
 print(f"Loaded price data from cache:  cache_file ")
else:
 dataframe = yfinance.download(
 TICKER,
 start=START_DATE,
 end=end_date,
 progress=False,
 auto_adjust=False
 )

 dataframe.to_parquet(cache_file)
 print(f"Fetched new price data from Yahoo Finance and cached to:  cache_file ")

The dataframe is a Pandas object with a powerful API. For example, to print a snippet from the beginning and the end of the dataframe to see what the data looks like, you can use:

python

print("First 5 rows of the raw data:")
print(df.head())
print("Last 5 rows of the raw data:")
print(df.tail())

Example output:

First 5 rows of the raw data
Price Adj Close Close High Low Open Volume
Ticker BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA
Date
2014-01-02 29.956285 55.540001 56.910000 55.349998 56.700001 316552
2014-01-03 30.031801 55.680000 55.990002 55.290001 55.580002 210044
2014-01-06 30.080338 55.770000 56.230000 55.529999 55.560001 185142
2014-01-07 30.943321 57.369999 57.619999 55.790001 55.880001 370397
2014-01-08 31.385597 58.189999 59.209999 57.750000 57.790001 489940
Last 5 rows of the raw data
Price Adj Close Close High Low Open Volume
Ticker BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA BNP.PA
Date
2025-12-11 78.669998 78.669998 78.919998 76.900002 76.919998 357918
2025-12-12 78.089996 78.089996 80.269997 78.089996 79.470001 280477
2025-12-15 79.080002 79.080002 79.449997 78.559998 78.559998 233852
2025-12-16 78.860001 78.860001 79.980003 78.809998 79.430000 283057
2025-12-17 80.080002 80.080002 80.150002 79.080002 79.199997 262818

Adding new columns to the dataframe is easy. For example, I used a custom function to calculate the Relative Strength Index (RSI). To add a new column RSI with a value for every row based on the price from that row, only one line of code is needed, without custom loops:

python

df["RSI"] = compute_rsi(df["price"], period=14)

After manipulating the data, the series can be converted into an array structure and printed as JSON into a placeholder in an HTML template:

python

 baseline_series = [
  "time": ts, "value": val 
 for ts, val in df_plot[["timestamp", BASELINE_LABEL]].itertuples(index=False)
 ]

 baseline_json = json.dumps(baseline_series)
 template = jinja2.Template("template.html")
 rendered_html = template.render(
 title=title,
 heading=heading,
 description=description_html,
 ...
 baseline_json=baseline_json,
 ...
 )

 with open("report.html", "w", encoding="utf-8") as f:
 f.write(rendered_html)
 print("Report generated!")

In the HTML template, the marker variable in Jinja syntax gets replaced with the actual JSON:

html

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <title>  title  </title>
 ...
</head>
<body>
 <h1>  heading  </h1>
 <div id="chart"></div>
 <script>
 // Ensure the DOM is ready before we initialise the chart
 document.addEventListener('DOMContentLoaded', () =>  
 // Parse the JSON data passed from Python
 const baselineData =   baseline_json   safe  ;
 const strategyData =   strategy_json   safe  ;
 const markersData =   markers_json   safe  ;

 // Create the chart
 const chart = LightweightCharts.createChart(document.getElementById('chart'),  
 width: document.getElementById('chart').clientWidth,
 height: 500,
 layout:  
 background:   color: "#222"  ,
 textColor: "#ccc"
  ,
 grid:  
 vertLines:   color: "#555"  ,
 horzLines:   color: "#555"  
  
  );

 // Add baseline series
 const baselineSeries = chart.addLineSeries( 
 title: '  baseline_label  ',
 lastValueVisible: false,
 priceLineVisible: false,
 priceLineWidth: 1
  );
 baselineSeries.setData(baselineData);

 baselineSeries.priceScale().applyOptions( 
 entireTextOnly: true
  );

 // Add strategy series
 const strategySeries = chart.addLineSeries( 
 title: '  strategy_label  ',
 lastValueVisible: false,
 priceLineVisible: false,
 color: '#FF6D00'
  );
 strategySeries.setData(strategyData);

 // Add buy/sell markers to the strategy series
 strategySeries.setMarkers(markersData);

 // Fit the chart to show the full data range (full zoom)
 chart.timeScale().fitContent();
  )
 </script>
</body>
</html>

There are also Python libraries built specifically for backtesting investment strategies, such as Backtrader and Zipline, but they do not seem to be actively maintained, and probably have too many features and complexity compared to what I needed for doing this simple test. The screenshot below shows an example of backtesting a strategy on the Waste Management Inc stock from January 2015 to December 2025. The baseline Buy and hold scenario is shown as the blue line and it fully tracks the stock price, while the orange line shows how the strategy would have performed, with markers for the sells and buys along the way. Backtest run example

Results I experimented with multiple strategies and tested them with various parameters, but I don t think I found a strategy that was consistently and clearly better than just buy-and-hold. It basically boils down to the fact that I was not able to find any way to calculate when the crash has bottomed based on historical data. You can only know in hindsight that the price has stopped dropping and is on a steady path to recovery, but at that point it is already too late to buy in. In my testing, most strategies underperformed buy-and-hold because they sold when the crash started, but bought back after it recovered at a slightly higher price. In particular when using narrow margins and selling on a 3-6% drawdown the strategy performed very badly, as those small dips tend to recover in a few days. Essentially, the strategy was repeating the pattern of selling 100 stocks at a 6% discount, then being able to buy back only 94 shares the next day, then again selling 94 shares at a 6% discount, and only being able to buy back maybe 90 shares after recovery, and so forth, never catching up to the buy-and-hold. The strategy worked better in large market crashes as they tended to last longer, and there were higher chances of buying back the shares while the price was still low. For example, in the 2020 crash selling at a 20% drawdown was a good strategy, as the stock I tested dropped nearly 50% and remained low for several weeks; thus, the strategy bought back the stocks while the price was still low and had not yet started to climb significantly. But that was just a lucky incident, as the delta between the trailing stop-loss margin of 20% and total crash of 50% was large enough. If the crash had been only 25%, the strategy would have missed the rebound and ended up buying back the stocks at a slightly higher price. Also, note that the simulation assumes that the trade itself is too small to affect the price formation. We should keep in mind that in reality, if many people have stop-loss orders in place, a large price drop would trigger all of them, creating a flood of sell orders, which in turn would affect the price and drive it lower even faster and deeper. Luckily, it seems that stop-loss orders are generally not a good strategy, and we don t need to fear that too many people will be using them.

Conclusion Even though using a trailing stop-loss strategy does not seem to help in getting consistently higher returns based on my backtesting, I would still say it is useful in protecting from the downside of stock investing. It can act as a kind of insurance policy to considerably decrease the chances of losing big while increasing the chances of losing a little bit. If you are risk-averse, which I think I probably am, this tradeoff can make sense. I d rather miss out on an initial 50% loss and an overall 3% gain on recovery than have to sit through weeks or months with a 50% loss before the price recovers to prior levels. Most notably, the trailing stop-loss strategy works best if used only once. If it is repeated multiple times, the small losses in gains will compound into big losses overall. Thus, I think I might actually put this automation in place at least on the stocks in my portfolio that have had the highest gains. If they keep going up, I will ride along, but once the crash happens, I will be out of those particular stocks permanently. Do you have a favorite open source investment tool or are you aware of any strategy that actually works? Comment below!

18 December 2025

Colin Watson: Preparing a transition in Debusine

We announced a public beta of Debusine repositories recently (Freexian blog, debian-devel-announce). One thing I m very keen on is being able to use these to prepare transitions : changes to multiple packages that need to be prepared together in order to land in testing. As I said in my DebConf25 talk:

We have distribution-wide CI in unstable, but there s only one of it and it s shared between all of us. As a result it s very possible to get into tangles when multiple people are working on related things at the same time, and we only avoid that as much as we do by careful coordination such as transition bugs. Experimental helps, but again, there s only one of it and setting up another one is far from trivial. So, what we want is a system where you can run experiments on possible Debian changes at a large scale without a high setup cost and without fear of breaking things for other people. And then, if it all works, push the whole lot into Debian.

Time to practice what I preach. Setup The setup process is documented on the Debian wiki. You need to decide whether you re working on a short-lived experiment, in which case you ll run the create-experiment workflow and your workspace will expire after 60 days of inactivity, or something that you expect to keep around for longer, in which case you ll run the create-repository workflow. Either one of those will create a new workspace for you. Then, in that workspace, you run debusine archive suite create for whichever suites you want to use. For the case of a transition that you plan to land in unstable, you ll most likely use create-experiment and then create a single suite with the pattern sid-<name>. The situation I was dealing with here was moving to Pylint 4. Tests showed that we needed this as part of adding Python 3.14 as a supported Python version, and I knew that I was going to need newer upstream versions of the astroid and pylint packages. However, I wasn t quite sure what the fallout of a new major version of pylint was going to be. Fortunately, the Debian Python ecosystem has pretty good autopkgtest coverage, so I thought I d see what Debusine said about it. I created an experiment called cjwatson-pylint (resulting in https://debusine.debian.net/debian/developers-cjwatson-pylint/ - I m not making that a proper link since it will expire in a couple of months) and a sid-pylint suite in it. Iteration From this starting point, the basic cycle involved uploading each package like this for each package I d prepared:

$ dput -O debusine_workspace=developers-cjwatson-pylint \
       -O debusine_workflow=publish-to-sid-pylint \
       debusine.debian.net foo.changes

I could have made a new dput-ng profile to cut down on typing, but it wasn t worth it here. Then I looked at the workflow results, figured out which other packages I needed to fix based on those, and repeated until the whole set looked coherent. Debusine automatically built each upload against whatever else was currently in the repository, as you d expect. I should probably have used version numbers with tilde suffixes (e.g. 4.0.2-1~test1) in case I needed to correct anything, but fortunately that was mostly unnecessary. I did at least run initial test-builds locally of just the individual packages I was directly changing to make sure that they weren t too egregiously broken, just because I usually find it quicker to iterate that way. I didn t take screenshots as I was going along, but here s what the list of top-level workflows in my workspace looked like by the end:

You can see that not all of the workflows are successful. This is because we currently just show everything in every workflow; we don t consider whether a task was retried and succeeded on the second try, or whether there s now a newer version of a reverse-dependency so tests of the older version should be disregarded, and so on. More fundamentally, you have to look through each individual workflow, which is a bit of a pain: we plan to add a dashboard that shows you the current state of a suite as a whole rather than the current workflow-oriented view, but we haven t started on that yet. Drilling down into one of these workflows, it looks something like this: astroid workflow

This was the first package I uploaded. The first pass of failures told me about pylint (expected), pylint-flask (an obvious consequence), and python-sphinx-autodoc2 and sphinx-autoapi (surprises). The slightly odd pattern of failures and errors is because I retried a few things, and we sometimes report retries in a slightly strange way, especially when there are workflows involved that might not be able to resolve their input parameters any more. The next level was: pylint workflow

Again, there were some retries involved here, and also some cases where packages were already failing in unstable so the failures weren t the fault of my change; for now I had to go through and analyze these by hand, but we ll soon have regression tracking to compare with reference runs and show you where things have got better or worse. After excluding those, that left pytest-pylint (not caused by my changes, but I fixed it anyway in unstable to clear out some noise) and spyder. I d seen people talking about spyder on #debian-python recently, so after a bit of conversation there I sponsored a rope upload by Aeliton Silva, upgraded python-lsp-server, and patched spyder. All those went into my repository too, exposing a couple more tests I d forgotten in spyder. Once I was satisfied with the results, I uploaded everything to unstable. The next day, I looked through the tracker as usual starting from astroid, and while there are some test failures showing up right now it looks as though they should all clear out as pieces migrate to testing. Success! Conclusions We still have some way to go before this is a completely smooth experience that I d be prepared to say that every developer can and should be using; there are all sorts of fit-and-finish issues that I can easily see here. Still, I do think we re at the point where a tolerant developer can use this to deal with the common case of a mid-sized transition, and get more out of it than they put in. Without Debusine, either I d have had to put much more effort into searching for and testing reverse-dependencies myself, or (more likely, let s face it) I d have just dumped things into unstable and sorted them out afterwards, resulting in potentially delaying other people s work. This way, everything was done with as little disruption as possible. This works best when the packages likely to be involved have reasonably good autopkgtest coverage (even if the tests themselves are relatively basic). This is an increasingly good bet in Debian, but we have plans to add installability comparisons (similar to how Debian s testing suite works) as well as optional rebuild testing. If this has got you interested, please try it out for yourself and let us know how it goes!

6 December 2025

Simon Josefsson: Reproducible Guix Container Images

Around a year ago I wrote about Guix Container Images for GitLab CI/CD and these images have served the community well. Besides continous use in CI/CD, these Guix container images are used to confirm reproducibility of the source tarball artifacts in the releases of Libtasn1 v4.20, InetUtils v2.6, Libidn2 v2.3.8, Libidn v1.43, SASL v2.2.2, Guile-GnuTLS v5.0.1, and OATH Toolkit v2.6.13. See how all those release announcements mention a Guix commit? That s the essential supply-chain information about the Guix build environment that allows the artifacts to be re-created. To make sure this is repeatable, the release tarball artifacts are re-created from source code every week in the verify-reproducible-artifacts project, that I wrote about earlier. Guix s time travelling feature make this sustainable to maintain, and hopefully will continue to be able to reproduce the exact same tarball artifacts for years to come. During the last year, unfortunately Guix was removed from Debian stable. My Guix container images were created from Debian with that Guix package. My setup continued to work since the old stage0 Debian+Guix containers were still available. Such a setup is not sustainable, as there will be bit-rot and we don t want to rely on old containers forever, which (after the removal of Guix in Debian) could not be re-produced any more. Let this be a reminder how user-empowering features such as Guix time-travelling is! I have reworked my Guix container image setup, and this post is an update on the current status of this effort. The first step was to re-engineer Debian container images with Guix, and I realized these were useful on their own, and warrant a separate project. A more narrowly scoped project makes will hopefully make it easier to keep them working. Now instead of apt-get install guix they use the official Guix guix-install.sh approach. Read more about that effort in the announcement of Debian with Guix. The second step was to reconsider my approach to generate the Guix images. The earlier design had several stages. First, Debian+Guix containers were created. Then from those containers, a pure Guix container was created. Finally, using the pure Guix container another pure Guix container was created. The idea behind that GCC-like approach was to get to reproducible images that were created from an image that had no Debian left on it. However, I never managed to finish this. Partially because I hadn t realized that every time you build a Guix container image from Guix, you effectively go back in time. When using Guix version X to build a container with Guix on it, it will not put Guix version X into the container but will put whatever version of Guix is available in its package archive, which will be an earlier version, such as version X-N. I had hope to overcome this somehow (running a guix pull in newly generated images may work), but never finished this before Guix was removed from Debian. So what could a better design look like? For efficiency, I had already started experimenting with generating the final images directly from the Debian+Guix images, and after reproducibility bugs were fixed I was able to get to reproducible images. However, I was still concerned that the Debian container could taint the process somehow, and was also concerned about the implied dependency on non-free software in Debian. I ve been using comparative rebuilds using similar distributions to confirm artifact reproducibility for my software projects, comparing builds on Trisquel 11 with Ubuntu 22.04, and AlmaLinux 9 with RockyLinux 9 for example. This works surprisingly well. Including one freedom-respecting distribution like Trisquel will detect if any non-free software has bearing on artifacts. Using different architectures, such as amd64 vs arm64 also help with deeper supply-chain concerns. My conclusion was that I wanted containers with the same Guix commit for both Trisquel and Ubuntu. Given the similarity with Debian, adapting and launching the Guix on Trisquel/Debian project was straight forward. So we now have Trisquel 11/12 and Ubuntu 22.04/24.04 images with the same Guix on them. Do you see where the debian-with-guix and guix-on-dpkg projects are leading to? We are now ready to look at the modernized Guix Container Images project. The tags are the same as before:

registry.gitlab.com/debdistutils/guix/container:latest
registry.gitlab.com/debdistutils/guix/container:slim
registry.gitlab.com/debdistutils/guix/container:extra
registry.gitlab.com/debdistutils/guix/container:gash

The method to create them is different. Now there is a build job that uses the earlier Guix+Trisquel container (for amd64) or Guix+Debian (for arm64, pending Trisquel arm64 containers). The build job create the final containers directly. Next a Ubuntu reproduce job is launched that runs the same commands, failing if it cannot generate the bit-by-bit identical container. Then single-arch images are tested (installing/building GNU hello and building libksba), and then pushed to the GitLab registry, adding multi-arch images in the process. Then the final multi-arch containers are tested by building Guile-GnuTLS and, on success, uploaded to the Docker Hub. How would you use them? A small way to start the container is like this:

jas@kaka:~$ podman run -it --privileged --entrypoint=/bin/sh registry.gitlab.com/debdistutils/guix/container:latest
sh-5.2# env HOME=/ guix describe # https://issues.guix.gnu.org/74949
  guix 21ce6b3
    repository URL: https://git.guix.gnu.org/guix.git
    branch: master
    commit: 21ce6b392ace4c4d22543abc41bd7c22596cd6d2
sh-5.2#

The need for --entrypoint=/bin/sh is because Guix s pack command sets up the entry point differently than most other containers. This could probably be fixed if people want that, and there may be open bug reports about this. The need for --privileged is more problematic, but is discussed upstream. The above example works fine without it, but running anything more elaborate with guix-daemon installing packages will trigger a fatal error. Speaking of that, here is a snippet of commands that allow you to install Guix packages in the container.

cp -rL /gnu/store/*profile/etc/* /etc/
echo 'root:x:0:0:root:/:/bin/sh' > /etc/passwd
echo 'root:x:0:' > /etc/group
groupadd --system guixbuild
for i in $(seq -w 1 10); do useradd -g guixbuild -G guixbuild -d /var/empty -s $(command -v nologin) -c "Guix build user $i" --system guixbuilder$i; done
env LANG=C.UTF-8 guix-daemon --build-users-group=guixbuild &
guix archive --authorize < /share/guix/ci.guix.gnu.org.pub
guix archive --authorize < /share/guix/bordeaux.guix.gnu.org.pub
guix install hello
GUIX_PROFILE="/var/guix/profiles/per-user/root/guix-profile"
. "$GUIX_PROFILE/etc/profile"
hello

This could be simplified, but we chose to not hard-code in our containers because some of these are things that probably shouldn t be papered over but fixed properly somehow. In some execution environments, you may need to pass --disable-chroot to guix-daemon. To use the containers to build something in a GitLab pipeline, here is an example snippet:

test-amd64-latest-wget-configure-make-libksba:
  image: registry.gitlab.com/debdistutils/guix/container:latest
  before_script:
  - cp -rL /gnu/store/*profile/etc/* /etc/
  - echo 'root:x:0:0:root:/:/bin/sh' > /etc/passwd
  - echo 'root:x:0:' > /etc/group
  - groupadd --system guixbuild
  - for i in $(seq -w 1 10); do useradd -g guixbuild -G guixbuild -d /var/empty -s $(command -v nologin) -c "Guix build user $i" --system guixbuilder$i; done
  - export HOME=/
  - env LANG=C.UTF-8 guix-daemon --build-users-group=guixbuild &
  - guix archive --authorize < /share/guix/ci.guix.gnu.org.pub
  - guix archive --authorize < /share/guix/bordeaux.guix.gnu.org.pub
  - guix describe
  - guix install libgpg-error
  - GUIX_PROFILE="//.guix-profile"
  - . "$GUIX_PROFILE/etc/profile"
  script:
  - wget https://www.gnupg.org/ftp/gcrypt/libksba/libksba-1.6.7.tar.bz2
  - tar xfa libksba-1.6.7.tar.bz2
  - cd libksba-1.6.7
  - ./configure
  - make V=1
  - make check VERBOSE=t V=1

More help on the project page for the Guix Container Images. That s it for tonight folks, and remember, Happy Hacking!

4 December 2025

Colin Watson: Free software activity in November 2025

My Debian contributions this month were all sponsored by Freexian. I had a bit less time than usual, because Freexian collaborators gathered in Marseille this month for our yearly sprint, doing some planning for next year. You can also support my work directly via Liberapay or GitHub Sponsors. OpenSSH I began preparing for the second stage of the GSS-API key exchange package split (some details have changed since that message). It seems that we ll need to wait until Ubuntu 26.04 LTS has been released, but that s close enough that it s worth making sure we re ready. This month I just did some packaging cleanups that would otherwise have been annoying to copy, such as removing support for direct upgrades from pre-bookworm. I m considering some other package rearrangements to make the split easier to manage, but haven t made any decisions here yet. This also led me to start on a long-overdue bug triage pass, mainly consisting of applying usertags to lots of our open bugs to sort them by which program they apply to, and also closing a few that have been fixed, since some bugs will eventually need to be reassigned to GSS-API packages and it would be helpful to make them easier to find. At the time of writing, about 30% of the bug list remains to be categorized this way. Python packaging I upgraded these packages to new upstream versions:

aioftp
basemap (fixing Cython 3.1 compatibility)
billiard
black
django-phonenumber-field
flask-security
flufl.i18n
langtable
mariadb-connector-python
peewee (fixing Cython 3.1 compatibility)
pydantic
pydantic-core
pydantic-settings
pylsqpack
pymongo
pymssql
pyodbc
python-auditwheel (contributed follow-up fix upstream)
python-avro
python-bleach
python-btrees
python-certifi
python-charset-normalizer
python-colorlog
python-django-channels
python-django-constance
python-django-contact-form
python-django-hashids
python-djvulibre (fixing Cython 3.1 compatibility)
python-evalidate
python-legacy-cgi
python-openstep-plist (fixing Cython 3.1 compatibility)
python-pytest-run-parallel
python-pytokens
python-tblib
python-webargs
pyupgrade
sqlfluff
trove-classifiers
ttconv
wcwidth
zope.hookable
zope.i18nmessageid (removing run-time dependency on setuptools)
zope.interface
zope.proxy
zope.security (removing run-time dependency on setuptools)
zope.sqlalchemy

I packaged django-pgtransaction and backported it to trixie, since we plan to use it in Debusine; and I adopted python-certifi for the Python team. I fixed or helped to fix several other build/test failures:

kivy
mypy
pytest-mypy-testing (contributed upstream)
python-cytoolz
python-discord
python-ncls, fixing some failures in pyranges
pyxdg
traitlets
twine

I fixed a couple of other bugs:

Other bits and pieces

Code reviews

base-passwd: French translation (merged and uploaded)
cdebconf: Consider using the standard dh sequence in d/rules (merged and uploaded)
cdebconf: Multiple select is not right aligned in Hebrew (patch fails to build; asked reporter for an update)
cdebconf: Allow building without libglib2.0-dev (merged and uploaded)
pydantic-core (merged and uploaded)
pymongo (sponsored adoption by Aryan Karamtoth)
python-maturin: Add missing build dependencies (merged and uploaded)

3 December 2025

Reproducible Builds: Reproducible Builds in November 2025

Welcome to the report for November 2025 from the Reproducible Builds project! These monthly reports outline what we ve been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As always, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website. In this report:

10 years of Reproducible Builds at SeaGL 2025 On Friday 8th November, Chris Lamb gave a talk called 10 years of Reproducible Builds at SeaGL in Seattle, WA. Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture. Chris talk:
[ ] introduces the concept of reproducible builds, its technical underpinnings and its potentially transformative impact on software security and transparency. It is aimed at developers, security professionals and policy-makers who are concerned with enhancing trust and accountability in our software. It also provides a history of the Reproducible Builds project, which is approximately ten years old. How are we getting on? What have we got left to do? Aren t all the builds reproducible now?

Distribution work In Debian this month, Jochen Sprickerhof created a merge request to replace the use of reprotest in Debian s Salsa Continuous Integration (CI) pipeline with debrebuild. Jochen cites the advantages as being threefold: firstly, that only one extra build needed ; it uses the same `sbuild` and `ccache` tooling as the normal build ; and works for any Debian release . The merge request was merged by Emmanuel Arias and is now active. kpcyrd posted to our mailing list announcing the initial release of `repro-threshold`, which implements an APT transport that defines a threshold of at least X of my N trusted rebuilders need to confirm they reproduced the binary before installing Debian packages. Configuration can be done through a config file, or through a curses-like user interface. Holger then merged two commits by Jochen Sprickerhof in order to address a `fakeroot`-related reproducibility issue in the `debian-installer`, and J rg Jaspert deployed a patch by Ivo De Decker for a bug originally filed by Holger in February 2025 related to some Debian packages not being archived on snapshot.debian.org. Elsewhere, Roland Clobus performed some analysis on the live Debian trixie images, which he determined were not reproducible. However, in a follow-up post, Roland happily reports that the issues have been handled. In addition, 145 reviews of Debian packages were added, 12 were updated and 15 were removed this month adding to our knowledge about identified issues. Lastly, Jochen Sprickerhof filed a bug announcing their intention to binary NMU a very large number of the R programming language after a reproducibility-related toolchain bug was fixed.
Bernhard M. Wiedemann posted another openSUSE monthly update for their work there.
Julien Malka and Arnout Engelen launched the new hash collection server for NixOS. Aside from improved reporting to help focus reproducible builds efforts within NixOS, it collects build hashes as individually-signed attestations from independent builders, laying the groundwork for further tooling.

Tool development diffoscope version `307` was uploaded to Debian unstable (as well as version `309`). These changes included further attempts to automatically attempt to deploy to PyPI by liaising with the PyPI developers/maintainers (with this experimental feature). [ ][ ][ ] In addition, reprotest versions `0.7.31` and `0.7.32` were uploaded to Debian unstable by Holger Levsen, who also made the following changes:

Do not vary the architecture personality if the kernel is not varied. (Thanks to Ra l Cumplido). [ ]

Drop the `debian/watch` file, as Lintian now flags this as error for native Debian packages. [ ][ ]

Bump `Standards-Version` to 4.7.2, with no changes needed. [ ]

Drop the `Rules-Requires-Root` header as it is no longer required.. [ ]

In addition, however, Vagrant Cascadian fixed a build failure by removing some extra whitespace from an older changelog entry. [ ]

Website updates Once again, there were a number of improvements made to our website this month including:

Bernhard M. Wiedemann updated the `SOURCE_DATE_EPOCH` page to fix the Lisp example syntax. [ ]

Holger Levsen updated a number of pages on our website related to our recent summit in Vienna [ ][ ][ ][ ][ ], and added a link to the YouTube video of his recent talk at Transparency.dev in Gothenburg, Sweden [ ].

James Addison replaced a broken link on the Reproducibility Troubleshooting page with one using Archive.org. [ ]

kpcyrd also updated the Vienna summit page in order to update group picture [ ] as well as to expand the project list [ ].

Robert Stupp added a new Helm page [ ][ ], and fleshed out some Gradle specifics, etc. on the JVM page [ ].

Miscellaneous news

It was noticed that the Comparison of Linux distributions Wikipedia page now has a Reproducible builds column.

The popular Ruby on Rails web development framework had a reproducibility-related test failure due to daylight savings time changes.

Debian Developer Otto Kek l inen appeared on the Open Source Security podcast, relating to their blog post about the XZ backdoor. The video, audio, as well as a full transcript of the show are available on the Open Source Security podcast page for this episode.

Thomas Wei schuh posted to our mailing list in order to look for feedback on their `CONFIG_MODULE_HASHES` patchset for the Linux kernel, which aims to enable reproducible kernel packages for Linux distributions .

kpcyrd also posted our list with a post entitled Github Actions and the hashFiles incident .

Simon Mudd posted to the list as well looking for reproducible RPM building / rebuilding tooling . Simon had watched a recent talk by Holger Levsen and was trying to ensure that he could rebuild various MySQL `.rpms`.

Lastly, there was a thread related to the hosting of the website powering this very report.

Software Supply Chain Security of Web3 Via our mailing list, Martin Monperrus let us know about their recently-published page on the Software Supply Chain Security of Web3. The abstract of their paper is as follows:
Web3 applications, built on blockchain technology, manage billions of dollars in digital assets through decentralized applications (dApps) and smart contracts. These systems rely on complex, software supply chains that introduce significant security vulnerabilities. This paper examines the software supply chain security challenges unique to the Web3 ecosystem, where traditional Web2 software supply chain problems intersect with the immutable and high-stakes nature of blockchain technology. We analyze the threat landscape and propose mitigation strategies to strengthen the security posture of Web3 systems.
Their paper lists reproducible builds as one of the mitigating strategies. A PDF of the full text is available to download.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`SARndbox` (race)

`clamav` (rust toolchain)

`contrast/identity/loupe/mousai` (need glib-macros update)

`cosmic` (cosmic* HashMap)

`dealers-choice` (nocheck)

`falcon` (python-falcon date)

`FreeDoko` (date)

`gnutls` (FTBFS-CPU)

`gods-deluxe` (jar mtimes)

`Kinect` (date)

`libplasma6` (qmlcachegen race)

`llvm` (rocm-omp date)

`rnp` (FTBFS-2041)

`rocsolver` (FTBFS-j1)

`switcheroo` (FTBFS-j1)

`vdrift` (date)

Arnout Engelen:

`ibus` (parallelism)

`qmlcachegen` (with Ulf Hermann)

Chris Lamb:

#1120066 filed against `python-gffutils`.

#1120068 filed against `python-biom-format`.

#1120069 filed against `python-requests-cache`.

#1120070 filed against `python-tld`.

#1120121 filed against `smart-open`.

#1120122 filed against `vanguards`.

#1120123 filed against `pycifrw`.

#1120124 filed against `golang-github-apptainer-container-library-client`.

#1120330 filed against `python-ofxhome`.

#1120331 filed against `python-lupa`.

#1120332 filed against `mu-editor`.

#1120340 filed against `python-spdx-tools`.

#1120342 filed against `python-django-waffle`.

#1120351 filed against `biosquid`.

#1120352 filed against `dateparser`.

#1120353 filed against `parsinsert`.

#1120357 filed against `rdf2rml`.

#1120405 filed against `python-et-xmlfile`.

#1120528 filed against `deblur`.

#1120529 filed against `ytcc`.

#1120530 filed against `pgpainless`.

#1120531 filed against `trillian`.

#1120532 filed against `pywavelets`.

#1120591 filed against `jsonpath-ng`.

#1120592 filed against `presto`.

#1120593 filed against `python-pyutil`.

#1120629 filed against `python-os-apply-config`.

#1120631 filed against `pydata-sphinx-theme`.

#1120632 filed against `python-ciso8601`.

#1120633 filed against `python-pymummer`.

#1120634 filed against `qcat`.

#1120870 filed against `tkgate`.

#1120871 filed against `tkgate`.

#1120872 filed against `ruby-gnuplot`.

#1120873 filed against `python-nixio`.

#1120874 filed against `python-altair`.

#1120875 filed against `python-graphene`.

#1120876 filed against `python-phabricator`.

#1120877 filed against `python-slimmer`.

#1120878 filed against `python-kafka`.

#1120879 filed against `python-sshsig`.

#1120880 filed against `python-babelgladeextractor`.

#1120881 filed against `python-genson`.

#1120882 filed against `flawfinder`.

#1120883 filed against `crasm`.

#1121064 filed against `insilicoseq`.

#1121065 filed against `pychopper`.

#1121066 filed against `pycparser`.

#1121067 filed against `whipper`.

#1121068 filed against `vt`.

#1121069 filed against `pyxnat`.

#1121070 filed against `golang-github-kshedden-statmodel`.

#1121071 filed against `nim-hts`.

#1121072 filed against `golang-github-emicklei-dot`.

#1121073 filed against `golang-gonum-v1-plot`.

#1121074 filed against `beangulp`.

#1121075 filed against `virulencefinder`.

#1121076 filed against `ansible-lint`.

#1121077 filed against `entropybroker`.

#1121078 filed against `namecheap`.

#1121141 filed against `spopt`.

#1121142 filed against `pyasn`.

#1121143 filed against `python-pyvcf`.

#1121147 filed against `python-pysaml2`.

Jochen Sprickerhof:

#1121500 filed against `cython`.

Vagrant Cascadian:

#1121406 filed against `clhep`.

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Mastodon: @reproducible_builds@fosstodon.org

Mailing list: `rb-general@lists.reproducible-builds.org`

Next.

Previous.

Search Results: "zed"

4 January 2026

3 January 2026

1 January 2026

31 December 2025

30 December 2025

Debian (E)LTS This month I have worked 72 hours on Debian Long Term Support (LTS) and on its sister Extended LTS project and did the following things:

23 December 2025

22 December 2025

19 December 2025

18 December 2025

6 December 2025

4 December 2025

3 December 2025

Debian (E)LTS

This month I have worked 72 hours on Debian Long Term Support (LTS) and on its sister Extended LTS project and did the following things: