Review: A Song for a New Day, by Sarah Pinsker

Publisher:	Berkley
Copyright:	September 2019
ISBN:	1-9848-0259-3
Format:	Kindle
Pages:	372

Luce Cannon was touring with a session band when the shutdown began. First came the hotel evacuation in the middle of the night due to bomb threats against every hotel in the state. Then came the stadium bombing just before they were ready to go on stage. Luce and most of the band performed anyway, with a volunteer crew and a shaken crowd. It was, people later decided, the last large stage show in the United States before the congregation laws shut down public gatherings. That was the end of Luce's expected career, and could have been the end of music, or at least public music. But Luce was stubborn and needed the music. Rosemary grew up in the aftermath: living at home with her parents well away from other people, attending school virtually, and then moving seamlessly into a virtual job for Superwally, the corporation that ran essentially everything. A good fix for some last-minute technical problems with StageHoloLive's ticketing system got her an upgraded VR hoodie and complimentary tickets to the first virtual concert she'd ever attended. She found the experience astonishing, prompting her to browse StageHoloLive job openings and then apply for a technical job and, on a whim, an artist recruiter role. That's how Rosemary found herself, quite nerve-wrackingly, traveling out into the unsafe world to look for underground musicians who could become StageHoloLive acts. A Song for a New Day was published in 2019 and had a moment of fame at the beginning of 2020, culminating in the Nebula Award for best novel, because it's about lockdowns, isolation, and the suppression of public performances. There's even a pandemic, although it's not a respiratory disease (it's some variety of smallpox or chicken pox) and is only a minor contributing factor to the lockdowns in this book. The primary impetus is random violence. Unfortunately, the subsequent two years have not been kind to this novel. Reading it in 2022, with the experience of the past two years fresh in my mind, was a frustrating and exasperating experience because the world setting is completely unbelievable. This is not entirely Pinsker's fault; this book was published in 2019, was not intended to be about our pandemic, and therefore could not reasonably predict its consequences. Still, it required significant effort to extract the premise of the book from the contradictory evidence of current affairs and salvage the pieces of it I still enjoyed. First, Pinsker's characters are the most astonishingly incurious and docile group of people I've seen in a recent political SF novel. This extends beyond the protagonists, where it could arguably be part of their characterization, to encompass the entire world (or at least the United States; the rest of the world does not appear in this book at all so far as I can recall). You may be wondering why someone bombs a stadium at the start of the book. If so, you are alone; this is not something anyone else sees any reason to be curious about. Why is random violence spiraling out of control? Is there some coordinated terrorist activity? Is there some social condition that has gotten markedly worse? Race riots? Climate crises? Wars? The only answer this book offers is a completely apathetic shrug. There is a hint at one point that the government may have theories that they're not communicating, but no one cares about that either. That leads to the second bizarre gap: for a book that hinges on political action, formal political structures are weirdly absent. Near the end of the book, one random person says that they have been inspired to run for office, which so far as I can tell is the first mention of elections in the entire book. The "government" passes congregation laws shutting down public gatherings and there are no protests, no arguments, no debate, but also no suppression, no laws against the press or free speech, no attempt to stop that debate. There's no attempt to build consensus for or against the laws, and no noticeable political campaigning. That's because there's no need. So far as one can tell from this story, literally everyone just shrugs and feels sad and vaguely compliant. Police officers exist and enforce laws, but changing those laws or defying them in other than tiny covert ways simply never occurs to anyone. This makes the book read a bit like a fatuous libertarian parody of a docile populace, but this is so obviously not the author's intent that it wouldn't be satisfying to read even as that. To be clear, this is not something that lasts only a few months in an emergency when everyone is still scared. This complete political docility and total incuriosity persists for enough years that Rosemary grows up within that mindset. The triggering event was a stadium bombing followed by an escalating series of random shootings and bombings. (The pandemic in the book only happens after everything is locked down and, apart from adding to Rosemary's agoraphobia and making people inconsistently obsessed with surface cleanliness, plays little role in the novel.) I lived through 9/11 and the Oklahoma City bombing in the US, other countries have been through more protracted and personally dangerous periods of violence (the Troubles come to mind), and never in human history has any country reacted to a shock of violence (or, for that matter, disease) like the US does in this book. At points it felt like one of those SF novels where the author is telling an apparently normal story and all the characters turn out to be aliens based on spiders or bats. I finally made sense of this by deciding that the author wasn't using sudden shocks like terrorism or pandemics as a model, even though that's what the book postulates. Instead, the model seems to be something implicitly tolerated and worked around: US school shootings, for instance, or the (incorrect but widespread) US belief in a rise of child kidnappings by strangers. The societal reaction here looks less like a public health or counter-terrorism response and more like suburban attitudes towards child-raising, where no child is ever left unattended for safety reasons but we routinely have school shootings no other country has at the same scale. We have been willing to radically (and ineffectually) alter the experience of childhood due to fears of external threat, and that's vaguely and superficially similar to the premise of this novel. What I think Pinsker still misses (and which the pandemic has made glaringly obvious) is the immense momentum of normality and the inability of adults to accept limitations on their own activities for very long. Even with school shootings, kids go to school in person. We now know that parts of society essentially collapse if they don't, and political pressure becomes intolerable. But by using school shootings as the model, I managed to view Pinsker's setup as an unrealistic but still potentially interesting SF extrapolation: a thought experiment that ignores countervailing pressures in order to exaggerate one aspect of society to an extreme. This is half of Pinsker's setup. The other half, which made less of a splash because it didn't have the same accident of timing, is the company Superwally: essentially "what if Amazon bought Walmart, Google, Facebook, Netflix, Disney, and Live Nation." This is a more typical SF extrapolation that left me with a few grumbles about realism, but that I'll accept as a plot device to talk about commercialization, monopolies, and surveillance capitalism. But here again, the complete absence of formal political structures in this book is not credible. Superwally achieves an all-pervasiveness that in other SF novels results in corporations taking over the role of national governments, but it still lobbies the government in much the same way and with about the same effectiveness as Amazon does in our world. I thought this directly undermined some parts of the end of the book. I simply did not believe that Superwally would be as benign and ineffectual as it is shown here. Those are a lot of complaints. I found reading the first half of this book to be an utterly miserable experience and only continued reading out of pure stubbornness and completionism. But the combination of the above-mentioned perspective shift and Pinsker's character focus did partly salvage the book for me. This is not a book about practical political change, even though it makes gestures in that direction. It's primarily a book about people, music, and personal connection, and Pinsker's portrayal of individual and community trust in all its complexity is the one thing the book gets right. Rosemary's character combines a sort of naive arrogance with self-justification in a way that I found very off-putting, but the pivot point of the book is the way in which Luce and her community extends trust to her anyway, as part of staying true to what they believe. The problem that I think Pinsker was trying to write about is atomization, which leads to social fragmentation into small trust networks with vast gulfs between them. Luce and Rosemary are both characters who are willing to bridge those gulfs in their own ways. Pinsker does an excellent job describing the benefits, the hurt, the misunderstandings, the risk, and the awkward process of building those bridges between communities that fundamentally do not understand each other. There's something deep here about the nature of solidarity, and how you need both people like Luce and people like Rosemary to build strong and effective communities. I've kept thinking about that part. It's also helpful for a community to have people who are curious about cause and effect, and who know how a bill becomes a law. It's hard to sum up this book, other than to say that I understand why it won a Nebula but it has deep world-building flaws that have become far more obvious over the past two years. Pinsker tries hard to capture the feeling of live music for both the listener and the performer and partly succeeded even for me, which probably means others will enjoy that part of the book immensely. The portrayal of the difficult dynamics of personal trust was the best part of the book for me, but you may have to build scaffolding and bracing for your world-building disbelief in order to get there. On the whole, I think A Song for a New Day is worth reading, but maybe not right now. If you do read it now, tell yourself at the start that this is absolutely not about the pandemic and that everything political in this book is a hugely simplified straw-man extrapolation, and hopefully you'll find the experience less frustrating than I found it. Rating: 6 out of 10

And back to relative normality : the usual suspects are in Cambridge. It's a glorious day across the UK and we're spending it indoors with laptops :)We'll also be releasing a point release of Buster as a wrap up of recent changes.Debian 10 should move from full support to LTS on July August 14th - one year after the release of Debian 11 - and there will be a final point release of Buster somewhere around that point.
All seems to be behaving itself well.Thanks to all for the hard work that goes into preparing each release and especially the security fixes of which there seem to be loads lately.

My current server is somewhat aged. I bought it new in July 2014 with a 6-core Xeon E5-2630L, 32 GB RAM and 4x 3.5 hot-swappable drives. Gladly I had the opportunity to extend the memory to 128 GB RAM at no additional cost by using memory from my ex-employer. It also has 4x 2 TB WD Red HDDs with 5400 rpm hooked up to the SATA backplane, but unfortunately only two of them are SATA-3 with 6 Gbit/s. The new server is a used/refurbished Supermicro server with 2x 14-core Xeon E5-2683 and 256 GB RAM and 4x 3.5 hot-swappable drives. It also came with a Hardware-RAID SAS/SATA 8-port controller with BBU. I also ordered two slim drive kits (MCP-220-81504-0N & MCP-220-81506-0N) to be able to use 2x 3.5 slots for rotational HDDs as a cheap storage. Right now I added 2x 128 GB Supermicro SATA DOMs, 4x WD Red 4 TB SSDs and a Sonnet Fusion 4 4 Silent and 4x 1 TB Seagate Firecuda 520 NVMe disks. And here the issue starts: The NVMe should be capable of 4-5 GB/s, but they are connected to a PCIe 3.0 x16 port via the Sonnet Fusion 4 4, which itself features a PCIe bridge, so bifurbacation is not necessary. When doing some tests with bonnie++ I get around 1 GB/s transfer rates out of a RAID10 setup with all 4 NVMes. In fact, regardless of the RAID level there are only transfer rates of about 1 1.2 GB/s with bonnie++. (All software RAIDs with mdadm.) But also when constructing a RAID each NVMe gives around 300-600 MB/s in sync speed except for one exception: RAID1. Regardless of how many NVMe disks in a RAID1 setup the sync speed is up to 2.5 GB/s for each of the NVMe disks. So the lower transfer rates with bonnie++ or other RAID levels shouldn t be limited by bus speed nor by CPU speed. Alas, atop shows upto 100% CPU usage for all tests. I even tested In my understanding RAID10 should perform similar to RAID1 in terms of syncing and better and while bonnie++ tests (up to 2x write and 4x read speed compared to a single disk). For the bonnie++ tests I even made some tests that are available here. You can find the test parameters listed in the hostname column: Baldur is the hostname, then followed by the layout (near-2, far-2, offset-2), chunk size and concurrency of bonnie++. In the end there was no big impact of the chunk size of the RAID. So, now I m wondering what the reason for the slow performance of those 4x NVMe disks is? Bus speed of the PCIe 3.0 x16 shouldn t be the cause, because I assume that the software RAID will need to transfer the blocks in RAID1 as well as in RAID10 over the bus. Same goes for the CPU: the amount of CPU work should be roughly the same for RAID1 and for RAID10. RAID10 should even have an advantage because the blocks only need to be synced to 2 disks in a stripe set. Bonnie++ tests are a different topic for sure. But when testing reading with dd from the md-devices I only get around 1-1.5 GB/s as well. Even when using LVM RAID instead of LVM on top of md RAID. All NVMe disks are already set to 4k and IO scheduler is set to mq-deadline. Is there anything I could do to improve the performance of the NVMe disks? On the other head, pure transfer rates are not that important to a server that runs a dozen of VMs. Here the improved IOPS performance over rotation disks is a clear performance gain. But I m still curious if I could get maybe 2 GB/s out of a RAID10 setup with the NVMe disks. Then again having two independent RAID1 setups for MariaDB and for PostgreSQL databases might be a better choice over a single RAID10 setup?

I enjoyed reading this article named "22 years of Emacs" recently. It's kind of fascinating, because I realised I don't exactly know for how long I've been using Emacs. It's lost in the mists of history. If I would have to venture a guess, it was back in the "early days", which in that history is mapped around 1996-1997, when I installed my very own "PC" with FreeBSD 2.2.x and painstakingly managed to make XFree86 run on it. Modelines. Those were the days... But I digress.

I am old... The only formal timestamp I can put is that my rebuilt .emacs.d git repository has its first commit in 2002. Some people reading this may be born after that time. This means I'm at least significantly older than those people, to put things gently. Clever history nerds will notice that the commit is obviously fake: Git itself did not exist until 2005. But ah-ah! I was already managing my home directory with CVS in 2001! I converted that repository into git some time in 2009, and therefore you can see all my embarrassing history, including changes from two decades ago. That includes my first known .emacs file which is just bizarre to read right now: 200 lines, most of which are "customize" stuff. Compare with the current, 1000+ lines init.el which is also still kind of a mess, but actually shares very little with the original, thankfully. All this to say that in those years (decades, really) of using Emacs, I have had a very different experience than credmp who wrote packages, sent patches, and got name dropping from other developers. My experience is just struggling to keep up with everything, in general, but also in Emacs.

... and Emacs is too fast for me It might sound odd to say, but Emacs is actually moving pretty fast right now. A lot of new packages are coming out, and I can hardly keep up.

• I am not using org mode, but did use it for time (and task) tracking for a while (and for invoicing too, funky stuff).
• I am not using mu4e, but maybe I'm using something better (notmuch) and yes, I am reading my mail in Emacs, which I find questionable from a security perspective. (Sandboxing untrusted inputs? Anyone?)
• I am using magit, but only when coding, so I do end up using git on the command line quite a bit anyways.
• I do have which-key enabled, and reading about it reminded me I wanted to turn it off because it's kind of noisy and I never remember I can actually use it for anything. Or, in other words, I don't even remember the prefix key or, when I do, there's too many possible commands after for it to be useful.
• I haven't setup lsp-mode, let alone Eglot, which I just learned about reading the article. I thought I would be super shiny and cool by setting up LSP instead of the (dying?) elpy package, but I never got around to it. And now it seems lsp-mode is uncool and I should really do eglot instead, and that doesn't help. UPDATE: I finally got tired and switched to lsp-mode. The main reason for choosing it over eglot is that it's in Debian (and eglot is not). (Apparently, eglot has more chance of being upstreamed, "when it's done", but I guess I'll cross that bridge when I get there.) lsp-mode feels slower than elpy but I haven't done any of the performance tuning and this will improve even more with native compilation (see below). I already had lsp-mode partially setup in Emacs so I only had to do this small tweak to switch and change the prefix key (because s-l or mod is used by my window manager). I also had to pin LSP packages to bookworm here and here.
• I am not using projectile. It's on some of my numerous todo lists somewhere, surely. I suspect it's important to getting my projects organised, but I still live halfway between the terminal and Emacs, so it's not quite clear what I would gain.
• I had to ask what native compilation was or why it mattered the first time I heard of it. And when I saw it again in the article, I had to click through to remember.

Overall, I feel there's a lot of cool stuff in Emacs out there. But I can't quite tell what's the best of which. I can barely remember which completion mechanism I use (company, maybe?) or what makes my mini-buffer completion work the way it does. Everything is lost in piles of customize and .emacs hacks that is constantly changing. Because a lot is in third-party packages, there are often many different options and it's hard to tell which one we should be using.

... or at least fast enough And really, Emacs feels fast enough for me. When I started, I was running Emacs on a Pentium I, 166MHz, with 8MB of RAM (eventually upgraded to 32MB, whoohoo!). Back in those days, the joke was that EMACS was an acronym for "Eight Megs, Always Scratching" and now that I write this down, I realize it's actually "Eight Megs, and Constantly Swapping", which doesn't sound as nice because you could actually hear Emacs running on those old hard drives back in the days. It would make a "scratching" noise as the hard drive heads would scramble maniacally to swap pages in and out of swap to make room for the memory-hungry editor. Now Emacs is pretty far down the list of processes in top(1) regardless of how you look at it. It's using 97MB of resident memory and close to 400MB of virtual memory, which does sound like an awful lot compared to my first computer... But it's absolutely nothing compared to things like Signal-desktop, which somehow manages to map a whopping 20.5GB virtual memory. (That's twenty Gigabytes of memory for old timers or time travelers from the past, and yes, that is now a thing.) I'm not exactly sure how much resident memory it uses (because it forks multiple processes), probably somewhere around 300MB of resident memory. Firefox also uses gigabytes of that good stuff, also spread around the multiple processes, per tab. Emacs "feels" super fast. Typing latency is noticeably better in Emacs than my web browser, and even beats most terminal emulators. It gets a little worse when font-locking is enabled, unfortunately, but it's still feels much better. And all my old stuff still works in Emacs, amazingly. (Good luck with your old Netscape or ICQ configuration from 2000.) I feel like an oldie, using Emacs, but I'm really happy to see younger people using it, and learning it, and especially improving it. If anything, one direction I would like to see it go is closer to what web browsers are doing (yes, I know how bad that sounds) and get better isolation between tasks. An attack on my email client shouldn't be able to edit my Puppet code, and/or all files on my system, for example. And I know, fundamentally, that's a really hard challenge in Emacs. But if you're going to treat your editor as your operating system (or vice versa, I lost track of where we are now that there's an Emacs Window Manager, which I do not use), at least we should get that kind of security. Otherwise I'll have to find a new mail client, and that's really something I try to limit to once a decade or so.

I concluded my previous post about the Purism Librem 5 [1] with the phone working as a Debian/GNOME system with SSH access over the LAN. Before I published that post I managed to render it unbootable, making a new computer unbootable on the first day of owning it isn t uncommon for me. In this case I tried to get SE Linux running on it and changing the kernel commandline parameter security=apparmor to security=selinux caused it to fail the checksum on kernel parameters and halt the boot. That seems to require a fresh install, it seems possible that I could setup my Librem5 to boot a recovery image from a SD card in such situations but that doesn t seem to be well documented and I didn t have any important data to lose. If I do figure out how to recover data by booting from a micro SD card I ll document it. Here s the documentation for reflashing the phone [2], you have to use the --variant luks option for the flashing tool to have an encrypted root filesystem (should default to on to match the default shipping configuration). There is an option --skip-cleanup to allow you to use the same image multiple times, but that probably isn t useful. The image that is available for download today has the latest kernel update that I installed yesterday so it seems that they quickly update the image which makes it convenient to get the latest (dpkg is slow on low power ARM systems). Overall the flash tool is nicely written, does the download and install and instructs you how to get the phone in flashing mode. It is a minor annoyance that the battery has to be removed as part of the flashing process, I will probably end up flashing my phone more often than I want to take the back off the case. A mitigating factor is that the back is well designed and doesn t appear prone to having it s plastic tabs breaking off when removed (as has happened to several other phones I ve owned). The camera doesn t seem to work well at this time, all photos have an unusually low brightness. The audio recording also doesn t work well, speaking clearly into the phone results in quiet recordings. I updated the Debian Wiki page on Mobile devices [3] to include a link to a page about the Librem5 [4] and to also have a section about applications known to work well on mobile devices. Hopefully other people will make some additions to that as most programs in Debian don t work well on mobile devices so we need a list of known good applications as well as applications that can be easily changed to work well. One thing I ve started looking at is the code for the Geary MUA (the default MUA for the Librem5 and the only one in Debian I know to be suitable for a phone). It needs the Thunderbird style autoconfig and it needs the ability to select which IMAP folders to scan as a common practice is to have some large IMAP folders that aren t used on mobile devices. I believe that Android runs each app in a separate UID to prevent them from messing with each other. The configuration on a standard Linux system and on PureOS is to have all apps running with the same permissions, I think this needs to be improved both for phones and for regular Linux systems which will probably benefit more than phones do. I ll write another blog post about this.

• [1] https://etbe.coker.com.au/2022/03/15/librem-5-first-impression/
• [2] https://tinyurl.com/y473wo3h
• [3] https://wiki.debian.org/Mobile
• [4] https://wiki.debian.org/Librem5

I'm happy to announce the Debian Clojure Team will hold a remote sprint from May 13th to May 14th 2022. The goal of this sprint is to improve various aspects of the Clojure ecosystem in Debian. As such, everyone is welcome to participate! Here are a few items we are planning to work on, in no particular order:

• Update leiningen to the latest upstream version, to let some libraries in experimental migrate to unstable.
• Work towards replacing our custom Clojure script with upstream's and package clj clojure-cli.
• Update clojure to the latest upstream version.
• Work on debugging autopkgtest failures on a bunch of puppetlabs-* libraries.
• Work on lintian tags for the Clojure Team.

You can register for the sprint on the Debian Wiki. We are planning to ask the DPL for a food budget. If you plan on joining and want your food to be sponsored, please register before April 2nd.

So, LibrePlanet, the FSF s conference, is coming! I much enjoyed attending this conference in person in March 2018. This year I submitted a talk again, and it got accepted of course, given the conference is still 100% online, I doubt I will be able to go 100% conference-mode (I hope to catch a couple of other talks, but well, we are all eager to go back to how things were before 2020!)

Anyway, what is my talk about? My talk is titled Current challenges for the OpenPGP keyserver network. Is there a way forward?. The abstract I submitted follows:

Many free projects use OpenPGP encryption or signatures for various important tasks, like defining membership, authenticating participation, asserting identity over a vote, etc. The Web-of-Trust upon which its operation is based is a model many of us hold dear, allowing for a decentralized way to assign trust to the identity of a given person. But both the Web-of-Trust model and the software that serves as a basis for the above mentioned uses are at risk due to attacks on the key distribution protocol (not on the software itself!) With this talk, I will try to bring awareness to this situation, to some possible mitigations, and present some proposals to allow for the decentralized model to continue to thrive towards the future.

I am on the third semester of my PhD, trying to somehow keep a decentralized infrastructure for the OpenPGP Web of Trust viable and usable for the future. While this is still in the early stages of my PhD work (and I still don t have a solution to present), I will talk about what the main problems are and will sketch out the path I intend to develop. What is the relevance? Mainly, I think, that many free software projects use the OpenPGP Web of Trust for their identity definitions Are we anachronistic? Are we using tools unfit for this century? I don t think so. I think we are in time to fix the main sore spots for this great example of a decentralized infrastructure.

When is my talk scheduled? This Saturday, 2022.03.19, at

GMT / UTC time

19:25 20:10

Conference schedule time (EDT/GMT-4)

15:25 16:10

Mexico City time (GMT-6)

13:25 14:10

How to watch it? The streams are open online. I will be talking in the Saturn room, feel free to just appear there and watch! The FSF asks people to [register for the conference](https://my.fsf.org/civicrm/event/info?reset=1&id=99) beforehand, in order to be able to have an active participation (i.e. ask questions and that). Of course, you might be interested in other talks take a look at the schedule! LibrePlanet keeps a video archive of their past conferences, and this talk will be linked from there. Of course, I will link to the recording once it is online. Update: As of 2022.03.30, LibrePlanet has posted the videos for all of their talks, all linked from the program. And of course, for convenience, I copied the talk over here: Current challenges for the OpenPGP keyserver network. Is there a way forward?

An even newer hot-fix release 1.0.8.3 of Rcpp follows the 1.0.8.2 release of a few days ago and got to CRAN this morning. A Debian upload will follow shortly, and Windows and macOS binaries will appear at CRAN in the next few days. This release again breaks with the six-months cycle started with release 1.0.5 in July 2020. When we addressed the CRAN request in 1.0.8.2 we forgot to dial testing down to their desired level (as three-part release numbers do automagically for us, whereas four-part do not). This is now taken care of, along with the hot-fix that was in 1.0.8.2 already. Rcpp has become the most popular way of enhancing R with C or C++ code. Right now, around 2522 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 239 in BioConductor. The full list of details for these two interim releases (and hence all changes accumulated since the last regular release, 1.0.8 in January) follows.

Changes in Rcpp hotfix release version 1.0.8.3 (2022-03-14)

• Changes in Rcpp API:
  • Accomodate C++98 compilation by adjusting attributes.cpp (Dirk in #1193 fixing #1192)
  • Accomodate newest compilers replacing deprecated std::unary_function and std::binary_function with std::function (Dirk in #1202 fixing #1201 and CRAN request)
• Changes in Rcpp Documentation:
  • Adjust one overflowing column (Bill Denney in #1196 fixing #1195)
• Changes in Rcpp Deployment:
  • Accomodate four digit version numbers in unit test (Dirk)
  • Do not run complete test suite to limit test time to CRAN preference (Dirk)

Thanks to my CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2843 previous questions. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Every month we review the work funded by Freexian s Debian LTS offering. Please find the report for February below. Debian project funding

• In February Rapha l and the LTS worked on a survey of Debian developers meant to solicit ideas for improvements in the Debian project at large. You can see the results of the initial discussion here in the list of ideas of which there are already over 30.
• The full survey is due to be emailed to Debian Developers shortly.
• In February 2250 was put aside to fund Debian projects.

Debian LTS contributors In February, 12 contributors were paid to work on Debian LTS, their reports are available below. If you re interested in participating in the LTS or ELTS teams, we welcome participation from the Debian community. Simply get in touch with Jeremiah or Rapha l if you are if you are interested in participating.

• Abhijith PA did 10h out of 10h available.
• Anton Gladky did 20h out of 20h available.
• Ben Hutchings did 16h out of 16h available.
• Chris Lamb did 18h out of 18h available.
• Emilio Pozuelo Monfort did 42.75h out of 42.75h available.
• Jeremiah Foster worked 20 hours out of 20 available on LTS administration and 2.9 hours on funded projects.
• Markus Koschany did 30h (out of 40h available), thus carrying over 10h to March.
• Roberto C. S nchez did 12h out of 12h available.
• Sylvain Beucler did 13h (out of 40h available), thus carrying over 27h to March.
• Thorsten Alteholz did 40h out of 40h available.
• Utkarsh Gupta did 15.75h (out of 42.75h available), thus carrying over 27h to March.

Evolution of the situation In February we released 24 DLAs. The security tracker currently lists 61 packages with a known CVE and the dla-needed.txt file has 26 packages needing an update. You can find out more about the Debian LTS project via the following video:

Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • TOSHIBA (for 78 months)
  • GitHub (for 68 months)
  • Civil Infrastructure Platform (CIP) (for 46 months)
• Gold sponsors:
  • Roche Diagnostics International AG (for 89 months)
  • Linode (for 83 months)
  • Babiel GmbH (for 72 months)
  • Plat Home (for 71 months)
  • University of Oxford (for 28 months)
  • Deveryware (for 15 months)
  • VyOS Inc (for 10 months)
• Silver sponsors:
  • The Positive Internet Company (for 94 months)
  • Domeneshop AS (for 93 months)
  • Nantes M tropole (for 87 months)
  • Univention GmbH (for 79 months)
  • Universit Jean Monnet de St Etienne (for 79 months)
  • Ribbon Communications, Inc. (for 73 months)
  • Exonet B.V. (for 63 months)
  • Leibniz Rechenzentrum (for 57 months)
  • CINECA (for 46 months)
  • Minist re de l Europe et des Affaires trang res (for 40 months)
  • Cloudways Ltd (for 29 months)
  • Dinahosting SL (for 27 months)
  • Platform.sh (for 22 months)
  • Bauer Xcel Media Deutschland KG (for 21 months)
  • Moxa Intelligence Co., Ltd. (for 16 months)
  • sipgate GmbH (for 13 months)
  • OVH US LLC (for 11 months)
  • Tilburg University (for 11 months)
  • GSI Helmholtzzentrum f r Schwerionenforschung GmbH
  • Telecats BV
  • Soliton Systems K.K.
• Bronze sponsors:
  • Evolix (for 94 months)
  • Seznam.cz, a.s. (for 94 months)
  • Linuxhotel GmbH (for 91 months)
  • Intevation GmbH (for 90 months)
  • Daevel SARL (for 89 months)
  • Bitfolk LTD (for 88 months)
  • Megaspace Internet Services GmbH (for 88 months)
  • Greenbone Networks GmbH (for 87 months)
  • NUMLOG (for 87 months)
  • WinGo AG (for 86 months)
  • Ecole Centrale de Nantes LHEEA (for 83 months)
  • Entr ouvert (for 78 months)
  • Adfinis AG (for 75 months)
  • Laboratoire LEGI UMR 5519 / CNRS (for 70 months)
  • Tesorion (for 70 months)
  • GNI MEDIA (for 69 months)
  • Bearstech (for 61 months)
  • LiHAS (for 61 months)
  • People Doc (for 57 months)
  • Catalyst IT Ltd (for 56 months)
  • Supagro (for 51 months)
  • Demarcq SAS (for 50 months)
  • Universit Grenoble Alpes (for 36 months)
  • TouchWeb SAS (for 28 months)
  • SPiN AG (for 24 months)
  • CoreFiling (for 20 months)
  • Institut des sciences cognitives Marc Jeannerod (for 15 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 11 months)
  • Tem Innovations GmbH (for 6 months)
  • WordFinder.pro (for 6 months)
  • CNRS DT INSU R sif (for 4 months)

Recently I had a look at a Hargassner wood chip boiler, and what kind of free software can be used to monitor and control it. The boiler can be connected to some cloud service via what the producer call an Internet Gateway, which seem to be a computer connecting to the boiler and passing the information gathered to the cloud. I discovered the boiler controller got an IP address on the local network and listen on TCP port 23 to provide status information as a text line of numbers. It also provide a HTTP server listening on port 80, but I have not yet figured out what it can do beside return an error code. If I am to believe various free software implementations talking to such boiler, the interpretation of the line of numbers differ between type of boiler and software version on the boiler. By comparing the list of numbers on the front panel of the boiler with the numbers returned via TCP, I have been able to figure out several of the numbers, but there are a lot left to understand. I've located several temperature measurements and hours running values, as well as oxygen measurements and counters. I decided to write a simple parser in Python for the values I figured out so far, and a simple MQTT injector publishing both the interpreted and the unknown values on a MQTT bus to make collecting and graphing simpler. The end result is available from the hargassner2mqtt project page on gitlab. I very much welcome patches extending the parser to understand more values, boiler types and software versions. I do not really expect very few free software developers got their hands on such unit to experiment, but it would be fun if others too find this project useful. As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

A new hot-fix release 1.0.8.2 of Rcpp just got to CRAN. It will also be uploaded to Debian shortly, and Windows and macOS binaries will appear at CRAN in the next few days. This release breaks with the six-months cycle started with release 1.0.5 in July 2020 as CRAN desired an update to silence nags from the newest clang version which turned a little loud over a feature deprecated in C++11 (namely std::unary_function() and std::binary_function()). This was easy to replace with std::function() which we did. The release also contains a minor bugfix relative to 1.0.8 and C++98 builds, and minor correction to one pdf vignette. The release was fully tested by us and CRAN as usual against all reverse dependencies. Rcpp has become the most popular way of enhancing R with C or C++ code. Right now, around 2519 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 239 in BioConductor. The full list of details for this interim release follows.

Changes in Rcpp hotfix release version 1.0.8.2 (2022-03-10)

• Changes in Rcpp API:
  • Accomodate C++98 compilation by adjusting attributes.cpp (Dirk in #1193 fixing #1192)
  • Accomodate newest compilers replacing deprecated std::unary_function and std::binary_function with std::function (Dirk in #1202 fixing #1201 and CRAN request)
• Changes in Rcpp Documentation:
  • Adjust one overflowing column (Bill Denney in #1196 fixing #1195)
• Changes in Rcpp Deployment:
  • Accomodate four digit version numbers in unit test (Dirk)

Thanks to my CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2843 previous questions. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Wait! Just like yesterday when I was accepted as an Outreachy intern and the first half of the internship is finished . How time flies when you are having a good time As part of the requirements for the final application during the contribution period for the Outreachy internship, I needed to provide a timeline to achieve our goal on my outreachy task which is transitioning of dependencies in node16 and webpack5. Having consulted my mentors who implied that the packages depending on webpack and nodejs combined are so numerous that its impossible to finish all within a space of three months but we have steps to guide us through the entire process to achieve most of our goals which are

• Find a list of packages(failing rebuild and testing with autopkgtest, of reverse dependencies of webpack and nodejs) to fix.
• See if new upstream versions are available that support nodejs16 and webpack5 respectively.
• See if the new upstream version works and doesn't fail while rebuilding or testing with autopkgtest.
• Report bug in Debian if any fails to rebuild or test with autopkgtest.
• Forward bugs upstream if needed.
• Fix packages and forward patches.

As of this writing(though a little late ) we have successfully rebuilt all reverse dependencies of webpack5 and split them equally each for I and my co-intern for all Javascript modules as ruby packages also depend on webpack which is a total of 44 packages. Filed a bug report on Debian bug tracking system for failing packages, also the original maintainer or uploader of the package to the Debian archive mostly Debian developers also get a mail in references to the package bug report. Sometimes the uploader who also receives the bug report decides to help out to fix the package and forward the patch upstream if need be. We have also filed an issue to upstream repo mostly via github where some respond and create PR to solve those errors and others are plain aversive to the whole idea. PR from the upstream developer is cherry-picked and a patch is created by us to incorporate the code into our own working repository. some package upstream maintainer rejects such issues or doesn't respond, we take it upon ourselves to fix the package. The total number of packages that are successfully updated and ready to be merged is 10 packages while 12 packages remain on my own end to be updated. One of the most challenging packages to update so far was prop-type as its runs its large test suite using jest of a lower version 19.0.2 compared to that of Debian OS which is version 27.5.1 updating and migrating its API's and methods to use the Debian updated version is so challenging after several googling, testing out the solution from StackOverflow, trials, and errors, reading documentations we eventually made progress with the help of my mentor, co-intern and the whole community member. It's so crazy that when I got it working I said to myself. phew it's not rocket science why can I figure it out sooner than expected I initially proposed that I would be halfway done with the project by now, I guess the reason am not able to achieve some of our goals which are finishing up with the packages for webpack and moving to transition some of the nodejs packages at all is DEBUGGING. Yes DEBUGGING! where you never can predict what the solution is. is the problem coming from Debian? or dependencies of the package you are working on, upstream bug, or dependencies of dependencies of the package you are working on, so many questions to answer. You can't easily find a solution to a bug as it takes time to try out so many guesses more of an educated guess, or even try out all the solutions from stack overflow and still no viable progress. Obviously, you cannot really know about something to set up a plan for unless you get right into it. One way of doing this, if I have to start again is the truly understand how the javascript package work under the hood, how its handles different interaction between packages, some of its dos and don't of transpiling, bundling, testing, e.t.c I guess my unrealistic goals need to be modified because some drawback that was not envisaged popped up and I underestimated the complexity of the tasks, which will be reducing the number of packages to update in transitioning of nodejs from what I planned My major focus for the second half of the internship is to fix bugs and errors I discover, file bug reports for future bugs to seek help from co-maintainer or developers, file issues upstream and close those whose bugs are already resolved for the remaining 12 packages, and ultimately successfully uploading all reverse dependencies. Also diving into transitioning of nodejs16. Thanks for stopping by

Leonardo and I are pleased to another update to our nanotime package bringing it to version 0.3.6 which landed on CRAN earlier today. nanotime relies on the RcppCCTZ package (as well as the RcppDate package for additional C++ operations) and offers efficient high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from a rigorous refactoring by Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations. This release corrects subsetting with %in% operator, integrates it better fit in the mixed S3/S4 setup, fixes a negative period parse, and updates class comparisons to rely on inherits(). The NEWS snippet has the full more details.

Changes in version 0.3.6 (2022-03-06)

• Fix incorrect subsetting with operator %in% (Leonardo in #100 fixing #99).
• Fix incorrect parsing for negative nanoperiod (Leonardo in #100 fixing #96).
• Test for class via inherits() (Dirk).

Thanks to my CRANberries there is also a diff to the previous version. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository and all documentation is provided at the nanotime documentation site. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Welcome to the February 2022 report from the Reproducible Builds project. In these reports, we try to round-up the important things we and others have been up to over the past month. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

---

Jiawen Xiong, Yong Shi, Boyuan Chen, Filipe R. Cogo and Zhen Ming Jiang have published a new paper titled Towards Build Verifiability for Java-based Systems (PDF). The abstract of the paper contains the following:

Various efforts towards build verifiability have been made to C/C++-based systems, yet the techniques for Java-based systems are not systematic and are often specific to a particular build tool (eg. Maven). In this study, we present a systematic approach towards build verifiability on Java-based systems.

---

GitBOM is a flexible scheme to track the source code used to generate build artifacts via Git-like unique identifiers. Although the project has been active for a while, the community around GitBOM has now started running weekly community meetings.

---

The paper Chris Lamb and Stefano Zacchiroli is now available in the March/April 2022 issue of IEEE Software. Titled Reproducible Builds: Increasing the Integrity of Software Supply Chains (PDF), the abstract of the paper contains the following:

We first define the problem, and then provide insight into the challenges of making real-world software build in a reproducible manner-this is, when every build generates bit-for-bit identical results. Through the experience of the Reproducible Builds project making the Debian Linux distribution reproducible, we also describe the affinity between reproducibility and quality assurance (QA).

---

In openSUSE, Bernhard M. Wiedemann posted his monthly reproducible builds status report.

---

On our mailing list this month, Thomas Schmitt started a thread around the SOURCE_DATE_EPOCH specification related to formats that cannot help embedding potentially timezone-specific timestamp. (Full thread index.)

---

The Yocto Project is pleased to report that it s core metadata (OpenEmbedded-Core) is now reproducible for all recipes (100% coverage) after issues with newer languages such as Golang were resolved. This was announced in their recent Year in Review publication. It is of particular interest for security updates so that systems can have specific components updated but reducing the risk of other unintended changes and making the sections of the system changing very clear for audit. The project is now also making heavy use of equivalence of build output to determine whether further items in builds need to be rebuilt or whether cached previously built items can be used. As mentioned in the article above, there are now public servers sharing this equivalence information. Reproducibility is key in making this possible and effective to reduce build times/costs/resource usage.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 203, 204, 205 and 206 to Debian unstable, as well as made the following changes to the code itself:

• Bug fixes:
  • Fix a file(1)-related regression where Debian .changes files that contained non-ASCII text were not identified as such, therefore resulting in seemingly arbitrary packages not actually comparing the nested files themselves. The non-ASCII parts were typically in the Maintainer or in the changelog text. [ ][ ]
  • Fix a regression when comparing directories against non-directories. [ ][ ]
  • If we fail to scan using binwalk, return False from BinwalkFile.recognizes. [ ]
  • If we fail to import binwalk, don t report that we are missing the Python rpm module! [ ]
• Testsuite improvements:
  • Add a test for recent file(1) issue regarding .changes files. [ ]
  • Use our assert_diff utility where we can within the test_directory.py set of tests. [ ]
  • Don t run our binwalk-related tests as root or fakeroot. The latest version of binwalk has some new security protection against this. [ ]
• Codebase improvements:
  • Drop the _PATH suffix from module-level globals that are not paths. [ ]
  • Tidy some control flow in Difference._reverse_self. [ ]
  • Don t print a warning to the console regarding NT_GNU_BUILD_ID changes. [ ]

In addition, Mattia Rizzolo updated the Debian packaging to ensure that diffoscope and diffoscope-minimal packages have the same version. [ ]

Debian-related updates Vagrant Cascadian wrote to the debian-devel mailing list after noticing that the binutils source package contained unreproducible logs in one of its binary packages. Vagrant expanded the discussion to one about all kinds of build metadata in packages and outlines a number of potential solutions that support reproducible builds and arbitrary metadata. Vagrant also started a discussion on debian-devel after identifying a large number of packages that embed build paths via RPATH when building with CMake, including a list of packages (grouped by Debian maintainer) affected by this issue. Maintainers were requested to check whether their package still builds correctly when passing the -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON directive. On our mailing list this month, kpcyrd announced the release of rebuilderd-debian-buildinfo-crawler a tool to parse the Packages.xz Debian package index file, attempts to discover the right .buildinfo file from buildinfos.debian.net and outputs it in a format that can be understood by rebuilderd. The tool, which is available on GitHub, solves a problem regarding correlating Debian version numbers with their builds. bauen1 provided two patches for debian-cd, the software used to make Debian installer images. This involved passing --invariant and -i deb00001 to mkfs.msdos(8) and avoided embedding timestamps into the gzipped Packages and Translations files. After some discussion, the patches in question were merged and will be included in debian-cd version 3.1.36. Roland Clobus wrote another in-depth status update about status of live Debian images, summarising the current situation that all major desktops build reproducibly with bullseye, bookworm and sid . The python3.10 package was uploaded to Debian by doko, fixing an issue where [.pyc files were not reproducible because the elements in frozenset data structures were not ordered reproducibly. This meant that to creating a bit-for-bit reproducible Debian chroot which included .pyc files was not reproducible. As of writing, the only remaining unreproducible parts of a standard chroot is man-db, but Guillem Jover has a patch for update-alternatives which will likely be part of the next release of dpkg. Elsewhere in Debian, 139 reviews of Debian packages were added, 29 were updated and 17 were removed this month adding to our knowledge about identified issues. A large number of issue types have been updated too, including the addition of captures_kernel_variant, erlang_escript_file, captures_build_path_in_r_rdb_rds_databases, captures_build_path_in_vo_files_generated_by_coq and build_path_in_vo_files_generated_by_coq.

Website updates There were quite a few changes to the Reproducible Builds website and documentation this month as well, including:

• Chris Lamb:
  • Considerably rework the Who is involved? page. [ ][ ]
  • Move the contributors.sh Bash/shell script into a Python script. [ ][ ][ ]
• Daniel Shahaf:
  • Try a different Markdown footnote content syntax to work around a rendering issue. [ ][ ][ ]
• Holger Levsen:
  • Make a huge number of changes to the Who is involved? page, including pre-populating a large number of contributors who cannot be identified from the metadata of the website itself. [ ][ ][ ][ ][ ]
  • Improve linking to sponsors in sidebar navigation. [ ]
  • drop sponsors paragraph as the navigation is clearer now. [ ]
  • Add Mullvad VPN as a bronze-level sponsor . [ ][ ]
• Vagrant Cascadian:
  • Remove a stray parenthesis from the Who is involved? page. [ ]

Upstream patches The Reproducible Builds project attempts to fix as many currently-unreproducible packages as possible. February s patches included the following:

• Bernhard M. Wiedemann:
  • btop (sort-related issue)
  • complexity (date)
  • giac (update the version with upstreamed date patch)
  • htcondor (use CMake timestamp)
  • libint (readdir system call related)
  • libnet (date-related issue)
  • librime-lua (sort filesystem ordering)
  • linux_logo (sort-related issue)
  • micro-editor (date-related issue)
  • openvas-smb (date-related issue)
  • ovmf (sort-related issue)
  • paperjam (date-related issue)
  • python-PyQRCode (date-related issue)
  • quimb (single-CPU build failure)
  • radare2 (Meson date/time-related issue)
  • radare2 (Rework SOURCE_DATE_EPOCH usage to be portable)
  • siproxd (date, with Sebastian Kemper + follow-up
  • xonsh (Address Space Layout Randomisation-related issue)
  • xsnow (date & tar(1)-related issue)
  • zip (toolchain issue related to filesystem ordering)
• Chris Lamb:
  • #1005029 filed against ltsp (forwarded upstream).
  • #1005197 filed against pcmemtest.
  • #1005825 filed against hatchling.
  • #1005826 filed against mpl-sphinx-theme (forwarded upstream)
  • #1005827 filed against gap-hapcryst.
  • #1005901 filed against tree-puzzle.
  • #1005954 filed against jcabi-aspects.
  • #1005955 filed against paper-icon-theme.
• Roland Clobus:
  • #1006358 filed against libxmlb.
• Vagrant Cascadian:
  • #1005408 filed against wcwidth.
  • #1005420 filed against xir.
  • #1005421 filed against xir.
  • #1005726 filed against ruby-github-markup.
  • #1005727 filed against ruby-tioga.
  • #1005792 filed against btop.
  • #1005793 filed against libadwaita-1.
  • #1005794 filed against snibbetracker.
  • #1006252 filed against cctbx.
  • #1006254 filed against mdnsd.
  • #1006256 filed against gmerlin.
  • #1006302 filed against beav.
  • #1006385 filed against krita.
  • #1006407 filed against qt6-base.
  • #1006455 filed against onevpl-intel-gpu.
  • #1006471 filed against ruby3.0.
  • #1006473 filed against nix.
  • #1006474 filed against foma.
  • #1006476 filed against ruby3.0.

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:

• Daniel Golle:
  • Update the OpenWrt configuration to not depend on the host LLVM, adding lines to the .config seed to build LLVM for eBPF from source. [ ]
  • Preserve more OpenWrt-related build artifacts. [ ]
• Holger Levsen:
• Temporary use a different Git tree when building OpenWrt as our tests had been broken since September 2020. This was reverted after the patch in question was accepted by Paul Spooren into the canonical openwrt.git repository the next day.
  • Various improvements to debugging OpenWrt reproducibility. [ ][ ][ ][ ][ ]
  • Ignore useradd warnings when building packages. [ ]
  • Update the script to powercycle armhf architecture nodes to add a hint to where nodes named virt-*. [ ]
  • Update the node health check to also fix failed logrotate and man-db services. [ ]
• Mattia Rizzolo:
  • Update the website job after contributors.sh script was rewritten in Python. [ ]
  • Make sure to set the DIFFOSCOPE environment variable when available. [ ]
• Vagrant Cascadian:
  • Various updates to the diffoscope timeouts. [ ][ ][ ]

Node maintenance was also performed by Holger Levsen [ ] and Vagrant Cascadian [ ].

Finally If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

I bought myself a new keyboard last November, a Logitech G213. True keyboard fans will tell me it s not a real mechanical keyboard, but it was a lot cheaper and met my requirements of having some backlighting and a few media keys (really all I use are the volume control keys). Oh, and being a proper UK layout. While the G213 isn t fully independent RGB per key it does have a set of zones that can be controlled. Also this has been reverse engineered, so there are tools to do this under Linux. All I really wanted was some basic backlighting to make things a bit nicer in the evenings, but with the ability to control colour I felt I should put it to good use. As previously mentioned I have a personal desktop / work laptop setup combined with a UGREEN USB 3.0 Sharing Switch Box, so the keyboard is shared between both machines. So I configured up both machines to set the keyboard colour when the USB device is plugged in, and told them to use different colours. Instant visual indication of which machine I m currently typing on! Running the script on USB detection is easy, a file in /etc/udev/rules.d/. I called it 99-keyboard-colour.rules: g213-set is a simple bit of Python: I did wonder about trying to make it turn red when I m in a root terminal, but that gets a bit more complicated (I m guessing I need to hook into GNOME Terminal some how?) and this simple hack gives me a significant win anyway.

TL;DR: procmail is a security liability and has been abandoned upstream for the last two decades. If you are still using it, you should probably drop everything and at least remove its SUID flag. There are plenty of alternatives to chose from, and conversion is a one-time, acceptable trade-off.

Procmail is unmaintained procmail is unmaintained. The "Final release", according to Wikipedia, dates back to September 10, 2001 (3.22). That release was shipped in Debian since then, all the way back from Debian 3.0 "woody", twenty years ago. Debian also ships 25 uploads on top of this, with 3.22-21 shipping the "3.23pre" release that has been rumored since at least the November 2001, according to debian/changelog at least:

procmail (3.22-1) unstable; urgency=low
  * New upstream release, which uses the  standard' format for Maildir
    filenames and retries on name collision. It also contains some
    bug fixes from the 3.23pre snapshot dated 2001-09-13.
  * Removed  sendmail' from the Recommends field, since we already
    have  exim' (the default Debian MTA) and  mail-transport-agent'.
  * Removed suidmanager support. Conflicts: suidmanager (<< 0.50).
  * Added support for DEB_BUILD_OPTIONS in the source package.
  * README.Maildir: Do not use locking on the example recipe,
    since it's wrong to do so in this case.
 -- Santiago Vila <sanvila@debian.org>  Wed, 21 Nov 2001 09:40:20 +0100

All Debian suites from buster onwards ship the 3.22-26 release, although the maintainer just pushed a 3.22-27 release to fix a seven year old null pointer dereference, after this article was drafted. Procmail is also shipped in all major distributions: Fedora and its derivatives, Debian derivatives, Gentoo, Arch, FreeBSD, OpenBSD. We all seem to be ignoring this problem. The upstream website (http://procmail.org/) has been down since about 2015, according to Debian bug #805864, with no change since. In effect, every distribution is currently maintaining its fork of this dead program. Note that, after filing a bug to keep Debian from shipping procmail in a stable release again, I was told that the Debian maintainer is apparently in contact with the upstream. And, surprise! they still plan to release that fabled 3.23 release, which has been now in "pre-release" for all those twenty years. In fact, it turns out that 3.23 is considered released already, and that the procmail author actually pushed a 3.24 release, codenamed "Two decades of fixes". That amounts to 25 commits since 3.23pre some of which address serious security issues, but none of which address fundamental issues with the code base.

Procmail is insecure By default, procmail is installed SUID root:mail in Debian. There's no debconf or pre-seed setting that can change this. There has been two bug reports against the Debian to make this configurable (298058, 264011), but both were closed to say that, basically, you should use dpkg-statoverride to change the permissions on the binary. So if anything, you should immediately run this command on any host that you have procmail installed on:

dpkg-statoverride --update --add root root 0755 /usr/bin/procmail

Note that this might break email delivery. It might also not work at all, thanks to usrmerge. Not sure. Yes, everything is on fire. This is fine. In my opinion, even assuming we keep procmail in Debian, that default should be reversed. It should be up to people installing procmail to assign it those dangerous permissions, after careful consideration of the risk involved. The last maintainer of procmail explicitly advised us (in that null pointer dereference bug) and other projects (e.g. OpenBSD, in [2]) to stop shipping it, back in 2014. Quote:

Executive summary: delete the procmail port; the code is not safe and should not be used as a basis for any further work.

I just read some of the code again this morning, after the original author claimed that procmail was active again. It's still littered with bizarre macros like:

#define bit_set(name,which,value) \
  (value?(name[bit_index(which)] =bit_mask(which)):\
  (name[bit_index(which)]&=~bit_mask(which)))

... from regexp.c, line 66 (yes, that's a custom regex engine). Or this one:

#define jj  (aleps.au.sopc)

It uses insecure functions like strcpy extensively. malloc() is thrown around gotos like it's 1984 all over again. (To be fair, it has been feeling like 1984 a lot lately, but that's another matter entirely.) That null pointer deref bug? It's fixed upstream now, in this commit merged a few hours ago, which I presume might be in response to my request to remove procmail from Debian. So while that's nice, this is the just tip of the iceberg. I speculate that one could easily find an exploitable crash in procmail if only by running it through a fuzzer. But I don't need to speculate: procmail had, for years, serious security issues that could possibly lead to root privilege escalation, remotely exploitable if procmail is (as it's designed to do) exposed to the network. Maybe I'm overreacting. Maybe the procmail author will go through the code base and do a proper rewrite. But I don't think that's what is in the cards right now. What I expect will happen next is that people will start fuzzing procmail, throw an uncountable number of bug reports at it which will get fixed in a trickle while never fixing the underlying, serious design flaws behind procmail.

Procmail has better alternatives The reason this is so frustrating is that there are plenty of modern alternatives to procmail which do not suffer from those problems. Alternatives to procmail(1) itself are typically part of mail servers. For example, Dovecot has its own LDA which implements the standard Sieve language (RFC 5228). (Interestingly, Sieve was published as RFC 3028 in 2001, before procmail was formally abandoned.) Courier also has "maildrop" which has its own filtering mechanism, and there is fdm (2007) which is a fetchmail and procmail replacement. Update: there's also mailprocessing, which is not an LDA, but processing an existing folder. It was, however, specifically designed to replace complex Procmail rules. But procmail, of course, doesn't just ship procmail; that would just be too easy. It ships mailstat(1) which we could probably ignore because it only parses procmail log files. But more importantly, it also ships:

• lockfile(1) - conditional semaphore-file creator
• formail(1) - mail (re)formatter

lockfile(1) already has a somewhat acceptable replacement in the form of flock(1), part of util-linux (which is Essential, so installed on any normal Debian system). It might not be a direct drop-in replacement, but it should be close enough. formail(1) is similar: the courier maildrop package ships reformail(1) which is, presumably, a rewrite of formail. It's unclear if it's a drop-in replacement, but it should probably possible to port uses of formail to it easily.

Update: the maildrop package ships a SUID root binary (two, even). So if you want only reformail(1), you might want to disable that with:

dpkg-statoverride --update --add root root 0755 /usr/bin/lockmail.maildrop 
dpkg-statoverride --update --add root root 0755 /usr/bin/maildrop

It would be perhaps better to have reformail(1) as a separate package, see bug 1006903 for that discussion.

The real challenge is, of course, migrating those old .procmailrc recipes to Sieve (basically). I added a few examples in the appendix below. You might notice the Sieve examples are easier to read, which is a nice added bonus.

Conclusion There is really, absolutely, no reason to keep procmail in Debian, nor should it be used anywhere at this point. It's a great part of our computing history. May it be kept forever in our museums and historical archives, but not in Debian, and certainly not in actual release. It's just a bomb waiting to go off. It is irresponsible for distributions to keep shipping obsolete and insecure software like this for unsuspecting users. Note that I am grateful to the author, I really am: I used procmail for decades and it served me well. But now, it's time to move, not bring it back from the dead.

Appendix

Previous work It's really weird to have to write this blog post. Back in 2016, I rebuilt my mail setup at home and, to my horror, discovered that procmail had been abandoned for 15 years at that point, thanks to that LWN article from 2010. I would have thought that I was the only weirdo still running procmail after all those years and felt kind of embarrassed to only "now" switch to the more modern (and, honestly, awesome) Sieve language. But no. Since then, Debian shipped three major releases (stretch, buster, and bullseye), all with the same vulnerable procmail release. Then, in early 2022, I found that, at work, we actually had procmail installed everywhere, possibly because userdir-ldap was using it for lockfile until 2019. I sent a patch to fix that and scrambled to remove get rid of procmail everywhere. That took about a day. But many other sites are now in that situation, possibly not imagining they have this glaring security hole in their infrastructure.

Procmail to Sieve recipes I'll collect a few Sieve equivalents to procmail recipes here. If you have any additions, do contact me. All Sieve examples below assume you drop the file in ~/.dovecot.sieve.

deliver mail to "plus" extension folder Say you want to deliver user+foo@example.com to the folder foo. You might write something like this in procmail:

MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
VERBOSE=off
EXTENSION=$1            # Need to rename it - ?? does not like $1 nor 1
:0
* EXTENSION ?? [a-zA-Z0-9]+
        .$EXTENSION/

That, in sieve language, would be:

require ["variables", "envelope", "fileinto", "subaddress"];
########################################################################
# wildcard +extension
# https://doc.dovecot.org/configuration_manual/sieve/examples/#plus-addressed-mail-filtering
if envelope :matches :detail "to" "*"  
  # Save name in $ name  in all lowercase
  set :lower "name" "$ 1 ";
  fileinto "$ name ";
  stop;
 

Subject into folder This would file all mails with a Subject: line having FreshPorts in it into the freshports folder, and mails from alternc.org mailing lists into the alternc folder:

:0
## mailing list freshports
* ^Subject.*FreshPorts.*
.freshports/
:0
## mailing list alternc
* ^List-Post.*mailto:.*@alternc.org.*
.alternc/

Equivalent Sieve:

if header :contains "subject" "FreshPorts"  
    fileinto "freshports";
  elsif header :contains "List-Id" "alternc.org"  
    fileinto "alternc";
 

Mail sent to root to a reports folder This double rule:

:0
* ^Subject: Cron
* ^From: .*root@
.rapports/

Would look something like this in Sieve:

if header :comparator "i;octet" :contains "Subject" "Cron"  
  if header :regex :comparator "i;octet"  "From" ".*root@"  
        fileinto "rapports";
   
 

Note that this is what the automated converted does (below). It's not very readable, but it works.

Bulk email I didn't have an equivalent of this in procmail, but that's something I did in Sieve:

if header :contains "Precedence" "bulk"  
    fileinto "bulk";
 

Any mailing list This is another rule I didn't have in procmail but I found handy and easy to do in Sieve:

if exists "List-Id"  
    fileinto "lists";
 

This or that I wouldn't remember how to do this in procmail either, but that's an easy one in Sieve:

if anyof (header :contains "from" "example.com",
           header :contains ["to", "cc"] "anarcat@example.com")  
    fileinto "example";
 

You can even pile up a bunch of options together to have one big rule with multiple patterns:

if anyof (exists "X-Cron-Env",
          header :contains ["subject"] ["security run output",
                                        "monthly run output",
                                        "daily run output",
                                        "weekly run output",
                                        "Debian Package Updates",
                                        "Debian package update",
                                        "daily mail stats",
                                        "Anacron job",
                                        "nagios",
                                        "changes report",
                                        "run output",
                                        "[Systraq]",
                                        "Undelivered mail",
                                        "Postfix SMTP server: errors from",
                                        "backupninja",
                                        "DenyHosts report",
                                        "Debian security status",
                                        "apt-listchanges"
                                        ],
           header :contains "Auto-Submitted" "auto-generated",
           envelope :contains "from" ["nagios@",
                                      "logcheck@",
                                      "root@"])
     
    fileinto "rapports";
 

Automated script There is a procmail2sieve.pl script floating around, and mentioned in the dovecot documentation. It didn't work very well for me: I could use it for small things, but I mostly wrote the sieve file from scratch.

Progressive migration Enrico Zini has progressively migrated his procmail setup to Sieve using a clever way: he hooked procmail inside sieve so that he could deliver to the Dovecot LDA and progressively migrate rules one by one, without having a "flag day". See this explanatory blog post for the details, which also shows how to configure Dovecot as an LMTP server with Postfix.

Other examples The Dovecot sieve examples are numerous and also quite useful. At the time of writing, they include virus scanning and spam filtering, vacation auto-replies, includes, archival, and flags.

Harmful considered harmful I am aware that the "considered harmful" title has a long and controversial history, being considered harmful in itself (by some people who are obviously not afraid of contradictions). I have nevertheless deliberately chosen that title, partly to make sure this article gets maximum visibility, but more specifically because I do not have doubts at this moment that procmail is, clearly, a bad idea at this moment in history.

Developing story I must also add that, incredibly, this story has changed while writing it. This article is derived from this bug I filed in Debian to, quite frankly, kick procmail out of Debian. But filing the bug had the interesting effect of pushing the upstream into action: as mentioned above, they have apparently made a new release and merged a bunch of patches in a new git repository. This doesn't change much of the above, at this moment. If anything significant comes out of this effort, I will try to update this article to reflect the situation. I am actually happy to retract the claims in this article if it turns out that procmail is a stellar example of defensive programming and survives fuzzing attacks. But at this moment, I'm pretty confident that will not happen, at least not in scope of the next Debian release cycle.

The filter rules preventing websites from portscanning the local machine have recently been tightened in Brave. It turns out there are a surprising number of ways to refer to the local machine in Chromium. localhost and friends 127.0.0.1 is the first address that comes to mind when thinking of the local machine. localhost is typically aliased to that address (via /etc/hosts), though that convention is not mandatory. The IPv6 equivalent is [::1].

• http://localhost/
• http://foo.localhost/
• http://127.0.0.1/
• http://0177.0000.0000.0001/ (127.0.0.1 in octal)
• http://0x7F000001/ (127.0.0.1 in hex)
• http://2130706433/ (127.0.0.1 in decimal)
• http://[::ffff:127.0.0.1]/ (IPv4-mapped IPv6 address)
• http://[::ffff:7f00:1]/ (alternate of IPv4-mapped IPv6 address)
• http://[0000:0000:0000:0000:0000:ffff:7f00:0001]/ (fully-expanded IPv4-mapped IPv6 address)
• http://[::1]/
• http://[0000:0000:0000:0000:0000:0000:0000:0001]/ (fully-expanded form of [::1])

0.0.0.0 0.0.0.0 is not a routable address, but that's what's used to tell a service to bind (listen) on all network interfaces. In Chromium, it resolves to the local machine, just like 127.0.0.1. The IPv6 equivalent is [::].

• http://0.0.0.0/
• http://0000.0000.0000.0000/ (0.0.0.0 in octal)
• http://0x00000000/(0.0.0.0 in hex)
• http://0/ (0.0.0.0 in decimal)
• http://[::ffff:0.0.0.0]/ (IPv4-mapped IPv6 address)
• http://[::ffff:0000:0000]/ (alternate form of IPv4-mapped IPv6 address)
• http://[0000:0000:0000:0000:0000:ffff:0000:0000]/ (fully-expanded IPv4-mapped IPv6 address)
• http://[::]/
• http://[0000:0000:0000:0000:0000:0000:0000:0000]/ (fully-expanded form of [::])

DNS-based Of course, another way to encode these numerical URLs is to create A / AAAA records for them under a domain you control. I've done this under my personal domain:

• http://t127.fmarier.org/ (127.0.0.1)
• http://t1aaaa.fmarier.org/ ([::1])
• http://t0.fmarier.org/ (0.0.0.0)
• http://t0aaaa.fmarier.org/ ([::])
• http://t127aaaam.fmarier.org/ ([::ffff:7f00:1])
• http://t0aaaam.fmarier.org/ ([::ffff:0000:0000])

For these to work, you'll need to:

• Make sure you can connect to IPv6-only hosts, for example by connecting to an appropriate VPN if needed.
• Put nameserver 8.8.8.8 in /etc/resolv.conf since you need a DNS server that will not filter these localhost domains. (For example, Unbound will do that if you use private-address: 127.0.0.0/8 in the server config.)
• Go into chrome://settings/security and disable Always use secure connections to make sure the OS resolver is used.
• Turn off the chrome://flags/#block-insecure-private-network-requests flag since that security feature (CORS-RFC1918) is designed to protect against these kinds of requests.

127.0.0.0/8 subnet Technically, the entire 127.0.0.0/8 subnet can used to refer to the local machine. However, it's not a reliable way to portscan a machine from a web browser because it only catches the services that listen on all interfaces (i.e. 0.0.0.0). For example, on my machine, if I nmap 127.0.0.1, I get:

PORT     STATE SERVICE   VERSION
22/tcp   open  ssh       OpenSSH 8.2p1
25/tcp   open  smtp      Postfix smtpd

whereas if I nmap 127.0.1.25, I only get:

PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 8.2p1

That's because I've got the following in /etc/postfix/main.cf:

inet_interfaces = loopback-only

which I assume is explicitly binding 127.0.0.1. Nevertheless, it would be good to get that fixed in Brave too.

This is a short statement on what I understand about Covid-19 and the current situation so that I can refer to it in other posts and don t need to repeat myself. What I do, think and eventually write is heavily influenced by this understanding, as outlined futher below. Also I want to set a counterpoint to the people that still publicly proclaim the official narrative. On November 15th 2020 I already wrote an email to debian-private@ with the subject Basic information about Corona in German that referred to my information collection on that topic. Since then my understanding has refined but not fundamentally changed:

• The western capitalistic system has finally reached end-of-life in 2019 after it almost collapsed already in 2008.
• The western oligarchs as organised in the World Economic Forum were well aware of this and planned a Great Reset of the system through a medical emergency. The goal is to create a new system where the old elite will remain in power.
• There has been no pandemic in 2020 but only a PCR-test that created cases . Age-normalized mortality in most countries was unspectacular in 2020.
• The so called Covid-19 vaccines are extremely dangerous in some batches with harmless batches in between to not raise suspicion: How Bad is My Batch.

I m happy to talk in detail about this topic and provide pointers. Right now a comprehensive overview is available at Grand-Jury.net. Why is this relevant? The internet in its current state is an instrument of control and suppression. This becomes especially obvious in the increase in censorship and the introduction of vaccine-passports which are planned to become universal passports for all aspects of our life. Therefor I want to work on initiatives for a free internet for the people. Right now I think about the freedombox, distributed search engines, re-decentralization in general and especially a decentralized alternative to Wikipedia.

As of this morning, Rcpp stands at 2501 reverse-dependencies on CRAN. The graph on the left depicts the growth of Rcpp usage (as measured by Depends, Imports and LinkingTo, but excluding Suggests) over time. Rcpp was first released in November 2008. It probably cleared 50 packages around three years later in December 2011, 100 packages in January 2013, 200 packages in April 2014, and 300 packages in November 2014. It passed 400 packages in June 2015 (when I tweeted about it), 500 packages in late October 2015, 600 packages in March 2016, 700 packages in July 2016, 800 packages in October 2016, 900 packages early January 2017, 1000 packages in April 2017, 1250 packages in November 2017, 1500 packages in November 2018, 1750 packages in August 2019, 2000 packages in July 2020, and 2250 package in March of last year. The chart extends to the very beginning via manually compiled data from CRANberries and checked with crandb. The next part uses manually saved entries. The core (and by far largest) part of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A (manually curated) list of packages using Rcpp is available too. Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited talk. We passed five percent in December of 2014, six percent July of 2015, seven percent just before Christmas 2015, eight percent in the summer of 2016, nine percent mid-December 2016, cracked ten percent in the summer of 2017, eleven percent in 2018 and passed 12.5 percent or one in every eight CRAN packages dependens on Rcpp along with the 2000 packages mark. Truly stunning. As before, there is more detail in the chart: how CRAN seems to be pushing back more and removing more aggressively (which my CRANberries tracks but not in as much detail as it could), how the growth of Rcpp seems to be slowing somewhat outright and even more so as a proportion of CRAN as one would expect a growth curve to. The Rcpp team continues to aim for keeping Rcpp as performant and reliable as it has been (and see e.g. here for some more details). A really big shoutout and Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Welcome to the 36th post of the really randomly reverberating R, or R⁴ for short, write-ups. Today s post is about using Redis, and especially RcppRedis, for live or (near) real-time monitoring with R. There is an saying that you can take the boy out of the valley, but you cannot the valley out of the boy so for those of us who spent a decade or two in finance and on trading floors, having some market price information available becomes second nature. And/or sometimes it is just good fun to program this. A good while back Josh posted a gist on a simple-yet-robust while loop. It (very cleverly) uses his quantmod package to access the SP500 in real-time . (I use quotes here because at the end of retail broadband one is not at the same market action as someone co-located in a New Jersey data center. It is however not delayed: as an index, it is not immediately tradeable as a stock, etf, or derivative may be all of which are only disseminated as delayed price information, usually by ten minutes.) I quite enjoyed the gist and used it and started tinkering with it. For example, it collects data but only saves (i.e. persists ) it after market close. If for whatever reason one needs to restart recent history is gone. In any event, I used his code and generalized it a little and published this about a year ago as function intradayMarketMonitor() in my dang package. (See this blog post announcing it.) The chart of the left shows this in action, the chart is a snapshot from a couple of days ago when the vignettes (more on them below) were written. As lovely as intradayMarketMonitor() is, it also limits itself to market hours. And sometimes you want to see, say, how the market opens on Sunday (futures usually restart at 17h Chicago time), or how news dissipates during the night, or where markets are pre-open, or . So I both wanted to complement this with futures, and also cache it locally so that, say, one machine might collect data and one (or several others) can visualize. For such tasks, Redis is unparalleled. (Yet I also always felt Redis could do with another, simple, short and sweet introduction stressing the key features of i) being multi-lingual: write in one language, consume in another and ii) loose coupling: no linking as one talks to Redis via standard tcp/ip networking. So I wrote a new intro vignette that is now in RcppRedis. I hope this comes in handy. Comments welcome!) Our RcppRedis package had long been used for such tasks, and it was easy to set it up. Standard use is to loop, fetch some data, push it to Redis, sleep, and start over. Clients do the same: fetch most recent data, plot or report it, sleep, start over. That works, but it has a dual delay as the client sleeping may miss the data update! The standard answer to this is called publish/pubscribe, or pub/sub. Libraries such as 0mq or zeromq specialise in this. But it turns out Redis already has it. I had some initial difficulty adding it to RcppRedis so for a trial I tested the marvellous rredis package by Bryan and simply instantiated two Redis clients. Now the data getter simply publishes a new data point in a given channel, by convention named after the security it tracks. Clients register with the Redis server which does all the actual work of keeping track of who listens to what. The clients now simply listen (which is a blocking operation) and as soon as data comes in receive it. This is quite mesmerizing when you just run two command-line clients (in a byobu session, say). As sone as the data is written (as shown on console log) it is consumed. No measruable overhead. Just lovely. Bryan and I then talked a litte as he may or may not retire rredis. Having implemented the pub/sub logic for both sides once, he took a good hard look at RcppRedis and just like that added it there. With some really clever wrinkles for (optional) per-symbol callback as closure attached to the instance. Truly amazeballs And once we had it in there, generalizing from publishing or subscribing to just one symbol easily generalizes to having one listener collect and publish for multiple symbols, and having one or more clients subscribe and listen one, more or even all symbol. All with ease thanks tp Redis. The second chart, also from a few days ago, shows four symbols for four (front-contract) futures for Bitcoin, Crude Oil, SP500, and Gold. As all this can get a little technical, I wrote a second vignette for RcppRedis on just this: market monitoring. Give this a read, if interested, feedback on this one is most welcome too! But all the code you need is included in the package just run a local Redis instance. Before closing, one sour note. I uploaded all this in a new and much improved updated RcppRedis 0.2.0 to CRAN on March 13 ten days ago. Not only is it still not there , but CRAN in their most delightful way also refuses to answer any emails of mine. Just lovely. The package exhibited just one compiler warning: a C++ compiler objected to the (embedded) C library hiredis (included as a fallback) for using a C language construct. Yes. A C++ compiler complaining about C. It s a non-issue. Yet it s been ten days and we still have nothing. So irritating and demotivating. Anyway, you can get the package off its GitHub repo. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.