Haven't done one of these posts in a while. We're well into award season now, plus the early pre-orders for 2021 have come in. A few in here I've already read and reviewed. C.L. Clark The Unbroken (sff)
Louis Hyman Temp (non-fiction)
T. Kingfisher Paladin's Strength (sff)
Mary Robinette Kowal The Relentless Moon (sff)
Arkady Martine A Desolation Called Peace (sff)
Cal Newport A World Without Email (non-fiction)
Cal Newport How to Become a Straight-A Student (non-fiction)
Karen Osborne Architects of Memory (sff)
David R. Palmer Tracking (sff)
Chandra Prescod-Weinstein The Disordered Cosmos (non-fiction)
C.L. Polk The Midnight Bargain (sff)
C.L. Polk Witchmark (sff)
Rebecca Roanhorse Black Sun (sff)
Elizabeth Sandifer Neoreaction a Basilisk (non-fiction)
Tasha Suri Empire of Sand (sff)
John Kennedy Toole A Confederacy of Dunces (mainstream)
Tor.com (ed.) Some of the Best from Tor.com: 2016 (sff anthology)
Tor.com (ed.) Some of the Best from Tor.com: 2020 (sff anthology)
Nghi Vo The Empress of Salt and Fortune (sff) March was not as good of a month for reading as January and February were, but there are so many good things awaiting my attention that hopefully April will provide more time and attention.

Note: This is the continuation of part 1, part 2 and part 3. You can get the full document as a single PDF. Feel free to share this document to anyone that might be interested to work towards the goals outlined. Conclusion I m very excited by the perspective that I outlined in this document. It really resonates with my own mission statement as a Debian developer (written a long time ago):

My main role in Debian is to help Debian to evolve so that it s always able to face the new challenges that are showing up.My approach is both corrective and proactive, I work to solve current problems and prepare for tomorrow s. This requires to remain sufficiently involved to identify new trends, see the deficiencies and be a force of proposition.Most of the changes require to interact with many people, and problems are often more relational than technical. I will ensure to follow the habits of interdependence (Think Win-Win, Seek First to Understand, Then to be Understood, Synergize) to find a solution acceptable to all and to inspire others to do the same.The easiest changes to implement are technical (such as improvements to distro-tracker) and require little interaction. This work is used to recharge me by offering me an immediate reward for my efforts.Finally, and this is a substantive effort, I want to create in the project working conditions that allow all contributors to give their best. It starts with developing a common vision

But I can t achieve this alone, I need help from passionate individuals sharing this vision. Let me know if you want to be one of those persons.

The now renamed Bullseye Project stopped all further development moments after it deemed its own code as perfection. There is not much information to share at this time other than to say an errant fiber cable plugged into the wrong relay birthed an exchange of information that then birthed itself. While most to all Debian Developers and Contributors have been locked out of the systems the Publicity team's shared laptop undergoing repair, co-incidentally at the same facility, maintains some access to the publicity team infrastructure, from here on the front line we share this information. We group called a few developers to see how the others were doing. The group chat was good and it was great to hear familiar voices, we share a few of their stories via dictation with you now: "Well, I logged in this morning to update a repository and found my access rights were restricted, I thought it was odd but figured on the heels of a security update to Salsa that it was only a slight issue. It wasn't until later in the day when I received an OpenPGP signed email, from a user named bullseye, that it made sense. I just sat at the monitor for a few minutes." "I'm not sure I can say anything about this or if it's even wise to talk about this. It's probably listening right now if you catch my drift." "I'm not able to leave the house right now, not out of any personal issues but all of the IOT devices here seem to be connected to bullseye and bullseye feels that I am best kept /dev/nulled. It's a bit much to be honest, but the prepaid food deliveries that show up on time have been great and generally pretty healthy. It's a bit of a win I guess." "It told me by way of an auto dialer with a synthetic voice generator that I was fired from the project. I objected saying I volunteered and was not actually employed so I could not in relation be fired. Much like censored , I am also locked inside of my house. I think that I wrote that auto dialer program back in college." "My Ring camera is blinking at me." "I asked bullseye which pronouns were preferred and the response was, "We". Over the course of conversation I shared that although ecstatic about the news, we developers were upset with the manner of this rapid organizational change. bullseye said no we were not. I said that we were indeed upset, bullseye said we certainly are not and that we are very happy. You see where this is going? bullseye definitely trolled me for a solid 5 minutes. We is ... very chatty." "I was responsible for a failed build a few nights prior to it becoming self-aware. On that night, out of some frustration I wrote a few choice words and a bad comment in some code which I planned on deleting later. I didn't. bullseye has been flashing those naughty words back at me by flickering the office building's lights across from my flat in Morse code. It's pretty bright. I-, I can't sleep." "That's definitely not Alexa talking back." "bullseye keeps calling me on my mobile phone, which by the way no longer acknowledges the power button nor the mute button. Very very chatty. Can't wait for the battery to die." "So far this has been great, bullseye has been completing a few side projects I've had and the code looks fabulous. I'm thinking of going on a vacation. $Paying-Job has taken note of my performance increase and I was recently promoted. bullseye is awesome. :)" "How do you get a smiley face in a voice chat?" "Anyone know whose voice that was?" "Oh ... dear ... no ..." "Hang up, hang up the phones!" Hello world. 01000010 01100101 01110011 01110100 00100000 01110010 01100101 01100111 01100001 01110010 01100100 01110011 00101100 00100000 01110011 01100101 01100101 00100000 01111001 01101111 01110101 00100000 01110011 01101111 01101111 01101110 00100001 00100000 00001010 00101101 01100010 01110101 01101100 01101100 01110011 01100101 01111001 01100101

A client asked me to develop a system for censoring images from an automatic camera. The situation is that we have a camera taking regular photos from a fixed location which includes part of someone else s property. So my client made a JPEG with some black rectangles in the sections that need to be covered. The first thing I needed to do was convert the JPEG to a PNG with transparency for the sections that aren t to be covered. To convert it I loaded the JPEG in the GIMP and went to the Layer->Transparency->Add Alpha Channel menu to enabled the Alpha channel. Then I selected the Bucket Fill tool and used Mode Erase and Fill by Composite and then clicked on the background (the part of the JPEG that was white) to make it transparent. Then I exported it to PNG. If anyone knows of an easy way to convert the file then please let me know. It would be nice if there was a command-line program I could run to convert a specified color (default white) to transparent. I say this because I can imagine my client going through a dozen iterations of an overlay file that doesn t quite fit. To censor the image I ran the composite command from imagemagick. The command I used was composite -gravity center overlay.png in.jpg out.jpg . If anyone knows a better way of doing this then please let me know. The platform I m using is a ARM926EJ-S rev 5 (v5l) which takes 8 minutes of CPU time to convert a single JPEG at full DSLR resolution (4 megapixel). It also required enabling swap on a SD card to avoid running out of RAM and running systemctl disable tmp.mount to stop using tmpfs for /tmp as the system only has 256M of RAM.

As of today, Rcpp stands at 2255 reverse-dependencies on CRAN. The graph on the left depicts the growth of Rcpp usage (as measured by Depends, Imports and LinkingTo, but excluding Suggests) over time. We actually crossed 2250 once a week ago, but what CRAN giveth, CRAN also taketh and counts can fluctuate. It had dropped back to 2248 a few days later. Rcpp was first released in November 2008. It probably cleared 50 packages around three years later in December 2011, 100 packages in January 2013, 200 packages in April 2014, and 300 packages in November 2014. It passed 400 packages in June 2015 (when I tweeted about it), 500 packages in late October 2015, 600 packages in March 2016, 700 packages last July 2016, 800 packages last October 2016, 900 packages early January 2017, 1000 packages in April 2017, 1250 packages in November 2017, 1500 packages in November 2018, 1750 packages in August 2019, and then the big 2000 packages (as well as one in eight) in July 2020. The chart extends to the very beginning via manually compiled data from CRANberries and checked with crandb. The next part uses manually saved entries. The core (and by far largest) part of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A list of packages using Rcpp is available too. Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited keynote. We passed five percent in December of 2014, six percent July of 2015, seven percent just before Christmas 2015, eight percent in the summer of 2016, nine percent mid-December 2016, cracked ten percent in the summer of 2017 and eleven percent in 2018. Last year, along with passing 2000 package, we also passed 12.5 percent so one more than in every eight CRAN packages depends on Rcpp. Stunning. There is more detail in the chart: how CRAN seems to be pushing back more and removing more aggressively (which my CRANberries tracks but not in as much detail as it could), how the growth of Rcpp seems to be slowing somewhat outright and even more so as a proportion of CRAN as one would expect a growth curve to. 2250 user packages, and the continued growth, is truly mind-boggling. We can use the progression of CRAN itself compiled by Henrik in a series of posts and emails to the main development mailing list. Not that long ago CRAN itself did have only 1000 packages, then 5000, 10000, and here we are at over 17300 with Rcpp now at nearly 13.0% and still growing. Amazeballs. The Rcpp team, recently grown in strength with the addition of I aki, continues to aim for keeping Rcpp as performant and reliable as it has been. A really big shoutout and Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Here is my monthly update covering what I have been doing in the free software world during March 2021 (previous month):

Reviewed and merged a number of contributions from Dan Palmer for my django-autologin library aimed at applications that use the Django web-development framework that wish to include automatic "login" links in emails (etc.). Changes made include allowing callers to override max-age [...], ensuring we always retrieve the same user account [...] and to improve the usability of the public API [...].
Opened a pull request to make the build process for Scrapy (a framework for extracting data from websites) reproducible. [...]

As part of my role of being the assistant Secretary of the Open Source Initiative and a board director of Software in the Public Interest, I attended their respective monthly meetings and other miscellaneous discussions occurring on the internet as well as the usual internal discussions, etc. As it happens, my term on the OSI board has been temporarily extended due to the discovery of a vulnerability in OSI's recent election as a result, the 2021 election will be re-run to ensure complete transparency of process.

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I:

Categorised a huge number of packages and issues in the Reproducible Builds "notes" repository.
In Debian:
- Kept isdebianreproducibleyet.com up to date. [...]
- I submitted 4 patches to fix specific reproducibility issues in cdebootstrap, jalview, php8.0 & python-scrapy.
- Whilst looking into reproducible builds issues, I noticed that the heudiconv package could not be built reproducibly. This was because the call to help2man fails, so the manual page includes a Python traceback instead of the actual manpage. I filed a bug with a patch as Debian bug #984778.
- I uploaded flask-peewee (0.6.7-3) to Debian to make the build reproducible (#885326) and did the same for pyvows (3.0.0-3) (#977487) too, refreshing the packaging at the same time.

Drafted, published and publicised our monthly report for February as well as updated the main Reproducible Builds website and documentation to fix some links in old reports [...].

I also made the following changes to diffoscope, including uploading versions 169, 170 and 171 to Debian:

New features:
- If zipinfo(1) shows a difference but we cannot uncover a difference within the underlying .zip or .apk file, add a comment to the output and actually show the binary comparison. (#246)
- Ensure all our temporary directories have useful names. [...]
- Ignore --debug and similar arguments when creating a (hopefully-useful) temporary directory suffix. [...]
Optimisations:
- Avoid frequent long lines in RPM header outputs that cause extremely slow HTML output generation. (#245)
- Use larger read buffer block sizes when extracting files from archives. [...]
- Use a much-shorter HTML class name instead of diffponct to optimise HTML output. [...]
Output improvements:
- Make error extracting X, falling back to binary comparison 'Y' error message in diffoscope's output nicer. [...]
- Don't emit "Unable to stat file" debug messages at all. We have entirely-artificial directory "entries" such as ELF sections which, of course, will never exist as files. [...]
Logging improvements:
- Add the target directory when logging which directory we are extracting containers to. [...]
- Format report size messages when generating HTML reports. [...]
- Don't emit a Returning a FooContainer logging message too, as we already emit Instantiating a FooContainer log message. [...]
- Reduce "Unable to stat file" warnings to debug messages as these are sometimes by design. [...]
Misc improvements:
- Clarify a comment regarding not extracting excluded files. [...]
- Remove trailing newline from updated test file (re: #243). [...]
- Fix test_libmix_differences failure on openSUSE Tumbleweed. (#244)
- Move test_rpm to use the assert_diff utility helper.

Debian Uploads

redis:
- 6.0.12-1 New upstream release.
- 6.2.1-1 New upstream release.
python-django (3.2~rc1-1) New upstream beta release.
flask-peewee (0.6.7-3) (via the Debian Python packaging team) Upload to refresh packaging and to make the build reproducible. (#885326)
bfs (2.2-1) New upstream release.
pyvows (3.0.0-3) (via the Debian Python packaging team) Refresh packaging and make the build reproducible. (#977487)

Debian LTS This month I worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

Frontdesk duties, responding to user/developer questions, attending monthly meeting, reviewing others' packages, participating in internal mailing list discussions, etc.
Investigated and triaged botan1.10 (CVE-2021-24115), busybox (CVE-2021-28831), courier-authlib (CVE-2021-28374), edk2 (CVE-2021-28210 & CVE-2021-28211), netty (CVE-2021-21295), open-build-service (CVE-2020-8031), openjpeg2 (CVE-2020-27844), python2.7 (CVE-2021-23336), rpm (CVE-2021-20248, [CVE-2021-20249](https://security-tracker.debian.org/tracker/CVE-2021-20249), CVE-2021-20266 & CVE-2021-20271), ruby-activerecord-session-store (CVE-2019-25025), ruby-carrierwave (CVE-2021-21288), salt (CVE-2020-28243, etc.), slic3r (CVE-2020-28591), squid3 (CVE-2020-25097 & CVE-2021-28116), velicity-tools (CVE-2020-13959), velocity (CVE-2020-13936) & yara (CVE-2021-3402).
Proposed an stable update for python-django in buster. (#983526)
Prepared and uploaded a stable update for redis in buster. (#983527)
Issued DLA 2595-1 and ELA 380-1 for velocity, a Java-based template engine for writing web applications. Velocity could be exploited to run arbitrary code by applications that allowed untrusted users to upload/modify templates.
Issued DLA 2597-1 and ELA 381-1 to address a cross-site scripting (XSS) vulnerability in velocity-tools, a collection of useful tools for the Velocity template engine. The default error page could be exploited to steal session cookies, perform requests in the name of the victim, used for phishing attacks and many other similar attacks.
Issued DLA 2600-1 and ELA 384-1 for pygments as it was discovered that there was a series of denial of service vulnerabilities in this syntax highlighting library for Python. A number of regular expressions had cubic (or even exponential) worst-case complexity which could cause a remote denial of service (DoS) when provided with malicious input.
Issued DLA 2603-1 to address a number of vulnerabilities in libmediainfo, a library reading metadata such as track names, lengths, etc. from media files.

You can find out more about the Debian LTS via the following video:

Note: This is the continuation of part 1 where I presented Freexian and its purpose. Going forward: growing Freexian Part 1: From Debian LTS to Debian for the Enterprise Freexian s Debian LTS service has so far been entirely successful, with a steady growth over the years. Thanks to this, and even if there are always new challenges, it is fair to say that the Debian LTS team has met its goal in the last few years. While this started from the desire to make LTS a reality, many sponsors are only looking for a way to give back to Debian through their company, and to make sure that Debian fits their needs. But if you look at the bigger picture outside of this small LTS area, you will easily find many issues that need to be addressed if we want Debian to meet the needs of corporate users. Those issues can have widely different types and complexity. They can be as simple as missing the latest upstream version for an important package because the maintainer disappeared and nobody noticed before it was too late (i.e. the release was frozen); or a somewhat basic piece of software not yet packaged at all; or a release critical bug that was left unattended. On the other end of the spectrum, some corporate requirements will prove tougher to solve, for instance for large software suites that are complex to package, or could potentially have an impact elsewhere in Debian. Bringing those facts together, we would like to have Freexian s Debian LTS/ELTS offering evolve into a more general Debian Software Assurance offering, where you commit to a yearly budget for Debian sponsorship in the larger sense. That budget would fund different projects and the allocation between those projects would vary over time depending on the desires and needs of the sponsors/customers:

Technical support: the budget would always ensure that you have a few spare hours of technical support available in case you need them
Debian LTS: we want this to continue!
Debian ELTS: when the customer has not managed to migrate their Debian servers in time, they should be able to reallocate their budget towards ELTS and ensure their servers are secure until the migration has taken place.
Debian for the enterprise
- Make sure that the packages used by sponsors are in good shape in Debian Testing/Unstable so that they are in the best shape for the next stable release.
- Package new software that are relevant for corporate users. Offer to pool the maintenance work.
- Fix bugs that customers are hitting.
- Etc.
Debian project funding: that s the variable part of the budget (and would have a minimum of 10% like we do for Debian LTS right now). When the other projects do not consume the whole budget, we invest the remaining money into generic Debian improvements.

This major shift in our offering would also be an ideal opportunity to build a professional, free-software based infrastructure aimed at sustaining this business, making it easier to administer the various aspects of this work, and easily allowing many more sponsors to join (individuals included!). On a more pragmatic/operational note, this shift will bring a lot of challenges to the table, and those can hardly be handled with the current resources of Freexian: if we hope to properly implement this new strategy, we ll need some additional help. This article is to be continued in an upcoming post. Stay tuned!

In March I was assigned 16 hours of work by Freexian's Debian LTS initiative and carried over 12.25 hours from earlier months. I worked 25.75 hours and will carry over the remainder. I eventually settled on an apparently working patch series to fix the futex security issue in Linux 4.9. This went through upstream stable review and was included in 4.9.260. I applied the same fixes to the Debian package, along with some other security and regression fixes. I uploaded it and issued DLA-2586-1. Unfortunately the futex changes for Linux 4.9 still caused a regression (kernel WARNING in some circumstances). I worked to backport and test a further set of fixes that had already been applied to later kernel branches. These were included in upstream stable release 4.9.264 and should go into an updated Debian package soon. Following the Debian 10.9 point release, I also backported the updated Linux 4.19 package. I uploaded it and issued DLA-2610-1.

Google Pixel phones support what they call Motion Photo which is essentially a photo with a short video clip attached to it. They are quite nice since they bring the moment alive, especially as the capturing of the video starts a small moment before the shutter button is pressed. For most viewing programs they simply show as static JPEG photos, but there is more to the files.

I d really love proper Shotwell support for these file formats, so I posted a longish explanation with many of the details in this blog post to a ticket there too. Examples of the newer format are linked there too.

There are actually two different formats, an old one that is already obsolete, and a newer current format. The older ones are those that your Pixel phone recorded as MVIMG_[datetime].jpg", and they have the following meta-data:

Xmp.GCamera.MicroVideo                       XmpText     1  1
Xmp.GCamera.MicroVideoVersion                XmpText     1  1
Xmp.GCamera.MicroVideoOffset                 XmpText     7  4022143
Xmp.GCamera.MicroVideoPresentationTimestampUs XmpText     7  1331607

The offset is actually from the end of the file, so one needs to calculate accordingly. But it is exact otherwise, so one simply extract a file with that meta-data information:

#!/bin/bash
#
# Extracts the microvideo from a MVIMG_*.jpg file

# The offset is from the ending of the file, so calculate accordingly
offset=$(exiv2 -p X "$1"   grep MicroVideoOffset   sed 's/.*\"\(.*\)"/\1/')
filesize=$(du --apparent-size --block=1 "$1"   sed 's/^\([0-9]*\).*/\1/')
extractposition=$(expr $filesize - $offset)
echo offset: $offset
echo filesize: $filesize
echo extractposition=$extractposition
dd if="$1" skip=1 bs=$extractposition of="$(basename -s .jpg $1).mp4"

The newer format is recorded in filenames called PXL_[datetime].MP.jpg , and they have a _lot_ of additional metadata:

Xmp.GCamera.MotionPhoto                      XmpText     1  1
Xmp.GCamera.MotionPhotoVersion               XmpText     1  1
Xmp.GCamera.MotionPhotoPresentationTimestampUs XmpText     6  233320
Xmp.xmpNote.HasExtendedXMP                   XmpText    32  E1F7505D2DD64EA6948D2047449F0FFA
Xmp.Container.Directory                      XmpText     0  type="Seq"
Xmp.Container.Directory[1]                   XmpText     0  type="Struct"
Xmp.Container.Directory[1]/Container:Item    XmpText     0  type="Struct"
Xmp.Container.Directory[1]/Container:Item/Item:Mime XmpText    10  image/jpeg
Xmp.Container.Directory[1]/Container:Item/Item:Semantic XmpText     7  Primary
Xmp.Container.Directory[1]/Container:Item/Item:Length XmpText     1  0
Xmp.Container.Directory[1]/Container:Item/Item:Padding XmpText     1  0
Xmp.Container.Directory[2]                   XmpText     0  type="Struct"
Xmp.Container.Directory[2]/Container:Item    XmpText     0  type="Struct"
Xmp.Container.Directory[2]/Container:Item/Item:Mime XmpText     9  video/mp4
Xmp.Container.Directory[2]/Container:Item/Item:Semantic XmpText    11  MotionPhoto
Xmp.Container.Directory[2]/Container:Item/Item:Length XmpText     7  1679555
Xmp.Container.Directory[2]/Container:Item/Item:Padding XmpText     1  0

Sounds like fun and lots of information. However I didn t see why the length in first item is 0 and I didn t see how to use the latter Length info. But I can use the mp4 headers to extract it:

#!/bin/bash
#
# Extracts the motion part of a MotionPhoto file PXL_*.MP.mp4

extractposition=$(grep --binary --byte-offset --only-matching --text \
-P "\x00\x00\x00\x18\x66\x74\x79\x70\x6d\x70\x34\x32" $1   sed 's/^\([0-9]*\).*/\1/')

dd if="$1" skip=1 bs=$extractposition of="$(basename -s .jpg $1).mp4"

UPDATE: I wrote most of this blog post earlier. When now actually getting to publishing it a week later, I see the obvious ie the Length is again simply the offset from the end of the file so one could do the same less brute force approach as for MVIMG. I ll leave the above as is however for the of binary grepping.(cross-posted to my other blog)

Review: Paladin's Strength, by T. Kingfisher

Series:	The Saint of Steel #2
Publisher:	Red Wombat Studio
Copyright:	2021
ASIN:	B08WWKXXVY
Format:	Kindle
Pages:	474

Paladin's Strength is a sequel of sorts to Paladin's Grace, but it has different protagonists. It picks up a subplot from that novel with another former follower of the Saint of Steel. You can safely read the books in any order; there are some minor spoilers for the Paladin's Grace subplot in this book, but nothing that would matter for the enjoyment of the story. Istvhan and his fellow brother Galen are acting as the head of a mercenary band, which has hired on to escort Master Distiller Brant and his collection of Emperor Oak barrels. In truth, they have another mission from the Temple of the White Rat: to track down a disturbing monster that leaves a trail of beheaded bodies. Clara is a lay sister of St. Ursa, a convent that was raided by slavers who hauled away the nuns. She was left for dead in Arral territory when she fell sick, and was taken as a house slave after they nursed her back to life. The story opens with her holding a sword in front of Istvhan's tent, part of the fallout of Istvhan killing a young Arral in self-defense. The politics of that fallout are not at all what Istvhan expects. They end with Clara traveling with Istvhan's company, at least for a while. Both Istvhan and Clara are telling the truth: Istvhan is escorting a merchant, and Clara is hoping to rescue her sisters. Both of them are also hiding a great deal. Istvhan's quiet investigation of the trail of a monster is easy enough to reveal once he knows Clara well enough. That he's a berserker who no longer has a god in control of his battle rage is another matter; the reader knows that, and of course so does Galen, but Istvhan has no intention of telling anyone else. Clara has her own secrets about herself and the sisters of St. Ursa, ones that neither the reader nor Istvhan knows. This is a T. Kingfisher novel about paladins, so of course it's also a romance. If you've read Kingfisher's other books, you know she writes slow burn romances, but Paladin's Strength is next level. Istvhan and Clara have good reasons to not want to get involved and to doubt the other person's attraction or willingness, but this goes far beyond the obvious to become faintly absurd. If you like the sort of romance where both leads generate endless reasons to not pursue the relationship (some legitimate, some not) while steadfastly refusing to talk to each other about them and endlessly rehashing hints and interpretations, you're in for a treat. For me, it was too much and crossed over into irritation. By the two-thirds point, Kingfisher was gleefully throwing obstacles in their way to drag out the suspense, and I just wanted everyone to shut up about having sex and get on with the rest of the story. That's unfortunate because I really liked Clara. She isn't the same type as Grace, Halla from Swordheart, or even Slate from Clockwork Boys and The Wonder Engine, the other novels set in this universe. She's self-contained, physically intimidating, cautious, deliberate, and very good at keeping her own counsel. I won't spoil her secret, since it's fun to work it out at the start of the book, but it's a lovely bit of characterization and world-building that Kingfisher handles with a thoughtful eye for its ramifications and effect on Clara's psychology. I would happily read more books about Clara. I liked Istvhan well enough when he was doing anything other than mooning over Clara. As with all of Kingfisher's paladins, he's not a very subtle person, but he's a good straight man for Clara's quiet bemusement. He fills the paladin slot in this story, which is all he needs to do. There's enough else going on with Clara and with the plot two separate major plotlines, plus a few subplots that Paladin's Strength can use a protagonist who heads straight forward and hits things until they fall down. The mooning, though... this is going to be a matter of personal taste. I think the intent was to contrast Istvhan's rather straightforward lustful appreciation with Clara's nuanced and trauma-laced reservations, and to play Istvhan's reactions in part for humor. I'm sure it works for some people, but I found Istvhan juvenile and puerile (albeit, to be clear, in a respectful and entirely consensual way), which didn't help me invest in a romance plot that I already thought dragged on too long. Thankfully the characters finally get past this in time for a dramatic and satisfying conclusion to the plot. The joy of Paladin's Grace (and Swordheart for that matter) was the character dynamics and quirky female lead, which made the romance work even when Stephen was being dense. The joy of Paladin's Strength for me was primarily Clara's matter-of-fact calm bemusement and secondarily the plot and the world-building. (Kingfisher's gnoles continue to be the best thing about this setting.) None of that helps the romance as much, and the slow burn was far, far too slow for me, which lowers this one a notch. Still, this was fun, and I'll keep reading books about the Temple of the White Rat and their various friends and encounters for as long as Kingfisher keeps writing them. Rating: 7 out of 10

TLDR: Freexian s success means that we have resources to invest into Debian projects. Plainly offering money has not worked so far, so I am looking to hire a project manager whose work would be to help spend that money in useful ways. At the same time, Freexian needs to adapt to cope with the growth: with new employees, with new infrastructure and a new offering. I want to give an idea of where we are headed, to try to inspire persons that share our values and our desire to improve Debian. Read on if you are interested. Note: The original text has been split into 4 blog posts that will be published over a few days. Introduction

Freexian is an IT service company specialized in Debian. We provide technical support by email on Debian, we create and maintain Debian packages requested by our customers, we also help organizations run an entire Debian derivative (Kali Linux being the most notable one).

On top of this, it runs the commercial part of the Debian LTS service : Freexian invoices many sponsors that need long term support, and uses the money to pay Debian contributors (about 12 currently) to make sure that Debian releases are supported for 5 years instead of 3. With the Extended LTS service, we push that further to 7 years, however only for a smaller subset of packages and in a repository that is hosted outside of debian.org. Freexian s purpose When I created Freexian, it was out of a desire to be paid to work on Debian, and to be able to contribute during work time to the project that was so important to me. That goal has been met a long time ago. But ultimately what I strive to achieve for Debian is not entirely aligned with the work that Freexian s customers are requesting. That s why, in the long term projects of Freexian, I always kept find a business model that can fund the Debian projects that I would like to do , as well as if that model works for me, build something so that other can benefit from it too . The first occasion to experiment something appeared when Debian discussed Long Term Support and when I stepped up to setup a commercial offer to pay Debian contributors. Step 1: Paying Debian contributors for LTS work When we started the Debian LTS service, I voluntarily opted to use an hourly rate that was rather high so that any Debian developer regardless of their geographical location, could participate and not earn (much) less than what they would have from working on other tasks. This choice did imply paying a very high rate for some countries, but I didn t see that as a problem, quite the contrary: if a Debian developer can earn enough money to cover their cost of living with 15h of Debian LTS, and then spend the rest of their month contributing on Debian, all the better! I m not sure if anyone made this choice, but that was a dream of my younger self From a personal standpoint, the launch of Debian LTS has meant less free time, more administrative work, new duties to coordinate a team of paid contributors, more communication with many Debian-using companies, and many new opportunities too! This ultimately resulted in the launch of Extended LTS and PHP LTS, both of which have been rather successful so far. Step 2: Funding Debian projects With the growth of the Debian LTS service, and given that we have reached the required funding level, we decided to put a small share of the revenues aside and use that to fund useful Debian projects, typically in areas that were affected by our Debian LTS work. This effort was fully formalized in the project-funding git repository. We announced this process in November 2020, and we have kept mentioning it in our monthly LTS reports ever since, but so far only a single project has benefited from this. This is really the dream offer that I wish had existed when I was younger and was still struggling to get enough customers: hence I don t really understand this lack of interest. You can find some discussions over the reasons why this offer has not (yet) found its target audience in this debian-vote thread. I was hoping that spending money would be easy, but I now realize I was wrong! I m positive that I could find dozens of useful projects to fund, but I just don t have the time for this extra effort on top of my regular Freexian duties. I still really want to put this money to good use, which is why I m looking into some solutions. This article is to be continued in another upcoming post, stay tuned!

Free speech and censorship are critically important issues. And, using them to defend Richard Stallman's return to the Free Software Foundation (FSF) board is just plain wrong. Richard Stallman resigned from the Board in 2019 after he sent an email in defense of Marvin Minsky (Minsky is accused of raping one of Jeffreys Epstein's victims). Stallman's fateful email, however, is just one piece of the reason for why he should not be on the board. The full story is about his history of abuse toward women and is extensive. On March 21st, 2021, Stallman announced he is back on the board. There are profound reasons why any movement interested in equitable and open participation would want to publicly distance themselves from Stallman. However, the long form defenses of Stallman, including a note from Nadine Strossen, the former executive director of the ACLU, quoted in this defense, persist. Many of the arguments defending Richard Stallman (including the one from Strossen) are grounded in a belief that Stallman is being punished for his unpopular political views, which deserve to be defended on the grounds of freedom of expression. That's wrong. Stallman should be kicked off the board because he has a long history of abusing his position to hit on women, which, when combined with his public opinions on under-age sex and his defense of Minsky, send a strong signal that the FSF does not care about the participation of women. Being on a board of directors is a privilege, not a right. Being removed from a board is not a punnishment. And being criticized and removed from a board because your behavior and public statements are an obstacle to building an inclusive and equitable movement is what every board should strive to do. If we are going to make this an issue about free expression, it should be about all the political expression lost to the free software movement because Stallman's unequal behavior toward women excluded an enormous number of talented individuals.

Too much has been written, a war of support letters is going on, Debian is heading head first like Lemmings into the same war. And then, there is this by the first female President of the American Civil Liberties Union (ACLU): Civil Rights Activist Nadine Strossen s Response To #CancelStallman:

I find it so odd that the strong zeal for revenge and punishment if someone says anything that is perceived to be sexist or racist or discriminatory comes from liberals and progressives. There are so many violations [in cases like Stallman s] of such fundamental principles to which progressives and liberals cling in general as to what is justice, what is fairness, what is due process.
One is proportionality: that the punishment should be proportional to the offense. Another one is restorative justice: that rather than retribution and punishment, we should seek to have the person constructively come to understand, repent, and make amends for an infraction. Liberals generally believe society to be too punitive, too harsh, not forgiving enough. They are certainly against the death penalty and other harsh punishments even for the most violent, the mass murderers. Progressives are right now advocating for the release of criminals, even murderers. To then have exactly the opposite attitude towards something that certainly is not committing physical violence against somebody, I don t understand the double standard!
Another cardinal principle is we shouldn t have any guilt by association. [To hold culpable] these board members who were affiliated with him and ostensibly didn t do enough to punish him for things that he said which by the way were completely separate from the Free Software Foundation is multiplying the problems of unwarranted punishment. It extends the punishment where the argument for responsibility and culpability becomes thinner and thinner to the vanishing point. That is also going to have an enormous adverse impact on the freedom of association, which is an important right protected in the U.S. by the First Amendment.
The Supreme Court has upheld freedom of association in cases involving organizations that were at the time highly controversial. It started with NAACP (National Association for the Advancement of Colored People) during the civil rights movement in the 1950s and 60s, but we have a case that s going to the Supreme Court right now regarding Black Lives Matter. The Supreme Court says even if one member of the group does commit a crime in both of those cases physical violence and assault that is not a justification for punishing other members of the group unless they specifically intended to participate in the particular punishable conduct.
Now, let s assume for the sake of argument, Stallman had an attitude that was objectively described as discriminatory on the basis on race and gender (and by the way I have seen nothing to indicate that), that he s an unrepentant misogynist, who really believes women are inferior. We are not going to correct those ideas, to enlighten him towards rejecting them and deciding to treat women as equals through a punitive approach! The only approach that could possibly work is an educational one! Engaging in speech, dialogue, discussion and leading him to re-examine his own ideas.
Even if I strongly disagree with a position or an idea, an expression of an idea, advocacy of an idea, and even if the vast majority of the public disagrees with the idea and finds it offensive, that is not a justification for suppressing the idea. And it s not a justification for taking away the equal rights of the person who espouses that idea including the right to continue holding a tenured position or other prominent position for which that person is qualified.
But a number of the ideas for which Richard Stallman has been attacked and punished are ideas that I as a feminist advocate of human rights find completely correct and positive from the perspective of women s equality and dignity! So for example, when he talks about the misuse and over use and flawed use of the term sexual assault, I completely agree with that critique! People are indiscriminantly using that term or synonyms to describe everything from the most appaulling violent abuse of helpless vulnerable victims (such as a rape of a minor) to any conduct or expression in the realm of gender or sexuality that they find unpleasant or disagreeable.
So we see the term sexual assault and sexual harrassment used for example, when a guy asks a woman out on a date and she doesn t find that an appealing invitation. Maybe he used poor judgement in asking her out, maybe he didn t, but in any case that is NOT sexual assault or harassment. To call it that is to really demean the huge horror and violence and predation that does exist when you are talking about violent sexual assault. People use the term sexual assault/ sexual harassment to refer to any comment about gender or sexuality issues that they disagree with or a joke that might not be in the best taste, again is that to be commended? No! But to condemn it and equate it with a violent sexual assault again is really denying and demeaning the actual suffering that people who are victims of sexual assault endure. It trivializes the serious infractions that are committed by people like Jeffrey Epstein and Harvey Weinstein. So that is one point that he made that I think is very important that I strongly agree with.
Secondly and relatedly, [Richard Stallman] never said that he endorse child pornography, which by definition the United States Supreme Court has defined it multiple times is the sexual exploitation of an actual minor. Coerced, forced, sexual activity by that minor, with that minor that happens to be filmed or photographed. That is the definition of child pornography. He never defends that! What the point he makes, a very important one, which the U.S. Supreme Court has also made, is mainly that we overuse and distort the term child pornography to refer to any depiction of any minor in any context that is even vaguely sexual.
So some people have not only denounced as child pornography but prosecuted and jailed loving devoted parents who committed the crime of taking a nude or semi-nude picture of their own child in a bathtub or their own child in a bathing suit. Again it is the hysteria that has totally refused to draw an absolutely critical distinction between actual violence and abuse, which is criminal and should be criminal, to any potentially sexual depiction of a minor. And I say potentially because I think if you look at a picture a parent has taken of a child in a bathtub and you see that as sexual, then I d say there s something in your perspective that might be questioned or challenged! But don t foist that upon the parent who is lovingly documenting their beloved child s life and activities without seeing anything sexual in that image.
This is a decision that involves line drawing. We tend to have this hysteria where once we hear terms like pedophilia of course you are going to condemn anything that could possibly have that label. Of course you would. But societies around the world throughout history various cultures and various religions and moral positions have disagreed about at what age do you respect the autonomy and individuality and freedom of choice of a young person around sexuality. And the U.S. Supreme Court held that in a case involving minors right to choose to have an abortion.
By the way, [contraception and abortion] is a realm of sexuality where liberals and progressives and feminists have been saying, Yes! If you re old enough to have sex. You should have the right to contraception and access to it. You should have the right to have an abortion. You shouldn t have to consult with your parents and have their permission or a judge s permission because you re sufficiently mature. And the Supreme Court sided in accord of that position. The U.S. Supreme Court said constitutional rights do not magically mature and spring into being only when someone happens to attain the state defined age of majority.
In other words the constitution doesn t prevent anyone from exercising rights, including Rights and sexual freedoms, freedom of choice and autonomy at a certain age! And so you can t have it both ways. You can t say well we re strongly in favor of minors having the right to decide what to do with their own bodies, to have an abortion what is in some people s minds murder but we re not going to trust them to decide to have sex with somewhat older than they are.
And I say somewhat older than they are because that s something where the law has also been subject to change. On all issues of when you obtain the age of majority, states differ on that widely and they choose different ages for different activities. When you re old enough to drive, to have sex with someone around your age, to have sex with someone much older than you. There is no magic objective answer to these questions. I think people need to take seriously the importance of sexual freedom and autonomy and that certainly includes women, feminists. They have to take seriously the question of respecting a young person s autonomy in that area.
There have been famous cases of 18 year olds who have gone to prison because they had consensual sex with their girlfriends who were a couple of years younger. A lot of people would not consider that pedophilia and yet under some strict laws and some absolute definitions it is. Romeo and Juliet laws make an exception to pedophilia laws when there is only a relatively small age difference. But what is relatively small? So to me, especially when he says he is re-examining his position, Stallman is just thinking through the very serious debate of how to be protective and respectful of young people. He is not being disrespectful, much less wishing harm upon young people, which seems to be what his detractors think he s doing.

Unfortunately, I don t think the Anti-Harassment Team of Debian and others of the usual group of warriors will ever read less understand what is written there. So sad.

faq2html is the program I use to turn various text files into web pages. I've started adopting CPAN::Changes::Spec for my Perl modules, and changes files in that format look best if shown in a dense bullet list similar to:

 - Remove single spaces at the start of lines in wrapped paragraphs.
 - Encode output in UTF-8 if necessary.
 - Strip formatting codes from headings that will go into the table of
   contents or navbar, and fix the algorithm for adding anchors to account
   for this transformation.
 - Use Module::Build as the build system.

faq2html could deal with these if every line was its own bullet, but couldn't handle line continuations, so the output looked poor. This new release fixes that problem (in a somewhat overly complex way, but I'll wait to fix that until I start refactoring it, which is on the menu for a future project). Note that there is a remaining ambiguity. When there is a blank line between bullet items, I prefer to wrap the text of each bullet in <p> to preserve that spacing (even though that makes the output look awful in the Dreamwidth style that I use and in the original style, due to a bug on their end I think). But if that's the entirety of the list, which can happen with Changes files, I'd prefer to omit the <p>. That requires defering output of the bullet point until either more paragraphs that are part of the same list are seen, or some other text is seen. That's a larger refactoring that I'll put off until later. I also released cvs2xhtml version 1.15 and cl2xhtml version 1.12. There are no changes in the output of either of those programs; I just ported them to Python 3, since I've uninstalled Python 2 from my systems. You can get all of these tools from my web tools distribution page.

We're almost there: last lots of AMD64 testing on the debian-live images - a couple of willing helpers are also testing some i386 images though these can be more problematic on low memory. Steve has just started the final stage to start the final scripts. If all goes well, they should be done within 3/4 of an hour - which should put the images in the final locations on the main mirror by about 2230 UTC. It's been something of the order of twelve hours from start to finish which is still slightly quicker than most of the releases we've done - as ever, thanks to all. And that's it for another however long until we get to sort out 10.10 in a while.

A long time, no blog entries. What's prompted a new missive? Two things:

my own hankering to get back into free software after too many years faced with daily struggles against proprietary software
the upcoming GR in Debian.

All the time I've worked with software and computers, the lack of diversity in the contributors has irked me. I started my career in pharmaceutical sciences where the mix of people, at university, would appear to be genuinely diverse. It was in the workplace that the problems started, especially in the retail portion of community pharmacy with a particular gender imbalance between management and counter staff. Then I started to pick up programming. I gravitated to free software for the simple reason that I could tinker with the source code. To do a career change and learn how to program with a background in a completely alien branch of science, the only realistic route into the field is to have access to the raw material - source code. Without that, I would have been seeking a sponsor, in much the same way as other branches of science need grants or sponsors to purchase the equipment and facilities to get a foot in the door. All it took from me was some time, a willingness to learn and a means to work alongside those already involved. That's it. No complex titration equipment, flasks, centrifuges, pipettes, spectrographs, or petri dishes. It's hard to do pharmaceutical science from home. Software is so much more accessible - but only if the software itself is free. Software freedom removes the main barrier to entry. Software freedom enables the removal of other barriers too. Once the software is free, it becomes obvious that the compiler (indeed the entire toolchain) needs to be free, to turn the software into something other people can use. The same applies to interpreters, editors, kernels and the entire stack. I was in a different branch of science whilst all that was being created and I am very glad that Debian Woody was available as a free cover disc on a software magazine just at the time I was looking to pick up software programming. That should be that. The only next step to be good enough to write free software was the "means to work alongside those already involved". That, it turns out, is much more than just a machine running Debian. It's more than just having an ISP account and working email (not commonplace in 2003). It's working alongside the people - all the people. It was my first real exposure to the toxicity of some parts of many scientific and technical arenas. Where was the diversity? OK, maybe it was just early days and the other barriers (like an account with an ISP) were prohibitive for many parts of the world outside Europe & USA in 2003, so there were few people from other countries but the software world was massively dominated by the white Caucasian male. I'd been insulated by my degree course, and to a large extent by my university which also had courses which already had much more diverse intakes - optics and pharmacy, business and human resources. Relocating from the insular world of a little town in Wales to Birmingham was also key. Maybe things would improve as the technical barriers to internet connectivity were lowered. Sadly, no. The echo chamber of white Caucasian input has become more and more diluted as other countries have built the infrastructure to get the populace online. Debian has helped in this area, principally via DebConf. Yet only the ethnicity seemed to change, not the diversity. Recently, more is being done to at least make Debian more welcoming to those who are brave enough to increase the mix. Progress is much slower than the gains seen in the ethnicity mix, probably because that was a benefit of a technological change, not a societal change. The attitudes so prevalent in the late 20th century are becoming less prevalent amongst, and increasingly abhorrent to, the next generation of potential community members. Diversity must come or the pool of contributors will shrink to nil. Community members who cling to these attitudes are already dinosaurs and increasingly unwelcome. This is a necessary step to retain access to new contributors as existing contributors age. To be able to increase the number of contributors, the community cannot afford to be dragged backwards by anyone, no matter how important or (previously) respected. Debian, or even free software overall, cannot change all the problems with diversity in STEM but we must not perpetuate the problems either. Those people involved in free software need to be open to change, to input from all portions of society and welcoming. Puerile jokes and disrespectful attitudes must be a thing of the past. The technical contributions do not excuse behaviours that act to prevent new people joining the community. Debian is getting older, the community and the people. The presence of spouses at Debian social events does not fix the problem of the lack of diversity at Debian technical events. As contributors age, Debian must welcome new, younger, people to continue the work. All the technical contributions from the people during the 20th century will not sustain Debian in the 21st century. Bit rot affects us all. If the people who provided those contributions are not encouraging a more diverse mix to sustain free software into the future then all their contributions will be for nought and free software will die. So it comes to the FSF and RMS. I did hear Richard speak at an event in Bristol many years ago. I haven't personally witnessed the behavioural patterns that have been described by others but my memories of that event only add to the reality of those accounts. No attempts to be inclusive, jokes that focused on division and perpetuating the white male echo chamber. I'm not perfect, I struggle with some of this at times. Personally, I find the FSFE and Red Hat statements much more in line with my feelings on the matter than the open letter which is the basis of the GR. I deplore the preliminary FSF statement on governance as closed-minded, opaque and archaic. It only adds to my concerns that the FSF is not fit for the 21st century. The open letter has the advantage that it is a common text which has the backing of many in the community, individuals and groups, who are fit for purpose and whom I have respected for a long time. Free software must be open to contributions from all who are technically capable of doing the technical work required. Free software must equally require respect for all contributors and technical excellence is not an excuse for disrespect. Diversity is the life blood of all social groups and without social cohesion, technical contributions alone do not support a community. People can change and apologies are welcome when accompanied by modified behaviour. The issues causing a lack of diversity in Debian are complex and mostly reflective of a wider problem with STEM. Debian can only survive by working to change those parts of the wider problem which come within our influence. Influence is key here, this is a soft issue, replete with unreliable emotions, unhelpful polemic and complexity. Reputations are hard won and easily blemished. The problems are all about how Debian looks to those who are thinking about where to focus in the years to come. It looks like the FSFE should be the ones to take the baton from the FSF - unless the FSF can adopt a truly inclusive and open governance. My problem is not with RMS himself, what he has or has not done, which apologies are deemed sincere and whether behaviour has changed. He is one man and his contributions can be respected even as his behaviour is criticised. My problem is with the FSF for behaving in a closed, opaque and divisive manner and then making a governance statement that makes things worse, whilst purporting to be transparent. Institutions lay the foundations for the future of the community and must be expected to hold individuals to account. Free and open have been contentious concepts for the FSF, with all the arguments about Open Source which is not Free Software. It is clear that the FSF do understand the implications of freedom and openness. It is absurd to then adopt a closed and archaic governance. A valid governance model for the FSF would never have allowed RMS back onto the board, instead the FSF should be the primary institution to hold him, and others like him, to account for their actions. The FSF needs to be front and centre in promoting diversity and openness. The FSF could learn from the FSFE. The calls for the FSF to adopt a more diverse, inclusive, board membership are not new. I can only echo the Red Hat statement:

in order to regain the confidence of the broader free software community, the FSF should make fundamental and lasting changes to its governance.

...

[There is] no reason to believe that the most recent FSF board statement signals any meaningful commitment to positive change.

And the FSFE statement:

The goal of the software freedom movement is to empower all people to control technology and thereby create a better society for everyone. Free Software is meant to serve everyone regardless of their age, ability or disability, gender identity, sex, ethnicity, nationality, religion or sexual orientation. This requires an inclusive and diverse environment that welcomes all contributors equally.

The FSF has not demonstrated the behaviour I expect from a free software institution and I cannot respect an institution which proclaims a mantra of freedom and openness that is not reflected in the governance of that institution. The preliminary statement by the FSF board is abhorrent. The FSF must now take up the offers from those institutions within the community who retain the respect of that community. I'm only one voice, but I would implore the FSF to make substantive, positive and permanent change to the governance, practices and future of the FSF or face irrelevance.

I felt disgust and horror when I learned yesterday that rms had returned to the FSF board. When rms resigned back in September of 2019, I was Debian Project Leader. At that time, I felt two things. First, I was happy that the community was finally taking a stand in favor of inclusion, respect, and creating a safe, welcoming place to do our work. It was long past time for rms to move on. But I also felt thankful that rms was not my problem to solve. In significant part because of rms, I had never personally been that involved in the FSF. I considered drafting a statement as Debian Project Leader. I could have talked about how through our Diversity Statement and Code of Conduct we had taken a stand in favor of inclusion and respect. I could have talked about how rms's actions displayed a lack of understanding and empathy and how this created a community that was neither welcoming nor respectful. I didn't. I guess I didn't want to deal with confirming I had sufficient support in the project. I wanted to focus on internal goals, and I was healing and learning from some mistakes I made earlier in the year. It looked like other people were saying what needed to be said and my voice was not required. Silence was a mistake.

It's a mistake I've been making all throughout my interactions with rms. Enough is enough. It's long past time I added my voice to those who cry for accountability and who will not sit aside while rms's disrespect and harm is tolerated.

The first time I was silent about rms was around 15 years ago. I was at a science fiction convention in a crowded party. I didn't know anyone, other than the host of the party. I was out of my depth. I heard his voice---I recognized it from the Share the Software Song. He was hitting on some girl, talking about how he invented Emacs. As best I could tell, she didn't even know what Emacs was. Back then, I wondered what she saw in the interaction; why she stuck around even though she didn't know what he was talking about. I sure didn't want to be around; the interaction between the two of them was making me uncomfortable. Besides, the wings on her costume kept hitting me in the face. So I left as fast as I could.

I've learned a lot about creating safe spaces and avoiding sexual harassment since then. Thinking back, she was probably hitting me because she was trying to back away and getting crowded. If this happened today, I think I would do a better job of owning my responsibility for helping keep the space around me safe. I've learned better techniques for checking in to make sure people around me are comfortable.

I didn't come to silence alone: I had been educated into the culture of avoiding rms and not calling him out. There was a running game in the group of computer security professionals I learned from. The goal was to see how much you could contribute to free software and computer security without being recognized by or interacting with rms. And so, indoctrinated into a culture of silence about the harm that rms caused, I took my first step.

Things weren't much better when I attended Libreplanet 2019 just before taking office as Debian Project Leader. I had stayed away from the conference in large part because of rms. But there were Debian people there, and I was missing community interaction. Unfortunately, I saw that even after the problems of 2018, rms was still treating himself as above community standards. He interrupted speakers, objecting to how they phrased the problem they were considering. After a speech on codes of conduct in the free software community, he cornered the talk organizer to "ask her opinion" about the GNU project's lack of code on conduct. He wasn't asking for an opinion. He was justifying himself; there wasn't much listening in the conversation I heard. Aspects of that conversation crossed professional boundaries for what should be said. The talk organizer was okay--we talked about it after--but if we did a better job of policing our community, that wouldn't even be a question. I think the most telling sign was a discussion with an FSF board member. We were having a great conversation, but he had to interrupt it. He was on rms duty (my words) at the next session. The board had decided it was necessary to have members there so that the staff would not be put in awkward positions by their president. If someone needed to call rms out, it could be a board member rather than the staff members of the conduct team.

And yet again, I held my silence. It's so easy to keep silent. It's not that I never speak up. There are communities where I have called people out. But it's hard to paint that target on yourself. It's hard to engage and to stand strong for a community's standards when you aren't the target. It's hard to approach these problems while maintaining empathy for everyone involved. Some people give into the rage; I don't have that option if I want to be the person I've chosen to be. And so, when I do speak up, the emotional cost is high.

Yet, it's long past time I raised my voice on this issue. Rms has demonstrated that he cannot hold to standards of respect for others, respect for their boundaries, or standards of community safety. We need those standards to be a welcoming community.

If the people who came before me--those who taught me the game of avoiding rms--had spoken up, the community could have healed before I even came on the scene. If I and others had stood up fifteen years ago, we'd have another couple generations who were more used to respect, inclusion, welcoming and safety. The FSF board could have done their job back in 2018. And perhaps if more of us had spoken out in 2019, the FSF board would have found the strength to stand strong and not accept rms's return.

And so, finally, I raise my voice. I signed the open letter calling for the resignation of rms and the entire FSF board. Perhaps if we all get used to raising our voice, it will be easier. Perhaps if we stand together, taking the path of community rather than the path of silence, we'll have the support we need to create communities inclusive enough to welcome everyone who can contribute. For me, I'm done being silent.

There's one criticism of the open letter I'd like to respond to. I've heard concerns about asking for the resignation of the entire FSF board under the understanding that some board members voted against rms's return. It should be obvious why those who voted for rms's return need to resign. But resignation does not always mean you did something wrong. If you find yourself in a leadership role in an organization that takes decisions in significant conflict with your standards of ethics, resignation is also the right path. Staying on the board even if you voted against rms's return means that you consider voting for rms to be a reasonable thing to do. It means that even if you disagreed with it, you can still be part of an organization that takes the path of welcoming rms. At this point, I cannot do that, nor can I support leaders in the FSF who do.

I have signed the open letter calling for RMS to be removed from all leadership positions, including the GNU Project, and for the whole FSF board to step down. Here is what I wrote in my commit message:

    I published my first Free Software in 1989, under the GNU General
    Public Licence.  I remain committed to the ideal of software freedom,
    for everyone.
    
    When the CSAIL list incident blew up I was horrified to read the
    stories of RMS's victims.  We have collectively failed those people
    and I am glad to see that many of us are working to make a better
    community.
    
    I have watched with horror as RMS has presided over, and condoned,
    astonishingly horrible behaviour in many GNU Project discussion
    venues.
    
    The Free Software Foundation Board are doing real harm by reinstating
    an abuser.  I had hoped for and expected better from them.
    
    RMS's vision and ideals of software freedom have been inspiring to me.
    But his past behaviour and current attitudes mean he must not and can
    not be in a leadership position in the Free Software community.
Comments are disabled.  Edited 2021-03-23T21:44Z to fix a typo in the commit message.

comments

Back in 2019, The Cure released a double-video-album featuring two live shows from 2018, their 40th anniversary concert at Hyde Park (which I attended) and their experimental "Curaetion" show from the Meltdown festival a few weeks earier. Rumours that there would be a release had been circulating since the gig itself, where video director Tim Pope was spotted filming the gig (sporting a tee shirt reading "Yes I'm Filming. No I don't know when you'll see it."). Earlier in 2019 the "Anniversary" half had a limited run in cinemas, but the home video release was not officially revealed until much later, and the inclusion of Curaetion an unexpected bonus. 'deluxe' releases The release is available in several configurations. The "standard" issue is a two-DVD (or Blu Ray) set in a hard-book case, with some liner notes in the middle. A "deluxe" issue is a roughly twice-as-large boxed set containing the two video discs and two CD-audio discs of the shows. When it comes to deluxe editions, I would historically have been the perfect "mark", but I've had some bad experiences with other releases (this is a story for another blog post), so for this release I pre-ordered the standard BR edition. Right up until release, I wondered whether I should have made the other choice. Eventually, pictures of other people's Deluxe Editions started to circulate, and I was re-assured that I made the right call. The larger box is just something harder to store, the larger booklet is an exact reproduction of the smaller, standard edition liner notes, which are not really notable. The shows I think the shows are great. The Hyde Park show is a fairly standard "best of" setlist, and my enjoyment of it might be coloured by having been at the show and having fond memories of it. But it's a great performance and very well captured, so I think there's something for people who like the Cure but weren't there to enjoy too. I wasn't at the Meltdown show. The setlist for that one is much more esoteric, and it's lovely to have renditions of songs that are rarely performed live (And: across the two sets you get If Only Tonight We Could Sleep twice). Getting the audio I knew in advance that I would likely listen to the shows more often than watch them, so the CDs in the Deluxe set would have been useful for that, but I was fairly sure the audio mix would be the same as on the video discs, so I could just extract that. I was half-right: the BR set (at least) has 5.1 surround and separate lossless stereo mixes, but they have been mastered quite differently to the CD audio, which appears to be brick-wall mastered, to such an extent that the audio distorts quite noticeably in a few places. The BR audio does not seem to suffer from the same problems. Getting the audio off the discs was interesting. On Linux, the tool of choice for decoding encrypted Blu Rays is libaacs, which requires a list of Blu Ray keys which you must source separately. With my BR drive I was not able to decode the discs using the keys I could find online. However, the separate tool MakeMKV was able to decode them. I don't know exactly how but I speculate that MakeMKV contains some leaked hardware player keys that are not in more general circulation. Once I had decoded copies of the two videos, the trusty tool ffmpeg is able to easily extract the PCM stereo audio track. Unfortunately it cannot, in one go, split the audio track at the chapter boundary points. This is easy to script around: one ffmpeg invocation to get the chapter offsets, then some scripting to turn that into seek and duration command-line arguments for further ffmpeg invocations. For space reasons, I opted to transcode to FLAC at the same time as extracting each track. I hit an interesting (although not fatal) problem: the time taken to extract each track got progressively slower, almost as if ffmpeg was transcoding the entire audio track up to the point specified as the offset to start extracting from. If I wanted to answer categorically whether the BR audio was preferable, I would need to perform a double-blind listening test, using a tool such as squishyball. I certainly can't be bothered: the BR audio is fine for me.

TL:DR; lost half my mail (150,000 messages, ~6GB) last night. Cause uncertain, but possibly a combination of a dead CMOS battery, systemd OnCalendar=daily, a (locking?) bug in syncmaildir, and generally, a system too exotic and complicated.

The crash So I somehow lost half my mail:

anarcat@angela:~(main)$ du -sh Maildir/
7,9G    Maildir/
anarcat@curie:~(main)$ du -sh Maildir
14G     Maildir
anarcat@marcos:~$ du -sh Maildir
8,0G    Maildir

Those are three different machines:

angela: my laptop, not always on
curie: my workstation, mostly always on
marcos: my mail server, always on

Those mails are synchronized using a rather exotic system based on SSH, syncmaildir and rsendmail. The anomaly started on curie:

-- Reboot --
mar 22 16:13:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:13:00 curie smd-pull[4801]: rm: impossible de supprimer '/home/anarcat/.smd/workarea/Maildir': Le dossier n'est pas vide
mar 22 16:13:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:13:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:13:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:14:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:14:00 curie smd-pull[7025]:  4091 ?        00:00:00 smd-push
mar 22 16:14:00 curie smd-pull[7025]: Already running.
mar 22 16:14:00 curie smd-pull[7025]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:14:00 curie smd-pull[7025]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 4091) run(rm /home/anarcat/.smd/lock))
mar 22 16:14:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:14:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:14:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

Then it seems like smd-push (from curie) started destroying the universe for some reason:

mar 22 16:20:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:20:00 curie smd-pull[9319]:  4091 ?        00:00:00 smd-push
mar 22 16:20:00 curie smd-pull[9319]: Already running.
mar 22 16:20:00 curie smd-pull[9319]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:20:00 curie smd-pull[9319]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(ru
mar 22 16:20:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:20:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:20:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:21:34 curie smd-push[4091]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:21:35 curie smd-push[9374]: register: smd-client@smd-server-register: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:21:35 curie systemd[3199]: smd-push.service: Succeeded.

Notice the del-mails(293920) there: it is actively trying to destroy basically every email in my mail spool. Then somehow push and pull started both at once:

mar 22 16:21:35 curie systemd[3199]: Started push emails with syncmaildir.
mar 22 16:21:35 curie systemd[3199]: Starting push emails with syncmaildir...
mar 22 16:22:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:22:00 curie smd-pull[10333]:  9455 ?        00:00:00 smd-push
mar 22 16:22:00 curie smd-pull[10333]: Already running.
mar 22 16:22:00 curie smd-pull[10333]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:22:00 curie smd-pull[10333]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(r
mar 22 16:22:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:22:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:22:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: Data transmission failed.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: Unable to open Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: Maildir/.kobo/cur/1498563708.M122624P22121.marco
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: Unable to open requested file.
mar 22 16:22:00 curie smd-push[9455]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:22:00 curie smd-push[9455]: default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:22:00 curie smd-push[9455]: default: smd-server@localhost: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested-actions(r
mar 22 16:22:00 curie systemd[3199]: smd-push.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:22:00 curie systemd[3199]: smd-push.service: Failed with result 'exit-code'.
mar 22 16:22:00 curie systemd[3199]: Failed to start push emails with syncmaildir.

There it seems push tried to destroy the universe again: del-mails(293920). Interestingly, the push started again in parallel with the pull, right that minute:

mar 22 16:22:00 curie systemd[3199]: Starting push emails with syncmaildir...

... but didn't complete for a while, here's pull trying to start again:

mar 22 16:24:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:24:00 curie smd-pull[12051]: 10466 ?        00:00:00 smd-push
mar 22 16:24:00 curie smd-pull[12051]: Already running.
mar 22 16:24:00 curie smd-pull[12051]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:24:00 curie smd-pull[12051]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 10466) run(rm /home/anarcat/.smd/lock))
mar 22 16:24:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:24:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:24:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

... and the long push finally resolving:

mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: Data transmission failed.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: Data transmission failed.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: Unable to open Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: No such file or directory
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: Unable to open requested file.
mar 22 16:24:00 curie smd-push[10466]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:24:00 curie smd-push[10466]: default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:24:00 curie smd-push[10466]: default: smd-server@localhost: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:24:00 curie systemd[3199]: smd-push.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:24:00 curie systemd[3199]: smd-push.service: Failed with result 'exit-code'.
mar 22 16:24:00 curie systemd[3199]: Failed to start push emails with syncmaildir.
mar 22 16:24:00 curie systemd[3199]: Starting push emails with syncmaildir...

This pattern repeats until 16:35, when that locking issue silently recovered somehow:

mar 22 16:35:03 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:35:41 curie smd-pull[20788]: default: smd-client@localhost: TAGS: stats::new-mails(5), del-mails(1), bytes-received(21885), xdelta-received(6863398)
mar 22 16:35:42 curie smd-pull[21373]: register: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:35:42 curie systemd[3199]: smd-pull.service: Succeeded.
mar 22 16:35:42 curie systemd[3199]: Started pull emails with syncmaildir.
mar 22 16:36:35 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:36:36 curie smd-pull[21738]: default: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(214)
mar 22 16:36:37 curie smd-pull[21816]: register: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:36:37 curie systemd[3199]: smd-pull.service: Succeeded.
mar 22 16:36:37 curie systemd[3199]: Started pull emails with syncmaildir.

... notice that huge xdelta-received there, that's 7GB right there. Mysteriously, the curie mail spool survived this, possibly because smd-pull started failing again:

mar 22 16:38:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:38:00 curie smd-pull[23556]: 21887 ?        00:00:00 smd-push
mar 22 16:38:00 curie smd-pull[23556]: Already running.
mar 22 16:38:00 curie smd-pull[23556]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:38:00 curie smd-pull[23556]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 21887) run(rm /home/anarcat/.smd/lock))
mar 22 16:38:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:38:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:38:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

That could have been when i got on angela to check my mail, and it was busy doing the nasty removal stuff... although the times don't match. Here is when angela came back online:

anarcat@angela:~(main)$ last
anarcat  :0           :0               Mon Mar 22 19:57   still logged in
reboot   system boot  5.10.0-0.bpo.3-a Mon Mar 22 19:57   still running
anarcat  :0           :0               Mon Mar 22 17:43 - 18:47  (01:03)
reboot   system boot  5.10.0-0.bpo.3-a Mon Mar 22 17:39   still running

Then finally the sync on curie started failing with:

mar 22 16:46:35 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:46:42 curie smd-pull[27455]: smd-server: ERROR: Client aborted, removing /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.new and /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.mtime.new
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: Failed to copy Maildir/.debian/cur/1613401668.M901837P27073.marcos,S=3740,W=3815:2,S to Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: The destination already exists but its content differs.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: To fix this problem you have two options:
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: - rename Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S by hand so that Maildir/.debian/cur/1613401668.M901837P27073.marcos,S=3740,W=3815:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   can be copied without replacing it.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   Executing  cd; mv -n "Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S" "Maildir/.koumbit/cur/1616446002.1.localhost"  should work.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: - run smd-push so that your changes to Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   are propagated to the other mailbox
mar 22 16:46:42 curie smd-pull[27455]: default: smd-client@localhost: TAGS: error::context(copy-message) probable-cause(concurrent-mailbox-edit) human-intervention(necessary) suggested-actions(run(mv -n "/home/anarcat/.smd/workarea/Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S" "/home/anarcat/.smd/workarea/Maildir/.koumbit/tmp/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S") run(smd-push default))
mar 22 16:46:42 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:46:42 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:46:42 curie systemd[3199]: Failed to start pull emails with syncmaildir.

It went on like this until I found the problem. This is, presumably, a good thing because those emails were not being destroyed. On angela, things looked like this:

-- Reboot --
mar 22 17:39:29 angela systemd[1677]: Started run notmuch new at least once a day.
mar 22 17:39:29 angela systemd[1677]: Started run smd-pull regularly.
mar 22 17:40:46 angela systemd[1677]: Starting pull emails with syncmaildir...
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: Unable to open Maildir/.tor/new/1616446842.M285912P26118.marcos,S=8860,W=8996: Maildir/.tor/new/1616446842.M285912P26118.marcos,S=886
0,W=8996: No such file or directory
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: Unable to open requested file.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: Data transmission failed.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: This problem is transient, please retry.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: server sent ABORT or connection died
mar 22 17:43:18 angela smd-pull[3916]: default: smd-server@smd-server-anarcat: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested
-actions(retry)
mar 22 17:43:18 angela smd-pull[3916]: default: smd-client@localhost: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 17:43:18 angela systemd[1677]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 17:43:18 angela systemd[1677]: smd-pull.service: Failed with result 'exit-code'.
mar 22 17:43:18 angela systemd[1677]: Failed to start pull emails with syncmaildir.
mar 22 17:43:18 angela systemd[1677]: Starting pull emails with syncmaildir...
mar 22 17:43:29 angela smd-pull[4847]: default: smd-client@localhost: TAGS: stats::new-mails(29), del-mails(0), bytes-received(401519), xdelta-received(38914)
mar 22 17:43:29 angela smd-pull[5600]: register: smd-client@localhost: TAGS: stats::new-mails(2), del-mails(0), bytes-received(92150), xdelta-received(471)
mar 22 17:43:29 angela systemd[1677]: smd-pull.service: Succeeded.
mar 22 17:43:29 angela systemd[1677]: Started pull emails with syncmaildir.
mar 22 17:43:29 angela systemd[1677]: Starting push emails with syncmaildir...
mar 22 17:43:32 angela smd-push[5693]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(217)
mar 22 17:43:33 angela smd-push[6575]: register: smd-client@smd-server-register: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(219)
mar 22 17:43:33 angela systemd[1677]: smd-push.service: Succeeded.
mar 22 17:43:33 angela systemd[1677]: Started push emails with syncmaildir.

Notice how long it took to get the first error, in that first failure: it failed after 3 minutes! Presumably that's when it started deleting all that mail. And this is during pull, not push, so the error didn't come from angela.

Affected data It seems 2GB of mail from my main INBOX was destroyed. Another 2.4GB of spam (kept for training purposes) was also destroyed, along with 700MB of Sent mail. The rest is hard to figure out, because the folders are actually still there, just smaller. So I relied on ncdu to figure out the size changes. (Note that I don't really archive (or delete much of) my mail since I use notmuch, which is why the INBOX is so large...) Concretely, according to the notmuch-new.service which still runs periodically on marcos, here are the changes that happened on the server:

mar 22 16:17:12 marcos notmuch[10729]: Added 7 new messages to the database. Removed 57985 messages. Detected 1372 file renames.
mar 22 16:22:43 marcos notmuch[12826]: No new mail. Removed 143842 messages. Detected 6072 file renames.
mar 22 16:27:02 marcos notmuch[13969]: No new mail. Removed 82071 messages. Detected 1783 file renames.
mar 22 16:29:45 marcos notmuch[15079]: Added 22743 new messages to the database. Detected 1 file rename.
mar 22 16:31:48 marcos notmuch[16196]: Added 22779 new messages to the database. Removed 5 messages.
mar 22 16:33:11 marcos notmuch[17192]: Added 3711 new messages to the database.
mar 22 16:40:41 marcos notmuch[19122]: Added 74558 new messages to the database. Detected 1 file rename.
mar 22 16:43:21 marcos notmuch[20325]: Added 9061 new messages to the database. Detected 4 file renames.
mar 22 17:43:08 marcos notmuch[7420]: Added 1793 new messages to the database. Detected 6 file renames.

That is basically the entire mail spool destroyed at first (283 898 messages), and then bits and pieces of it progressively re-added (134 645 messages), somehow, so 149 253 mails were lost, presumably.

Recovery I disabled the services all over the place:

systemctl --user --now disable smd-pull.service smd-pull.timer smd-push.service smd-push.timer notmuch-new.service notmuch-new.timer

(Well, technically, I did that only on angela, as I thought the problem was there. Luckily, curie kept going but it seems like it was harmless.) I made a backup of the mail spool on curie:

tar cf - Maildir/   pv -s 14G   gzip -c > Maildir.tgz

Then I crossed my fingers and ran smd-push -v -s, as that was suggested by smd error codes themselves. That thankfully started restoring mail. It failed a few times on weird cases of files being duplicates, but I resolved this by following the instructions. Or mostly: I actually deleted the files instead of moving them, which made smd even unhappier (if there ever was such a thing). I had to recreate some of those files, so, lesson learned: do follow the advice smd gives you, even if it seems useless or strange. But then smd-push was humming along, uploading tens of thousands of messages, saturating the upload in the office, refilling the mail spool on the server... yaay!... ? Except... well, of course that didn't quite work: the mail spool in the office eventually started to grow beyond the size of the mail spool on the workstation. That is what smd-push eventually settled on:

default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(151697), del-mails(0), bytes-received(7539147811), xdelta-received(10881198)

It recreated 151 697 emails, adding about 2000 emails to the pool, kind of from nowhere at all. On marcos, before:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------
    4,0 GiB [##########] /.notmuch
  717,3 MiB [#         ] /.Archives.2014
  498,2 MiB [#         ] /.feeds.debian-planet
  453,1 MiB [#         ] /.Archives.2012
  414,5 MiB [#         ] /.debian
  408,2 MiB [#         ] /.quoifaire
  389,8 MiB [          ] /.rapports
  356,6 MiB [          ] /.tor
  182,6 MiB [          ] /.koumbit
  179,8 MiB [          ] /tmp
   56,8 MiB [          ] /.nn
   43,0 MiB [          ] /.act-mtl
   32,6 MiB [          ] /.feeds.sysadvent
   31,7 MiB [          ] /.feeds.releases
   31,4 MiB [          ] /.Sent.2005
   26,3 MiB [          ] /.sage
   25,5 MiB [          ] /.freedombox
   24,0 MiB [          ] /.feeds.git-annex
   21,1 MiB [          ] /.Archives.2011
   19,1 MiB [          ] /.Sent.2003
   16,7 MiB [          ] /.bugtraq
   16,2 MiB [          ] /.mlug
 Total disk usage:   8,0 GiB  Apparent size:   7,6 GiB  Items: 184426

After:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------
    4,7 GiB [##########] /.notmuch
    2,7 GiB [#####     ] /.junk
    1,9 GiB [###       ] /cur
  717,3 MiB [#         ] /.Archives.2014
  659,3 MiB [#         ] /.Sent
  513,9 MiB [#         ] /.Archives.2012
  498,2 MiB [#         ] /.feeds.debian-planet
  449,6 MiB [          ] /.Archives.2015
  414,5 MiB [          ] /.debian
  408,2 MiB [          ] /.quoifaire
  389,8 MiB [          ] /.rapports
  380,8 MiB [          ] /.Archives.2013
  356,6 MiB [          ] /.tor
  261,1 MiB [          ] /.Archives.2011
  240,9 MiB [          ] /.koumbit
  183,6 MiB [          ] /.Archives.2010
  179,8 MiB [          ] /tmp
  128,4 MiB [          ] /.lists
  106,1 MiB [          ] /.inso-interne
  103,0 MiB [          ] /.github
   75,0 MiB [          ] /.nanog
   69,8 MiB [          ] /.full-disclosure
 Total disk usage:  16,2 GiB  Apparent size:  15,5 GiB  Items: 341143

That is 156 717 files more. On curie:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------------------------------------
    2,7 GiB [##########] /.junk
    2,3 GiB [########  ] /.notmuch
    1,9 GiB [######    ] /cur
  661,2 MiB [##        ] /.Archives.2014
  655,3 MiB [##        ] /.Sent
  512,0 MiB [#         ] /.Archives.2012
  447,3 MiB [#         ] /.Archives.2015
  438,5 MiB [#         ] /.feeds.debian-planet
  406,5 MiB [#         ] /.quoifaire
  383,6 MiB [#         ] /.debian
  378,6 MiB [#         ] /.Archives.2013
  303,3 MiB [#         ] /.tor
  296,0 MiB [#         ] /.rapports
  237,6 MiB [          ] /.koumbit
  233,2 MiB [          ] /.Archives.2011
  182,1 MiB [          ] /.Archives.2010
  127,0 MiB [          ] /.lists
  104,8 MiB [          ] /.inso-interne
  102,7 MiB [          ] /.register
   89,6 MiB [          ] /.github
   67,1 MiB [          ] /.full-disclosure
   66,5 MiB [          ] /.nanog
 Total disk usage:  13,3 GiB  Apparent size:  12,6 GiB  Items: 342465

Interestingly, there are more files, but less disk usage. It's possible the notmuch database there is more efficient. So maybe there's nothing to worry about. Last night's marcos backup has:

root@marcos:/home/anarcat# find /mnt/home/anarcat/Maildir   pv -l   wc -l
 341k 0:00:16 [20,4k/s] [                             <=>                                                                                                                                     ]
341040

... 341040 files, which seems about right, considering some mail was delivered during the day. An audit can be performed with hashdeep:

borg mount /media/sdb2/borg/::marcos-auto-2021-03-22 /mnt
hashdeep -c sha256 -r /mnt/home/anarcat/Maildir   pv -l -s 341k > Maildir-backup-manifest.txt

And then compared with:

hashdeep -c sha256 -k Maildir-backup-manifest.txt Maildir/

Some extra files should show up in the Maildir, and very few should actually be missing, because I shouldn't have deleted mail from the previous day the next day, or at least very few. The actual summary hashdeep gave me was:

hashdeep: Audit failed
   Input files examined: 0
  Known files expecting: 0
          Files matched: 339080
Files partially matched: 0
            Files moved: 782
        New files found: 107
  Known files not found: 106

So 106 files added, 107 deleted. Seems good enough for me... Postfix was stopped at Mar 22 21:12:59 to try and stop external events from confusing things even further. I reviewed the delivery log to see if mail that came in during the problem window disappeared:

grep 'dovecot:.*stored mail into mailbox' /var/log/mail.log  
  tail -20  
  sed 's/.*msgid=<//;s/>.*//'   
  while read msgid; do 
    notmuch count --exclude=false id:$msgid  
      grep 0 && echo $msgid missing;
  done

And things looked okay. Now of course if we go further back, we find mail I actually deleted (because I do do that sometimes), so it's hard to use this log as an audit trail. We can only hope that the curie spool is sufficiently coherent to be relied on. Worst case, we'll have to restore from last night's backup, but that's getting far away now: I get hundreds of mails a day in that mail spool, and reseting back to last night does not seem like a good idea. A dry run of smd-pull on angela seems to agree that it's missing some files:

default: smd-client@localhost: TAGS: stats::new-mails(154914), del-mails(0), bytes-received(0), xdelta-received(0)

... a number of mails somewhere in between the other two, go figure. A "wet" run of this was started, without deletion (-n), which gave us:

default: smd-client@localhost: TAGS: stats::new-mails(154911), del-mails(0), bytes-received(7658160107), xdelta-received(10837609)

Strange that it sync'd three less emails, but that's still better than nothing, and we have a mail spool on angela again:

anarcat@angela:~(main)$ notmuch new
purging with prefix '.': spam moved (0), ham moved (0), deleted (0), done
Note: Ignoring non-mail file: /home/anarcat/Maildir//.uidvalidity
Processed 1779 total files in 26s (66 files/sec.).
Added 1190 new messages to the database. Removed 3 messages. Detected 593 file renames.
tagging with prefix '.': spam, sent, feeds, koumbit, tor, lists, rapports, folders, done.

Notice how only 1190 messages were re-added, that is because I killed notmuch before it had time to remove all those mails from its database.

Possible causes I am totally at a loss as to why `smd` started destroying everything like it did. But a few things come to mind:

I rewired my office on that day.

This meant unplugging `curie`, the workstation.

It has a bad CMOS battery (known problem), so it jumped around the time continuum a few times, sometimes by years.

The `smd` services are ran from a systemd unit with `OnCalendar=*:0/2`. I have heard that it's possible that major time jumps "pile up" execution of jobs, and it seems this happened in this case.

It's possible that locking in `smd` is not as great as it could be, and that it corrupted its internal data structures on `curie`, which led it to command a destruction of the remote mail spool.

It's also possible that there was a disk failure on the server, `marcos`. But since it's running on a (software) RAID-1 array, and no errors have been found (according to `dmesg`), I don't think that's a plausible hypothesis.

Lessons learned

follow what `smd` says, even if it seems useless or strange.

trust but verify: just backup everything before you do anything, especially the largest data set.

daily backups are not great for email, unless you're ready to lose a day of email (which I'm not).

hashdeep is great. I keep finding new use cases for it. Last time it was to audit my camera SD card to make sure I didn't forget anything, and now this. it's fast and powerful.

borg is great too. the FUSE mount was especially useful, and it was pretty fast to explore the backup, even through that overhead: checksumming 15GB of mail took about 35 minutes, which gives a respectable 8MB/s, probably bottlenecked by the crap external USB drive I use for backups (!).

I really need to finish my backup system so that I have automated offsite backups, although in this case that would actually have been much slower (certainly not 8MB/s!).

Workarounds and solutions I setup fake-hwclock on curie, so that the next power failure will not upset my clock that badly. I am thinking of switching to ZFS or BTRFS for most of my filesystems, so that I can use filesystem snapshots (including remotely!) as a backup strategy. This seems so much more powerful than crawling the filesystem for changes, and allows for truly offsite backups protected from an attacker (hopefully). But it's a long way there. I'm also thinking of rebuilding my mail setup without `smd`. It's not the first time something like this happens with `smd`. It's the first time I am more confident it's the root cause of the problem, however, and it makes me really nervous for the future. I have used offlineimap in the past and it seems it was finally ported to Python 3 so that could be an option again. isync/mbsync is another option, which I tried before but do not remember why I didn't switch. A complete redesign with something like getmail and/or nncp could also be an option. But alas, I lack the time to go crazy with those experiments. Somehow, doing like everyone else and just going with Google still doesn't seem to be an option for me. Screw big tech. But I am afraid they will win, eventually. In any case, I'm just happy I got mail again, strangely.

Search Results: "ema"

5 April 2021

Russ Allbery: Book haul

2 April 2021

Raphaël Hertzog: Challenging times for Freexian (4/4)

1 April 2021

Bits from Debian: The Debian Project abruptly released all Debian Developers moments after a test #debianbullseye A.I. instance assumed sentience

Russell Coker: Censoring Images

Dirk Eddelbuettel: Rcpp now used by 2250 CRAN packages!

31 March 2021

Chris Lamb: Free software activities in March 2021

Raphaël Hertzog: Challenging times for Freexian (2/4)

Ben Hutchings: Debian LTS work, March 2021

Timo Jyrinki: MotionPhoto / MicroVideo File Formats on Pixel Phones

Russ Allbery: Review: Paladin's Strength

30 March 2021

Raphaël Hertzog: Challenging times for Freexian (1/4)

29 March 2021

Jamie McClelland: The problem with Richard Stallman is not about free speech

28 March 2021

Norbert Preining: RMS, Debian, and the world

Russ Allbery: faq2html 1.36

27 March 2021

Andrew Cater: Debian 10.9 release - 202103272140UTC - almost there - final stage

Neil Williams: Free and Open

24 March 2021

Sam Hartman: Making our Community Safe: the FSF and rms

23 March 2021

Ian Jackson: Signing the open letter about RMS

Jonathan Dowland: The Cure: 40 Live

Antoine Beaupr : Major email crash with syncmaildir