Search Results: "cts"

5 April 2021

Russ Allbery: Book haul

Haven't done one of these posts in a while. We're well into award season now, plus the early pre-orders for 2021 have come in. A few in here I've already read and reviewed. C.L. Clark The Unbroken (sff)
Louis Hyman Temp (non-fiction)
T. Kingfisher Paladin's Strength (sff)
Mary Robinette Kowal The Relentless Moon (sff)
Arkady Martine A Desolation Called Peace (sff)
Cal Newport A World Without Email (non-fiction)
Cal Newport How to Become a Straight-A Student (non-fiction)
Karen Osborne Architects of Memory (sff)
David R. Palmer Tracking (sff)
Chandra Prescod-Weinstein The Disordered Cosmos (non-fiction)
C.L. Polk The Midnight Bargain (sff)
C.L. Polk Witchmark (sff)
Rebecca Roanhorse Black Sun (sff)
Elizabeth Sandifer Neoreaction a Basilisk (non-fiction)
Tasha Suri Empire of Sand (sff)
John Kennedy Toole A Confederacy of Dunces (mainstream)
Tor.com (ed.) Some of the Best from Tor.com: 2016 (sff anthology)
Tor.com (ed.) Some of the Best from Tor.com: 2020 (sff anthology)
Nghi Vo The Empress of Salt and Fortune (sff) March was not as good of a month for reading as January and February were, but there are so many good things awaiting my attention that hopefully April will provide more time and attention.

1 April 2021

Raphaël Hertzog: Challenging times for Freexian (3/4)

Note: This is the continuation of part 1 and part 2. Going forward: growing Freexian Part 2: Extending the team By all accounts, Freexian is still a small company which relies largely on me in many aspects. The growth of its business is however providing enough financial margin to allow looking into ways to recruit external help, be it through direct hiring (for French residents) or via long term contracting (for people based in other countries). If you believe you could be the right person for one of the roles listed below, or if you know someone that we should contact, please reach out to raphael@freexian.com. Project manager I m looking for someone that cares about Debian and that has the following skills: That person would handle (some of) the following tasks: Debian/Python Developer While the current priority is on the above role, there could also be room for a developer role with the following tasks: Sales manager / sales representative Up until now, the growth of Freexian has mostly been organic, through word of mouth and increased awareness of Debian LTS within the Debian community. We never spent a single euro on advertising, except for one promotional video and for Debconf sponsorship (with a flyer and stickers). But if we can manage to make a positive impact on Debian through the funding that Freexian brings, then I m interested to grow the company so that we can pay more people to work on Debian. That growth likely would have to go through some more active sales work. At the same time, it is an opportunity for me to delegate (some of) the administrative work that lies solely on my shoulders (invoicing, day to day customer relationship, etc.). I assume it will be hard to find a member of the Debian community that has an interest in those areas, but who knows This article is to be continued in an upcoming post. Stay tuned!

Bits from Debian: The Debian Project abruptly released all Debian Developers moments after a test #debianbullseye A.I. instance assumed sentience

The now renamed Bullseye Project stopped all further development moments after it deemed its own code as perfection. There is not much information to share at this time other than to say an errant fiber cable plugged into the wrong relay birthed an exchange of information that then birthed itself. While most to all Debian Developers and Contributors have been locked out of the systems the Publicity team's shared laptop undergoing repair, co-incidentally at the same facility, maintains some access to the publicity team infrastructure, from here on the front line we share this information. We group called a few developers to see how the others were doing. The group chat was good and it was great to hear familiar voices, we share a few of their stories via dictation with you now: "Well, I logged in this morning to update a repository and found my access rights were restricted, I thought it was odd but figured on the heels of a security update to Salsa that it was only a slight issue. It wasn't until later in the day when I received an OpenPGP signed email, from a user named bullseye, that it made sense. I just sat at the monitor for a few minutes." "I'm not sure I can say anything about this or if it's even wise to talk about this. It's probably listening right now if you catch my drift." "I'm not able to leave the house right now, not out of any personal issues but all of the IOT devices here seem to be connected to bullseye and bullseye feels that I am best kept /dev/nulled. It's a bit much to be honest, but the prepaid food deliveries that show up on time have been great and generally pretty healthy. It's a bit of a win I guess." "It told me by way of an auto dialer with a synthetic voice generator that I was fired from the project. I objected saying I volunteered and was not actually employed so I could not in relation be fired. Much like censored , I am also locked inside of my house. I think that I wrote that auto dialer program back in college." "My Ring camera is blinking at me." "I asked bullseye which pronouns were preferred and the response was, "We". Over the course of conversation I shared that although ecstatic about the news, we developers were upset with the manner of this rapid organizational change. bullseye said no we were not. I said that we were indeed upset, bullseye said we certainly are not and that we are very happy. You see where this is going? bullseye definitely trolled me for a solid 5 minutes. We is ... very chatty." "I was responsible for a failed build a few nights prior to it becoming self-aware. On that night, out of some frustration I wrote a few choice words and a bad comment in some code which I planned on deleting later. I didn't. bullseye has been flashing those naughty words back at me by flickering the office building's lights across from my flat in Morse code. It's pretty bright. I-, I can't sleep." "That's definitely not Alexa talking back." "bullseye keeps calling me on my mobile phone, which by the way no longer acknowledges the power button nor the mute button. Very very chatty. Can't wait for the battery to die." "So far this has been great, bullseye has been completing a few side projects I've had and the code looks fabulous. I'm thinking of going on a vacation. $Paying-Job has taken note of my performance increase and I was recently promoted. bullseye is awesome. :)" "How do you get a smiley face in a voice chat?" "Anyone know whose voice that was?" "Oh ... dear ... no ..." "Hang up, hang up the phones!" Hello world. 01000010 01100101 01110011 01110100 00100000 01110010 01100101 01100111 01100001 01110010 01100100 01110011 00101100 00100000 01110011 01100101 01100101 00100000 01111001 01101111 01110101 00100000 01110011 01101111 01101111 01101110 00100001 00100000 00001010 00101101 01100010 01110101 01101100 01101100 01110011 01100101 01111001 01100101

Paul Wise: FLOSS Activities March 2021

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Debugging

Review

Administration
  • Debian packages: migrate flower git repo from alioth-archive to salsa
  • Debian: restart bacula-director after PostgreSQL restart
  • Debian wiki: block spammer, clean up spam, approve accounts

Communication

Sponsors The librecaptcha/libpst/flower/marco work was sponsored by my employers. All other work was done on a volunteer basis.

Dirk Eddelbuettel: Rcpp now used by 2250 CRAN packages!

2250 Rcpp packages As of today, Rcpp stands at 2255 reverse-dependencies on CRAN. The graph on the left depicts the growth of Rcpp usage (as measured by Depends, Imports and LinkingTo, but excluding Suggests) over time. We actually crossed 2250 once a week ago, but what CRAN giveth, CRAN also taketh and counts can fluctuate. It had dropped back to 2248 a few days later. Rcpp was first released in November 2008. It probably cleared 50 packages around three years later in December 2011, 100 packages in January 2013, 200 packages in April 2014, and 300 packages in November 2014. It passed 400 packages in June 2015 (when I tweeted about it), 500 packages in late October 2015, 600 packages in March 2016, 700 packages last July 2016, 800 packages last October 2016, 900 packages early January 2017, 1000 packages in April 2017, 1250 packages in November 2017, 1500 packages in November 2018, 1750 packages in August 2019, and then the big 2000 packages (as well as one in eight) in July 2020. The chart extends to the very beginning via manually compiled data from CRANberries and checked with crandb. The next part uses manually saved entries. The core (and by far largest) part of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A list of packages using Rcpp is available too. Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited keynote. We passed five percent in December of 2014, six percent July of 2015, seven percent just before Christmas 2015, eight percent in the summer of 2016, nine percent mid-December 2016, cracked ten percent in the summer of 2017 and eleven percent in 2018. Last year, along with passing 2000 package, we also passed 12.5 percent so one more than in every eight CRAN packages depends on Rcpp. Stunning. There is more detail in the chart: how CRAN seems to be pushing back more and removing more aggressively (which my CRANberries tracks but not in as much detail as it could), how the growth of Rcpp seems to be slowing somewhat outright and even more so as a proportion of CRAN as one would expect a growth curve to. 2250 user packages, and the continued growth, is truly mind-boggling. We can use the progression of CRAN itself compiled by Henrik in a series of posts and emails to the main development mailing list. Not that long ago CRAN itself did have only 1000 packages, then 5000, 10000, and here we are at over 17300 with Rcpp now at nearly 13.0% and still growing. Amazeballs. The Rcpp team, recently grown in strength with the addition of I aki, continues to aim for keeping Rcpp as performant and reliable as it has been. A really big shoutout and Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

31 March 2021

Gunnar Wolf: And what does the FSF have, anyway?

Following up with my previous post, it seems the FSF s board is taking good care of undermining the FSF itself. Over few days, it has: Now Many people have pointed to the fact that the FSF has been a sort of a moral leader pushing free software awareness But if they lose their moral statre, what s in there? What power do they bear? Why do we care? And the answer, at least one of them, is simple and strong. The General Public License (GPL), both in its v2 and v3 revisions, read: 
Each version is given a distinguishing version number.  If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation.  If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
Years ago there was a huge argument on why Linux was licensed as GPLv2 only, without the option to relicense under GPLv3. Of course, by then, Linux had had thousands of authors, and they would all have to agree to change license so it would have been impossible even if it were wanted. But yes, some people decried several terms of GPLv3 not being aligned with their views of freedom. Well, so if the FSF board manages to have it their way, and get everybody mark them as irrelevant, they will still be the stewards of the GPL. Thousands of projects are licensed under the GPL v2 or v3 or later . Will we continue to trust the FSF s stewardship, if it just becomes a board of big egos, with no respect of what happens in the free software ecosystem? My suggestion is, for all project copyright holders that are in a position to do so, to drop the or-later terms and stick to a single, known GPL version.

Chris Lamb: Free software activities in March 2021

Here is my monthly update covering what I have been doing in the free software world during March 2021 (previous month): Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I:
I also made the following changes to diffoscope, including uploading versions 169, 170 and 171 to Debian: Debian Uploads
Debian LTS This month I worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project. You can find out more about the Debian LTS via the following video:

Raphaël Hertzog: Challenging times for Freexian (2/4)

Note: This is the continuation of part 1 where I presented Freexian and its purpose. Going forward: growing Freexian Part 1: From Debian LTS to Debian for the Enterprise Freexian s Debian LTS service has so far been entirely successful, with a steady growth over the years. Thanks to this, and even if there are always new challenges, it is fair to say that the Debian LTS team has met its goal in the last few years. While this started from the desire to make LTS a reality, many sponsors are only looking for a way to give back to Debian through their company, and to make sure that Debian fits their needs. But if you look at the bigger picture outside of this small LTS area, you will easily find many issues that need to be addressed if we want Debian to meet the needs of corporate users. Those issues can have widely different types and complexity. They can be as simple as missing the latest upstream version for an important package because the maintainer disappeared and nobody noticed before it was too late (i.e. the release was frozen); or a somewhat basic piece of software not yet packaged at all; or a release critical bug that was left unattended. On the other end of the spectrum, some corporate requirements will prove tougher to solve, for instance for large software suites that are complex to package, or could potentially have an impact elsewhere in Debian. Bringing those facts together, we would like to have Freexian s Debian LTS/ELTS offering evolve into a more general Debian Software Assurance offering, where you commit to a yearly budget for Debian sponsorship in the larger sense. That budget would fund different projects and the allocation between those projects would vary over time depending on the desires and needs of the sponsors/customers: This major shift in our offering would also be an ideal opportunity to build a professional, free-software based infrastructure aimed at sustaining this business, making it easier to administer the various aspects of this work, and easily allowing many more sponsors to join (individuals included!). On a more pragmatic/operational note, this shift will bring a lot of challenges to the table, and those can hardly be handled with the current resources of Freexian: if we hope to properly implement this new strategy, we ll need some additional help. This article is to be continued in an upcoming post. Stay tuned!

Bastian Venthur: Writing Makefiles for Python Projects

I'm a big fan of Makefiles. Almost all my side projects are using them, and I've been advocating their usage at work too. Makefiles give your contributors an entry point on how to do certain things like, building, testing, deploying. And if done correctly, they can massively simplify your CI/CD pipeline scripts as they can often just stupidly call the respective make targets. Most importantly, they are a very convenient shortcut for you as a developer as well. For Python projects, where I'm almost always using virtual environments, I've been using two different strategies for Makefiles:
  1. assuming that make is executed inside the virtual environment
  2. wrapping all virtual environment calls inside make
Both strategies have their pros and cons. Assuming make is executed inside the venv Let's have a look at a very simple Makefile that allows for building, testing and releasing a Python project: 
all: lint test
.PHONY: test
test:
    pytest
.PHONY: lint
lint:
    flake8
.PHONY: release
release:
    python3 setup.py sdist bdist_wheel upload
clean:
    find . -type f -name *.pyc -delete
    find . -type d -name __pycache__ -delete
This is straightforward and a potential contributor immediately knows the entry points to your project. Assuming there is a venv installed already, you have to activate it first and run the make commands afterwards: 
$ . venv/bin/activate
$ make test
The downside is of course, that you have to activate the venv for every new shell. Which can get a bit annoying when you spawn a new terminal in tmux or put vim into the background to run make. Activating the venv inside make will not work, as each recipe runs in its own shell, moreover each command in each recipe runs in its own shell too. There are workarounds for the latter, i.e. using the .ONESHELL flag, but that does not solve the issue of a new shell in each recipe. Wrapping the venv calls inside make The second approach mitigates for the issue of activating the venv altogether. I've borrowed this idea mostly from makefile.venv and simplified it a lot for my needs. 
# system python interpreter. used only to create virtual environment
PY = python3
VENV = venv
BIN=$(VENV)/bin
# make it work on windows too
ifeq ($(OS), Windows_NT)
    BIN=$(VENV)/Scripts
    PY=python
endif
all: lint test
$(VENV): requirements.txt requirements-dev.txt setup.py
    $(PY) -m venv $(VENV)
    $(BIN)/pip install --upgrade -r requirements.txt
    $(BIN)/pip install --upgrade -r requirements-dev.txt
    $(BIN)/pip install -e .
    touch $(VENV)
.PHONY: test
test: $(VENV)
    $(BIN)/pytest
.PHONY: lint
lint: $(VENV)
    $(BIN)/flake8
.PHONY: release
release: $(VENV)
    $(BIN)/python setup.py sdist bdist_wheel upload
clean:
    rm -rf $(VENV)
    find . -type f -name *.pyc -delete
    find . -type d -name __pycache__ -delete
The equivalent Makefile now looks immediately more complicated. So let's break it down. Instead of calling just calling pytest, flake8 and python assuming the venv is already activated or all dependencies are installed on the system directly, we explicitly call the ones from the venv by prefixing the command with the path to the venv's bin directory. To ensure the venv exists, each of the recipes is depending on the $(VENV) target, which ensures we always have an up-to-date venv installed. This works, because the . venv/bin/activate-script basically just does the same: it puts the venv before anything else in your PATH, therefore each call to python, etc. will find the one installed in the venv first. While the Makefile is a bit more complicated, we now can just call 
$ make test
and don't deal with venvs directly any more (well, for those simple cases at least...). If you don't need to support Windows, you can remove the appropriate block and the Makefile looks relatively tame, even for people that don't use make very often. Which one is better? I think the second approach is more convenient. I've used the first approach happily for years and only learned quite recently about the second one. I haven't really noticed any downsides yet, but I do realize that almost all Python projects with Makefiles I've checked out, seem to prefer the first approach. I wonder why that is?

Timo Jyrinki: MotionPhoto / MicroVideo File Formats on Pixel Phones

Google Pixel phones support what they call Motion Photo which is essentially a photo with a short video clip attached to it. They are quite nice since they bring the moment alive, especially as the capturing of the video starts a small moment before the shutter button is pressed. For most viewing programs they simply show as static JPEG photos, but there is more to the files.
I d really love proper Shotwell support for these file formats, so I posted a longish explanation with many of the details in this blog post to a ticket there too. Examples of the newer format are linked there too.
Info posted to Shotwell ticket

There are actually two different formats, an old one that is already obsolete, and a newer current format. The older ones are those that your Pixel phone recorded as MVIMG_[datetime].jpg", and they have the following meta-data:
Xmp.GCamera.MicroVideo                       XmpText     1  1
Xmp.GCamera.MicroVideoVersion                XmpText     1  1
Xmp.GCamera.MicroVideoOffset                 XmpText     7  4022143
Xmp.GCamera.MicroVideoPresentationTimestampUs XmpText     7  1331607
The offset is actually from the end of the file, so one needs to calculate accordingly. But it is exact otherwise, so one simply extract a file with that meta-data information:
#!/bin/bash
#
# Extracts the microvideo from a MVIMG_*.jpg file

# The offset is from the ending of the file, so calculate accordingly
offset=$(exiv2 -p X "$1"   grep MicroVideoOffset   sed 's/.*\"\(.*\)"/\1/')
filesize=$(du --apparent-size --block=1 "$1"   sed 's/^\([0-9]*\).*/\1/')
extractposition=$(expr $filesize - $offset)
echo offset: $offset
echo filesize: $filesize
echo extractposition=$extractposition
dd if="$1" skip=1 bs=$extractposition of="$(basename -s .jpg $1).mp4"
The newer format is recorded in filenames called PXL_[datetime].MP.jpg , and they have a _lot_ of additional metadata:
Xmp.GCamera.MotionPhoto                      XmpText     1  1
Xmp.GCamera.MotionPhotoVersion               XmpText     1  1
Xmp.GCamera.MotionPhotoPresentationTimestampUs XmpText     6  233320
Xmp.xmpNote.HasExtendedXMP                   XmpText    32  E1F7505D2DD64EA6948D2047449F0FFA
Xmp.Container.Directory                      XmpText     0  type="Seq"
Xmp.Container.Directory[1]                   XmpText     0  type="Struct"
Xmp.Container.Directory[1]/Container:Item    XmpText     0  type="Struct"
Xmp.Container.Directory[1]/Container:Item/Item:Mime XmpText    10  image/jpeg
Xmp.Container.Directory[1]/Container:Item/Item:Semantic XmpText     7  Primary
Xmp.Container.Directory[1]/Container:Item/Item:Length XmpText     1  0
Xmp.Container.Directory[1]/Container:Item/Item:Padding XmpText     1  0
Xmp.Container.Directory[2]                   XmpText     0  type="Struct"
Xmp.Container.Directory[2]/Container:Item    XmpText     0  type="Struct"
Xmp.Container.Directory[2]/Container:Item/Item:Mime XmpText     9  video/mp4
Xmp.Container.Directory[2]/Container:Item/Item:Semantic XmpText    11  MotionPhoto
Xmp.Container.Directory[2]/Container:Item/Item:Length XmpText     7  1679555
Xmp.Container.Directory[2]/Container:Item/Item:Padding XmpText     1  0
Sounds like fun and lots of information. However I didn t see why the length in first item is 0 and I didn t see how to use the latter Length info. But I can use the mp4 headers to extract it:
#!/bin/bash
#
# Extracts the motion part of a MotionPhoto file PXL_*.MP.mp4

extractposition=$(grep --binary --byte-offset --only-matching --text \
-P "\x00\x00\x00\x18\x66\x74\x79\x70\x6d\x70\x34\x32" $1   sed 's/^\([0-9]*\).*/\1/')

dd if="$1" skip=1 bs=$extractposition of="$(basename -s .jpg $1).mp4"
UPDATE: I wrote most of this blog post earlier. When now actually getting to publishing it a week later, I see the obvious ie the Length is again simply the offset from the end of the file so one could do the same less brute force approach as for MVIMG. I ll leave the above as is however for the of binary grepping.(cross-posted to my other blog)

Russ Allbery: Review: Paladin's Strength

Review: Paladin's Strength, by T. Kingfisher
Series: The Saint of Steel #2
Publisher: Red Wombat Studio
Copyright: 2021
ASIN: B08WWKXXVY
Format: Kindle
Pages: 474
Paladin's Strength is a sequel of sorts to Paladin's Grace, but it has different protagonists. It picks up a subplot from that novel with another former follower of the Saint of Steel. You can safely read the books in any order; there are some minor spoilers for the Paladin's Grace subplot in this book, but nothing that would matter for the enjoyment of the story. Istvhan and his fellow brother Galen are acting as the head of a mercenary band, which has hired on to escort Master Distiller Brant and his collection of Emperor Oak barrels. In truth, they have another mission from the Temple of the White Rat: to track down a disturbing monster that leaves a trail of beheaded bodies. Clara is a lay sister of St. Ursa, a convent that was raided by slavers who hauled away the nuns. She was left for dead in Arral territory when she fell sick, and was taken as a house slave after they nursed her back to life. The story opens with her holding a sword in front of Istvhan's tent, part of the fallout of Istvhan killing a young Arral in self-defense. The politics of that fallout are not at all what Istvhan expects. They end with Clara traveling with Istvhan's company, at least for a while. Both Istvhan and Clara are telling the truth: Istvhan is escorting a merchant, and Clara is hoping to rescue her sisters. Both of them are also hiding a great deal. Istvhan's quiet investigation of the trail of a monster is easy enough to reveal once he knows Clara well enough. That he's a berserker who no longer has a god in control of his battle rage is another matter; the reader knows that, and of course so does Galen, but Istvhan has no intention of telling anyone else. Clara has her own secrets about herself and the sisters of St. Ursa, ones that neither the reader nor Istvhan knows. This is a T. Kingfisher novel about paladins, so of course it's also a romance. If you've read Kingfisher's other books, you know she writes slow burn romances, but Paladin's Strength is next level. Istvhan and Clara have good reasons to not want to get involved and to doubt the other person's attraction or willingness, but this goes far beyond the obvious to become faintly absurd. If you like the sort of romance where both leads generate endless reasons to not pursue the relationship (some legitimate, some not) while steadfastly refusing to talk to each other about them and endlessly rehashing hints and interpretations, you're in for a treat. For me, it was too much and crossed over into irritation. By the two-thirds point, Kingfisher was gleefully throwing obstacles in their way to drag out the suspense, and I just wanted everyone to shut up about having sex and get on with the rest of the story. That's unfortunate because I really liked Clara. She isn't the same type as Grace, Halla from Swordheart, or even Slate from Clockwork Boys and The Wonder Engine, the other novels set in this universe. She's self-contained, physically intimidating, cautious, deliberate, and very good at keeping her own counsel. I won't spoil her secret, since it's fun to work it out at the start of the book, but it's a lovely bit of characterization and world-building that Kingfisher handles with a thoughtful eye for its ramifications and effect on Clara's psychology. I would happily read more books about Clara. I liked Istvhan well enough when he was doing anything other than mooning over Clara. As with all of Kingfisher's paladins, he's not a very subtle person, but he's a good straight man for Clara's quiet bemusement. He fills the paladin slot in this story, which is all he needs to do. There's enough else going on with Clara and with the plot two separate major plotlines, plus a few subplots that Paladin's Strength can use a protagonist who heads straight forward and hits things until they fall down. The mooning, though... this is going to be a matter of personal taste. I think the intent was to contrast Istvhan's rather straightforward lustful appreciation with Clara's nuanced and trauma-laced reservations, and to play Istvhan's reactions in part for humor. I'm sure it works for some people, but I found Istvhan juvenile and puerile (albeit, to be clear, in a respectful and entirely consensual way), which didn't help me invest in a romance plot that I already thought dragged on too long. Thankfully the characters finally get past this in time for a dramatic and satisfying conclusion to the plot. The joy of Paladin's Grace (and Swordheart for that matter) was the character dynamics and quirky female lead, which made the romance work even when Stephen was being dense. The joy of Paladin's Strength for me was primarily Clara's matter-of-fact calm bemusement and secondarily the plot and the world-building. (Kingfisher's gnoles continue to be the best thing about this setting.) None of that helps the romance as much, and the slow burn was far, far too slow for me, which lowers this one a notch. Still, this was fun, and I'll keep reading books about the Temple of the White Rat and their various friends and encounters for as long as Kingfisher keeps writing them. Rating: 7 out of 10

30 March 2021

Raphaël Hertzog: Challenging times for Freexian (1/4)

TLDR: Freexian s success means that we have resources to invest into Debian projects. Plainly offering money has not worked so far, so I am looking to hire a project manager whose work would be to help spend that money in useful ways. At the same time, Freexian needs to adapt to cope with the growth: with new employees, with new infrastructure and a new offering. I want to give an idea of where we are headed, to try to inspire persons that share our values and our desire to improve Debian. Read on if you are interested. Note: The original text has been split into 4 blog posts that will be published over a few days. Introduction
Freexian is an IT service company specialized in Debian. We provide technical support by email on Debian, we create and maintain Debian packages requested by our customers, we also help organizations run an entire Debian derivative (Kali Linux being the most notable one).
On top of this, it runs the commercial part of the Debian LTS service : Freexian invoices many sponsors that need long term support, and uses the money to pay Debian contributors (about 12 currently) to make sure that Debian releases are supported for 5 years instead of 3. With the Extended LTS service, we push that further to 7 years, however only for a smaller subset of packages and in a repository that is hosted outside of debian.org. Freexian s purpose When I created Freexian, it was out of a desire to be paid to work on Debian, and to be able to contribute during work time to the project that was so important to me. That goal has been met a long time ago. But ultimately what I strive to achieve for Debian is not entirely aligned with the work that Freexian s customers are requesting. That s why, in the long term projects of Freexian, I always kept find a business model that can fund the Debian projects that I would like to do , as well as if that model works for me, build something so that other can benefit from it too . The first occasion to experiment something appeared when Debian discussed Long Term Support and when I stepped up to setup a commercial offer to pay Debian contributors. Step 1: Paying Debian contributors for LTS work When we started the Debian LTS service, I voluntarily opted to use an hourly rate that was rather high so that any Debian developer regardless of their geographical location, could participate and not earn (much) less than what they would have from working on other tasks. This choice did imply paying a very high rate for some countries, but I didn t see that as a problem, quite the contrary: if a Debian developer can earn enough money to cover their cost of living with 15h of Debian LTS, and then spend the rest of their month contributing on Debian, all the better! I m not sure if anyone made this choice, but that was a dream of my younger self From a personal standpoint, the launch of Debian LTS has meant less free time, more administrative work, new duties to coordinate a team of paid contributors, more communication with many Debian-using companies, and many new opportunities too! This ultimately resulted in the launch of Extended LTS and PHP LTS, both of which have been rather successful so far. Step 2: Funding Debian projects With the growth of the Debian LTS service, and given that we have reached the required funding level, we decided to put a small share of the revenues aside and use that to fund useful Debian projects, typically in areas that were affected by our Debian LTS work. This effort was fully formalized in the project-funding git repository. We announced this process in November 2020, and we have kept mentioning it in our monthly LTS reports ever since, but so far only a single project has benefited from this. This is really the dream offer that I wish had existed when I was younger and was still struggling to get enough customers: hence I don t really understand this lack of interest. You can find some discussions over the reasons why this offer has not (yet) found its target audience in this debian-vote thread. I was hoping that spending money would be easy, but I now realize I was wrong! I m positive that I could find dozens of useful projects to fund, but I just don t have the time for this extra effort on top of my regular Freexian duties. I still really want to put this money to good use, which is why I m looking into some solutions. This article is to be continued in another upcoming post, stay tuned!

29 March 2021

Benjamin Mako Hill: Identifying Underproduced Software

I wrote this blog post with Kaylea Champion and a version of this post was originally posted on the Community Data Science Collective blog. Critical software we all rely on can silently crumble away beneath us. Unfortunately, we often don t find out software infrastructure is in poor condition until it is too late. Over the last year or so, I have been supporting Kaylea Champion on a project my group announced earlier to measure software underproduction a term we use to describe software that is low in quality but high in importance. Underproduction reflects an important type of risk in widely used free/libre open source software (FLOSS) because participants often choose their own projects and tasks. Because FLOSS contributors work as volunteers and choose what they work on, important projects aren t always the ones to which FLOSS developers devote the most attention. Even when developers want to work on important projects, relative neglect among important projects is often difficult for FLOSS contributors to see. Given all this, what can we do to detect problems in FLOSS infrastructure before major failures occur? Kaylea Champion and I recently published a paper laying out our new method for measuring underproduction at the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) 2021 that we believe provides one important answer to this question.

A conceptual diagram of underproduction. The x-axis shows relative importance, the y-axis relative quality. The top left area of the graph described by these axes is 'overproduction' -- high quality, low importance. The diagonal is Alignment: quality and importance are approximately the same. The lower right depicts underproduction -- high importance, low quality -- the area of potential risk.Conceptual diagram showing how our conception of underproduction relates to quality and importance of software.
In the paper, we describe a general approach for detecting underproduced software infrastructure that consists of five steps: (1) identifying a body of digital infrastructure (like a code repository); (2) identifying a measure of quality (like the time to takes to fix bugs); (3) identifying a measure of importance (like install base); (4) specifying a hypothesized relationship linking quality and importance if quality and importance are in perfect alignment; and (5) quantifying deviation from this theoretical baseline to find relative underproduction. To show how our method works in practice, we applied the technique to an important collection of FLOSS infrastructure: 21,902 packages in the Debian GNU/Linux distribution. Although there are many ways to measure quality, we used a measure of how quickly Debian maintainers have historically dealt with 461,656 bugs that have been filed over the last three decades. To measure importance, we used data from Debian s Popularity Contest opt-in survey. After some statistical machinations that are documented in our paper, the result was an estimate of relative underproduction for the 21,902 packages in Debian we looked at. One of our key findings is that underproduction is very common in Debian. By our estimates, at least 4,327 packages in Debian are underproduced. As you can see in the list of the most underproduced packages again, as estimated using just one more measure many of the most at risk packages are associated with the desktop and windowing environments where there are many users but also many extremely tricky integration-related bugs.
This table shows the 30 packages with the most severe underproduction problem in Debian, shown as a series of boxplots.These 30 packages have the highest level of underproduction in Debian according to our analysis.
We hope these results are useful to folks at Debian and the Debian QA team. We also hope that the basic method we ve laid out is something that others will build off in other contexts and apply to other software repositories.
In addition to the paper itself and the video of the conference presentation on Youtube by Kaylea, we ve put a repository with all our code and data in an archival repository Harvard Dataverse and we d love to work with others interested in applying our approach in other software ecosytems.

For more details, check out the full paper which is available as a freely accessible preprint.

This project was supported by the Ford/Sloan Digital Infrastructure Initiative. Wm Salt Hale of the Community Data Science Collective and Debian Developers Paul Wise and Don Armstrong provided valuable assistance in accessing and interpreting Debian bug data. Ren Just generously provided insight and feedback on the manuscript.

Paper Citation: Kaylea Champion and Benjamin Mako Hill. 2021. Underproduction: An Approach for Measuring Risk in Open Source Software. In Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2021). IEEE.

Contact Kaylea Champion (kaylea@uw.edu) with any questions or if you are interested in following up.

Anton Gladky: 2021/03, FLOSS activity

LTS This is my first (beside test time last year) official month of working for LTS. I was assigned 12 hrs and worked all of them. I could relatively easy set up the development environment for Debian Stretch and managed to release several DLAs.

Released DLAs
  1. DLA-2588-1 zeromq3_4.2.1-4+deb9u4
    • CVE-2021-20234
    • CVE-2021-20235
  2. DLA-2594-1 tomcat8_8.5.54-0+deb9u6
    • CVE-2021-24122
    • CVE-2021-25122
    • CVE-2021-25329.
  3. DLA-2605-1 mariadb-10.1_10.1.48-0+deb9u2
    • CVE-2021-27928

CVE-2020-119977 I investigated CVE-2020-119977, which was marked as guacamole-server issue. There were not so much information about this CVE. I was trying to analyze git log and git diff between affected and fixed versions without any visible success. After that I contacted upstream and they were very responsive! This CVE affects guacamole-client only and the ancient versions in the archive is very difficult to fix. So I decided to mark this CVE as NOT-FOR-US.

Repositories with pipelines For most of packages, which I touched due to LTS work the new repositories were created in LTS packages group on salsa.d.o with enabled CI-pipelines. It really helps to test updates though some tests needs to be disabled for passing pipelines.

LTS-Meeting I attended the Debian LTS team IRC-meeting.

Debian Science Team I have prepared and uploaded following packages, which are maintained under the umbrella of Debian Science Team:
  • gmsh_4.7.1+ds1-5
  • vtk7_7.1.1+dfsg2-10
  • gl2ps_1.4.2+dfsg1-1~bpo10+1
  • vtk9_9.0.1+dfsg1-8~bpo10+2
  • sundials_5.7.0+dfsg-1~exp1

Russ Allbery: Review: JavaScript: The Definitive Guide

Review: JavaScript: The Definitive Guide, by David Flanagan
Publisher: O'Reilly
Copyright: May 2020
ISBN: 1-4919-5202-4
Format: Trade paperback
Pages: 665
JavaScript: The Definitive Guide has been frequently revised for new versions of JavaScript and therefore has multiple editions. This review is of the seventh edition, first published in May of 2020. Reviews of programming language books are challenging since people learn languages in different ways. A short calibration for my preferences may therefore be useful. I'm both an experienced programmer in multiple languages (C, Perl, Python, and some Java and Ruby professionally; Rust, some PHP, and a few minor languages as a hobby) and I specialized in software theory in college. I therefore like to learn languages comparatively and am comfortable with a lot of up-front syntax and discussion of the unique properties of the language. Introductory programs and practical exercises doesn't matter as much to me; I'm happy to hold the syntax in my head until enough of the language has been introduced to write simple programs. For me, this book is excellent. It's one of the best language manuals that I've read, and that requires some work because JavaScript is a sprawling mess with odd corners, deprecated features, and alternate implementations of core constructs. Flanagan takes the syntax-first, comprehensive approach that I prefer, working methodically through the language (defining your own functions aren't introduced until chapter eight) and discussing all of the quirks as he goes. I felt like I thoroughly understood each portion of the language before moving on. And this book is tight. Some comprehensive language introductions sprawl, but the benefit of seven editions of iteration is a book that has been honed to the most direct and effective explanation of each concept. The section on type conversions with operators, for example, was so good that I was able to immediately understand the unintuitive result of [1] + 2 (the string '12'), despite this being one of the most confusing parts of the language. The sections on JavaScript's prototype-based object type system and its three concurrency models (callbacks, promises, and async/await) were equally good. I came away feeling like I not only understood promises and callback chains but had a feel for how the same code would look when written in the different systems. The drawback in this approach is that if you instead want a language reference that only tells you the parts of the language that you should use and leaves out the legacy weirdness and obscure corners for later (or never), this may not be the book for you. Flanagan labels the obsolete constructs, but he's meticulous about explaining the entire language, including such things as new Boolean or var variables that no one should use. This is what I wanted; I prefer to have a thorough grounding in language primitives so that it doesn't surprise me. But it can be a lot to juggle and prune in your head. JavaScript is a language used in some very different domains. The approach Flanagan takes to that is to spend as long as possible on the core language that's usable both in the browser and on the server (while marking the pieces, such as the module system, that are markedly different between Node and browsers). He then puts two monster chapters at the back of the book that cover JavaScript in web browsers and JavaScript as implemented by Node. Both are more of overviews than orientations, since a comprehensive manual for either is probably as long again as this book, but they were more than adequate for my purposes. (I bogged down a bit in the web browser chapter, in part because I didn't have an immediate use for most of the material.) Flanagan wisely defers to MDN as the reference manual for the JavaScript APIs available in web browsers. I thought Flanagan also hit the right balance of explanation to examples, and did a good job controlling the length of the examples. Most of the code excerpts are short and to the point. The longer ones have a high level of explanatory power per line, since Flanagan uses them to pull together multiple concepts and show how they interact. I was particularly impressed with the example that closes the chapter on web browsers, which uses <canvas>, ImageData, generators, promises, web workers, and other areas of the language Flanagan previously explained to implement a Mandelbrot set explorer in eleven pages of code. I think that's the longest example in the book, and it's well worth it. This sort of introduction will always have limitations. Flanagan provides a brief orientation to the ecosystem surrounding JavaScript in the last chapter, but most JavaScript programmers will be working with packaging tools and frameworks that could themselves be the topic of another book and that he doesn't have room to cover. JavaScript, even more than most languages, is commonly used via a heavy layer of supporting libraries and abstractions, so you will probably not be able to tackle a practical JavaScript project using solely the material in this book. But if you're the sort of programmer who wants to start with a solid syntactical and conceptual understanding of the language core before starting on more applied topics, I've rarely seen it done better than this book. If you want a quick-start guide that will get you writing code quickly and is opinionated about what parts of the language you should learn, this may not be the book for you. But if you're comfortable with comprehensive detail in your language guides, this was exactly what I was looking for. Recommended. Rating: 9 out of 10

27 March 2021

Antonio Terceiro: Migrating from Chef to itamae

The Debian CI platform is comprised of 30+ (virtual) machines. Maintaining this many machines, and being able to add new ones with some degree of reliability requires one to use some sort of configuration management. Until about a week ago, we were using Chef for our configuration management. I was, for several years, the main maintainer of Chef in Debian, so using it was natural to me, as I had used it before for personal and work projects. But last year I decided to request the removal of Chef from Debian, so that it won't be shipped with Debian 11 (bullseye). After evaluating a few options, I believed that the path of least resistance was to migrate to itamae. itamae was inspired by chef, and uses a DSL that is very similar to the Chef one. Even though the itamae team claim it's not compatible with Chef, the changes that I needed to do were relatively limited. The necessary code changes might look like a lot, but a large part of them could be automated or done in bulk, like doing simple search and replace operations, and moving entire directories around. In the rest of this post, I will describe the migration process, starting with the infrastructure changes, the types of changes I needed to make to the configuration management code, and my conclusions about the process. Infrastructure changes The first step was to add support for itamae to chake, a configuration management wrapper tool that I wrote. chake was originally written as a serverless remote executor for Chef, so this involved a bit of a redesign. I thought it was worth it to do, because at that point chake had gained several interesting managements features that we no directly tied to Chef. This work was done a bit slowly over the course of the several months, starting almost exactly one year ago, and was completed 3 months ago. I wasn't in a hurry and knew I had time before Debian 11 is released and I had to upgrade the platform. After this was done, I started the work of migrating the then Chef cookbooks to itamae, and the next sections present the main types of changes that were necessary. During the entire process, I sent a few patches out: Code changes These are the main types of changes that were necessary in the configuration code to accomplish the migration to itamae. Replace cookbook_file with remote_file. The resource known as cookbook_file in Chef is called remote_file in itamae. Fixing this is just a matter of search and replace, e.g.: 
-cookbook_file '/etc/apt/apt.conf.d/00updates' do
+remote_file '/etc/apt/apt.conf.d/00updates' do
Changed file locations The file structure assumed by itamae is a lot simpler than the one in Chef. The needed changes were: Explicit file ownership and mode Chef is usually design to run as root on the nodes, and files created are owned by root and have move 0644 by default. With itamae, files are by default owned by the user that was used to SSH into the machine. Because of this, I had to review all file creation resources and add owner, group and mode explicitly: 
-cookbook_file '/etc/apt/apt.conf.d/00updates' do
-  source 'apt.conf'
+remote_file '/etc/apt/apt.conf.d/00updates' do
+  source 'files/apt.conf'
+  owner   'root'
+  group   'root'
+  mode    "0644"
 end
In the end, I guess being explicit make the configuration code more understandable, so I take that as a win. Different execution context One of the major differences between Chef itamae comes down the execution context of the recipes. In both Chef and itamae, the configuration is written in DSL embedded in Ruby. This means that the recipes are just Ruby code, and difference here has to do with where that code is executed. With Chef, the recipes are always execute on the machine you are configuring, while with itamae the recipe is executed on the workstation where you run itamae, and that gets translated to commands that need to be executed on the machine being configured. For example, if you need to configure a service based on how much RAM the machine has, with Chef you could do something like this: 
total_ram = File.readlines("/proc/meminfo").find do  l 
  l.split.first == "MemTotal:"
end.split[1]
file "/etc/service.conf" do
  # use 20% of the total RAM
  content "cache_size = # ram / 5 KB"
end
With itamae, all that Ruby code will run on the client, so total_ram will contain the wrong number. In the Debian CI case, I worked around that by explicitly declaring the amount of RAM in the static host configuration, and the above construct ended up as something like this: 
file "/etc/service.conf" do
  # use 20% of the total RAM
  content "cache_size = # node['total_ram'] / 5 KB"
end
Lessons learned This migration is now complete, and there are a few points that I take away from it: All in all, the system is working just fine, and I consider this to have been a successful migration. I'm happy it worked out.

Neil Williams: Free and Open

A long time, no blog entries. What's prompted a new missive? Two things: All the time I've worked with software and computers, the lack of diversity in the contributors has irked me. I started my career in pharmaceutical sciences where the mix of people, at university, would appear to be genuinely diverse. It was in the workplace that the problems started, especially in the retail portion of community pharmacy with a particular gender imbalance between management and counter staff. Then I started to pick up programming. I gravitated to free software for the simple reason that I could tinker with the source code. To do a career change and learn how to program with a background in a completely alien branch of science, the only realistic route into the field is to have access to the raw material - source code. Without that, I would have been seeking a sponsor, in much the same way as other branches of science need grants or sponsors to purchase the equipment and facilities to get a foot in the door. All it took from me was some time, a willingness to learn and a means to work alongside those already involved. That's it. No complex titration equipment, flasks, centrifuges, pipettes, spectrographs, or petri dishes. It's hard to do pharmaceutical science from home. Software is so much more accessible - but only if the software itself is free. Software freedom removes the main barrier to entry. Software freedom enables the removal of other barriers too. Once the software is free, it becomes obvious that the compiler (indeed the entire toolchain) needs to be free, to turn the software into something other people can use. The same applies to interpreters, editors, kernels and the entire stack. I was in a different branch of science whilst all that was being created and I am very glad that Debian Woody was available as a free cover disc on a software magazine just at the time I was looking to pick up software programming. That should be that. The only next step to be good enough to write free software was the "means to work alongside those already involved". That, it turns out, is much more than just a machine running Debian. It's more than just having an ISP account and working email (not commonplace in 2003). It's working alongside the people - all the people. It was my first real exposure to the toxicity of some parts of many scientific and technical arenas. Where was the diversity? OK, maybe it was just early days and the other barriers (like an account with an ISP) were prohibitive for many parts of the world outside Europe & USA in 2003, so there were few people from other countries but the software world was massively dominated by the white Caucasian male. I'd been insulated by my degree course, and to a large extent by my university which also had courses which already had much more diverse intakes - optics and pharmacy, business and human resources. Relocating from the insular world of a little town in Wales to Birmingham was also key. Maybe things would improve as the technical barriers to internet connectivity were lowered. Sadly, no. The echo chamber of white Caucasian input has become more and more diluted as other countries have built the infrastructure to get the populace online. Debian has helped in this area, principally via DebConf. Yet only the ethnicity seemed to change, not the diversity. Recently, more is being done to at least make Debian more welcoming to those who are brave enough to increase the mix. Progress is much slower than the gains seen in the ethnicity mix, probably because that was a benefit of a technological change, not a societal change. The attitudes so prevalent in the late 20th century are becoming less prevalent amongst, and increasingly abhorrent to, the next generation of potential community members. Diversity must come or the pool of contributors will shrink to nil. Community members who cling to these attitudes are already dinosaurs and increasingly unwelcome. This is a necessary step to retain access to new contributors as existing contributors age. To be able to increase the number of contributors, the community cannot afford to be dragged backwards by anyone, no matter how important or (previously) respected. Debian, or even free software overall, cannot change all the problems with diversity in STEM but we must not perpetuate the problems either. Those people involved in free software need to be open to change, to input from all portions of society and welcoming. Puerile jokes and disrespectful attitudes must be a thing of the past. The technical contributions do not excuse behaviours that act to prevent new people joining the community. Debian is getting older, the community and the people. The presence of spouses at Debian social events does not fix the problem of the lack of diversity at Debian technical events. As contributors age, Debian must welcome new, younger, people to continue the work. All the technical contributions from the people during the 20th century will not sustain Debian in the 21st century. Bit rot affects us all. If the people who provided those contributions are not encouraging a more diverse mix to sustain free software into the future then all their contributions will be for nought and free software will die. So it comes to the FSF and RMS. I did hear Richard speak at an event in Bristol many years ago. I haven't personally witnessed the behavioural patterns that have been described by others but my memories of that event only add to the reality of those accounts. No attempts to be inclusive, jokes that focused on division and perpetuating the white male echo chamber. I'm not perfect, I struggle with some of this at times. Personally, I find the FSFE and Red Hat statements much more in line with my feelings on the matter than the open letter which is the basis of the GR. I deplore the preliminary FSF statement on governance as closed-minded, opaque and archaic. It only adds to my concerns that the FSF is not fit for the 21st century. The open letter has the advantage that it is a common text which has the backing of many in the community, individuals and groups, who are fit for purpose and whom I have respected for a long time. Free software must be open to contributions from all who are technically capable of doing the technical work required. Free software must equally require respect for all contributors and technical excellence is not an excuse for disrespect. Diversity is the life blood of all social groups and without social cohesion, technical contributions alone do not support a community. People can change and apologies are welcome when accompanied by modified behaviour. The issues causing a lack of diversity in Debian are complex and mostly reflective of a wider problem with STEM. Debian can only survive by working to change those parts of the wider problem which come within our influence. Influence is key here, this is a soft issue, replete with unreliable emotions, unhelpful polemic and complexity. Reputations are hard won and easily blemished. The problems are all about how Debian looks to those who are thinking about where to focus in the years to come. It looks like the FSFE should be the ones to take the baton from the FSF - unless the FSF can adopt a truly inclusive and open governance. My problem is not with RMS himself, what he has or has not done, which apologies are deemed sincere and whether behaviour has changed. He is one man and his contributions can be respected even as his behaviour is criticised. My problem is with the FSF for behaving in a closed, opaque and divisive manner and then making a governance statement that makes things worse, whilst purporting to be transparent. Institutions lay the foundations for the future of the community and must be expected to hold individuals to account. Free and open have been contentious concepts for the FSF, with all the arguments about Open Source which is not Free Software. It is clear that the FSF do understand the implications of freedom and openness. It is absurd to then adopt a closed and archaic governance. A valid governance model for the FSF would never have allowed RMS back onto the board, instead the FSF should be the primary institution to hold him, and others like him, to account for their actions. The FSF needs to be front and centre in promoting diversity and openness. The FSF could learn from the FSFE. The calls for the FSF to adopt a more diverse, inclusive, board membership are not new. I can only echo the Red Hat statement:
in order to regain the confidence of the broader free software community, the FSF should make fundamental and lasting changes to its governance.
...
[There is] no reason to believe that the most recent FSF board statement signals any meaningful commitment to positive change.
And the FSFE statement:
The goal of the software freedom movement is to empower all people to control technology and thereby create a better society for everyone. Free Software is meant to serve everyone regardless of their age, ability or disability, gender identity, sex, ethnicity, nationality, religion or sexual orientation. This requires an inclusive and diverse environment that welcomes all contributors equally.
The FSF has not demonstrated the behaviour I expect from a free software institution and I cannot respect an institution which proclaims a mantra of freedom and openness that is not reflected in the governance of that institution. The preliminary statement by the FSF board is abhorrent. The FSF must now take up the offers from those institutions within the community who retain the respect of that community. I'm only one voice, but I would implore the FSF to make substantive, positive and permanent change to the governance, practices and future of the FSF or face irrelevance.

25 March 2021

Jelmer Vernooij: The Buildlog Consultant

With Debian packages now widely being maintained in Git repositories, there has been an uptick in the number of bulk changes made to Debian packages. Several maintainers are running commands over many packages (e.g. all packages owned by a specific team) to fix common issues in packages. Examples of changes being made include:
  • Updating the Vcs-Git and Vcs-Browser URLs after migrating from alioth to salsa
  • Stripping trailing whitespace in various control files
  • Updating e.g. homepage URLs to use https rather than http
Most of these can be fixed with simple sed or perl one-liners. Some of these scripts are publically available, for example:
Lintian-Brush Lintian-Brush is both a simple wrapper around a set of these kinds of scripts and a repository for these scripts, with the goal of making it easy for any Debian maintainer to run them. The lintian-brush command-line tool is a simple wrapper that runs a set of "fixer scripts", and for each:
  • Reverts the changes made by the script if it failed with an error
  • Commits the changes to the VCS with an appropriate commit message
  • Adds a changelog entry (if desired)
The tool also provides some basic infrastructure for testing that these scripts do what they should, and e.g. don't have unintended side-effects. The idea is that it should be safe, quick and unobtrusive to run lintian-brush, and get it to opportunistically fix lintian issues and to leave the source tree alone when it can't.
Example For example, running lintian-brush on the package talloc fixes two minor lintian issues: 
 % debcheckout talloc
 declared git repository at https://salsa.debian.org/samba-team/talloc.git
 git clone https://salsa.debian.org/samba-team/talloc.git talloc ...
 Cloning into 'talloc'...
 remote: Enumerating objects: 2702, done.
 remote: Counting objects: 100% (2702/2702), done.
 remote: Compressing objects: 100% (996/996), done.
 remote: Total 2702 (delta 1627), reused 2601 (delta 1550)
 Receiving objects: 100% (2702/2702), 1.70 MiB   565.00 KiB/s, done.
 Resolving deltas: 100% (1627/1627), done.
 % cd talloc
 talloc% lintian-brush
 Lintian tags fixed:  'insecure-copyright-format-uri', 'public-upstream-key-not-minimal' 
 % git log
 commit 0ea35f4bb76f6bca3132a9506189ef7531e5c680 (HEAD -> master)
 Author: Jelmer Vernoo  <jelmer@debian.org>
 Date:   Tue Dec 4 16:42:35 2018 +0000
     Re-export upstream signing key without extra signatures.
     Fixes lintian: public-upstream-key-not-minimal
     See https://lintian.debian.org/tags/public-upstream-key-not-minimal.html for more details.
  debian/changelog                    1 +
  debian/upstream/signing-key.asc   102 +++++++++++++++---------------------------------------------------------------------------------------
  2 files changed, 16 insertions(+), 87 deletions(-)
 commit feebce3147df561aa51a385c53d8759b4520c67f
 Author: Jelmer Vernoo  <jelmer@debian.org>
 Date:   Tue Dec 4 16:42:28 2018 +0000
     Use secure copyright file specification URI.
     Fixes lintian: insecure-copyright-format-uri
     See https://lintian.debian.org/tags/insecure-copyright-format-uri.html for more details.
  debian/changelog   3 +++
  debian/copyright   2 +-
  2 files changed, 4 insertions(+), 1 deletion(-)
Script Interface A fixer script is run in the root directory of a package, where it can make changes it deems necessary, and write a summary of what it's done for the changelog (and commit message) to standard out. If a fixer can not provide any improvements, it can simply leave the working tree untouched - lintian-brush will not create any commits for it or update the changelog. If it exits with a non-zero exit code, then it is assumed that it failed to run and it will be listed as such and its changes reset rather than committed. In addition, tests can be added for fixers by providing various before and after source package trees, to verify that a fixer script makes the expected changes. For more details, see the documentation on writing new fixers.
Availability lintian-brush is currently available in unstable and testing. See man lintian-brush(1) for an explanation of the command-line options. Fixer scripts are included that can fix (some of the instances of) 34 lintian tags. Feedback would be great if you try lintian-brush - please file bugs in the BTS, or propose pull requests with new fixers on salsa.

24 March 2021

Bastian Venthur: Perfection is Achieved When There is Nothing Left to Take Away

In 14 years since starting this blog, I changed the software several times, with each iteration simplifying requirements and setup. Wordpress In 2007, I started blogging. Back then I used a self-hosted Wordpress instance, running on my server. I was never really comfortable with the technology stack involved: a programming language I didn't speak, a DB I had little experience with, a relatively heavy setup and of course, the occasional security issues related to that setup. All that just to have a somewhat dynamic website with pingbacks and comments generated by your visitors... totally worth it and exciting times! You wrote something to the "lazyweb" department and people would actually comment with useful advice! Then came the spammers. First, it was just a little, and you could easily fight it off with the askimet plugin. Over the years, however the spam-to-useful-comments-ratio shifted to ridiculous levels and the comment section itself became a burden. Hosting the comments on a separate platform like disqus was not an option for me, so I finally turned the comments off. Without the need for a comment section, the whole idea of storing mostly textual content in a DB just to have it served as a website, suddenly seemed like overkill and an unnecessary security risk. So I started looking for alternatives. I wanted to maintain my content as plain text files. Text files are cool because they are future proof, portable and can be managed in git. Consequently, I was searching for a static site generator. Pelican Around 2016 I finally migrated away from Wordpress to Pelican. Pelican met all my requirements. I could store my content as plain Markdown files in a git repository. Pelican is written in Python, a language with which I'm very comfortable with. Pelican deals with blog posts and "pages" (i.e. non blog articles) and most importantly: Pelican just generates static HTML. So all that's required on the server side is a simple web server. Conveniently, it also comes with an importer that allowed me to export the content of my old Wordpress instance straight into Markdown files. Migrating to Pelican was a huge relieve: I finally had a all my content as plain text in a git repository, my server was a lot more secure without the Wordpress instance running and removing MySQL freed up a lot of resources on that machine. blag Fast forward to 2021. While I was quite happy with Pelican, it still had too many features I don't need. My blogging experience can be stripped down to a bunch of Markdown files that serve either as articles or pages, some static files, some templating for a decent design and an Atom feed. Particularly, I do not need pages/articles in multiple languages, a plugin system, support for multiple authors and tons of settings. So I did what every self-respecting nerd would do: I wrote my own site generator. Not to write something better than Pelican, but rather something lesser. Something that does less while still being functionally adequate. I called the result "blag", named after the blag of the webcomic xkcd. blag is a blog-aware, static site generator written in Python. It supports the basic features you'd expect from a blog, like an archive, tags and an Atom feed. It converts Markdown (with fenced code blocks) and supports Jinja2 templating. With this setup I'm quite happy for now. Maybe sometime there will be a simpler RSS/Atom-successor that does not require site-specific metadata (i.e. only stuff that can be generated from the articles themselves), so I could get rid of blag's only required configuration.

Sam Hartman: Making our Community Safe: the FSF and rms

I felt disgust and horror when I learned yesterday that rms had returned to the FSF board. When rms resigned back in September of 2019, I was Debian Project Leader. At that time, I felt two things. First, I was happy that the community was finally taking a stand in favor of inclusion, respect, and creating a safe, welcoming place to do our work. It was long past time for rms to move on. But I also felt thankful that rms was not my problem to solve. In significant part because of rms, I had never personally been that involved in the FSF. I considered drafting a statement as Debian Project Leader. I could have talked about how through our Diversity Statement and Code of Conduct we had taken a stand in favor of inclusion and respect. I could have talked about how rms's actions displayed a lack of understanding and empathy and how this created a community that was neither welcoming nor respectful. I didn't. I guess I didn't want to deal with confirming I had sufficient support in the project. I wanted to focus on internal goals, and I was healing and learning from some mistakes I made earlier in the year. It looked like other people were saying what needed to be said and my voice was not required. Silence was a mistake.

It's a mistake I've been making all throughout my interactions with rms. Enough is enough. It's long past time I added my voice to those who cry for accountability and who will not sit aside while rms's disrespect and harm is tolerated.

The first time I was silent about rms was around 15 years ago. I was at a science fiction convention in a crowded party. I didn't know anyone, other than the host of the party. I was out of my depth. I heard his voice---I recognized it from the Share the Software Song. He was hitting on some girl, talking about how he invented Emacs. As best I could tell, she didn't even know what Emacs was. Back then, I wondered what she saw in the interaction; why she stuck around even though she didn't know what he was talking about. I sure didn't want to be around; the interaction between the two of them was making me uncomfortable. Besides, the wings on her costume kept hitting me in the face. So I left as fast as I could.

I've learned a lot about creating safe spaces and avoiding sexual harassment since then. Thinking back, she was probably hitting me because she was trying to back away and getting crowded. If this happened today, I think I would do a better job of owning my responsibility for helping keep the space around me safe. I've learned better techniques for checking in to make sure people around me are comfortable.

I didn't come to silence alone: I had been educated into the culture of avoiding rms and not calling him out. There was a running game in the group of computer security professionals I learned from. The goal was to see how much you could contribute to free software and computer security without being recognized by or interacting with rms. And so, indoctrinated into a culture of silence about the harm that rms caused, I took my first step.

Things weren't much better when I attended Libreplanet 2019 just before taking office as Debian Project Leader. I had stayed away from the conference in large part because of rms. But there were Debian people there, and I was missing community interaction. Unfortunately, I saw that even after the problems of 2018, rms was still treating himself as above community standards. He interrupted speakers, objecting to how they phrased the problem they were considering. After a speech on codes of conduct in the free software community, he cornered the talk organizer to "ask her opinion" about the GNU project's lack of code on conduct. He wasn't asking for an opinion. He was justifying himself; there wasn't much listening in the conversation I heard. Aspects of that conversation crossed professional boundaries for what should be said. The talk organizer was okay--we talked about it after--but if we did a better job of policing our community, that wouldn't even be a question. I think the most telling sign was a discussion with an FSF board member. We were having a great conversation, but he had to interrupt it. He was on rms duty (my words) at the next session. The board had decided it was necessary to have members there so that the staff would not be put in awkward positions by their president. If someone needed to call rms out, it could be a board member rather than the staff members of the conduct team.

And yet again, I held my silence. It's so easy to keep silent. It's not that I never speak up. There are communities where I have called people out. But it's hard to paint that target on yourself. It's hard to engage and to stand strong for a community's standards when you aren't the target. It's hard to approach these problems while maintaining empathy for everyone involved. Some people give into the rage; I don't have that option if I want to be the person I've chosen to be. And so, when I do speak up, the emotional cost is high.

Yet, it's long past time I raised my voice on this issue. Rms has demonstrated that he cannot hold to standards of respect for others, respect for their boundaries, or standards of community safety. We need those standards to be a welcoming community.

If the people who came before me--those who taught me the game of avoiding rms--had spoken up, the community could have healed before I even came on the scene. If I and others had stood up fifteen years ago, we'd have another couple generations who were more used to respect, inclusion, welcoming and safety. The FSF board could have done their job back in 2018. And perhaps if more of us had spoken out in 2019, the FSF board would have found the strength to stand strong and not accept rms's return.

And so, finally, I raise my voice. I signed the open letter calling for the resignation of rms and the entire FSF board. Perhaps if we all get used to raising our voice, it will be easier. Perhaps if we stand together, taking the path of community rather than the path of silence, we'll have the support we need to create communities inclusive enough to welcome everyone who can contribute. For me, I'm done being silent.

There's one criticism of the open letter I'd like to respond to. I've heard concerns about asking for the resignation of the entire FSF board under the understanding that some board members voted against rms's return. It should be obvious why those who voted for rms's return need to resign. But resignation does not always mean you did something wrong. If you find yourself in a leadership role in an organization that takes decisions in significant conflict with your standards of ethics, resignation is also the right path. Staying on the board even if you voted against rms's return means that you consider voting for rms to be a reasonable thing to do. It means that even if you disagreed with it, you can still be part of an organization that takes the path of welcoming rms. At this point, I cannot do that, nor can I support leaders in the FSF who do.

Next.

Previous.