Search Results: "tar"

21 June 2021

Russ Allbery: Review: Demon Lord of Karanda

Review: Demon Lord of Karanda, by David Eddings
Series: The Malloreon #3
Publisher: Del Rey
Copyright: September 1988
Printing: February 1991
ISBN: 0-345-36331-0
Format: Mass market
Pages: 404
This is the third book of the Malloreon, which in turn is a sequel trilogy to The Belgariad. Eddings, unlike most series authors, does a great job of reminding you what's happening with prologues in each book, but you definitely do not want to start reading here. When we last left our heroes, they had been captured. (This is arguably a spoiler for King of the Murgos, but it's not much of one, nor one that really matters.) This turns out to be an opportunity to meet the Emperor of Mallorea, the empire from which their adversary Zandramas (and, from the earlier trilogy, the god Torak) comes. This goes much better than one might expect, continuing the trend in this series of showing the leaders of the enemy countries as substantially similar to the leaders of the supposedly good countries. This sounds like open-mindedness on Eddings's part, and I suppose it partly is. It's at least a change from the first series, in which the bad guys were treated more like orcs. But the deeper I read into this series, the more obvious how invested Eddings is in a weird sort of classism. Garion and the others get along with Zakath in part because they're all royalty, or at least run in those circles. They just disagree about how to be a good ruler (and not as much as one might think, or hope). The general population of any of the countries is rarely of much significance. Zakath is a bit more cynical than Garion and company and has his own agenda, but he's not able to overcome the strong conviction of this series that the Prophecy and the fight between the Child of Light and the Child of Dark is the only thing of importance that's going on, and other people matter only to the extent that they're involved in that story. When I first read these books as a teenager, I was one of the few who liked the second series and didn't mind that its plot was partly a rehash of the first. I found, and still find, the blatantness with which Eddings manipulates the plot by making prophecy a character in the novel amusing. What I had forgotten, however, was how much of a slog the middle of this series is. It takes about three quarters of this book before there are any significant plot developments, and that time isn't packed with interesting diversions. It's mostly the heroes having conversations with each other or with Zakath, being weirdly sexist, rehashing their personality quirks, or shrugging about horrific events that don't matter to them personally. The last is a reference to the plague that appears in this book, and which I had completely forgotten. To be fair to my memory, that's partly because none of the characters seem to care much about it either. They're cooling their heels in a huge city, a plague starts killing people, they give Zakath amazingly brutal and bloodthirsty advice to essentially set fire to all the parts of the city with infected people, and then they blatantly ignore all the restrictions on movement because, well, they're important unlike all those other people and have places to go. It's rather stunningly unempathetic under the best of circumstances and seems even more vile in a 2021 re-read. Eddings also manages to make Ce'Nedra even more obnoxious than she has been by turning her into a walking zombie with weird fits where she's obsessed with her child, and adding further problems (which would be a spoiler) on top of that. I have never been a Ce'Nedra fan (that Garion's marriage ever works at all appears to be by authorial decree), but in this book she's both useless and irritating while supposedly being a tragic figure. You might be able to tell that I'm running sufficiently low on patience for Eddings's character quirks that I'm losing my enthusiasm for re-reading this bit of teenage nostalgia. This is the third book of a five-book series, so while there's a climax of sorts just like there was in the third book of the Belgariad, it's a false climax. One of the secondary characters is removed, but nothing is truly resolved; the state of the plot isn't much different at the end of this book than it was at the start. And to get there, one has to put up with Garion being an idiot, Ce'Nedra being a basket case, the supposed heroes being incredibly vicious about a plague, and one character pretending to have an absolutely dreadful Irish accent for pages upon pages upon pages. (His identity is supposedly a mystery, but was completely obvious a hundred pages before Garion figured it out. Garion isn't the sharpest knife in the drawer.) The one redeeming merit to this series is the dry voice in Garion's head and the absurd sight of the prophecy telling all the characters what to do, and we barely get any of that in this book. When it wasn't irritating or offensive, it was just a waste of time. The worst book of the series so far. Followed by Sorceress of Darshiva. Rating: 3 out of 10

20 June 2021

Julian Andres Klode: Migrating away from apt-key

This is an edited copy of an email I sent to provide guidance to users of apt-key as to how to handle things in a post apt-key world. The manual page already provides all you need to know for replacing apt-key add usage:
Note: Instead of using this command a keyring should be placed directly in the /etc/apt/trusted.gpg.d/ directory with a descriptive name and either gpg or asc as file extension
So it s kind of surprising people need step by step instructions for how to copy/download a file into a directory. I ll also discuss the alternative security snakeoil approach with signed-by that s become popular. Maybe we should not have added signed-by, people seem to forget that debs still run maintainer scripts as root. Aside from this email, Debian users should look into extrepo, which manages curated external repositories for you.

Direct translation Assume you currently have:
wget -qO- https://myrepo.example/myrepo.asc   sudo apt-key add  
To translate this directly for bionic and newer, you can use:
sudo wget -qO /etc/apt/trusted.gpg.d/myrepo.asc https://myrepo.example/myrepo.asc
or to avoid downloading as root:
wget -qO-  https://myrepo.example/myrepo.asc   sudo tee -a /etc/apt/trusted.gpg.d/myrepo.asc
Older (and all) releases only support unarmored files with an extension .gpg. If you care about them, provide one, and use
sudo wget -qO /etc/apt/trusted.gpg.d/myrepo.gpg https://myrepo.example/myrepo.gpg
Some people will tell you to download the .asc and pipe it to gpg --dearmor, but gpg might not be installed, so really, just offer a .gpg one instead that is supported on all systems. wget might not be available everywhere so you can use apt-helper:
sudo /usr/lib/apt/apt-helper download-file https://myrepo.example/myrepo.asc /etc/apt/trusted.gpg.d/myrepo.asc
or, to avoid downloading as root:
/usr/lib/apt/apt-helper download-file https://myrepo.example/myrepo.asc /tmp/myrepo.asc && sudo mv /tmp/myrepo.asc /etc/apt/trusted.gpg.d

Pretending to be safer by using signed-by People say it s good practice to not use trusted.gpg.d and install the file elsewhere and then refer to it from the sources.list entry by using signed-by=<path to the file>. So this looks a lot safer, because now your key can t sign other unrelated repositories. In practice, security increase is minimal, since package maintainer scripts run as root anyway. But I guess it s better for publicity :) As an example, here are the instructions to install signal-desktop from signal.org. As mentioned, gpg --dearmor use in there is not a good idea, and I d personally not tell people to modify /usr as it s supposed to be managed by the package manager, but we don t have an /etc/apt/keyrings or similar at the moment; it s fine though if the keyring is installed by the package. You can also just add the file there as a starting point, and then install a keyring package overriding it (pretend there is a signal-desktop-keyring package below that would override the .gpg we added).
# NOTE: These instructions only work for 64 bit Debian-based
# Linux distributions such as Ubuntu, Mint etc.
# 1. Install our official public software signing key
wget -O- https://updates.signal.org/desktop/apt/keys.asc   gpg --dearmor > signal-desktop-keyring.gpg
cat signal-desktop-keyring.gpg   sudo tee -a /usr/share/keyrings/signal-desktop-keyring.gpg > /dev/null
# 2. Add our repository to your list of repositories
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main'  \
  sudo tee -a /etc/apt/sources.list.d/signal-xenial.list
# 3. Update your package database and install signal
sudo apt update && sudo apt install signal-desktop
I do wonder why they do wget gpg --dearmor, pipe that into the file and then cat sudo tee it, instead of having that all in one pipeline. Maybe they want nicer progress reporting.

Scenario-specific guidance We have three scenarios: For system image building, shipping the key in /etc/apt/trusted.gpg.d seems reasonable to me; you are the vendor sort of, so it can be globally trusted. Chrome-style debs and repository config debs: If you ship a deb, embedding the sources.list.d snippet (calling it $myrepo.list) and shipping a $myrepo.gpg in /usr/share/keyrings is the best approach. Whether you ship that in product debs aka vscode/chromium or provide a repository configuration deb (let s call it myrepo-repo.deb) and then tell people to run apt update followed by apt install <package inside the repo> depends on how many packages are in the repo, I guess. Manual instructions (signal style): The third case, where you tell people to run wget themselves, I find tricky. As we see in signal, just stuffing keyring files into /usr/share/keyrings is popular, despite /usr supposed to be managed by the package manager. We don t have another dir inside /etc (or /usr/local), so it s hard to suggest something else. There s no significant benefit from actually using signed-by, so it s kind of extra work for little gain, though.

Addendum: Future work This part is new, just for this blog post. Let s look at upcoming changes and how they make things easier.

Bundled .sources files Assuming I get my merge request merged, the next version of APT (2.4/2.3.something) will do away with all the complexity and allow you to embed the key directly into a deb822 .sources file (which have been available for some time now):
Types: deb
URIs: https://myrepo.example/ https://myotherrepo.example/
Suites: stable not-so-stable
Components: main
Signed-By:
 -----BEGIN PGP PUBLIC KEY BLOCK-----
 .
 mDMEYCQjIxYJKwYBBAHaRw8BAQdAD/P5Nvvnvk66SxBBHDbhRml9ORg1WV5CvzKY
 CuMfoIS0BmFiY2RlZoiQBBMWCgA4FiEErCIG1VhKWMWo2yfAREZd5NfO31cFAmAk
 IyMCGyMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQREZd5NfO31fbOwD6ArzS
 dM0Dkd5h2Ujy1b6KcAaVW9FOa5UNfJ9FFBtjLQEBAJ7UyWD3dZzhvlaAwunsk7DG
 3bHcln8DMpIJVXht78sL
 =IE0r
 -----END PGP PUBLIC KEY BLOCK-----
Then you can just provide a .sources files to users, they place it into sources.list.d, and everything magically works Probably adding a nice apt add-source command for it I guess. Well, python-apt s aptsources package still does not support deb822 sources, and never will, we ll need an aptsources2 for that for backwards-compatibility reasons, and then port software-properties and other users to it.

OpenPGP vs aptsign We do have a better, tighter replacement for gpg in the works which uses Ed25519 keys to sign Release files. It s temporarily named aptsign, but it s a generic signer for single-section deb822 files, similar to signify/minisign. We believe that this solves the security nightmare that our OpenPGP integration is while reducing complexity at the same time. Keys are much shorter, so the bundled sources file above will look much nicer.

Russ Allbery: Review: The Magician's Nephew

Review: The Magician's Nephew, by C.S. Lewis
Illustrator: Pauline Baynes
Series: Chronicles of Narnia #6
Publisher: Collier Books
Copyright: 1955
Printing: 1978
ISBN: 0-02-044230-0
Format: Mass market
Pages: 186
The Magician's Nephew is the sixth book of the Chronicles of Narnia in the original publication order, but it's a prequel, set fifty years before The Lion, the Witch and the Wardrobe. It's therefore put first in the new reading order. I have always loved world-building and continuities and, as a comics book reader (Marvel primarily), developed a deep enjoyment of filling in the pieces and reconstructing histories from later stories. It's no wonder that I love reading The Magician's Nephew after The Lion, the Witch and the Wardrobe. The experience of fleshing out backstory with detail and specifics makes me happy. If that's also you, I recommend the order in which I'm reading these books. Reading this one first is defensible, though. One of the strongest arguments for doing so is that it's a much stronger, tighter, and better-told story than The Lion, the Witch and the Wardrobe, and therefore might start the series off on a better foot for you. It stands alone well; you don't need to know any of the later events to enjoy this, although you will miss the significance of a few things like the lamp post and you don't get the full introduction to Aslan. The Magician's Nephew is the story of Polly Plummer, her new neighbor Digory Kirke, and his Uncle Andrew, who fancies himself a magician. At the start of the book, Digory's mother is bed-ridden and dying and Digory is miserable, which is the impetus for a friendship with Polly. The two decide to explore the crawl space of the row houses in which they live, seeing if they can get into the empty house past Digory's. They don't calculate the distances correctly and end up in Uncle Andrew's workroom, where Digory was forbidden to go. Uncle Andrew sees this as a golden opportunity to use them for an experiment in travel to other worlds. MAJOR SPOILERS BELOW. The Magician's Nephew, like the best of the Narnia books, does not drag its feet getting started. It takes a mere 30 pages to introduce all of the characters, establish a friendship, introduce us to a villain, and get both of the kids into another world. When Lewis is at his best, he has an economy of storytelling and a grasp of pacing that I wish was more common. It's also stuffed to the brim with ideas, one of the best of which is the Wood Between the Worlds. Uncle Andrew has crafted pairs of magic rings, yellow and green, and tricks Polly into touching one of the yellow ones, causing her to vanish from our world. He then uses her plight to coerce Digory into going after her, carrying two green rings that he thinks will bring people back into our world, and not incidentally also observing that world and returning to tell Uncle Andrew what it's like. But the world is more complicated than he thinks it is, and the place where the children find themselves is an eerie and incredibly peaceful wood, full of grass and trees but apparently no other living thing, and sprinkled with pools of water. This was my first encounter with the idea of a world that connects other worlds, and it remains the most memorable one for me. I love everything about the Wood: the simplicity of it, the calm that seems in part to be a defense against intrusion, the hidden danger that one might lose one's way and confuse the ponds for each other, and even the way that it tends to make one lose track of why one is there or what one is trying to accomplish. That quiet forest filled with pools is still an image I use for infinite creativity and potential. It's quiet and nonthreatening, but not entirely inviting either; it's magnificently neutral, letting each person bring what they wish to it. One of the minor plot points of this book is that Uncle Andrew is wrong about the rings because he's wrong about the worlds. There aren't just two worlds; there are an infinite number, with the Wood as a nexus, and our reality is neither the center nor one of an important pair. The rings are directional, but relative to the Wood, not our world. The kids, who are forced to experiment and who have open minds, figure this out quickly, but Uncle Andrew never shifts his perspective. This isn't important to the story, but I've always thought it was a nice touch of world-building. Where this story is heading, of course, is the creation of Narnia and the beginning of all of the stories told in the rest of the series. But before that, the kids's first trip out of the Wood is to one of the best worlds of children's fantasy: Charn. If the Wood is my mental image of a world nexus, Charn will forever be my image of a dying world: black sky, swollen red sun, and endless abandoned and crumbling buildings as far as the eye can see, full of tired silences and eerie noises. And, of course, the hall of statues, with one of the most memorable descriptions of history and empire I've ever read (if you ignore the racialized description):
All of the faces they could see were certainly nice. Both the men and women looked kind and wise, and they seemed to come of a handsome race. But after the children had gone a few steps down the room they came to faces that looked a little different. These were very solemn faces. You felt you would have to mind your P's and Q's, if you ever met living people who looked like that. When they had gone a little farther, they found themselves among faces they didn't like: this was about the middle of the room. The faces here looked very strong and proud and happy, but they looked cruel. A little further on, they looked crueller. Further on again, they were still cruel but they no longer looked happy. They were even despairing faces: as if the people they belonged to had done dreadful things and also suffered dreadful things.
The last statue is of a fierce, proud woman that Digory finds strikingly beautiful. (Lewis notes in an aside that Polly always said she never found anything specially beautiful about her. Here, as in The Silver Chair, the girl is the sensible one and things would have gone better if the boy had listened to her, a theme that I find immensely frustrating because Susan was the sensible one in the first two books of the series but then Lewis threw that away.) There is a bell in the middle of this hall, and the pillar that holds that bell has an inscription on it that I think every kid who grew up on Narnia knows by heart.
Make your choice, adventurous Stranger;
Strike the bell and bide the danger,
Or wonder, till it drives you mad,
What would have followed if you had.
Polly has no intention of striking the bell, but Digory fights her and does it anyway, waking Jadis from where she sat as the final statue in the hall and setting off one of the greatest reimaginings of a villain in children's literature. Jadis will, of course, become the White Witch who holds Narnia in endless winter some thousand Narnian years later. But the White Witch was a mediocre villain at best, the sort of obvious and cruel villain common in short fairy tales where the author isn't interested in doing much characterization. She exists to be evil, do bad things, and be defeated. She has a few good moments in conflict with Aslan, but that's about it. Jadis in this book is another matter entirely: proud, brilliant, dangerous, and creative. The death of everything on Charn was Jadis's doing: an intentional spell, used to claim a victory of sorts from the jaws of defeat by her sister in a civil war. (I find it fascinating that Lewis puts aside his normally sexist roles here.) Despite the best attempts of the kids to lose her both in Charn and in the Wood (which is inimical to her, in another nice bit of world-building), she manages to get back to England with them. The result is a remarkably good bit of villain characterization. Jadis is totally out of her element, used to a world-spanning empire run with magic and (from what hints we get) vaguely medieval technology. Her plan to take over their local country and eventually the world should be absurd and is played somewhat for laughs. Her magic, which is her great weapon, doesn't even work in England. But Jadis learns at a speed that the reader can watch. She's observant, she pays attention to things that don't fit her expectations, she changes plans, and she moves with predatory speed. Within a few hours in London she's stolen jewels and a horse and carriage, and the local police seem entirely overmatched. There's no way that one person without magic should be a real danger to England around the turn of the 20th century, but by the time the kids manage to pull her back into the Wood, you're not entirely sure England would have been safe. A chaotic confrontation, plus the ability of the rings to work their magic through transitive human contact, ends up with the kids, Uncle Andrew, Jadis, a taxicab driver and his horse all transported through the Wood to a new world. In this case, literally a new world: Narnia at the point of its creation. Here again, Lewis translates Christian myth, in this case the Genesis creation story, into a more vivid and in many ways more beautiful story than the original. Aslan singing the world into existence is an incredible image, as is the newly-created world so bursting with life that even things that normally could not grow will do so. (Which, of course, is why there is a lamp post burning in the middle of the western forest of Narnia for the Pevensie kids to find later.) I think my favorite part is the creation of the stars, but the whole sequence is great. There's also an insightful bit of human psychology. Uncle Andrew can't believe that a lion is singing, so he convinces himself that Aslan is not singing, and thus prevents himself from making any sense of the talking animals later.
Now the trouble about trying to make yourself stupider than you really are is that you very often succeed.
As with a lot in Lewis, he probably meant this as a statement about faith, but it generalizes well beyond the religious context. What disappointed me about the creation story, though, is the animals. I didn't notice this as a kid, but this re-read has sensitized me to how Lewis consistently treats the talking animals as less than humans even though he celebrates them. That happens here too: the newly-created, newly-awakened animals are curious and excited but kind of dim. Some of this is an attempt to show that they're young and are just starting to learn, but it also seems to be an excuse for Aslan to set up a human king and queen over them instead of teaching them directly how to deal with the threat of Jadis who the children inadvertently introduced into the world. The other thing I dislike about The Magician's Nephew is that the climax is unnecessarily cruel. Once Digory realizes the properties of the newly-created world, he hopes to find a way to use that to heal his mother. Aslan points out that he is responsible for Jadis entering the world and instead sends him on a mission to obtain a fruit that, when planted, will ward Narnia against her for many years. The same fruit would heal his mother, and he has to choose Narnia over her. (It's a fairly explicit parallel to the Garden of Eden, except in this case Digory passes.) Aslan, in the end, gives Digory the fruit of the tree that grows, which is still sufficient to heal his mother, but this sequence made me angry when re-reading it. Aslan knew all along that what Digory is doing will let him heal his mother as well, but hides this from him to make it more of a test. It's cruel and mean; Aslan could have promised to heal Digory's mother and then seen if he would help Narnia without getting anything in return other than atoning for his error, but I suppose that was too transactional for Lewis's theology or something. Meh. But, despite that, the only reason why this is not the best Narnia book is because The Voyage of the Dawn Treader is the only Narnia book that also nails the ending. The Magician's Nephew, up through Charn, Jadis's rampage through London, and the initial creation of Narnia, is fully as good, perhaps better. It sags a bit at the end, partly because it tries to hard to make the Narnian animals humorous and partly because of the unnecessary emotional torture of Digory. But this still holds up as the second-best Narnia book, and one I thoroughly enjoyed re-reading. If anything, Jadis and Charn are even better than I remembered. Followed by the last book of the series, the somewhat notorious The Last Battle. Rating: 9 out of 10

19 June 2021

Andrew Cater: Fixing Wayland failing to start when a desktop environment is installed but your machine needs firmware. ...

This came up in an install that I was just doing for Debian 10.10 media testing. I hadn't seen this before and it would be disconcerting to other people, though it is a known bug, I think. I was installing an image that had no network and no firmware. KDE failed to run and dropped me to a text mode prompt. This was because the Zotac SBC I'm using requires Radeon R600 firmware to work. There was a warning message on screen to that effect.
The way round this was to plug in a network cable and edit /etc/apt/sources.list. Editing /etc/apt/sources.list was to add contrib and non-free to the appropriate lines to allow me to install firmware-linux-nonfree and firmware-misc-nonfree which includes the appropriate AMD firmware for the embedded Radeon chipset.
Since the machine hadn't been connected to a network at install time, I also needed to run a dhclient command to obtain a network lease and allow me to install the non-free metapackages over the network.Result: success: a full KDE desktop. [The machine is an old Zotac SBC with embedded graphics hardware: AMD E350 - specifically, it requires firmware-amd-graphics and amd64-microcode].

Chris Lamb: *Raiders of the Lost Ark*: 40 Years On

"Again, we see there is nothing you can possess which I cannot take away."
The cinema was a rare and expensive treat in my youth, so I first came across Raiders of the Lost Ark by recording it from television onto a poor quality VHS. I only mention this as it meant I watched a slightly different film to the one intended, as my copy somehow missed off the first 10 minutes. For those not as intimately familiar with the film as me, this is just in time to see a Belloq demand Dr. Jones hand over the Peruvian head (see above), just in time to learn that Indy loathes snakes, and just in time to see the inadvertent reproduction of two Europeans squabbling over the spoils of a foreign land. What this truncation did to my interpretation of the film (released thirty years ago today on June 19th 1981) is interesting to explore. Without Jones' physical and moral traits being demonstrated on-screen (as well as missing the weighing the gold head and the rollercoaster boulder scene), it actually made the idea of 'Indiana Jones' even more of a mythical archetype. The film wisely withholds Jones' backstory, but my directors cut deprived him of even more, and counterintuitively imbued him with even more of a legendary hue as the elision made his qualities an assumption beyond question. Indiana Jones, if you can excuse the clich , needed no introduction at all. Good artists copy, great artists steal. And oh boy, does Raiders steal. I've watched this film about twenty times over the past two decades and it's now firmly entered into my personal canon. But watching it on its thirtieth anniversary was different not least because I could situate it in a broader cinematic context. For example, I now see the Gestapo officer in Major Strasser from Casablanca (1942), in fact just as I can with many of Raiders' other orientalist tendencies: not only in its breezy depictions of backwards sand people, but also of North Africa as an entrep t and playground for a certain kind of Western gangster. The opening as well, set in an equally reductionist pseudo-Peru, now feels like Werner Herzog's Aguirre, the Wrath of God (1972) but without, of course, any self-conscious colonial critique.
The imagery of the ark appears to be borrowed from James Tissot's The Ark Passes Over the Jordan, part of the fin de siecle fascination with the occult and (ironically enough given the background of Raiders' director), a French Catholic revival.
I can now also appreciate some of the finer edges that make this film just so much damn fun to watch. For instance, the comic book conceit that Jones and Belloq are a 'shadowy reflection' of one other and that they need 'only a nudge' to make one like the other. As is the idea that Belloq seems to be actually enjoying being evil. I also spotted Jones rejecting the martini on the plane. This feels less like a comment on corrupting effect of alcohol (he drinks rather heavily elsewhere in the film), but rather a subtle distancing from James Bond. This feels especially important given that the action-packed cold open is, let us be honest for a second, ripped straight from the 007 franchise. John William's soundtracks are always worth mentioning. The corny Raiders March does almost nothing for me, but the highly-underrated 'Ark theme' certainly does. I delight in its allusions to Gregorian chant, the diabolus in musica and the Hungarian minor scale, fusing the Christian doctrine of the Holy Trinity (the stacked thirds, get it?), the ars antiqua of the Middle Ages with an 'exotic' twist that the Russian Five associated with central European Judaism.
The best use of the ark leitmotif is, of course, when it is opened. Here, Indy and Marion are saved by not opening their eyes whilst the 'High Priest' Belloq and the rest of the Nazis are all melted away. I'm no Biblical scholar, but I'm almost certain they were alluding to Leviticus 16:2 here:
The Lord said to Moses: Tell your brother Aaron that he is not to come whenever he chooses into the Most Holy Place behind the curtain in front of the atonement cover on the ark, or else he will die, for I will appear in the cloud above the mercy seat.
But would it be too much of a stretch to also see the myth of Orpheus and Eurydices too? Orpheus's wife would only be saved from the underworld if he did not turn around until he came to his own house. But he turned round to look at his wife, and she instantly slipped back into the depths:
For he who overcome should turn back his gaze
Towards the Tartarean cave,
Whatever excellence he takes with him
He loses when he looks on those below.
Perhaps not, given that Marion and the ark are not lost in quite the same way. But whilst touching on gender, it was interesting to update my view of archaeologist Ren Belloq. To countermand his slight queer coding (a trope of Disney villains such as Scar, Jafar, Cruella, etc.), there is a rather clumsy subplot involving Belloq repeatedly (and half-heartedly) failing to seduce Marion. This disavows any idea that Belloq isn't firmly heterosexual, essential for the film's mainstream audience, but it is especially important in Raiders because, if we recall the relationship between Belloq and Jones: 'it would take only a nudge to make you like me'. (This would definitely put a new slant on 'Top men'.)
However, my favourite moment is where the Nazis place the ark in a crate in order to transport it to the deserted island. On route, the swastikas on the side of the crate spontaneously burn away, and a disturbing noise is heard in the background. This short scene has always fascinated me, partly because it's the first time in the film that the power of the ark is demonstrated first-hand but also because gives the object an other-worldly nature that, to the best of my knowledge, has no parallel in the rest of cinema. Still, I had always assumed that the Aak disfigured the swastikas because of their association with the Nazis, interpreting the act as God's condemnation of the Third Reich. But now I catch myself wondering whether the ark would have disfigured any iconography as a matter of principle or whether their treatment was specific to the swastika. We later get a partial answer to this question, as the 'US Army' inscriptions in the Citizen Kane warehouse remain untouched. Far from being an insignificant concern, the filmmakers appear to have wandered into a highly-contested theological debate. As in, if the burning of the swastika is God's moral judgement of the Nazi regime, then God is clearly both willing and able to intervene in human affairs. So why did he not, to put it mildly, prevent Auschwitz? From this perspective, Spielberg appears to be limbering up for some of the academic critiques surrounding Holocaust representations that will follow Schindler's List (1993). Given my nostalgic and somewhat ironic attachment to Raiders, it will always be difficult for me to objectively appraise the film. Even so, it feels like it is underpinned by an earnest attempt to entertain the viewer, largely absent in the affected cynicism of contemporary cinema. And when considered in the totality of Hollywood's output, its tonal and technical flaws are not actually that bad or at least Marion's muddled characterisation and its breezy chauvinism (for example) clearly have far worse examples. Perhaps the most remarkable thing about the film in 2021 is that it hasn't changed that much at all. It spawned one good sequel (The Last Crusade), one bad one (The Temple of Doom), and one hardly worth mentioning at all, yet these adventures haven't affected the original Raiders in any meaningful way. In fact, if anything has affected the original text it is, once again, George Lucas himself, as knowing the impending backlash around the Star Wars prequels adds an inadvertent paratext to all his earlier works. Yet in a 1978 discussion prior to the creation of Raiders, you can get a keen sense of how Lucas' childlike enthusiasm will always result in something either extremely good or something extremely bad somehow no middle ground is quite possible. Yes, it's easy to rubbish his initial ideas 'We'll call him Indiana Smith! but hasn't Lucas actually captured the essence of a heroic 'Americana' here, and that the final result is simply a difference of degree, not kind?

Chris Lamb: Raiders of the Lost Ark: 40 Years On

"Again, we see there is nothing you can possess which I cannot take away."
The cinema was a rare and expensive treat in my youth, so I first came across Raiders of the Lost Ark by recording it from television onto a poor quality VHS. I only mention this as it meant I watched a slightly different film to the one intended, as my copy somehow missed off the first 10 minutes. For those not as intimately familiar with the film as me, this is just in time to see a Belloq demand Dr. Jones hand over the Peruvian head (see above), just in time to learn that Indy loathes snakes, and just in time to see the inadvertent reproduction of two Europeans squabbling over the spoils of a foreign land. What this truncation did to my interpretation of the film (released thirty years ago today on June 19th 1981) is interesting to explore. Without Jones' physical and moral traits being demonstrated on-screen (as well as missing the weighing the gold head and the rollercoaster boulder scene), it actually made the idea of 'Indiana Jones' even more of a mythical archetype. The film wisely withholds Jones' backstory, but my directors cut deprived him of even more, and counterintuitively imbued him with even more of a legendary hue as the elision made his qualities an assumption beyond question. Indiana Jones, if you can excuse the clich , needed no introduction at all. Good artists copy, great artists steal. And oh boy, does Raiders steal. I've watched this film about twenty times over the past two decades and it's now firmly entered into my personal canon. But watching it on its thirtieth anniversary was different not least because I could situate it in a broader cinematic context. For example, I now see the Gestapo officer in Major Strasser from Casablanca (1942), in fact just as I can with many of Raiders' other orientalist tendencies: not only in its breezy depictions of backwards sand people, but also of North Africa as an entrep t and playground for a certain kind of Western gangster. The opening as well, set in an equally reductionist pseudo-Peru, now feels like Werner Herzog's Aguirre, the Wrath of God (1972) but without, of course, any self-conscious colonial critique.
The imagery of the ark appears to be borrowed from James Tissot's The Ark Passes Over the Jordan, part of the fin de siecle fascination with the occult and (ironically enough given the background of Raiders' director), a French Catholic revival.
I can now also appreciate some of the finer edges that make this film just so much damn fun to watch. For instance, the comic book conceit that Jones and Belloq are a 'shadowy reflection' of one other and that they need 'only a nudge' to make one like the other. As is the idea that Belloq seems to be actually enjoying being evil. I also spotted Jones rejecting the martini on the plane. This feels less like a comment on corrupting effect of alcohol (he drinks rather heavily elsewhere in the film), but rather a subtle distancing from James Bond. This feels especially important given that the action-packed cold open is, let us be honest for a second, ripped straight from the 007 franchise. John William's soundtracks are always worth mentioning. The corny Raiders March does almost nothing for me, but the highly-underrated 'Ark theme' certainly does. I delight in its allusions to Gregorian chant, the diabolus in musica and the Hungarian minor scale, fusing the Christian doctrine of the Holy Trinity (the stacked thirds, get it?), the ars antiqua of the Middle Ages with an 'exotic' twist that the Russian Five associated with central European Judaism.
The best use of the ark leitmotif is, of course, when it is opened. Here, Indy and Marion are saved by not opening their eyes whilst the 'High Priest' Belloq and the rest of the Nazis are all melted away. I'm no Biblical scholar, but I'm almost certain they were alluding to Leviticus 16:2 here:
The Lord said to Moses: Tell your brother Aaron that he is not to come whenever he chooses into the Most Holy Place behind the curtain in front of the atonement cover on the ark, or else he will die, for I will appear in the cloud above the mercy seat.
But would it be too much of a stretch to also see the myth of Orpheus and Eurydices too? Orpheus's wife would only be saved from the underworld if he did not turn around until he came to his own house. But he turned round to look at his wife, and she instantly slipped back into the depths:
For he who overcome should turn back his gaze
Towards the Tartarean cave,
Whatever excellence he takes with him
He loses when he looks on those below.
Perhaps not, given that Marion and the ark are not lost in quite the same way. But whilst touching on gender, it was interesting to update my view of archaeologist Ren Belloq. To countermand his slight queer coding (a trope of Disney villains such as Scar, Jafar, Cruella, etc.), there is a rather clumsy subplot involving Belloq repeatedly (and half-heartedly) failing to seduce Marion. This disavows any idea that Belloq isn't firmly heterosexual, essential for the film's mainstream audience, but it is especially important in Raiders because, if we recall the relationship between Belloq and Jones: 'it would take only a nudge to make you like me'. (This would definitely put a new slant on 'Top men'.)
However, my favourite moment is where the Nazis place the ark in a crate in order to transport it to the deserted island. On route, the swastikas on the side of the crate spontaneously burn away, and a disturbing noise is heard in the background. This short scene has always fascinated me, partly because it's the first time in the film that the power of the ark is demonstrated first-hand but also because gives the object an other-worldly nature that, to the best of my knowledge, has no parallel in the rest of cinema. Still, I had always assumed that the Aak disfigured the swastikas because of their association with the Nazis, interpreting the act as God's condemnation of the Third Reich. But now I catch myself wondering whether the ark would have disfigured any iconography as a matter of principle or whether their treatment was specific to the swastika. We later get a partial answer to this question, as the 'US Army' inscriptions in the Citizen Kane warehouse remain untouched. Far from being an insignificant concern, the filmmakers appear to have wandered into a highly-contested theological debate. As in, if the burning of the swastika is God's moral judgement of the Nazi regime, then God is clearly both willing and able to intervene in human affairs. So why did he not, to put it mildly, prevent Auschwitz? From this perspective, Spielberg appears to be limbering up for some of the academic critiques surrounding Holocaust representations that will follow Schindler's List (1993). Given my nostalgic and somewhat ironic attachment to Raiders, it will always be difficult for me to objectively appraise the film. Even so, it feels like it is underpinned by an earnest attempt to entertain the viewer, largely absent in the affected cynicism of contemporary cinema. And when considered in the totality of Hollywood's output, its tonal and technical flaws are not actually that bad or at least Marion's muddled characterisation and its breezy chauvinism (for example) clearly have far worse examples. Perhaps the most remarkable thing about the film in 2021 is that it hasn't changed that much at all. It spawned one good sequel (The Last Crusade), one bad one (The Temple of Doom), and one hardly worth mentioning at all, yet these adventures haven't affected the original Raiders in any meaningful way. In fact, if anything has affected the original text it is, once again, George Lucas himself, as knowing the impending backlash around the Star Wars prequels adds an inadvertent paratext to all his earlier works. Yet in a 1978 discussion prior to the creation of Raiders, you can get a keen sense of how Lucas' childlike enthusiasm will always result in something either extremely good or something extremely bad somehow no middle ground is quite possible. Yes, it's easy to rubbish his initial ideas 'We'll call him Indiana Smith! but hasn't Lucas actually captured the essence of a heroic 'Americana' here, and that the final result is simply a difference of degree, not kind?

Andrew Cater: Debian 10.10 release 202106191548

Late blogging on this one.Even as we wait for the final release of Bullseye [Debian 11], we're still producing updates for Debian 10 [Buster].Today has thrown up a few problems: working with Steve, RattusRattus and Isy in Cambridge, Schweer and Linux-Fan somewhere else in the world.A couple of build problems have meant that we've started later than we otherwise might have been and a couple of image runs have had to be redone. We're there now and happily running tests.As ever, it's good to be doing this. With practice, I can now repeat mistakes with 100% reliability and in shorter time :)More updates later.

Joachim Breitner: Leaving DFINITY

Last Thursday was my last working day at DFINITY. There are various reasons why I felt that after almost three years the DFINITY Foundation isn t quite the right place for me anymore, and this plan has been in the making for a while. Primarily, there are personal pull factors that strongly suggest that I ll take a break from full time employment, so I decided to see the launch of the Internet Computer through and then leave. DFINITY has hired some amazing people, and it was a great pleasure to work with them. I learned a lot (some Rust, a lot of Nix, and just how merciless Conway s law is), and I dare say I had the opportunity to do some good work, contributing my part to make the Internet Computer a reality. I am especially proud of the Interface Specification and the specification-driven design principles behind it. It even comes with a model reference implementation and acceptance test suite, and although we didn t quite get to do formalization, those familiar with the DeepSpec project will recognize some influence of their concept of deep specifications . Besides that, there is of course my work on the Motoko programming language, where I build the backend,a the Candid interoperability layer, where I helped with the formalism, formulated the a generic soundness criterion for Interface Description Languages in a higher order settings and formally verified that in Coq. Fortunately, all of this work is now Free Software or at least Open Source. With so much work poured into this project, I continue to care about it, and you ll see me post on the the developer forum and hack on Motoko. As the Internet Computer becomes gradually more open, I hope I can be gradually more involved again. But even without me contributing full-time I am sure that DFINITY and the Internet Computer will do well; when I left there were still plenty of smart, capable and enthusiastic people forging ahead. So what s next? So far, I have rushed every professional transition in my life: When starting my PhD, when starting my postdoc, when starting my job at DFINITY, and every time I regretted it. So this time, I will take a proper break and will explore the world a bit (as far as that is possible given the pandemic). I will stresslessly contribute to various open source projects. I also hope to do more public outreach and teaching, writing more blog posts again, recording screencasts and giving talks and lectures. If you want to invite me to your user group/seminar/company/workshop, please let me know! Also, I might be up for small interesting projects in a while. Beyond these, I have no concrete plans and am looking forward to the inspiration I might get from hiking through the Scandinavian wilderness. If you happen to stumble across my tent, please stop for a tea.

18 June 2021

Enrico Zini: Playbooks, host vars, group vars

This is part of a series of posts on ideas for an ansible-like provisioning system, implemented in Transilience. Host variables Ansible allows to specify per-host variables, and I like that. Let's try to model a host as a dataclass:
@dataclass
class Host:
    """
    A host to be provisioned.
    """
    name: str
    type: str = "Mitogen"
    args: Dict[str, Any] = field(default_factory=dict)
    def _make_system(self) -> System:
        cls = getattr(transilience.system, self.type)
        return cls(self.name, **self.args)
This should have enough information to create a connection to the host, and can be subclassed to add host-specific dataclass fields. Host variables can then be provided as default constructor arguments when instantiating Roles:
    # Add host/group variables to role constructor args
    host_fields =  f.name: f for f in fields(host) 
    for field in fields(role_cls):
        if field.name in host_fields:
            role_kwargs.setdefault(field.name, getattr(host, field.name))
    role = role_cls(**role_kwargs)
Group variables It looks like I can model groups and group variables by using dataclasses as mixins:
@dataclass
class Webserver:
    server_name: str = "www.example.org"
@dataclass
class Srv1(Webserver):
    ...
Doing things like filtering all hosts that are members of a given group can be done with a simple isinstance or issubclass test. Playbooks So far Transilience is executing on one host at a time, and Ansible can execute on a whole host inventory. Since the most part of running a playbook is I/O bound, we can parallelize hosts using threads, without worrying too much about the performance impact of GIL. Let's introduce a Playbook class as the main entry point for a playbook:
class Playbook:
    def setup_logging(self):
        ...
    def make_argparser(self):
        description = inspect.getdoc(self)
        if not description:
            description = "Provision systems"
        parser = argparse.ArgumentParser(description=description)
        parser.add_argument("-v", "--verbose", action="store_true",
                            help="verbose output")
        parser.add_argument("--debug", action="store_true",
                            help="verbose output")
        return parser
    def hosts(self) -> Sequence[Host]:
        """
        Generate a sequence with all the systems on which the playbook needs to run
        """
        return ()
    def start(self, runner: Runner):
        """
        Start the playbook on the given runner.
        This method is called once for each system returned by systems()
        """
        raise NotImplementedError(f" self.__class__.__name__ .start is not implemented")
    def main(self):
        parser = self.make_argparser()
        self.args = parser.parse_args()
        self.setup_logging()
        # Start all the runners in separate threads
        threads = []
        for host in self.hosts():
            runner = Runner(host)
            self.start(runner)
            t = threading.Thread(target=runner.main)
            threads.append(t)
            t.start()
        # Wait for all threads to complete
        for t in threads:
            t.join()
And an actual playbook will now look like something like this:
from dataclasses import dataclass
import sys
from transilience import Playbook, Host
@dataclass
class MyServer(Host):
    srv_root: str = "/srv"
    site_admin: str = "enrico@enricozini.org"
class VPS(Playbook):
    """
    Provision my VPS
    """
    def hosts(self):
        yield MyServer(name="server", args= 
            "method": "ssh",
            "hostname": "host.example.org",
            "username": "root",
         )
    def start(self, runner):
        runner.add_role("fail2ban")
        runner.add_role("prosody")
        runner.add_role(
                "mailserver",
                postmaster="enrico",
                myhostname="mail.example.org",
                aliases= ... )
if __name__ == "__main__":
    sys.exit(VPS().main())
It looks quite straightforward to me, works on any number of hosts, and has a proper command line interface:
./provision  --help
usage: provision [-h] [-v] [--debug]
Provision my VPS
optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose  verbose output
  --debug        verbose output
Next step: check mode!

17 June 2021

Elana Hashman: I'm hosting a Bug Scrub for Kubernetes SIG Node

It's been a long while since I last hosted a BSP, but 'tis the season. Kubernetes SIG Node will be holding a bug scrub on June 24-25, and this is a great opportunity for you to get involved if you're interested in contributing to Kubernetes or SIG Node! We will be hosting a global event with region captains for all timezones. I am one of the NASA captains (~17:00-01:00 UTC) and I'll be leading the kickoff. We will be working on Slack and Zoom. I hope you'll be able to drop in! Details I'm an existing contributor, what should I work on? Work on triaging and closing SIG Node bugs. We have a lot of bugs!! The goal of our event is to categorize, clean up, and resolve some of the 450+ issues in k/k for SIG Node. Check out the event docs for more instructions. I'm a new contributor and want to join but I have no idea what I'm doing! At some point, that was all of us! This is a great opportunity to get involved if you've never contributed to Kubernetes. We'll have dedicated mentors available to coordinate and help out new contributors. If you've never contributed to Kubernetes before, I recommend you check out the Getting Started and Contributor Guide resources in advance of the event. You will want to ensure you've signed the contributor license agreement (CLA). Remember, you don't have to code to make valuable contributions! Triaging the bug tracker is a great example of this. See you there! Happy hacking.

Enrico Zini: Reimagining Ansible variables

This is part of a series of posts on ideas for an ansible-like provisioning system, implemented in Transilience. While experimenting with Transilience, I've been giving some thought about Ansible variables. My gripes I like the possibility to define host and group variables, and I like to have a set of variables that are autodiscovered on the target systems. I do not like to have everything all blended in a big bucket of global variables. Let's try some more prototyping. My fiddlings First, Role classes could become dataclasses, too, and declare the variables and facts that they intend to use (typed, even!):
class Role(role.Role):
    """
    Postfix mail server configuration
    """
    # Postmaster username
    postmaster: str = None
    # Public name of the mail server
    myhostname: str = None
    # Email aliases defined on this mail server
    aliases: Dict[str, str] = field(default_factory=dict)
Using dataclasses.asdict() I immediately gain context variables for rendering Jinja2 templates:
class Role:
    # [...]
    def render_file(self, path: str, **kwargs):
        """
        Render a Jinja2 template from a file, using as context all Role fields,
        plus the given kwargs.
        """
        ctx = asdict(self)
        ctx.update(kwargs)
        return self.template_engine.render_file(path, ctx)
    def render_string(self, template: str, **kwargs):
        """
        Render a Jinja2 template from a string, using as context all Role fields,
        plus the given kwargs.
        """
        ctx = asdict(self)
        ctx.update(kwargs)
        return self.template_engine.render_string(template, ctx)
I can also model results from fact gathering into dataclass members:
# From ansible/module_utils/facts/system/platform.py
@dataclass
class Platform(Facts):
    """
    Facts from the platform module
    """
    ansible_system: Optional[str] = None
    ansible_kernel: Optional[str] = None
    ansible_kernel: Optional[str] = None
    ansible_kernel_version: Optional[str] = None
    ansible_machine: Optional[str] = None
    # [...]
    ansible_userspace_architecture: Optional[str] = None
    ansible_machine_id: Optional[str] = None
    def summary(self):
        return "gather platform facts"
    def run(self, system: transilience.system.System):
        super().run(system)
        # ... collect facts
I like that this way, one can explicitly declare what variables a Facts action will collect, and what variables a Role needs. At this point, I can add machinery to allow a Role to declare what Facts it needs, and automatically have the fields from the Facts class added to the Role class. Then, when facts are gathered, I can make sure that their fields get copied over to the Role classes that use them. In a way, variables become role-scoped, and Facts subclasses can be used like some kind of Role mixin, that contributes only field members:
# Postfix mail server configuration
@role.with_facts([actions.facts.Platform])
class Role(role.Role):
    # Postmaster username
    postmaster: str = None
    # Public name of the mail server
    myhostname: str = None
    # Email aliases defined on this mail server
    aliases: Dict[str, str] = field(default_factory=dict)
    # All fields from actions.facts.Platform are inherited here!
    def have_facts(self, facts):
        # self.ansible_domain comes from actions.facts.Platform
        self.add(builtin.command(
            argv=["certbot", "certonly", "-d", f"mail. self.ansible_domain ", "-n", "--apache"],
            creates=f"/etc/letsencrypt/live/mail. self.ansible_domain /fullchain.pem"
        ), name="obtain mail.* certificate")
        # the template context will have the Role variables, plus the variables
        # of all the Facts the Role uses
        with self.notify(ReloadPostfix):
            self.add(builtin.copy(
                dest="/etc/postfix/main.cf",
                content=self.render_file("roles/mailserver/templates/main.cf"),
            ), name="configure /etc/postfix/main.cf")
One can also fill in variables when instantiating Roles, making parameterized generic Roles possible and easy:
    runner.add_role(
            "mailserver",
            postmaster="enrico",
            myhostname="mail.enricozini.org",
            aliases= 
                "me": "enrico",
             ,
    )
Outcomes I like where this is going: having well defined variables for facts and roles, means that the variables that get into play can be explicitly defined, well known, and documented. I think this design lends itself quite well to role reuse: I have a feeling that, this way, it may be much easier to create generic libraries of Roles that one can reuse to compose complex playbooks. Since roles are just Python modules, we even already know how to package and distribute them! Next step: Playbooks, host vars, group vars.

Rapha&#235;l Hertzog: Submit your ideas for Debian and +1 those that you find important

A while ago, I got a request from Kentaro Hayashi on the project I use to manage funding requests addressed to Freexian. He was keen to see some improvements on the way reimbursement requests are handled in Debian. In my opinion, the idea is certainly good but he s not part of the treasurer team and was not willing to implement the project either, so it was not really ready to be submitted to us. To be able to fund a useful project, we need either someone that is willing to do the work and try to push it further in Debian, or we need a Debian team interested in the result of the project (and in that case, we can try to find someone willing to implement the project). In this case, it s a bit sad that the treasurer team didn t comment at all but in general, what should we do with those suggestions ? It would still be interesting to have a list of such suggestions and have Debian developers be able to advocate (+1) those suggestions. It s in this spirit that Kentaro created the Grow your ideas project on salsa.debian.org. Browse the list of issues, submit your own and click on  (+1) to bump the suggestions that you find interesting. And click on the Watch button on the homepage to receive notifications when others submit suggestions so that you can evaluate those suggestions and possibly improve them through your comments. In the end, this can only help those that are looking for ways to contribute to Debian: maybe some of them will formalize a concrete action plan and request funding for it! We have a pile of money sitting unused that s clearly targeted to pay contributors to improve our infrastructure. And we have endless lists of (wishlist) bugs on all our infrastructure projects (dak, distro-tracker, ). What are you waiting ?

Joey Hess: typed pipes in every shell

Powershell and nushell take unix piping beyond raw streams of text to structured or typed data. Is it possible to keep a traditional shell like bash and still get typed pipes? I think it is possible, and I'm now surprised noone seems to have done it yet. This is a fairly detailed design for how to do it. I've not implemented it yet. RFC. Let's start with a command called typed. You can use it in a pipeline like this:
typed foo   typed bar   typed baz
What typed does is discover the types of the commands to its left and its right, while communicating the type of the command it runs back to them. Then it checks if the types match, and runs the command, communicating the type information to it. Pipes are unidirectional, so it may seem hard to discover the type to the right, but I'll explain how it can be done in a minute. Now suppose that foo generates json, and bar filters structured data of a variety of types, and baz consumes csv and pretty-prints a table. Then bar will be informed that its input is supposed to be json, and that its output should be csv. If bar didn't support json, typed foo and typed bar would both fail with a type error. Writing "typed" in front of everything is annoying. But it can be made a shell alias like "t". It also possible to wrap programs using typed:
cat >~/bin/foo <<EOF
#/usr/bin/typed /usr/bin/foo
EOF
Or program could import a library that uses typed, so it natively supports being used in typed pipelines. I'll explain one way to make such a library later on, once some more details are clear. Which gets us back to a nice simple pipeline, now automatically typed.
foo   bar   baz
If one of the commands is not actually typed, the other ones in the pipe will treat it as having a raw stream of text as input or output. Which will sometimes result in a type error (yay, I love type errors!), but in other cases can do something useful.
find   bar   baz
# type error, bar expected json or csv
foo   bar   less
# less displays csv 
So how does typed discover the types of the commands to the left and right? That's the hard part. It has to start by finding the pids to its left and right. There is no really good way to do that, but on Linux, it can be done: Look at what /proc/self/fd/0 and /proc/self/fd/1 link to, which contains the unique identifiers of the pipes. Then look at other processes' fd/0 and fd/1 to find matching pipe identifiers. (It's also possible to do this on OSX, I believe. I don't know about BSDs.) Searching through all processes would be a bit expensive (around 15 ms with an average number of processes), but there's a nice optimisation: The shell will have started the processes close together in time, so the pids are probably nearby. So look at the previous pid, and the next pid, and fan outward. Also, check isatty to detect the beginning and end of the pipeline and avoid scanning all the processes in those cases. To indicate the type of the command it will run, typed simply opens a file with an extension of ".typed". The file can be located anywhere, and can be an already existing file, or can be created as needed (eg in /run). Once it discovers the pid at the other end of a pipe, typed first looks at /proc/$pid/cmdline to see if it's also running typed. If it is, it looks at its open file handles to find the first ".typed" file. It may need to wait for the file handle to get opened, which is why it needs to verify the pid is running typed. There also needs to be a way for typed to learn the type of the command it will run. Reading /usr/share/typed/$command.typed is one way. Or it can be specified at the command line, which is useful for wrapper scripts:
cat >~/bin/bar <<EOF
#/usr/bin/typed --type="JSON   CSV" --output-type="JSON   CSV" /usr/bin/bar
EOF
And typed communicates the type information to the command that it runs. This way a command like bar can know what format its input should be in, and what format to use as output. This might be done with environment variables, eg INPUT_TYPE=JSON and OUTPUT_TYPE=CSV I think that's everything typed needs, except for the syntax of types and how the type checking works. Which I should probably not try to think up off the cuff. I used Haskell ADT syntax in the example above, but don't think that's necessarily the right choice. Finally, here's how to make a library that lets a program natively support being used in a typed pipeline. It's a bit tricky, because it has to run typed, because typed checks /proc/$pid/cmdline as detailed above. So, check an environment variable. When not set yet, set it, and exec typed, passing it the path to the program, which it will re-exec. This should be done before program does anything else.
This work was sponsored by Mark Reidenbach on Patreon.

16 June 2021

Abhijith PA: Changing LCD screen of car infotainment system

I have a 2013 model used car that I bought two years ago. It came with a 7 inch touch screen infotainment system on its dash board with features like navigation, Bluetooth phone connectivity and a good FM AM radio. Except the radio I rarely use navigation or Bluetooth phone sync. After couple of months the touch started to become non-responsive. Since all important things such as call termination, mute and volume control have physical switches, I was happy with it. During a periodic car check up on a local workshop, mechanic pulled battery terminals making the infotainment system locked. Now it ask for 4 digit pass code. Its one of their ant-theft mechanism and in order to enter those digits you need a touch responsive screen. So now I am completed locked out. I visited service center of this car to get it changed, turns out they don t repair it and only change by the unit. And will cost me Rs 50,000. Considering my usage is restricted to radio. That price is way too much. So my next options were,
  1. Use a normal cheap car radio player. I dropped this plan, since stereo players comes in very small size and I might need plastic placeholders to close rest of the area from the 7 inch screen. This can mess aesthetics of dash board. And it also affect value of the car if I will sell in future.
  2. Use a 7 inch third party infotainment system available in the market. I did enquired with local car accessories shop and I dropped this plan as well because most of them are android and comes with enormous number of pre installed apps which I never gonna use it anyway. And I don t need one more device that need to be connected to the Internet. Also the wiring at the back of these devices are different than what I have already so it need rewiring.
  3. Last option was to change the infotainment system parts my own. I was under the impression that these units are in-house made, tightly assembled and sealed with screws that you never see in your life and you will never able to be open it up, let alone fix parts.
Couple of articles showed me that car infotainment system sizes and wiring at the back are actually standardized. This particular system s OEM is LG After gaining confidence from several youtube videos and articles, I chose the 3rd option and started to disassemble head unit. Luckily all those screw driver heads were with me. After taking everything apart I could see back of touch screen glass bulged and pushing against the display. The back of the display had a label with its manufacturers name Innolux display and its model number AT070TN94. You might know this company from all those raspberry pi displays in the market. Also Tesla car makers use Innolux display in their infotainment unit. I was able to spot an online reseller in Hong Kong. I only need a touch screen display but to be on the safe side I also placed order for display as well(Price difference were negligible) Took 1 month to reach at my hand due to COVID restrictions. I assembled everything together and collected radio pass code the very same day I received the item. Everything is working now. yay! The touch screen + display cost me Rs 6000/-. I can easily get an average android infotainment around this price. But it was the price I paid for avoiding e-waste. So if you have a broken head unit in your pre 2015 era car. Try to change the parts yourself. Its not complicated like our mobile phones. Back of the infotainment system inside infotainment damaged screen After reassembling with new screen Finally

14 June 2021

Enrico Zini: Pipelining

This is part of a series of posts on ideas for an ansible-like provisioning system, implemented in Transilience. Running actions on a server is nice, but a network round trip for each action is not very efficient. If I need to run a linear sequence of actions, I can stream them all to the server, and then read replies streamed from the server as they get executed. This technique is called pipelining and one can see it used, for example, in Redis, or Mitogen. Roles Ansible has the concept of "Roles" as a series of related tasks: I'll play with that. Here's an example role to install and setup fail2ban:
class Role(role.Role):
    def main(self):
        self.add(builtin.apt(
            name=["fail2ban"],
            state="present",
        ))
        self.add(builtin.copy(
            content=inline("""
                [postfix]
                enabled = true
                [dovecot]
                enabled = true
            """),
            dest="/etc/fail2ban/jail.local",
            owner="root",
            group="root",
            mode=0o644,
        ), name="configure fail2ban")
I prototyped roles as classes, with methods that push actions down the pipeline. If an action fails, all further actions for the same role won't executed, and will be marked as skipped. Since skipping is applied per-role, it means that I can blissfully stream actions for multiple roles to the server down the same pipe, and errors in one role will stop executing that role and not others. Potentially I can get multiple roles going with a single network round-trip:
#!/usr/bin/python3
import sys
from transilience.system import Mitogen
from transilience.runner import Runner
@Runner.cli
def main():
    system = Mitogen("my server", "ssh", hostname="server.example.org", username="root")
    runner = Runner(system)
    # Send roles to the server
    runner.add_role("general")
    runner.add_role("fail2ban")
    runner.add_role("prosody")
    # Run until all roles are done
    runner.main()
if __name__ == "__main__":
    sys.exit(main())
That looks like a playbook, using Python as glue rather than YAML. Decision making in roles Besides filing a series of actions, a role may need to take decisions based on the results of previous actions, or on facts discovered from the server. In that case, we need to wait until the results we need come back from the server, and then decide if we're done or if we want to send more actions down the pipe. Here's an example role that installs and configures Prosody:
from transilience import actions, role
from transilience.actions import builtin
from .handlers import RestartProsody
class Role(role.Role):
    """
    Set up prosody XMPP server
    """
    def main(self):
        self.add(actions.facts.Platform(), then=self.have_facts)
        self.add(builtin.apt(
            name=["certbot", "python-certbot-apache"],
            state="present",
        ), name="install support packages")
        self.add(builtin.apt(
            name=["prosody", "prosody-modules", "lua-sec", "lua-event", "lua-dbi-sqlite3"],
            state="present",
        ), name="install prosody packages")
    def have_facts(self, facts):
        facts = facts.facts  # Malkovich Malkovich Malkovich!
        domain = facts["domain"]
        ctx =  
            "ansible_domain": domain
         
        self.add(builtin.command(
            argv=["certbot", "certonly", "-d", f"chat. domain ", "-n", "--apache"],
            creates=f"/etc/letsencrypt/live/chat. domain /fullchain.pem"
        ), name="obtain chat certificate")
        with self.notify(RestartProsody):
            self.add(builtin.copy(
                content=self.template_engine.render_file("roles/prosody/templates/prosody.cfg.lua", ctx),
                dest="/etc/prosody/prosody.cfg.lua",
            ), name="write prosody configuration")
            self.add(builtin.copy(
                src="roles/prosody/templates/firewall-ruleset.pfw",
                dest="/etc/prosody/firewall-ruleset.pfw",
            ), name="write prosody firewall")
    # ...
This files some general actions down the pipe, with a hook that says: when the results of this action come back, run self.have_facts(). At that point, the role can use the results to build certbot command lines, render prosody's configuration from Jinja2 templates, and use the results to file further action down the pipe. Note that this way, while the server is potentially still busy installing prosody, we're already streaming prosody's configuration to it. If anything goes wrong with the installation of prosody's package, the role will be marked as failed and all further actions of the same role, even those filed by have_facts() will be skipped. Notify and handlers In the previous example self.notify() also appears: that's my attempt to model the equivalent of Ansible's handlers. If any of the actions inside the with produce changes, then the RestartProsody role will be executed, potentially filing more actions ad the end of the playbook. The runner will take care of collecting all the triggered role classes in a set, which discards duplicates, and then running the main() method of all resulting roles, which will cause more actions to be filed down the pipe. Action conditions Sometimes some actions are only meaningful as consequences of other actions. Let's take, for example, enabling buster-backports as an extra apt source:
        a = self.add(builtin.copy(
            owner="root",
            group="root",
            mode=0o644,
            dest="/etc/apt/sources.list.d/debian-buster-backports.list",
            content="deb [arch=amd64] https://mirrors.gandi.net/debian/ buster-backports main contrib",
        ), name="enable backports")
        self.add(builtin.apt(
            update_cache=True
        ), name="update after enabling backports",
           # Run only if the previous copy changed anything
           when= a: ResultState.CHANGED ,
        )
Here we want to update Apt's cache, which is a slow operation, only after we actually write /etc/apt/sources.list.d/debian-buster-backports.list. If the file was already there from a previous run, we can skip downloading the new package lists. The when= attributes adds an annotation to the action that is sent town the pipeline, that says that it should only be run if the state of a previous action matches the given one. In this case, when on the remote it's the turn of "update after enabling backports", it gets skipped unless the state of the previous "enable backports" action is CHANGED. Effects of pipelining I ported enough of Ansible's modules to be able to run the provisioning scripts of my VPS entirely via ansible. This is the playbook run as plain Ansible:
$ time ansible-playbook vps.yaml
[...]
servername       : ok=55   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
real    2m10.072s
user    0m33.149s
sys 0m10.379s
This is the same playbook run with Ansible speeded up via the Mitogen backend, which makes Ansible more bearable:
$ export ANSIBLE_STRATEGY=mitogen_linear
$ time ansible-playbook vps.yaml
[...]
servername       : ok=55   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
real    0m24.428s
user    0m8.479s
sys 0m1.894s
This is the same playbook ported to Transilience:
$ time ./provision
[...]
real    0m2.585s
user    0m0.659s
sys 0m0.034s
Doing nothing went from 2 minutes down to 3 seconds! That's the kind of running time that finally makes me comfortable with maintaining my VPS by editing the playbook only, and never logging in to mess with the system configuration by hand! Next steps I'm quite happy with what I have: I can now maintain my VPS with a simple script with quick iterative cycles. I might use it to develop new playbooks, and port them to ansible only when they're tested and need to be shared with infrastructure that needs to rely on something more solid and battle tested than a prototype provisioning system. I might also keep working on it as I have more interesting ideas that I'd like to try. I feel like Ansible reached some architectural limits that are hard to overcome without a major redesign, and are in many way hardcoded in its playbook configuration. It's nice to be able to try out new designs without that baggage. I'd love it if even just the library of Transilience actions could grow, and gain widespread use. Ansible modules standardized a set of management operations, that I think became the way people think about system management, and should really be broadly available outside of Ansible. If you are interesting in playing with Transilience, such as: do get in touch or send a pull request! :) Next step: Reimagining Ansible variables.

Enrico Zini: Use ansible actions in a script

This is part of a series of posts on ideas for an ansible-like provisioning system, implemented in Transilience. I like many of the modules provided with Ansible: they are convenient, platform-independent implementations of common provisioning steps. They'd be fantastic to have in a library that I could use in normal programs. This doesn't look easy to do with Ansible code as it is. Also, the code quality of various Ansible modules doesn't fit something I'd want in a standard library of cross-platform provisioning functions. Modeling Actions I want to keep the declarative, idempotent aspect of describing actions on a system. A good place to start could be a hierarchy of dataclasses that hold the same parameters as ansible modules, plus a run() method that performs the action:
@dataclass
class Action:
    """
    Base class for all action implementations.
    An Action is the equivalent of an ansible module: a declarative
    representation of an idempotent operation on a system.
    An Action can be run immediately, or serialized, sent to a remote system,
    run, and sent back with its results.
    """
    uuid: str = field(default_factory=lambda: str(uuid.uuid4()))
    result: Result = field(default_factory=Result)
    def summary(self):
        """
        Return a short text description of this action
        """
        return self.__class__.__name__
    def run(self, system: transilience.system.System):
        """
        Perform the action
        """
        self.result.state = ResultState.NOOP
I like that Ansible tasks have names, and I hate having to give names to trivial tasks like "Create directory /foo/bar", so I added a summary() method so that trivial tasks like that can take care of naming themselves. Dataclasses allow to introspect fields and annotate them with extra metadata, and together with docstrings, I can make actions reasonably self-documeting. I ported some of Ansible's modules over: see complete list in the git repository. Running Actions in a script With a bit of glue code I can now run Ansible-style functions from a plain Python script:
#!/usr/bin/python3
from transilience.runner import Script
script = Script()
for i in range(10):
    script.builtin.file(state="touch", path=f"/tmp/test i ")
Running Actions remotely Dataclasses have an asdict function that makes them trivially serializable. If their members stick to data types that can be serialized with Mitogen and the run implementation doesn't use non-pure, non-stdlib Python modules, then I can trivially run actions on all sorts of remote systems using Mitogen:
#!/usr/bin/python3
from transilience.runner import Script
from transilience.system import Mitogen
script = Script(system=Mitogen("my server", "ssh", hostname="machine.example.org", username="user"))
for i in range(10):
    script.builtin.file(state="touch", path=f"/tmp/test i ")
How fast would that be, compared to Ansible?
$ time ansible-playbook test.yaml
[...]
real    0m15.232s
user    0m4.033s
sys 0m1.336s
$ time ./test_script
real    0m4.934s
user    0m0.547s
sys 0m0.049s
With a network round-trip for each single operation I'm already 3x faster than Ansible, and it can run on nspawn containers, too! I always wanted to have a library of ansible modules useable in normal scripts, and I've always been angry with Ansible for not bundling their backend code in a generic library. Well, now there's the beginning of one! Sweet! Next step, pipelining.

Enrico Zini: My gripes with Ansible

This is part of a series of posts on ideas for an ansible-like provisioning system, implemented in Transilience. Musing about Ansible I like infrastructure as code. I like to be able to represent an entire system as text files in a git repositories, and to be able to use that to recreate the system, from my Virtual Private Server, to my print server and my stereo, to build machines, to other kind of systems I might end up setting up. I like that the provisioning work I do on a machine can be self-documenting and replicable at will. The good For that I quite like Ansible, in principle: simple (in theory) YAML files describe a system in (reasonably) high-level steps, and it can be run on (almost) any machine that happens to have a simple Python interpreter installed. I also like many of the modules provided with Ansible: they are convenient, platform-independent implementations of common provisioning steps. They'd be fantastic to have in a library that I could use in normal programs. The bad Unfortunately, Ansible is slow. Running the playbook on my VPS takes about 3 whole minutes even if I'm just changing a line in a configuration file. This means that most of the time, instead of changing that line in the playbook and running it, to then figure out after 3 minutes that it was the wrong line, or I made a spelling mistake in the playbook, I end up logging into the server and editing in place. That defeats the whole purpose, but that level of latency between iterations is just unacceptable to me. The ugly I also think that Ansible has outgrown its original design, and the supposedly declarative, idempotent YAML has become a full declarative scripting language in disguise, whose syntax is extremely awkward and verbose. If I'm writing declarative descriptions, YAML is great. If I'm writing loops and conditionals, I want to write code, not templated YAML. I also keep struggling trying to use Ansible to provision chroots and nspawn containers. A personal experiment: Transilience There's another thing I like in Ansible: it's written in Python, which is a language I'm comfortable with. Compared to other platforms, it's one that I'm more likely to be able to control beyond being a simple user. What if I can port Ansible modules into a library of high-level provisioning functions, that I can just run via normal Python scripts? What if I can find a way to execute those scripts remotely and not just locally? I've started writing some prototype code, and the biggest problem is, of course, finding a name. Ansible comes from Ursula K. Le Guin's Hainish Cycle novels, where it is a device that allows its users to communicate near-instantaneously over interstellar distances. Traveling, however, is still constrained by the speed of light. Later in the same universe, the novels A Fisherman of the Inland Sea and The Shobies' Story, talk about experiments with instantaneous interstellar travel, as a science Ursula Le Guin called transilience:
Transilience: n. A leap across or from one thing to another [1913 Webster]
Transilience. I like everything about this name. Now that the hardest problem is solved, the rest is just a simple matter of implementation details.

Fran ois Marier: Self-hosting an Ikiwiki blog

8.5 years ago, I moved my blog to Ikiwiki and Branchable. It's now time for me to take the next step and host my blog on my own server. This is how I migrated from Branchable to my own Apache server.

Installing Ikiwiki dependencies Here are all of the extra Debian packages I had to install on my server:
apt install ikiwiki ikiwiki-hosting-common gcc libauthen-passphrase-perl libcgi-formbuilder-perl libcrypt-sslauthen-passphrase-perl libcgi-formbuilder-perl libcrypt-ssleay-perl libjson-xs-perl librpc-xml-perl python-docutils libxml-feed-perl libsearch-xapian-perl libmailtools-perl highlight-common libsearch-xapian-perl xapian-omega
apt install --no-install-recommends ikiwiki-hosting-web libgravatar-url-perl libmail-sendmail-perl libcgi-session-perl
apt purge libnet-openid-consumer-perl
Then I enabled the CGI module in Apache:
a2enmod cgi
and un-commented the following in /etc/apache2/mods-available/mime.conf:
AddHandler cgi-script .cgi

Creating a separate user account Since Ikiwiki needs to regenerate my blog whenever a new article is pushed to the git repo or a comment is accepted, I created a restricted user account for it:
adduser blog
adduser blog sshuser
chsh -s /usr/bin/git-shell blog

git setup Thanks to Branchable storing blogs in git repositories, I was able to import my blog using a simple git clone in /home/blog (the srcdir):
git clone --bare git://feedingthecloud.branchable.com/ source.git
Note that the name of the directory (source.git) is important for the ikiwikihosting plugin to work. Then I pulled the .setup file out of the setup branch in that repo and put it in /home/blog/.ikiwiki/FeedingTheCloud.setup. After that, I deleted the setup branch and the origin remote from that clone:
git branch -d setup
git remote rm origin
Following the recommended git configuration, I created a working directory (the repository) for the blog user to modify the blog as needed:
cd /home/blog/
git clone /home/blog/source.git FeedingTheCloud
I added my own ssh public key to /home/blog/.ssh/authorized_keys so that I could push to the srcdir from my laptop. Finaly, I generated a new ssh key without a passphrase:
ssh-keygen -t ed25519
and added it as deploy key to the GitHub repo which acts as a read-only mirror of my blog.

Ikiwiki config While I started with the Branchable setup file, I changed the following things in it:
adminemail: webmaster@fmarier.org
srcdir: /home/blog/FeedingTheCloud
destdir: /var/www/blog
url: https://feeding.cloud.geek.nz
cgiurl: https://feeding.cloud.geek.nz/blog.cgi
cgi_wrapper: /var/www/blog/blog.cgi
cgi_wrappermode: 675
add_plugins:
- goodstuff
- lockedit
- comments
- blogspam
- sidebar
- attachment
- favicon
- format
- highlight
- search
- theme
- moderatedcomments
- flattr
- calendar
- headinganchors
- notifyemail
- anonok
- autoindex
- date
- relativedate
- htmlbalance
- pagestats
- sortnaturally
- ikiwikihosting
- gitpush
- emailauth
disable_plugins:
- brokenlinks
- fortune
- more
- openid
- orphans
- passwordauth
- progress
- recentchanges
- repolist
- toggle
- txt
sslcookie: 1
cookiejar:
  file: /home/blog/.ikiwiki/cookies
useragent: ikiwiki
git_wrapper: /home/blog/source.git/hooks/post-update
urlalias:
- http://feeds.cloud.geek.nz/
- http://www.feeding.cloud.geek.nz/
owner: francois@fmarier.org
hostname: feeding.cloud.geek.nz
emailauth_sender: login@fmarier.org
allowed_attachments: admin()
Then I created the destdir:
mkdir /var/www/blog
chown blog:blog /var/www/blog
and generated the initial copy of the blog as the blog user:
ikiwiki --setup .ikiwiki/FeedingTheCloud.setup --wrappers --rebuild
One thing that failed to generate properly was the tag cloug (from the pagestats plugin). I have not been able to figure out why it fails to generate any output when run this way, but if I push to the repo and let the git hook handle the rebuilding of the wiki, the tag cloud is generated correctly. Consequently, fixing this is not high on my list of priorities, but if you happen to know what the problem is, please reach out.

Apache config Here's the Apache config I put in /etc/apache2/sites-available/blog.conf:
<VirtualHost *:443>
    ServerName feeding.cloud.geek.nz
    SSLEngine On
    SSLCertificateFile /etc/letsencrypt/live/feeding.cloud.geek.nz/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/feeding.cloud.geek.nz/privkey.pem
    Header set Strict-Transport-Security: "max-age=63072000; includeSubDomains; preload"
    Include /etc/fmarier-org/blog-common
</VirtualHost>
<VirtualHost *:443>
    ServerName www.feeding.cloud.geek.nz
    ServerAlias feeds.cloud.geek.nz
    SSLEngine On
    SSLCertificateFile /etc/letsencrypt/live/feeding.cloud.geek.nz/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/feeding.cloud.geek.nz/privkey.pem
    Redirect permanent / https://feeding.cloud.geek.nz/
</VirtualHost>
<VirtualHost *:80>
    ServerName feeding.cloud.geek.nz
    ServerAlias www.feeding.cloud.geek.nz
    ServerAlias feeds.cloud.geek.nz
    Redirect permanent / https://feeding.cloud.geek.nz/
</VirtualHost>
and the common config I put in /etc/fmarier-org/blog-common:
ServerAdmin webmaster@fmarier.org
DocumentRoot /var/www/blog
LogLevel core:info
CustomLog $ APACHE_LOG_DIR /blog-access.log combined
ErrorLog $ APACHE_LOG_DIR /blog-error.log
AddType application/rss+xml .rss
<Location /blog.cgi>
        Options +ExecCGI
</Location>
before enabling all of this using:
a2ensite blog
apache2ctl configtest
systemctl restart apache2.service
The feeds.cloud.geek.nz domain used to be pointing to Feedburner and so I need to maintain it in order to avoid breaking RSS feeds from folks who added my blog to their reader a long time ago.

Server-side improvements Since I'm now in control of the server configuration, I was able to make several improvements to how my blog is served. First of all, I enabled the HTTP/2 and Brotli modules:
a2enmod http2
a2enmod brotli
and enabled Brotli compression by putting the following in /etc/apache2/conf-available/compression.conf:
<IfModule mod_brotli.c>
  <IfDefine !TRANSFER_COMPRESSION>
    Define TRANSFER_COMPRESSION BROTLI_COMPRESS
  </IfDefine>
</IfModule>
<IfModule mod_deflate.c>
  <IfDefine !TRANSFER_COMPRESSION>
    Define TRANSFER_COMPRESSION DEFLATE
  </IfDefine>
</IfModule>
<IfDefine TRANSFER_COMPRESSION>
  <IfModule mod_filter.c>
    AddOutputFilterByType $ TRANSFER_COMPRESSION  text/html text/plain text/xml text/css text/javascript
    AddOutputFilterByType $ TRANSFER_COMPRESSION  application/x-javascript application/javascript application/ecmascript
    AddOutputFilterByType $ TRANSFER_COMPRESSION  application/rss+xml
    AddOutputFilterByType $ TRANSFER_COMPRESSION  application/xml
  </IfModule>
</IfDefine>
and replacing /etc/apache2/mods-available/deflate.conf with the following:
# Moved to /etc/apache2/conf-available/compression.conf as per https://bugs.debian.org/972632
before enabling this new config:
a2enconf compression
Next, I made my blog available as a Tor onion service by putting the following in /etc/apache2/sites-available/blog.conf:
<VirtualHost *:443>
    ServerName feeding.cloud.geek.nz
    ServerAlias xfdug5vmfi6oh42fp6ahhrqdjcf7ysqat6fkp5dhvde4d7vlkqixrsad.onion
    Header set Onion-Location "http://xfdug5vmfi6oh42fp6ahhrqdjcf7ysqat6fkp5dhvde4d7vlkqixrsad.onion% REQUEST_URI s"
    Header set alt-svc 'h2="xfdug5vmfi6oh42fp6ahhrqdjcf7ysqat6fkp5dhvde4d7vlkqixrsad.onion:443"; ma=315360000; persist=1'
    ... 
<VirtualHost *:80>
    ServerName xfdug5vmfi6oh42fp6ahhrqdjcf7ysqat6fkp5dhvde4d7vlkqixrsad.onion
    Include /etc/fmarier-org/blog-common
</VirtualHost>
Then I followed the Mozilla Observatory recommendations and enabled the following security headers:
Header set Content-Security-Policy: "default-src 'none'; report-uri https://fmarier.report-uri.com/r/d/csp/enforce ; style-src 'self' 'unsafe-inline' ; img-src 'self' https://seccdn.libravatar.org/ ; script-src https://feeding.cloud.geek.nz/ikiwiki/ https://xfdug5vmfi6oh42fp6ahhrqdjcf7ysqat6fkp5dhvde4d7vlkqixrsad.onion/ikiwiki/ http://xfdug5vmfi6oh42fp6ahhrqdjcf7ysqat6fkp5dhvde4d7vlkqixrsad.onion/ikiwiki/ 'unsafe-inline' 'sha256-pA8FbKo4pYLWPDH2YMPqcPMBzbjH/RYj0HlNAHYoYT0=' 'sha256-Kn5E/7OLXYSq+EKMhEBGJMyU6bREA9E8Av9FjqbpGKk=' 'sha256-/BTNlczeBxXOoPvhwvE1ftmxwg9z+WIBJtpk3qe7Pqo=' ; base-uri 'self'; form-action 'self' ; frame-ancestors 'self'"
Header set X-Frame-Options: "SAMEORIGIN"
Header set Referrer-Policy: "same-origin"
Header set X-Content-Type-Options: "nosniff"
Note that the Mozilla Observatory is mistakenly identifying HTTP onion services as insecure, so you can ignore that failure. I also used the Mozilla TLS config generator to improve the TLS config for my server. Then I added security.txt and gpc.json to the root of my git repo and then added the following aliases to put these files in the right place:
Alias /.well-known/gpc.json /var/www/blog/gpc.json
Alias /.well-known/security.txt /var/www/blog/security.txt
I also followed these instructions to create a sitemap for my blog with the following alias:
Alias /sitemap.xml /var/www/blog/sitemap/index.rss
Finally, I simplified a few error pages to save bandwidth:
ErrorDocument 301 " "
ErrorDocument 302 " "
ErrorDocument 404 "Not Found"

Monitoring 404s Another advantage of running my own web server is that I can monitor the 404s easily using logcheck by putting the following in /etc/logcheck/logcheck.logfiles:
/var/log/apache2/blog-error.log 
Based on that, I added a few redirects to point bots and users to the location of my RSS feed:
Redirect permanent /atom /index.atom
Redirect permanent /comments.rss /comments/index.rss
Redirect permanent /comments.atom /comments/index.atom
Redirect permanent /FeedingTheCloud /index.rss
Redirect permanent /feed /index.rss
Redirect permanent /feed/ /index.rss
Redirect permanent /feeds/posts/default /index.rss
Redirect permanent /rss /index.rss
Redirect permanent /rss/ /index.rss
and to tell them to stop trying to fetch obsolete resources:
Redirect gone /~ff/FeedingTheCloud
Redirect gone /gittip_button.png
Redirect gone /ikiwiki.cgi
I also used these 404s to discover a few old Feedburner URLs that I could redirect to the right place using archive.org:
Redirect permanent /feeds/1572545745827565861/comments/default /posts/watch-all-of-your-logs-using-monkeytail/comments.atom
Redirect permanent /feeds/1582328597404141220/comments/default /posts/news-feeds-rssatom-for-mythtvorg-and/comments.atom
...
Redirect permanent /feeds/8490436852808833136/comments/default /posts/recovering-lost-git-commits/comments.atom
Redirect permanent /feeds/963415010433858516/comments/default /posts/debugging-openwrt-routers-by-shipping/comments.atom
I also put the following robots.txt in the git repo in order to stop a bunch of authentication errors coming from crawlers:
User-agent: *
Disallow: /blog.cgi
Disallow: /ikiwiki.cgi

Future improvements There are a few things I'd like to improve on my current setup. The first one is to remove the iwikihosting and gitpush plugins and replace them with a small script which would simply git push to the read-only GitHub mirror. Then I could uninstall the ikiwiki-hosting-common and ikiwiki-hosting-web since that's all I use them for. Next, I would like to have proper support for signed git pushes. At the moment, I have the following in /home/blog/source.git/config:
[receive]
    advertisePushOptions = true
    certNonceSeed = "(random string)"
but I'd like to also reject unsigned pushes. While my blog now has a CSP policy which doesn't rely on unsafe-inline for scripts, it does rely on unsafe-inline for stylesheets. I tried to remove this but the actual calls to allow seemed to be located deep within jQuery and so I gave up. Update: now fixed. Finally, I'd like to figure out a good way to deal with articles which don't currently have comments. At the moment, if you try to subscribe to their comment feed, it returns a 404. For example:
[Sun Jun 06 17:43:12.336350 2021] [core:info] [pid 30591:tid 140253834704640] [client 66.249.66.70:57381] AH00128: File does not exist: /var/www/blog/posts/using-iptables-with-network-manager/comments.atom
This is obviously not ideal since many feed readers will refuse to add a feed which is currently not found even though it could become real in the future. If you know of a way to fix this, please let me know.

12 June 2021

Kentaro Hayashi: fabre.debian.net has moved to Debian.net Team Infrastructure

Today, fabre.debian.net has moved to Debian.net Team Infrastructure So far, fabre.debian.net was sponsored by FOSSHOST which provides us a VPS instance since Jan, 2021. It was located at OSU Open Source Lab. It worked pretty well, Thanks FOSSHOST sponsorship since ever! Now, fabre.debian.net uses the VPS instance which is provided by Debian.net Team Infrastructure. (still non-DSA managed) It is hosted at HETZNER Cloud. About fabre.debian.net fabre.debian.net is a experimental service to demonstrate how to improve user experience with finding and fixing Debian unstable related bugs for making "unstable life" comfortable. Thank Debian.net Team for sponsoring,

11 June 2021

Colin Watson: SSH quoting

A while back there was a thread on one of our company mailing lists about SSH quoting, and I posted a long answer to it. Since then a few people have asked me questions that caused me to reach for it, so I thought it might be helpful if I were to anonymize the original question and post my answer here. The question was why a sequence of commands involving ssh and fiddly quoting produced the output they did. The first example was this:
$ ssh user@machine.local bash -lc "cd /tmp;pwd"
/home/user
Oh hi, my dubious life choices have been such that this is my specialist subject! This is because SSH command-line parsing is not quite what you expect. First, recall that your local shell will apply its usual parsing, and the actual OS-level execution of ssh will be like this:
[0]: ssh
[1]: user@machine.local
[2]: bash
[3]: -lc
[4]: cd /tmp;pwd
Now, the SSH wire protocol only takes a single string as the command, with the expectation that it should be passed to a shell by the remote end. The OpenSSH client deals with this by taking all its arguments after things like options and the target, which in this case are:
[0]: bash
[1]: -lc
[2]: cd /tmp;pwd
It then joins them with a single space:
bash -lc cd /tmp;pwd
This is passed as a string to the server, which then passes that entire string to a shell for evaluation, so as if you d typed this directly on the server:
sh -c 'bash -lc cd /tmp;pwd'
The shell then parses this as two commands:
bash -lc cd /tmp
pwd
The directory change thus happens in a subshell (actually it doesn t quite even do that, because bash -lc cd /tmp in fact ends up just calling cd because of the way bash -c parses multiple arguments), and then that subshell exits, then pwd is called in the outer shell which still has the original working directory. The second example was this:
$ ssh user@machine.local bash -lc "pwd;cd /tmp;pwd"
/home/user
/tmp
Following the logic above, this ends up as if you d run this on the server:
sh -c 'bash -lc pwd; cd /tmp; pwd'
The third example was this:
$ ssh user@machine.local bash -lc "cd /tmp;cd /tmp;pwd"
/tmp
And this is as if you d run:
sh -c 'bash -lc cd /tmp; cd /tmp; pwd'
Now, I wouldn t have implemented the SSH client this way, because I agree that it s confusing. But /usr/bin/ssh is used as a transport for other things so much that changing its behaviour now would be enormously disruptive, so it s probably impossible to fix. (I have occasionally agitated on openssh-unix-dev@ for at least documenting this better, but haven t made much headway yet; I need to get round to preparing a documentation patch.) Once you know about it you can use the proper quoting, though. In this case that would simply be:
ssh user@machine.local 'cd /tmp;pwd'
Or if you do need to specifically invoke bash -l there for some reason (I m assuming that the original example was reduced from something more complicated), then you can minimise your confusion by passing the whole thing as a single string in the form you want the remote sh -c to see, in a way that ensures that the quotes are preserved and sent to the server rather than being removed by your local shell:
ssh user@machine.local 'bash -lc "cd /tmp;pwd"'
Shell parsing is hard.

Next.

Previous.