Search Results: "rra"

16 March 2022

Michael Ablassmeier: python logging messages and exit codes

Everyone knows that an application exit code should change based on the success, error or maybe warnings that happened during execution. Lately i came along some python code that was structured the following way:
#!/usr/bin/python3
import sys
import logging
def warnme():
    # something bad happens
    logging.warning("warning")
    sys.exit(2)
def evil():
    # something evil happens
    logging.error("error")
    sys.exit(1)
def main():
    logging.basicConfig(
        level=logging.DEBUG,
    )   
    [..]
the situation was a little bit more complicated, some functions in other modules also exited the application, so sys.exit() calls were distributed in lots of modules an files. Exiting the application in some random function of another module is something i dont consider nice coding style, because it makes it hard to track down errors. I expect:

How to do better? As the application is using the logging module, we have a single point to collect warnings and errors that might happen accross all modules. This works by passing a custom handler to the logging module which tracks emitted messages. Heres an small example:
#!/usr/bin/python3
import sys
import logging
class logCount(logging.Handler):
    class LogType:
        def __init__(self):
            self.warnings = 0
            self.errors = 0
    def __init__(self):
        super().__init__()
        self.count = self.LogType()
    def emit(self, record):
        if record.levelname == "WARNING":
            self.count.warnings += 1
        if record.levelname == "ERROR":
            self.count.errors += 1
            
def infome():
    logging.info("hello world")
def warnme():
    logging.warning("help, an warning")
def evil():
    logging.error("yikes")
def main():
    EXIT_WARNING = 2
    EXIT_ERROR = 1
    counter = logCount()
    logging.basicConfig(
        level=logging.DEBUG,
        handlers=[counter, logging.StreamHandler(sys.stderr)],
    )
    infome()
    warnme()
    evil()
    if counter.count.errors != 0:
        raise SystemExit(EXIT_ERROR)
    if counter.count.warnings != 0:
        raise SystemExit(EXIT_WARNING)
if __name__ == "__main__":
    main()
python3 count.py ; echo $?
INFO:root:hello world
WARNING:root:help, an warning
ERROR:root:yikes
1
This also makes easy to define something like:
  • hey, got 2 warnings, change exit code to error?
  • got 3 warnings, but no strict passed, ingore those, exit with success!
  • etc..

15 March 2022

Kunal Mehta: How to mirror the Russian Wikipedia with Debian and Kiwix

It has been reported that the Russian government has threatened to block access to Wikipedia for documenting narratives that do not agree with the official position of the Russian government. One of the anti-censorship strategies I've been working on is Kiwix, an offline Wikipedia reader (and plenty of other content too). Kiwix is free and open source software developed by a great community of people that I really enjoy working with. With threats of censorship, traffic to Kiwix has increased fifty-fold, with users from Russia accounting for 40% of new downloads! You can download copies of every language of Wikipedia for offline reading and distribution, as well as hosting your own read-only mirror, which I'm going to explain today. Disclaimer: depending on where you live it may be illegal or get you in trouble with the authorities to rehost Wikipedia content, please be aware of your digital and physical safety before proceeding. With that out of the way, let's get started. You'll need a Debian (or Ubuntu) server with at least 30GB of free disk space. You'll also want to have a webserver like Apache or nginx installed (I'll share the Apache config here). First, we need to download the latest copy of the Russian Wikipedia.
$ wget 'https://download.kiwix.org/zim/wikipedia/wikipedia_ru_all_maxi_2022-03.zim'
If the download is interrupted or fails, you can use wget -c $url to resume it. Next let's install kiwix-serve and try it out. If you're using Ubuntu, I strongly recommend enabling our Kiwix PPA first.
$ sudo apt update
$ sudo apt install kiwix-tools
$ kiwix-serve -p 3004 wikipedia_ru_all_maxi_2022-03.zim
At this point you should be able to visit http://yourserver.com:3004/ and see the Russian Wikipedia. Awesome! You can use any available port, I just picked 3004. Now let's use systemd to daemonize it so it runs in the background. Create /etc/systemd/system/kiwix-ru-wp.service with the following:
[Unit]
Description=Kiwix Russian Wikipedia
[Service]
Type=simple
User=www-data
ExecStart=/usr/bin/kiwix-serve -p 3004 /path/to/wikipedia_ru_all_maxi_2022-03.zim
Restart=always
[Install]
WantedBy=multi-user.target
Now let's start it and enable it at boot:
$ sudo systemctl start kiwix-ru-wp
$ sudo systemctl enable kiwix-ru-wp
Since we want to expose this on the public internet, we should put it behind a more established webserver and configure HTTPS. Here's the Apache httpd configuration I used:
<VirtualHost *:80>
        ServerName ru-wp.yourserver.com
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        ErrorLog $ APACHE_LOG_DIR /error.log
        CustomLog $ APACHE_LOG_DIR /access.log combined
        <Proxy *>
                Require all granted
        </Proxy>
        ProxyPass / http://127.0.0.1:3004/
        ProxyPassReverse / http://127.0.0.1:3004/
</VirtualHost>
Put that in /etc/apache2/sites-available/kiwix-ru-wp.conf and run:
$ sudo a2ensite kiwix-ru-wp
$ sudo systemctl reload apache2
Finally, I used certbot to enable HTTPS on that subdomain and redirect all HTTP traffic over to HTTPS. This is an interactive process that is well documented so I'm not going to go into it in detail. You can see my mirror of the Russian Wikipedia, following these instructions, at https://ru-wp.legoktm.com/. Anyone is welcome to use it or distribute the link, though I am not committing to running it long-term. This is certainly not a perfect anti-censorship solution, the copy of Wikipedia that Kiwix provides became out of date the moment it was created, and the setup described here will require you to manually update the service when the new copy is available next month. Finally, if you have some extra bandwith, you can also help seed this as a torrent.

2 March 2022

Antoine Beaupr : procmail considered harmful

TL;DR: procmail is a security liability and has been abandoned upstream for the last two decades. If you are still using it, you should probably drop everything and at least remove its SUID flag. There are plenty of alternatives to chose from, and conversion is a one-time, acceptable trade-off.

Procmail is unmaintained procmail is unmaintained. The "Final release", according to Wikipedia, dates back to September 10, 2001 (3.22). That release was shipped in Debian since then, all the way back from Debian 3.0 "woody", twenty years ago. Debian also ships 25 uploads on top of this, with 3.22-21 shipping the "3.23pre" release that has been rumored since at least the November 2001, according to debian/changelog at least:
procmail (3.22-1) unstable; urgency=low
  * New upstream release, which uses the  standard' format for Maildir
    filenames and retries on name collision. It also contains some
    bug fixes from the 3.23pre snapshot dated 2001-09-13.
  * Removed  sendmail' from the Recommends field, since we already
    have  exim' (the default Debian MTA) and  mail-transport-agent'.
  * Removed suidmanager support. Conflicts: suidmanager (<< 0.50).
  * Added support for DEB_BUILD_OPTIONS in the source package.
  * README.Maildir: Do not use locking on the example recipe,
    since it's wrong to do so in this case.
 -- Santiago Vila <sanvila@debian.org>  Wed, 21 Nov 2001 09:40:20 +0100
All Debian suites from buster onwards ship the 3.22-26 release, although the maintainer just pushed a 3.22-27 release to fix a seven year old null pointer dereference, after this article was drafted. Procmail is also shipped in all major distributions: Fedora and its derivatives, Debian derivatives, Gentoo, Arch, FreeBSD, OpenBSD. We all seem to be ignoring this problem. The upstream website (http://procmail.org/) has been down since about 2015, according to Debian bug #805864, with no change since. In effect, every distribution is currently maintaining its fork of this dead program. Note that, after filing a bug to keep Debian from shipping procmail in a stable release again, I was told that the Debian maintainer is apparently in contact with the upstream. And, surprise! they still plan to release that fabled 3.23 release, which has been now in "pre-release" for all those twenty years. In fact, it turns out that 3.23 is considered released already, and that the procmail author actually pushed a 3.24 release, codenamed "Two decades of fixes". That amounts to 25 commits since 3.23pre some of which address serious security issues, but none of which address fundamental issues with the code base.

Procmail is insecure By default, procmail is installed SUID root:mail in Debian. There's no debconf or pre-seed setting that can change this. There has been two bug reports against the Debian to make this configurable (298058, 264011), but both were closed to say that, basically, you should use dpkg-statoverride to change the permissions on the binary. So if anything, you should immediately run this command on any host that you have procmail installed on:
dpkg-statoverride --update --add root root 0755 /usr/bin/procmail
Note that this might break email delivery. It might also not work at all, thanks to usrmerge. Not sure. Yes, everything is on fire. This is fine. In my opinion, even assuming we keep procmail in Debian, that default should be reversed. It should be up to people installing procmail to assign it those dangerous permissions, after careful consideration of the risk involved. The last maintainer of procmail explicitly advised us (in that null pointer dereference bug) and other projects (e.g. OpenBSD, in [2]) to stop shipping it, back in 2014. Quote:
Executive summary: delete the procmail port; the code is not safe and should not be used as a basis for any further work.
I just read some of the code again this morning, after the original author claimed that procmail was active again. It's still littered with bizarre macros like:
#define bit_set(name,which,value) \
  (value?(name[bit_index(which)] =bit_mask(which)):\
  (name[bit_index(which)]&=~bit_mask(which)))
... from regexp.c, line 66 (yes, that's a custom regex engine). Or this one:
#define jj  (aleps.au.sopc)
It uses insecure functions like strcpy extensively. malloc() is thrown around gotos like it's 1984 all over again. (To be fair, it has been feeling like 1984 a lot lately, but that's another matter entirely.) That null pointer deref bug? It's fixed upstream now, in this commit merged a few hours ago, which I presume might be in response to my request to remove procmail from Debian. So while that's nice, this is the just tip of the iceberg. I speculate that one could easily find an exploitable crash in procmail if only by running it through a fuzzer. But I don't need to speculate: procmail had, for years, serious security issues that could possibly lead to root privilege escalation, remotely exploitable if procmail is (as it's designed to do) exposed to the network. Maybe I'm overreacting. Maybe the procmail author will go through the code base and do a proper rewrite. But I don't think that's what is in the cards right now. What I expect will happen next is that people will start fuzzing procmail, throw an uncountable number of bug reports at it which will get fixed in a trickle while never fixing the underlying, serious design flaws behind procmail.

Procmail has better alternatives The reason this is so frustrating is that there are plenty of modern alternatives to procmail which do not suffer from those problems. Alternatives to procmail(1) itself are typically part of mail servers. For example, Dovecot has its own LDA which implements the standard Sieve language (RFC 5228). (Interestingly, Sieve was published as RFC 3028 in 2001, before procmail was formally abandoned.) Courier also has "maildrop" which has its own filtering mechanism, and there is fdm (2007) which is a fetchmail and procmail replacement. Update: there's also mailprocessing, which is not an LDA, but processing an existing folder. It was, however, specifically designed to replace complex Procmail rules. But procmail, of course, doesn't just ship procmail; that would just be too easy. It ships mailstat(1) which we could probably ignore because it only parses procmail log files. But more importantly, it also ships:
  • lockfile(1) - conditional semaphore-file creator
  • formail(1) - mail (re)formatter
lockfile(1) already has a somewhat acceptable replacement in the form of flock(1), part of util-linux (which is Essential, so installed on any normal Debian system). It might not be a direct drop-in replacement, but it should be close enough. formail(1) is similar: the courier maildrop package ships reformail(1) which is, presumably, a rewrite of formail. It's unclear if it's a drop-in replacement, but it should probably possible to port uses of formail to it easily.
Update: the maildrop package ships a SUID root binary (two, even). So if you want only reformail(1), you might want to disable that with:
dpkg-statoverride --update --add root root 0755 /usr/bin/lockmail.maildrop 
dpkg-statoverride --update --add root root 0755 /usr/bin/maildrop
It would be perhaps better to have reformail(1) as a separate package, see bug 1006903 for that discussion.
The real challenge is, of course, migrating those old .procmailrc recipes to Sieve (basically). I added a few examples in the appendix below. You might notice the Sieve examples are easier to read, which is a nice added bonus.

Conclusion There is really, absolutely, no reason to keep procmail in Debian, nor should it be used anywhere at this point. It's a great part of our computing history. May it be kept forever in our museums and historical archives, but not in Debian, and certainly not in actual release. It's just a bomb waiting to go off. It is irresponsible for distributions to keep shipping obsolete and insecure software like this for unsuspecting users. Note that I am grateful to the author, I really am: I used procmail for decades and it served me well. But now, it's time to move, not bring it back from the dead.

Appendix

Previous work It's really weird to have to write this blog post. Back in 2016, I rebuilt my mail setup at home and, to my horror, discovered that procmail had been abandoned for 15 years at that point, thanks to that LWN article from 2010. I would have thought that I was the only weirdo still running procmail after all those years and felt kind of embarrassed to only "now" switch to the more modern (and, honestly, awesome) Sieve language. But no. Since then, Debian shipped three major releases (stretch, buster, and bullseye), all with the same vulnerable procmail release. Then, in early 2022, I found that, at work, we actually had procmail installed everywhere, possibly because userdir-ldap was using it for lockfile until 2019. I sent a patch to fix that and scrambled to remove get rid of procmail everywhere. That took about a day. But many other sites are now in that situation, possibly not imagining they have this glaring security hole in their infrastructure.

Procmail to Sieve recipes I'll collect a few Sieve equivalents to procmail recipes here. If you have any additions, do contact me. All Sieve examples below assume you drop the file in ~/.dovecot.sieve.

deliver mail to "plus" extension folder Say you want to deliver user+foo@example.com to the folder foo. You might write something like this in procmail:
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
VERBOSE=off
EXTENSION=$1            # Need to rename it - ?? does not like $1 nor 1
:0
* EXTENSION ?? [a-zA-Z0-9]+
        .$EXTENSION/
That, in sieve language, would be:
require ["variables", "envelope", "fileinto", "subaddress"];
########################################################################
# wildcard +extension
# https://doc.dovecot.org/configuration_manual/sieve/examples/#plus-addressed-mail-filtering
if envelope :matches :detail "to" "*"  
  # Save name in $ name  in all lowercase
  set :lower "name" "$ 1 ";
  fileinto "$ name ";
  stop;
 

Subject into folder This would file all mails with a Subject: line having FreshPorts in it into the freshports folder, and mails from alternc.org mailing lists into the alternc folder:
:0
## mailing list freshports
* ^Subject.*FreshPorts.*
.freshports/
:0
## mailing list alternc
* ^List-Post.*mailto:.*@alternc.org.*
.alternc/
Equivalent Sieve:
if header :contains "subject" "FreshPorts"  
    fileinto "freshports";
  elsif header :contains "List-Id" "alternc.org"  
    fileinto "alternc";
 

Mail sent to root to a reports folder This double rule:
:0
* ^Subject: Cron
* ^From: .*root@
.rapports/
Would look something like this in Sieve:
if header :comparator "i;octet" :contains "Subject" "Cron"  
  if header :regex :comparator "i;octet"  "From" ".*root@"  
        fileinto "rapports";
   
 
Note that this is what the automated converted does (below). It's not very readable, but it works.

Bulk email I didn't have an equivalent of this in procmail, but that's something I did in Sieve:
if header :contains "Precedence" "bulk"  
    fileinto "bulk";
 

Any mailing list This is another rule I didn't have in procmail but I found handy and easy to do in Sieve:
if exists "List-Id"  
    fileinto "lists";
 

This or that I wouldn't remember how to do this in procmail either, but that's an easy one in Sieve:
if anyof (header :contains "from" "example.com",
           header :contains ["to", "cc"] "anarcat@example.com")  
    fileinto "example";
 
You can even pile up a bunch of options together to have one big rule with multiple patterns:
if anyof (exists "X-Cron-Env",
          header :contains ["subject"] ["security run output",
                                        "monthly run output",
                                        "daily run output",
                                        "weekly run output",
                                        "Debian Package Updates",
                                        "Debian package update",
                                        "daily mail stats",
                                        "Anacron job",
                                        "nagios",
                                        "changes report",
                                        "run output",
                                        "[Systraq]",
                                        "Undelivered mail",
                                        "Postfix SMTP server: errors from",
                                        "backupninja",
                                        "DenyHosts report",
                                        "Debian security status",
                                        "apt-listchanges"
                                        ],
           header :contains "Auto-Submitted" "auto-generated",
           envelope :contains "from" ["nagios@",
                                      "logcheck@",
                                      "root@"])
     
    fileinto "rapports";
 

Automated script There is a procmail2sieve.pl script floating around, and mentioned in the dovecot documentation. It didn't work very well for me: I could use it for small things, but I mostly wrote the sieve file from scratch.

Progressive migration Enrico Zini has progressively migrated his procmail setup to Sieve using a clever way: he hooked procmail inside sieve so that he could deliver to the Dovecot LDA and progressively migrate rules one by one, without having a "flag day". See this explanatory blog post for the details, which also shows how to configure Dovecot as an LMTP server with Postfix.

Other examples The Dovecot sieve examples are numerous and also quite useful. At the time of writing, they include virus scanning and spam filtering, vacation auto-replies, includes, archival, and flags.

Harmful considered harmful I am aware that the "considered harmful" title has a long and controversial history, being considered harmful in itself (by some people who are obviously not afraid of contradictions). I have nevertheless deliberately chosen that title, partly to make sure this article gets maximum visibility, but more specifically because I do not have doubts at this moment that procmail is, clearly, a bad idea at this moment in history.

Developing story I must also add that, incredibly, this story has changed while writing it. This article is derived from this bug I filed in Debian to, quite frankly, kick procmail out of Debian. But filing the bug had the interesting effect of pushing the upstream into action: as mentioned above, they have apparently made a new release and merged a bunch of patches in a new git repository. This doesn't change much of the above, at this moment. If anything significant comes out of this effort, I will try to update this article to reflect the situation. I am actually happy to retract the claims in this article if it turns out that procmail is a stellar example of defensive programming and survives fuzzing attacks. But at this moment, I'm pretty confident that will not happen, at least not in scope of the next Debian release cycle.

Keith Packard: picolibc-testing

Testing Picolibc with the glibc tests Picolibc has a bunch of built-in tests, but more testing is always better, right? I decided to see how hard it would be to run some of the tests provided in the GNU C Library (glibc). Parallel meson build files Similar to how Picolibc uses meson build files to avoid modifying the newlib autotools infrastructure, I decided to take the glibc code and write meson build rules that would compile the tests against Picolibc header files and link against Picolibc libraries. I decided to select a single target for this project so I could focus on getting things building and not worry too much about making it portable. I wanted to pick something that had hardware floating point so that I would have rounding modes and exception support, so I picked the ARM Cortex M7 with hard float ABI:
$ arm-none-eabi-gcc -mcpu=cortex-m7 -mfloat-abi=hard
It should be fairly easy to extend this to other targets, but for now, that's what I've got working. There's a cross-cortex-m7.txt file containing all of the cross compilation details which is used when running meson setup. All of the Picolibc-specific files live in a new picolibc directory so they are isolated from the main glibc code. Pre-loading a pile of hacks Adapt Picolibc to support the Glibc test code required a bunch of random hacks, from providing _unlocked versions of the stdio macros to stubbing out various unsupportable syscalls (like sleep and chdir). Instead of modifying the Glibc code, I created a file called hacks.h which is full of this stuff and used the gcc -include parameter to read that into the compiler before starting compilation on all of the files. Supporting command line parameters The glibc tests all support a range of command line parameters, some of which turned out to be quite useful for this work. Picolibc had limited semihosting support for accessing the command line, but that required modifying applications to go fetch the command line using a special semihosting function. To make this easier, I added a new crt0 variant for picolibc called (oddly) semihost. This extends the existing hosted variant by adding a call to the semihosting library to fetch the current command line and splitting that into words at each space. It doesn't handle any quoting, but it's sufficient for the needs here. Avoiding glibc headers The glibc tests use some glibc-specific extensions to the standard POSIX C library, so I needed to include those in the test builds. Headers for those extensions are mixed in with the rest of the POSIX standard headers, which conflict with the Picolibc versions. To work around this, I stuck stub #include files in the picolibc directory which directly include the appropriate headers for the glibc API extensions. This includes things like argp.h and array_length.h. For other headers which weren't actually needed for picolibc, I created empty files. Adding more POSIX to Picolibc At this point, code was compiling but failing to find various standard POSIX functions which aren't available in Picolibc. That included some syscalls which could be emulated using semihosting, like gettimeofday and getpagesize. It also included some generally useful additions, like replacing ecvtbuf and fcvtbuf with ecvt_r and fcvt_r. The _r variants provide a target buffer size instead of assuming that it was large enough as the Picolibc buf variants did. Which tests are working? So far, I've got some of the tests in malloc, math, misc and stdio-common running. There are a lot of tests in the malloc directory which cover glibc API extensions or require POSIX syscalls not supported by semihosting. I think I've added all of the tests which should be supported. For the math tests, I'm testing the standard POSIX math APIs in both float and double forms, except for the Bessel and Gamma functions. Picolibc's versions of those are so bad that they violate some pretty basic assumptions about error bounds built into the glibc test code. Until Picolibc gets better versions of these functions, we'll have to skip testing them this way. In the misc directory, I've only added tests for ecvt, fcvt, gcvt, dirname and hsearch. I don't think there are any other tests there which should work. Finally, for stdio-common, almost all of the tests expect a fully functioning file system, which semihosting really can't support. As a result, we're only able to run the printf, scanf and some of the sprintf tests. All in all, we're running 78 of the glibc test programs, which is a small fraction of the total tests, but I think it's the bulk of the tests which cover APIs that don't depend too much on the underlying POSIX operating system. Bugs found and fixed in Picolibc This exercise has resulted in 17 fixes in Picolibc, which can be classified as:
  1. Math functions taking sNaN and not raising FE_INVALID and returning qNaN. Almost any operation on an sNaN value is supposed to signal an invalid operand and replace that with a qNaN so that further processing doesn't raise another exception. This was fairly easy to fix, just need to use return x + x; instead of return x;.
  2. Math functions failing to set errno. I'd recently restructured the math library to get rid of the separate IEEE version of the functions which didn't set errno and missed a couple of cases that needed to use the errno-setting helper functions.
  3. Corner cases in string/float conversion, including the need to perform round-to-even for '%a' conversions in printf and supporting negative decimal values for fcvt. This particular exercise led to replacing the ecvtbuf and fcvtbuf APIs with glibc's ecvt_r and fcvt_r versions as those pass explicit buffer lengths, making overflow prevention far more reliable.
  4. A bunch of malloc entry points were not handling failure correctly; allocation values close to the maximum possible size caused numerous numeric wrapping errors with the usual consequences (allocations "succeed", but return a far smaller buffer than requested). Also, realloc was failing to check the return value from malloc before copying the old data, which really isn't a good idea.
  5. Tinystdio's POSIX support code was returning an error instead of EOF at end of file.
  6. Error bounds for the Picolibc math library aren't great; I had to generate Picolibc-specific ulps files. Most functions are between 0 and 3 ulps, but for some reason, the float version of erfc (ercf) has errors as large as 64 ulps. That needs investigation.
Tests added to Picolibc With all of the fixes applied to Picolibc, I added some tests to verify them without relying on running the glibc tests, that includes sNaN vs qNaN tests for math functions, testing fopen and mktemp, checking the printf %a rounding support and adding a ecvt/fcvt tests. I also discovered that the github-based CI system was compiling but not testing when using clang with a riscv target, so that got added in. Where's the source code? The Picolibc changes are sitting on the glibc-testing branch. I'll merge them once the current CI pass finishes. The hacked-up Glibc bits are in a glibc mirror at github in the picolibc project on the picolibc-testing branch. It would be interesting to know what additional tests should be usable in this environment. And, perhaps, finding a way to use this for picolibc CI testing in the future. Concluding thoughts Overall, I'm pretty pleased with these results. The bugs in malloc are fairly serious and could easily result in trouble, but the remaining issues are mostly minor and shouldn't have a big impact on applications using Picolibc. I'll get the changes merged and start thinking about doing another Picolibc release.

26 February 2022

Thomas Koch: Corona Plandemic

Posted on February 26, 2022
Tags: debian, life, peace
This is a short statement on what I understand about Covid-19 and the current situation so that I can refer to it in other posts and don t need to repeat myself. What I do, think and eventually write is heavily influenced by this understanding, as outlined futher below. Also I want to set a counterpoint to the people that still publicly proclaim the official narrative. On November 15th 2020 I already wrote an email to debian-private@ with the subject Basic information about Corona in German that referred to my information collection on that topic. Since then my understanding has refined but not fundamentally changed: I m happy to talk in detail about this topic and provide pointers. Right now a comprehensive overview is available at Grand-Jury.net. Why is this relevant? The internet in its current state is an instrument of control and suppression. This becomes especially obvious in the increase in censorship and the introduction of vaccine-passports which are planned to become universal passports for all aspects of our life. Therefor I want to work on initiatives for a free internet for the people. Right now I think about the freedombox, distributed search engines, re-decentralization in general and especially a decentralized alternative to Wikipedia.

22 February 2022

Russ Allbery: Review: Children of Earth and Sky

Review: Children of Earth and Sky, by Guy Gavriel Kay
Publisher: New American Library
Copyright: 2016
ISBN: 0-698-18327-4
Format: Kindle
Pages: 572
Nine hundred years have passed since the events of Lord of Emperors. Twenty-five years ago, Sarantium, queen of cities, fell to the Osmanlis, who have renamed it Asharias in honor of their Asherite faith. The repercussions are still echoing through the western world, as the Osmanlis attempt each spring to push farther west and the forces of Rodolfo, Holy Emperor in Obravic and defender of the Jaddite faith, hold them back. Seressa and Dubrava are city-state republics built on the sea trade. Seressa is the larger and most renown, money-lenders to Rodolfo and notorious for their focus on business and profit, including willingness to trade with the Osmanlis. Dubrava has a more tenuous position: smaller, reliant on trade and other assistance from Seressa, but also holding a more-favored trading position with Asharias. Both are harassed by piracy from Senjan, a fiercely Jaddite raiding city north up the coast from Dubrava and renown for its bravery against the Asherites. The Senjani are bad for business. Seressa would love to wipe them out, but they have the favor of the Holy Emperor. They settled for attempting to starve the city with a blockade. As Children of Earth and Sky opens, Seressa is sending out new spies. One is a woman named Leonora Valeri, who will present herself as the wife of a doctor that Seressa is sending to Dubrava. She is neither his wife nor Seressani, but this assignment gets her out of the convent to which her noble father exiled her after an unapproved love affair. The other new spy is the young artist Pero Villani, a minor painter whose only notable work was destroyed by the woman who commissioned it for being too revealing. Pero's destination is farther east: Grand Khalif Gur u the Destroyer, the man whose forces took Sarantium, wants to be painted in the western style. Pero will do so, and observe all he can, and if the opportunity arises to do more than that, well, so much the better. Pero and Leonora are traveling on a ship owned by Marin Djivo, the younger son of a wealthy Dubravan merchant family, when their ship is captured by Senjani raiders. Among the raiders is Danica Gradek, the archer who broke the Seressani blockade of Senjan. This sort of piracy, while tense, should be an economic transaction: some theft, some bargaining, some ransom, and everyone goes on their way. That is not what happens. Moments later, two men lie dead, and Danica's life has become entangled with Dubravan merchants and Seressani spies. Children of Earth and Sky is in some sense a sequel to the Sarantine Mosaic, and knowing the events of that series adds some emotional depth and significant moments to this story, but you can easily read it as a stand-alone novel. (That said, I recommend the Sarantine Mosaic regardless.) As with nearly all of Kay's work, it's historical fiction with the names changed (less this time than in most of this books) and a bit of magic added. The setting is the middle of the 15th century. Seressa is, of course, Venice. The Osmanlis are the Ottoman Turks, and Asharias is Istanbul, the captured Constantinople. Rodolfo is a Habsburg Holy Roman Emperor, holding court in an amalgam of northern cities that (per the afterward) is primarily Prague. Dubrava, which is central to much of this book, is Dubrovnik in Croatia. As usual with Kay's novels, you don't need to know this to enjoy the story, but it may spark some fun secondary reading. The touch of magic is present in several places, but comes primarily from Danica, whose grandfather resides as a voice in her head. He is the last of her family that she is in contact with. Her father and older brother were killed by Osmanli raiders, and her younger brother taken as a slave to be raised as a djanni warrior in the khalif's infantry. (Djannis are akin to Mamluks in our world.) Damaz, as he is now known, is the remaining major viewpoint character I've not mentioned. There are a couple of key events in the book that have magic at the center, generally involving Danica or Damaz, but most of the story is straight historical fiction (albeit with significant divergences from our world). I'd talked myself out of starting this novel several times before I finally picked it up. Like most of Kay's, it's a long book, and I wasn't sure if I was in the mood for epic narration and a huge cast. And indeed, I found it slow at the start. Once the story got underway, though, I was as enthralled as always. There is a bit of sag in the middle of the book, in part because Kay didn't follow up on some relationships that I wish were more central to the plot and in part because he overdoes the narrative weight in one scene, but the ending is exceptional. Guy Gavriel Kay is the master of a specific type of omniscient tight third person narration, one in which the reader sees what a character is thinking but also gets narrative commentary, foreshadowing, and emotional emphasis apart from the character's thoughts. It can feel heavy-handed; if something is important, Kay tells you, explicitly and sometimes repetitively, and the foreshadowing frequently can be described as portentous. But in return, Kay gets fine control of pacing and emphasis. The narrative commentary functions like a soundtrack in a movie. It tells you when to pay close attention and when you can relax, what moments are important, where to slow down, when to brace yourself, and when you can speed up. That in turn requires trust; if you're not in the mood for the author to dictate your reading pace to the degree Kay is attempting, it can be irritating. If you are in the mood, though, it makes his novels easy to relax into. The narrator will ensure that you don't miss anything important, and it's an effective way to build tension. Kay also strikes just the right balance between showing multiple perspectives on a single moment and spending too much time retelling the same story. He will often switch viewpoint characters in the middle of a scene, but he avoids the trap of replaying the scene and thus losing the reader's interest. There is instead just a moment of doubled perspective or retrospective commentary, just enough information for the reader to extrapolate the other character's experience backwards, and then the story moves on. Kay has an excellent feel for when I badly wanted to see another character's perspective on something that just happened. Some of Kay's novels revolve around a specific event or person. Children of Earth and Sky is not one of those. It's a braided novel following five main characters, each with their own story. Some of those stories converge; some of them touch for a while and then diverge again. About three-quarters of the way through, I wasn't sure how Kay would manage a satisfying conclusion for the numerous separate threads that didn't feel rushed, but I need not have worried. The ending had very little of the shape that I had expected, focused more on the small than the large (although there are some world-changing events here), but it was an absolute delight, with some beautiful moments of happiness that took the rest of the novel to set up. This is not the sort of novel with a clear theme, but insofar as it has one, it's a story about how much of the future shape and events of the world are unknowable. All we can control is our own choices, and we may never know their impact. Each individual must decide who they want to be and attempt to live their life in accordance with that decision, hopefully with some grace towards others in the world. The novel does, alas, still have some of Kay's standard weaknesses. There is (at last!) an important female friendship, and I had great hopes for a second one, but sadly it lasted only a scant handful of pages. Men interact with each other and with women; women interact almost exclusively with men. Kay does slightly less awarding of women to male characters than in some previous books (although it still happens), but this world is still weirdly obsessed with handing women to men for sex as a hospitality gesture. None of this is too belabored or central to the story, or I would be complaining more, but as soon as one sees how regressive the gender roles typically are in a Kay novel, it's hard to unsee. And, as always for Kay, the sex in this book is weirdly off-putting to me. I think this goes hand in hand with Kay's ability to write some of the best conversations in fantasy. Kay's characters spar and thrust with every line and read nuance into small details of wording. Frequently, the turn of the story rests on the outcome of a careful conversation. This is great reading; it's the part of Kay's writing I enjoy the most. But I'm not sure he knows how to turn it off between characters who love and trust each other. The characters never fully relax; sex feels like another move in ongoing chess games, which in turn makes it feel weirdly transactional or manipulative instead of open-hearted and intimate. It doesn't help that Kay appears to believe that arousal is a far more irresistible force for men than I do. Those problems did get in the way of my enjoyment occasionally, but I didn't think they ruined the book. The rest of the story is too good. Danica in particular is a wonderful character: thoughtful, brave, determined, and deeply honest with herself in that way that is typical of the best of Kay's characters. I wanted to read the book where Danica's and Leonora's stories stayed more entwined; alas, that's not the story Kay was writing. But I am in awe at Kay's ability to write characters who feel thoughtful and insightful even when working at cross purposes, in a world that mostly avoids simple villains, with a plot that never hinges on someone doing something stupid. I love reading about these people. Their triumphs, when they finally come, are deeply satisfying. Children of Earth and Sky is probably not in the top echelon of Kay's works with the Sarantine Mosaic and Under Heaven, but it's close. If you like his other writing, you will like this as well. Highly recommended. Rating: 9 out of 10

16 February 2022

Robert McQueen: Forward the Foundation

Earlier this week, Neil McGovern announced that he is due to be stepping down as the Executive Director as the GNOME Foundation later this year. As the President of the board and Neil s effective manager together with the Executive Committee, I wanted to take a moment to reflect on his achievements in the past 5 years and explain a little about what the next steps would be. Since joining in 2017, Neil has overseen a productive period of growth and maturity for the Foundation, increasing our influence both within the GNOME project and the wider Free and Open Source Software community. Here s a few highlights of what he s achieved together with the Foundation team and the community: Recognizing and appreciating the amazing progress that GNOME has made with Neil s support, the search for a new Executive Director provides the opportunity for the Foundation board to set the agenda and next high-level goals we d like to achieve together with our new Executive Director. In terms of the desktop, applications, technology, design and development processes, whilst there are always improvements to be made, the board s general feeling is that thanks to the work of our amazing community of contributors, GNOME is doing very well in terms of what we produce and publish. Recent desktop releases have looked great, highly polished and well-received, and the application ecosystem is growing and improving through new developers and applications bringing great energy at the moment. From here, our largest opportunity in terms of growing the community and our user base is being able to articulate the benefits of what we ve produced to a wider public audience, and deliver impact which allows us to secure and grow new and sustainable sources of funding. For individuals, we are able to offer an exceedingly high quality desktop experience and a broad range of powerful applications which are affordable to all, backed by a nonprofit which can be trusted to look after your data, digital security and your best interests as an individual. From the perspective of being a public charity in the US, we also have the opportunity to establish programs that draw upon our community, technology and products to deliver impact such as developing employable skills, incubating new Open Source contributors, learning to program and more. For our next Executive Director, we will be looking for an individual with existing experience in that nonprofit landscape, ideally with prior experience establishing and raising funds for programs that deliver impact through technology, and appreciation for the values that bring people to Free, Open Source and other Open Culture organizations. Working closely with the existing members, contributors, volunteers and whole GNOME community, and managing our relationships with the Advisory Board and other key partners, we hope to find a candidate that can build public awareness and help people learn about, use and benefit from what GNOME has built over the past two decades. Neil has agreed to stay in his position for a 6 month transition period, during which he will support the board in our search for a new Executive Director and support a smooth hand-over. Over the coming weeks we will publish the job description for the new ED, and establish a search committee who will be responsible for sourcing and interviewing candidates to make a recommendation to the board for Neil s successor a hard act to follow! I m confident the community will join me and the board in personally thanking Neil for his 5 years of dedicated service in support of GNOME and the Foundation. Should you have any queries regarding the process, or offers of assistance in the coming hiring process, please don t hesitate to join the discussion or reach out directly to the board.

12 February 2022

Ritesh Raj Sarraf: apt-offline 1.8.4

apt-offline 1.8.4 apt-offline version 1.8.4 has been released. This release includes many bug fixes but the important ones are:
  • Better GPG signature handling
  • Support for verifying InRelease files

Changelog
apt-offline (1.8.4-1) unstable; urgency=medium
  [ Debian Janitor ]
  * Update standards version to 4.5.0, no changes needed.
  [ Paul Wise ]
  * Clarify file type in unknown file message
  * Fix typos
  * Remove trailing whitespace
  * Update LICENSE file to match official GNU version
  * Complain when there are no valid keyrings instead of missing keyrings
  * Make all syncrhronised files world readable
  * Fix usage of indefinite articles
  * Only show the APT Offline GUI once in the menu
  * Update out of date URLs
  * Fix date and whitespace issues in the manual page
  * Replace stereotyping with an appropriate word
  * Switch more Python shebangs to Python 3
  * Correct usage of the /tmp/ directory
  * Fix YAML files
  * Fix usage of the log API
  * Make the copying of changelog lines less brittle
  * Do not split keyring paths on whitespace
  [ Ritesh Raj Sarraf ]
  * Drop the redundant import of the apt module.
    Thanks to github/dandelionred
  * Fix deprecation of get_bugs() in debianbts
  * Drop the unused IgnoredBugTypes
  * Set encoding for files when opening
  * Better error logging when apt fails
  * Don't mandate a default option
  * Demote metadata errors to verbose
  * Also log an error message for every failed .deb url
  * Check hard for the url type
  * Check for ascii armored signature files.
    Thanks to David Klnischkies
  * Add MIME type for InRelease files
  * Drop patch 0001-Drop-the-redundant-import-of-the-apt-module.patch.
    Now part of the 1.8.4 release
  * Prepare release 1.8.3
  * Prepare release 1.8.4
  * debian packaging
    + Bump debhelper compatibility to 13
    + Update install files
  [ Dean Anderson ]
  * [#143] Added support for verifying InRelease files
 -- Ritesh Raj Sarraf <rrs@debian.org>  Sat, 12 Feb 2022 18:52:58 +0530

Resources
  • Tarball and Zip archive for apt-offline are available here
  • Packages should be available in Debian.
  • Development for apt-offline is currently hosted here

11 February 2022

Ingo Juergensmann: Old Buildd.Net Database

Since March/April 2000 I was deeply involved in Debian m68k and operated multiple m68k autobuilder for over a decade. In fact my Amiga 3000 named arrakis was the second buildd for m68k in addition to the Debian owned Amiga 3000UX named kullervo . Back in that time there was some small website running on Kullervo to display some information about the Debian autobuilder. After some time we (as m68k porters) moved that webpage away from Kullervo to my root server. Step by step this site evolved to Buildd.Net and extended to other archs and suites beside unstable like backports or non-volatile. The project got more and more complex and beyond my ability to do a complete necessary rewrite. So, in 2016 I asked for adoption of the project and in 2018 I shut it down, because (apparently) there was nobody taking over. From November 2005 until January 2018 I do have entries in my PostgreSQL database for Buildd.Net. I think the data in the database might be interesting for those that want to examine that data. You can use the data to see how build times have increased over time, which e.g. led to the expulsion of m68k as release arch, because the arch couldn t keep up anymore. I could imagine that you could do other interesting analysis with that data. For example how new versions of the toolchain increased the build times, maybe even if a specific version of e.g. binutils or gcc had a positive effect on certain archs, but a negative effect on other archs. If there is interest in this data I could open the database to the public or even upload the dump of the database so that you can download and install it on your own.

7 February 2022

Gustavo R. Montesino: Hardware failures...

The Bad: When I got to work today, I've found our server (white box with Debian etch responsible for networking, files, printers, etc) powered off - most likely the UPS' battery didn't survive a power outage during the night. When I turned the server on I was greeted by all those nice lines telling me I had a hard disk problem. The Ugly: Instead off marking the disk showing the read errors as bad, the RAID stack (device mapper?) somehow concluded the "good" disk of the RAID 1 array wasn't synched and kicked it out... The good The bad sectors did take only some unimportant collectd status files with them. After some poking with dd trying to force the HD to redirect the bad sectors, the read errors vanished and the Reallocated Sector Count didn't increase according to smartctl, which seems like a good signal.

Russ Allbery: Review: Embers of War

Review: Embers of War, by Gareth L. Powell
Series: Embers of War #1
Publisher: Titan Books
Copyright: February 2018
ISBN: 1-78565-519-1
Format: Kindle
Pages: 312
The military leadership of the Outward faction of humanity was meeting on the forest world of Pelapatarn, creating an opportunity for the Conglomeration to win the war at a stroke. Resistance was supposed to be minimal, since the Outward had attempted to keep the conference secret rather than massing forces to protect it. But the Outward resistance was stronger than expected, and Captain Deal's forces would not be able to locate and assassinate the Outward leadership before they could escape. She therefore followed orders from above her and ordered the four incoming Carnivore heavy cruisers to jump past the space battle and bomb the planet. The entire planet. The Carnivores' nuclear and antimatter weaponry reduced the billion-year-old sentient jungle of Pelapatarn to ash. Three years later, Sal Konstanz is a ship captain for the House of Reclamation, a strictly neutral search and rescue force modeled after a long-vanished alien fleet that prioritizes preservation of life above all else. Anyone can join Reclamation, provided that they renounce their previous alliances and devote themselves to the Reclamation cause. Sal and her crew member Alva Clay were Outward. The ship medic was Conglomeration. So was the ship: the Trouble Dog, a sentient AI heavy cruiser built to carry an arsenal of weapons and three hundred crew, and now carrying three humans and a Druff mechanic. More precisely, the Trouble Dog was one of the four Carnivores that destroyed Pelapatarn. Meanwhile, Ona Sudak, a popular war poet, is a passenger on a luxury cruise on the liner Geest van Amsterdam, which is making an unscheduled stop in the star system known as the Gallery. The Gallery is the home of the Objects: seven planets that, ten thousand years earlier, were carved by unknown aliens into immense sculptures for unknown reasons. The Objects appear to be both harmless and mysterious, making them an irresistible tourist attraction for the liner passengers. The Gallery is in disputed space, but no one was expecting serious trouble. They certainly weren't expecting the Geest van Amsterdam to be attacked and brought down on the Object known as the Brain, killing nearly everyone aboard. The Trouble Dog is the closest rescue ship. This book was... fine. It's a perfectly serviceable science fiction novel that didn't stand out for me, which I think says more about the current excellent state of the science fiction field than about this book. When I was a teenager reading Asimov, Niven, and Heinlein, I would have devoured this. It compares favorably to minor Niven or Heinlein (The Integral Trees, for example, or Double Star), but the bar for excellent science fiction is just so much higher now. The best character in this book (and the reason why I read it) is the Trouble Dog. I love science fiction about intelligent space ships, and she did not disappoint. The AI ships in this book are partly made from human and dog neurons, so their viewpoint is mostly human but with some interesting minor variations. And the Trouble Dog would be a great character even if she weren't an intelligent ship: ethical, aggressive, daring, and introspective, with a nuanced relationship with her human crew. Unfortunately, Embers of War has four other viewpoint characters, and Powell chose to write them all in the first person. First person narration depends heavily on a memorable and interesting character because the reader is so thoroughly within their perspective. This works great for the Trouble Dog, and is fine for Nod, the Druff who serves as the ship mechanic. (Nod's perspective is intriguing, short, almost free-verse musings, rather than major story segments.) Sal, the ship captain, is a bit of a default character, but I didn't mind her much. Neither of the other two viewpoint characters are interesting enough to warrant the narrative attention. The Conglomerate agent Ashton Childe has such an uninteresting internal monologue that I would have liked the character better if he'd only been seen through other people's viewpoints, and although Powell needs some way to show Ona Sudak's view of events, I didn't think her thoughts added much to the story. The writing is adequate but a bit clunky: slightly flat descriptions, a bit too predictable at the sentence level, and rarely that memorable. There is a bit of fun world-building of the ancient artifact variety and a couple of decent set pieces (and one rather-too-obvious Matrix homage that I didn't think was as effective as the author did), but most of the story is focused on characters navigating their lives and processing trauma from the war. The story kept me turning the pages with interest, but I also doubt it will surprise anyone who has read much science fiction. I suspect a lot of it is setup for the following two books of the trilogy, and there are plenty of hooks for more stories in this universe. I really wanted a first-person story from the perspective of the Trouble Dog, possibly with some tight third-person interludes showing Ona Sudak's story. What I got instead was entertaining but not memorable enough to stand out in the current rich state of science fiction. I think I'm invested enough in this story to want to read the next book, though, so that's still a recommendation of sorts. Followed by Fleet of Knives. Rating: 6 out of 10

5 February 2022

Louis-Philippe V ronneau: Migrating from ledger to hledger

I first started using ledger the original plain-text accounting software in 2017. Since then, I had been pretty happy with my accounting routine, but grew a little annoyed by the repetitive manual work I had to do to assign recurring transactions to the right account. To make things easier, I had a collection of bash scripts to parse and convert the CSV files from my bank's website1 into ledger entries. They were of course ugly, unreadable piles of sed-grep-regex and did not let met achieve the automation complexity I longed for. Dissatisfied with ledger's features, I decided to migrate to hledger. Contrary to ledger, hledger comes with a CSV parser one can use to import and classify recurring transactions automagically. Having a proper DSL for this makes all the difference: I can easily add new rules and understand the old ones. In the end, I get a very consistent result, something my old bash scripts weren't great at. Here is what my debit.csv.rules file looks like. It is used to tell hledger how CSV data from my debit card should be parsed:
# skip the headings line:
skip 1
# assign fields
fields , , , date, , description, , amount2-in, amount2-out
# assign account where the money comes from
# 99 to get it at the bottom of the transaction
account99 assets:checking
include matchers.rules
As you can see, the rules matching transactions to specific accounts are imported from another file, matchers.rules. I'm doing this since I want to be able to use the same set of rules for debit and credit and use the cards interchangeably. Here's a snippet of my matchers.rules file:
# house
if Loyer/bail
  account2  expenses:rent
# leisure
if PAYPAL .*STEAM GAMES
  account2  expenses:leisure:videogame
if PAYPAL .*BANDCAMP
  account2  expenses:leisure:music
Using this ruleset, a transaction looking like this:
"SOME ACCOUNT DESC","111111","EOP","2022/01/03",00002,"Loyer/bail","",521.00,"","","","",""
Would be transformed into:
2022-01-03 Loyer/bail
    expenses:rent                               521.00
    assets:checking
Sadly, hledger's CSV rules won't let you do arithmetics. This can be useful when you know a certain transaction needs to be split between accounts. This is where auto postings come in. They are a way to specify arbitrary rules when an account is encountered. Going back to my previous rent example, I split it 50/50 with my SO using this rule:
= expenses:rent
    assets:receivable:rent    *0.5
    assets:checking
After it is applied, the final transaction looks like this:
2022-01-03 Loyer/bail  ; modified:
    expenses:rent                               521.00
    assets:receivable:rent                      260.50  ; generated-posting: = expenses:rent
    assets:checking
Neat eh? Here is the little bash script I've written to automate all these steps:
 #!/bin/bash
#
# Convert CSV to ledger using hledger
declare -a assets=("credit" "debit")
# Read the array values with space
for i in "$ assets[@] "
do
  if test -f "$i.csv"
  then
    # convert to ledger
    LEDGER=$(hledger -f "$i.csv" --rules-file rules/"$i.csv".rules print)
    # add auto_postings
    LEDGER=$(printf "include rules/auto_postings.ledger\n\n$LEDGER\n"   hledger print -f- --auto)
    # remove superfluous assets:checking lines
    LEDGER=$(printf "$LEDGER"   sed '/assets:checking \+; generated.\+/d')
    printf "$LEDGER" > "$i.ledger"
  else
    printf "File $i.csv does not exist\n"
  fi
done
Migrating to hledger, I've cut down the time I spend on accounting from 1 hour per month to about 10 minutes, all while making my workflow much cleaner. Many thanks to the kind folks on #hledger @irc.libera.chat for the help!

  1. One for my checking+savings account and another one for my credit card.

31 January 2022

Russ Allbery: Review: The Story of the Treasure Seekers

Review: The Story of the Treasure Seekers, by E. Nesbit
Publisher: Amazon
Copyright: 1899
Printing: May 2012
ASIN: B0082ZBXSI
Format: Kindle
Pages: 136
The Story of the Treasure Seekers was originally published in 1899 and is no longer covered by copyright. I read the free Amazon Kindle version because it was convenient. My guess is that Amazon is republishing the Project Gutenberg version, but they only credit "a community of volunteers." There are six Bastable children: Dora, Oswald, Dicky, the twins Alice and Noel, and Horace Octavius (H.O.), the youngest. Their mother is dead and the family's finances have suffered in the wake of her death (or, as the first-person narrator puts it, "the fortunes of the ancient House of Bastable were really fallen"), which means that their father works long hours and is very absorbed with his business. That leaves the six kids largely to fend for themselves, since they can't afford school. Clearly the solution is to find treasure. This is a fix-up novel constructed from short stories that were originally published in various periodicals, reordered and occasionally rewritten for the collected publication. To be honest, calling it a fix-up novel is generous; there are some references to previous events, but the first fourteen chapters can mostly stand alone. The last two chapters are closely related and provide an ending. More on that in a moment. What grabs the reader's attention from the first paragraph is the writing style:
This is the story of the different ways we looked for treasure, and I think when you have read it you will see that we were not lazy about the looking. There are some things I must tell before I begin to tell about the treasure-seeking, because I have read books myself, and I know how beastly it is when a story begins, "Alas!" said Hildegarde with a deep sigh, "we must look our last on this ancestral home" and then some one else says something and you don't know for pages and pages where the home is, or who Hildegarde is, or anything about it.
The first-person narrator of The Story of the Treasure Seekers is one of the six kids.
It is one of us that tells this story but I shall not tell you which: only at the very end perhaps I will.
The narrator then goes on to elaborately praise one of the kids, occasionally accidentally uses "I" instead of their name, and then remembers and tries to hide who is telling the story again. It's beautifully done and had me snickering throughout the book. It's not much of a mystery (you will figure out who is telling the story very quickly), but Nesbit captures the writing style of a kid astonishingly well without making the story poorly written. Descriptions of events have a headlong style that captures a child's sense of adventure and heedless immortality mixed with quiet observations that remind the reader that kids don't miss as much as people think they do. I think the most skillful part of this book is the way Nesbit captures a kid's disregard of literary convention. The narrator in a book written by an adult tends to fit into a standard choice of story-telling style and follow it consistently. Even first-person narrators who break some of those rules feel like intentionally constructed characters. The Story of the Treasure Seekers is instead half "kid telling a story" and half "kid trying to emulate the way stories are told in books" and tends to veer wildly between the two when the narrator gets excited, as if they're vaguely aware of the conventions they're supposed to be following but are murky on the specifics. It feels exactly like the sort of book a smart and well-read kid would write (with extensive help from an editor). The other thing that Nesbit handles exceptionally well is the dynamic between the six kids. This is a collection of fairly short stories, so there isn't a lot of room for characterization. The kids are mostly sketched out with one or two memorable quirks. But Nesbit puts a lot of effort into the dynamics that arise between the children in a tight-knit family, properly making the group of kids as a whole and in various combinations a sort of character in their own right. Never for a moment does either the reader or the kids forget that they have siblings. Most adventures involve some process of sorting out who is going to come along and who is going to do other things, and there's a constant but unobtrusive background rhythm of bickering, making up, supporting each other, being frustrated by each other, and getting exasperated at each other's quirks. It's one of the better-written sibling dynamics that I've read. I somehow managed to miss Nesbit entirely as a kid, probably because she didn't write long series and child me was strongly biased towards books that were part of long series. (One book was at most a pleasant few hours; there needed to be a whole series attached to get any reasonable amount of reading out of the world.) This was nonetheless a fun bit of nostalgia because it was so much like the books I did read: kids finding adventures and making things up, getting into various trouble but getting out of it by being honest and kind, and only occasional and spotty adult supervision. Reading as an adult, I can see the touches of melancholy of loss that Nesbit embeds into this quest for riches, but part of the appeal of the stories is that the kids determinedly refuse to talk about it except as a problem to be solved. Nesbit was a rather famous progressive, but this is still a book of its time, which means there's one instance of the n-word and the kids have grown up playing the very racist version of cowboys and indians. The narrator also does a lot of stereotyping of boys and girls, although Nesbit undermines that a bit by making Alice a tomboy. I found all of this easier to ignore because the story is narrated by one of the kids who doesn't know any better, but your mileage may vary. I am always entertained by how anyone worth writing about in a British children's novel of this era has servants. You know the Bastables have fallen upon hard times because they only have one servant. The kids don't have much respect for Eliza, which I found a bit off-putting, and I wondered what this world looks like from her perspective. She clearly did a lot of the work of raising these motherless kids, but the kids view her as the hired help or an obstacle to be avoided, and there's not a lot of gratitude present. As the stories unfold, it becomes more and more clear that there's a quiet conspiracy of surrounding adults to watch out for these kids, which the kids never notice. This says good things about society, but it does undermine the adventures a little, and by the end of the book the sameness of the stories was wearing a bit thin. The high point of the book is probably chapter eight, in which the kids make their own newspaper, the entirety of which is reproduced in the book and is a note-perfect recreation of what an enterprising group of kids would come up with. In the last two stories, Nesbit tacks on an ending that was probably obligatory, but which I thought undermined some of the emotional subtext of the rest of the book. I'm not sure how else one could have put an ending on this book, but the ending she chose emphasized the degree to which the adventures really were just play, and the kids are rewarded in these stories for their ethics and their circumstances rather than for anything they concretely do. It's a bit unsatisfying. This is mostly a nostalgia read, but I'm glad I read it. If this book was not part of your childhood, it's worth reading if only for how well Nesbit captures a child's narrative voice. Rating: 7 out of 10

22 January 2022

Steve Kemp: Visiting the UK was difficult, but worth it

So in my previous post I mentioned that we were going to spend the Christmas period in the UK, which we did. We spent a couple of days there, meeting my parents, and family. We also persuaded my sister to drive us to Scarborough so that we could hang out on the beach for an afternoon. Finland has lots of lakes, but it doesn't have proper waves. So it was surprisingly good just to wade in the sea and see waves! Unfortunately our child was a wee bit too scared to ride on a donkey! Unfortunately upon our return to Finland we all tested positive for COVID-19, me first, then the child, and about three days later my wife. We had negative tests in advance of our flights home, so we figure that either the tests were broken, or we were infected in the airplane/airport. Thankfully things weren't too bad, we stayed indoors for the appropriate length of time, and a combination of a couple of neighbours and online shopping meant we didn't run out of food. Since I've been back home I've been automating AWS activities with aws-utils, and updating my simple host-automation system, marionette. Marionette is something that was inspired by puppet, the configuration management utility, but it runs upon localhost only. Despite the small number of integrated primitives it actually works surprisingly well, and although I don't expect it will ever become popular it was an interesting research project. The aws-utilities? They were specifically put together because I've worked in a few places where infrastructure is setup with terraform, or cloudformation, but there are always the odd thing that is configured manually. Typically we'll have an openvpn gateway which uses a manually maintained IP allow-list, or some admin-server which has a security-group maintained somewhat manually. Having the ability to update a bunch of rules with your external IP, as a single command, across a number of AWS accounts/roles, and a number of security-groups is an enormous time-saver when your home IP changes. I'd quite like to add more things to that collection, but there's no particular rush.

16 January 2022

Chris Lamb: Favourite films of 2021

In my four most recent posts, I went over the memoirs and biographies, the non-fiction, the fiction and the 'classic' novels that I enjoyed reading the most in 2021. But in the very last of my 2021 roundup posts, I'll be going over some of my favourite movies. (Saying that, these are perhaps less of my 'favourite films' than the ones worth remarking on after all, nobody needs to hear that The Godfather is a good movie.) It's probably helpful to remark you that I took a self-directed course in film history in 2021, based around the first volume of Roger Ebert's The Great Movies. This collection of 100-odd movie essays aims to make a tour of the landmarks of the first century of cinema, and I watched all but a handul before the year was out. I am slowly making my way through volume two in 2022. This tome was tremendously useful, and not simply due to the background context that Ebert added to each film: it also brought me into contact with films I would have hardly come through some other means. Would I have ever discovered the sly comedy of Trouble in Paradise (1932) or the touching proto-realism of L'Atalante (1934) any other way? It also helped me to 'get around' to watching films I may have put off watching forever the influential Battleship Potemkin (1925), for instance, and the ur-epic Lawrence of Arabia (1962) spring to mind here. Choosing a 'worst' film is perhaps more difficult than choosing the best. There are first those that left me completely dry (Ready or Not, Written on the Wind, etc.), and those that were simply poorly executed. And there are those that failed to meet their own high opinions of themselves, such as the 'made for Reddit' Tenet (2020) or the inscrutable Vanilla Sky (2001) the latter being an almost perfect example of late-20th century cultural exhaustion. But I must save my most severe judgement for those films where I took a visceral dislike how their subjects were portrayed. The sexually problematic Sixteen Candles (1984) and the pseudo-Catholic vigilantism of The Boondock Saints (1999) both spring to mind here, the latter of which combines so many things I dislike into such a short running time I'd need an entire essay to adequately express how much I disliked it.

Dogtooth (2009) A father, a mother, a brother and two sisters live in a large and affluent house behind a very high wall and an always-locked gate. Only the father ever leaves the property, driving to the factory that he happens to own. Dogtooth goes far beyond any allusion to Josef Fritzl's cellar, though, as the children's education is a grotesque parody of home-schooling. Here, the parents deliberately teach their children the wrong meaning of words (e.g. a yellow flower is called a 'zombie'), all of which renders the outside world utterly meaningless and unreadable, and completely mystifying its very existence. It is this creepy strangeness within a 'regular' family unit in Dogtooth that is both socially and epistemically horrific, and I'll say nothing here of its sexual elements as well. Despite its cold, inscrutable and deadpan surreality, Dogtooth invites all manner of potential interpretations. Is this film about the artificiality of the nuclear family that the West insists is the benchmark of normality? Or is it, as I prefer to believe, something more visceral altogether: an allegory for the various forms of ontological violence wrought by fascism, as well a sobering nod towards some of fascism's inherent appeals? (Perhaps it is both. In 1972, French poststructuralists Gilles and F lix Guattari wrote Anti-Oedipus, which plays with the idea of the family unit as a metaphor for the authoritarian state.) The Greek-language Dogtooth, elegantly shot, thankfully provides no easy answers.

Holy Motors (2012) There is an infamous scene in Un Chien Andalou, the 1929 film collaboration between Luis Bu uel and famed artist Salvador Dal . A young woman is cornered in her own apartment by a threatening man, and she reaches for a tennis racquet in self-defence. But the man suddenly picks up two nearby ropes and drags into the frame two large grand pianos... each leaden with a dead donkey, a stone tablet, a pumpkin and a bewildered priest. This bizarre sketch serves as a better introduction to Leos Carax's Holy Motors than any elementary outline of its plot, which ostensibly follows 24 hours in the life of a man who must play a number of extremely diverse roles around Paris... all for no apparent reason. (And is he even a man?) Surrealism as an art movement gets a pretty bad wrap these days, and perhaps justifiably so. But Holy Motors and Un Chien Andalou serve as a good reminder that surrealism can be, well, 'good, actually'. And if not quite high art, Holy Motors at least demonstrates that surrealism can still unnerving and hilariously funny. Indeed, recalling the whimsy of the plot to a close friend, the tears of laughter came unbidden to my eyes once again. ("And then the limousines...!") Still, it is unclear how Holy Motors truly refreshes surrealism for the twenty-first century. Surrealism was, in part, a reaction to the mechanical and unfeeling brutality of World War I and ultimately sought to release the creative potential of the unconscious mind. Holy Motors cannot be responding to another continental conflagration, and so it appears to me to be some kind of commentary on the roles we exhibit in an era of 'post-postmodernity': a sketch on our age of performative authenticity, perhaps, or an idle doodle on the function and psychosocial function of work. Or perhaps not. After all, this film was produced in a time that offers the near-universal availability of mind-altering substances, and this certainly changes the context in which this film was both created. And, how can I put it, was intended to be watched.

Manchester by the Sea (2016) An absolutely devastating portrayal of a character who is unable to forgive himself and is hesitant to engage with anyone ever again. It features a near-ideal balance between portraying unrecoverable anguish and tender warmth, and is paradoxically grandiose in its subtle intimacy. The mechanics of life led me to watch this lying on a bed in a chain hotel by Heathrow Airport, and if this colourless circumstance blunted the film's emotional impact on me, I am probably thankful for it. Indeed, I find myself reduced in this review to fatuously recalling my favourite interactions instead of providing any real commentary. You could write a whole essay about one particular incident: its surfaces, subtexts and angles... all despite nothing of any substance ever being communicated. Truly stunning.

McCabe & Mrs. Miller (1971) Roger Ebert called this movie one of the saddest films I have ever seen, filled with a yearning for love and home that will not ever come. But whilst it is difficult to disagree with his sentiment, Ebert's choice of sad is somehow not quite the right word. Indeed, I've long regretted that our dictionaries don't have more nuanced blends of tragedy and sadness; perhaps the Ancient Greeks can loan us some. Nevertheless, the plot of this film is of a gambler and a prostitute who become business partners in a new and remote mining town called Presbyterian Church. However, as their town and enterprise booms, it comes to the attention of a large mining corporation who want to bully or buy their way into the action. What makes this film stand out is not the plot itself, however, but its mood and tone the town and its inhabitants seem to be thrown together out of raw lumber, covered alternatively in mud or frozen ice, and their days (and their personalities) are both short and dark in equal measure. As a brief aside, if you haven't seen a Roger Altman film before, this has all the trappings of being a good introduction. As Ebert went on to observe: This is not the kind of movie where the characters are introduced. They are all already here. Furthermore, we can see some of Altman's trademark conversations that overlap, a superb handling of ensemble casts, and a quietly subversive view of the tyranny of 'genre'... and the latter in a time when the appetite for revisionist portrays of the West was not very strong. All of these 'Altmanian' trademarks can be ordered in much stronger measures in his later films: in particular, his comedy-drama Nashville (1975) has 24 main characters, and my jejune interpretation of Gosford Park (2001) is that it is purposefully designed to poke fun those who take a reductionist view of 'genre', or at least on the audience's expectations. (In this case, an Edwardian-era English murder mystery in the style of Agatha Christie, but where no real murder or detection really takes place.) On the other hand, McCabe & Mrs. Miller is actually a poor introduction to Altman. The story is told in a suitable deliberate and slow tempo, and the two stars of the film are shown thoroughly defrocked of any 'star status', in both the visual and moral dimensions. All of these traits are, however, this film's strength, adding up to a credible, fascinating and riveting portrayal of the old West.

Detour (1945) Detour was filmed in less than a week, and it's difficult to decide out of the actors and the screenplay which is its weakest point.... Yet it still somehow seemed to drag me in. The plot revolves around luckless Al who is hitchhiking to California. Al gets a lift from a man called Haskell who quickly falls down dead from a heart attack. Al quickly buries the body and takes Haskell's money, car and identification, believing that the police will believe Al murdered him. An unstable element is soon introduced in the guise of Vera, who, through a set of coincidences that stretches credulity, knows that this 'new' Haskell (ie. Al pretending to be him) is not who he seems. Vera then attaches herself to Al in order to blackmail him, and the world starts to spin out of his control. It must be understood that none of this is executed very well. Rather, what makes Detour so interesting to watch is that its 'errors' lend a distinctively creepy and unnatural hue to the film. Indeed, in the early twentieth century, Sigmund Freud used the word unheimlich to describe the experience of something that is not simply mysterious, but something creepy in a strangely familiar way. This is almost the perfect description of watching Detour its eerie nature means that we are not only frequently second-guessed about where the film is going, but are often uncertain whether we are watching the usual objective perspective offered by cinema. In particular, are all the ham-fisted segues, stilted dialogue and inscrutable character motivations actually a product of Al inventing a story for the viewer? Did he murder Haskell after all, despite the film 'showing' us that Haskell died of natural causes? In other words, are we watching what Al wants us to believe? Regardless of the answers to these questions, the film succeeds precisely because of its accidental or inadvertent choices, so it is an implicit reminder that seeking the director's original intention in any piece of art is a complete mirage. Detour is certainly not a good film, but it just might be a great one. (It is a short film too, and, out of copyright, it is available online for free.)

Safe (1995) Safe is a subtly disturbing film about an upper-middle-class housewife who begins to complain about vague symptoms of illness. Initially claiming that she doesn't feel right, Carol starts to have unexplained headaches, a dry cough and nosebleeds, and eventually begins to have trouble breathing. Carol's family doctor treats her concerns with little care, and suggests to her husband that she sees a psychiatrist. Yet Carol's episodes soon escalate. For example, as a 'homemaker' and with nothing else to occupy her, Carol's orders a new couch for a party. But when the store delivers the wrong one (although it is not altogether clear that they did), Carol has a near breakdown. Unsure where to turn, an 'allergist' tells Carol she has "Environmental Illness," and so Carol eventually checks herself into a new-age commune filled with alternative therapies. On the surface, Safe is thus a film about the increasing about of pesticides and chemicals in our lives, something that was clearly felt far more viscerally in the 1990s. But it is also a film about how lack of genuine healthcare for women must be seen as a critical factor in the rise of crank medicine. (Indeed, it made for something of an uncomfortable watch during the coronavirus lockdown.) More interestingly, however, Safe gently-yet-critically examines the psychosocial causes that may be aggravating Carol's illnesses, including her vacant marriage, her hollow friends and the 'empty calorie' stimulus of suburbia. None of this should be especially new to anyone: the gendered Victorian term 'hysterical' is often all but spoken throughout this film, and perhaps from the very invention of modern medicine, women's symptoms have often regularly minimised or outright dismissed. (Hilary Mantel's 2003 memoir, Giving Up the Ghost is especially harrowing on this.) As I opened this review, the film is subtle in its messaging. Just to take one example from many, the sound of the cars is always just a fraction too loud: there's a scene where a group is eating dinner with a road in the background, and the total effect can be seen as representing the toxic fumes of modernity invading our social lives and health. I won't spoiler the conclusion of this quietly devasting film, but don't expect a happy ending.

The Driver (1978) Critics grossly misunderstood The Driver when it was first released. They interpreted the cold and unemotional affect of the characters with the lack of developmental depth, instead of representing their dissociation from the society around them. This reading was encouraged by the fact that the principal actors aren't given real names and are instead known simply by their archetypes instead: 'The Driver', 'The Detective', 'The Player' and so on. This sort of quasi-Jungian erudition is common in many crime films today (Reservoir Dogs, Kill Bill, Layer Cake, Fight Club), so the critics' misconceptions were entirely reasonable in 1978. The plot of The Driver involves the eponymous Driver, a noted getaway driver for robberies in Los Angeles. His exceptional talent has far prevented him from being captured thus far, so the Detective attempts to catch the Driver by pardoning another gang if they help convict the Driver via a set-up robbery. To give himself an edge, however, The Driver seeks help from the femme fatale 'Player' in order to mislead the Detective. If this all sounds eerily familiar, you would not be far wrong. The film was essentially remade by Nicolas Winding Refn as Drive (2011) and in Edgar Wright's 2017 Baby Driver. Yet The Driver offers something that these neon-noir variants do not. In particular, the car chases around Los Angeles are some of the most captivating I've seen: they aren't thrilling in the sense of tyre squeals, explosions and flying boxes, but rather the vehicles come across like wild animals hunting one another. This feels especially so when the police are hunting The Driver, which feels less like a low-stakes game of cat and mouse than a pack of feral animals working together a gang who will tear apart their prey if they find him. In contrast to the undercar neon glow of the Fast & Furious franchise, the urban realism backdrop of the The Driver's LA metropolis contributes to a sincere feeling of artistic fidelity as well. To be sure, most of this is present in the truly-excellent Drive, where the chase scenes do really communicate a credible sense of stakes. But the substitution of The Driver's grit with Drive's soft neon tilts it slightly towards that common affliction of crime movies: style over substance. Nevertheless, I can highly recommend watching The Driver and Drive together, as it can tell you a lot about the disconnected socioeconomic practices of the 1980s compared to the 2010s. More than that, however, the pseudo-1980s synthwave soundtrack of Drive captures something crucial to analysing the world of today. In particular, these 'sounds from the past filtered through the present' bring to mind the increasing role of nostalgia for lost futures in the culture of today, where temporality and pop culture references are almost-exclusively citational and commemorational.

The Souvenir (2019) The ostensible outline of this quietly understated film follows a shy but ambitious film student who falls into an emotionally fraught relationship with a charismatic but untrustworthy older man. But that doesn't quite cover the plot at all, for not only is The Souvenir a film about a young artist who is inspired, derailed and ultimately strengthened by a toxic relationship, it is also partly a coming-of-age drama, a subtle portrait of class and, finally, a film about the making of a film. Still, one of the geniuses of this truly heartbreaking movie is that none of these many elements crowds out the other. It never, ever feels rushed. Indeed, there are many scenes where the camera simply 'sits there' and quietly observes what is going on. Other films might smother themselves through references to 18th-century oil paintings, but The Souvenir somehow evades this too. And there's a certain ring of credibility to the story as well, no doubt in part due to the fact it is based on director Joanna Hogg's own experiences at film school. A beautifully observed and multi-layered film; I'll be happy if the sequel is one-half as good.

The Wrestler (2008) Randy 'The Ram' Robinson is long past his prime, but he is still rarin' to go in the local pro-wrestling circuit. Yet after a brutal beating that seriously threatens his health, Randy hangs up his tights and pursues a serious relationship... and even tries to reconnect with his estranged daughter. But Randy can't resist the lure of the ring, and readies himself for a comeback. The stage is thus set for Darren Aronofsky's The Wrestler, which is essentially about what drives Randy back to the ring. To be sure, Randy derives much of his money from wrestling as well as his 'fitness', self-image, self-esteem and self-worth. Oh, it's no use insisting that wrestling is fake, for the sport is, needless to say, Randy's identity; it's not for nothing that this film is called The Wrestler. In a number of ways, The Sound of Metal (2019) is both a reaction to (and a quiet remake of) The Wrestler, if only because both movies utilise 'cool' professions to explore such questions of identity. But perhaps simply when The Wrestler was produced makes it the superior film. Indeed, the role of time feels very important for the Wrestler. In the first instance, time is clearly taking its toll on Randy's body, but I felt it more strongly in the sense this was very much a pre-2008 film, released on the cliff-edge of the global financial crisis, and the concomitant precarity of the 2010s. Indeed, it is curious to consider that you couldn't make The Wrestler today, although not because the relationship to work has changed in any fundamentalway. (Indeed, isn't it somewhat depressing the realise that, since the start of the pandemic and the 'work from home' trend to one side, we now require even more people to wreck their bodies and mental health to cover their bills?) No, what I mean to say here is that, post-2016, you cannot portray wrestling on-screen without, how can I put it, unwelcome connotations. All of which then reminds me of Minari's notorious red hat... But I digress. The Wrestler is a grittily stark darkly humorous look into the life of a desperate man and a sorrowful world, all through one tragic profession.

Thief (1981) Frank is an expert professional safecracker and specialises in high-profile diamond heists. He plans to use his ill-gotten gains to retire from crime and build a life for himself with a wife and kids, so he signs on with a top gangster for one last big score. This, of course, could be the plot to any number of heist movies, but Thief does something different. Similar to The Wrestler and The Driver (see above) and a number of other films that I watched this year, Thief seems to be saying about our relationship to work and family in modernity and postmodernity. Indeed, the 'heist film', we are told, is an understudied genre, but part of the pleasure of watching these films is said to arise from how they portray our desired relationship to work. In particular, Frank's desire to pull off that last big job feels less about the money it would bring him, but a displacement from (or proxy for) fulfilling some deep-down desire to have a family or indeed any relationship at all. Because in theory, of course, Frank could enter into a fulfilling long-term relationship right away, without stealing millions of dollars in diamonds... but that's kinda the entire point: Frank needing just one more theft is an excuse to not pursue a relationship and put it off indefinitely in favour of 'work'. (And being Federal crimes, it also means Frank cannot put down meaningful roots in a community.) All this is communicated extremely subtly in the justly-lauded lowkey diner scene, by far the best scene in the movie. The visual aesthetic of Thief is as if you set The Warriors (1979) in a similarly-filthy Chicago, with the Xenophon-inspired plot of The Warriors replaced with an almost deliberate lack of plot development... and the allure of The Warriors' fantastical criminal gangs (with their alluringly well-defined social identities) substituted by a bunch of amoral individuals with no solidarity beyond the immediate moment. A tale of our time, perhaps. I should warn you that the ending of Thief is famously weak, but this is a gritty, intelligent and strangely credible heist movie before you get there.

Uncut Gems (2019) The most exhausting film I've seen in years; the cinematic equivalent of four cups of double espresso, I didn't even bother even trying to sleep after downing Uncut Gems late one night. Directed by the two Safdie Brothers, it often felt like I was watching two films that had been made at the same time. (Or do I mean two films at 2X speed?) No, whatever clumsy metaphor you choose to adopt, the unavoidable effect of this film's finely-tuned chaos is an uncompromising and anxiety-inducing piece of cinema. The plot follows Howard as a man lost to his countless vices mostly gambling with a significant side hustle in adultery, but you get the distinct impression he would be happy with anything that will give him another high. A true junkie's junkie, you might say. You know right from the beginning it's going to end in some kind of disaster, the only question remaining is precisely how and what. Portrayed by an (almost unrecognisable) Adam Sandler, there's an uncanny sense of distance in the emotional chasm between 'Sandler-as-junkie' and 'Sandler-as-regular-star-of-goofy-comedies'. Yet instead of being distracting and reducing the film's affect, this possibly-deliberate intertextuality somehow adds to the masterfully-controlled mayhem. My heart races just at the memory. Oof.

Woman in the Dunes (1964) I ended up watching three films that feature sand this year: Denis Villeneuve's Dune (2021), Lawrence of Arabia (1962) and Woman in the Dunes. But it is this last 1964 film by Hiroshi Teshigahara that will stick in my mind in the years to come. Sure, there is none of the Medician intrigue of Dune or the Super Panavision-70 of Lawrence of Arabia (or its quasi-orientalist score, itself likely stolen from Anton Bruckner's 6th Symphony), but Woman in the Dunes doesn't have to assert its confidence so boldly, and it reveals the enormity of its plot slowly and deliberately instead. Woman in the Dunes never rushes to get to the film's central dilemma, and it uncovers its terror in little hints and insights, all whilst establishing the daily rhythm of life. Woman in the Dunes has something of the uncanny horror as Dogtooth (see above), as well as its broad range of potential interpretations. Both films permit a wide array of readings, without resorting to being deliberately obscurantist or being just plain random it is perhaps this reason why I enjoyed them so much. It is true that asking 'So what does the sand mean?' sounds tediously sophomoric shorn of any context, but it somehow applies to this thoughtfully self-contained piece of cinema.

A Quiet Place (2018) Although A Quiet Place was not actually one of the best films I saw this year, I'm including it here as it is certainly one of the better 'mainstream' Hollywood franchises I came across. Not only is the film very ably constructed and engages on a visceral level, I should point out that it is rare that I can empathise with the peril of conventional horror movies (and perhaps prefer to focus on its cultural and political aesthetics), but I did here. The conceit of this particular post-apocalyptic world is that a family is forced to live in almost complete silence while hiding from creatures that hunt by sound alone. Still, A Quiet Place engages on an intellectual level too, and this probably works in tandem with the pure 'horrorific' elements and make it stick into your mind. In particular, and to my mind at least, A Quiet Place a deeply American conservative film below the surface: it exalts the family structure and a certain kind of sacrifice for your family. (The music often had a passacaglia-like strain too, forming a tombeau for America.) Moreover, you survive in this dystopia by staying quiet that is to say, by staying stoic suggesting that in the wake of any conflict that might beset the world, the best thing to do is to keep quiet. Even communicating with your loved ones can be deadly to both of you, so not emote, acquiesce quietly to your fate, and don't, whatever you do, speak up. (Or join a union.) I could go on, but The Quiet Place is more than this. It's taut and brief, and despite cinema being an increasingly visual medium, it encourages its audience to develop a new relationship with sound.

Russell Coker: SSD Endurance

I previously wrote about the issue of swap potentially breaking SSD [1]. My conclusion was that swap wouldn t be a problem as no normally operating systems that I run had swap using any significant fraction of total disk writes. In that post the most writes I could see was 128GB written per day on a 120G Intel SSD (writing the entire device once a day). My post about swap and SSD was based on the assumption that you could get many thousands of writes to the entire device which was incorrect. Here s a background on the terminology from WD [2]. So in the case of the 120G Intel SSD I was doing over 1 DWPD (Drive Writes Per Day) which is in the middle of the range of SSD capability, Intel doesn t specify the DWPD or TBW (Tera Bytes Written) for that device. The most expensive and high end NVMe device sold by my local computer store is the Samsung 980 Pro which has a warranty of 150TBW for the 250G device and 600TBW for the 1TB device [3]. That means that the system which used to have an Intel SSD would have exceeded the warranty in 3 years if it had a 250G device. My current workstation has been up for just over 7 days and has averaged 110GB written per day. It has some light VM use and the occasional kernel compile, a fairly typical developer workstation. It s storage is 2*Crucial 1TB NVMe devices in a BTRFS RAID-1, the NVMe devices are the old series of Crucial ones and are rated for 200TBW which means that they can be expected to last for 5 years under the current load. This isn t a real problem for me as the performance of those devices is lower than I hoped for so I will buy faster ones before they are 5yo anyway. My home server (and my wife s workstation) is averaging 325GB per day on the SSDs used for the RAID-1 BTRFS filesystem for root and for most data that is written much (including VMs). The SSDs are 500G Samsung 850 EVOs [4] which are rated at 150TBW which means just over a year of expected lifetime. The SSDs are much more than a year old, I think Samsung stopped selling them more than a year ago. Between the 2 SSDs SMART reports 18 uncorrectable errors and btrfs device stats reports 55 errors on one of them. I m not about to immediately replace them, but it appears that they are well past their prime. The server which runs my blog (among many other things) is averaging over 1TB written per day. It currently has a RAID-1 of hard drives for all storage but it s previous incarnation (which probably had about the same amount of writes) had a RAID-1 of enterprise SSDs for the most written data. After a few years of running like that (and some time running with someone else s load before it) the SSDs became extremely slow (sustained writes of 15MB/s) and started getting errors. So that s a pair of SSDs that were burned out. Conclusion The amounts of data being written are steadily increasing. Recent machines with more RAM can decrease storage usage in some situations but that doesn t compare to the increased use of checksummed and logged filesystems, VMs, databases for local storage, and other things that multiply writes. The amount of writes allowed under warranty isn t increasing much and there are new technologies for larger SSD storage that decrease the DWPD rating of the underlying hardware. For the systems I own it seems that they are all going to exceed the rated TBW for the SSDs before I have other reasons to replace them, and they aren t particularly high usage systems. A mail server for a large number of users would hit it much earlier. RAID of SSDs is a really good thing. Replacement of SSDs is something that should be planned for and a way of swapping SSDs to less important uses is also good (my parents have some SSDs that are too small for my current use but which work well for them). Another thing to consider is that if you have a server with spare drive bays you could put some extra SSDs in to spread the wear among a larger RAID-10 array. Instead of having a 2*SSD BTRFS RAID-1 for a server you could have 6*SSD to get a 3* longer lifetime than a regular RAID-1 before the SSDs wear out (BTRFS supports this sort of thing). Based on these calculations and the small number of errors I ve seen on my home server I ll add a 480G SSD I have lying around to the array to spread the load and keep it running for a while longer.

Russ Allbery: DocKnot 6.01

This release of my static site generator and software release manager finishes incorporating the last piece of my old release script that I was still using: copying a new software release into a software distribution archive tree, updating symlinks, updating the version database used to generate my web pages, and archiving the old version. I also added a new docknot update-spin command that updates an input tree for the spin static site generator, fixing any deprecations or changes in the input format. Currently, all this does is convert the old-style *.rpod pointer files to new-style *.spin pointers. This release also has a few other minor bug fixes, including for an embarrassing bug that required docknot spin be run from a package source tree because it tried to load per-package metadata (even though it doesn't use that data). You can get the latest release from CPAN or from the DocKnot distribution page.

14 January 2022

Dirk Eddelbuettel: Rcpp 1.0.8: Updated, Strict Headers

rcpp logo The Rcpp team is thrilled to share the news of the newest release 1.0.8 of Rcpp which hit CRAN today, and has already been uploaded to Debian as well. Windows and macOS builds should appear at CRAN in the next few days. This release continues with the six-months cycle started with release 1.0.5 in July 2020. As a reminder, interim dev or rc releases will alwasys be available in the Rcpp drat repo; this cycle there were once again seven (!!) times two as we also tested the modified header (more below). These rolling release tend to work just as well, and are also fully tested against all reverse-dependencies. Rcpp has become the most popular way of enhancing R with C or C++ code. Right now, around 2478 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 242 in BioConductor. This release finally brings a change we have worked on quite a bit over the last few months. The idea of enforcing the setting of STRICT_R_HEADERS was prososed years ago in 2016 and again in 2018. But making such a chance against a widely-deployed code base has repurcussions, and we were not ready then. Last April, this was revisited in issue #1158. Over the course of numerous lengthy runs of tests of a changed Rcpp package against (essentially) all reverse-dependencies (i.e. packages which use Rcpp) we identified ninetyfour packages in total which needed a change. We provided either a patch we emailed, or a GitHub pull request, to all ninetyfour. And we are happy to say that eighty cases were resolved via a new CRAN upload, with a seven more having merged the pull request but not yet uploaded. Hence, we could make the case to CRAN (who were always CC ed on the monthly nag emails we sent to maintainers of packages needing a change) that an upload was warranted. And after a brief period for their checks and inspection, our January 11 release of Rcpp 1.0.8 arrived on CRAN on January 13. So with that, a big and heartfelt Thank You! to all eighty maintainers for updating their packages to permit this change at the Rcpp end, to CRAN for the extra checking, and to everybody else who I bugged with the numerous emails and updated to the seemingly never-ending issue #1158. We all got this done, and that is a Good Thing (TM). Other than the aforementioned change which will not automatically set STRICT_R_HEADERS (unless opted out which one can), a number of nice pull request by a number of contributors are included in this release: The full list of details follows.

Changes in Rcpp release version 1.0.8 (2022-01-11)
  • Changes in Rcpp API:
    • STRICT_R_HEADERS is now enabled by default, see extensive discussion in #1158 closing #898.
    • A new #define allows default setting of finalizer calls for external pointers (I aki in #1180 closing #1108).
    • Rcpp:::CxxFlags() now quotes the include path generated, (Kevin in #1189 closing #1188).
    • New header files Rcpp/Light, Rcpp/Lighter, Rcpp/Lightest and default Rcpp/Rcpp for fine-grained access to features (and compilation time) (Dirk #1191 addressing #1168).
  • Changes in Rcpp Attributes:
    • A new option signature allows customization of function signatures (Travers Ching in #1184 and #1187 fixing #1182)
  • Changes in Rcpp Documentation:
    • The Rcpp FAQ has a new entry on how not to grow a vector (Dirk in #1167).
    • Some long-spurious calls to RNGSope have been removed from examples (Dirk in #1173 closing #1172).
    • DOI reference in the bibtex files have been updated per JSS request (Dirk in #1186).
  • Changes in Rcpp Deployment:
    • Some continuous integration components have been updated (Dirk in #1174, #1181, and #1190).

Thanks to my CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2822 previous questions. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 January 2022

Bits from Debian: New Debian Developers and Maintainers (November and December 2021)

The following contributors got their Debian Developer accounts in the last two months: The following contributors were added as Debian Maintainers in the last two months: Congratulations!

11 January 2022

Ritesh Raj Sarraf: ThinkPad AMD Debian

After a hiatus of 6 years, it was nice to be back with the ThinkPad. This blog post briefly touches upon my impressions with the current generation ThinkPad T14 Gen2 AMD variant.
ThinkPad T14 Gen2 AMD
ThinkPad T14 Gen2 AMD

Lenovo It took 8 weeks to get my hands on the machine. Given the pandemic, restrictions and uncertainities, not sure if I should call it an ontime delivery. This was a CTO - Customise-to-order; so was nice to get rid of things I really didn t care/use much. On the other side, it also meant I could save on some power. It also came comparatively cheaper overall.
  • No fingerprint reader
  • No Touch screen
There s still parts where Lenovo could improve. Or less frustate a customer. I don t understand why a company would provide a full customization option on their portal, while at the same time, not provide an explicit option to choose the make/model of the hardware one wants. Lenovo deliberately chooses to not show/specify which WiFi adapter one could choose. So, as I suspected, I ended up with a MEDIATEK Corp. Device 7961 wifi adapter.

AMD For the first time in my computing life, I m now using AMD at the core. I was pretty frustrated with annoying Intel Graphics bugs, so decided to take the plunge and give AMD/ATI a shot, knowing that the radeon driver does have decent support. So far, on the graphics side of things, I m glad that things look bright. The stock in-kernel radeon driver has been working perfect for my needs and I haven t had to tinker even once so far, in my 30 days of use. On the overall system performance, I have not done any benchmarks nor do I want to do. But wholly, the system performance is smooth.

Power/Thermal This is where things need more improvement on the AMD side. This AMD laptop terribly draws a lot of power in suspend mode. And it isn t just this machine, but also the previous T14 Gen1 which has similar problems. I m not sure if this is a generic ThinkPad problem, or an AMD specific problem. But coming from the Dell XPS 13 9370 Intel, this does draw a lot lot more power. So much, that I chose to use hibernation instead. Similarly, on the thermal side, this machine doesn t cool down well as compared the the Dell XPS Intel one. On an idle machine, its temperature are comparatively higher. Looking at powertop reports, it does show to consume an average of 10 watts power even while idle. I m hoping these are Linux ingeration issues and that Lenovo/AMD will improve things in the coming months. But given the user feedback on the ThinkPad T14 Gen1 thread, it may just be wishful thinking.

Linux The overall hardware support has been surprisingly decent. The MediaTek WiFi driver had some glitches but with Linux 5.15+, things have considerably improved. And I hope the trend will continue with forthcoming Linux releases. My previous device driver experience with MediaTek wasn t good but I took the plunge, considering that in the worst scenario I d have the option to swap the card. There s a lot of marketing about Linux + Intel. But I took a jibe with Linux + AMD. There are glitches but nothing so far that has been a dealbreaker. If anything, I wish Lenovo/AMD would seriously work on the power/thermal issues.

Migration Other than what s mentioned above, I haven t had any serious issues. I may have had some rare occassional hangs but they ve been so infrequent that I haven t spent time to investigate those. Upon receiving the machine, my biggest requirement was how to switch my current workstation from Dell XPS to Lenovo ThinkPad. I ve been using btrfs for some time now. And over the years, built my own practise on how to structure it. Things like, provisioning [sub]volumes, based on use cases is one thing I see. Like keeping separate subvols for: cache/temporary data, copy-on-write data , swap etc. I wish these things could be simplified; either on the btrfs tooling side or some different tool on top of it. Below is filtered list of subvols created over years, that were worthy of moving to the new machine.
rrs@priyasi:~$ cat btrfs-volume-layout 
ID 550 gen 19166 top level 5 path home/foo/.cache
ID 552 gen 1522688 top level 5 path home/rrs
ID 553 gen 1522688 top level 552 path home/rrs/.cache
ID 555 gen 1426323 top level 552 path home/rrs/rrs-home/Libvirt-Images
ID 618 gen 1522672 top level 5 path var/spool/news
ID 634 gen 1522670 top level 5 path var/tmp
ID 635 gen 1522688 top level 5 path var/log
ID 639 gen 1522226 top level 5 path var/cache
ID 992 gen 1522670 top level 5 path disk-tmp
ID 1018 gen 1522688 top level 552 path home/rrs/NoBackup
ID 1196 gen 1522671 top level 5 path etc
ID 23721 gen 775692 top level 5 path swap
18:54                      

btrfs send/receive This did come in handy but I sorely missed some feature. Maybe they aren t there, or are there and I didn t look close enough. Over the years, different attributes were set to different subvols. Over time I forget what feature was added where. But from a migration point of view, it d be nice to say, Take this volume and take it with all its attributes . I didn t find that functionality in send/receive. There s get/set-property which I noticed later but by then it was late. So some sort of tooling, ideally something like btrfs migrate or somesuch would be nicer. In the file system world, we already have nice tools to take care of similar scenarios. Like with rsync, I can request it to carry all file attributes. Also, iirc, send/receive works only on ro volumes. So there s more work one needs to do in:
  1. create ro vol
  2. send
  3. receive
  4. don t forget to set rw property
  5. And then somehow find out other properties set on each individual subvols and [re]apply the same on the destination
I wish this all be condensed into a sub-command. For my own sake, for this migration, the steps used were:
user@debian:~$ for volume in  sudo btrfs sub list /media/user/TOSHIBA/Migrate/   cut -d ' ' -f9   grep -v ROOTVOL   grep -v etc   grep -v btrbk ; do echo $volume; sud
o btrfs send /media/user/TOSHIBA/$volume   sudo btrfs receive /media/user/BTRFSROOT/ ; done            
Migrate/snapshot_disk-tmp
At subvol /media/user/TOSHIBA/Migrate/snapshot_disk-tmp
At subvol snapshot_disk-tmp
Migrate/snapshot-home_foo_.cache
At subvol /media/user/TOSHIBA/Migrate/snapshot-home_foo_.cache
At subvol snapshot-home_foo_.cache
Migrate/snapshot-home_rrs
At subvol /media/user/TOSHIBA/Migrate/snapshot-home_rrs
At subvol snapshot-home_rrs
Migrate/snapshot-home_rrs_.cache
At subvol /media/user/TOSHIBA/Migrate/snapshot-home_rrs_.cache
At subvol snapshot-home_rrs_.cache
ERROR: crc32 mismatch in command
Migrate/snapshot-home_rrs_rrs-home_Libvirt-Images
At subvol /media/user/TOSHIBA/Migrate/snapshot-home_rrs_rrs-home_Libvirt-Images
At subvol snapshot-home_rrs_rrs-home_Libvirt-Images
ERROR: crc32 mismatch in command
Migrate/snapshot-var_spool_news
At subvol /media/user/TOSHIBA/Migrate/snapshot-var_spool_news
At subvol snapshot-var_spool_news
Migrate/snapshot-var_lib_machines
At subvol /media/user/TOSHIBA/Migrate/snapshot-var_lib_machines
At subvol snapshot-var_lib_machines
Migrate/snapshot-var_lib_machines_DebianSidTemplate
..... snipped .....
And then, follow-up with:
user@debian:~$ for volume in  sudo btrfs sub list /media/user/BTRFSROOT/   cut -d ' ' -f9 ; do echo $volume; sudo btrfs property set -ts /media/user/BTRFSROOT/$volume ro false; done
ROOTVOL
ERROR: Could not open: No such file or directory
etc
snapshot_disk-tmp
snapshot-home_foo_.cache
snapshot-home_rrs
snapshot-var_spool_news
snapshot-var_lib_machines
snapshot-var_lib_machines_DebianSidTemplate
snapshot-var_lib_machines_DebSidArmhf
snapshot-var_lib_machines_DebianJessieTemplate
snapshot-var_tmp
snapshot-var_log
snapshot-var_cache
snapshot-disk-tmp
And then finally, renaming everything to match proper:
user@debian:/media/user/BTRFSROOT$ for x in snapshot*; do vol=$(echo $x   cut -d '-' -f2   sed -e "s _ / g"); echo $x $vol; sudo mv $x $vol; done
snapshot-var_lib_machines var/lib/machines
snapshot-var_lib_machines_Apertisv2020ospackTargetARMHF var/lib/machines/Apertisv2020ospackTargetARMHF
snapshot-var_lib_machines_Apertisv2021ospackTargetARM64 var/lib/machines/Apertisv2021ospackTargetARM64
snapshot-var_lib_machines_Apertisv2022dev3ospackTargetARMHF var/lib/machines/Apertisv2022dev3ospackTargetARMHF
snapshot-var_lib_machines_BusterArm64 var/lib/machines/BusterArm64
snapshot-var_lib_machines_DebianBusterTemplate var/lib/machines/DebianBusterTemplate
snapshot-var_lib_machines_DebianJessieTemplate var/lib/machines/DebianJessieTemplate
snapshot-var_lib_machines_DebianSidTemplate var/lib/machines/DebianSidTemplate
snapshot-var_lib_machines_DebianSidTemplate_var_lib_portables var/lib/machines/DebianSidTemplate/var/lib/portables
snapshot-var_lib_machines_DebSidArm64 var/lib/machines/DebSidArm64
snapshot-var_lib_machines_DebSidArmhf var/lib/machines/DebSidArmhf
snapshot-var_lib_machines_DebSidMips var/lib/machines/DebSidMips
snapshot-var_lib_machines_JenkinsApertis var/lib/machines/JenkinsApertis
snapshot-var_lib_machines_v2019 var/lib/machines/v2019
snapshot-var_lib_machines_v2019LinuxSupport var/lib/machines/v2019LinuxSupport
snapshot-var_lib_machines_v2020 var/lib/machines/v2020
snapshot-var_lib_machines_v2021dev3Slim var/lib/machines/v2021dev3Slim
snapshot-var_lib_machines_v2021dev3SlimTarget var/lib/machines/v2021dev3SlimTarget
snapshot-var_lib_machines_v2022dev2OspackMinimal var/lib/machines/v2022dev2OspackMinimal
snapshot-var_lib_portables var/lib/portables
snapshot-var_log var/log
snapshot-var_spool_news var/spool/news
snapshot-var_tmp var/tmp

snapper Entirely independent of this, but indirectly related. I use snapper as my snapshotting tool. It worked perfect on my previous machine. While everything got migrated, the only thing that fell apart was snapper. It just wouldn t start/run proper. Funny thing is that I just removed the snapper configs and reinitialized with the exact same config again, and voila snapper was happy.

Conclusion That was pretty much it. With the above and then also migrating /boot and then just chroot to install the boot loader. At some time, I d like to explore other boot options but given that that is such a non-essential task, it is low on the list. The good part was that I booted into my new machine with my exact workstation setup as it was. All the way to the user cache and the desktop session. So it was nice on that part. But I surely think there s room for a better migration experience here. If not directly as btrfs migrate, then maybe as an independent tool. The problem is that such a tool is going to be used once in years, so I didn t find the motivation to write one. But this surely would be a good use case for the distribution vendors.

Next.

Previous.