Search Results: "rra"

23 October 2021

Arthur Diniz: Dropdown for GitHub workflows input parameters

Dropdown for GitHub workflows input parameters Sometimes when we look at CI/CD tools embedded within git-based software repository manager like GitHub, GitLab or Bitbucket, we ran into a lack of some features. This time me and my DevOps/SRE team were facing a pain of not being able to have the option to create drop-downs within GitHub workflows using input parameters. Although this functionality is already available on other platforms such as Bitbucket, the specific client we were working on stored the code inside GitHub. At first I thought that someone has already solved this problem somehow, but doing an extensive search on the internet I found several angry GitHub users opening requests within the Support Community and even in the stack overflow. comment-1 comment-2 comment-3 comment-5 comment-4 So I decided to create a solution for this, always thinking about simplicity and in a way that makes it easy to get this missing functionality. I started by creating an input array pattern using commas and using a tag (the selector) e.g brackets as the default value marker. Here is an example of what an input string would look like:
name: gh-action-dropdown-list-input
on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment'
        required: true
        default: 'dev,staging,[uat],prod'
Now the final question that would turn out to be the most complicated to deal with. How can I change the GitHub Actions interface to replace the input pattern we created earlier to a dropdown? The simplest answer I thought was to create a chrome and firefox extension that would do all this logic behind the scenes and replace the HTML input element with the selected tag containing the array values and leaving the tag value (selector) always as the default. All code was developed in pure JavaScript, open-source licensed under Apache 2.0 and available at https://github.com/arthurbdiniz/gh-action-dropdown-list-input.

Install extension Once installed, the extension is ready to use and the final result we see is the Actions interface with drop-downs. :) showcase-1 showcase-2

Configuring selectors Go to the top right corner of the browser you are using and click on the extension logo. A screen will popup with tag options. Choose the right tags for you and save it.
This action might require reloading the GitHub workflow tab. config

Have fun using drop-downs inside GitHub. If you liked this project please share this post and if possible star within the repository. Also feel free to connect with me on LinkedIn: https://www.linkedin.com/in/arthurbdiniz

References
  • https://github.community/t/can-workflow-dispatch-input-be-option-list/127338
  • https://stackoverflow.com/questions/69296314/dropdown-for-github-workflows-input-parameters

18 October 2021

Russ Allbery: rra-c-util 10.0

It's been a while since I pushed out a release of my collection of utility libraries and test suite programs, so I've accumulated quite a lot of chanages. Here's a summary; for more, see the NEWS file. You can get the latest version from the rra-c-util distribution page.

9 October 2021

Ritesh Raj Sarraf: Lotus to Lily

The Lotus story so far My very first experience with water flowering plants was pretty good. I learnt a good deal of things; from setting up the pond, germinating the lotus seeds, setting up the right soil, witnessing the growth of the lotus plant, fish eco-system to take care of the pond. Overall, a lot of things learnt. But I couldn t succeed in getting the Lotus flower. A lot many reasons. The granite container developed some leakage, which I had to fix by emptying it, which might have caused some shock to the lotus. But more than that, in my understanding, the reason for not being able to flower the lotus, was the amount of sunlight. From what I have learned, these plants need a minimum of 6-8 hrs of sunlight to really give you with the flowering result, whereas the setup of my pond was on the ground with hardly 3-4 hrs of sun. And that too, with all the plants growing, resulted in indirect sunlight.

Lotus to Lily For my new setup, I chose a large oval container. And this one, I placed on my terrace, carefully choosing a spot where it d get 6-8 hrs of very bright sun on usual days. Other than that, the rest of the setup is pretty similar to my previous setup in the garden. Guppies, Solar Water Fountain etc.
Initial lily pond setup
Initial lily pond setup
The good thing about the terrace is that the setup gets ample amount of sun. You can see that in the picture above, with the amount of algae that has been formed. Something that is vital for the plant s ecosystem. I must thank my wonderful neighbor who kindly shared a sapling from their lily plant. They already had had success with flowering the lily. So I had high hopes to see the day come when I d be happy to write down my experience in this blog post. Though, a lot of patience is needed. I got the lily some time in January this year. And it blossomed now, in October. So, here s me sharing my happiness here, in particular order of how I documented the process.
Monday morning greeted with a blossomed lily
Monday morning greeted with a blossomed lily
Lily Blossom Closeup
Lily Blossom Closeup
Beautiful water reflection
Beautiful water reflection

Dawn to Dusk The other thing that I learned in this whole lily episode is that the flower goes back to sleeping at dusk. And back to flowering again at dawn. There s so much to learn in the surrounding, only if you spare some time to the little things with mother nature.
Lily status at dusk
Lily status at dusk
Lily the next day
Lily the next day
Not sure how long this phenomenon is to last, but overall witnessing this whole process has been mesmerizing. This past week has been great.

3 October 2021

Ritesh Raj Sarraf: Human Society

In my past, I ve had experiences that have had me thinking. My experiences have been mostly in the South Asian Indian Sub-Continent, so may not be fair to generalize it.

23 September 2021

Dirk Eddelbuettel: prrd 0.0.5: Incremental Mode

prrd facilitates the parallel running [of] reverse dependency [checks] when preparing R packages. It is used extensively for Rcpp, RcppArmadillo, RcppEigen, BH, and others. prrd screenshot image The key idea of prrd is simple, and described in some more detail on its webpage and its GitHub repo. Reverse dependency checks are an important part of package development that is easily done in a (serial) loop. But these checks are also generally embarassingly parallel as there is no or little interdependency between them (besides maybe shared build depedencies). See the (dated) screenshot (running six parallel workers, arranged in a split byobu session). This release brings some new features I used of late when testing and re-testing reverse dependencies for Rcpp. Enqueuing jobs can now consider the most recent prior job queue file. This allows us to find new packages that were not part of the previous runs. We added a second toggle to also add those packages who failed in the previous run. Finally, the dequeue interface allows to specify a date (rather than defaulting to the current date, useful for long-running jobs or restarts). The release is summarised in the NEWS entry:

Changes in prrd version 0.0.5 (2021-09-22)
  • Some remaing http URLs were changed to https.
  • The dequeueJobs script has a new argument date to help specify a queue file.
  • The enqueueJobs can now compute just a delta of (new) packages relative to a given prior queuefile and run.
  • When running in delta mode, previously failed packages can also be selected.

My CRANberries provides the usual summary of changes to the previous version. See the aforementioned webpage and its repo for details. For more questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 September 2021

Clint Adams: Outrage culture killed my dog

Why can't Debian have embarrassing flamewars like this thread?
Posted on 2021-09-21
Tags: barks

16 September 2021

Chris Lamb: On Colson Whitehead's Harlem Shuffle

Colson Whitehead's latest novel, Harlem Shuffle, was always going to be widely reviewed, if only because his last two books won Pulitzer prizes. Still, after enjoying both The Underground Railroad and The Nickel Boys, I was certainly going to read his next book, regardless of what the critics were saying indeed, it was actually quite agreeable to float above the manufactured energy of the book's launch. Saying that, I was encouraged to listen to an interview with the author by Ezra Klein. Now I had heard Whitehead speak once before when he accepted the Orwell Prize in 2020, and once again he came across as a pretty down-to-earth guy. Or if I were to emulate the detached and cynical tone Whitehead embodied in The Nickel Boys, after winning so many literary prizes in the past few years, he has clearly rehearsed how to respond to the cliched questions authors must be asked in every interview. With the obligatory throat-clearing of 'so, how did you get into writing?', for instance, Whitehead replies with his part of the catechism that 'It seemed like being a writer could be a cool job. You could work from home and not talk to people.' The response is the right combination of cute and self-effacing... and with its slight tone-deafness towards enforced isolation, it was no doubt honed before Covid-19. Harlem Shuffle tells three separate stories about Ray Carney, a furniture salesman and 'fence' for stolen goods in New York in the 1960s. Carney doesn't consider himself a genuine criminal though, and there's a certain logic to his relativistic morality. After all, everyone in New York City is on the take in some way, and if some 'lightly used items' in Carney's shop happened to have had 'previous owners', well, that's not quite his problem. 'Nothing solid in the city but the bedrock,' as one character dryly observes. Yet as Ezra pounces on in his NYT interview mentioned abov, the focus on the Harlem underworld means there are very few women in the book, and Whitehead's circular response ah well, it's a book about the criminals at that time! was a little unsatisfying. Not only did it feel uncharacteristically slippery of someone justly lauded for his unflinching power of observation (after all, it was the author who decided what to write about in the first place), it foreclosed on the opportunity to delve into why the heist and caper genres (from The Killing, The Feather Thief, Ocean's 11, etc.) have historically been a 'male' mode of storytelling. Perhaps knowing this to be the case, the conversation quickly steered towards Ray Carney's wife, Elizabeth, the only woman in the book who could be said possesses some plausible interiority. The following off-hand remark from Whitehead caught my attention:
My wife is convinced that [Elizabeth] knows everything about Carney's criminal life, and is sort of giving him a pass. And I'm not sure if that's true. I have to have to figure out exactly what she knows and when she knows it and how she feels about it.
I was quite taken by this, although not simply due to its effect on the story it self. As in, it immediately conjured up a charming picture of Whitehead's domestic arrangements: not only does Whitehead's wife feel free to disagree with what one of Whitehead's 'own' characters knows or believes, but that Colson has no problem whatsoever sharing that disagreement with the public at large. (It feels somehow natural that Whitehead's wife believes her counterpart knows more than she lets on, whilst Whitehead himself imbues the protagonist's wife with a kind of neo-Victorian innocence.) I'm minded to agree with Whitehead's partner myself, if only due to the passages where Elizabeth is studiously ignoring Carney's otherwise unexplained freak-outs. But all of these meta-thoughts simply underline just how emancipatory the Death of the Author can be. This product of academic literary criticism (the term was coined by Roland Barthes' 1967 essay of the same name) holds that the original author's intentions, ideas or biographical background carry no especial weight in determining how others should interpret their work. It is usually understood as meaning that a writer's own views are no more valid or 'correct' than the views held by someone else. (As an aside, I've found that most readers who encounter this concept for the first time have been reading books in this way since they were young. But the opposite is invariably true with cinephiles, who often have a bizarre obsession with researching or deciphering the 'true' interpretation of a film.) And with all that in mind, can you think of a more wry example of how freeing (and fun) nature of the Death of the Author than an author's own partner dissenting with their (Pulitzer Prize-winning) husband on the position of a lynchpin character?
The 1964 Harlem riot began after James Powell, a 15-year-old African American, was shot and killed by Thomas Gilligan, an NYPD police officer in front of 10s of witnesses. Gilligan was subsequently cleared by a grand jury.
As it turns out, the reviews for Harlem Shuffle have been almost universally positive, and after reading it in the two days after its release, I would certainly agree it is an above-average book. But it didn't quite take hold of me in the way that The Underground Railroad or The Nickel Boys did, especially the later chapters of The Nickel Boys that were set in contemporary New York and could thus make some (admittedly fairly explicit) connections from the 1960s to the present day that kind of connection is not there in Harlem Shuffle, or at least I did not pick up on it during my reading. I can see why one might take exception to that, though. For instance, it is certainly true that the week-long Harlem Riot forms a significant part of the plot, and some events in particular are entirely contingent on the ramifications of this momentous event. But it's difficult to argue the riot's impact are truly integral to the story, so not only is this uprising against police brutality almost regarded as a background event, any contemporary allusion to the murder of George Floyd is subsequently watered down. It's nowhere near the historical rubbernecking of Forrest Gump (1994), of course, but that's not a battle you should ever be fighting. Indeed, whilst a certain smoothness of affect is to be priced into the Whitehead reading experience, my initial overall reaction to Harlem Shuffle was fairly flat, despite all the action and intrigue on the page. The book perhaps belies its origins as a work conceived during quarantine after all, the book is essentially comprised of three loosely connected novellas, almost as if the unreality and mental turbulence of lockdown prevented the author from performing the psychological 'deep work' of producing a novel-length text with his usual depth of craft. A few other elements chimed with this being a 'lockdown novel' as well, particularly the book's preoccupation with the sheer physicality of the city compared to the usual complex interplay between its architecture and its inhabitants. This felt like it had been directly absorbed into the book from the author walking around his deserted city, and thus being able to take in details for the first time:
The doorways were entrances into different cities no, different entrances into one vast, secret city. Ever close, adjacent to all you know, just underneath. If you know where to look.
And I can't fail to mention that you can almost touch Whitehead's sublimated hunger to eat out again as well:
Stickups were chops they cook fast and hot, you re in and out. A stakeout was ribs fire down low, slow, taking your time. [ ] Sometimes when Carney jumped into the Hudson when he was a kid, some of that stuff got into his mouth. The Big Apple Diner served it up and called it coffee.
More seriously, however, the relatively thin personalities of minor characters then reminded me of the simulacrum of Zoom-based relationships, and the essentially unsatisfactory endings to the novellas felt reminiscent of lockdown pseudo-events that simply fizzle out without a bang. One of the stories ties up loose ends with: 'These things were usually enough to terminate a mob war, and they appeared to end the hostilities in this case as well.' They did? Well, okay, I guess.
The corner of 125th Street and Morningside Avenue in 2019, the purported location of Carney's fictional furniture store. Signage plays a prominent role in Harlem Shuffle, possibly due to the author's quarantine walks.
Still, it would be unfair to characterise myself as 'disappointed' with the novel, and none of this piece should be taken as really deep criticism. The book certainly was entertaining enough, and pretty funny in places as well:
Carney didn t have an etiquette book in front of him, but he was sure it was bad manners to sit on a man s safe. [ ] The manager of the laundromat was a scrawny man in a saggy undershirt painted with sweat stains. Launderer, heal thyself.
Yet I can't shake the feeling that every book you write is a book that you don't, and so we might need to hold out a little longer for Whitehead's 'George Floyd novel'. (Although it is for others to say how much of this sentiment is the expectations of a White Reader for The Black Author to ventriloquise the pain of 'their' community.) Some room for personal critique is surely permitted. I dearly missed the junk food energy of the dry and acerbic observations that run through Whitehead's previous work. At one point he had a good line on the model tokenisation that lurks behind 'The First Negro to...' labels, but the callbacks to this idea ceased without any payoff. Similar things happened with the not-so-subtle critiques of the American Dream:
Entrepreneur? Pepper said the last part like manure. That s just a hustler who pays taxes. [ ] One thing I ve learned in my job is that life is cheap, and when things start getting expensive, it gets cheaper still.
Ultimately, though, I think I just wanted more. I wanted a deeper exploration of how the real power in New York is not wielded by individual street hoodlums or even the cops but in the form of real estate, essentially serving as a synecdoche for Capital as a whole. (A recent take of this can be felt in Jed Rothstein's 2021 documentary, WeWork: Or the Making and Breaking of a $47 Billion Unicorn and it is perhaps pertinent to remember that the US President at the time this novel was written was affecting to be a real estate tycoon.). Indeed, just like the concluding scenes of J. J. Connolly's Layer Cake, although you can certainly pull off a cool heist against the Man, power ultimately resides in those who control the means of production... and a homespun furniture salesman on the corner of 125 & Morningside just ain't that. There are some nods to kind of analysis in the conclusion of the final story ('Their heist unwound as if it had never happened, and Van Wyck kept throwing up buildings.'), but, again, I would have simply liked more. And when I attempted then file this book away into the broader media landscape, given the current cultural visibility of 1960s pop culture (e.g. One Night in Miami (2020), Judas and the Black Messiah (2021), Summer of Soul (2021), etc.), Harlem Shuffle also seemed like a missed opportunity to critically analyse our (highly-qualified) longing for the civil rights era. I can certainly understand why we might look fondly on the cultural products from a period when politics was less alienated, when society was less atomised, and when it was still possible to imagine meaningful change, but in this dimension at least, Harlem Shuffle seems to merely contribute to this nostalgic escapism.

15 September 2021

Ian Jackson: Get source to Debian packages only via dgit; "official" git links are beartraps

tl;dr dgit clone sourcepackage gets you the source code, as a git tree, in ./sourcepackage. cd into it and dpkg-buildpackage -uc -b. Do not use: "VCS" links on official Debian web pages like tracker.debian.org; "debcheckout"; searching Debian's gitlab (salsa.debian.org). These are good for Debian experts only. If you use Debian's "official" source git repo links you can easily build a package without Debian's patches applied.[1] This can even mean missing security patches. Or maybe it can't even be built in a normal way (or at all). OMG WTF BBQ, why? It's complicated. There is History. Debian's "most-official" centralised source repository is still the Debian Archive, which is a system based on tarballs and patches. I invented the Debian source package format in 1992/3 and it has been souped up since, but it's still tarballs and patches. This system is, of course, obsolete, now that we have modern version control systems, especially git. Maintainers of Debian packages have invented ways of using git anyway, of course. But this is not standardised. There is a bewildering array of approaches. The most common approach is to maintain git tree containing a pile of *.patch files, which are then often maintained using quilt. Yes, really, many Debian people are still using quilt, despite having git! There is machinery for converting this git tree containing a series of patches, to an "official" source package. If you don't use that machinery, and just build from git, nothing applies the patches. [1] This post was prompted by a conversation with a friend who had wanted to build a Debian package, and didn't know to use dgit. They had got the source from salsa via a link on tracker.d.o, and built .debs without Debian's patches. This not a theoretical unsoundness, but a very real practical risk. Future is not very bright In 2013 at the Debconf in Vaumarcus, Joey Hess, myself, and others, came up with a plan to try to improve this which we thought would be deployable. (Previous attempts had failed.) Crucially, this transition plan does not force change onto any of Debian's many packaging teams, nor onto people doing cross-package maintenance work. I worked on this for quite a while, and at a technical level it is a resounding success. Unfortunately there is a big limitation. At the current stage of the transition, to work at its best, this replacement scheme hopes that maintainers who update a package will use a new upload tool. The new tool fits into their existing Debian git packaging workflow and has some benefits, but it does make things more complicated rather than less (like any transition plan must, during the transitional phase). When maintainers don't use this new tool, the standardised git branch seen by users is a compatibility stub generated from the tarballs-and-patches. So it has the right contents, but useless history. The next step is to allow a maintainer to update a package without dealing with tarballs-and-patches at all. This would be massively more convenient for the maintainer, so an easy sell. And of course it links the tarballs-and-patches to the git history in a proper machine-readable way. We held a "git packaging requirements-gathering session" at the Curitiba Debconf in 2019. I think the DPL's intent was to try to get input into the git workflow design problem. The session was a great success: my existing design was able to meet nearly everyone's needs and wants. The room was obviously keen to see progress. The next stage was to deploy tag2upload. I spoke to various key people at the Debconf and afterwards in 2019 and the code has been basically ready since then. Unfortunately, deployment of tag2upload is mired in politics. It was blocked by a key team because of unfounded security concerns; positive opinions from independent security experts within Debian were disregarded. Of course it is always hard to get a team to agree to something when it's part of a transition plan which treats their systems as an obsolete setup retained for compatibility. Current status If you don't know about Debian's git packaging practices (eg, you have no idea what "patches-unapplied packaging branch without .pc directory" means), and don't want want to learn about them, you must use dgit to obtain the source of Debian packages. There is a lot more information and detailed instructions in dgit-user(7). Hopefully either the maintainer did the best thing, or, if they didn't, you won't need to inspect the history. If you are a Debian maintainer, you should use dgit push-source to do your uploads. This will make sure that users of dgit will see a reasonable git history.
edited 2021-09-15 14:48 Z to fix a typo


comment count unavailable comments

14 September 2021

Joachim Breitner: A Candid explainer: Quirks

This is the fifth and final post in a series about the interface description language Candid. If you made it this far, you now have a good understanding of what Candid is, what it is for and how it is used. For this final post, I ll put the spotlight on specific aspects of Candid that are maybe surprising, or odd, or quirky. This section will be quite opinionated, and could maybe be called what I d do differently if I d re-do the whole thing . Note that these quirks are not serious problems, and they don t invalidate the overall design. I am writing this up not to discourage the use of Candid, but merely help interested parties to understand it better.

References in the wire format When the work on Candid began at DFINITY, the Internet Computer was still far away from being a thing, and many fundamental aspects about it were still in the air. I particular, there was still talk about embracing capabilities as a core feature of the application model, which would be implemented as opaque references on the system level, likely building on WebAssembly s host reference type proposal (which only landed recently), and could be used to model access permissions, custom tokens and many other things. So Candid is designed with that in mind, and you ll find that its wire format is not just a type table and a value table, but actually
a triple (T, M, R), where T ( type ) and M ( memory ) are sequences of bytes and R ( references ) is a sequence of references.
Also the wire format for values of function service tyeps have an extra byte to distinguish between public references (represented by a principal and possible a method name in the data part), and these opaque references. Alas, references never made it into the Internet Computer, so all Candid implementations simply ignore that part of the specification. But it s still in the spec, and if it confused you before, now you know why.

Hashed field names Candid record and variant types look like they have textual field names:
type T = record   syndactyle : nat; trustbuster: bool  
But that is actually only true superficially. The wire format for Candid only stores hashes of field names. So the above is actually equivalent to
type T = record   4260381820 : nat; 3504418361 : bool  
or, for that matter, to
type T = record   destroys : bool; rectum : nat  
(Yes, I used an english word list to find these hash collisions. There aren t that many actually.) The benefit of such hashing is that the messages are a bit smaller in most (not all) cases, but it is a big annoyance for dynamic uses of Candid. It s the reason why tools like dfx, if they don t know the Candid interface of a service, will print the result with just the numerical hash, letting you guess which field is which. It also complicates host languages that derive Candid types from the host language, like Motoko, as some records (e.g. record trustbuster: bool; destroys : int ) with field name hash collisions can not be represented in Candid, and either the host language s type system needs to be Candid aware now (as is the case of Motoko), or serialization/deserialization will fail at runtime, or odd bugs can happen. (More discussion of this issue).

Tuples Many languages have a built-in notion of a tuple type (e.g. (Int, Bool)), but Candid does not have such a type. The only first class product type is records. This means that tuples have to encoded as records somehow. Conveniently(?) record fields are just numbers after all, so the type (Int, Bool) would be mapped to the type
record   0 : int; 1 : bool  
So tuples can be expressed. But from my experience implementing the type mappings for Motoko and Haskell this is causing headaches. To get a good experience when importing from Candid, the tools have to try to reconstruct which records may have been tuples originally, and turn them into tuples. The main argument for the status quo is that Candid types should be canonical, and there should not be more than one product type, and records are fine, and there needs to be no anonymous product type. But that has never quite resonated with me, given the practical reality of tuple types in many host languages.

Argument sequences Did I say that Candid does not have tuple types? Turns out it does, sort of. There is no first class anonymous product, but since functions take sequences of arguments and results, there is a tuple type right there:
func foo : (bool, int) -> (int, bool)
Again, I found that ergonomic interaction with host languages becomes relatively unwieldy by requiring functions to take and return sequences of values. This is especially true for languages where functions take one argument value or return one result type (the latter being very common). Here, return sequences of length one are turned into that type directly, longer argument sequences turn into the host language s tuple type, and nullary argument sequences turn into the idiomatic unit type. But this means that the types (int, bool) -> () and (record 0: int, 1: bool ) -> () may be mapped to the same host language type, which causes problems when you hope to encode all necessary Candid type information in the host language. Another oddity with argument and result sequences is that you can give names to the entries, e.g. write
func hello : (last_name : text; first_name : text) -> ()
but these names are completely ignored! So while this looks like you can, for example, add new optional arguments in the middle, such as
func hello : (last_name : text; middle_name: opt text, first_name : text) -> ()
without breaking clients, this does not have the effect you think it has and will likely break. My suggestion is to never put names on function arguments and result values in Candid interfaces, and for anything that might be extended with new fields or where you want to name the arguments, use a single record type as the only argument:
func hello : (record   last_name : text; first_name : text ) -> ()
This allows you to add and remove arguments more easily and reliably.

Type shorthands The Candid specification defines a system of types, and then adds a number of syntactic short-hands . For example, if you write blob in a Candid type description, it ought to means the same as vec nat8. My qualm with that is that it doesn t always mean the same. A Candid type description is interpreted by a number of, say, consumers . Two such consumers are part of the Candid specification:
  • The specification that defines the wire format for that type
  • The upgrading (subtyping) rules
But there are more! For every host language, there is some mapping from Candid types to host language types, and also generic tools like Candid UI are consumers of the type algebra. If these were to take the Candid specification as gospel, they would be forced to treat blob and vec nat8 the same, but that would be quite unergonomic and might cause performance regressions (most language try to map blob to some compact binary data type, while vec t tends to turn into some form of array structure). So they need to be pragmatic and treat blob and vec nat8 differently. But then, for all practical purposes, blob is not just a short-hand of vec nat8. They are different types that just happens to have the same wire representations and subtyping relations. This affects not just blob, but also tuples (record blob; int; bool ) and field names , as discussed above.

The value text format For a while after defining Candid, the only implementation was in Motoko, and all the plumbing was automatic, so there was never a need for users to to explicitly handle Candid values, as all values were Motoko values. Still, for debugging and testing and such things, we eventually needed a way to print out Candid values, so the text format was defined ( To enable convenient debugging, the following grammar specifies a text format for values ). But soon the dfx tool learned to talk to canisters, and now users needed to enter Candid value on the command line, possibly even when talking to canisters for which the interface was not known to dfx. And, sure enough, the textual interface defined in the Candid spec was used. Unfortunately, since it was not designed for that use case, it is rather unwieldy:
  • It is quite verbose. You have to write record , not just . Vectors are written vec ; instead of some conventional syntax like [ , ]. Variants are written as variant error = " " with braces that don t any value here, and something like #error " " might have worked as well. With a bit more care, a more concise and ergonomic syntax might have been possible.
  • It wasn t designed to be sufficient to create a Candid value from it. If you write 5 it s unclear whether that s a nat or an int16 or what (and all of these have different wire representations). Type annotations were later added, but are relatively unwieldy, and don t cover all cases (e.g. a service reference with a recursive type cannot be represented in the textual format at the moment).
  • Not really the fault of the textual format, but some useful information about the types is not reflected in the type description that s part of the wire format. In particular not the field names, and whether a value was intended to be binary data (blob) or a list of small numbers (vec nat8), so pretty-printing such values requires guesswork. The Haskell library even tries to brute-force the hash to guess the field name, if it is short or in a english word list!
In hindsight I think it was too optimistic to assume that correct static type information is always available, and instead of actively trying to discourage dynamic use, Candid might be better if we had taken these (unavoidable?) use cases into account.

Custom wire format At the beginning of this post, I have a Candid is list. The list is relatively long, and the actual wire format is just one bullet point. Yes, defining a wire format that works isn t rocket science, and it was easiest to just make one up. But since most of the interesting meat of Candid is in other aspects (subtyping rules, host language integration), I wonder if it would have been better to use an existing, generic wire format, such as CBOR, and build Candid as a layer on top. This would give us plenty of tools and libraries to begin with. And maybe it would have reduced barrier of entry for developers, which now led to the very strange situation that DFINITY advocates for the universal use of Candid on the Internet Computer, so that all services can smoothly interact, but two of the most important services on the Internet Computer (the registry and the ledger) use Protobuf as their primary interface format, with Candid interfaces missing or an afterthought.

Sideways Interface Evolution This is not a quirk of Candid itself, but rather an idiom of how you can use Candid that emerged from our solution for record extensions and upgrades. Consider our example from before, a service with interface
service   add_user : (record   login : text; name : text  ) -> ()  
where you want to add an age field, which should be a number. The official way of doing that is to add that field with an optional type:
service   add_user : (record   login : text; name : text; age : opt nat  ) -> ()  
As explained above, this will not break old clients, as the decoder will treat a missing argument as null. So far so good. But often when adding such a field you don t want to bother new clients with the fact that this age was, at some point in the past, not there yet. And you can do that! The trick is to distinguish between the interface you publish and the interface you implement. You can (for example in your documentation) state that the interface is
service   add_user : (record   login : text; name : text; age : nat  ) -> ()  
which is not a subtype of the old type, but it is the interface you want new clients to work with. And then your implementation uses the type with opt nat. Calls from old clients will come through as null, and calls from new clients will come through as opt 42. We can see this idiom used in the Management Canister of the Internet Computer. The current documented interface only mentions a controllers : vec principal field in the settings, but the implementation still can handle both the old controller : principal and the new controllers field. It s probably advisable to let your CI system check that new versions of your service continue to implement all published interfaces, including past interfaces. But as long as the actual implementation s interface is a subtype of all interfaces ever published, this works fine. This pattern is related to when your service implements, say, http_request (so its implemented interface is a subtype of that common interface), but does not include that method in the published documentation (because clients of your service don t need to call it).

Self-describing Services As you have noticed, Candid was very much designed assuming that all parties always have the service type of services they want to interact with. But the Candid specification does not define how one can obtain the interface of a given service, and there isn t really a an official way to do that on the Internet Computer. That is unfortunate, because many interesting features depend on that: Such as writing import C "ic:7e6iv-biaaa-aaaaf-aaada-cai" in your Motoko program, and having it s type right there. Or tools like ic.rocks, that allow you to interact with any canister right there. One reason why we don t really have that feature yet is because of disagreements about how dynamic that feature should be. Should you be able to just ask the canister for its interface (and allow the canister to vary the response, for example if it can change its functionality over time, even without changing the actual wasm code)? Or is the interface a static property of the code, and one should be able to query the system for that data, without the canister s active involvement. Or, completely different, should interfaces be distributed out of band, maybe together with API documentation, or in some canister registry somewhere else? I always leaned towards the first of these options, but not convincingly enough. The second options requires system assistance, so more components to change, more teams to be involved that maybe intrinsically don t care a lot about this feature. And the third might have emerged as the community matures and builds such infrastructure, but that did not happen yet. In the end I sneaked in an implementation of the first into Motoko, arguing that even if we don t know yet how this feature will be implemented eventually, we all want the feature to exist somehow, and we really really want to unblock all the interesting applications it enables (e.g. Candid UI). That s why every Motoko canister, and some rust canisters too, implements a method
__get_candid_interface_tmp_hack : () -> (text)
that one can use to get the Candid interface file. The name was chosen to signal that this may not be the final interface, but like all good provisional solutions, it may last longer than intended. If that s the case, I m not sorry. This concludes my blog post series about Candid, for now. If you want to know more, feel free to post your question on the DFINTY developer forum, and I ll probably answer.

13 September 2021

Bits from Debian: New Debian Developers and Maintainers (July and August 2021)

The following contributors got their Debian Developer accounts in the last two months: The following contributors were added as Debian Maintainers in the last two months: Congratulations!

10 September 2021

Enrico Zini: A nightmare of confcalls and microphones

I had this nightmare where I had a very, very important confcall. I joined with Chrome. Chrome said Failed to access your microphone - Cannot use microphone for an unknown reason. Could not start audio source. I joined with Firefox. Firefox chose Monitor of Built-in Audio Analog Stereo as a microphone, and did not let me change it. Not in the browser, not in pavucontrol. I joined with the browser on my phone, and the webpage said This meeting needs to use your microphone and camera. Select *Allow* when your browser asks for permissions. But the question never came. I could hear people talking. I had very important things to say. I tried typing them in the chat window, but they weren't seeing it. The meeting ended. I was on the verge of tears.
Tell me, Mr. Anderson, what good is a phone call when you are unable to speak?
Since this nightmare happened for real, including the bit about tears in the end, let's see that it doesn't happen again. I should now have three working systems, which hopefully won't all break again all at the same time. Fixing Chrome I can reproduce this reliably, on Bullseye's standard Chromium 90.0.4430.212-1, just launched on an empty profile, no extensions. The webpage has camera and microphone allowed. Chrome doesn't show up in the recording tab of pulseaudio. Nothing on Chrome's stdout/stderr. JavaScript console has:
Logger.js:154 2021-09-10Txx:xx:xx.xxxZ [features/base/tracks] Failed to create local tracks
Array(2)
DOMException: Could not start audio source
I found the answer here:
I had the similar problem once with chromium. i could solve it by switching in preferences->microphone-> from "default" to "intern analog stereo".
Opening the little popup next to the microphone/mute button allows choosing other microphones, which work. Only "Same as system (Default)" does not work. Fixing Firefox I have firefox-esr 78.13.0esr-1~deb11u1. In Jitsi, microphone selection is disabled on the toolbar and in the settings menu. In pavucontrol, changing the recording device for Firefox has no effect. If for some reason the wrong microphone got chosen, those are not ways of fixing it. What I found works is to click on the camera permission icon, remove microphone permission, then reload the page. At that point Firefox will ask for permission again, and that microphone selection seems to work. Relevant bugs: on Jitsi and on Firefox. Since this is well known (once you find the relevant issues), I'd have appreciated Jitsi at least showing a link to an explanation of workarounds on Firefox, instead of just disabling microphone selection. Fixing Jitsi on the phone side I really don't want to preemptively give camera and microphone permissions to my phone browser. I noticed that there's the Jitsi app on F-Droid and much as I hate to use an app when a website would work, at least in this case it's a way to keep the permission sets separate, so I installed that. Fixing pavucontrol? I tried to find out why I can't change input device for FireFox on pavucontrol. I only managed to find an Ask Ubuntu question with no answer and a Unix StackExchange question with no answer.

5 September 2021

Reproducible Builds: Reproducible Builds in August 2021

Welcome to the latest report from the Reproducible Builds project. In this post, we round up the important things that happened in the world of reproducible builds in August 2021. As always, if you are interested in contributing to the project, please visit the Contribute page on our website.
There were a large number of talks related to reproducible builds at DebConf21 this year, the 21st annual conference of the Debian Linux distribution (full schedule):
PackagingCon (@PackagingCon) is new conference for developers of package management software as well as their related communities and stakeholders. The virtual event, which is scheduled to take place on the 9th and 10th November 2021, has a mission is to bring different ecosystems together: from Python s pip to Rust s cargo to Julia s Pkg, from Debian apt over Nix to conda and mamba, and from vcpkg to Spack we hope to have many different approaches to package management at the conference . A number of people from reproducible builds community are planning on attending this new conference, and some may even present. Tickets start at $20 USD.
As reported in our May report, the president of the United States signed an executive order outlining policies aimed to improve the cybersecurity in the US. The executive order comes after a number of highly-publicised security problems such as a ransomware attack that affected an oil pipeline between Texas and New York and the SolarWinds hack that affected a large number of US federal agencies. As a followup this month, however, a detailed fact sheet was released announcing a number large-scale initiatives and that will undoubtedly be related to software supply chain security and, as a result, reproducible builds.
Lastly, We ran another productive meeting on IRC in August (original announcement) which ran for just short of two hours. A full set of notes from the meeting is available.

Software development kpcyrd announced an interesting new project this month called I probably didn t backdoor this which is an attempt to be:
a practical attempt at shipping a program and having reasonably solid evidence there s probably no backdoor. All source code is annotated and there are instructions explaining how to use reproducible builds to rebuild the artifacts distributed in this repository from source. The idea is shifting the burden of proof from you need to prove there s a backdoor to we need to prove there s probably no backdoor . This repository is less about code (we re going to try to keep code at a minimum actually) and instead contains technical writing that explains why these controls are effective and how to verify them. You are very welcome to adopt the techniques used here in your projects. ( )
As the project s README goes on the mention: the techniques used to rebuild the binary artifacts are only possible because the builds for this project are reproducible . This was also announced on our mailing list this month in a thread titled i-probably-didnt-backdoor-this: Reproducible Builds for upstreams. kpcyrd also wrote a detailed blog post about the problems surrounding Linux distributions (such as Alpine and Arch Linux) that distribute compiled Python bytecode in the form of .pyc files generated during the build process.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made a number of changes, including releasing version 180), version 181) and version 182) as well as the following changes:
  • New features:
    • Add support for extracting the signing block from Android APKs. [ ]
    • If we specify a suffix for a temporary file or directory within the code, ensure it starts with an underscore (ie. _ ) to make the generated filenames more human-readable. [ ]
    • Don t include short GCC lines that differ on a single prefix byte either. These are distracting, not very useful and are simply the strings(1) command s idea of the build ID, which is displayed elsewhere in the diff. [ ][ ]
    • Don t include specific .debug-like lines in the ELF-related output, as it is invariably a duplicate of the debug ID that exists better in the readelf(1) differences for this file. [ ]
  • Bug fixes:
    • Add a special case to SquashFS image extraction to not fail if we aren t the superuser. [ ]
    • Only use java -jar /path/to/apksigner.jar if we have an apksigner.jar as newer versions of apksigner in Debian use a shell wrapper script which will be rejected if passed directly to the JVM. [ ]
    • Reduce the maximum line length for calculating Wagner-Fischer, improving the speed of output generation a lot. [ ]
    • Don t require apksigner in order to compare .apk files using apktool. [ ]
    • Update calls (and tests) for the new version of odt2txt. [ ]
  • Output improvements:
    • Mention in the output if the apksigner tool is missing. [ ]
    • Profile diffoscope.diff.linediff and specialize. [ ][ ]
  • Logging improvements:
    • Format debug-level messages related to ELF sections using the diffoscope.utils.format_class. [ ]
    • Print the size of generated reports in the logs (if possible). [ ]
    • Include profiling information in --debug output if --profile is not set. [ ]
  • Codebase improvements:
    • Clarify a comment about the HUGE_TOOLS Python dictionary. [ ]
    • We can pass -f to apktool to avoid creating a strangely-named subdirectory. [ ]
    • Drop an unused File import. [ ]
    • Update the supported & minimum version of Black. [ ]
    • We don t use the logging variable in a specific place, so alias it to an underscore (ie. _ ) instead. [ ]
    • Update some various copyright years. [ ]
    • Clarify a comment. [ ]
  • Test improvements:
    • Update a test to check specific contents of SquashFS listings, otherwise it fails depending on the test systems user ID to username passwd(5) mapping. [ ]
    • Assign seen and expected values to local variables to improve contextual information in failed tests. [ ]
    • Don t print an orphan newline when the source code formatting test passes. [ ]

In addition Santiago Torres Arias added support for Squashfs version 4.5 [ ] and Felix C. Stegerman suggested a number of small improvements to the output of the new APK signing block [ ]. Lastly, Chris Lamb uploaded python-libarchive-c version 3.1-1 to Debian experimental for the new 3.x branch python-libarchive-c is used by diffoscope.

Distribution work In Debian, 68 reviews of packages were added, 33 were updated and 10 were removed this month, adding to our knowledge about identified issues. Two new issue types have been identified too: nondeterministic_ordering_in_todo_items_collected_by_doxygen and kodi_package_captures_build_path_in_source_filename_hash. kpcyrd published another monthly report on their work on reproducible builds within the Alpine and Arch Linux distributions, specifically mentioning rebuilderd, one of the components powering reproducible.archlinux.org. The report also touches on binary transparency, an important component for supply chain security. The @GuixHPC account on Twitter posted an infographic on what fraction of GNU Guix packages are bit-for-bit reproducible: Finally, Bernhard M. Wiedemann posted his monthly reproducible builds status report for openSUSE.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: Elsewhere, it was discovered that when supporting various new language features and APIs for Android apps, the resulting APK files that are generated now vary wildly from build to build (example diffoscope output). Happily, it appears that a patch has been committed to the relevant source tree. This was also discussed on our mailing list this month in a thread titled Android desugaring and reproducible builds started by Marcus Hoffmann.

Website and documentation There were quite a few changes to the Reproducible Builds website and documentation this month, including:
  • Felix C. Stegerman:
    • Update the website self-build process to not use the buster-backports suite now that Debian Bullseye is the stable release. [ ]
  • Holger Levsen:
    • Add a new page documenting various package rebuilder solutions. [ ]
    • Add some historical talks and slides from DebConf20. [ ][ ]
    • Various improvements to the history page. [ ][ ][ ]
    • Rename the Comparison protocol documentation category to Verification . [ ]
    • Update links to F-Droid documentation. [ ]
  • Ian Muchina:
    • Increase the font size of titles and de-emphasize event details on the talk page. [ ]
    • Rename the README file to README.md to improve the user experience when browsing the Git repository in a web browser. [ ]
  • Mattia Rizzolo:
    • Drop a position:fixed CSS statement that is negatively affecting with some width settings. [ ]
    • Fix the sizing of the elements inside the side navigation bar. [ ]
    • Show gold level sponsors and above in the sidebar. [ ]
    • Updated the documentation within reprotest to mention how ldconfig conflicts with the kernel variation. [ ]
  • Roland Clobus:
    • Added a ticket number for the issue with the live Cinnamon image and diffoscope. [ ]

Testing framework The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
  • Holger Levsen:
    • Debian-related changes:
      • Make a large number of changes to support the new Debian bookworm release, including adding it to the dashboard [ ], start scheduling tests [ ], adding suitable Apache redirects [ ] etc. [ ][ ][ ][ ][ ]
      • Make the first build use LANG=C.UTF-8 to match the official Debian build servers. [ ]
      • Only test Debian Live images once a week. [ ]
      • Upgrade all nodes to use Debian Bullseye [ ] [ ]
      • Update README documentation for the Debian Bullseye release. [ ]
    • Other changes:
      • Only include rsync output if the $DEBUG variable is enabled. [ ]
      • Don t try to install mock, a tool used to build Fedora packages some time ago. [ ]
      • Drop an unused function. [ ]
      • Various documentation improvements. [ ][ ]
      • Improve the node health check to detect zombie jobs. [ ]
  • Jessica Clarke (FreeBSD-related changes):
    • Update the location and branch name for the main FreeBSD Git repository. [ ]
    • Correctly ignore the source tarball when comparing build results. [ ]
    • Drop an outdated version number from the documentation. [ ]
  • Mattia Rizzolo:
    • Block F-Droid jobs from running whilst the setup is running. [ ]
    • Enable debugging for the rsync job related to Debian Live images. [ ]
    • Pass BUILD_TAG and BUILD_URL environment for the Debian Live jobs. [ ]
    • Refactor the master_wrapper script to use a Bash array for the parameters. [ ]
    • Prefer YAML s safe_load() function over the unsafe variant. [ ]
    • Use the correct variable in the Apache config to match possible existing files on disk. [ ]
    • Stop issuing HTTP 301 redirects for things that not actually permanent. [ ]
  • Roland Clobus (Debian live image generation):
    • Increase the diffoscope timeout from 120 to 240 minutes; the Cinnamon image should now be able to finish. [ ]
    • Use the new snapshot service. [ ]
    • Make a number of improvements to artifact handling, such as moving the artifacts to the Jenkins host [ ] and correctly cleaning them up at the right time. [ ][ ][ ]
    • Where possible, link to the Jenkins build URL that created the artifacts. [ ][ ]
    • Only allow only one job to run at the same time. [ ]
  • Vagrant Cascadian:
    • Temporarily disable armhf nodes for DebConf21. [ ][ ]

Lastly, if you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. You can get in touch with us via:

1 September 2021

Utkarsh Gupta: FOSS Activites in August 2021

Here s my (twenty-third) monthly but brief update about the activities I ve done in the F/L/OSS world.

Debian
This was my 32nd month of actively contributing to Debian. I became a DM in late March 2019 and a DD on Christmas 19! \o/ Tough month but I mostly spent on it churning through the immense backlog. But that somewhat backfired and I have even more backlog than ever. :D Anyway, I did the following stuff in Debian:

Uploads and bug fixes:
  • ruby3.0 (3.0.0-2) - Upload to unstable! \o/

Other $things:
  • Mentoring for newcomers.
  • Moderation of -project mailing list.

Ubuntu
This was my 7th month of actively contributing to Ubuntu. Now that I ve joined Canonical to work on Ubuntu full-time, there s a bunch of things I do! \o/ I mostly worked on different things, I guess. But mostly on packaging keylime and some Google Agents upload(s) and SRU(s). Also did a lot of reviewing, et al. I was too lazy to maintain a list of things I worked on so there s no concrete list atm. Maybe I ll get back to this section later or will start to list stuff from next month onward, as I ve been doing before. :D

Debian (E)LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the Jessie release (+2 years after LTS support). This was my twenty-third month as a Debian LTS and eleventh month as a Debian ELTS paid contributor.
I was assigned 23.75 hours for LTS and 40.00 hours for ELTS and worked on the following things:
(however, I only worked for 23.75h on ELTS work, thereby, carrying the rest to next month)

LTS CVE Fixes and Announcements:

ELTS CVE Fixes and Announcements:

Other (E)LTS Work:
  • Front-desk duty from 26-07 until 01-08 and from 30-08 until 05-09 for both LTS and ELTS.
  • Triaged haproxy, ntfs-3g, and cyrus-imapd, and exiv2, ffmpeg, git, gpac, inetutils, mc, modsecurity-crs, node-object-path, php-pear, systemd-cron, node-tar, ruby2.3, gst-plugins-bad0.10, jsoup, libxstream-java, qemu, tomcat7, ruby2.1, prototypejs, pillow, cpio, and qtbase-opensource-src, and amd64-microcode.
  • Mark CVE-2021-39240/haproxy as not-affected for stretch and jessie.
  • Mark CVE-2021-39241/haproxy as not-affected for stretch and jessie.
  • Mark CVE-2021-39242/haproxy as not-affected for stretch and jessie.
  • Mark CVE-2021-33582/cyrus-imapd as no-dsa for stretch.
  • Mark CVE-2020-18771/exiv2 as no-dsa for exiv2 for stretch.
  • Mark CVE-2020-18899/exiv2 as no-dsa for exiv2 for stretch.
  • Mark CVE-2021-38171/ffmpeg as postponed for stretch.
  • Mark CVE-2021-40330/git as no-dsa for stretch and jessie.
  • Mark CVE-2020-19481/gpac as ignored for stretch.
  • Mark CVE-2021-40491/inetutils as no-dsa for stretch.
  • Mark CVE-2021-36370/mc as no-dsa for stretch and jessie.
  • Mark CVE-2021-35368/modsecurity-crs as no-dsa for stretch.
  • Mark CVE-2021-23434/node-object-path as end-of-life for stretch.
  • Mark CVE-2021-32610/php-pear as no-dsa for stretch.
  • Mark CVE-2017-9525/systemd-cron as no-dsa for stretch.
  • Mark CVE-2021-37701/node-tar as end-of-life for stretch.
  • Mark CVE-2021-37712/node-tar as end-of-life in stretch.
  • Mark CVE-2021-3750/qemu as postponsed for jessie.
  • Mark CVE-2021-27511/prototypejs as postponsed for jessie.
  • Mark CVE-2021-23437/pillow as postponed for stretch and jessie.
  • Auto EOL ed gpac, cacti, openscad, cgal, cyrus-imapd-2.4, libsolv, mosquitto, atomicparsley, gtkpod, node-tar, libapache2-mod-auth-openidc, neutron, inetutils and linux for jessie.
  • Drop cpio from ela-needed; open issues don t warrant an ELA.
  • Attended monthly Debian LTS meeting.
  • Answered questions (& discussions) on IRC (#debian-lts and #debian-elts).
  • General and other discussions on LTS private and public mailing list.

Until next time.
:wq for today.

23 August 2021

Vincent Bernat: ThinkPad X1 Carbon (Gen 7): 2 years later

Two years ago, I replaced my ThinkPad X1 Carbon 2014 with the latest generation. The new configuration embeds an Intel Core i7-8565U, 16 Gib of RAM, a 1 Tib NVMe disk, and a WQHD display (2560 1440). I did not ask for a WWAN card. I think it is easier and more reliable to use the wifi hotspot feature of a phone instead: no unreliable firmware and unsupported drivers.1 Here is my opinion on this model.
ThinkPad X1 Carbon 7th Gen with the lid closed
ThinkPad X1 Carbon with its lid closed
While the second generation got a very odd keyboard, this one got a classic one with a full row of function keys. I don t know if my model was defective, but the keyboard skips one keypress from time to time. I have got used to it, but the space key still has a hard time registering when hitting it with my right thumb. The travel course is also shorter and it is less comfortable to type on it than it was on the 2014 version. The trackpoint2 works well. The physical buttons are a welcome addition. I am only using the trackpad for scrolling with the two-finger gesture.
Keyboard of the X1 Carbon 7th Gen
Keyboard with an ANSI QWERTY layout (aka English EU for Lenovo). The LEDs on the speaker and microphone keys work out of the box on Linux.
The screen does not suffer from ghosting effects like the one in my previous ThinkPad. To avoid leaving marks on the screen, I use a piece of cloth on top of the keyboard when closing the laptop. The 720p webcam has a built-in mechanical cover. Its quality is not great, but it does the job.
X1 Carbon 7th Gen
ThinkPad X1 Carbon with its lid opened
Battery life effortlessly reaches about 8 hours on a full charge. This is the main reason this laptop feels like a solid upgrade compared to the previous one: no need to carry a power brick during the day. This is the first Lenovo model with a sound card requiring the Sound Open Firmware. Without the appropriate firmware and the related userspace components (ALSA UCM and PulseAudio), the microphone does not work. If everything is set up correctly, the speakers produce a very decent sound, better than the 2014 model. It should now work out-of-the-box with Debian Bullseye. Just install the firmware-sof-signed package.3 The BIOS can be updated directly from Linux, thanks to the Linux Vendor Firmware Service. I was using the ThinkPad USB-C Dock Gen 2 as a docking station. Everything works out-of-the-box. However, from time to time, I got issues reliably getting an image on the two screens. I was using a couple of 10-year LG monitors with DVI connectors, so I relied on DP HDMI DVI adapters. This may have been the source of some of the problems.
ThinkPad USB-C Dock Gen 2
ThinkPad USB-C Dock Gen 2 with network, keyboard, mouse, power and two screens plugged

In summary, this is a fine laptop plagued with a bad keyboard. I did appreciate the good battery life and the fact there is still one HDMI and two USB-A ports, so I didn t need to travel with dongles. I was disappointed by how small the performance gap was with my 5-year older laptop. I am unsure to get a Lenovo for the next one: HiDPI screens are mostly unavailable and current prices are high. Other possibilities include: Until then, I am using my previous ThinkPad.

  1. The seventh generation moved from Sierra Wireless to Fibocom. While Linux support for modems is good, thanks to ModemManager, they are usually driven over the USB bus. The Fibocom L850-GL can use either the USB bus or the PCI bus but Lenovo s BIOS blacklists the former. It took quite some time to find how to switch it to USB after booting. A PCI driver is in progress.
  2. I did replace the trackpoint with a low-profile one from SaotoTech.
  3. The package is in non-free, despite being open-source: many platforms require the firmware to be signed by Intel.

17 August 2021

Russ Allbery: Review: Black Sun

Review: Black Sun, by Rebecca Roanhorse
Series: Between Earth and Sky #1
Publisher: Saga Press
Copyright: October 2020
ISBN: 1-5344-3769-X
Format: Kindle
Pages: 454
Serapio has been crafted and trained to be the vessel for a god. He grew up in Obregi land, far from his ancestral home, but he will return to Tova at the appropriate time and carry the hopes of the Carrion Crow clan with him. Xiala is a ship captain, a woman, and a Teek. That means she's a target. Teek have magic, which makes them uncanny and dangerous. They're also said to carry that magic in their bones, which makes them valuable in ways that are not pleasant for the Teek. Running afoul of the moral codes of Cuecola is therefore even more dangerous to her than it would be to others, which is why she accepts a bargain to run errands for a local lord for twelve years, paid at the end of that time with ownership of a ship and crew. The first task: ferry a strange man to the city of Tova. Meanwhile, in Tova, the priestess Naranpa has clawed her way to the top of the Sky Made hierarchy from an inauspicious beginning in the poor district of Coyote's Maw. She's ruthlessly separated herself from her despised beginnings and focused her attention on calming Tova in advance of the convergence, a rare astronomical alignment at the same time as the winter solstice. But Carrion Crow holds a deep-seated grudge at their slaughter by the priesthood during the Night of Knives, and Naranpa's position atop the religious order that partly rules Tova's fractious politics is more precarious than she thinks. I am delighted that more fantasy is drawing on mythologies and histories other than the genre default of western European. It's long overdue for numerous reasons and a trend to be rewarded. But do authors writing fantasy in English who reach for Mesoamerican cultures have to gleefully embrace the excuse to add more torture? I'm developing an aversion to this setting (which I do not want to do!) because every book seems to feature human sacrifice, dismemberment, or some other horror show. Roanhorse at least does not fill the book with that (there's lingering child abuse but nothing as sickening as the first chapter), but that makes the authorial choice to make the torture one's first impression of this book even odder. Our introduction to Serapio is a scene that I would have preferred to have never read, and I don't think it even adds much to the plot. Huge warnings for people who don't want to read about a mother torturing her son, or about eyes in that context. Once past that introduction, Black Sun settles into a two-thread fantasy, one following Xiala and Serapio's sea voyage and the other following Naranpa and the political machinations in Tova. Both the magic systems and the political systems are different enough to be refreshing, and there are a few bits of world-building I enjoyed (a city built on top of rocks separated by deep canyons and connected with bridges, giant intelligent riding crows, everything about the Teek). My problem was that I didn't care what happened to any of the characters. Naranpa spends most of the book dithering and whining despite a backstory that should have promised more dynamic and decisive responses. The other character from Tova introduced somewhat later in the book is clearly "character whose story will appear in the next volume"; here, he's just station-keeping and representing the status quo. And while it's realistic given the plot that Serapio is an abused sociopath, that didn't mean I enjoyed reading his viewpoint or his childhood abuse. Xiala is the best character in the book by far and I was warming to the careful work she has to do to win over an unknown crew, but apparently Roanhorse was not interested in that. Instead, the focus of Xiala's characterization turns to a bad-boy romance that did absolutely nothing for me. This will be a matter of personal taste; I know this is a plot feature for many readers. But it had me rolling my eyes and turning the pages to get to something more interesting (which, sadly, was not forthcoming). It also plays heavily on magical disabled person cliches, like the blind man being the best fighter anyone has met. I did not enjoy this book very much, but there were some neat bits of world-building and I could see why other people might disagree. What pushed me into actively recommending against it (at least for now) is the publishing structure. This is the first book of a trilogy, so one can expect the major plot to not be resolved by the end of the book. But part of the contract with the reader when publishing a book series is that each volume should reach some sense of closure and catharsis. There will be cliffhangers and unanswered questions, but there should also be enough plot lines that are satisfactorily resolved to warrant publishing a book as a separate novel. There is none of that here. This is the first half (or third) of a novel. It introduces a bunch of plot lines, pulls them together, describes an intermediate crisis, and then simply stops. Not a single plot line is resolved. This is made worse by the fact this series (presumably, as I have only seen the first book) has a U-shaped plot: everything gets worse and worse until some point of crisis, and then presumably the protagonists will get their shit together and things will start to improve. I have soured on U-shaped plots since the first half of the story often feels like a tedious grind (eat your vegetables and then you can have dessert), but it's made much worse by cutting the book off at the bottom of the U. You get a volume, like Black Sun, that's all setup and horror and collapse, with no payoff or optimism. After two tries, I have concluded that Roanhorse is not for me. This is clearly a me problem rather than a Roanhorse problem, given how many other people love both Black Sun and her Sixth World series, but this is the second book of hers where I mildly enjoyed the world building but didn't care about any of the characters. Ah well, tastes will differ. Even if you get along with Roanhorse, though, I recommend against starting this book until the second half of it is published (currently scheduled for 2022). As it stands, it's a wholly unsatisfying reading experience. Followed by the not-yet-published Fevered Star. Rating: 4 out of 10

16 August 2021

Russ Allbery: Review: The Galaxy, and the Ground Within

Review: The Galaxy, and the Ground Within, by Becky Chambers
Series: Wayfarers #4
Publisher: Harper Voyager
Copyright: April 2021
ISBN: 0-06-293605-0
Format: Kindle
Pages: 325
The name of the planet Gora is the Hanto word for useless. It's a bone-dry, resource-poor planet with nothing to recommend it except that it happened to be conveniently located between five other busy systems and had well-established interspacial tunnels. Gora is therefore a transit hub: a place lots of people visit for about half a day while waiting for a departure time, but where very few people stay. It is the interstellar equivalent of an airport. Ouloo is a Laru, a species physically somewhat similar to a giant dog. She is the owner of the Five-Hop One-Stop in a habitat dome on Gora, where she lives with her child (who is not yet old enough to choose a gender). On the day when this novel begins, she's expecting three ships to dock: one Aeluon, one Quelin, and, to Ouloo's significant surprise and moderate discomfort, an Akarak. But apart from that, it's a normal day. A normal day, that is, until maintenance work on the solar satellite array leads to a Kessler syndrome collision cascade that destroys most of the communication satellites and makes it unsafe to leave the surface of the planet. Ouloo and her guests are stuck with each other for longer than they expected. In a typical SF novel, you would expect the characters to have to fix the satellite cascade, or for it to be a sign of something more nefarious. That is not the case here; the problem is handled by the Goran authorities, the characters have no special expertise, and there is no larger significance to the accident. Instead, the accident functions as storm in a very old story-telling frame: three travelers and their host and her child, trapped together by circumstance and forced to entertain each other until they can go on their way. Breaking from the formula, they do not do that primarily by telling stories to each other, although the close third-person narration that moves between the characters reveals their backgrounds over the course of the book. Instead, a lot of this book is conversation, sometimes prompted by Ouloo's kid Tupo (who I thought was a wonderfully-written tween, complete with swings between curiosity and shyness, random interests, occasionally poor impulse control, and intense but unpredictable learning interest). That leads to some conflict (and some emergencies), but, similar to Record of a Spaceborn Few, this is more of a character study book than a things-happen book. An interesting question, then, is why is this story science fiction? A similar story could be written (and has been, many times) with human travelers in a mundane inn or boarding house in a storm. None of the aliens are all that alien; despite having different body shapes and senses, you could get more variation from a group of university students. And even more than with Chambers's other books, the advanced technology is not the point and is described only enough to provide some background color and a bit of characterization. The answer, for me, is that the cognitive estrangement of non-human characters relieves my brain of the baggage that I bring to human characters and makes it easier for me to empathize with the characters as individuals rather than representatives of human archetypes. With human characters, I would be fitting them into my knowledge of history and politics, and my reaction to each decision the characters make would be influenced by the assumptions prompted by that background. I enjoy the distraction of invented worlds and invented histories in part because they're simplified compared to human histories and therefore feel more knowable and less subtle. I'm not trying to understand the political angle from which the author is writing or wondering if I'm missing a reference that's important to the story. In other words, the science fiction setting gives the narrator more power. The story tells me the important details of the background; there isn't some true history lurking beneath that I'm trying to ferret out. When that's combined with interesting physical differences, I find myself imagining what it would be like to be the various aliens, trying to insert myself into their worlds, rather than placing them in a historical or political context. That puts me in a curious and empathetic mindset, and that, in turn, is the best perspective from which to enjoy Chambers's stories. The characters in this story don't solve any large-scale problems. They do make life decisions, some quite significant, but only on a personal scale. They also don't resolve all of their suspicions and disagreements. This won't be to everyone's taste, but it's one of the things I most enjoyed about the book: it shows a small part of the lives of a collection of average individuals, none of whom are close to the levers of power and none of whom are responsible for fixing their species or galactic politics. They are responsible for their own choices, and for how their lives touch the lives of others. They can make the people they encounter happier or sadder, they can chose how to be true to their own principles, and they can make hard choices without right answers. When I describe a mainstream fiction book that way, I often find it depressing, but I came away from The Galaxy, and the Ground Within feeling better about the world and more open-hearted towards other people. I'm not sure what Chambers does to produce that reaction, so I'm not sure if it will have the same effect on other people. Perhaps part of it is that while there is some drama, her characters do not seek drama for its own sake, none of the characters are villains, and she has a way of writing sincerity that clicks with my brain. There is a scene, about two-thirds of the way through the book, where the characters get into a heated argument about politics, and for me this is the moment where you will either love this book or it will not work for you. The argument doesn't resolve anything, and yet it's one of the most perceptive, accurate, and satisfying portrayals of a political argument among normal people that I've seen in fiction. It's the sort of air-clearing conversation in which every character is blunt with both their opinion and their emotions rather than shading them for politeness. Those positions are not necessarily sophisticated or deeply philosophical, but they are deeply honest.
"And you know what? I truly don't care which of them is right so long as it fixes everything. I don't have an... an ideology. I don't know the right terms to discuss these things. I don't know the science behind any of it. I'm sure I sound silly right now. But I just want everyone to get along, and to be well taken care of. That's it. I want everybody to be happy and I do not care how we get there." She exhaled, her broad nostrils flaring. "That's how I feel about it."
I am not Ouloo, but I think she represents far more people than fiction normally realizes, and I found something deeply satisfying and revealing in seeing that position presented so clearly in the midst of a heated argument. If you like what Chambers does, I think you will like this book. If it's not for you, this is probably not the book that will change your mind, although there is a bit less hand-wavy technology to distract the people whom that distracts. The Galaxy, and the Ground Within didn't have the emotional resonance that Record of a Spaceborn Few had for me, or the emotional gut punch of A Closed and Common Orbit. But I loved every moment of reading it. This will apparently be the last novel in the Wayfarers universe, at least for the time being. Chambers will be moving on to other settings (starting with A Psalm for the Wild-Built). Rating: 8 out of 10

9 August 2021

Russ Allbery: Review: The Last Battle

Review: The Last Battle, by C.S. Lewis
Illustrator: Pauline Baynes
Series: Chronicles of Narnia #7
Publisher: Collier Books
Copyright: 1956
Printing: 1978
ISBN: 0-02-044210-6
Format: Mass market
Pages: 184
The Last Battle is the seventh and final book of the Chronicles of Narnia in every reading order. It ties together (and spoils) every previous Narnia book, so you do indeed want to read it last (or skip it entirely, but I'll get into that). In the far west of Narnia, beyond the Lantern Waste and near the great waterfall that marks Narnia's western boundary, live a talking ape named Shift and a talking donkey named Puzzle. Shift is a narcissistic asshole who has been gaslighting and manipulating Puzzle for years, convincing the poor donkey that he's stupid and useless for anything other than being Shift's servant. At the start of the book, a lion skin washes over the waterfall and into the Cauldron Pool. Shift, seeing a great opportunity, convinces Puzzle to retrieve it. The king of Narnia at this time is Tirian. I would tell you more about Tirian except, despite being the protagonist, that's about all the characterization he gets. He's the king, he's broad-shouldered and strong, he behaves in a correct kingly fashion by preferring hunting lodges and simple camps to the capital at Cair Paravel, and his close companion is a unicorn named Jewel. Other than that, he's another character like Rilian from The Silver Chair who feels like he was taken from a medieval Arthurian story. (Thankfully, unlike Rilian, he doesn't talk like he's in a medieval Arthurian story.) Tirian finds out about Shift's scheme when a dryad appears at Tirian's camp, calling for justice for the trees of Lantern Waste who are being felled. Tirian rushes to investigate and stop this monstrous act, only to find the beasts of Narnia cutting down trees and hauling them away for Calormene overseers. When challenged on why they would do such a thing, they reply that it's at Aslan's orders. The Last Battle is largely the reason why I decided to do this re-read and review series. It is, let me be clear, a bad book. The plot is absurd, insulting to the characters, and in places actively offensive. It is also, unlike the rest of the Narnia series, dark and depressing for nearly all of the book. The theology suffers from problems faced by modern literature that tries to use the Book of Revelation and related Christian mythology as a basis. And it is, most famously, the site of one of the most notorious authorial betrayals of a character in fiction. And yet, The Last Battle, probably more than any other single book, taught me to be a better human being. It contains two very specific pieces of theology that I would now critique in multiple ways but which were exactly the pieces of theology that I needed to hear when I first understood them. This book steered me away from a closed, judgmental, and condemnatory mindset at exactly the age when I needed something to do that. For that, I will always have a warm spot in my heart for it. I'm going to start with the bad parts, though, because that's how the book starts. MAJOR SPOILERS BELOW. First, and most seriously, this is a second-order idiot plot. Shift shows up with a donkey wearing a lion skin (badly), only lets anyone see him via firelight, claims he's Aslan, and starts ordering the talking animals of Narnia to completely betray their laws and moral principles and reverse every long-standing political position of the country... and everyone just nods and goes along with this. This is the most blatant example of a long-standing problem in this series: Lewis does not respect his animal characters. They are the best feature of his world, and he treats them as barely more intelligent than their non-speaking equivalents and in need of humans to tell them what to do. Furthermore, despite the assertion of the narrator, Shift is not even close to clever. His deception has all the subtlety of a five-year-old who doesn't want to go to bed, and he offers the Narnians absolutely nothing in exchange for betraying their principles. I can forgive Puzzle for going along with the scheme since Puzzle has been so emotionally abused that he doesn't know what else to do, but no one else has any excuse, especially Shift's neighbors. Given his behavior in the book, everyone within a ten mile radius would be so sick of his whining, bullying, and lying within a month that they'd never believe anything he said again. Rishda and Ginger, a Calormene captain and a sociopathic cat who later take over Shift's scheme, do qualify as clever, but there's no realistic way Shift's plot would have gotten far enough for them to get involved. The things that Shift gets the Narnians to do are awful. This is by far the most depressing book in the series, even more than the worst parts of The Silver Chair. I'm sure I'm not the only one who struggled to read through the first part of this book, and raced through it on re-reads because everything is so hard to watch. The destruction is wanton and purposeless, and the frequent warnings from both characters and narration that these are the last days of Narnia add to the despair. Lewis takes all the beautiful things that he built over six books and smashes them before your eyes. It's a lot to take, given that previous books would have treated the felling of a single tree as an unspeakable catastrophe. I think some of these problems are due to the difficulty of using Christian eschatology in a modern novel. An antichrist is obligatory, but the animals of Narnia have no reason to follow an antichrist given their direct experience with Aslan, particularly not the aloof one that Shift tries to give them. Lewis forces the plot by making everyone act stupidly and out of character. Similarly, Christian eschatology says everything must become as awful as possible right before the return of Christ, hence the difficult-to-read sections of Narnia's destruction, but there's no in-book reason for the Narnians' complicity in that destruction. One can argue about whether this is good theology, but it's certainly bad storytelling. I can see the outlines of the moral points Lewis is trying to make about greed and rapacity, abuse of the natural world, dubious alliances, cynicism, and ill-chosen prophets, but because there is no explicable reason for Tirian's quiet kingdom to suddenly turn to murderous resource exploitation, none of those moral points land with any force. The best moral apocalypse shows the reader how, were they living through it, they would be complicit in the devastation as well. Lewis does none of that work, so the reader is just left angry and confused. The book also has several smaller poor authorial choices, such as the blackface incident. Tirian, Jill, and Eustace need to infiltrate Shift's camp, and use blackface to disguise themselves as Calormenes. That alone uncomfortably reveals how much skin tone determines nationality in this world, but Lewis makes it far worse by having Tirian comment that he "feel[s] a true man again" after removing the blackface and switching to Narnian clothes. All of this drags on and on, unlike Lewis's normally tighter pacing, to the point that I remembered this book being twice the length of any other Narnia book. It's not; it's about the same length as the rest, but it's such a grind that it feels interminable. The sum total of the bright points of the first two-thirds of the book are the arrival of Jill and Eustace, Jill's one moment of true heroism, and the loyalty of a single Dwarf. The rest is all horror and betrayal and doomed battles and abject stupidity. I do, though, have to describe Jill's moment of glory, since I complained about her and Eustace throughout The Silver Chair. Eustace is still useless, but Jill learned forestcraft during her previous adventures (not that we saw much sign of this previously) and slips through the forest like a ghost to steal Puzzle and his lion costume out from the under the nose of the villains. Even better, she finds Puzzle and the lion costume hilarious, which is the one moment in the book where one of the characters seems to understand how absurd and ridiculous this all is. I loved Jill so much in that moment that it makes up for all of the pointless bickering of The Silver Chair. She doesn't get to do much else in this book, but I wish the Jill who shows up in The Last Battle had gotten her own book. The end of this book, and the only reason why it's worth reading, happens once the heroes are forced into the stable that Shift and his co-conspirators have been using as the stage for their fake Aslan. Its door (for no well-explained reason) has become a door to Aslan's Country and leads to a reunion with all the protagonists of the series. It also becomes the frame of Aslan's final destruction of Narnia and judging of its inhabitants, which I suspect would be confusing if you didn't already know something about Christian eschatology. But before that, this happens, which is sufficiently and deservedly notorious that I think it needs to be quoted in full.
"Sir," said Tirian, when he had greeted all these. "If I have read the chronicle aright, there should be another. Has not your Majesty two sisters? Where is Queen Susan?" "My sister Susan," answered Peter shortly and gravely, "is no longer a friend of Narnia." "Yes," said Eustace, "and whenever you've tried to get her to come and talk about Narnia or do anything about Narnia, she says 'What wonderful memories you have! Fancy your still thinking about all those funny games we used to play when we were children.'" "Oh Susan!" said Jill. "She's interested in nothing nowadays except nylons and lipstick and invitations. She always was a jolly sight too keen on being grown-up." "Grown-up indeed," said the Lady Polly. "I wish she would grow up. She wasted all her school time wanting to be the age she is now, and she'll waste all the rest of her life trying to stay that age. Her whole idea is to race on to the silliest time of one's life as quick as she can and then stop there as long as she can."
There are so many obvious and dire problems with this passage, and so many others have written about it at length, that I will only add a few points. First, I find it interesting that neither Lucy nor Edmund says a thing. (I would like to think that Edmund knows better.) The real criticism comes from three characters who never interacted with Susan in the series: the two characters introduced after she was no longer allowed to return to Narnia, and a character from the story that predated hers. (And Eustace certainly has some gall to criticize someone else for treating Narnia as a childish game.) It also doesn't say anything good about Lewis that he puts his rather sexist attack on Susan into the mouths of two other female characters. Polly's criticism is a somewhat generic attack on puberty that could arguably apply to either sex (although "silliness" is usually reserved for women), but Jill makes the attack explicitly gendered. It's the attack of a girl who wants to be one of the boys on a girl who embraces things that are coded feminine, and there's a whole lot of politics around the construction of gender happening here that Lewis is blindly reinforcing and not grappling with at all. Plus, this is only barely supported by single sentences in The Voyage of the Dawn Treader and The Horse and His Boy and directly contradicts the earlier books. We're expected to believe that Susan the archer, the best swimmer, the most sensible and thoughtful of the four kids has abruptly changed her whole personality. Lewis could have made me believe Susan had soured on Narnia after the attempted kidnapping (and, although left unstated, presumably eventual attempted rape) in The Horse and His Boy, if one ignores the fact that incident supposedly happens before Prince Caspian where there is no sign of such a reaction. But not for those reasons, and not in that way. Thankfully, after this, the book gets better, starting with the Dwarfs, which is one of the two passages that had a profound influence on me. Except for one Dwarf who allied with Tirian, the Dwarfs reacted to the exposure of Shift's lies by disbelieving both Tirian and Shift, calling a pox on both their houses, and deciding to make their own side. During the last fight in front of the stable, they started killing whichever side looked like they were winning. (Although this is horrific in the story, I think this is accurate social commentary on a certain type of cynicism, even if I suspect Lewis may have been aiming it at atheists.) Eventually, they're thrown through the stable door by the Calormenes. However, rather than seeing the land of beauty and plenty that everyone else sees, they are firmly convinced they're in a dark, musty stable surrounded by refuse and dirty straw. This is, quite explicitly, not something imposed on them. Lucy rebukes Eustace for wishing Tash had killed them, and tries to make friends with them. Aslan tries to show them how wrong their perceptions are, to no avail. Their unwillingness to admit they were wrong is so strong that they make themselves believe that everything is worse than it actually is.
"You see," said Aslan. "They will not let us help them. They have chosen cunning instead of belief. Their prison is only in their own minds, yet they are in that prison; and so afraid of being taken in that they cannot be taken out."
I grew up with the US evangelical version of Hell as a place of eternal torment, which in turn was used to justify religious atrocities in the name of saving people from Hell. But there is no Hell of that type in this book. There is a shadow into which many evil characters simply disappear, and there's this passage. Reading this was the first time I understood the alternative idea of Hell as the absence of God instead of active divine punishment. Lewis doesn't use the word "Hell," but it's obvious from context that the Dwarfs are in Hell. But it's not something Aslan does to them and no one wants them there; they could leave any time they wanted, but they're too unwilling to be wrong. You may have to be raised in conservative Christianity to understand how profoundly this rethinking of Hell (which Lewis tackles at greater length in The Great Divorce) undermines the system of guilt and fear that's used as motivation and control. It took me several re-readings and a lot of thinking about this passage, but this is where I stopped believing in a vengeful God who will eternally torture nonbelievers, and thus stopped believing in all of the other theology that goes with it. The second passage that changed me is Emeth's story. Emeth is a devout Calormene, a follower of Tash, who volunteered to enter the stable when Shift and his co-conspirators were claiming Aslan/Tash was inside. Some time after going through, he encounters Aslan, and this is part of his telling of that story (and yes, Lewis still has Calormenes telling stories as if they were British translators of the Arabian Nights):
[...] Lord, is it then true, as the Ape said, that thou and Tash are one? The Lion growled so that the earth shook (but his wrath was not against me) and said, It is false. Not because he and I are one, but because we are opposites, I take to me the services which thou hast done to him. For I and he are of such different kinds that no service which is vile can be done to me, and none which is not vile can be done to him. Therefore if any man swear by Tash and keep his oath for the oath's sake, it is by me that he has truly sworn, though he know it not, and it is I who reward him. And if any man do a cruelty in my name, then, though he says the name Aslan, it is Tash whom he serves and by Tash his deed is accepted. Dost thou understand, Child? I said, Lord, thou knowest how much I understand. But I said also (for the truth constrained me), Yet I have been seeking Tash all my days. Beloved, said the Glorious One, unless thy desire had been for me, thou wouldst not have sought so long and so truly. For all find what they truly seek.
So, first, don't ever say this to anyone. It's horribly condescending and, since it's normally said by white Christians to other people, usually explicitly colonialist. Telling someone that their god is evil but since they seem to be a good person they're truly worshiping your god is only barely better than saying yours is the only true religion. But it is better, and as someone who, at the time, was wholly steeped in the belief that only Christians were saved and every follower of another religion was following Satan and was damned to Hell, this passage blew my mind. This was the first place I encountered the idea that someone who followed a different religion could be saved, or that God could transcend religion, and it came with exactly the context and justification that I needed given how close-minded I was at the time. Today, I would say that the Christian side of this analysis needs far more humility, and fobbing off all the evil done in the name of the Christian God by saying "oh, those people were really following Satan" is a total moral copout. But, nonetheless, Lewis opened a door for me that I was able to step through and move beyond to a less judgmental, dismissive, and hostile view of others. There's not much else in the book after this. It's mostly Lewis's charmingly Platonic view of the afterlife, in which the characters go inward and upward to truer and more complete versions of both Narnia and England and are reunited (very briefly) with every character of the series. Lewis knows not to try too hard to describe the indescribable, but it remains one of my favorite visions of an afterlife because it makes so explicit that this world is neither static or the last, but only the beginning of a new adventure. This final section of The Last Battle is deeply flawed, rather arrogant, a little bizarre, and involves more lectures on theology than precise description, but I still love it. By itself, it's not a bad ending for the series, although I don't think it has half the beauty or wonder of the end of The Voyage of the Dawn Treader. It's a shame about the rest of the book, and it's a worse shame that Lewis chose to sacrifice Susan on the altar of his prejudices. Those problems made it very hard to read this book again and make it impossible to recommend. Thankfully, you can read the series without it, and perhaps most readers would be better off imagining their own ending (or lack of ending) to Narnia than the one Lewis chose to give it. But the one redeeming quality The Last Battle will always have for me is that, despite all of its flaws, it was exactly the book that I needed to read when I read it. Rating: 4 out of 10

2 August 2021

Colin Watson: Launchpad now runs on Python 3!

After a very long porting journey, Launchpad is finally running on Python 3 across all of our systems. I wanted to take a bit of time to reflect on why my emotional responses to this port differ so much from those of some others who ve done large ports, such as the Mercurial maintainers. It s hard to deny that we ve had to burn a lot of time on this, which I m sure has had an opportunity cost, and from one point of view it s essentially running to stand still: there is no single compelling feature that we get solely by porting to Python 3, although it s clearly a prerequisite for tidying up old compatibility code and being able to use modern language facilities in the future. And yet, on the whole, I found this a rewarding project and enjoyed doing it. Some of this may be because by inclination I m a maintenance programmer and actually enjoy this sort of thing. My default view tends to be that software version upgrades may be a pain but it s much better to get that pain over with as soon as you can rather than trying to hold back the tide; you can certainly get involved and try to shape where things end up, but rightly or wrongly I can t think of many cases when a righteously indignant user base managed to arrange for the old version to be maintained in perpetuity so that they never had to deal with the new thing (OK, maybe Perl 5 counts here). I think a more compelling difference between Launchpad and Mercurial, though, may be that very few other people really had a vested interest in what Python version Launchpad happened to be running, because it s all server-side code (aside from some client libraries such as launchpadlib, which were ported years ago). As such, we weren t trying to do this with the internet having Strong Opinions at us. We were doing this because it was obviously the only long-term-maintainable path forward, and in more recent times because some of our library dependencies were starting to drop support for Python 2 and so it was obviously going to become a practical problem for us sooner or later; but if we d just stayed on Python 2 forever then fundamentally hardly anyone else would really have cared directly, only maybe about some indirect consequences of that. I don t follow Mercurial development so I may be entirely off-base, but if other people were yelling at me about how late my project was to finish its port, that in itself would make me feel more negatively about the project even if I thought it was a good idea. Having most of the pressure come from ourselves rather than from outside meant that wasn t an issue for us. I m somewhat inclined to think of the process as an extreme version of paying down technical debt. Moving from Python 2.7 to 3.5, as we just did, means skipping over multiple language versions in one go, and if similar changes had been made more gradually it would probably have felt a lot more like the typical dependency update treadmill. I appreciate why not everyone might want to think of it this way: maybe this is just my own rationalization. Reflections on porting to Python 3 I m not going to defend the Python 3 migration process; it was pretty rough in a lot of ways. Nor am I going to spend much effort relitigating it here, as it s already been done to death elsewhere, and as I understand it the core Python developers have got the message loud and clear by now. At a bare minimum, a lot of valuable time was lost early in Python 3 s lifetime hanging on to flag-day-type porting strategies that were impractical for large projects, when it should have been providing for bilingual strategies (code that runs in both Python 2 and 3 for a transitional period) which is where most libraries and most large migrations ended up in practice. For instance, the early advice to library maintainers to maintain two parallel versions or perhaps translate dynamically with 2to3 was entirely impractical in most non-trivial cases and wasn t what most people ended up doing, and yet the idea that 2to3 is all you need still floats around Stack Overflow and the like as a result. (These days, I would probably point people towards something more like Eevee s porting FAQ as somewhere to start.) There are various fairly straightforward things that people often suggest could have been done to smooth the path, and I largely agree: not removing the u'' string prefix only to put it back in 3.3, fewer gratuitous compatibility breaks in the name of tidiness, and so on. But if I had a time machine, the number one thing I would ask to have been done differently would be introducing type annotations in Python 2 before Python 3 branched off. It s true that it s technically possible to do type annotations in Python 2, but the fact that it s a different syntax that would have to be fixed later is offputting, and in practice it wasn t widely used in Python 2 code. To make a significant difference to the ease of porting, annotations would need to have been introduced early enough that lots of Python 2 library code used them so that porting code didn t have to be quite so much of an exercise of manually figuring out the exact nature of string types from context. Launchpad is a complex piece of software that interacts with multiple domains: for example, it deals with a database, HTTP, web page rendering, Debian-format archive publishing, and multiple revision control systems, and there s often overlap between domains. Each of these tends to imply different kinds of string handling. Web page rendering is normally done mainly in Unicode, converting to bytes as late as possible; revision control systems normally want to spend most of their time working with bytes, although the exact details vary; HTTP is of course bytes on the wire, but Python s WSGI interface has some string type subtleties. In practice I found myself thinking about at least four string-like types (that is, things that in a language with a stricter type system I might well want to define as distinct types and restrict conversion between them): bytes, text, ordinary native strings (str in either language, encoded to UTF-8 in Python 2), and native strings with WSGI s encoding rules. Some of these are emergent properties of writing in the intersection of Python 2 and 3, which is effectively a specialized language of its own without coherent official documentation whose users must intuit its behaviour by comparing multiple sources of information, or by referring to unofficial porting guides: not a very satisfactory situation. Fortunately much of the complexity collapses once it becomes possible to write solely in Python 3. Some of the difficulties we ran into are not ones that are typically thought of as Python 2-to-3 porting issues, because they were changed later in Python 3 s development process. For instance, the email module was substantially improved in around the 3.2/3.3 timeframe to handle Python 3 s bytes/text model more correctly, and since Launchpad sends quite a few different kinds of email messages and has some quite picky tests for exactly what it emits, this entailed a lot of work in our email sending code and in our test suite to account for that. (It took me a while to work out whether we should be treating raw email messages as bytes or as text; bytes turned out to work best.) 3.4 made some tweaks to the implementation of quoted-printable encoding that broke a number of our tests in ways that took some effort to fix, because the tests needed to work on both 2.7 and 3.5. The list goes on. I got quite proficient at digging through Python s git history to figure out when and why some particular bit of behaviour had changed. One of the thorniest problems was parsing HTTP form data. We mainly rely on zope.publisher for this, which in turn relied on cgi.FieldStorage; but cgi.FieldStorage is badly broken in some situations on Python 3. Even if that bug were fixed in a more recent version of Python, we can t easily use anything newer than 3.5 for the first stage of our port due to the version of the base OS we re currently running, so it wouldn t help much. In the end I fixed some minor issues in the multipart module (and was kindly given co-maintenance of it) and converted zope.publisher to use it. Although this took a while to sort out, it seems to have gone very well. A couple of other interesting late-arriving issues were around pickle. For most things we normally prefer safer formats such as JSON, but there are a few cases where we use pickle, particularly for our session databases. One of my colleagues pointed out that I needed to remember to tell pickle to stick to protocol 2, so that we d be able to switch back and forward between Python 2 and 3 for a while; quite right, and we later ran into a similar problem with marshal too. A more surprising problem was that datetime.datetime objects pickled on Python 2 require special care when unpickling on Python 3; rather than the approach that ended up being implemented and documented for Python 3.6, though, I preferred a custom unpickler, both so that things would work on Python 3.5 and so that I wouldn t have to risk affecting the decoding of other pickled strings in the session database. General lessons Writing this over a year after Python 2 s end-of-life date, and certainly nowhere near the leading edge of Python 3 porting work, it s perhaps more useful to look at this in terms of the lessons it has for other large technical debt projects. I mentioned in my previous article that I used the approach of an enormous and frequently-rebased git branch as a working area for the port, committing often and sometimes combining and extracting commits for review once they seemed to be ready. A port of this scale would have been entirely intractable without a tool of similar power to git rebase, so I m very glad that we finished migrating to git in 2019. I relied on this right up to the end of the port, and it also allowed for quick assessments of how much more there was to land. git worktree was also helpful, in that I could easily maintain working trees built for each of Python 2 and 3 for comparison. As is usual for most multi-developer projects, all changes to Launchpad need to go through code review, although we sometimes make exceptions for very simple and obvious changes that can be self-reviewed. Since I knew from the outset that this was going to generate a lot of changes for review, I therefore structured my work from the outset to try to make it as easy as possible for my colleagues to review it. This generally involved keeping most changes to a somewhat manageable size of 800 lines or less (although this wasn t always possible), and arranging commits mainly according to the kind of change they made rather than their location. For example, when I needed to fix issues with / in Python 3 being true division rather than floor division, I did so in one commit across the various places where it mattered and took care not to mix it with other unrelated changes. This is good practice for nearly any kind of development, but it was especially important here since it allowed reviewers to consider a clear explanation of what I was doing in the commit message and then skim-read the rest of it much more quickly. It was vital to keep the codebase in a working state at all times, and deploy to production reasonably often: this way if something went wrong the amount of code we had to debug to figure out what had happened was always tractable. (Although I can t seem to find it now to link to it, I saw an account a while back of a company that had taken a flag-day approach instead with a large codebase. It seemed to work for them, but I m certain we couldn t have made it work for Launchpad.) I can t speak too highly of Launchpad s test suite, much of which originated before my time. Without a great deal of extensive coverage of all sorts of interesting edge cases at both the unit and functional level, and a corresponding culture of maintaining that test suite well when making new changes, it would have been impossible to be anything like as confident of the port as we were. As part of the porting work, we split out a couple of substantial chunks of the Launchpad codebase that could easily be decoupled from the core: its Mailman integration and its code import worker. Both of these had substantial dependencies with complex requirements for porting to Python 3, and arranging to be able to do these separately on their own schedule was absolutely worth it. Like disentangling balls of wool, any opportunity you can take to make things less tightly-coupled is probably going to make it easier to disentangle the rest. (I can see a tractable way forward to porting the code import worker, so we may well get that done soon. Our Mailman integration will need to be rewritten, though, since it currently depends on the Python-2-only Mailman 2, and Mailman 3 has a different architecture.) Python lessons Our database layer was already in pretty good shape for a port, since at least the modern bits of its table modelling interface were already strict about using Unicode for text columns. If you have any kind of pervasive low-level framework like this, then making it be pedantic at you in advance of a Python 3 port will probably incur much less swearing in the long run, as you won t be trying to deal with quite so many bytes/text issues at the same time as everything else. Early in our port, we established a standard set of __future__ imports and started incrementally converting files over to them, mainly because we weren t yet sure what else to do and it seemed likely to be helpful. absolute_import was definitely reasonable (and not often a problem in our code), and print_function was annoying but necessary. In hindsight I m not sure about unicode_literals, though. For files that only deal with bytes and text it was reasonable enough, but as I mentioned above there were also a number of cases where we needed literals of the language s native str type, i.e. bytes in Python 2 and text in Python 3: this was particularly noticeable in WSGI contexts, but also cropped up in some other surprising places. We generally either omitted unicode_literals or used six.ensure_str in such cases, but it was definitely a bit awkward and maybe I should have listened more to people telling me it might be a bad idea. A lot of Launchpad s early tests used doctest, mainly in the style where you have text files that interleave narrative commentary with examples. The development team later reached consensus that this was best avoided in most cases, but by then there were far too many doctests to conveniently rewrite in some other form. Porting doctests to Python 3 is really annoying. You run into all the little changes in how objects are represented as text (particularly u'...' versus '...', but plenty of other cases as well); you have next to no tools to do anything useful like skipping individual bits of a doctest that don t apply; using __future__ imports requires the rather obscure approach of adding the relevant names to the doctest s globals in the relevant DocFileSuite or DocTestSuite; dealing with many exception tracebacks requires something like zope.testing.renormalizing; and whatever code refactoring tools you re using probably don t work properly. Basically, don t have done that. It did all turn out to be tractable for us in the end, and I managed to avoid using much in the way of fragile doctest extensions aside from the aforementioned zope.testing.renormalizing, but it was not an enjoyable experience. Regressions I know of nine regressions that reached Launchpad s production systems as a result of this porting work; of course there were various other regressions caught by CI or in manual testing. (Considering the size of this project, I count it as a resounding success that there were only nine production issues, and that for the most part we were able to fix them quickly.) Equality testing of removed database objects One of the things we had to do while porting to Python 3 was to implement the __eq__, __ne__, and __hash__ special methods for all our database objects. This was quite conceptually fiddly, because doing this requires knowing each object s primary key, and that may not yet be available if we ve created an object in Python but not yet flushed the actual INSERT statement to the database (most of our primary keys are auto-incrementing sequences). We thus had to take care to flush pending SQL statements in such cases in order to ensure that we know the primary keys. However, it s possible to have a problem at the other end of the object lifecycle: that is, a Python object might still be reachable in memory even though the underlying row has been DELETEd from the database. In most cases we don t keep removed objects around for obvious reasons, but it can happen in caching code, and buildd-manager crashed as a result (in fact while it was still running on Python 2). We had to take extra care to avoid this problem. Debian imports crashed on non-UTF-8 filenames Python 2 has some unfortunate behaviour around passing bytes or Unicode strings (depending on the platform) to shutil.rmtree, and the combination of some porting work and a particular source package in Debian that contained a non-UTF-8 file name caused us to run into this. The fix was to ensure that the argument passed to shutil.rmtree is a str regardless of Python version. We d actually run into something similar before: it s a subtle porting gotcha, since it s quite easy to end up passing Unicode strings to shutil.rmtree if you re in the process of porting your code to Python 3, and you might easily not notice if the file names in your tests are all encoded using UTF-8. lazr.restful ETags We eventually got far enough along that we could switch one of our four appserver machines (we have quite a number of other machines too, but the appservers handle web and API requests) to Python 3 and see what happened. By this point our extensive test suite had shaken out the vast majority of the things that could go wrong, but there was always going to be room for some interesting edge cases. One of the Ubuntu kernel team reported that they were seeing an increase in 412 Precondition Failed errors in some of their scripts that use our webservice API. These can happen when you re trying to modify an existing resource: the underlying protocol involves sending an If-Match header with the ETag that the client thinks the resource has, and if this doesn t match the ETag that the server calculates for the resource then the client has to refresh its copy of the resource and try again. We initially thought that this might be legitimate since it can happen in normal operation if you collide with another client making changes to the same resource, but it soon became clear that something stranger was going on: we were getting inconsistent ETags for the same object even when it was unchanged. Since we d recently switched a quarter of our appservers to Python 3, that was a natural suspect. Our lazr.restful package provides the framework for our webservice API, and roughly speaking it generates ETags by serializing objects into some kind of canonical form and hashing the result. Unfortunately the serialization was dependent on the Python version in a few ways, and in particular it serialized lists of strings such as lists of bug tags differently: Python 2 used [u'foo', u'bar', u'baz'] where Python 3 used ['foo', 'bar', 'baz']. In lazr.restful 1.0.3 we switched to using JSON for this, removing the Python version dependency and ensuring consistent behaviour between appservers. Memory leaks This problem took the longest to solve. We noticed fairly quickly from our graphs that the appserver machine we d switched to Python 3 had a serious memory leak. Our appservers had always been a bit leaky, but now it wasn t so much a small hole that we can bail occasionally as the boat is sinking rapidly : A serious memory leak (Yes, this got in the way of working out what was going on with ETags for a while.) I spent ages messing around with various attempts to fix this. Since only a quarter of our appservers were affected, and we could get by on 75% capacity for a while, it wasn t urgent but it was definitely annoying. After spending some quality time with objgraph, for some time I thought traceback reference cycles might be at fault, and I sent a number of fixes to various upstream projects for those (e.g. zope.pagetemplate). Those didn t help the leaks much though, and after a while it became clear to me that this couldn t be the sole problem: Python has a cyclic garbage collector that will eventually collect reference cycles as long as there are no strong references to any objects in them, although it might not happen very quickly. Something else must be going on. Debugging reference leaks in any non-trivial and long-running Python program is extremely arduous, especially with ORMs that naturally tend to end up with lots of cycles and caches. After a while I formed a hypothesis that zope.server might be keeping a strong reference to something, although I never managed to nail it down more firmly than that. This was an attractive theory as we were already in the process of migrating to Gunicorn for other reasons anyway, and Gunicorn also has a convenient max_requests setting that s good at mitigating memory leaks. Getting this all in place took some time, but once we did we found that everything was much more stable: A rather flat memory graph This isn t completely satisfying as we never quite got to the bottom of the leak itself, and it s entirely possible that we ve only papered over it using max_requests: I expect we ll gradually back off on how frequently we restart workers over time to try to track this down. However, pragmatically, it s no longer an operational concern. Mirror prober HTTPS proxy handling After we switched our script servers to Python 3, we had several reports of mirror probing failures. (Launchpad keeps lists of Ubuntu archive and image mirrors, and probes them every so often to check that they re reasonably complete and up to date.) This only affected HTTPS mirrors when probed via a proxy server, support for which is a relatively recent feature in Launchpad and involved some code that we never managed to unit-test properly: of course this is exactly the code that went wrong. Sadly I wasn t able to sort out that gap, but at least the fix was simple. Non-MIME-encoded email headers As I mentioned above, there were substantial changes in the email package between Python 2 and 3, and indeed between minor versions of Python 3. Our test coverage here is pretty good, but it s an area where it s very easy to have gaps. We noticed that a script that processes incoming email was crashing on messages with headers that were non-ASCII but not MIME-encoded (and indeed then crashing again when it tried to send a notification of the crash!). The only examples of these I looked at were spam, but we still didn t want to crash on them. The fix involved being somewhat more careful about both the handling of headers returned by Python s email parser and the building of outgoing email notifications. This seems to be working well so far, although I wouldn t be surprised to find the odd other incorrect detail in this sort of area. Failure to handle non-ISO-8859-1 URL-encoded form input Remember how I said that parsing HTTP form data was thorny? After we finished upgrading all our appservers to Python 3, people started reporting that they couldn t post Unicode comments to bugs, which turned out to be only if the attempt was made using JavaScript, and was because I hadn t quite managed to get URL-encoded form data working properly with zope.publisher and multipart. The current standard describes the URL-encoded format for form data as in many ways an aberrant monstrosity , so this was no great surprise. Part of the problem was some very strange choices in zope.publisher dating back to 2004 or earlier, which I attempted to clean up and simplify. The rest was that Python 2 s urlparse.parse_qs unconditionally decodes percent-encoded sequences as ISO-8859-1 if they re passed in as part of a Unicode string, so multipart needs to work around this on Python 2. I m still not completely confident that this is correct in all situations, but at least now that we re on Python 3 everywhere the matrix of cases we need to care about is smaller. Inconsistent marshalling of Loggerhead s disk cache We use Loggerhead for providing web browsing of Bazaar branches. When we upgraded one of its two servers to Python 3, we immediately noticed that the one still on Python 2 was failing to read back its revision information cache, which it stores in a database on disk. (We noticed this because it caused a deployment to fail: when we tried to roll out new code to the instance still on Python 2, Nagios checks had already caused an incompatible cache to be written for one branch from the Python 3 instance.) This turned out to be a similar problem to the pickle issue mentioned above, except this one was with marshal, which I didn t think to look for because it s a relatively obscure module mostly used for internal purposes by Python itself; I m not sure that Loggerhead should really be using it in the first place. The fix was relatively straightforward, complicated mainly by now needing to cope with throwing away unreadable cache data. Ironically, if we d just gone ahead and taken the nominally riskier path of upgrading both servers at the same time, we might never have had a problem here. Intermittent bzr failures Finally, after we upgraded one of our two Bazaar codehosting servers to Python 3, we had a report of intermittent bzr branch hangs. After some digging I found this in our logs:
Traceback (most recent call last):
  ...
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/twisted/conch/ssh/channel.py", line 136, in addWindowBytes
    self.startWriting()
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/lazr/sshserver/session.py", line 88, in startWriting
    resumeProducing()
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/twisted/internet/process.py", line 894, in resumeProducing
    for p in self.pipes.itervalues():
builtins.AttributeError: 'dict' object has no attribute 'itervalues'
I d seen this before in our git hosting service: it was a bug in Twisted s Python 3 port, fixed after 20.3.0 but unfortunately after the last release that supported Python 2, so we had to backport that patch. Using the same backport dealt with this. Onwards!

Russ Allbery: Review: Piranesi

Review: Piranesi, by Susanna Clarke
Publisher: Bloomsbury Publishing
Copyright: 2020
ISBN: 1-63557-564-8
Format: Kindle
Pages: 245
Piranesi is a story told in first-person journal entries by someone who lives in a three-floored world of endless halls full of statues. The writing style is one of the most distinctive things about this book (and something you'll have to get along with to enjoy it), so it's worth quoting a longer passage from the introductory description of the world:
I am determined to explore as much of the World as I can in my lifetime. To this end I have travelled as far as the Nine-Hundred-and-Sixtieth Hall to the West, the Eight-Hundred-and-Ninetieth Hall to to the North and the Seven-Hundred-and-Sixty-Eighth Hall to the South. I have climbed up to the Upper Halls where Clouds move in slow procession and Statues appear suddenly out of the Mists. I have explored the Drowned Halls where the Dark Waters are carpeted with white water lilies. I have seen the Derelict Halls of the East where Ceilings, Floors sometimes even Walls! have collapsed and the dimness is split by shafts of grey Light. In all these places I have stood in Doorways and looked ahead. I have never seen any indication that the World was coming to an End, but only the regular progression of Halls and Passageways into the Far Distance. No Hall, no Vestibule, no Staircase, no Passage is without its Statues. In most Halls they cover all the available space, though here and there you will find an Empty Plinth, Niche or Apse, or even a blank space on a Wall otherwise encrusted with Statues. These Absences are as mysterious in their way as the Statues themselves.
So far as the protagonist knows, the world contains only one other living person, the Other, and thirteen dead ones who exist only as bones. The Other is a scientist searching for Great and Secret Knowledge, and calls the protagonist Piranesi, which is odd because that is not the protagonist's name. Be warned that I'm skating around spoilers for the rest of this review. I don't think I'm giving away anything that would ruin the book, but the nature of the story takes some sharp turns. If knowing anything about that would spoil the book for you and you want to read this without that knowledge, you may want to stop reading here. I also want to disclose early in this review that I wanted this to be a different book than it is, and that had a significant impact on how much I enjoyed it. Someone who came to it with different expectations may have a different and more enjoyable experience. I was engrossed by the strange world, the atmosphere, and the mystery of the halls full of statues. The protagonist is also interested in the same things, and the early part of the book is full of discussion of exploration, scientific investigation, and attempts to understand the nature of the world. That led me to hope for the sort of fantasy novel in which the setting is a character and where understanding the setting is a significant part of the plot. Piranesi is not that book. The story that Clarke wants to tell is centered on psychology rather than setting. The setting does not become a character, nor do we learn much about it by the end of the book. While we do learn how the protagonist came to be in this world, my first thought when that revelation starts halfway through the book was "this is going to be disappointing." And, indeed, it was. I say all of this because I think Piranesi looks, from both its synopsis and from the first few chapters, like it's going to be a world building and exploration fantasy. I think it runs a high risk of disappointing readers in the way that it disappointed me, and that can lead to disliking a book one may have enjoyed if one had read it in a different mood and with a different set of expectations. Piranesi is, instead, about how the protagonist constructs the world, about the effect of trauma on that construction, and about the complexities hidden behind the idea of recovery. And there is a lot to like here: The ending is complex and subtle and does not arrive at easy answers (although I also found it very sad), and although Clarke, by the end of the book, is using the setting primarily as metaphor, the descriptions remain vivid and immersive. I still want the book that I thought I was reading, but I want that book in large part because the fragments of that book that are in this one are so compelling and engrossing. What did not work for me was every character in the book except for the protagonist and one supporting character. The relationship between the protagonist and the Other early in the book is a lovely bit of unsettling complexity. It's obvious that the Other has a far different outlook on the world than the protagonist, but the protagonist seems unaware of it. It's also obvious that the Other is a bit of a jerk, but I was hoping for a twist that showed additional complexity in his character. Sadly, when we get the twist, it's not in the direction of more complexity. Instead, it leads to a highly irritating plot that is unnecessarily prolonged through the protagonist being gullible and child-like in the face of blatantly obvious gaslighting. This is a pattern for the rest of the book: Once villains appear on stage, they're one-note narcissists with essentially no depth. There is one character in Piranesi that I liked as well or better than the protagonist, but they only show up late in the story and get very little character development. Clarke sketches the outline of a character I wanted to learn much more about, but never gives us the details on the page. That leads to what I thought was too much telling rather than showing in the protagonist's relationships at the end of the book, which is part of why I thought the ending was so sad. What the protagonist loses is obvious to me (and lines up with the loss I felt when the book didn't turn out to be what I was hoping it would be); what the protagonist gains is less obvious, is working more on the metaphorical level of the story than the literal level, and is more narrated than shown. In other words, this is psychological fantasy with literary sensibilities told in a frame that looks like exploration fantasy. Parts of it, particularly the descriptions and the sense of place, are quite skillful, but the plot, once revealed, is superficial, obvious, and disappointing. I think it's possible this shift in the reader's sense of what type of book they're reading is intentional on Clarke's part, since it works with the metaphorical topic of the book. But it's not the existence of a shift itself that is my primary objection. I like psychological fantasy as well as exploration fantasy. It's that I thought the book after the shift was shallower, less interesting, and more predictable than the book before the shift. The one thing that is excellent throughout Piranesi, though, is the mood. It takes a bit to get used to the protagonist's writing style (and I continue to dislike the Affectation of capitalizing Nouns when writing in English), but it's open-hearted, curious, thoughtful, observant, and capable in a way I found delightful. Some of the events in this book are quite dark, but it never felt horrifying or oppressive because the protagonist remains so determinedly optimistic and upbeat, even when yanked around by the world's most obvious and blatant gaslighting. That persistent hopefulness and lightness is a good feature in a book published in 2020 and is what carried me through the parts of the story I didn't care for. I wish this had been a different book than it was, or failing that, a book with more complex and interesting supporting characters and plot to fit its complex and interesting psychological arc. I also wish that Clarke had done something more interesting with gender in this novel; it felt like she was setting that up for much of the book, and then it never happened. Ah well. As is, I can't recommend Piranesi, but I can say the protagonist, atmosphere, and sense of place are very well done and I think it will work for some other readers better than it did for me. Rating: 6 out of 10

1 August 2021

Paul Wise: FLOSS Activities July 2021

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Review

Administration
  • libusbgx/gt: triage issues
  • Debian packages: triaged bugs for reintroduced packages
  • Debian servers: debug lists mail issue, debug lists subscription issue
  • Debian wiki: unblock IP addresses, approve accounts

Communication
  • Respond to queries from Debian users and contributors on the mailing lists and IRC

Sponsors The microsoft-authentication-library-for-python and purple-discord work was sponsored by my employer. All other work was done on a volunteer basis.

Next.

Previous.