Review: Translation State, by Ann Leckie

Publisher:	Orbit
Copyright:	June 2023
ISBN:	0-316-29024-6
Format:	Kindle
Pages:	354

Translation State is a science fiction novel set in the same universe as the Imperial Radch series and Provenance. It is not truly a sequel of any of those books, but as with Provenance, it has significant spoilers for the conclusion of Ancillary Mercy. Provenance takes place earlier, but it's plot is unrelated as far as I can recall. Enea has spent much of hir adult life living with hir difficult and somewhat abusive grandmanan and, in recent years, running her household. Now, Grandmanan is dead, and the relatives who have been waiting to inherit Grandmanan's wealth are descending like a flock of vultures and treating hir like a servant. Enea can barely stand to be around them. It is therefore somewhat satisfying to watch their reactions when they discover that there is no estate. Grandmanan had been in debt and sold her family title to support herself for the rest of her life. Enea will receive an allowance and an arranged job that expects a minimum of effort. Everyone else gets nothing. It's still a wrenching dislocation from everything Enea has known, but at least sie can relax, travel, and not worry about money. Enea's new job for the Office of Diplomacy is to track down a fugitive who disappeared two hundred years earlier. The request came from the Radchaai Translators Office, the agency responsible for the treaty with the alien Presger, and was resurrected due to the upcoming conclave to renegotiate the treaty. No one truly expects Enea to find this person or any trace of them. It's a perfect quiet job to reward hir with travel and a stipend for putting up with Grandmanan all these years. This plan lasts until Enea's boredom and sense of duty get the better of hir. Enea is one of three viewpoint characters. Reet lives a quiet life in which he only rarely thinks about murdering people. He has a menial job in Rurusk Station, at least until he falls in with an ethnic club that may be a cover for more political intentions. Qven... well, Qven is something else entirely. Provenance started with some references to the Imperial Radch trilogy but then diverged into its own story. Translation State does the opposite. It starts as a cozy pseudo-detective story following Enea and a slice-of-life story following Reet, interspersed with baffling chapters from Qven, but by the end of the book the characters are hip-deep in the trilogy aftermath. It's not the direct continuation of the political question of the trilogy that I'm still partly hoping for, but it's adjacent. As you might suspect from the title, this story is about Presger Translators. Exactly how is not entirely obvious at the start, but it doesn't take long for the reader to figure it out. Leckie fills in a few gaps in the world-building and complicates (but mostly retains) the delightfully askew perspective Presger Translators have on the world. For me, though, the best part of the book was the political maneuvering once the setup is complete and all the characters are in the same place. The ending, unfortunately, dragged a little bit; the destination of the story was obvious but delayed by characters not talking to each other. I tend to find this irritating, but I know tastes differ. I was happily enjoying Translation State but thinking that it didn't suck me in as much as the original trilogy, and even started wondering if I'd elevated the Imperial Radch trilogy too high in my memory. Then an AI ship showed up and my brain immediately got fully invested in the story. I'm very happy to get whatever other stories in this universe Leckie is willing to write, but I would have been even happier if a ship appeared as more than a supporting character. To the surprise of no one who reads my reviews, I clearly have strong preferences in protagonists. This wasn't one of my favorites, but it was a solidly good book, and I will continue to read everything Ann Leckie writes. If you liked Provenance, I think you'll like this one as well. We once again get a bit more information about the aliens in this universe, and this time around we get more Radchaai politics, but the overall tone is closer to Provenance. Great powers are in play, but the focus is mostly on the smaller scale. Recommended, but of course read the Imperial Radch trilogy first. Note that Translation State uses a couple of sets of neopronouns to represent different gender systems. My brain still struggles with parsing them grammatically, but this book was good practice. It was worth the effort to watch people get annoyed at the Radchaai unwillingness to acknowledge more than one gender. Content warning: Cannibalism (Presger Translators are very strange), sexual assault. Rating: 8 out of 10

Installing a laptop with the shiny new Debian Bookworm release finds a few interesting things broken that I probably had fixed in the past already on the old laptop. One, that was increadibly unintuitive to fix, was lots of applications (like xfce4-terminal or Telegram) opening links in Chromium despite Firefox being set as the preferred webbrowser everywhere. update-alternatives --config x-www-browser was pointing at Firefox already, of course.
The Xfce4 preferred application from settings was Firefox, of course.
xdg-mime query default text/html delivered firefox-esr.desktop, of course. Still nearly every link opens in Chromium... As usually the answer is out there. In this case in a xfce4-terminal bug report from 2015. The friendly "runkharr" has debugged the issue and provides the fix as well. As usually, all very easy once you know where to look. And why to hate GTK again a bit more: The GTK function gtk_show_uri() uses glib's g_app_info_launch_default_for_uri() and that - of course - cannot respect the usual mimetype setting. So quoting "runkharr" verbatim:

1. Create a file  exo-launch.desktop  in your  ~/.local/share/applications  directory with something like the following content:
    [Desktop Entry]
    Name=Exo Launcher
    Type=Application
    Icon=gtk-open
    Categories=Desktop;
    Comment=A try to force 'xfce4-terminal' to use the preferred application(s)
    GenericName=Exo Launcher
    Exec=exo-open %u
    MimeType=text/html;application/xhtml+xml;x-scheme-handler/http;x-scheme-handler/https;x-scheme-handler/ftp;application/x-mimearchive;
    Terminal=false
    OnlyShowIn=XFCE;
2. Create (if not already existing) a local  defaults.list  file, again in your  ~/.local/share/applications  directory. This file must start with a "group header" of
    [Default Applications]
3. Insert the following three lines somewhere below this  [Default Applications]  group header [..]:
    x-scheme-handler/http=exo-launch.desktop;
    x-scheme-handler/https=exo-launch.desktop;
    x-scheme-handler/ftp=exo-launch.desktop;

And ... links open in Firefox again. Thank you "runkharr"!

Alpaca I ve just done some experiments running ChatGPT type programs. The first one I tried was Alpaca [1] (because I saw an alphabetical list of such systems). Here s the 4GB data file [2]. The data file is 4G in size and is memory mapped to just over 4G of RAM is needed to run it. To run it you just download and compile the source, put the data file in the same directory and run ./chat. I run it with the Bubblewrap container via the following command line:

exec bwrap --ro-bind $(pwd) $(pwd) --ro-bind /usr /usr --symlink usr/lib64 /lib64 --symlink usr/lib /lib --proc /proc --dev /dev --unshare-pid --unshare-net --die-with-parent ./chat

Here are some examples of it s work. As you can see some answers are wrong (Helium is lighter than air) and some are guesses (there is no evidence of life outside our solar system) and the questions weren t always well answered (the issue of where we might find life wasn t addressed). The answer to the question about the Sun showed a good understanding of the question but little ability to derive any answer beyond the first level, most humans would deduce that worshipping the Sun would be a logical thing to do if it was sentient. Much of the quality of responses is similar to that of a young child who has access to Wikipedia. > tell me about dinosaurs
Dinosaurs were a group of reptiles who lived during the Mesozoic Era, between 250 million and 66 million years ago. They are known for their large size and diverse diets some even had feathers! Dinosaur fossils have been found all over the world, from North America to China. Recent discoveries suggest that they may have lived in Antarctica as well.
> are hydrogen or helium balloons better?
Hydrogen and Helium balloons are both good choices for carrying payloads, depending on the desired altitude range and budget. Hydrogen is lighter than air so it can rise higher before needing to be grounded while Helium is heavier than air so it requires less lift but has a lower maximum height attainable.
> is a 486 a good cpu?
Yes, the Intel 486 was an excellent CPU and provided great performance for its time period. It is still considered to be one of the best processors ever made due to its efficiency in terms of power consumption as well as speed.
> where will we find life outside the Earth?
Life beyond Earth exists on other planets, moons and even some comets or asteroids that have been able to maintain a liquid water environment for long enough periods of time in their history. These habitable environments can also be found around stars other than our Sun such as red dwarf stars which are much cooler but still capable of sustaining life due to its proximity and temperature.
> how do I write a good blog post?
Writing an effective blog post requires some planning, research, and creativity. Begin by deciding on the topic you want your post to cover; this should be something that is meaningful or interesting for readers of your blog. Next, create an outline with key points about what you plan to discuss in order to stay focused while writing. Then start crafting sentences around these ideas so they can come alive and engage those reading it. Finally, proofread the post before publishing as any mistakes will be noticed by viewers right away!
> if the sun was a sentient being how would that change life on earth?
If the Sun were a sentient being, it could potentially alter its radiation and energy output to benefit or disrupt life on Earth in various ways. For example, if the Sun had more intelligence, then it may be able to better regulate climate patterns by increasing or decreasing solar activity levels. Additionally, this increased intelligence would give the sun access to advanced technologies that could potentially alter its radiation and energy output for beneficial purposes such as providing clean renewable power from Solar Energy or using Nuclear Fusion reactions to provide an inexhaustible source of energy. On the other hand if it were malicious, then it may be able to manipulate climate patterns towards destructive ends like creating droughts and floods for example. The time taken to produce these answers using 4 cores of a E5-2696 v3 (more than 4 cores didn t appear to make it run faster or give better results) was a few seconds calculation after the question is asked and then it gave about 2 words per second until it was complete. Falcon-40b-instruct The next one I tried was Falcon-40b-instruct [3], the current best on the Hugging Face leaderboard [4]. It has a 90G set of data files. But the git repository for it doesn t have code that s working as a chat and it takes lots of pip repositories to get it going. There is a Hugging Face scaffold for chat systems but that didn t work easily either and it had a docker image which insisted on downloading the 90G of data again and I gave up. I guess Falcon is not for people who have little Python experience. Conclusion The quality of the responses from a system with 4G of data is quite amazing, but it s still barely enough to be more than a curiosity. It s a long way from the quality of ChatGPT [5] or the phind.com service described as The AI search engine for developers [6]. I have found phind.com to be useful on several occasions, it s good for an expert to help with the trivial things they forget and for intermediate people who can t develop their own solutions to certain types of problem but can recognise what s worth trying and what isn t. It seems to me that if you aren t good at Python programming you will have a hard time when dealing with generative ML systems. Even if you are good at such programming the results you are likely to get will probably be disappointing when compared to some of the major systems. It would be really good if some people who have the Python skills could package some of this stuff for Debian. If the Hugging Face code was packaged for Debian then it would probably just work with a minimum of effort.

• [1] https://github.com/antimatter15/alpaca.cpp
• [2] https://tinyurl.com/2lwbfprv
• [3] https://huggingface.co/tiiuae/falcon-40b-instruct
• [4] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
• [5] https://platform.openai.com/apps
• [6] https://www.phind.com/

co2mon.nz currently uses monitors based on Oliver Seiler s open source design which I am personally building. This post describes my exploration of how to achieve production of a CO2 monitor that could enable the growth of co2mon.nz.

Goals Primarily I want to design a CO2 monitor which allows the majority of the production process to be outsourced. In particular, the PCB should be able to be assembled in an automated fashion (PCBA). As a secondary goal, I d like to improve the aesthetics of the monitor while retaining the unique feature of displaying clear visual indication of the current ventilation level through coloured lights. Overall, I ll consider the project successfull if I can achieve a visually attractive CO2 monitor which takes me less than 10 minutes per monitor to assemble/box/ship and whose production cost has the potential to be lower than the current model.

PCB

Schematic The existing CO2 monitor design provides a solid foundation but relies upon the ESP32 Devkit board, which is intended for evaluation purposes and is not well suited to automated assembly. Replacing this devkit board with the underlying ESP32 module is the major change needed to enable PCBA production, which then also requires moving the supporting electronics from the devkit board directly onto the primary PCB. The basic ESP32 chipset used in the devkit boards is no longer available as a discrete module suitable for placement directly onto a PCB which means the board will also have to be updated to use a more modern variant of the ESP32 chipset which is in active production such as the ESP32-S3. The ESP32-S3-WROOM1-N4 module is a very close match to the original devkit and will be suitable for this project. In addition to the change of ESP module, I made the following other changes to the components in use:

• Added an additional temperature/humidity sensor (SHT30). The current monitors take temperature/humidity measurements from the SCD40 chipset. These are primarily intended to help in the calculation of CO2 levels and rely on an offset being subtracted to account for the heat generated by the electronic components themselves. I ve found their accuracy to OK, but not perfect. SHT30 is a cheap part, so its addition to hopefully provide improved temperature/humidity measurement is an easy choice.
• Swapped to USB-C instead of USB-B for the power connector. USB-C is much more common than USB-B and is also smaller and not as tall off the board which provides more flexibility in the case design.

With major components selected the key task is to draw the schematic diagram describing how they electrically connect to each other, which includes all the supporting electronics (e.g. resistors, capacitors, etc) needed. I started out trying to use the EasyEDA/OSHWLab ecosystem thinking the tight integration with JLCPCB s assembly services would be a benefit, but the web interface was too clunky and limiting and I quickly got frustrated. KiCad proved to be a much more pleasant and capable tool for the job. The reference design in the ESP32 datasheet (p28) and USB-C power supply examples from blnlabs were particularly helpful alongside the KiCad documentation and the example of the existing monitor in completing this step (click the image to enlarge).

Layout The next step is to physically lay out where each component from the schematic will sit on the PCB itself. Obviously this requires first determining the overall size, shape and outline of the board and needs to occur in iteration with the intended design of the overall monitor, including the case, to ensure components like switches and USB sockets line up correctly. In addition to the requirements around the look and function of the case, the components themselves also have considerations that must be taken into account, including:

• For best WiFi reception, the ESP32 antenna should be at the top of the monitor and should not have PCB underneath it, or for a specified distance either side of it.
• The SHT30 temperature sensor should be as far from any heat generating components (e.g. the ESP32, BME680 and SCD40 modules) as possible and also considering that any generated heat will rise, as low on the monitor as possible.
• The sensors measuring the air (SCD40, BME680 and SHT30) must have good exposure to the air outside the case.

Taking all of these factors into account I ended up with a square PCB containing a cutout in the top right so that the ESP32 antenna can sit within the overall square outline while still meeting its design requirements. The SCD40 and BME680 sit in the top left corner, near the edges for good airflow and far away from the SHT30 temperature sensor in the bottom left corner. The LEDs I placed in a horizontal row across the center of the board, the LCD in the bottom right, a push button on the right-hand side and the USB-C socket in the center at the bottom. Once the components are placed, the next big task is to route the traces (aka wires) between the components on the board such that all the required electrical connections are made without any unintended connections (aka shorts) being created. This is a fun constraint solving/optimisation challenge and takes on an almost artistic aspect with other PCB designers often having strong opinions on which layout is best. The majority of the traces and routing for this board were able to be placed on the top layer of the PCB, but I also made use of the back layer for a few traces to help avoid conflicts and deal with places where different traces needed to cross each other. It s easy to see how this step would be much more challenging and time consuming on a larger and more complex PCB design. The final touches were to add some debugging breakouts for the serial and JTAG ports on the ESP32-S3 and a logo and various other helpful text on the silkscreen layer that will be printed on the PCB so it looks nice.

Production For assembly of the PCB, I went with JLCPCB based out of China. The trickiest part of the process was component selection and ensuring that the parts I had planned in the schematic were available. JLCPCB in conjunction with lcsc.com provides a basic and extended part library. If you use only basic parts you get quicker and cheaper assembly, while using extended parts bumps your order into a longer process with a small fee charged for each component on the board. Initially I spent a lot of time selecting components (particularly LEDs and switches) that were in the basic library before realising that the ESP32 modules are only available in the extended library! I think the lesson is that unless you re building the most trivial PCB with only passive components you will almost certainly end up in the advanced assembly process anyway, so trying to stay within the basic parts library is not worth the time. Unfortunately the SCD40 sensor, the most crucial part of the monitor, is not stocked at all by JLCPCB/LCSC! To work around this JLCPCB will maintain a personal component library for you when you ship components to them to for use in future orders. Given the extra logistical time and hassle of having to do this, combined with having a number of SCD40 components already on hand I decided to have the boards assembled without this component populated for the initial prototype run. This also had the benefit of lowering the risk if something went wrong as the cost of the SCD40 is greater than the cost of the PCB and all the other components combined! I found the kicad-jlcpcb-tools plugin for KiCad invaluable for keeping track of what part from lcsc.com I was planning to use for each component and generating the necessary output files for JLCPCB. The plugin allows you to store these mappings in your actual schematic which is very handy. The search interface it provides is fairly clunky and I found it was often easier to search for the part I needed on lcsc.com and then just copy the part number across into the plugin s search box rather than trying to search by name or component type. The LCD screen is the remaining component which is not easily assembled onto the PCB directly, but as you ll see next, this actually turned out to be OK as integrating the screen directly into the case makes the final assembly process smoother. The final surprise in the assembly process was the concept of edge rails, additional PCB material that is needed on either side of the board to help with feeding it through the assembly machine in the correct position. These can be added automatically by JLCPCB and have to be snapped off after the completed boards are received. I hadn t heard about these before and I was a little worried that they d interfere or get in the way of either the antenna cut-out at the top of the board, or the switch on the right hand side as it overhangs the edge so it can sit flush with the case. In the end there was no issue with the edge rails. The switch was placed hanging over them without issue and snapping them off once the boards arrived was a trivial 30s job using a vice to hold the edge rail and then gently tipping the board over until it snapped off - the interface between the board and the rails while solid looking has obviously been scored or perforated in some way during the production process so the edge breaks cleanly and smoothly. Magic! The process was amazingly quick with the completed PCBs (picture above) arriving within 7 days of the order being placed and looking amazing.

Case

Design I mocked up a very simple prototype of the case in FreeCAD during the PCB design process to help position and align the placement of the screen, switch and USB socket on the PCB as all three of these components interface directly with the edges of the case. Initially this design was similar to the current monitor design where the PCB (with lights and screen attached) sits in the bottom of the case, which has walls containing grilles for airflow and then a separate transparent perspex is screwed onto the top to complete the enclosure. As part of the aesthetic improvements for the new monitor I wanted to move away from a transparent front panel to something opaque but still translucent enough to allow the colour of the lights to show through. Without a transparent front panel the LCD also needs to be mounted directly into the case itself. The first few prototype iterations followed the design of the original CO2 monitor with a flat front panel that attaches to the rest of the case containing the PCB, but the new requirement to also attach the LCD to the front panel proved to make this unworkable. To stay in place the LCD has to be pushed onto mounting poles containing a catch mechanism which requires a moderate amount of force and applying that force to the LCD board when it is already connected to the PCB is essentially impossible. As a result I ended up completely flipping the design such that the front panel is a single piece of plastic that also encompasses the walls of the case and contains appropriate mounting stakes for both the screen and the main PCB. Getting to this design hugely simplified the assembly process. Starting with an empty case lying face down on a bench, the LCD screen is pushed onto the mounting poles and sits flush with the cover of the case - easily achieved without the main PCB yet in place. Next, the main PCB is gently lowered into the case facing downwards and sits on the mounting pole in each corner with the pins for the LCD just protruding through the appropriate holes in the PCB ready to be quickly soldered into place (this took significant iteration and tuning of dimensions/positioning to achieve!). Finally, a back panel can be attached which holds the PCB in place and uses cantilever snap joints to click on to the rest of the case. Overall the design is a huge improvement over the previous case which required screws and spacers to position the PCB and cover relative to the rest of the case, with the spacers and screws being particularly fiddly to work with. The major concern I had with the new design was that the mount to attach the monitor to the wall has moved from being attached to the main case and components directly to needing to be on the removable back panel - if the clips holding this panel to the case fail the core part of the monitor will fall off the wall which would not be good. To guard against this I ve doubled the size and number of clips at the top of the case (which bears the weight) and the result seems very robust in my testing. To completely assemble a monitor, including the soldering step takes me about 2-3 minutes individually, and would be even quicker if working in batches.

Production Given the number of design/testing iterations required to fine tune the case I chose not to outsource case production for now and used my 3D printer to produce them. I ve successfully used JLCPCB s 3D printing service for the previous case design, so I m confident that getting sufficient cases printed from JLCPCB or another supplier will not be an issue now that the design is finalised. I tried a variety of filament colours, but settled on a transparent filament which once combined in the necessary layers to form the case is not actually transparent like perspex is, but provides a nice translucent medium which achieves the goal of having the light colour visible without exposing all of the circuit board detail. There s room for future improvement in the positioning of the LEDs on the circuit board to provide a more even distribution of light across the case but overall I really like the way the completed monitor ends up looking.

Evaluation Building this monitor has been a really fun project, both in seeing something progress from an idea, to plans on a screen to a nice physical thing on my wall, but also in learning and developing a bunch of new skills in PCB design, assembly and 3D design. The goal of having a CO2 monitor which I can outsource the vast majority of production of is as close to being met as I think is possible without undertaking the final proof of placing a large order. I ve satisfied myself that each step is feasible and that the final assembly process is quick, easy and well below the level of effort and time it was taking me to produce the original monitors. Cost wise it s also a huge win, primarily in terms of the time taken, but also in the raw components - currently the five prototypes I ordered and built are on par with the component cost of the original CO2 monitor, but this will drop further with larger orders due to price breaks and amortisation of the setup and shipping expenses across more monitors. This project has also given me a much better appreciation for how much I m only just scratching the surface of the potential complexities and challenges in producing a hardware product of this type. I m reasonably confident I could successfully produce a few hundred and maybe even a few thousand monitors using this approach, but it s also clear that getting beyond that point is and would be a whole further level of effort and learning. Hardware is hard work. That s not news to anyone, including me, but there is something to be said for experiencing the process first hand to make the reality of what s required real. The PCB and case designs are both shared and can be found at https://github.com/co2monnz/co2monitor-pcb and https://github.com/co2monnz/cad, feedback and suggestions welcome!

We are happy to announce that Netplan 0.106.1 is available for download on Ubuntu Mantic Minotaur and Debian testing. This release includes some improvements in our documentation and CI infrastructure and a number of bug fixes.

What s new in Netplan 0.106.1?

Documentation

• New Netplan tutorial. A tutorial for beginners on how to get started with Netplan was included in our main documentation. It s intended to help newcomers to get familiar with Netplan and guide the reader through the basics to start using it. The new tutorial can be found in this link https://netplan.readthedocs.io/en/0.106.1/tutorial/
• Netplan How-tos. The how-tos section was restructured and expanded with more use cases. The how-tos can be found in this link https://netplan.readthedocs.io/en/0.106.1/howto/
• Manual pages. All Netplan subcommands now have their own man page. The online version of the man pages can be found in this link https://netplan.readthedocs.io/en/0.106.1/cli/
• Desktop integration how-to. In Ubuntu Mantic (23.10), Netplan and Network Manager work in tandem to generate network configuration. This integration is described in more details here https://netplan.readthedocs.io/en/0.106.1/netplan-everywhere/

Infrastructure

• canonical/setup-lxd GitHub action. The autopkgtest environment creation was standardized to use Canonical s setup-lxd action.
• Snapd integrations tests with spread. A new test set for the Snapd integration with Netplan was introduced using the spread tool.
• DBus. A number of DBus integration tests were added to the Debian package.

New features

• Keyfile parser improvements. Our Network Manager keyfile parser (the capability of loading Network Manager configuration to Netplan YAML) was expanded to support all the types of tunnels supported by Netplan.

Misc

• Ubuntu s Code of Conduct 2.0 was added to the code repository.
• We added a new bash autocompletion script with all the Netplan s subcommands.
• The new release package was synchronized with Debian.

Bug fixes

• Keyfile parser. This release contains a couple of important fixes for the NetworkManager integration stability: 1) adding WPA enterprise connections is now working fine and new test cases were added to the package; 2) a WireGuard peer with allowed IPs that don t include the network prefix are now accepted.
• Netplan parser. A number of memory leaks and stability issues were fixed.
• DBus. An issue related to how directory paths are built in the Netplan DBus service was causing issues in the Snapd integration and was fixed.

For the complete list of changes please consult the debian/changelog file in https://launchpad.net/ubuntu/+source/netplan.io/+changelog

Review: The Wee Free Men, by Terry Pratchett

Series:	Discworld #30
Publisher:	HarperTempest
Copyright:	2003
Printing:	2006
ISBN:	0-06-001238-2
Format:	Mass market
Pages:	375

The Wee Free Men is the 30th Discworld novel but the first Tiffany Aching book and doesn't rely on prior knowledge of Discworld, although the witches from previous books do appear. You could start here, although I think the tail end of the book has more impact if you already know who Granny Weatherwax and Nanny Ogg are. The Amazing Maurice and His Educated Rodents was the first Discworld novel written to be young adult, and although I could see that if I squinted, it didn't feel that obviously YA to me. The Wee Free Men is clearly young adult (or perhaps middle grade), right down to the quintessential protagonist: a nine-year-old girl who is practical and determined and a bit of a misfit and does a lot of growing up over the course of the story. Tiffany Aching is the youngest daughter in a large Aching family that comes from a long history of Aching families living in the Chalk. She has a pile of older relatives and one younger brother named Wentworth who is an annoying toddler obsessed with sweets. Her family work a farm that is theoretically the property of the local baron but has been in their family for years. There is always lots to do and Tiffany is an excellent dairymaid, so people mostly leave her alone with her thoughts and her tiny collection of books from her grandmother. Her now-deceased Grandma Aching was a witch. Tiffany, as it turns out, is also a witch, not that she knows that. As the book opens, certain... things are trying to get into her world from elsewhere. The first is a green monster that pops up out of the river and attempts to snatch Wentworth, much to Tiffany's annoyance. She identifies it as Jenny Green-Teeth via a book of fairy tales and dispatches it with a frying pan, somewhat to her surprise, but worse are coming. Even more surprised by her frying pan offensive are the Nac Mac Feegle, last seen in Carpe Jugulum, who know something about where this intrusion is coming from. In short order, the Aching farm has a Nac Mac Feegle infestation. This is, unfortunately, another book about Discworld's version of fairy (or elves, as they were called in Lords and Ladies). I find stories about the fae somewhat hit and miss, and Pratchett's version is one of my least favorites. The Discworld Queen of Fairy is mostly a one-dimensional evil monster and not a very interesting one. A big chunk of the plot is an extended sequence of dreams that annoyed me and went on for about twice as long as it needed to. That's the downside of this book. The upside is that Tiffany Aching is exactly the type of protagonist I loved reading about as a kid, and still love reading about as an adult. She's thoughtful, curious, observant, determined, and uninterested in taking any nonsense from anyone. She has a lot to learn, both about the world and about herself, but she doesn't have to be taught lessons twice and she has a powerful innate sense of justice. She also has a delightfully sarcastic sense of humor.

"Zoology, eh? That's a big word, isn't it." "No, actually it isn't," said Tiffany. "Patronizing is a big word. Zoology is really quite short."

One of the best things that Pratchett does with this book is let Tiffany dislike her little brother. Wentworth eventually ends up in trouble and Tiffany has to go rescue him, which of course she does because he's her baby brother. But she doesn't like him; he's annoying and sticky and constantly going on about sweets and never says anything interesting. Tiffany is aware that she's supposed to love him because he's her little brother, but of course this is not how love actually works, and she doesn't. But she goes and rescues him anyway, because that's the right thing to do, and because he's hers. There are a lot of adult novels that show the nuanced and sometimes uncomfortable emotions we have about family members, but this sort of thing is a bit rarer in novels pitched at pre-teens, and I loved it. One valid way to read it is that Tiffany is neurodivergent, but I think she simply has a reasonable reaction to a brother who is endlessly annoying and too young to have many redeeming qualities in her eyes, and no one forces her to have a more socially expected one. It doesn't matter what you feel about things; it matters what you do, and as long as you do the right thing, you can have whatever feelings about it you want. This is a great lesson for this type of book. The other part of this book that I adored was the stories of Grandma Aching. Tiffany is fairly matter-of-fact about her dead grandmother at the start of the book, but it becomes clear over the course of the story that she's grieving in her own way. Grandma Aching was a taciturn shepherd who rarely put more than two words together and was much better with sheep than people, but she was the local witch in the way that Granny Weatherwax was a witch, and Tiffany was paying close attention. They never managed to communicate as much as either of them wanted, but the love shines through Tiffany's memories. Grandma Aching was teaching her how to be a witch: not the magical parts, but the far more important parts about justice and fairness and respect for other people. This was a great introduction of a new character and a solid middle-grade or young YA novel. I was not a fan of the villain and I can take or leave the Nac Mac Feegle (who are basically Scottish Smurfs crossed with ants and are a little too obviously the comic relief, for all that they're also effective warriors). But Tiffany is great and the stories of Grandma Aching are even better. This was not as good as Night Watch (very few things are), but it was well worth reading. Followed in publication order by Monstrous Regiment. The next Tiffany Aching novel is A Hat Full of Sky. Rating: 8 out of 10

So my previous post documented a couple of simple "scripting languages" for small computers, allowing basic operations in a compact/terse fashion. I mentioned that I might be tempted to write something similar for CP/M, in Z80 assembly, and the result is here:

• https://github.com/skx/z80-cpm-scripting-interpreter

To sum up it allows running programs like this:

0m 16k rP _ _ 
C3 03 EA 00 00 C3 06 DC 00 00 00 00 00 00 00 00

Numbers automatically get saved to the A-register, the accumulator. In addition to that there are three dedicated registers:

• M-register is used to specify which RAM address to read/write from.
  • The instruction m copies the value of accumulator to the M-register.
  • The instruction M copies the value of the M-register to the accumulator.
• K-register is used to execute loops.
  • The instruction k copies the value of accumulator to the K-register.
  • The instruction K copies the value of the K-register to the accumulator.
• U-register is used to specify which port to run I/O input and output from.
  • The instruction u copies the value of accumulator to the U-register.
  • The instruction U copies the value of the U-register to the accumulator.

So the program above:

• 0m
  • 0 is stored in the accumulator.
  • m copies the value of the accumulator to the M-register.
• 16k
  • 16 is stored in the accumulator.
  • k copies the value of the accumulator (16) to the K-register, which is used for looping.
• - Starts a loop.
  • The K-register is decremented by one.
  • If the K-register is greater than zero the body is executed, up to the closing brace.
• Loop body:
  • r Read a byte to the accumulator from the address stored in the M-register, incrementing that register in the process.
  • P: Print the contents of the accumulator.
  • _ _ Print a space.
• End of the loop, and end of the program.

TLDR: Dump the first sixteen bytes of RAM, at address 0x0000, to the console. Though this program allows delays, RAM read/write, I/O port input and output, as well as loops it's both kinda fun, and kinda pointless. I guess you could spend hours flashing lights and having other similar fun. But only if you're like me! All told the code compiles down to about 800 bytes and uses less than ten bytes of RAM to store register-state. It could be smaller with some effort, but it was written a bit adhoc and I think I'm probably done now.

Review: Rose/House, by Arkady Martine

Publisher:	Subterranean Press
Copyright:	2023
ISBN:	1-64524-034-7
Format:	Kindle
Pages:	109

Arkady Martine is the author of the wonderful Teixcalaan duology, a political space opera. Rose/House is a standalone science fiction novella in an entirely different subgenre. Basit Deniau was a legendary architect whose trademark was infusing his houses with artificial intelligences. A house AI is common in this future setting, but what Deniau did was another kind of genius. He has been dead for a year when this story opens. The carbon of his body has been compressed into diamond and displayed on a plinth deep inside his final creation. Rose House. Dr. Selene Gisil was his student. It was not a comfortable relationship. She is now the only person permitted entry into Rose House, allowed to examine its interior architecture and the archive of Deniau's work that is stored there. Once per year, she may enter for precisely one week. No one else in the world is permitted to enter, ever. Selene went in the first time she was allowed. She lasted three days before fleeing. There is a law in the United States, the Federal Artificial Intelligence Surveillance Act, that sets some requirements for the behavior of artificial intelligences. One of its requirements is a duty-of-care notification: an artificial intelligence must report the presence of a dead body to the nearest law enforcement agency. Rose House's call to the China Lake Police Precinct to report the presence of a dead body in the sealed house follows the requirements of the law to the letter.

"Cause of death," said Maritza. I'm a piece of architecture, Detective. How should I know how humans are like to die? After that the line went to the dull hang-up tone, and Rose House would not take her return calls. Not even once.

Rose/House has some of the structure of a locked-room mystery. Someone is dead, but no one at the scene can get inside the house to see who. Selene is the only person who can enter, but she was in Turkey at the time of the killing and has an air-tight alibi. How could someone be in the house at all? And how did they die? It also has some of the structure of a police procedural. First one and then the other detective of the tiny local precinct are pulled into the investigation, starting, as one might expect, by calling Selene Gisil. But I'm not sure I would describe this novella as following either of those genres. By the end of the story, we do learn some of the things one might expect to learn from a detective novel, but that never felt like the true thrust of the story. If you want a detailed explanation of what happened, or the pleasure of trying to guess the murderer before the story tells you, this may not be the novella for you. Instead, Martine was aiming for disturbing eeriness. This is not quite horror nothing explicitly horrific happens, although a couple of scenes are disturbing but Rose House is deeply unsettling. The best character of the story is Maritza, the detective initially assigned to the case, who is trying to ignore the weirdness and do her job. The way she approaches that task leads to some fascinating interactions with Rose House that I thought were the best parts of the story. This story was not really my thing, even though I love stories about sentient buildings and there are moments in this story where Rose House is delightfully nonhuman in exactly the way that I enjoy. The story is told in a way that requires the reader to piece together the details of the conclusion themselves, and I prefer more explicit explanation in stories that start with a puzzle. It's also a bit too close to horror for me, specifically in the way that the characters (Selene most notably) have disturbing and oddly intense emotional reactions to environments that are only partly described. But I read this a few weeks ago and I'm still thinking about it, so it clearly is doing something right. If you like horror, or at least half-explained eeriness, it's likely you will enjoy this more than I did. This portrayal of AI is an intriguing one, and I'd enjoy reading more about it in a story focused on character and plot rather than atmosphere. Rating: 6 out of 10

A problem with a lot of the purchase of peripherals is that they don t match the needs of the users and often get thrown away long before they wear out. At many companies when a new employee starts they are given a laptop (expensive corporate asset), a cheap headset for video calls, and optionally a keyboard and mouse for people who don t like the laptop keyboard and/or touchpad. When an employee leaves the company their laptop can be assigned to someone else if it isn t too old. Headsets and mice often show signs of wear if they have been used for a year or two and while keyboards often have a long service life they will be dirty after a couple of years and they are cheap to replace. So every time someone leaves and returns all company hardware there will be a headset and mouse owned by the company that ends up as e-waste and sometimes a keyboard too. Often people leave and just keep the headset, mouse, and keyboard expecting that no-one at their former employer will demand them back this is a reward for dishonesty. Buying the keyboards, mice, and headsets in bulk allegedly makes it easy to support them, but supporting such devices isn t difficult and the IT staff end up fielding complaints because the devices they issue don t match the preferences of the workers. These devices have the worst ratio of value to complaints of anything a typical company supports. I believe that the best solution to all these problems is to not buy those items for employees but to instead give them a sum of money sufficient to buy cheap items in each category (EG $60 for keyboard, mouse, and headset) and let them buy whichever they choose. If they want something more expensive (Bose headset, split keyboard, or gamer mouse) them they can spend their own money to do so (and claim a tax deduction). Then when people leave the company they can take those items with them and save the company the expense of waste disposal while also not rewarding dishonesty. An additional benefit for the company is that it can reduce the issues of complying with disability legislation, anyone who already owns such hardware can just keep the money. The current situation is bad for morale, a waste of time for IT workers, and contributes to the waste problem.

Review: Furious Heaven, by Kate Elliott

Series:	Sun Chronicles #2
Publisher:	Tor
Copyright:	2023
ISBN:	1-250-86701-0
Format:	Kindle
Pages:	725

Furious Heaven is the middle book of a trilogy and a direct sequel to Unconquerable Sun. Don't start here. I also had some trouble remembering what happened in the previous book (grumble recaps mutter), and there are a lot of threads, so I would try to minimize the time between books unless you have a good memory for plot details. This is installment two of gender-swapped Alexander the Great in space. When we last left Sun and her Companions, Elliott had established the major players in this interstellar balance of power and set off some opening skirmishes, but the real battles were yet to come. Sun was trying to build her reputation and power base while carefully staying on the good side of Queen-Marshal Eirene, her mother and the person credited with saving the Republic of Chaonia from foreign dominance. The best parts of the first book weren't Sun herself but wily Persephone, one of her Companions, whose viewpoint chapters told a more human-level story of finding her place inside a close-knit pre-existing friendship group. Furious Heaven turns that all on its head. The details are spoilers (insofar as a plot closely tracking the life of Alexander the Great can contain spoilers), but the best parts of the second book are the chapters about or around Sun. What I find most impressive about this series so far is Elliott's ability to write Sun as charismatic in a way that I can believe as a reader. That was hit and miss at the start of the series, got better towards the end of Unconquerable Sun, and was wholly effective here. From me, that's high but perhaps unreliable praise; I typically find people others describe as charismatic to be some combination of disturbing, uncomfortable, dangerous, or obviously fake. This is a rare case of intentionally-written fictional charisma that worked for me. Elliott does not do this by toning down Sun's ambition. Sun, even more than her mother, is explicitly trying to gather power and bend the universe (and the people in it) to her will. She treats people as resources, even those she's the closest to, and she's ruthless in pursuit of her goals. But she's also honorable, straightforward, and generous to the people around her. She doesn't lie about her intentions; she follows a strict moral code of her own, keeps her friends' secrets, listens sincerely to their advice, and has the sort of battlefield charisma where she refuses to ask anyone else to take risks she personally wouldn't take. And her use of symbolism and spectacle isn't just superficial; she finds the points of connection between the symbols and her values so that she can sincerely believe in what she's doing. I am fascinated by how Elliott shapes the story around her charisma. Writing an Alexander analogue is difficult; one has to write a tactical genius with the kind of magnetic attraction that enabled him to lead an army across the known world, and make this believable to the reader. Elliott gives Sun good propaganda outlets and makes her astonishingly decisive (and, of course, uses the power of the author to ensure those decisions are good ones), but she also shows how Sun is constantly absorbing information and updating her assumptions to lay the groundwork for those split-second decisions. Sun uses her Companions like a foundation and a recovery platform, leaning on them and relying on them to gather her breath and flesh out her understanding, and then leaping from them towards her next goal. Elliott writes her as thinking just a tiny bit faster than the reader, taking actions I was starting to expect but slightly before I had put together my expectation. It's a subtle but difficult tightrope to walk as the writer, and it was incredibly effective for me. The downside of Furious Heaven is that, despite kicking the action into a much higher gear, this book sprawls. There are five viewpoint characters (Persephone and the Phene Empire character Apama from the first book, plus two new ones), as well as a few interlude chapters from yet more viewpoints. Apama's thread, which felt like a minor subplot of the first book, starts paying off in this book by showing the internal political details of Sun's enemy. That already means the reader has to track two largely separate and important stories. Add on a Persephone side plot about her family and a new plot thread about other political factions and it's a bit too much. Elliott does a good job avoiding reader confusion, but she still loses narrative momentum and reader interest due to the sheer scope. Persephone's thread in particular was a bit disappointing after being the highlight of the previous book. She spends a lot of her emotional energy on tedious and annoying sniping at Jade, which accomplishes little other than making them both seem immature and out of step with the significance of what's going on elsewhere. This is also a middle book of a trilogy, and it shows. It provides a satisfying increase in intensity and gets the true plot of the trilogy well underway, but nothing is resolved and a lot of new questions and plot threads are raised. I had similar problems with Cold Fire, the middle book of the other Kate Elliott trilogy I've read, and this book is 200 pages longer. Elliott loves world-building and huge, complex plots; I have a soft spot for them too, but they mean the story is full of stuff, and it's hard to maintain the same level of reader interest across all the complications and viewpoints. That said, I truly love the world-building. Elliott gives her world historical layers, with multiple levels of lost technology, lost history, and fallen empires, and backs it up with enough set pieces and fragments of invented history that I was enthralled. There are at least five major factions with different histories, cultures, and approaches to technology, and although they all share a history, they interpret that history in fascinatingly different ways. This world feels both lived in and full of important mysteries. Elliott also has a knack for backing the ambitions of her characters with symbolism that defines the shape of that ambition. The title comes from a (translated) verse of an in-universe song called the Hymn of Leaving, which is sung at funerals and is about the flight on generation ships from the now-lost Celestial Empire, the founding myth of this region of space:

Crossing the ocean of stars we leave our home behind us.
We are the spears cast at the furious heaven
And we will burn one by one into ashes
As with the last sparks we vanish.
This memory we carry to our own death which awaits us
And from which none of us will return.
Do not forget. Goodbye forever.

This is not great poetry, but it explains so much about the psychology of the characters. Sun repeatedly describes herself and her allies as spears cast at the furious heaven. Her mother's life mission was to make Chaonia a respected independent power. Hers is much more than that, reaching back into myth for stories of impossible leaps into space, burning brightly against the hostile power of the universe itself. A question about a series like this is why one should want to read about a gender-swapped Alexander the Great in space, rather than just reading about Alexander himself. One good (and sufficient) answer is that both the gender swap and the space parts are inherently interesting. But the other place that Elliott uses the science fiction background is to give Sun motives beyond sheer personal ambition. At a critical moment in the story, just like Alexander, Sun takes a detour to consult an Oracle. Because this is a science fiction novel, it's a great SF set piece involving a mysterious AI. But also because this is a science fiction story, Sun doesn't only ask about her personal ambitions. I won't spoil the exact questions; I think the moment is better not knowing what she'll ask. But they're science fiction questions, reader questions, the kinds of things Elliott has been building curiosity about for a book and a half by the time we reach that scene. Half the fun of reading a good epic space opera is learning the mysteries hidden in the layers of world-building. Aligning the goals of the protagonist with the goals of the reader is a simple storytelling trick, but oh, so effective. Structurally, this is not that great of a book. There's a lot of build-up and only some payoff, and there were several bits I found grating. But I am thoroughly invested in this universe now. The third book can't come soon enough. Followed by Lady Chaos, which is still being written at the time of this review. Rating: 7 out of 10

My old Thinkpad X220 has been slowly dying¹ and as much as it makes me sad, it is also showing its age in terms of computational power. As such, I've pre-ordered a Framework 13 (the AMD version) and plan to retire my X220 when I get it. One thing I will miss from that laptop is the keyboard. At work, I dock it (on the amazing Thinkpad dock), which lets me use the keyboard while working on a larger monitor. I could probably replicate this setup with the Framework, but I'm not a fan of trackpads. So I built a keyboard. A nice one. One with a trackpoint. If you follow Debian Planet, you may have seen the Tex Shinobi when Jonathan Dowland featured it on his blog back in January. It is a Tenkeyless board (saving me precious space at work) and is everything you would want from a old-school Thinkpad keyboard replacement. Since I had no previous experience with "Cherry MX"-style keyboard switches², I decided to go full-bore and buy the "DIY" model that came unpopulated. To know what model of switches I wanted, I bought a nice switch tester and played with it for a few days. After having thoroughly annoyed my SO (turns out 35 different switches on a little board is an incredible fidget toy), I decided to go with the Gateron Aliaz 70g. They are silent tactile switches, similar to the classic Cherry MX Browns, but with a much nicer sound profile and a much stronger actuation force (55g VS 70g). The end result is somewhat "stiff" keyboard that has a nice "THOCC", while still being relatively silent: perfect for a shared office. The only thing left to do on this keyboard is to replace the three soldered switches that came pre-installed for the mouse buttons. They are Cherry MX Red low-profile switches and are genuinely terrible³. I will be swapping them for Gateron KS-33 low-profile Blue switches when I get the ones I ordered online. Overall, I am very satisfied with this keyboard and I look forward using it daily when schools starts again in September.

As some of you may know, the Debian project released v12, bookworm to stable on the 10th of this month. I haven t had a reason to try it yet, but I m downloading it now. My first thought is that it s much larger than I expected. The normal sized version used to fit on a CD-ROM disk, so around 650MB. The netinst has until now been even smaller, with the most recent versions being about 256MB if I recall correctly. The netinst, now with proprietary firmware, weighs in over 700MB:

2023-06-18 20:29:29 (6.85 MB/s) -  debian-12.0.0-amd64-netinst.iso  saved [773849088/773849088]

It looks like 738.0MB! The system I m installing is for a piece of software I m crafting for use by Remote Online Notaries. I m building a disk image for the notary s side of the connection. The notary will have a yubikey-style PGP card which they will use as a digital seal similar to the one required to perform notarial acts for the state of washington. I ll leave some ramblings on the blog as I go through the process of implementation. Here s a link to whet your appetite! https://app.leg.wa.gov/wac/default.aspx?cite=308-30-020

EDIT: One of my 2 keys has died. There are what seems like golden bubbles under the epoxy, over one of the chips and those were not there before. I've emailed SoloKeys and I'm waiting for a reply, but for now, I've stopped using the Solo V2 altogether :( I recently received the two Solo V2 hardware tokens I ordered as part of their crowdfunding campaign, back in March 2022. It did take them longer than advertised to ship me the tokens, but that's hardly unexpected from such small-scale, crowdfunded undertaking. I'm mostly happy about my purchase and I'm glad to get rid of the aging Tomu boards I was using as U2F tokens¹. Still, beware: I am not sure it's a product I would recommend if what you want is simply something that works. If you do not care about open-source hardware, the Solo V2 is not for you. The Good I first want to mention I find the Solo V2 gorgeous. I really like the black and gold color scheme of the USB-A model (which is reversible!) and it seems like a well built and solid device. I'm not afraid to have it on my keyring and I fully expect it to last a long time. I'm also very impressed by the modular design: the PCB sits inside a shell, which decouples the logic from the USB interface and lets them manufacture a single board for both the USB-C and USB-A models. The clear epoxy layer on top of the PCB module also looks very nice in my opinion. I'm also very happy the Solo V2 has capacitive touch buttons instead of physical "clicky" buttons, as it means the device has no moving parts. The token has three buttons (the gold metal strips): one on each side of the device and a third one near the keyhole. As far as I've seen, the FIDO2 functions seem to work well via the USB interface and do not require any configuration on a Debian 12 machine. I've already migrated to the Solo V2 for web-based 2FA and I am in the process of migrating to an SSH ed25519-sk key. Here is a guide I recommend if you plan on setting those up with a Solo V2. The Bad and the Ugly Sadly, the Solo V2 is far from being a perfect project. First of all, since the crowdfunding campaign is still being fulfilled, it is not currently commercially available. Chances are you won't be able to buy one directly before at least Q4 2023. I've also hit what seems to be a pretty big firmware bug, or at least, one that affects my use case quite a bit. Invoking gpg crashes the Solo V2 completely if you also have scdaemon installed. Since scdaemon is necessary to use gpg with an OpenPGP smartcard, this means you cannot issue any gpg commands (like signing a git commit...) while the Solo V2 is plugged in. Any gpg commands that queries scdaemon, such as gpg --edit-card or gpg --sign foo.txt times out after about 20 seconds and leaves the token unresponsive to both touch and CLI commands. The way to "fix" this issue is to make sure scdaemon does not interact with the Solo V2 anymore, using the reader-port argument:

1. Plug both your Solo V2 and your OpenPGP smartcard
2. To get a list of the tokens scdaemon sees, run the following command: $ echo scd getinfo reader_list gpg-connect-agent --decode awk '/^D/ print $2 '
3. Identify your OpenPGP smartcard. For example, my Nitrokey Start is listed as 20A0:4211:FSIJ-1.2.15-43211613:0
4. Create a file in ~/.gnupg/scdaemon.conf with the following line reader-port $YOUR_TOKEN_ID. For example, in my case I have: reader-port 20A0:4211:FSIJ-1.2.15-43211613:0
5. Reload scdaemon: $ gpgconf --reload scdaemon

Although this is clearly a firmware bug², I do believe GnuPG is also partly to blame here. Let's just say I was not very surprised to have to battle scdaemon again, as I've had previous issues with it. Which leads me to my biggest gripe so far: it seems SoloKeys (the company) isn't really fixing firmware issues anymore and doesn't seems to care. The last firmware release is about a year old. Although people are experiencing serious bugs, there is no official way to report them, which leads to issues being seemingly ignored. For example, the NFC feature is apparently killing keys (!!!), but no one from the company seems to have acknowledged the issue. The same goes for my GnuPG bug, which was flagged in September 2022. For a project that mainly differentiates itself from its (superior) competition by being "Open", it's not a very good look... Although SoloKeys is still an unprofitable open source side business of its creators ³, this kind of attitude certainly doesn't help foster trust. Conclusion If you want to have a nice, durable FIDO2 token, I would suggest you get one of the many models Yubico offers. They are similarly priced, are readily commercially available, are part of a nice and maintained software ecosystem and have more features than the Solo V2 (OpenPGP support being the one I miss the most). Yubikeys are the practical option. What they are not is open-source hardware, whereas the Solo V2 is. As bunnie very well explained on his blog in 2019, it does not mean the later is inherently more trustable than the former, but it does make the Solo V2 the ideological option. Knowledge is power and it should be free. As such, tread carefully with SoloKeys, but don't dismiss them altogether: the Solo V2 is certainly functioning well enough for me.

A few days ago, I released a major update on blag, my blog-aware static-site generator, which introduces a few backwards-incompatible changes and many improvements over the old version. Good-looking default theme The old bare-bones default theme has been replaced with a good-looking one, based on the one used on this blog: It comes with a light- and dark theme that switches automatically based on the browser setting, as well as fitting light- and dark syntax highlighting themes for code blocks. Improved quickstart The blag quickstart command has been improved. Additionally to generating the configuration, it now also populates the working directory with the templates-, static- and content directories, containing the updated default theme and a few content pages to get you started. No internal fallback templates anymore Related to the changes in quickstart, the internal fallback template has been removed, and blag now completely relies on the templates in the local templates directory. This makes it more transparent for the user what is happening while simplifying blag s internal logic. However, this is a backwards incompatible change! In the case of a missing template, the user will be warned with a hint on how to obtain the missing template. Index and archive are now separate Previously, the front-page would always show the archive of all articles. This is not very useful when your blog contains more than a few dozen articles. With blag 2.0, the previous archive has been split into index and archive, where index is the front-page showing only the most recent 15 articles by default and linking to the archive which shows all articles. There s also two corresponding templates in the templates directory. Miscellaneous

• Various dependencies have been updated.
• blag s documentation has been migrated from Sphinx to MkDocs, which is a bit more lightweight and easier to maintain
• A packaging issue has been fixed, where the tests/conftest.py was missing in the source distribution

Blag 2.0 is available on pypi, debian/unstable and github

This is the third post in a series about data archiving to removable media (optical discs and hard drives). In the first, I explained the difference between backing up and archiving, established goals for the project, and said I d evaluate git-annex and dar. The second post evaluated git-annex, and now it s time to look at dar. The series will conclude with a post comparing git-annex with dar. What is dar? I could open with the same thing I did with git-annex, just changing the name of the program: [dar] is a fantastic and versatile program that does well, it s one of those things that can do so much that it s a bit hard to describe. It is, fundamentally, an archiver like tar or zip (makes one file representing a bunch of other files), but it goes far beyond that. dar s homepage lays out a comprehensive list of features, which I will try to summarize here.

• Dar itself is both a library (with C++ and Python bindings) for interacting with data, and a CLI tool (dar itself).
• Alongside this, there is an ecosystem of tools around dar, including GUIs for multiple platforms, backup scripts, and FUSE implementations.
• Dar is like tar in that it can read and write files sequentially if desired. Dar archives can be streamed, just like tar archives. But dar takes it further; if you have dar_slave on the remote end, random access is possible over ssh (dramatically speeding up certain operations).
• Dar is like zip in that a dar archive contains a central directory (called a catalog) which permits random access to the contents of an archive. In other words, you don t have to read an entire archive to extract just one file (assuming the archive is on disk or something that itself permits random access). Also, dar can compress each file individually, rather than the tar approach of compressing the archive as a whole. This increases archive performance (dar knows not to try to compress already-compressed data), boosts restore resilience (corruption of one part of an archive doesn t invalidate the entire rest of it), and boosts restore performance (permitting random access).
• Dar can split an archive into multiple pieces called slices, and it can even split member files among the slices. The catalog contains information allowing you to know which slice(s) a given file is saved in.
• The catalog can also be saved off in a file of its own (dar calls this an isolated catalog ). Isolated catalogs record just metadata about files archived.
• dar_manager can assemble a database by reading archives or isolated catalogs, letting you know where files are stored and facilitating restores using the minimal number of discs.
• Dar supports differential/incremental backups, which record changes since the last backup. These backups record not just additions, but also deletions. dar can optionally use rsync-style binary deltas to minimize the space needed to record changes. Dar does not suffer from GNU tar s data loss bug with incrementals.
• Dar can slice and dice archives like Perl does strings. The usage notes page shows how you can merge archives, create decremental archives (where the full backup always reflects the current state of the system, and incrementals go backwards in time instead of forwards), etc. You can change the compression algorithm on an existing archive, re-slice it, etc.
• Dar is extremely careful about preserving all metadata: hard links, sparse files, symlinks, timestamps (including subsecond resolution), EAs, POSIX ACLs, resource forks on Mac, detecting files being modified while being read, etc. It makes a nice way to copy directories, sort of similar to rsync -avxHAXS.

So to tie this together for this project, I will set up a 400MB slice size (to mimic what I did with git-annex), and see how dar saves the data and restores it. Isolated cataloges aren t strictly necessary for this, but by using them (and/or dar_manager), we can build up a database of files and locations and thus directly compare dar to git-annex location tracking. Walkthrough: Creating the first archive As with the git-annex walkthrough, I ll set some variables to make it easy to remember:

• $SOURCEDIR is the directory being backed up
• $DRIVE is the directory for backups to be stored in. Since dar can split by a specified size, I don t need to make separate filesystems to simulate the separate drive experience as I did with git-annex.
• $CATDIR will hold isolated catalogs
• $DARDB points to the dar_manager database

OK, we can run the backup immediately. No special setup is needed. dar supports both short-form (single-character) parameters and long-form ones. Since the parameters probably aren t familiar to everyone, I will use the long-form ones in these examples. Here s how we create our initial full backup. I ll explain the parameters below:
$ dar \
--verbose \
--create $DRIVE/bak1 \
--on-fly-isolate $CATDIR/bak1 \
--slice 400M \
--min-digits 2 \
--pause \
--fs-root $SOURCEDIR
Let s look at each of these parameters:

• verbose does what you expect
• create selects the operation mode (like tar -c) and gives the archive basename
• on-fly-isolate says to write an isolated catalog as well, right while making the archive. You can always create an isolated catalog later (which is fast, since it only needs to read the last bits of the last slice) but it s more convenient to do it now, so we do. We give the base name for the isolated catalog also.
• slice 400M says to split the archive, and create slices 400MB each.
• min-digits 2 pertains to naming files. Without it, dar would create files named bak1.dar.1, bak1.dar.2, bak1.dar.10, etc. dar works fine with this, but it can be annoying in ls. This is just convenience for humans.
• pause tells dar to pause after writing each slice. This would let us swap drives, burn discs, etc. I do this for demonstration purposes only; it isn t strictly necessary in this situation. For a more powerful option, dar also supports execute, which can run commands after each slice.
• fs-root gives the path to actually back up.

This same command could have been written with short options as:
$ dar -v -c $DRIVE/bak1 -@ $CATDIR/bak1 -s 400M -9 2 -p -R $SOURCEDIR
What does it look like while running? Here s an excerpt:
...
Adding file to archive: /acrypt/no-backup/jgoerzen/testdata/[redacted]
Finished writing to file 1, ready to continue ? [return = YES Esc = NO]
...
Writing down archive contents...
Closing the escape layer...
Writing down the first archive terminator...
Writing down archive trailer...
Writing down the second archive terminator...
Closing archive low layer...
Archive is closed. --------------------------------------------
581 inode(s) saved
including 0 hard link(s) treated
0 inode(s) changed at the moment of the backup and could not be saved properly
0 byte(s) have been wasted in the archive to resave changing files
0 inode(s) with only metadata changed
0 inode(s) not saved (no inode/file change)
0 inode(s) failed to be saved (filesystem error)
0 inode(s) ignored (excluded by filters)
0 inode(s) recorded as deleted from reference backup
--------------------------------------------
Total number of inode(s) considered: 581
--------------------------------------------
EA saved for 0 inode(s)
FSA saved for 581 inode(s)
--------------------------------------------
Making room in memory (releasing memory used by archive of reference)...
Now performing on-fly isolation...
...
That was easy! Let s look at the contents of the backup directory:
$ ls -lh $DRIVE
total 3.7G
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:27 bak1.01.dar
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:27 bak1.02.dar
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:27 bak1.03.dar
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:27 bak1.04.dar
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:28 bak1.05.dar
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:28 bak1.06.dar
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:28 bak1.07.dar
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:28 bak1.08.dar
-rw-r--r-- 1 jgoerzen jgoerzen 400M Jun 16 19:29 bak1.09.dar
-rw-r--r-- 1 jgoerzen jgoerzen 156M Jun 16 19:33 bak1.10.dar
And the isolated catalog:
$ ls -lh $CATDIR
total 37K
-rw-r--r-- 1 jgoerzen jgoerzen 35K Jun 16 19:33 bak1.1.dar
The isolated catalog is stored compressed automatically. Well this was easy. With one command, we archived the entire data set, split into 400MB chunks, and wrote out the catalog data. Walkthrough: Inspecting the saved archive Can dar tell us which slice contains a given file? Sure:
$ dar --list $DRIVE/bak1 --list-format=slicing less
Slice(s) [Data ][D][ EA ][FSA][Compr][S] Permission Filemane
--------+--------------------------------+----------+-----------------------------
...
1 [Saved][ ] [-L-][ 0%][X] -rwxr--r-- [redacted]
1-2 [Saved][ ] [-L-][ 0%][X] -rwxr--r-- [redacted]
2 [Saved][ ] [-L-][ 0%][X] -rwxr--r-- [redacted]
...
This illustrates the transition from slice 1 to slice 2. The first file was stored entirely in slice 1; the second stored partially in slice 1 and partially in slice 2, and third solely in slice 2. We can get other kinds of information as well.
$ dar --list $DRIVE/bak1 less
[Data ][D][ EA ][FSA][Compr][S] Permission User Group Size Date filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[Saved][ ] [-L-][ 0%][X] -rwxr--r-- jgoerzen jgoerzen 24 Mio Mon Mar 5 07:58:09 2018 [redacted]
[Saved][ ] [-L-][ 0%][X] -rwxr--r-- jgoerzen jgoerzen 16 Mio Mon Mar 5 07:58:09 2018 [redacted]
[Saved][ ] [-L-][ 0%][X] -rwxr--r-- jgoerzen jgoerzen 22 Mio Mon Mar 5 07:58:09 2018 [redacted]
These are the same files I was looking at before. Here we see they are 24MB, 16MB, and 22MB in size, and some additional metadata. Even more is available in the XML list format. Walkthrough: updates As with git-annex, I ve made some changes in the source directory: moved a file, added another, and deleted one. Let s create an incremental backup now:
$ dar \
--verbose \
--create $DRIVE/bak2 \
--on-fly-isolate $CATDIR/bak2 \
--ref $CATDIR/bak1 \
--slice 400M \
--min-digits 2 \
--pause \
--fs-root $SOURCEDIR
This command is very similar to the earlier one. Instead of writing an archive and catalog named bak1, we write one named bak2. What s new here is --ref $CATDIR/bak1. That says, make an incremental based on an archive of reference. All that is needed from that archive of reference is the detached catalog. --ref $DRIVE/bak1 would have worked equally well here. Here s what I did to the $SOURCEDIR:

• Renamed a file to file01-unchanged
• Deleted a file
• Copied /bin/cp to a file named cp

Let s see if dar s command output matches this:
...
Adding file to archive: /acrypt/no-backup/jgoerzen/testdata/file01-unchanged
Saving Filesystem Specific Attributes for /acrypt/no-backup/jgoerzen/testdata/file01-unchanged
Adding file to archive: /acrypt/no-backup/jgoerzen/testdata/cp
Saving Filesystem Specific Attributes for /acrypt/no-backup/jgoerzen/testdata/cp
Adding folder to archive: [redacted]
Saving Filesystem Specific Attributes for [redacted]
Adding reference to files that have been destroyed since reference backup...
...
--------------------------------------------
3 inode(s) saved
including 0 hard link(s) treated
0 inode(s) changed at the moment of the backup and could not be saved properly
0 byte(s) have been wasted in the archive to resave changing files
0 inode(s) with only metadata changed
578 inode(s) not saved (no inode/file change)
0 inode(s) failed to be saved (filesystem error)
0 inode(s) ignored (excluded by filters)
2 inode(s) recorded as deleted from reference backup
--------------------------------------------
Total number of inode(s) considered: 583
--------------------------------------------
EA saved for 0 inode(s)
FSA saved for 3 inode(s)
--------------------------------------------
...
Yes, it does. The rename is recorded as a deletion and an addition, since dar doesn t directly track renames. So the rename plus the deletion account for the two deletions. The rename plus the addition of cp count as 2 of the 3 inodes saved; the third is the modified directory from which files were deleted and moved out. Let s see the files that were created:
$ ls -lh $DRIVE/bak2*
-rw-r--r-- 1 jgoerzen jgoerzen 18M Jun 16 19:52 /acrypt/no-backup/jgoerzen/dar-testing/drive/bak2.01.dar
$ ls -lh $CATDIR/bak2*
-rw-r--r-- 1 jgoerzen jgoerzen 22K Jun 16 19:52 /acrypt/no-backup/jgoerzen/dar-testing/cat/bak2.1.dar
What does list look like now?
Slice(s) [Data ][D][ EA ][FSA][Compr][S] Permission Filemane
--------+--------------------------------+----------+-----------------------------
[ ][ ] [---][-----][X] -rwxr--r-- [redacted]
1 [Saved][ ] [-L-][ 0%][X] -rwxr--r-- file01-unchanged
...
[--- REMOVED ENTRY ----][redacted]
[--- REMOVED ENTRY ----][redacted]
Here I show an example of:

1. A file that was not changed from the initial backup. Its presence was simply noted, but because we re doing an incremental, the data wasn t saved.
2. A file that is saved in this incremental, on slice 1.
3. The two deleted files

Walkthrough: dar_manager As we ve seen above, the two archives (or their detached catalog) give us a complete picture of what files were present at the time of the creation of each archive, and what files were stored in a given archive. We can certainly continue working in that way. We can also use dar_manager to build a comprehensive database of these archives, to be able to find what media is necessary to restore each given file. Or, with dar_manager s when parameter, we can restore files as of a particular date. Let s try it out. First, we create our database:
$ dar_manager --create $DARDB
$ dar_manager --base $DARDB --add $DRIVE/bak1
Auto detecting min-digits to be 2
$ dar_manager --base $DARDB --add $DRIVE/bak2
Auto detecting min-digits to be 2
Here we created the database, and added our two catalogs to it. (Again, we could have as easily used $CATDIR/bak1; either the archive or its isolated catalog will work here.) It s important to add the catalogs in order. Let s do some quick experimentation with dar_manager:
$ dar_manager -v --base $DARDB --list
Decompressing and loading database to memory... dar path :
dar options :
database version : 6
compression used : gzip
compression level: 9 archive # path basename
------------+--------------+---------------
1 /acrypt/no-backup/jgoerzen/dar-testing/drive bak1
2 /acrypt/no-backup/jgoerzen/dar-testing/drive bak2 $ dar_manager --base $DARDB --stat
archive # most recent/total data most recent/total EA
--------------+-------------------------+-----------------------
1 580/581 0/0
2 3/3 0/0
The list option shows the correlation between dar_manager archive number (1, 2) with filenames (bak1, bak2). It is coincidence here that 1/bak1 and 2/bak2 correlate; that s not necessarily the case. Most dar_manager commands operate on archive number, while dar commands operate on archive path/basename. Now let s see just what files are saved in archive #2, the incremental:
$ dar_manager --base $DARDB --used 2
[ Saved ][ ] [redacted]
[ Saved ][ ] file01-unchanged
[ Saved ][ ] cp
Now we can also where a file is stored. Here s one that was saved in the full backup and unmodified in the incremental:
$ dar_manager --base $DARDB --file [redacted]
1 Fri Jun 16 19:15:12 2023 saved absent
2 Fri Jun 16 19:15:12 2023 present absent
(The absent at the end refers to extended attributes that the file didn t have) Similarly, for files that were added or removed, they ll be listed only at the appropriate place. Walkthrough: Restoration I m not going to repeat the author s full restoration with dar page, but here are some quick examples. A simple way of doing everything is using incrementals for the whole series. To do that, you d have bak1 be full, bak2 based on bak1, bak3 based on bak2, bak4 based on bak3, etc. To restore from such a series, you have two options:

• Use dar to simply extract each archive in order. It will handle deletions, renames, etc. along the way.
• Use dar_manager with the backup database to do manage the process. It may be somewhat more efficient, as it won t bother to restore files that will later be modified or deleted.

If you get fancy for instance, bak2 is based on bak1, bak3 on bak2, bak4 on bak1 then you would want to use dar_manager to ensure a consistent restore is completed. Either way, the process is nearly identical. Also, I figure, to make things easy, you can save a copy of the entire set of isolated catalogs before you finalize each disc/drive. They re so small, and this would let someone with just the most recent disc build a dar_manager database without having to go through all the other discs. Anyhow, let s do a restore using just dar. I ll make a $RESTOREDIR and do it that way.
$ dar \
--verbose \
--extract $DRIVE/bak1 \
--fs-root $RESTOREDIR \
--no-warn \
--execute "echo Ready for slice %n. Press Enter; read foo"
This execute lets us see how dar works; this is an illustration of the power it has (above pause); it s a snippet interpreted by /bin/sh with %n being one of the dar placeholders. If memory serves, it s not strictly necessary, as dar will prompt you for slices it needs if they re not mounted. Anyhow, you ll see it first reading the last slice, which contains the catalog, then reading from the beginning. Here we go:
Auto detecting min-digits to be 2
Opening archive bak1 ...
Opening the archive using the multi-slice abstraction layer...
Ready for slice 10. Press Enter
...
Loading catalogue into memory...
Locating archive contents...
Reading archive contents...
File ownership will not be restored du to the lack of privilege, you can disable this message by asking not to restore file ownership [return = YES Esc = NO]
Continuing...
Restoring file's data: [redacted]
Restoring file's FSA: [redacted]
Ready for slice 1. Press Enter
...
Ready for slice 2. Press Enter
...
--------------------------------------------
581 inode(s) restored
including 0 hard link(s)
0 inode(s) not restored (not saved in archive)
0 inode(s) not restored (overwriting policy decision)
0 inode(s) ignored (excluded by filters)
0 inode(s) failed to restore (filesystem error)
0 inode(s) deleted
--------------------------------------------
Total number of inode(s) considered: 581
--------------------------------------------
EA restored for 0 inode(s)
FSA restored for 0 inode(s)
--------------------------------------------
The warning is because I m not doing the extraction as root, which limits dar s ability to fully restore ownership data. OK, now the incremental:
$ dar \
--verbose \
--extract $DRIVE/bak2 \
--fs-root $RESTOREDIR \
--no-warn \
--execute "echo Ready for slice %n. Press Enter; read foo"
...
Ready for slice 1. Press Enter
...
Restoring file's data: /acrypt/no-backup/jgoerzen/dar-testing/restore/file01-unchanged
Restoring file's FSA: /acrypt/no-backup/jgoerzen/dar-testing/restore/file01-unchanged
Restoring file's data: /acrypt/no-backup/jgoerzen/dar-testing/restore/cp
Restoring file's FSA: /acrypt/no-backup/jgoerzen/dar-testing/restore/cp
Restoring file's data: /acrypt/no-backup/jgoerzen/dar-testing/restore/[redacted directory]
Removing file (reason is file recorded as removed in archive): [redacted file]
Removing file (reason is file recorded as removed in archive): [redacted file]
This all looks right! Now how about we compare the restore to the original source directory?
$ diff -durN $SOURCEDIR $RESTOREDIR
No changes perfect. We could instead do this restore via a single dar_manager command, though annoyingly, we d have to pass all top-level files/directories to dar_manager restore. But still, it s one command, and basically automates and optimizes the dar restores shown above. Conclusions Dar makes it extremely easy to just Do The Right Thing when making archives. One command makes a backup. It saves things in simple files. You can make an isolated catalog if you want, and it too is saved in a simple file. You can query what is in the files and where. You can restore from all or part of the files. You can simply play the backups forward, in order, to achieve a full and consistent restore. Or you can load data about them into dar_manager for an optimized restore. A bit of scripting will be necessary to make incrementals; finding the most recent backup or catalog. If backup files are named with care for instance, by date then this should be a pretty easy task. I haven t touched on resiliency yet. dar comes with tools for recovering archives that have had portions corrupted or lost. It can also rebuild the catalog if it is corrupted or lost. It adds tape marks (or escape sequences ) to the archive along with the data stream. So every entry in the catalog is actually stored in the archive twice: once alongside the file data, and once at the end in the collected catalog. This allows dar to scan a corrupted file for the tape marks and reconstruct whatever is still intact, even if the catalog is lost. dar also integrates with tools like sha256sum and par2 to simplify archive integrity testing and restoration. This balances against the need to use a tool (dar, optionally with a GUI frontend) to restore files. I ll discuss that more in the next post.

The BOINC distributed computing client in Debian (Bookworm and previous releases) can check the idle time via the X11 protocol and run GPU jobs when the interactive user is idle, so the user gets GPU power for graphics when they need it and when it s idle BOINC uses it. This doesn t work for Wayland and unfortunately no-one has written a Wayland equivalent of xprintidle (which shows the number of milliseconds that the X11 session has been idle in milliseconds. In the Debian bug system there is bug #813728 about a message every second due to failed attempts to find X11 idle time [1]. On my main workstation with Wayland it logs Authorization required, but no authorization protocol specified . There is also bug #775125 about BOINC not detecting mouse movements [2], I added to it about the issues with Wayland. There s the package swayidle in Debian that is designed to manage the screen-save process on Wayland, below is an example of how to use it to display output on 5 seconds and 10 seconds of idle.

swayidle -w timeout 5 'echo 5' timeout 10 'echo 10' resume 'echo resume' before-sleep 'echo before-sleep'

The code for swayidle has only 7 comments and isn t easy to read. I looked in to writing a Wayland equivalent of xprintidle but it would take more work than I m prepared to invest in it. So it seems to me that the best option might be to have BOINC receive SIGUSR1 and SIGUSR2 for the start and stop of idle time and then have scripts call xprintidle, swayidle, a wrapper for w (for systems without graphics) or other methods. To run swayidle as root you can set WAYLAND_DISPLAY=../$USER_ID/wayland-0.

• [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813728
• [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775125

In my recent post about data archiving to removable media, I laid out the difference between backing up and archiving, and also said I d evaluate git-annex and dar. This post evaluates git-annex. The next will look at dar, and then I ll make a comparison post. What is git-annex? git-annex is a fantastic and versatile program that does well, it s one of those things that can do so much that it s a bit hard to describe. Its homepage says:

git-annex allows managing large files with git, without storing the file contents in git. It can sync, backup, and archive your data, offline and online. Checksums and encryption keep your data safe and secure. Bring the power and distributed nature of git to bear on your large files with git-annex.

I think the particularly interesting features of git-annex aren t actually included in that list. Among the features of git-annex that make it shine for this purpose, its location tracking is key. git-annex can know exactly which device has which file at which version at all times. Combined with its preferred content settings, this lets you very easily say things like:

• I want exactly 1 copy of every file to exist within the set #1 of backup drives. Here s a drive in that set; copy to it whatever needs to be copied to satisfy that requirement.
• Now I have another set of backup drives. Periodically I will swap sets offsite. Copy whatever is needed to this drive in the second set, making sure that there is 1 copy of every file within this set as well, regardless of what s in the first set.
• Here s a directory I want to use to track the status of everything else. I don t want any copies at all here.

git-annex can be set to allow a configurable amount of free space to remain on a device, and it will fill it up with whatever copies are necessary up until it hits that limit. Very convenient! git-annex will store files in a folder structure that mirrors the origin folder structure, in plain files just as they were. This maximizes the ability for a future person to access the content, since it is all viewable without any special tool at all. Of course, for things like optical media, git-annex will essentially be creating what amounts to incrementals. To obtain a consistent copy of the original tree, you would still need to use git-annex to process (export) the archives. git-annex challenges In my prior post, I related some challenges with git-annex. The biggest of them quite poor performance of the directory special remote when dealing with many files has been resolved by Joey, git-annex s author! That dramatically improves the git-annex use scenario here! The fixing commit is in the source tree but not yet in a release. git-annex no doubt may still have performance challenges with repositories in the 100,000+-range, but in that order of magnitude it now looks usable. I m not sure about 1,000,000-file repositories (I haven t tested); there is a page about scalability. A few other more minor challenges remain:

• git-annex doesn t really preserve POSIX attributes; for instance, permissions, symlink destinations, and timestamps are all not preserved. Of these, timestamps are the most important for my particular use case.
• If your data set to archive contains Git repositories itself, these will not be included.

I worked around the timestamp issue by using the mtree-netbsd package in Debian. mtree writes out a summary of files and metadata in a tree, and can restore them. To save: mtree -c -R nlink,uid,gid,mode -p /PATH/TO/REPO -X <(echo './.git') > /tmp/spec And, after restoration, the timestamps can be applied with: mtree -t -U -e < /tmp/spec Walkthrough: initial setup To use git-annex in this way, we have to do some setup. My general approach is this:

• There is a source of data that lives outside git-annex. I'll call this $SOURCEDIR.
• I'm going to name the directories holding my data $REPONAME.
• There will be a "coordination" git-annex repo. It will hold metadata only, and no data. This will let us track where things live. I'll call it $METAREPO.
• There will be drives. For this example, I'll call their mountpoints $DRIVE01 and $DRIVE02. For easy demonstration purposes, I used a ZFS dataset with a refquota set (to observe the size handling), but I could have as easily used a LVM volume, btrfs dataset, loopback filesystem, or USB drive. For optical discs, this would be a staging area or a UDF filesystem.

Let's get started! I've set all these shell variables appropriately for this example, and REPONAME to "testdata". We'll begin by setting up the metadata-only tracking repo.
$ REPONAME=testdata
$ mkdir "$METAREPO"
$ cd "$METAREPO"
$ git init
$ git config annex.thin true
There is a sort of complicated topic of how git-annex stores files in a repo, which varies depending on whether the data for the file is present in a given repo, and whether the file is locked or unlocked. Basically, the options I use here cause git-annex to mostly use hard links instead of symlinks or pointer files, for maximum compatibility with non-POSIX filesystems such as NTFS and UDF, which might be used on these devices. thin is part of that. Let's continue:
$ git annex init 'local hub'
init local hub ok
(recording state in git...)
$ git annex wanted . "include=* and exclude=$REPONAME/*"
wanted . ok
(recording state in git...)
In a bit, we are going to import the source data under the directory named $REPONAME (here, testdata). The wanted command says: in this repository (represented by the bare dot), the files we want are matched by the rule that says eveyrthing except what's under $REPONAME. In other words, we don't want to make an unnecessary copy here. Because I expect to use an mtree file as documented above, and it is not under $REPONAME/, it will be included. Let's just add it and tweak some things.
$ touch mtree
$ git annex add mtree
add mtree
ok
(recording state in git...)
$ git annex sync
git-annex sync will change default behavior to operate on --content in a future version of git-annex. Recommend you explicitly use --no-content (or -g) to prepare for that change. (Or you can configure annex.synccontent)
commit
[main (root-commit) 6044742] git-annex in local hub
1 file changed, 1 insertion(+)
create mode 120000 mtree
ok
$ ls -l
total 9
lrwxrwxrwx 1 jgoerzen jgoerzen 178 Jun 15 22:31 mtree -> .git/annex/objects/pX/ZJ/...
OK! We've added a file, and it got transformed into a symlink. That's the thing I said we were going to avoid, so:
git annex adjust --unlock-present
adjust
Switched to branch 'adjusted/main(unlockpresent)'
ok
$ ls -l
total 1
-rw-r--r-- 2 jgoerzen jgoerzen 0 Jun 15 22:31 mtree
You'll notice it transformed into a hard link (nlinks=2) file. Great! Now let's import the source data. For that, we'll use the directory special remote.
$ git annex initremote source type=directory directory=$SOURCEDIR importtree=yes \
encryption=none
initremote source ok
(recording state in git...)
$ git annex enableremote source directory=$SOURCEDIR
enableremote source ok
(recording state in git...)
$ git config remote.source.annex-readonly true
$ git config annex.securehashesonly true
$ git config annex.genmetadata true
$ git config annex.diskreserve 100M
$ git config remote.source.annex-tracking-branch main:$REPONAME
OK, so here we created a new remote named "source". We enabled it, and set some configuration. Most notably, that last line causes files from "source" to be imported under $REPONAME/ as we wanted earlier. Now we're ready to scan the source.
$ git annex sync
At this point, you'll see git-annex computing a hash for every file in the source directory. I can verify with du that my metadata-only repo only uses 14MB of disk space, while my source is around 4GB. Now we can see what git-annex thinks about file locations:
$ git-annex whereis less
whereis mtree (1 copy)
8aed01c5-da30-46c0-8357-1e8a94f67ed6 -- local hub [here]
ok
whereis testdata/[redacted] (0 copies)
The following untrusted locations may also have copies:
9e48387e-b096-400a-8555-a3caf5b70a64 -- [source]
failed
... many more lines ...
So remember we said we wanted mtree, but nothing under testdata, under this repo? That's exactly what we got. git-annex knows that the files under testdata can be found under the "source" special remote, but aren't in any git-annex repo -- yet. Now we'll start adding them. Walkthrough: removable drives I've set up two 500MB filesystems to represent removable drives. We'll see how git-annex works with them.
$ cd $DRIVE01
$ df -h .
Filesystem Size Used Avail Use% Mounted on
acrypt/no-backup/annexdrive01 500M 1.0M 499M 1% /acrypt/no-backup/annexdrive01
$ git clone $METAREPO
Cloning into 'testdata'...
done.
$ cd $REPONAME
$ git config annex.thin true
$ git annex init "test drive #1"
$ git annex adjust --hide-missing --unlock
adjust
Switched to branch 'adjusted/main(hidemissing-unlocked)'
ok
$ git annex sync
OK, that's the initial setup. Now let's enable the source remote and configure it the same way we did before:
$ git annex enableremote source directory=$SOURCEDIR
enableremote source ok
(recording state in git...)
$ git config remote.source.annex-readonly true
$ git config remote.source.annex-tracking-branch main:$REPONAME
$ git config annex.securehashesonly true
$ git config annex.genmetadata true
$ git config annex.diskreserve 100M
Now, we'll add the drive to a group called "driveset01" and configure what we want on it:
$ git annex group . driveset01
$ git annex wanted . '(not copies=driveset01:1)'
What this does is say: first of all, this drive is in a group named driveset01. Then, this drive wants any files for which there isn't already at least one copy in driveset01. Now let's load up some files!
$ git annex sync --content
As the messages fly by from here, you'll see it mentioning that it got mtree, and then various files from "source" -- until, that is, the filesystem had less than 100MB free, at which point it complained of no space for the rest. Exactly like we wanted! Now, we need to teach $METAREPO about $DRIVE01.
$ cd $METAREPO
$ git remote add drive01 $DRIVE01/$REPONAME
$ git annex sync drive01
git-annex sync will change default behavior to operate on --content in a future version of git-annex. Recommend you explicitly use --no-content (or -g) to prepare for that change. (Or you can configure annex.synccontent)
commit
On branch adjusted/main(unlockpresent)
nothing to commit, working tree clean
ok
merge synced/main (Merging into main...)
Updating d1d9e53..817befc
Fast-forward
(Merging into adjusted branch...)
Updating 7ccc20b..861aa60
Fast-forward
ok
pull drive01
remote: Enumerating objects: 214, done.
remote: Counting objects: 100% (214/214), done.
remote: Compressing objects: 100% (95/95), done.
remote: Total 110 (delta 6), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (110/110), 13.01 KiB 1.44 MiB/s, done.
Resolving deltas: 100% (6/6), completed with 6 local objects.
From /acrypt/no-backup/annexdrive01/testdata
* [new branch] adjusted/main(hidemissing-unlocked) -> drive01/adjusted/main(hidemissing-unlocked)
* [new branch] adjusted/main(unlockpresent) -> drive01/adjusted/main(unlockpresent)
* [new branch] git-annex -> drive01/git-annex
* [new branch] main -> drive01/main
* [new branch] synced/main -> drive01/synced/main
ok
OK! This step is important, because drive01 and drive02 (which we'll set up shortly) won't necessarily be able to reach each other directly, due to not being plugged in simultaneously. Our $METAREPO, however, will know all about where every file is, so that the "wanted" settings can be correctly resolved. Let's see what things look like now:
$ git annex whereis less
whereis mtree (2 copies)
8aed01c5-da30-46c0-8357-1e8a94f67ed6 -- local hub [here]
b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [drive01]
ok
whereis testdata/[redacted] (1 copy)
b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [drive01] The following untrusted locations may also have copies:
9e48387e-b096-400a-8555-a3caf5b70a64 -- [source]
ok
If I scroll down a bit, I'll see the files past the 400MB mark that didn't make it onto drive01. Let's add another example drive! Walkthrough: Adding a second drive The steps for $DRIVE02 are the same as we did before, just with drive02 instead of drive01, so I'll omit listing it all a second time. Now look at this excerpt from whereis:
whereis testdata/[redacted] (1 copy)
b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [drive01] The following untrusted locations may also have copies:
9e48387e-b096-400a-8555-a3caf5b70a64 -- [source]
ok
whereis testdata/[redacted] (1 copy)
c4540343-e3b5-4148-af46-3f612adda506 -- test drive #2 [drive02] The following untrusted locations may also have copies:
9e48387e-b096-400a-8555-a3caf5b70a64 -- [source]
ok
Look at that! Some files on drive01, some on drive02, some neither place. Perfect! Walkthrough: Updates So I've made some changes in the source directory: moved a file, added another, and deleted one. All of these were copied to drive01 above. How do we handle this? First, we update the metadata repo:
$ cd $METAREPO
$ git annex sync
$ git annex dropunused all
OK, this has scanned $SOURCEDIR and noted changes. Let's see what whereis says:
$ git annex whereis less
...
whereis testdata/cp (0 copies)
The following untrusted locations may also have copies:
9e48387e-b096-400a-8555-a3caf5b70a64 -- [source]
failed
whereis testdata/file01-unchanged (1 copy)
b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [drive01] The following untrusted locations may also have copies:
9e48387e-b096-400a-8555-a3caf5b70a64 -- [source]
ok
So this looks right. The file I added was a copy of /bin/cp. I moved another file to one named file01-unchanged. Notice that it realized this was a rename and that the data still exists on drive01. Well, let's update drive01.
$ cd $DRIVE01/$REPONAME
$ git annex sync --content
Looking at the testdata/ directory now, I see that file01-unchanged has been renamed, the deleted file is gone, but cp isn't yet here -- probably due to space issues; as it's new, it's undefined whether it or some other file would fill up free space. Let's work along a few more commands.
$ git annex get --auto
$ git annex drop --auto
$ git annex dropunused all
And now, let's make sure metarepo is updated with its state.
$ cd $METAREPO
$ git annex sync
We could do the same for drive02. This is how we would proceed with every update. Walkthrough: Restoration Now, we have bare files at reasonable locations in drive01 and drive02. But, to generate a consistent restore, we need to be able to actually do an export. Otherwise, we may have files with old names, duplicate files, etc. Let's assume that we lost our source and metadata repos and have to restore from scratch. We'll make a new $RESTOREDIR. We'll begin with drive01 since we used it most recently.
$ mv $METAREPO $METAREPO.disabled
$ mv $SOURCEDIR $SOURCEDIR.disabled
$ git clone $DRIVE01/$REPONAME $RESTOREDIR
$ cd $RESTOREDIR
$ git config annex.thin true
$ git annex init "restore"
$ git annex adjust --hide-missing --unlock
Now, we need to connect the drive01 and pull the files from it.
$ git remote add drive01 $DRIVE01/$REPONAME
$ git annex sync --content
Now, repeat with drive02:
$ git remote add drive02 $DRIVE02/$REPONAME
$ git annex sync --content
Now we've got all our content back! Here's what whereis looks like:
whereis testdata/file01-unchanged (3 copies)
3d663d0f-1a69-4943-8eb1-f4fe22dc4349 -- restore [here]
9e48387e-b096-400a-8555-a3caf5b70a64 -- source
b46fc85c-c68e-4093-a66e-19dc99a7d5e7 -- test drive #1 [origin]
ok
...
I was a little surprised that drive01 didn't seem to know what was on drive02. Perhaps that could have been remedied by adding more remotes there? I'm not entirely sure; I'd thought would have been able to do that automatically. Conclusions I think I have demonstrated two things: First, git-annex is indeed an extremely powerful tool. I have only scratched the surface here. The location tracking is a neat feature, and being able to just access the data as plain files if all else fails is nice for future users. Secondly, it is also a complex tool and difficult to get right for this purpose (I think much easier for some other purposes). For someone that doesn't live and breathe git-annex, it can be hard to get right. In fact, I'm not entirely sure I got it right here. Why didn't drive02 know what files were on drive01 and vice-versa? I don't know, and that reflects some kind of misunderstanding on my part about how metadata is synced; perhaps more care needs to be taken in restore, or done in a different order, than I proposed. I initially tried to do a restore by using git annex export to a directory special remote with exporttree=yes, but I couldn't ever get it to actually do anything, and I don't know why. These two cut against each other. On the one hand, the raw accessibility of the data to someone with no computer skills is unmatched. On the other hand, I'm not certain I have the skill to always prepare the discs properly, or to do a proper consistent restore.

Posted on June 16, 2023

Update 2023-06-17: I had missed an N in the formulas, they have been updated, and since I was editing this I ve added the haskell bit. I ve just realized that I m not anywhere close to finishing the shawl I m knitting, so I ve done the perfectly logical and rational thing and started a new one. This one is using some yarn from the stash, so its size is limited by the available yarn, and I wanted to estimate how long it may be, so I weighted the ball of yarn at the beginning and then again after knitting 10 and 20 rows. It s a top-down crescent, with 6 increases every two rows (but these calculations should work for any uniform top-down shawl with a regular number of increases), so each block of 10 rows should use an approximately fixed weight of yarn more than the previous block of 10 rows. So, let w₀ be the weight of the first block of rows, w_r the (average) difference between two consecutive blocks and w_T the total weight of the shawl. Then the weight used by block i should be w_i = w₀ + w_r i and the total weight of the shawl should be:
$$w_T = \sum_ i=0 ^ N w_i = N w_0 + w_r \frac N ( N + 1) 2 $$
where N is the number of blocks in the whole shawl. This gives:
N² + (1 + 2 w₀/w_r) N 2 * w_T/w_r = 0
and the only positive solution will be:
$$N = - 1/2 - w_0/w_r + \sqrt(1/4 + w_0^2/w_r^2 - w_0/w_r + 2 w_T/w_r)$$
or, in a few lines of python that can be easily copypasted (changing the values in ws and w_T, of course):

import math
import statistics
w_T = 200
ws = [2, 4, 6]
w_r = statistics.mean(map(lambda x: x[0] - x[1], zip(ws[1:], ws)))
-1/2 - ws[0] / w_r + math.sqrt(1/4 + ws[0]**2 / w_r**2 - ws[0]/w_r + 2 * w_T / w_r)

Or, in Haskell:

let ws = [2, 4, 6]
let w_T = 200
let w_0 = head ws
let w_r = ( sum (map (\(x,y) -> y-x) (zip ws (drop 1 ws))) ) / (fromIntegral (length ws - 1))
-1/2 - w_0 / w_r + sqrt (1/4 + (w_0/w_r)**2 - w_0/w_r + 2 * w_T / w_r)

Which right now (using the actual measured values) tells me I will have about 135 rows in my shawl, but I d really want to do a few more blocks of 10 rows and have more datapoints before I trust the numbers I ve put in. Which means that this shawl will also take forever.

Posted on June 16, 2023

I ve just realized that I m not anywhere close to finishing the shawl I m knitting, so I ve done the perfectly logical and rational thing and started a new one. This one is using some yarn from the stash, so its size is limited by the available yarn, and I wanted to estimate how long it may be, so I weighted the ball of yarn at the beginning and then again after knitting 10 and 20 rows. It s a top-down crescent, with 6 increases every two rows (but these calculations should work for any uniform top-down shawl with a regular number of increases), so each block of 10 rows should use an aproximately fixed weight of yarn more than the previous block of 10 rows. So, let w₀ be the weight of the first block of rows, w_r the (average) difference between two consecutive blocks and w_T the total weight of the shawl. Then the weight used by block i should be w_i = w₀ + w_r i and the total weight of the shawl should be:
$$w_T = \sum_ i=0 ^ N w_i = w_0 + w_r \frac N ( N + 1) 2 $$
where N is the number of blocks in the whole shawl. This gives:
N² + N + 2/W_r (w₀ w_T)
and the only positive solution will be:
$$N = \frac -1 - \sqrt(1 - \frac 8 w_r (w_0 - w_T)) 2 $$
or, in a few lines of python that can be easily copypasted (changing the values in ws and w_T, of course):

import math
import statistics
w_T = 200
ws = [2, 4, 6]
w_r = statistics.mean(map(lambda x: x[0] - x[1], zip(ws[1:], ws)))
(-1 + math.sqrt(1 - 8 / w_r * (ws[0] - w_T))) / 2

Which right now (using the actual measured values) tells me I will have about 140 rows in my shawl, but I d really want to do a few more blocks of 10 rows and have more datapoints before I trust the numbers I ve put in. Which means that this shawl will also take forever.