Search Results: "jmtd"

20 November 2021

Jonathan Dowland: hledger footguns

I wrote in budgeting tools that I was taking a look at Plain Text Accounting and in particular, hledger. My Jury's still out on the tools, but in the time I've been looking at them I've come across a couple of foot-guns I thought it was worth writing down. hledger's ledger format is derived from that of its predecessor ledger, and so some of the problems might be inherited. 1. significant white space delimiters The basic syntax for a transaction looks like this
2020-03-15 client payment
    assets:checking         $ 2000
    income:consulting       $-2000
There's some significant white space delimiters in play. The most subtle is what separates the account names from the values: it is two or more spaces. A single space, and the value is treated as part of the account name. For some reason I hit this frequently with trying to encode opening balances: the account name used as the source of the initial balances is something not otherwise generally referred to again (something like equity:opening balances) and the transaction amount is inferred where possible, so I ended up with a bunch of accounts named equity:opening balances 100 and similar. 2. flexible decimal delimiter The value of transactions can be interspersed with commas and periods to make it more readable: e.g. $2000 could be written as $2,000. Different locales have different conventions here: It seems some(/most/all?) of Europe use periods to separate out the units and a comma to delimit the fractional part, whereas the US and the UK do the opposite. There is no built-in association between the currency symbol you are using and the period/comma convention: it's quite possible to accidentally write a number which is interpreted differently to how you intended, and it doesn't matter if you are using $ or etc. 3. new syntax has unexpected results in old versions Finally, my favourite. hledger has a notion of rules that can be used to match transactions when importing from CSV. The format looks like this:
if (match rule)
& (another rule)
account1 some:account:from
account2 some:account:to
By default, multiple rules in sequence like above are OR'd: any of them can match. The & prefix switches the behaviour to AND. But, & is a relatively new addition: it's not supported in 1.18.1, the version in Debian stable, which upstream released in June 2020. In prior versions the & prefix is not a syntax error, or at least, not one that's reported: it's silently ignored; meaning, the line with the & does nothing, and any of the other rules in the set will match. This is easy to miss, and means imports could be incorrectly posted.

12 November 2021

Jonathan Dowland: Frictionless external backups with systemd

Here's a description of how my monthly external backups are managed at a technical level. I didn't realise I hadn't written this all down anywhere yet. What
blinkenlights! blinkenlights!
I plug in one of two (prepared) external hard drives into my headless NAS. The NAS contains my primary data backup. A job automatically decrypts the encrypted filesystem on the drive, mounts it and synchronises the copy of my backup data on the drive from that on the NAS. Whilst this is going on, the blinkstick LED on the NAS switches to a colour to signal "in progress". When it's done, the light changes to green to signal "done" and I can remove it. If something goes wrong, it turns red and I get mail. Why I want a third-strand, off-site backup of my and my family's data in case of a disaster in our house. For it to be useful it has to be regular, so I needed to remove as much of the friction of performing the backup as possible. I use two drives alternately so that I don't have all my eggs in one basket in the window when I bring one of them home and perform the backup. How As much as possible I lean on systemd and its ability to trigger actions based on events.
  1. External drive is plugged in. systemd instantiates a corresponding device unit, named dev-disk-by\x2duuid-aaaaaaaa\x2daaaa\x2daaaa\x2daaaa\x2daaaaaaaaaaaa.device, where aaaa is the UUID of a partition on the device
  2. The backup job is a systemd service which has a WantedBy relationship on the device unit, so when the device appears, systemd starts the backup service.
  3. The backup service has Requires and After relationships on systemd-cryptsetup@extbackup.service, a service created by systemd's cryptsetup generator on start-up (but slightly customised, see below). The encrypted device is therefore unlocked.
  4. The backup service defines multiple start and stop commands with ExecStart and ExecStop. These are used to:
    1. set the blinkstick to the working colour (blue-ish)
    2. mount the now-decrypted filesystem
    3. get a lock on the backup repository (so nothing else writes to it) and synchronise the files
    4. unmount the filesystem
    5. set the blinkstick to the success colour (green)
  5. Finally, the systemd-cryptsetup@extbackup.service unit realises it is not required any more. It has been customised with StopWhenUnneeded=true1, so the encrypted filesystem is closed, ready for the drive to be removed.
  6. I notice the LED colour is green, remove the drive, and take it to its off-site home.
If anything goes wrong, all my custom systemd units have, as a matter of course,
OnFailure=status-email-user@%n.service blinkstick-fail.service
Preparing a new backup disk This is mostly just a standard dm-crypt/cryptsetup/LUKS encrypted device, on top of a standard partition on the underlying disk, with a normal filesystem sitting on top: Basically, the most common way to encrypt a drive in Linux. See places like the cryptsetup docs for how to set something like that up. The key things here are The backup service Here's the backup service unit definition in its entirety:
 [Unit]
 OnFailure=status-email-user@%n.service blinkstick-fail.service
 Requires=systemd-cryptsetup@extbackup.service backup.mount
 After=systemd-cryptsetup@extbackup.service backup.mount
 [Service]
 Type=oneshot
 ExecStart=/usr/local/bin/blinkstick --index 1 --limit 10 --set-color 33c280
 ExecStart=/bin/mount /extbackup
 ExecStart=/home/jon/bin/phobos-backup-monthly
 ExecStop=/bin/umount /extbackup
 ExecStop=/usr/local/bin/blinkstick --index 1 --limit 10 --set-color green
 [Install]
 WantedBy=dev-disk-by\x2duuid-aaaaaaaa\x2daaaa\x2daaaa\x2daaaa\x2daaaaaaaaaaaa.device
The dashes in the UUID in WantedBy= need to be encoded as \x2d and then the slashes from the path bit as dashes. Using dashes to encode slashes is possibly the single most frustrating systemd design decision. Issues Sadly (as detailed in Blinkenlights, part 2) there are 2 some frustrating limitations with trying to handle the mount (and unmount) of the filesystem in systemd-land, so instead, it's done using the traditional mount, umount and fstab. If you can point out any improvements to this approach, please let me know!

  1. I customized mine a while ago by copying the generated service file to a static file, but nowadays I think you could do systemd edit systemd-cryptsetup@extbackup.service to add the StopWhenUnneeded to an override file and not need the rest.
  2. or at least were. It's been a while since I revisited this part.

10 November 2021

Jonathan Dowland: LEGO Princess Castle-books

The set The set
My eldest daughter and I visited a LEGO shop recently and I wanted to buy her a gift. The catch was that we were going to be flying on an airplane the next day, so I wanted to find something with the lowest risk of losing parts on the plane. We settled on Ariel, Belle, Cinderella and Tiana's Storybook Adventures which had a number of things going for it: It was reasonably priced at under 20, for the size of the set; it included four human minifigs (albeit in a sub-minifig size, some kind of munchkin size, but that did not seem to matter) and an assortment of animal accompaniments; but mostly, it folded up into a self-contained mock fairytail "book", and opened up into an enclosed "tray" play area, minimising the risk of losses on the flight.
The set in its resting state The set in its resting state
Lego have done a few of these styles of sets, all Disney princess themed, and it looks like they have a few more on their product roadmap. The newer ones incorporate a locking mechanism with a cute Lego key. I love the concept and think it should be extended to other themes/properties. I can imagine a Lego Star Wars-themed version with a little Death Star trench in the middle, or even an original IP like Classic Space, or Medieval.
Exploring a DIY Lego book frame Exploring a DIY Lego book frame
I really liked the Book device which reminded me of hollow books as a child. The cover and spine pieces are bespoke Lego bricks made for purpose, but I thought you could create something similar with generic parts. Holly and I had a go at the concept with what bricks we had to hand. It's definitely viable (and you could do a lot better with a wider selection of bricks / more skilled builders) and it will be fun to pick something to try and build on the spine.

5 November 2021

Jonathan Dowland: 25 things I would like to 3D print

Last year I started collecting ideas of things I would like to 3D print one day, on Twitter. Twitter is fundamentally ephemeral, so I'll collect it here instead. I got up to 14 items on Twitter, and now I'm up to 25. I don't own a 3D printer, but I have access to one at the work office. Perhaps this list is my subconcious trying to convince me to buy one. What am I missing? What else should I be thinking of printing? Let me know!
  1. Some kind of 45 leaning prong to dry bottles and flasks on
  2. A tea tray and coasters
  3. a replacement prop arm/foot for my computer keyboard (something like this but for the Lenovo Ultranav)
  4. some attempted representation of Borges Library of Babel, a la @jwz The Library of Babel, again
  5. an exploration of the geometry of Susanna Clarke s Piranesi
  6. further iterations of my castle
  7. Small tins to keep loose-leaf tea in
  8. Who am I kidding, bound to be a map from DOOM. E1M1 perhaps, or something more regular (MAP07? E2M8?) See also this amazing print of Quake 3: Arena's "Camping Grounds"
  9. replacement bits for kids toy sets, e.g. a bolt with long shank from Early Learning Centre Build It Deluxe Set, without all 4 of which you can't build much of anything
  10. A stand for a decorative Christmas bauble (kid's hand print on it) A roll of cellotape works pretty well in the mean time
  11. DIY bits-and-bobs sorter/storage (nuts and bolts etc)
  12. A space ship from Elite/Frontier. Probably a Cobra mk3 or maybe a Viper mk2. In a glow-in-the-dark PLA which i d overpaint with gunmetal except for the fuselage
  13. A watch stand/holder/storage thing Except it would look nicer in wood (And I m more inclined to get rid of all but one watch instead)
  14. Little tabbed 7 dividers for vinyl records, with A-Z cut into the tabs 12 ones might be a stretch (something like this)
  15. A low-profile custom trackpoint cover like the ones by SaotoTech (e.g.)
  16. A vinyl record. (Not sure that any 3D printer I would have access to would have the necessary resolution. I haven't done any research yet.)
  17. A free-standing inclined vinyl record display stand (e.g.)
  18. A "Settlers of Catan" set. I've got the travel edition which is great but it would be nicer to have a larger-sized set. There are some things I really like about the travel set that the full-size set lacks; so designing and printing a larger set myself could incorporate them. Also I don't feel inclined to buy the full-size set for 50 or so to end up with essentially the same game I already bought. No doubt I'd spent at least that much in PLA.
  19. Little kids trinkets. Pacman ghosts, that sort-of thing. Whatever my daughters come up with next.
  20. Lego storage/sorters.
  21. Some kind of lenticular picture. Perhaps a gift or Christmas card combined in one.
  22. A bracket to install a Gotek drive in my Amiga 500 (e.g.). I've managed without but the fit isn't great.
  23. An attempt at using the 3D printer for 2D drawing. I would never get the same kind of quality results you can get from a proper plotter, but still Take a look at some proper plotter art!
  24. Garden decorations. I like the idea of porous geometric shapes that you can plant mosses or ferns into, but also things which might be taken over and "used" by nature in ways I hadn't thought of.
  25. Floor plans / 3D plans of my house (including variations if I remodelled)

Jonathan Dowland: Mastodon again

I've been an occasional Mastodon user for (apparently) four years. I experimented with running my own instance for a while before giving that up, then I shopped around for an interesting little instance and settled on one, which was lovely until the maintainer decided (quite reasonably) it wasn't worth the trouble any more and turned it off. So I'm in need of a new instance! The first account I had was @Jmtd@mastodon.cloud, but that instance was too busy for my liking. I'd like to find something with small but vibrant local traffic, ideally matching to my interests. In fact one of the problems with my last instance1 was it was too quiet, and the local timeline was mostly empty. So something between "empty" and "overwhelmingly busy" as mastodon.cloud is, or was. Recommendations welcome! I'd set @jmtd@mastodon.cloud to forward to my instance for most of the last four years. I've turned that off now, so I am (at least temporarily) back to that instance. That's got me thinking: perhaps it's less onerous to run a local instance that only forwards, and to use that as a sort-of "permalink", allowing me to migrate from real instance-to-instance when required, but giving me a Mastodon "perma-link". Something to look at. (It's also worth considering that you should back up the list of accounts you are following off whichever instance you are on. I'm starting again with almost a clean slate.)

  1. I'm deliberately omitting the name of my last instance to give the former maintainer a break.

14 September 2021

Jonathan Dowland: GHC rewrite rules

The Glasgow Haskell Compiler (GHC) has support for user-supplied rewrite rules, which applied during one of the compiler optimisation stages. An example rule is
 -# RULES
      "streamFilter fuse" forall f g xs.
          streamFilter f (streamFilter g xs) = streamFilter (f.g) xs
  #- 
I spent some time today looking at these more closely. In order for rewrite rules to be applied, optimisation needs to be enabled. This conflicts with interactive use of GHC, so you can't explore these things in GHCi. I think rewrite rules are enabled by default (with optimisation), but you can ask for them explicitly. When investigating these it's also useful to ask ghc to always recompile, otherwise you have to change the source or manually remove .hi or .o files (etc.) between invocations. A starting set of command-line options to use then, are
-O -fenable-rewrite-rules -fforce-recomp
GHC runs several compilation stages, and the source program is transformed into several different languages or language dialects as it goes. Before the phase where rewrite rules are applied, some other optimisations take place, and the source gets desugared. You can see the results of the desugaring by passing the argument -ddump-ds. Here's an example program
main = print (unStream (streamFilter (>3) (streamFilter (<10)
    (mkStream [0..20]))))
And here's what it looks like after the first pass optimisation and desugaring:
main
  = print
      ($fShow[] $fShowInteger)
      (unStream
         (streamFilter
            (let  
               ds
               ds = 3   in
             \ ds -> > $fOrdInteger ds ds)
            (streamFilter
               (let  
                  ds
                  ds = 10   in
                \ ds -> < $fOrdInteger ds ds)
               (mkStream (enumFromTo $fEnumInteger 0 20)))))
(Note: I used -dsuppress-all and -dsuppress-uniques to improve the clarity of the above output. See Suppressing unwanted information for further details). Those short-hand sections ((<3)) in the input program are expanded to something quite a lot longer. Out of curiosity I tried it again with plain lambdas, not sections, and the result was smaller
main
  = print
      ($fShow[] $fShowInteger)
      (unStream
         (streamFilter
            (\ x -> > $fOrdInteger x 3)
            (streamFilter
               (\ x -> < $fOrdInteger x 10)
               (mkStream (enumFromTo $fEnumInteger 0 20)))))
Rewrite rules happen after this. Once they're done (and several other passes), the program is translated into an intermediate representation called Tiny Core. This language faintly resembles the input Haskell. GHC will output the Tiny Core program if you supply the argument -ddump-simpl. Here's (most) of the program in Tiny Core (I've substituted some constants for clarity):
main  = hPutStr' stdout main1 True
main1 = $fShowInteger_$cshowList (catMaybes1 (main_go 0)) []
main_go
  = \ x ->
      case gtInteger# x 20 of  
        DEFAULT ->
          case ltInteger# x 10 of  
            DEFAULT -> main_go (plusInteger x 1);
            1# ->
              case gtInteger# x 3 of  
                DEFAULT -> main_go (plusInteger x 1);
                1# -> : (Just x) (main_go (plusInteger x 1))
               
           ;
        1# -> []
       
After Tiny Core, GHC continues to translate the program into other forms (including STG, CMM, ASM) and you can ask GHC to dump those representations too, but this is all downstream from the rewrites so not relevant to them. The rewrite rule at the top of this blog post is never applied: It doesn't get a chance, because the function it operates on (streamFilter) is inlined by GHC. GHC can detect this in some circumstances (-Winline-rule-shadowing). You instruct GHC to report on which rules fired with -ddump-rule-firings and can see before-and-after snippets of Tiny Core for each rule applied with -ddump-rule-rewrites. I played around with adding -# NOINLINE functionName #- pragmas to disable inlining various functions to try and provoke a situation where the above rule could match, but it was a losing battle: GHC's built-in optimisations are just too good. But, it's also moot: the outcome I want (the filters to be fused) is happening, it's just the built-in rewrite rules are achieving it, once striot's functions have been inlined away.

20 August 2021

Jonathan Dowland: PhD year 4 progression

I've now completed my 4th year on my part-time PhD (a.k.a. Stage 2). I'm 2/3 of the way through. Here's my stage 2 / year 4 report. As I've written before, this isn't graded so it might not be an example of a good one, but I did pass progression. I had hoped to write some more blog posts about the actual content of my work before another administrative document. I spent a couple of hours drafting one a few weeks back but didn't finish it!

19 August 2021

Jonathan Dowland: Budgeting tools

YNAB On the advice of many friends, I tried to use You Need A Budget. I gave it a seriously long, proper evaluation: over a year. But I just couldn't get it to work for me. I don't want to try and explain why. To be honest, those same friends who advocated for it fairly strongly, also gave me a pretty hard time for giving up on it! Despite it not clicking for me, there are a few concepts from YNAB that I quite like: GNUCash Jimmy Kaplowitz suggested back in 2012 that I should take a look at GNUCash. It took me a few more years before I did. The eventual trigger point for me was organising an event where I paid for a load of things on behalf of others and needed to track who had paid me back. It excelled for that. I've continued to use GNUCash to manage my personal money that is, my "play money" and anything I've accumulated but I haven't committed to it for my family finances. Practically speaking that would lock my wife out of them, which wouldn't be fair. But also because GNUCash's shortcomings (and despite its strengths, it certainly has some) mean that I don't expect I will be using it into the indefinite future, even for my personal stuff. The most significant drawback, in my opinion, is GNUCash's support for scripting. Sometimes, there's a laborious but easily-mechanisable (in theory) task I need to perform that would be ideal to script. GNUCash has built-in scripting support using Guile the GNU lisp/scheme dialect but this is limited to Reports only, I don't think it can be used for a task such as "match a series of transactions using one or more filters or regular expressions, and apply a transformation to them, such as change the account to which they are posted", etc. It also has a C library and auto-generated bindings for other languages. This has a horrible API, which is carried over into the language bindings. Documentation for the whole lot is basically non-existent too. Plain-text accounting For that reason I set out to find some better tools. There's a lot of interest and activity in plain-text accounting (PTA), including tools such as beancount, ledger or the Haskell re-implementation hledger. In a future post I'll write about PTA and hledger.

30 July 2021

Jonathan Dowland: Accounting: pooling income

I wrote about budgeting nine years ago and I've been a little reluctant to write about it again: by far, it's the blog post that has attracted the most requests from people asking me to link to their blog, site, or service. I wasn't good at budgeting then and I'm still not good at it now, although I have learned a few things in the intervening time. Those things more properly relate to accounting than budgeting (so there's the first thing: I learned the difference!). I wanted to write about some of the things I've learned since then, starting with our family's approach to pooling income. Pooling From talking to friends about how they manage stuff, this doesn't seem to be a common approach. We pay all our income into a shared account. We agree on an amount of "play money" that we can individually spend on whatever we like, and we pay that amount to ourselves from the shared account every month. Crucially, the amount we pick is the same for each of us, irrespective of our relative incomes. All of our shared family expenses come out of the shared account. Some of my friends, especially (exclusively) the bread-winners, find this a bit alarming. One of the things I like about it is that whichever partner earns less than the other is not disadvantaged in terms of their discretionary spending. When my wife earned less than me, and I believe structural sexism was a contributing factor to that, that impacted us both equally. When my wife was not earning a salary at all, but was doing the lion's share of bringing up our children, she has the same discretionary spend as I do. Apart from the equity of it, there's a whole class of gripes and grumbles that some of my friends have about their partner's spending habits or money management that we completely avoid.

15 July 2021

Jonathan Dowland: Small tweaks to git branch behaviour

Despite my best efforts, I often end up with a lot of branches in my git repositories, many of which need cleaning up, but even so, may which don't. Two git configuration tweaks make the output of git branch much more useful for me. Motivational example, default git behaviour:
 git branch
  2021-apr-cpu-proposed
  OPENJDK-159-openj9-FROM
  OPENJDK-312-passwd
  OPENJDK-407-dnf-modules-fonts
  create_override_files_in_redhat_189
* develop
  inline-container-yaml
  local-modules
  mdrafiur-pr185-jolokia
  openjdk-containers-1.9
  openjdk-rm-jolokia
  osbs-openjdk
  release
  signing-intent-release
  ubi-1.3-mergedown
  ubi-11-singleton-jdk
  ubi8.2
  update-FROM-lines
  update-for-cct-module-changes-maven-etc
The default sort order is alphabetical, but that's never useful for the repositories I work in. The age of the branch is generally more useful. This particular example isn't that long, but often the number of branches can fill the screen. git can be configured to use columns for branch listings, which I think generally improves readability.
 git config --global branch.sort authordate
 git config --global column.branch auto
After:
 git branch
  update-for-cct-module-changes-maven-etc   signing-intent-release
  openjdk-rm-jolokia                        local-modules
  ubi8.2                                    mdrafiur-pr185-jolokia
  ubi-11-singleton-jdk                      OPENJDK-312-passwd
  ubi-1.3-mergedown                         create_override_files_in_redhat_189
  OPENJDK-159-openj9-FROM                   2021-apr-cpu-proposed
  openjdk-containers-1.9                    OPENJDK-407-dnf-modules-fonts
  inline-container-yaml                     release
  update-FROM-lines                       * develop
  osbs-openjdk

5 July 2021

Jonathan Dowland: Photos and WhatsApp

I woke up this morning to a lovely little gallery of pictures of our children that my wife had sent me via WhatsApp. This has become the most common way we interact with family photos. We regularly send and receive photos to and from our families via WhatsApp, which re-compresses them for transit and temporary storage across their network. The original photos, wherever they are, will be in a very high quality (as you get on most modern cameras) and will be backed up in perfect fidelity to either Apple or Google s photo storage solutions. But all of that seems moot, when the most frequent way we engage with the pictures is via a method which compresses so aggressively that you can clearly see the artefacts, even thumbnailed on a phone screen. I still don t feel particularly happy with the solution in place for backing up the photos (or even: getting them off the phone). Both Apple and Google make it less than convenient to get them out of their respective walled gardens. I ve been evaluating the nextCloud app and a Nextcloud instance on my home NAS as a possible alternative.

14 June 2021

Jonathan Dowland: Opinionated IkiWiki v1

It's been more than a year since I wrote about Opinionated IkiWiki, a pre-configured, containerized deployment of Ikiwiki with opinions. My intention was to make something that is easy to get up and running if you are more experienced with containers than IkiWiki. I haven't yet switched to Opinionated IkiWiki for this site, but that was my goal, and I think it's mature enough now that I can migrate over at some point, so it seems a good time to call it Version 1.0. I have been using it for my own private PIM systems for a while now. You can pull built images from quay.io, here: https://quay.io/repository/jdowland/opinionated-ikiwiki The source lives here: https://github.com/jmtd/opinionated-ikiwiki A description of some of the changes made to the IkiWiki version lives here: https://github.com/jmtd/ikiwiki/blob/opinionated-doc/README.md

8 June 2021

Jonathan Dowland: LaTeX draft documents

I'm writing up a PhD deliverable (which will show up here eventually) using LaTeX, which is my preferred tool for such things, since I also use it for papers, and will eventually be using it for my thesis itself. For this last document, I experimented with a few packages and techniques for organising the document which I found useful, so I thought I'd share them. What version is this anyway? I habitually store and develop my documents in git repositories. From time to time I generate a PDF and copy it off elsewhere to review (e.g., an iPad). Later on it can be useful to be able to figure out exactly what source built the PDF. I achieve this using
\newcommand \version \input "git describe --always --dirty" 
And \version\ somewhere in the header of the document. Draft mode The common document classes all accept a draft argument, to enable Draft Mode.
\documentclass[12pt,draft] article 
Various other packages behave differently if Draft Mode is enabled. The graphicx package, for example, doesn't actually draw pictures in draft mode, which I don't find useful. So for that package, I force it to behave as if we were in "Final Mode" at all times:
\usepackage[final] graphicx 
I want to also include some different bits and pieces in Draft Mode. Although the final version won't need it, I find having a Table of Contents very helpful during the writing process. The ifdraft package adds a convenience macro to query whether we are in draft or not. I use it like so:
\ifdraft 
This page will be cut from the final report.
\tableofcontents
\newpage
 
For this document, I have been given the section headings I must use and the number of pages each section must run to. When drafting, I want to include the page budget in the section names (e.g. Background (2 pages)). I also force new pages at the beginning of each Section, to make it easier to see how close I am to each section's page budget.
\ifdraft \newpage 
\section Work completed\ifdraft  (1 page)  % 1 Page
todonotes
Two TODO items in the margin Two TODO items in the margin
Collated TODOs in a list Collated TODOs in a list
The todonotes package package is one of many that offers macros to make managing in-line TODO notes easier. Within the source of my document, I can add a TODO right next to the relevant text with \todo something to do . In the document, by default, this is rendered in the right-hand margin. With the right argument, the package will only render the notes in draft mode.
\usepackage[textsize=small,obeyDraft] todonotes 
todonotes can also collate all the TODOs together into a single list. The list items are hyperlinked back to the page where the relevant item appears.
\ifdraft 
\newpage
\section TODO 
  This page will be cut from the final report.
  \listoftodos
 

28 May 2021

Jonathan Dowland: Queueing theory

Last year I began looking at queuing theory, to try and see if I could use it as a robust underpinning for a cost model to evaluate rewritten stream-processing programs. I started to work with my co-supervisor, Dr. Paul Ezhilchelvan, who is an expert in this area. In order to brief him on the work I'd done so far as well as my initial efforts to map queueing theory concepts onto Striot's concepts, I prepared a brief presentation:

Jonathan Dowland: Queuing theory

Last year I began looking at queuing theory, to try and see if I could use it as a robust underpinning for a cost model to evaluate rewritten stream-processing programs. I started to work with my co-supervisor, Dr. Paul Ezhilchelvan, who is an expert in this area. In order to brief him on the work I'd done so far as well as my initial efforts to map queueing theory concepts onto Striot's concepts, I prepared a brief presentation:

24 May 2021

Jonathan Dowland: OpenJDK Runtime Containers

The UBI OpenJDK containers are full fat developer containers: they've got the full OpenJDK distribution in them (including javac etc.); they include Maven and there are various quality-of-life additions such as a Prometheus agent for metrics gathering in OpenShift and a run-java script to control launching the eventual application, as well as OpenShift Source-To-Image (S2I) integration. We've had a number of customers request that we produce some runtime-only images. It took a while to properly scope the request, as each person who wanted such a thing wanted a different subset of functionality included from the existing images. One consistent request though related to image size: It became clear that the runtime images should be as minimal as possible. Last week we launched ubi8/openjdk-8-runtime and ubi8/openjdk-11-runtime containers which weight in at 106.M1iB and 119.7 MiB respectively. They are extremely bare-bones, containing just a subset of the OpenJDK distribution (that which is packaged as java-N-openjdk-headless in RHEL) on top of the UBI minimal flavour base images. To explain one way these could be use in an OpenShift environment I wrote a blog post for Red Hat Developer: Build lean Java containers with the new Red Hat Universal Base Images OpenJDK runtime images. We've got a couple of avenues under exploration in this area which should result in even leaner or more tailored images in the future. I'll write more when we have something to show!

Jonathan Dowland: Answering my own Template Haskell question

A couple of weeks ago I stumbled across a Template Haskell question on the Haskell sub-reddit. This was quite exciting because I rarely see TH questions, and this one was by someone working on something which was quite similar to what I had done: they were writing a code generator, and had pure functions of type Exp -> Exp -> Exp, but wanted to leverage TH's quasi-quotation syntax (e.g. [ \p -> p + 1 ]) as a syntactic short-hand. Alas these quasi-quotes are of type Q Exp, so the question becomes, how to escape Q? I had to do exactly this, so I knew an answer: use unsafePerformIO. Another commenter pointed at this Overloaded Quotation Brackets GHC Proposal which I was not aware of and is a much more elegant solution, available since GHC 9.0.1. There was something odd about the question, though. Sometime later on, I checked back to find that the thread had been deleted. It turns out it was one of a set of duplicate posts with exactly the same question. Spammers are copying old questions and reposting them, presumably in an attempt to harvest "Reddit karma". When I learned that, my sense of oddness about the question remained, so I went to find the original thread, to discover I wrote the original question a year ago and had therefore literally answered my own question a year later. Back to the more elegant solution. In GHC versions 9 onwards, quasi-quotation returns a different type that is not bound up in the Q Monad: GHC 8.8.4:
 > :t [  42  ]
[  42  ] :: Language.Haskell.TH.Lib.Internal.ExpQ
 > [  42  ]
<interactive>:5:1: error:
      No instance for (Show Language.Haskell.TH.Lib.Internal.ExpQ)
        arising from a use of  print 
      In a stmt of an interactive GHCi command: print it
GHC 9.0.1:
 > :t [  42  ]
[  42  ]
  :: Language.Haskell.TH.Syntax.Quote m =>
     m Language.Haskell.TH.Syntax.Exp
 > [  42  ]
LitE (IntegerL 42)

11 April 2021

Jonathan Dowland: 2020 in short fiction

Cover for *Episodes*
Following on from 2020 in Fiction: In 2020 I read a couple of collections of short fiction from some of my favourite authors. I started the year with Christopher Priest's Episodes. The stories within are collected from throughout his long career, and vary in style and tone. Priest wrote new little prologues and epilogues for each of the stories, explaining the context in which they were written. I really enjoyed this additional view into their construction.
Cover for *Adam Robots*
By contrast, Adam Robert's Adam Robots presents the stories on their own terms. Each of the stories is written in a different mode: one as golden-age SF, another as a kind of Cyberpunk, for example, although they all blend or confound sub-genres to some degree. I'm not clever enough to have decoded all their secrets on a first read, and I would have appreciated some "Cliff's Notes on any deeper meaning or intent.
Cover for *Exhalation*
Ted Chiang's Exhalation was up to the fantastic standard of his earlier collection and had some extremely thoughtful explorations of philosophical ideas. All the stories are strong but one stuck in my mind the longest: Omphalos) With my daughter I finished three of Terry Pratchett's short story collections aimed at children: Dragon at Crumbling Castle; The Witch's Vacuum Cleaner and The Time-Travelling Caveman. If you are a Pratchett fan and you've overlooked these because they're aimed at children, take another look. The quality varies, but there are some true gems in these. Several stories take place in common settings, either the town of Blackbury, in Gritshire (and the adjacent Even Moor), or the Welsh border-town of Llandanffwnfafegettupagogo. The sad thing was knowing that once I'd finished them (and the fourth, Father Christmas's Fake Beard) that was it: there will be no more.
Cover for Interzone, issue 277
8/31 of the "books" I read in 2020 were issues of Interzone. Counting them as "books" for my annual reading goal has encouraged me to read full issues, whereas before I would likely have only read a couple of stories from each issue. Reading full issues has rekindled the enjoyment I got out of it when I first discovered the magazine at the turn of the Century. I am starting to recognise stories by authors that have written stories in other issues, as well as common themes from the current era weaving their way into the work (Trump, Brexit, etc.) No doubt the Pandemic will leave its mark on 2021's stories.

23 March 2021

Jonathan Dowland: The Cure: 40 Live

Back in 2019, The Cure released a double-video-album featuring two live shows from 2018, their 40th anniversary concert at Hyde Park (which I attended) and their experimental "Curaetion" show from the Meltdown festival a few weeks earier. Rumours that there would be a release had been circulating since the gig itself, where video director Tim Pope was spotted filming the gig (sporting a tee shirt reading "Yes I'm Filming. No I don't know when you'll see it."). Earlier in 2019 the "Anniversary" half had a limited run in cinemas, but the home video release was not officially revealed until much later, and the inclusion of Curaetion an unexpected bonus. 'deluxe' releases The release is available in several configurations. The "standard" issue is a two-DVD (or Blu Ray) set in a hard-book case, with some liner notes in the middle. A "deluxe" issue is a roughly twice-as-large boxed set containing the two video discs and two CD-audio discs of the shows. When it comes to deluxe editions, I would historically have been the perfect "mark", but I've had some bad experiences with other releases (this is a story for another blog post), so for this release I pre-ordered the standard BR edition. Right up until release, I wondered whether I should have made the other choice. Eventually, pictures of other people's Deluxe Editions started to circulate, and I was re-assured that I made the right call. The larger box is just something harder to store, the larger booklet is an exact reproduction of the smaller, standard edition liner notes, which are not really notable. The shows I think the shows are great. The Hyde Park show is a fairly standard "best of" setlist, and my enjoyment of it might be coloured by having been at the show and having fond memories of it. But it's a great performance and very well captured, so I think there's something for people who like the Cure but weren't there to enjoy too. I wasn't at the Meltdown show. The setlist for that one is much more esoteric, and it's lovely to have renditions of songs that are rarely performed live (And: across the two sets you get If Only Tonight We Could Sleep twice). Getting the audio I knew in advance that I would likely listen to the shows more often than watch them, so the CDs in the Deluxe set would have been useful for that, but I was fairly sure the audio mix would be the same as on the video discs, so I could just extract that. I was half-right: the BR set (at least) has 5.1 surround and separate lossless stereo mixes, but they have been mastered quite differently to the CD audio, which appears to be brick-wall mastered, to such an extent that the audio distorts quite noticeably in a few places. The BR audio does not seem to suffer from the same problems. Getting the audio off the discs was interesting. On Linux, the tool of choice for decoding encrypted Blu Rays is libaacs, which requires a list of Blu Ray keys which you must source separately. With my BR drive I was not able to decode the discs using the keys I could find online. However, the separate tool MakeMKV was able to decode them. I don't know exactly how but I speculate that MakeMKV contains some leaked hardware player keys that are not in more general circulation. Once I had decoded copies of the two videos, the trusty tool ffmpeg is able to easily extract the PCM stereo audio track. Unfortunately it cannot, in one go, split the audio track at the chapter boundary points. This is easy to script around: one ffmpeg invocation to get the chapter offsets, then some scripting to turn that into seek and duration command-line arguments for further ffmpeg invocations. For space reasons, I opted to transcode to FLAC at the same time as extracting each track. I hit an interesting (although not fatal) problem: the time taken to extract each track got progressively slower, almost as if ffmpeg was transcoding the entire audio track up to the point specified as the offset to start extracting from. If I wanted to answer categorically whether the BR audio was preferable, I would need to perform a double-blind listening test, using a tool such as squishyball. I certainly can't be bothered: the BR audio is fine for me.

13 March 2021

Jonathan Dowland: Internal Gotek

This is the eleventh part in a series of blog posts. The previous post was FlashFloppy OLED display. The whole series is available here: Amiga.
It's still mostly all quiet on the Amiga project front. The last few months with both my kids at home have been too difficult and busy with no time for side projects. Primary Schools in the UK went back this week, and I've finally got around to installing the Gotek floppy emulator inside the Amiga A500's chassis. I followed A short "Internal Gotek Floppy" Installation Guide , which explains that you don't need to print (or buy) a special mount for it: Simply removing the Gotek's top plastic case is enough for it to fit inside, and you can support it with some nuts and bolts.
Gotek bolted into place Gotek bolted into place
Easy in theory, but it still took me a while, with plenty of trial and error to line it up just right. I ended up using some M4 bolts and smaller (the kind that you use in electrical sockets) to support the bottom part of the Gotek case in three places: one, at the front, re-using an original vertical support for the old floppy drive; and two at the back, screwing through two holes in the A500's outer case and mainboard. If I'd chosen short enough bolts, the back of the Gotek case could have probably just rested on the ends of those bolts, but the ones I had were another few cm longer, so I drilled holes in the Gotek case to run the bolts through, and secured it with nuts.
Knob, display and mount, external Knob, display and mount, external
In that guide, the author has an OLED display for their Gotek that they wanted on the top of the Amiga case, so they fed the four tiny wires through the grill. I had a go at that, but I've got the OLED display and a rotary encoder, so that's 8 wires total, and damn fiddly. Instead for now I've just routed them all out through the hole left behind by the floppy drive's Button. I'm going to try and tidy it up with some heat shrink or similar. So finally, my Amiga is one "unit" again, without dangly bits hanging off it (if you ignore those 8 tiny wires for now) that I can move around to store or use much more easily.

Next.