Search Results: "jmtd"

20 April 2016

Jonathan Dowland: mount-on-demand backups

Last week, someone posted a request for help on the popular Server Fault Q&A site: they had apparently accidentally deleted their entire web hosting business, and all their backups. The post (now itself deleted) was a reasonably obvious fake, but mainstream media reported on it anyway, and then life imitated art and 123-reg went and did actually delete all their hosted VMs, and their backups. I was chatting to some friends from $job-2 and we had a brief smug moment that we had never done anything this bad, before moving on to incredulity that we had never done anything this bad in the 5 years or so we were running the University web servers. Some time later I realised that my personal backups were at risk from something like this because I have a permanently mounted /backup partition on my home NAS. I decided to fix it. I already use Systemd to manage mounting the /backup partition (via a backup.mount file) and its dependencies. I'll skip the finer details of that for now. I planned to define some new Systemd units for each backup job which was previously scheduled via Cron in order that I could mark them as depending on the /backup mount. I needed to adjust that mount definition by adding StopWhenUnneeded=true. This ensures that /backup will be unmounted when it is not in use by another job, and not at risk of a stray rm -rf. The backup jobs are all simple shell scripts that convert quite easily into services. An example: backup-home.service:
[Unit]
Requires=backup.mount
After=backup.mount
[Service]
User=backupuser
Group=backupuser
ExecStart=/home/backupuser/bin/phobos-backup-home
To schedule this, I also need to create a timer: backup-home.timer:
[Timer]
OnCalendar=*-*-* 04:01:00
[Install]
WantedBy=timers.target
To enable the timer, you have to both enable and start it:
systemctl enable backup-home.timer
systemctl start backup-home.timer
I created service and timer units for each of my cron jobs. The other big difference to driving these from Cron is that by default I won't get any emails if the jobs generate output - in particular, if they fail. I definitely do want mail if things fail. The Arch Wiki has an interesting proposed solution to this which I took a look at. It's a bit clunky, and my initial experiments with a derivation from this (using mail(1) not sendmail(1)) have not yet generated any mail. Pros and Cons The Systemd timespec is more intuitive than Cron's. It's a shame you need a minimum of three more lines of boilerplate for the simplest of timers. I think WantedBy=timers.target should probably be an implicit default for all .timer type units. Here I think clarity suffers in the name of consistency. With timers, start doesn't kick-off the job, it really means "enable" in the context of timers, which is clumsy considering the existing enable verb, which seems almost superfluous, but is necessary for consistency, since Systemd units need to be enabled before they can be started As Simon points out in the comments, this is not true. Rather, "enable" is needed for the timer to be active upon subsequent boots, but won't enable it in the current boot. "Start" will enable it for the current boot, but not for subsequent ones. Since I need a .service and a .unit file for each active line in my crontab, that's a lot of small files (twice as many as the number of jobs being defined) and they're all stored in system-wide folder because of the dependency on the necessarily system-level units defining the mount. It's easy to forget the After= line for the backup services. On the one hand, it's a shame that After= doesn't imply Require=, so you don't need both; or alternatively there was a convenience option that did both. On the other hand, there are already too many Systemd options and adding more conjoined ones would just make it even more complicated. It's a shame I couldn't use user-level units to achieve this, but they could not depend on the system-level ones, nor activate /backup. This is a sensible default, since you don't want any user to be able to start any service on-demand, but some way of enabling it for these situations would be good. I ruled out systemd.automount because a stray rm -rf would trigger the mount which defeats the whole exercise. Apparently this might be something you solve with Polkit, as the Arch Wiki explains, which looks like it has XML disease. I need to get mail-on-error working reliably.

3 April 2016

Jonathan Dowland: My first DSA

I've been a Debian developer for 7 years now, but this last week was the first time I've had to deal with a security bug severe enough to warrant a Debian Security Advisory: DSA 3540-1, for lhasa, the lzh archive decompressor. If you use lhasa via the Debian package, please upgrade. (Debian timeline duly updated)

1 April 2016

Jonathan Dowland: April Fools

I'm not a big fan of April Fools' day, but this year I decided to try one. My modest contribution: a Pull Request for Chocolate Doom titled Bring back the classic Doom "hit enter" behaviour:
When playing PWADs with classic Doom, you always got the "this game has been modified" banner and had to hit enter to continue. If you played a lot of PWADs, you got used to hitting enter regularly. It became a core part of the Doom-playing experience. Even if you used one of those lovely launchers that came on the shovelware CDs which streamlined the PWAD choosing experience, you always had to hit the enter key. This commit restores this classic behaviour and allows those who still have the muscle-memory "hit enter" behaviour permanently burned into their nervous systems to finally be at peace. TODO for a future commit: the coloured top-line of text (red for 1.666, blue for 1.9 iirc)
Some feedback from IRC:
<Linguica> its funny because its nearly indistinguishable from real chocolate doom feature requests <jmickle> i actually first thought "huh, kinda surprised this isn't in already <ryan> best chocolate-doom feature yet! ;)

3 February 2016

Jonathan Dowland: Comparing Docker images

I haven't written much yet about what I've been up to at work. Right now, I'm making changes to the sources of a set of Docker images. The changes I'm making should not result in any changes to the actual images: it's just a re-organisation of the way in which they are built. I've been using the btrfs storage driver for Docker which makes comparing image filesystems very easy from the host machine, as all the image filesystems are subvolumes. I use a bash script like the following to make sure I haven't broken anything:
oldid="$1"; newid="$2";
id_in_canonical_form()  
    echo "$1"   grep -qE '^[a-f0-9] 64 $'
 
canonicalize_id()  
    docker inspect --format '  .Id  ' "$1"
 
id_in_canonical_form "$oldid"   oldid="$(canonicalize_id "$oldid")"
id_in_canonical_form "$newid"   newid="$(canonicalize_id "$newid")"
cd "/var/lib/docker/btrfs/subvolumes"
sumpath()  
    cd "$1" && find . -printf "%M %4U %4G %16s %h/%f\n"   sort
 
diff -ruN "$oldid" "newid"
diff -u <(sumpath "$oldid") <(sumpath "$newid")
Using -printf means I can ignore changes in the timestamps on files which is something I am not interested in. If it is available in your environment, Lars Wirzenius' tool Summain generates manifests that include a file checksum and could be very useful for this use-case.

21 January 2016

Jonathan Dowland: Network Attached Storage

little and large little and large
I've just upgraded my home Network Storage box. Until now, it was very discreet, about the size of a car radio. It lived on one of my bookcases and I was very happy with it but it could only take 2.5" sized hard drives and I've outgrown what you can fit on them. I've ended up moving to 2x4T 3.5" drives, a Coolermaster Elite 110 case and re-using the mainboard/CPU and RAM. This is hopefully enough capacity that I don't need to look at another upgrade for a few years, and the case could take more drives if I need it to. The Elite 110 is - by Mini-ITX case terms - pretty small, but still 6 times larger in volume than the old case. In the consumer NAS market, there are plenty of appliances that takes two 3.5" drives and are much smaller than that (such as the WD MyBook Duo, or Synology DS214). Unfortunately I could not find any cases that were of this size for a DIY build. Did I miss anything obvious? (As to why I choose DIY, that could be the subject for a future post.) The full boring details of the build are on the NAS system page.

8 January 2016

Jonathan Dowland: Game boxes

a big pile of computer game boxes computer game boxes
My brother and I found these whilst tidying up the stuff that we've left at our parent's house. What on earth should we do with them? I have a small number of older game boxes (Doom, Doom 2, Civ 2, etc.) which I use as drawer dividers (both the lid and the 'seat' of the box), but most of these are newer designs, more like a box of cereal, with their openings at the top, so they can't be used as such. (and there's only so many drawer dividers that you need...) This is loosely related to my stalled Amiga floppy disk recovery project, which hasn't made any progress since I last wrote about it.

7 December 2015

Jonathan Dowland: Download codes are distinct copies of albums

In CDs should come with download codes, I argued that CDs should come with a coupon to obtain a digital copy of the album, much like a lot of modern vinyl releases do. What didn't occur to me at the time is that there's a difference in how we think about copies of albums that we obtain that way, versus ripping a CD. When we rip a CD, we are making a personal copy of the media. We've bought a physical "token" that represents our "ownership" of the music, and we've made a copy of that privately for our own convenience. If we decide to get rid of the CD, then morally we really should get rid of the digital copy, too, as we've transferred our "ownership" to someone else. Of course we don't really own the music on the CD in the first place and in the UK it's technically not legal to make a copy, even a private copy for personal use, but that's another story. With a download code, the situation is quite different. If you buy a record with a download code, you have (at least) two copies of the album. You don't make a second copy yourself, it has already been made. And if you decide to pass on the physical album, I don't think there is any moral expectation that you should delete your digital copy.

23 November 2015

Jonathan Dowland: CDs should come with download codes

boxes of CDs & the same data on MicroSD boxes of CDs & the same data on MicroSD
There's a Vinyl resurgence going on, with vinyl record sales growing year-on-year. Many of the people buying records don't have record players. Many records are sold including a download code, granting the owner an (often one-time) opportunity to download a digital copy of the album they just bought. Some may be tempted to look down upon those buying vinyl records, especially those who don't have a means to play them. The record itself is, now more than ever, a physical totem rather than a media for the music. But is this really that different to how we've treated audio CDs this century? For at least 15 years, I've ripped every CD I've bought and then stored it in a shoebox. (I'm up to 10 shoeboxes). The ripped copy is the only thing I listen to. The CD is little more than a totem, albeit one which I have to use in a relatively inconvenient ritual in order to get something I can conveniently listen to. The process of ripping CDs has improved a lot in this time, but it's still a pain. CD-ROM drives are also becoming a lot more scarce. Ripping is not necessary reliable, either. The best tool to verify a rip is AccurateRip, a privately-owned database of track checksums. The private status is a problem for the community (Remember what happened to CDDB?) and is only useful if other people using an AccurateRip-supported ripper have already successfully ripped the CD. Then there's things like CD pre-emphasis. It turns out that the Red Book standard defines a rarely-used flag that means the CD (or individual tracks) have had pre-emphasis applied to the treble-end of the frequency spectrum. The CD player is supposed to apply de-emphasis on playback. This doesn't happen if you fetch the audio data digitally, so it becomes the CD rippers responsibility to handle this. CD rippers have only relatively recently grown support for it. Awareness has been pretty low, so low that nobody has a good idea about how many CDs actually have pre-emphasis set: it's thought to be very rare, but (as far as I know) MusicBrainz doesn't (yet) track it. So some proportion of my already-ripped CDs may have actually been ripped incorrectly, and I can't easily determine which ones without re-ripping them all. I know that at least my Quake computer game CD has it set, and I have suspicions about some other releases. Going forward, this could be avoided entirely if CDs were treated more like totems, as vinyl records are, than the media delivering the music itself, and if record labels routinely included download cards with audio CDs. For just about anyone, no matter how the music was obtained, media-less digital is the canonical form for engaging with it. Attention should also be paid to make sure that digital releases are of a high quality: but that's a topic for another blog post.

Jonathan Dowland: On BBC 6 Music

Back in July I had a question of mine read out on the Radcliffe and Maconie programme on BBC 6 Music. The pair were interviewing Stephen Morris of New Order and I took the opportunity to ask a question about backing vocals on the 1989 song "Run2". Here's the question and answer (318K MP3, 21s):

20 November 2015

Jonathan Dowland: smartmontools

It's been at least a year since I last did any work on Debian, but this week I finally uploaded a new version of squishyball, an audio sample comparison tool, incorporating a patch from Thibaut Girka which fixes the X/X/Y test method. Shamefully Thibaut's patch is nearly a year old too. Better late than never... I've also uploaded a new version of smartmontools which updates the package to the new upstream version. I'm not the regular maintainer for this package, but it is in the set of packages covered by the collab-maint team. To be polite I uploaded it to DELAYED-7, so it will take a week to hit unstable. I've temporarily put a copy of the package here in the meantime.

11 November 2015

Jonathan Dowland: Useful Mac programs

A little over a year ago I wrote about how I'd been using a Mac as my main work machine. I hadn't written anything on the subject since. Here are four useful Mac programs that I can recommend to people. Sizeup, from Irradiated Software. A bit Like Windows' aero snap, but on steroids. I love this. I regularly move windows between two desktops (external and internal display), resize and centre them or put them in one quarter of the screen with just a few presses. The "snapback" feature is also great. X Lossless Decoder (XLD). A handy transcoder that can use QuickTime encoders and so can write out Apple/Quicktime/iTunes-encoded AAC/MP4 files en-mass, translating file meta-data.
LimeChat LimeChat
LimeChat, an IRC client. I theme it to look like Colloquy. The net result is pretty clean. I like using a proportional-font for text things, so getting IRC out of a terminal is a win for me. LimeChat automatically fetches and thumbnails URIs to images that people paste in channels, which is either incredibly convenient or a curse, depending on the channel. You can toggle that behaviour but only across the whole client, not on a channel or network basis. Finally, 1Password is an incredibly slick password manager. I use it in a very basic fashion: no mobile client, no syncing to anything outside of my workstation. You could also use LastPass which is similar and has a Linux client. I haven't tried it, but there's a third party tool to read 1Password password stores on Linux written by Ryan Coleman of SDL fame called 1pass.

16 October 2015

Jonathan Dowland: Ikiwiki-in-a-box Docker container

For a while I've been wanting to tidy up some of my existing ikiwiki hacks, as well as test some changes I'd like to make to this site, but I hadn't had much success with setting up environments to play with it. That was until earlier in the year, when as a bit of an experiment I tried to put ikiwiki in docker. I've put up a preliminary image for anyone who is curious. It's in a very early state. When you start up the image, you get a HTTP daemon listening on port 8080 and serving a basic ikiwiki from /. A single administrative user is defined, user admin and password password. The CGI is set up (path /cgi) and websetup is enabled. The ikiwiki instance is configured with git as the vcs backend and the git HTTP backend is also enabled at path /git. It is set up for anonymous pull and HTTP basic authentication for push, with admin/password. There's still a lot to do before this is useful for what I want to do with it, and I would strongly advise against using it for anything except testing yourself, unless you are going to layer on some changes (HTTPS and change the credentials at the very least!) but I thought I'd put it out there. If you have any suggestions for it, please either file an issue or leave a comment here. It's built on top of the stuff I mentioned last time but it should rebase on top of the "semi-official" Debian images without problems.

12 October 2015

Jonathan Dowland: Johnny Marr

Last night I saw Johnny Marr play at Newcastle University Student's Union. This is the first time I've been to a gig at the refurbished NUSU. I think the last gig I saw there was Nouvelle Vague in around 2007. It was a great gig! The accoustics in the Union are pretty good. I seem to recall they've always been pretty good. It used to be very cheap, too, which is sadly no longer true. Johnny Marr is an eminently likeable character. He did a set with a fairly balanced mix of his solo work from across two albums, Smiths songs and a few other bits and pieces, including a surprise cover of "Crash" by the Primitives, which by a seeming coincidence I had re-discovered a few weeks ago and can't stop listening to. The stand-out tracks for me were * Strikingly similar to Gary Numan's Metal. I think this must be intentional, because the guitar work at the end of the verses also sounds a lot like Numan's Complex, which is the track that immediately follows Metal on The Pleasure Principle. Speak Out Reach Out is a great song, and not just due to the similarity.)

25 September 2015

Jonathan Dowland: WadC 2.0 released

WadC
This week I released version 2.0 of Wad Compiler, a lazy functional programming language and IDE for the construction of Doom maps. Version 2.0 is the first version in about four years and adds a fair number of features, most notably the ability to compose textures in your code and a basic command-line interface. For more information see the release notes and the reference.

20 September 2015

Jonathan Dowland: Kolo ep, Dubrovnik, Dalmatia

Trsteno Trsteno (near Botanical gardens)
This year, Sarah and I spent two weeks off the coast of Dubrovnik, on Kolo ep island. We'd never been to Croatia before, and I was a little nervous that if there wasn't much to do on the island, we wouldn't be able to get elsewhere particularly easily. I needn't have worried: We thoroughly enjoyed our stay. It might even be my favourite holiday so far. We did travel around a bit: twice to Dubrovnik's old town and once to a botanical gardens in Trsteno but there was also loads to see on the island itself. One of our favourite trips was simply to a cave on the opposite side of the island known as a "blue cave" because you looked like a Smurf on the inside. We had to jump into the sea from the boat and swim into and out of the cave before continuing on our trip. Simply puttering about on the boat, either around the island or to Trsteno was a great experience in itself. Our guide Sammi was very friendly, the weather was great and swimming around or just enjoying the sun on the back of the boat was enough for me to want to check property prices out there. On Kolo ep island is a little restaurant called Villa Ruza. We managed to eat there twice during our stay. I think it is considered to be one of the top ten restaurants in Croatia. It's one of the best places I've ever eaten in my life. In terms of price, it was about half that of restaurants in Dubrovnik itself.
Kolo ep Kolo ep bay, from Villa Ruza
Dubrovnik itself was, quite predictably, much busier than Kolo ep. It is a stunningly attractive old town. On our second visit we took a tour. There are a variety of different themed tours that you can take, including Game Of Thrones sight-seeing tours, and there were Game Of Thrones tat shops all over the place. We opted for A Story About The War. One thing we love to do when visiting foreign places is to get an idea of what life is like for people there, and this war was not all that long ago - I can remember the news coverage from the time. We were the only two people on that particular tour, so we had a fairly tailored experience. I wish I could remember our guide's name. She was the same age as us and grew up as a child in the city when it was under siege. Her personal story included details about where her family stayed; what life was like for kids growing up at the time; how people got supplies; which areas were badly affected and much more. It was a very heartfelt tour and gave us an intimate and personal portrait of what life was like for people there at a time that most tour operators tend to prefer to ignore. This was exactly what we were looking for. Sarah and I were both a little teary at the end! We tried to do things a little different on this holiday. Normally we keep ourselves to ourselves and don't socialise much with other guests. We also rarely do trips and excursions, preferring to sort things out for ourselves. This time we made an effort to be more sociable and I'm glad we did because we met some really nice people. The trips we did were great fun and on the last day I blew through most of the rest of our currency by hiring a jet ski for half an hour. Those things are FUN. Top speed on the one I was using was 57 land mph. Every muscle in my body was screaming for a few days afterwards! Often on a two-week holiday we reach a point around 10 days where we've pretty much had enough. This time around I could have stayed for twice as long. This was the first proper test of my new camera and I've uploaded a set of photographs from our holiday to a Flickr gallery.

28 July 2015

Jonathan Dowland: Sound effect pitch-shifting in Doom

My previous blog posts about deterministic Doom proved very popular. The reason I was messing around with Doom's RNG was I was studying how early versions of Doom performed random pitch-shifting of sound effects, a feature that was removed early on in Doom's history. By fixing the random number table and replacing the game's sound effects with a sine wave, one second long and tuned to middle-c, I was able to determine the upper and lower bounds of the pitch shift. Once I knew that, I was able to write some patches to re-implement pitch shifting in Chocolate Doom, which I'm pleased to say have been accepted. The patches have also made their way into the related projects Crispy Doom and Doom Retro. I'm pleased with the final result. It's the most significant bit of C code I've ever released publically, as well as my biggest Doom hack and the first time I've ever done any audio manipulation in code. There was a load of other notes and bits of code that I produced in the process. I've put them together on a page here: More than you ever wanted to know about pitch-shifting.

21 July 2015

Jonathan Dowland: New camera

Sony RX100-III
Earlier in the year I treated myself to a new camera. It's been many years since I bought one, which was a perfectly serviceable Panasonic FS-15 compact, to replace my lost-or-stolen Panasonic TZ3, which I loved. The FS-15 didn't have a "wow" factor and with the advent of smartphones and fantastic smartphone cameras, it rarely left a drawer at home. Last year I upgraded my mobile from an iPhone to a Motorola Moto G, which is a great phone in many respects, but has a really poor camera. I was given a very generous gift voucher when I left my last job and so had the perfect excuse to buy a dedicated camera. I'd been very tempted by a Panasonic CSC camera ever since I read this review of the GF1 years ago, and the GM1 was high on my list, but there were a lot of compromises: no EVF... In the end I picked up a Sony RX 100 Mark 3 which had the right balance of compromises for me. I haven't posted a lot of photos to this site in the past but I hope to do so in future. I've got to make some alterations to the software first. Post-script: Craig Mod, who wrote that GF1 review, wrote another interesting essay a few years later: Cameras, Goodbye, where he discusses whether smartphone cameras are displacing even the top end of the Camera market.

13 May 2015

Jonathan Dowland: Amiga floppy recovery project

I've still got it! I've still got it!
Recovered floppy disks Recovered floppy disks
My first computer was an Amiga A500, and my brother and I spent a fair chunk of our childhoods creating things with it. These things are locked away on 3.5" floppy disks, but they were also lost a long time ago. A few weeks ago my dad found them in a box in his loft, so a disk-reading project is now on the horizon! Step one is to catalogue what we've got, which I've done here. Step two is to check which, if any, of these are not already in circulation amongst archivists. Thanks to Matthew Garrett for pointing me at the Software Preservation Society, which is a good first place to check. When we get to the reading step, there are quite a few approaches I could take. Which one to use depends to some extent on which disks we need to read, and whether they employ any custom sector layout or other copy protection schemes. I think the easiest method using equipment I already have is probably Amiga Explorer and a null-modem cable, as this approach will work on an A500 with Workbench 1.3. There are a variety of hardware tools and projects for reading Amiga floppies on a PC, but the most interesting one to me is DiscFerret, which is open hardware and software.

28 April 2015

Jonathan Dowland: Deterministic Doom video

My last blog post proved popular, and I've had some requests for a video demonstrating it, so I've put one together:

23 April 2015

Jonathan Dowland: Deterministic Doom

What happens if you remove randomness from Doom? For some reason, recently I have been thinking about Doom. This evening I was wanting to explore some behaviour in an old version of Doom and to do so, I hex-edited the binary and replaced the random number lookup table with static values. Rather than consume system randomness, Doom has a fixed 256-value random number table from which numbers are pulled by aspects of the game logic. By replacing the whole table with a constant value, you essentially make the game entirely deterministic. What does it play like? I tried two values, 0x00 and 0xFF. With either value, the screen "melt" effect that is used at the end of levels is replaced with a level vertical wipe: the randomness was used to offset each column. Monsters do not make different death noises at different times; only one is played for each category of monster. The bullet-based (hitscan) weapons have no spread at all: the shotgun becomes like a sniper rifle, and the chain-gun is likewise always true. You'd think this would make the super-shotgun a pretty lethal weapon, but it seems to have been nerfed: the spread pattern is integral to its function. With 0x00, monsters never make their idle noises (breathing etc.) On the other hand, with 0xFF, they always do: so often, that each sample collides with the previous one, and you just get a sort-of monster drone. This is quite overwhelming with even a small pack of monsters. With 0xFF, any strobing sectors (flickering lights etc.), are static. However, with 0x00, they strobe like crazy. With 0x00, monsters seem to choose to attack much more frequently than usual. Damage seems to be worst-case. The most damaging floor type ("super hellslime"/20%) can hurt you even if you are wearing a radiation suit: There was a very low chance of it hurting whilst wearing the suit (~2.6%) each time the game checked; this is rounded up to 100%. Various other aspects of the game become weird. A monster may always choose to use a ranged attack, regardless of how close you are. They might give up pursuing you. I've seen them walk aimlessly in circles if they are obstructed by another thing. The chance of monster in-fighting is never, or a certainty. The player is either mute, or cries in pain whenever he's hurt. If you want to try this yourself, the easiest way is to hack the m_random.c file in the source, but you can hex-edit a binary. Look for a 256-byte sequence beginning beginning ['0x0', '0x8', '0x6d', '0xdc', '0xde', '0xf1'] and ending ['0x78', '0xa3', '0xec', '0xf9'].

Next.

Previous.