Search Results: "jgoerzen"

25 August 2021

John Goerzen: Excellent Experience with Debian Bullseye

I ve appreciated the bullseye upgrade, like most Debian upgrades. I m not quite sure how, since I was already running a backports kernel, but somehow the entire system is snappier. Maybe newer X or something? I m really pleased with it. Hardware integration is even nicer now, particularly the automatic driverless support for scanners in addition to the existing support for printers. All in all, a very nice upgrade, and pretty painless. I experienced a few odd situations. For one, I had been using Gnome Flashback. Since xmonad-log-applet didn t compile there (due to bitrot in the log applet, not flashback), and I had been finding Gnome Flashback to be a rather dusty and forgotten corner of Gnome for a long time, I decided to try Mate. Mate just seemed utterly unable to handle a situation with a laptop and an external monitor very well. I want to use only the external monitor with the laptop lid is closed, and it just couldn t remember how to do the right thing external monitor on, laptop monitor off, laptop not put into suspend. gdm3 also didn t seem to be able to put the external monitor to sleep, either, causing a few nights of wasted power. So off I went to XFCE, which I had been using for years on my workstation anyhow. Lots more settings available in XFCE, plus things Just Worked there. Odd that XFCE, the thin and light DE, is now the one that has the most relevant settings. It seems the Gnome let s remove a bunch of features approach has extended to MATE as well. When I switched to XFCE, I also removed gdm3 from my system, leaving lightdm as the only DM on it. That matched what my desktop machine was using, and also what task-xfce-desktop called for. But strangely, the XFCE settings for lightdm were completely different between the laptop and the desktop. It turns out that with lightdm, you can have the lightdm-gtk-greeter and the accompanying lightdm-gtk-greeter-settings, or slick-greeter and the accompanying lightdm-settings. One machine had one greeter and settings, and the other had the other. Why, I don t know. But lightdm-gtk-greeter-settings had the necessary options for putting monitors to sleep on the login screen, so I went with it. This does highlight a bit of a weakness in Debian upgrades. There is SO MUCH choice in Debian, which I highly value. At some point, almost certainly without my conscious choice, one machine got one greeter and another got the other. Despite both having task-xfce-desktop installed, they got different desktop experiences. There isn t a great way to say OK, I know I had a bunch of things installed before, but NOW I want the default bullseye experience . But overall, it is an absolutely fantastic distribution. It is great to see this nonprofit community distribution continue to have such quality on such an immense scale. And hard to believe I ve been a Debian developer for 25 years. That seems almost impossible!

18 August 2021

John Goerzen: Distributed, Asynchronous Git Syncing with NNCP

I have a problem. I have a directory that I use with org-mode and org-roam. I want it to be synced across multiple machines. I also want to keep the history with git. And, I want to use end-to-end encryption (no storing a plain git repo on a remote server), have a serverless setup, not require any two machines to be up simultaneously, and be resilient in the face of races and conflicts. Whew. I ve tried a number of setups git-remote-gcrypt on a remote server (fragile), some complicated scripts around a separate repo in syncthing (requires one machine to be in charge ), etc. They all were subpar. Then NNCP introdoced asynchronous multicast and I was intrigued. So, I wrote gitsync-nncp, which uses NNCP to distribute git bundles to all the participating machines. The comprehensive documentation for gitsync-nncp goes into a lot more detail about how it works and what problems it solves. It s working quite well for me!

30 December 2020

John Goerzen: Airgapped / Asynchronous Backups with ZFS over NNCP

In my previous articles in the series on asynchronous communication with the modern NNCP tool, I talked about its use for asynchronous, potentially airgapped, backups. The first article, How & Why To Use Airgapped Backups laid out the foundations for this. Now let s dig into the details. Today s post will cover ZFS, because it has a lot of features that make it very easy to support in this setup. Non-ZFS backups will be covered later. The setup is actually about as simple as it is for SSH, but since people are less familiar with this kind of communication, I m going to try to go into more detail here. Assumptions I am assuming a setup where: Hardware Let s start with hardware for the machine to hold the backups. I initially considered a Raspberry Pi 4 with 8GB of RAM. That would probably have been a suitable machine, at least for smaller backup sets. However, none of the Raspberry Pi machines support hardware AES encryption acceleration, and my Pi4 benchmarks as about 60MB/s for AES encryption. I want my backups to be encrypted, and decided this would just be too slow for my purposes. Again, if you don t need encrypted backups or don t care that much about performance may people probably fall into this category you can have a fully-functional Raspberry Pi 4 system for under $100 that would make a fantastic backup server. I wound up purchasing a Qotom-Q355G4 micro PC with a Core i5 for about $315. It has USB 3 ports and is designed as a rugged, long-lasting system. I have been using one of their older Celeron-based models as my router/firewall for a number of years now and it s been quite reliable. For backup storage, you can get a USB 3 external drive. My own preference is to get a USB 3 toaster (device that lets me plug in SATA drives) so that I have more control over the underlying medium and can save the expense and hassle of a bunch of power supplies. In a future post, I will discuss drive rotation so you always have an offline drive. Then, there is the question of transport to the backup machine. A simple solution would be to have a heavily-firewalled backup system that has no incoming ports open but makes occasional outgoing connections to one specific NNCP daemon on the spooling machine. However, for airgapped operation, it would also be very simple to use nncp-xfer to transport the data across on a USB stick or some such. You could set up automounting for a specific USB stick plug it in, all the spooled data is moved over, then plug it in to the backup system and it s processed, and any outbound email traffic or whatever is copied to the USB stick at that point too. The NNCP page has some more commentary about this kind of setup. Both are fairly easy to set up, and NNCP is designed to be transport-agnostic, so in this article I m going to focus on how to integrate ZFS with NNCP. Operating System Of course, it should be no surprise that I set this up on Debian. As an added step, I did all the configuration in Ansible stored in a local git repo. This adds a lot of work, but it means that it is trivial to periodically wipe and reinstall if any security issue is suspected. The git repo can be copied off to another system for storage and takes the system from freshly-installed to ready-to-use state. Security There is, of course, nothing preventing you from running NNCP as root. The zfs commands, obviously, need to be run as root. However, from a privilege separation standpoint, I have chosen to run everything relating to NNCP as a nncp user. NNCP already does encryption, but if you prefer to have zero knowledge of the data even to NNCP, it s trivial to add gpg to the pipeline as well, and in fact I ll be demonstrating that in a future post for other reasons. Software Besides NNCP, there needs to be a system that generates the zfs send streams. For this project, I looked at quite a few. Most were designed to inspect the list of snapshots on a remote end, compare it to a list on the local end, and calculate a difference from there. This, of course, won t work for this situation. I realized my own simplesnap project was very close to being able to do this. It already used an algorithm of using specially-named snapshots on the machine being backed up, so never needed any communication about what snapshots were present where. All it needed was a few more options to permit sending to a stream instead of zfs receive. I made those changes and they are available in simplesnap 2.0.0 or above. That version has also been uploaded to sid, and will work fine as-is on buster as well. Preparing NNCP I m going to assume three hosts in this setup: The basic NNCP workflow documentation covers the basic steps. You ll need to run nncp-cfgnew on each machine. This generates a basic configuration, along with public and private keys for that machine. You ll copy the public key sets to the configurations of the other machines as usual. On the laptop, you ll add a via line like this:
  id: ....
  exchpub: ...
  signpub: ...
  noisepub: ...
  via: ["spooler"]
This tells NNCP that data destined for backupsvr should always be sent via spooler first. You can then arrange for the nncp-daemon to run on the spooler, and nncp-caller or nncp-call on the backupsvr. Or, alternatively, airgapped between the two with nncp-xfer. Generating Backup Data Now, on the laptop, install simplesnap (2.0.0 or above). Although you won t be backing up to the local system, simplesnap still maintains a hostlock in ZFS. Prepate a dataset for it:
zfs create tank/simplesnap
zfs set org.complete.simplesnap:exclude=on tank/simplesnap
Then, create a script /usr/local/bin/runsimplesnap like this:
set -e
simplesnap --store tank/simplesnap --setname backups --local --host  hostname  \
   --receivecmd /usr/local/bin/simplesnap-queue \
su nncp -c '/usr/local/nncp/bin/nncp-toss -noprogress -quiet'
if ip addr   grep -q; then
  su nncp -c '/usr/local/nncp/bin/nncp-call -noprogress -quiet -onlinedeadline 1 spooler'
The call to simplesnap sets it up to send the data to simplesnap-queue, which we ll create in a moment. The receivmd, plus noreap, sets it up to run without ZFS on the local system. The call to nncp-toss will process any previously-received inbound NNCP packets, if there are any. Then, in this example, we do a very basic check to see if we re on the LAN (checking, and if so, will establish a connection to the spooler to transmit the data. If course, you could also do this over the Internet, with tor, or whatever, but in my case, I don t want to automatically do this in case I m tethered to mobile. I figure if I want to send backups in that case, I can fire up nncp-call myself. You can also use nncp-caller to set up automated connections on other schedules; there are a lot of options. Now, here s what /usr/local/bin/simplesnap-queue looks like:
set -e
set -o pipefail
DEST=" echo $1   sed 's,^tank/simplesnap/,,' "
echo "Processing $DEST" >&2
# stdin piped to this
su nncp -c "/usr/local/nncp/bin/nncp-exec -nice B -noprogress backupsvr zfsreceive '$DEST'" >&2
echo "Queued for $DEST" >&2
This is a pretty simple script. simplesnap will call it with a path based on the store, with the hostname after; so, for instance, tank/simplesnap/laptop/root or some such. This script strips off the leading tank/simplesnap (which is a local fragment), leaving the host and dataset paths. Then it just pipes it to nncp-exec. -nice B classifies it as low-priority bulk data (so if you have some more important interactive data, it would be sent first), then passes it to whatever the backupsvr defines as zfsreceive. Receiving ZFS backups In the NNCP configuration on the recipient s side, in the laptop section, we define what command it s allowed to run as zfsreceive:
        zfsreceive: ["/usr/bin/sudo", "-H", "/usr/local/bin/nncp-zfs-receive"]
We authorize the nncp user to run this under sudo in /etc/sudoers.d/local nncp:
Defaults env_keep += "NNCP_SENDER"
nncp ALL=(root) NOPASSWD: /usr/local/bin/nncp-zfs-receive
The NNCP_SENDER is the public key ID of the sending node when nncp-toss processes the incoming data. We can use that for sanity checking later. Now, here s a basic nncp-zfs-receive script:
set -e
set -o pipefail
# now process stdin
runcommand zfs receive -o readonly=on -x mountpoint "$STORE/$DEST"
And there you have it all the basics are in place. Update 2020-12-30: An earlier version of this article had zfs receive -F instead of zfs receive -o readonly=on -x mountpoint . These changed arguments are more robust.
Update 2021-01-04: I am now recommending zfs receive -u -o readonly=on ; see my successor article for more. Enhancements You could enhance the nncp-zfs-receive script to improve logging and error handling. For instance:
set -e
set -o pipefail
# $1 will be the host/dataset
HOST=" echo "$1"   sed 's,/.*,,g' "
if [ -z "$HOST" ]; then
   echo "Malformed command line"
   exit 5
# Log a message
logit ()  
   logger -p info -t " basename "$0" [$$]" "$1"
# Log an error message
logerror ()  
   logger -p err -t " basename "$0" [$$]" "$1"
# Log stdin with the given code.  Used normally to log stderr.
logstdin ()  
   logger -p info -t " basename "$0" [$$/$1]"
# Run command, logging stderr and exit code
runcommand ()  
   logit "Running $*"
   if "$@" 2> >(logstdin "$1") ; then
      logit "$1 exited successfully"
      return 0
       logerror "$1 exited with error $RETVAL"
       return "$RETVAL"
exiterror ()  
   logerror "$1"
   echo "$1" 1>&2
   exit 10
# Sanity check
if [ "$HOST" = "laptop" ]; then
  if [ "$NNCP_SENDER" != "12345678" ]; then
    exiterror "Host $HOST doesn't match sender $NNCP_SENDER"
  exiterror "Unknown host $HOST"
runcommand zfs receive -F "$STORE/$DEST"
Now you ll capture the ZFS receive output in syslog in a friendly way, so you can look back later why things failed if they did. Further notes on NNCP nncp-toss will examine the exit code from an invocation. If it is nonzero, it will keep the command (and associated stdin) in the queue and retry it on the next invocation. NNCP does not guarantee order of execution, so it is possible in some cases that ZFS streams may be received in the wrong order. That is fine here; zfs receive will exit with an error, and nncp-toss will just run it again after the dependent snapshots have been received. For non-ZFS backups, a simple sequence number can handle this issue.

16 December 2020

John Goerzen: How To Join the Fediverse and Cast Off the Attention Economy

In a recent post, I wrote about how the attention economy in use at big social networks hurts you. In this post, I m going to suggest what to do about it. Mastodon and the Fediverse When you use email, you can send a message from an account at Google to one at Yahoo, Microsoft, or any of millions of businesses and organizations running their own mail server. Unlike, say, Facebook, email isn t a single service, but rather a whole bunch of independent systems that can communicate (or federate) with each other. The Fediverse is similar, and the most advanced Fediverse client is Mastodon. Mastodon: It s easy to get started! Head over to and click Get Started . Pick a community don t worry, this isn t a hugely consequential decision, as you can always move or change later. You can browse activity from across the Fediverse, or just on your local community, so if you find a community with similar interests, it can be a neat way to find others to follow. If you re looking for more details, has a nice guide. Defeating the Attention Economy So, why does Mastodon make a difference? First of all, you get to pick your host (and even software). With Twitter, you pretty much are using Twitter (yes, I know of things like Hootsuite, but for the vast majority of people, it s only). With Mastodon, you have choice. Pick the host that runs the software and has the kind of moderation you like. Secondly, Mastodon is not for profit. There is no money to be made in keeping you on the site. Almost all Mastodon instances are ad-free. And Mastodon s completely open protocols make it easy to go elsewhere if you like. It s Not Just Mastodon! There are plenty of other programs in the Fediverse. And, this is really key, they all interact with each other. You can share photos in Pixelfed (sort of like a federated Instagram) and see them and comment in Mastodon! Some things to point out: And there are many others. This blog, for instance, runs WordPress and uses an ActivityPub connector; comments from the Fediverse integrate here. Find me in the Fediverse You can look me up: just type in in the search box of any Mastodon instance and click Follow. You can also follow this blog at

14 August 2020

John Goerzen: In Which COVID-19 Misinformation Leads To A Bunch of Graphs Made With Rust

A funny and by funny, I mean sad thing has happened. Recently the Kansas Department of Health and Environment (KDHE) has been analyzing data from the patchwork implementation of mask requirements in Kansas. They came to a conclusion that shouldn t be surprising to anyone: masks help. They published a chart showing this. A right-wing propaganda publication got ahold of this, and claimed the numbers were doctored because there were two-different Y-axes. I set about to analyze the data myself from public sources, and produced graphs of various kinds using a single Y-axis and supporting the idea that the graphs were not, in fact, doctored. Here s one graph that s showing that:
In order to do that, I had imported COVID-19 data from various public sources. Many states in the US are large enough to have significant variation in COVID-19 conditions, and many of the source people look at don t show county-level data over time. I wanted to do that. Eventually, I wrote covid19db, which ingests data from a number of public sources and generates a SQLite database file. Using Github Actions, this file is automatically updated every morning and available for download. Or, you can download the code and generate a database yourself locally. Then, I wrote covid19ks, which generates various pretty graphs covering the data. These graphs, incidentally, turn out to highlight just how poorly the United States is doing compared to the rest of the industrialized world. I hope that these resources, and especially covid19db, might be useful to others that would like to analyze the data. The code isn t the prettiest since it was done in a hurry, but I think that functionally this is useful.

6 November 2017

John Goerzen: The Yellow House Phone Company (Featuring Asterisk and an 11-year-old)

Well Jacob, do you think we should set up our own pretend phone company in the house? We can DO THAT? Yes! Then yes. Yes! YES YES YESYESYESYES YES! Let s do it, dad! Not long ago, my parents had dug up the old phone I used back in the day. We still have a landline, and Jacob was having fun discovering how an analog phone works. I told him about the special number he could call to get the time and temperature read out to him. He discovered what happens if you call your own number and hang up. He figured out how to play Mary Had a Little Lamb using touchtone keys (after a slightly concerned lecture from me setting out some rules to make sure his musical dialing wouldn t result in any, well, dialing.) He was hooked. So I thought that taking it to the next level would be a good thing for a rainy day. I have run Asterisk before, though I had unfortunately gotten rid of most of my equipment some time back. But I found a great deal on a Cisco 186 ATA (Analog Telephone Adapter). It has two FXS lines (FXS ports simulate the phone company, and provide dialtone and ring voltage to a connected phone), and of course hooks up to the LAN. We plugged that in, and Jacob was amazed to see its web interface come up. I had to figure out how to configure it (unfortunately, it uses SCCP rather than SIP, and figuring out Asterisk s chan_skinny took some doing, but we got there.) I set up voicemail. He loved it. He promptly figured out how to record his own greetings. We set up a second phone on the other line, so he could call between them. The cordless phones in our house support SIP, so I configured one of them as a third line. He spent a long time leaving himself messages. IMG_3465 Pretty soon we both started having ideas. I set up extension 777, where he could call for the time. Then he wanted a way to get the weather forecast. Well, weather-util generates a text-based report. With it, a little sed and grep tweaking, the espeak TTS engine, and a little help from sox, I had a shell script worked up that would read back a forecast whenever he called a certain extension. He was super excited! That s great, dad! Can it also read weather alerts too? Sure! weather-util has a nice option just for that. Both boys cackled as the system tried to read out the NWS header (their timestamps like 201711031258 started with two hundred one billion ) Then I found an online source for streaming NOAA Weather Radio feeds Jacob enjoys listening to weather radio and I set up another extension he could call to listen to that. More delight! But it really took off when I asked him, Would you like to record your own menu? You mean those things where it says press 1 or 2 for this or that? Yes. WE CAN DO THAT? Oh yes! YES, LET S DO IT RIGHT NOW! So he recorded a menu, then came and hovered by me while I hacked up extensions.conf, then eagerly went back to the phone to try it. Oh the excitement of hearing hisown voice, and finding that it worked! Pretty soon he was designing sub-menus ( OK Dad, so we ll set it up so people can press 2 for the weather, and then choose if they want weather radio or the weather report. I m recording that now. Got it? ) He has informed me that next Saturday we will build an intercom system like we have at school. I m going to have to have some ideas on how to tie Squeezebox in with Asterisk to make that happen, I think. Maybe this will do.

22 August 2017

John Goerzen: The Eclipse

Highway US-81 in northern Kansas and southern Nebraska is normally a pleasant, sleepy sort of drive. It was upgraded to a 4-lane road not too long ago, but as far as 4-lane roads go, its traffic is typically light. For drives from Kansas to South Dakota, it makes a pleasant route. Yesterday was eclipse day. I strongly suspect that highway 81 had more traffic that day than it ever has before, or ever will again. For nearly the entire 3-hour drive to Geneva, NE, it was packed though mostly still moving at a good speed. And for our entire drive back, highway 81 and every other southbound road we used was so full it felt like rush hour in Dallas. (Well, not quite. Traffic was still moving.) I believe scenes like this were played out across the continent. I ve been taking a lot of photos, and writing about our new baby Martha lately. Now it s time to write a bit about some more adventures with Jacob and Oliver they re now in third and fifth grades in school. We had been planning to fly, and airports I called were either full, or were planning to park planes in the grass, or even shut down some runways to use for parking. The airport in the little town of Beatrice, NE (which I had visited twice before) was even going to have a temporary FAA control tower. At the last minute, due to some storm activity near home at departure time, we unloaded the plane and drove instead. The atmosphere at the fairgrounds in Geneva was festive. One family had brought bubbles for their kids and extras to share. IMG_20170821_113229 I had bought the boys a book about the eclipse, which they were reading before and during the event. They were both great, safe users of their eclipse glasses. IMG_20170821_124809 Jacob caught a toad, and played with it for awhile. He wanted to bring it home with us, but I convinced him to let me take a picture of him with his toad friend instead. IMG_20170821_124553 While we were waiting for totality, a number of buses from the local school district arrived. So by the time the big moment arrived, we could hear the distant roar of delight and applause from the school children gathered at the far end of the field, plus all the excitement nearby. Both boys were absolutely ecstatic to be witnessing it (and so was I!) Wow! Awesome! And simple cackles of delight were heard. On the drive home, they both kept talking about how amazing it was, and it was once in a lifetime. We enjoyed our eclipse neighbors the woman from San Antonio next to us, the surprise discovery of another family from just a few miles from us parked two cars down, even running into relatives at a restaurant on the way home. The applause from all around when it started and when it ended. And the feeling, which is hard to describe, of awe and amazement at the wonders of our world and our universe. There are many problems with the world right now, but somehow there s something right about people coming together from all over to enjoy it.

10 August 2017

John Goerzen: A new baby and deep smiles

IMG_2059 A month ago, we were waiting for our new baby; time seemed to stand still. Now she is here! Martha Goerzen was born recently, and she is doing well and growing! Laura and I have enjoyed moments of cuddling her, watching her stare at our faces, hearing her (hopefully) soft sounds as she falls asleep in our arms. It is also heart-warming to see Martha s older brothers take such an interest in her. Here is the first time Jacob got to hold her: IMG_1846 Oliver, who is a boy very much into sports, play involving police and firefighters, and such, has started adding aww and she s so cute! to his common vocabulary. He can be very insistent about interrupting me to hold her, too.

9 June 2017

John Goerzen: Fixing the Problems with Docker Images

I recently wrote about the challenges in securing Docker container contents, and in particular with keeping up-to-date with security patches from all over the Internet. Today I want to fix that. Besides security, there is a second problem: the common way of running things in Docker pretends to provide a traditional POSIX API and environment, but really doesn t. This is a big deal. Before diving into that, I want to explain something: I have often heard it said the Docker provides single-process containers. This is unambiguously false in almost every case. Any time you have a shell script inside Docker that calls cp or even ls, you are running a second process. Web servers from Apache to whatever else use processes or threads of various types to service multiple connections at once. Many Docker containers are single-application, but a process is a core part of the POSIX API, and very little software would work if it was limited to a single process. So this is my little plea for more precise language. OK, soapbox mode off. Now then, in a traditional Linux environment, besides your application, there are other key components of the system. These are usually missing in Docker containers. So today, I will fix this also. In my docker-debian-base images, I have prepared a system that still has only 11MB RAM overhead, makes minimal changes on top of Debian, and yet provides a very complete environment and API. Here s what you get: The above goes into my minimal image. Additional images add layers on top of it, and here are some of the features they add: All of the above, including the optional features, has an 11MB overhead on start. Not bad for so much, right? From here, you can layer on top all your usual Dockery things. You can still run one application per container. But you can now make sure your disk doesn t fill up from logs, run your database vacuuming commands at will, have your blog download its RSS feeds every few minutes, etc all from within the container, as it should be. Furthermore, you don t have to reinvent the wheel, because Debian already ships with things to take care of a lot of this out of the box and now those tools will just work. There is some popular work done in this area already by phusion s baseimage-docker. However, I made my own for these reasons: Finally a word on the choice to use sysvinit. It would have been simpler to use systemd here, since it is the default in Debian these days. Unfortunately, systemd requires you to poke some holes in the Docker security model, as well as mount a cgroups filesystem from the host. I didn t consider this acceptable, and sysvinit ran without these workarounds, so I went with it. With all this, Docker becomes a viable replacement for KVM for various services on my internal networks. I ll be writing about that later.

6 June 2017

John Goerzen: Family Spring: A Story in Photos

This has been a spring with times to relax, times to be busy, times of anticipation of a new baby, and times of enjoying our family. Rather than write a lot of words about it, I m telling the story in photos. To view, click here, then click Show Info in the upper right to see captions. You can pause it with the button in the lower left, and use arrow keys to advance. Alternatively, there s a captionless slideshow available here. Here s one photo to get you started: Happy about the little sister on the way

5 June 2017

John Goerzen: Flying with my brothers

Picture one Sunday morning. Three guys are seemingly-randomly walking into a Mennonite church in rural Nebraska. One with long hair and well-maintained clothes from the 70s. Another dressed well enough to be preaching. And the third simply dressed to be comfortable, with short hair showing evidence of having worn a headset for a couple of hours that morning. This was the scene as we made a spur-of-the-moment visit to that church which resulted in quite some surprise all around, since my brother knew a number of people there. For instance:
Pastor: Peter! What are you doing here? Peter: [jokingly] Is that how you greet visitors here?
And then, of course, Peter would say, Well, we were flying home from South Dakota and figured we d stop in at Beatrice for fuel. And drop in on you. Followed by some surprise that we would stop at their little airport (which is quite a nice one). This all happened because it was windy. This is the fun adventure of aviation. Sometimes you plan to go to Texas, but the weather there is terrible, so you discover a 100-year-old landmark in Indiana instead. Or sometimes, like a couple of weeks ago, we planned to fly straight home but spent a few hours exploring rural Nebraska. The three of us flew to Sioux Falls, SD, in a little Cessna to visit my uncle and aunt up there. On our flight up, we stopped at the little airport in Seward, NE. It was complete with this unique elevated deck. In my imagination, this is used for people to drink beer while watching the planes land. IMG_20170512_113323 In South Dakota, we had a weekend full of card and board games, horseshoes, and Crokinole with my uncle and aunt, who are always fun to visit. We had many memories of visits up there as children and the pleasant enjoyment of the fact that we didn t need an 8-hour drive to get there. We flew back with a huge bag of large rhubarb from their garden (that too is something of a tradition!) It was a fun weekend to spend with my brothers first time we d been able to do this in a long while. And it marked the 11th state I ve flown into, and over 17,000 miles of flying.

22 December 2016

John Goerzen: Singing with Kids

For four years now, we ve had a tradition: I go up to the attic one night, make a lot of noise, and pretend to be Santa. The boys don t think Santa is real, but they get a huge kick out of this anyway. The other day, this wound up with me singing a duet with my 7-year-old Oliver, and seeing a hugely delighted 10-year-old Jacob. All last week, the boys had been lobbying for me to be Santa . They aren t going to be able to be here on Christmas day this year, so I thought why not let them have some fun. I chose one present to give them early too. So, Saturday night, I said they could get ready for Santa. They found some cookies somewhere, got out some milk. And Oliver wrote this wonderful note to Santa : IMG_20161217_204244_cropped That is a note I m going to keep for a long time. He helpfully drew arrows pointing to the milk, cookies, and even the pen. He even started Santa s reply at the bottom! So, Saturday night, I snuck up to the attic, pretended to be Santa, and ate some cookies, drank some milk, and wrote Oliver a note. And I left a present. Jacob has been really getting into music lately, and Laura suggested I find something for the boys. I went looking for something that could record also, and came up with what has got to be a kid s dream: a karaoke machine. The particular one I found came with two microphones, a CD player, audio recording onto SD card (though it s a little dodgy), and a screen for showing words on any music that s karaoke-enhanced. Cue gasps of awe and excitement from the boys when we came down in our PJs and sweats at 6:45 Sunday morning to check it out. IMG_8895 Jacob excitedly began exploring all the knobs and options on it (they were particularly fond of the echo feature), while Oliver wanted to sing. So we found one of his favorite Christmas songs, and here he is singing it with me. IMG_8908 When you have a system with a line in, line out, and several microphone jacks, you can get creative. With a few bits of adapters from my attic, the headset I use for amateur radio worked with it perfectly. Add on a little mic extension cord, and pretty soon Oliver was pretending to be an announcer for a football game! IMG_8919 Then, Oliver decided he would act out a football game while Jacob and I were the announcers. Something tells me there will be much fun had with this over the next while! Just wait until I show them how to hook up a handheld radio to it in order to make a remotely-activated loudspeaker

9 December 2016

John Goerzen: Giant Concrete Arrows, Old Maps, and Fascinated Kids

Let me set a scene for you. Two children, ages 7 and 10, are jostling for position. There s a little pushing and shoving to get the best view. This is pretty typical for siblings this age. But what, you may wonder, are they trying to see? A TV? Video game? No. Jacob and Oliver were in a library, trying to see a 98-year-old map of the property owners in Township 23, range 1 East, Harvey County, Kansas. And they were super excited about it, somewhat to the astonishment of the research librarian, who I am sure is more used to children jostling for position over the DVDs in the youth section than poring over maps in the non-circulating historical archives! All this started with giant concrete arrows in the middle of nowhere. Nearly a century ago, the US government installed a series of arrows on the ground in Kansas. These were part of a primitive air navigation system that led to the first transcontinental airmail service. Every so often, people stumble upon these abandoned arrows and there is a big discussion online. Even Snopes has had to verify their authenticity (verdict: true). Entire websites exist to tracking and locating the remnants of these arrows. And as one of the early air mail routes went through Kansas, every so often people find these arrows around here. I got the idea that it would be fun to replicate a journey along the old routes. Maybe I d spot a few old arrows and such. So I started collecting old maps: a Contract Airmail Route #34 (CAM 34) map from 1927, aviation sectionals from 1933 and 1946, etc. I noticed an odd thing on these maps: the Newton, KS airport was on the other side of the city from its present location, sometimes even several miles outside the city. What was going on? 1927 Airway Map
(1927 Airway Map) 1946 Wichita Sectional
(1946 Wichita sectional) So one foggy morning, I explained my puzzlement to the boys. I highlighted all the mysteries: were these maps correct? Were there really two Newton airports at one time? How many airports were there, and where were they? Why did they move? What was the story behind them? And I offered them the chance to be history detectives with me. And oh my goodness, were they ever excited! We had some information from a very helpful person at the Harvey County Historical Museum (thanks Kris!) So we suspected one airport at least was established in 1927. We also had a description of its location, though given in terms of township maps. So the boys and I made the short drive over to the museum. We reviewed their property maps, though they were all a little older than the time period we needed. We looked through books and at pictures. Oliver pored over a railroad map of Newton from a century ago, fascinated. Jacob was excited to discover on one map that there used to be a train track down the middle of Main Street! I was interested that the present Newton Airport was once known as Wirt Field, rather to my surprise. I somehow suspect most 2nd and 4th graders spend a lot less excited time on their research floor! Then on to the Newton Public Library to see if they d have anything more and that s when the map that produced all the excitement came out. It, by itself, didn t answer the question, but by piecing together a number of pieces of information newspaper stories, information from the museum, and the maps we were able to come up with a pretty good explanation, much to their excitement. Apparently, a man named Tangeman owned a golf course (the golf links according to the paper), and around 1927 the city of Newton purchased it, because of all the planes that were landing there. They turned it into a real airport. Later, they bought land east of the city and moved the airport there. However, during World War II, the Navy took over that location, so they built a third airport a few miles west of the city but moved back to the current east location after the Navy returned that field to them. Of course, a project like this just opens up all sorts of extra questions: why isn t it called Wirt Field anymore? What s the story of Frank Wirt? What led the Navy to take over Newton s airport? Why did planes start landing on the golf course? Where precisely was the west airport located? How long was it there? (I found an aerial photo from 1956 that looks like it may have a plane in that general area, but it seems later than I d have expected) So now I have the boys interested in going to the courthouse with me to research the property records out there. Jacob is continually astounded that we are discovering things that aren t in Wikipedia, and also excited that he could be the one to add them. To be continued, apparently!

12 November 2016

John Goerzen: Morning in the Skies

IMG_8515 This is morning. Time to fly. Two boys, happy to open the hangar door and get the plane ready. It s been a year since I passed the FAA exam and became a pilot. Memories like these are my favorite reminders why I did. It is such fun to see people s faces light up with the joy of flying a few thousand feet above ground, of the beauty and freedom and peace of the skies. I ve flown 14 different passengers in that time; almost every flight I ve taken has been with people, which I enjoy. I ve heard wow or beautiful so many times, and said it myself even more times. IMG_6083 I ve landed in two state parks, visited any number of wonderful small towns, seen historic sites and placid lakes, ascended magically over forests and plains. I ve landed at 31 airports in 10 states, flying over 13,000 miles. airports Not once have I encountered anyone other than friendly, kind, and outgoing. And why not? After all, we re working around magic flying carpet machines, right? IMG_7867_bw (That s my brother before a flight with me, by the way) Some weeks it is easy to be glum. This week has been that way for many, myself included. But then, whether you are in the air or on the ground, if you pay attention, you realize we still live in a beautiful world with many wonderful people. And, in fact, I got a reminder of that this week. Not long after the election, I got in a plane, pushed in the throttle, and started the takeoff roll down a runway in the midst of an Indiana forest. The skies were the best kind of clear blue, and pretty soon I lifted off and could see for miles. Off in the distance, I could see the last cottony remnants of the morning s fog, lying still in the valleys, surrounding the little farms and houses as if to give them a loving hug. Wow. Sometimes the flight is bumpy. Sometimes the weather doesn t cooperate, and it doesn t happen at all. Sometimes you can fly across four large states and it feels as smooth as glass the whole way. Whatever happens, at the end of the day, the magic flying carpet machine gets locked up again. We go home, rest our heads on our soft pillows, and if we so choose, remember the beauty we experienced that day. Really, this post is not about being a pilot. This post is a reminder to pay attention to all that is beautiful in this world. It surrounds us; the smell of pine trees in the forest, the delight in the faces of children, the gentle breeze in our hair, the kind word from a stranger, the very sunrise. I hope that more of us will pay attention to the moments of clear skies and wind at our back. Even at those moments when we pull the hangar door shut. IMG_20160716_093627

13 September 2016

John Goerzen: Two Boys, An Airplane, Plus Hundreds of Old Computers

Was there anything you didn t like about our trip? Jacob s answer: That we had to leave so soon! That s always a good sign. When I first heard about the Vintage Computer Festival Midwest, I almost immediately got the notion that I wanted to go. Besides the TRS-80 CoCo II up in my attic, I also have fond memories of an old IBM PC with CGA monitor, a 25MHz 486, an Alpha also in my attic, and a lot of other computers along the way. I didn t really think my boys would be interested. But I mentioned it to them, and they just lit up. They remembered the Youtube videos I d shown them of old line printers and punch card readers, and thought it would be great fun. I thought it could be a great educational experience for them too and it was. It also turned into a trip that combined being a proud dad with so many of my other interests. Quite a fun time. IMG_20160911_061456 (Jacob modeling his new t-shirt) Captain Jacob Chicago being not all that close to Kansas, I planned to fly us there. If you re flying yourself, solid flight planning is always important. I had already planned out my flight using electronic tools, but I always carry paper maps with me in the cockpit for backup. I got them out and the boys and I planned out the flight the old-fashioned way. Here s Oliver using a scale ruler (with markings for miles corresponding to the scale of the map) and Jacob doing calculating for us. We measured the entire route and came to within one mile of the computer s calculation for each segment those boys are precise! 20160904_175519 We figured out how much fuel we d use, where we d make fuel stops, etc. The day of our flight, we made it as far as Davenport, Iowa when a chance of bad weather en route to Chicago convinced me to land there and drive the rest of the way. The boys saw that as part of the exciting adventure! Jacob is always interested in maps, and had kept wanting to use my map whenever we flew. So I dug an old Android tablet out of the attic, put Avare on it (which has aviation maps), and let him use that. He was always checking it while flying, sometimes saying this over his headset: DING. Attention all passengers, this is Captain Jacob speaking. We are now 45 miles from St. Joseph. Our altitude is 6514 feet. Our speed is 115 knots. We will be on the ground shortly. Thank you. DING Here he is at the Davenport airport, still busy looking at his maps: IMG_20160909_183813 Every little airport we stopped at featured adults smiling at the boys. People enjoyed watching a dad and his kids flying somewhere together. Oliver kept busy too. He loves to help me on my pre-flight inspections. He will report every little thing to me a scratch, a fleck of paint missing on a wheel cover, etc. He takes it seriously. Both boys love to help get the plane ready or put it away. The Computers Jacob quickly gravitated towards a few interesting things. He sat for about half an hour watching this old Commodore plotter do its thing (click for video): VID_20160910_142044 His other favorite thing was the phones. Several people had brought complete analog PBXs with them. They used them to demonstrate various old phone-related hardware; one had several BBSs running with actual modems, another had old answering machines and home-security devices. Jacob learned a lot about phones, including how to operate a rotary-dial phone, which he d never used before! IMG_20160910_151431 Oliver was drawn more to the old computers. He was fascinated by the IBM PC XT, which I explained was just about like a model I used to get to use sometimes. They learned about floppy disks and how computers store information. IMG_20160910_195145 He hadn t used joysticks much, and found Pong ( this is a soccer game! ) interesting. Somebody has also replaced the guts of a TRS-80 with a Raspberry Pi running a SNES emulator. This had thoroughly confused me for a little while, and excited Oliver. Jacob enjoyed an old TRS-80, which, through a modern Ethernet interface and a little computation help in AWS, provided an interface to Wikipedia. Jacob figured out the text-mode interface quickly. Here he is reading up on trains. IMG_20160910_140524 I had no idea that Commodore made a lot of adding machines and calculators before they got into the home computer business. There was a vast table with that older Commodore hardware, too much to get on a single photo. But some of the adding machines had their covers off, so the boys got to see all the little gears and wheels and learn how an adding machine can do its printing. IMG_20160910_145911 And then we get to my favorite: the big iron. Here is a VAX a working VAX. When you have a computer that huge, it s easier for the kids to understand just what something is. IMG_20160910_125451 When we encountered the table from the Glenside Color Computer Club, featuring the good old CoCo IIs like what I used as a kid (and have up in my attic), I pointed out to the boys that we have a computer just like this that can do these things and they responded wow! I think they are eager to try out floppy disks and disk BASIC now. Some of my favorites were the old Unix systems, which are a direct ancestor to what I ve been working with for decades now. Here s AT&T System V release 3 running on its original hardware: IMG_20160910_144923 And there were a couple of Sun workstations there, making me nostalgic for my college days. If memory serves, this one is actually running on m68k in the pre-Sparc days: IMG_20160910_153418 Returning home After all the excitement of the weekend, both boys zonked out for awhile on the flight back home. Here s Jacob, sleeping with his maps still up. IMG_20160911_132952 As we were nearly home, we hit a pocket of turbulence, the kind that feels as if the plane is dropping a bit (it s perfectly normal and safe; you ve probably felt that on commercial flights too). I was a bit concerned about Oliver; he is known to get motion sick in cars (and even planes sometimes). But what did I hear from Oliver? Whee! That was fun! It felt like a roller coaster! Do it again, dad!

9 August 2016

John Goerzen: Easily Improving Linux Security with Two-Factor Authentication

2-Factor Authentication (2FA) is a simple way to help improve the security of your systems. It restricts the scope of damage if a machine is compromised. If, for instance, you have a security token or authenticator app on your phone that is required for ssh to a remote machine, then even if every laptop you use to connect to the remote is totally owned, an attacker cannot establish a new ssh session on their own. There are a lot of tutorials out there on the Internet that get you about halfway there, so here is some more detail. Background In this article, I will be focusing on authentication in the style of Google Authenticator, which is a special case of OATH HOTP or TOTP. You can use the Google Authenticator app, FreeOTP, or a hardware token like Yubikey to generate tokens with this. They are all 100% compatible with Google Authenticator and libpam-google-authenticator. The basic idea is that there is a pre-shared secret key. At each login, a different and unique token is required, which is generated based on the pre-shared secret key and some other information. With TOTP, the other information is the current time, implying that both machines must be reasably well in-sync time-wise. With HOTP, the other information is a count of the number of times the pre-shared key has been used. Both typically have a window on the server side that can let times within a certain number of seconds, or a certain number of login accesses, work. The beauty of this system is that after the initial setup, no Internet access is required on either end to validate the key (though TOTP requires both ends to be reasonably in sync time-wise). The basics: user account setup and ssh authentication You can start with the basics by reading one of these articles: one, two, three. Debian/Ubuntu users will find both the pam module and the user account setup binary in libpam-google-authenticator. For many, you can stop there. You re done. But if you want to kick it up a notch, read on: Enhancement 1: Requiring 2FA even when ssh public key auth is used Let s consider a scenario in which your system is completely compromised. Unless your ssh keys are also stored in something like a Yubikey Neo, they could wind up being compromised as well if someone can read your files and sniff your keyboard, your ssh private keys are at risk. So we can configure ssh and PAM so that a OTP token is required even for this scenario. First off, in /etc/ssh/sshd_config, we want to change or add these lines:
UsePAM yes
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
This forces all authentication to pass two verification methods in ssh: publickey and keyboard-interactive. All users will have to supply a public key and then also pass keyboard-interactive auth. Normally keyboard-interactive auth prompts for a password, but we can change /etc/pam.d/sshd on this. I added this line at the very top of /etc/pam.d/sshd:
auth [success=done new_authtok_reqd=done ignore=ignore default=bad]
This basically makes Google Authenticator both necessary and sufficient for keyboard-interactive in ssh. That is, whenever the system wants to use keyboard-interactive, rather than prompt for a password, it instead prompts for a token. Note that any user that has not set up google-authenticator already will be completely unable to ssh into their account. Enhancement 1, variant 2: Allowing automated processes to root On many of my systems, I have ~root/.ssh/authorized_keys set up to permit certain systems to run locked-down commands for things like backups. These are automated commands, and the above configuration will break them because I m not going to be typing in codes at 3AM. If you are very restrictive about what you put in root s authorized_keys, you can exempt the root user from the 2FA requirement in ssh by adding this to sshd_config:
Match User root
  AuthenticationMethods publickey
This says that the only way to access the root account via ssh is to use the authorized_keys file, and no 2FA will be required in this scenario. Enhancement 1, variant 2: Allowing non-pubkey auth On some multiuser systems, some users may still want to use password auth rather than publickey auth. There are a few ways we can support that:
  1. Users without public keys will have to supply a OTP and a password, while users with public keys will have to supply public key, OTP, and a password
  2. Users without public keys will have to supply OTP or a password, while users with public keys will have to supply public key, OTP, or a password
  3. Users without public keys will have to supply OTP and a password, while users with public keys only need to supply the public key
The third option is covered in any number of third-party tutorials. To enable options 1 or 2, you ll need to put this in sshd_config:
AuthenticationMethods publickey,keyboard-interactive keyboard-interactive
This means that to authenticate, you need to pass either publickey and then keyboard-interactive auth, or just keyboard-interactive auth. Then in /etc/pam.d/sshd, you put this:
auth required
As a sub-variant for option 1, you can add nullok to here to permit auth from people that do not have a Google Authenticator configuration. Or for option 2, change required to sufficient . You should not add nullok in combination with sufficient, because that could let people without a Google Authenticator config authenticate completely without a password at all. Enhancement 2: Configuring su A lot of other tutorials stop with ssh (and maybe gdm) but forget about the other ways we authenticate or change users on a system. su and sudo are the two most important ones. If your root password is compromised, you don t want anybody to be able to su to that account without having to supply a token. So you can set up google-authenticator for root. Then, edit /etc/pam.d/su and insert this line after the line:
auth       required nullok
The reason you put this after is because you want to be able to su from root to any account without having to input a token. We add nullok to the end of this, because you may want to su to accounts that don t have tokens. Just make sure to configure tokens for the root account first. Enhancement 3: Configuring sudo This one is similar to su, but a little different. This lets you, say, secure the root password for sudo. Normally, you might sudo from your user account to root (if so configured). You might have sudo configured to require you to enter in your own password (rather than root s), or to just permit you to do whatever you want as root without a password. Our first step, as always, is to configure PAM. What we do here depends on your desired behavior: do you want to require someone to supply both a password and a token, or just a token, or require a token? If you want to require a token, put this at the top of /etc/pam.d/sudo:
auth [success=done new_authtok_reqd=done ignore=ignore default=bad]
If you want to require a token and a password, change the bracketed string to required , and if you want a token or a password, change it to sufficient . As before, if you want to permit people without a configured token to proceed, add nullok , but do not use that with sufficient or the bracketed example here. Now here comes the fun part. By default, if a user is required to supply a password to sudo, they are required to supply their own password. That does not help us here, because a user logged in to the system can read the ~/.google_authenticator file and easily then supply tokens for themselves. What you want to do is require them to supply root s password. Here s how I set that up in sudoers:
Defaults:jgoerzen rootpw
jgoerzen ALL=(ALL) ALL
So now, with the combination of this and the PAM configuration above, I can sudo to the root user without knowing its password but only if I can supply root s token. Pretty slick, eh? Further reading In addition to the basic tutorials referenced above, consider: Edit: additional comments Here are a few other things to try: First, the libpam-google-authenticator module supports putting the Google Authenticator files in different locations and having them owned by a certain user. You could use this to, for instance, lock down all secret keys to be readable only by the root user. This would prevent users from adding, changing, or removing their own auth tokens, but would also let you do things such as reusing your personal token for the root account without a problem. Also, the pam-oath module does much of the same things as the libpam-google-authenticator module, but without some of the help for setup. It uses a single monolithic root-owned password file for all accounts. There is an oathtool that can be used to generate authentication codes from the command line.

3 August 2016

John Goerzen: All Aboard

Aaaaaall Aboard! *chug* *chug* And so began a trip aboard our hotel train in Indianapolis, conducted by our very own Jacob and Oliver. IMG_20160703_101438 Because, well, what could be more fun than spending a few days in the world s only real Pullman sleeping car, on its original service track, inside a hotel? IMG_20160703_101520 We were on a family vacation to Indianapolis, staying in what two railfan boys were sure to enjoy: a hotel actually built into part of the historic Indianapolis Union Station complex. This is the original train track and trainshed. They moved in the Pullman cars, then built the hotel around them. Jacob and Oliver played for hours, acting as conductors and engineers, sending their train all across the country to pick up and drop off passengers. Opa! Have you ever seen a kid s face when you introduce them to something totally new, and they think it is really exciting, but a little scary too? That was Jacob and Oliver when I introduced them to saganaki (flaming cheese) at a Greek restaurant. The conversation went a little like this: Our waitress will bring out some cheese. And she will set it ON FIRE right by our table! Will it burn the ceiling? No, she ll be careful. Will it be a HUGE fire? About a medium-sized fire. Then what will happen? She ll yell OPA! and we ll eat the cheese after the fire goes out. Does it taste good? Oh yes. My favorite! It turned out several tables had ordered saganaki that evening, so whenever I saw it coming out, I d direct their attention to it. Jacob decided that everyone should call it opa instead of saganaki because that s what the waitstaff always said. Pretty soon whenever they d see something appear in the window from the kitchen, there d be craning necks and excited jabbering of maybe that s our opa! And when it finally WAS our opa , there were laughs of delight and I suspect they thought that was the best cheese ever. Giggling Elevators IMG_20160703_205544 Fun times were had pressing noses against the glass around the elevator. Laura and I sat on a nearby sofa while Jacob and Oliver sat by the elevators, anxiously waiting for someone to need to go up and down. They point and wave at elevators coming down, and when elevator passengers waved back, Oliver would burst out giggling and run over to Laura and me with excitement. Some history IMG_20160704_161550 We got to see the grand hall of Indianapolis Union Station what a treat to be able to set foot in this magnificent, historic space, the world s oldest union station. We even got to see the office where Thomas Edison worked, and as a hotel employee explained, was fired for doing too many experiments on the job. Water and walkways Indy has a system of elevated walkways spanning quite a section of downtown. It can be rather complex navigating them, and after our first day there, I offered to let Jacob and Oliver be the leaders. Boy did they take pride in that! They stopped to carefully study maps and signs, and proudly announced this way or turn here and were usually correct. 20160702_164754_Richtone(HDR) And it was the same in the paddleboat we took down the canal. Both boys wanted to be in charge of steering, and we only scared a few other paddleboaters. Fireworks IMG_20160704_220332 Our visit ended with the grand fireworks show downtown, set off from atop a skyscraper. I had been scouting for places to watch from, and figured that a bridge-walkway would be great. A couple other families had that thought too, and we all watched the 20-minute show in the drizzle. Loving brothers By far my favorite photo from the week is this one, of Jacob and Oliver asleep, snuggled up next to each other under the covers. They sure are loving and caring brothers, and had a great time playing together. IMG_20160702_071015

28 June 2016

John Goerzen: A great day for a flight with the boys

I tend to save up my vacation time to use in summer for family activities, and today was one of those days. Yesterday, Jacob and Oliver enjoyed planning what they were going to do with me. They ruled out all sorts of things nearby, but they decided they would like to fly to Ponca City, explore the oil museum there, then eat at Enrique s before flying home. Of course, it is not particularly hard to convince me to fly somewhere. So off we went today for some great father-son time. The weather on the way was just gorgeous. We cruised along at about a mile above ground, which gave us pleasantly cool air through the vents and a smooth ride. Out in the distance, a few clouds were trying to form. IMG_20160627_141614 Whether I m flying or driving, a pilot is always happy to pass a small airport. Here was the Winfield, KS airport (KWLD): IMG_20160627_142106 This is a beautiful time of year in Kansas. The freshly-cut wheat fields are still a vibrant yellow. Other crops make a bright green, and colors just pop from the sky. A camera can t do it justice. They enjoyed the museum, and then Oliver wanted to find something else to do before we returned to the airport for dinner. A little exploring yielded the beautiful and shady Garfield Park, complete with numerous old stone bridges. IMG_20160627_162121 Of course, the hit of any visit to Enrique s is their ice cream tacos (sopapillas with ice cream). Here is Oliver polishing off his. IMG_20160627_174345 They had both requested sightseeing from the sky on our way back, but both fell asleep so we opted to pass on that this time. Oliver slept through the landing, and I had to wake him up when it was time to go. I always take it as a compliment when a 6-year-old sleeps through a landing! IMG_20160627_191524 Most small airports have a bowl of candy setting out somewhere. Jacob and Oliver have become adept at finding them, and I will usually let them talk me into a piece of candy at one of them. Today, after we got back, they were intent at exploring the small gift shop back home, and each bought a little toy helicopter for $1.25. They may have been too tired to enjoy it though. They ve been in bed for awhile now, and I m still smiling about the day. Time goes fast when you re having fun, and all three of us were. It is fun to see them inheriting my sense of excitement at adventure, and enjoying the world around them as they go. The lady at the museum asked how we had heard about them, and noticed I drove up in an airport car (most small airports have an old car you can borrow for a couple hours for free if you re a pilot). I told the story briefly, and she said, So you flew out to this small town just to spend some time here? Yep. Wow, that s really neat. I don t think we ve ever had a visitor like you before. Then she turned to the boys and said, You boys are some of the luckiest kids in the world. And I can t help but feel like the luckiest dad in the world.

16 June 2016

John Goerzen: Mud, Airplanes, Arduino, and Fun

The last few weeks have been pretty hectic in their way, but I ve also had the chance to take some time off work to spend with family, which has been nice. Memorial Day: breakfast and mud For Memorial Day, I decided it would be nice to have a cookout for breakfast rather than for dinner. So we all went out to the fire ring. Jacob and Oliver helped gather kindling for the fire, while Laura chopped up some vegetables. Once we got a good fire going, I cooked some scrambled eggs in a cast iron skillet, mixed with meat and veggies. Mmm, that was tasty. Then we all just lingered outside. Jacob and Oliver enjoyed playing with the cats, and the swingset, and then . water. They put the hose over the slide and made a water slide (more mud slide maybe). IMG_7688 Then we got out the water balloon fillers they had gotten recently, and they loved filling up water balloons. All in all, we all just enjoyed the outdoors for hours. MVI_7738 Flying to Petit Jean, Arkansas Somehow, neither Laura nor I have ever really been to Arkansas. We figured it was about time. I had heard wonderful things about Petit Jean State Park from other pilots: it s rather unique in that it has a small airport right in the park, a feature left over from when Winthrop Rockefeller owned much of the mountain. And what a beautiful place it was! Dense forests with wonderful hiking trails, dotted with small streams, bubbling springs, and waterfalls all over; a nice lake, and a beautiful lodge to boot. Here was our view down into the valley at breakfast in the lodge one morning: IMG_7475 And here s a view of one of the trails: IMG_7576 The sunset views were pretty nice, too: IMG_7610 And finally, the plane we flew out in, parked all by itself on the ramp: IMG_20160522_171823 It was truly a relaxing, peaceful, re-invigorating place. Flying to Atchison Last weekend, Laura and I decided to fly to Atchison, KS. Atchison is one of the oldest cities in Kansas, and has quite a bit of history to show off. It was fun landing at the Amelia Earhart Memorial Airport in a little Cessna, and then going to three museums and finding lunch too. Of course, there is the Amelia Earhart Birthplace Museum, which is a beautifully-maintained old house along the banks of the Missouri River. IMG_20160611_134313 I was amused to find this hanging in the county historical society museum: IMG_20160611_153826 One fascinating find is a Regina Music Box, popular in the late 1800s and early 1900s. It operates under the same principles as those that you might see that are cylindrical. But I am particular impressed with the effort that would go into developing these discs in the pre-computer era, as of course the holes at the outer edge of the disc move faster than the inner ones. It would certainly take a lot of careful calculation to produce one of these. I found this one in the Cray House Museum: VID_20160611_151504 An Arduino Project with Jacob One day, Jacob and I got going with an Arduino project. He wanted flashing blue lights for his police station , so we disassembled our previous Arduino project, put a few things on the breadboard, I wrote some code, and there we go. Then he noticed an LCD in my Arduino kit. I hadn t ever gotten around to using it yet, and of course he wanted it immediately. So I looked up how to connect it, found an API reference, and dusted off my C skills (that was fun!) to program a scrolling message on it. Here is Jacob showing it off: VID_20160614_074802.mp4

6 June 2016

John Goerzen: How git-annex replaces Dropbox + encfs with untrusted providers

git-annex has been around for a long time, but I just recently stumbled across some of the work Joey has been doing to it. This post isn t about it s traditional roots in git or all the features it has for partial copies of large data sets, but rather for its live syncing capabilities like Dropbox. It takes a bit to wrap your head around, because git-annex is just a little different from everything else. It s sort of like a different-colored smell. The git-annex wiki has a lot of great information both low-level reference and a high-level 10-minute screencast showing how easy it is to set up. I found I had to sort of piece together the architecture between those levels, so I m writing this all down hoping it will benefit others that are curious. Ir you just want to use it, you don t need to know all this. But I like to understand how my tools work. Overview git-annex lets you set up a live syncing solution that requires no central provider at all, or can be used with a completely untrusted central provider. Depending on your usage pattern, this central provider could require only a few MBs of space even for repositories containing gigabytes or terabytes of data that is kept in sync. Let s take a look at the high-level architecture of the tool. Then I ll illustrate how it works with some scenarios. Three Layers Fundamentally, git-annex takes layers that are all combined in Dropbox and separates them out. There is the storage layer, which stores the literal data bytes that you are interested in. git-annex indexes the data in storage by a hash. There is metadata, which is for things like a filename-to-hash mapping and revision history. And then there is an optional layer, which is live signaling used to drive the real-time syncing. git-annex has several modes of operation, and the one that enables live syncing is called the git-annex assistant. It runs as a daemon, and is available for Linux/POSIX platforms, Windows, Mac, and Android. I ll be covering it here. The storage layer The storage layer simply is blobs of data. These blobs are indexed by a hash, and can be optionally encrypted at rest at remote backends. git-annex has a large number of storage backends; some examples include rsync, a remote machine with git-annex on it that has ssh installed, WebDAV, S3, Amazon Glacier, removable USB drive, etc. There s a huge list. One of the git-annex features is that each client knows the state of each storage repository, as well as the capability set of each storage repository. So let s say you have a workstation at home and a laptop you take with you to work or the coffee shop. You d like changes on one to be instantly recognized on another. With something like Dropbox or OwnCloud, every file in the set you want synchronized has to reside on a server in the cloud. With git-annex, it can be configured such that the server in the cloud only contains a copy of a file until every client has synced it up, at which point it gets removed. Think about it that is often what you want anyhow, so why maintain an unnecessary copy after it s synced everywhere? (This behavior is, of course, configurable.) git-annex can also avoid storing in the cloud entirely if the machines are able to reach each other directly at least some of the time. The metadata layer Metadata about your files includes a mapping from the file names to the storage location (based on hashes), change history, and information about the status of each machine that participates in the syncing. On your clients, git-annex stores this using git. This detail is very useful to some, and irrelevant to others. Some of the git-annex storage backends can support only storage (S3, for instance). Some can support both storage and metadata (rsync, ssh, local drives, etc.) You can even configure a backend to support only metadata (more on why that may be useful in a bit). When you are working with a git-backed repository for git-annex, it can hold data, metadata, or both. So, to have a working sync system, you must have a way to transport both the data and the metadata. The transport for the metadata is generally rsync or git, but it can also be XMPP in which Git changesets are basically wrapped up in XMPP presence messages. Joey says, however, that there are some known issues with XMPP servers sometimes dropping or reordering some XMPP messages, so he doesn t encourage that method currently. The live signaling layer So once you have your data and metadata, you can already do syncs via git annex sync --contents. But the real killer feature here will be automatic detection of changes, both on the local and the remote. To do that, you need some way of live signaling. git-annex supports two methods. The first requires ssh access to a remote machine where git-annex is installed. In this mode of operation, when the git-annex assistant fires up, it opens up a persistent ssh connection to the remote and runs the git-annex-shell over there, which notifies it of changes to the git metadata repository. When a change is detected, a sync is initiated. This is considered ideal. A substitute can be XMPP, and git-annex actually converts git commits into a form that can be sent over XMPP. As I mentioned above, there are some known reliability issues with this and it is not the recommended option. Encryption When it comes to encryption, you generally are concerned about all three layers. In an ideal scenario, the encryption and decryption happens entirely on the client side, so no service provider ever has any details about your data. The live signaling layer is encrypted pretty trivially; the ssh sessions are, of course, encrypted and TLS support in XMPP is pervasive these days. However, this is not end-to-end encryption; those messages are decrypted by the service provider, so a service provider could theoretically spy on metadata, which may include change times and filenames, though not the contents of files themselves. The data layer also can be encrypted very trivially. In the case of the dumb backends like S3, git-annex can use symmetric encryption or a gpg keypair and all that ever shows up on the server are arbitrarily-named buckets. You can also use a gcrypt-based git repository. This can cover both data and metadata and, if the target also has git-annex installed, the live signalling layer. Using a gcrypt-based git repository for the metadata and live signalling is the only way to accomplish live syncing with 100% client-side encryption. All of these methods are implemented in terms of gpg, and can support symmetric of public-key encryption. It should be noted here that the current release versions of git-annex need a one-character patch in order to fix live syncing with a remote using gcrypt. For those of you running jessie, I recommend the version in jessie-backports, which is presently 5.20151208. For your convenience, I have compiled an amd64 binary that can drop in over /usr/bin/git-annex if you have this version. You can download it and a gpg signature for it. Note that you only need this binary on the clients; the server can use the version from jessie-backports without issue. Putting the pieces together: some scenarios Now that I ve explained the layers, let s look at how they fit together. Scenario 1: Central server In this scenario, you might have a workstation and a laptop that sync up with each other by way of a central server that also has a full copy of the data. This is the scenario that most closely resembles Dropbox, box, or OwnCloud. Here you would basically follow the steps in the git-assistant screencast: install git-annex on a server somewhere, and point your clients to it. If you want full end-to-end encryption, I would recommend letting git-annex generate a gpg keypair for you, which you would then need to copy to both your laptop and workstation (but not the server). Every change you make locally will be synced to the server, and then from the server to your other PC. All three systems would be configured in the client transfer group. Scenario 1a: Central server without a full copy of the data In this scenario, everything is configured the same except the central server is configured with the transfer transfer group. This means that the actual data synced to it is deleted after it has been propagated to all clients. Since git-annex can verify which repository has received a copy of which data, it can easily enough delete the actual file content from the central server after it has been copied to all the clients. Many people use something like Dropbox or OwnCloud as a multi-PC syncing solution anyhow, so once the files have been synced everywhere, it makes sense to remove them from the central server. This is often a good ideal for people. There are some obvious downsides that are sometimes relevant. For instance, to add a third sync client, it must be able to initially copy down from one of the existing clients. Or, if you intend to access the data from a device such as a cell phone where you don t intend for it to have a copy of all data all the time, you won t have as convenient way to download your data. Scenario 1b: Split data/metadata central servers Imagine that you have a shell or rsync account on some remote system where you can run git-annex, but don t have much storage space. Maybe you have a cheap VPS or shell account somewhere, but it s just not big enough to hold your data. The answer to this would be to use this shell or rsync account for the metadata, but put the data elsewhere. You could, for instance, store the data in Amazon S3 or Amazon Glacier. These backends aren t capable of storing the git-annex metadata, so all you need is a shell or rsync account somewhere to sync up the metadata. (Or, as below, you might even combine a fully distributed approach with this.) Then you can have your encrypted data pushed up to S3 or some such service, which presumably will grow to whatever size you need. Scenario 2: Fully distributed Like git itself, git-annex does not actually need a central server at all. If your different clients can reach each other directly at least some of the time, that is good enough. Of course, a given client will not be able to do fully automatic live sync unless it can reach at least one other client, so changes may not propagate as quickly. You can simply set this up by making ssh connections available between your clients. git-annex assistant can automatically generate appropriate ~/.ssh/authorized_keys entries for you. Scenario 2a: Fully distributed with multiple disconnected branches You can even have a graph of connections available. For instance, you might have a couple machines at home and a couple machines at work with no ability to have a direct connection between them (due to, say, firewalls). The two machines at home could sync with each other in real-time, as could the two machines at work. git-annex also supports things like USB drives as a transport mechanism, so you could throw a USB drive in your pocket each morning, pop it in to one client at work, and poof both clients are synced up over there. Repeat when you get home in the evening, and you re synced there. The USB drive s repository can, of course, be of the transport type so data is automatically deleted from it once it s been synced everywhere. Scenario 3: Hybrid git-annex can support LAN sync even if you have a central server. If your laptop, say, travels around but is sometimes on the same LAN as your PC, git-annex can easily sync directly between the two when they are reachable, saving a round-trip to the server. You can assign a cost to each remote, and git-annex will always try to sync first to the lowest-cost path that is available. Drawbacks of git-annex There are some scenarios where git-annex with the assistant won t be as useful as one of the more traditional instant-sync systems. The first and most obvious one is if you want to access the files without the git-annex client. For instance, many of the other tools let you generate a URL that you can email to people, and then they can download files without any special client software. This is not directly possible with git-annex. You could, of course, make something like a public_html directory be managed with git-annex, but it wouldn t provide things like obfuscated URLs, password-protected sharing, time-limited sharing, etc. that you get with other systems. While you can share your repositories with others that have git-annex, you can t share individual subdirectories; for a given repository, it is all or nothing. The Android client for git-annex is a pretty interesting thing: it is mostly a small POSIX environment, providing a terminal, git, gpg, and the same web interface that you get on a standalone machine. This means that the git-annex Android client is fully functional compared to a desktop one. It also has a quick setup process for syncing off your photos/videos. On the other hand, the integration with the Android ecosystem is poor compared to most other tools. Other git-annex features git-annex has a lot to offer besides the git-annex assistant. Besides the things I ve already mentioned, any given git-annex repository including your client repository can have a partial copy of the full content. Say, for instance, that you set up a git-annex repository for your music collection, which is quite large. You want some music on your netbook, but don t have room for it all. You can tell git-annex to get or drop files from the netbook s repository without deleting them remotely. git-annex has quite a few ways to automate and configure this, including making sure that at least a certain number of copies of a file exist in your git-annex ecosystem. Conclusion I initially started looking at git-annex due to the security issues with encfs, and the difficulty with setting up ecryptfs in this way. (I had been layering encfs atop OwnCloud). git-annex certainly ticks the box for me security-wise, and obviously anything encrypted with encfs wasn t going to be shared with others anyhow. I ll be using git-annex more in the future, I m sure.