I ve been hacking on a project on and off for my LAN called DNSync. This will take a DNSMasq leases file and sync it to Amazon Route 53.
I ve added a new feature, which will create A reccords for each MAC address on the LAN.
Since DNSync won t touch CNAME records, I use CNAME records (manually) to point to the auto-synced A records for services on my LAN (such as my Projector, etc).
Since It s easy for two machines to have the same name, I ve decided to add A records for each MAC as well as their client name. They take the fomm of something like ab-cd-ef-ab-cd-ef.by-mac.paultag.house., which is harder to accedentally collide.
While setting up my new network at my house, I figured I d do things right and set up an IPSec VPN (and a few other fancy bits). One thing that became annoying when I wasn t on my LAN was I d have to fiddle with the DNS Resolver to resolve names of machines on the LAN.
Since I hate fiddling with options when I need things to just work, the easiest way out was to make the DNS names actually resolve on the public internet.
A day or two later, some Golang glue, and AWS Route 53, and I wrote code that would sit on my dnsmasq.leases, watch inotify for IN_MODIFY signals, and sync the records to AWS Route 53.
I pushed it up to my GitHub as DNSync.
In the spirit of blogging about some of the code i ve written in the past year or two, I wrote a small utility library called go-haversine, which uses the Haversine Forumla to compute the distance between two points.
This is super helpful when working with GPS data - but remember, this assumes everything s squarely on the face of the planet.
A few weeks ago, I hacked up go-wmata, some golang bindings to the WMATA API. This is super handy if you are in the DC area, and want to interface to the WMATA data.
As a proof of concept, I wrote a yo bot called @WMATA, where it returns the closest station if you Yo it your location. For hilarity, feel free to Yo it from outside DC.
For added fun, and puns, I wrote a dbus proxy for the API as weel, at wmata-dbus, so you can query the next train over dbus. One thought was to make a GNOME Shell extension to tell me when the next train is. I d love help with this (or pointers on how to learn how to do this right).
A while back, I found myself in need of some TLS certificates set up and issued for a testing environment.
I remembered there was some code for issuing TLS certs in Docker, so I yanked some of that code and made a sensable CLI API over it.
Thus was born minica!
Something as simple as minica email@example.com domain.tld will issue two TLS certs (one with a Client EKU, and one server) issued from a single CA.
Next time you re in need of a few TLS keys (without having to worry about stuff like revocation or anything), this might be the quickest way out!
PKCS#11 is a standard API to interface with HSMs, Smart Cards, or other types
of random hardware backed crypto. On my travel laptop, I use a few Yubikeys in
PKCS#11 mode using OpenSC to handle system login. libpam-pkcs11 is a pretty
easy to use module that will let you log into your system locally using a
PKCS#11 token locally.
One of the least documented things, though, was how to use an OpenSC PKCS#11
token in Chrome. First, close all web browsers you have open.
About a year ago, I bought a Projector after I finally admitted that I could
actually use a TV in my apartment. I settled on buying a
It was a really great value, and has been nothing short of a delight to own.
I was always a bit curious about the DB9 connector on the back of the unit,
so I dug into the user manual, and found some hex code strings in there. So,
last year, between my last gig at the
Sunlight Foundtion and
USDS, I spent some time wandering around the US,
hitting up DebConf, and exploring Washington
DC. Between trips, I set out to figure out exactly what was going on with my
Projector, and see if I could make it do anything fun.
So, I started off with basics, and tried to work out how these command codes
were structured. I had a few working codes, but to write clean code, I'd be
better off understanding why the codes looked like they do. Let's look at
the "Power On" code.
0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D
Some were 10 bytes, other were 11, and most started with similar looking
things. The first byte was usually a 0x06 or 0x07, followed by two
bytes 0x14 0x00, and either a 0x04 or 0x05. Since the first few bytes
were similarly structured, I assumed the first octet (either 0x06 or 0x07)
was actually a length, since the first 4 octets seemed always present.
So, my best guess is that we have a Length byte at index 0, followed by
two bytes for the Protocol, a flag for if you're Reading or Writing (best
guess on that one), and opaque data following that. Sometimes it's a const
of sorts, and sometimes an octet (either little or big endian, confusingly).
Right. OK. So, let's get to work. In the spirit of code is data, data is code,
I hacked up some of the projector codes into a s-expression we can use later.
The structure of this is boring, but it'll let us both store the command
code to issue, as well as define the handler to read the data back.
(setv*commands*; function type family control'((power-onnilnil(0x060x140x000x040x000x340x110x000x000x5D))(power-offnilnil(0x060x140x000x040x000x340x110x010x000x5E))(power-statusconstpower(0x070x140x000x050x000x340x000x000x110x000x5E))(resetnilnil(0x060x140x000x040x000x340x110x020x000x5F))...
As well as defining some of the const responses that come back from the
Projector itself. These are pretty boring, but it's helpful to put a
name to the int that falls out.
After defining a few simple functions to write the byte arrays to the serial
port as well as reading and understanding responses from the projector, I could
start elaborating on some higher order functions that can talk projector. So
the first thing I wrote was to make a function that converts the command
entry into a native Hy function.
Cool. So, now we can import things like power-on from *commands* which
takes a single argument (serial) for the serial port, and it'll send a
command, and return the response. The best part about all this is you only
have to define the data once in a list, and the rest comes for free.
Finally, I do want to be able to turn my projector on and off over the network
so I went ahead and make a Flask "API" on top of this. First, let's define
a macro to define Flask routes:
I ll be at HOPE 11 this year - if anyone else will be around, feel free to send me an email! I won t have a phone on me (so texting only works if you use Signal!)
Looking forward for a chance to see everyone soon!
Around a year ago, I started hacking together a machine readable version
of the OSI approved licenses list, and casually picking parts up until it
was ready to launch. A few weeks ago, we officially announced
the osi license api, which is now
live at api.opensource.org.
I also took a whack at writing a few API bindings, in
and using the models from the API implementation itself in
Go. In the following
few weeks, Clint wrote one in Haskell,
Eriol wrote one in Rust,
and Oliver wrote one in R.
The data is sourced from a repo on GitHub,
the licenses repo under OpenSourceOrg. Pull Requests against that repo are
wildly encouraged! Additional data ideas, cleanup or more hand collected data
would be wonderful!
In the meantime, use-cases for using this API range from language package
managers pulling OSI approval of a licence programatically to using a license
identifier as defined in one dataset (SPDX, for exampele), and using that
to find the identifer as it exists in another system (DEP5, Wikipedia,
Patches are hugly welcome, as are bug reports or ideas! I'd also love more
API wrappers for other languages!
A while back, I found myself in need of two webservers that would terminate TLS (with different rules). I wanted to run some custom code I d written (which uses TLS peer authentication), and also nginx on port 443.
The best way I figured out how to do this was to write a tool to sit on port 443, and parse TLS Client Hello packets, and dispatch to the correct backend depending on the SNI name.
SNI, or Server Name Indication allows the client to announce (yes over cleartext!) what server it s looking for, similar to the HTTP Host header. Sometimes, like in the case above, the Host header won t work, since you ve already done a TLS handshake by the time you figure out who they re looking for.
I also spun the Client Hello parser out into its own importable package, just in case someone else finds themselves in this same boat.
The code s up on github.com/paultag/sniff!
Last week, I posted about python-sense,
and API wrapper for the internal Sense API. I wrote this so that I could
pull data about myself into my own databases, allowing me to use that
information for myself.
One way I'm doing this is by pulling my room data into an
InfluxDB database, letting me run time series
queries against my environmental data.
A while back, I saw a Kickstarter
for one of the most well designed and pretty sleep trackers on the market. I
fell in love with it, and it has stuck with me since.
A few months ago, I finally got my hands on one and started to track my data.
Naturally, I now want to store this new data with the rest of the data I have
on myself in my own databases.
I went in search of an API, but I found that the Sense API hasn't been published
yet, and is being worked on by the team. Here's hoping it'll land soon!
After some subdomain guessing, I hit on api.hello.is.
So, naturally, I went to take a quick look at their Android app and network
traffic, lo and behold, there was a pretty nicely designed API.
This API is clearly an internal API, and as such, it's something that
should not be considered stable. However, I'm OK with a fragile API,
so I've published a quick and dirty API wrapper for the Sense API
to my GitHub..
I've published it because I've found it useful, but I can't promise the world,
(since I'm not a member of the Sense team at Hello!), so here are a few ground
rules of this wrapper:
I make no claims to the stability or completeness.
I have no documentation or assurances.
I will not provide the client secret and ID. You'll have to find them on
This may stop working without any notice, and there may even be really nasty
bugs that result in your alarm going off at 4 AM.
Send PRs! This is a side-project for me.
This module is currently Python 3 only. If someone really needs Python 2
support, I'm open to minimally invasive patches to the codebase using
six to support Python 2.7.
Working with the API:
First, let's go ahead and log in using python -m sense.
$ python -m sense
Sense OAuth Client ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense OAuth Client Secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense email: firstname.lastname@example.org
Attempting to log into Sense's APISuccess!Attempting to query the Sense APIThe humidity is **just right**.The air quality is **just right**.The light level is **just right**.It's **pretty hot** in here.
The noise level is **just right**.
Now, let's see if we can pull up information on my Sense:
>>>fromsenseimportSense>>>sense=Sense()>>>sense.devices()'senses':[ 'id':'xxxxxxxxxxxxxxxx','firmware_version':'11a1','last_updated':1466991060000,'state':'NORMAL','wifi_info':'rssi':0,'ssid':'Pretty Fly for a WiFi (2.4 GhZ)','condition':'GOOD','last_updated':1462927722000 ,'color':'BLACK' ],'pills':[ 'id':'xxxxxxxxxxxxxxxx','firmware_version':'2','last_updated':1466990339000,'battery_level':87,'color':'BLUE','state':'NORMAL' ]
Neat! Pretty cool. Look, you can even see my WiFi AP! Let's try some more
and pull some trends out.
I plan to keep maintaining it as long as it's needed, so I welcome
co-maintainers, and I'd love to see what people build with it! So far, I'm
using it to dump my room data into InfluxDB, pulling information on my room
into Grafana. Hopefully more to come!
As some of the world knows full well by now, I've been noodling with Go
for a few years, working through its pros, its cons, and thinking a lot
about how humans use code to express thoughts and ideas. Go's got a lot of
neat use cases, suited to particular problems, and used in the right place,
you can see some clear massive wins.
I've started writing Debian tooling in Go, because it's a pretty natural fit.
Go's fairly tight, and overhead shouldn't be taken up by your operating system.
After a while, I wound up hitting the usual blockers, and started to build up
abstractions. They became pretty darn useful, so, this blog post is announcing
(a still incomplete, year old and perhaps API changing) Debian package for Go.
The Go importable name is pault.ag/go/debian. This contains a lot of utilities
for dealing with Debian packages, and will become an edited down "toolbelt"
for working with or on Debian packages.
Currently, the package contains 4 major sub packages. They're a changelog
parser, a control file parser, deb file format parser, dependency parser
and a version parser. Together, these are a set of powerful building blocks
which can be used together to create higher order systems with reliable
understandings of the world.
The first (and perhaps most incomplete and least tested) is a changelog file
parser.. This provides the
programmer with the ability to pull out the suite being targeted in the
changelog, when each upload was, and the version for each. For example, let's
look at how we can pull when all the uploads of Docker to sid took place:
funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/changelogs/main/d/docker.io/unstable_changelog")iferr!=nilpanic(err)allEntries,err:=changelog.Parse(resp.Body)iferr!=nilpanic(err)for_,entry:=rangeallEntriesfmt.Printf("Version %s was uploaded on %s\n",entry.Version,entry.When)
The output of which looks like:
Version 1.8.3~ds1-2 was uploaded on 2015-11-04 00:09:02 -0800 -0800
Version 1.8.3~ds1-1 was uploaded on 2015-10-29 19:40:51 -0700 -0700
Version 1.8.2~ds1-2 was uploaded on 2015-10-29 07:23:10 -0700 -0700
Version 1.8.2~ds1-1 was uploaded on 2015-10-28 14:21:00 -0700 -0700
Version 1.7.1~dfsg1-1 was uploaded on 2015-08-26 10:13:48 -0700 -0700
Version 1.6.2~dfsg1-2 was uploaded on 2015-07-01 07:45:19 -0600 -0600
Version 1.6.2~dfsg1-1 was uploaded on 2015-05-21 00:47:43 -0600 -0600
Version 1.6.1+dfsg1-2 was uploaded on 2015-05-10 13:02:54 -0400 EDT
Version 1.6.1+dfsg1-1 was uploaded on 2015-05-08 17:57:10 -0600 -0600
Version 1.6.0+dfsg1-1 was uploaded on 2015-05-05 15:10:49 -0600 -0600
Version 1.6.0+dfsg1-1~exp1 was uploaded on 2015-04-16 18:00:21 -0600 -0600
Version 1.6.0~rc7~dfsg1-1~exp1 was uploaded on 2015-04-15 19:35:46 -0600 -0600
Version 1.6.0~rc4~dfsg1-1 was uploaded on 2015-04-06 17:11:33 -0600 -0600
Version 1.5.0~dfsg1-1 was uploaded on 2015-03-10 22:58:49 -0600 -0600
Version 1.3.3~dfsg1-2 was uploaded on 2015-01-03 00:11:47 -0700 -0700
Version 1.3.3~dfsg1-1 was uploaded on 2014-12-18 21:54:12 -0700 -0700
Version 1.3.2~dfsg1-1 was uploaded on 2014-11-24 19:14:28 -0500 EST
Version 1.3.1~dfsg1-2 was uploaded on 2014-11-07 13:11:34 -0700 -0700
Version 1.3.1~dfsg1-1 was uploaded on 2014-11-03 08:26:29 -0700 -0700
Version 1.3.0~dfsg1-1 was uploaded on 2014-10-17 00:56:07 -0600 -0600
Version 1.2.0~dfsg1-2 was uploaded on 2014-10-09 00:08:11 +0000 +0000
Version 1.2.0~dfsg1-1 was uploaded on 2014-09-13 11:43:17 -0600 -0600
Version 1.0.0~dfsg1-1 was uploaded on 2014-06-13 21:04:53 -0400 EDT
Version 0.11.1~dfsg1-1 was uploaded on 2014-05-09 17:30:45 -0400 EDT
Version 0.9.1~dfsg1-2 was uploaded on 2014-04-08 23:19:08 -0400 EDT
Version 0.9.1~dfsg1-1 was uploaded on 2014-04-03 21:38:30 -0400 EDT
Version 0.9.0+dfsg1-1 was uploaded on 2014-03-11 22:24:31 -0400 EDT
Version 0.8.1+dfsg1-1 was uploaded on 2014-02-25 20:56:31 -0500 EST
Version 0.8.0+dfsg1-2 was uploaded on 2014-02-15 17:51:58 -0500 EST
Version 0.8.0+dfsg1-1 was uploaded on 2014-02-10 20:41:10 -0500 EST
Version 0.7.6+dfsg1-1 was uploaded on 2014-01-22 22:50:47 -0500 EST
Version 0.7.1+dfsg1-1 was uploaded on 2014-01-15 20:22:34 -0500 EST
Version 0.6.7+dfsg1-3 was uploaded on 2014-01-09 20:10:20 -0500 EST
Version 0.6.7+dfsg1-2 was uploaded on 2014-01-08 19:14:02 -0500 EST
Version 0.6.7+dfsg1-1 was uploaded on 2014-01-07 21:06:10 -0500 EST
Next is one of the most complex, and one of the oldest parts of go-debian,
which is the control file parser
(otherwise sometimes known as deb822). This module was inspired by the way
that the json module works in Go, allowing for files to be defined in code
with a struct. This tends to be a bit more declarative, but also winds up
putting logic into struct tags, which can be a nasty anti-pattern if used too
The first primitive in this module is the concept of a Paragraph, a struct
containing two values, the order of keys seen, and a map of string to string.
All higher order functions dealing with control files will go through this
type, which is a helpful interchange format to be aware of. All parsing of
meaning from the Control file happens when the Paragraph is unpacked into
a struct using reflection.
The idea behind this strategy that you define your struct, and let the Control
parser handle unpacking the data from the IO into your container, letting you
maintain type safety, since you never have to read and cast, the conversion
will handle this, and return an Unmarshaling error in the event of failure.
Additionally, Structs that define an anonymous member of control.Paragraph
will have the raw Paragraph struct of the underlying file, allowing the
programmer to handle dynamic tags (such as X-Foo), or at least, letting
them survive the round-trip through go.
The default decoder
contains an argument, the ability to verify the input control file using an
OpenPGP keyring, which is exposed to the programmer through the
(*Decoder).Signer() function. If the passed argument is nil, it will not
check the input file signature (at all!), and if it has been passed, any
signed data must be found or an error will fall out of the NewDecoder call.
On the way out, the opposite happens, where the struct is introspected,
turned into a control.Paragraph, and then written out to the io.Writer.
Here's a quick (and VERY dirty) example showing the basics of reading and
writing Debian Control files with go-debian.
packagemainimport("fmt""io""net/http""strings""pault.ag/go/debian/control")typeAllowedPackagestructPackagestringFingerprintstringfunc(a*AllowedPackage)UnmarshalControl(instring)errorin=strings.TrimSpace(in)chunks:=strings.SplitN(in," ",2)iflen(chunks)!=2returnfmt.Errorf("Syntax sucks: '%s'",in)a.Package=chunksa.Fingerprint=chunks[1:len(chunks)-1]returnniltypeDMUAstructFingerprintstringUidstringAllowedPackagesAllowedPackage control:"Allow" delim:"," funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/dm.txt")iferr!=nilpanic(err)decoder,err:=control.NewDecoder(resp.Body,nil)iferr!=nilpanic(err)fordmua:=DMUAiferr:=decoder.Decode(&dmua);err!=niliferr==io.EOFbreakpanic(err)fmt.Printf("The DM %s is allowed to upload:\n",dmua.Uid)for_,allowedPackage:=rangedmua.AllowedPackagesfmt.Printf(" %s [granted by %s]\n",allowedPackage.Package,allowedPackage.Fingerprint)
Output (truncated!) looks a bit like:
The DM Allison Randal <email@example.com> is allowed to upload:
parrot [granted by A4F455C3414B10563FCC9244AFA51BD6CDE573CB]
The DM Benjamin Barenblat <firstname.lastname@example.org> is allowed to upload:
boogie [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
dafny [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
transmission-remote-gtk [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
urweb [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
The DM <email@example.com> is allowed to upload:
covered [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
dico [granted by 6ADD5093AC6D1072C9129000B1CCD97290267086]
drawtiming [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
fonts-hosny-amiri [granted by BD838A2BAAF9E3408BD9646833BE1A0A8C2ED8FF]
Next up, we've got the deb module. This contains code to handle reading
Debian 2.0 .deb files. It contains a wrapper that will parse the control
member, and provide the data member through the
Here's an example of how to read a .deb file, access some metadata, and
iterate over the tar archive, and print the filenames of each of the
The dependency package provides an interface to parse and compute
dependencies. This package is a bit odd in that, well, there's no other
library that does this. The issue is that there are actually two different
parsers that compute our Dependency lines, one in Perl (as part of dpkg-dev)
and another in C (in dpkg).
To date, this has resulted in me filing
I also found a broken package in the
which actually resulted in another bug being (totally accidentally)
I hope to continue to run the archive through my parser in hopes of finding
more bugs! This package is a bit complex, but it basically just returns what
amounts to be an AST
for our Dependency lines. I'm positive there are bugs, so file them!
funcmain()dep,err:=dependency.Parse("foo bar, baz, foobar [amd64] bazfoo [!sparc], fnord:armhf [gnu-linux-sparc]")iferr!=nilpanic(err)anySparc,err:=dependency.ParseArch("sparc")iferr!=nilpanic(err)for_,possi:=rangedep.GetPossibilities(*anySparc)fmt.Printf("%s (%s)\n",possi.Name,possi.Arch)
Gives the output:
Right off the bat, I'd like to thank
Michael Stapelberg for letting me graft this
out of dcs and into the go-debian package.
This was nearly entirely his work (with a one or two line function I added
later), and was amazingly helpful to have. Thank you!
This module implements Debian version comparisons and parsing, allowing for
sorting in lists, checking to see if it's native or not, and letting the
programmer to implement smart(er!) logic based on upstream (or Debian)
This module is extremely easy to use and very straightforward, and not worth
writing an example for.
This is more of a "Yeah, OK, this has been useful enough to me at this point
that I'm going to support this" rather than a "It's stable!" or even
"It's alive!" post. Hopefully folks can report bugs and help iterate on
this module until we have some really clean building blocks to build
solid higher level systems on top of. Being able to have multiple libraries
interoperate by relying on go-debian will be a massive ease.
I'm in need of more documentation, and to finalize some parts of the older
sub package APIs, but I'm hoping to be at a "1.0" real soon now.
As nearly anyone who's worked with me will attest to, I've long since
touted nedbat's talk
Pragmatic Unicode, or, How do I stop the pain?
as one of the most foundational talks, and required watching for all programmers.
The reason is because netbat hits on something bigger - something more
fundamental than how to handle Unicode -- it's how to handle data which is
For those who want the TL;DR, the argument is as follows:
Facts of Life:
Computers work with Bytes. Bytes go in, Bytes go out.
I've started to think more about why we do the things we do when we write
code, and one thing that continues to be a source of morbid schadenfreude
is watching code break by failing to handle Unicode right. It's hard! However,
watching what breaks lets you gain a bit of insight into how the author
thinks, and what assumptions they make.
When you send someone Unicode, there are a lot of assumptions that have to be
made. Your computer has to trust what you (yes, you!) entered into your web
browser, your web browser has to pass that on over the network (most of the
time without encoding information), to a server which reads that bytestream,
and makes a wild guess at what it should be. That server might save it to a
database, and interpolate it into an HTML template in a different encoding
(called Mojibake), resulting
in a bad time for everyone involved.
Everything's awful, and the fact our computers can continue to display
text to us is a goddamn miracle. Never forget that.
When it comes down to it, when I see a byte sitting on a page, I don't know
(and can't know!) if it's Windows-1252, UTF-8, Latin-1, or EBCDIC. What's
a poem to me is terminal garbage to you.
Over the years, hacks have evolved. We have
and plain ole' hacks to just guess based on the content. Of course, like
all good computer programs, this has lead to its fair share of hilarious
and there's nothing stopping files from (validly!) being multiple things at the
Like many things, it's all in the eye of the beholder.
Just like Unicode, this is a word that can put your friendly neighborhood
programmer into a series of profanity laden tirades. Go find one in the wild,
and ask them about what they think about timezone handling bugs they've seen.
I'll wait. Go ahead.
Rants are funny things. They're fun to watch. Hilarious to give. Sometimes
just getting it all out can help. They can tell you a lot about the true
nature of problems.
It's funny to consider the isomorphic nature of Unicode rants and Timezone
I don't think this is an accident.
U n i c o d e timezone Sandwich
Ned's Unicode Sandwich applies -- As early as we can, in the lowest level
we can (reading from the database, filesystem, wherever!), all datetimes
must be timezone qualified with their correct timezone. Always. If you mean
UTC, say it's in UTC.
Treat any unqualified datetimes as "bytes". They're not to be trusted.
Never, never, never trust 'em. Don't
process any datetimes until you're sure they're in the right timezone.
This lets the delicious inside of your datetime sandwich handle timezones
with grace, and finally, as late as you can, turn it back into bytes
(if at all!). Treat locations as tzdb entries, and qualify datetime
objects into their absolute timezone (EST, EDT, PST, PDT)
It's not until you want to show the datetime to the user again should you
consider how to re-encode your datetime to bytes. You should think about
what flavor of bytes, what encoding -- what timezone -- should I be
Just like Unicode, testing that your code works with datetimes is important.
Every time I think about how to go about doing this, I think about that
one time that mjg59 couldn't book a flight
starting Tuesday from AKL, landing in HNL on Monday night, because
United couldn't book the last leg to SFO. Do you ever assume dates only go
forward as time goes on? Remember timezones.
Construct test data, make sure someone in New Zealand's
+13:45 can correctly talk with
their friends in
Baker Island's -12:00,
and that the events sort right.
Just because it's Noon on New Years Eve in England doesn't mean it's not
1 AM the next year in New Zealand. Places a few miles apart may go on Daylight
savings different days. Indian Standard Time is not even aligned on the hour
to GMT (+05:30)!
Test early, and test often. Memorize a few timezones, and challenge
your assumptions when writing code that has to do with time. Don't use
wall clocks to mean monotonic time. Remember there's a whole world out there,
and we only deal with part of it.
It's also worth remembering, as Andrew Pendleton
pointed out to me, that it's possible that a datetime isn't even unique for a
place, since you can never know if 2016-11-06 01:00:00 in America/New_York
(in the tzdb) is the first one, or second one. Storing EST or EDT along
with your datetime may help, though!
Improper handling of timezones can lead to some interesting things, and failing
to be explicit (or at least, very rigid) in what you expect will lead to an
unholy class of bugs we've all come to hate. At best, you have confused
users doing math, at worst, someone misses a critical event, or our
security code fails.
I recently found what I regard to be a pretty bad
bug in apt (which David has prepared a
for and is pending upload, yay! Thank you!), which boiled down to documentation
and code expecting datetimes in a timezone, but accepting any timezone, and
silently treating it as UTC.
The solution is to hard-fail, which is an interesting choice to me (as a vocal
fan of timezone aware code), but at the least it won't fail by
misunderstanding what the server is trying to communicate, and I do understand
and empathize with the situation the apt maintainers are in.
Overall, my main point is although most modern developers know how to deal
with Unicode pain, I think there is a more general lesson to learn -- namely,
you should always know what data you have, and always remember what it is.
Understand assumptions as early as you can, and always store them with the data.
Back in 2014, Mako ran a Boston Iron Blogger chapter, where you had to blog once a week, or you owed $5 into the pot. A while later, I ran it (along with Molly and Johns), and things were great.
When I moved to DC, I had already talked with Tom Lee and Eric Mill about running a DC Iron Blogger chapter, but it hasn t happened in the year and a half I ve been in DC.
This week, I make good on that, with a fantastic group set up at dc.iron-blogger.com; with more to come (I m sure!).
Looking forward to many parties and though provoking blog posts in my future. I m also quite pleased I ll be resuming my blogging. Hi, again, planet Debian!
(You can Follow or Tweet about this blog on Twitter)
Over the last week, people have been saying a lot about the wonderful life of Ian Murdock and his contributions to Debian and the world of free software. According to one news site, a San Francisco police officer, Grace Gatpandan, has been doing the opposite, starting a PR spin operation, leaking snippets of information about what may have happened during Ian's final 24 hours. Sadly, these things are now starting to be regurgitated without proper scrutiny by the mainstream press (note the erroneous reference to SFGate with link to SFBay.ca, this is British tabloid media at its best).
The report talks about somebody (no suggestion that it was even Ian) "trying to break into a residence". Let's translate that from the spin-doctor-speak back to English: it is the silly season, when many people have a couple of extra drinks and do silly things like losing their keys. "a residence", or just their own home perhaps? Maybe some AirBNB guest arriving late to the irritation of annoyed neighbours? Doesn't the choice of words make the motive sound so much more sinister? Nobody knows the full story and nobody knows if this was Ian, so snippets of information like this are inappropriate, especially when somebody is deceased.
Did they really mean to leave people with the impression that one of the greatest visionaries of the Linux world was also a cat burglar? That somebody who spent his life giving selflessly and generously for the benefit of the whole world (his legacy is far greater than Steve Jobs, as Debian comes with no strings attached) spends the Christmas weekend taking things from other people's houses in the dark of the night? The report doesn't mention any evidence of a break-in or any charges for breaking-in.
If having a few drinks and losing your keys in December is such a sorry state to be in, many of us could potentially be framed in the same terms at some point in our lives. That is one of the reasons I feel so compelled to write this: somebody else could be going through exactly the same experience at the moment you are reading this. Any of us could end up facing an assault as unpleasant as the tweets imply at some point in the future. At least I can console myself that as a privileged white male, the risk to myself is much lower than for those with mental illness, the homeless, transgender, Muslim or black people but as the tweets suggest, it could be any of us.
The story reports that officers didn't actually come across Ian breaking in to anything, they encountered him at a nearby street corner. If he had weapons or drugs or he was known to police that would have almost certainly been emphasized. Is it right to rush in and deprive somebody of their liberties without first giving them an opportunity to identify themselves and possibly confirm if they had a reason to be there?
The report goes on, "he was belligerent", "he became violent", "banging his head" all by himself. How often do you see intelligent and successful people like Ian Murdock spontaneously harming themselves in that way? Can you find anything like that in any of the 4,390 Ian Murdock videos on YouTube? How much more frequently do you see reports that somebody "banged their head", all by themselves of course, during some encounter with law enforcement? Do police never make mistakes like other human beings?
If any person was genuinely trying to spontaneously inflict a head injury on himself, as the police have suggested, why wouldn't the police leave them in the hospital or other suitable care? Do they really think that when people are displaying signs of self-harm, rounding them up and taking them to jail will be in their best interests?
Now, I'm not suggesting this started out with some sort of conspiracy. Police may have been at the end of a long shift (and it is a disgrace that many US police are not paid for their overtime) or just had a rough experience with somebody far more sinister. On the other hand, there may have been a mistake, gaps in police training or an inappropriate use of a procedure that is not always justified, like a strip search, that causes profound suffering for many victims.
A select number of US police forces have been shamed around the world for a series of incidents of extreme violence in recent times, including the death of Michael Brown in Ferguson, shooting Walter Scott in the back, death of Freddie Gray in Baltimore and the attempts of Chicago's police to run an on-shore version of Guantanamo Bay. Beyond those highly violent incidents, the world has also seen the abuse of Ahmed Mohamed, the Muslim schoolboy arrested for his interest in electronics and in 2013, the suicide of Aaron Swartz which appears to be a direct consequence of the "Justice" department's obsession with him.
What have the police learned from all this bad publicity? Are they changing their methods, or just hiring more spin doctors? If that is their response, then doesn't it leave them with a cruel advantage over those people who were deceased?
Isn't it standard practice for some police to simply round up anybody who is a bit lost and write up a charge sheet for resisting arrest or assaulting an officer as insurance against questions about their own excessive use of force?
When British police executed Jean Charles de Menezes on a crowded tube train and realized they had just done something incredibly outrageous, their PR office went to great lengths to try and protect their image, even photoshopping images of Menezes to make him look more like some other suspect in a wanted poster. To this day, they continue to refer to Menezes as a victim of the terrorists, could they be any more arrogant? While nobody believes the police woke up that morning thinking "let's kill some random guy on the tube", it is clear they made a mistake and like many people (not just police), they immediately prioritized protecting their reputation over protecting the truth.
Nobody else knows exactly what Ian was doing and exactly what the police did to him. We may never know. However, any disparaging or irrelevant comments from the police should be viewed with some caution.
The horrors of incarceration
It would be hard for any of us to understand everything that an innocent person goes through when detained by the police. The recently released movie about The Stanford Prison Experiment may be an interesting place to start, a German version produced in 2001, Das Experiment, is also very highly respected.
The United States has the largest prison population in the world and the second-highest per-capita incarceration rate. Many, including some on death row, are actually innocent, in the wrong place at the wrong time, without the funds to hire an attorney. The system, and the police and prison officers who operate it, treat these people as packages on a conveyor belt, without even the most basic human dignity. Whether their encounter lasts for just a few hours or decades, is it any surprise that something dies inside them when they discover this cruel side of American society?
Worldwide, there is an increasing trend to make incarceration as degrading as possible. People may be innocent until proven guilty, but this hasn't stopped police in the UK from locking up and strip-searching over 4,500 children in a five year period, would these children go away feeling any different than if they had an encounter with Jimmy Saville or Rolf Harris? One can only wonder what they do to adults.
What all this boils down to is that people shouldn't really be incarcerated unless it is clear the danger they pose to society is greater than the danger they may face in a prison.
What can people do for Ian and for justice?
Now that these unfortunate smears have appeared, it would be great to try and fill the Internet with stories of the great things Ian has done for the world. Write whatever you feel about Ian's work and your own experience of Debian.
While the circumstances of the final tweets from his Twitter account are confusing, the tweets appear to be consistent with many other complaints about US law enforcement. Are there positive things that people can do in their community to help reduce the harm?
Sending books to prisoners (the UK tried to ban this) can make a difference. Treat them like humans, even if the system doesn't.
Recording incidents of police activities can also make a huge difference, such as the video of the shooting of Walter Scott or the UK police making a brutal unprovoked attack on a newspaper vendor. Don't just walk past a situation and assume everything is under control. People making recordings may find themselves in danger, it is recommended to use software that automatically duplicates each recording, preferably to the cloud, so that if the police ask you to delete such evidence, you can let them watch you delete it and still have a copy.
Can anybody think of awards that Ian Murdock should be nominated for, either in free software, computing or engineering in general? Some, like the prestigious Queen Elizabeth Prize for Engineering can't be awarded posthumously but others may be within reach. Come and share your ideas on the debian-project mailing list, there are already some here.
Best of all, Ian didn't just build software, he built an organization, Debian. Debian's principles have helped to unite many people from otherwise different backgrounds and carry on those principles even when Ian is no longer among us. Find out more, install it on your computer or even look for ways to participate in the project.
Combine Soylent and Simple Syrup. Create what I m going to start to call Soylent Syrup . Enjoy that one, folks.
Add ice to a rocks glass, pour Soylent Syrup over ice. Add Sherry, Vermouth and Campari. Stir. Garnish with an orange twist.
Big thanks to Matthew Garrett for sparking this one.
A friend recently commented on my rather unusual portrait on my (out of date) page on the Berkman website. Here s the story.
I joined Berkman as a fellow with a fantastic class of fellows that included, among many other incredibly accomplished people, Vivek Kundra: first Chief Information Officer of the United States. At Berkman, all the fellows are all asked for photos and Vivek apparently sent in his official government portrait.
You are probably familiar with the genre. In the US at least, official government portraits are mostly pictures of men in dark suits, light shirts, and red or blue ties with flags draped blurrily in the background.
Not unaware of the fact that Vivek sat right below me on the alphabetically sorted Berkman fellows page, a small group that included Paul Tagliamonte very familiar with the genre from his work with government photos in Open States decided to create a government portrait of me using the only flag we had on hand late one night.
The result shown in the screenshot above and in the WayBack Machine was almost entirely unnoticed (at least to my knowledge) but was hopefully appreciated by those who did see it.
Been a while since my last blog post - things have been a bit hectic lately, and I ve not really had the time.
Now that things have settled down a bit I m in DC! I ve moved down south to join the rest of my colleagues at Sunlight to head up our State & Local team.
Leaving behind the brilliant Free Software community in Boston won t be easy, but I m hoping to find a similar community here in DC.