Since I wrote my first post on this topic, Docker has switched from apt.dockerproject.org to download.docker.com, so this post revisits my original steps, but tailored for the new repo. There will be less commentary this time (straight to the beef). For further commentary on why for any step, see my previous post.

These steps should be fairly similar to what s found in upstream s Install Docker on Debian document, but do differ slightly in a few minor ways.

grab Docker s APT repo GPG key

# "Docker Release (CE deb)"
export GNUPGHOME="$(mktemp -d)"
gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
# stretch+
gpg --export --armor 9DC858229FC7DD38854AE2D88D81803C0EBFCD88   sudo tee /etc/apt/trusted.gpg.d/docker.gpg.asc
# jessie
# gpg --export 9DC858229FC7DD38854AE2D88D81803C0EBFCD88   sudo tee /etc/apt/trusted.gpg.d/docker.gpg > /dev/null
rm -rf "$GNUPGHOME"

Verify:

$ apt-key list
...
/etc/apt/trusted.gpg.d/docker.gpg.asc
-------------------------------------
pub   rsa4096 2017-02-22 [SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           [ unknown] Docker Release (CE deb) <docker@docker.com>
sub   rsa4096 2017-02-22 [S]
...

add Docker s APT source With the switch to download.docker.com, HTTPS is now mandated:

$ apt-get update && apt-get install apt-transport-https

Setup sources.list:

echo "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable"   sudo tee /etc/apt/sources.list.d/docker.list

Add edge component for every-month releases and test for release candidates (ie, ... stretch stable edge). Replace stretch with jessie for Jessie installs. At this point, you should be safe to run apt-get update to verify the changes:

$ sudo apt-get update
...
Get:5 https://download.docker.com/linux/debian stretch/stable amd64 Packages [1227 B]
...
Reading package lists... Done

(There shouldn t be any warnings or errors about missing keys, etc.)

configure Docker

This step could be done after Docker s installed (and indeed, that s usually when I do it because I forget that I should until I ve got Docker installed and realize that my configuration is suboptimal), but doing it before ensures that Docker doesn t have to be restarted later.

sudo mkdir -p /etc/docker
sudo sensible-editor /etc/docker/daemon.json

(sensible-editor can be replaced by whatever editor you prefer, but that command should choose or prompt for a reasonable default)

I then fill daemon.json with at least a default storage-driver. Whether I use aufs or overlay2 depends on my kernel version and available modules if I m on Ubuntu, AUFS is still a no-brainer (since it s included in the default kernel if the linux-image-extra-XXX/linux-image-extra-virtual package is installed), but on Debian AUFS is only available in either 3.x kernels (jessie s default non-backports kernel) or recently in the aufs-dkms package (as of this writing, still only available on stretch and sid no jessie-backports option).

If my kernel is 4.x+, I m likely going to choose overlay2 (or if that errors out, the older overlay driver).

Choosing an appropriate storage driver is a fairly complex topic, and I d recommend that for serious production deployments, more research on pros and cons is performed than I m including here (especially since AUFS and OverlayFS are not the only options they re just the two I personally use most often).

 
	"storage-driver": "overlay2"
 

configure boot parameters

I usually set a few boot parameters as well (in /etc/default/grub s GRUB_CMDLINE_LINUX_DEFAULT option run sudo update-grub after adding these, space-separated).

• cgroup_enable=memory enable memory accounting for containers (allows docker run --memory for setting hard memory limits on containers)
• swapaccount=1 enable swap accounting for containers (allows docker run --memory-swap for setting hard swap memory limits on containers)
• systemd.legacy_systemd_cgroup_controller=yes newer versions of systemd may disable the legacy cgroup interfaces Docker currently uses; this instructs systemd to keep those enabled (for more details, see systemd/systemd#4628, opencontainers/runc#1175, docker/docker#28109)
• vsyscall=emulate allow older binaries to run (debian:wheezy, etc.; see docker/docker#28705)

All together:

...
GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1 systemd.legacy_systemd_cgroup_controller=yes vsyscall=emulate"
...

install Docker! Finally, the time has come.

$ sudo apt-get install -V docker-ce
...
   docker-ce (17.03.1~ce-0~debian-stretch)
...
$ sudo docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:07:28 2017
 OS/Arch:      linux/amd64
Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:07:28 2017
 OS/Arch:      linux/amd64
 Experimental: false
$ sudo usermod -aG docker "$(id -un)"

Two days ago one of the most influential logician of the 20th century has passed away, Gaisi Takeuti ( ). I had the pleasure to meet this excellent man, teacher, writer, thinker several times while he was the president of the Kurt G del Society. I don t want to recall his achievements in mathematical logic, in particular proof theory, because I am not worth to write about such a genius. I want to recall a few personal stories from my own experience. I came into contact with Prof. Takeuti via is famous book Proof Theory, which my then Professor, now colleague and friend Matthias Baaz used for teaching us students proof theory. Together with Shoenfield s Mathematical Logic these two books became the foundation of my whole logic education. Now again in print, back then the Proof Theory was a rare precious. Few prints did remain in the library, and over the years one by one disappeared, until the last copy we had access to was my copy where I had scribbled pages and pages of notes and proofs. Matthias later on used these copies for his lectures, I should have written on the back-side! I remember well my first meeting with Prof. Takeuti: I was on the Conference on Internationalization in 2003 in Tsukuba, long before I moved to Japan. Back then I was just finishing my PhD and without much experience. When I arrived in the hotel, without fail there was a message of Prof. Takeuti inviting me for dinner the following day. We had dinner in a specialty restaurant of his area, together with is lovely wife. I was soo nervous about Japanese manners and stuttered Japanese phrases just to be stopped by Prof. Takeuti pouring himself a glass of sake and telling me: Relax, and forget the rules and fill your own glass when you want to. I am well aware that this liberal attitude didn t extend to Japanese colleagues, where he, descendant from a Samurai family, was at times very, extremely strict. The dinner was decided upon already, not easy since I was still strict vegetarian back than (now I would have enjoyed the dinner much more!), but for the last course we could decide. I remember with a smile how Prof. Takeuti suggested in Japanese various sweets, just to be interrupted by his wife with No Gaisi, no! . I asked what is going on and she explained that he wants to order a Japanese sweet for me I agreed, and that was probably the worst dish I had in Japan. Slippy noodles swimming in a cold broth, to be picked with chopsticks and put into a semi-sweet soja-sauce. I finished it, but it wasn t good. I should have thought twice when Prof. Takeuti s wife ordered a normal fruit salad. Scientifically he was simply a genius and famous for not reading a lot but reinventing everything. One of my research areas, G del logics, was reinvented by him as Intuitionistic Fuzzy Logic (for an overview see my talk at the Collegium Logicum 2016: G del Logics a short survey). But I want to recall one of my favorite articles of him: A Conservative Extension of Peano Arithmetic . This was published as part 2 of Volume 17 of Publications of the Mathematical Society of Japan, retypeset pdf is available here, JSTOR page. Therein he develops classical (real and complex) analysis over Peano s arithmetic. He shows that any arithmetical theorem proved in analytic number theory is a theorem in Peano s arithmetic. The proof uses Gentzen s cut elimination theorem, the center piece of modern proof theory. With Georg Kreisel having passed away in 2015, and now Gaisi Takeuti, we loose two of the greatest, if not the greatest minds in logic.

Cockpit has now been in Debian unstable and Ubuntu 17.04 and devel, which means it s now a simple

$ sudo apt install cockpit

away for you to try and use. This metapackage pulls in the most common plugins, which are currently NetworkManager and udisks/storaged. If you want/need, you can also install cockpit-docker (if you grab docker.io from jessie-backports or use Ubuntu) or cockpit-machines to administer VMs through libvirt. Cockpit upstream also has a rather comprehensive Kubernetes/Openstack plugin, but this isn t currently packaged for Debian/Ubuntu as kubernetes itself is not yet in Debian testing or Ubuntu. After that, point your browser to https://localhost:9090 (or the host name/IP where you installed it) and off you go.

What is Cockpit? Think of it as an equivalent of a desktop (like GNOME or KDE) for configuring, maintaining, and interacting with servers. It is a web service that lets you log into your local or a remote (through ssh) machine using normal credentials (PAM user/password or SSH keys) and then starts a normal login session just as gdm, ssh, or the classic VT logins would. The left side bar is the equivalent of a task switcher , and the applications (i. e. modules for administering various aspects of your server) are run in parallel. The main idea of Cockpit is that it should not behave special in any way - it does not have any specific configuration files or state keeping and uses the same Operating System APIs and privileges like you would on the command line (such as lvmconfig, the org.freedesktop.UDisks2 D-Bus interface, reading/writing the native config files, and using sudo when necessary). You can simultaneously change stuff in Cockpit and in a shell, and Cockpit will instantly react to changes in the OS, e. g. if you create a new LVM PV or a network device gets added. This makes it fundamentally different to projects like webmin or ebox, which basically own your computer once you use them the first time. It is an interface for your operating system, which even reflects in the branding: as you see above, this is Debian (or Ubuntu, or Fedora, or wherever you run it on), not Cockpit .

Remote machines In your home or small office you often have more than one machine to maintain. You can install cockpit-bridge and cockpit-system on those for the most basic functionality, configure SSH on them, and then add them on the Dashboard (I add a Fedora 26 machine here) and from then on can switch between them on the top left, and everything works and feels exactly the same, including using the terminal widget: The Fedora 26 machine has some more Cockpit modules installed, including a lot of playground ones, thus you see a lot more menu entries there.

Under the hood Beneath the fancy Patternfly/React/JavaScript user interface is the Cockpit API and protocol, which particularly fascinates me as a developer as that is what makes Cockpit so generic, reactive, and extensible. This API connects the worlds of the web, which speaks IPs and host names, ports, and JSON, to the local host only world of operating systems which speak D-Bus, command line programs, configuration files, and even use fancy techniques like passing file descriptors through Unix sockets. In an ideal world, all Operating System APIs would be remotable by themselves, but they aren t. This is where the cockpit bridge comes into play. It is a JSON (i. e. ASCII text) stream protocol that can control arbitrarily many channels to the target machine for reading, writing, and getting notifications. There are channel types for running programs, making D-Bus calls, reading/writing files, getting notified about file changes, and so on. Of course every channel can also act on a remote machine. One can play with this protocol directly. E. g. this opens a (local) D-Bus channel named d1 and gets a property from systemd s hostnamed:

$ cockpit-bridge --interact=---
  "command": "open", "channel": "d1", "payload": "dbus-json3", "name": "org.freedesktop.hostname1"  
---
d1
  "call": [ "/org/freedesktop/hostname1", "org.freedesktop.DBus.Properties", "Get",
          [ "org.freedesktop.hostname1", "StaticHostname" ] ],
  "id": "hostname-prop"  
---

and it will reply with something like

d1
 "reply":[[ "t":"s","v":"donald" ]],"id":"hostname-prop" 
---

( donald is my laptop s name). By adding additional parameters like host and passing credentials these can also be run remotely through logging in via ssh and running cockpit-bridge on the remote host. Stef Walter explains this in detail in a blog post about Web Access to System APIs. Of course Cockpit plugins (both internal and third-party) don t directly speak this, but use a nice JavaScript API. As a simple example how to create your own Cockpit plugin that uses this API you can look at my schroot plugin proof of concept which I hacked together at DevConf.cz in about an hour during the Cockpit workshop. Note that I never before wrote any JavaScript and I didn t put any effort into design whatsoever, but it does work .

Next steps Cockpit aims at servers and getting third-party plugins for talking to your favourite part of the system, which means we really want it to be available in Debian testing and stable, and Ubuntu LTS. Our CI runs integration tests on all of these, so each and every change that goes in is certified to work on Debian 8 (jessie) and Ubuntu 16.04 LTS, for example. But I d like to replace the external PPA/repository on the Install instructions with just it s readily available in -backports ! Unfortunately there s some procedural blockers there, the Ubuntu backport request suffers from understaffing, and the Debian stable backport is blocked on getting it in to testing first, which in turn is blocked by the freeze. I will soon ask for a freeze exception into testing, after all it s just about zero risk - it s a new leaf package in testing. Have fun playing around with it, and please report bugs! Feel free to discuss and ask questions on the Google+ post.

Recently, Daniel visited us in Helsinki. In addition to enjoying local food and scenerey, we spent some time together in front of a whiteboard to sketch out designs for Ick2. Ick is my continuous integration system, and it's all Daniel's fault for suggesting the name. Ahem. I am currently using the first generation of Ick and it is a rigid, cumbersome, and fragile thing. It works well enough that I don't miss Jenkins, but I would like something better. That's the second generation of Ick, or Ick2, and that's what we discussed with Daniel. Where pretty much everything in Ick1 is hardcoded, everything in Ick2 will be user-configurable. It's my last, best chance to go completely overboard in the second system syndrome manner. Where Ick1 was written in a feverish two-week hacking session, rushed because my Jenkins install at the time had broken one time too many, we're taking our time with Ick2. Slow and careful is the tune this time around. Our "minimum viable product" or MVP for Ick2 is defined like this:

Ick2 builds static websites from source in a git repository, using ikiwiki, and published to a web server using rsync. A change to the git repository triggers a new build. It can handle many separate websites, and if given enough worker machines, can build many of them concurrently.

This is a real task, and something we already do with Ick1 at work. It's a reasonable first step for the new program. Some decisions we made:

• The Ick2 controller, which decides which projects to build, and what's the next build step at any one time, will be reactive only. It will do nothing except in response to an HTTP API request. This includes things like timed events. An external service will need to poke the controller at the right time.
• The controller will be accompanied by worker manager processes, which fetch instructions of what to do next, and control actual worker over ssh.
• Provisioning of the workers is out of scope for the MVP. For the MVP we are OK with a static list of workers. In the future we might make worker registration be a dynamic things, but not for the MVP. (Parts or all of this decision may be changed in the future, but we need to start somewhere.)
• The MVP publishing will happen by running rsync to a web server. Providing credentials for the workers to do that is the sysadmin's problem, not something the MVP will handle itself.
• The MVP needs to handle more than one worker, and more than one pipelines, and needs to build things concurrently when there's call for it.
• The MVP will need to read the pipelines (and their steps and any other info) from YAML config files, and can't have that stuff hardcoded.
• The MVP API will have no authentication or authorization stuff yet.

The initial pipelines will be basically like this, but expressed in some way by the user:

1. Clone the source repoistory.
2. Run ikiwiki --build to build the website.
3. Run rsync to publish the website on a server.

Assumptions:

• Every worker can clone from the git server.
• Every worker has all the build tools.
• Every worker has rsync and access to every web server.
• Every pipeline run is clean.

Actions the Ick2 controller API needs to support:

• List all existing projects.
• Trigger a project to build.
• Query what project builds are running.
• Get build logs for a project: current log (from the running build), and the most recent finished build.

A sketch API:

• POST /projects/foo/+trigger Trigger build of project foo. If the git hasn't changed, the build runs anyway.
• GET /projects List names of all projects.
• GET /projects/foo On second thought, I can't think of anything useful for this to return for the MVP. Scratch.
• GET /projects/foo/logs/current Return entire known build log captured so far for the currently running build.
• GET /projects/foo/logs/previous Return entire build log for latest finished build.
• GET /work/bar Used by worker bar: return next not-yet-finished step to run as a JSON object containing fields "project" (name of project for which to run the step) and "shell" (a shell command to run). The call will return the same JSON object until the worker reports it as having finished.
• POST /work/bar/snippet Used by worker bar to report progress on the currently running step: a JSON object containing fields "stdout" (string with output from the shell command's stdout), "stderr" (ditto but stderr), and "exit_code" (the shell command's exit code, if it's finished, or null).

Sequence:

• Git server has a hook that calls "GET /projects/foo/+trigger" (or else this is simulated by user).
• Controller add a build of project foo to queue.
• Worker manager calls "GET /work/bar", gets a shell command to run, and starts running it on its worker.
• While worker runs shell command, every second or so, worker manager calls "POST /work/bar/snippet" to report progress including collected output, if any.
• Controller responds with OK or KILL, and if the latter, worker kills the command it is running. Worker manager continues reporting progress via snippet until shell command is finished (on its own or by having been killed).
• Controller appends any output reported via .../snippet. When it learns a shell command has finished, it updates its idea of the next step to run.
• When controller learns a project has finished building, it rotates the current build log to be the previous one.

The next step will probably be to sketch a yarn test suite of the API and implement a rudimentary one.

Here is my monthly update covering what I have been doing in the free software world (previous month):

• I was elected Debian Project Leader for 2017. I'd like to sincerely thank everyone who voted for me as well as everyone who took part in the election in general especially Mehdi Dogguy for being a worthy opponent. The result was covered on LWN, Phoronix, DistroWatch, iTWire, etc.
• Added support for the Monzo banking API in social-core, a Python library to allow web applications to authenticate using third-parties. (#68)
• Fixed a HTML injection attack in a demo of Russell Keith-Magee's BeeWare presentation library. (#3)
• Updated systemd's documentation to explain why we suggest explicitly calling make all despite the Makefile's "check" target calling it. (#5830)
• Updated the documentation of a breadth-first version of find(1) called bfs to refer to the newly-uploaded Debian package. (#23)
• Updated the configuration for the ticketbot IRC bot (zwiebelbot on OFTC) to identify #reproducible-builds as a Debian-related channel. This is so that bug Debian bug numbers are automatically expanded by the bot. (#7)

---

There is much to like about Docker. Much has been written about it, and about how secure the containerization is. This post isn t about that. This is about keeping what s inside each container secure. I believe we have a fundamental problem here. Earlier this month, a study on security vulnerabilities on Docker Hub came out, and the picture isn t pretty. One key finding:

Over 80% of the :latest versions of official images contained at least on high severity vulnerability!

And it s not the only one raising questions. Let s dive in and see how we got here. It s hard to be secure, but Debian makes it easier Let s say you want to run a PHP application like WordPress under Apache. Here are the things you need to keep secure:

• WordPress itself
• All plugins, themes, customizations
• All PHP libraries it uses (MySQL, image-processing, etc.)
• MySQL
• Apache
• All libraries MySQL or Apache use: OpenSSL, libc, PHP itself, etc.
• The kernel
• All containerization tools

On Debian (and most of its best-known derivatives), we are extremely lucky to have a wonderful security support system. If you run a Debian system, the combination of unattended-updates, needrestart, debsecan, and debian-security-support will help one keep a Debian system secure and verify it is. When the latest OpenSSL bug comes out, generally speaking by the time I wake up, unattended-updates has already patched it, needrestart has already restarted any server that uses it, and I m protected. Debian s security team generally backports fixes rather than just say here s the new version , making it very safe to automatically apply patches. As long as I use what s in Debian stable, all layers mentioned above will be protected using this scheme. This picture is much nicer than what we see in Docker. Problems We have a lot of problems in the Docker ecosystem:

1. No built-in way to know when a base needs to be updated, or to automatically update it
2. Diverse and complicated vendor security picture
3. No way to detect when intermediate libraries need to be updated
4. Complicated final application security picture

Let s look at them individually. Problem #1: No built-in way to know when a base needs to be updated, or to automatically update it First of all, there is nothing in Docker like unattended-updates. Although a few people have suggested ways to run unattended-updates inside containers, there are many reasons that approach doesn t work well. The standard advice is to update/rebuild containers. So how do you know when to do that? It is not all that obvious. Theoretically, official OS base images will be updated when needed, and then other Docker hub images will detect the base update and be rebuilt. So, if a bug in a base image is found, and if the vendors work properly, and if you are somehow watching, then you could be protected. There is work in this area; tools such as watchtower help here. But this can lead to a false sense of security, because: Problem #2: Diverse and complicated vendor security picture Different images can use different operating system bases. Consider just these official images, and the bases they use: (tracking latest tag on each)

• nginx: debian:stretch-slim (stretch is pre-release at this date!)
• mysql: debian:jessie
• mongo: debian:wheezy-slim (previous release)
• apache httpd: debian:jessie-backports
• postgres: debian:jessie
• node: buildpack-deps:jessie, eventually depends on debian:jessie
• wordpress: php:5.6-apache, eventually depends on debian:jessie

And how about a few unofficial images?

• oracle/openjdk: oraclelinux:latest
• robotamer/citadel: debian:testing (dangerous, because testing is an alias for different distros at different times)
• docker.elastic.co/kibana: ubuntu of some sort

The good news is that Debian jessie seems to be pretty popular here. The bad news is that you see everything from Oracle Linux, to Ubuntu, to Debian testing, to Debian oldstable in just this list. Go a little further, and you ll see Alpine Linux, CentOS, and many more represented. Here s the question: what do you know about the security practices of each of these organizations? How well updated are their base images? Even if it s Debian, how well updated is, for instance, the oldstable or the testing image? The attack surface here is a lot larger than if you were just using a single OS. But wait, it gets worse: Problem #3: No way to detect when intermediate libraries need to be updated Let s say your Docker image is using a base that is updated immediately when a security problem is found. Let s further assume that your software package (WordPress, MySQL, whatever) is also being updated. What about the intermediate dependencies? Let s look at the build process for nginx. The Dockerfile for it begins with Debian:stretch-slim. But then it does a natural thing: it runs an apt-get install, pulling in packages from both Debian and an nginx repo. I ran the docker build across this. Of course, the apt-get command brings in not just the specified packages, but also their dependencies. Here are the ones nginx brought in: fontconfig-config fonts-dejavu-core gettext-base libbsd0 libexpat1 libfontconfig1 libfreetype6 libgd3 libgeoip1 libicu57 libjbig0 libjpeg62-turbo libpng16-16 libssl1.1 libtiff5 libwebp6 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxml2 libxpm4 libxslt1.1 nginx nginx-module-geoip nginx-module-image-filter nginx-module-njs nginx-module-xslt ucf Now, what is going to trigger a rebuild if there s a security fix to libssl1.1 or libicu57? (Both of these have a history of security holes.) The answer, for the vast majority of Docker images, seems to be: nothing automatic. Problem #4: Complicated final application security picture And that brings us to the last problem: Let s say you want to run an application in Docker. exim, PostgreSQL, Drupal, or maybe something more obscure. Who is watching for security holes in it? If you re using Debian packages, the Debian security team is. If you re using a Docker image, well, maybe it s the random person that contributed it, maybe it s the vendor, maybe it s Docker, maybe it s nobody. You have to take this burden on yourself, to validate the security support picture for each image you use. Conclusion All this adds up to a lot of work, which is not taken care of for you by default in Docker. It is no surprise that many Docker images are insecure, given this picture. The unfortunate reality is that many Docker containers are running with known vulnerabilities that have known fixes, but just aren t, and that s sad. I wonder if there are any practices people are using that can mitigate this better than what the current best-practice recommendations seem to be?

A bug fix release of RcppTOML arrived on CRAN today. Table arrays were (wrongly) not allowing for nesting; a simply recursion fix addresses this. RcppTOML brings TOML to R. TOML is a file format that is most suitable for configurations, as it is meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now. TOML is making good inroads with newer and more flexible projects such as the Hugo static blog compiler, or the Cargo system of Crates (aka "packages") for the Rust language.

Changes in version 0.1.3 (2017-04-25)

• Nested TableArray types are now correctly handled (#16)

Courtesy of CRANberries, there is a diffstat report for this release. More information is on the RcppTOML page page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

How hard it is to write a compiler for Haskell Core? Not too hard, actually! I wish we had a formally verified compiler for Haskell, or at least for GHC s intermediate language Core. Now formalizing that part of GHC itself seems to be far out of reach, with the many phases the code goes through (Core to STG to CMM to Assembly or LLVM) and optimizations happening at all of these phases and the many complicated details to the highly tuned GHC runtime (pointer tagging, support for concurrency and garbage collection).

Introducing Veggies So to make that goal of a formally verified compiler more feasible, I set out and implemented code generation from GHC s intermediate language Core to LLVM IR, with simplicity as the main design driving factor. You can find the result in the GitHub repository of veggies (the name derives from verifiable GHC ). If you clone that and run ./boot.sh some-directory, you will find that you can use the program some-directory/bin/veggies just like like you would use ghc. It comes with the full base library, so your favorite variant of HelloWorld might just compile and run. As of now, the code generation handles all the Core constructs (which is easy when you simply ignore all the types). It supports a good number of primitive operations, including pointers and arrays I implement these as need and has support for FFI calls into C.

Why you don't want to use Veggies Since the code generator was written with simplicity in mind, performance of the resulting code is abysmal: Everything is boxed, i.e. represented as pointer to some heap-allocated data, including unboxed integer values and unboxed tuples. This is very uniform and simplifies the code, but it is also slow, and because there is no garbage collection (and probably never will be for this project), will fill up your memory quickly. Also, the code is currently only supports 64bit architectures, and this is hard-coded in many places. There is no support for concurrency.

Why it might be interesting to you nevertheless So if it is not really usable to run programs with, should you care about it? Probably not, but maybe you do for one of these reasons:

• You always wondered how a compiler for Haskell actually works, and reading through a little over a thousands lines of code is less daunting than reading through the 34k lines of code that is GHC s backend.
• You have wacky ideas about Code generation for Haskell that you want to experiment with.
• You have wacky ideas about Haskell that require special support in the backend, and want to prototype that.
• You want to see how I use the GHC API to provide a ghc-like experience. (I copied GHC s Main.hs and inserted a few hooks, an approach I copied from GHCJS).
• You want to learn about running Haskell programs efficiently, and starting from veggies, you can implement all the trick of the trade yourself and enjoy observing the speed-ups you get.
• You want to compile Haskell code to some weird platform that is supported by LLVM, but where you for some reason cannot run GHC s runtime. (Because there are no threads and no garbage collection, the code generated by veggies does not require a runtime system.)
• You want to formally verify Haskell code generation. Note that the code generator targets the same AST for LLVM IR that the vellvm2 project uses, so eventually, veggies can become a verified arrow in the top right corner map of the DeepSpec project.

So feel free to play around with veggies, and report any issues you have on the GitHub repository.

Here's what happened in the Reproducible Builds effort between Sunday April 2 and Saturday April 8 2017: Media coverage

• Daniel Peebles reported that tar still has problem with deterministic output and long filenames.
• Dustin Kirkland wrote a blog post referencing reproducible builds as a followup to a recent thread on Hacker News.

Toolchain development and fixes

• Ximin Luo uploaded custom version of GCC6 to our repository to test BUILD_PATH_PREFIX_MAP patches.

Reviews of unreproducible packages 27 package reviews have been added, 14 have been updated and 17 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Aaron M. Ucko (1)
• Adrian Bunk (1)
• Chris Lamb (2)

tests.reproducible-builds.org

• Chris Lamb added support for a .bz2 version of reproducible.json, available at https://reproducible.debian.net/reproducible.json.bz2
• FreeGeek donated a network switch to make room for some additional boards on the armhf build network.

Misc. This week's edition was written by Chris Lamb, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

A new programmer asked on a work chat room how timezones are handled in databases. He asked if it was a good idea to store things in UTC. The senior programmers all laughed as we told some of our horror stories with timezones. Yes, UTC is great; if only it were that simple.
About a week later I was designing the schema for a blue sky project I'm implementing. I had to confront time in all its Pythonic horror.
Let's start with the datetime.datetime class. Datetime objects optionally include a timezone. If no timezone is present, several methods such as timestamp treat the object as a local time in the system's timezone. The timezone method returns a POSIX timestamp, which is always expressed in UTC, so knowing the input timezone is important. The now method constructs such an object from the current time.
However other methods act differently. The utcnow method constructs a datetime object that has the UTC time, but is not marked with a timezone. So, for example datetime.fromtimestamp(datetime.utcnow().timestamp()) produces the wrong result unless your system timezone happens to have the same offset as UTC.
It's also possible to construct a datetime object that includes a UTC time and is marked as having a UTC time. The utcnow method never does this, but you can pass the UTC timezone into the now method and get that effect. As you'd expect, the timestamp method returns the correct result on such a datetime.
Now enter SQLAlchemy, one of the more popular Python ORMs. Its DATETIME type has an argument that tries to request a column capable of storing a a timezone from the underlying database. You aren't guaranteed to get this though; some databases don't provide that functionality. With PostgreSQL, I do get such a column, although something in SQLAlchemy is not preserving the timezones (although it is correctly adjusting the time). That is, I'll store a UTC time in an object, flush it to my session, and then read back the same time represented in my local timezone (marked as my local timezone). You'd think this would be safe.
Enter SQLite. SQLite makes life hard for people wanting to store time; it seems to want to store things as strings. That's fairly incompatible with storing a timezone and doing any sort of comparisons on dates. SQLAlchemy does not try to store a timezone in SQLite. It just trims any timezone information from the datetime. So, if I do something like

d = datetime.now(timezone.utc)
obj.date_col = d
session.add(obj)
session.flush()
assert obj.date_col == d # fails
assert obj.date_col.timestamp() == d.timestamp() # fails
assert d == obj.date_col.replace(tzinfo = timezone.utc) # finally succeeds

There are some unfortunate consequences of this. If you mark your datetimes with timezone information (even if it is always the same timezone), whether two datetimes representing the same datetime compare equal depends on whether objects have been flushed to the session yet. If you don't mark your objects with timezones, then you may not store timezone information on other databases.
At least if you use only the methods we've discussed so far, you're reasonably safe if you use local time everywhere in your application and don't mark your datetimes with timezones. That's undesirable because as our new programmer correctly surmised, you really should be using UTC. This is particularly true if users of your database might span multiple timezones.
You can use UTC time and not mark your objects as UTC. This will give the wrong data with a database that actually does support timezones, but will sort of work with SQLite. You need to be careful never to convert your datetime objects into POSIX time as you'll get the wrong result.
It turns out that my life was even more complicated because parts of my project serialize data into JSON. For that serialization, I've chosen ISO 8601. You've probably seen that format: '2017-04-09T18:17:27.340410+00:00. Datetime provides the convenient isoformat method to print timestamps in the ISO 8601 format. If the datetime has a timezone indication, it is included in the ISO formatted string. If not, then no timezone indication is included. You know how I mentioned that datetime takes a string without a timezone marker as local time? Yeah, well, that's not what 8601 does: UTC all the way, baby! And at least the parser in the iso8601 module will always include timezone markers. So, if you use datetime to print a timestamp without a timezone marker and then read that back in to construct a new datetime on the deserialization side, then you'll get the wrong time. OK, so mark things with timezones then. Well, if you use local time, then the time you get depends on whether you print the ISO string before or after session flush (before or after SQLAlchemy trims the timezone information as it goes to SQLite).
It turns out that I had the additional complication of one side of my application using SQLite and one side using PostgreSQL. Remember how I mentioned that something between SQLAlchemy and PostgreSQL was recasting my times in local timezone (although keeping the time the same)? Well, consider how that's going to work. I serialize with the timezone marker on the PostgreSQL side. I get a ISO8601 localtime marked with the correct timezone marker. I deserialize on the SQLite side. Before session flush, I get a local time marked as localtime. After session flush, I get a local time with no marking. That's bad. If I further serialize on the SQLite side, I'll get that local time incorrectly marked as UTC. Moreover, all the times being locally generated on the SQLite side are UTC, and as we explored, SQLite really only wants one timezone in play.
I eventually came up with the following approach:

1. If I find myself manipulating a time without a timezone marking, assert that its timezone is UTC not localtime.


2. Always use UTC for times coming into the system.


3. If I'm generating an ISO 8601 time from a datetime that has a timezone marker in a timezone other than UTC, represent that time as a UTC-marked datetime adjusting the time for the change in timezone.

This is way too complicated. I think that both datetime and SQLAlchemy's SQLite time handling have a lot to answer for. I think SQLAlchemy's core time handling may also have some to answer for, but I'm less sure of that.

Ikea recently launched their Tr dfri smart lighting platform in the US. The idea of Ikea plus internet security together at last seems like a pretty terrible one, but having taken a look it's surprisingly competent. Hardware-wise, the device is pretty minimal - it seems to be based on the Cypress[1] WICED IoT platform, with 100MBit ethernet and a Silicon Labs Zigbee chipset. It's running the Express Logic ThreadX RTOS, has no running services on any TCP ports and appears to listen on two single UDP ports. As IoT devices go, it's pleasingly minimal.

That single port seems to be a COAP server running with DTLS and a pre-shared key that's printed on the bottom of the device. When you start the app for the first time it prompts you to scan a QR code that's just a machine-readable version of that key. The Android app has code for using the insecure COAP port rather than the encrypted one, but the device doesn't respond to queries there so it's presumably disabled in release builds. It's also local only, with no cloud support. You can program timers, but they run on the device. The only other service it seems to run is an mdns responder, which responds to the _coap._udp.local query to allow for discovery.

From a security perspective, this is pretty close to ideal. Having no remote APIs means that security is limited to what's exposed locally. The local traffic is all encrypted. You can only authenticate with the device if you have physical access to read the (decently long) key off the bottom. I haven't checked whether the DTLS server is actually well-implemented, but it doesn't seem to respond unless you authenticate first which probably covers off a lot of potential risks. The SoC has wireless support, but it seems to be disabled - there's no antenna on board and no mechanism for configuring it.

However, there's one minor issue. On boot the device grabs the current time from pool.ntp.org (fine) but also hits http://fw.ota.homesmart.ikea.net/feed/version_info.json . That file contains a bunch of links to firmware updates, all of which are also downloaded over http (and not https). The firmware images themselves appear to be signed, but downloading untrusted objects and then parsing them isn't ideal. Realistically, this is only a problem if someone already has enough control over your network to mess with your DNS, and being wired-only makes this pretty unlikely. I'd be surprised if it's ever used as a real avenue of attack.

Overall: as far as design goes, this is one of the most secure IoT-style devices I've looked at. I haven't examined the COAP stack in detail to figure out whether it has any exploitable bugs, but the attack surface is pretty much as minimal as it could be while still retaining any functionality at all. I'm impressed.

[1] Formerly Broadcom

comments

A book about being woman, mother in a modern but still traditional society. About happiness and fulfillment, love and sex, responsibility and dependency, about life: Stella Stejskal s Im Mezzanin (in German).
Written by a friend back from my old times in Vienna, Stella Stejskal, like me an emigrant, I saw parts of this book during writing, and I was happy to see the final product and read it. The first novel of Stella turns around Anna, the protagonist, a wife, mother, and woman, trying to find her balance between the house, family, kids, work, and her incredible power and thrive to live, live to the fullest.

Dein Leben in der Vorstadt, im Einfamilienhaus mit dem Garten, macht Dich nicht gl cklich. Vordergr ndig hast Du alles, was eine Frau sich w nscht und doch fehlt Dir etwas Wesentliches: Verlangen und Leidenschaft.

(Your life in the suburb, in the one-family home with garden, it doesn t make you happy. On the surface you do have everything what a woman could wish for, but you are missing something essential: desire and passion) The author does not shy away from explicit language without ever dropping into the Vernacolo, the banalities. She manages to convey the incredible tension many of those being stretched out between the necessities of daily life and the need for a more personal life. Last but not least, I loved this book for quoting one of my most favorite lines from a song:

Konstantin Weckers Was passiert in den Jahren, drangen leise durch den Raum. Komm, wir gehen mit der Flut und verwandeln mit den Wellen unsere Angst in neuen Mut , sang ich mit und dachte an den Sommer

For those capable of German, very recommendable.

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Fixed two issues in try.diffoscope.org, a web-based version of the diffoscope in-depth and content-aware diff utility:
  • Fix command-line API breakage. (commit)
  • Use deb.debian.org over httpredir.debian.org. (commit)
• Made a number of improvements to travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every code change) travis.debian.net, including:
  • Correctly detecting the distribution to build with for some tags. (commit)
  • Use Lintian from the backports repository where appropriate. (#44)
  • Don't build upstream/ branches even if they contain .travis.yml files. (commit)
• Fixed an issue in django-staticfiles-dotd, my Django staticfiles adaptor to concatentate .d-style directories, where some .d directories were being skipped. This was caused by modifying the contents of a Python list during iteration. (#3)
• Performed some miscelleanous cleanups in django12factor, a Django utility to make projects adhere better to the 12-factor web-application philosophy. (#58)
• Submitted a pull request for Doomsday-Engine, a portable, enhanced source port of Doom, Heretic and Hexen, to make the build reproducible (#16)
• Created a pull request for gdata-python-client (a Python client library for Google APIs) to make the build reproducible. (#56)
• Authored a pull request for the MochaJS JavaScript test framework to make the build reproducible. (#2727)
• Filed a pull request against vine, a Python promises library, to avoid non-determinstic default keyword argument appearing in the documentation. (#12)
• Filed an issue for the Redis key-value database addressing build failures on the MIPS architecture. (#3874)
• Submitted a bug report against xdotool a tool to automate window and keyboard interactions reporting a crash when searching after binding an action with behave. (#169)
• Reviewed a pull request from Dan Palmer for django-email-from-template, a library to send emails in Django generated entirely from the templating system, which intends to add an option to send mails upon transaction commit.

---

A new release of RcppTOML is now on CRAN. This release fixes a few parsing issues for less frequently-used inputs: vectors of boolean or date(time) types, as well as table array input. RcppTOML brings TOML to R. TOML is a file format that is most suitable for configurations, as it is meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now. TOML is making good inroads with newer and more flexible projects such as the Hugo static blog compiler, or the Cargo system of Crates (aka "packages") for the Rust language.

Changes in version 0.1.2 (2017-03-26)

• Dates and Datetimes in arrays in the input now preserve their types instead of converting to numeric vectors (#13)
• Boolean vectors are also correctly handled (#14)
• TableArray types are now stored as lists in a single named list (#15)
• The README.md file was expanded with an example and screenshot.
• Added file init.c with calls to R_registerRoutines() and R_useDynamicSymbols(); also use .registration=TRUE in useDynLib in NAMESPACE
• Two example files were updated.

Courtesy of CRANberries, there is a diffstat report for this release. More information is on the RcppTOML page page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dear lazyweb, I am trying to get a good way to present the categorization of several cases studied with a fitting graph. I am rating several vulnerabilities / failures according to James Cebula et. al.'s paper, A taxonomy of Operational Cyber Security Risks; this is a somewhat deep taxonomy, with 57 end items, but organized in a three levels deep hierarchy. Copying a table from the cited paper (click to display it full-sized): My categorization is binary: I care only whether it falls within a given category or not. My first stab at this was to represent each case using a star or radar graph. As an example: As you can see, to a "bare" star graph, I added a background color for each top-level category (blue for actions of people, green for systems and technology failures), red for failed internal processes and gray for external events), and printed out only the labels for the second level categories; for an accurate reading of the graphs, you have to refer to the table and count bars. And, yes, according to the Engineering Statistics Handbook:

Star plots are helpful for small-to-moderate-sized multivariate data sets. Their primary weakness is that their effectiveness is limited to data sets with less than a few hundred points. After that, they tend to be overwhelming.

I strongly agree with the above statement And stating that "a few hundred points" can be understood is even an overstatement. 50 points are just too much. Now, trying to increase usability for this graph, I came across the Sunburst diagram. One of the proponents for this diagram, John Stasko, has written quite a bit about it. Now... How to create my beautiful Sunburst diagram? That's a tougher one. Even though the page I linked to in the (great!) Data visualization catalogue presents even some free-as-in-software tools to do this... They are Javascript projects that will render their beautiful plots (even including an animation)... To the browser. I need them for a static (i.e. to be printed) document. Yes, I can screenshot and all, but I want them to be automatically generated, so I can review and regenerate them all automatically. Oh, I could just write JSON and use SaaS sites such as Aculocity to do the heavy-lifting, but if you know me, you will understand why I don't want to. So... I set out to find a Gunnar-approved way to display the information I need. Now, as the Protovis documentation says, an icicle is simply a sunburst transformed from polar to cartesian coordinates... But I came to a similar conclusion: The tools I found are not what I need. OK, but an icicle graph seems much simpler to produce I fired up my Emacs, and started writing using Ruby, RMagick and RVG... I decided to try a different way. This is my result so far: So... What do you think? Does this look right to you? Clearer than the previous one? Worst? Do you have any idea on how I could make this better? Oh... You want to tell me there is something odd about it? Well, yes, of course! I still need to tweak it quite a bit. Would you believe me if I told you this is not really a left-to-right icicle graph, but rather a strangely formatted Graphviz non-directed graph using the dot formatter? I can assure you you don't want to look at my Graphviz sources... But in case you insist... Take them and laugh. Or cry. Of course, this file comes from a hand-crafted template, but has some autogenerated bits to it. I have still to tweak it quite a bit to correct several of its usability shortcomings, but at least it looks somewhat like what I want to achieve. Anyway, I started out by making a "dear lazyweb" question. So, here it goes: Do you think I'm using the right visualization for my data? Do you have any better suggestions, either of a graph or of a graph-generating tool? Thanks! [update] Thanks for the first pointer, Lazyweb! I found a beautiful solution; we will see if it is what I need or not (it is too space-greedy to be readable... But I will check it out more thoroughly). It lays out much better than anything I can spew out by myself Writing it as a mindmap using TikZ directly from within LaTeX, I get the following result:

Here's what happened in the Reproducible Builds effort between Sunday February 26 and Saturday March 4 2017: Upcoming Events Ed Maste will present Reproducible Builds in FreeBSD at AsiaBSDCon 2017. Ximin Luo will present Reproducible builds, its uses and the future at Open Source Days in Copenhagen on March 18. Holger Levsen will give a talk at the German Unix User Group's "Fr hjahrsfachgespr ch" in Darmstadt, Germany, about Reproducible Builds everywhere on March 23. Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th. Media coverage Aspiration Tech published a very detailed report on our Reproducible Builds World Summit 2016 in Berlin. Reproducible work in other projects Duncan published a very thorough post on the Rust Programming Language Forum about reproducible builds in the Rust compiler and toolchain. In particular, he produced a table recording the reproducibility of different build products under different individual variations, totalling 187 build+variation combinations. Packages reviewed and fixed, and bugs filed Chris Lamb:

• #856614 filed against dask.distributed.
• #856807 filed against node-mocha, forwarded and merged upstream
• #856834 filed against tendermint-go-rpc.
• #856860 filed against archvsync.

Dhole:

• #856257 filed against tunnelx.

Reviews of unreproducible packages 60 package reviews have been added, 8 have been updated and 13 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been added:

• timestamp_in_fonts_generated_by_opentype

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Chris Lamb (3)

diffoscope development diffoscope 78 was uploaded to unstable and jessie-backports by Mattia Rizzolo. It included contributions from:

• Chris Lamb:
  • Make tests that call xxd work on jessie again. (Closes: #855239)
  • tests: Move normalize_zeros to more generic utils.data module.
• Brett Smith:
  • comparators.json: Catch bad JSON errors on Python pre-3.5. (Closes: #855233)
• Ed Maste:
  • Use BSD-style stat(1) on FreeBSD. (Closes: #855169)

In addition, the following changes were made on the experimental branch:

• Chris Lamb (4):
  • Tidy cbfs tests.
  • Correct "exercice" -> "exercise" typo.
  • Support newer versions of cbfstool to avoid test failure. (Closes: #856446)
  • Skip icc test that varies on endian if the (Debian-specific) patch is not present. (Closes: #856447)

reproducible-website development

• anonmos1:
  • Replace root with 0 when giving UIDs/GIDs to GNU tar.
• Holger Levsen and Chris Lamb:
  • Publish report by Aspiration Tech about RWS Berlin 2016.

tests.reproducible-builds.org

• Ed Maste continued his work on testing FreeBSD for reproducibility but hasn't reached the magical 100% mark yet.
• Holger Levsen adjusted the Debian builders scheduling frequency, mostly to adopt to armhf having become faster due to the two new nodes.

Misc. This week's edition was written by Ximin Luo, Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

RProtoBuf provides R bindings for the Google Protocol Buffers ("Protobuf") data encoding and serialization library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects. The RProtoBuf 0.4.9 release is the fourth and final update this weekend following the request by CRAN to not use package= in .Call() when PACKAGE= is really called for. Some of the code in RProtoBuf 0.4.9 had this bug; some other entry points had neither (!!). With the ongoing drive to establish proper registration of entry points, a few more issues were coming up, all of which are now addressed. And we had some other unreleased minor cleanup, so this made for a somewhat longer (compared to the other updates this weekend) NEWS list:

Changes in RProtoBuf version 0.4.9 (2017-03-06)

• A new file init.c was added with calls to R_registerRoutines() and R_useDynamicSymbols()
• Symbol registration is enabled in useDynLib
• Several missing PACKAGE= arguments were added to the corresponding .Call invocations
• Two (internal) C++ functions were renamed with suffix _cpp to disambiguate them from R functions with the same name
• All of above were part of #26
• Some editing corrections were made to the introductory vignette (David Kretch in #25)
• The 'configure.ac' file was updated, and renamed from the older converntion 'configure.in', along with 'src/Makevars'. (PR #24 fixing #23)

CRANberries also provides a diff to the previous release. The RProtoBuf page has an older package vignette, a 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

RcppSMC provides Rcpp-based bindings to R for them Sequential Monte Carlo Template Classes (SMCTC) by Adam Johansen described in his JSS article. RcppSMC release 0.1.5 is one of several maintenance releases this weekend to fix an issue flagged by CRAN: calls to .Call() sometimes used package= where PACKAGE= was meant. This came up now while the registration mechanism is being reworked. Hence RcppSMC was updated, and we took the opportunity to bring several packaging aspects up to the newest standards, including support for the soon-to-be required registration of routines. No new code or features were added. The NEWS file entries follow below:

Changes in RcppSMC version 0.1.5 (2017-03-03)

• Correct .Call to use PACKAGE= argument
• DESCRIPTION, NAMESPACE, README.md changes to comply with current R CMD check levels
• Added file init.c with calls to R_registerRoutines() and R_useDynamicSymbols()
• Updated .travis.yml file for continuous integration

Courtesy of CRANberries, there is a diffstat report for this release. More information is on the RcppSMC page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

These are notes from my research that led to the publication of the password hashers article. This article is more technical than the previous ones and compares the various cryptographic primitives and algorithms used in the various software I have reviewed. The criteria for inclusion on this list is fairly vague: I mostly included a password hasher if it was significantly different from the previous implementations in some way, and I have included all the major ones I could find as well.

The first password hashers Nic Wolff claims to be the first to have written such a program, all the way back in 2003. Back then the hashing algorithm was MD5, although Wolff has now updated the algorithm to use SHA-1 and still maintains his webpage for public use. Another ancient but unrelated implementation, is the Standford University Applied Cryptography's pwdhash software. That implementation was published in 2004 and unfortunately, that implementation was not updated and still uses MD5 as an hashing algorithm, but at least it uses HMAC to generate tokens, which makes the use of rainbow tables impractical. Those implementations are the simplest password hashers: the inputs are simply the site URL and a password. So the algorithms are, basically, for Wolff's:

token = base64(SHA1(password + domain))

And for Standford's PwdHash:

token = base64(HMAC(MD5, password, domain)))

SuperGenPass Another unrelated implementation that is still around is supergenpass is a bookmarklet that was created around 2007, originally using MD5 as well but now supports SHA512 now although still limited to 24 characters like MD5 (which needlessly limits the entropy of the resulting password) and still defaults MD5 with not enough rounds (10, when key derivation recommendations are more generally around 10 000, so that it's slower to bruteforce). Note that Chris Zarate, the supergenpass author, actually credits Nic Wolff as the inspiration for his implementation. Supergenpass is still in active development and is available for the browser (as a bookmarklet) or mobile (as an webpage). Supergenpass allows you to modify the password length, but also add an extra profile secret which adds to the password and generates a personalized identicon presumably to prevent phishing but it also introduces the interesting protection, the profile-specific secret only found later in Password Hasher Plus. So the Supergenpass algorithm looks something like this:

token = base64(SHA512(password + profileSecret + ":" + domain, rounds))

The Wijjo Password Hasher Another popular implementation is the Wijjo Password Hasher, created around 2006. It was probably the first shipped as a browser extension which greatly improved the security of the product as users didn't have to continually download the software on the fly. Wijjo's algorithm also improved on the above algorithms, as it uses HMAC-SHA1 instead of plain SHA-1 or HMAC-MD5, which makes it harder to recover the plaintext. Password Hasher allows you to set different password policies (use digits, punctuation, mixed case, special characters and password length) and saves the site names it uses for future reference. It also happens that the Wijjo Password Hasher, in turn, took its inspiration on different project, hashapass.com, created in 2006 and also based on HMAC-SHA-1. Indeed, hashapass "can easily be generated on almost any modern Unix-like system using the following command line pattern":

echo -n parameter \
  openssl dgst -sha1 -binary -hmac password \
  openssl enc -base64 \
  cut -c 1-8

So the algorithm here is obviously:

token = base64(HMAC(SHA1, password, domain + ":" + counter)))[:8]

... although in the case of Password Hasher, there is a special routine that takes the token and inserts random characters in locations determined by the sum of the values of the characters in the token.

Password Hasher Plus Years later, in 2010, Eric Woodruff ported the Wijjo Password Hasher to Chrome and called it Password Hasher Plus. Like the original Password Hasher, the "plus" version also keeps those settings in the extension and uses HMAC-SHA-1 to generate the password, as it is designed to be backwards-compatible with the Wijjo Password Hasher. Woodruff did add one interesting feature: a profile-specific secret key that gets mixed in to create the security token, like what SuperGenPass does now. Stealing the master password is therefore not enough to generate tokens anymore. This solves one security concern with Password Hasher: an hostile page could watch your keystrokes and steal your master password and use it to derive passwords on other sites. Having a profile-specific secret key, not accessible to the site's Javascript works around that issue, but typing the master password directly in the password field, while convenient, is just a bad idea, period. The final algorithm looks something like:

token = base64(HMAC(SHA1, password, base64(HMAC(SHA1, profileSecret, domain + ":" + counter))))

Honestly, that seems rather strange, but it's what I read from the source code, which is available only after decompressing the extension nowadays. I would have expected the simplest version:

token = base64(HMAC(SHA1, HMAC(SHA1, profileSecret, password), domain + ":" + counter))

The idea here would be "hide" the master password from bruteforce attacks as soon as possible... But maybe this is all equivalent. Regardless, Password Hasher Plus then takes the token and applies the same special character insertion routine as the Password Hasher.

LessPass Last year, Guillaume Vincent a french self-described "humanist and scuba diving fan" released the lesspass extension for Chrome, Firefox and Android. Lesspass introduces several interesting features. It is probably the first to include a commandline version. It also uses a more robust key derivation algorithm (PBKDF2) and takes into account the username on the site, allowing multi account support. The original release (version 1) used only 8192 rounds which is now considered too low. In the bug report it was interesting to note that LessPass couldn't do the usual practice of running the key derivation for 1 second to determine the number of rounds needed as the results need to be deterministic. At first glance, the LessPass source code seems clear and easy to read which is always a good sign, but of course, the devil is in the details. One key feature that is missing from Password Hasher Plus is the profile-specific seed, although it should be impossible, for a hostile web page to steal keystrokes from a browser extension, as far as I know. The algorithm then gets a little more interesting:

entropy = PBKDF2(SHA256, masterPassword, domain + username + counter, rounds, length)
where
    rounds=10000
    length=32

entropy is then used to pick characters to match the chosen profile. Regarding code readability, I got quickly confused by the PBKDF2 implementation: SubtleCrypto.ImportKey() doesn't seem to support PBKDF2 in the API, yet it's how it is used there... Is it just something to extract key material? We see later what looks like a more standard AES-based PBKDF2 implementation, but this code looks just strange to me. It could be me unfamilarity with newer Javascript coding patterns, however. There is also a lesspass-specific character picking routing that is also not base64, and different from the original Password Hasher algorithm.

Master Password A review of password hashers would hardly be complete without mentioning the Master Password and its elaborate algorithm. While the applications surrounding the project are not as refined (there is no web browser plugin and the web interface can't be easily turned into a bookmarklet), the algorithm has been well developed. Of all the password managers reviewed here, Master Password uses one of the strongest key derivation algorithms out there, scrypt:

key = scrypt( password, salt, cost, size, parallelization, length )
where
salt = "com.lyndir.masterpassword" + len(username) + name
cost = 32768
size = 8
parallelization = 2
length = 64
entropy = hmac-sha256(key, "com.lyndir.masterpassword" + len(domain) + domain + counter )

Master Password the uses one of 6 sets of "templates" specially crafted to be "easy for a user to read from a screen and type using a keyboard or smartphone" and "compatible with most site's password policies", our "transferable" criteria defined in the first passwords article. For example, the default template mixes vowels, consonants, numbers and symbols, but carefully avoiding possibly visibly similar characters like O and 0 or i and 1 (although it does mix 1 and l, oddly enough). The main strength of Master Password seems to be the clear definition of its algorithm (although Hashpass.com does give out OpenSSL commandline examples...), which led to its reuse in another application called freepass. The Master Password app also doubles as a stateful password manager...

Other implementations I have also considered including easypasswords, which uses PBKDF2-HMAC-SHA1, in my list of recommendations. I discovered only recently that the author wrote a detailed review of many more password hashers and scores them according to their relative strength. In the end, I ended up covering more LessPass since the design is very similar and LessPass does seem a bit more usable. Covering LessPass also allowed me to show the contrast and issues regarding the algorithm changes, for example. It is also interesting to note that the EasyPasswords author has criticized the Master Password algorithm quite severely:

[...] scrypt isn t being applied correctly. The initial scrypt hash calculation only depends on the username and master password. The resulting key is combined with the site name via SHA-256 hashing then. This means that a website only needs to break the SHA-256 hashing and deduce the intermediate key as long as the username doesn t change this key can be used to generate passwords for other websites. This makes breaking scrypt unnecessary[...]

During a discussion with the Master Password author, he outlined that "there is nothing "easy" about brute-force deriving a 64-byte key through a SHA-256 algorithm." SHA-256 is used in the last stage because it is "extremely fast". scrypt is used as a key derivation algorithm to generate a large secret and is "intentionnally slow": "we don't want it to be easy to reverse the master password from a site password". "But it' unnecessary for the second phase because the input to the second phase is so large. A master password is tiny, there are only a few thousand or million possibilities to try. A master key is 8^64, the search space is huge. Reversing that doesn't need to be made slower. And it's nice for the password generation to be fast after the key has been prepared in-memory so we can display site passwords easily on a mobile app instead of having to lock the UI a few seconds for every password." Finally, I considered covering Blum's Mental Hash (also covered here and elsewhere). This consists of an algorithm that can basically be ran by the human brain directly. It's not for the faint of heart, however: if I understand it correctly, it will require remembering a password that is basically a string of 26 digits, plus compute modulo arithmetics on the outputs. Needless to say, most people don't do modulo arithmetics every day...

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Submitted a number of pull requests to the Django web development framework:
  • Add a --mode=unified option to the "diffsettings" management command. (#8113)
  • Fix a crash in setup_test_environment() if ALLOWED_HOSTS is a tuple. (#8101)
  • Use Python 3 "shebangs" now that the master branch is Python 3 only. (#8105)
  • URL namespacing warning should consider nested namespaces. (#8102)
• Created an experimental patch against the Python interpreter in order to find reproducibility-related assumptions in dict handling in arbitrary Python code. (#29431)
• Filed two issues against dh-virtualenv, a tool to package Python virtualenv environments in Debian packages:
  • Fix "upgrage-pip" typo in usage documentation. (#195)
  • Missing DH_UPGRADE_SETUPTOOLS equivalent for dh_virtualenv (#196)
• Fixed a large number of spelling corrections in Samba, a free-software re-implementation of the Windows networking protocols.
• Reviewed and merged a pull request by @jheld for django-slack (my library to easily post messages to the Slack group-messaging utility) to support per-message backends and channels. (#63)
• Created a pull request for django-two-factor-auth, a complete Two-Factor Authentication (2FA) framework for projects using the Django web development framework to drop use of the @lazy_property decorator to ensure compatibility with Django 1.11. (#195)
• Filed, triaged and eventually merged a change from @evgeni to fix an autopkgtest-related issue in travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every code change) travis.debian.net. (#41)
• Submitted a pull request against social-core a library to allow Python applications to authenticate against third-party web services such as Facebook, Twitter, etc. to use the more-readable X if Y else Z construction over Y and X or Z. (#44)
• Filed an issue against freezegun (a tool to make it easier to write Python tests involving times) to report that dateutils was missing from requirements.txt. (#173)
• Submitted a pull request against the Hypothesis "QuickCheck"-like testing framework to make the build reproducible. (#440)
• Fixed an issue reported by @davidak in trydiffoscope (a web-based version of the diffoscope in-depth and content-aware diff utility) where the maximum upload size was incorrectly calculated. (#22)
• Created a pull request for the Mars Simulation Project to remove some embedded timestamps from the changelog.gz and mars-sim.1.gz files in order to make the build reproducible. (#24)
• Filed a bug against the cpio archiving utility to report that the testsuite fails when run in the UTC +1300 timezone. (Thread)
• Submitted a pull request against the "pnmixer" system-tray volume mixer in order to make the build reproducible. (#153)
• Sent a patch to Testfixtures (a collection of helpers and mock objects that are useful when writing Python unit tests or doctests) to make the build reproducible. (#56)
• Created a pull request for the "Cloud" Sphinx documentation theme in order to make the output reproducible. (#22)

---