I farm bits and pieces out to the guys who are much more brilliant than I am. I say, "build me a laser", this. "Design me a molecular analyzer", that. They do, and I just stick 'em together. (Seth Brundle, "The Fly")

When I decided to try and turn siterefactor into staticsite, I decided that I would go ahead only for as long as it could be done with minimal work, writing code in the most straightforward way on top of existing and stable components. I am pleased by how far that went. Python-Markdown It works fast enough, already comes with extensions for most of what I needed, and can be extended in several ways. One of the extension methods is a hook for manipulating the ElementTree of the rendered document before serializing it to HTML, which made it really easy to go and process internal links in all <a href= and <img src= attributes. To tell an internal link from an external link I just use the standard python urlparse and see if the link has a scheme or a netloc component. If it does not, and if it has a path, then it is an internal link. This also means that I do not need to invent new Markdown syntax for internal references, avoiding the need for remembering things like [text]( < relref "blog/post.md" > ) or [text]( filename /blog/post.md). In staticsite, it's just [text](/blog/post.md) or [text](post.md) if the post is nearby. This feels nicely clean to me: if I wanted to implement fancy markdown features, I could do it as Python-Markdown extensions and submit them upstream. If I wanted to implement fancy interlinking features, I could do it with a special url scheme in links. For example, it would be straigtforward to implement a ssite: url scheme that expanded the url with elements from staticsite's settings using a call to python's string.format (ssite: SETTING_NAME /bar maybe?), except I do not currently see any use cases for extending internal linking from what it is now. Jinja2 Jina2 is a template engine that I already knew, it is widely used, powerful and pleasant to use, both on the templating side and on the API's side. It is not HTML specific, so I can also use it to generate Atom, RSS2, "dynamic" site content, and even new site Markdown pages. Implementing RSS and Atom feeds was just a matter of writing and testing these Jinja2 macros and then reusing them anywhere. toml, yaml, json No need to implement my own front matter parsing. Also, reusing the same syntax as Hugo allows me to just link to its documentation. python-slugify I found python-slugify so I did not bother writing a slug-generating function. As a side effect, now things works better than I would even have thought to implement, including transliteration of non-ascii characters: (I just filed an RFP) python-livereload Implementing ssite serve which monitors the file system and autoreloads when content changes and renders everything on the fly, took about an hour. Most of that hour went into implementing rendering pages on demand. Then I discovered that it autoreloads even when I edit staticsite's source code. Then I discovered that it communicates with the browser and even automatically triggers a page refresh. I can keep vim on half my screen and a browser in the other half, and I get live preview for free every time I save, without ever leaving the editor. Bootstrap I already use Bootstrap at work, so creating the default theme templates with it took about 10 minutes. This morning I tried looking at my website using my mobile phone, and I pleasantly saw it automatically turning into a working mobile version of itself. Pygments Python-Markdown uses Pygments for syntax highlighting, and it can be themed just by loading a .css. So, without me really doing anything, even staticsite's syntax highligthing is themable, and there's even a nice page with a list of themes to choose from. Everything else... Command line parsing? Straight argparse. Logging? python's logging support. Copying static resource files? shutil.copy2. Parsing dates? dateutil.parser. Timing execution? time.perf_counter. Timezone handling? pytz. Building the command to run an editor? string.format. Matching site pages? fnmatch.translate. ...and then some. If I ever decide to implement incremental rendering, how do I implement tracking which source files have changed? Well, for example, how about just asking git?

As most distributions have switched to systemd as default "init", some people have asked why we actually keep sysvinit around, as it's old and crusty, and systemd can do so much more. My answer is: systemd and sysvinit solve entirely different problems, and neither can ever replace the other fully. The big argument in favour of systemd is integration: when you keep everything in one process, it is easy to make sure that components talk to each other. That is nice, but kind of misses the point: integration could be done via IPC as well, and systemd uses a lot of it to separate privileges anyway. The tighter integration comes from something else: the configuration format. Where sysvinit uses a lot of shell scripts ("imperative" format), systemd uses explicit keywords ("descriptive" format), and this is the important design decision here: descriptive formats cannot do anything unanticipated, neither good nor bad. This is important for UI programmers: if there is a button that says "if I close the lid switch, suspend the laptop", then this needs to be connected to functionality that behaves this way, and this connection needs to work in both directions, so the user will find the active setting there when opening the dialog. So, we need to limit ourselves to a closed set of choices, so the UI people can prepare translations. This is the design tradeoff systemd makes: better integration in exchange for less flexibility. Of course, it is simple to allow "custom" actions everywhere, but that starts to break the "well-integrated" behaviour. There is an old blog post from Joey Hess about how to implement a simple alarm clock, using systemd, and this shows both how powerful and how limited the system is. On one hand, the single line WakeSystem=true takes care of the entire dependency chain attached to making sure the system is actually on -- when the system is going to suspend mode, some program needs to determine that there are scheduled jobs that should wake up the system, determined which of these is the next one, program the wakeup time into the hardware and only then allow the suspend to continue. On the other hand, the descriptive framework already breaks down in this simple example, because the default behaviour of the system is to go to sleep if the lid switch is closed, and the easiest way to fix this is to tell systemd to ignore the lid switch while the program is running. The IPC permissions disallow scheduled user jobs from setting this, so a "custom" action is taken that sets up the "ignore the lid switch" policy, changes to an unprivileged user, and runs the actual job. So, it is possible to get back flexibility, but this is traded for integration: while the job is running, any functionality that the "suspend laptop when the lid is closed" button had, is broken. For something simple as a switch on a personal laptop that breaks if the user configures something that is non-standard, that is acceptable. However, the keywords in the service unit files, the timer files, etc. are also part of a descriptive framework, and the services using them expect that the things they promise are kept by the framework, so there is a limit to how many exceptions you can grant. SystemV init really comes from the other direction here. Nothing is promised to init scripts but a vague notion that they will be run with sufficient privileges. There is no dependency tracking that ensures that certain services will always start before others, just priorities. You can add a dependency system like insserv if you like and make sure that all the requirements for it to work (i.e. services declaring their dependencies) are given, and there is a safe fallback of not parallelizing so we can always play it safe. Because there is so little promised to scripts, writing them becomes a bit tedious -- all of them have a large case statement where they parse the command line, all of them need to reimplement dropping privileges and finding the PID of a running process. There are lots of helpers for the more common things, like the LSB standard functions for colorful and matching console output, and Debian's start-stop-daemon for privilege and PID file handling, but in the end it remains each script's responsibility. Systemd will never be able to provide the same flexibility to admins while at the same time keeping the promises made in the descriptive language, and sysvinit will never reach the same amount of easy integration. I think it is fairly safe to predict both of these things, and I'd even go a step further: if any one of them tried, I'd ask the project managers what they were thinking. The only way to keep systems as complex as these running is to limit the scope of the project. With systemd, the complexity is kept in the internals, and it is important for manageability that all possible interactions are well-defined. That is a huge issue when writing adventure games, where you need to define interaction rules for each player action in combination with each object, and each pair of objects that can be combined with the "Use" action. Where adventure games would default to "This doesn't seem to work." or "I cannot use this.", this would not be a helpful error message when we're trying to boot, so we really need to cut down on actions and object types here. Sysvinit, on the other hand, is both more extreme in limiting the scope to very few actions (restart, wait, once) and only one object type (process that can be started and may occasionally terminate) in the main program, and a lot more open in what scripts outside the main program are allowed to do. The prime example for this is that the majority of configuration takes place outside the inittab even -- a script is responsible for the interface that services are registered by dropping init scripts into specific directories, and scripts can also create additional interfaces, but none of this is intrinsic to the init system itself. Which system is the right choice for your system is just that: your choice. You would choose a starting point, and customize what is necessary for your computer to do what you want it to do. Here's the thing: most users will be entirely happy with fully uncustomized systemd. It will suspend your laptop if you close the lid, and even give your download manager veto power. I fully support Debian's decision to use systemd as the default init for new installs, because it makes sense to use the default that is good for the vast majority of users. However, my opening statement still stands: systemd can never support all use cases, because it is constrained by its design. Trying to change that would turn it into an unmaintainable mess, and the authors are right in rejecting this. There are use cases that define the project, and use cases that are out of scope. This is what we need sysvinit for: the bits that the systemd project managers rightfully reject, but that are someone's use case. Telling these users to get lost would be extremely bad style for the "universal operating system." For example, if someone needs a service that asks a database server for a list of virtual machines to start, runs each in its private network namespace that is named after replacing part of the virtual machine name with the last two octets of the host machine's IP address, then binds a virtual Ethernet device into the network namespace and sets up a NAT rule that allows devices to talk to the public Internet and a firewall rule to stop VMs from talking to each other. Such a beast would live outside of systemd's world view. You can easily start it, but systemd would not know how to monitor it (as long as there is some process still running, is that a good sign), not know how to shut down one instance, not know how to map processes to instances and so on. SystemV init has the advantage here that none of these things are defined by the system, and as long as my init script has some code to handle "status", I can easily check up on my processes with the same interface I'd use for everything else. This requires me to have more knowledge of how the init system works, that is true, however I'd also venture that the learning curve for sysvinit is shallower. If you know shell scripting, you can understand what init scripts do, without memorizing special rules that govern how certain keywords interact, and keeping track of changes to these rules as the feature set is expanded. That said, I'm looking at becoming the new maintainer of the sysvinit package so we can continue to ship it and give our users a choice, but I'd definitely need help, as I only have a few systems left that won't run properly with systemd because of its nasty habit to keep lots of code in memory where sysvinit would unload it after use (by letting the shell process end), and these don't make good test systems. The overall plan is to set up a CI system that

• tests sysvinit -> systemd transition
• tests systemd -> sysvinit transition
• tests debootstrap with systemd
• tests debootstrap with sysvinit
• tests whether service start works for a list of services, so we can catch regressions
• scans packages that have systemd unit files for init scripts, and vice versa

Ideally, I'd like to reach a state where we can actually ensure that the default configuration is somewhat similar regardless of which init system we use, and that it is possible to install with either (this is a nasty regression in jessie: bootstrapping a foreign-architecture system broke). This will take time, and step one is probably looking at bugs first -- I'm grateful for any help I can get here, primarily I'd like to hear about regressions via bug reports. tl;dr: as a distribution, we need both systemd and sysvinit. If you suggest that systemd can replace sysvinit, I will probably understand that you think that the systemd maintainers don't know project management. I'm reluctantly looking at becoming the official maintainer because I think it is important to keep around.

As I have a few hours of downtime on the train, I'm working on one of my favourite KiCad features again: the pin table. This silently appeared in the last KiCad release -- when you have a schematic component open in the editor, there is a new icon, currently on the right of the toolbar, that brings up a table of all pins. Currently, I have the pin number, name and type in the table (as well as the coordinates on the grid, but that's probably not that useful). For the release, this table is read-only, but can at least be useful for proofreading: left mouse click on a column header sorts (as would be expected), and right mouse click on a column header creates a grouping (i.e. all pins that have the same value in this column are collapsed into a single row). The latter is entirely unintuitive, but I have no idea how to show the user that this option exists at all. The good old DirectoryOpus on Amiga had a convention that buttons that had right-click functionality had an earmark to indicate that a "backside" exists, but that is a fairly obscure UI pattern, and would be hard to reproduce across all platforms. I could make the right click drop down a menu instead of immediately selecting this column for grouping -- at least this way it would be explained what is happening, but advanced users may get annoyed at the extra step. I could add a status bar with a hint, but that will always cost me screen space, and this only papers over the issue. I could add a menu with sorting/grouping options, which would allow novice and intermediate users to find the function, but that still does not hint at the shortcut, and the functions to set the sorting would have to be disabled on Linux because wxGTK cannot set the sort column from a program. For editing, another problem turns up in the collapsed rows: what should editing do when we are basically editing multiple pins at once. The most useful mapping would be:

• Editing the pin numbers adds and removes pins. The column is displayed and interpreted as a comma-separated list of pins, so going to edit mode and adding ",5" at the end is an obvious way to quickly create a pin with the number 5.
• Editing the pin name would ask for confirmation if not all the pins had the same name before, and then set the same name for all.
• Editing the pin type would do the same -- set the same type for all, possibly after confirmation.

However, I'd like to later on introduce multiple-function pins, which would add multiple (partially) exclusive pin names per pin number, which would change the semantics:

• As before, editing the pin number adds and removes pins.
• Editing the pin name adds and removes functions
• Having commas in both the pin name and pin number is legal if e.g. we've collapsed on pin type (so all the outputs are in one line), but editing either of them now will create a broken state.

At the moment, I'm mostly inclined to have separate modes for "grouped by pin number" and "grouped by pin name" -- these modes could be switched by the current grouping column, but that is neither intuitive, nor does it provide a full solution (because grouping by pin type is also supported). Time to look at the use cases:

• I want to quickly enter pins from tables. There are two major cases here:
  1. I have pins with the same function, e.g. Vcc or GND. I'd like to enter these as "1,5,17GNDp".
  2. I have pins with multiple configurations. I'd like to enter these via "3IO3,DIFFIO4+,CONF_DONEb,b,o".
• When I'm done entering data, I want a quick overview whether the data entered is complete.
• For existing components, I want to quickly check and correct data.

At 32C3, I've implemented a quick completeness check, consisting of a summary line that collapses the pin number column into a comma-separated list of ranges (where a range is defined as consecutive numbers in the last group of digits). I wonder if there is a group of developers with an UI focus who are happy about solving problems like these...

Day 3 was again a full of useful work. Since the beginning of the sprint, we were able to fix more than 50 FTBFS bugs, alongside general quality improvements in the packages.

in the Debian jargon, FTBFS means that a package fails to build from source , which in Debian is a critical bug because users need to be able to produce binary packages from their source code to fully exercise the free software principles.

An important milestone that was also achieved on day 3 was the upload of ruby-defaults 1:2.3.0+1, making ruby2.3 the new default version of Ruby. That is the version that will shipped in the next Debian release, codenamed stretch. This is the culmination of a joint effort between the Ruby team and Debian Release Team that involves rebuilding a little more than 130 packages that use the Ruby C API to make sure everything will just work on upgrades, both from the previous stable release, and from earlier snapshots of the current development release. Another small change that will have a big impact for Debian and for free software was an improvement to gem2deb that fixes a reproducibility issue in Ruby packages and will help currently more than 100 Ruby packages become reproducible. The full list of items that have been worked on is this:

• sponsor ruby-github-linguist. Closes: #815908. Thanks balasankar
• rebuild remaining failing packages 55 remaining!
• uploaded ruby-em-synchrony (FTBFS #800029)
• day 2 report blog post
• ruby-certificate-authority-0.1.6-1 (#816506)
• filed for ruby-grack removal
• uploaded ruby-em-synchrony
• uploaded ruby-mathml (FTBS #800029)
• uploaded new upstream version of ruby-redis which claims better ruby2.2 support
• uploaded pry (0.10.3-2), hoping it will build correctly on the buildd
• uploaded ruby-hashery
• rails security issues: CVE-2016-2098, CVE-2016-2097
  • fixed in unstable on 2:4.2.5.2-1
  • contacted security team for jessie update
• uploaded ruby-ruby-engine, ruby-ruby-version, ruby-rspec-pending-for to NEW
• updated ruby-hashie to new upstream version
• ruby-remcached (fix FTBFS)
• made the switch to ruby2.3 in unstable *********
• filed https://bugs.ruby-lang.org/issues/12137 as ruby-awesome-print triggers this
• uploaded ruby-rspec 3.4
• uploaded ruby-rspec-rails 3.4
• uploaded ruby-grack 2.0.2-2 (fix FTBFS)
• ruby-memcache-client (#797256) => RoM ruby-memcache-client (#816538) + https://github.com/rack/rack/issues/1025
• upload ruby-acts-as-lists (Closes: FTBS #816482)
• ruby-beautify: fixed and uploaded
• filed for ruby-foreigner removal (+ fixed diaspora dependency)
• uploaded newer ruby-origin to fix FTBFS
• uploaded ruby-responders
• gem2deb hacking
  • dh-make-ruby will not overwrite debian/copyright anymore
  • force generated gemspecs to use date from debian/changelog. fixes https://tests.reproducible-builds.org/issues/unstable/timestamps_in_gemspec_files_issue.html
  • uploaded version 0.27 to unstable
• make ruby-riddle tests run and work during build
• ruby-flexmock 2.0.4-1 -> uploaded
• RoM ruby-memcache-client (done #816538)
• upload nanoc 4.1.4 and fix tests for ruby2.3
• Upload new upstream version of ruby-test-after-commit
• uploaded newer upstream for ruby-origin fixing FTBFS bug

The day ended at Outback, where we had an amount of beer that led us to formulate what we will now call the One-Sided Dinner Booth Problem. In a party arranged like above, when the people closest to wall need to go alleviate themselves of some beer, you basically have to perform a removal from the bottom of a stack, which requires popping all the elements at the top. When they come back, you have to options:

• place them back at their original position, in which case
  • the people who stayed have to either stay standing or get up all over again (not sure what is worse)
  • the odds are that the next person to leave will require less effort from the others
• shift everyone to places against the wall and place the one coming back at the very end of the bench away from the wall, in which case
  • the people who stayed are perhaps less annoyed, but you have to rearrange plates and pints
  • it is less likely that the next time will require less effort from the others

The One-Sided Dinner Booth Problem is finding the optimal data structure and algorithm for this situation. It is postulated that this is an NP-complete problem, and that only probabilistic solutions are cost-effective.

I would like to give an overview of what is the current status of iptables & nftables in Debian.
Some information contained in this article is not new, this post is about what happened to iptables & nftables in the last 3 months, focusing on the integration with Debian.
versionsBy the time of this article, upstream versions are:

• iptables 1.6.0
• nftables 0.5
• libnftnl 1.0.5

iptables maintenanceready for stable?

I ve been using VULTR for a little while now and have been generally very pleased (especially with the very recent facelift the management portal received). I don t want to waste too much time talking about it, but the killer feature for me (over some of their competitors like DigitalOcean) is that I can provide a raw ISO and provision my VM directly using it as I would any local VM (which also means that once my VM is up and working, I get to use the OS s standard kernel, which is especially important for using Debian Unstable well). Anyhow, already too much about that let s get to the cool stuff. Getting right down to the beef, let s assume I ve got a VULTR instance already created, my OS is already installed and working, I ve enabled IPv6 within VULTR, ensured that my VM is able to ping6 google.com (to verify at least basic routability), and have Docker version 1.10.2 installed. For the sake of demonstration, we ll assume that VULTR has assigned my IPv6 as follows: (available under the VM details via Settings > IPv6)

• Default IP: 2001:db8::5400:00ff:fe20:2295
• Network: 2001:db8::
• CIDR: 64

(The astute reader may recognize RFC3849 here. ) The relevant documentation which helped me get to the working state outlined below is in the IPv6 with Docker section. The first step I took was creating a systemd drop-in file so that I could modify the daemon startup parameters (to include --ipv6 and --fixed-cidr-v6): I chose to use just /80 for Docker any other reasonable prefix (assuming it is routed to your host / host network) should also work; the documentation I linked above has an example using a /125, for example. With this half in place, I can systemctl daemon-reload and systemctl restart docker.service, and when I start a container it will be automatically assigned an IPv6 address from within that prefix. Excellent. An important caveat to note is that this will break discovery on our host due to Docker enabling forwarding for us, so (assuming your internet-facing interface is named ens3 for the sake of illustration; it might just as easily be eth0, eth1, enps3, lan0, wlan0, etc) I had to sysctl net.ipv6.conf.ens3.accept_ra=2, and I added it to /etc/sysctl.d/docker-ipv6.conf for good measure (so that I don t lose it after I reboot). The second half of our IPv6 to containers problem is routing. The nitty-gritty details of this are discussed in the Using NDP proxying section of the documentation, but the gist is that my containers have IPv6 addresses, but the outside world doesn t have a route that leads to them, and that we need to tell the kernel to respond to solicitations for our container s IPv6 addresses appropriately. The kernel has a mechanism for doing so (via ip -6 neigh ...), but it is limited to individual addresses and is thus not especially great for having a solution that works magically without further manual labor per-container. This is where ndppd (also packaged for Debian as ndppd) came in. After getting this configuration in place and restarting ndppd (systemctl restart ndppd), magic happened. My containers could ping6 google.com, and my other IPv6 hosts could ping6 the IPv6 addresses of my individual containers! You ve probably noted that this configuration isn t exactly secure, since it means that each of my individual containers has a publicly routable IPv6 address, but for this specific use case, I m OK with that! Update (2015-03-09): thanks to for correcting my systemd drop-in file usage!

Day 1 ended with dinner at a Yamato, my preferred Japanese restaurant in the city. Curitiba has a very large Japanese community, and lots of Japanese restaurants. Yamato, however, is the only one were you will stumble upon senior Japanese people, probably first or second generation immigrants, what I guess says something about its authenticity. Right after breaking for lunch, but before actually going out, we made what so far is official group photo (I might try again as the shot was not a really good one). Of course the most interesting part was the actual work that was done, and day 2 list is not less impressive than the day before:

• ruby-albino 1.3.3-4 (#813644) @sbadia
• #816256: ruby-versionomy: FTBFS: require : cannot load such file blockenspiel/unmixer_mri (LoadError)
• investiage ruby2.1 removal
• upload ruby-beautify
• upload ruby-aws-sdk
• made ruby-blockenspiel arch:all again
• filed RC bug against pcs depending on ruby2.1-dev
• ruby-dev in experimental now supports pkg-config ruby so packages can use pkg-config to build against the current default Ruby
• ruby-libxml (some tests skipped)
• fixed FTBFS #804794 in subtle delaying ruby2.1 removal (now in DELAYED/2)
• mailed debian-ruby and ask people to test ruby2.3 as default
• #816162: ruby-zentest: FTBFS: Failure: TestZenTest#test_testcase9
• ruby-sshkit 1.9.0~rc1-1 (795118)
  • capistrano 3.4.0-1 + ruby-sshkit 1.9.0~rc1-1 => http://paste.debian.net/plain/410699
• filed for ruby-patron removal
• updated and uploaded ruby-pygments.rb updated + patch for tests (fix #812940)
• discussed dependency cycle resolution
  • will keep cycle between ruby2.3 and rake (as they really need each other)
  • will break cycle from pure ruby packages to ruby when they are depended on by ruby2.3
  • for same package set, enable Multi-Arch: foreign when needed
• uploaded ruby-power-assert (M-A foreign, break ruby cycle)
• uploaded ruby-did-you-mean (M-A foreign, break ruby cycle)
• uploaded ruby-minitest (M-A foreign, break ruby cycle)
• uploaded ruby-test-unit (M-A foreign, break ruby cycle)
• uploaded rake (M-A foreign)
• uploaded ruby-net-telnet (M-A foreign)
• discussed removal of libruby metapackage, filed bug against dh-make-drupal (only rdep) #816417
• uploaded yard (Closes: #812809)
• rerun failed builds
• closed #798539 (not applicable on current version of ruby-webmock)
• closed #816321 fixed by ruby2.3 upload
• uploaded ruby-cliver (by @lucasmoura)
• uploaded ruby-celluloid-supervision ( closes: #810770)
• uploaded ruby-celluloid-fsm ( closes: #815107)
• uploaded ruby-raindrops
• investigated status of dhelp, ruby-mysql again
• ruby-http-form-data (1.0.1+gemwatch-1) @sbadia
• nmu ohcount (DELAYED/2)
• updated trocla to newest upstream, fixing #816257
• reassign #816358 from ruby-safe-yaml to ruby2.3
• close #816120 (transient bug in ruby-default-value-for, waiting for ruby-sqlite3 binNMU)
• upload updated ruby-rc4
• uploaded new upstream version of pry, fixing FTBFS with ruby2.3
• ruby-rb-inotify , required to new ruby-listen( need ruby-rb-inotify >= 0.9.7)
• applied patch from upstream to ruby2.3 for #816358
• uploaded new upstream version for coderay supporting ruby2.3

On Monday C dric told us that he and Sebastien had brought a bottle of French wine and some smelly French cheeses, and suggested that in the best Debian tradition we should have a Mini Cheese and Wine Party . Sure thing! Luckily there is a farmer s market 2 blocks from home on Tuesdays mornings, where I usually buy my fruits, vegetables, and cheese & friends, so the timing was perfect. I went shopping early in the morning, and bought a few things, and was back before it was the time to go to UTFPR. After the day-long hacking session we stopped by another store nearby to buy a few extra bottles of wine and other snacks. At night, in my place, I ended up playing cheese master. There was enough food that at the end we were all very full. And with the spokesperson task of the day done, off to hacking I am!

Recently I have been very hopeful about the 96boards Hikey SBC and as Evan Esar predicted I have therefore been very disappointed. I was given a Hikey after a Linaro connect event some time ago by another developer who could not get the system working usefully and this is the tale of what followed.
The Standard DesignThis board design was presented as Linaro creating a standard for the 64bit Single Board Computer (SBC) market. So I had expected that a project with such lofty goals would have considered many factors and provided at least as good a solution as the existing 32bit boards.

The lamentable hubris of creating a completely new form factor unfortunately sets a pattern for the whole enterprise. Given the aim towards "makers" I would have accepted that the system would not be a ATX PC size motherboard, but mini/micro/nano and pico ITX have been available for several years.

If opting for a smaller "credit card" form factor why not use one of the common ones that have already been defined by systems such as the Raspberry Pi B+? Instead now every 96Board requires special cases and different expansion boards.

Not content with defining their own form factor the design also uses a 8-18V supply, this is the only SBC I own that is not fed from a 5V supply. I understand that a system might require more current than a micro USB connector can provide, but for example the Banana Pi manages with a DC barrel jack readily capable of delivering 25W which would seem more than enough

The new form factor forced the I/O connectors to be placed differently to other SBC, given the opportunity to concentrate all connectors on one edge (like ATX designs) and avoid the issues where two or three sides are used. The 96board design instead puts connectors on two edges removing any possible benefit this might have given.

The actual I/O connectors specified are rather strange. There is a mandate for HDMI removing the possibility of future display technology changes. The odd USB arrangement of two single sockets instead of a stacked seems to be an attempt to keep height down but the expansion headers and CPU heatsink mean this is largely moot.

The biggest issue though is mandating WIFI but not Ethernet (even as an option), everything else in the design I could logically understand but this makes no sense. It means the design is not useful for many applications without adding USB devices.

Expansion is presented as a 2mm pitch DIL socket for "low speed" signals and a high density connector for "high speed" signals. The positioning and arrangement of these connectors proffered an opportunity to improve upon previous SBC designs which was not taken. The use of 2mm pitch and low voltage signals instead of the more traditional 2.54mm pitch 3.3v signals means that most maker type applications will need adapting from the popular Raspberry Pi and Arduino style designs.

In summary the design appears to have been a Linaro project to favour one of their members which took a Hisilicon Android phone reference design and put it onto a board with no actual thought beyond getting it done. Then afterwards attempted to turn that into a specification, this has simply not worked as an approach.

My personal opinion is that this specification is fatally flawed and, is a direct cause of, the bizarre situation where the "consumer" specification exists alongside the "enterprise" edition which itself has an option of microATX form factor anyhow!
The ImplementationIf we ignore the specification appearing to be nothing more than a codification of the original HiKey design we can look at the HiKey as an implementation.

Initially the board required modifying to add headers to attach a USB to 1.8V LVTTL serial adaptor on the UART0 serial port. Once Andy Simpkins had made this change for me I was able to work through the instructions and attempt to install a bootloader and OS image.

The initial software was essentially HiSilicon vendor code using the Android fastboot system to configure booting. There was no source and the Ubuntu OS images were an obvious afterthought to the Android images. Just getting these images installed required a great deal of effort, repetition and debugging. It was such a dreadful experience this signalled the commencement one of the repeated hiatuses throughout this project, the allure of 64 bit ARM computing has its limits even for me.

When I returned to the project I attempted to use the system from the on-board eMMC but the pre-built binary only kernel and OS image were very limited. Building a replacement kernel , or even modules for the existing one proved fruitless and the system was dreadfully unstable.

I wanted to use the system as a builder for some Open Source projects but the system instability ruled this out. I considered attempting to use virtualisation which would also give better system isolation for builder systems. By using KVM running a modern host kernel and OS as a guest this would also avoid issues with the host systems limitations. At which point I discovered the system had no virtualisation enabled apparently because the bootloader lacked support.

In addition to these software issues there were hardware problems, despite forcing the use of USB for all additional connectivity the USB implementation was dreadful. For a start all USB peripherals have to run at the same speed! One cannot mix full (12Mbit) and high speed (480Mbit) devices which makes adding a USB Ethernet and SATA device challenging when you cannot use a keyboard.

And because I needed more challenges only one of the USB root hubs was functional. In effect this made the console serial port critical as it was the only reliable way to reconfigure the system without a keyboard or network link (and WIFI was not reliable either)

After another long pause in proceedings I decided that I should house all the components together and that perhaps being out on my bench might be the cause of some instability. I purchased a powered Amazon basics USB 2 hub, an Ethernet adaptor and a USB 2 SATA interface in the hope of accessing some larger mass storage.

The USB hub power supply was 12V DC which matched the Hikey requirements so I worked out I could use a single 4A capable supply and run a 3.5inch SATA hard drive too. I designed a laser cut enclosure and mounted all the components. As it turned out I only had a 2.5inch hard drive handy so the enclosure is a little over size. If I were redoing this design I would attempt to make it fit in 1U of height and be mountable in a 19inch rack instead it is an 83mm high (under 2U) box.

A new software release had also become available which purported to use an UEFI bootloader after struggling to install this version unsuccessfully, made somewhat more problematic by the undocumented change from UART0 (unpopulated header) to UART3 on the low speed 2mm pitch header. The system seemed to start the kernel which panicked and hung either booting from eMMC or SD card. Once again the project went on hold after spending tens of hours trying to make progress.
Third time's a charmAs the year rolled to a close I once again was persuaded to look at the hikey, I followed the much improved instructions and installed the shiny new November software release which appears to have been made for the re-release of the Hikey through LeMaker. This time I obtained a Debian "jessie" system that booted from the eMMC.

Having a booted system meant I could finally try and use it. I had decided to try and use the system to host virtual machines used as builders within the NetSurf CI system.

The basic OS uses a mixture of normal Debian packages with some replacements from Linaro repositories. I would have prefered to see more use of Debain packages even if they were from the backports repositories but on the positive side it is good to see the use of Debian instead of Ubuntu.

The kernel is a heavily patched 3.18 built in a predominantly monolithic (without modules) manner with the usual exceptions such as the mali and wifi drivers (both of which appear to be binary blobs). The use of a non-mainline capable SoC means the standard generic distribution kernels cannot be used and unless Linaro choose to distribute a kernel with the feature built in it is necessary to compile your own from sources.

The default install has a linaro user which I renamed to my user and ensured all the ssh keys and passwords on the system were changed. This is an important step when using this pre-supplied images as often a booted system is identical to every other copy.

To access mass storage my only option was via USB, indeed to add any additional expansion that is the only choice. The first issue here is that the USB host support is compiled in so when the host ports are initialised it is not possible to select a speed other than 12MBit. The speed is changed to 480Mbit by using a user space application found in the users home directory (why this is not a tool provided by a package and held in sbin I do not know).

When the usb_speed tool is run there is a chance that the previously enumerated devices will be rescanned and what was /dev/sda has become /dev/sdb if this happens there is a high probability that the system must be rebooted to prevent random crashes due to the "zombie" device.

Because the speed change operation is unreliable it cannot be reliably placed in the boot sequence so this must be executed by hand on each boot to get access to the mass storage.

NetSurf project already uses a x86_64 virtual host system which runs an LLVM physical volume from which we allocate logical volumes for each VM. I initially hoped to do this with the hikey but as soon as I tried to use the logical volume with a VM the system locked up with nothing shown on console. I did not really try very hard to discover why and instead simply used files on disc for virtual drives which seemed to work.

To provide reliable network access I used a USB attached Ethernet device, this like the mass storage suffered from unreliable enumeration and for similar reasons could not be automated requiring manually using the serial console to start the system.

Once the system was started I needed to install the guest VM. I had hoped I might be able to install locally from Debian install media as I do for x86 using the libvirt tools. After a great deal of trial and error I finally was forced to abandon this approach when I discovered the Linaro kernel is lacking iso9660 support so installing from standard media was not possible.

Instead I used the instructions provided by Leif Lindholm to create a virtual machine image on my PC and copied the result across. These instructions are great except I used version 2.5 of Qemu instead of 2.2 which had no negative effect. I also installed the Debian backports for Jessie to get an up to date 4.3 kernel.

After copying the image to the Hikey I started it by hand from the command line as a four core virtual machine and was successfully able to log in. The guest would operate for up to a day before stopping with output such as

$
Message from syslogd@ciworker13 at Jan 29 07:45:28 ...
 kernel:[68903.702501] BUG: soft lockup - CPU#0 stuck for 27s! [mv:24089]

Message from syslogd@ciworker13 at Jan 29 07:45:28 ...

 kernel:[68976.958028] BUG: soft lockup - CPU#2 stuck for 74s! [swapper/2:0]

Message from syslogd@ciworker13 at Jan 29 07:47:39 ...
 kernel:[69103.199724] BUG: soft lockup - CPU#3 stuck for 99s! [swapper/3:0]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69140.321145] BUG: soft lockup - CPU#3 stuck for 30s! [rs:main Q:Reg:505]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69192.880804] BUG: soft lockup - CPU#0 stuck for 21s! [jbd2/vda3-8:107]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69444.805235] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0]

Message from syslogd@ciworker13 at Jan 29 07:55:21 ...
 kernel:[69570.177600] BUG: soft lockup - CPU#1 stuck for 112s! [systemd:1]

Timeout, server 192.168.7.211 not responding.

After this output the host system would not respond and had to be power cycled never mind the guest!

Once I changed to single core operation the system would run for some time until the host suffered from the dreaded kernel OOM killer. I was at a loss as to why the oom killer was running as the VM was only allocated half the physical memory (512MB) allowing the host what I presumed to be an adequate amount.

By adding a 512MB swapfile the system was able to push the few hundred kilobytes it wanted to swap and the system was now stable! The swapfile of course has to be started by hand as the external storage is unreliable and unavailable at boot.

I converted the qemu command line to a libvirt config using the virsh tool

virsh domxml-from-native qemu-argv cmdln.args

The converted configuration required manual editing to get a working system but now I have a libvirt based VM guest I can control along with all my other VM using the virt-manager tool.

This system is now stable and has been in production use for a month at time of writing. The one guest VM is a single core 512MB aarch64 system which takes over 1100 seconds (19 minutes) to do what a Banana Pi 2 dual core 1GB memory 32bit native ARM system manages in 300 seconds.

It seems the single core limited memory system with USB SATA attached storage is very, very slow.

I briefly attempted to run the CI system job natively within the host system but within minutes it crashed hard and required a power cycle to retrieve, it had also broken the UEFI boot. I must thank Leif for walking me through recovering the system otherwise I would have needed to start over.
ConclusionsI must stress these conclusions and observations are my own and do not represent anyone else.

My main conclusions are:

• My experience is of a poorly conceived, designed and implemented product rushed to market before it was ready.
• It was the first announced 64bit ARM single board computer to market but that lead was squandered with issues around availability, reliability and software.
• Value for money appears poor. Product is 70 plus an additional 50 for USB hubs, power supplies, USB Ethernet and USB SATA. Other comparable SBC are around the 30 mark and require fewer extras.
• The limited I/O within the core product yields a heavy reliance on USB
• The USB system is poorly implemented resulting in large additional administrative burdens.
• Limited memory of 1Gigabyte reduces the scope for making use of the benefits of 64bit processors.
• Recent change to UEFI bootloader is very welcome and something I would like to see across all aarch64 platforms. This is the time for there to be a single unified boot method, having to deal with multiple bad copies of bootloaders in the 32bit world was painful, perhaps this mistake can be avoided in the future.
• The kernel provision is abysmal and nothing like the quality I would expect from Linaro. The non-upstream kernel combined with missing features after almost a year of development is inexplicable.
• The Pine64 with 2G of memory and the Raspberry Pi 3 with its de-facto standard form factor are both preferable despite their limitations in other areas.

I decided to rethink the state of my personal site, and try out some of the new static site generators that are available now. To do that, I jotted down a series of things that I want in a static site generator, then wrote a tool to convert my ikiwiki site to other formats, and set out to evaluate things. As a benchmark I did a full rebuild of my site, which currently contains 1164 static files and 458 markdown pages. My requirements Free layout for my site My / is mostly a high-level index to the site contents. Blog posts are at /blog. My talk archive is organised like a separate blog at /talks. I want the freedom to create other sections of the site, each with its own rss feed, located wherever I want in the site hierarchy. Assets next to posts I occasionally blog just a photo with a little comment, and I would like the .md page with the comment to live next to the image in the file system. I did not even know that I had this as a requirement until I found static site generators that mandated a completely different directory structure for markdown contents and for static assets. Multiple RSS/Atom feeds I want at least one RSS/Atom feed per tag, because I use tags for marking which articles go to http://planet.debian.org. I also want RSS/Atom feeds for specific parts of the site, like the blog and talks. Arbitrary contents in /index.html I want total control over the contents of the main home page of the site. Quick preview while editing contents I do not like to wait several seconds for the site to be rebuilt at every review iteration of the pages I write. This makes me feel like the task of editing is harder than it should, and makes me lose motivation to post. Reasonable time for a full site rebuild I want to be able to run a full rebuild of the site in a reasonable time. I could define "reasonable" in this case as how long I can stare at the screen without getting bored, starting to do something else, and forgetting what it was that I was doing with the site. It is ok if a rebuild takes something like 10 or 30 seconds. It is not ok if it takes minutes. Code and dependency ecosystems that I can work with I can deal with Python and Go. I cannot deal with Ruby or JavaScript. I forgot all about Perl. Also, if it isn't in Debian it does not exist. Decent themes out of the box One of my hopes in switching to a more mainstream generator is to pick and choose themes and easily give my site a more modern look. Hugo Hugo is written in Go and is in Debian testing. Full rebuild time for my site is acceptable, and it can even parallelize: Free layout for my site was hard to get. I could replace /index.html by editing the template page for it, but then I did not find out how to create another similar index in an arbitrary place. Also, archetypes are applied only on the first path component of new posts, but I would like them instead to be matched on the last path component first, and failing that traveling up to the path until the top. This should be easy to fix by reorganizing the content a bit around here For example, a path for a new blog post of mine could be blog/2016/debian/ and I would like it to match the debian archetype first, and failing that the blog archetype. Assets next to posts almost work. Hugo automatically generates one feed per taxonomy element, and one feed per section. This would be currently sufficient for me, although I don't like the idea that sections map 1 to 1 to toplevel directories in the site structure. Hugo has a server that watches the file system and rerenders pages as they are modified, so the quick preview while editing works fine. About themes, it took me several tries to find a theme that would render navigation elements for both sections and tags, and most themes would render by pages with white components all around, and expect me to somehow dig in and tweak them. That frustrated me, because for quite a while I could not tell if I had misconfigured Hugo's taxonomies or if the theme was just somehow incomplete. Nikola Nikola is written in Python and is in Debian testing. Full rebuild time for my site is almost two orders of magnitude more than Hugo, and I am miffed to find the phrases "Nikola is fast." or "Fast building process" in its front page and package description: Free layout could be achieved fiddling with the site configuration to tell it where to read sources. Assets next to post work after tweaking the configuration, but they require to write inconsistent links in the markdown source: https://github.com/getnikola/nikola/issues/2266 I have a hard time accepting that that, because I want to author content with consistent semantic interlinking, because I want to be able 10 years from now to parse it and convert it to something else if a new technology comes out. Nikola generates one RSS/Atom feed per tag just fine. I have not tried generating feeds for different sections of the site. Incremental generation inside its built in server works fine. Pelican Pelican is written in Python and is in Debian testing. Full rebuild time for my site is acceptable: By default, pelican seems to put generate a single flat directory of html files regardless of the directory hierarchy of the sources. To have free layout, pelican needs some convincing in the configuration: but even if I do that, the urls that it generates still point to just slug /index.html and I have not trivially found a configuration option to fix that accordingly. I got quite uncomfortable at the idea of needing to configure content generation and linking to match, instead of having one automatically being in sync with the other. Having assets next to posts seems to be possible (also setting STATIC_PATHS = ["."]), but I do not recall making progress on this front. I did not manage to generate a feed for each tag out of the box, and probably there is some knob in the configuration for it. I gave up with Pelican as trying it out felt like a constant process of hacking the configuration from defaults that do not make any sense for me, withouth even knowing if a configuration exists that would do what I need Ikiwiki Ikiwiki is written in Perl and is in Debian. Although I am not anymore proficient with Perl, I was already using it, so it was worth considering. Full rebuild time feels a bit on the slow side but is still acceptable: In terms of free site structure, all feeds for all or part of the site, ikiwiki just excels. I even considered writing a python web server that monitors the file system and calls ikiwiki --refresh when anything changes, and calling it a day. However, when I tried to re-theme my website around a simple bootstrap boilerplate, I found that to be hard, as a some of the HTML structure is hardcoded in Perl (and it's also my fault) and there is only so much that can be done by tweaking the (rather unreadable) templates. siterefactor During all these experiments I had built siterefactor to generate contents for all those static site engines, and it was going through all the contents quite fast: So I wondered how slow it would become if, instead of making it write markdown, I made it write HTML via python markdown and Jinja2: I then started wondering how slower it would become if I implemented postprocessing of all local URLs generated by Markdown to make sure they are kept consistent even if the path of a generated page is different than the path of its source. Not much slower, really. I then added taxonomies. And arbitrary Jinja2 templates in the input, able to generate page lists and RSS/Atom feeds. And theming. And realised that reading all the sources and cross-linking them took 0.2 seconds, and the rest was generation time. And that after cross-linking, each page can be generated independently from all the others. staticsite So my site is now generated with staticsite: It's comparable with Hugo, and on a single process.

Most of the time when you are using lambdas, you are not even doing functional programming, because you often are violating one key rule of functional programming: no side effects.

For example:

collection.forEach(System.out::println);

is of course very cute to use, and is (wow) 10 characters shorter than:

for (Object o : collection) System.out.println(o);

but this is not functional programming because it has side effects.

What you are doing are anonymous methods/objects, using a shorthand notion. It's sometimes convenient, it is usually short, and unfortunately often unreadable, once you start cramming complex problems into this framework.

It does not offer efficiency improvements, unless you have the propery of side-effect freeness (and a language compiler that can exploit this, or parallelism that can then call the function concurrently in arbitrary order and still yield the same result).

Here is an examples of how to not use lambdas:
DZone Java 8 Factorial (with boilerplate such as the Pair class omitted):

Stream<Pair> allFactorials = Stream.iterate(
  new Pair(BigInteger.ONE, BigInteger.ONE),
  x -> new Pair(
    x.num.add(BigInteger.ONE),
    x.value.multiply(x.num.add(BigInteger.ONE))));
return allFactorials.filter(
  (x) -> x.num.equals(num)).findAny().get().value;

When you are fresh out of the functional programming class, this may seem like a good idea to you... (and in contrast to the examples mentioned above, this is really a functional program).
But such code is a pain to read, and will not scale well either. Rewriting this to classic Java yields:

BigInteger cur = BigInteger.ONE, acc = BigInteger.ONE;
while(cur.compareTo(num) <= 0)  
  cur = cur.add(BigInteger.ONE); // Unfortunately, BigInteger is immutable!
  acc = acc.multiply(cur);
 
return acc;

Sorry, but the traditional loop is much more readable. It will still not perform very well (because of BigInteger not being designed for efficiency - it does not even make sense to allow BigInteger for num - the factorial of 2**63-1, the maximum of a Java long, needs 10²⁰ bytes to store, i.e. about 500 exabyte.

For some, I did some benchmarking. One hundred random values num (of course the same for all methods) from the range 1 to 1000.

I also included this even more traditional version:

BigInteger acc = BigInteger.ONE;
for(long i = 2; i <=x; i++)  
  acc = acc.multiply(BigInteger.valueOf(i));
 
return acc;

Here are the results (Microbenchmark, using JMH, 10 warum iterations, 20 measurement iterations of 1 second each):

functional    1000     100  avgt   20  9748276,035   222981,283  ns/op
biginteger    1000     100  avgt   20  7920254,491   247454,534  ns/op
traditional   1000     100  avgt   20  6360620,309   135236,735  ns/op

As you can see, this "functional" approach above is about 50% slower than the classic for-loop. This will be mostly due to the Pair and additional BigInteger objects created and garbage collected.

Apart from being substantially faster, the iterative approach is also much simpler to follow. (To some extend it is faster because it is also easier for the compiler!)

There was a recent blog post by Robert Br utigam that discussed exception throwing in Java lambdas and the pitfalls associated with this. The discussed approach involves abusing generics for throwing unknown checked exceptions in the lambdas, ouch.

---

Don't get me wrong. There are cases where the use of lambdas is perfectly reasonable. There are also cases where it adheres to the "functional programming" principle. For example, a stream.filter(x -> x.name.equals("John Doe")) can be a readable shorthand when selecting or preprocessing data. If it is really functional (side-effect free), then it can safely be run in parallel and give you some speedup.

Also, Java lambdas were carefully designed, and the hotspot VM tries hard to optimize them. That is why Java lambdas are not closures - that would be much less performant. Also, the stack traces of Java lambdas remain somewhat readable (although still much worse than those of traditional code). This blog post by Takipi showcases how bad the stacktraces become (in the Java example, the stream function is more to blame than the actual lambda - nevertheless, the actual lambda application shows up as the cryptic LmbdaMain$$Lambda$1/821270929.apply(Unknown Source) without line number information). Java 8 added new bytecodes to be able to optimize Lambdas better - earlier JVM-based languages may not yet make good use of this.

But you really should use lambdas only for one-liners. If it is a more complex method, you should give it a name to encourage reuse and improve debugging.

Beware of the cost of .boxed() streams!

And do not overuse lambdas. Most often, non-Lambda code is just as compact, and much more readable. Similar to foreach-loops, you do lose some flexibility compared to the "raw" APIs such as Iterators:

for(Iterator<Something>> it = collection.iterator(); it.hasNext(); )  
  Something s = it.next();
  if (someTest(s)) continue; // Skip
  if (otherTest(s)) it.remove(); // Remove
  if (thirdTest(s)) process(s); // Call-out to a complex function
  if (fourthTest(s)) break; // Stop early
 

In many cases, this code is preferrable to the lambda hacks we see pop up everywhere these days. Above code is efficient, and readable.
If you can solve it with a for loop, use a for loop!

Code quality is not measured by how much functionality you can do without typing a semicolon or a newline!

On the contrary: the key ingredient to writing high-performance code is the memory layout (usually) - something you need to do low-level.

Instead of going crazy about Lambdas, I'm more looking forward to real value types (similar to a struct in C, reference-free objects) maybe in Java 9 (Project Valhalla), as they will allow reducing the memory impact for many scenarios considerably. I'd prefer a mutable design, however - I understand why this is proposed, but the uses cases I have in mind become much less elegant when having to overwrite instead of modify all the time.