I would like to give an overview of what is the current status of iptables & nftables in Debian.
Some information contained in this article is not new, this post is about what happened to iptables & nftables in the last 3 months, focusing on the integration with Debian.
versionsBy the time of this article, upstream versions are:

• iptables 1.6.0
• nftables 0.5
• libnftnl 1.0.5

iptables maintenanceready for stable?

February marked the end of Debian 6.0 "squeeze" LTS, though there was some initial confusion (partly my fault) over which day of month it should end. I carried over 10 hours from January and was assigned another 11.25 hours of work by Freexian's Debian LTS initiative. I worked a total of 14 hours. I started the month with "the final" update to the linux-2.6 kernel package (DLA-412-1), fixing 5 CVEs and incorporating upstream stable update 2.6.32.70. After it was agreed that support would continue to the end of the month, I made one more update on the last day (DLA-439-1), fixing 4 CVEs, a regression, and some issues I had found lacking CVEs (one is now CVE-2012-6701). I helped to edit the end-of-life announcement for Squeeze LTS. I spent a fairly quiet week in the front desk, triaging new issues for squeeze. As I've volunteered to handle embargoed issues, I prepared security updates to libssh and libssh2 and issued them minutes after the public disclosure of the similarly disastrous CVE-2016-0739 and CVE-2016-0787 issues.

A good week after the update to Eigen 3.2.8 in our RcppEigen package 0.3.2.8.0, we have another (local) update with RcppEigen 0.3.2.8.1 which should fix another UBSAN issue. This new version in now on CRAN and in Debian. Big thanks once again to Yixuan Qiu for liasing with Eigen upstream, reporting the bug and preparing a PR with the fix. The NEWS file entry follows.

Changes in RcppEigen version 0.3.2.8.1 (2016-02-29)

• Applied another upstream UBSAN fix (PR #30 by Yixuan)

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Recently I have been very hopeful about the 96boards Hikey SBC and as Evan Esar predicted I have therefore been very disappointed. I was given a Hikey after a Linaro connect event some time ago by another developer who could not get the system working usefully and this is the tale of what followed.
The Standard DesignThis board design was presented as Linaro creating a standard for the 64bit Single Board Computer (SBC) market. So I had expected that a project with such lofty goals would have considered many factors and provided at least as good a solution as the existing 32bit boards.

The lamentable hubris of creating a completely new form factor unfortunately sets a pattern for the whole enterprise. Given the aim towards "makers" I would have accepted that the system would not be a ATX PC size motherboard, but mini/micro/nano and pico ITX have been available for several years.

If opting for a smaller "credit card" form factor why not use one of the common ones that have already been defined by systems such as the Raspberry Pi B+? Instead now every 96Board requires special cases and different expansion boards.

Not content with defining their own form factor the design also uses a 8-18V supply, this is the only SBC I own that is not fed from a 5V supply. I understand that a system might require more current than a micro USB connector can provide, but for example the Banana Pi manages with a DC barrel jack readily capable of delivering 25W which would seem more than enough

The new form factor forced the I/O connectors to be placed differently to other SBC, given the opportunity to concentrate all connectors on one edge (like ATX designs) and avoid the issues where two or three sides are used. The 96board design instead puts connectors on two edges removing any possible benefit this might have given.

The actual I/O connectors specified are rather strange. There is a mandate for HDMI removing the possibility of future display technology changes. The odd USB arrangement of two single sockets instead of a stacked seems to be an attempt to keep height down but the expansion headers and CPU heatsink mean this is largely moot.

The biggest issue though is mandating WIFI but not Ethernet (even as an option), everything else in the design I could logically understand but this makes no sense. It means the design is not useful for many applications without adding USB devices.

Expansion is presented as a 2mm pitch DIL socket for "low speed" signals and a high density connector for "high speed" signals. The positioning and arrangement of these connectors proffered an opportunity to improve upon previous SBC designs which was not taken. The use of 2mm pitch and low voltage signals instead of the more traditional 2.54mm pitch 3.3v signals means that most maker type applications will need adapting from the popular Raspberry Pi and Arduino style designs.

In summary the design appears to have been a Linaro project to favour one of their members which took a Hisilicon Android phone reference design and put it onto a board with no actual thought beyond getting it done. Then afterwards attempted to turn that into a specification, this has simply not worked as an approach.

My personal opinion is that this specification is fatally flawed and, is a direct cause of, the bizarre situation where the "consumer" specification exists alongside the "enterprise" edition which itself has an option of microATX form factor anyhow!
The ImplementationIf we ignore the specification appearing to be nothing more than a codification of the original HiKey design we can look at the HiKey as an implementation.

Initially the board required modifying to add headers to attach a USB to 1.8V LVTTL serial adaptor on the UART0 serial port. Once Andy Simpkins had made this change for me I was able to work through the instructions and attempt to install a bootloader and OS image.

The initial software was essentially HiSilicon vendor code using the Android fastboot system to configure booting. There was no source and the Ubuntu OS images were an obvious afterthought to the Android images. Just getting these images installed required a great deal of effort, repetition and debugging. It was such a dreadful experience this signalled the commencement one of the repeated hiatuses throughout this project, the allure of 64 bit ARM computing has its limits even for me.

When I returned to the project I attempted to use the system from the on-board eMMC but the pre-built binary only kernel and OS image were very limited. Building a replacement kernel , or even modules for the existing one proved fruitless and the system was dreadfully unstable.

I wanted to use the system as a builder for some Open Source projects but the system instability ruled this out. I considered attempting to use virtualisation which would also give better system isolation for builder systems. By using KVM running a modern host kernel and OS as a guest this would also avoid issues with the host systems limitations. At which point I discovered the system had no virtualisation enabled apparently because the bootloader lacked support.

In addition to these software issues there were hardware problems, despite forcing the use of USB for all additional connectivity the USB implementation was dreadful. For a start all USB peripherals have to run at the same speed! One cannot mix full (12Mbit) and high speed (480Mbit) devices which makes adding a USB Ethernet and SATA device challenging when you cannot use a keyboard.

And because I needed more challenges only one of the USB root hubs was functional. In effect this made the console serial port critical as it was the only reliable way to reconfigure the system without a keyboard or network link (and WIFI was not reliable either)

After another long pause in proceedings I decided that I should house all the components together and that perhaps being out on my bench might be the cause of some instability. I purchased a powered Amazon basics USB 2 hub, an Ethernet adaptor and a USB 2 SATA interface in the hope of accessing some larger mass storage.

The USB hub power supply was 12V DC which matched the Hikey requirements so I worked out I could use a single 4A capable supply and run a 3.5inch SATA hard drive too. I designed a laser cut enclosure and mounted all the components. As it turned out I only had a 2.5inch hard drive handy so the enclosure is a little over size. If I were redoing this design I would attempt to make it fit in 1U of height and be mountable in a 19inch rack instead it is an 83mm high (under 2U) box.

A new software release had also become available which purported to use an UEFI bootloader after struggling to install this version unsuccessfully, made somewhat more problematic by the undocumented change from UART0 (unpopulated header) to UART3 on the low speed 2mm pitch header. The system seemed to start the kernel which panicked and hung either booting from eMMC or SD card. Once again the project went on hold after spending tens of hours trying to make progress.
Third time's a charmAs the year rolled to a close I once again was persuaded to look at the hikey, I followed the much improved instructions and installed the shiny new November software release which appears to have been made for the re-release of the Hikey through LeMaker. This time I obtained a Debian "jessie" system that booted from the eMMC.

Having a booted system meant I could finally try and use it. I had decided to try and use the system to host virtual machines used as builders within the NetSurf CI system.

The basic OS uses a mixture of normal Debian packages with some replacements from Linaro repositories. I would have prefered to see more use of Debain packages even if they were from the backports repositories but on the positive side it is good to see the use of Debian instead of Ubuntu.

The kernel is a heavily patched 3.18 built in a predominantly monolithic (without modules) manner with the usual exceptions such as the mali and wifi drivers (both of which appear to be binary blobs). The use of a non-mainline capable SoC means the standard generic distribution kernels cannot be used and unless Linaro choose to distribute a kernel with the feature built in it is necessary to compile your own from sources.

The default install has a linaro user which I renamed to my user and ensured all the ssh keys and passwords on the system were changed. This is an important step when using this pre-supplied images as often a booted system is identical to every other copy.

To access mass storage my only option was via USB, indeed to add any additional expansion that is the only choice. The first issue here is that the USB host support is compiled in so when the host ports are initialised it is not possible to select a speed other than 12MBit. The speed is changed to 480Mbit by using a user space application found in the users home directory (why this is not a tool provided by a package and held in sbin I do not know).

When the usb_speed tool is run there is a chance that the previously enumerated devices will be rescanned and what was /dev/sda has become /dev/sdb if this happens there is a high probability that the system must be rebooted to prevent random crashes due to the "zombie" device.

Because the speed change operation is unreliable it cannot be reliably placed in the boot sequence so this must be executed by hand on each boot to get access to the mass storage.

NetSurf project already uses a x86_64 virtual host system which runs an LLVM physical volume from which we allocate logical volumes for each VM. I initially hoped to do this with the hikey but as soon as I tried to use the logical volume with a VM the system locked up with nothing shown on console. I did not really try very hard to discover why and instead simply used files on disc for virtual drives which seemed to work.

To provide reliable network access I used a USB attached Ethernet device, this like the mass storage suffered from unreliable enumeration and for similar reasons could not be automated requiring manually using the serial console to start the system.

Once the system was started I needed to install the guest VM. I had hoped I might be able to install locally from Debian install media as I do for x86 using the libvirt tools. After a great deal of trial and error I finally was forced to abandon this approach when I discovered the Linaro kernel is lacking iso9660 support so installing from standard media was not possible.

Instead I used the instructions provided by Leif Lindholm to create a virtual machine image on my PC and copied the result across. These instructions are great except I used version 2.5 of Qemu instead of 2.2 which had no negative effect. I also installed the Debian backports for Jessie to get an up to date 4.3 kernel.

After copying the image to the Hikey I started it by hand from the command line as a four core virtual machine and was successfully able to log in. The guest would operate for up to a day before stopping with output such as

$
Message from syslogd@ciworker13 at Jan 29 07:45:28 ...
 kernel:[68903.702501] BUG: soft lockup - CPU#0 stuck for 27s! [mv:24089]

Message from syslogd@ciworker13 at Jan 29 07:45:28 ...

 kernel:[68976.958028] BUG: soft lockup - CPU#2 stuck for 74s! [swapper/2:0]

Message from syslogd@ciworker13 at Jan 29 07:47:39 ...
 kernel:[69103.199724] BUG: soft lockup - CPU#3 stuck for 99s! [swapper/3:0]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69140.321145] BUG: soft lockup - CPU#3 stuck for 30s! [rs:main Q:Reg:505]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69192.880804] BUG: soft lockup - CPU#0 stuck for 21s! [jbd2/vda3-8:107]

Message from syslogd@ciworker13 at Jan 29 07:53:21 ...
 kernel:[69444.805235] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0]

Message from syslogd@ciworker13 at Jan 29 07:55:21 ...
 kernel:[69570.177600] BUG: soft lockup - CPU#1 stuck for 112s! [systemd:1]

Timeout, server 192.168.7.211 not responding.

After this output the host system would not respond and had to be power cycled never mind the guest!

Once I changed to single core operation the system would run for some time until the host suffered from the dreaded kernel OOM killer. I was at a loss as to why the oom killer was running as the VM was only allocated half the physical memory (512MB) allowing the host what I presumed to be an adequate amount.

By adding a 512MB swapfile the system was able to push the few hundred kilobytes it wanted to swap and the system was now stable! The swapfile of course has to be started by hand as the external storage is unreliable and unavailable at boot.

I converted the qemu command line to a libvirt config using the virsh tool

virsh domxml-from-native qemu-argv cmdln.args

The converted configuration required manual editing to get a working system but now I have a libvirt based VM guest I can control along with all my other VM using the virt-manager tool.

This system is now stable and has been in production use for a month at time of writing. The one guest VM is a single core 512MB aarch64 system which takes over 1100 seconds (19 minutes) to do what a Banana Pi 2 dual core 1GB memory 32bit native ARM system manages in 300 seconds.

It seems the single core limited memory system with USB SATA attached storage is very, very slow.

I briefly attempted to run the CI system job natively within the host system but within minutes it crashed hard and required a power cycle to retrieve, it had also broken the UEFI boot. I must thank Leif for walking me through recovering the system otherwise I would have needed to start over.
ConclusionsI must stress these conclusions and observations are my own and do not represent anyone else.

My main conclusions are:

• My experience is of a poorly conceived, designed and implemented product rushed to market before it was ready.
• It was the first announced 64bit ARM single board computer to market but that lead was squandered with issues around availability, reliability and software.
• Value for money appears poor. Product is 70 plus an additional 50 for USB hubs, power supplies, USB Ethernet and USB SATA. Other comparable SBC are around the 30 mark and require fewer extras.
• The limited I/O within the core product yields a heavy reliance on USB
• The USB system is poorly implemented resulting in large additional administrative burdens.
• Limited memory of 1Gigabyte reduces the scope for making use of the benefits of 64bit processors.
• Recent change to UEFI bootloader is very welcome and something I would like to see across all aarch64 platforms. This is the time for there to be a single unified boot method, having to deal with multiple bad copies of bootloaders in the 32bit world was painful, perhaps this mistake can be avoided in the future.
• The kernel provision is abysmal and nothing like the quality I would expect from Linaro. The non-upstream kernel combined with missing features after almost a year of development is inexplicable.
• The Pine64 with 2G of memory and the Raspberry Pi 3 with its de-facto standard form factor are both preferable despite their limitations in other areas.

Categories:

• Debian-Blog
• Computing
• Technology
• Tools

Keywords:

• power saving
• laptop-mode-tools
• laptop
• battery

Like:

I decided to rethink the state of my personal site, and try out some of the new static site generators that are available now. To do that, I jotted down a series of things that I want in a static site generator, then wrote a tool to convert my ikiwiki site to other formats, and set out to evaluate things. As a benchmark I did a full rebuild of my site, which currently contains 1164 static files and 458 markdown pages. My requirements Free layout for my site My / is mostly a high-level index to the site contents. Blog posts are at /blog. My talk archive is organised like a separate blog at /talks. I want the freedom to create other sections of the site, each with its own rss feed, located wherever I want in the site hierarchy. Assets next to posts I occasionally blog just a photo with a little comment, and I would like the .md page with the comment to live next to the image in the file system. I did not even know that I had this as a requirement until I found static site generators that mandated a completely different directory structure for markdown contents and for static assets. Multiple RSS/Atom feeds I want at least one RSS/Atom feed per tag, because I use tags for marking which articles go to http://planet.debian.org. I also want RSS/Atom feeds for specific parts of the site, like the blog and talks. Arbitrary contents in /index.html I want total control over the contents of the main home page of the site. Quick preview while editing contents I do not like to wait several seconds for the site to be rebuilt at every review iteration of the pages I write. This makes me feel like the task of editing is harder than it should, and makes me lose motivation to post. Reasonable time for a full site rebuild I want to be able to run a full rebuild of the site in a reasonable time. I could define "reasonable" in this case as how long I can stare at the screen without getting bored, starting to do something else, and forgetting what it was that I was doing with the site. It is ok if a rebuild takes something like 10 or 30 seconds. It is not ok if it takes minutes. Code and dependency ecosystems that I can work with I can deal with Python and Go. I cannot deal with Ruby or JavaScript. I forgot all about Perl. Also, if it isn't in Debian it does not exist. Decent themes out of the box One of my hopes in switching to a more mainstream generator is to pick and choose themes and easily give my site a more modern look. Hugo Hugo is written in Go and is in Debian testing. Full rebuild time for my site is acceptable, and it can even parallelize: Free layout for my site was hard to get. I could replace /index.html by editing the template page for it, but then I did not find out how to create another similar index in an arbitrary place. Also, archetypes are applied only on the first path component of new posts, but I would like them instead to be matched on the last path component first, and failing that traveling up to the path until the top. This should be easy to fix by reorganizing the content a bit around here For example, a path for a new blog post of mine could be blog/2016/debian/ and I would like it to match the debian archetype first, and failing that the blog archetype. Assets next to posts almost work. Hugo automatically generates one feed per taxonomy element, and one feed per section. This would be currently sufficient for me, although I don't like the idea that sections map 1 to 1 to toplevel directories in the site structure. Hugo has a server that watches the file system and rerenders pages as they are modified, so the quick preview while editing works fine. About themes, it took me several tries to find a theme that would render navigation elements for both sections and tags, and most themes would render by pages with white components all around, and expect me to somehow dig in and tweak them. That frustrated me, because for quite a while I could not tell if I had misconfigured Hugo's taxonomies or if the theme was just somehow incomplete. Nikola Nikola is written in Python and is in Debian testing. Full rebuild time for my site is almost two orders of magnitude more than Hugo, and I am miffed to find the phrases "Nikola is fast." or "Fast building process" in its front page and package description: Free layout could be achieved fiddling with the site configuration to tell it where to read sources. Assets next to post work after tweaking the configuration, but they require to write inconsistent links in the markdown source: https://github.com/getnikola/nikola/issues/2266 I have a hard time accepting that that, because I want to author content with consistent semantic interlinking, because I want to be able 10 years from now to parse it and convert it to something else if a new technology comes out. Nikola generates one RSS/Atom feed per tag just fine. I have not tried generating feeds for different sections of the site. Incremental generation inside its built in server works fine. Pelican Pelican is written in Python and is in Debian testing. Full rebuild time for my site is acceptable: By default, pelican seems to put generate a single flat directory of html files regardless of the directory hierarchy of the sources. To have free layout, pelican needs some convincing in the configuration: but even if I do that, the urls that it generates still point to just slug /index.html and I have not trivially found a configuration option to fix that accordingly. I got quite uncomfortable at the idea of needing to configure content generation and linking to match, instead of having one automatically being in sync with the other. Having assets next to posts seems to be possible (also setting STATIC_PATHS = ["."]), but I do not recall making progress on this front. I did not manage to generate a feed for each tag out of the box, and probably there is some knob in the configuration for it. I gave up with Pelican as trying it out felt like a constant process of hacking the configuration from defaults that do not make any sense for me, withouth even knowing if a configuration exists that would do what I need Ikiwiki Ikiwiki is written in Perl and is in Debian. Although I am not anymore proficient with Perl, I was already using it, so it was worth considering. Full rebuild time feels a bit on the slow side but is still acceptable: In terms of free site structure, all feeds for all or part of the site, ikiwiki just excels. I even considered writing a python web server that monitors the file system and calls ikiwiki --refresh when anything changes, and calling it a day. However, when I tried to re-theme my website around a simple bootstrap boilerplate, I found that to be hard, as a some of the HTML structure is hardcoded in Perl (and it's also my fault) and there is only so much that can be done by tweaking the (rather unreadable) templates. siterefactor During all these experiments I had built siterefactor to generate contents for all those static site engines, and it was going through all the contents quite fast: So I wondered how slow it would become if, instead of making it write markdown, I made it write HTML via python markdown and Jinja2: I then started wondering how slower it would become if I implemented postprocessing of all local URLs generated by Markdown to make sure they are kept consistent even if the path of a generated page is different than the path of its source. Not much slower, really. I then added taxonomies. And arbitrary Jinja2 templates in the input, able to generate page lists and RSS/Atom feeds. And theming. And realised that reading all the sources and cross-linking them took 0.2 seconds, and the rest was generation time. And that after cross-linking, each page can be generated independently from all the others. staticsite So my site is now generated with staticsite: It's comparable with Hugo, and on a single process.

Review: The Lost Steersman, by Rosemary Kirstein

Series:	Steerswomen #3
Publisher:	Rosemary Kirstein
Copyright:	2003, 2014
Printing:	2014
ISBN:	0-9913546-2-1
Format:	Kindle
Pages:	432

This is the third book in the Steerswomen series and a direct follow-up to the events of The Outskirter's Secret. It does, marvel of marvels, feature an in-character summary of the events of the series to date! I do love when authors do this; it helps immensely if you come back to a series after a bit of a break between books. But this whole series is so good, and the emotional tone and development of Rowan as a character is so strong, that I recommend against starting here. After the events of the last book, Rowan has returned to the Inner Lands. She's sent her report back to the Archives, but stopped at the Annex in Alemeth. This is an auxiliary library that should have copies of the journals and other research that Rowan wants to search, and stopping there saves substantial travel time. However, she finds the steerswoman who was custodian of the Annex is deceased and the Annex is, from Rowan's perspective, a mess. Nothing is organized, the books aren't properly cared-for, and Mira's interactions with the townsfolk were far different than Rowan's natural attitude. The start of this book was a surprising shift. After the large-scale revelations at the end of The Outskirter's Secret, and the sense of escalating danger, Rowan's return to small-town life in the Inner Lands comes as a shock. That's true for both the reader and for Rowan, and the parallels make it a remarkably effective bit of writing. At the beginning of the story, the reader is already familiar with Rowan (at least if you've read the previous books) and how she thinks of being a steerswoman. Rowan is very much on edge and in a hurry given what's going on in the broader world. But the town is used to Mira: a gregarious socializer who cared far more about town gossip and her role as coordinator of it than she cared about most of her steerswoman duties. (At least as seen from Rowan's perspective. By the end of the book, we have a few hints that something else might be going on, but the damage to the books at least feels unforgivable.) Rowan is resented and even disliked at first, particularly by Steffie and Gwen who did most of the chores at the Annex and were closest to Mira. One of the reasons why I love this series so much is that Kirstein has a gift for characterization. Rowan (and Bel, who largely doesn't appear in this book) are brilliant characters, but it's not just them. At the start of this book, the reader tends to share Rowan's opinion of the town: a sort of half-bemused, half-exasperated indifference. Even as the characters start to grow on one, it seems like a backwater and a diversion from the larger story. But it becomes clear that Rowan is very on-edge from her experience in the Outskirts, that she's underestimated the relevance of Mira to at least the town's happiness, and she's greatly underestimated the ability of the townsfolk to help her. Steffie, in particular, is a wonderful character; by the end of the book, he had become one of my favorite people in the series so far. He doesn't think he's particularly smart, and his life before Rowan is very simple, but there are depths to him that no one, including him, expected. There is a plot here, apart from small town politics and Rowan's slow relaxation. (Although those were so compelling that I'm not sure I would have minded if that were the entire book.) The lost steersman of the title is an old student friend of Rowan that she unexpectedly meets in town, a former steersman who quit the order and refused to explain why. Rowan, of course, cannot resist trying to fix this situation. The second plot driver is a dangerous invasion of Outskirts monsters into the town. Those who have read the previous books will have some immediate guesses as to why this might be, and Rowan does as well. But there's more going on than it might first appear. This book is not entirely a diversion. It returns to the main plot of the series by the end of the book, and we learn much more about Rowan's world. But, somewhat surprisingly, that was my least favorite part of the book. It has some nice bits of exploration and puzzle-solving, and Rowan is always a delight to spend time with. But the last section of the book is similar to many other genre books I've read before well-written, to be sure, but not as unique. It also features a rather long section following a character who is severely physically ill, which is something I always find very hard to read (a personal quirk). But there's a lot of meat here for the broader plot, and I have no idea what will happen in the next book. The part of The Lost Steersman that I'm going to remember, though, are the town bits, up through the arrival of Zenna (another delightful character who adds even more variety to Kirstein's presentation of steerswomen). Kirstein is remarkably good at mixing small-town characters with the scientific investigation of the steerswomen and letting them bounce off of each other to reveal more about the character of both. If it weren't for the end of the book, which bothered me for partly idiosyncratic reasons, I think this would have been my favorite book of the series. Followed by The Language of Power, and be warned that this book ends on something close to a cliffhanger. Rating: 8 out of 10

The following is a short summary of my open source work in February, just like in previous months.

Debian

• Uploaded mopidy-dirble 1.3.0-1: New upstream release. Moved to pkg-mopidy team.
• Uploaded mopidy-tunein 0.3.0-1: New upstream release. Moved to pkg-mopidy team.
• Uploaded python-tunigo 1.0.0-1: New package. ITP. Maintained in the Debian Python Modules Team (DPMT).
• Uploaded mopidy 2.0.0-1: New upstream release. Fixed RC bug #785910.
• Uploaded mopidy-soundcloud 2.0.2-2: Fixed RC bug #799089.

Mopidy

• Released Mopidy-Spotify 2.3.1: Works around search being broken in libspotify by using the Spotify Web API for the search functionality, and nothing else.
• Wrapped up and released Mopidy 2.0 release. This was a quite large release that ports to GStreamer 1.x and adds support for gapless playback. It was uploaded to Debian in time to be part of Ubuntu 16.04 LTS.
• Released Mopidy-Spotify 3.0.0: A couple of small compatibility fixes to work with Mopidy 2.x and GStreamer 1.x.
• Updates to packages not in Debian, only at apt.mopidy.com:
  • Uploaded mopidy-spotify-tunigo 1.0.0-0mopidy1. Far down the chain, it depends on libspotify, which is not in Debian.
  • Uploaded mopidy-spotify 2.0.0-0mopidy1.
• Updated homebrew-mopidy with lots of new releases:
  • Mopidy 2.0.0
  • Mopidy-Dirble 1.3.0
  • Mopidy-SoundCloud 2.0.2
  • Mopidy-Spotify 2.3.0, 2.3.1 and 3.0.0
  • python-backports-abc 0.4
  • python-backports-ssl-match-hostname 3.5.0.1
  • python-certifi 2015.11.20.1
  • python-cffi 1.5.0 and 1.5.2
  • python-requests 2.9.1
  • python-singledispatch 3.4.0.3
  • python-six 1.10.0
  • python-tornado 4.3
• Rebased my feature/py3-compat branch on top of Mopidy 2.0.
• Planned breaking changes for Mopidy 3.0: unbundling of Mopidy.js, removal of deprecated core APIs, model APIs, audio APIs and removal of the http/static_dir config.
• Planned breaking changes to modernize Mopidy.js: change default calling convention to by-position-or-by-name, replace When.js with ES6 Promise, replace Bane.js with EventEmitter, and replace Buster.js with Karma+Mocha.

Comics

• Started working on the upgrade to Django 1.8. Most deps are upgraded. The major remaining pieces is to upgrade django-registration from 0.8 to 2.0 and to replace the years-old vendored copy of django-invitation (singular) with django-invitations (plural).

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

• Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
  • Added explicit support to send messages asynchronously via the Celery distributed task queue. (#29)
  • Worked with Patrick Clope to add an escapeslack template filter to ensure the correct characters for Slack's API are escaped using Django's built-in safe is too invasive. (#36)
  • Corrected an issue where custom endpoints and channel names were incompatible. (#27)
  • Overhaul of the field/option/block parsing. (49ff3c4)
  • Moved away from using a django.template.Context instance, preventing a deprecation warning. (#31)
• Added a create-folders subcommand to my tickle-me-email Getting Things Done (GTD) email toolbox to create numbered folders for the rotate "tickler" functionality. (#3)
• Wrote and released a Django template tag to collapse multiple whitespace/newline characters in order to correctly format, for example, plaintext emails that make extensive use of for-loops. These would normally require careful and fragile placement of the % for .. % and % endfor % tags. (Repo)
• Pushed a number of updates to my Strava Enhancement Suite, a Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker:
  • Fix unit conversion where the element was wrapped in an another HTML element. (#48)
  • Hide premium indicator from top-level navigation. (#45)
  • Correct matching of the "shop" link in top-level navigation and footer module. (#47 & #46)
  • Remove "premium" pages on Club pages. (#44)
• Added a security-oriented warning to Ansible's documentation regarding the behaviour if an UFW firewall application profile is added and subsequently removed. (#1740)
• Added support to my django-staticfiles-dotd "staticfiles" library which concatenates Javascript and CSS files from .d-style directories to support an arbitrary file rendering method. This allows the use of media pre-processors such as SASS within such directories. (#1)
• Corrected my Chrome extension for the FastMail web interface to correctly hook into the internal "send" mechanism. (#2)
• Moved my stravabot IRC bot to use dh-virtualenv and systemd, improving security and reliability. (1c48709)
• Updated django-pedantic-http-methods a tool to raise an exception during development when attempting to perform side effects in GET and HEAD HTTP methods to support the latest version of Django. (#1)

---

This was my tenth month as a Freexian sponsored LTS contributor. I was assigned 8 hours for the month of February. As I did last month, I worked on updating clamav in wheezy and squeeze-lts. As with previous updates to clamav, we updated it to the new upstream version[1]. As an added complexity, this version bumped soname, so it s now libclamav7 instead of libclamav6. This bump necessitated a small transition in jessie/wheezy-proposed-updates and squeeze-lts. The update for Jessie (included for completeness here) was done early in the month by other pkg-clamav team members. It and the rebuilt/update libclamav reverse-depends will be included in the next Jessie point release. For wheezy, I uploaded libclamunrar (which bumped soname as well) and worked with other pkg-clamav team members on getting clamav to build on sparc and preparing a fix for c-icap. It and the rebuilt/update libclamav reverse-depends will be included in the next Wheezy point release. As a result of the amount of time it took, the squeeze-lts update landed later than I hoped it would, but it is there. As documented in DLA 437-1, there are new packages for clamav, libclamunrar, python-clamav, and klamav. The last squeeze libclamav reverse-depend, dansguardian, took more work, but it too is updated, see DLA 440-1.
[1] The primary reason for this is that anti-virus is an arms race. Unlike other types of packages being stable with only fixes for severe bugs and security issues does not result in a stable capability. It will regress over time. In order to keep up, the new version is needed.

This is a bit of an odd posting since it's about something I've done but is also here to help me explain why I did it and thus perhaps encourage some discussion around the topic within the Kicad community...

Recently (as you will know if you follow this blog anywhere it is syndicated) I have started playing with Kicad for the development of some hardware projects I've had a desire for. In addition, some of you may be aware that I used to work for a hardware/software consultancy called Simtec, and there I got to play for a while with an EDA tool called Mentor Designview. Mentor was an expensive, slow, clunky, old-school EDA tool, but I grew to understand and like the workflow. I spent time looking at gEDA and Eagle when I wanted to get back into hardware hacking for my own ends; but neither did I really click with. On the other hand, a mere 10 minutes with Kicad and I knew I had found the tool I wanted to work with long-term. I designed the beer'o'meter project (a flow meter for the pub we are somehow intimately involved with) and then started on my first personal surface-mount project -- SamDAC which is a DAC designed to work with our HiFi in our study at home. As I worked on the SamDAC project, I realised that I was missing a very particular thing from Mentor, something which I had low-level been annoyed by while looking at other EDA tools -- Kicad lacks a mechanism to mark a wire as being linked to somewhere else on the same sheet. Almost all of the EDA tools I've looked at seem to lack this nicety, and honestly I miss it greatly, so I figured it was time to see if I could successfully hack on Kicad. Kicad is written in C++, and it has been mumble mumble years since I last did any C++, either for personal hacking or professionally, so it took a little while for that part of my brain to kick back in enough for me to grok the codebase. Kicad is not a small project, taking around ten minutes to build on my not-inconsiderable computer. And while it beavered away building, I spent time looking around the source code, particularly the schematic editor eeschema. To skip ahead a bit, after a couple of days of hacking around, I had a proof-of-concept for the intra-sheet links which I had been missing from my days with Mentor, and some ERC (electrical rules checking) to go alongside that to help produce schematics without unwanted "sharp corners". In total, I added:

• an "intra-sheet link" schematic element, including:
  • the schematic element itself
  • loading and saving of the element (which required a bump in schematic file format versioning)
  • UI for adding intra-sheet links to your schematic
  • netlist build support to ensure that the links found their way into the internal netlist content.

• an ERC tickybox "Strict wiring checks" and behind that I hid:
  • checks for wires with dangling ends (dangles are bad)
  • junctions which have no point in being present (pointless junctions are confusing to readers)
  • checks that intra-sheet links exist in groups of two or more on any given net, since individual links are pointless.

I forked the Kicad mirror on Github and pushed my own branch with this work to my Kicad fork. All of this is meant to allow schematic capture engineers to more clearly state their intentions regarding what they are drawing. The intra-sheet link could be thought of like a no-connect element, except instead of saying "this explicitly goes nowhere" we're saying "this explicitly goes somewhere else on this sheet, you can go look for it". Obviously, people who dislike (or simply don't want to use) such intra-sheet link elements can just disable that ERC tickybox and not be bothered by them in the least (well except for the toolbar button and menu item I suppose). Whether this work gets accepted into Kicad, or festers and dies on the vine, it was good fun developing it and I'd like to illustrate how it could help you, and why I wrote it in the first place:

A contrived story

Note, while this story is meant to be taken seriously, it is somewhat contrived, the examples are likely electrical madness, but please just think about the purpose of the checks etc.

To help to illustrate the feature and why it exists, I'd like to tell you a somewhat contrived story about Fred. Fred is a schematic capture engineer and his main job is to review schematics generated by his colleagues. Fred and his colleagues work with Kicad (hurrah) but of late they've been having a few issues with being able to cleanly review schematics. Fred's colleagues are not the neatest of engineers. In particular they tend to be quite lazy when it comes to running busses, which are not (for example) address and data busses, around their designs and they tend to simply have wires which end in mid-space and pick up somewhere else on the sheet. All this is perfectly reasonable of course, and Kicad handles it with aplomb. Sadly it seems quite error prone for Fred's workplace. As an example, Fred's colleague Ben has been designing the power supply for a particular board. As with most power supplies, plenty of capacitors are needed to stabilise the regulators and smooth the output. In the example below, the intent is that all of the capacitors are on the FOO net. Sadly there's a missing junction and/or slightly misplaced label in the upper section which means that C2 and C3 simply don't join to the FOO net. This could easily be missed, but the ERC can't spot it at all since there's more than one thing on each net, so the pins of the capacitors are connected to something. Fred is very sad that this kind of problem can sometimes escape notice by the schematic designer Ben, Fred himself, and the layout engineer, resulting in boards which simply do not work. Fred takes it upon himself to request that the strict wiring checks ERC is made mandatory for all designs, and that the design engineers be required to use intra-sheet link symbols when they have signals which wander off to other parts of the sheet like FOO does in the example. Without any further schematic changes, strict wiring checks enabled gives the following points of ERC concern for Ben to think about: As you can see, the ERC is pointing at the wire ends and the warnings are simply that the wires are dangling and that this is not acceptable. This warning is very like the pin-not-connected warnings which can be silenced with an explicit no-connect schematic element. Ben, being a well behaved and gentle soul, obeys the design edicts from Fred and seeks out the intra-sheet link symbols, clearing off the ERC markers and then adding intra-sheet links to his design: This silences the dangling end ERC check, which is good, however it results in another ERC warning: This time, the warning for Ben to deal with is that the intra-sheet links are pointless. Each exists without a companion to link to because of the net name hiccough in the top section. It takes Ben a moment to realise that the mistake which has been made is that a junction is missing in the top section. He adds the junction and bingo the ERC is clean once more: Now, this might not seem like much gain for so much effort, but Ben can now be more confident that the FOO net is properly linked across his design and Fred can know, when he looks at the top part of the design, that Ben intended for the FOO net to go somewhere else on the sheet and he can look for it.

Why do this at all? Okay, dropping out of our story now, let's discuss why these ERC checks are worthwhile and why the intra-sheet link schematic element is needed.

Note: This bit is here to remind me of why I did the work, and to hopefully explain a little more about why I think it's worth adding to Kicad...

Designers are (one assumes) human beings. As humans we (and I count myself here too) are prone to mistakes. Sadly mistakes are often subtle and could easily be thought of as deliberate if the right thought processes are not followed carefully when reviewing. Anyone who has ever done code review, proofread a document, or performed any such activity, will be quite familiar with the problems which can be introduced by a syntactically and semantically valid construct which simply turns out to be wrong in the greater context. When drawing designs, I often end up with bits of wire sticking out of schematic sections which are not yet complete. Sadly if I sleep between design sessions, I often lose track of whether such a dangling wire is meant to be attached to more stuff, or is simply left because the net is picked up elsewhere on the sheet. With intra-sheet link elements available, if I had intended the latter, I'd have just dropped such an element on the end of the wire before I stopped for the day. Also, when drawing designs, I sometimes forget to label a wire, especially if it has just passed through a filter or current-limiting resistor or similar. As such, even with intra-sheet link elements to show me when I mean for a net to go bimbling off across the sheet, I can sometimes end up with unnamed nets whose capacitors end up not used for anything useful. This is where the ERC comes in. By having the ERC complain if a wire dangles -- the design engineer won't forget to add links (or check more explicitly if the wire is meant to be attached to something else). By having junctions which don't actually link anything warned about, the engineer can't just slap a junction blob down on the end of a wire to silence that warning, since that doesn't mean anything to a reviewer later down the line. By having the ERC warn if a net has exactly one intra-sheet link attached to it, missing net names and errors such as that shown in my contrived example above can be spotted and corrected. Ultimately this entire piece of work is about ensuring that the intent of the design engineer is captured clearly in the schematic. If the design engineer meant to leave that wire dangling because it's joining to another bit of wire elsewhere on the sheet, they can put the intra-sheet links in to show this. The associated ERC checks are there purely to ensure that the validation of this intent is not bypassed accidentally, or deliberately, in order to make the use of this more worthwhile and to increase the usefulness of the ERC on designs where signals jump around on sheets where wiring them up directly would just create a mess.

This is based on joint work with Gregor Aisch and Josh Keller of the New York Times. A new package gunsales is now on the CRAN network for R. It is based the NYTimes/gunsales repository underlying the excellent New York Times visualizations, first published first in December 2015 and updated with more recent data since. The analysis takes public government data on gun sales from the National Instant Criminal Background Check System (NICS). The original data is scraped from the pdf, included in the package, and analysed in a cross-section and time-series manner. The standard US Census tool X-13ARIMA-SEATS is used to deseasonalize the timeseries at the national or state level. (Note that Buzzfeed also published data and (Python) code in another GitHub repo.) As an aside, it was the use of X-13ARIMA-SEATS here -- and its somewhat awkward and manual installation also seen in the initial versions of the code in the NYTimes/gunsales repo -- which lead to the recent work by Christoph Sax and myself. We now provide a new package x13binary on CRAN so that Christoph's excellent seasonal package can simply depend upon it and have a working binary provided and installed ready to use; see the recent blog post for more. The net result is that a package like this new gunsales project can simply depend upon seasonal and also be assurred that x13binary "just works". As Martha would say, "A Good Thing". Back to the gunsales project. Following the initial publication of the repository with the data and R code in a simple script, I felt compelled to reorganize it as a package. Packages for R, as we teach our students, colleagues, or anybody else who wants to listen are really the best way to bundle code, data, documentation (i.e. vignettes) and tests. All that exists now in the gunsales package. The package now has one main function, analysis(), which returns a single dataframe object. This dataframe object can then be fed to two plotting functions. The first, plot_gunsales(), will then recreate all the (base R) plots from the original code base. The second, ggplot_gunsales(), does the same but via ggplot2. This should give anybody the ability to look at the data, study the transformations done, form and maybe test new hypotheses and visualize in manner comparable to the original publication. As an amuse gueule, here are the key plots also shown in the main README.md at GitHub: We look forward to more remixes and analysis of this data. The plan of the GitHub repository is to keep the data set updated as new data points are published.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I maintain an application for bridging various non-Hue lighting systems to something that looks enough like a Hue that an Amazon Echo will still control them. One thing I hadn't really worked on was colour support, so I picked up some cheap bulbs and a bridge. The kit is badged as an iSuper iRainbow001, and it's terrible.

Things seemed promising enough at first, although the bulbs were alarmingly heavy (there's a significant chunk of heatsink built into them, which seems to get a lot warmer than I'd expect from something that claims a 7W power consumption). The app was a bit clunky, but eh - I wasn't planning on using it for long. I pressed the button on the bridge, launched the app and could control the bulbs. The first thing I noticed was that they had a separate "white" and "colour" mode. White mode was pretty bright, but colour mode massively less so - presumably the white LEDs are entirely independent of the RGB ones, and much higher intensity. Still, potentially useful as mood lighting.

Anyway. Next step was to start playing with the protocol, which meant finding the device on my network. I checked anything that had picked up a DHCP lease recently and nmapped them. The OS detection reported Linux, which wasn't hugely surprising - there was no GPL notice or source code included with the box, but I'm way past the point of shock at that. It also reported that there was a telnet daemon running. I connected and got a login prompt. And then I typed admin as the username and admin as the password and got a root prompt. So, there's that. The copy of Busybox included even came with tftp, so it was easy to get copies of tcpdump and strace on there to see what was up.

So. Next step. Protocol sniffing. I wanted to know how discovery worked, so reset the device to factory and watched what happens. The app on my phone sent out a discovery packet on UDP port 18602 which looked like this:

INLAN:CLIP:23.21.212.48:CLPT:22345:MAC:02:00:00:00:00:00

The CLIP and CLPT fields refer to the cloud server that allows for management when you're not on the local network. The mac field contains an utterly fake address. If you send out a discovery packet and your mac hasn't been registered with the device, you get a denial back. If your mac has been (and by "mac" here I mean "utterly fake mac that's the same for all devices"), you get back a response including the device serial number and IP address. And if you just leave out the mac field entirely, you get back a response no matter whether your address is registered or not. So, that's a start. Once you've registered one copy of the app with the device, anything can communicate with it by just using the same fake mac in the discovery packets.

Onwards. The command format turns out to be simple. They start ##, are followed by two ascii digits encoding a command, four ascii digits containing a bulb id, two ascii digits containing the number of following bytes and then the command data (in ascii). An example is:

##05010002ff

which decodes as command 5 (set white intensity) on bulb 1 with two bytes of data following, each of which is an f. This translates as "Set bulb 1 to full white intensity". I worked out the rest pretty quickly - command 03 sets the RGB colour of the bulb, 0A asks the bridge to search for new bulbs, 0B tells you which bulbs are available and their capabilities and 0E gives you the MAC addresses of the bulbs( ). 0C crashes the server process, and 06 spews a bunch of garbage back at you and potentially crashes the bulb in a hilarious way that involves it flickering at about 15Hz. It turns out that 06 is actually the "Rename bulb" command, and if you send it less data than it's expecting something goes hilariously wrong in string parsing and everything is awful.

Ok. Easy enough, but not hugely exciting. What about the remote protocol? This turns out to involve sending a login packet and then a wrapped command packet. The login has some length data, a header of "MQIsdp", a long bunch of ascii-encoded hex, a username and a password.

The username is w13733 and the password is gbigbi01. These are hardcoded in the app. The ascii-encoded hex can be replaced with 0s and the login succeeds anyway.

Ok. What about the wrapping on the command? The login never indicated which device we wanted to talk to, so presumably there's some sort of protection going on here oh wait. The command packet is a length, the serial number of the bridge and then a normal command packet. As long as you know the serial number of the device (which it tells you in response to a discovery packet, even if you're unauthenticated), you can use the cloud service to send arbitrary commands to the device (including the one that crashes the service). One of which involves the device then doing some kind of string parsing that doesn't appear to work that well. What could possibly go wrong?

Ok, so that seemed to be the entirety of the network protocol. Anything else to do? Some poking around on the bridge revealed (a) that it had an active wireless device and (b) a running DHCP server. They wouldn't, would they?

Yes. Yes, they would.

The devices all have a hardcoded SSID of "iRainbow", although they don't broadcast it. There's no security - anybody can associate. It'll then hand out an IP address. It's running telnetd on that interface as well. You can bounce through there to the owner's internal network.

So, in summary: it's a device that infringes my copyright, gives you root access in response to trivial credentials, has access control that depends entirely on nobody ever looking at the packets, is sufficiently poorly implemented that you can crash both it and the bulbs, has a cloud access protocol that has no security whatsoever and also acts as an easy mechanism for people to circumvent your network security. This may be the single worst device I've ever bought.

I called the manufacturer and they insisted that the device was designed in 2012, was no longer manufactured or supported, that they had no source code to give me and there would be no security fixes. The vendor wants me to pay for shipping back to them and reserves the right to deduct the cost of the original "free" shipping from the refund. Everything is awful, which is why I just ordered four more random bulbs to play with over the weekend.

comments

Another minor release of RcppEigen is on CRAN and getting into Debian. The main focus is an upgrade to the recent 3.2.7 release of Eigen which should address another UBSAN issue. And once again Yixuan Qiu did all the heavy lifting. The NEWS file entry follows.

Changes in RcppEigen version 0.3.2.8.0 (2016-02-23)

• Updated to version 3.2.8 of Eigen (PR #29 by Yixuan)

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

The binary code, although necessary for major social and technological developments, is annihilating the Homo Technologicus, stifling his innate freedom for analog connections.

The first widespread use of the binary code was the Morse code alphabet. For decades, this communication system allowed to transmit information over long distances, between ships in the ocean and the mainland, between one continent and another, and today its use is still active in emergency situations. In its disarming simplicity (connect or interrupt two electrical wires), this system proved to be the shortest way to transcribe an alphabet.

Then, the Boolean algebra (binary) represented the language of communication between humans and computers, enabling the translation of instructions and commands towards the processing unit (microprocessor / CPU). This determined the Big Bang of the computers and the technological era

<Read More...>

Coming from a country that has taken lots of refugees, especially in the years after the second world war (although not any more), and while living in Japan still feeling connected to Europe and what is going on there, the current situation of refugees from the near east gives lots of material to think about. One of these things recently was passed around on various SNS sites. It is about the number of refugees a country can accept . The situation is difficult, and after an initial wave of empathy it seems that especially governments are retracting and trying to stop the influx of refugees:

We cannot accommodate any more refugees in Europe, that s not possible.
Manuel Valls, Prime Minister of France

I am well aware about the difficult situation in Europe, and consider the dealing with the refugee crisis the biggest challenge of these decade. In this blog, without many words, I only want to focus on how Japan deals with refugees. Europe might be bad, but Japan I am bare of words.

	Lebanon	European Community	Japan
Population	4.467.000 (2013)	503.000.000	127.300.000 (2013)
Refugees 2015	~1.000.000	850.000	27
Ratio	4.4 : 1	590 : 1	4714814 : 1

Sources of the data:

• Japan: Japan s Ministry of Justice via Japan Association for refugees
• Lebanon: UNHCR data Syria
• Europe: UNHCR Europe Response Center as well as the previous page

No words are necessary.

I recently stumbled across an XML format specified in a DTD that I wanted to work with from within C++. The XML format is document centric, which is a bit of a pain with existing data binding compilers according to my limited experience. So to learn something new, and to keep control over generated code, I started to investigate what it would take to write my own little custom data binding compiler.

I recently stumbled across an XML format specified in a DTD that I wanted to work with from within C++. The XML format is document centric, which is a bit of a pain with existing data binding compilers according to my limited experience. So to learn something new, and to keep control over generated code, I started to investigate what it would take to write my own little custom data binding compiler.

The biggest change we ve seen in the Linux kernel for ARM over the past few years has been the transition to providing descriptions of the hardware in systems via device tree. This splits out the description of the devices in the system that can t be automatically enumerated from the kernel into a separate binary instead of being part of the kernel binary. Currently for most systems that are actively used upstream the device tree source code is kept in the kernel but the goal is to allow people to use device trees that are distributed separately to the kernel, especially device trees that are shipped as part of the board firmware. This is something that other platforms have done for a long time, PowerPC Macs and Sun SPARC systems use device tree as the mechanism for describing the hardware to the operating system. One consequence of this desire to allow the kernel and device tree to be shipped separately is that the device tree becomes an ABI. This is a really big change for people working in the embedded and consumer electronics areas where ARM has been most widely deployed, it means that any descriptions of the hardware need to be something that can stand the test of time. Anything we release is something we have to expect to carry code for indefinitely. When everything was done as part of the kernel binary we could easily do something that doesn t quite represent the hardware with the intention of replacing it later, now it is much harder to do that. An example of this is the SAW in Qualcomm SoCs. This is a block in the SoC which provides control of some regulators used for the CPU cores in the PMIC in very low power states and also allows the CPU to control the regulator with fast memory mapped registers rather than the slower buses used to control the PMIC. Unfortunately it doesn t fully replace direct access to the PMIC, it supports a subset of the control we need for the PMIC but not all of it. We could represent the SAW as an independent regulator but from a system integration point of view it is functioning as an extra control interface for the external PMIC and if we want to use the extra functionality that is only available via direct access to the PMIC we need to consider that and represent the SAW as an extension of it. If we don t need that extra PMIC functionality at the current time this means we need to do some extra work to make sure we describe the PMIC before we can use the SAW even if we have no intention to use anything other than the SAW. Now, few if any people are actually using the device tree as an ABI at present so those working on enabling platforms often forget about the requirement and find it an obstacle to getting things done they have pressure to get things done, they don t have quite the same pressure to make sure that attention is paid to device tree compatibility so it can easily get forgotten. Over time this may change, especially if people start to take advantage of the device tree as an ABI that become more and more important, but for now if we want to enable that in the future it s something we have to actively think about and work on, accepting that this means we won t always be able to do the most expedient thing.

A little more than 3 months after our latest minor release, here is the new major version of Gnocchi, stamped 2.0.0. It contains a lot of new and exciting features, and I'd like to talk about some of them to celebrate! You may notice that this release happens in the middle of the OpenStack release cycle. Indeed, Gnocchi does not follow that 6-months cycle, and we release whenever our code is ready. That forces us to have a more iterative approach, less disruptive for other projects and allow us to achieve a higher velocity. Applying the good old mantra release early, release often. Documentation This version features a large documentation update. Gnocchi is still the only OpenStack server project that implements a "no doc, no merge" policy, meaning any code must come with the documentation addition or change included in the patch. The full documentation is included in the source code and available online at gnocchi.xyz. Data split & compression I've already covered this change extensively in my last blog about timeseries compression. Long story short, Gnocchi now splits timeseries archives in small chunks that are compressed, increasing speed and decreasing data size. Measures batching support Gnocchi now supports batching, which allow submitting several measures for different metric in a single request. This is especially useful in the context where your application tends to cache metrics for a while and is able to send them in a batch. Usage is fully documented for the REST API. Group by support in aggregation One of the most demanded features was the ability to do measure aggregation no resource, using a group by type query. This is now possible using the new groupby parameter to aggregation queries. Ceph backend optimization We improved the Ceph back-end a lot. Mehdi Abaakouk wrote a new Python binding for Ceph, called Cradox, that is going to replace the current Python rados module in the subsequent Ceph releases. Gnocchi makes usage of this new module to speed things up, making the Ceph based driver really, really faster than before. We also implemented asynchronous data deletion, which improves performance a bit. The next step will be to run some new benchmarks like I did a few months ago and compare with the Gnocchi 1.3 series. Stay tuned!