The following is a short summary of my open source work in February, just like in previous months.

Debian

• Uploaded mopidy-dirble 1.3.0-1: New upstream release. Moved to pkg-mopidy team.
• Uploaded mopidy-tunein 0.3.0-1: New upstream release. Moved to pkg-mopidy team.
• Uploaded python-tunigo 1.0.0-1: New package. ITP. Maintained in the Debian Python Modules Team (DPMT).
• Uploaded mopidy 2.0.0-1: New upstream release. Fixed RC bug #785910.
• Uploaded mopidy-soundcloud 2.0.2-2: Fixed RC bug #799089.

Mopidy

• Released Mopidy-Spotify 2.3.1: Works around search being broken in libspotify by using the Spotify Web API for the search functionality, and nothing else.
• Wrapped up and released Mopidy 2.0 release. This was a quite large release that ports to GStreamer 1.x and adds support for gapless playback. It was uploaded to Debian in time to be part of Ubuntu 16.04 LTS.
• Released Mopidy-Spotify 3.0.0: A couple of small compatibility fixes to work with Mopidy 2.x and GStreamer 1.x.
• Updates to packages not in Debian, only at apt.mopidy.com:
  • Uploaded mopidy-spotify-tunigo 1.0.0-0mopidy1. Far down the chain, it depends on libspotify, which is not in Debian.
  • Uploaded mopidy-spotify 2.0.0-0mopidy1.
• Updated homebrew-mopidy with lots of new releases:
  • Mopidy 2.0.0
  • Mopidy-Dirble 1.3.0
  • Mopidy-SoundCloud 2.0.2
  • Mopidy-Spotify 2.3.0, 2.3.1 and 3.0.0
  • python-backports-abc 0.4
  • python-backports-ssl-match-hostname 3.5.0.1
  • python-certifi 2015.11.20.1
  • python-cffi 1.5.0 and 1.5.2
  • python-requests 2.9.1
  • python-singledispatch 3.4.0.3
  • python-six 1.10.0
  • python-tornado 4.3
• Rebased my feature/py3-compat branch on top of Mopidy 2.0.
• Planned breaking changes for Mopidy 3.0: unbundling of Mopidy.js, removal of deprecated core APIs, model APIs, audio APIs and removal of the http/static_dir config.
• Planned breaking changes to modernize Mopidy.js: change default calling convention to by-position-or-by-name, replace When.js with ES6 Promise, replace Bane.js with EventEmitter, and replace Buster.js with Karma+Mocha.

Comics

• Started working on the upgrade to Django 1.8. Most deps are upgraded. The major remaining pieces is to upgrade django-registration from 0.8 to 2.0 and to replace the years-old vendored copy of django-invitation (singular) with django-invitations (plural).

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

• Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
  • Added explicit support to send messages asynchronously via the Celery distributed task queue. (#29)
  • Worked with Patrick Clope to add an escapeslack template filter to ensure the correct characters for Slack's API are escaped using Django's built-in safe is too invasive. (#36)
  • Corrected an issue where custom endpoints and channel names were incompatible. (#27)
  • Overhaul of the field/option/block parsing. (49ff3c4)
  • Moved away from using a django.template.Context instance, preventing a deprecation warning. (#31)
• Added a create-folders subcommand to my tickle-me-email Getting Things Done (GTD) email toolbox to create numbered folders for the rotate "tickler" functionality. (#3)
• Wrote and released a Django template tag to collapse multiple whitespace/newline characters in order to correctly format, for example, plaintext emails that make extensive use of for-loops. These would normally require careful and fragile placement of the % for .. % and % endfor % tags. (Repo)
• Pushed a number of updates to my Strava Enhancement Suite, a Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker:
  • Fix unit conversion where the element was wrapped in an another HTML element. (#48)
  • Hide premium indicator from top-level navigation. (#45)
  • Correct matching of the "shop" link in top-level navigation and footer module. (#47 & #46)
  • Remove "premium" pages on Club pages. (#44)
• Added a security-oriented warning to Ansible's documentation regarding the behaviour if an UFW firewall application profile is added and subsequently removed. (#1740)
• Added support to my django-staticfiles-dotd "staticfiles" library which concatenates Javascript and CSS files from .d-style directories to support an arbitrary file rendering method. This allows the use of media pre-processors such as SASS within such directories. (#1)
• Corrected my Chrome extension for the FastMail web interface to correctly hook into the internal "send" mechanism. (#2)
• Moved my stravabot IRC bot to use dh-virtualenv and systemd, improving security and reliability. (1c48709)
• Updated django-pedantic-http-methods a tool to raise an exception during development when attempting to perform side effects in GET and HEAD HTTP methods to support the latest version of Django. (#1)

---

This year s Debian Ruby team sprint started today here at Curitiba. Everyone arrived fine, and we started working at the meeting room we have booked for the week at Curitiba campus of the Federal Technical University of Paran . The room is at the Department of Business and Community Relations , what makes a lot of sense! :-) The day started with a quick setup, with a simple 8-port switch, and a couple of power strips. It tooks us a few minutes to figure what was blocked or not on the corporate network, and almost everyone who needs connections that are usually blocked in such environments already had their VPN setups so we were able to get started right after that. We are taking notes live on mozilla s piblic etherpad site Today we accomplished quite a lot:

• analyzed the pending issues for the ruby2.3 transition issues, categorizing the missing packages into needs a binNMU now , needs a binNMU after switching the default to ruby2.3 , and broken .
• ruby-defaults uploaded to experimental switching to ruby2.3 as default, and dropping support for ruby2.2
• ruby-gsl fixed to build against GSL 2.x (was blocking ruby2.3 transition)
• #816253: ruby-fast-gettext: fix FTBFS issue and import a new upstream
• ruby-aws: several issues fixed
• ruby-binding-of-caller: fixed rubygems-integration
• fixed ruby-kakasi-ffi (again)
• made ruby-blockenspiel arch:any again
• ruby-fast-gettext 1.0.0-1 (fix ftbfs 816253)
• ruby-aws-sdk: new upstream, debcheck fix and several bumps
• ruby-fcgi dropped transitional packages + refreshed packaging with -w
• ruby-sshkit 1.8.0
• ruby-beautify: new upstream and few minor fixes
• asciidoctor (new version sponsored)
• capistrano 3.4.0 (ftbfs #795724, #802734)
• #816254: ruby-packetfu: FTBFS: expected NameError with uninitialized constant PacketFu::FakePacket , got #
• updated rake to 10.5.0; making it not include -I/usr/lib/ruby/vendor_ruby when running tests
• triaged an closed open bugs on ruby-httpclient that do not apply anymore.
• updated ruby-httpclient to get rid of warnings in apt-listbugs under Ruby 2.3
• ruby-packetfu (fix ftbfs #816254) by @kanashiro
• investigated extension/rdepends build failing with ruby2.3-only
• applied upstream patch to ruby to fix extension ftbfs when extension uses c++
• ruby-albino 1.3.3-4 (#813644)
• basic user-level testing using ruby2.3 as default:
  • chef WORKS mostly; it seems ohai segfaults some times
  • rails autopkgtest FIXED Could not find gem binding_of_caller (>= 0.7.2) , which is required by gem web-console (~> 2.0) , in any of the sources.
  • vagrant WORKS
  • redmine autopkgtest WORKS
  • apt-listbugs FIXED warnings (e.g. try apt-listbugs list $pkg ); caused by ruby-httpclient
  • nanoc WORKS
• capistrano 3.4.0-1 (#795724, #802734) by @sbadia + @terceiro

Is SFLC Shooting Open Source in the Foot? The academic article by SFLC about ZFS is troubling and may unintentionally shoot free software licensing in the foot. When I was at Sun (as part of the team that released the Java Programming Language by starting the OpenJDK project) I often heard community concerns about the CDDL license. At the time the big complaint was about the "Choice of Venue" clause. I got involved because Sun had developed many essential Java libraries and distributed them under CDDL. The community requested a more permissive license and I was able to convince internal project leaders (and Sun's lawyers) to make a licensing change for a handful of these projects. And there was much rejoicing. Based on my experience in helping Java to become open source I came to appreciate the legal hacks on copyright which make open source possible. It's the free software license which uses copyright to enable sharing (vs. the default of disabling sharing).


And so I have appreciated many of the writings and speeches from SFLC on the mechanisms of software freedom. I was particularly moved by the talks about the "Freedom Box" concept. That's why this SFLC post on ZFS sounds so off key: if open source works because of free software licenses it seems weird to weaken that foundation by prioritizing the "equity" (or intended spirit) of the license. Allow me to mention that as I do most of my computing these days on GNU/Linux I miss the super cool features of ZFS from Solaris. I did try an early version of btrfs and was quite disappointed (but that's another story). In this happy case the source code for ZFS is available, but what about the future, when we aren't so lucky and someone asserts in court that the "you know, the software license was really about the spirit of sharing and that means we are allowed to use it -- and not be held to the pesky details as written in the license". A lawyer I respect called this out: "Equity" has no place in US law. The point is that for lawyers software licenses work because they have clear, written rules to guarantee the spirit is upheld; but spirit doesn't work in front of a judge -- clear rules do. Free and open source software has made so much progress in all facets of life why on earth would we second guess the licensing tools that made it possible? And why would SFLC try to shift the spotlight (and in this case the legal burden) to "a good-faith belief that the conduct falls within the equity of the license". Especially given the earlier comment which clearly states "[the combination] is inconsistent with the literal meaning of GPLv2 section 2(b)."
Wat?

The entire raison d' tre for open source software licenses was so that developers (and users) would have clarity and wouldn't have to ask permission to use the software!!! As stated elsewhere (and like I did with those Java libraries) the easy solution is to have the ZFS copyright holder (now Oracle) reclicense (or dual license) the code under a compatible license (permissive or copyleft). If OpenSolaris was still a thing I might understand some hesitancy, but why not liberate ZFS now? So we have to wonder what could possibly be motivating this odd "spirit of the license" position on the part of SFLC? Fortunately charities that enjoy non-profit status are required to make public filings of their income in something called a "Form 990". The latest SFLC 990 I could find shows SFLC getting 78% (or just over $5 million) from "non public support" (see page 14). A number with "two commas" would even be interesting to for-profit companies. Just whom is making these "donations" and what exactly do they get in return? Apparently I'm not the only one wondering about this question. On one hand it's important to know if SFLC as a non-profit is, indeed, acting in the public interest (as the IRS requires). Yet the even bigger issue here is would "asking for a consensus about the spirit" trump the written copyright license and set a scary precedent for open source software in general?

Only one week has passed since the last Debian/TeX Live update, and there is already a new one? The explanation is simple: consolidation of packages and preparation for 2016. The update comprises of uploads of texlive-base, texlive-lang, texlive-extra, texlive-bin, musixtex, pmx, m-tx, xmltex, jadetex. Besides the usual changes it brings a merge back in of several hitherto separately packaged packages. The details are as follows:

• Music related packages (musixtex, pmx, m-tx) have been merged into texlive-music, and temporary upgrade packages for the old packages have been uploaded
• HTML/SGML/XML related packages (jadetex, xmltex) have been merged into the new package texlive-htmlxml, and temporary upgrade packages for the old ones have been uploaded

Furthermore, since some packages ship binaries, texlive-bin has been updated to build the new binaries, too. Lots of changes, but in the long run I see less work and better upgrades. Especially keeping the music-related packages uptodate was a bit a pain, and using the standard TeX Live update mechanisms looks much more promising. And, not to forget, a few more new and updated packages: New packages bibletext, cochineal, hyphen-occitan, libertinegc, lroundrect, miama, mparrows, scrlttr2copy, visualpstricks. Updated packages animate, babel-friulan, babel-russian, bxjscls, caption, celtic, cmtiup, crimson, crossrefware, droit-fr, dvips, fibeamer, fira, fithesis, forest, geschichtsfrkl, gost, hobby, hyperxmp, hyphen-base, hyph-utf8, inconsolata, ipaex-type1, mcf2graph, media9, reledmac, roundrect, showhyphens, spath3, splitindex, unicode-data. Enjoy.

This was my tenth month as a Freexian sponsored LTS contributor. I was assigned 8 hours for the month of February. As I did last month, I worked on updating clamav in wheezy and squeeze-lts. As with previous updates to clamav, we updated it to the new upstream version[1]. As an added complexity, this version bumped soname, so it s now libclamav7 instead of libclamav6. This bump necessitated a small transition in jessie/wheezy-proposed-updates and squeeze-lts. The update for Jessie (included for completeness here) was done early in the month by other pkg-clamav team members. It and the rebuilt/update libclamav reverse-depends will be included in the next Jessie point release. For wheezy, I uploaded libclamunrar (which bumped soname as well) and worked with other pkg-clamav team members on getting clamav to build on sparc and preparing a fix for c-icap. It and the rebuilt/update libclamav reverse-depends will be included in the next Wheezy point release. As a result of the amount of time it took, the squeeze-lts update landed later than I hoped it would, but it is there. As documented in DLA 437-1, there are new packages for clamav, libclamunrar, python-clamav, and klamav. The last squeeze libclamav reverse-depend, dansguardian, took more work, but it too is updated, see DLA 440-1.
[1] The primary reason for this is that anti-virus is an arms race. Unlike other types of packages being stable with only fixes for severe bugs and security issues does not result in a stable capability. It will regress over time. In order to keep up, the new version is needed.

Recently I had a conversation with a programmer who repeated the adage that programming in perl consists of writing line-noise. This isn't true but it reminded me of my love of fuzzers. Fuzzers are often used to generate random input files which are fed to tools, looking for security problems, segfaults, and similar hilarity. To the untrained eye the output of most fuzzers is essentially line-noise, since you often start with a valid input file and start flipping bits, swapping bytes, and appending garbage. Anyway this made me wonder what happens if you fed random garbage into a perl interpreter? I wasn't brave enough to try it, because knowing my luck the fuzzer would write a program like so:

system( "rm -rf /home/steve" );

But I figured it was still an interesting idea, and I could have a go at fuzzing something else. I picked gawk, the GNU implementation of awk because the codebase is pretty small, and I understand it reasonably well. Almost immediately my fuzzer found some interesting segfaults and problems. Here's a nice simple example:

 $ gawk 'for (i = ) in steve kemp rocks'
 ..
 gawk: cmd. line:1: fatal error: internal error: segfault
 Aborted

I look forward to seeing what happens when other people fuzz perl..

This is a bit of an odd posting since it's about something I've done but is also here to help me explain why I did it and thus perhaps encourage some discussion around the topic within the Kicad community...

Recently (as you will know if you follow this blog anywhere it is syndicated) I have started playing with Kicad for the development of some hardware projects I've had a desire for. In addition, some of you may be aware that I used to work for a hardware/software consultancy called Simtec, and there I got to play for a while with an EDA tool called Mentor Designview. Mentor was an expensive, slow, clunky, old-school EDA tool, but I grew to understand and like the workflow. I spent time looking at gEDA and Eagle when I wanted to get back into hardware hacking for my own ends; but neither did I really click with. On the other hand, a mere 10 minutes with Kicad and I knew I had found the tool I wanted to work with long-term. I designed the beer'o'meter project (a flow meter for the pub we are somehow intimately involved with) and then started on my first personal surface-mount project -- SamDAC which is a DAC designed to work with our HiFi in our study at home. As I worked on the SamDAC project, I realised that I was missing a very particular thing from Mentor, something which I had low-level been annoyed by while looking at other EDA tools -- Kicad lacks a mechanism to mark a wire as being linked to somewhere else on the same sheet. Almost all of the EDA tools I've looked at seem to lack this nicety, and honestly I miss it greatly, so I figured it was time to see if I could successfully hack on Kicad. Kicad is written in C++, and it has been mumble mumble years since I last did any C++, either for personal hacking or professionally, so it took a little while for that part of my brain to kick back in enough for me to grok the codebase. Kicad is not a small project, taking around ten minutes to build on my not-inconsiderable computer. And while it beavered away building, I spent time looking around the source code, particularly the schematic editor eeschema. To skip ahead a bit, after a couple of days of hacking around, I had a proof-of-concept for the intra-sheet links which I had been missing from my days with Mentor, and some ERC (electrical rules checking) to go alongside that to help produce schematics without unwanted "sharp corners". In total, I added:

• an "intra-sheet link" schematic element, including:
  • the schematic element itself
  • loading and saving of the element (which required a bump in schematic file format versioning)
  • UI for adding intra-sheet links to your schematic
  • netlist build support to ensure that the links found their way into the internal netlist content.

• an ERC tickybox "Strict wiring checks" and behind that I hid:
  • checks for wires with dangling ends (dangles are bad)
  • junctions which have no point in being present (pointless junctions are confusing to readers)
  • checks that intra-sheet links exist in groups of two or more on any given net, since individual links are pointless.

I forked the Kicad mirror on Github and pushed my own branch with this work to my Kicad fork. All of this is meant to allow schematic capture engineers to more clearly state their intentions regarding what they are drawing. The intra-sheet link could be thought of like a no-connect element, except instead of saying "this explicitly goes nowhere" we're saying "this explicitly goes somewhere else on this sheet, you can go look for it". Obviously, people who dislike (or simply don't want to use) such intra-sheet link elements can just disable that ERC tickybox and not be bothered by them in the least (well except for the toolbar button and menu item I suppose). Whether this work gets accepted into Kicad, or festers and dies on the vine, it was good fun developing it and I'd like to illustrate how it could help you, and why I wrote it in the first place:

A contrived story

Note, while this story is meant to be taken seriously, it is somewhat contrived, the examples are likely electrical madness, but please just think about the purpose of the checks etc.

To help to illustrate the feature and why it exists, I'd like to tell you a somewhat contrived story about Fred. Fred is a schematic capture engineer and his main job is to review schematics generated by his colleagues. Fred and his colleagues work with Kicad (hurrah) but of late they've been having a few issues with being able to cleanly review schematics. Fred's colleagues are not the neatest of engineers. In particular they tend to be quite lazy when it comes to running busses, which are not (for example) address and data busses, around their designs and they tend to simply have wires which end in mid-space and pick up somewhere else on the sheet. All this is perfectly reasonable of course, and Kicad handles it with aplomb. Sadly it seems quite error prone for Fred's workplace. As an example, Fred's colleague Ben has been designing the power supply for a particular board. As with most power supplies, plenty of capacitors are needed to stabilise the regulators and smooth the output. In the example below, the intent is that all of the capacitors are on the FOO net. Sadly there's a missing junction and/or slightly misplaced label in the upper section which means that C2 and C3 simply don't join to the FOO net. This could easily be missed, but the ERC can't spot it at all since there's more than one thing on each net, so the pins of the capacitors are connected to something. Fred is very sad that this kind of problem can sometimes escape notice by the schematic designer Ben, Fred himself, and the layout engineer, resulting in boards which simply do not work. Fred takes it upon himself to request that the strict wiring checks ERC is made mandatory for all designs, and that the design engineers be required to use intra-sheet link symbols when they have signals which wander off to other parts of the sheet like FOO does in the example. Without any further schematic changes, strict wiring checks enabled gives the following points of ERC concern for Ben to think about: As you can see, the ERC is pointing at the wire ends and the warnings are simply that the wires are dangling and that this is not acceptable. This warning is very like the pin-not-connected warnings which can be silenced with an explicit no-connect schematic element. Ben, being a well behaved and gentle soul, obeys the design edicts from Fred and seeks out the intra-sheet link symbols, clearing off the ERC markers and then adding intra-sheet links to his design: This silences the dangling end ERC check, which is good, however it results in another ERC warning: This time, the warning for Ben to deal with is that the intra-sheet links are pointless. Each exists without a companion to link to because of the net name hiccough in the top section. It takes Ben a moment to realise that the mistake which has been made is that a junction is missing in the top section. He adds the junction and bingo the ERC is clean once more: Now, this might not seem like much gain for so much effort, but Ben can now be more confident that the FOO net is properly linked across his design and Fred can know, when he looks at the top part of the design, that Ben intended for the FOO net to go somewhere else on the sheet and he can look for it.

Why do this at all? Okay, dropping out of our story now, let's discuss why these ERC checks are worthwhile and why the intra-sheet link schematic element is needed.

Note: This bit is here to remind me of why I did the work, and to hopefully explain a little more about why I think it's worth adding to Kicad...

Designers are (one assumes) human beings. As humans we (and I count myself here too) are prone to mistakes. Sadly mistakes are often subtle and could easily be thought of as deliberate if the right thought processes are not followed carefully when reviewing. Anyone who has ever done code review, proofread a document, or performed any such activity, will be quite familiar with the problems which can be introduced by a syntactically and semantically valid construct which simply turns out to be wrong in the greater context. When drawing designs, I often end up with bits of wire sticking out of schematic sections which are not yet complete. Sadly if I sleep between design sessions, I often lose track of whether such a dangling wire is meant to be attached to more stuff, or is simply left because the net is picked up elsewhere on the sheet. With intra-sheet link elements available, if I had intended the latter, I'd have just dropped such an element on the end of the wire before I stopped for the day. Also, when drawing designs, I sometimes forget to label a wire, especially if it has just passed through a filter or current-limiting resistor or similar. As such, even with intra-sheet link elements to show me when I mean for a net to go bimbling off across the sheet, I can sometimes end up with unnamed nets whose capacitors end up not used for anything useful. This is where the ERC comes in. By having the ERC complain if a wire dangles -- the design engineer won't forget to add links (or check more explicitly if the wire is meant to be attached to something else). By having junctions which don't actually link anything warned about, the engineer can't just slap a junction blob down on the end of a wire to silence that warning, since that doesn't mean anything to a reviewer later down the line. By having the ERC warn if a net has exactly one intra-sheet link attached to it, missing net names and errors such as that shown in my contrived example above can be spotted and corrected. Ultimately this entire piece of work is about ensuring that the intent of the design engineer is captured clearly in the schematic. If the design engineer meant to leave that wire dangling because it's joining to another bit of wire elsewhere on the sheet, they can put the intra-sheet links in to show this. The associated ERC checks are there purely to ensure that the validation of this intent is not bypassed accidentally, or deliberately, in order to make the use of this more worthwhile and to increase the usefulness of the ERC on designs where signals jump around on sheets where wiring them up directly would just create a mess.

This is based on joint work with Gregor Aisch and Josh Keller of the New York Times. A new package gunsales is now on the CRAN network for R. It is based the NYTimes/gunsales repository underlying the excellent New York Times visualizations, first published first in December 2015 and updated with more recent data since. The analysis takes public government data on gun sales from the National Instant Criminal Background Check System (NICS). The original data is scraped from the pdf, included in the package, and analysed in a cross-section and time-series manner. The standard US Census tool X-13ARIMA-SEATS is used to deseasonalize the timeseries at the national or state level. (Note that Buzzfeed also published data and (Python) code in another GitHub repo.) As an aside, it was the use of X-13ARIMA-SEATS here -- and its somewhat awkward and manual installation also seen in the initial versions of the code in the NYTimes/gunsales repo -- which lead to the recent work by Christoph Sax and myself. We now provide a new package x13binary on CRAN so that Christoph's excellent seasonal package can simply depend upon it and have a working binary provided and installed ready to use; see the recent blog post for more. The net result is that a package like this new gunsales project can simply depend upon seasonal and also be assurred that x13binary "just works". As Martha would say, "A Good Thing". Back to the gunsales project. Following the initial publication of the repository with the data and R code in a simple script, I felt compelled to reorganize it as a package. Packages for R, as we teach our students, colleagues, or anybody else who wants to listen are really the best way to bundle code, data, documentation (i.e. vignettes) and tests. All that exists now in the gunsales package. The package now has one main function, analysis(), which returns a single dataframe object. This dataframe object can then be fed to two plotting functions. The first, plot_gunsales(), will then recreate all the (base R) plots from the original code base. The second, ggplot_gunsales(), does the same but via ggplot2. This should give anybody the ability to look at the data, study the transformations done, form and maybe test new hypotheses and visualize in manner comparable to the original publication. As an amuse gueule, here are the key plots also shown in the main README.md at GitHub: We look forward to more remixes and analysis of this data. The plan of the GitHub repository is to keep the data set updated as new data points are published.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Recently I started getting SMART warnings from on of the disks in my home NAS (a QNAP TS-419P II armel/kirkwood device running Debian Jessie):

Device: /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 [SAT], Self-Test Log error count increased from 0 to 1

Meaning it was now time to switch out that disk from the RAID5 array. Since everytime this happens I have to go and lookup again what to do I've decided to write it down this time. I configure SMART to talk about devices by-id (giving me their name and model number) so first I needed to figure out what the kernel was calling this device (although mdadm is happy with the by-id path, various other bits are not):

# readlink /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 
../../sdd

Next I needed to mark the device as failed in the array:

# mdadm --detail /dev/md0 
/dev/md0:
[...]
          State : clean 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
[...]
    Number   Major   Minor   RaidDevice State
       5       8       48        0      active sync   /dev/sdd
       1       8       32        1      active sync   /dev/sdc
       6       8       16        2      active sync   /dev/sdb
       4       8        0        3      active sync   /dev/sda
# mdadm --fail /dev/md0 /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 
mdadm: set /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 faulty in /dev/md0
# mdadm --detail /dev/md0 
/dev/md0:
[...]
 Active Devices : 3
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 0
[...]
Number   Major   Minor   RaidDevice State
   0       0        0        0      removed
   1       8       32        1      active sync   /dev/sdc
   6       8       16        2      active sync   /dev/sdb
   4       8        0        3      active sync   /dev/sda
   5       8       48        -      faulty   /dev/sdd

If it had been the RAID subsystem rather than SMART monitoring which had first spotted the issue then this would have happened already (and I would had received a different mail from the RAID checks instead of SMART). Once the disk is marked as failed then actually remove it from the array:

# mdadm --remove /dev/md0 /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 
mdadm: hot removed /dev/disk/by-id/ata-ST3000DM001-1CH166_W1F2QSV6 from /dev/md0

And finally tell the kernel to delete the device:

# echo 1 > /sys/block/sdd/device/delete 

At this point I can physically swap the disks. At this point I noticed there were some interesting messages in dmesg, either from the echo to the delete node in sysfs or from the physical switch of the disks:

[1779238.656459] md: unbind<sdd>
[1779238.659455] md: export_rdev(sdd)
[1779258.686720] sd 3:0:0:0: [sdd] Synchronizing SCSI cache
[1779258.700507] sd 3:0:0:0: [sdd] Stopping disk
[1779259.377589] ata4.00: disabled
[1779371.126202] ata4: exception Emask 0x10 SAct 0x0 SErr 0x180000 action 0x6 frozen
[1779371.133740] ata4: edma_err_cause=00000020 pp_flags=00000000, SError=00180000
[1779371.141003] ata4: SError:   10B8B Dispar  
[1779371.145309] ata4: hard resetting link
[1779371.468708] ata4: SATA link down (SStatus 0 SControl 300)
[1779371.474340] ata4: EH complete
[1779557.416735] ata4: exception Emask 0x10 SAct 0x0 SErr 0x4010000 action 0xe frozen
[1779557.424356] ata4: edma_err_cause=00000010 pp_flags=00000000, dev connect
[1779557.431264] ata4: SError:   PHYRdyChg DevExch  
[1779557.436008] ata4: hard resetting link
[1779563.357089] ata4: link is slow to respond, please be patient (ready=0)
[1779567.449096] ata4: SRST failed (errno=-16)
[1779567.453316] ata4: hard resetting link

I wonder if I should have used another method to detach the disk, perhaps poking the controller rather than the disk (which rang a vague bell in my memory from last time this happened) but in the end the disk is broken and the kernel seems to have coped so I'm not too worried about it. It looked like the new disk had already been recognised:

[1779572.593471] scsi 3:0:0:0: Direct-Access     ATA      HGST HDN724040AL A5E0 PQ: 0 ANSI: 5
[1779572.604187] sd 3:0:0:0: [sdd] 7814037168 512-byte logical blocks: (4.00 TB/3.63 TiB)
[1779572.612171] sd 3:0:0:0: [sdd] 4096-byte physical blocks
[1779572.618252] sd 3:0:0:0: Attached scsi generic sg3 type 0
[1779572.626754] sd 3:0:0:0: [sdd] Write Protect is off
[1779572.631771] sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[1779572.632588] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[1779572.665609]  sdd: unknown partition table
[1779572.671522] sd 3:0:0:0: [sdd] Attached SCSI disk
[1779855.362331]  sdd: unknown partition table

So I skipped trying to figure out how to perform a SCSI rescan and went straight to identifying that the new disk was called:

/dev/disk/by-id/ata-HGST_HDN724040ALE640_PK1338P4GY8ENB

and then tried to do a SMART conveyancing self-test with:

# smartctl -t conveyance /dev/disk/by-id/ata-HGST_HDN724040ALE640_PK1338P4GY8ENB

But this particular drive seems to not support that, so I went straight to editing /etc/smartd.conf to replace the old disk with the new one and:

# service smartmontools reload

With all that I was ready to add the new disk to the array:

# mdadm --add /dev/md0 /dev/disk/by-id/ata-HGST_HDN724040ALE640_PK1338P4GY8ENB
mdadm: added /dev/disk/by-id/ata-HGST_HDN724040ALE640_PK1338P4GY8ENB
# mdadm --detail /dev/md0 
/dev/md0:
[...] 
      State : clean, degraded, recovering 
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1
[...]
 Rebuild Status : 0% complete
[...]
Number   Major   Minor   RaidDevice State
   5       8       48        0      spare rebuilding   /dev/sdd
   1       8       32        1      active sync   /dev/sdc
   6       8       16        2      active sync   /dev/sdb
   4       8        0        3      active sync   /dev/sda
# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdd[5] sda[4] sdb[6] sdc[1]
      5860538880 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
      [>....................]  recovery =  0.0% (364032/1953512960) finish=1162.4min speed=28002K/sec

So now all that was left was to wait about 20 hours (with fingers crossed a second disk didn't die! spoiler: it didn't)

I write documentation first and code second. I've mentioned this from time to time (previously, previously) but a reader pointed out that I've never really explained why I work that way. It's a way to make my thinking more concrete without diving all the way into the complexities of the code right away. So sometimes, what I write down is design documentation, and sometimes it's notes on a bug report[1], but if what I'm working on is user-visible, I start by writing down the end user documentation. Writing things down lets me interact with them as words on a page, which are more concrete than muddled thoughts in the head, and much easier to edit and reason about. Code constrains to existing structures; a blank page frees you to explore and build up new ideas. It's the essay writing process, applied to software development, with a side effect of making sure everything is documented. Also, end-user documentation is best when it doesn't assume that the user has any prior knowledge. The point in time when I'm closest to perfect lack of knowledge about something is before I've built it[2]. So, that's the best time to document it. I understand what I'm trying to tell you better now that I've written it down than I did when I started. Hopefully you do too.

---

[1] I'll often write a bug report down even if I have found the bug myself and am going to fix it myself on the same day. (example) This is one place where it's nice to have bug reports as files in the same repository as the code, so that the bug report can be included in the commit fixing it. Often the bug report has lots of details that don't need to go into the commit message, but explain more about my evolving thinking about a problem. [2] Technically I'm even more clueless ten years later when I've totally forgotten whatever, but it's not practical to wait. ;-)

If you bought an North India travel guide by Vanessa Betts and Victoria McCulloch, and tried to figure out where is Phulud and how to get there from Deogarh (and how to get to Deogarh itself from Udaipur), don t waste your time googling, as it s not Phulud, but Phulad. It does seem that the narrow gauge journey from Deogarh to Phulad is indeed beautiful: Meanwhile, I have also found this very interesting post by Mary Anne Erickson: Impressions of India: Udaipur to Deogarh. I m not yet sure we re going to follow that route, but it seems promising. P.S. Despite what the guide said about the airport in Jaisalmer, which is due to open in 2013, according to the reports, it is still not open, so we have to skip that city. Oh well. EDIT: The guide actually also says, on page 10: Fly from Jaisalmer back to Delhi to connect with your flight home. Fact checking? No, who needs that? :)

---

---

Postfix 3.0 recently hit Debian Unstable (and Ubuntu Xenial for those that care about that). It s been a bit of a bumpy road, but it seems to mostly be there for new installs. For package upgrades, there s still issues. We hope to have that sorted shortly, but in the meantime, all you should need to do to get an upgraded system working is add or adjust two parameters in your main.cf shlib_directory=/usr/lib/postfix
daemon_directory=/usr/lib/postfix/sbin You can either edit the file directly or use postconf: postconf -e shlib_directory=/usr/lib/postfix
postconf -e daemon_directory=/usr/lib/postfix/sbin No need to file more bugs and yes, we also know postfix 3.1 was just released. One thing at a time.

Systemd has a new way of specifying names for Ethernet interfaces as documented in systemd.link(5). The Debian package should keep working with the old 70-persistent-net.rules file, but I had a problem with this that forced me to learn about systemd.link(5). Below is a little shell script I wrote to convert a basic 70-persistent-net.rules (that only matches on MAC address) to systemd.link files. #!/bin/bash RULES=/etc/udev/rules.d/70-persistent-net.rules for n in $(grep ^SUB $RULES sed -e s/^.*NAME..// -e s/.$//) ; do
NAME=/etc/systemd/network/10-$n.link
LINE=$(grep $n $RULES)
MAC=$(echo $LINE sed -e s/^.*address .// -e s/ ATTR.*$//)
echo "[Match]" > $NAME
echo "MACAddress=$MAC" >> $NAME
echo "[Link]" >> $NAME
echo "Name=$n" >> $NAME
done

I maintain an application for bridging various non-Hue lighting systems to something that looks enough like a Hue that an Amazon Echo will still control them. One thing I hadn't really worked on was colour support, so I picked up some cheap bulbs and a bridge. The kit is badged as an iSuper iRainbow001, and it's terrible.

Things seemed promising enough at first, although the bulbs were alarmingly heavy (there's a significant chunk of heatsink built into them, which seems to get a lot warmer than I'd expect from something that claims a 7W power consumption). The app was a bit clunky, but eh - I wasn't planning on using it for long. I pressed the button on the bridge, launched the app and could control the bulbs. The first thing I noticed was that they had a separate "white" and "colour" mode. White mode was pretty bright, but colour mode massively less so - presumably the white LEDs are entirely independent of the RGB ones, and much higher intensity. Still, potentially useful as mood lighting.

Anyway. Next step was to start playing with the protocol, which meant finding the device on my network. I checked anything that had picked up a DHCP lease recently and nmapped them. The OS detection reported Linux, which wasn't hugely surprising - there was no GPL notice or source code included with the box, but I'm way past the point of shock at that. It also reported that there was a telnet daemon running. I connected and got a login prompt. And then I typed admin as the username and admin as the password and got a root prompt. So, there's that. The copy of Busybox included even came with tftp, so it was easy to get copies of tcpdump and strace on there to see what was up.

So. Next step. Protocol sniffing. I wanted to know how discovery worked, so reset the device to factory and watched what happens. The app on my phone sent out a discovery packet on UDP port 18602 which looked like this:

INLAN:CLIP:23.21.212.48:CLPT:22345:MAC:02:00:00:00:00:00

The CLIP and CLPT fields refer to the cloud server that allows for management when you're not on the local network. The mac field contains an utterly fake address. If you send out a discovery packet and your mac hasn't been registered with the device, you get a denial back. If your mac has been (and by "mac" here I mean "utterly fake mac that's the same for all devices"), you get back a response including the device serial number and IP address. And if you just leave out the mac field entirely, you get back a response no matter whether your address is registered or not. So, that's a start. Once you've registered one copy of the app with the device, anything can communicate with it by just using the same fake mac in the discovery packets.

Onwards. The command format turns out to be simple. They start ##, are followed by two ascii digits encoding a command, four ascii digits containing a bulb id, two ascii digits containing the number of following bytes and then the command data (in ascii). An example is:

##05010002ff

which decodes as command 5 (set white intensity) on bulb 1 with two bytes of data following, each of which is an f. This translates as "Set bulb 1 to full white intensity". I worked out the rest pretty quickly - command 03 sets the RGB colour of the bulb, 0A asks the bridge to search for new bulbs, 0B tells you which bulbs are available and their capabilities and 0E gives you the MAC addresses of the bulbs( ). 0C crashes the server process, and 06 spews a bunch of garbage back at you and potentially crashes the bulb in a hilarious way that involves it flickering at about 15Hz. It turns out that 06 is actually the "Rename bulb" command, and if you send it less data than it's expecting something goes hilariously wrong in string parsing and everything is awful.

Ok. Easy enough, but not hugely exciting. What about the remote protocol? This turns out to involve sending a login packet and then a wrapped command packet. The login has some length data, a header of "MQIsdp", a long bunch of ascii-encoded hex, a username and a password.

The username is w13733 and the password is gbigbi01. These are hardcoded in the app. The ascii-encoded hex can be replaced with 0s and the login succeeds anyway.

Ok. What about the wrapping on the command? The login never indicated which device we wanted to talk to, so presumably there's some sort of protection going on here oh wait. The command packet is a length, the serial number of the bridge and then a normal command packet. As long as you know the serial number of the device (which it tells you in response to a discovery packet, even if you're unauthenticated), you can use the cloud service to send arbitrary commands to the device (including the one that crashes the service). One of which involves the device then doing some kind of string parsing that doesn't appear to work that well. What could possibly go wrong?

Ok, so that seemed to be the entirety of the network protocol. Anything else to do? Some poking around on the bridge revealed (a) that it had an active wireless device and (b) a running DHCP server. They wouldn't, would they?

Yes. Yes, they would.

The devices all have a hardcoded SSID of "iRainbow", although they don't broadcast it. There's no security - anybody can associate. It'll then hand out an IP address. It's running telnetd on that interface as well. You can bounce through there to the owner's internal network.

So, in summary: it's a device that infringes my copyright, gives you root access in response to trivial credentials, has access control that depends entirely on nobody ever looking at the packets, is sufficiently poorly implemented that you can crash both it and the bulbs, has a cloud access protocol that has no security whatsoever and also acts as an easy mechanism for people to circumvent your network security. This may be the single worst device I've ever bought.

I called the manufacturer and they insisted that the device was designed in 2012, was no longer manufactured or supported, that they had no source code to give me and there would be no security fixes. The vendor wants me to pay for shipping back to them and reserves the right to deduct the cost of the original "free" shipping from the refund. Everything is awful, which is why I just ordered four more random bulbs to play with over the weekend.

comments

Without further words:

% for i in $(seq 1 90) ; do doveadm mailbox status messages debian.buildd.archive.2011.05   column -t ;  doveadm deduplicate mailbox debian.buildd.archive.2011.05 ; done
debian.buildd.archive.2011.05  messages=8094
debian.buildd.archive.2011.05  messages=7939
debian.buildd.archive.2011.05  messages=7816
debian.buildd.archive.2011.05  messages=7698
debian.buildd.archive.2011.05  messages=7610
debian.buildd.archive.2011.05  messages=7529
debian.buildd.archive.2011.05  messages=7455
debian.buildd.archive.2011.05  messages=7375
debian.buildd.archive.2011.05  messages=7294
debian.buildd.archive.2011.05  messages=7215
debian.buildd.archive.2011.05  messages=7136
debian.buildd.archive.2011.05  messages=7032
debian.buildd.archive.2011.05  messages=6941
debian.buildd.archive.2011.05  messages=6839
debian.buildd.archive.2011.05  messages=6721
debian.buildd.archive.2011.05  messages=6631
debian.buildd.archive.2011.05  messages=6553
debian.buildd.archive.2011.05  messages=6476
debian.buildd.archive.2011.05  messages=6388
debian.buildd.archive.2011.05  messages=6301
debian.buildd.archive.2011.05  messages=6211
debian.buildd.archive.2011.05  messages=6140
debian.buildd.archive.2011.05  messages=6056
debian.buildd.archive.2011.05  messages=6007
debian.buildd.archive.2011.05  messages=5955
debian.buildd.archive.2011.05  messages=5887
debian.buildd.archive.2011.05  messages=5826
debian.buildd.archive.2011.05  messages=5752
debian.buildd.archive.2011.05  messages=5706
debian.buildd.archive.2011.05  messages=5657
debian.buildd.archive.2011.05  messages=5612
debian.buildd.archive.2011.05  messages=5570
debian.buildd.archive.2011.05  messages=5523
debian.buildd.archive.2011.05  messages=5474
debian.buildd.archive.2011.05  messages=5422
debian.buildd.archive.2011.05  messages=5382
debian.buildd.archive.2011.05  messages=5343
debian.buildd.archive.2011.05  messages=5308
debian.buildd.archive.2011.05  messages=5256
debian.buildd.archive.2011.05  messages=5221
debian.buildd.archive.2011.05  messages=5168
debian.buildd.archive.2011.05  messages=5133
debian.buildd.archive.2011.05  messages=5092
debian.buildd.archive.2011.05  messages=5058
debian.buildd.archive.2011.05  messages=5030
debian.buildd.archive.2011.05  messages=4994
debian.buildd.archive.2011.05  messages=4964
debian.buildd.archive.2011.05  messages=4935
debian.buildd.archive.2011.05  messages=4900
debian.buildd.archive.2011.05  messages=4868
debian.buildd.archive.2011.05  messages=4838
debian.buildd.archive.2011.05  messages=4811
debian.buildd.archive.2011.05  messages=4778
debian.buildd.archive.2011.05  messages=4748
debian.buildd.archive.2011.05  messages=4722
debian.buildd.archive.2011.05  messages=4686
debian.buildd.archive.2011.05  messages=4661
debian.buildd.archive.2011.05  messages=4637
debian.buildd.archive.2011.05  messages=4613
debian.buildd.archive.2011.05  messages=4593
debian.buildd.archive.2011.05  messages=4570
debian.buildd.archive.2011.05  messages=4554
debian.buildd.archive.2011.05  messages=4536
debian.buildd.archive.2011.05  messages=4520
debian.buildd.archive.2011.05  messages=4500
debian.buildd.archive.2011.05  messages=4481
debian.buildd.archive.2011.05  messages=4466
debian.buildd.archive.2011.05  messages=4445
debian.buildd.archive.2011.05  messages=4430
debian.buildd.archive.2011.05  messages=4417
debian.buildd.archive.2011.05  messages=4405
debian.buildd.archive.2011.05  messages=4390
debian.buildd.archive.2011.05  messages=4376
debian.buildd.archive.2011.05  messages=4366
debian.buildd.archive.2011.05  messages=4360
debian.buildd.archive.2011.05  messages=4350
debian.buildd.archive.2011.05  messages=4336
debian.buildd.archive.2011.05  messages=4329
debian.buildd.archive.2011.05  messages=4320
debian.buildd.archive.2011.05  messages=4315
debian.buildd.archive.2011.05  messages=4312
debian.buildd.archive.2011.05  messages=4311
debian.buildd.archive.2011.05  messages=4309
debian.buildd.archive.2011.05  messages=4308
debian.buildd.archive.2011.05  messages=4308
debian.buildd.archive.2011.05  messages=4308
debian.buildd.archive.2011.05  messages=4308
debian.buildd.archive.2011.05  messages=4308
debian.buildd.archive.2011.05  messages=4308
debian.buildd.archive.2011.05  messages=4308

Given the HA stack is now in very good shape in Debian stretch [testing by now] we can start playing a bit with these tools.

There are 2 main points in doing so now:

1. reporting bugs
2. systemd integration

Packages
Basic configurationYou will need to edit the /etc/corosync/corosync.conf file:


Note that if you choose the proper bindnetaddr directive, you can use the exact same file for all the nodes (see corosync.conf(5))

System, daemons, servicesYou may want to check that the corosync and pacemaker services are auto-started at boot time by systemd.
Do a reboot and check it:


Also, check that no firewall blocks the communication between both nodes.

Now, the status of the cluster should show both nodes working together (with no resources):


ResourcesNow, you can start playing with the cluster resources. Below is an example of a virtual IPv6 address:

$ sudo crm configure primitive test ocf:heartbeat:IPv6addr params ipv6addr="fe00::200" cidr_netmask="64" nic="eth0"

Which should give this cluster status:


Congratulations, now you have a working HA cluster with a virtual IP address, using an active/backup approach.

Please, report any bug you may find.

Since several years ago we've been using php-gettext in phpMyAdmin, but it's about time to change it. The change will not be that big, we're just moving to own fork of that library :-). You probably ask why to fork? You will probably guess it easily, but to name some reasons:

• The php-gettext library is not maintained anymore
• It doesn't work with recent PHP version (phpMyAdmin has patched version)
• It's not possible to install it using Composer
• There was place for performance improvements in the library

So I've taken existing php-gettext codebase and turned that into motranslator and now 0.1 release of it is out. The recommended way to install it is from Packagist and it has no additional dependencies. What changes you can expect? First of all it supports all current PHP versions. It also performs way better - in my tests loading of mo file is 4-5 times faster and memory consumption went down about 10 percent. You can additionally use object API instead of traditional function based. On the other side some features we don't need were removed - there is no support for using native Gettext, it doesn't do any encoding conversion (assuming that UTF-8 is on both sides these days) and it doesn't support delayed loading of messages. The last change means that it's not suitable for applications with huge MO files. Any feedback is welcome, the code is still fresh and probably needs some polishing.

Filed under: English phpMyAdmin 0 comments