After two pre-releases in the last few days, Conrad finalised a new Armadillo version 4.400 today. I had kept up with the pre-releases, tested twice against all eighty (!!) CRAN dependents of RcppArmadillo and have hence uploaded RcppArmadillo 0.4.400.0 to CRAN and into Debian. This release brings a number of new upstream features which are detailed below. As included is s bugfix for sparse matrix creation at the RcppArmadillo end which was found by the ASAN tests at CRAN --- which are similar to the sanitizers tests I recently blogged. I was able to develop and test the fix in the very docker r-devel-san images I had written about which was nice. Special thanks also to Ryan Curtin for help with the fix.

Changes in RcppArmadillo version 0.4.400.0 (2014-08-19)

• Upgraded to Armadillo release Version 4.400 (Winter Shark Alley)
  • added gmm_diag class for statistical modelling using Gaussian Mixture Models; includes multi-threaded implementation of k-means and Expectation-Maximisation for parameter estimation
  • added clamp() for clamping values to be between lower and upper limits
  • expanded batch insertion constructors for sparse matrices to add values at repeated locations
  • faster handling of subvectors by dot()
  • faster handling of aliasing by submatrix views
• Corrected a bug (found by the g++ Address Sanitizer) in sparse matrix initialization where space for a sentinel was allocated, but the sentinel was not set; with extra thanks to Ryan Curtin for help
• Added a few unit tests for sparse matrices

Courtesy of CRANberries, there is also a diffstat report for the most recent release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Today I did something interesting with the Debian packaging for propellor, which seems like it could be a useful technique for other Debian packages as well. Propellor is configured by a directory, which is maintained as a local git repository. In propellor's case, it's ~/.propellor/. This contains a lot of haskell files, in fact the entire source code of propellor! That's really unusual, but I think this can be generalized to any package whose configuration is maintained in its own git repository on the user's system. For now on, I'll refer to this as the config repo. The config repo is set up the first time a user runs propellor. But, until now, I didn't provide an easy way to update the config repo when the propellor package was updated. Nothing would break, but the old version would be used until the user updated it themselves somehow (probably by pulling from a git repository over the network, bypassing apt's signature validation). So, what I wanted was a way to update the config repo, merging in any changes from the new version of the Debian package, while preserving the user's local modifications. Ideally, the user could just run git merge upstream/master, where the upstream repo was included in the Debian package. But, that can't work! The Debian package can't reasonably include the full git repository of propellor with all its history. So, any git repository included in the Debian binary package would need to be a synthetic one, that only contains probably one commit that is not connected to anything else. Which means that if the config repo was cloned from that repo in version 1.0, then when version 1.1 came around, git would see no common parent when merging 1.1 into the config repo, and the merge would fail horribly. To solve this, let's assume that the config repo's master branch has a parent commit that can be identified, somehow, as coming from a past version of the Debian package. It doesn't matter which version, although the last one merged with will be best. (The easy way to do this is to set refs/heads/upstream/master to point to it when creating the config repo.) Once we have that parent commit, we have three things:

1. The current content of the config repo.
2. The content from some old version of the Debian package.
3. The new content of the Debian package.

Now git can be used to merge #3 onto #2, with -Xtheirs, so the result is a git commit with parents of #3 and #2, and content of #3. (This can be done using a temporary clone of the config repo to avoid touching its contents.) Such a git commit can be merged into the config repo, without any conflicts other than those the user might have caused with their own edits. So, propellor will tell the user when updates are available, and they can simply run git merge upstream/master to get them. The resulting history looks like this:

* Merge remote-tracking branch 'upstream/master'
 \  
  * merging upstream version
   \  
    * upstream version
*   user change
 /  
* upstream version

So, generalizing this, if a package has a lot of config files, and creates a git repository containing them when the user uses it (or automatically when it's installed), this method can be used to provide an easily mergable branch that tracks the files as distributed with the package. It would perhaps not be hard to get from here to a full git-backed version of ucf. Note that the Debian binary package doesn't have to ship a git repisitory, it can just as easily ship the current version of the config files somewhere in /usr, and check them into a new empty repository as part of the generation of the upstream/master branch.

I've switched my Git repositories to GitHub recently, and started to watch my contributions statistics, which were very low considering I spend my days hacking on open source software, especially OpenStack.

OpenStack hosts its Git repositories on its own infrastructure at git.openstack.org, but also mirrors them on GitHub. Logically, I was expecting GitHub to track my commits there too, as I'm using the same email address everywhere. It turns out that it was not the case, and the help page about that on GitHub describes the rule in place to compute statistics. Indeed, according to GitHub, I had no relations to the OpenStack repositories, as I never forked them nor opened a pull request on them (OpenStack uses Gerrit). Starring a repository is enough to build a relationship between a user and a repository, so this is was the only thing needed to inform GitHub that I have contributed to those repositories. Considering OpenStack has hundreds of repositories, I decided to star them all by using a small Python script using pygithub. And voil , my statistics are now including all my contributions to OpenStack!

For a while, I've been wanting to set up configuration management for my home network. With half a dozen servers, a VPS and a workstation it is not big, but large enough to make it annoying to manually log into each machine for network-wide changes. Most of the servers I have are low-end ARM machines, each responsible for a couple of tasks. Most of my machines run Debian or something derived from Debian. Oh, and I'm a member of the declarative school of configuration management.

A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning. In this article, I would like to relate what I've been doing with other Ceilometer developers in the last 5 months. I've lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it's largely time to take a break and talk about it. Ceilometer early design For the last years, Ceilometer didn't change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called samples. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster ) constructs and emits a sample headed for the storage component that we call the collector. This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support. The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want. The scalability issue We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each sample generated by Ceilometer and is stored as is). That basically means that the sample data structure is stored in most drivers in just one table or collection, in order to be able to scan them at once, and there's no good "perfect" sharding solution, making data storage scalability painful. It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are O(n) where n is the number of samples recorded (see big O notation if you're unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable. During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.

Thinking outside the box Unfortunately, the PTL job doesn't leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn't able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box. When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics. Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built into oslo.messaging. Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth usage, the number of images that Glance is storing for you, etc These are not events, since nothing is happening. These are facts, states we need to meter. Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right. I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there's still no silver bullet this made it a good challenge. The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API as we do in all OpenStack services. This should cover the metric storage pretty well. Cooking Gnocchi

A cloud of gnocchis!

At the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a "pasta" name, right? The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process. The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I've named these things "entities", as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn't care and should be agnostic in this regard. One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the RRDtool have been doing for a long time on the fly, and I decided it was a good trail to follow. I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both. The first driver written to achieve that and store those metrics inside Gnocchi was based on whisper. Whisper is the file format used to store metrics for the Graphite project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable. Storing metrics for each entities in a different whisper file and putting them in Swift turned out to have a fantastic algorithm complexity: it was O(1). Indeed, the complexity needed to store and retrieve metrics doesn't depends on the number of metrics you have nor on the number of things you are metering. Which is already a huge win compared to the current Ceilometer collector design. However, it turned out that whisper has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (time.time()). I've started to hack on that in my own fork, but then everything broke. The whisper project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform whisper into the time-series format I wanted, without being sure I wasn't going to break everything (remember, no test coverage). I decided to take a break and look into alternatives, and stumbled upon Pandas, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it carbonara (a wink to both the Carbon project and pasta, how clever!). The code is quite small (a third of whisper's, 200 SLOC vs 600 SLOC), does not have many of the whisper limitations and it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers. Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It's a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a InfluxDB driver, working closely with the upstream developer of that database. Dina Belova started to write an OpenTSDB driver. This helps to make sure the API is designed directly in the right way. Handling resources Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.

Relationship of entities and resources

Once this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now. Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network. All of these requirements led to to the design of what's called the indexer. The indexer is responsible for indexing entities, resources, and link them together. The initial implementation is based on SQLAlchemy and should be pretty efficient. It's easy enough to index the most requested attributes (columns), and they are also correctly typed. We plan to establish a model for all known OpenStack resources (instances, volumes, networks, ) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It'd be up to the users to store extra attributes. Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.

The indexer classes and their relations

REST API All of this is exported via a REST API that was partially designed and documented in the Gnocchi specification in the Ceilometer repository; though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer. The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.

Macroscopic view of the Gnocchi architecture

Roadmap & Ceilometer integration All of this plan has been exposed and discussed with the Ceilometer team during the last OpenStack summit in Atlanta in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue. It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to. Early benchmarks Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July. The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.

For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of O(1), and not O(n) anymore. Next steps

Cl ment drawing the logo

While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now. The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We'll discuss how to integrate it in the project in a more permanent and strong manner probably during the OpenStack Summit for Kilo that will take place next November in Paris.

Now that I've had my solar generation system for a little while, I thought I'd write a follow up post on how it's all going. Energex came out a week ago last Saturday and swapped my electricity meter over for a new digital one that measures grid consumption and excess energy exported. Prior to that point, it was quite fun to watch the old analog meter going backwards. I took a few readings after the system was installed, through to when the analog meter was disconnected, and the meter had a value 26 kWh lower than when I started. I've really liked how the excess energy generated during the day has effectively masked any relatively small overnight power consumption. Now that I have the new digital meter things are less exciting. It has a meter measuring how much power I'm buying from the grid, and how much excess power I'm exporting back to the grid. So far, I've bought 32 kWh and exported 53 kWh excess energy. Ideally I want to minimise the excess because what I get paid for it is about a third of what I have to pay to buy it from the grid. The trick is to try and shift around my consumption as much as possible to the daylight hours so that I'm using it rather than exporting it. On a good day, it seems I'm generating about 10 kWh of energy. I'm still impatiently waiting for PowerOne to release their WiFi data logger card. Then I'm hoping I can set up something automated to submit my daily production to PVOutput for added geekery.

Update 2014-08-20: apcid needs tweaking. See update section below. Somehow both my regular work laptop and home entertainment computers (both running Debian Jessie) were switched to systemd without me noticing. Judging from by dpkg.log it may have happened quite a while ago. I'm pretty sure that's a compliment to the backwards compatibility efforts made by the systemd developers and a criticism of me (I should be paying closer attention to what's happening on my own machines!). In any event, I've started trying to pay more attention now - particularly learning how to take advantage of this new software. I'll try to keep this blog updated as I learn. For now, I have made a few changes and discoveries. First - I have a convenient bash wrapper I use that both starts my OpenVPN client to a samba server and then mounts the drive. I only connect when I need to and rarely do I properly disconnect (the OpenVPN connection does not start automatically). So, I've written the script to carefully check if my openvpn connection is present and either restart or start depending on the results. I had something like this:

if ps -eFH   egrep [o]penvpn; then
  sudo /etc/init.d/openvpn restart
else
  sudo /etc/init.d/openvpn start
fi

One of the advertised advantages of systemd is the ability to more accurately detect if a service is running. So, first I changed this to:

if systemctl -q is-active openvpn.service; then
  sudo systemctl restart openvpn.service
else
  sudo systemctl start openvpn.service
fi

However, after reviewing the man page I realized I can shorten if further to simply:

  sudo systemctl restart openvpn.service

According to the man page, restart means:

Restart one or more units specified on the command line. If the units are not
running yet, they will be started.

After discovering this meaning for "restart" in systemd, I tested and realized that "restart" works the same way for openvpn using the old sysv style init system. Oh well. At least there's a man page and a stronger guarantee that it will work with all services, not just the ones that happen to respect that convention in their init.d scripts. The next step was to disable openvpn on start up. I confess, I never bothered to really learn update-rc.d. Everytime I read the manpage I ended up throwing up my hands and renaming symlinks by hand. In the case of openvpn I had previously edited /etc/default/openvpn to indicate that "none" of the virtual private networks should be started. Now, I've returned that file to the default configuration and instead I ran:

systemctl disable openvpn.service

UPDATES 2014-08-20: I've recently noticed strange behavior when I wake my laptop. Seems to sometimes go right back to sleep. After doing some digging I traced the problem to some customizations I have made to my laptop's acpid behavior combined with systemd taking over some apci events. Up to now, I have created my own /etc/acpi files so I have full control over the acpi events. In particular, I don't want my laptop to suspend when I close the lid. I only want it to suspend when I press the suspend key. And, when it does suspend, I want it to run my own personal suspend script so I can do things like lock the screen and restart tint2. I've found that systemd launches it's own acpi event monitoring that ignores my /etc/acpid rules (the systemd "unit" that monitors events is called acpid.socket which exists in addition to acpid.service). The systemd reaction to events is controlled by the systemd-logind.service which has a configuration file: /etc/systemd/logind.conf. By default, systemd-logind.service will put my laptop to sleep when the lid is closed and when the suspend button is pushed. systemd seems to get the signal first, putting the laptop to sleep. After I wake it up, acpid gets the signal - so it goes right back to sleep. Reading man logind.conf helps. I was able to restore my desired behavior by adding these lines to /etc/systemd/logind.conf:

HandleSuspendKey=ignore
HandleLidSwitch=ignore

Then: sudo systemctl restart systemd-logind.service.

Yet another update from my internship at Mozilla, as part of the OPW. An online triage workshop One of the most interesting thing I've done during the last weeks has been to held an online Bug Triage Workshop on the #testday channel at irc.mozilla.org.
That was a first time for me: I had been a moderator for a series of training sessions on IRC organized by Debian Women, but never a "speaker".
The experience turned out to be a good one: creating the material for the workshop had me basically summarize (not too much, I'm way too verbose!) all what I've learned in this past months about triaging in Mozilla, and speaking of it on IRC was a sort of challenge to my usual shyness. And I was so very lucky that a participant was able to reproduce the bug I picked as example, thus confirming it! How cool is that? ;) The workshop was about the very basics of triaging for Firefox, and we mostly focused on a simplified lifecycle of bugs, a guided tour of bugzilla (including the quicksearch and the advanced one, the list view, the individual bug view) and an explanation of the workflow of the triager. I still have my notes, and I plan to upload them to the wiki, sooner or later. I'm pretty satisfied of the outcome: the only regret is that the promoting wasn't enough, so we have few participants.
Will try to promote it better next time! :) about:crashes Another thing that had me quite busy in the last weeks was to learn more about crashes and stability in general.
If you are unfortunate enough to experience a crash with Firefox, you're probably familiar with the Mozilla Crash Reporter dialog box asking you to submit the crash report. But how does it works? From the client-side, Mozilla uses Breakpad as set of libraries for crash reporting. The Mozilla specific implementation adds to that a crash-reporting UI, a server to collect and process crash reported data (and particularly to convert raw dumps into readable stack traces) and a web interface, Socorro to view and parse crash reports. Curious about your crashes? The about:crashes page will show you a list of the submitted and unsubmitted crash reports. (And by the way, try to type about:about in the location bar, to find all the super-secret about pages!) For the submitted ones clicking on the CrashID will take you to the crash report on crash-stats, the website where the reports are stored and analyzed. The individual crash report page on crash-stats is awesome: it shows you the reported bug numbers if any bug summaries match the crash signature, as well as many other information. If crash-stats does not show a bug number, you really should file one! The CrashKill team works on these reports tracking the general stability of the various channels, triaging the top crashes, ensuring that the crash bugs have enough information and are reproducible and actionable by the devs.
The crash-stats site is a mine of information: take a look at the Top Crashes for Firefox 34.0a1.
If you click on a individual crash, you will see lots of details about it: just on the first tab ("Signature Summary") you can find a breakdown of the crashes by OS, by graphic vendors or chips or even by uptime range.
A very useful one is the number of crashes per install, so that you know how widespread is the crashing for that particular signature. You can also check the comments the users have submitted with the crash report, on the "Comments" tab. One and Done tasks review Last week I helped the awesome group of One and Done developers, doing some reviewing of the tasks pages. One and Done is a brilliant idea to help people contribute to the QA Mozilla teams.
It's a website proposing the user a series of tasks of different difficulty and on different topics to contribute to Mozilla. Each task is self-contained and can last few minutes or be a bit more challenging. The team has worked hard on developing it and they have definitely done an awesome job! :) I'm not a coding person, so I just know that they're using Django for it, but if you are interested in all the dirty details take a look at the project repository. My job has been only to check all the existent tasks and verify that the description and instruction are correct, that the task is properly tagged and so on. My impression is that this an awesome tool, well written and well thought with a lot of potential for helping people in their first steps into Mozilla. Something that other projects should definitely imitate (cough Debian cough). What's next? Next week I'll be back on working on bugs. I kind of love bugs, I have to admit it. And not squashing them: not being a coder make me less of a violent person toward digital insects. Herding them is enough for me. I'm feeling extremely non-violent toward bugs. I'll try to help Liz with the Test Plan for Firefox 34, on the triaging/verifying bugs part.
I'll also try to triage/reproduce some accessibility bugs (thanks Mario for the suggestion!).

The GnuTLS28 transition is making progress, more than 60% done: Thanks to a national holiday combined with rainy weather this should look even better soon:
ametzler@argenau:~$ w3m -dump https://ftp-master.debian.org/deferred.html grep changes wc -l
26
ametzler@argenau:~$ w3m -dump https://ftp-master.debian.org/deferred.html grep Metzler wc -l
18


Firstly, thanks to all who responded to my previous rant. It turns out exactly what I wanted does exist in the form of a ID-000 format smartcard combined with a USB reader. Perfect. No idea why I couldn t find that on my own prior to ranting, but very happy to have found it now. Secondly, now that I ve got my keys and management practices in order, it is time to begin transitioning to my new key. Click this link to find the properly signed, full transition statement. I m not going to paste the full statement into this post, but my new key is:

pub   4096R/A48F065A 2014-07-27 [expires: 2016-07-26]
      Key fingerprint = DBB7 64A1 797D 2E21 C799  3CD7 A628 CB5F A48F 065A
uid                  Matthew Brown <matt @mattb.net.nz>
uid                  Matthew Brown <mattb @debian.org>
sub   4096R/1937883F 2014-07-27 [expires: 2016-07-26]

If you signed my old key, I d very much appreciate a signature on my new key, details and instructions in the transition statement. I m happy to reciprocate if you have a similarly signed transition statement to present.

Hi Everybody, So I already explained that we decided to go in a new direction and patch ucf to allow automatic configuration file merging with any custom command. Today I wanted to explain how to use ucf s new three-way-merge-command functionality in
conjunction with Elektra in order to ultilize Elektra s powerful tools in order to allow automatic three-way merges of your package s configuration during upgrades in a way that is more reliable than a diff3 merge. This guide assumes that you are fimilar with ucf already and are just trying to impliment the --three-way-merge-command option using Elektra. The addition of the --three-way-merge-command option was a part of my Google Summer of Code Project. This option takes the form:

--three-way-merge-command command [New File] [Destination]

Where command is the command you would like to use for the merge. New File and Destination are the same as always. We added a new script to Elektra called elektra-merge for use with this new option in ucf. This script acts as a liaison between ucf and Elektra, allowing a regular ucf command to run a kdb merge even though ucf commands only pass New File and Destination whereas kdb merge requires ourpath, theirpath, basepath, and resultpath. Since ucf already performs a three-way merge, it keeps track of all the necessary files to do so, even though it only takes in New File and Destination. In order to use elektra-merge, the current configuration file must be mounted to KDB to serve as ours in the merge. The script automatically mounts theirs, base, and result using the kdb remount command in order to use the same backend as ours (since all versions of the same file should use the same backend anyway) and this way users don t need to worry about specifying the backend for each version of the file. Then the script attempts a merge on the newly mounted KeySets. Once this is finished, either on success or not, the script finishes by unmouting all but our copy of the file to cleanup KDB. Then, if the merge was successful ucf will replace ours with the result providing the package with an automatically merged configuration which will also be updated in KDB itself. Additionally, we added two other scripts, elektra-mount and elektra-umount which act as simple wrappers for kdb mount and kdb umount. That work identically but are more script friendly The full command to use elektra-merge to perform a three-way merge on a file managed by ucf is:

ucf --three-way --threeway-merge-command elektra-merge [New File] [Destination]

Thats it! As described above, elektra-merge is smart enough to run the whole merge off of the information from that command and utilizes the new kdb remount command to do so. Integrating elektra-merge into a package that already uses ucf is very easy! In postinst you should have a line similar to:

ucf [New File] [Destination]

or perhaps:

ucf --three-way [New File] [Destination]
All you must do is in postinst, when run with the configure option you must mount the config file to Elektra:

kdb elektra-mount [New File] [Mounting Destination] [Backend]
Next, you must update the line containing ucf with the options --three-way and --threeway-merge-command like so:

ucf --three-way --threeway-merge-command elektra-merge [New File] [Destination]
Then, in your postrm script, during a purge, you must umount the config file before deleting it:

kdb lektra-umount [Name]
That s it! With those small changes you can use Elektra to perform automatic three-way merges on any files that your package uses ucf to handle! I just wanted to show a quick example, below is a diff representing the changes we made to the samba-common package in order to allow automatic configuration merging for smb.conf using Elektra. We chose this package because it already
uses ucf to handle smb.conf but it frequently requires users to manually merge changes across versions. Here is the patch showing what we changed:

diff samba_orig/samba-3.6.6/debian/samba-common.postinst samba/samba-3.6.6/debian/samba-common.postinst
92c92,93
< ucf --three-way --debconf-ok "$NEWFILE" "$CONFIG"
---
> kdb elektra-mount "$CONFIG" system/samba/smb ini
> ucf --three-way --threeway-merge-command elektra-merge --debconf-ok "$NEWFILE" "$CONFIG"
Only in samba/samba-3.6.6/debian/: samba-common.postinst~
diff samba_orig/samba-3.6.6/debian/samba-common.postrm samba/samba-3.6.6/debian/samba-common.postrm
4a5
> kdb elektra-umount system/samba/smb
As you can see, all we had to do was add the line to mount smb.conf during install, update the ucf command to include the new --threeway-merge-command option, and unmount system/samba/smb during a purge. It really is that easy! Sincerely,
Ian S. Donnelly

Loled by MadameZou (inspired by this tweet)

There hasn t been a progress-report on DEP-11 for some time, but that doesn t mean there was no work going on on it. DEP-11 is Debian s implementation of AppStream, as well as an effort to enhance the metadata available about software in Debian. While initially, AppStream was only about applications, DEP-11 was designed with a larger scope, to collect data about libraries, binaries and things like Python modules. Now, since AppStream 0.6, DEP-11 and AppStream have essentially the same scope, with the difference of DEP-11 metadata being described in YAML, while official AppStream data is XML. That was due to a request by our ftpmasters team, which doesn t like XML (which is also not used anywhere in Debian, as opposed to YAML). But this doesn t mean that people will have to deal with the YAML file format: The libappstream library will just take DEP-11 data as another data source for it s Xapian database, allowing anything using libappstream to access that data just like the XML stuff. Richards libappstream-glib will also receive support for the DEP-11 format soon, filling it s in-memory data cache and enabling the use of GNOME-Software on Debian. So, what has been done so far? The past months, my Google Summer of Code student. Abhishek Bhattacharjee, was working hard to integrate DEP-11 support into dak, the Debian Archive Kit, which maintains the whole Debian archive. The result will be an additional metadata table in our internal Postgres database, storing detailed information about the software available in a Debian package, as well as Components-<arch>.yml.gz files in the Debian repositories. Dak will also produce an application icon-cache and a screenshots repository. During the time of the SoC, Abhishek focused mainly on the applications part of things, and less on the other components (like extracting data about Python modules or libraries) these things can easily be implemented later. The remaining steps will be to polish the code and make it merge-ready for Debian s dak (as soon as it has received enough testing, we will likely give it a try on the Tanglu Debian derivative). Following that, Apt will be extended to fetch the DEP-11 data on-demand on systems where it is useful (which is currently mostly desktop-systems) if you want to save a little bit of space, you will be able to disable downloading this extra metadata in Apt. From there, libappstream will take the data for it s Xapian db. This will lead to the removal of the much-hated (from ftpmasters and maintainers side) app-install-data package, which has not been updated for two years and only contains a small fraction of the metadata provided by DEP-11. What Debian will ultimately gain from this effort is support for software-centers like GNOME-Software, and improved support for tools like Apper and Muon in displaying applications. Long-term, with more metadata being available, It would be cool to add support for it to specialized package managers , like Python s pip, npm or gem, to make them fetch information about available distribution software and install that instead of their own copies from 3rd-party repositories, if possible. This should ultimatively lead to less code duplication on distributions and will likely result in fewer security issues, since the officially maintained and integrated distribution packages can easily be used, if possible. This is no attempt to make tools like pip obsolete, but an attempt to have the different tools installing software on your machine communicate better, instead of creating parallel worlds in terms of software management. Another nice sideeffect of more metadata will be options to search for tools handling mimetypes in the software repos (in case you can t open a file), smart software centers installing missing firmware, and automatic suggestions for developers which software they need to install in order to build a specific software package. Also, the data allows us to match software across distributions, for that, I will have some news soon (not sure how soon though, as I am currently in thesis-writing-mode, and therefore have not that much spare time). Since the goal is to have these features available on all distributions supporting AppStream, it will take longer to realize but we are on a good way. So, if you want some more information about my student s awesome work, you can read his blogpost about it. He will also be at Debconf 14 (Portland). (I can t make it this time, but I surely won t miss the next Debconf) Sadly, I only see a very small chance to have the basic DEP-11 stuff land in-time for Jessie (lots of review work needs to be done, and some more code needs to be written), but we will definitively have it in Jessie+1. A small example on how this data will look like can be found here a larger, actual file is available here. Any questions and feedback are highly appreciated.

I ll be there this year! Talks look amazing, I can t wait to hit up all the talks. Looks really well organized! Talk schedule has a bunch that I want to hit, I hope they re recorded to watch later! If anyone s heading to PyGotham, let me know, I ll be there both days, likely floating around the talks.

Last time I changed my desktop computer I bought a CPU from the Intel Haswell family, the one available on the market at that time. I carefully selected the CPU to make sure it supports as many instructions extensions as possible in this family (Intel likes segmentation, even high-end CPUs like the Core i7-4770k do not support all possible instructions). I ended-up choosing the Core i7-4771 as it supports the Transactional Synchronization Extensions (Intel TSX) instructions, which provide transactional memory support. Support for it has been recently added in the GNU libc, and has been activated in Debian. By choosing this CPU, I wanted to be sure that I can debug this support in case of bug report, like for example in bug#751147. Recently some computing websites started to mention that the TSX instructions have bugs on Xeon E3 v3 family (and likely on Core i7-4771 as they share the same silicon and stepping), quoting this Intel document. Indeed one can read on page 49:

HSW136. Software Using Intel TSX May Result in Unpredictable System Behavior

Problem: Under a complex set of internal timing conditions and system events, software using the Intel TSX (Transactional Synchronization Extensions) instructions may result in unpredictable system behavior.
Implication: This erratum may result in unpredictable system behavior.
Workaround: It is possible for the BIOS to contain a workaround for this erratum.

And later on page 51:

Due to Erratum HSw136, TSX instructions are disabled and are only supported for software development. See your Intel representative for details.

The same websites tell that Intel is going to disable the TSX instructions via a microcode update. I hope it won t be the case and that they are going to be able to find a microcode fix. Otherwise it would mean I will have to upgrade my desktop computer earlier than expected. It s a bit expensive to upgrade it every year and that s a the reason why I skipped the Ivy Bridge generation which didn t bring a lot from the instructions point of view. Alternatively I can also skip microcode and BIOS updates, in the hope I won t need another fix from them at some point.

This is a random post inspired by recent purchases. Some things we buy are practical, others are a little arbitrary. I tend to avoid buying things for the sake of it, and have explicitly started decluttering our house over the past few years. That said sometimes things just seem sufficiently "cool" that they get bought without too much thought. This entry is about two things. A couple of years ago my bathroom was ripped apart and refitted. Gone was the old and nasty room, and in its place was a glorious space. There was only one downside to the new bathroom - you turn on the light and the fan comes on too. When your wife works funny shifts at the hospital you can find that the (quiet) fan sounds very loud in the middle of the night and wakes you up.. So I figured we could buy a couple of LED lights and scatter them around the place - when it is dark the movement sensors turn on the lights. These things are amazing. We have one sat on a shelf, one velcroed to the bottom of the sink, and one on the floor, just hidden underneath the toilet. Due to the shiny-white walls of the room they're all you need in the dark. By contrast my second purchase was a mistake - The Logitech Harmony 650 Universal Remote Control should be great. It clearly has the features I want - Able to power:

• Our TV.
• Our Sky-box.
• OUr DVD player.

The problem is solely due to the horrific software. You program the device via an application/website which works only under Windows. I had to resort to installing Windows in a virtual machine to make it run:

# Get the Bus/ID for the USB device
bus=$(lsusb  grep -i Harmony   awk ' print $2 '   tr -d 0)
id=$(lsusb  grep -i Harmony   awk ' print $4 '   tr -d 0:)
# pass to kvm
kvm -localtime ..  -usb -device usb-host,hostbus=$bus,hostaddr=$id ..

That allows the device to be passed through to windows, though you'll later have to jump onto the Qemu console to re-add the device as the software disconnects and reconnects it at random times, and the bus changes. Sigh. I guess I can pretend it works, and has cut down on the number of remotes sat on our table, but .. The overwhelmingly negative setup and configuration process has really soured me on it. There is a linux application which will take a configuration file and squirt it onto the device, when attached via a USB cable. This software, which I found during research prior to buying it, is useful but not as much as I'd expected. Why? Well the software lets you upload the config file, but to get a config file you must fully complete the setup on Windows. It is impossible to configure/use this device solely using GNU/Linux. (Apparently there is MacOS software too, I don't use macs. *shrugs*) In conclusion - Motion-activated LED lights, more useful than expected, but Harmony causes Discord.

This is the last week of Google Summer of Code 2014 - all good things must come to an end. To wrap things up, I ve merged all my work on JSCommunicator into a new version with all the added features. You can now demo the new and improved (or at least so I hope) JSCommunicator on rtc.debian.org! JSCommunicator 2.0 has an assortment of new add-ons, the most important new features are the Instant Messaging component and the internationalization support. The UI has been reorganized but we are currently not using a skin for color scheme - will be posting about that in a bit. The idea is to have a more neutral look that can be easily customized and integrated with other web apps. A chat session is automatically opened when you begin a call with someone - unless you already started a chat session with said someone. Sound alerts for new incoming messages are optional in the config file, visual alerts occur when an inactive chat tab receives a new message. Future work includes multiple user chat sessions and adapting the layout to a large amount of chat tabs. Currently it only handles 6. (Should I allow more? Who chats with more than 6 people at once? 14 year old me would, but now I just can t handle that. Anyway, I welcome advice on how to go about this. Should we do infinite tabs or if not, what s the cut-off?) About internationalization, I m uber proud to say we currently run in 6 languages! The 6 are English (default), Spanish, French, Portuguese, Hebrew and German. But one thing I must mention is that since I added new stuff to JSCommunicator, some of the new stuff doesn t have a translation. I took care of the Portuguese translation and Yehuda Korotkin quickly turned in the Hebrew translation, but we are still missing an update for Spanish, French and German. If you can contribute, please do. There are about 10 new labels to translate, you can fix the issue here. Or if you re short on time, shoot me an email with the translation for what s on the right side of the = : welcome = Welcome, call = Call chat = Chat enter_contact = Enter contact type_to_chat = type to chat start_chat = start chat me = me logout = Logout no_contact = Please enter a contact. remember_me = Remember me I ll merge it myself but I ll be sure to add you to the authors list.

perl 5.20 got uploaded to debian unstable a few minutes ago; be prepared for some glitches when upgrading sid machines/chroots in the next days, while all 557 reverse dependencies are rebuilt via binNMUs. how does this relate to this blog post's title? it does, since during the last weeks I was mostly trying to help with the preparation of this transition. & we managed to fix quite a few bugs while they were not bumped to serious yet, otherwise the list below would be a bit longer :) anyway, here are the the RC bugs I've worked on in the last 20 or so weeks:

• #711614 src:libscriptalicious-perl: "libscriptalicious-perl: FTBFS with perl 5.18: test hang"
  upload new upstream release (pkg-perl)
• #711616 src:libtest-refcount-perl: "libtest-refcount-perl: FTBFS with perl 5.18: test failures"
  build-depend on fixed version (pkg-perl)
• #719835 libdevel-findref-perl: "libdevel-findref-perl: crash in XS_Devel__FindRef_find_ on Perl 5.18"
  upload new upstream release (pkg-perl)
• #720021 src:libhtml-template-dumper-perl: "libhtml-template-dumper-perl: FTBFS with perl 5.18: test failures"
  mark fragile test as TODO (pgk-perl)
• #720271 src:libnet-jabber-perl: "libnet-jabber-perl: FTBFS with perl 5.18: test failures"
  add patch to sort hash (pkg-perl)
• #726948 libmath-bigint-perl: "libmath-bigint-perl: uninstallable in sid - obsoleted by perl 5.18"
  upload new upstream release (pkg-perl)
• #728634 src:fusesmb: "fusesmb: FTBFS: configure: error: Please install libsmbclient header files."
  finally upload to DELAYED/2 with patch from November (using pkg-config)
• #730936 src:libaudio-mpd-perl: "libaudio-mpd-perl: FTBFS: Tests errors"
  upload new upstream release (pkg-perl)
• #737434 src:libmojomojo-perl: "[src:libmojomojo-perl] Sourceless file (minified)"
  add unminified version of javascript file to source package (pkg-perl)
• #739505 libcgi-application-perl: "libcgi-application-perl: CVE-2013-7329: information disclosure flaw"
  upload with patch prepared by carnil (pkg-perl)
• #739809 src:libgtk2-perl: "libgtk2-perl: FTBFS: Test failure"
  add patch from Colin Watson (pkg-perl)
• #743086 src:libmousex-getopt-perl: "libmousex-getopt-perl: FTBFS: Tests failures"
  add patch from CPAN RT (pkg-perl)
• #743099 src:libclass-refresh-perl: "libclass-refresh-perl: FTBFS: Tests failures"
  upload new upstream release (pkg-perl)
• #745792 encfs: "[PATCH] Fixing FTBFS on i386 and kfreebsd-i386"
  use DEB_HOST_MULTIARCH to find libraries, upload to DELAYED/2
• #746148 src:redshift: "redshift: FTBFS: configure: error: missing dependencies for VidMode method"
  add missing build dependency, upload to DELAYED/2
• #747771 src:bti: "bti: FTBFS: configure: line 3571: syntax error near unexpected token PKG_CHECK_MODULES'"
  add missing build dependency
• #748996 libgd-securityimage-perl: "libgd-securityimage-perl: should switch to use libgd-perl"
  update (build) dependency (pkg-perl)
• #749509 src:visualvm: "visualvm: FTBFS: debian/visualvm/...: Directory nonexistent"
  use override_dh_install-indep in debian/rules (pkg-java)
• #749825 src:libtime-parsedate-perl: "libtime-parsedate-perl: trying to overwrite '/usr/share/man/man3/Time::ParseDate.3pm.gz', which is also in package libtime-modules-perl 2011.0517-1"
  add missing Breaks/Replaces (pkg-perl)
• #749938 libnet-ssh2-perl: "libnet-ssh2-perl: FTBFS: libgcrypt20 vs. libcrypt11"
  upload package with fixed build-dep, prepared by Daniel Lintott (pkg-perl)
• #750276 libhttp-async-perl: "libhttp-async-perl: FTBFS: Tests failures"
  upload new upstream release prepared by Daniel Lintott (pkg-perl)
• #750283 src:xacobeo: "xacobeo: FTBFS: Tests failures when network is accessible"
  add missing build dependency (pkg-perl)
• #750305 src:libmoosex-app-cmd-perl: "libmoosex-app-cmd-perl: FTBFS: Tests failures"
  add patch to fix test regexps (pkg-perl)
• #750325 src:libtemplate-plugin-latex-perl: "libtemplate-plugin-latex-perl: FTBFS: Tests failures"
  upload new upstream releases prepared by Robert James Clay (pkg-perl)
• #750341 src:cpanminus: "cpanminus: FTBFS: Trying to write outside builddir"
  set HOME for tests (pkg-perl)
• #750564 obexftp: "missing license in debian/copyright"
  add missing license to debian/copyright, QA upload
• #750770 libsereal-decoder-perl: "libsereal-decoder-perl: FTBFS on various architectures"
  upload new upstream development release (pkg-perl)
• #751044 packaging-tutorial: "packaging-tutorial: FTBFS - File bxcjkjatype.sty' not found."
  send a patch (updated build-depends) to the BTS
• #751563 src:tuxguitar: "tuxguitar: depends on xulrunner which is no more"
  do some triaging (pkg-java)
• #752171 src:pcp: "pcp: Build depends on autoconf"
  upload NMU prepared by Xilin Sun, adding missing build dependency
• #752347 highlight: "highlight: hardcodes /usr/lib/perl5"
  use executable .install file for perl library path, upload to DELAYED/5
• #752349 src:nflog-bindings: "nflog-bindings: hardcodes /usr/lib/perl5"
  use executable .install file for perl library path, upload to DELAYED/5
• #752469 clearsilver: "clearsilver: hardcodes /usr/lib/perl5"
  use executable .install file for perl library path, upload to DELAYED/5
• #752470 ekg2: "ekg2: hardcodes /usr/lib/perl5"
  calculate perl lib path at build time, QA upload
• #752472 fwknop: "fwknop: hardcodes /usr/lib/perl5"
  use $Config vendorarch in debian/rules, upload to DELAYED/5
• #752476 handlersocket: "handlersocket: hardcodes /usr/lib/perl5"
  create .install from .install.in at build time, QA upload
• #752704 lcgdm: "lcgdm: hardcodes /usr/lib/perl5"
  create .install from .install.in at build time, upload to DELAYED/5
• #752705 libbuffy-bindings: "libbuffy-bindings: hardcodes /usr/lib/perl5"
  pass value of $Config vendorarch to dh_install in debian/rules, upload to DELAYED/5
• #752710 liboping: "liboping: hardcodes /usr/lib/perl5"
  use executable .install file for perl library path, upload to DELAYED/5
• #752714 lockdev: "lockdev: hardcodes /usr/lib/perl5"
  use $Config vendorarch in debian/rules, upload to DELAYED/5
• #752716 ming: "ming: hardcodes /usr/lib/perl5"
  NMU with the minimal changes from the next release
• #752799 obexftp: "obexftp: hardcodes /usr/lib/perl5"
  calculate perl lib path at build time, QA upload
• #752810 src:razor: "razor: hardcodes /usr/lib/perl5"
  use $Config vendorarch in debian/rules, upload to DELAYED/5
• #752812 src:redland-bindings: "redland-bindings: hardcodes /usr/lib/perl5"
  use $Config vendorarch in debian/rules, upload to DELAYED/5
• #752815 src:stfl: "stfl: hardcodes /usr/lib/perl5"
  create .install from .install.in at build time, upload to DELAYED/5
• #752924 libdbix-class-perl: "libdbix-class-perl: FTBFS: Failed test 'Cascading delete on Ordered has_many works'"
  add patch from upstream git (pkg-perl)
• #752928 libencode-arabic-perl: "libencode-arabic-perl: FTBFS with newer Encode: Can't locate object method "export_to_level" via package "Encode""
  add patch from Niko Tyni (pkg-perl)
• #752982 src:libwebservice-musicbrainz-perl: "libwebservice-musicbrainz-perl: hardcodes /usr/lib/perl5"
  pass create_packlist=0 to Build.PL, upload to DELAYED/5
• #752988 libnet-dns-resolver-programmable-perl: "libnet-dns-resolver-programmable-perl: broken with newer Net::DNS"
  add patch from CPAN RT (pkg-perl)
• #752989 libio-callback-perl: "libio-callback-perl: FTBFS with Perl 5.20: alternative dependencies"
  versioned close (pkg-perl)
• #753026 libje-perl: "libje-perl: FTBFS with Perl 5.20: test failures"
  upload new upstream release (pkg-perl)
• #753038 libplack-test-anyevent-perl: "libplack-test-anyevent-perl: FTBFS with Perl 5.20: alternative dependencies"
  versioned close (pkg-perl)
• #753057 libinline-java-perl: "libinline-java-perl: broken symlinks when built under perl 5.20"
  fix symlinks to differing paths in perl 5.18 vs. 5.20 (pkg-perl)
• #753144 src:net-snmp: "net-snmp: FTBFS on kfreebsd-amd64 - 'struct kinfo_proc' has no member named 'kp_eproc'"
  add patch from Niko Tyni, upload to DELAYED/5, later reschedules to 0-day with maintainer's approval
• #753214 src:license-reconcile: "license-reconcile: FTBFS: Tests failures"
  make (build) dependency versioned (pkg-perl)
• #753237 src:libcgi-application-plugin-ajaxupload-perl: "libcgi-application-plugin-ajaxupload-perl: Tests failures"
  make (build) dependency versioned (pkg-perl)
• #754125 libimager-perl: "libimager-perl: FTBFS on s390x"
  close bug, package builds again after libpng upload (pkg-perl)
• #754691 src:libio-interface-perl: "libio-interface-perl: FTBFS on kfreebsd-*: invalid storage class for function 'XS_IO__Interface_if_flags'"
  add patch which adds a missing (pkg-perl)
• #754993 libdevice-usb-perl: "libdevice-usb-perl: FTBFS with newer Inline(::C)"
  workaround an Inline bug in debian/rules
• #755028 src:libtk-tablematrix-perl: "libtk-tablematrix-perl: hardcodes /usr/lib/perl5"
  use $Config vendorarch in debian/rules, upload to DELAYED/5
• #755324 src:pinto: "pinto: FTBFS: Tests failures"
  add patch to "use" required module (pkg-perl)
• #755332 src:libdevel-nytprof-perl: "libdevel-nytprof-perl: FTBFS: Tests failures"
  mark failing tests temporarily as TODO (pkg-perl)
• #757754 obexftp: "obexftp: FTBFS: format not a string literal and no format arguments [-Werror=format-security]"
  add patch with format argument, QA upload
• #757774 src:libwx-glcanvas-perl: "libwx-glcanvas-perl: hardcodes /usr/lib/perl5"
  build-depend on new libwx-perl (pkg-perl)
• #757855 libwx-perl: "libwx-perl: embeds exact wxWidgets version, needs stricter dependencies"
  use virtual package provided by alien-wxwidgets (pkg-perl)
• #758127 src:libwx-perl: "libwx-perl: FTBFS on arm*"
  report and try to debug new build failure (pkg-perl)

p.s.: & now, go & enjoy the new perl 5.20 features :)