A little more than 3 months after our latest minor release, here is the new major version of Gnocchi, stamped 2.0.0. It contains a lot of new and exciting features, and I'd like to talk about some of them to celebrate! You may notice that this release happens in the middle of the OpenStack release cycle. Indeed, Gnocchi does not follow that 6-months cycle, and we release whenever our code is ready. That forces us to have a more iterative approach, less disruptive for other projects and allow us to achieve a higher velocity. Applying the good old mantra release early, release often. Documentation This version features a large documentation update. Gnocchi is still the only OpenStack server project that implements a "no doc, no merge" policy, meaning any code must come with the documentation addition or change included in the patch. The full documentation is included in the source code and available online at gnocchi.xyz. Data split & compression I've already covered this change extensively in my last blog about timeseries compression. Long story short, Gnocchi now splits timeseries archives in small chunks that are compressed, increasing speed and decreasing data size. Measures batching support Gnocchi now supports batching, which allow submitting several measures for different metric in a single request. This is especially useful in the context where your application tends to cache metrics for a while and is able to send them in a batch. Usage is fully documented for the REST API. Group by support in aggregation One of the most demanded features was the ability to do measure aggregation no resource, using a group by type query. This is now possible using the new groupby parameter to aggregation queries. Ceph backend optimization We improved the Ceph back-end a lot. Mehdi Abaakouk wrote a new Python binding for Ceph, called Cradox, that is going to replace the current Python rados module in the subsequent Ceph releases. Gnocchi makes usage of this new module to speed things up, making the Ceph based driver really, really faster than before. We also implemented asynchronous data deletion, which improves performance a bit. The next step will be to run some new benchmarks like I did a few months ago and compare with the Gnocchi 1.3 series. Stay tuned!

Making packages for Debian requires quite a lot of attention to details. And one of the details is the content of the debian/copyright file, which should list all relevant licenses used by the code in the package in question, preferably in machine readable DEP5 format. For large packages with lots of contributors it is hard to write and update this file manually, and if you get some detail wrong, the package is normally rejected by the ftpmasters. So getting it right the first time around get the package into Debian faster, and save both you and the ftpmasters some work.. Today, while trying to figure out what was wrong with the zfsonlinux copyright file, I decided to spend some time on figuring out the options for doing this job automatically, or at least semi-automatically. Lucikly, there are at least two tools available for generating the file based on the code in the source package, debmake and cme. I'm not sure which one of them came first, but both seem to be able to create a sensible draft file. As far as I can tell, none of them can be trusted to get the result just right, so the content need to be polished a bit before the file is OK to upload. I found the debmake option in a blog posts from 2014. To generate using debmake, use the -cc option:

debmake -cc > debian/copyright

Note there are some problems with python and non-ASCII names, so this might not be the best option. The cme option is based on a config parsing library, and I found this approach in a blog post from 2015. To generate using cme, use the 'update dpkg-copyright' option:

cme update dpkg-copyright

This will create or update debian/copyright. The cme tool seem to handle UTF-8 names better than debmake. When the copyright file is created, I would also like some help to check if the file is correct. For this I found two good options, debmake -k and license-reconcile. The former seem to focus on license types and file matching, and is able to detect ineffective blocks in the copyright file. The latter reports missing copyright holders and years, but was confused by inconsistent license names (like CDDL vs. CDDL-1.0). I suspect it is good to use both and fix all issues reported by them before uploading. But I do not know if the tools and the ftpmasters agree on what is important to fix in a copyright file, so the package might still be rejected. The devscripts tool licensecheck deserve mentioning. It will read through the source and try to find all copyright statements. It is not comparing the result to the content of debian/copyright, but can be useful when verifying the content of the copyright file. Are you aware of better tools in Debian to create and update debian/copyright file. Please let me know, or blog about it on planet.debian.org. As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b. Update 2016-02-20: I got a tip from Mike Gabriel on how to use licensecheck and cdbs to create a draft copyright file

licensecheck --copyright -r  find * -type f    \
  /usr/lib/cdbs/licensecheck2dep5 > debian/copyright.auto

He mentioned that he normally check the generated file into the version control system to make it easier to discover license and copyright changes in the upstream source. I will try to do the same with my packages in the future. Update 2016-02-21: The cme author recommended against using -quiet for new users, so I removed it from the proposed command line.

I had no access to the internet for most of my childhood. Nobody in my area knew anything about programming. I learned a great deal from a number of CDs that included free software source code and archives of project mailing lists. When I got to university, I learned even more by being able to develop a Debian-based OS for use in our computer facilities. That gave me the experience and knowledge that I needed to become involved in Debian development, which in turn gave me the background required to be able to help Ubuntu become the first free software operating system to work out of the box on modern laptops. From there, I've been able to build my career around developing free software.

Ubuntu can be translated as "I am who I am because of who we all are". I am who I am because people made the choice to release their software under licenses that permitted examination, modification and redistribution. I am who I am because I was able to participate in communities that took advantages of those freedoms to produce new and better software. I am who I am because when my priorities differed from those of existing communities, it was still possible for me to benefit from their work and for them to benefit from mine.

Free software doesn't mean that the software is entirely free of restrictions. While a core aspect is the right to distribute modified versions of code, it has never been fundamental to free software that you be able to do so while still claiming that the code is the original version. Various approaches have been taken to make it possible for users to distinguish modified versions, ranging from simply including license terms that require modified versions be marked as such, to licenses that require that you change the name of the package if you modify it. However, what's probably the most effective approach has been to apply trademark law to the problem. Mozilla's trademark policy is an example of this - if you modify the code in ways that aren't approved by Mozilla, you aren't entitled to use the trademarks.

A requirement that you avoid use of trademarks in an infringing way is reasonable. Mozilla products include support for building with branding disabled, which makes it very straightforward for a user to build a modified version of Firefox that can be redistributed without any trademark issues. Red Hat have a similar policy for Fedora and RHEL[1] - you simply replace the packages that contain the branding and you're done.

Canonical's IP policy around Ubuntu is fundamentally different. While Mozilla make it clear that you simply no longer have a right to use the trademarks under trademark law, Canonical appear to require that you remove all trademarks entirely even if using them wouldn't be a violation of trademark law. While Mozilla restrict the redistribution of modified binaries that include their trademarks, Canonical insist that you rebuild everything even if the package doesn't contain any trademarks. And while Mozilla give you a single build option that creates binaries that conform with their trademark requirements, Canonical will refuse to tell you what you have to do.

When asked about this at SCALE earlier this year, Mark Shuttleworth claimed that Ubuntu's policy was consistent with that of other projects. This is inaccurate. Nobody else requires that you rebuild every package before you can redistribute it in a modified distribution - such a restriction is a violation of freedom 2 of the Free Software Definition, and as a result the binary distributions of Ubuntu are not free software. Nobody else refuses to discuss whether you're required to remove non-infringing trademarks in order to be able to redistribute. Nobody else responds to offers to make it easier for users to produce non-infringing derivatives with a flat refusal.

Mark claims that I'm only raising this issue because I work for a competitor and wish to harm Canonical. Nothing could be further from the truth. I began discussing this before working for my current employers - my previous employers had no meaningful market overlap with Canonical at all. The reason I care is because I care about free software. I care about people being able to derive new and interesting things from existing code. I care about a small team of people being able to take Ubuntu and make something better in the same way that Ubuntu did with Debian. I care about ensuring that users receive the freedom to do this without having to jump through a significant number of hoops in the process. Ubuntu has been a spectacularly successful vehicle for getting free software into the hands of users. Mark's generosity in funding this experiment has undoubtedly made the world a better place. Canonical employs a large number of talented developers writing high quality software, many of whom I'm fortunate enough to be able to call friends. And Canonical are squandering that by restricting the rights of their users and alienating the free software community.

I want others to be who they are because of my work and the work of all the others like me. Anything that makes that more difficult saddens me, and so I do what I can to fix it. I criticise Canonical's policies in the hope that we, as a community, can convince Canonical to agree that this kind of artificial barrier to modification hurts us more than it helps them. In many ways, Canonical remain one of our best hopes for broadening the reach of free software, and this is why it's unfortunate that they do so in a way that makes it more difficult for people to have the same experiences that I did.

[1] While it's easy to turn a trademark infringing version of RHEL into a non-infringing one, Red Hat don't provide publicly available binary packages for RHEL. If you get hold of them somehow you're entitled to redistribute them freely, but Red Hat's subscriber agreement indicates that if you do this as a Red Hat customer you will lose access to further binary updates - a provision that I find utterly repugnant. Its inclusion reduces my respect for Red Hat and my enthusiasm for working with them, and given the official Red Hat support for CentOS it appears to make no sense whatsoever. Red Hat should drop it.

comments

The day has finally come again to release a new version of my plotting program, ctioga2. Version 0.14 features, among others:

• a new binner, useful to make histograms;
• a command to hide elements already presents, that can be very useful to make animations;
• a whole set of functionalities to make it easier to draw complex grids, including easy ways to style grid elements and a way to switch to the next grid element (picture);
• a whole series of command-file functions to obtain informations about loaded datasets;
• a way to draw legends separately by hand;
• and more, including bugfixes, better error reporting, such as when some of the numbers are infinite, and much much faster debug output.

As usual, the best way to update the package is running:

~ gem update ctioga2

An updated Debian package should find its way to the archive later on today. Have fun !

Debugging output of GUI apps I have been hit by #643726 again. I was happily working on a terminal while an apt upgrade was running on another, and suddenly kbuildsycoca4 started vomiting on my command line because I had run okular once on that terminal last tuesday. Among the possible workarounds, one can run kdebugdialog from package kde-runtime and check the "Disable all debug output" button. This is the output of kdebugdialog when run on a terminal after enabling "Disable all debug output":

$ kdebugdialog
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.

I see it as debugging output turned into an art performance, and I have an appreciation for this kind of things. kdebugdialog, my hat's off to you.

Debugging output of GUI apps I have been hit by #643726 again. I was happily working on a terminal while an apt upgrade was running on another, and suddenly kbuildsycoca4 started vomiting on my command line because I had run okular once on that terminal last tuesday. Among the possible workarounds, one can run kdebugdialog from package kde-runtime and check the "Disable all debug output" button. This is the output of kdebugdialog when run on a terminal after enabling "Disable all debug output":

$ kdebugdialog
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.

I see it as debugging output turned into an art performance, and I have an appreciation for this kind of things. kdebugdialog, my hat's off to you.

I have been hit by #643726 again. I was happily working on a terminal while an apt upgrade was running on another, and suddenly kbuildsycoca4 started vomiting on my command line because I had run okular once on that terminal last tuesday. Among the possible workarounds, one can run kdebugdialog from package kde-runtime and check the "Disable all debug output" button. This is the output of kdebugdialog when run on a terminal after enabling "Disable all debug output": I see it as debugging output turned into an art performance, and I have an appreciation for this kind of things. kdebugdialog, my hat's off to you.

This is the story of libspotify, as experienced by a Spotify customer and libspotify developer for six years.

Step 1: Support your API for years Since April 2009, libspotify has been a mostly nice although proprietary C library for integrating with the Spotify music streaming service, providing both music metadata and full playback capabilities. Language wrappers have been written for a plenitude of programming languages (including my own pyspotify). libspotify is integrated into numerous open source projects (including my own Mopidy), networked AV receivers, and rumour has it: even cars. In addition to being closed source, there was another catch: in order to use it, you need the Spotify premium subscription. Speaking from my own experience supporting a project integrating with Spotify for the last six years: lots of end users upgraded to premium in order to use the projects built on top of libspotify.

Step 2: Stop accepting bug reports and stop releasing updates In May 2012, Spotify released libspotify 12.1.51 for all supported architectures, which now even included Android. After months of endless requests from the quickly growing hoards of Raspberry Pi users, Spotify released libspotify 12.1.103 for hardfloat armv6 on January 22 2013. The release included minor API additions, like the addition of explicit lyrics and means for accepting Spotify s Terms of Service. The additions in 12.1.103 never reached the other supported architectures and 12.1.51 remains the last release for those. There was no clear communication about this being the end of libspotify, releases have just ceased. libspotify has not been mentioned in a single developer blog post or developer email newsletter since. That s now more than three years ago.

Step 3: Create new APIs Over the next couple of years Spotify released a new Web API and iOS and Android SDKs. Through their developer blog, they communicated quite a bit both about these and about the deprecation of the apps inside the Spotify desktop client. Here are some pieces from their developer blog to illustrate:

• 2014-02-03: Deprecation of Spotify Apps API 0.x
• 2014-03-06: First Spotify iOS SDK beta release
• 2014-03-24: No longer accepting new Spotify Apps
• 2014-06-17: First Spotify Web API release
• 2014-06-25: First Spotify Android SDK beta release

The Web API grew steadily, as documented by the Web APIs changelog. Over time, its feature list improved, supporting anonymous endpoints for generic music metadata as well as OAuth protected endpoints for managing your own music collection and playlists. The only major part missing that prevented developers replacing libspotify was music playback, as the Web API only provides 30 second previews. From what I recall of 2014 IRC discussions with Spotify employees, they had built a new much smaller library for streaming music. The iOS and Android SDKs were simply platform specific wrappers around this new playback library and the new Web API. My impression is that, after a number of beta releases, the iOS and Android SDKs seamlessly replaced libspotify on those platforms. The plan was apparently to release the new playback library for desktop too, so that it, in combination with the Web API, could replace libspotify there as well. That was all back in 2014.

Step 4: Deprecate old APIs At some point, Spotify deprecated their old web API, the Metadata API. There was no blog post about it and it was not mentioned in any Spotify Developer News email I ve received in recent years. I guess they only published this fact as a note on the Metadata API web page; it is therefore hard to track down exactly when it was deprecated and how the deprecation was communicated. Anyway, it was deprecated, and the natural upgrade path was to the new Web API. Spotify even has a migration guide mapping the concepts of the Metadata API to the new Web API. All good. In May 2015, after 28 months without a single release or any news, libspotify is officially deprecated. The communication channel of choice? A note on the libspotify web page. No post on the developer blog. No Spotify Developer News email. To be able to use libspotify, all developers have registered and requested a personal application key. To my knowledge, none of these registered library users were notified directly.

Step 5: Shut down an old API On January 20 2016 Spotify end-of-lifed their Metadata API. I don t know if this date was communicated on the Metadata API web page well in advance of this shutdown. There s no blog post on the topic. There s no news email. There s no tweet from @SpotifyPlatform or @SpotifyStatus. I can t find anything except users commenting on various Spotify documentation pages that the text is outdated because the API is no longer merely deprecated, but entirely shut down. Hopefully they did publish the shutdown date ahead of time. It had a full replacement API and a migration guide. Fair and square. Nothing much to complain about here. It s time to get porting your code to the Web API! But frankly, I don t really care about this Metadata API because I don t use the Metadata API, do I?

Step 6: Break your API without warning February 3 2016, Mopidy users start reporting General transient error when searching Spotify. This is a libspotify error message known as SP_ERROR_OTHER_TRANSIENT with error code 16. This error usually means that there is some unspecified issue with the Spotify service. We ve actually seen it before and like its name suggests, it is indeed transient and soon goes away. According to a series of tweets between my fellow Mopidy developer Nick Steel and @SpotifyCares it appears that the libspotify search functionality was backed by the Metadata API s search functionality. Who knew? I m not aware of any relation between these APIs ever being mentioned publicly. No libspotify user could possibly have known that the Metadata API shutdown would affect them. Let s summarize: libspotify hasn t had any releases for three years and it was officially deprecated in May last year. But, there have been no warnings about any shutdown. The libspotify project is likely no longer staffed at all. Did someone just forget that libspotify depended on the Metadata API they were shutting down? Many newer networked AV receivers use the Spotify Connect libraries that are exclusive to Spotify s commercial partners. But some receivers probably still use libspotify. If so, did Spotify just break hardware sold with Spotify support as a major feature? Something that people are paying a monthly fee to use? It seems they did! In the Spotify Community forums there are numerous reports of broken devices and software that all have two things in common: the date the problem started occurring, and that search is the only feature that is broken.

• Denon receivers, including models: CEOL N8
• Logitech Squeezebox media players
• Marantz receivers, including models: M-CR510, M-CR610, NR1504, NR1603, SR5007, SR6008
• Onkyo receivers, including CR-N755, TX-NR509, TX-NR609, TX-8050
• Roku media players

Surely, Spotify will quickly explain themselves and fix this?

Step 7: Stay quiet Since the API shutdown, Spotify has been very quiet. No posts on the developer blog. No email newsletter. No tweets from @SpotifyPlatform since December 1 last year. Some users at The Spotify Community forum are quoting responses from Spotify. Most responses seem to link to the Metadata API migration docs and state that application developers must migrate. In other cases the responses point the finger at the commercial partner which only adds to the confusion. Some of these devices may use the Metadata API directly, and in those cases the application maintainers are probably to blame. However, I m quite certain that the libspotify search issue is responsible for a fair share of these complaints. libspotify, in contrast to the Metadata API, has never been officially shut down. The responsibility for any breakage caused by libspotify is Spotify s alone.

What I want from Spotify

1. Apologize to your paying customers and supporters.
2. Get substantially better at communicating deprecations and shutdowns. We want all news, not just good news.
3. Give us a new supported C library for audio playback. Now! Not later this year. I ve heard that promise for three consecutive years now. We can live with pre-release software quality initially. Getting the library out sooner rather than later shows us that there is a road ahead for all the projects that stream music from Spotify. Let us get started making new language wrappers.
4. Release Spotify Connect specifications and libraries. We want to make music players that can be controlled by the Spotify mobile app. We want to make music controllers that can play music on all the Spotify Connect software and hardware players we ve invested in. You ve created this technology, now let us use it.

Thanks to Nick Steel for reviewing this blog post.

Since I've now been using Linux for 20 years, I've dug up some artifacts from the early journey.

1. First the book, from late 1995. This from before Tux, so the penguin in the cover is just a co-incidence. The book came with a slackware 3.0 CD, which was my entrance to Linux. Today, almost all of the book is outdated - slackware and lilo install? printing with lpr? mtools and dosemu? ftp, telnet with SLIP dialup? Manually configuring XFree86 and fvwm? How I miss those times!* The only parts of the book are still valid are: shell and vi guides. I didn't read latter, and instead imported my favorite editor from dos FTE.
2. Fast forward some years, into my first programming job. Ready to advertise the Linux revolution, I bought the mug on right. Nobody else would have a Tux mug, so nobody would accidentally take my mug from the office dishwasher. That only worked for my first work place (a huge and nationally hated IT consultant house). The next workplace, a mobile gaming startup (in 2001, I was there before it was trendy!) - and there was already plenty of Linux mugs when I joined...
3. While today it may be hard to imagine, those days using Microsoft office tools was mandatory. That leads to the third memorabilia in the picture. Wordperfect for Linux existed for a brief while, and in the box (can you imagine, software came in physical boxes?) came a Tux plush.

* Wait no, I don't miss those times at all

Great news! The HA stack has been finally updated and you can find now both pacemaker & corosync in Debian stretch.

This is thanks to the hard work of some people, specially Christoph Berg, Ferenc W gner, Martin Loschwitz and others.

By the time of this blogpost, in testing (stretch) you have:

• pacemaker 1.1.14-1
• corosync 2.3.5-3
• crmsh 2.2.0-1

• pcs 0.9.148-1

what to do now

Debian Long Term Support (LTS) This is my third month working on Debian LTS, started by Raphael Hertzog at Freexian. This month was my first month working on the frontdesk duty and did a good bunch of triage. I also performed one upload and reviewed a few security issues.

Frontdesk duties I spent some time trying to get familiar with the frontdesk duty. I still need to document a bit of what I learned, which did involve asking around for parts of the process. The following issues were triaged:

• roundcube in squeeze was happily not vulnerable to CVE-2015-8794 and CVE-2015-8793, as the code affected was not present. roundcube is also not shipped with jessie but the backport is vulnerable
• the php-openid vulnerability was actually just a code sample, a bug report comment clarified all of CVE-2016-2049
• ffmpeg issues were closed, as it is not supported in squeeze
• libxml2 was marked as needing work (CVE-2016-2073)
• asterisk was triaged for all distros before i found out it is also unsupported in squeeze (CVEs coming up, AST-2016-001, AST-2016-001, AST-2016-001)
• libebml and libmatroska were marked as unsupported, although an upload of debian-security-support will be necessary to complete that work (bug #814557 filed)

Uploads and reviews I only ended up doing one upload, of the chrony package (CVE-2016-1567), thanks to the maintainer which provided the patch. I tried my best trying to sort through the issues with tiff (CVE-2015-8668 and CVE-2015-7554), which didn't have obvious fixes available. OpenSUSE seem to have patches, but it is really hard to find them through their issue trackers, which were timing out on me. Hopefully someone else can pick that one up. I also tried and failed to reproduce the cpio issue (CVE-2016-2037), which, at the time, didn't have a patch for a fix. This ended up being solved and Santiago took up the upload. I finally spent some time trying to untangle the mess that is libraw, or more precisely, all the packages that embed dcraw code instead of linking against libraw. Now I really feel even more strongly for the Debian policy section 4.13 which states that Debian packages should not ship with copies of other packages code. It made it really hard to figure out which packages were vulnerable, especially because it was hard to figure out which versions of libraw/dcraw were actually vulnerable to the bug, but also just plain figure out which packages were copying code from libraw. I wish I had found out about secure-testing/data/embedded-code-copies earlier... Still, it was interesting to get familiar with codesearch.debian.net to try to find copies of the vulnerable code, which was not working so well. Kudos to darktable 2.0 for getting rid of their embedded copy of libraw, by the way - it made it completely not vulnerable to the issue, the versions in stretch and sid not having the code at all and older versions having non-vulnerable copies of the code.

Issues with VMs again I still had problems running a squeeze VM - not wanting to use virtualbox because of the overhead, I got lost for a bit trying to use libvirt and KVM. A bunch of issues crept up: using virt-manager would just fail on startup with an error saying interface mtu value is improper, which is a very unhelpful error message (what is a proper value??) - and, for the record, the MTU on eth0 and wlan0 is the fairly standard 1500, while lo is at 65536 bytes, nothing unusual there as far as I know. Then the next problem was actually running a VM - I still somewhat expected to be able to boot off a chroot, something I should definitely forget about it seems like (boot loader missing? not sure). I ended up calling virt-install with the live ISO image I was previously using:

virt-install --virt-type kvm --name squeeze-amd64 --memory 512 --cdrom ~/iso/Debian/cdimage.debian.org_mirror_cdimage_archive_6.0.10_live_amd64_iso_hybrid_debian_live_6.0.10_amd64_gnome_desktop.iso --disk size=4 --os-variant debiansqueeze

At least now I have an installed squeeze VM, something I didn't get to do in Virtualbox (mostly because I didn't want to wait through the install, because it was so slow). Finally, I still have trouble getting a commandline console on the VM: somehow, running virtsh console squeeze-amd64 doesn't give me a login terminal, and worse, it actually freezes the terminal that I can actually get on virt-viewer squeeze-amd64, which definitely sounds like a bug. I documented a bit more of that setup in the Debian wiki KVM page so hopefully this will be useful for others.

Other free software work I continued my work on improving timetracking with ledger in my ledger-timetracking git repository, which now got a place on the new plaintextaccounting.org website, which acts as a portal for ledger-like software projects and documentation.

Darktable 2.0 I had the pleasure of trying the new Darktable 2.0 release, which only recently entered Debian. I built a backport for jessie, which works beautifully: much faster thumbnail rendering, no dropping of history when switching views... The new features are great, but I also appreciate how they are being very conservative in their approach. Darktable is great software: I may have trouble approaching the results other are having with lightroom and snapseed, but those are proprietary software that I can't use anyways. I also suspect that I just don't have enough of a clue of what I'm doing to get the results I need in Darktable. Maybe with hand-holding, one day, I will surpass the results I get with the JPEGs from my Canon camera. Until then, I turned off RAW exports in my camera to try and control the explosion of disk use I saw since I got that camera:

41M     2004
363M    2005
937M    2006
2,2G    2007
894M    2008
800M    2009
1,8G    2010
1,4G    2011
9,8G    2012
31G     2013
26G     2014
9,8G    2015

The drop in 2015 is mostly due to me taking less pictures in the last year, for some reason...

Markdown mode hacks I ended up writing some elisp for the markdown mode. It seems I am always writing links like [text](link) which seems more natural at first, but then the formatting looks messier, as paragraph wrapping is all off because of the long URLs. So I always ended up converting those links, which was a painful series of keystrokes. So I made a macro, and while I'm a it, why not rewrite it as a lisp function. Twice. Then I was told by the markdown-mode.el developers that they had already fixed that (in the 2.1 version, not in Debian jessie) and that the C-c C-a r key binding actually recognized existing links and conveniently converted them. I documented my adventures in bug #94, but it seems I wrote this code for nothing else than re-learning Emacs lisp, which was actually quite fun.

More emacs hacking Another thing I always wasted time doing by and is "rename file and buffer". Often, you visit a file but it's named wrong. My most common case is a .txt file that i rename to .mdwn. I would then have to do:

M-x rename-file <ret> newfile
M-x rename-buffer <ret> newfile
C-x C-s <ret> newfile

Really annoying. Turns out that set-visited-file-name actually does most of the job, but doesn't actually rename the file, which is really silly. So I wrote this small function instead:

(defun rename-file-and-buffer (newfname)
  "combine rename-file and rename-buffer
set-visited-file-name does most of the job, but unfortunately
doesn't actually rename the file. rename-file does that, but
doesn't rename the buffer. rename-buffer only renames the buffer,
which is pretty pointless.
only operates on current buffer because set-visited-file-name
also does so and we don't bother doing excursions around.
"
  (interactive "GRename file and bufer: ")
  (let ((oldfname (buffer-file-name)))
    (set-visited-file-name newfname nil t)
    (rename-file oldfname newfname)
    )
  )

Not bound to any key, really trivial, but doing this without that function is really non-trivial, especially since set-visited-file-name needs special arguments to not mark the file as modified.

IRC packages updates I updated the Sopel IRC bot package to the latest release, 6.3.0. They have finally switched to Requests, but apart from that, no change was necessary. I am glad to finally see SNI support working everywhere in the bot! I also update the Charydbis IRC server package to the latest 3.5.0 stable release. This release is great news, as I was able to remove 5 of the 7 patches I was dragging along the Debian package. The previous Charybdis stable release was over 3 years old, as 3.4.2 was released in (December) 2012! I spend a good chunk of time making the package reproducible. I filed a bug upstream and eventually made a patch to make it possible to hardcode a build timestamp, which seems to have been the only detectable change in the reproducible build infrastructure. Charybdis had been FTBS for a while in sid now, and the upload should fix that as well. Unfortunately, Charybdis still doesn't build with hardening flags - but hopefully a future update of the package should fix that. It is probably because CFLAGS are not passed around properly. There's really interesting stuff going on in the IRC world. Even though IRC is one of the oldest protocols still in operation (1988, even before the Web, but after SMTP and the even more venerable FTP), it is still being actively developed, with a working group drafting multiple extensions to the IRCv3 protocol defined in RFC 1459. For example, IRCv3.3 includes a Strict Transport Security extension, which tries to ensure users use encrypted channels as much as possible, through warnings and STARTTLS support. Charybdis goes even further by proposing a reversal of the +S ("secure channel" flag) where all channels are secure by default, and you need to deliberately mark a channel as insecure with the +U flag if you actually want to allow users on an clear-text connection to join the channel. A transition mechanism is also proposed.

Miscellaneous bug reports En vrac... I fell face-first in this amazing game that is endless-sky. I made a small pull request on the documentation, a bug report and a feature request. I forwarded a bug report, originally filed against monkeysign, to the pyqrencode maintainers. I filed a usability bug against tails-installer, which just entered Debian, mostly usability issues. I discovered the fim image viewer, which re-entered Debian recently. It seemed perfect to adjust my photos-import workflow, so I added it to my script, to be able to review photos prior to importing them into Darktable and git-annex.

As we're getting closer to release of new major release (4.6.0), the focus moves to the bug fixing. Most of the fixed issues came from our error reporting server, which collects error reports from the installations. Vast majority if bugs were affecting older releases as well, so these were fixed for upcoming 4.5.5 as well. Besides bugfixing there was also some cleanups in the master branch - shared code for processing sprites and covering it by tests and removed caching of MySQL server information as that didn't bring any speed improvements and lead to cache consistency issues. All handled issues:

• #11955 Undefined variable: format
• #11953 Warning on realpath failure
• #11952 Warning on invalid file upload
• #11951 Warning open_basedir restriction in effect
• #11950 Undefined index: is_ajax_request
• #11948 Javascript error on longer location hash
• #11946 Notice when file upload fails
• #11945 Help me please with error code
• #11938 Document some known issues
• #11944 Error with disabled php_uname
• #11940 Warning on checking docs
• #11941 Warning when error reporting server is unreachable
• #11939 Not submitting errors without curl
• #11928 [doc] Installation with Composer - add --no-dev option
• #11933 Attempt to change table structure results in exception "file not found" in HHVM
• #11930 Can't change AUTO_INCREMENT
• #11931 Fix Can't change AUTO_INCREMENT, Issue #11930
• #11880 Issue#11872 Resolved
• #11872 Missing parameter error doesn't render HTML
• #11868 v 4.5.3.1 When editing field with primary key: "table does not contain a unique column" (OS X)
• #11887 Discrepancy between the calculated result and the displayed result of a query.
• #11905 Row count displayed by PMA does not match real result
• #11914 Undefined constant PHPSECLIB_INC_DIR with autoloader
• #11927 Remove MySQL version caching
• #11924 Database Server Information on start page wrong
• #11897 Remove direct support for openssl in cookie auth [WIP]
• #11929 No table data after inline edit
• #11916 Cookie auth keep asking for the password
• #18 Use port 80 instead of port 8080
• #19 Switch to port 80 (instead of 8080)

Filed under: English phpMyAdmin 0 comments

About one month has passed and here is the usual updated of TeX Live packages for Debian. While I am not really calling for testers at the moment, building of preliminary packages for TeX Live 2016 has begone. The binaries are already uploaded to experimental, and arch=all packages for experimental will follow soon. As with the last time, here is the list of new and updated pacakges with (auto-generated) links to the package pages on CTAN. New packages adtrees, babel-occitan, crimson, emisa, glossaries-extra, gloss-occitan, libertinus, luatex-def, mynsfc, nihbiosketch, signchart, smartunits, tikz-feynman, uhrzeit, xduthesis. Updated packages achemso, acro, animate, apnum, archaeologie, babel, babel-french, babel-greek, babel-macedonian, bangorcsthesis, biblatex-bookinarticle, biblatex-gost, biblatex-manuscripts-philology, br-lex, caption, chemgreek, chemmacros, chemnum, cmtiup, csplain, csquotes, ctanify, ctex, cweb, datatool, datetime2, datetime2-english, delimseasy, droit-fr, dvips, erewhon, exsheets, fibeamer, fira, fontspec, forest, geschichtsfrkl, glossaries, graphics, greek-fontenc, greektonoi, hyph-utf8, inconsolata, isodoc, l2tabu, l3build, l3experimental, l3kernel, l3packages, latex, latex-bin, latex-make, leadsheets, leaflet, luaotfload, luatexja, mathastext, mcf2graph, mcmthesis, media9, metrix, mhchem, ndsu-thesis, newpx, newtx, ocgx2, pas-tableur, pdftex, pdfx, pict2e, pst-perspective, pstricks, pstricks-add, ptex, quran, ran_toks, realscripts, schemata, showhyphens, siunitx, sr-vorl, sttools, tempora, tetex, tex4ht, texfot, texlive-scripts, thinsp, thuthesis, tkz-orm, tools, uantwerpendocs, ulthese, unicode-data, uptex, xassoccnt, xcjk2uni, xebaposter, xecjk, xetex, xetexko, xltxtra, xpinyin, zhnumber, zhspacing. Enjoy.

To state the obvious: my personal preference is to run Debian GNU/Linux. My current workplace is a CentOS shop and usually I'm the first to claim that it doesn't matter at all, and distribution specific implementation details are irrelevant for what we do (running a JVM). Let's take a short detour to the RedHat network-scripts. Two weeks ago we found some systems we originally installed in a different network segment, with different DNS servers and different search domains, came back up after a reboot with a rewritten 'resolv.conf'. Later on cfengine replaced the generated 'resolv.conf' with the intended one, so it wasn't that obvious to spot in the first place. A colleague found the origin of the rewritten 'resolv.conf' in a device specific configuration file that defined the 'DNS 1,2 ' variables with the installation time DNS server IPs from the other segment. I expected to experience the same behaviour (resolv.conf rewritten during startup and replaced by cfengine later on) in other locations, but assumed we just didn't notice it because the main difference would be a slightly different list of search domains. And I was wrong. I checked the timestamps of several 'resolv.conf' files and their cfengine backup file. None were recently created or related somehow to a reboot. grep-ing through parts of the network-scripts I found the following conditional in '/etc/sysconfig/network-scripts/ifup'

if [ "$PEERDNS" != "no" ]  [ -n "$RESOLV_MODS" -a "$RESOLV_MODS" != "no" ]; then
   [ -n "$MS_DNS1" ] && DNS1=$MS_DNS1
   [ -n "$MS_DNS2" ] && DNS2=$MS_DNS2
      if [ -n "$DNS1" ] && ! grep -q "^nameserver $DNS1" /etc/resolv.conf &&
         tr=$(mktemp /tmp/XXXXXX) ; then
 ...

So if you adjust only the second nameserver IP you stored in "DNS2" in your configuration you end up without an update to your 'resolv.conf'. Now knowing that, I'd say this is relevant distribution specific knowledge, and I'm wondering how many of such subtle behaviours we've hidden in Debian specific solutions? Maybe knowledge about distribution specific implementation details even matters after all. Regardig the 'resolv.conf' issue itself the fault is on us. We as the responsible team did not read the documentation properly and thus deployed a configuration that later on let to some unexpected consequences. I try to remember those issues as an example for the next NetworkManager/systemd-networkd vs old-school-network-scripts argument.

During the last days I've completed quite long time challenge - make Weblate compatible with Python 3. It's probably still not in shape you would use it on production server, but it runs fine and the testsuite completes without errors. As usual, the major obstacle are dependencies. The translate-toolkit doesn't support Python 3 in any released version, so you need latest Git snapshot. Another problematic bit is Mercurial, which will not support Python 3 in near future. But in case you need it, you can still install Python 2 version and use it from Weblate (it uses command line interface). The porting was mostly dealing with encoding mess - in Python 2 you really didn't have to distinguish between binary and text data, while Python 3 quite enforces that. This turned out to be especially tricky with CSV files where csv module expects text streams in Python 3, while translate-toolkit CSV parser expects binary ones. All this has one downside as well - the test matrix got bigger again, what makes the testsuite run longer with 5 workers available. While doing these changes, I've also decided to drop support for Django 1.7, this will allow us to get rid of lot of deprecation warnings in Django 1.9 and prepare codebase for Django 1.10.

Filed under: English Weblate 0 comments

The first major version of the scalable timeserie database I work on, Gnocchi was a released a few months ago. In this first iteration, it took a rather naive approach to data storage. We had little ideas about if and how our distributed back-ends were going to be heavily used, so we stuck to the code of the first proof-of-concept written a couple of years ago. Recently we got more feedbacks from our users, ran a few benchmarks. That gave us enough feedback to start investigating in improving our storage strategy. Data split Up to Gnocchi 1.3, all data for a single metric are stored in a single gigantic file per aggregation method (min, max, average ). This means that the file can grow to several megabytes in size, which make it slow to manipulate. For the next version of Gnocchi, our first work has been to rework that storage and split the data into smaller parts. The diagram above shows how data are organized inside Gnocchi. Until version 1.3, there would have been only one file for each aggregation methods. In the upcoming 2.0 version, Gnocchi will split all these data into smaller parts, where each data split is stored in a file/object. This allows to manipulate smaller pieces of data and to increase the parallelism of the CRUD operations on the back-end leading to large speed improvement. In order to split timeseries into several chunks, Gnocchi defines a maximum number of N points to keep per chunk, to limit their maximum size. It then defines a hash function that produces a non-unique key for any timestamp. It makes it easy to find in which chunk any timestamp should be stored or retrieved. Data compression Up to Gnocchi 1.3, the data stored for each metric is simply serialized using msgpack, a fast and small serialization format. Though, this format does not provide any compression. That means that storing data points needs 8 bytes for a timestamp (64 bits timestamp with nanosecond precision) and 8 bytes for a value (64 bits double-precision floating-point), plus some overhead (extra information and msgpack itself). After looking around on how to compress all these measures, I stumbled upon a paper from some Facebook engineers called about Gorilla, their in-memory timeserie database, entitled "Gorilla: A Fast, Scalable, In-Memory Time Series Database". For reference, part of this encoding is also used by InfluxDB in its new storage engine. The first technique I implemented is easy enough, and it's inspired from delta-of-delta encoding. Instead of storing each timestamp for each data point, and since all the data points are aggregated on a regular interval, we transpose points to be the time difference divided by the interval. For example, the suite of timestamps timestamps = [41230, 41235, 41240, 41250, 41255] is encoded into timestamps = [41230, 1, 1, 2, 1], interval = 5. This allows regular compression algorithms to reduce the size of the integer list using run-length encoding. To actually compress the values, I tried two different algorithms:

• [LZ4](https://en.wikipedia.org/wiki/LZ4_(compression_algorithm), a fast compression/decompression algorithm
• The XOR based compression scheme described in the Gorilla paper mentioned above that I had to implement myself. For reference, it also exists a Go implementation in go-tsz.

I then benchmarked these solutions: The XOR algorithm implemented in Python is pretty slow, compared to LZ4. Truth is that python-lz4 is fully implemented in C, which makes it fast. I've profiled my XOR implementation in Python, to discover that one operation took 20 % of the time: count_lead_and_trail_zeroes, which is in charge of counting the number of leading and trailing zeroes in a binary number. I tried 2 Python implementations of the same algorithm (and submitted them to my friend and Python developer Victor Stinner by the way). The first version using string search with .index() is 10 faster than the second one that only do integer computation. Ah, Python As Victor explained, each Python operation is slow and there's a lot in the second version, whereas .index() is implemented in C and really well optimized and only needs 2 Python operations. Finally, I ended up optimizing that code by leveraging cffi to use directly ffsll() and flsll(). That decreased the run-time of count_lead_and_trail_zeroes by 45 %, making the entire XOR compression code speed increased by a small 7 %. This is not enough to catch up with LZ4 speed. At this stage, the only solution to achieve a high-speed would probably to go with a full C implementation. Considering the compression ratio of the different algorithms, they are pretty much identical. The worst case scenario (random values) for LZ4 compress down to 9 bytes per data point, whereas XOR can go down to 7.38 bytes per data point. In general XOR encoding beats LZ4 by 15 %, except for cases where all values are 0 or 1. However, LZ4 is faster than XOR by a factor of 4 -70 depending on cases. That means that we'll use LZ4 for data compression in Gnocchi 2.0. It's possible that we could achieve as fast compression/decompression algorithm, but I don't think it's worth the effort right now it'd represent a lot of code to write and to maintain.

What happened in the reproducible builds effort between February 7th and February 13th 2016:

Toolchain fixes

• James McCoy uploaded devscripts/2.16.1 which makes dcmd supports .buildinfo files. Original patch by josch.
• Lisandro Dami n Nicanor P rez Meyer uploaded qt4-x11/4:4.8.7+dfsg-6 which make files created by qch reproducible by using a fixed date instead of the current time. Original patch by Dhole.

Norbert Preining rejected the patch submitted by Reiner Herrmann to make the CreationDate not appear in comments of DVI / PS files produced by TeX. He also mentioned that some timestamps can be replaced by using the -output-comment option and that the next version of pdftex will have patches inspired by reproducible build to mitigate the effects (see SOURCE_DATE_EPOCH patches) .

Packages fixed The following packages have become reproducible due to changes in their build dependencies: abntex, apt-dpkg-ref, arduino, c++-annotations, cfi, chaksem, clif, cppreference-doc, dejagnu, derivations, ecasound, fdutils, gnash, gnu-standards, gnuift, gsequencer, gss, gstreamer0.10, gstreamer1.0, harden-doc, haskell98-report, iproute2, java-policy, libbluray, libmodbus, lizardfs, mclibs, moon-buggy, nurpawiki, php-sasl, shishi, stealth, xmltex, xsom. The following packages became reproducible after getting fixed:

• adblock-plus/2.7.1+dfsg-1 uploaded by David Pr vot, original patch by Dhole.
• gyoto/1.0.2-2 uploaded by Thibaut Paumard, original patch by Chris Lamb.
• libosmocore/0.9.0-4 by Ruben Undheim.
• libsyncml/0.5.4-2.3 uploaded by Mattia Rizzolo, original patch by akira.
• ltsp/5.5.6-2 by Vagrant Cascadian.
• mira/4.9.5-5 by Michael R. Crusoe.
• pagekite/0.5.8a-1 uploaded by Petter Reinholdtsen, original patch by Chris Lamb.
• plexus-containers/1.0~beta3.0.7-8 by Emmanuel Bourg.
• propellor/2.15.4-1 by Sean Whitton.
• salmon/0.4.2+ds1-2 uploaded by Michael R. Crusoe, original patch by Chris Lamb.
• wmii-doc/1:1-15 by Reiner Herrmann.

Some uploads fixed some reproducibility issues, but not all of them:

• dipy/0.10.1-1 uploaded by Yaroslav Halchenko, original patch by Juna Picca.
• suomi-malaga/2.0-1 uploaded by Timo Jyrinki, original patch by Chris Lamb.
• west-chamber/20100405+svn20111107.r124-7 by Ying-Chun Liu, original patch by Chris Lamb.

Patches submitted which have not made their way to the archive yet:

• #813944 on cvm by Reiner Herrmann: remove gzip headers, fix permissions of some directories and the order of the md5sums.
• #814019 on latexdiff by Reiner Herrmann: remove the current build date from documentation.
• #814214 on rocksdb by Chris Lamb: add support for SOURCE_DATE_EPOCH.

reproducible.debian.net A new armhf build node has been added (thanks to Vagrant Cascadian) and integrated into the Jenkins setup for 4 new armhf builder jobs. (h01ger) All packages for Debian testing (Stretch) have been tested on armhf in just 42 days. It took 114 days to get the same point for unstable back when the armhf test infrastructure was much smaller. Package sets have been enabled for testing on armhf. (h01ger) Packages producing architecture-independent ( Arch:all ) binary packages together with architecture dependent packages targeted for specific architectures will now only be tested on matching architectures. (Steven Chamberlain, h01ger) As the Jenkins setup is now made of 252 different jobs, the overview has been split into 11 different smalller views. (h01ger)

Package reviews 222 reviews have been removed, 110 added and 50 updated in the previous week. 35 FTBFS reports were made by Chris Lamb, Danny Edel, and Niko Tyni.

Misc. The recordings of Ludovic Court s' talk at FOSDEM 16 about reproducible builds and GNU Guix is now available. One can also have a look at slides from Fabian Keil's talk about ElecrtroBSD and Baptiste Daroussin's talk about FreeBSD packages.

There are three of us living here, and the plan is to switch all three of us from wireless to wired ethernet Internet connections by transporting cabling through the ventilation system using this remote-controlled car. Here s an action shot we took. My housemate put on his Gaming playlist and it reached a very suitable crescendo as we guided the car through its final journey, pulling the last piece of cotton which we would then use to pull the final cabal through. The timing of the music couldn t have been better. It reflected the risk of losing control of the car as we moved the controller around to enable the car to pick up the radio signals. A further risk was caused by the car only being able to turn right, not left, and there not being enough room in the air duct for it to make a full-circle rotation. Our backup plan was to try tossing a ball tied to a piece of string down the duct. Unfortunately I m still on a wireless connection because to reach my room we d have had to send the cable via the furnace/air conditioning unit which we decided against. Does a SATA-to-USB converter, for moving data off my old laptop hard drive which I had to replace this week, need to be this complicated?

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In December, 113.50 work hours have been dispatched among 9 paid contributors. Their reports are available:

• Antoine Beaupr did 11.5h.
• Ben Hutchings did 15 hours (out of 15 hours allocated + 10 extra hours remaining, meaning that he still has 10 extra hours to do over February).
• Chris Lamb did 18 hours.
• Guido G nther did 13 hours (out of 12 hours allocated + 1 remaining).
• Mike Gabriel did 16 hours (8 hours allocated + 8h remaining).
• Santiago Ruano Rinc n did 18 hours (out of 12h allocated + 8.50 remaining, thus keeping 2.50 extra hours for February).
• Scott Kitterman did 8 hours.
• Thorsten Alteholz did 30 hours.

Evolution of the situation As expected, we had a small drop in the amount of hours sponsored. New sponsors (re-)joined but others stopped too (Gree this time) mostly balancing the result. We only lost 2 hours of sponsored work. It would be nice if we could invert that curve and actually start again to get closer to our objective of funding the equivalent of a full time position. Let s hope that the switch to wheezy as the version supported by the LTS team will motivate many companies relying on Debian 7 in their IT system. In terms of security updates waiting to be handled, the situation is close to last month(17 packages in dla-needed.txt, 27 in the list of CVE). It looks like that having about 20 packages needing an update is the normal situation and that we can t really get further down given the time required to process some updates (sometimes we wait until the upstream authors provides a patch, and so on). Thanks to our sponsors New sponsors are in bold.

• Platinum sponsors:
• TOSHIBA (for 4 months)
• Gold sponsors:
• The Positive Internet (for 20 months)
• Blablacar (for 19 months)
• Linode LLC (for 9 months)
• Silver sponsors:
• Domeneshop AS (for 19 months)
• Universit Lille 3 (for 19 months)
• Trollweb Solutions (for 17 months)
• Nantes M tropole (for 13 months, renewal after a small break)
• University of Luxembourg (for 11 months)
• Dalenys (for 9 months)
• Univention GmbH (for 5 months)
• Universit Jean Monnet de St Etienne (for 5 months)
• Bronze sponsors:
• David Ayers IntarS Austria (for 20 months)
• Offensive Security (for 20 months)
• Seznam.cz, a.s. (for 20 months)
• Evolix (for 19 months)
• Freeside Internet Service (for 19 months)
• MyTux (for 19 months)
• Linuxhotel GmbH (for 17 months)
• Intevation GmbH (for 16 months)
• Daevel SARL (for 15 months)
• Bitfolk LTD (for 14 months)
• Megaspace Internet Services GmbH (for 14 months)
• Greenbone Networks GmbH (for 13 months)
• NUMLOG (for 13 months)
• WinGo AG (for 12 months)
• Ecole Centrale de Nantes LHEEA (for 8 months)
• Sig-I/O (for 6 months)
• Entr ouvert (for 4 months)
• Adfinis SyGroup AG

No comment Liked this article? Click here. My blog is Flattr-enabled.