This is the story of libspotify, as experienced by a Spotify customer and libspotify developer for six years.

Step 1: Support your API for years Since April 2009, libspotify has been a mostly nice although proprietary C library for integrating with the Spotify music streaming service, providing both music metadata and full playback capabilities. Language wrappers have been written for a plenitude of programming languages (including my own pyspotify). libspotify is integrated into numerous open source projects (including my own Mopidy), networked AV receivers, and rumour has it: even cars. In addition to being closed source, there was another catch: in order to use it, you need the Spotify premium subscription. Speaking from my own experience supporting a project integrating with Spotify for the last six years: lots of end users upgraded to premium in order to use the projects built on top of libspotify.

Step 2: Stop accepting bug reports and stop releasing updates In May 2012, Spotify released libspotify 12.1.51 for all supported architectures, which now even included Android. After months of endless requests from the quickly growing hoards of Raspberry Pi users, Spotify released libspotify 12.1.103 for hardfloat armv6 on January 22 2013. The release included minor API additions, like the addition of explicit lyrics and means for accepting Spotify s Terms of Service. The additions in 12.1.103 never reached the other supported architectures and 12.1.51 remains the last release for those. There was no clear communication about this being the end of libspotify, releases have just ceased. libspotify has not been mentioned in a single developer blog post or developer email newsletter since. That s now more than three years ago.

Step 3: Create new APIs Over the next couple of years Spotify released a new Web API and iOS and Android SDKs. Through their developer blog, they communicated quite a bit both about these and about the deprecation of the apps inside the Spotify desktop client. Here are some pieces from their developer blog to illustrate:

• 2014-02-03: Deprecation of Spotify Apps API 0.x
• 2014-03-06: First Spotify iOS SDK beta release
• 2014-03-24: No longer accepting new Spotify Apps
• 2014-06-17: First Spotify Web API release
• 2014-06-25: First Spotify Android SDK beta release

The Web API grew steadily, as documented by the Web APIs changelog. Over time, its feature list improved, supporting anonymous endpoints for generic music metadata as well as OAuth protected endpoints for managing your own music collection and playlists. The only major part missing that prevented developers replacing libspotify was music playback, as the Web API only provides 30 second previews. From what I recall of 2014 IRC discussions with Spotify employees, they had built a new much smaller library for streaming music. The iOS and Android SDKs were simply platform specific wrappers around this new playback library and the new Web API. My impression is that, after a number of beta releases, the iOS and Android SDKs seamlessly replaced libspotify on those platforms. The plan was apparently to release the new playback library for desktop too, so that it, in combination with the Web API, could replace libspotify there as well. That was all back in 2014.

Step 4: Deprecate old APIs At some point, Spotify deprecated their old web API, the Metadata API. There was no blog post about it and it was not mentioned in any Spotify Developer News email I ve received in recent years. I guess they only published this fact as a note on the Metadata API web page; it is therefore hard to track down exactly when it was deprecated and how the deprecation was communicated. Anyway, it was deprecated, and the natural upgrade path was to the new Web API. Spotify even has a migration guide mapping the concepts of the Metadata API to the new Web API. All good. In May 2015, after 28 months without a single release or any news, libspotify is officially deprecated. The communication channel of choice? A note on the libspotify web page. No post on the developer blog. No Spotify Developer News email. To be able to use libspotify, all developers have registered and requested a personal application key. To my knowledge, none of these registered library users were notified directly.

Step 5: Shut down an old API On January 20 2016 Spotify end-of-lifed their Metadata API. I don t know if this date was communicated on the Metadata API web page well in advance of this shutdown. There s no blog post on the topic. There s no news email. There s no tweet from @SpotifyPlatform or @SpotifyStatus. I can t find anything except users commenting on various Spotify documentation pages that the text is outdated because the API is no longer merely deprecated, but entirely shut down. Hopefully they did publish the shutdown date ahead of time. It had a full replacement API and a migration guide. Fair and square. Nothing much to complain about here. It s time to get porting your code to the Web API! But frankly, I don t really care about this Metadata API because I don t use the Metadata API, do I?

Step 6: Break your API without warning February 3 2016, Mopidy users start reporting General transient error when searching Spotify. This is a libspotify error message known as SP_ERROR_OTHER_TRANSIENT with error code 16. This error usually means that there is some unspecified issue with the Spotify service. We ve actually seen it before and like its name suggests, it is indeed transient and soon goes away. According to a series of tweets between my fellow Mopidy developer Nick Steel and @SpotifyCares it appears that the libspotify search functionality was backed by the Metadata API s search functionality. Who knew? I m not aware of any relation between these APIs ever being mentioned publicly. No libspotify user could possibly have known that the Metadata API shutdown would affect them. Let s summarize: libspotify hasn t had any releases for three years and it was officially deprecated in May last year. But, there have been no warnings about any shutdown. The libspotify project is likely no longer staffed at all. Did someone just forget that libspotify depended on the Metadata API they were shutting down? Many newer networked AV receivers use the Spotify Connect libraries that are exclusive to Spotify s commercial partners. But some receivers probably still use libspotify. If so, did Spotify just break hardware sold with Spotify support as a major feature? Something that people are paying a monthly fee to use? It seems they did! In the Spotify Community forums there are numerous reports of broken devices and software that all have two things in common: the date the problem started occurring, and that search is the only feature that is broken.

• Denon receivers, including models: CEOL N8
• Logitech Squeezebox media players
• Marantz receivers, including models: M-CR510, M-CR610, NR1504, NR1603, SR5007, SR6008
• Onkyo receivers, including CR-N755, TX-NR509, TX-NR609, TX-8050
• Roku media players

Surely, Spotify will quickly explain themselves and fix this?

Step 7: Stay quiet Since the API shutdown, Spotify has been very quiet. No posts on the developer blog. No email newsletter. No tweets from @SpotifyPlatform since December 1 last year. Some users at The Spotify Community forum are quoting responses from Spotify. Most responses seem to link to the Metadata API migration docs and state that application developers must migrate. In other cases the responses point the finger at the commercial partner which only adds to the confusion. Some of these devices may use the Metadata API directly, and in those cases the application maintainers are probably to blame. However, I m quite certain that the libspotify search issue is responsible for a fair share of these complaints. libspotify, in contrast to the Metadata API, has never been officially shut down. The responsibility for any breakage caused by libspotify is Spotify s alone.

What I want from Spotify

1. Apologize to your paying customers and supporters.
2. Get substantially better at communicating deprecations and shutdowns. We want all news, not just good news.
3. Give us a new supported C library for audio playback. Now! Not later this year. I ve heard that promise for three consecutive years now. We can live with pre-release software quality initially. Getting the library out sooner rather than later shows us that there is a road ahead for all the projects that stream music from Spotify. Let us get started making new language wrappers.
4. Release Spotify Connect specifications and libraries. We want to make music players that can be controlled by the Spotify mobile app. We want to make music controllers that can play music on all the Spotify Connect software and hardware players we ve invested in. You ve created this technology, now let us use it.

Thanks to Nick Steel for reviewing this blog post.

Great news! The HA stack has been finally updated and you can find now both pacemaker & corosync in Debian stretch.

This is thanks to the hard work of some people, specially Christoph Berg, Ferenc W gner, Martin Loschwitz and others.

By the time of this blogpost, in testing (stretch) you have:

• pacemaker 1.1.14-1
• corosync 2.3.5-3
• crmsh 2.2.0-1

• pcs 0.9.148-1

what to do now

During data processing in our team, we often subtract plain polynomial baselines to our data, drawn by hand using the b command. Note that this is legitimate because we know that the signal we are looking for quickly drops to 0 outside a given range. The usual workflow is to just load the data, use b to draw a baseline and subtract it, and then have further fun with the baseline-subtracted data. This was the only possibility in the old SOAS. However, I don't like so much this kind of workflow, because such a baseline is arbitrary, and once it's subtracted, it's hard to reassess the baseline afterwards.
This is why I've designed a better way in QSoas: when you are done creating a baseline, push p: this exits pushing only the interpolation nodes:
Above, the baseline subtraction interface, and below, the resulting nodes:
The nodes can be reloaded by hitting uppercase L from within b and giving a buffer number (admittedly, the interface as of now isn't that good, I'll have to work on it). Alternatively, you can greenerate the baseline by using interpolate, and then further subtract it using S:

QSoas> interpolate 1 0

where 1 is the original buffer (in fact, only its X values will be used, so you don't have to use the same buffer) and 0 is the buffer containing the interpolation nodes. Saving only the interpolation nodes makes it possible to reedit the baseline easily, and regenerate the baseline-subtracted data using a script. This is my favorite approach, because I like to be able to trace every single step from the raw data to processed data. Another neat feature is that, from within b, you can reload only the X values of nodes using lowercase l (and giving buffer number): the Y values are computed as if you had clicked at the corresponding positions. This is very useful is you have a whole series of similar baselines to make: most often, you won't even need manual adjustment to get a good baseline. I hope you'll find this tip useful !

To state the obvious: my personal preference is to run Debian GNU/Linux. My current workplace is a CentOS shop and usually I'm the first to claim that it doesn't matter at all, and distribution specific implementation details are irrelevant for what we do (running a JVM). Let's take a short detour to the RedHat network-scripts. Two weeks ago we found some systems we originally installed in a different network segment, with different DNS servers and different search domains, came back up after a reboot with a rewritten 'resolv.conf'. Later on cfengine replaced the generated 'resolv.conf' with the intended one, so it wasn't that obvious to spot in the first place. A colleague found the origin of the rewritten 'resolv.conf' in a device specific configuration file that defined the 'DNS 1,2 ' variables with the installation time DNS server IPs from the other segment. I expected to experience the same behaviour (resolv.conf rewritten during startup and replaced by cfengine later on) in other locations, but assumed we just didn't notice it because the main difference would be a slightly different list of search domains. And I was wrong. I checked the timestamps of several 'resolv.conf' files and their cfengine backup file. None were recently created or related somehow to a reboot. grep-ing through parts of the network-scripts I found the following conditional in '/etc/sysconfig/network-scripts/ifup'

if [ "$PEERDNS" != "no" ]  [ -n "$RESOLV_MODS" -a "$RESOLV_MODS" != "no" ]; then
   [ -n "$MS_DNS1" ] && DNS1=$MS_DNS1
   [ -n "$MS_DNS2" ] && DNS2=$MS_DNS2
      if [ -n "$DNS1" ] && ! grep -q "^nameserver $DNS1" /etc/resolv.conf &&
         tr=$(mktemp /tmp/XXXXXX) ; then
 ...

So if you adjust only the second nameserver IP you stored in "DNS2" in your configuration you end up without an update to your 'resolv.conf'. Now knowing that, I'd say this is relevant distribution specific knowledge, and I'm wondering how many of such subtle behaviours we've hidden in Debian specific solutions? Maybe knowledge about distribution specific implementation details even matters after all. Regardig the 'resolv.conf' issue itself the fault is on us. We as the responsible team did not read the documentation properly and thus deployed a configuration that later on let to some unexpected consequences. I try to remember those issues as an example for the next NetworkManager/systemd-networkd vs old-school-network-scripts argument.

The first major version of the scalable timeserie database I work on, Gnocchi was a released a few months ago. In this first iteration, it took a rather naive approach to data storage. We had little ideas about if and how our distributed back-ends were going to be heavily used, so we stuck to the code of the first proof-of-concept written a couple of years ago. Recently we got more feedbacks from our users, ran a few benchmarks. That gave us enough feedback to start investigating in improving our storage strategy. Data split Up to Gnocchi 1.3, all data for a single metric are stored in a single gigantic file per aggregation method (min, max, average ). This means that the file can grow to several megabytes in size, which make it slow to manipulate. For the next version of Gnocchi, our first work has been to rework that storage and split the data into smaller parts. The diagram above shows how data are organized inside Gnocchi. Until version 1.3, there would have been only one file for each aggregation methods. In the upcoming 2.0 version, Gnocchi will split all these data into smaller parts, where each data split is stored in a file/object. This allows to manipulate smaller pieces of data and to increase the parallelism of the CRUD operations on the back-end leading to large speed improvement. In order to split timeseries into several chunks, Gnocchi defines a maximum number of N points to keep per chunk, to limit their maximum size. It then defines a hash function that produces a non-unique key for any timestamp. It makes it easy to find in which chunk any timestamp should be stored or retrieved. Data compression Up to Gnocchi 1.3, the data stored for each metric is simply serialized using msgpack, a fast and small serialization format. Though, this format does not provide any compression. That means that storing data points needs 8 bytes for a timestamp (64 bits timestamp with nanosecond precision) and 8 bytes for a value (64 bits double-precision floating-point), plus some overhead (extra information and msgpack itself). After looking around on how to compress all these measures, I stumbled upon a paper from some Facebook engineers called about Gorilla, their in-memory timeserie database, entitled "Gorilla: A Fast, Scalable, In-Memory Time Series Database". For reference, part of this encoding is also used by InfluxDB in its new storage engine. The first technique I implemented is easy enough, and it's inspired from delta-of-delta encoding. Instead of storing each timestamp for each data point, and since all the data points are aggregated on a regular interval, we transpose points to be the time difference divided by the interval. For example, the suite of timestamps timestamps = [41230, 41235, 41240, 41250, 41255] is encoded into timestamps = [41230, 1, 1, 2, 1], interval = 5. This allows regular compression algorithms to reduce the size of the integer list using run-length encoding. To actually compress the values, I tried two different algorithms:

• [LZ4](https://en.wikipedia.org/wiki/LZ4_(compression_algorithm), a fast compression/decompression algorithm
• The XOR based compression scheme described in the Gorilla paper mentioned above that I had to implement myself. For reference, it also exists a Go implementation in go-tsz.

I then benchmarked these solutions: The XOR algorithm implemented in Python is pretty slow, compared to LZ4. Truth is that python-lz4 is fully implemented in C, which makes it fast. I've profiled my XOR implementation in Python, to discover that one operation took 20 % of the time: count_lead_and_trail_zeroes, which is in charge of counting the number of leading and trailing zeroes in a binary number. I tried 2 Python implementations of the same algorithm (and submitted them to my friend and Python developer Victor Stinner by the way). The first version using string search with .index() is 10 faster than the second one that only do integer computation. Ah, Python As Victor explained, each Python operation is slow and there's a lot in the second version, whereas .index() is implemented in C and really well optimized and only needs 2 Python operations. Finally, I ended up optimizing that code by leveraging cffi to use directly ffsll() and flsll(). That decreased the run-time of count_lead_and_trail_zeroes by 45 %, making the entire XOR compression code speed increased by a small 7 %. This is not enough to catch up with LZ4 speed. At this stage, the only solution to achieve a high-speed would probably to go with a full C implementation. Considering the compression ratio of the different algorithms, they are pretty much identical. The worst case scenario (random values) for LZ4 compress down to 9 bytes per data point, whereas XOR can go down to 7.38 bytes per data point. In general XOR encoding beats LZ4 by 15 %, except for cases where all values are 0 or 1. However, LZ4 is faster than XOR by a factor of 4 -70 depending on cases. That means that we'll use LZ4 for data compression in Gnocchi 2.0. It's possible that we could achieve as fast compression/decompression algorithm, but I don't think it's worth the effort right now it'd represent a lot of code to write and to maintain.

Just over a year ago I implemented an optimization to the SPI core code in Linux that avoids some needless context switches to a worker thread in the main data path that most clients use. This was really nice, it was simple to do but saved a bunch of work for most drivers using SPI and made things noticeably faster. The code got merged in v4.0 and that was that, I kept on kicking a few more ideas for optimizations in this area around but that was that until the past month. What happened then was that for whatever reason people started picking up v4.0 and using it in production more. On some systems people started seeing problems when there was heavy SPI flash usage, often during things like distribution installation. In some cases the lockup detector fired, but the most entertaining error was that on Marvell Orion systems (which are single core) when the flash was being heavily used the SATA controller started having trouble handling interrupts. These problems all bisected down to the key commit in that series, 0461a4149836c79 (spi: Pump transfers inside calling context for spi_sync()). The problem is that there are a number of widely deployed SPI controllers out there that don t support DMA and instead require the CPU to explicitly read and write everything sent to and from registers in the controller. To make matters worse these accesses to the controller will usually take many CPU cycles to complete, each one stalling the CPU while they happen. This is fine for short transfers or if the CPU has nothing else to do but on a busy multitasking system it s an issue. Before the optimization the switches between the worker thread interacting with the hardware and the thread initiating the SPI operations provided breaks in this activity which allowed other things to switch in. Unfortunately when we optimize those away then if there s a lot of work for the controller being done from one thread then that thread can run for a long time without pause. The fix for affected drivers if there are no less CPU intensive ways of driving the hardware is to add some explicit sleeps into the driver itself, either at the end of the transfer_one() or perhaps in an unprepare_message() function. In a way I was quite pleased to see this, it was a clear demonstration that the optimization was having the intended effect though obviously users of affected systems will not find that so comforting. It s not the first time that making things faster or fixing a bug has revealed an underlying problem, I m sure it won t be the last.

I found a bug in amanda-3.3.8 that was quickly fixed by upstream. Please upgrade to this version. The updates will go to collab-maint.The debs files for jessie are here: amanda-common_3.3.8-1_cal1_i386.deb amanda-server_3.3.8-1_cal1_i386.deb amanda-client_3.3.8-1_cal1_i386.deb Here comes the changelog:

amanda (1:3.3.8-1~cal1) unstable; urgency=low
  * New Upstream version
    * Changes for 3.3.8
      * s3 devices
          New NEARLINE S3-STORAGE-CLASS for Google storage.
          New AWS4 STORAGE-API
      * amcryptsimple
          Works with newer gpg2.
      * amgtar
          Default SPARSE value is NO if tar < 1.28.
          Because a bug in tar with some filesystem.
      * amstar
          support include in backup mode.
      * ampgsql
          Add FULL-WAL property.
      * Many bugs fix.
    * Changes for 3.3.7p1
      * Fix build in 3.3.7
    * Changes for 3.3.7
      * amvault
          new --no-interactivity argument.
          new --src-labelstr argument.
      * amdump
          compute crc32 of the streams and write them to the debug files.
      * chg-robot
          Add a BROKEN-DRIVE-LOADED-SLOT property.
      * Many bugs fix.
  * Refreshed patches.
  * Dropped patches that were applied by the upstream: fix-misc-typos,
    automake-add-missing, fix-amcheck-M.patch,
    fix-device-src_rait-device.c, fix-amreport-perl_Amanda_Report_human.pm
  * Change the email of the maintainer.
  * "wrap-and-sort -at" all control files.
  * swig is a new build depend.
  * Bump standard version to 3.9.6, no changes needed.
  * Replace deprecated dependency perl5 by perl, (Closes: #808209), thank
    you Gregor Herrmann for the NMU.
  * New patch fix-amgtar-exclude to fix amgtar when the exclude file don't
    exist.
 -- Jose M Calhariz <jose@calhariz.com>  Thu, 11 Feb 2016 17:52:08 +0000

This article describes how to use systemd-networkd on Debian Edu 8.x (aka jessie) notebooks. What we have to deal with? At the schools we support we have several notebooks running Debian Edu 8.x (aka jessie) in the field. For school notebooks (classroom sets) we install the Debian Edu Workstation Profile. Those machines are mostly used over wireless network. We know that Debian Edu also offers a Roaming Workstation Profile at installation time, but with that profile chosen, user logins create local user accounts and local home directories on the notebooks (package: libpam-mklocaluser). For our customers, we do not want that. People using the school notebooks shall always work on their NFS home directories. School notebooks shall not be usable outside of the school network. Our woes... The default setup on Debian Edu jessie workstations regarding networking is this:

• systemd runs as PID 1
• ifupdown manages static network interfaces (eth0, etc.)
• NetworkManager manages wireless network interfaces
• for our customers we configured NetworkManager with a system-wide WiFi (WPA2-PSK) profile

We have observed various problems with that setup:

• By default, network interface eth0 is managed by ifupdown (via /etc/network/interfaces):
  

  auto eth0
  iface eth0 inet dhcp
  

  Woe no. 1: In combination with systemd, this results in a 120sec delay at system startup.

read more

As well as the simple reading of a musical score is sufficient to an experienced musician to recognize the most velvety harmonic variations of an orchestral piece, so the apparent coldness of a fragment of program code can stimulate emotions of ecstatic contemplation in the developer.

Don t be misled by the simplicity of the laconic definition of the code as a sequence of instructions to be given to a computer to solve a specific problem. Generally, a problem has multiple solutions, the most simple and fast to implement, the most economical from the point of view of machine cycles or memory, the elegant solution and the makeshift one.

However, there is always a poetic solution, the one that has a particular and unusual beauty and that is always generated by the inexhaustible forge of the human intuition .

[Read More ]

In the last week of January, the project IT-Zukunft Schule (Logo EDV-Systeme GmbH and DAS-NETZWERKTEAM) had visitors from Norway: Klaus Ade Johnstad and Linnea Skogtvedt from LinuxAvdelingen [1] came by for exchanging insights, knowledge, technology, stories regarding IT services at school in Norway and Nothern Germany. This was our schedule... Tuesday

• 3pm Arrival of Klaus Ade and Linnea, meet up at LOGO with coffee and cake
• 4pm Planning the workshop, coming up with an agenda for the next two days (Klaus Ade, Andreas, Mike)
• 5pm Preparing OPSI demo sites (Mike, Linnea)
• 8pm Gr nkohl and Rotkohl and ... at Traum GmbH, Kiel (Klaus Ade, Linnea, Andreas, Mike)

Wednesday

• 8.30am more work on the OPSI demo site (Mike, Linnea)
• 10am pfSense (esp. captive portal functionality), backup solutions (Klaus Ade, all)
• 11am ITZkS overlay packages, basic principles of Debian packaging (Mike, special guests: Torsten, Lucian, Benni)
• 12am-2pm lunch break
• 2pm OPSI demonstration, discussion, foundation of the OpsiPackages project [2] (Mike)
• 4pm Puppet (Linnea)
• 7pm dinner time (eating in, Thai fast food :-) )
• 20pm Sepiida hacking (Mike, Linnea), customer care (Andreas, Klaus Ade)
• 22:30pm zZzZzZ time...

Thursday

read more

About 7 months ago, I wrote about we had improved Lintian s performance. In 2.5.41, we are doing another memory reduction, where we primarily reduce the memory consumption of data about ELF binaries. Like previously, memory reductions follows the less is more pattern. My initial test subject was linux-image-4.4.0-trunk-rt-686-pae_4.4-1~exp1_i386.deb. It had a somewhat unique property that the ELF data made up a little over half the cache.

• We do away with a lot of unnecessary default values [f4c57bb, 470875f]
  • That removed about ~3MB (out of 10.56MB) of that ELF data cache
• Discard section information we do not use [3fd98d9]
  • This reduced the ELF data cache to 2MB (down from the 7MB).
• Then we stop caching the output of file(1) twice [7c2bee4]
  • While a fairly modest reduction (only 0.80MB out of 16MB total), it also affects packages without ELF binaries.

At this point, we had reduced the total memory usage from 18.35MB to 8.92MB (the ELF data going from 10.56MB to 1.98MB)[1]. At this point, I figured that I was happy with the improvement and discarded my test subject. While impressive, the test subject was unsurprisingly a special case. The improvement in regular packages[2] (with ELF binaries) were closer to 8% in total. Not being satisfied with that, I pulled one more trick.

• Keep only UND and .text symbols [2b21621]
  • This brought coreutils (just the lone deb) another 10% memory reduction in total.

In the grand total, coreutils 8.24-1 amd64 went from 4.09MB to 3.48MB. The ELF data cache went from 3.38MB to 2.84MB. Similarly, libreoffice/4.2.5-1 (including its ~170 binaries) has also seen a 8.5% reduction in total cache size[3] and is now down to 260.48MB (from 284.83MB). [1] If you are wondering why I in 3fd98d9 wrote The total cache memory usage is approaching 1/3 of the original for that package , then you are not alone. I am not sure myself any more, but it seems obviously wrong. [2] FTR: The sample size of regular packages is 2 in this case. Of which one of them being coreutils [3] Admittedly, since take 2 and not since 2.5.40.2 like the rest.
Filed under: Debian, Lintian

Before the rest of the family was up, I took a brief excursion to explore the first kilometre of the Bluff Trail and check out conditions. I turned at the ridge, satisfied I had seen enough to give an idea of what it s like out there, and then walked back the four kilometres home on the BLT Trail. I saw three joggers and their three dogs just before I exited the Bluff Trail on the way back, and later, two young men on the BLT with day packs approaching. The parking lot had gained two more cars for a total of three as I headed home. Exercising appropriate caution and judgement, the first loop is beautiful and rewarding, and I m not alone in feeling the draw of its delights this crisp morning. Click the first photo below to start the slideshow.

Click to start the slideshow

While I sort out a sponsor, my sponsor is very busy, here is a preview of the new packages. So anyone can install and test them on jessie. The source of the packages is in collab-maint.The debs files for jessie are here: amanda-common_3.3.8-1_cal0_i386.deb amanda-server_3.3.8-1_cal0_i386.deb amanda-client_3.3.8-1_cal0_i386.deb Here comes the changelog:

amanda (1:3.3.8-1~cal0) unstable; urgency=low
  * New Upstream version
    * Changes for 3.3.8
      * s3 devices
          New NEARLINE S3-STORAGE-CLASS for Google storage.
          New AWS4 STORAGE-API
      * amcryptsimple
          Works with newer gpg2.
      * amgtar
          Default SPARSE value is NO if tar < 1.28.
          Because a bug in tar with some filesystem.
      * amstar
          support include in backup mode.
      * ampgsql
          Add FULL-WAL property.
      * Many bugs fix.
    * Changes for 3.3.7p1
      * Fix build in 3.3.7
    * Changes for 3.3.7
      * amvault
          new --no-interactivity argument.
          new --src-labelstr argument.
      * amdump
          compute crc32 of the streams and write them to the debug files.
      * chg-robot
          Add a BROKEN-DRIVE-LOADED-SLOT property.
      * Many bugs fix.
  * Refreshed patches.
  * Dropped patches that were applied by the upstream: fix-misc-typos,
    automake-add-missing, fix-amcheck-M.patch,
    fix-device-src_rait-device.c, fix-amreport-perl_Amanda_Report_human.pm
  * Change the email of the maintainer.
  * "wrap-and-sort -at" all control files.
  * swig is a new build depend.
  * Bump standard version to 3.9.6, no changes needed.
  * Replace deprecated dependency perl5 by perl, (Closes: #808209), thank
    you Gregor Herrmann for the NMU.
 -- Jose M Calhariz <jose@calhariz.com>  Tue, 02 Feb 2016 19:56:12 +0000

In January I carried over 10 hours from December and was assigned another 15 hours of work by Freexian's Debian LTS initiative. I worked a total of 15 hours. I had a few days on 'front desk' at the start of the month, as my week in that role spanned the new year. I fixed a regression in the kernel that was introduced to all stable suites in December. I uploaded this along with some minor security fixes, and issued DLA 378-1. I finished backporting and testing fixes to sudo for CVE-2015-5602. I uploaded an update and issued DLA 382-1, which was followed by DSA 3440-1 for wheezy and jessie. I finished backporting and testing fixes to Claws Mail for CVE-2015-8614 and CVE-2015-8708. I uploaded an update and issued DLA 383-1. This was followed by DSA 3452-1 for wheezy and jessie, although the issues are less serious there. I also apent a little time on InspIRCd, though this isn't a package that Freexian's customers care about and it seems to have been broken in squeeze for several years due to a latent bug in the build system. I had already backported the security fix by the time I discovered this, so I went ahead with an update fixing that regression as well, and issued DLA 384-1. Finally, I diagnosed the regression in the update to isc-dhcp in DLA 385-1.

At LCA I attended a talk about Unikernels. Here are the reasons why I think that they are a bad idea: Single Address Space According to the Unikernel Wikipedia page [1] a significant criteria for a Unikernel system is that it has a single address space. This gives performance benefits as there is no need to change CPU memory mappings when making system calls. But the disadvantage is that any code in the application/kernel can access any other code directly. In a typical modern OS (Linux, BSD, Windows, etc) every application has a separate address space and there are separate memory regions for code and data. While an application can request the ability to modify it s own executable code in some situations (if the OS is configured to allow that) it won t happen by default. In MS-DOS and in a Unikernel system all code has read/write/execute access to all memory. MS-DOS was the least reliable OS that I ever used. It was unreliable because it performed tasks that were more complex than CP/M but had no memory protection so any bug in any code was likely to cause a system crash. The crash could be delayed by some time (EG corrupting data structures that are only rarely accessed) which would make it very difficult to fix. It would be possible to have a Unikernel system with non-modifyable executable areas and non-executable data areas and it is conceivable that a virtual machine system like Xen could enforce that. But that still wouldn t solve the problem of all code being able to write to all data. On a Linux system when an application writes to the wrong address there is a reasonable probability that it will not have write access and you will immediately get a SEGV which is logged and informs the sysadmin of the address of the crash. When Linux applications have bugs that are difficult to diagnose (EG buffer overruns that happen in production and can t be reproduced in a test environment) there are a variety of ways of debugging them. Tools such as Valgrind can analyse memory access and tell the developers which code had a bug and what the bug does. It s theoretically possible to link something like Valgrind into a Unikernel, but the lack of multiple processes would make it difficult to manage. Debugging A full Unix environment has a rich array of debugging tools, strace, ltrace, gdb, valgrind and more. If there are performance problems then tools like sysstat, sar, iostat, top, iotop, and more. I don t know which of those tools I might need to debug problems at some future time. I don t think that any Internet facing service can be expected to be reliable enough that it will never need any sort of debugging. Service Complexity It s very rare for a server to have only a single process performing the essential tasks. It s not uncommon to have a web server running CGI-BIN scripts or calling shell scripts from PHP code as part of the essential service. Also many Unix daemons are not written to run as a single process, at least threading is required and many daemons require multiple processes. It s also very common for the design of a daemon to rely on a cron job to clean up temporary files etc. It is possible to build the functionality of cron into a Unikernel, but that means more potential bugs and more time spent not actually developing the core application. One could argue that there are design benefits to writing simple servers that don t require multiple programs. But most programmers aren t used to doing that and in many cases it would result in a less efficient result. One can also argue that a Finite State Machine design is the best way to deal with many problems that are usually solved by multi-threading or multiple processes. But most programmers are better at writing threaded code so forcing programmers to use a FSM design doesn t seem like a good idea for security. Management The typical server programs rely on cron jobs to rotate log files and monitoring software to inspect the state of the system for the purposes of graphing performance and flagging potential problems. It would be possible to compile the functionality of something like the Nagios NRPE into a Unikernel if you want to have your monitoring code running in the kernel. I ve seen something very similar implemented in the past, the CA Unicenter monitoring system on Solaris used to have a kernel module for monitoring (I don t know why). My experience was that Unicenter caused many kernel panics and more downtime than all other problems combined. It would not be difficult to write better code than the typical CA employee, but writing code that is good enough to have a monitoring system running in the kernel on a single-threaded system is asking a lot. One of the claimed benefits of a Unikernel was that it s supposedly risky to allow ssh access. The recent ssh security issue was an attack against the ssh client if it connected to a hostile server. If you had a ssh server only accepting connections from management workstations (a reasonably common configuration for running servers) and only allowed the ssh clients to connect to servers related to work (an uncommon configuration that s not difficult to implement) then there wouldn t be any problems in this regard. I think that I m a good programmer, but I don t think that I can write server code that s likely to be more secure than sshd. On Designing It Yourself One thing that everyone who has any experience in security has witnessed is that people who design their own encryption inevitably do it badly. The people who are experts in cryptology don t design their own custom algorithm because they know that encryption algorithms need significant review before they can be trusted. The people who know how to do it well know that they can t do it well on their own. The people who know little just go ahead and do it. I think that the same thing applies to operating systems. I ve contributed a few patches to the Linux kernel and spent a lot of time working on SE Linux (including maintaining out of tree kernel patches) and know how hard it is to do it properly. Even though I m a good programmer I know better than to think I could just build my own kernel and expect it to be secure. I think that the Unikernel people haven t learned this.

• [1] https://en.wikipedia.org/wiki/Unikernel

This post was written by Dirk Eddelbuettel and Christoph Sax and will be posted on both author's respective blogs. The seasonal package by Christoph Sax brings a very featureful and expressive interface for working with seasonal data to the R environment. It uses the standard tool of the trade: X-13ARIMA-SEATS. This powerful program is provided by the statisticians of the US Census Bureau based on their earlier work (named X-11 and X-12-ARIMA) as well as the TRAMO/SEATS program by the Bank of Spain. X-13ARIMA-SEATS is probably the best known tool for de-seasonalization of timeseries, and used by statistical offices around the world. Sadly, it also has a steep learning curve. One interacts with a basic command-line tool which users have to download, install and properly reference (by environment variables or related means). Each model specification has to be prepared in a special 'spec' file that uses its own, cumbersome syntax. As seasonal provides all the required functionality to use X-13ARIMA-SEATS from R --- see the very nice seasonal demo site --- it still required the user to manually deal with the X-13ARIMA-SEATS installation. So we decided to do something about this. A pair of GitHub repositories provide both the underlying binary in a per-operating system form (see x13prebuilt) as well as a ready-to- use R package (see x13binary) which uses the former to provide binaries for R. And the latter is now on CRAN as package x13binary ready to be used on Windows, OS-X or Linux. And the seasonal package (in version 1.2.0 -- now on CRAN -- or later) automatically makes use of it. Installing seasaonal and x13binary in R is now as easy as:

install.packages("seasonal")

which opens the door for effortless deployment of powerful deasonalization. By default, the principal function of the package employs a number of automated techniques that work well in most circumstances. For example, the following code produces a seasonal adjustment of the latest data of US retail sales (by the Census Bureau) downloaded from Quandl:

library(seasonal) 
library(Quandl)   ## not needed for seasonal but has some niceties for Quandl data
rs <- Quandl(code="USCENSUS/BI_MARTS_44000_SM", type="ts")/1e3
m1 <- seas(rs)
plot(m1, main = "Retail Trade: U.S. Total Sales", ylab = "USD (in Billions)")

This tests for log-transformation, performs an automated ARIMA model search, applies outlier detection, tests and adjusts for trading day and easter effects, and invokes the SEATS method to perform seasonal adjustment. And this is how the adjusted series looks like: Of course, you can access all available options of X-13ARIMA-SEATS as well. Here is an example where we adjust the latest data for Chinese exports (as tallied by the US FED), taking into account the different effects of Chinese New Year before, during and after the holiday:

xp <- Quandl(code="FRED/VALEXPCNM052N", type="ts")/1e9
m2 <- seas(window(xp, start = 2000),
          xreg = cbind(genhol(cny, start = -7, end = -1, center = "calendar"), 
                       genhol(cny, start = 0, end = 7, center = "calendar"), 
                       genhol(cny, start = 8, end = 21, center = "calendar")
          ),
          regression.aictest = c("td", "user"),
          regression.usertype = "holiday")
plot(m2, main = "Goods, Value of Exports for China", ylab = "USD (in Billions)")

which generates the following chart demonstrating a recent flattening in export activity measured in USD. We hope this simple examples illustrates both how powerful a tool X-13ARIMA-SEATS is, but also just how easy it is to use X-13ARIMA-SEATS from R now that we provide the x13binary package automating its installation.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS I did not ask for any paid hours this month and won t be requesting paid hours for the next 5 months as I have a big project to handle with a deadline in June. That said I still did a few LTS related tasks:

• I uploaded a new version of debian-security-support (2016.01.07) to officialize that virtualbox-ose is no longer supported in Squeeze and that redmine was not really supportable ever since we dropped support for rails.
• Made a summary of the discussion about what to support in wheezy and started a new round of discussions with some open questions. I invited contributors to try to pickup one topic, study it and bring the discussion to some conclusion.
• I wrote a blog post to recruit new paid contributors. Brian May, Markus Koschany and Damyan Ivanov candidated and will do their first paid hours over February.

Distro Tracker Due to many nights spent on playing Splatoon (I m at level 33, rank B+, anyone else playing it?), I did not do much work on Distro Tracker. After having received the bug report #809211, I investigated the reasons why SQLite was no longer working satisfactorily in Django 1.9 and I opened the upstream ticket 26063 and I had a long discussion with two upstream developers to find out the best fix. The next point release (1.9.2) will fix that annoying regression. I also merged a couple of contributions (two patches from Christophe Siraut, one adding descriptions to keywords, cf #754413, one making it more obvious that chevrons in action items are actionable to show more data, a patch from Balasankar C in #810226 fixing a bad URL in an action item). I fixed a small bug in the unsubscribe command of the mail bot, it was not properly recognizing source packages. I updated the task notifying of new upstream versions to use the data generated by UDD (instead of the data generated by Christoph Berg s mole-based implementation which was suffering from a few bugs). Debian Packaging Testing experimental sbuild. While following the work of Johannes Schauer on sbuild, I installed the version from experimental to support his work and give him some feedback. In the process I uncovered #810248. Python sponsorship. I reviewed and uploaded many packages for Daniel Stender who keeps doing great work maintaining prospector and all its recursive dependencies: pylint-common, python-requirements-detector, sphinx-argparse, pylint-django, prospector. He also prepared an upload of python-bcrypt which I requested last month for Django. Django packaging. I uploaded Django 1.8.8 to jessie-backports.
My stable updates for Django 1.7.11 was not handled before the release of Debian 8.3 even though it was filed more than 1.5 months before. Misc stuff. My stable update for debian-handbook has been accepted fairly shortly after my last monthly report (thank you Adam!) so I uploaded the package once acked by a release manager. I also sponsor a backports upload of zim prepared by Joerg Desch. Kali related work Kernel work. The switch to Linux 4.3 in Kali resulted in a few bug reports that I investigated with the help of #debian-kernel and where I reported my findings back so that the Debian kernel could also benefit from the fixes I uploaded to Kali: first we included a patch for a regression in the vmwgfx video driver used by VMWare virtual machines (which broke the gdm login screen), then we fixed the input-modules udeb to fix support of some Logitech keyboards in debian-installer (see #796096). Misc work. I made a non-maintainer upload of python-maxminddb to fix #805689 which had been removed from stretch and that we needed in Kali. I also had to NMU libmaxminddb since it was no longer available on armel and we actually support armel in Kali. During that NMU, it occurred to me that dh-exec could offer a feature of optional install , that is installing a file that exists but not failing if it doesn t exist. I filed this as #811064 and it stirred up quite some debate. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

In January 2016 I was finally able to work on various FLOSS topics again (after two months of heavily focussed local customer work):

• Upload of MATE 1.12 to Debian unstable
• Debian LTS packaging and front desk work
• Other Debian activies
• Edu Workshop in Kiel
• Yet another OPSI Packaging Project

Upload of MATE 1.12 to Debian testing/unstable At the beginning of the new year, I finalized the bundle upload of MATE 1.12 to Debian unstable [1]. All uploaded packages are available in Debian testing (stretch) and Ubuntu xenial by now. MATE 1.12 will also be the version shipped in Ubuntu MATE 16.04 LTS. Additionally, I finally uploaded caja-dropbox to Debian unstable (non-free), thanks to Vangelis Mouhtsis and Martin Wimpress for doing first steps preparations. The package has already left Debian's NEW queue, but unfortunately has been removed from Debian testing (stretch) again due to build failures in one of its dependencies. Debian LTS work In January 2016 I did my first round of Debian LTS front desk work [2]. Before actually starting with my front desk duty, I worked myself through the documentation and found it difficult to understand the output of the lts-cve-triage.py script. So, I proposed various improvments to the output of that script (all committed by now). During the second week of January then, I triaged the following packages regarding known/open CVE issues:

• isc-dhcp (CVE-2015-8605)
• gosa (CVE-2015-8771, CVE-2014-9760)
• roundcube (CVE-2015-8770)
  

read more

As part of my master class on Free and Open Source (FOSS) Software at University Paris Diderot, I invite guest lecturers to present to my students the point of views of various actors of the FOSS ecosystem --- companies, non-profits, activists, lawyers, etc. Tomorrow, Tuesday 2 February 2016, the students will have the pleasure to have as guest lecturer John Sullivan, Executive Director of the Free Software Foundation, talking about Overthrowing the tyranny of software: Why (and how) free societies respect computer user freedom. The lecture is open to everyone interested, but registration is recommended. Logistic and registration information, as well as the lecture abstract in both English and French is reported below.

---

John Sullivan's Lecture at University Paris Diderot - Overthrowing the tyranny of software: Why (and how) free societies respect computer user freedom John Sullivan, Executive Director of the Free Software Foundation will give a lecture titled "Overthrowing the tyranny of software: Why (and how) free societies respect computer user freedom" at University Paris Diderot next Tuesday, 2 February 2016, at 12:30 in the Amphi 3B, Halle aux Farines building, Paris 75013. Map at: http://www.openstreetmap.org/way/62378611#map=19/48.82928/2.38183 The lecture will be in English and open to everyone, but registration is recommended at https://framadate.org/iPqfjNTz2535F8u4 or via email writing to zack@pps.univ-paris-diderot.fr. Abstract: Anyone who has used a computer for long has at least sometimes felt like a helpless subject under the tyrant of software, screaming (uselessly) in frustration at the screen to try and get the desired results. But with driverless cars, appliances which eavesdrop on conversations in our homes, mobile devices that transmit our location when we are out and about, and computers with unexpected hidden "features", our inability to control the software supposedly in our possession has become a much more serious problem than the superficial blue-screen-of-death irritations of the past. Software which is free "as in freedom" allows anyone who has it to inspect the code and even modify it -- or ask someone trained in the dark arts of computer programming to do it for them -- so that undesirable behaviors can be removed or defused. This characteristic, applied to all software, should be a major part of foundation of free societies moving forward. To get there, we'll need individual developers, nonprofit organizations, governments, and companies all working together -- with the first two groups leading the way.

---

Cours Magistral de John Sullivan l'Universit Paris Diderot - Surmonter la tyrannie du logiciel: pourquoi (et comment) les soci t s libres respectent les libert s des utilisateurs John Sullivan, Directeur Ex cutif de la Free Software Foundation donnera un cours magistral ayant pour titre "Surmonter la tyrannie du logiciel: pourquoi (et comment) les soci t s libres respectent les libert s des utilisateurs" l'Universit Paris Diderot Mardi prochain, 2 f vrier 2016, 12h30 dans l'Amphi 3B de la Halle aux Farines, Paris 75013. Plan: http://www.openstreetmap.org/way/62378611#map=19/48.82928/2.38183 Le cours (en langue Anglaise) sera ouvert toutes et tous, mais l'inscription est recommand via le formulaire https://framadate.org/iPqfjNTz2535F8u4 ou par mail l'adresse zack@pps.univ-paris-diderot.fr. R sum : Chacun de nous, au moins une fois dans sa vie, a pest contre son ordinateur dans l'espoir (vain) d'obtenir un r sultat attendu, se sentant d poss de par un tyran logiciel. Mais au jour d'aujourd'hui - avec des voitures autonomes, des dispositifs "intelligents" que nous coutent chez nous, des portables qui transmettent notre position quand nous nous baladons, et des ordinateurs pleins des fonctionnalit s cach es - notre incapacit de contr ler nos biens devient une question beaucoup plus s rieuse par rapport a l'irritation qu'auparavant nous causait l' cran bleu de la mort. Le logiciel libre permet chaque utilisateur d' tudier son fonctionnement et de le modifier --- ou de demander des experts dans la magie noire de la programmation de le faire a sa place --- supprimant, ou du moins r duisant, les comportements ind sir s du logiciel. Cette caract ristique du logiciel libre devrait tre appliqu e chaque type de logiciel et devrait constituer un pilier des soci t s se pr tendant libres. Pour achever cet id al, d veloppeurs, organisations but non lucratif, gouvernements et entreprises doivent travailler ensemble. Et les d veloppeurs et les ONG doivent se positionner au premier rang dans ce combat.