There have been a two long threads on the debian-devel mailing list about the representation of the changes to upstream source code made by Debian maintainers. Here are a few notes for my own reference. I spent a lot of time defending the workflow I described in dgit-maint-merge(7) (which was inspired by this blog post). However, I came to be convinced that there is a case for a manually curated series of patches for certain classes of package. It will depend on how upstream uses git (rebasing or merging) and on whether the Debian delta from upstream is significant and/or long-standing. I still think that we should be using dgit-maint-merge(7) for leaf or near-leaf packages, because it saves so much volunteer time that can be better spent on other things. When upstream does use a merging workflow, one advantage of the dgit-maint-merge(7) workflow is that Debian s packaging is just another branch of development. Now consider packages where we do want a manually curated patch series. It is very hard to represent such a series in git. The only natural way to do it is to continually rebase the patch series against an upstream branch, but public branches that get rebased are not a good idea. The solution that many people have adopted is to represent their patch series as a folder full of .diff files, and then use gbp pq to convert this into a rebasing branch. This branch is not shared. It is edited, rebased, and then converted back to the folder of .diff files, the changes to which are then committed to git. One of the advantages of dgit is that there now exists an official, non-rebasing git history of uploads to the archive. It would be nice if we could represent curated patch series as branches in the dgit repos, rather than as folders full of .diff files. But as I just described, this is very hard. However, Ian Jackson has the beginnings of a workflow that just might fit the bill.

Did you ever wonder where the web trafic really flow to reach the web servers, and who own the network equipment it is flowing through? It is possible to get a glimpse of this from using traceroute, but it is hard to find all the details. Many years ago, I wrote a system to map the Norwegian Internet (trying to figure out if our plans for a network game service would get low enough latency, and who we needed to talk to about setting up game servers close to the users. Back then I used traceroute output from many locations (I asked my friends to run a script and send me their traceroute output) to create the graph and the map. The output from traceroute typically look like this:

traceroute to www.stortinget.no (85.88.67.10), 30 hops max, 60 byte packets
 1  uio-gw10.uio.no (129.240.202.1)  0.447 ms  0.486 ms  0.621 ms
 2  uio-gw8.uio.no (129.240.24.229)  0.467 ms  0.578 ms  0.675 ms
 3  oslo-gw1.uninett.no (128.39.65.17)  0.385 ms  0.373 ms  0.358 ms
 4  te3-1-2.br1.fn3.as2116.net (193.156.90.3)  1.174 ms  1.172 ms  1.153 ms
 5  he16-1-1.cr1.san110.as2116.net (195.0.244.234)  2.627 ms he16-1-1.cr2.oslosda310.as2116.net (195.0.244.48)  3.172 ms he16-1-1.cr1.san110.as2116.net (195.0.244.234)  2.857 ms
 6  ae1.ar8.oslosda310.as2116.net (195.0.242.39)  0.662 ms  0.637 ms ae0.ar8.oslosda310.as2116.net (195.0.242.23)  0.622 ms
 7  89.191.10.146 (89.191.10.146)  0.931 ms  0.917 ms  0.955 ms
 8  * * *
 9  * * *
[...]

This show the DNS names and IP addresses of (at least some of the) network equipment involved in getting the data traffic from me to the www.stortinget.no server, and how long it took in milliseconds for a package to reach the equipment and return to me. Three packages are sent, and some times the packages do not follow the same path. This is shown for hop 5, where three different IP addresses replied to the traceroute request. There are many ways to measure trace routes. Other good traceroute implementations I use are traceroute (using ICMP packages) mtr (can do both ICMP, UDP and TCP) and scapy (python library with ICMP, UDP, TCP traceroute and a lot of other capabilities). All of them are easily available in Debian. This time around, I wanted to know the geographic location of different route points, to visualize how visiting a web page spread information about the visit to a lot of servers around the globe. The background is that a web site today often will ask the browser to get from many servers the parts (for example HTML, JSON, fonts, JavaScript, CSS, video) required to display the content. This will leak information about the visit to those controlling these servers and anyone able to peek at the data traffic passing by (like your ISP, the ISPs backbone provider, FRA, GCHQ, NSA and others). Lets pick an example, the Norwegian parliament web site www.stortinget.no. It is read daily by all members of parliament and their staff, as well as political journalists, activits and many other citizens of Norway. A visit to the www.stortinget.no web site will ask your browser to contact 8 other servers: ajax.googleapis.com, insights.hotjar.com, script.hotjar.com, static.hotjar.com, stats.g.doubleclick.net, www.google-analytics.com, www.googletagmanager.com and www.netigate.se. I extracted this by asking PhantomJS to visit the Stortinget web page and tell me all the URLs PhantomJS downloaded to render the page (in HAR format using their netsniff example. I am very grateful to Gorm for showing me how to do this). My goal is to visualize network traces to all IP addresses behind these DNS names, do show where visitors personal information is spread when visiting the page.

When I had a look around for options, I could not find any good free software tools to do this, and decided I needed my own traceroute wrapper outputting KML based on locations looked up using GeoIP. KML is easy to work with and easy to generate, and understood by several of the GIS tools I have available. I got good help from by NUUG colleague Anders Einar with this, and the result can be seen in my kmltraceroute git repository. Unfortunately, the quality of the free GeoIP databases I could find (and the for-pay databases my friends had access to) is not up to the task. The IP addresses of central Internet infrastructure would typically be placed near the controlling companies main office, and not where the router is really located, as you can see from the KML file I created using the GeoLite City dataset from MaxMind.

I also had a look at the visual traceroute graph created by the scrapy project, showing IP network ownership (aka AS owner) for the IP address in question. The graph display a lot of useful information about the traceroute in SVG format, and give a good indication on who control the network equipment involved, but it do not include geolocation. This graph make it possible to see the information is made available at least for UNINETT, Catchcom, Stortinget, Nordunet, Google, Amazon, Telia, Level 3 Communications and NetDNA.

In the process, I came across the web service GeoTraceroute by Salim Gasmi. Its methology of combining guesses based on DNS names, various location databases and finally use latecy times to rule out candidate locations seemed to do a very good job of guessing correct geolocation. But it could only do one trace at the time, did not have a sensor in Norway and did not make the geolocations easily available for postprocessing. So I contacted the developer and asked if he would be willing to share the code (he refused until he had time to clean it up), but he was interested in providing the geolocations in a machine readable format, and willing to set up a sensor in Norway. So since yesterday, it is possible to run traces from Norway in this service thanks to a sensor node set up by the NUUG assosiation, and get the trace in KML format for further processing.

Here we can see a lot of trafic passes Sweden on its way to Denmark, Germany, Holland and Ireland. Plenty of places where the Snowden confirmations verified the traffic is read by various actors without your best interest as their top priority. Combining KML files is trivial using a text editor, so I could loop over all the hosts behind the urls imported by www.stortinget.no and ask for the KML file from GeoTraceroute, and create a combined KML file with all the traces (unfortunately only one of the IP addresses behind the DNS name is traced this time. To get them all, one would have to request traces using IP number instead of DNS names from GeoTraceroute). That might be the next step in this project. Armed with these tools, I find it a lot easier to figure out where the IP traffic moves and who control the boxes involved in moving it. And every time the link crosses for example the Swedish border, we can be sure Swedish Signal Intelligence (FRA) is listening, as GCHQ do in Britain and NSA in USA and cables around the globe. (Hm, what should we tell them? :) Keep that in mind if you ever send anything unencrypted over the Internet. PS: KML files are drawn using the KML viewer from Ivan Rublev, as it was less cluttered than the local Linux application Marble. There are heaps of other options too. As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

I use openssh every day, be it the ssh command for connecting to remote hosts, or the scp command for uploading/downloading files. Once a day, or more, I forget that scp uses the non-obvious -P flag for specifying the port, not the -p flag that ssh uses. Enough is enough. I shall not file a bug report against the Debian openssh-client page, because no doubt compatibility with both upstream, and other distributions, is important. But damnit I've had enough. apt-get source openssh-client shows the appropriate code:

    fflag = tflag = 0;
    while ((ch = getopt(argc, argv, "dfl:prtvBCc:i:P:q12346S:o:F:")) != -1)
          switch (ch)  
          ..
          ..
            case 'P':
                    addargs(&remote_remote_args, "-p");
                    addargs(&remote_remote_args, "%s", optarg);
                    addargs(&args, "-p");
                    addargs(&args, "%s", optarg);
                    break;
          ..
          ..
            case 'p':
                    pflag = 1;
                    break;
          ..
          ..
          ..

Swapping those two flags around, and updating the format string appropriately, was sufficient to do the necessary. In other news I've done some hardware development, using both Arduino boards and the WeMos D1-mini. I'm still at the stage where I'm flashing lights, and doing similarly trivial things:

• Absolute Arduino Basic Tutorial
  • Recognizing the board.
  • Getting the development environment setup.
  • Running your first project.
• Absolute WeMos D1 Mini (ESP8266) Basic Tutorial
  • Recognizing the board.
  • Getting the development environment setup.
  • Installing the libraries/headers/examples
  • Writing your first WiFi project.

I have more complex projects planned for the future, but these are on-hold until the appropriate parts are delivered:

• MP3 playback.
• Bluetooth-speakers.
• Washing machine alarm.
• LCD clock, with time set by NTP, and relay control.

Even with a few LEDs though I've had fun, for example writing a trivial binary display.

Candy from Strangers

A few days ago I gave a talk at ESC https://www.endsummercamp.org/ about some reasons why I think that using software and especially libraries from the packages of a community managed distribution is important and much better than alternatives such as pypi, nmp etc. This article is a translation of what I planned to say before forgetting bits of it and luckily adding it back as an answer to a question :)

When I was young, my parents taught me not to accept candy from strangers, unless they were present and approved of it, because there was a small risk of very bad things happening. It was of course a simplistic rule, but it had to be easy enough to follow for somebody who wasn't proficient (yet) in the subtleties of social interactions.

One of the reasons why it worked well was that following it wasn't a big burden: at home candy was plenty and actual offers were rare: I only remember missing one piece of candy because of it, and while it may have been a great one, the ones I could have at home were also good.

Contrary to candy, offers of gratis software from random strangers are quite common: from suspicious looking websites to legit and professional looking ones, to platforms that are explicitly designed to allow developers to publish their own software with little or no checks.

Just like candy, there is also a source of trusted software in the Linux distributions, especially those lead by a community: I mention mostly Debian because it's the one I know best, but the same principles apply to Fedora and, to some measure, to most of the other distributions. Like good parents, distributions can be wrong, and they do leave room for older children (and proficient users) to make their own choices, but still provide a safe default.

Among the unsafe sources there are many different cases and while they do share some of the risks, they have different targets with different issues; for brevity the scope of this article is limited to the ones that mostly concern software developers: language specific package managers and software distribution platforms like PyPi, npm and rubygems etc.

These platforms are extremely convenient both for the writers of libraries, who are enabled to publish their work with minor hassles, and for the people who use such libraries, because they provide an easy way to install and use an huge amount of code. They are of course also an excellent place for distributions to find new libraries to package and distribute, and this I agree is a good thing.

What I however believe is that getting code from such sources and using it without carefully checking it is even more risky than accepting candy from a random stranger on the street in an unfamiliar neighbourhood.

The risk aren't trivial: while you probably won't be taken as an hostage for ransom, your data could be, or your devices and the ones who run your programs could be used in some criminal act causing at least some monetary damage both to yourself and to society at large.

If you're writing code that should be maintained in time there are also other risks even when no malice is involved, because each package on these platform has a different policy with regards to updates, their backwards compatibility and what can be expected in case an old version is found to have security issues.

The very fact that everybody can publish anything on such platforms is both their biggest strength and their main source of vulnerability: while most of the people who publish their libraries do so with good intentions, attacks have been described and publicly tested, such as the fun typo-squatting http://incolumitas.com/2016/06/08/typosquatting-package-managers/ one (archived on http://web.archive.org/web/20160801161807/http://incolumitas.com/2016/06/08/typosquatting-package-managers) that published harmless malicious code under common typos for famous libraries.

Contrast this with Debian, where everybody can contribute, but before they are allowed full unsupervised access to the archive they have to establish a relationship with the rest of the community, which includes meeting other developers in real life, at the very least to get their gpg keys signed.

This doesn't prevent malicious people from introducing software, but raises significantly the effort required to do so, and once caught people can usually be much more effectively prevented from repeating it than a simple ban on an online-only account can do.

It is true that not every Debian maintainer actually does a full code review of everything that they allow in the archive, and in some cases it would be unreasonable to expect it, but in most cases they are at least reasonably familiar with the code to do at least bug triage, and most importantly they are in an excellent position to establish a relationship of mutual trust with the upstream authors.

Additionally, package maintainers don't work in isolation: a growing number of packages are being maintained by a team of people, and most importantly there are aspects that involve potentially the whole community, from the fact that new packages that enter the distribution are publicity announced on a mailing list to the various distribution-wide QA efforts.

Going back to the language specific distribution platforms, sometimes even the people who manage the platform themselves can't be fully trusted to do the right thing: I believe everybody in the field remembers the npm fiasco https://lwn.net/Articles/681410/ where a lawyer letter requesting the removal of a package started a series of events that resulted in potentially breaking a huge amount of automated build systems.

Here some of the problems were caused by some technical policies that caused the whole ecosystem to be especially vulnerable, but one big issue was the fact that the managers of the npm platform are a private entity with no oversight from the user community.

Here not all distributions are equal, but contrast this with Debian, where the distribution is managed by a community that is based on a social contract https://www.debian.org/social_contract and is governed via democratic procedures established in its https://www.debian.org/devel/constitution.

Additionally, the long history of the distribution model means that many issues have already been met, the errors have already been done, and there are established technical procedures to deal with them in a better way.

So, shouldn't we use language specific distribution platforms at all? No! As developers we aren't children, we are adults who have the skills to distinguish between safe and unsafe libraries just as well as the average distribution maintainer can do. What I believe we should do is stop treating them as a safe source that can be used blindly and reserve that status to actual trustful sources like Debian, falling back to the language specific platforms only when strictly needed, and in that case:

actually check carefully what we are using, both by reading the code and by analysing the development and community practices of the authors;
if possible, share that work by becoming ourselves maintainers of that library in our favourite distribution, to prevent duplication of effort and to give back to the community whose work we get advantage from.

Edit: fixed broken typosquatting url

This weekend Toby Churchill kindly hosted a hacking weekend for OpenTAC myself, Michael Grzeschik, Steve McIntyre and Andy Simpkins got together to bring up the remaining bits of the hardware on the current board revision and get some of the low level tooling like production flashing for the FTDI serial ports on the board up and running. It was a very productive weekend, we verified that everything was working with only few small mods needed for the board . Personally the main thing I worked on was getting most of an initial driver for the EMC1701 written. That was the one component without Linux support and allowed us to verify that the power switching and measurement for the systems under test was working well. There s still at least one more board revision and quite a bit of software work to do (I m hoping to get the EMC1701 upstream for v4.8) but it was great to finally see all the physical components of the system working well and see it managing a system under test, this board revision should support all the software development that s going to be needed for the final board. Thanks to all who attended, Pengutronix for sponsoring Michael s attendance and Toby Churchill for hosting!

Big news: RcppTOML now works on Windows too! This package had an uneventful 2016 without a single update. Release 0.0.5 had come out in late 2015 and we had no bugs or issues to fix. We use the package daily in production: a key part of our parameterisation is in TOML files In the summer, I took one brief stab at building on Windows now that R sports itself a proper C++11 compiler on Windows too. I got stuck over the not-uncommon problem of incomplete POSIX and/or C++11 support with MinGW and g++-4.9. And sadly ... I appears I wasn't quite awake enough to realize that the missing functionality was right there exposed by Rcpp! Having updated that date / datetime functionality very recently, I was in a better position to realize this when Devin Pastoor asked two days ago. I was able to make a quick suggestion which he tested, which I then refined ... here we are: RcppTOML on Windows too! (For the impatient: CRAN has reported that it has built the Windows binaries, they should hit mirrors such as this CRAN package for RcppTOML shortly.) So what is this TOML thing, you ask? A file format, very suitable for configurations, meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now. But TOML is making good inroads with newer and more flexible projects. The Hugo static blog compiler is one example; the Cargo system of Crates (aka "packages") for the Rust language is another example. The new release updates the included cpptoml template header by Chase Geigle, brings the aforementioned Windows support and updates the Travis configuration. We also added a NEWS file for the first time so here are all changes so far:

Changes in version 0.1.0 (2017-01-05)

• Added Windows support by relying on Rcpp::mktime00() (#6 and #8 closing #5 and #3)
• Synchronized with cpptoml upstream (#9)
• Updated Travis CI support via newer run.sh

Changes in version 0.0.5 (2015-12-19)

• Synchronized with cpptoml upstream (#4)
• Improved and extended examples

Changes in version 0.0.4 (2015-07-16)

• Minor update of upstream cpptoml.h
• More explicit call of utils::str()
• Properly cope with empty lists (#2)

Changes in version 0.0.3 (2015-04-27)

• First CRAN release after four weeks of initial development

Courtesy of CRANberries, there is a diffstat report for this release. More information and examples are on the RcppTOML page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Falling off the cliff is always a surprise. I know it's there; I've been living next to it for months. I chart its outline daily. Avoiding it has become routine, and so comfortable, and so failing to avoid it surprises. Monday evening around 10 pm, the laptop starts draining down from 100%. House battery, which has been steady around 11.5-10.8 volts since well before Winter solstice, and was in the low 10's, has plummeted to 8.5 volts. With the old batteries, the cliff used to be at 7 volts, but now, with new batteries but fewer, it's in a surprising place, something like 10 volts, and I fell off it. Weather forecast for the week ahead is much like the previous week: Maybe a couple sunny afternoons, but mostly no sun at all. Falling off the cliff is not all bad. It shakes things up. It's a good opportunity to disconnect, to read paper books, and think long winter thoughts. It forces some flexability. I have an auxillary battery for these situations. With its own little portable solar panel, it can charge the laptop and run it for around 6 hours. But it takes it several days of winter sun to charge back up. That's enough to get me through the night. Then I take a short trip, and glory in one sunny afternoon. But I know that won't get me out of the hole, the batteries need a sunny week to recover. This evening, I expect to lose power again, and probably tomorrow evening too. Luckily, my goal for the week was to write slides for two talks, and I've been able to do that despite being mostly offline, and sometimes decomputered. And, in a few days I will be jetting off to Australia! That should give the batteries a perfect chance to recover. Previously: battery bank refresh late summer

Today Gammu 1.38.1 has been released. This is bugfix release fixing several minor bugs which were discovered in 1.38.0. The Windows binaries will be available shortly. These are built using AppVeyor and will help bring Windows users back to latest versions. Full list of changes and new features can be found on Gammu 1.38.1 release page. Would you like to see more features in Gammu? You an support further Gammu development at Bountysource salt or by direct donation.

Filed under: Debian English Gammu 0 comments

Weblate probably would not exist (or at least would be much harder to manage) without several services that help us to develop, improve and fix bugs in our code base. Over the time the development world has become very relying on cloud services. As every change this has both sides - you don't have to run the service, but you also don't have control on the service. Personally I'd prefer to use more free software services, on the other side I really love this comfort and I'm lazy to setup things which I can get for free. The list was written down mostly for showing up how we work and the services are not listed in any particular order. All of the services provide free offerings for free software projects or for limited usage. GitHub I guess there is not much to say here, it has become standard place to develop software - it has Git repositories, issue tracker, pull requests and several other features. Travis CI Running tests on every commit is something what will make you feel confident that you didn't break anything. Of course you still need to write the tests, but having them run automatically is really great help. Especially great for automatically checking pull requests. AppVeyor Continuous integration on Windows - it's still widely used platform with it's quirks, so it's really good idea to test there as well. With AppVeyor you can do that and it works pretty nicely. Codecov When running tests it's good to know how much of your code is covered by them. Codecov is one of the best interfaces I've seen for this. They are also able to merge coverage reports from multiple builds and platforms (for example for wlc we have combined coverage for Linux, OSX and Windows coming from Travis CI and AppVeyor builds). SauceLabs Unit testing is good, but the frontend testing in browser is also important. We run Selenium tests in several browsers in SauceLabs to verify that we haven't screwed up something from the user interface. Read the Docs Documentation is necessary for every project and having it built automatically is nice bonus. Landscape Doing code analysis is a way to avoid some problems which are not spot during testing. These can be code paths not covered by test or simply coding style issues. There are several such services, but Landscape is my favorite one right now.

Filed under: Debian English phpMyAdmin SUSE Weblate 0 comments

Hello Systemd is powerful, but creating a new service is a task that require creating several files in non obvious location (like /etc/systemd/system or ~/.local/share/systemd/user/). Each file features 2 or more sections (e.g. [Unit], [Service]). And each section supports a lot of parameters. Creating such Systemd configuration files can be seen as a daunting task for beginners. cme project aims to make this task easier by providing a GUI that:

• shows all existing services in a single screen
• shows all possible sections and parameters with their documentation
• validates the content of each parameter (if possible)

For instance, on my laptop, the command cme edit systemd-user shows 2 custom services ( free-imap-tunnel@ and gmail-imap-tunnel@ ) with: The GUI above shows the units for my custom systemd files:

$ ls ~/.config/systemd/user/
free-imap-tunnel@.service
free-imap-tunnel.socket
gmail-imap-tunnel@.service
gmail-imap-tunnel.socket
sockets.target.wants

and the units installed by Debian packages:

$ find /usr/lib/systemd/user/ -maxdepth 1 \
  '(' -name '*.service' -o -name '*.socket' ')' \
  -printf '%f\n'  sort  head -15
at-spi-dbus-bus.service
colord-session.service
dbus.service
dbus.socket
dirmngr.service
dirmngr.socket
glib-pacrunner.service
gpg-agent-browser.socket
gpg-agent-extra.socket
gpg-agent.service
gpg-agent.socket
gpg-agent-ssh.socket
obex.service
pulseaudio.service
pulseaudio.socket

The screenshot above shows the content of the service defined by the following file:

$ cat ~/.config/systemd/user/free-imap-tunnel@.service
[Unit]
Description=Tunnel IMAPS connections to Free with Systemd
[Service]
StandardInput=socket
# no need to install corkscrew
ExecStart=-/usr/bin/socat - PROXY:127.0.0.1:imap.free.fr:993,proxyport=8888

Note that empty parameters are not shown because the hide empty value checkbox on top right is enabled. Likewise, cme is able to edit system files like user files with sudo cme edit systemd: For more details on how to use the GUI to edit systemd files, please see:

• Managing systemd configuration with cme
• Using cme

Using a GUI may not be your cup of tea. cme can also be used as a validation tool. Let s add a parameter with an excessive value to my service:

$ echo "CPUShares = 1000000" >> ~/.local/share/systemd/user/free-imap-tunnel@.service

And check the file with cme:

$ cme check systemd-user 
cme: using Systemd model
loading data
Configuration item 'service:"free-imap-tunnel@" Service CPUShares' has a wrong value:
        value 1000000 > max limit 262144

ok, let s fix this with cme. The wrong value can either be deleted:

$ cme modify systemd-user 'service:"free-imap-tunnel@" Service CPUShares~'
cme: using Systemd model
Changes applied to systemd-user configuration:
- service:"free-imap-tunnel@" Service CPUShares: '1000000' -> ''

Or modified:

$ cme modify systemd-user 'service:"free-imap-tunnel@" Service CPUShares=2048'
cme: using Systemd model
Changes applied to systemd-user configuration:
- service:"free-imap-tunnel@" Service CPUShares: '1000000' -> '2048'

You can also view the specification of a service using cme:

$ cme dump systemd-user 'service:"free-imap-tunnel@"'---
Service:
  CPUShares: 2048
  ExecStart:
    - '-/usr/bin/socat -  PROXY:127.0.0.1:imap.free.fr:993,proxyport=8888'
  StandardInput: socket
Unit:
  Description: Tunnel IMAPS connections to Free with Systemd

The output above matches the content of the service configuration file:

$ cat ~/.local/share/systemd/user/free-imap-tunnel@.service
## This file was written by cme command.
## You can run 'cme edit systemd-user' to modify this file.
## You may also modify the content of this file with your favorite editor.
[Unit]
Description=Tunnel IMAPS connections to Free with Systemd
[Service]
StartupCPUWeight=100
CPUShares=2048
StartupCPUShares=1024
StandardInput=socket
# no need to install corkscrew now
ExecStart=-/usr/bin/socat -  PROXY:127.0.0.1:imap.free.fr:993,proxyport=8888

Last but not least, you can use cme shell if you want an interactive ui but cannot use a graphical interface:

$ cme shell systemd-user 
cme: using Systemd model
 >:$ cd service:"free-imap-tunnel@"  Service  
 >: service:"free-imap-tunnel@" Service $ ll -nz Exec*
name        type   value                                                             
 
ExecStart   list   -/usr/bin/socat -  PROXY:127.0.0.1:imap.free.fr:993,proxyport=8888
 >: service:"free-imap-tunnel@" Service $ ll -nz
name               type      value                                                             
 
StartupCPUWeight   integer   100                                                               
CPUShares          integer   2048                                                              
StartupCPUShares   integer   1024                                                              
StandardInput      enum      socket                                                            
ExecStart          list      -/usr/bin/socat -  PROXY:127.0.0.1:imap.free.fr:993,proxyport=8888
 >: service:"free-imap-tunnel@" Service $ set CPUShares=1024
 >: service:"free-imap-tunnel@" Service $ ll -nz CPUShares 
name        type      value
 
CPUShares   integer   1024 
 >: service:"free-imap-tunnel@" Service $ quit
Changes applied to systemd-user configuration:
- service:"free-imap-tunnel@" Service CPUShares: '2048' -> '1024'
write back data before exit ? (Y/n)

Currently, only service, socket and timer units are supported. Please create a bug report on github if you need more. Installation instructions are detailed at the beginning of Managing Systemd configuration with cme wiki page. As all softwares, cme probably has bugs. Please report any issue you might have with it. For more information:

• Managing Systemd configuration with cme wiki page
• Config::Model::Systemd documentation
• Source code on GitHub. Feel free to star this project
• rate this project on Perl CPAN system

All in all, systemd is quite complex to setup. I hope I made a little bit easier to deal with. All the best
Tagged: config-model, configuration, Perl, systemd

What happened in the Reproducible Builds effort between Sunday December 25 and Saturday December 31 2016: Media coverage

• Brett Smith wrote a report on our recent summit meeting on behalf of the Software Freedom Conservancy.

Reproducible bugs filed Chris West:

• #849667 filed against htslib.

Chris Lamb:

• #849314 filed against node-gulp.
• #849333 filed against faker.

Rob Browning:

• #849692 filed against oneliner-el.

Reviews of unreproducible packages 7 package reviews have been added, 12 have been updated and 14 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• build_id_differences_only
• cython_captures_build_path

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris West (19)
• Chris Lamb (7)
• Rob Browning (1)

diffoscope development

• Chris Lamb:
  • Optimisations:
    • Avoid unnecessary string manipulation writing --text output (~20x speedup).
    • Avoid n iterations over archive files (~8x speedup).
    • Memoize calls to distutils.spawn.find_executable to avoid excessive stat(1) syscalls.
  • Progress bar:
    • Show current file / ELF section under analysis etc. in progress bar.
    • Move the --status-fd output to use JSON and to include the current filename.
  • Code tidying:
    • Split out the try.diffoscope.org client so that it can be released separately on PyPI.
    • Completely rework the diffoscope and diffoscope.comparators modules, grouping similar utilities into their own modules, etc.
  • Miscellaneous:
    • Ensure that running from Git will always use that checkout's Python modules.
• Mattia Rizzolo:
  • Follow the rename of the debian package 'python-jsbeautifier' to 'jsbeautifier'
• Reiner Herrmann:
  • apk: Extend recognition regex to also match zip archives
• siamezzze:
  • Fixed no newline being classified as order-like difference.

strip-nondeterminism development

• strip-nondeterminism 0.029-2 was uploaded to unstable by Mattia Rizzolo to include a fix by Chris Lamb to fix the autopkgtest tests.

try.diffoscope.org development

• Chris Lamb:
  • Show progress bar and position in queue, etc.
  • Promote command-line client with PyPI instructions.
  • Increase comparison time limit to 90 seconds.

tests.reproducible-builds.org

• Run half of the arm64 build nodes in the future. (h01ger)
• Resume testing scheduling (on i386 and armhf) now #846564 and #844701 bugs in dpkg are fixed in that suite. (h01ger)

Misc. This week's edition was written by Chris Lamb, Holger Levsen and was reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

I apologize to all potential readers of this blog for not writing a comprehensive Introduction post with details of the project I am taking part in during my internship, as well as some story about how I ended up there. Let me just say that I was a Debian user for years when I discovered it is taking part in Outreachy as one of organisations. Their Reproducible Builds effort has a noble goal and a bunch of great people behind it - I had no chances not to get excited by it. Looking for a place where my skills could be of any use, I discovered diffoscope - the tool for in-depth comparassion of files, archives etc. My mentor, Mattia Rizzolo, supported my decision to work on it, so now I am concentrating my efforts on improving diffoscope. As my first steps, I am doing small (but hopefully still somewhat important) job of fixing existing bugs. It helps me to better understand how diffoscope works, as well as introduces me to the workflow of opensource development. During December, I have done several small contributions, mostly fixing bugs.

Test data and jessie-backports First of them could be somewhat called cleaning up after my own mistake, although that mistake wasn t trivial. During the application period, I have fixed a bug with diffoscope failing while comparing symlinks to directory. That was a small change, but I included some tests for that case anyway. And that actually caused problems. With these tests, I included test data: two folders with symlinks. All was good in unstable version of Debian, but in jessie-backports, that commit caused build to fail. After some digging, I discovered the problem was caused by build process including copying that data. That was done using shutils Python module, and older version of that module, included in jessie, could not handle copying symlinks to directory properly. Thanks to my mentor for giving me a hint on how to resolve this: using temporary folders and creating these symlinks at runtime. That way, we ensured tests run without problems during build process on jessie. What have I learned: A great deal, actually. I spent too much time on that one, but I learned how to build packages, what happens during dpkg-buildpackage run and what debhelper tools are for. I also learned a bit about what chroot is and how to use it for testing.

ICC profile files and file type recognizing regexp Another one was also about failing tests and, therefore, failing build. Failing tests were all due to ICC files were not recognized by diffoscope. Turned out libmagic got an update which changed the description of ICC profile files. Diffoscope was relying on regexp applied to file type description to recognize the file, so I changed regexp to reflect the changes in libmagic. What have I learned: How diffoscope recognizes file types. Got me thinking: maybe there is a better way? That regexp-based approach is doomed to cause problems with every file type description change. I have this question still lingering in my mind - maybe I will come up with an idea later.

Order-like difference in text files Next, I decided to do something a bit bigger and fullfilled a feature request. That request was for detecting order-like difference in text files (when files has the same lines, but in different order). I did it by collecting added and removed lines in diff output in lists, sorting and then comparing them. Sadly, I forgot about one particular case - when one of the files is missing the newline at the end of file. I was kindly reminded of that quite soon in comments on the bug-tracker (thanks danielsh!) and have already fixed that. I also recieved feedback on how better implement it deeper in the diffoscope - not using the results of diff, but rather comparing sum of hashes of the lines directly in the difference module. I am yet to try that. What have I learned: That a call to diff is actually the slowest part of the diffoscope run when done on two big text files. Could it help somehow in speeding it up? I don t know yet. I also learned to comment on bugs in Debian bugtracker and was surprised by how much feedback I got. Thanks to my mentor for pushing me to do that - I definetely need to overcome my fear of communications to be more effective!

Random FTBFS There was also a very nasty bug that caused diffoscope to fail to be built from source randomly, failing with non-informative Fatal Python error: deallocated None. It already seemed strange when it was first reported; It got only more strange when suddenly that bug ceased to be reproducible. We hoped that would mean that bug was caused by some external tool, and was fixed there. Turns out it was not that easy. I tested this on two separate computers and on virtual machine; I used different versions of diffoscope. Well. Seems like that bug is still somehow tied to diffoscope version and not some external tool version - I still can do git checkout 64 and be able to reproduce the bug (still randomly, though). Although I spent quite a lot of time on that one, the only result was the information about connection between bug apperances and diffoscope version. I still wasn t able to get to the root of the problem - hopefully, someone else will be able to, given the information I found. What have I learned: git-bisect! Thanks to my friend for pointing me to it, that tool came handy in that situation. Also, got some experience in catching nasty bugs like that (pity that no experience in squashing them). I had some extra time commitements in December, one of them (Reproducible Builds Summit II) connected to my internship and one (my exam session in university) not. In January, I should be able to allocate more time to that work - I hope it will help me achieve more significant results. Many thanks to Mattia Rizzolo, Chris Lamb, Holger Levsen and all other folks of Reproducible Builds project - I cannot stress enough how important your support is to me. Wish you all a great 2017!

A Safe and Happy New Year to all. While Debconf India is still a pipe-dream as of now, did see that India has been gradually doing it easier for tourists and casual business visitors to come visit India. This I take as very positive development for India itself. The 1st condition is itself good for anybody visiting India

Eligibility International Travellers whose sole objective of visiting India is recreation , sight-seeing , casual visit to meet friends or relatives, short duration medical treatment or casual business visit.

https://indianvisaonline.gov.in/visa/tvoa.html That this facility is being given to 130 odd countries is better still

Albania, Andorra, Anguilla, Antigua & Barbuda, Argentina, Armenia, Aruba, Australia, Austria, Bahamas, Barbados, Belgium, Belize, Bolivia, Bosnia & Herzegovina, Botswana, Brazil, Brunei, Bulgaria, Cambodia, Canada, Cape Verde, Cayman Island, Chile, China, China- SAR Hong-Kong, China- SAR Macau, Colombia, Comoros, Cook Islands, Costa Rica, Cote d lvoire, Croatia, Cuba, Czech Republic, Denmark, Djibouti, Dominica, Dominican Republic, East Timor, Ecuador, El Salvador, Eritrea, Estonia, Fiji, Finland, France, Gabon, Gambia, Georgia, Germany, Ghana, Greece, Grenada, Guatemala, Guinea, Guyana, Haiti, Honduras, Hungary, Iceland, Indonesia, Ireland, Israel, Jamaica, Japan, Jordan, Kenya, Kiribati, Laos, Latvia, Lesotho, Liberia, Liechtenstein, Lithuania, Luxembourg, Madagascar, Malawi, Malaysia, Malta, Marshall Islands, Mauritius, Mexico, Micronesia, Moldova, Monaco, Mongolia, Montenegro, Montserrat, Mozambique, Myanmar, Namibia, Nauru, Netherlands, New Zealand, Nicaragua, Niue Island, Norway, Oman, Palau, Palestine, Panama, Papua New Guinea, Paraguay, Peru, Philippines, Poland, Portugal, Republic of Korea, Republic of Macedonia, Romania, Russia, Saint Christopher and Nevis, Saint Lucia, Saint Vincent & the Grenadines, Samoa, San Marino, Senegal, Serbia, Seychelles, Singapore, Slovakia, Slovenia, Solomon Islands, South Africa, Spain, Sri Lanka, Suriname, Swaziland, Sweden, Switzerland, Taiwan, Tajikistan, Tanzania, Thailand, Tonga, Trinidad & Tobago, Turks & Caicos Island, Tuvalu, UAE, Ukraine, United Kingdom, Uruguay, USA, Vanuatu, Vatican City-Holy See, Venezuela, Vietnam, Zambia and Zimbabwe.

This should make it somewhat easier for any Indian organizer as well as any participants from any of the member countries shared. There is possibility that this list would even get longer, provided we are able to scale our airports and all and any necessary infrastructure that would be needed for International Visitors to have a good experience. What has been particularly interesting is to know which ports of call are being used by International Visitors as well as overall growth rate

The Percentage share of Foreign Tourist Arrivals (FTAs) in India during November, 2016 among the top 15 source countries was highest from USA (15.53%) followed by UK (11.21%), Bangladesh (10.72%), Canada (4.66%), Russian Fed (4.53%), Australia (4.04%), Malaysia (3.65%), Germany (3.53%), China (3.14%), France (2.88%), Sri Lanka (2.49%), Japan (2.49%), Singapore (2.16%), Nepal (1.46%) and Thailand (1.37%).

And port of call

The Percentage share of Foreign Tourist Arrivals (FTAs) in India during November 2016 among the top 15 ports was highest at Delhi Airport (32.71%) followed by Mumbai Airport (18.51%), Chennai Airport (6.83%), Bengaluru Airport (5.89%), Haridaspur Land check post (5.87%), Goa Airport (5.63%), Kolkata Airport (3.90%), Cochin Airport (3.29%), Hyderabad Airport (3.14%), Ahmadabad Airport (2.76%), Trivandrum Airport (1.54%), Trichy Airport (1.53%), Gede Rail (1.16%), Amritsar Airport (1.15%), and Ghojadanga land check post (0.82%) .

The Ghojadanga land check post seems to be between West Bengal, India and Bangladesh. Gede Railway Station is also in West Bengal as well. So all and any overlanders could take any of those ways.Even Hardispur Land Check post comes in the Bengal-Bangladesh border only. In the airports, Delhi Airport seems to be attracting lot more business than the Mumbai Airport. Part of the reason I *think* is the direct link of Delhi Airport to NDLS via the Delhi Airport Express Line . The same when it will happen in Mumbai should be a game-changer for city too. Now if you are wondering why I have been suddenly talking about visas and airports in India, it came because Hong Kong is going to Withdraw Visa Free Entry Facility For Indians. Although, as rightly pointed out in the article doesn t make sense from economic POV and seems to be somewhat politically motivated. Not that I or anybody else can do anything about that. Seeing that, I thought it was a good opportunity to see how good/Bad our Government is and it seems to be on the right path. Although the hawks (Intelligence and Counter-Terrorist Agencies) will probably become a bit more paranoid , their work becomes tougher.
Filed under: Miscellenous Tagged: #Airport Metro Line 3, #CSIA, #Incredible India, #India, #International Tourism

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Celebrated my 10-year anniversary of contributing to Debian. An excerpt of this post was quoted on LWN.
• Made a number of improvements to AptFS, my FUSE-based filesystem that provides a view on unpacked Debian source packages as regular folders, including move from the popen2 Python module to subprocess and correcting the parsing of package lists.
• Corrected an UnboundLocalError exception in the Finnish social security number generator in faker, a tool to generate test data in Python applications. (#441)
• Made a small change to travis.debian.net (my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every code change) to fix an issue with malformed YAML.
• Added the ability to specify the clone target to gbp-import-dsc etc. in git-buildpackage, a tool to build Debian packages using Git. (commit)
• Filed three issues against the Redis key-value database:
  • Tests fail on the alpha architecture due to "memory efficiency". (#3666)
  • Please update hiredis (#3687)
  • Correct "whenever" typo. (#3652)

---

Debian Long Term Support (LTS) Those were 8th and 9th months working on Debian LTS started by Raphael Hertzog at Freexian. I had trouble resuming work in November as I had taken a long break during the month and started looking at issues only during the last week of November.

Imagemagick, again I have, again, spent a significant amount of time fighting the ImageMagick (IM) codebase. About 15 more vulnerabilities were found since the last upload, which resulted in DLA-756-1. In the advisory, I unfortunately forgot to mention CVE-2016-8677 and CVE-2016-9559, something that was noticed by my colleague Roberto after the upload... More details about the upload are available in the announcement. When you consider that I worked on IM back in october, which lead to an upload near the end of November covering around 80 more vulnerabilities, it doesn't look good for the project at all. Of the 15 vulnerabilities I worked on, only 6 had CVEs assigned and I had to request CVEs for the other 9 vulnerabilities plus 11 more that were still unassigned. This lead to the assignment of 25 distinct CVE identifiers as a lot of issues were found to be distinct enough to warrant their own CVEs. One could also question how many of those issues affect the fork, Graphicsmagick. A lot of the vulnerabilities were found through fuzzing searches that may not have been tested on Graphicsmagick. It seems clear to me that a public corpus of test data should be available to test regressions and cross-project vulnerabilities. It's already hard enough to track issues withing IM itself, I can't imagine what it would be for the fork to keep track of those issues, especially since upstream doesn't systematically request CVEs for issues that they find, a questionable practice considering the number of issues we all need to keep track of.

Nagios I have also worked on the Nagios package and produced DLA 751-1 which fixed two fairly major issues (CVE-2016-9565 and CVE-2016-9566) that could allow remote root access under certain conditions. Fortunately, the restricted permissions setup by default in the Debian package made both exploits limited to information disclosure and privilege escalation if the debug log is enabled. This says a lot about how proper Debian packaging can help in limiting the attack surface of certain vulnerabilities. It was also "interesting" to have to re-learn dpatch to add patches to the package: I regret not converting it to quilt, as the operation is simple and quilt is so much easier to use. People new to Debian packaging may be curious to learn about the staggering number of patching systems historically used in Debian. On that topic, I started a conversation about how much we want to reuse existing frameworks when we work on those odd packages, and the feedback was interesting. Basically, the answer is "it depends"...

NSS I had already worked on the package in November and continued the work in December. Most of the work was done by Raphael, which fixed a lot of issues with the test suite. I tried to wrap this up by fixing CVE-2016-9074, the build on armel and the test suite. Unfortunately, I had to stop again because I ran out of hours and the fips test suite was still failing, but fortunately Raphael was able to complete the work with DLA-759-1. As things stand now, the package is in better shape than in other suites as tests (Debian bug #806639) and autopkgtest (Debian bug #806207) are still not shipped in the sid or stable releases.

Other work For the second time, I forgot to formally assign myself a package before working on it, which meant that I wasted part of my hours working on the monit package. Those hours, of course, were not counted in my regular hours. I still spent some time reviewing mejo's patch to ensure it was done properly and it turned out we both made similar patches working independently, always a good sign. As I reported in my preliminary November report, I have also triaged issues in libxml2, ntp, openssl and tiff. Finally, I should mention my short review of the phpMyAdmin upload, among the many posts i sent to the LTS mailing list.

Other free software work One reason why I had so much trouble getting paid work done in November is that I was busy with unpaid work...

manpages.debian.org A major time hole for me was trying to tackle the manpages.debian.org service, which had been offline since August. After a thorough evaluation of the available codebases, I figured the problem space wasn't so hard and it was worth trying to do an implementation from scratch. The result is a tool called debmans. It took, obviously, way longer than I expected, as I experimented with Python libraries I had been keeping an eye on for a while. For the commanline interface, I used the click library, which is really a breeze to use, but a bit heavy for smaller scripts. For a web search service prototype, I looked at flask, which was also very interesting, as it is light and simple enough to use that I could get started quickly. It also, surprisingly, fares pretty well in the global TechEmpower benchmarking tests. Those interested in those tools may want to look at the source code, in particular the main command (using an interesting pattern itself, __main__.py) and the search prototype. Debmans is the first project for which I have tried the CII Best Practices Badge program, an interesting questionnaire to review best practices in software engineering. It is an excellent checklist I recommend every project manager and programmer to get familiar with. I still need to complete my work on Debmans: as I write this, I couldn't get access to the new server the DSA team setup for this purpose. It was a bit of a frustrating experience to wait for all the bits to get into place while I had a product ready to test. In the end, the existing manpages.d.o maintainer decided to deploy the existing codebase on the new server while the necessary dependencies are installed and accesses are granted. There's obviously still a bunch of work to be done for this to be running in production so I have postponed all this work to January. My hope is that this tool can be reused by other distributions, but after talking with Ubuntu folks, I am not holding my breath: it seems everyone has something that is "good enough" and that they don't want to break it...

Monkeysign I spent a good chunk of time giving a kick in the Monkeysign project, with the 2.2.2 release, which features contributions from two other developers, which may be a record for a single release. I am especially happy to have adopted a new code of conduct - it has been an interesting process to adapt the code of conduct for such a relatively small project. Monkeysign is becoming a bit of a template on how to do things properly for my Python projects: documentation on readthedocs.org including a code of conduct, support and contribution information, and so on. Even though the code now looks a bit old to me and I am embarrassed to read certain parts, I still think it is a solid project that is useful for a lot of people. I would love to have more time to spend on it.

LWN publishing As you may have noticed if you follow this blog, I have started publishing articles for the LWN magazine, filed here under the lwn tag. It is a way for me to actually get paid for some of my blogging work that used to be done for free. Reports like this one, for example, take up a significant amount of my time and are done without being paid. Converting parts of this work into paid work is part of my recent effort to reduce the amount of time I spend on the computer. An funny note: I always found the layout of the site to be a bit odd, until I looked at my articles posted there in a different web browser, which didn't have my normal ad blocker configuration. It turns out LWN uses ads, and Google ones too, which surprised me. I definitely didn't want to publish my work under banner ads, and will never do so on this blog. But it seems fair that, since I get paid for this work, there is some sort of revenue stream associated with it. If you prefer to see my work without ads, you can wait for it to be published here or become a subscriber which allows you to get rid of the ads on the site. My experience with LWN is great: they're great folks, and very supportive. It's my first experience with a real editor and it really pushed me in improving my writing to make better articles that I normally would here. Thanks to the LWN folks for their support! Expect more of those quality articles in 2017.

Debian packaging I have added a few packages to the Debian archive:

• magic-wormhole: easy file-transfer tool, co-maintained with Jamie Rollins
• slop: screenshot tool
• xininfo: utility used by teiler
• teiler (currently in NEW): GUI for screenshot and screencast tools

I have also updated sopel and atheme-services.

Other work Against my better judgment, I worked again on the borg project. This time I tried to improve the documentation, after a friend asked me for help on "how to make a quick backup". I realized I didn't have any good primer to send regular, non-sysadmin users to and figured that, instead of writing a new one, I could improve the upstream documentation instead. I generated a surprising 18 commits of documentation during that time, mainly to fix display issues and streamline the documentation. My final attempt at refactoring the docs eventually failed, unfortunately, again reminding me of the difficulty I have in collaborating on that project. I am not sure I succeeded in making the project more attractive to non-technical users, but maybe that's okay too: borg is a fairly advanced project and not currently aimed at such a public. This is yet another project I am thinking of creating: a metabackup program like backupninja that would implement the vision created by liw in his A vision for backups in Debian post, which was discarded by the Borg project. Github also tells me that I have opened 19 issues in 14 different repositories in November. I would like to particularly bring your attention to the linkchecker project which seems to be dead upstream and for which I am looking for collaborators in order to create a healthy fork. Finally, I started on reviving the stressant project and changing all my passwords, stay tuned for more!

A friend of mine choose for $reasons to install the latest OpenSuSE 42.2 release as his new laptop operating system. It's been a while that I had contact with the SuSE Linux distribution. Must be around 12 years or so. The unsual part here is that I've to support a somewhat eccentric, but mostly ordinary user of computers. And to my surprise it's still hard to just plug in your existing stuff and expect it work. I've done so many dirty things to this installation in the last three days, my system egineering heart is bleeding. printing with a Canon Pixma iP100 printer This is a small portable Canon printer, about four years old. It provides a decent quality and its main strength is that it's small and really portable. Sadly the gutenprint driver just pushes through a blank page. No ink wasted on it at all. So the only reasonable other choice was a four year old binary rpm package provided by Canon. It has a file dependency on "libtiff.so.3" which is no longer available in recent GNU/Linux distributions. So I cheated and

- unpacked the tarball
- installed the rpm from the "packages" folder
zypper install cnijfilter-common-3.70-1.x86_64.rpm cnijfilter-ip100series-3.70-1.x86_64.rpm
... and choose to ignore the missing file dependency on libtiff.so.5.
ln -s /usr/lib64/libtiff.so /usr/lib64/libtiff.so.5
- re-ran the ./install.sh which registered the printer with cups and does whatever else
  magic is included in 1906 lines of shell.

To my surprise this driver still works and provides the expected quality. Though it's just a question of time until this setup will break. Be it an incompatible ABI change in libtiff or another lib in use by those Canon provided tools. QGIS and gdal with ECW support While the printer stuff is a rather common use case, having a map viewer for map files in the ECW format is the eccentric part. I found some hints on stackoverflow and subsequently https://trac.osgeo.org/gdal/wiki/ECW that a non-free library is required and a specific build of gdal. Then QGIS should be able to work with ECW files. Lucky us there is at least a OpenSuSE repository for gdal and QGIS. So I did the following:

zypper addrepo http://download.opensuse.org/repositories/Application:/Geo/openSUSE_Leap_42.2/Application:Geo.repo
zypper install qgis

Then I had to download the non-free ECW SDK from http://download.hexagongeospatial.com/downloads/erdas-ecw-jp2-sdk-v5.3-%28linux%29 - you'll and up with a '.bin' installer file. The installation process left me with "ERDAS-ECW_JPEG_2000_SDK-5.3.0" folder in my $HOME. I moved that one to /opt. Next step is adding the library to the ldconfig search path.

echo "/opt/ERDAS-ECW_JPEG_2000_SDK-5.3.0/Desktop_Read-Only/lib/x64/release/" > /etc/ld.so.conf.d/ecw.conf; ldconfig

Now it was "just" about rebuild gdal with ECW support. So I downloaded the required source packages with "zypper source-install gdal", edited the spec somewhere in "/usr/src/" to make the following modifications

--with-ecw=/opt/ERDAS-ECW_JPEG_2000_SDK-5.3.0/Desktop_Read-Only

added to the "./configure" invocation. And somewhere at the top we had to relax the requirement that all installed files have to be referenced inside the package.

%define _unpackaged_files_terminate_build 0

As a last step I had to "rpmbuild -ba" the package and force the installation via zypper once more, because this time we have a file depedency on the libecw stuff and it's obviously not listed in the rpm database. Last but not least I tried to put the gdal build on hold with

zypper addlock gdal libgdal20

to ensure it's not removed on the next update. Other non-free tools Beside of those two issues I had to install a range of other non-free tools, but currently they work without further issues or modifications. One is Teamviewer (i686 multiarch rpm) and the other one is XnViewMP. XnView is also able to show ECW files, but only the smaller ones. It crashes on bigger ones but that's also the case on Windows. Then there is also (required by some Italian map related websites) the ugly Adobe Flash Plugin for Firefox, but that one is sadly still a widespread issue. We also tried to try out the nvidia graphic drivers but at the moment we could only get the build in Intel card to work. Usually the preferred solution from my point of view but sometimes we see rendering glitches and I'm not sure if it's the driver or something else. my personal take away I hate to admit it but it's nothing extraordinary that was requested here. But still it took me the better part of two evenings to figure everything out. And even now it's not properly integrated and doomed to fail any day due to various updates and changes in the surounding ecosystem. I've full sympathy for every average user that would give up after two hours of research and try&error on this journey. For the printer drivers I'm happy to blame Canon. The printer situation as a whole improved from my point of view during the last decade, but it's still a pain in the ass with the very short shelf life you usually see with consumer models. For the ECW case one could discuss if it would be legally possible and helpful to do ugly dlopen() stuff to dynamcially load the shared libs. But then again someone has to make his hands dirty during the build and discussions about the legal use of header files will be the next chapter (hello Oracle). It's just ugly. Actually I know too little about the world of image formats to judge if someone has a good reason to keep this format commercial or not. From my personal point of view it's not useful and maybe even morally wrong. Technically one could argue if it would make sense to keep a local copy of the gdal build in "/opt" and start QGIS with a modified library path to prefer the private gdal build. Not sure if that is any better. On the other hand there are evolving mechanism like flatpack that would ease the handling of such situations. Buth then again we would be catering non-free software. It feels a lot like giving up. While my private working environment is except for firmware blobs free, I now created for someone a real "FrankenSuSE" to satisfy his everyday needs. On the one hand we now have another mostly satisfied user of a mostly free operating system. On the other hand that was only possible by adding a vast amount of non-free software. For sure we did not win the war, I'm not even sure if we've won a single battle here. It's just frustrating to see what is required to get someone up and running. With my personal attitude towards open source software it even feels wrong to invest so much time into fiddling with non-free components. What is still missing We currently lack an image viewer that allows us to print only a selection of an image, which is useful to print parts of a map. That usually works with XnView on Windows but does not work with the Linux version at the moment. I also tried gwenview and geeqie and had the same issue. Not sure if it's maybe a bug in XnView or one of the Qt parts (gwenview is also Qt based). I did not research that yet. Update: I spent quite some time looking into open bug reports for geeqie and gwenview. Seems the feature to print only a section of an image is something new. I've created #374299 (gwenview) and #457 (geeqie). For XnView I expect it's a difference between XnViewMP (the portable version) and the Windows only XnView Classic. Needs to be clarified and it might be worth to try XnView Classic with wine. Maybe printing with wine via cups works, I found at least some results for it on the internet.

What happened in the Reproducible Builds effort between Sunday December 18 and Saturday December 24 2016: Media coverage 100% Of The 289 Coreboot Images Are Now Built Reproducibly by Phoronix, with more details in German by Pro-Linux.de. We have further reports on our Reproducible Builds World summit #2 in Berlin from Rok Garbas of NixOS as well as Clemens Lang of MacPorts Debian infrastructure work Dak now archives buildinfo files thanks to a patch from Chris Lamb. We also have mostly finalised a design of how they will be distributed by the Debian FTP mirror network which we will start implementing soon. This is great for the future of Debianb but unfortunately this also means that we won't have .buildinfo files for Stretch as Debian will not rebuild its source packages and because these binary packages currently in the archive were mostly built with dpkg > 1.18.11. reprepro/5.0.0-1 has added support for dealing with .buildinfo files that are included in .changes files. (Closes: #843402) Reproducible work in other projects The Chromium project is now working on making their build process (mostly) deterministic. Their motivation is to save both "[money] (less hardware is required) and developer time (reduced latency by having less work to do on the TS and CI)". Unreproducible bugs filed

• Chris Lamb:
  • #848721 filed against apt.
  • #848866 filed against libcorelinux.
  • #849314 filed against node-gulp.
  • #849333 filed against faker.
• Dhole:
  • #848633 filed against sugar-toolkit-gtk3.

Reviews of unreproducible packages 39 package reviews have been added, 75 have been updated and 44 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• Updated captures_build_path
• Added nondeterministic_ordering_in_desktop_files_by_python_sugar3

Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by:

• Adrian Bunk (1)
• Chris Lamb (7)
• Lucas Nussbaum (4)

diffoscope development diffoscope 66 was uploaded to unstable by Chris Lamb. It included contributions from:

• Emanuel Bronshtein:
  • Use ssh-keygen for comparing OpenSSH public keys
  • Use js-beautify as JavaScript code beautifier for .js files (with tests).
  • Many CSS & HTML improvements.
  • Change all HTTP URLs to HTTPS where applicable.
• anthraxx:
  • Enable the use of ssh-keygen on Arch Linux.
• Maria Glukhova:
  • Add detection of order-only difference in plain text format. (Closes: #848049)
  • Change icc-recognizing regexp to reflect changes in file type description. (Closes: #848814)
• Chris Lamb:
  • Update tests for compatibility with enjarify >= 1.0.3. (Closes: #849142)
  • When skipping tests because the version of an external is too low, print the detected version.
  • Avoid unpacking packages twice when comparing .changes. (Closes: #843531)
  • Add a simple profiling framework (enabled via --profile).
  • Various code quality and reliability improvements.
  • Document how to sign PyPI uploads.

strip-nondeterminism development strip-nondeterminism 0.029-1 was uploaded to unstable by Chris Lamb. It included no new content from this week, but rather included contributions from previous weeks. reproducible-website development The website is now also accessible via the https://www.reproducible-builds.org URL.

• Clemens Lang:
  • Add the definition of "reproducible", as drafted at the reproducible builds world summit in Berlin. Thanks to all participants in the sessions that worked these out!
• Valerie R Young:
  • Force ordering of titles.
  • Various formatting improvements.
• Holger Levsen:
  • Various usability and formatting improvements.
  • https://www.reproducible-builds.org
• Chris Lamb:
  • Various usability, style and wording improvements.
  • Add Debconf15, Skroutz.gz and MiniDebconfCambridge15 talks to resouces page.

tests.reproducible-builds.org

• We changed the data storage backend from a single sqlite3 database file (651 MB) to a PostgreSQL database. With this change we'll be able to scale a lot more and add testing of the arm64 architecture.
  • Valerie Young wrote most of the code, Mattia Rizzolo reviewed and helped improve the code and Holger deployed it and found some minor bugs which have been fixed.
• We are now testing the arm64 architecture for all packages on all suites, arranged by Holger. Many thanks to codethink for providing us with access to eight 8-core arm64 machines with 64GB memory, which allows us to rebuild Debian very fast!

Misc. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC and the mailing lists.

There s a pattern that comes up from time to time in the release management of free software projects. To allow for big, disruptive changes, a new development branch is created. Most of the developers focus moves to the development branch. However at the same time, the users focus stays on the stable branch. As a result:

• The development branch lacks user testing, and tends to make slower progress towards stabilization.
• Since users continue to use the stable branch, it is tempting for developers to spend time backporting new features to the stable branch instead of improving the development branch to get it stable.

This situation can grow up to a quasi-deadlock, with people questioning whether it was a good idea to do such a massive fork in the first place, and if it is a good idea to even spend time switching to the new branch. To make things more unclear, the development branch is often declared stable by its developers, before most of the libraries or applications have been ported to it. This has happened at least three times. First, in the Linux 2.4 / 2.5 era. Wikipedia describes the situation like this:

Before the 2.6 series, there was a stable branch (2.4) where only relatively minor and safe changes were merged, and an unstable branch (2.5), where bigger changes and cleanups were allowed. Both of these branches had been maintained by the same set of people, led by Torvalds. This meant that users would always have a well-tested 2.4 version with the latest security and bug fixes to use, though they would have to wait for the features which went into the 2.5 branch. The downside of this was that the stable kernel ended up so far behind that it no longer supported recent hardware and lacked needed features. In the late 2.5 kernel series, some maintainers elected to try backporting of their changes to the stable kernel series, which resulted in bugs being introduced into the 2.4 kernel series. The 2.5 branch was then eventually declared stable and renamed to 2.6. But instead of opening an unstable 2.7 branch, the kernel developers decided to continue putting major changes into the 2.6 branch, which would then be released at a pace faster than 2.4.x but slower than 2.5.x. This had the desirable effect of making new features more quickly available and getting more testing of the new code, which was added in smaller batches and easier to test. Then, in the Ruby community. In 2007, Ruby 1.8.6 was the stable version of Ruby. Ruby 1.9.0 was released on 2007-12-26, without being declared stable, as a snapshot from Ruby s trunk branch, and most of the development s attention moved to 1.9.x. On 2009-01-31, Ruby 1.9.1 was the first release of the 1.9 branch to be declared stable. But at the same time, the disruptive changes introduced in Ruby 1.9 made users stay with Ruby 1.8, as many libraries (gems) remained incompatible with Ruby 1.9.x. Debian provided packages for both branches of Ruby in Squeeze (2011) but only changed the default to 1.9 in 2012 (in a stable release with Wheezy 2013). Finally, in the Python community. Similarly to what happened with Ruby 1.9, Python 3.0 was released in December 2008. Releases from the 3.x branch have been shipped in Debian Squeeze (3.1), Wheezy (3.2), Jessie (3.4). But the python command still points to 2.7 (I don t think that there are plans to make it point to 3.x, making python 3.x essentially a different language), and there are talks about really getting rid of Python 2.7 in Buster (Stretch+1, Jessie+2). In retrospect, and looking at what those projects have been doing in recent years, it is probably a better idea to break early, break often, and fix a constant stream of breakages, on a regular basis, even if that means temporarily exposing breakage to users, and spending more time seeking strategies to limit the damage caused by introducing breakage. What also changed since the time those branches were introduced is the increased popularity of automated testing and continuous integration, which makes it easier to measure breakage caused by disruptive changes. Distributions are in a good position to help here, by being able to provide early feedback to upstream projects about potentially disruptive changes. And distributions also have good motivations to help here, because it is usually not a great solution to ship two incompatible branches of the same project. (I wonder if there are other occurrences of the same pattern?) Update: There s a discussion about this post on HN

it's amazing how many new bugs appear; luckily in the Debian Perl Group we're not too bad at fixing them as well. here's the list of my contributions over the last weeks:

• #828387 src:libcrypt-openssl-rsa-perl: "libcrypt-openssl-rsa-perl: FTBFS with openssl 1.1.0"
  add patch from Sebastian Andrzej Siewior (pkg-perl)
• #839322 src:libmoops-perl: "libmoops-perl: FTBFS: build-dependency not installable: libkavorka-perl"
  close, as dependency is fixed (pkg-perl)
• #840305 src:libatteanx-store-memorytriplestore-perl: "libatteanx-store-memorytriplestore-perl: depends on unavailable AtteanX::RDFQueryTranslator"
  investigate (pkg-perl)
• #842753 libnet-sip-perl: "FTBFS: test failures"
  import new upstream release (pkg-perl)
• #844141 src:libsys-cpuaffinity-perl: "libsys-cpuaffinity-perl: FTBFS: Failed test 'bind to all processors successful 18446744073709551615 == 18446744073709551616-1'"
  import new upstream release, later reopen & downgrade (pkg-perl)
• #844538 src:libdbd-mysql-perl: "libdbd-mysql-perl: FTBFS on some architectures"
  import new upstream release (pkg-perl)
• #844950 src:libdatetime-format-duration-perl: "libdatetime-format-duration-perl must build-depend on libparams-validate-perl"
  add missing (build) dependency, upload to DELAYED/5
• #846059 src:libmath-bigint-gmp-perl: "libmath-bigint-gmp-perl: FTBFS: Failed 1/12 test programs (t/bigintpm.t)"
  import new upstream release (pkg-perl)
• #846721 src:libaudio-mpd-perl: "libaudio-mpd-perl: FTBFS: Test failures"
  skip a brittle test (pkg-perl)
• #846731 src:libbio-scf-perl: "libbio-scf-perl: FTBFS: Test failures"
  add patch from ntyni (pkg-perl)
• #847176 libbusiness-ismn-perl: "libbusiness-ismn-perl: FTBFS: t/prereq.t failure"
  import new upstream release (pkg-perl)
• #847188 libtest-data-perl: "libtest-data-perl: FTBFS: t/prereq.t failure"
  import new upstream release (pkg-perl)
• #847189 libbusiness-issn-perl: "libbusiness-issn-perl: FTBFS: t/prereq.t failure"
  import new upstream release (pkg-perl)
• #847397 src:libimager-perl: "libimager-perl FTBFS on mips/mipsel: Failed 10/66 test programs"
  build with -O0 on mips/mipsel (pkg-perl)
• #848060 src:libx11-protocol-other-perl: "libx11-protocol-other-perl: FTBFS randomly (failing tests)"
  disable fragile test (pkg-perl)
• #848266 src:libparanoid-perl: "libparanoid-perl: FTBFS: tests timeout"
  skip a test (pkg-perl)
• #848346 libio-socket-ssl-perl: "libio-socket-ssl-perl: default certificate path lost with libssl1.1"
  import new upstream release (pkg-perl)
• #848427 libanyevent-perl: "libanyevent-perl: FTBFS with openssl 1.1: t/80_ssltest.t failure"
  apply upstream patch (pkg-perl)

I recently had call to play with some 1-Wire devices at work (more of which in a future post). It was taking a while for the appropriate programmer to turn up, so of course I pulled out my trusty BusPirate. It turned out the devices in question would only talk in overdrive mode, while the Bus Pirate could only offer standard mode. So I set about trying to figure out how to add the appropriate support. This is is a huge endorsement for test equipment with Free Software firmware. Rather than giving up I was able to go and grab the current firmware, which has been adopted by the community since Dangerous Prototypes have discontinued development. What let me down was the ecosystem around the PIC24FJ64GA002. My previous recent experience with microcontrollers has been with the ATTiny range and the STM32. Getting up and running with both of these was fairly easy - the tool chains necessary were already present in Debian, so all it took was a simple apt invocation to install everything I needed to compile code and program it to the devices. Not so with the PIC series, which surprised me. There seems to be some basic support for the earlier PIC16 range, but for later chips there s nothing that works out of the box with Debian. Investigation revealed that this was because there s nothing maintained that enabled Free development for the PIC range. The accepted solution is the closed MPLAB X. Now, in one sense fair play to Microchip for making this available. But in another, shame on you. I can t imagine ever choosing to build something based on a chip that only had a closed source tool chain available. I want things I can use in Makefiles and properly script, that are available in my distro of choice and that generally work in the same fashion as the tool chains I m used to. I understand there might be some benefit in a closed compiler in terms of performance (and have HPC friends who would never trust a benchmark provided using GCC), but in general that s not the space I move in. Nor does it seem to be the sort of attitude you should be taking if you are trying to attract the hobbyist and small production run market. Any yet this seems common amongst hardware manufacturers. People whose core business is selling physical items, where the software is only relevant in terms of being able to use those items, seem to consider the software to be precious. Instead of opening up programming specifications and allowing a more widespread use of the hardware, increasing sales. I understand there are some cases where this isn t practical, but the default attitude is definitely one of being closed rather than open, which is a terrible shame. Anyway. I do have some Bus Pirate 1-Wire overdrive support now working (pending some testing to ensure standard mode still works), but I am glad I never spent a lot of time getting involved with PICs now.