Some news about Calibre in Debian: I have been added to the list of maintainers, thanks Martin, and the recent release of Calibre 3.4 into Debian/unstable brought some fixes concerning the desktop integration. Now I am working on Calibre 3.5. Calibre 3.5 separates out one module, html5-parser, into a separate package which needs to be included into Debian first. I have prepared and uploaded a version, but NEW processing will keep this package from entering Debian for a while. Other things I am currently doing is going over the list of bugs and try to fix or close as many as possible. Finally the endless Rar support story still continues. I still don t have any response from the maintainer of unrar-nonfree in Debian, so I am contemplating to package my own version of libunrar. As I wrote in the previous post, Calibre now checks whether the Python module unrardll is available, and if, uses it to decode rar-packed ebooks. I have a package for unrardll ready to be uploaded, but it needs the shared library version, and thus I am stuck waiting for unrar-nonfree. Anyway, to help all those wanting to play with the latest Calibre, my archive nowadays contains:

• preliminary packages for Calibre 3.5
• updated package for unrar-nonfree that ship the shared library
• the new pacakge unrardll to be uploaded after unrar-nonfree is updated
• the new package html5-parser which is in NEW

Together these packages provide the newest Calibre with Rar support.

deb http://www.preining.info/debian/ calibre main
deb-src http://www.preining.info/debian/ calibre main

The releases are signed with my Debian key 0x6CACA448860CDC13 Enjoy

For most packages, the dh short-hand rules (possibly with a few overrides) work great. It can often auto-detect the buildsystem and handle all the trivial parts. With one notably exception: What if you need to compile the upstream code twice (or more) with different flags? This is the case for all source packages building both regular debs and udebs. In that case, you would previously need to override about 5-6 helpers for this to work at all. The five dh_auto_* helpers and usually also dh_install (to call it with different sourcedir for different packages). This gets even more complex if you want to support Build-Profiles such as noudeb and nodoc . The best way to support nodoc in debhelper is to move documentation out of dh_install s config files and use dh_installman, dh_installdocs, and dh_installexamples instead (NB: wait for compat 11 before doing this). This in turn will mean more overrides with sourcedir and -p/-N. And then there is noudeb , which currently requires manual handling in debian/rules. Basically, you need to use make or shell if-statements to conditionally skip the udeb part of the builds. All of this is needlessly complex. Improving the situation In an attempt to make things better, I have made a new prototype feature in debhelper called buildlabels in experimental. The current prototype is designed to deal with part (but not all) of the above problems:

• It will remove the need for littering your rules file for supporting noudeb (and in some cases also other noX profiles).
• It will remove the need for overriding the dh_install* tools just to juggle with sourcedir and -p/-N.

However, it currently not solve the need for overriding the dh_auto_* tools and I am not sure when/if it will. The feature relies on being able to relate packages to a given series of calls to dh_auto_*. In the following example, I will use udebs for the secondary build. However, this feature is not tied to udebs in any way and can be used any source package that needs to do two or more upstream builds for different packages. Assume our example source builds the following binary packages:

• foo
• libfoo1
• libfoo-dev
• foo-udeb
• libfoo1-udeb

And in the rules file, we would have something like:

[...]
override_dh_auto_configure:
    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure -B build-udeb -- --without-feature1 --without-feature2
[...]

What is somewhat obvious to a human is that the first configure line is related to the regular debs and the second configure line is for the udebs. However, debhelper does not know how to infer this and this is where buildlabels come in. With buildlabels, you can let debhelper know which packages and builds that belong together. How to use buildlabels To use buildlabels, you have to do three things:

1. Pick a reasonable label name for the secondary build. In the example, I will use udeb .
2. Add buildlabel=$LABEL to all dh_auto_* calls related to your secondary build.
3. Tag all packages related to my-label with X-DH-Buildlabel: $LABEL in debian/control. (For udeb packages, you may want to add Build-Profiles: <!noudeb> while you are at it).

For the example package, we would change the debian/rules snippet to:

[...]
override_dh_auto_configure:
    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure --buildlabel=udeb -B build-udeb -- --without-feature1 --without-feature2
[...]

(Remember to update *all* calls to dh_auto_* helpers; the above only lists dh_auto_configure to keep the example short.) And then add X-DH-Buildlabel: udeb in the stanzas for foo-udeb + libfoo1-udeb. With those two minor changes:

• debhelper will skip the calls to dh_auto_* with buildlabel=udeb if the udeb packages are skipped.
• dh_auto_install will automatically pick a separate destination directory by default for the udeb build (assuming you do not explicitly override it with destdir).
• dh_install will now automatically pick up files from the destination directory.that dh_auto_install used for the given package (even if you overwrote it with destdir). Note that you have to remove any use of sourcedir first as this disables the auto-detection. This also works for other dh_install* tools supporting sourcedir in compat 11 or later.

Real example Thanks to Michael Biebl, I was able to make an branch in the systemd git repository to play with this feature. Therefore I have an real example to use as a show case. The gist of it is in the following three commits:

• Apply labels to packages and dh_auto_* tools: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=880a80bc392a298bed033c047ccbef65e50acc58
• Clean up most make conditionals: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=96c0ab8b35ce04eab031a4a9ee4c4ef686f76977
• Merge multiple dh_install calls into one: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=67994cb5e5b451ebcebb0dccfdd97e606ac4a172
  ( fail-missing was never applied to the udeb before; I kept it that way by passing sourcedir to dh_missing because I was to lazy to review the missing files).

Full branch can be seen at: https://anonscm.debian.org/git/pkg-systemd/systemd.git/log/?h=wip-dh-prototype-smarter-multi-builds Request for comments / call for testing This prototype is now in experimental (debhelper/10.7+exp.buildlabels) and you are very welcome to take it for a spin. Please let me know if you find the idea useful and feel free to file bugs or feature requests. If deemed useful, I will merge into master and include in a future release. If you have any questions or comments about the feature or need help with trying it out, you are also very welcome to mail the debhelper-devel mailing list. Known issues / the fine print:

• It is experimental feature and may change without notice.
• The current prototype may break existing packages as it is not guarded by a compat bump to ease your testing. I am still very curious to hear about any issues you may experience.
• The default build directory is re-used even with different buildlabels, so you still need to use explicit build dirs for buildsystems that prefer building in a separate directory (e.g. meson).
• udebs are not automatically tagged with an udeb buildlabel. This is partly by design as some source packages only build udebs (and no regular debs). If they were automatically tagged, the existing packages would break.
• Label names are used in path names, so you may want to refrain from using too exciting label names.
• It is experimental feature and may change without notice. (Yes, I thought it deserved repeating)

Filed under: Debhelper, Debian

Debian Long Term Support (LTS) This is my monthly working on Debian LTS. This time I worked on various hairy issues surrounding ca-certificates, unattended-upgrades, apache2 regressions, libmtp, tcpdump and ipsec-tools.

ca-certificates updates I've been working on the removal of the Wosign and StartCom certificates (Debian bug #858539) and, in general, the synchronisation of ca-certificates across suites (Debian bug #867461) since at least last march. I have made an attempt at summarizing the issue which led to a productive discussion and it seems that, in the end, the maintainer will take care of synchronizing information across suites. Guido was right in again raising the question of synchronizing NSS across all suites (Debian bug #824872) which itself raised the other question of how to test reverse dependencies. This brings me back to Debian bug #817286 which, basically proposed the idea of having "proposed updates" for security issues. The problem is while we can upload test packages to stable proposed-updates, we can't do the same in LTS because the suite is closed and we operate only on security packages. This issue came up before in other security upload and we need to think better about how to solve this.

unattended-upgrades Speaking of security upgrades brings me to the question of a bug (Debian bug #867169) that was filed against the wheezy version of unattended-upgrades, which showed that the package simply stopped working since the latest stable release, because wheezy became "oldoldstable". I first suggested using the "codename" but that appears to have been introduced only after wheezy. In the end, I proposed a simple update that would fix the configuration files and uploaded this as DLA-1032-1. This is thankfully fixed in later releases and will not require such hackery when jessie becomes LTS as well.

libmtp Next up is the work on the libmtp vulnerabilities (CVE-2017-9831 and CVE-2017-9832). As I described in my announcement, the work to backport the patch was huge, as upstream basically backported a whole library from the gphoto2 package to fix those issues (and probably many more). The lack of a test suite made it difficult to trust my own work, but given that I had no (negative) feedback, I figured it was okay to simply upload the result and that became DLA-1029-1.

tcpdump I then looked at reproducing CVE-2017-11108, a heap overflow triggered tcpdump would parse specifically STP packets. In Debian bug #867718, I described how to reproduce the issue across all suites and opened an issue upstream, given that the upstream maintainers hadn't responded responded in weeks according to notes in the RedHat Bugzilla issue. I eventually worked on a patch which I shared upstream, but that was rejected as they were already working on it in their embargoed repository. I can explain this confusion and duplication of work with:

1. the original submitter didn't really contact security@tcpdump.org
2. he did and they didn't reply, being just too busy
3. they replied and he didn't relay that information back

I think #2 is most likely: the tcpdump.org folks are probably very busy with tons of reports like this. Still, I should probably have contacted security@tcpdump.org directly before starting my work, even though no harm was done because I didn't divulge issues that were already public. Since then, tcpdump has released 4.9.1 which fixes the issue, but then new CVEs came out that will require more work and probably another release. People looking into this issue must be certain to coordinate with the tcpdump security team before fixing the actual issues.

ipsec-tools Another package that didn't quite have a working solution is the ipsec-tools suite, in which the racoon daemon was vulnerable to a remotely-triggered DOS attack (CVE-2016-10396). I reviewed and fixed the upstream patch which introduced a regression. Unfortunately, there is no test suite or proof of concept to control the results. The reality is that ipsec-tools is really old, and should maybe simply be removed from Debian, in favor of strongswan. Upstream hasn't done a release in years and various distributions have patched up forks of those to keep it alive... I was happy, however, to know that a maintainer will take care of updating the various suites, including LTS, with my improved patch. So this fixes the issue for now, but I would strongly encourage users to switch away from ipsec-tools in the future.

apache2 Finally, I was bitten by the old DLA-841-1 upload I did all the way back in February, as it introduced a regression (Debian bug #858373). It turns out it was possible to segfault Apache workers with a trivial HTTP request, in certain (rather exotic, I might add) configurations (ErrorDocument 400 directive pointing to a cgid script in worker mode). Still, it was a serious regression and I found a part of the nasty long patch we worked on back then that was faulty, and introduced a small fix to correct that. The proposed package unfortunately didn't yield any feedback, and I can only assume it will work okay for people. The result is the DLA-841-2 upload which fixes the regression. I unfortunately didn't have time to work on the remaining CVEs affecting apache2 in LTS at the time of writing.

Triage I also did some miscellaneous triage by filing Debian bug #867477 for poppler in an effort to document better the pending issue. Next up was some minor work on eglibc issues. CVE-2017-8804 has a patch, but it's been disputed. since the main victim of this and the core of the vulnerability (rpcbind) has already been fixed, I am not sure this vulnerability is still a thing in LTS at all. I also looked at CVE-2014-9984, but the code is so different in wheezy that I wonder if LTS is affected at all. Unfortunately, the eglibc gymnastics are a little beyond me and I do not feel confident enough to just push those issues aside for now and let them open for others to look at.

Other free software work And of course, there's my usual monthly volunteer work. My ratio is a little better this time, having reached an about even ratio between paid and volunteer work, whereas this was 60% volunteer work in march.

Announcing ecdysis I recently published ecdysis, a set of template and code samples that I frequently reuse across project. This is probably the least pronounceable project name I have ever chosen, but this is somewhat on purpose. The goal of this project is not collaboration or to become a library: it's just a personal project which I share with the world as a curiosity. To quote the README file:

The name comes from what snakes and other animals do to "create a new snake": they shed their skin. This is not so appropriate for snakes, as it's just a way to rejuvenate their skin, but is especially relevant for anthropods since then "ecdysis" may be associated with a metamorphosis:

Ecdysis is the moulting of the cuticle in many invertebrates of the clade Ecdysozoa. Since the cuticle of these animals typically forms a largely inelastic exoskeleton, it is shed during growth and a new, larger covering is formed. The remnants of the old, empty exoskeleton are called exuviae. Wikipedia

So this project is metamorphosed into others when the documentation templates, code examples and so on are reused elsewhere. For that reason, the license is an unusally liberal (for me) MIT/Expat license. The name also has the nice property of being absolutely unpronounceable, which makes it unlikely to be copied but easy to search online.

It was an interesting exercise to go back into older projects and factor out interesting code. The process is not complete yet, as there are older projects I'm still curious in reviewing. A bunch of that code could also be factored into upstream project and maybe even the Python standard library. In short, this is stuff I keep on forgetting how to do: a proper setup.py config, some fancy argparse extensions and so on. Instead of having to remember where I had written that clever piece of code, I now shove it in the crazy chaotic project where I can find it again in the future.

Beets experiments Since I started using Subsonic (or Libresonic) to manage the music on my phone, album covers are suddenly way more interesting. But my collection so far has had limited album covers: my other media player (gmpc) would download those on the fly on its own and store them in its own database - not on the filesystem. I guess this could be considered to be a limitation of Subsonic, but I actually appreciate the separation of duty here. Garbage in, garbage out: the quality of Subsonic's rendering depends largely on how well setup your library and tags are. It turns out there is an amazing tool called beets to do exactly that kind of stuff. I originally discarded that "media library management system for obsessive-compulsive [OC] music geeks", trying to convince myself i was not an "OC music geek". Turns out I am. Oh well. Thanks to beets, I was able to download album covers for a lot of the albums in my collection. The only covers that are missing now are albums that are not correctly tagged and that beets couldn't automatically fix up. I still need to go through those and fix all those tags, but the first run did an impressive job at getting album covers. Then I got the next crazy idea: after a camping trip where we forgot (again) the lyrics to Georges Brassens, I figured I could start putting some lyrics on my ebook reader. "How hard can that be?" of course, being the start of another crazy project. A pull request and 3 days later, I had something that could turn a beets lyrics database into a Sphinx document which, in turn, can be turned into an ePUB. In the process, I probably got blocked from MusixMatch a hundred times, but it's done. Phew! The resulting e-book is about 8000 pages long, but is still surprisingly responsive. In the process, I also happened to do a partial benchmark of Python's bloom filter libraries. The biggest surprise there was the performance of the set builtin: for small items, it is basically as fast as a bloom filter. Of course, when the item size grows larger, its memory usage explodes, but in this case it turned out to be sufficient and bloom filter completely overkill and confusing. Oh, and thanks to those efforts, I got admitted in the beetbox organization on GitHub! I am not sure what I will do with that newfound power: I was just scratching an itch, really. But hopefully I'll be able to help here and there in the future as well.

Debian package maintenance I did some normal upkeep on a bunch of my packages this month, that were long overdue:

• uploaded slop 6.3.47-1: major new upstream release
• uploaded an NMU for maim 5.4.64-1.1: maim was broken by the slop release
• uploaded pv 1.6.6-1: new upstream release
• uploaded kedpm 1.0+deb8u1 to jessie (oldstable): one last security fix (Debian bug #860817, CVE-2017-8296) for that derelict password manager
• uploaded charybdis 3.5.5-1: new minor upstream release, with optional support for mbedtls
• filed Debian bug #866786 against cryptsetup to make the remote initramfs SSH-based unlocking support multiple devices: thanks to the maintainer, this now works flawlessly in buster and may be backported to stretch
• expanded on Debian bug #805414 against gdm3 and Debian bug #845938 against pulseaudio, because I had trouble connecting my computer to this new Bluetooth speaker. turns out this is a known issue in Pulseaudio: whereas it releases ALSA devices, it doesn't release Bluetooth devices properly. Documented this more clearly in the wiki page
• filed Debian bug #866790 regarding old stray Apparmor profiles that were lying around my system after an upgrade, which got me interested in Debian bug #830502 in turn
• filed Debian bug #868728 against cups regarding a weird behavior I had interacting with a network printer. turns out the other workstation was misconfigured... why are printers still so hard?
• filed Debian bug #870102 to automate sbuild schroots upgrades
• after playing around with rash tried to complete the packaging (Debian bug #754972) of percol with this pull request upstream. this ended up to be way too much overhead and I reverted to my old normal history habits.

At the Gtk+ hackfest in London earlier this year, we stole an afternoon from the toolkit folks (sorry!) to talk about Flatpak, and how we could establish a critical mass behind the Flatpak format. Bringing Linux container and sandboxing technology together with ostree, we ve got a technology which solves real world distribution, technical and security problems which have arguably held back the Linux desktop space and frustrated ISVs and app developers for nearly 20 years. The problem we need to solve, like any ecosystem, is one of users and developers without stuff you can easily get in Flatpak format, there won t be many users, and without many users, we won t have a strong or compelling incentive for developers to take their precious time to understand a new format and a new technology. As Alex Larsson said in his GUADEC talk yesterday: Decentralisation is good. Flatpak is a tool that is totally agnostic of who is publishing the software and where it comes from. For software freedom, that s an important thing because we want technology to empower users, rather than tell them what they can or can t do. Unfortunately, decentralisation makes for a terrible user experience. At present, the Flatpak webpage has a manually curated list of links to 10s of places where you can find different Flatpaks and add them to your system. You can t easily search and browse to find apps to try out so it s clear that if the current situation remains we re not going to be able to get a critical mass of users and developers around Flatpak. Enter Flathub. The idea is that by creating an obvious center of gravity for the Flatpak community to contribute and build their apps, users will have one place to go and find the best that the Linux app ecosystem has to offer. We can take care of the boring stuff like running a build service and empower Linux application developers to choose how and when their app gets out to their users. After the London hackfest we sketched out a minimum viable system Github, Buildbot and a few workers and got it going over the past few months, culminating in a mini-fundraiser to pay for the hosting of a production-ready setup. Thanks to the 20 individuals who supported our fundraiser, to Mythic Beasts who provided a server along with management, monitoring and heaps of bandwidth, and to Codethink and Scaleway who provide our ARM and Intel workers respectively. We inherit our core principles from the Flatpak project we want the Flatpak technology to succeed at alleviating the issues faced by app developers in targeting a diverse set of Linux platforms. None of this stops you from building and hosting your own Flatpak repos and we look forward to this being a wide and open playing field. We care about the success of the Linux desktop as a platform, so we are open to proprietary applications through Flatpak s extra data feature where the client machine downloads 3rd party binaries. They are correctly labeled as such in the AppStream, so will only be shown if you or your OS has configured GNOME Software to show you apps with proprietary licenses, respecting the user s preference. The new infrastructure is up and running and I put it into production on Thursday. We rebuilt the whole repository on the new system over the course of the week, signing everything with our new 4096-bit key stored on a Yubikey smartcard USB key. We have 66 apps at the moment, although Alex is working on bringing in the GNOME apps at present we hope those will be joined soon by the KDE apps, and Endless is planning to move over as many of our 3rd party Flatpaks as possible over the coming months. So, thanks again to Alex and the whole Flatpak community, and the individuals and the companies who supported making this a reality. You can add the repository and get downloading right away. Welcome to Flathub! Go forth and flatten

Like all good blog posts, this one starts with an apology about not blogging for ages in my case it looks like it s been about 7 years which is definitely a new personal best (perhaps the equally or more remarkable thing is that I have diligently kept WordPress running in the meantime). In that time, as you might expect, a few things have happened, like I met a wonderful woman and fell in love and we have two wonderful children. I also decided to part ways with my first baby and leave my role as CTO & co-founder of Collabora. This was obviously a very tough decision it s a fantastic team where I met and made many life-long friends, and they are still going strong and doing awesome things with Open Source. However, shortly after that, in February last year, I was lucky enough to land what is basically a dream job working at Endless Computers as the VP of Deployment. As I m sure most readers know, Endless is building an OS to bring personal computing to millions of new users across the world. It s based on Free Software like GNOME, Debian, ostree and Flatpak, and the more successful Endless is, the more people who get access to education, technology and opportunity and the more FOSS users and developers there will be in the world. But in my role, I get to help define the product, understand our users and how technology might help them, take Open Source out to new people, solve commercial problems to get Endless OS out into the world, manage a fantastic team and work with bright people, learn from great managers and mentors, and still find time to squash bugs, review and write more code than I used to. Like any startup, we have a lot to do and not enough time to do it, so although there aren t quite enough days in the week, I m really happy! In any case, the main point of this blog post is that I m at GUADEC in Manchester right now, and I d like to blog about Flathub, but I thought it would be weird to just show up and say that after 7 years of silence without saying hello again.

I decided to try packaging Mesos for Debian/Stretch. I had a spare system with a i7-930 CPU, 48G of RAM, and SSDs to use for building. The i7-930 isn t really fast by today s standards, but 48G of RAM and SSD storage mean that overall it s a decent build system faster than most systems I run (for myself and for clients) and probably faster than most systems used by Debian Developers for build purposes. There s a github issue about the lack of an upstream package for Debian/Stretch [1]. That upstream issue could probably be worked around by adding Jessie sources to the APT sources.list file, but a package for Stretch is what is needed anyway. Here is the documentation on building for Debian [2]. The list of packages it gives as build dependencies is incomplete, it also needs zlib1g-dev libapr1-dev libcurl4-nss-dev openjdk-8-jdk maven libsasl2-dev libsvn-dev. So BUILDING this software requires Java + Maven, Ruby, and Python along with autoconf, libtool, and all the usual Unix build tools. It also requires the FPM (Fucking Package Management) tool, I take the choice of name as an indication of the professionalism of the author. Building the software on my i7 system took 79 minutes which includes 76 minutes of CPU time (I didn t use the -j option to make). At the end of the build it turned out that I had mistakenly failed to install the Fucking Package Management gem and it aborted. At this stage I gave up on Mesos, the pain involved exceeds my interest in trying it out. How to do it Better One of the aims of Free Software is that bugs are more likely to get solved if many people look at them. There aren t many people who will devote 76 minutes of CPU time on a moderately fast system to investigate a single bug. To deal with this software should be prepared as components. An example of this is the SE Linux project which has 13 source modules in the latest release [3]. Of those 13 only 5 are really required. So anyone who wants to start on SE Linux from source (without considering a distribution like Debian or Fedora that has it packaged) can build the 5 most important ones. Also anyone who has an issue with SE Linux on their system can find the one source package that is relevant and study it with a short compile time. As an aside I ve been working on SE Linux since long before it was split into so many separate source packages and know the code well, but I still find the separation convenient I rarely need to work on more than a small subset of the code at one time. The requirement of Java, Ruby, and Python to build Mesos could be partly due to language interfaces to call Mesos interfaces from Ruby and Python. Ohe solution to that is to have the C libraries and header files to call Mesos and have separate packages that depend on those libraries and headers to provide the bindings for other languages. Another solution is to have autoconf detect that some languages aren t installed and just not try to compile bindings for them (this is one of the purposes of autoconf). The use of a tool like Fucking Package Management means that you don t get help from experts in the various distributions in making better packages. When there is a FOSS project with a debian subdirectory that makes barely functional packages then you will be likely to have an experienced Debian Developer offer a patch to improve it (I ve offered patches for such things on many occasions). When there is a FOSS project that uses a tool that is never used by Debian developers (or developers of Fedora and other distributions) then the only patches you will get will be from inexperienced people. A software build process should not download anything from the Internet. The source archive should contain everything that is needed and there should be dependencies for external software. Any downloads from the Internet need to be protected from MITM attacks which means that a responsible software developer has to read through the build system and make sure that appropriate PGP signature checks etc are performed. It could be that the files that the Mesos build downloaded from the Apache site had appropriate PGP checks performed but it would take me extra time and effort to verify this and I can t distribute software without being sure of this. Also reproducible builds are one of the latest things we aim for in the Debian project, this means we can t just download files from web sites because the next build might get a different version. Finally the fpm (Fucking Package Management) tool is a Ruby Gem that has to be installed with the gem install command. Any time you specify a gem install command you should include the -v option to ensure that everyone is using the same version of that gem, otherwise there is no guarantee that people who follow your documentation will get the same results. Also a quick Google search didn t indicate whether gem install checks PGP keys or verifies data integrity in other ways. If I m going to compile software for other people to use I m concerned about getting unexpected results with such things. A Google search indicates that Ruby people were worried about such things in 2013 but doesn t indicate whether they solved the problem properly.

• [1] https://github.com/mesosphere/mesos-deb-packaging/issues/109
• [2] https://github.com/mesosphere/mesos-deb-packaging
• [3] https://github.com/SELinuxProject/selinux/wiki/Releases

I normally don t play survival games or walking simulators, but The Long Dark by Hinterland Games, which I purchased back then when it was still in early access on Steam, took me into new realms. You are tossed out into the Canadian wilderness with hardly anything, and your only aim is to survive, find shelter, food, craft tools, hunt for food, explore. And while everything by now is Sandbox mode, on August 1st the first episode of Story mode is released. Best time to get the game! You will be greeted with some icy nights, but also with great vistas, relaxed evenings at a fire place, you will try to survive on moldy food and rotten energy bars, but also feast on the fireside while reading a good book. A real treat this game! Sandbox mode features five different areas to explore. Each one is large enough to spend weeks (in game time) wandering around. The easiest area to start with is Mystery Lake, with plenty of shelter (several huts) and abundance of resources. And just in case you are getting bored, all the areas are connected via tunnels or caves and one can wander of into the neighboring places. My home in Mystery Lake was always the Camp Office, the usual suspect. Nice views, fishing huts nearby to get fresh fish, lots of space. After managing to get from your starting place (which is arbitrary as far as I see) to one of the shelters, one starts collecting food, wood, savaging every accessible place for tools, weapons, burning material. And soon the backpack becomes to heavy, and one needs to store stuff and decide what to take. This is a very well done part of the game. The backpack is not limited by number of items, but you are limited in weight you can carry. That includes clothes (which can get quite heavy) and all the items in your backpack. In addition, the longer the day and the more tired you become, the less weight you can carry. And if the backpack starts getting too heavy you crawl to a very slow movement. There are many influences of the outside world on the player s condition: temperature, the wetness of your clothes, hunger, thirst, exhaustion, but also infections and bruises, all need to be taken care of, otherwise the end is coming faster than one wishes for. I have only two things to complain: First, if one walks outside, or runs outside, the own body temperature does not rise. This is unrealistically and should have been taken into account. The other thing is the difficulty: I have played weeks in game time in the easiest level, without any problem. But the moment I switched to the second level of difficulty (of 5!), I not even manage it for 2(!) days. Wolves, starvation, thirst, any of those kills me within an instant. I don t want to know how the hardest level feels, but it has a certain steep step here. The game takes a very realistic view onto the weather: Every day is different, sunny, foggy, blizzard, windy, often changing very quickly. It is wise to plan one s activities according to the weather, as it is very unforgiving. With beautifully crafted landscapes, loads of areas to explore, your own pride to survive for at least a few weeks, and lots of tools to find and craft and try out, this games, even while it is still in Sandbox mode, is a real feat. My absolute favorite since I have finished the Talos Principle and Portal series, absolutely recommendable!

Earlier today, I demonstrated how to work with coinduction in the theorem provers Isabelle, Coq and Agda, with a very simple example. This reminded me of a discussion I had in Karlsruhe with my then colleague Denis Lohner: If coinduction is the dual of induction, why do the induction principles look so different? I like what we observed there, so I d like to share this. The following is mostly based on my naive understanding of coinduction based on what I observe in the implementation in Isabelle. I am sure that a different, more categorial presentation of datatypes (as initial resp. terminal objects in some category of algebras) makes the duality more obvious, but that does not necessarily help the working Isabelle user who wants to make sense of coninduction.

Inductive lists I will use the usual polymorphic list data type as an example. So on the one hand, we have normal, finite inductive lists:

datatype 'a list = nil   cons (hd : 'a) (tl : "'a list")

with the well-known induction principle that many of my readers know by heart (syntax slightly un-isabellized):

P nil   ( x xs. P xs   P (cons x xs))     xs. P xs

Coinductive lists In contrast, if we define our lists coinductively to get possibly infinite, Haskell-style lists, by writing

codatatype 'a llist = lnil   lcons (hd : 'a)  (tl : "'a llist")

we get the following coinduction principle:

(  xs ys.
    R xs ys'   (xs = lnil) = (ys = lnil)  
               (xs   lnil   ys'   lnil  
	         hd xs = hd ys   R (tl xs) (tl ys)))  
  (  xs ys. R xs ys   xs = ys)

This is less scary that it looks at first. It tell you if you give me a relation R between lists which implies that either both lists are empty or both lists are nonempty, and furthermore if both are non-empty, that they have the same head and tails related by R, then any two lists related by R are actually equal. If you think of the infinte list as a series of states of a computer program, then this is nothing else than a bisimulation. So we have two proof principles, both of which make intuitive sense. But how are they related? They look very different! In one, we have a predicate P, in the other a relation R, to point out just one difference.

Relation induction To see how they are dual to each other, we have to recognize that both these theorems are actually specializations of a more general (co)induction principle. The datatype declaration automatically creates a relator:

rel_list :: ('a   'b   bool)   'a list   'b list   bool

The definition of rel_list R xs ys is that xs and ys have the same shape (i.e. length), and that the corresponding elements are pairwise related by R. You might have defined this relation yourself at some time, and if so, you probably introduced it as an inductive predicate. So it is not surprising that the following induction principle characterizes this relation:

Q nil nil  
( x xs y ys. R x y   Q xs ys   Q (cons x xs) (cons y ys))  
( xs ys   rel_list R xs ys   Q xs ys)

Note how how similar this lemma is in shape to the normal induction for lists above! And indeed, if we choose Q xs ys (P xs xs = ys) and R x y (x = y), then we obtain exactly that. In that sense, the relation induction is a generalization of the normal induction.

Relation coinduction The same observation can be made in the coinductive world. Here, as well, the codatatype declaration introduces a function

rel_llist :: ('a   'b   bool)   'a llist   'b llist   bool

which relates lists of the same shape with related elements only that this one also relates infinite lists, and therefore is a coinductive relation. The corresponding rule for proof by coinduction is not surprising and should remind you of bisimulation, too:

( xs ys.
    R xs ys   (xs = lnil) = (ys = lnil)  
              (xs   lnil   ys   lnil  
	        Q (hd xs) (hd ys)   R (tl xs) (tl ys)))  
(  xs ys   R xs ys   rel_llist Q xs ys)

It is even more obvious that this is a generalization of the standard coinduction principle shown above: Just instantiate Q with equality, which turns rel_llist Q into equality on the lists, and you have the theorem above.

The duality With our induction and coinduction principle generalized to relations, suddenly a duality emerges: If you turn around the implication in the conclusion of one you get the conclusion of the other one. This is an example of cosomething is something with arrows reversed . But what about the premise(s) of the rules? What happens if we turn around the arrow here? Although slighty less immediate, it turns out that they are the same as well. To see that, we start with the premise of the coinduction rule, reverse the implication and then show that to be equivalent to the two premises of the induction rule:

( xs ys.
    R xs ys   (xs = lnil) = (ys = lnil)  
              (xs   lnil   ys   lnil  
	        Q (hd xs) (hd ys)   R (tl xs) (tl ys)))
=   case analysis (the other two cases are vacuously true)  
  ( xs ys.
    xs = lnil   ys = lnil  
    R xs ys   (xs = lnil) = (ys = lnil)  
              (xs   lnil   ys   lnil  
	        Q (hd xs) (hd ys)   R (tl xs) (tl ys)))
  ( xs ys.
    xs   lnil   ys   lnil
    R xs ys   (xs = lnil) = (ys = lnil)  
              (xs   lnil   ys   lnil  
	        Q (hd xs) (hd ys)   R (tl xs) (tl ys)))
=   simplification  
  ( xs ys.  xs = lnil   ys = lnil   R xs ys
  ( x xs y ys.  R (cons x xs) (cons y ys)   (Q x y   R xs ys))
=   more rewriting  
  R nil nil
  ( x xs y ys. Q x y   R xs ys   R (cons x xs) (cons y ys))

Conclusion The coinduction rule is not the direct dual of the induction rule, but both are specializations of more general, relational proof methods, where the duality is clearly present. More generally, this little excursion shows that it is often beneficial to think of types less as sets, and more as relations this way of thinking is surprisingly fruitful, and led to proofs of parametricity and free theorems and other nice things.

The DeepSpec Summer School is almost over, and I have had a few good discussions. One revolved around coinduction: What is it, how does it differ from induction, and how do you actually prove something. In the course of the discussion, I came up with a very simple coinductive exercise, and solved it both in Coq and Isabelle

The task Define the extended natural numbers coinductively. Define the min function and the relation. Show that min(n, m) n holds.

Coq The definitions are straight forward. Note that in Coq, we use the same command to define a coinductive data type and a coinductively defined relation:

CoInductive ENat :=
    N : ENat
    S : ENat -> ENat.
CoFixpoint min (n : ENat) (m : ENat)
  :=match n, m with   S n', S m' => S (min n' m')
                      _, _       => N end.
CoInductive le : ENat -> ENat -> Prop :=
    leN : forall m, le N m
    leS : forall n m, le n m -> le (S n) (S m).

The lemma is specified as

Lemma min_le: forall n m, le (min n m) n.

and the proof method of choice to show that some coinductive relation holds, is cofix. One would wish that the following proof would work:

Lemma min_le: forall n m, le (min n m) n.
Proof.
  cofix.
  destruct n, m.
  * apply leN.
  * apply leN.
  * apply leN.
  * apply leS.
    apply min_le.
Qed.

but we get the error message

Error:
In environment
min_le : forall n m : ENat, le (min n m) n
Unable to unify "le N ?M170" with "le (min N N) N

Effectively, as Coq is trying to figure out whether our proof is correct, i.e. type-checks, it stumbled on the equation min N N = N, and like a kid scared of coinduction, it did not dare to run the min function. The reason it does not just run a CoFixpoint is that doing so too daringly might simply not terminate. So, as Adam explains in a chapter of his book, Coq reduces a cofixpoint only when it is the scrutinee of a match statement. So we need to get a match statement in place. We can do so with a helper function:

Definition evalN (n : ENat) :=
  match n with   N => N
                 S n => S n end.
Lemma evalN_eq : forall n, evalN n = n.
Proof. intros. destruct n; reflexivity. Qed.

This function does not really do anything besides nudging Coq to actually evaluate its argument to a constructor (N or S _). We can use it in the proof to guide Coq, and the following goes through:

Lemma min_le: forall n m, le (min n m) n.
Proof.
  cofix.
  destruct n, m; rewrite <- evalN_eq with (n := min _ _).
  * apply leN.
  * apply leN.
  * apply leN.
  * apply leS.
    apply min_le.
Qed.

Isabelle In Isabelle, definitions and types are very different things, so we use different commands to define ENat and le:

theory ENat imports  Main begin
codatatype ENat =  N   S  ENat
primcorec min where
   "min n m = (case n of
       N   N
       S n'   (case m of
        N   N
        S m'   S (min n' m')))"
coinductive le where
  leN: "le N m"
  leS: "le n m   le (S n) (S m)"

There are actually many ways of defining min; I chose the one most similar to the one above. For more details, see the corec tutorial. Now to the proof:

lemma min_le: "le (min n m) n"
proof (coinduction arbitrary: n m)
  case le
  show ?case
  proof(cases n)
    case N then show ?thesis by simp
  next
    case (S n') then show ?thesis
    proof(cases m)
      case N then show ?thesis by simp
    next
      case (S m')  with  n = _  show ?thesis
        unfolding min.code[where n = n and m = m]
        by auto
    qed
  qed
qed

The coinduction proof methods produces this goal:

proof (state)
goal (1 subgoal):
 1.  n m. ( m'. min n m = N   n = m')  
          ( n' m'.
               min n m = S n'  
               n = S m'  
	       (( n m. n' = min n m   m' = n)   le n' m'))

I chose to spell the proof out in the Isar proof language, where the outermost proof structure is done relatively explicity, and I proceed by case analysis mimiking the min function definition. In the cases where one argument of min is N, Isabelle s simplifier (a term rewriting tactic, so to say), can solve the goal automatically. This is because the primcorec command produces a bunch of lemmas, one of which states n = N m = N min n m = N. In the other case, we need to help Isabelle a bit to reduce the call to min (S n) (S m) using the unfolding methods, where min.code contains exactly the equation that we used to specify min. Using just unfolding min.code would send this method into a loop, so we restrict it to the concrete arguments n and m. Then auto can solve the remaining goal (despite all the existential quantifiers).

Summary Both theorem provers are able to prove the desired result. To me it seems that it is slightly more convenient in Isabelle because a lot of Coq infrastructure relies on the type checker being able to effectively evaluate expressions, which is tricky with cofixpoints, wheras evaluation plays a much less central role in Isabelle, where rewriting is the crucial technique, and while one still cannot simply throw min.code into the simpset, so working with objects that do not evaluate easily or completely is less strange.

Agda I was challenged to do it in Agda. Here it is:

module ENat where
open import Coinduction
data ENat : Set where
  N : ENat
  S :   ENat   ENat
min : ENat   ENat   ENat
min (S n') (S m') = S (  (min (  n') (  m')))
min _ _ = N
data le : ENat   ENat   Set where
  leN :    m    le N m
  leS :    n m      (le (  n) (  m))   le (S n) (S m)
min_le :    n m    le (min n m) n
min_le  S n'   S m'  = leS (  min_le)
min_le  N      S m'  = leN
min_le  S n'   N  = leN
min_le  N      N  = leN

I will refrain from commenting it, because I do not really know what I have been doing here, but it typechecks, and refer you to the official documentation on coinduction in Agda. But let me note that I wrote this using plain inductive types and recursion, and added , and until it worked.

Weblate 2.16 is almost ready (I expect no further code changes), so it's really great time to contribute to it's translations! Weblate 2.16 will be probably released during my presence at DebConf 17. As you might expect, Weblate is translated using Weblate, so the contributions should be really easy. In case there is something unclear, you can look into Weblate documentation. I'd especially like to see improvements in the Italian translation which was one of the first in Weblate beginnings, but hasn't received much love in past years.

Filed under: Debian English SUSE Weblate

Is it possible to do software development, mathematical or not, as mathematician in academics? This is a question I was asking myself recently a lot, seeing my own development from logician at a state university getting rid of foreigners to software developer. And then, a friend pointed me to this very depressing document: The origins of SageMath by William Stein, the main developer of SageMath. And I realized that it seems to be a global phenomenon that mathematicians who are interested in software development have to leave academics. What a sad affair. SageMath has a clear mission:

Creating a viable free open source alternative to Magma, Maple, Mathematica and Matlab.

All the Ma -software packages are commercial, and expensive. On the other hand they often have very good algorithms implemented. The Sage developers invested lots of time, energy, and brain power to develop excellent algorithm in an open source project for the mathematical researcher, but this investment wasn t honored in academic life. To quote from the presentation:

Issues with software dev in academia

• Hard money for software development is virtually nonexistent: I can t think of anyone I know who got tenured based on his or her software.
• Researchers on soft money are systematically discriminated against in favor of tenure-track and tenured faculty.
• Researchers are increasingly evaluated solely on bibliometric counts rather than an informed assessment of their overall portfolio of papers, code, software, industry engagement, or student supervision.

The origins of SageMath, p.31

I can fully agree to this. Both from my own experience as well as from those around me. The presentation slides are full of other examples, from the developers of NumPy, Jupyter, as well as statements by Stephen Wolfram from Mathematica about this issue. A textbook how to not setup academia. My assumption was that this hits only on non-tenured staff, the academic precariat. It is shocking to see that even William Stein with a tenure position is leaving academics. It seems the times are not ready

Every great open source math library is built on the ashes of someone s academic career.
The origins of SageMath, p.32

In this wiki entry, I describe how to extend a FAI nfsroot, so it can be used as a file system for a diskless client or a Live CD. A host can mount it via NFS when booting via PXE. You can create a Live CD easily by using the command fai-cd. This works also nicely with a Xfce desktop, and I've prepared a ISO image for easy testing. You can log in as user demo, and the password is fai. The next thing is to check, if we can use FAI's dirinstall or install method for creating the same environment, so it will be easy to create customized Live images. FAI

I finally received a copy of the Norwegian Bokm l edition of "The Debian Administrator's Handbook". This test copy arrived in the mail a few days ago, and I am very happy to hold the result in my hand. We spent around one and a half year translating it. This paperbook edition is available from lulu.com. If you buy it quickly, you save 25% on the list price. The book is also available for download in electronic form as PDF, EPUB and Mobipocket, as can be read online as a web page. This is the second book I publish (the first was the book "Free Culture" by Lawrence Lessig in English, French and Norwegian Bokm l), and I am very excited to finally wrap up this project. I hope "H ndbok for Debian-administratoren" will be well received.

I have forked the Mon network/server monitoring system. Here is a link to the new project page [1]. There hasn t been an upstream release since 2010 and I think we need more frequent releases than that. I plan to merge as many useful monitoring scripts as possible and support them well. All Perl scripts will use strict and use other best practices. The first release of etbe-mon is essentially the same as the last release of the mon package in Debian. This is because I started work on the Debian package (almost all the systems I want to monitor run Debian) and as I had been accepted as a co-maintainer of the Debian package I put all my patches into Debian. It s probably not a common practice for someone to fork upstream of a package soon after becoming a comaintainer of the Debian package. But I believe that this is in the best interests of the users. I presume that there are other collections of patches out there and I hope to merge them so that everyone can get the benefits of features and bug fixes that have been separate due to a lack of upstream releases. Last time I checked mon wasn t in Fedora. I believe that mon has some unique features for simple monitoring that would be of benefit to Fedora users and would like to work with anyone who wants to maintain the package for Fedora. I am also interested in working with any other distributions of Linux and with non-Linux systems. While setting up the mailing list for etbemon I wrote an article about DKIM and mailing lists (primarily Mailman) [2]. This explains how to setup Mailman for correct operation with DKIM and also why that seems to be the only viable option.

• [1] https://doc.coker.com.au/projects/etbe-mon/
• [2] https://doc.coker.com.au/internet/dkim-and-mailing-lists/

The Kit project in Go is the common project containing all the standard libraries or packages used across all the Go projects in the organization. Motivation I attended GopherCon India 2017, there was a talk on Package Oriented Design In Go by William Kennedy. In that talk, William explained some really important and thoughtful design principles which We can apply in our day to day life, while writing Go. Hence, I wanted to apply these design philosophies to the Go projects in which I have been working on as a part of OpenEBS project. I learnt a good practice of having a Go Kit project at the organization level from William s talk. What is the Kit Project? The Kit project in Go is the common project containing all the standard libraries or packages used across all the Go projects in the organization. Packages in the Kit project should follow design philosophies. Need for a Kit project Sometimes, We write same Go packages again and again to do the same task at different levels in the different Go projects under the same organization. For example, we write custom logger package in the different Go projects. If the custom logger package is same across the organization and can be reused by simply importing it, then this custom logger package is the perfect fit for Kit project. You can sense how much time and cost it will save for us when we have a Kit project. How to convert existing projects to have kit Maya is the kit project in the progress. I will walk through our journey of creating a Kit project called maya for OpenEBS organization from existing Go projects. At OpenEBS, as a open source and growing Go project, We value Go principles and We try hard to leverage Go s offerings. Maya is the Kit project for the Application projects like maya-apiserver, maya-storage-bot etc. Maya contains all the kubernetes & nomad API s, common utilities etc. needed for development of maya-apiserver and maya-storage-bot. In the near future, we are trying to push our custom libraries to maya. So that, it will become a promising Go kit project for OpenEBS community. We have specifically followed the package oriented design principles in Go to create maya as a kit project.

• Usability We moved common packages such as orchprovider, types, pkg to maya from maya-apiserver. These packages are very generic and can be used in most of the Go projects in OpenEBS organization. Brief details about new packages in Maya.
  • Orchprovider : orchprovider contains packages of different orchestrators such as kubernetes and nomad.
  • types: types provides all the generic types related to orchestrator.
  • pkg: pkg contains packages like nethelper, util etc.
  • volumes: volumes contain packages related to volume provisioner and profiles.
• Purpose Packages in the Kit project should be purposeful. These packages must provide, not contain. In maya, we have packages like types, orchprovider, volumes etc. name of these packages suggests the functionality provided by them.
• Portability Portability is important factor for packages in kit project. Hence, we are making maya in such a way that it will be easy to import and use in any Go project. Packages in the Maya are not single point of dependency and all the packages are independent of each other.

Project structure for the Maya as Kit project

• Without having Maya as a kit project:
  • Project structure for the Maya

  
      buildscripts
          docker
      command
      docs
      example
      scripts
          config
      templates
  

  • Project structure for the Maya-apiserver

      buildscripts
        docker
      cmd
      docs
      lib
        api
          v1
        artifacts
          docker
          docker-compose
          k8s
        config
        flaghelper
        loghelper
        mockit
          etcmayaserver
        nethelper
        orchprovider
          k8s
          nomad
        profile
          orchprovider
          volumeprovisioner
        server
        util
        volumeprovisioner
            jiva
      proposals
  

• Maya as a kit project
  • Project structure for the Maya (Kit Project)

  
      buildscripts
          docker
      command
      docs
      example
      orchprovider
          k8s
              v1
          nomad
              v1
      pkg
          nethelper
          util
      scripts
          config
      templates
      types
          v1
              profile
                  orchestrator
      volumes
          profile
              volumeprovisioner
          provisioner
              jiva
  

  • Project structure for the Maya-apiserver (Application Project)

      buildscripts
        docker
      cmd
      docs
      lib
        artifacts
          docker
          docker-compose
          k8s
        config
        flaghelper
        loghelper
        server
      proposals
  

Example usage of maya kit project in maya-apiserver Maya-apiserver uses maya as a Kit project. Maya-apiserver exposes OpenEBS operations in form of REST APIs. This allows multiple clients e.g. volume related plugins to consume OpenEBS storage operations exposed by Maya API server.

• Need for orchestration providers in maya-apiserver Maya-apiserver has been designed keeping storage inside containers in mind. In other words, containerized storage. This leads to maya-apiserver using best of breed container orchestrators. Hence maya-apiserver abstracts the container related orchestration needs in form of orchprovider namespace. Therefore,to achieve above need maya-api server uses orchprovider package provided by Maya.
• Need for volumes in the maya-apiserver There can be multiple persistent volume provisioners e.g. jiva & cstor which has the details of workings of its volume. maya-apiserver tries to abstract these into volumes operations whose specifics will reside in individual namespaces i.e. jiva, cstor, etc. Hence, maya-apiserver will make use of volumes package from Maya to satisfy above requirements.

Conclusion Go Kit project should contain packages which are usable, purposeful and portable. Go Kit projects will improve the efficiency of the organization at both human and code level.

(image shamelessly copied from Noodles' Emptiness) This year I will only make it to DebConf, not to DebCamp. But, still, I am very very happy and excited as the travel date looms nearer! I have ordered some of the delicacies for the Cheese and Wine party, signed up for the public bicycle system of Montreal, and done a fair share of work with the Content Team; finally today we sent out the announcement for the schedule of talks. Of course, there are several issues yet to fix, and a lot of things to do before traveling... But, no doubt about this: It will be an intense week! Oh, one more thing while we are at it: The schedule as it was published today does not really look like we have organized stuff into tracks But we have! This will be soon fixed, adding some color-coding to make tracks clearer on the schedule. This year, I pushed for the Content Team to recover the notion of tracks as an organizative measure, and as something that delivers value to DebConf as a whole. Several months ago, I created a Wiki page for the DebConf tracks, asking interested people to sign up for them. We currently have the following tracks registered:

Blends

Andreas Tille

Debian Science

Michael Banck

Cloud and containers

Luca Filipozzi

Embedded

Pending

Systems administration, automation and orchestation

Pending

Security

Gunnar Wolf

We have two tracks still needing a track coordinator. Do note that most of the tasks mentioned by the Wiki have already been carried out; what a track coordinator will now do is to serve as some sort of moderator, maybe a recurring talkmeister, ensuring continuity and probably providing for some commentary, giving some unity to its sessions. So, the responsibilities for a track coordinator right now are quite similar to what is expected for video team volunteers but to a set of contiguous sessions. If you are interested in being the track coordinator/moderator for Embedded or for Systems administration, automation and orchestation or even to share the job with any of the other, registered, coordinators, please speak up! Mail content@debconf.org and update the table in the Wiki page. See you very soon in Montreal!

The DebConf17 orga team is proud to announce that over 120 activities have been scheduled so far, including 45- and 20-minute talks, team meetings, and workshops, as well as a variety of other events. Most of the talks and BoFs will be streamed and recorded, thanks to our amazing video team! We'd like to remind you that Saturday August 5th is also our much anticipated Open Day! This means a program for a wider audience, including special activities for newcomers, such as an AMA session about Debian, a beginners workshop on packaging, a thoughtful talk about freedom with regard to today's popular gadgets and more. In addition to the published schedule, we'll provide rooms for ad-hoc sessions where attendees will be able to schedule activities at any time during the whole conference. The current schedule is available at https://debconf17.debconf.org/schedule/ This is also available through an XML feed. You can use ConfClerk in Debian to consume this, or Giggity on Android devices: https://debconf17.debconf.org/schedule/mobile/ We look forward to seeing you in Montreal!

Some month ago I upgrade my (European) fenix3 to a (Japanese) fenix 5x, looking forward to the built-in maps as well as support for Japanese. I was about to write a great review, how content I have been with the fenix 3 and how much better the 5x is. Well, until I realized that Garmin s engineers seem to be brain-damaged and shipping broken by design devices: Just one word: Set an alarm on the watch, and wonder .. because you will never wake up the next day, since the alarm was deleted due to an unattended (so-called) sync operation. Happened to me, several times, worst was when I was with clients working as guide. So the problem is well known, see this link and this link and this link, and it is not restricted to the fenix. The origin of the problem is that Garmin is incapable of implementing a synchronization protocol. They have three sources of data: The Connect website, the Connect application on the smartphone, and the watch itself. And interestingly they only do pushes from Web to application to device, overwriting settings on the lower end. Which means, any alarm that I create on the watch will be overwritten, ie deleted, on every sync sorry, not sync, on every forced push from the mobile. It is a bit depressing, but I think the urgent solution is to exempt alarms from the synchronization completely, and remove them from both the connect web page and the application, until a better, a real, synchronization is implemented. I found a work-around in one of the threads on the Garmin forum, that is: create an alarm on the web or application, better alarms for each of the times you might need it. These will be synced to the device, and can be activated/deactivated on the device without problems. But to be honest this is not a solution I can really trust in when I need to get up in the morning. Besides this issue, which is unfortunately rather severe a restriction for me as mountaineer and mountain guide, I really love the watch and will write a more detailed review soon.

In a few days, I ll be attending DebCamp17 in Montr al, Canada. Here are a few things I plan to pay attention to:

• Calamares: My main goal is to get Calamares in a great state for inclusion in the archives, along with sane default configuration for Debian (calamares-settings-debian) that can easily be adapted for derivatives. Calamares itself is already looking good and might make it into the archives before DebCamp even starts, but the settings package will still need some work.
• Gnome Shell Extensions: During the stretch development cycle I made great progress in packaging some of the most popular shell extensions, but there are a few more that I d like to get in for buster: apt-update-indicator, dash-to-panel (done), proxy-switcher, tilix-dropdown, tilix-shortcut
• zram-tools: Fix a few remaining issues in zram-tools and get it packaged into the archive.
• DebConf Committee: Since January, I ve been a delegate on the DebConf committee, I m hoping that we get some time to catch up in person before DebConf starts. We ve been working on some issues together recently and so far we ve been working together really well. We re working on keeping DebConf organisation stable and improving community processes, and I hope that by the end of DebConf we ll have some proposals that will prevent some re-occurring problems and also help mend old wounds from previous years.
• ISO Image Writer: I plan to try out Jonathan Riddell s iso image writer tool and check whether it works with the use cases I m interested in (Debian install and live media, images created with Calamares, boots UEFI/BIOS on optical and usb media). If it does what I want I ll probably package it too if Jonathan Riddell didn t get to it yet.
• Hashcheck: Kyle Robertze wrote a tool called Hashcheck for checking install media checksums from a live environment. If he gets a break during DebCamp from video team stuff, I m hoping we can look at some improvements for it and also getting it packaged in Debian.

Since debhelper/10.3, there has been a number of performance related changes. The vast majority primarily improves bulk performance or only have visible effects at larger input sizes. Most visible cases are:

• dh + dh_* now scales a lot better for large number of binary packages. Even more so with parallel builds.
• Most dh_* tools are now a lot faster when creating many directories or installing files.
• dh_prep and dh_clean now bulk removals.
• dh_install can now bulk some installations. For a concrete corner-case, libssl-doc went from approximately 11 seconds to less than a second. This optimization is implicitly disabled with exclude (among other).
• dh_installman now scales a lot better with many manpages. Even more so with parallel builds.
• dh_installman has restored its performance under fakeroot (regression since 10.2.2)

For debhelper, this mostly involved:

• avoiding fork+exec of commands for things doable natively in perl. Especially, when each fork+exec only process one file or dir.
• bulking as many files/dirs into the call as possible, where fork+exec is still used.
• caching / memorizing slow calls (e.g. in parts of pkgfile inside Dh_Lib)
• adding an internal API for dh to do bulk check for pkgfiles. This is useful for dh when checking if it should optimize out a helper.
• and, of course, doing things in parallel where trivially possible.

How to take advantage of these improvements in tools that use Dh_Lib:

• If you use install_ file,prog,lib,dir , then it will come out of the box. These functions are available in Debian/stable. On a related note, if you use doit to call install (or mkdir ), then please consider migrating to these functions instead.
• If you need to reset owner+mode (chown 0:0 FILE + chmod MODE FILE), consider using reset_perm_and_owner. This is also available in Debian/stable.
  • CAVEAT: It is not recursive and YMMV if you do not need the chown call (due to fakeroot).
• If you have a lot of items to be processed by a external tool, consider using xargs(). Since 10.5.1, it is now possible to insert the items anywhere in the command rather than just in the end.
• If you need to remove files, consider using the new rm_files function. It removes files and silently ignores if a file does not exist. It is also available since 10.5.1.
• If you need to create symlinks, please consider using make_symlink (available in Debian/stable) or make_symlink_raw_target (since 10.5.1). The former creates policy compliant symlinks (e.g. fixup absolute symlinks that should have been relative). The latter is closer to a ln -s call.
• If you need to rename a file, please consider using rename_path (since 10.5). It behaves mostly like mv -f but requires dest to be a (non-existing) file.
• Have a look at whether on_pkgs_in_parallel() / on_items_in_parallel() would be suitable for enabling parallelization in your tool.
  • The emphasis for these functions is on making parallelization easy to add with minimal code changes. It pre-distributes the items which can lead to unbalanced workloads, where some processes are idle while a few keeps working.

Credits: I would like to thank the following for reporting performance issues, regressions or/and providing patches. The list is in no particular order:

• Helmut Grohne
• Kurt Roeckx
• Gianfranco Costamagna
• Iain Lane
• Sven Joachim
• Adrian Bunk
• Michael Stapelberg

Should I have missed your contribution, please do not hesitate to let me know.
Filed under: Debhelper, Debian