This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• gnome-shell-extension-shortcuts shows a keyboard shortcut overlay
• gnome-shell-extension-show-ip shows ip address in the notification area
• gnome-shell-extension-autohidetopbar hide the top panel
• gnome-shell-extension-caffeine prevents computer from suspending when enabled
• gnome-shell-extension-mediaplayer control mediaplayer from the system menu
• gnome-shell-extension-redshift change colour temperature to improve attention span and sleep paterns
• gnome-shell-extension-suspend-button adds suspend shortcut
• gnome-shell-extension-taskbar adds plenty of options for the top panel and optional bottom taskbar
• ugnome-shell-extension-top-icons-plus moves system tray icons from the bottom hidden menu to the top
• gnome-shell-extension-weather weather report in panel
• gnome-shell-extensions-gpaste clipbook manager
• gnome-shell-extension-onboard on screen keyboard manager

• gnome-shell-extension-remove-round-corners remove round corners on top panel
• gnome-shell-extension-proxy-switcher adds menu in system menu to quickly switch between proxy settings
• gnome-shell-extension-apt-update-indicator show apt status and available upgrades
• gnome-shell-extension-uptime-indicator- indicator that displays uptime
• gnome-shell-extension-dash-to-panel great extension that combines the dock into the top panel
• gnome-shell-extension-tilix-dropdown shortcut configurator for tilix s quake mode

• A new upstream release of apktool was uploaded to experimental.

• I packaged new upstream releases of megaglest and megaglest-data.
• I fixed a bug in pangzero (#857474) that crashed the game when someone pressed the pause key. The updated package will be part of Stretch.
• The severity was inflated and the issue debatable but since it took less time to fix bug #857801 in dopewars than writing this sentence, I did it anyway.
• I fixed bug #857236 and #857845 in holotz-castle. Background: There are various packages in Debian that ship a considerable amount of documentation which is usually a good thing. We always strive to optimize packages and reducing the package size is one option. In the past people thought that symlinking the doc directory of an arch:all (architecture-independent) package to an an arch:any (architecture-dependent) package saves disk space because it is not necessary to duplicate the same content on every architecture. Unfortunately this feature, dh-installdocs link-doc, is broken by design (#766711) and in its current state not usable for this use case. As a consequence I filed a bug report against tracker.debian.org #857851, asked for an improvement of piuparts status reports and also filed #857852 against dpkg which was later cloned into #858036 for debhelper. In a nutshell I would like to see better documentation how to use dh-maintscript-helper and *.maintscript files. I also believe it would be nice to simplify the latter by using only one file.

• I packaged version 5.4 of sweethome3d and added myself to Uploaders and closed two bugs (#854030),(#856769)
• I fixed an RC bug (#856626) in lucene-solr, more precisely in one of the configuration files of solr-tomcat, a search engine with Tomcat integration, that prevented the server from starting.
• I am still investigating an RC issue (#857343) in logback. This is a potential security vulnerability that allows remote attackers to execute arbitrary code. My first patch was incomplete and more backported code from the latest upstream release is required. Unfortunately upstream was not very helpful in tracking down the necessary code changes. My question still remains unanswered.
• Netbeans (#837081): Netbeans has been crashing from time to time. It is not easy to trigger the issue but it is related to libatk-wrapper-java-jni and the Accessibility ToolKit (ATK). I have cloned bug number #837081 as #858700 for now because I don t think it can be fixed in Netbeans.

• From 06. March until 13. March I was in charge of our LTS frontdesk. I triaged security issues in qbittorrent, imagemagick, freetype, glibc, vim, suricada, texlive-base, web2py, lxc, r-base, mysql-5.5, partclone, irrsi, wordpress, mupdf and php5.
• DLA-846-1. Issued a security update for libzip-ruby fixing 1 CVE.
• DLA-853-1. Issued a security update for pidgin fixing 1 CVE.
• DLA-855-1. Issued a security update for roundcube fixing 1 CVE.
• DLA-860-1. Issued a security update for wordpress fixing 3 CVE.
• DLA-870-1. Issued a security update for libplist fixing 3 CVE.
• DLA-872-1. Issued a security update for xrdp fixing 1 CVE.
• DLA-875-1. Issued a security update for php5 fixing 3 CVE.

• March 2017 also saw a new version of MediathekView (now in experimental).

Looking into self-financing Before I begin, I should mention that I started tracking my time working on free software more systematically. I spend a lot of time on the computer, as regular readers of this blog might remember so I wanted to know exactly how much time was paid vs free work. I was already using org-mode's time clock system to keep track of my work hours, so I just extended this to my regular free software contributions, which also helps in writing those reports. It turns out that over 60% of my computer time is spent working on free software. That's huge! I was expecting something more along the range of 20 to 40% of my time. So I started thinking about ways of financing this work. I created a Patreon page but I'm hesitant into launching such a campaign: the only thing worse than "no patreon page" is "a patreon page with failed goals and no one financing it". So before starting such an effort, I'd like to get a feeling of what other people's experience with it are. I know that joeyh is close to achieving his goals, but I can't compare with the guy that invented git-annex or debhelper, so I'm concerned I wouldn't be able to raise the same level of funding. So any advice you have, feel free to contact me in private or in the comments. If you would be ready to fund my work, I'd love to know about it, obviously, but I guess I wouldn't get real numbers until I actually open up such a page... Now, onto the regular report.

Wallabako I spent a good chunk of time completing most of the things I had in mind for Wallabako, which I mentioned quickly in the previous report. Wallabako is now much easier to installed, with clearer instructions, an easier to use configuration file, more reliable synchronization and read status propagation. As usual the Wallabako README file has all the details. I've also looked at better integration with Koreader, the free software e-reader that forms the basis of the okreader free software distribution which has been able to port Debian to the Kobo e-readers, a project I am really excited about. This project has the potential of supporting Kobo readers beyond the lifetime that upstream grants it and removes a lot of proprietary software and spyware that ships with the Kobo readers. So I have made a few contributions to okreader and also on koreader, the ebook reader okreader is based on.

Stressant I rewrote stressant, my simple burn-in and stress-testing tool. After struggling in turn with Debirf, live-build, vmdebootstrap and even FAI, I just figured maybe it wasn't the best idea to try and reinvent that particular wheel: instead of reinventing how to build yet another Debian system build tool, maybe I should just reuse what's already there. It turns out there's a well known, succesful and fairly complete recovery system called Grml. It is a Debian Derivative, so all I needed to do was to stop procrastinating and actually write the actual stressant tool instead of just creating a distribution with a bunch of random tools shipped in. This allowed me to focus on which tools were the best to stress test different components. This selection ended up being:

• lshw and smartmon-tools (smartctl) for hardware inventory
• coreutils's famous dd, hdparm, fio and (again) smartctl) for disk testing
• stress-ng for CPU testing
• iperf3 for network testing

fio can also be used to overwrite disk drives with the proper options (--overwrite and --size=100%), although grml also ships with nwipe for wiping old spinning disks and hdparm to do a secure erase of SSD disks (whatever that's worth). Stressant still needs to be shipped with grml for this transition to be complete. In the meantime, I was able to configure the excellent public Gitlab CI service to provide ISO images with Stressant built-in as a stopgap measure. I also need to figure out a way to automate starting stressant from a boot menu to automate deployments on a larger scale, although because I have little need for the feature at this moment in time, this will likely wait for a sponsor to show up for this to be implemented. Still, stressant has useful features like the capability of sending logs by email using a fresh new implementation of the Python SMTPHandler (BufferedSMTPHandler) which waits for logging to complete before sending a single email. Another interesting piece of code in there is the NegateAction argparse handler that enables the use of "toggle flags" (e.g. --flag / --no-flag). I'm so happy with the code that I figure I could just share it here directly:

class NegateAction(argparse.Action):
    '''add a toggle flag to argparse

    this is similar to 'store_true' or 'store_false', but allows
    arguments prefixed with --no to disable the default. the default
    is set depending on the first argument - if it starts with the
    negative form (define by default as '--no'), the default is False,
    otherwise True.
    '''
    negative = '--no'
    def __init__(self, option_strings, *args, **kwargs):
        '''set default depending on the first argument'''
        default = not option_strings[0].startswith(self.negative)
        super(NegateAction, self).__init__(option_strings, *args,
                                           default=default, nargs=0, **kwargs)
    def __call__(self, parser, ns, values, option):
        '''set the truth value depending on whether
        it starts with the negative form'''
        setattr(ns, self.dest, not option.startswith(self.negative))

Short and sweet. I wonder why stuff like this is not in the standard library yet - maybe just because no one bothered yet? It'd be great to get feedback of more experienced Pythonistas on this one. I hope that my work on Stressant is complete. I get zero funding for this work, and have little use for it myself: I manage only a few machines and such a tool really shines when you regularly put new hardware online, which is (fortunately?) not my case anymore. I'd be happy, of course, to accompany organisations and people that wish to further develop and use such a tool. A short demo of stressant as well as detailed description of how it works is of course available in its README file.

Standard third party repositories After looking at improvements for the grml repository instructions, I realized there was no real "best practices" document on how to configure an Apt repository. Sure, there are tools like reprepro and others, but those hardly qualify as policy: they are very flexible and there are lots of ways to create insecure repositories or curl sh style instructions, which we of course generally want to avoid. While the larger problem of Unstrusted Debian packages remain generally unsolved (e.g. when you install any .deb file, it can get root on your system), it seemed to me one critical part of this problem was how to add a random third-party repository to your machine while limiting, as much as possible, what possible attackers could do with such a repository. In other words, to solve the more general problem of insecure .deb files, we also need to solve the distribution problem, otherwise fixing the .deb files themselves will be useless. This lead to the creation of standardized repository instructions that define:

1. how to distribute the repository's public signing key (ie. over HTTPS)
2. how to name suites and components (e.g. use stable and main unless you have a good reason, and explain yourself)
3. recommend a healthy does of apt preferences pinning
4. how to distribute keys (e.g. with a derive-archive-keyring package)

I've seen so many third party repositories get this wrong. For example, a lot of repositories recommend this type of command to intialize the OpenPGP trust path:

curl http://example.com/key.asc   apt-key add -

This has the following problems:

• the key is transfered in plaintext and can easily be manipulated by an active attacker (e.g. a router on your path to the server or a neighbor in a Wifi cafe)
• the key is added to the main trust root, which allows the key to authentify as the real Debian archive, therefore giving it all rights over all packages
• since it's part of the global archive, it's difficult for a package to remove/add the key when a key rollover is necessary (and repositories generally don't provide a deriv-archive-keyring to do that process anyways)

An example of this are the Docker install instructions that, at least, manage to do this over HTTPS. Some other repositories don't even bother teaching people about the proper way of adding those keys. We settled for:

wget -O /usr/share/keyrings/deriv-archive-keyring.gpg https://deriv.example.net/debian/deriv-archive-keyring.gpg

That location was explicitly chosen to be out of the main trust directory, so that it needs to be explicitly added to the sources.list as well:

deb [signed-by=/usr/share/keyrings/deriv-archive-keyring.gpg] https://deriv.example.net/debian/ stable main

Similarly, we highly recommend users setup "apt pinning" to restrict what a given repository can do. Since pinning is so confusing, most people don't actually bother even configuring it and I have yet to see a single repo advise its users to configure those preferences, which are essential to limit what a repository can do. To keep configuration simple, we recommend this:

Package: *
Pin: origin deriv.example.net
Pin-Priority: 100

Obviously, for a single-package repository, the actual package name should be listed, e.g.:

Package: foo
Pin: origin deriv.example.net
Pin-Priority: 100

And the priority should probably be set to 1 unless you want to allow automatic upgrades. It is my hope that this design will get more traction in the years to come and become a de-facto standard that will be a key part in safely adding third party repositories. There is obviously much more work to be done to improve security when installing untrusted .deb files, and I encourage Debian developers to consider contributing to the UntrustedDebs discussions and particularly to the Teams/Dpkg/Spec/DeclarativePackaging work.

Signal R&D I spent a significant amount of time this month struggling with the Signal project on my phone. I'm still ambivalent on Signal: it's a centralized designed, too dependent on phone numbers, but I must admit they get a lot of things right and it's the only free-software platform that allows for easy-to-use, multi-platform videoconferencing that my family can use. I've been following Signal for a while: up until now, I had been using the LibreSignal rebuild of the official client, as it is distributed on a F-Droid repository. Because I try to avoid Google (proprietary) software on my phone, it's basically the only way I could even install Signal. Unfortunately, the repository is out of date and introduces another point of trust in the distribution model: now you not only need to trust the Signal authors to do the right thing, you also need to trust that F-Droid repo not to inject nasty code on your phone. I've therefore started a discussion about how Signal could be distributed outside of the Google Play Store. I'd like to think it's one of the things that led the Signal people to distribute an official copy of Signal outside of the playstore. After much struggling, I was able to upgrade to this official client and will be able to upgrade easily by just downloading the APK. (Do note that I ended up reinstalling and re-registering Signal, which unfortunately changed my secret keys.) I do hope Signal enters F-Droid one day, but it could take a while because it still doesn't work without Google services and barely works with MicroG, the free software alternative to the Google services clients. Moxie also set a list of requirements like crash reporting and statistics that need to be implemented on F-Droid's side before he agrees to the deployment, so this could take a while. I've also participated in the, ahem, discussion on the JWZ blog regarding a supposed vulnerability in Signal where it would leak previously unknown phone numbers to third parties. I reviewed the way the phone number is uploaded and, while it's possible to create a rainbow table of phone numbers (which are hashed with a truncated SHA-1 checksum), I couldn't verify the claims of other participants in the thread. For me, Signal still does the right thing with contacts, although I do question the way "read status" notifications get transmitted, but that belong in another bug report / blog post.

Debian Long Term Support (LTS) It's been more than a year working on Debian LTS, started by Raphael Hertzog at Freexian. I didn't work much in February so I had a lot of hours to catchup with, and was unfortunately unable to do so, partly because I was busy with other projects, and partly because my colleagues are doing a great job at resolving the most important issues. So one my concerns this month was finding work. It seemed that all the hard packages were either taken (e.g. my usual favorites, tiff and imagemagick, we done by others) or just too challenging (e.g. I don't feel quite comfortable tackling the LTS branch of the Linux kernel yet). I spent quite a bit of time trying to figure out what was wrong with pcre3, only to realise the "32" in the report was not about the architecture, but about the character width. Because of thise, I marked 4 CVEs (CVE-2017-7186, CVE-2017-7244, CVE-2017-7245, CVE-2017-7246) as "not-affected", since the 32-bith character support wasn't enabled in wheezy (or jessie, for that matter). I still spent some time trying to reproduce the issues, which require a compiler with an AddressSanitizer, something that was introduced in both Clang and GCC after Wheezy was released, which makes reproducing this fairly complicated... This allowed me to experiment more with Vagrant, however, and I have provided the Debian cloud team with a 32-bit Vagrant box that was merged in shortly after, although it doesn't show up yet in the official list of Debian images. Then I looked at the apparmor situation (CVE-2017-6507), Debian bug #858768). That was one tricky bug as well, since it's not a security issue in apparmor per se, but more an issue with things that assume a certain behavior from apparmor. I have concluded that Wheezy was not affected because there are no assumptions of proper isolation there - which are provided only starting from LXC 1.0 - and Docker is not in Wheezy. I also couldn't reproduce the issue on Jessie, but, as it turns out, the issue was sysvinit-specific, which is why I couldn't reproduce it under the default systemd configuration shipped with Jessie. I also looked at the various binutils security issues: as I reported on the mailing list, I didn't see anything serious enough in there to warrant a a security release and followed the lead of both the stable and Red Hat security teams by marking this "no-dsa". I similiarly reviewed the mp3splt security issues (specifically CVE-2017-5666) and was fairly puzzled by that issue, which seems to be triggered only the same address sanitization extensions than PCRE, although there was some pretty wild interplay with debugging flags in there. All in all, it seems we can't reproduce that issue in wheezy, but I do not feel confident enough in the results to push that issue aside for now. I finally uploaded the pending graphicsmagick issue (DLA-547-2), a regression update to fix a crash that was introduced in the previous release (DLA-547-1, mistakenly named DLA-574-1). Hopefully that release should clear up some of the confusion and fix the regression. I also released DLA-879-1 for the CVE-2017-6369 in firebird2.5 which was an interesting experiment: I couldn't reproduce the issue in a local VM. After following the Ubuntu setup tutorial, as I wasn't too familiar with the Firebird database until now (hint: the default username and password is sysdba/masterkey), I ended up assuming we were vulnerable and just backporting the patch after seeing the jessie folks push out a release just in case. I also looked at updating the ca-certificates package to deal with the pending WoSign/Startcom removal: I made an explicit list of the CAs that need to be removed after reviewing the Mozilla list. I also sent a patch for an unrelated issue where ca-certificates is writing to /usr/local (!!) in Debian bug #843722. I have also done some "meta" work in starting a discussion about fixing the missing DLA links in the tracker, as you will notice all of the above links lead to nowhere. Thanks to pabs, there are now some links but unfortunately there are about 500 DLAs missing from the website. We also discussed ways to Debian bug #859123, something which is currently a manual process. This is now in the hands of the excellent webmaster team. I have also filed a few missing security bugs (Debian bug #859135, Debian bug #859136), partly because I wanted to help the security team. But it turned out that I felt the script needed some improvements, so I submitted a patch to improve the script so it is easier to run.

Other projects As usual, there's the usual mixed bags of chaos:

• converted the Charybdis IRC operator guide from SGML (yuck) to RST (PR)
• patches and discussions on the restic backup software review and backup data generation program in the hope of reducing duplication with the other two similar projects (Obnam's genbackupdata and Borg's backupdata)
• atheme-services security upload (still need to do the same with charybdis
• neat cleanup of the Python Cryptography table of contents (PR)
• basic maintenance on the Linkchecker project, including particularly a set of community guidelines which I'd like to standardize as well
• raised a red flag on the security of the Weechat project by reporting that most users are not aware that Weechat relays provide arbitrary remote code execution to clients, upstream thinks users are to blame, issue still opened
• git-annex documentation: semi-synchronized remotes and antipatterns
• tried to unblock the mess that TLS verification still is in Emacs (Debian bug #816063)

More stuff on Github...

• Clone the existing Debian system in a new partition and provide a dual boot using Grub.
• Change the desktop wallpaper from the previous Spacefun theme to one in rainbow colors.
• Fork all the packages whose source code was present in the initial Debian system, patch multiple bugs in those packages and some patches more for some tricky security problems.
• Add ten new language locales that do not correspond to any language spoken in Earth, with full translation for four of them.
• A copy of the Debian website repository, migrated to the git version control system and perfectly running, has been found in the /home/earth0/Documents folder. This new repo includes code to show the Debian micronews in the home page and many other improvements, keeping the style of not needing JavaScript and providing a nice control of up-to-date/outdated translations, similar to the one existing in Debian.

• Moving the mercurial to git SHA1 mapping to the helper process (Finding a git bug in the process).
• Tracking mercurial manifest heads in the helper process.
• Removing most of the synchronous calls to the helper happening during a clone.

• The overall clone is now about 11 minutes faster than 0.4.0 (and about 50 minutes faster than master as of be75326!)
• Non-shared memory use of the git-remote-hg process stays well under 2GB during the whole clone, with no spike at the end.
• git-cinnabar-helper now uses more memory, but the sum of both processes is less than what it used to be, even when compensating for the glibc memory allocator issue. One thing to note is that while the git-cinnabar-helper memory use goes above 2GB at the end of the clone, a very large part is due to the pack window size being 1GB on 64-bits (vs. 32MB on 32-bits). Memory usage should stay well under the 2GB address space limit on a 32-bits system.
• CPU usage is well above 100% for most of the clone.

• The Import manifests phase is now 13 minutes faster than it was in 0.4.0.
• The Read and import files phase is still almost 4 minutes slower than in 0.4.0.
• The Import changesets phase is still almost 2 minutes slower than in 0.4.0.
• But the Finalization phase is now 3 minutes faster than in 0.4.0.

• The allocated memory (requested bytes) is well below what it was, and the spike at the end is entirely gone. It also more closely follows the amount of raw data we re holding on to (which makes sense since most of the bookkeeping was moved to the helper)
• The number of live allocations (allocated memory pointers that haven t been free()d yet) has gone significantly down as well.
• The cumulated[*] bytes are now in a much more reasonable range, with the lower bound close to the total amount of data we re dealing with during the clone, and the upper bound slightly over twice that amount (the upper bound for the be75326 is not shown here, but it was around 45TB; less than 7TB is a big improvement).
• There are less allocator calls during the first phases and the Importing changesets phase, but more during the Reading and importing files and Importing manifests phases.

void *p = malloc(42);
free(p);

void *p = malloc(42);
void *other = malloc(42);
free(p);

void *p = malloc(42);
p = realloc(p, 84);
free(p);

• As expected, there are more short-lived allocations in 35c18e7.
• Around 90% of allocations have a lifetime spanning 10% of the process life or less. This is a rather surprisingly large amount of allocations with a very large lifetime.
• Around 80% of allocations have a lifetime spanning 0.01% of the process life or less.
• The median lifetime is around 0.0000002% (2*10^-7%) of the process life, which, in absolute terms is around 500 allocator function calls between a malloc and a free.
• If we consider every imported changeset, manifest and file to require a similar number of allocations, and considering there are about 2.7M of them in total, each spans about 3.7*10^-7%. About 53% of all allocations on be75326 and 57% on 35c18e7 have a lifetime below that. Whenever I get to look more closely to memory usage again, I ll probably look at the data separately for each individual phase.
• One surprising fact, that doesn t appear on the graph because of the logarithmic scale not showing 0 on the horizontal axis, is that 9.3% on be75326 and 7.3% on 35c18e7 of all allocations have a lifetime of 0. That is, whatever the code using them is doing, it s not allocating or freeing anything else, and not reallocating them either.

• As mentioned in a previous post, there are bigger wins to be had from not keeping manifests data around in memory at all, and by importing it directly instead.
• In time, a lot of the import code is meant to move to the helper, where the constraints are completely different, and it might not be worth spending time now on reducing the memory usage of python code that might go away soon(ish). The situation was bad and necessitated action rather quickly, but we re now in a place where it s not as bad anymore.

Changes

• gitk: colour of remote refs
• myrepos: accumulated patches
• check-all-the-things: fix the yamllint cmd, add TODO items for haxelint and Dockerfile checkers, add checks for dodgy stuff, dead code, GIFs, Ruby (rubocop, roodi), Mono CIL binaries, FITS images and TeX
• gnome-shell-extension-show-ip: came up with the idea to fix the main blocker to deforking it, cleanups, show the primary IP address and show IP addresses in the menu
• Spelling:
  • codespell: broser, optionnal, optionnaly, reuest, ressource (hi samba folks), swtich, assumme(s), checg(ed), workes, accidentially, uninitalized, permision(s), whe, gruop(s), determing, imilar, tigthly, standar, iff, nott, technlogy, playble, redeable, diagnotic, langauage, mor, soundard, firmwware, containt(s), mazilla (hi malware authors), mozila, cilyndre(s), sepatae
  • lintian: broser, optionnaly, kenrel, swtich, assumme(s), checg(ed), workes, whe, gruops, imilar, tigthly, standar, iff, nott, technlogy, playble, debia, redeable, diagnotic, mor, soundard, firmwware, containt, mazilla, mozila, cilyndre(s), sepatae
• Wikipedia: Message-ID
• Debian wiki: AndroidTools, AppArmor/Contribute/Upstream, BSP/2017/05/ch/Zurich, CrossCompiling, DateTime, DebianEdu/Status/Stretch, DebianEvents/br/2017/MiniDebconfCuritiba/Atividades/CFP, DebianKernel/UserspaceTools, DebianMultimedia/FFmpeg (Unstripped), DebianRepository (Unofficial, UseThirdParty), DebianScience/UnofficialRepository, DebianUnofficial, dedup.debian.net, Derivatives/Census (QA (1, 2), Qubes, SparkyLinux), Exploits, fr/Simple-CDD, Games/Unsuitable, Hardening/RepoAndImages, Ideas/Ports, iMacG5, IRC, it/UltimateDebianDatabase, Merchandise, PaulWise/InterestingSoftware (1, 2), piuparts/FAQ, pkg-escience, qa.debian.org, QuickInstall, RepositoryLocal, Services/PublicUddMirror, Simple-CDD/Howto, SuitesAndReposExtension, SummerOfCode2006, SummerOfCode2014/Projects/DebianMetricsPortal, SummerOfCode2017/Projects/QA_BiologyApps, sv/IRC, SystemBuildTools, UltimateDebianDatabase, Vangelis Mouhtsis
• Debian website: Remove old unused and not working website hosting sponsor functionality (1 2 3 4 5 6 7), use correct SSL CA cert store, fix and https links to debian.ch and fix HTML entity escaping in the mirrors list (1 2)
• Debian ports: help rescue the #debian-ports IRC channel, remove dead mirror, update mirrors page, remove d-p.o reference, update bits.d.o and update packages.d.o
• Debian packages site: remove some git cruft
• Debian security tracker: remove NFU from webkit issue and add source link to DLAs
• Planet Debian derivatives: Add Qubes
• Debian package uploads: needrestart-session 0.3-4.1 and valgrind 1:3.10.0-4~bpo7+1

Issues

• Crashes in evolution, hexchat, gpaste, pidgin and eog
• Perl @INC issue in libthrift-perl
• Regression in samba-common-bin
• Debian migration unblocks of needrestart-session (1, 2)
• Debian Policy violation in selenium-firefoxdriver
• Important issue in needrestart-session
• Minor issues in gnupg and giflib-tools
• Features in file, debdelta, diffoscope, nss-passwords and linux

Review

• Spam: reported 5 Debian bug reports and 289 Debian mailing list posts
• Debian wiki: RecentChanges for the month
• Debian screenshots:
  • approved spectrwm (2), ranger, i3, kupfer, zathura (3), gimagereader, yubioath-desktop (3), fluxbox, libreoffice-writer, gtklp, terminator, wireshark-qt, wireshark-gtk, synaptic, empathy, gnome-control-center, shutter (2), gnome-online-accounts, gnome-documents, gnome-contacts, network-manager-openvpn-gnome, gnome-shell-extension-system-monitor, neofetch, octave-sparsersb, audacious, smplayer, bluefish, gpodder, delaboratory, geeqie, fracplanet and braillegraph
  • rejected hotssh (artefacts in screenshot), libsixel-bin (copied from elsewhere), spectrwm (not a screenshot) and geeqie (worse version of accepted image)
  • approved deletion (despite bogus reason) of snmp (screenshots site not a support forum, was a diagram not a screenshot)
  • rejected deletion of kupfer (we do not remove old versions), vlc (external UI in the screenshot is minimal), vokoscreen/fonts-ecolier-lignes-court (garbage in reason field), aptitude (discussing non-English screenshots with Christoph Haas), hydra-gtk (greetings are not valid reasons) and oxygen-icon-theme (the screenshot is not a reCAPTCHA)

Administration

• Debian systems: apply a patch to userdir-ldap, ask a local admin to reset a dead powerpc buildd, remove dead SH4 porterboxen from LDAP, fix perms on www.d.o OC static mirror, report false positives in an an automated abuse report, redirect 1 student to FAQs/support/DebianEdu, redirect 1 event organiser to partners/trademark/merchandise/DPL, redirect 1 guest account seeker to NM, redirect 1 @debian.org desirer to NM, redirect 1 email bounce to a changes@db.d.o user, redirect 2 people to the listmasters, redirect 1 person to Debian Pure Blends, redirect 1 user to a service admin and redirect 2 users to support
• Debian packages site: deploy my ports/cruft changes
• Debian wiki: poke at HP page history and advise a contributor, whitelist 13 email address, whitelist 1 domain, check out history of a banned IP, direct 1 hoster to DebConf17 sponsors team, direct 1 user to OpenStack packaging, direct 1 user to InstallingDebianOn and h-node.org, direct 2 users to different ways to help Debian and direct 1 emeritus DD on repository wiki page reorganisation
• Debian QA: fix an issue with the PTS news, remove some debugging cruft I left behind, fix the usertags on a QA bug and deploy some code fixes
• Debian mentors: security upgrades and service restarts
• Openmoko: security upgrades and reboots

Communication

• Invite GALPon MiniNo, Scibian, Roche Linux and ubilinux to the Debian derivatives census
• Welcome Qubes to the Debian derivatives census
• Start discussion about how many people contact the wrong part of Debian for their query

Sponsors The valgrind backport, samba and libthrift-perl bug reports were sponsored by my employer. All other work was done on a volunteer basis.

1. Reference counting is hard. I'm creating a new drm_master, which means creating a new file. There were a bunch of reference counting errors in my first pass; I've cleaned those up while also reducing the changes needed to the rest of the DRM code.
2. Added a 'mask_lease' value to the leases -- this controls whether resources in the lease are hidden from the lessor, allowing the lessor to continue to do operations on the leased resources if it likes.
3. Hacked the mutex locking assertions to not crash in normal circumstances. I'm now doing: BUG_ON(__mutex_owner(&master->dev->mode_config.idr_mutex) != current); to make sure the mutex is held by the current thread, instead of just making sure some thread holds the mutex. I have this memory of a better way to do this, but now I can't dig it up. Suggestions welcome, of course.

• What should happen when a Lessor is closed? Should all access to controlled resources be revoked from all descendant Lessees? Answer: The lessor is referenced by the lessee and isn't destroyed until it has been closed and all lessees are destroyed.
• How about when a Lessee is closed? Should the Lessor be notified in some way? Answer: I think so? Need to figure out a mechanism here.
• CRTCs and Encoders have properties. Should these properties be automatically included in the lease? Answer -- no, userspace is responsible for constructing the entire lease.

1. Changing leases should update sub-leases. When you reduce the resources in one lease, the kernel should walk any sub-leases and clear out resources which the lessor no longer has access to.
2. Sending events when leases are created/destroyed. When a lease is created, if the mask_lease value is set, then the lessor should get regular events describing the effective change. Similarly, both lessor and lessee should get events when a lease is changed.
3. Refactoring the patch series to squash intermediate versions of the new code.

• Extend RandR to add a lease-creation request that returns a file descriptor. This will take a mode and the server will set that mode before returning.
• Provide some EDID-based matching for HMD displays in the X server. The goal is to 'hide' these from the regular desktop so that the HMD screen doesn't flicker with desktop content before the lease is created. I think what I want is to let an X client provide some matching criteria and for it to get an event when a output is connected with matching EDID information.

• linux: https://cgit.freedesktop.org/~keithp/linux/log/?h=drm-leases
• libdrm: https://cgit.freedesktop.org/~keithp/drm/log/?h=drm-lease
• kmscube: https://cgit.freedesktop.org/~keithp/kmscube/log/?h=drm-lease

• Fixed two issues in try.diffoscope.org, a web-based version of the diffoscope in-depth and content-aware diff utility:
  • Fix command-line API breakage. (commit)
  • Use deb.debian.org over httpredir.debian.org. (commit)
• Made a number of improvements to travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every code change) travis.debian.net, including:
  • Correctly detecting the distribution to build with for some tags. (commit)
  • Use Lintian from the backports repository where appropriate. (#44)
  • Don't build upstream/ branches even if they contain .travis.yml files. (commit)
• Fixed an issue in django-staticfiles-dotd, my Django staticfiles adaptor to concatentate .d-style directories, where some .d directories were being skipped. This was caused by modifying the contents of a Python list during iteration. (#3)
• Performed some miscelleanous cleanups in django12factor, a Django utility to make projects adhere better to the 12-factor web-application philosophy. (#58)
• Submitted a pull request for Doomsday-Engine, a portable, enhanced source port of Doom, Heretic and Hexen, to make the build reproducible (#16)
• Created a pull request for gdata-python-client (a Python client library for Google APIs) to make the build reproducible. (#56)
• Authored a pull request for the MochaJS JavaScript test framework to make the build reproducible. (#2727)
• Filed a pull request against vine, a Python promises library, to avoid non-determinstic default keyword argument appearing in the documentation. (#12)
• Filed an issue for the Redis key-value database addressing build failures on the MIPS architecture. (#3874)
• Submitted a bug report against xdotool a tool to automate window and keyboard interactions reporting a crash when searching after binding an action with behave. (#169)
• Reviewed a pull request from Dan Palmer for django-email-from-template, a library to send emails in Django generated entirely from the templating system, which intends to add an option to send mails upon transaction commit.

1. I downloaded a Raspbian Jessie Lite image
2. I put it on an SD card
3. I created an empty ssh file on the boot partition
4. I put the SD card on the Raspberry Pi and turned on the stereo.
5. ssh pi@raspberrypi password raspberry
6. sudo raspi-config to change the hostname, the password, and to enlarge the root partition to include all the rest of the space available in the SD card.

apt install mpd

bind_to_address         "any"

zeroconf_enabled                "yes"
zeroconf_name                   "stereo"

password                        "SECRET@read,add,control,admin"
default_permissions             "read,add,control"

audio_output  
        type            "alsa"
        name            "My ALSA Device"
        device          "hw:0,0"        # optional
        use_mmap        "yes"
 

#!/bin/sh
rsync -avz --filter=". sync-stereo.filter" --copy-links --prune-empty-dirs --delete ./ pi@stereo:/var/lib/mpd/music
ssh pi@stereo "chmod u=rwX,go=rX -R /var/lib/mpd/music"

hide /_archive
include */
include **.mp3
hide *

$ mpc -h stereo status
UltraCat - Unexpected Little Happenings
[playing] #15/22   0:03/4:06 (1%)
volume: 80%   repeat: off   random: on    single: off   consume: off

#!/usr/bin/r
library(tools)
.find_calls_in_package_code <- tools:::.find_calls_in_package_code
.read_description <- tools:::.read_description
## all what follows is from R-devel aka R 3.4.0 to be
package_ff_call_db <- function(dir)  
    ## A few packages such as CDM use base::.Call
    ff_call_names <- c(".C", ".Call", ".Fortran", ".External",
                       "base::.C", "base::.Call",
                       "base::.Fortran", "base::.External")
    predicate <- function(e)  
        (length(e) > 1L) &&
            !is.na(match(deparse(e[[1L]]), ff_call_names))
     
    calls <- .find_calls_in_package_code(dir,
                                         predicate = predicate,
                                         recursive = TRUE)
    calls <- unlist(Filter(length, calls))
    if(!length(calls)) return(NULL)
    attr(calls, "dir") <- dir
    calls
 
native_routine_registration_db_from_ff_call_db <- function(calls, dir = NULL, character_only = TRUE)  
    if(!length(calls)) return(NULL)
    ff_call_names <- c(".C", ".Call", ".Fortran", ".External")
    ff_call_args <- lapply(ff_call_names,
                           function(e) args(get(e, baseenv())))
    names(ff_call_args) <- ff_call_names
    ff_call_args_names <-
        lapply(lapply(ff_call_args,
                      function(e) names(formals(e))), setdiff,
               "...")
    if(is.null(dir))
        dir <- attr(calls, "dir")
    package <- # drop name
        as.vector(.read_description(file.path(dir, "DESCRIPTION"))["Package"])
    symbols <- character()
    nrdb <-
        lapply(calls,
               function(e)  
                   if (startsWith(deparse(e[[1L]]), "base::"))
                       e[[1L]] <- e[[1L]][3L]
                   ## First figure out whether ff calls had '...'.
                   pos <- which(unlist(Map(identical,
                                           lapply(e, as.character),
                                           "...")))
                   ## Then match the call with '...' dropped.
                   ## Note that only .NAME could be given by name or
                   ## positionally (the other ff interface named
                   ## arguments come after '...').
                   if(length(pos)) e <- e[-pos]
                   ## drop calls with only ...
                   if(length(e) < 2L) return(NULL)
                   cname <- as.character(e[[1L]])
                   ## The help says
                   ##
                   ## '.NAME' is always matched to the first argument
                   ## supplied (which should not be named).
                   ##
                   ## But some people do (Geneland ...).
                   nm <- names(e); nm[2L] <- ""; names(e) <- nm
                   e <- match.call(ff_call_args[[cname]], e)
                   ## Only keep ff calls where .NAME is character
                   ## or (optionally) a name.
                   s <- e[[".NAME"]]
                   if(is.name(s))  
                       s <- deparse(s)[1L]
                       if(character_only)  
                           symbols <<- c(symbols, s)
                           return(NULL)
                        
                     else if(is.character(s))  
                       s <- s[1L]
                     else   ## expressions
                       symbols <<- c(symbols, deparse(s))
                       return(NULL)
                    
                   ## Drop the ones where PACKAGE gives a different
                   ## package. Ignore those which are not char strings.
                   if(!is.null(p <- e[["PACKAGE"]]) &&
                      is.character(p) && !identical(p, package))
                       return(NULL)
                   n <- if(length(pos))  
                            ## Cannot determine the number of args: use
                            ## -1 which might be ok for .External().
                            -1L
                          else  
                            sum(is.na(match(names(e),
                                            ff_call_args_names[[cname]]))) - 1L
                         
                   ## Could perhaps also record whether 's' was a symbol
                   ## or a character string ...
                   cbind(cname, s, n)
                )
    nrdb <- do.call(rbind, nrdb)
    nrdb <- as.data.frame(unique(nrdb), stringsAsFactors = FALSE)
    if(NROW(nrdb) == 0L   length(nrdb) != 3L)
        stop("no native symbols were extracted")
    nrdb[, 3L] <- as.numeric(nrdb[, 3L])
    nrdb <- nrdb[order(nrdb[, 1L], nrdb[, 2L], nrdb[, 3L]), ]
    nms <- nrdb[, "s"]
    dups <- unique(nms[duplicated(nms)])
    ## Now get the namespace info for the package.
    info <- parseNamespaceFile(basename(dir), dirname(dir))
    ## Could have ff calls with symbols imported from other packages:
    ## try dropping these eventually.
    imports <- info$imports
    imports <- imports[lengths(imports) == 2L]
    imports <- unlist(lapply(imports,  [[ , 2L))
    info <- info$nativeRoutines[[package]]
    ## Adjust native routine names for explicit remapping or
    ## namespace .fixes.
    if(length(symnames <- info$symbolNames))  
        ind <- match(nrdb[, 2L], names(symnames), nomatch = 0L)
        nrdb[ind > 0L, 2L] <- symnames[ind]
      else if(!character_only &&
              any((fixes <- info$registrationFixes) != ""))  
        ## There are packages which have not used the fixes, e.g. utf8latex
        ## fixes[1L] is a prefix, fixes[2L] is an undocumented suffix
        nrdb[, 2L] <- sub(paste0("^", fixes[1L]), "", nrdb[, 2L])
        if(nzchar(fixes[2L]))
            nrdb[, 2L] <- sub(paste0(fixes[2L]), "$", "", nrdb[, 2L])
     
    ## See above.
    if(any(ind <- !is.na(match(nrdb[, 2L], imports))))
        nrdb <- nrdb[!ind, , drop = FALSE]
    ## Fortran entry points are mapped to l/case
    dotF <- nrdb$cname == ".Fortran"
    nrdb[dotF, "s"] <- tolower(nrdb[dotF, "s"])
    attr(nrdb, "package") <- package
    attr(nrdb, "duplicates") <- dups
    attr(nrdb, "symbols") <- unique(symbols)
    nrdb
 
format_native_routine_registration_db_for_skeleton <- function(nrdb, align = TRUE, include_declarations = FALSE)  
    if(!length(nrdb))
        return(character())
    fmt1 <- function(x, n)  
        c(if(align)  
              paste(format(sprintf("     \"%s\",", x[, 1L])),
                    format(sprintf(if(n == "Fortran")
                                       "(DL_FUNC) &F77_NAME(%s),"
                                   else
                                       "(DL_FUNC) &%s,",
                                   x[, 1L])),
                    format(sprintf("%d ,", x[, 2L]),
                           justify = "right"))
            else  
              sprintf(if(n == "Fortran")
                          "     \"%s\", (DL_FUNC) &F77_NAME(%s), %d ,"
                      else
                          "     \"%s\", (DL_FUNC) &%s, %d ,",
                      x[, 1L],
                      x[, 1L],
                      x[, 2L])
           ,
          "     NULL, NULL, 0 ")
     
    package <- attr(nrdb, "package")
    dups <- attr(nrdb, "duplicates")
    symbols <- attr(nrdb, "symbols")
    nrdb <- split(nrdb[, -1L, drop = FALSE],
                  factor(nrdb[, 1L],
                         levels =
                             c(".C", ".Call", ".Fortran", ".External")))
    has <- vapply(nrdb, NROW, 0L) > 0L
    nms <- names(nrdb)
    entries <- substring(nms, 2L)
    blocks <- Map(function(x, n)  
                      c(sprintf("static const R_%sMethodDef %sEntries[] =  ",
                                n, n),
                        fmt1(x, n),
                        " ;",
                        "")
                   ,
                  nrdb[has],
                  entries[has])
    decls <- c(
        "/* FIXME: ",
        "   Add declarations for the native routines registered below.",
        "*/")
    if(include_declarations)  
        decls <- c(
            "/* FIXME: ",
            "   Check these declarations against the C/Fortran source code.",
            "*/",
            if(NROW(y <- nrdb$.C))  
                 args <- sapply(y$n, function(n) if(n >= 0)
                                paste(rep("void *", n), collapse=", ")
                                else "/* FIXME */")
                c("", "/* .C calls */",
                  paste0("extern void ", y$s, "(", args, ");"))
            ,
            if(NROW(y <- nrdb$.Call))  
                args <- sapply(y$n, function(n) if(n >= 0)
                               paste(rep("SEXP", n), collapse=", ")
                               else "/* FIXME */")
               c("", "/* .Call calls */",
                  paste0("extern SEXP ", y$s, "(", args, ");"))
             ,
            if(NROW(y <- nrdb$.Fortran))  
                 args <- sapply(y$n, function(n) if(n >= 0)
                                paste(rep("void *", n), collapse=", ")
                                else "/* FIXME */")
                c("", "/* .Fortran calls */",
                  paste0("extern void F77_NAME(", y$s, ")(", args, ");"))
             ,
            if(NROW(y <- nrdb$.External))
                c("", "/* .External calls */",
                  paste0("extern SEXP ", y$s, "(SEXP);"))
            )
     
    headers <- if(NROW(nrdb$.Call)   NROW(nrdb$.External))
        c("#include <R.h>", "#include <Rinternals.h>")
    else if(NROW(nrdb$.Fortran)) "#include <R_ext/RS.h>"
    else character()
    c(headers,
      "#include <stdlib.h> // for NULL",
      "#include <R_ext/Rdynload.h>",
      "",
      if(length(symbols))  
          c("/*",
            "  The following symbols/expresssions for .NAME have been omitted",
            "", strwrap(symbols, indent = 4, exdent = 4), "",
            "  Most likely possible values need to be added below.",
            "*/", "")
       ,
      if(length(dups))  
          c("/*",
            "  The following name(s) appear with different usages",
            "  e.g., with different numbers of arguments:",
            "", strwrap(dups, indent = 4, exdent = 4), "",
            "  This needs to be resolved in the tables and any declarations.",
            "*/", "")
       ,
      decls,
      "",
      unlist(blocks, use.names = FALSE),
      ## We cannot use names with '.' in: WRE mentions replacing with "_"
      sprintf("void R_init_%s(DllInfo *dll)",
              gsub(".", "_", package, fixed = TRUE)),
      " ",
      sprintf("    R_registerRoutines(dll, %s);",
              paste0(ifelse(has,
                            paste0(entries, "Entries"),
                            "NULL"),
                     collapse = ", ")),
      "    R_useDynamicSymbols(dll, FALSE);",
      " ")
 
package_native_routine_registration_db <- function(dir, character_only = TRUE)  
    calls <- package_ff_call_db(dir)
    native_routine_registration_db_from_ff_call_db(calls, dir, character_only)
 
package_native_routine_registration_db <- function(dir, character_only = TRUE)  
    calls <- package_ff_call_db(dir)
    native_routine_registration_db_from_ff_call_db(calls, dir, character_only)
 
package_native_routine_registration_skeleton <- function(dir, con = stdout(), align = TRUE,
                                                         character_only = TRUE, include_declarations = TRUE)  
    nrdb <- package_native_routine_registration_db(dir, character_only)
    writeLines(format_native_routine_registration_db_for_skeleton(nrdb,
                align, include_declarations),
               con)
 
package_native_routine_registration_skeleton(".")  ## when R 3.4.0 is out you only need this line

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Filed under: Debian English Gammu 0 comments

Context Last month, Brian Ripley announced on r-devel that registration of routines would now be tested for by R CMD check in r-devel (which by next month will become R 3.4.0). A NOTE will be issued now, this will presumably turn into a WARNING at some point. Writing R Extensions has an updated introduction) of the topic. Package registration has long been available, and applies to all native (i.e. "compiled") function via the .C(), .Call(), .Fortran() or .External() interfaces. If you use any of those -- and .Call() may be the only truly relevant one here -- then is of interest to you. Brian Ripley and Kurt Hornik also added a new helper function: tools::package_native_routine_registration_skeleton(). It parses the R code of your package and collects all native function entrypoints in order to autogenerate the registration. It is available in R-devel now, will be in R 3.4.0 and makes adding such registration truly trivial. But as of today, it requires that you have R-devel. Once R 3.4.0 is out, you can call the helper directly. As for R-devel, there have always been at least two ways to use it: build it locally (which we may cover in another R⁴ installment), or using Docker. Here will focus on the latter by relying on the Rocker project by Carl and myself.

Use the Rocker drd Image We assume you can run Docker on your system. How to add it on Windows, macOS or Linux is beyond our scope here today, but also covered extensively elsewhere. So we assume you can execute docker and e.g. bring in the 'daily r-devel' image drd from our Rocker project via

~$ docker pull rocker/drd

With that, we can use R-devel to create the registration file very easily in a single call (which is a long command-line we have broken up with one line-break for the display below). The following is real-life example when I needed to add registration to the RcppTOML package for this week's update:

~/git/rcpptoml(master)$ docker run --rm -ti -v $(pwd):/mnt rocker/drd \  ## line break
             RD --slave -e 'tools::package_native_routine_registration_skeleton("/mnt")'
#include <R.h>
#include <Rinternals.h>
#include <stdlib.h> // for NULL
#include <R_ext/Rdynload.h>
/* FIXME: 
   Check these declarations against the C/Fortran source code.
*/
/* .Call calls */
extern SEXP RcppTOML_tomlparseImpl(SEXP, SEXP, SEXP);
static const R_CallMethodDef CallEntries[] =  
     "RcppTOML_tomlparseImpl", (DL_FUNC) &RcppTOML_tomlparseImpl, 3 ,
     NULL, NULL, 0 
 ;
void R_init_RcppTOML(DllInfo *dll)
 
    R_registerRoutines(dll, NULL, CallEntries, NULL, NULL);
    R_useDynamicSymbols(dll, FALSE);
 
edd@max:~/git/rcpptoml(master)$ 

Decompose the Command We can understand the docker command invocation above through its components:

• docker run is the basic call to a container
• --rm -ti does subsequent cleanup (--rm) and gives a terminal (-t) that is interactive (-i)
• -v $(pwd):/mnt uses the -v a:b options to make local directory a available as b in the container; here $(pwd) calls print working directory to get the local directory which is now mapped to /mnt in the container
• rocker/drd invokes the 'drd' container of the Rocker project
• RD is a shorthand to the R-devel binary inside the container, and the main reason we use this container
• -e 'tools::package_native_routine_registration_skeleton("/mnt") calls the helper function of R (currently in R-devel only, so we use RD) to compute a possible init.c file based on the current directory -- which is /mnt inside the container

That it. We get a call to the R function executed inside the Docker container, examining the package in the working directory and creating a registration file for it which is display to the console.

Copy the Output to src/init.c We simply copy the output to a file src/init.c; I often fold one opening brace one line up.

Update NAMESPACE We also change one line in NAMESPACE from (for this package) useDynLib("RcppTOML") to useDynLib("RcppTOML", .registration=TRUE). Adjust accordingly for other package names.

That's it! And with that we a have a package which no longer provokes the NOTE as seen by the checks page. Calls to native routines are now safer (less of a chance for name clashing), get called quicker as we skip the symbol search (see the WRE discussion) and best of all this applies to all native routines whether written by hand or written via a generator such as Rcpp Attributes. So give this a try to get your package up-to-date.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.