What happened in the reproducible builds effort between April 3rd and April 9th 2016: Media coverage Emily Ratliff wrote an article for SecurityWeek called Establishing Correspondence Between an Application and its Source Code - How Combining Two Completely Separate Open Source Projects Can Make Us All More Secure. Tails have started work on a design for freezable APT repositories to make it easier and practical to perform reproductions of an entire distribution at a given point in time, which will be needed to create reproducible installation- or live-media. Toolchain fixes Alexis Bienven e submitted patches adding support for SOURCE_DATE_EPOCH in several tools: transfig, imagemagick, rdtool, and asciidoctor. boyska submitted one for python-reportlab. Packages fixed The following packages have become reproducible due to changes in their build dependencies: atinject-jsr330 brailleutils cglib3 gnugo libcobra-java libgnumail-java libjchart2d-java libjcommon-java libjfreechart-java libjide-oss-java liblaf-widget-java liblastfm-java liboptions-java octave-control octave-mpi octave-nan octave-parallel octave-stk octave-struct octave-tsa oar The following packages became reproducible after getting fixed:

• apt-listchanges/2.88 by Robert Luberda.
• cgal/4.8-1 by Joachim Reichel.
• erlang-bitcask/2.0.2+dfsg-1 by Nobuhiro Iwamatsu.
• geneweb/6.08+git20160228+dfsg-2 by Guillaume Brochu.
• glance/2:12.0.0~rc2-1 uploaded by Thomas Goirand, original patch by Chris Lamb.
• gle-graphics/4.2.5-4 by Christian T. Steigies.
• maude uploaded by Andreas Tille, patch by Alexis Bienven e.
• psqlodbc/1:09.05.0100-3 by Christoph Berg.
• resource-agents/1:3.9.7-3 by Christoph Berg.
• vgabios/0.7a-6 by Reiner Herrmann.
• vim/2:7.4.1689-2 by James McCoy.
• xmhtml/1.1.10-2 by Graham Inggs with Reiner Herrmann's help.
• xscreensaver/5.34-2 uploaded by Tormod Volden, original patch by Sascha Steinbiss.

Several uploads fixed some reproducibility issues, but not all of them:

• rkward/0.6.5-1 uploaded by Thomas Friedrichsmeier, original patch by Philip Rinn.
• mailfilter/0.8.4-1 uploaded by Elimar Riesebieter, original patch by Chris Lamb.
• bind9/1:9.10.3.dfsg.P4-6 uploaded by Michael Gilbert, original patch by Reiner Herrmann.
• bzr/2.7.0- 3,4 by Jelmer Vernooij.
• samba/2:4.3.6+dfsg-2 uploaded by Mathieu Parent, fix by Jelmer Vernooij.
• fwupdate/0.5-3 by Mario Limonciello.
• paraview/5.0.1+dfsg1-1 by Anton Gladky.

Patches submitted which have not made their way to the archive yet:

• #819883 on debootstrap by Reiner Herrmann: tell tar to sort the archive members.
• #819885 on chktex by Sascha Steinbiss: use the time of latest debian/changelog entry as documentation timestamp.
• #819915 on kannel by Alexis Bienven e: use the time of latest debian/changelog entry as documentation timestamp.
• #819921 on basket by Alexis Bienven e: remove build date from debug info.
• #819965 on openarena-data by Alexandre Detiste: normalize file permissions before creating .pk3 archive.
• #820016 on gabedit by Alexis Bienven e: sort object files used to build the executable.
• #820032 on bibledit-gtk by Alexis Bienven e: remove useless included Makefile.
• #820072 on synfig by Alexis Bienven e: remove build date from info output.
• #820148 on autopkgtest by Alexis Bienven e: fix install order to cope with locales with case insensitive globbing.
• #820152 on anope by Alexis Bienven e: remove build date from the version string.
• #820179 on aodh by Alexis Bienven e: remove build date from the documentation.
• #820183 on cython by Alexis Bienven e: add support SOURCE_DATE_EPOCH.
• #820194 on nasm by rain1: sorts keys when traversing hash tables used to build the documentation.
• #820226 on chrony by Alexis Bienven e: add support for SOURCE_DATE_EPOCH to preset the ntp_era_split parameter.
• #820457 on recode by Alexis Bienven e: use system help2man.
• #820522 on gtkspell by Alexis Bienven e: force shell to /bin/sh in example Makefile.

Other upstream fixes Alexander Batischev made a commit to make newsbeuter reproducible. tests.reproducible-builds.org

• An architecture agnostic summary has been added to the reproducible-tracker.json by Valerie Young to make it easy to parse whether a package is unreproducible anywhere.
• To find more reproducibility issues a new variation was added to the i386 builders, so that one build is done using a 32 bit kernel (686-PAE) and the other build is using a 64 bit kernel (amd64). (h01ger)
  • Niko Tyni was the first to notice a bug due to this: #821182 perl: embeds kernel architecture information
• The 2nd builds are now done in fr_CH on amd64, de_CH on i386 and it_CH on armhf. (h01ger)
• The variation table has been updated to reflect the recent changes and various small bugs have been fixed. (h01ger)

Package reviews 93 reviews have been removed, 66 added and 21 updated in the previous week. 12 new FTBFS bugs have been reported by Chris Lamb and Niko Tyni. Misc. This week's edition was written by Lunar, Holger Levsen, Reiner Herrmann, Mattia Rizzolo and Ximin Luo. With the departure of Lunar as a full-time contributor, Reproducible Builds Weekly News (this thing you're reading) has moved from his personal Debian blog on Debian People to the Reproducible Builds team web site on Debian Alioth. You may want to update your RSS or Atom feeds. Very many thanks to Lunar for writing and publishing this weekly news for so long, well & continously!

On my Debian machines I run stunnel to create an secure connection to my e-mail provider s SMTP gateway. Postfix sends mail through that TLS tunnel. Recently I stopped receiving e-mail from rss2email and I discovered tonight that the reason was that the tunnel has caved in on the machine which rss2email was running on. Unfortunately, some mail was permanently discarded from the postfix queue because it turns out that postfix will by default keep mail in the queue for a maximum of only 5 days. Since the connection to the gateway was down, postfix couldn t return the mail to its sender (i.e. me). Fortunately, I m not smart enough to have any log rotation going on, so I could easily find the message that were lost:

grep "status=expired, returned to sender" /var/log/mail.log \
      awk ' print $6 ' \
      while read id; do grep "$id" -m1 /var/log/mail.log; done

The first grep determines the queue id of the messages that were expired, and then the second grep finds the first entry in the mail log for that message, which provides the time the message was sent. Replacing -m1 with -m4 gave me the message-id of the messages and the intended recipient of the messages. This allowed me to restore them from backups or bounce them from my sent mail folder for those that I tried to send myself. To prevent this from happening again, I ve extended the maximum lifetime of messages in the queue from 5 days to 10:

postconf -e maximal_queue_lifetime=10d

I ve incorporated a check for clogged mail queues on my machines into my weekly backup routine.

I recently had to sign someone s GPG key. I ve long used the caff tool from the signing-party package to help me with this. Unfortunately, I m using a new laptop and hadn t yet configured caff on it. Moreover, caff uses the system MTA by default, normally found at /usr/sbin/sendmail, and I hadn t yet properly configured it to send mail to the outside world. Since I have multiple email accounts and use mutt as my mail client, I use msmtp as my SMTP client / sendmail drop-in. This post describes how to configure caff to use msmtp. Reading the caff man page, one sees the following tantalising hint, which leads you to believe it s completely trivial to specify your own MTA: Unfortunately, substituting in a custom path, e.g., /home/rak/bin/msmtp/msmtp-default, for sendmail in the above line results in a long sequence of errors when caff tries to mail the signed keys: Fortunately, it s still straightforward to accomplish after reading the perldoc for Mail::Mailer. In my case, it was sufficient to add the following to my ~/.caffrc: To specify alternate paths to your sendmail-style MTA, simply modify the value of $ENV 'PERL_MAILERS' after the initial sendmail: bit. If you re using stock msmtp, I believe the following pair of lines will work, though I haven t tested it: If there s a more elegant way to accomplish this, I d be happy to hear about it!

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In February, 111.75 work hours have been dispatched among 10 paid contributors. Their reports are available:

• Antoine Beaupr did 8h.
• Ben Hutchings did 12.75 hours (out of 11 hours allocated + 7.25 extra hours remaining, meaning that he still has 5.50 extra hours to do over April).
• Brian May did 10 hours.
• Chris Lamb did 7 hours (instead of the 14.25 hours he was allocated +, compensating the extra hours he did last month).
• Damyan Ivanov did nothing out of the 7.25 remaining hours he had, he opted to give them back and come back to LTS work later.
• Guido G nther did 13 hours (out of 12 hours allocated + 4.25 remaining hours, leaving 3.25 extra hours for April).
• Markus Koschany did 14.25 hours.
• Mike Gabriel did nothing and opted to give back the 8 hours allocated. He will stop LTS work for now as he has other projects taking all his time.
• Santiago Ruano Rinc n did 10 hours (out of 12h allocated + 1.50 remaining, thus keeping 3.50 extra hours for April).
• Scott Kitterman did a few hours but was not able to provide his report in time due to sickness. His next report will cover two months.
• Thorsten Alteholz did 14.25 hours.

Evolution of the situation The number of sponsored hours started to increase for April (116.75 hours, thanks to Sonus Networks) and should increase even further for May (with a new Gold sponsor currently joining us, Babiel GmbH). Hopefully the trend will continue so that we can reach our objective of funding the equivalent of a full-time position. At the end of the month the LTS team will be fully responsible of all Debian 7 Wheezy updates. For now paid contributors are still helping the security team by fixing packages that were fixed in squeeze already but that are still outstanding in wheezy. They are also looking for ways to ensure that some of the most complicated packages can be supported over the wheezy LTS timeframe. It is likely that we will seek external help (possibly from credativ which is already handling support of PostgreSQL) for the maintenance of Xen and that some other packages (like libav, vlc, maybe qemu?) will be upgraded to newer versions which are still maintained (either upstream or in Debian Jessie by the Debian maintainers). Thanks to our sponsors New sponsors are in bold.

• Platinum sponsors:
• TOSHIBA (for 6 months)
• Gold sponsors:
• The Positive Internet (for 22 months)
• Blablacar (for 21 months)
• Linode LLC (for 11 months)
• Silver sponsors:
• Domeneshop AS (for 21 months)
• Universit Lille 3 (for 21 months)
• Trollweb Solutions (for 19 months)
• Nantes M tropole (for 15 months)
• University of Luxembourg (for 13 months)
• Dalenys (for 11 months)
• Univention GmbH (for 7 months)
• Universit Jean Monnet de St Etienne (for 7 months)
• Sonus Networks
• Bronze sponsors:
• David Ayers IntarS Austria (for 22 months)
• Offensive Security (for 22 months)
• Seznam.cz, a.s. (for 22 months)
• Evolix (for 21 months)
• Freeside Internet Service (for 21 months)
• MyTux (for 21 months)
• Linuxhotel GmbH (for 19 months)
• Intevation GmbH (for 18 months)
• Daevel SARL (for 17 months)
• Bitfolk LTD (for 16 months)
• Megaspace Internet Services GmbH (for 16 months)
• Greenbone Networks GmbH (for 15 months)
• NUMLOG (for 15 months)
• WinGo AG (for 14 months)
• Ecole Centrale de Nantes LHEEA (for 11 months)
• Sig-I/O (for 8 months)
• Entr ouvert (for 6 months)
• Adfinis SyGroup AG (for 3 months)

No comment Liked this article? Click here. My blog is Flattr-enabled.

(Edit to add: this issue is restricted to the mobile SKUs. Desktop parts have very different power management behaviour)

Linux 4.5 seems to have got Intel's Skylake platform (ie, 6th-generation Core CPUs) to the point where graphics work pretty reliably, which is great progress (4.4 tended to lose all my windows every so often, especially over suspend/resume). I'm even running Wayland happily. Unfortunately one of the reasons I have a laptop is that I want to be able to do things like use it on battery, and power consumption's an important part of that. Skylake continues the trend from Haswell of moving to an SoC-type model where clock and power domains are shared between components that were previously entirely independent, and so you can't enter deep power saving states unless multiple components all have the correct power management configuration. On Haswell/Broadwell this manifested in the form of Serial ATA link power management being involved in preventing the package from going into deep power saving states - setting that up correctly resulted in a reduction in full-system power consumption of about 40%[1].

I've now got a Skylake platform with a nice shiny NVMe device, so Serial ATA policy isn't relevant (the platform doesn't even expose a SATA controller). The deepest power saving state I can get into is PC3, despite Skylake supporting PC8 - so I'm probably consuming about 40% more power than I should be. And nobody seems to know what needs to be done to fix this. I've found no public documentation on the power management dependencies on Skylake. Turning on everything in Powertop doesn't improve anything. My battery life is pretty poor and the system is pretty warm.

The best thing about this is the following statement from page 64 of the 6th Generation Intel Processor Datasheet for U-Platforms:

Caution: Long term reliability cannot be assured unless all the Low-Power Idle States are enabled.

which is pretty concerning. Without support for states deeper than PC3, Linux is running in a configuration that Intel imply may trigger premature failure. That's obviously not good. Until this situation is improved, you probably shouldn't buy any Skylake systems if you're planning on running Linux.

[1] These patches never went upstream. Someone reported that they resulted in their SSD throwing errors and I couldn't find anybody with deeper levels of SATA experience who was interested in working on the problem. Intel's AHCI drivers for Windows do the right thing, but I couldn't find anybody at Intel who could get any information from their Windows driver team.

comments

That s a longer title than I d like, but I want to try and catch the attention of anyone who might have missed more directed notifications about this. If you re not an SPI contributing member there s probably nothing to see here Although I decided not to stand for re-election at the Software in the Public Interest (SPI) board elections last July, I haven t stopped my involvement with the organisation. In particular I ve spent some time working on an overhaul of the members website and rolling it out. One of the things this has enabled is implementation of 2009-11-04.jmd.1: Contributing membership expiry, by tracking activity in elections and providing an easy way for a member to indicate they consider themselves active even if they haven t voted. The plan is that this will run at some point after the completion of every board election. A first pass of cleanups was completed nearly a month ago, contacting all contributing members who d never been seen to vote and asking them to update their status if they were still active. A second round, of people who didn t vote in the last board election (in 2014), is currently under way. Affected members will have been emailed directly and there was a mail to spi-announce, but I m aware people often overlook these things or filter mail off somewhere that doesn t get read often. If you are an SPI Contributing member who considers themselves an active member I strongly recommend you login to the SPI Members Website and check the Last active date displayed is after 2014-07-14 (i.e. post the start of the last board election). If it s not, click on the Update link beside the date. The updated date will be shown once you ve done so. Why does pruning inactive members matter? The 2015 X.Org election results provide at least one indication of why ensuring you have an engaged membership is important - they failed to make a by-laws change that a vast majority of votes were in favour of, due to failing to make quorum. (If you re an X.org member, go vote!)

That s a longer title than I d like, but I want to try and catch the attention of anyone who might have missed more directed notifications about this. If you re not an SPI contributing member there s probably nothing to see here Although I decided not to stand for re-election at the Software in the Public Interest (SPI) board elections last July, I haven t stopped my involvement with the organisation. In particular I ve spent some time working on an overhaul of the members website and rolling it out. One of the things this has enabled is implementation of 2009-11-04.jmd.1: Contributing membership expiry, by tracking activity in elections and providing an easy way for a member to indicate they consider themselves active even if they haven t voted. The plan is that this will run at some point after the completion of every board election. A first pass of cleanups was completed nearly a month ago, contacting all contributing members who d never been seen to vote and asking them to update their status if they were still active. A second round, of people who didn t vote in the last board election (in 2014), is currently under way. Affected members will have been emailed directly and there was a mail to spi-announce, but I m aware people often overlook these things or filter mail off somewhere that doesn t get read often. If you are an SPI Contributing member who considers themselves an active member I strongly recommend you login to the SPI Members Website and check the Last active date displayed is after 2014-07-14 (i.e. post the start of the last board election). If it s not, click on the Update link beside the date. The updated date will be shown once you ve done so. Why does pruning inactive members matter? The 2015 X.Org election results provide at least one indication of why ensuring you have an engaged membership is important - they failed to make a by-laws change that a vast majority of votes were in favour of, due to failing to make quorum. (If you re an X.org member, go vote!)

Let's see if I can scrounge through all of my now-organized directories of ebooks and figure out what I haven't recorded here yet. At least the paper books make that relatively easy, since I don't shelve them until I post them. (Yeah, yeah, I should actually make a database.) Hugh Aldersey-Williams Periodic Tales (nonfiction)
Sandra Ulbrich Almazan SF Women A-Z (nonfiction)
Radley Balko Rise of the Warrior Cop (nonfiction)
Peter V. Brett The Warded Man (sff)
Lois McMaster Bujold Gentleman Jole and the Red Queen (sff)
Fred Clark The Anti-Christ Handbook Vol. 2 (nonfiction)
Dave Duncan West of January (sff)
Karl Fogel Producing Open Source Software (nonfiction)
Philip Gourevitch We Wish to Inform You That Tomorrow We Will Be Killed With Our Families (nonfiction)
Andrew Groen Empires of EVE (nonfiction)
John Harris @ Play (nonfiction)
David Hellman & Tevis Thompson Second Quest (graphic novel)
M.C.A. Hogarth Earthrise (sff)
S.L. Huang An Examination of Collegial Dynamics... (sff)
S.L. Huang & Kurt Hunt Up and Coming (sff anthology)
Kameron Hurley Infidel (sff)
Kevin Jackson-Mead & J. Robinson Wheeler IF Theory Reader (nonfiction)
Rosemary Kirstein The Lost Steersman (sff)
Rosemary Kirstein The Language of Power (sff)
Merritt Kopas Videogames for Humans (nonfiction)
Alisa Krasnostein & Alexandra Pierce (ed.) Letters to Tiptree (nonfiction)
Mathew Kumar Exp. Negatives (nonfiction)
Ken Liu The Grace of Kings (sff)
Susan MacGregor The Tattooed Witch (sff)
Helen Marshall Gifts for the One Who Comes After (sff collection)
Jack McDevitt Coming Home (sff)
Seanan McGuire A Red-Rose Chain (sff)
Seanan McGuire Velveteen vs. The Multiverse (sff)
Seanan McGuire The Winter Long (sff)
Marc Miller Agent of the Imperium (sff)
Randal Munroe Thing Explainer (graphic nonfiction)
Marguerite Reed Archangel (sff)
J.K. Rowling Harry Potter: The Complete Collection (sff)
K.J. Russell Tides of Possibility (sff anthology)
Robert J. Sawyer Starplex (sff)
Bruce Schneier Secrets & Lies (nonfiction)
Mike Selinker (ed.) The Kobold Game to Board Game Design (nonfiction)
Douglas Smith Chimerascope (sff collection)
Jonathan Strahan Fearsome Journeys (sff anthology)
Nick Suttner Shadow of the Colossus (nonfiction)
Aaron Swartz The Boy Who Could Change the World (essays)
Caitlin Sweet The Pattern Scars (sff)
John Szczepaniak The Untold History of Japanese Game Developers I (nonfiction)
John Szczepaniak The Untold History of Japanese Game Developers II (nonfiction)
Jeffrey Toobin The Run of His Life (nonfiction)
Hayden Trenholm Blood and Water (sff anthology)
Coen Teulings & Richard Baldwin (ed.) Secular Stagnation (nonfiction)
Ursula Vernon Book of the Wombat 2015 (graphic nonfiction)
Ursula Vernon Digger (graphic novel) Phew, that was a ton of stuff. A bunch of these were from two large StoryBundle bundles, which is a great source of cheap DRM-free ebooks, although still rather hit and miss. There's a lot of just fairly random stuff that's been accumulating for a while, even though I've not had a chance to read very much. Vacation upcoming, which will be a nice time to catch up on reading.

My previous blog post was about fuzzing and finding segfaults in GNU Awk. At the time of this update they still remain unfixed. Reading about a new release of mutt I've seen a lot of complaints about how it handles HTML mail, by shelling out to lynx or w3m. As I have a vested interest in console based mail-clients I wanted to have a quick check to see how dangerous that could be. After all it wasn't so long ago that I discovered that printing a fingerprint of an SSH key could be dangerous, so the idea of parsing untrusted HTML is something I could see. In fact back in 2005 I reported that some specific HTML could crash Mozilla's firefox. Due to some ordering issues my Firefox bug was eventually reported as a duplicate, and although it seemed to qualify for the Mozilla bug-bounty and a CVE assignment I never received any actual cash. Shame. I'd have been more interested in testing the browser if I had a cheque to hang on my wall (and never cash). Anyway full-circle. Fuzzing the w3m console-based browser resulted in a bunch of segfaults when running this:

 w3m -dump $file.html

Anyway each of the two bugs I reported were fixed in a day or two, and both involved gnarly UTF-8/encoding transformations. Many thanks to Tatsuya Kinoshita for such prompt attention and excellent debugging skills. And lynx? Still no segfaults. I'll leave the fuzzer running over the weekend and if there are no faults found by Monday I guess I'll move on to links.

Also titled minimal images - maximal effort! The OpenStack Infrastructure Team manages a large continuous-integration system that provides the broad range of testing the OpenStack project requires. Tests are run thousands of times a day across every project, on multiple platforms and on multiple cloud-providers. There are essentially no manual steps in any part of the process, with every component being automated via scripting, a few home-grown tools and liberal doses of Puppet and Ansible. More importantly, every component resides in the public git trees right alongside every other OpenStack project, with contributions actively encouraged. As with any large system, technical debt can build up and start to affect stability and long-term maintainability. OpenStack Infrastructure can see some of this debt accumulating as more testing environments across more cloud-providers are being added to support ever-growing testing demands. Thus a strong focus of recent work has been consolidating testing platforms to be smaller, better defined and more maintainable. This post illustrates some of the background to the issues and describes how these new platforms are more reliable and maintainable.

There's a piece of software called XScreenSaver. It attempts to fill two somewhat disparate roles:

• Provide a functioning screen lock on systems using the X11 windowing system, a job made incredibly difficult due to a variety of design misfeatures in said windowing system[1]
• Provide cute graphical output while the screen is locked

XScreenSaver does an excellent job of the second of these[2] and is pretty good at the first, which is to say that it only suffers from a disastrous security flaw once very few years and as such is certainly not appreciably worse than any other piece of software.

Debian ships an operating system that prides itself on stability. The Debian definition of stability is a very specific one - rather than referring to how often the software crashes or misbehaves, it refers to how often the software changes behaviour. Debian is very reluctant to upgrade software that is part of a stable release, to the extent that developers will attempt to backport individual security fixes to the version they shipped rather than upgrading to a release that contains all those security fixes but also adds a new feature. The argument here is that the new release may also introduce new bugs, and Debian's users desire stability (in the "things don't change" sense) more than new features. Backporting security fixes keeps them safe without compromising the reason they're running Debian in the first place.

This all makes plenty of sense at a theoretical level, but reality is sometimes less convenient. The first problem is that security bugs are typically also, well, bugs. They may make your software crash or misbehave in annoying but apparently harmless ways. And when you fix that bug you've also fixed a security bug, but the ability to determine whether a bug is a security bug or not is one that involves deep magic and a fanatical devotion to the cause so given the choice between maybe asking for a CVE and dealing with embargoes and all that crap when perhaps you've actually only fixed a bug that makes the letter "E" appear in places it shouldn't and not one that allows the complete destruction of your intergalactic invasion fleet means people will tend to err on the side of "Eh fuckit" and go drinking instead. So new versions of software will often fix security vulnerabilities without there being any indication that they do so[3], and running old versions probably means you have a bunch of security issues that nobody will ever do anything about.

But that's broadly a technical problem and one we can apply various metrics to, and if somebody wanted to spend enough time performing careful analysis of software we could have actual numbers to figure out whether the better security approach is to upgrade or to backport fixes. Conversations become boring once we introduce too many numbers, so let's ignore that problem and go onto the second, which is far more handwavy and social and so significantly more interesting.

The second problem is that upstream developers remain associated with the software shipped by Debian. Even though Debian includes a tool for reporting bugs against packages included in Debian, some users will ignore that and go straight to the upstream developers. Those upstream developers then have to spend at least 15 or so seconds telling the user that the bug they're seeing has been fixed for some time, and then figure out how to explain that no sorry they can't make Debian include a fixed version because that's not how things work. Worst case, the stable release of Debian ends up including a bug that makes software just basically not work at all and everybody who uses it assumes that the upstream author is brutally incompetent, and they end up quitting the software industry and I don't know running a nightclub or something.

From the Debian side of things, the straightforward solution is to make it more obvious that users should file bugs with Debian and not bother the upstream authors. This doesn't solve the problem of damaged reputation, and nor does it entirely solve the problem of users contacting upstream developers. If a bug is filed with Debian and doesn't get fixed in a timely manner, it's hardly surprising that users will end up going upstream. The Debian bugs list for XScreenSaver does not make terribly attractive reading.

So, coming back to the title for this entry. The most obvious failure of the commons is where a basically malicious actor consumes while giving nothing back, but if an actor with good intentions ends up consuming more than they contribute that may still be a problem. An upstream author releases a piece of software under a free license. Debian distributes this to users. Debian's policies result in the upstream author having to do more work. What does the upstream author get out of this exchange? In an ideal world, plenty. The author's software is made available to more people. A larger set of developers is willing to work on making improvements to the software. In a less ideal world, rather less. The author has to deal with bug mail about already fixed bugs. The author's reputation may be harmed by user exposure to said fixed bugs. The author may get less in the way of useful bug fixes or features because people are running old versions rather than fixing new ones. If the balance tips towards the latter, the author's decision to release their software under a free license has made their life more difficult.

Most discussions about Debian's policies entirely ignore the latter scenario, focusing more on the fact that the author chose to release their software under a free license to begin with. If the author is unwilling to handle the consequences of that, goes the argument, why did they do it in the first place? The unfortunate logical conclusion to that argument is that the author realises that they made a huge mistake and never does so again, and woo uh oops.

The irony here is that one of Debian's foundational documents, the Debian Free Software Guidelines, makes allowances for this. Section 4 allows for distribution of software in Debian even if the author insists that modified versions[4] are renamed. This allows for an author to make a choice - allow themselves to be associated with the Debian version of their work and increase (a) their userbase and (b) their support load, or try to distinguish what Debian ship from their identity. But that document was ratified in 1997 and people haven't really spent much time since then thinking about why it says what it does, and so this tradeoff is rarely considered.

Free software doesn't benefit from distributions antagonising their upstreams, even if said upstream is a cranky nightclub owner. Debian's users are Debian's highest priority, but those users are going to suffer if developers decide that not using free licenses improves their quality of life. Kneejerk reactions around specific instances aren't helpful, but now is probably a good time to start thinking about what value Debian bring to its upstream authors and how that can be increased. Failing to do so doesn't serve users, Debian itself or the free software community as a whole.

[1] The X server has no fundamental concept of a screen lock. This is implemented by an application asking that the X server send all keyboard and mouse input to it rather than to any other application, and then that application creating a window that fills the screen. Due to some hilarious design decisions, opening a pop-up menu in an application prevents any other application from being able to grab input and so it is impossible for the screensaver to activate if you open a menu and then walk away from your computer. This is merely the most obvious problem - there are others that are more subtle and more infuriating. The only fix in this case is to nuke the site from orbit.

[2] There's screenshots here. My favourites are the one that emulate the electrical characteristics of an old CRT in order to present a more realistic depiction of the output of an Apple 2 and the one that includes a complete 6502 emulator.

[3] And obviously new versions of software will often also introduce new security vulnerabilities without there being any indication that they do so, because who would ever put that in their changelog. But the less ethically challenged members of the security community are more likely to be looking at new versions of software than ones released three years ago, so you're probably still tending towards winning overall

[4] There's a perfectly reasonable argument that all packages distributed by Debian are modified in some way

comments

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. I skipped my monthly report last time so this one will cover two months. I will try to list only the most important things to not make it too long. The Debian Handbook I worked with Ryuunosuke Ayanokouzi to prepare a paperback version of the Japanese translation of my book. Thanks to the efforts of everybody, it s now available. Unfortunately, Lulu declined to take it in distribution program so it won t be available on traditional bookstores (like Amazon, etc.). The reason is that they do not support non-latin character sets in the meta-data. I tried to cheat a little bit by inputting the description in English (still explaining that the book was in Japanese) but they rejected it nevertheless because the English title could mislead people. So the paperback is only available on lulu.com. Fortunately, the shipping costs are reasonable if you pick the most economic offer. Following this I invited the Italian, Spanish and Brazilian Portuguese translators to complete the work (they were close will all the strings already translated, mainly missing translated screenshots and some backcover content) so that we can also release paperback versions in those languages. It s getting close to completion for them. Hopefully we will have those available until next month. Distro Tracker In early February, I tweaked the configuration to send (by email) exceptions generated by incoming mails and by routine task. Before this they were logged but I did not take the time to look into them. This quickly brought a few issues into light and I fixed them as they appeared: for instance the bounce handling code was getting confused when the character case was not respected, and it appears that some emails come back to us after having been lowercased. Also the code was broken when the References field used more than one line on incoming control emails. This brought into light a whole class of problems with the database storing twice the same email with only differing case. So I did further work to merge all those duplicate entries behind a single email entry. Later, the experimental Sources files changed and I had to tweak the code to work with the removal of the Files field (relying instead on Checksums-* to find out the various files part of the entry). At some point, I also fixed the login form to not generate an exception when the user submits an empty form. I also decided that I no longer wanted to support Django 1.7 in distro tracker as Django 1.8 is the current LTS version. I asked the Debian system administrators to update the package on tracker.debian.org with the version in jessie-backports. This allowed me to fix a few deprecation warnings that I kept triggering because I wanted the code to work with Django 1.7. One of those warnings was generated by django-jsonfield though and I could not fix it immediately. Instead I prepared a pull request that I submitted to the upstream author. Oh, and a last thing, I tweaked the CSS to densify the layout on the package page. This was one of the most requested changes from the people who were still preferring packages.qa.debian.org over tracker.debian.org. Kali and new pkg-security team As part of my Kali work, I have been fixing RC bugs in Debian packages that we use in Kali. But in many cases, I stumbled upon packages whose maintainers were really missing in action (MIA). Up to now, we were only doing non-maintainers upload (NMU) but I want to be able to maintain those packages more effectively so we created a new pkg-security team (we re only two right now and we have no documentation yet, but if you want to join, you re welcome, in particular if you maintain a package which is useful in the security field). arm64 work. The first 3 packages that we took over (ssldump, sucrack, xprobe) are actually packages that were missing arm64 builds. We just started our arm64 port on Kali and we fixed them for that architecture. Since they were no longer properly maintained, in most cases it was just a matter of using dh_autoreconf to get up-to-date config. sub,guess files. We still miss a few packages on arm64: vboot-utils (that we will likely take over soon since it s offered for adoption), ruby-libv8 and ruby-therubyracer, ntopng (we have to wait a new luajit which is only in experimental right now). We also noticed that dh-make-golang was not available on arm64, after some discussion on #debian-buildd, I filed two bugs for this: #819472 on dh-make-golang and #819473 on dh-golang. RC bug fixing. hdparm was affected by multiple RC bugs and the release managers were trying to get rid of it from testing. This removed multiple packages that were used by Kali and its users. So I investigated the situation of that package, convinced the current maintainers to orphan it, asked for new maintainers on debian-devel, reviewed multiple updates prepared by the new volunteers and sponsored their work. Now hdparm is again RC-bug free and has the latest upstream version. We also updated jsonpickle to 0.9.3-1 to fix RC bug #812114 (that I forwarded upstream first). Systemd presets support in init-system-helpers. I tried to find someone (to hire) to implement the system preset feature I requested in #772555 but I failed. Still Andreas Henriksson was kind enough to give it a try and sent a first patch. I tried it and found some issues so I continued to improve it and simplify it I submitted an updated patch and pinged Martin Pitt. He pointed me to the DEP-8 test failures that my patch was creating. I quickly fixed those afterwards. This patch is in use in Kali and lets us disable network services by default. I would like to see it merged in Debian so that everybody can setup systemd preset file and have their desire respected at installation time. Misc bug reports. I filed #813801 to request a new upstream release of kismet. Same for masscan in #816644 and for wkhtmltopdf in #816714. We packaged (before Debian) a new upstream release of ruby-msgpack and found out that it was not building on armel/armhf so we filed two upstream tickets (with a suggested fix). In #814805, we asked the pyscard maintainer to reinstate python-pyscard that was dropped (keeping only the Python3 version) as we use the Python 2 version in Kali. And there s more: I filed #816553 (segfault) and #816554 against cdebootstrap. I asked for dh-python to have a better behaviour after having being bitten by the fact that dh with python3 was not doing what I expected it to do (see #818175). And I reported #818907 against live-build since it is failing to handle a package whose name contains an upper case character (it s not policy compliant but dpkg supports them). Misc packaging I uploaded Django 1.9.2 to unstable and 1.8.9 to jessie-backports. I provided the supplementary information that Julien Cristau asked me in #807654 but despite this, this jessie update has been ignored for the second point release in a row. It is now outdated until I update it to include the security fixes that have been released in the mean time but I m not yet sure that I will do it the lack of cooperation of the release team for that kind of request is discouraging. I sponsored multiple uploads of dolibarr (on security update notably) and tcpdf (to fix one RC bug). Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

FTP assistant This month I marked 226 packages for accept and rejected 22. I also sent 5 emails to maintainers asking questions. It seems to be that a rather quiet month is behind us. As I have seen some packages with strange debian/copyright in binNEW, I wonder whether also the archive should be checked regularly. Maybe it is time to file some bugs Debian LTS This was my twenty-first month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. Due to outstanding hours that were redistributed, my all in all workload had been 14.25h. As Wheezy LTS didn t start yet and I am not able to do normal security uploads, I sent debdiffs to the security team. Btw. this can be done by everybody and the way to go is described in chapter 5.8.5 of the Debian Developer s Reference. Altogether I sent the following debdiffs for

• extplorer to fix CVE-2015-0896
• inspircd to fix CVE-2015-8702
• libmatroska to fix CVE-2015-8792
• libstruts1.2-java to fix CVE-2015-0899
• fuseiso to fix two temporary issues
• minissdpd to fix CVE-2016-3178 and CVE-2016-3179
• tlslite to fix CVE-2015-3220

As the security team wants to update Wheezy and Jessie with only one DSA, whenever applicable I created debdiffs for both releases. Up to now the results can be seen in DSA 3526-1, DSA 3527-1 and DSA 3536-1. As tlslite has been removed from Wheezy during today s point release, I am afraid that was a wasted effort. Other stuff My node activities this month involved uploads of: node-component-consoler, node-generator-supported, node-xmlhttprequest-ssl, node-co, node-uid-umber, node-url-join, node-uri-path, node-read-file, node-nth-check, node-base62, node-require-dir, node-for-in, node-obj-util, node-normalize-it-url, node-delve, node-function-bind, node-seq, node-json-localizer, node-through, node-addressparser, node-ansi-regex, node-crypto-cacerts, node-decamelize, node-array-find-index, node-require-main-filename, node-invert-kv, node-starttls. To fix one or the other bug I also uploaded: node-connect, node-mysql. I also forwarded bug #809252, which is tagged as security relevant in the BTS, to the Node Security Project. I even got one answer stating that the report arrived. We will see what happens next. At least after 45 days another email might arrive

The paths of life to guide us towards the knowledge are many and a straight; well-defined line does not always represent them. Among these, the theory of Fuzzy Logic can help us to better understand some of the methods that maybe for too long we have lost the habit to use, if not even forgotten, because of the residuals of the positivist scientism.

Brief mathematical introduction

All the elements making up the majority of the categories of everyday life (feelings, emotions, opinions, values, quality, etc.) cannot be defined in a unique or exact way. Let s try to define an event that causes severe pain: a hammer blow on the finger? The loss of a loved one? Tooth extraction? It is evident that the answer can vary in a very consistent way, based on the subjectivity of the individual, the environment, boundary conditions and the general context. This is because the Strong Pain is a fuzzy set (which we will call SP), characterized by a function of degree of membership which maps the elements of a universe in a real continuous interval between 0 and 1. In practical terms

<Read More [by Fabio Marzocca]>

Yesterday my contract with my university JAIST ended. I was working there as Associate Professor for 6.5 years, but the university decided not to extend my contract (financial problems, foreigner in Japan). That means, I am now out on the job market again not the greatest fun for an academic at my age. Thanks to JAIST for circumventing the ominous 5-year clause of permanent employment.
My prime interests are academic work, that is research and teaching, where research is in a variety of fields, mostly mathematical logic, theoretical computer science, and formal methods. Recently I am working on an extension of the Curry-Howard correspondence to cover different calculi, in particular Hyper Sequent Calculi, and relate them to some kind of process algebra. For more details see the logic dedicated page. My other focal point is programming and software development, where I have contributed to many open source projects, and act as main developer of the TeX Live infrastructure and distribution system (tlmgr). Main languages are Perl, PHP, C, but also Python, Vala, Lisp, CafeOBJ, and whatever comes around. For more details see the software page. I recently became father, so I would prefer either an academic position in Hokuriku, Japan, or a software developer position where I can work (most of the time) remotely. In case one of the readers here has some hints, please drop me a line. In the meantime I enjoy my unemployed life with my baby!

Debian Long Term Support (LTS) This is my 4th month working on Debian LTS, started by Raphael Hertzog at Freexian. I spent half of the month away on a vacation so little work was done, especially since I tried to tackle rather large uploads like NSS and Xen. I also worked on the frontdesk shift last week.

Frontdesk That work mainly consisted of figuring out how to best help the security team with the last uploads to the Wheezy release. For those who don't know, Debian 7 Wheezy, or "oldstable", is going to be unsupported by the security team starting end of april, and the Debian 6 Squeeze (the previous LTS) is now unsupported. The PGP signatures on the archived release have started yielding expiration errors which can be ignored but that are really a strong reminder that it is really time to upgrade. So the LTS team is now working towards backporting a few security issues from squeeze to wheezy, and this is what I focused on during triage work. I have identified the following high priority packages I will work on after I complete my work on the Xen and NSS packages (detailed below):

• libidn
• icu
• phpmyadmin
• tomcat6
• optipng
• srtp
• dhcpcd
• python-tornado

Updates to NSS and Xen I have spent a lot of time testing and building packages for NSS and Xen. To be fair, Brian May did most of the work on the Xen packages, and I merely did some work to test the packages on Koumbit's infrastructure, something which I will continue doing in the next month. For NSS, wheezy and jessie are in this weird state where patches were provided to the security team all the way back in November yet were never tested. Since then, yet more issues came up and I worked hard to review and port patches for those new security issues to wheezy. I'll followup on both packages in the following month.

Other free software work

Android TL;DR: there's an even longer version of this with the step-by-step procedures and that I will update as time goes on in my wiki. I somehow inherited an Android phone recently, on a loan from a friend because the phone broke one too many times and she got a new one from her provider. This phone is a HTC One S "Ville", which is fairly old, but good enough to play with and give me a mobile computing platform to listen to podcasts, play music, access maps and create GPS traces. I was previously doing this with my N900, but that device is really showing its age: very little development is happening on it, the SDK is closed source and the device itself is fairly big, compared to the "Ville". Plus, the SIM card actually works on the Ville so, even though I do not have an actual contract with a cell phone provider (too expensive, too invasive on my privacy), I can still make emergency phone calls (911)! Plus, since there is good wifi on the machine, I can use it to connect to the phone system with the built-in SIP client, send text messages through SMS (thanks to VoIP.ms SMS support) or Jabber. I have also played around with LibreSignal, the free software replacement for Signal, which uses proprietary google services. Yes, the VoIP.ms SMS app also uses GCM, but hopefully that can be fixed. (While I was writing this, another Debian Developer wrote a good review of Signal so I am happy to skip that step. Go read that.)

See also my apps list for a more complete list of the apps I have installed on the phone. I welcome recommendations on cool free software apps I should use!

I have replaced the stock firmware on the phone with Cyanogenmod 12.1, which was a fairly painful experience, partly because of the difficult ambiance on the #cyanogenmod channel on Freenode, where I had extreme experiences: a brave soul helped me through the first flashing process for around 2 hours, nicely holding my hand at every step. Other times, I have seen flames and obtuse comments from people being vulgar, brutal, obnoxious, if not sometimes downright homophobic and sexist from other users. It is clearly a community that needs to fix their attitude. I have documented everything I could in details in this wiki page, in case others want to resuscitate their old phones, but also because I ended up reinstalling the freaking phone about 4 times, and was getting tired of forgetting how to do it every time. I am somewhat fascinated by Android: here is the Linux-based device that should save us all from the proprietary Apple nightmares of fenced in gardens and censorship. Yet, public Android downloads are hidden behind license agreements, even though the code itself is free, which has led fellow Debian developers to work on making libre rebuilds of Androids to workaround this insanity. But worse: all phones are basically proprietary devices down to the core. You need custom firmware to be loaded on the machine for it to boot at all, from the bootloader all the way down to the GSM baseband and Wifi drivers. It is a minefield of closed source software, and trying to run free software on there is a bit of a delusion, especially since the baseband has so much power over the phone. Still, I think it is really interesting to run free software on those machines, and help people that are stuck with cell phones get familiar with software freedom. It seems especially important to me to make Android developers aware of software freedom considering how many apps are available for free yet on which it is not possible to contribute significantly because the source code is not published at all, or published only on the Google Store, instead of the more open and public F-Droid repository which publishes only free software. So I did contribute. This month, I am happy to announce that I contributed to the following free software projects on Android:

• listmyapps: feature request, patch
• Locker: bug report, added to f-droid
• Sensorium: bug report
• LibreSignal: feature request and testing

I have also reviewed the literature surrounding Telegram, a popular messaging app rival to Signal and Whatsapp. Oddly enough, my contributions to Wikipedia on that subject were promptly reverted which made me bring up the subject on the page's Talk page. This lead to an interesting response from the article's main editors which at least added that the "its security features have been contested by security researchers and cryptography experts". Considering the history of Telegram, I would keep people away from it and direct people to use Signal instead, even though Signal has similar metadata issues, mostly because Telegram's lack of response to the security issues that were outlined by fellow security researchers. Both systems suffer from a lack of federation as well, which is a shame in this era of increasing centralization. I am not sure I will put much more work in developing for Android for now. It seems like a fairly hostile platform to work on, and unless I have specific pain points I want to fix, it feels so much better to work on my own stuff in Debian. Which brings me to my usual plethora of free software projects I got stuck in this month.

IRC projects irssi-plugin-otr had a bunch of idiotic bugs lying around, and I had a patch that I hadn't submitted upstream from the Debian package, which needed a rebuild because the irssi version changed, which is a major annoyance. The version in sid is now a snapshot because upstream needs to make a new release but at least it should fix things for my friends running unstable and testing. Hopefully those silly uploads won't be necessary in the future. That's for the client side. On the server side, I have worked on updating the Atheme-services package to the latest version, which actually failed because the upstream libmowgli is missing release tags, which means the Debian package for it is not up-to-date either. Still, it is nice to have a somewhat newer version, even though it is not the latest and some bugs were fixed. I have also looked at making atheme reproducible but was surprised at the hostility of the upstream. In the end, it looks like they are still interested in patches, but they will be a little harder to deploy than for Charybdis, so this could take some time. Hopefully I will find time in the coming weeks to test the new atheme services daemon on the IRC network I operate.

Syncthing I have also re-discovered Syncthing, a file synchronization software. Amazingly, I was having trouble transferring a single file between two phones. I couldn't use Bluetooth (not sure why), the "Wifi sharing" app was available only on one phone (and is proprietary, and has a limit of 4MB files), and everything else requires an account, the cloud, or cabling. So. Just heading to f-droid, install syncthing, flash a few qr-codes around and voil : files are copied over! Pretty amazing: the files were actually copied over the local network, using IPv6 link-local addresses, encryption and the DHT. Which is a real geeky way of saying it's completely fast, secure and fast. Now, I found a few usability issues, so much that I wrote a whole usability story for the developers, which were really appreciative of my review. Some of the issues were already fixed, others were pretty minor. Syncthing has a great community, and it seems like a great project I encourage everyone to get familiar with.

Battery stats The battery-status project I mentionned previously has been merged with the battery-stats project (yes, the names are almost the same, which is confusing) and so I had to do some work to fix my Python graph script, which was accepted upstream and will be part of Debian officially from now on, which is cool. The previous package was unofficial. I have also noticed that my battery has a significantly than when I wrote the script. Whereas it was basically full back then, it seems now it has lost almost 15% of its capacity in about 6 months. According to the calculations of the script:

this battery will reach end of life (5%) in 935 days, 19:07:58.336480, on 2018-10-23 12:06:07.270290

Which is, to be fair, a good life: it will somewhat work for about three more years.

Playlist, git-annex and MPD in Python On top of my previously mentioned photos-import script, I have worked on two more small programs. One is called get-playlist and is an extension to git-annex to easily copy to the local git-annex repository all files present in a given M3U playlist. This is useful for me because my phone cannot possibly fit my whole MP3 collection, and I use playlists in GMPC to tag certain files, particularly the Favorites list which is populated by the "star" button in the UI. I had a lot of fun writing this script. I started using elpy as an IDE in Emacs. (Notice how Emacs got a new webpage, which is a huge improvement was had been basically unchanged since the original version, now almost 20 years old, and probably written by RMS himself.) I wonder how I managed to stay away from Elpy for so long, as it glues together key components of Emacs in an elegant and functional bundle:

• Company: the "vim-like" completion mode i had been waiting for forever
• Jedi: context-sensitive autocompletion for Python
• Flymake: to outline style and syntax errors (unfortunately not the more modern Flycheck)
• inline documentation...

In short, it's amazing and makes everything so much easier to work with that I wrote another script. The first program wouldn't work very well because some songs in the playlists had been moved, so I made another program, this time to repair playlists which refer to missing files. The script is simply called fix-playlists, and can operate transparently on multiple playlists. It has a bunch of heuristics to find files and uses a MPD server as a directory to search into. It can edit files in place or just act as a filter.

Useful snippets Writing so many scripts, so often, I figured I needed to stop wasting time always writing the same boilerplate stuff on top of every file, so I started publishing Yasnippet-compatible file snippets, in my snippets repository. For example, this report is based on the humble lts snippet. I also have a base license snippet which slaps the AGPLv3 license on top of a Python file. But the most interesting snippet, for me, is this simple script snippet which is a basic scaffolding for a commandline script that includes argument processing, logging and filtering of files, something which I was always copy-pasting around.

Other projects And finally, a list of interesting issues en vrac:

• there's a new Bootstrap 4 theme for Ikiwiki. It was delivering content over CDNs, which is bad for privacy issues, so I filed an issue which was almost immediately fixed by the author!
• I found out about BitHub, a tool to make payments with Bitcoins for patches submitted on OWS projects. It wasn't clear to me where to find the demo, but when I did, I quickly filed a PR to fix the documentation. Given the number of open PRs there and the lack of activity, I wonder if the model is working at all...
• a fellow Debian Developer shared his photos workflow, which was interesting to me because I have my own peculiar script, which I never mentionned here, but which I ended up sharing with the author

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

• Submitted a number of pull requests to the Ansible sever configuration tool:
  • Add the ability to specify the --allow-unauthenticated to APT. (#2023)
  • Ignore EPIPE when flushing standard input and output to avoid unnecessary and ugly tracebacks. (#14774)
  • Add an option for ansible-playbook to exit with an error if no plays were matched. (#14742)
• Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
  • Merged a patch from Joshua Blum to update the logging integration documentation. (#39)
  • Fixed an issue when rendering various exceptions would in themselves raise an exception. (#40)
  • Improved and merged a patch from Patrick Cloke to actually allow settings to be overriden using the @override_settings decorator. (#37)
• Updated local-debian-mirror, a package that attempts to make it easy to maintain a partial local mirror of the Debian archive:
  • Added an option to exclude DEP-11 files. (ee03b21)
  • Added an option to exclude Translation-* files. (28bbf34)
  • Corrected the phrasing of the debconf prompt to not ask a direct question to appease Lintian. (13fccc7b)
• Updated django-keyerror a library to post exceptions to the KeyError.com error tracking service to support pushing tracebacks from the Celery queue processor using the task_failure signal. (#1)
• Submitted a number of pull requests for django-zebra, a collection of utilities that make it easier to integrate the Stripe payment processor and the Django web development framework:
  • Add the ability to test an incoming Stripe webhook. (#44)
  • Prefer @require_POST over an explicit check in the view. (#43)
  • Move away from the deprecated django.conf.urls.patterns method. (#45)
• Updated django-template-tests, a tool to perform simple static analysis on Django templates prior to production to improve the robustness of dynamic test generation. (e98f9fc)
• Fixed django-force-logout a library to allow administrators to log user's out of Django projects to not require a custom JSON serialiser. (c0e5d64)
• Corrected the documentation for travis.debian.net my hosted script to easily test and build Debian packages on the Travis CI continuous integration platform to reflect the actual default mirror. (4c862f0)
• Blogged about generating dynamic Python tests using metaclasses.

---

TL;DR; If you want to know why cloning several of the X.org repositories to Github or GitLab instances fails and how this can be worked around, you may want to continue reading. Why we stumbled over this issue... As a joint effort of the Arctica Project, TheQVD and X2Go, Ulrich Sibiller and I are currently preparing a build workflow for the nxagent X-server (version 3) [1] that allows building nxagent against the modular X.org 7.0 (using autoconf and automake) rather than the monolithic build workflow of X.org 6.9 (using ancient imake). Our goal is to rewind all X.org components required for building nxagent back to a state where nxagent successfully builds and runs. Then we will go through various, (probably) monthly cycles of

• bumping commit level of required X.org components
• rebuild and test nxagent
• fix FTBFS, if any
• fix broken functionality, if any

Our first hurdle... After Ulrich now has a functioning nxagent-against-X.org-7.0-workflow locally, we want to get everything into the Arctica Project's Github namespace. Which fails...

[mike@minobo xorg.upstream]$ git clone https://anongit.freedesktop.org/git/xorg/lib/libX11.git
Cloning into 'libX11'...
remote: Counting objects: 18520, done.
remote: Compressing objects: 100% (3441/3441), done.
remote: Total 18520 (delta 15094), reused 18325 (delta 14954)
Receiving objects: 100% (18520/18520), 5.98 MiB   549.00 KiB/s, done.
Resolving deltas: 100% (15094/15094), done.
Checking connectivity... done.
[mike@minobo xorg.upstream]$ cd libX11/
[mike@minobo libX11 (master)]$ git remote rename origin upstream 
[mike@minobo libX11 (master)]$ git remote add origin git@github.com:ArcticaProject/libX11.git
[mike@minobo libX11 (master)]$ git push --mirror origin 
Counting objects: 18520, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3301/3301), done.
remote: error: object 70d5e4d45dd7bf1e05b099cb5a4dd529344084f0: missingSpaceBeforeDate: invalid author/committer line - missing space before date
remote: fatal: Error in object
error: pack-objects died of signal 13
error: failed to push some refs to 'git@github.com:ArcticaProject/libX11.git'

This issue should be a known issue at X.org / Mesa upstream [2]. The underlying problem occurs with Git 2.5 and above... If you run git-fsck from Git (>= 2.5) against several of the X.org Git repositories, you will stumble over the above issue. For libX11.git, we see these errors reported when running git-fsck from current Debian unstable:

mike@sid:~/xorg/libX11.git$ git fsck 
Checking object directories: 100% (256/256), done.
error in tag 70d5e4d45dd7bf1e05b099cb5a4dd529344084f0: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a3bfe698090a5d41f1e9acb1b57a049085d6b04e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag c1fc8c9aec1bc92d03d08d5e986dd40b194a7a3e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag d4c27f7e7d8e2ea32418f341ad85c33f3b76862d: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 79780379aee21bf4d4bcb046a3b54774893ea6b1: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a638de80ec20dc1de625cd323ed19a6646fecf2e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f6a6414e4e52a971a35fd118c350567e2383d034: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b5c6810e21be2e7ccfac8f0539d46bf75dbe50a0: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 86cec9d20428cc190ec7278a0abe481b180288ac: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ac912988091574790c7cffbbc2c60b8c59f8fcef: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2d01674092951cd3284b3b099978b2436ea468ad: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 922d0534154668918eaa09a36cc5cd53fc6b71b7: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 8ef91b9cc0a4a7b0361f6b40b3f735221c8272cb: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1516173727005a87aa501e1ad708d72e6ae6e753: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 77c9e63a3abff1593ccbed249b2c3ae7f68e1833: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 58044c7087137ce2b521297dbb31934ccaedc94e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2e0d061c6830891dcc856a04aac900e5bb6d0779: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2f58c7cc4cbf3d30927f6301039b6bf018ed38fd: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 77fc2e1e26019c3ba92275d249fe1979570255d2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9b2e06fa262f23dbcbbccd42be6477e08a452268: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 7f99410bb9cc9f7cbbc2d43d8ad044771a6eec0a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 99b59b83f126b4abb9da89f577a5b8147d543c25: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 97170c33f99761acd932f968e21d6d7664e7ba80: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b39966d452127c992987d082f69321f950ae4c39: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 05e20c3bd8ac6b7e873607db73894486a4ac0519: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f478497b5a3e9d6b92237e0f408c9f0f18108f52: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag c5ed936f28ce70c8932e33d67a6e06d140e35ba8: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2b6beba295a6ff997dfe15a732c95ae6577fa735: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 32edbb1e9c5b7d1c6f8639dab85e5368d303df7f: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 490fbe2afb9ffc7057c0eb150f7cdd4d4fa8686c: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag fc605ac48bd7d675d6d04162a0f3bfebe27f2037: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6289ecc26d1d505f395c59da744a9354d7aad2a2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 8c9c30dac72aebd1a2c31a55a47c0ac11bc328db: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 3b7aa57994924be0dad693e837559d9ee900299c: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b02ad9a8ec96e3c1027ca71289aeef5b8748e17e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 4402ab1a86226a9cd0ce84fb8147b1c4958c684d: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag bb02b4fbde316644c771d28d5edeeb9b213a6d6a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 95b6462c70213b562c9d2b8726eda706edc9c456: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a99fea43ad53907623f269942c55237c238645e0: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 0c417ce98a6cf927ce7a1359b198b2bb746c707e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ce3ff6b102f360fc9dcd3a85361f44da054de0b1: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2fb7b16b38a365c5009bd170d4463634c1f3de26: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 95570f484bfde5fc2ed8548cb17b87d8114a2866: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag d6e5895c104cfe0d135494a605013dd2910a93a1: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 49be7144523c4a56afcb389a9aa021d167710d2c: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 020353d1b982ac938d229039ca13bfea5793fcd9: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 991792d4f8477ab65ad4d8a0245c9a3bbb1f0294: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag e8037555efe299a7143370b485f754f6b08ffac7: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 4fdc8e36a5022f3e32d5567305d8e9cde4011f1a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9c93a19a76163328e30a7306bb0babe3175fb9a2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9f90bf4424250f80340b3e2803d343c74d2adb0b: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2e7a8c7974777e3e869de9013025dc3f61633e1e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 59b7c06da2a64d72668394e9239e3dfe8a5cff59: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 0263c4f684faa9a5126a0db524d5d35be7277e52: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1514e654c1a9b346cbfca5a81f6d331f106242aa: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6aa83ec3d1d82ee332cedcf4e99d3cbd56e73edd: missingSpaceBeforeDate: invalid author/committer line - missing space before date
Checking objects: 100% (18523/18523), done.

What actually is going wrong here... When looking at one of those tags (luckily, only tags are affected), we see that something's not ok with the date string of the tag (I am not a Git plumber, so sorry for being superficial here):

mike@sid:~/xorg/libX11.git$ git show 2fb7b16b38a365c5009bd170d4463634c1f3de26
tag XORG-RELEASE-1-TM-MERGE
Tagger: Alan Coopersmith 
Date:   Thu Jan 1 00:00:00 1970 +0000
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
commit 84adc60cae8e944777563501dd633622a28bdb3b
Author: Alan Coopersmith 
Date:   Sat Mar 27 07:08:16 2004 +0000
    Fix typo (IsModiferKey should be IsModifierKey)
[... not quoting the diff for the above commit ...]

How to work around that issue... We then got the idea that we could redo those tags with the date string derived from the commit that this tag points at. This can be scripted (credits go to an unnamed person in Ulrich's surrounding!!!):

mike@sid:~/xorg/libX11$ cat repair.sh
#!/bin/bash
set -x
git fsck 2>&1   tee git-fsck.log   grep "error in tag"   sed -r -e 's/error in tag ([0-9a-f]+):.*/\1/p'   while read broken; do
         echo $broken
         commit=$(git rev-parse $ broken ^ commit )
         tag=$(git cat-file tag $broken   sed -n 's/^tag //p')
         tag_msg=$(git cat-file tag $broken   sed -n '/^$/,$p'   tail -n +2)
         export GIT_COMMITTER_NAME="$(git log -1 --format='%cn' $commit)"
         export GIT_COMMITTER_EMAIL="$(git log -1 --format='%ce' $commit)"
         export GIT_COMMITTER_DATE="$(git log -1 --format='%cD' $commit)"
         git tag -a -f -m "$tag_msg" $tag $commit
done
# drop all old content...
git reflog expire --expire=all --all
git gc --prune=all
# no more errors should be reported...
git fsck

Copy this repair.sh into the Git repo itself and let the script walk over your tags. After this script has been applied you can push the X.org Git repo with broken tags redone to Git hosters like Github or GitLab. light+love,
Mike [1] https://github.com/ArcticaProject/nx-libs
[2] https://lists.freedesktop.org/archives/mesa-dev/2016-February/106268.html

When I started contributing to OpenStack, almost five years ago, it was a small ecosystem. There were no foundation, a handful of projects and you could understand the code base in a few days. Fast forward 2016, and it is a totally different beast. The project grew to no less than 54 teams, each team providing one or more deliverable. For example, the Nova and Swift team each one produces one service and its client, whereas the Telemetry team produces 3 services and 3 different clients. In 5 years, OpenStack went to a few IaaS projects, to 54 different teams tackling different areas related to cloud computing. Once upon a time, OpenStack was all about starting some virtual machines on a network, backed by images and volumes. Nowadays, it's also about orchestrating your network deployment over containers, while managing your application life-cycle using a database service, everything being metered and billed for. This exponential growth has been made possible with the decision of the OpenStack Technical Committee to open the gates with the project structure reform voted at the end of 2014. This amendment suppresses the old OpenStack model of "integrated projects" (i.e. Nova, Glance, Swift ). The big tent, as it's called, allowed OpenStack to land new projects every month, growing from the 20 project teams of December 2014 to the 54 we have today multiplying the number of projects by 2.7 in a little more than a year. Amazing growth, right? And this was clearly a good change. I sat at the Technical Committee in 2013, when projects were trying to apply to be "integrated", after Ceilometer and Heat were. It was painful to see how the Technical Committee was trying to assess whether new projects should be brought in or not. But what I notice these days, is how OpenStack is still stuck between its old and new models. On one side, it accepted a lot of new teams, but on the other side, many are considered as second-class citizens. Efforts are made to continue to build an OpenStack project that does not exist anymore. For example, there is a team trying to define what's OpenStack core, named DefCore. That is looking to define which projects are, somehow, actually OpenStack. This leads to weird situations, such as having non-DefCore projects seeing their doc rejected from installation guides. Again, I reiterated my proposal to publish documentation as part of each project code to solve that dishonest situation and put everything on a level playing field Some cross-projects specs are also pushed without implication of all OpenStack projects. For example, The deprecate-cli spec which proposes to deprecate command-line interface tools proposed by each project had a lot of sense in the old OpenStack sense, where the goal was to build a unified and ubiquitous cloud platform. But when you now have tens of projects with largely different scopes, this start making less sense. Still, this spec was merged by the OpenStack Technical Committee this cycle. Keystone is the first project to proudly force users to rely on openstack-client, removing its old keystone command line tool. I find it odd to push that specs when it's pretty clear that some projects (e.g. Swift, Gnocchi ) have no intention to go down that path. Unfortunately, most specs pushed by the Technical Committee are in the realm of wishful thinking. It somehow makes sense, since only a few of the members are actively contributing to OpenStack projects, and they can't by themselves implement all of that magically. But OpenStack is no exception in the free software world and remains a do-ocracy. There is good cross-project content in OpenStack, such as the API working group. While the work done should probably not be OpenStack specific, there's a lot that teams have learned by building various HTTP REST API with different frameworks. Compiling this knowledge and offering it as a guidance to various teams is a great help. My fellow developer Chris Dent wrote a post about what he would do on the Technical Committee. In this article, he points to a lot of the shortcomings I described here, and his confusion between OpenStack being a product or being a kit is quite understandable. Indeed, the message broadcasted by OpenStack is still very confusing after the big tent openness. There's no enough user experience improvement being done. The OpenStack Technical Committee election is opened for April 2016, and from what I read so far, many candidates are proposing to now clean up the big tent, kicking out projects that do not match certain criteria anymore. This is probably a good idea, there is some inactive project laying around. But I don't think that will be enough to solve the identity crisis that OpenStack is experiencing. So this is why, once again this cycle, I will throw my hat in the ring and submit my candidacy for OpenStack Technical Committee.

Julian has written about their efforts to strengthen security in APT, and shortly before that notified us that Launchpad s signatures on PPAs use weak SHA-1 digests. Unfortunately we hadn t noticed that before; GnuPG s defaults tend to result in weak digests unless carefully tweaked, which is a shame. I started on the necessary fixes for this immediately we heard of the problem, but it s taken a little while to get everything in place, and I thought I d explain why since some of the problems uncovered are interesting in their own right. Firstly, there was the relatively trivial matter of using SHA-512 digests on new signatures. This was mostly a matter of adjusting our configuration, although writing the test was a bit tricky since PyGPGME isn t as helpful as it could be. (Simpler repository implementations that call gpg from the command line should probably just add the --digest-algo SHA512 option instead of imitating this.) After getting that in place, any change to a suite in a PPA will result in it being re-signed with SHA-512, which is good as far as it goes, but we also want to re-sign PPAs that haven t been modified. Launchpad hosts more than 50000 active PPAs, though, a significant percentage of which include packages for sufficiently recent Ubuntu releases that we d want to re-sign them for this. We can t expect everyone to push new uploads, and we need to run this through at least some part of our usual publication machinery rather than just writing a hacky shell script to do the job (which would have no idea which keys to sign with, to start with); but forcing full reprocessing of all those PPAs would take a prohibitively long time, and at the moment we need to interrupt normal PPA publication to do this kind of work. I therefore had to spend some quality time working out how to make things go fast enough. The first couple of changes (1, 2) were to add options to our publisher script to let us run just the one step we need in careful mode: that is, forcibly re-run the Release file processing step even if it thinks nothing has changed, and entirely disable the other steps such as generating Packages and Sources files. Then last week I finally got around to timing things on one of our staging systems so that we could estimate how long a full run would take. It was taking a little over two seconds per archive, which meant that if we were to re-sign all published PPAs then that would take more than 33 hours! Obviously this wasn t viable; even just re-signing xenial would be prohibitively slow. The next question was where all that time was going. I thought perhaps that the actual signing might be slow for some reason, but it was taking about half a second per archive: not great, but not enough to account for most of the slowness. The main part of the delay was in fact when we committed the database transaction after processing each archive, but not in the actual PostgreSQL commit, rather in the ORM invalidate method called to prepare for a commit. Launchpad uses the excellent Storm for all of its database interactions. One property of this ORM (and possibly of others; I ll cheerfully admit to not having spent much time with other ORMs) is that it uses a WeakValueDictionary to keep track of the objects it s populated with database results. Before it commits a transaction, it iterates over all those alive objects to note that if they re used in future then information needs to be reloaded from the database first. Usually this is a very good thing: it saves us from having to think too hard about data consistency at the application layer. But in this case, one of the things we did at the start of the publisher script was: That innocuous-looking filter means that we do all the public/private filtering of PPAs up-front and return a list of all the PPAs we intend to operate on. This means that all those objects are alive as far as Storm is concerned and need to be considered for invalidation on every commit, and the time required for that stacks up when many thousands of objects are involved: this is essentially accidentally quadratic behaviour, because all archives are considered when committing changes to each archive in turn. Normally this isn t too bad because only a few hundred PPAs need to be processed in any given run; but if we re running in a mode where we re processing all PPAs rather than just ones that are pending publication, then suddenly this balloons to the point where it takes a couple of seconds. The fix is very simple, using an iterator instead so that we don t need to keep all the objects alive: After that, I turned to that half a second for signing. A good chunk of that was accounted for by the signContent method taking a fingerprint rather than a key, despite the fact that we normally already had the key in hand; this caused us to have to ask GPGME to reload the key, which requires two subprocess calls. Converting this to take a key rather than a fingerprint gets the per-archive time down to about a quarter of a second on our staging system, about eight times faster than where we started. Using this, we ve now re-signed all xenial Release files in PPAs using SHA-512 digests. On production, this took about 80 minutes to iterate over around 70000 archives, of which 1761 were modified. Most of the time appears to have been spent skipping over unmodified archives; even a few hundredths of a second per archive adds up quickly there. The remaining time comes out to around 0.4 seconds per modified archive. There s certainly still room for speeding this up a bit. We wouldn t want to do this procedure every day, but it s acceptable for occasional tasks like this. I expect that we ll similarly re-sign wily, vivid, and trusty Release files soon in the same way.