The indoor garden at Coll ge de Maisonneuve, the DebConf 17 venue

I'm currently at DebConf 17 in Montr al, back at DebConf for the first time in 10 years (last time was DebConf 7 in Edinburgh). It's great to put names to faces and meet more of my co-developers in person! On Monday I gave a talk entitled A Debian maintainer's guide to Flatpak , aiming to introduce Debian developers to Flatpak, and show how Flatpak and Debian (and Debian derivatives like SteamOS) can help each other. It seems to have been quite well received, with people generally positive about the idea of using Flatpak to deliver backports and faster-moving leaf packages (games!) onto the stable base platform that Debian is so good at providing. A video of the talk is available from the Debian Meetings Archive. I've also put up my slides in the DebConf git-annex repository, with some small edits to link to more source code: A Debian maintainer's guide to Flatpak. Source code for the slides is also available from Collabora's git server. The next step is to take my proof-of-concept for building Flatpak runtimes and apps from Debian and SteamOS packages, flatdeb, get it a bit more production-ready, and perhaps start publishing some sample runtimes from a cron job on a Debian or Collabora server. (By the way, if you downloaded that source right after my talk, please update - I've now pushed some late changes that were necessary to fix the 3D drivers for my OpenArena demo.) I don't think Debian will be going quite as far as Endless any time soon: as Cosimo outlined in the talk right before mine, they deploy their Debian derivative as an immutable base OS with libOSTree, with all the user-installable modules above that coming from Flatpak. That model is certainly an interesting thing to think about for Debian derivatives, though: at Collabora we work on a lot of appliance-like embedded Debian derivatives, with a lot of flexibility during development but very limited state on deployed systems, and Endless' approach seems a perfect fit for those situations. [Edited 2017-08-16 to fix the link for the slides, and add links for the video]

Today is Debian's 24th anniversary. If you are close to any of the cities celebrating Debian Day 2017, you're very welcome to join the party! If not, there's still time for you to organize a little celebration or contribution to Debian. For example, spread the word about Debian Day with this nice piece of artwork created by Debian Developer Daniel Lenharo de Souza and Valessio Brito, taking inspiration from the desktop themes Lines and softWaves by Juliette Belin: If you also like graphics design, or design in general, have a look at https://wiki.debian.org/Design and join the team! Or you can visit the general list of Debian Teams for many other opportunities to participate in Debian development. Thanks to everybody who has contributed to develop our beloved operating system in these 24 years, and happy birthday Debian!

We have been announcing it: we are going to remove Qt 4 during the Buster cycle.

Or at least that's the best outcome we can expect. Removing a very highly used library is hard, as Qt4's Webkit has proved. Qt 4 is long dead upstream and we have already started to need to patch it with untested patches as in the OpenSSL 1.1 case (will be in experimental in a few hours after this post).

We will try to put as less effort as possible in keeping it alive meaning that from now on if we need to patch it to make it support a newer lib or alike we will simply remove its support if possible. Using the OpenSSL case as an example, if we need to support any version > 1.1 we will simply remove the SSL support. That means things will break.

So, if you depend on FLOSS which is still based on Qt 4 be sure to try to port it. If you depend on a proprietary vendor software which uses Qt 4 then you better start telling them it's really time to update it. Really.

We will soon start filing bugs against packages using Qt 4. I'll update this blog post later to add that info.

For the Qt/KDE team, Lisandro.

Here's what happened in the Reproducible Builds effort between Sunday July 30 and Saturday August 5 2017: Media coverage We were mentioned on Late Night Linux Episode 17, around 29:30. Packages reviewed and fixed, and bugs filed Upstream packages:

• Bernhard M. Wiedemann:
  • efl (merged), unique ids based on memory address
  • 389-ds (merged), SOURCE_DATE_EPOCH support.
  • plowshare, SOURCE_DATE_EPOCH support
  • sphinx, file ordering
  • sphinx, SOURCE_DATE_EPOCH support

Debian packages:

• Adrian Bunk:
  • #870212 filed against glib2.0.
  • #870213 filed against pajeng.
  • #870733 filed against openhpi.
  • #870738 filed against gtk2-engines-xfce.
  • #870741 filed against libgda5.
  • #870742 filed against libgnome.
  • #870749 filed against glade.
  • #870821 filed against esys-particle.
• Chris Lamb:
  • #870131 filed against autopep8.
  • #870221 filed against dpkg.
  • #870573 filed against grap.
• Johannes Schauer:
  • #870585 filed against hevea.
• Logan Rosen:
  • #870586 filed against behave.
• Lucas Nussbaum:
  • #871059 filed against fltk1.3.
  • #871155 filed against brltty.
  • #871159 filed against texlive-extra.
• jathan:
  • #870890 filed against apg.

Reviews of unreproducible packages 29 package reviews have been added, 72 have been updated and 151 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been updated:

• Added timestamps_generated_by_hevea.
• Added timestamps_in_source_generated_by_rcc.
• Updated build_id_differences_only: remove an obsolete example.
• Updated golang_compiler_captures_build_path_in_binary: mark as not deterministic, because the patch fixing it is not yet upstreamed.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (36)
• Andreas Beckmann (2)
• Daniel Schepler (2)
• Logan Rosen (1)
• Lucas Nussbaum (93)

diffoscope development Version 85 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Mattia Rizzolo:
  • Add an explicit Recommends: on the defusedxml python package.
  • Various other code quality tweaks.
• Juliana Oliveira Rodrigues:
  • Fix test_ico_image for ImageMagick identify >= 6.9.8.
  • Use the defusedxml XML library by default in the XML comparator, if it's available. This protects against various XML parser DoS attacks and other security holes, which other Python XML libraries are vulnerable to.
• Ximin Luo:
  • Force a flush when writing output to diff. (Closes: #870049).

as well as previous weeks' contributions, summarised in the changelog. There were also further commits in git, which will be released in a later version:

• Guangyuan Yang:
  • tests/iso9660: support isoinfo's output coming from cdrtools' version instead of genisoimage's
• Mattia Rizzolo:
  • Code quality and test fixes.
• Chris Lamb:
  • Code quality and test fixes.

Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Today, Saturday 12 August 2017, the annual Debian Developers and Contributors Conference came to a close. With over 405 people attending from all over the world, and 169 events including 89 talks, 61 discussion sessions or BoFs, 6 workshops and 13 other activities, DebConf17 has been hailed as a success. Highlights included DebCamp with 117 participants, the Open Day,
where events of interest to a broader audience were offered, talks from invited speakers (Deb Nicholson, Matthew Garrett and Katheryn Sutter), the traditional Bits from the DPL, lightning talks and live demos and the announcement of next year's DebConf (DebConf18 in Hsinchu, Taiwan). The schedule has been updated every day, including 32 ad-hoc new activities, planned
by attendees during the whole conference. For those not able to attend, talks and sessions were recorded and live streamed, and videos are being made available at the Debian meetings archive website. Many sessions also facilitated remote participation via IRC or a collaborative pad. The DebConf17 website will remain active for archive purposes, and will continue to offer links to the presentations and videos of talks and events. Next year, DebConf18 will be held in Hsinchu, Taiwan, from 29 July 2018 until 5 August 2018. It will be the first DebConf held in Asia. For the days before DebConf the local organisers will again set up DebCamp (21 July - 27 July), a session for some intense work on improving the distribution, and organise the Open Day on 28 July 2018, aimed at the general public. DebConf is committed to a safe and welcome environment for all participants. See the DebConf Code of Conduct and the Debian Code of Conduct for more details on this. Debian thanks the commitment of numerous sponsors to support DebConf17, particularly our Platinum Sponsors Savoir-Faire Linux, Hewlett Packard Enterprise, and Google. About Savoir-faire Linux Savoir-faire Linux is a Montreal-based Free/Open-Source Software company with offices in Quebec City, Toronto, Paris and Lyon. It offers Linux and Free Software integration solutions in order to provide performance, flexibility and independence for its clients. The company actively contributes to many free software projects, and provides mirrors of Debian, Ubuntu, Linux and others. About Hewlett Packard Enterprise Hewlett Packard Enterprise (HPE) is one of the largest computer companies in the world, providing a wide range of products and services, such as servers, storage, networking, consulting and support, software, and financial services. HPE is also a development partner of Debian, and provides hardware for port development, Debian mirrors, and other Debian services. About Google Google is one of the largest technology companies in the world, providing a wide range of Internet-related services and products as online advertising technologies, search, cloud computing, software, and hardware. Google has been supporting Debian by sponsoring DebConf since more than ten years, at gold level since DebConf12, and at platinum level for this DebConf17.

I used to think I was a programmer who did "sysadmin-stuff". Nowadays I interact with too many real programmers to believe that. Or rather I can code/program/develop, but I'm not often as good as I could be. These days I'm getting more consistent with writing tests, and I like it when things are thoroughly planned and developed. But too often if I'm busy, or distracted, I think to myself "Hrm .. compiles? Probably done. Oops. Bug, you say?" I was going to write about working with golang today. The go language is minimal and quite neat. I like the toolset:

• go fmt
  • Making everything consistent.
• go test
  • Making testing an obvious and natural part of libraries and code.
  • My personal puppet dashboard went from 0% code-coverage to ~60% coverage in the space of a few hours.

Instead I think today I'm going to write about something else. Since having a child a lot of my life is different. Routine becomes something that is essential, as is planning and scheduling. So an average week-day goes something like this:

• 6:00AM
  • Wake up (naturally).
• 7:00AM
  • Wake up Oiva and play with him for 45 minutes.
• 7:45AM
  • Prepare breakfast for my wife, and wake her up, then play with Oiva for another 15 minutes while she eats.
• 8:00AM
  • Take tram to office.
• 8:30AM
  • Make coffee, make a rough plan for the day.
• 9:00AM
  • Work, until lunchtime which might be 1pm, 2pm, or even 3pm.
• 5:00PM
  • Leave work, and take bus home.
  • Yes I go to work via tram, but come back via bus. There are reasons.
• 5:40PM
  • Arrive home, and relax in peace for 20 minutes.
• 6:00PM-7:00PM
  • Take Oiva for a walk, stop en route to relax in a hammock for 30 minutes reading a book.
• 7:00-7:20PM
  • Feed Oiva his evening meal.
• 7:30PM
  • Give Oiva his bath, then pass him over to my wife to put him to bed.
• 7:30PM - 8:00pm
  • Relax
• 8:00PM - 10:00PM
  • Deal with Oiva waking up, making noises, or being unsettled.
  • Try to spend quality time with my wife, watch TV, read a book, do some coding, etc.
• 10:00PM ~ 11:30PM
  • Go to bed.

In short I'm responsible for Oiva from 6ish-8ish in the morning, then from 6PM-10PM (with a little break while he's put to bed.) There are some exceptions to this routine - for example I work from home on Monday/Friday afternoons, and Monday evenings he goes to his swimming classes. But most working-days are the same. Weekends are a bit different. There I tend to take him 6AM-8AM, then 1PM-10PM with a few breaks for tea, and bed. At the moment we're starting to reach the peak-party time of year, which means weekends often involve negotiation(s) about which parent is having a party, and which parent is either leaving early, or not going out at all. Today I have him all day, and it's awesome. He's just learned to say "Daddy" which makes any stress, angst or unpleasantness utterly worthwhile.

Weblate 2.16 has been released today while I'm at DebConf17. There are quite some performance improvements (and more of that is scheduled for 2.17), new file formats support and various other improvements. Full list of changes:

• Various performance improvements.
• Added support for nested JSON format.
• Added support for WebExtension JSON format.
• Fixed git exporter authentication.
• Improved CSV import in certain situations.
• Improved look of Other translations widget.
• The max-length checks is now enforcing length of text in form.
• Make the commit_pending age configurable per component.
• Various user interface cleanups.
• Fixed component/project/sitewide search for translations.

If you are upgrading from older version, please follow our upgrading instructions. You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects. Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English SUSE Weblate

Last Monday, I gave a 20min talk about our little FLOSS school project "IT-Zukunft Schule" at the Debian Conference 17 in Montreal. The talk had video coverage, so may want to peek in, if you couldn't manage to watch the life stream: http://meetings-archive.debian.net/pub/debian-meetings/2017/debconf17/su... I'd like to share some major outcomes (so far) of this talk.

1. I realized how attached I am to "IT-Zukunft Schule" and how much it means to me that our kids grow up in a world of freedom and choice. Also and esp. when it comes to choosing your daily communication tools and computer working environment
2. I met Foteini Tsiami and Alkis Georgopoulos from Greece. They work on LTSP and have deployed 1000+ schools in Greece with LTSP + Debian GNU/Linux + MATE Desktop Environment
3. I met Vagrant Cascadian who is the maintainer of LTSP in Debian and also a major LTSP upstream contributor
4. I received a lot of fine feedback that was very encouraging to go on with our local work in Schleswig-Holstein

If you have some more time for watching DebConf talks on video, I dearly recommend the talk given by Alkis and Foteini on their Greek FLOSS success story. If you don't have that much time, please skip through the video until you are at 26:15 and enjoy the map that shows how much Debian + LTSP has spread over all of Greece. http://meetings-archive.debian.net/pub/debian-meetings/2017/debconf17/lt... Unfortunately, the schools in Greece are so much smaller than schools in Germany. Most schools there have between 50 and 300 students. So at the Greek schools, it is possible to have a teacher machine being the server for one computer lab. This teacher / server machine provides the infrastructure for a room full of LTSP fat clients (no hard drive inside) and that's it. For German schools, unfortunately, we need a larger scale setup. German schools often have 800+ students and network services need to be spread over more than one server machine. So, the current approach with one server running LDAP, Kerberos etc. is quite appropriate, but also extendible, possibly on municipality level or on county level. We (from IT-Zukunft Schule) are quite positive that there will be opportunities for introducing FLOSS approaches more on the county level in Schleswig-Holstein in the near future. So stay tuned...

I recently ran into problems trying to package the latest version of my planetfilter tool. This is how I was able to temporarily work-around bugs in my tools and still produce a package that can be built reproducibly from source and that contains a verifiable upstream signature.

pristine-tar being is unable to reproduce a tarball After importing the latest upstream tarball using gbp import-orig, I tried to build the package but ran into this pristine-tar error:

$ gbp buildpackage
gbp:error: Pristine-tar couldn't checkout "planetfilter_0.7.4.orig.tar.gz": xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3 decode failed! at /usr/share/perl5/Pristine/Tar/DeltaTools.pm line 56.
pristine-tar: command failed: pristine-gz --no-verbose --no-debug --no-keep gengz /tmp/user/1000/pristine-tar.mgnaMjnwlk/wrapper /tmp/user/1000/pristine-tar.EV5aXIPWfn/planetfilter_0.7.4.orig.tar.gz.tmp
pristine-tar: failed to generate tarball

So I decided to throw away what I had, re-import the tarball and try again. This time, I got a different pristine-tar error:

$ gbp buildpackage
gbp:error: Pristine-tar couldn't checkout "planetfilter_0.7.4.orig.tar.gz": xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
xdelta3: normally this indicates that the source file is incorrect
xdelta3: please verify the source file with sha1sum or equivalent
xdelta3 decode failed! at /usr/share/perl5/Pristine/Tar/DeltaTools.pm line 56.
pristine-tar: command failed: pristine-gz --no-verbose --no-debug --no-keep gengz /tmp/user/1000/pristine-tar.mgnaMjnwlk/wrapper /tmp/user/1000/pristine-tar.EV5aXIPWfn/planetfilter_0.7.4.orig.tar.gz.tmp
pristine-tar: failed to generate tarball

I filed bug 871938 for this. As a work-around, I simply symlinked the upstream tarball I already had and then built the package using the tarball directly instead of the upstream git branch:

ln -s ~/deve/remote/planetfilter/dist/planetfilter-0.7.4.tar.gz ../planetfilter_0.7.4.orig.tar.gz
gbp buildpackage --git-tarball-dir=..

Given that only the upstream and master branches are signed, the .delta file on the pristine-tar branch could be fixed at any time in the future by committing a new .delta file once pristine-tar gets fixed. This therefore seems like a reasonable work-around.

git-buildpackage doesn't import the upstream tarball signature The second problem I ran into was a missing upstream signature after building the package with git-buildpackage:

$ lintian -i planetfilter_0.7.4-1_amd64.changes
E: planetfilter changes: orig-tarball-missing-upstream-signature planetfilter_0.7.4.orig.tar.gz
N: 
N:    The packaging includes an upstream signing key but the corresponding
N:    .asc signature for one or more source tarballs are not included in your
N:    .changes file.
N:    
N:    Severity: important, Certainty: certain
N:    
N:    Check: changes-file, Type: changes
N: 

This problem (and the lintian error I suspect) is fairly new and hasn't been solved yet. So until gbp import-orig gets proper support for upstream signatures, my work-around was to copy the upstream signature in the export-dir output directory (which I set in ~/.gbp.conf) so that it can be picked up by the final stages of gbp buildpackage:

ln -s ~/deve/remote/planetfilter/dist/planetfilter-0.7.4.tar.gz.asc ../build-area/planetfilter_0.7.4.orig.tar.gz.asc

If there's a better way to do this, please feel free to leave a comment (authentication not required)!

I am still far from being a professor, but I recently got a glimps of what awaits you in that role

From: Sebastian R. < @gmail.com>
To: joachim@cis.upenn.edu
Subject: re: Errors I've spotted a basic error in your course on Haskell (https://www.seas.upenn.edu/~cis194/fall16/). Before I proceed, it's cool if you're not receptive to errors being indicated; I've come across a number of professors who would rather take offense than admit we're all human and thus capable of making mistakes... My goal is to find a resource that might be useful well into the future, and a good indicator of that is how responsive the author is to change. In your introduction note you have written:

n contrast to a classical intro into Haskell, we do not start with numbers, booleans, tuples, lists and strings, but we start with pictures. These are of course library-defined (hence the input CodeWorld) and not part of the language . But that does not make them less interesting, and in fact, even the basic boolean type is library defined it just happens to be the standard library.

Howeverm there is no input CodeWorld in the code above. Have you been made aware of this error earlier? Regards, ...

Nice. I like when people learn from my lectures. The introduction is a bit werid, but ok, maybe this guy had some bad experiences. Strangley, I don t see a mistake in the material, so I respond:

From: Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript>
To: Sebastian R. < @gmail.com>
Subject: Re: Errors Dear Sebastian, thanks for pointing out errors. But the first piece of code under Basic Haskell starts with

 -# LANGUAGE OverloadedStrings #- 
import CodeWorld

so I am not sure what you are referring to. Note that these are lecture notes, so you have to imagine a lecturer editing code live on stage along with it. If you only have the notes, you might have to infer a few things. Regards, Joachim

A while later, I receive this response:

From: Sebastian R. < @gmail.com>
To: Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript>
Subject: Re: Errors Greetings, Joachim. Kindly open the lecture slides and search for "input CodeWorld" to find the error; it is not in the code, but in the paragraph that implicitly refers back to the code. You might note that I quoted this precisely from the lectures... and so I repeat myself... this came from your lectures; they're not my words!

In contrast to a classical intro into Haskell, we do not start with numbers, booleans, tuples, lists and strings, but we start with pictures. These are of course library-defined (hence the input CodeWorld) and not part of the language . But that does not make them less interesting, and in fact, even the basic boolean type is library defined it just happens to be the standard library.

This time around, I've highlighted the issue. I hope that made it easier for you to spot... Nonetheless, I got my answer. Don't reply if you're going to fight tooth and nail about such a basic fix; it's simply a waste of both of our time. I'd rather learn from somewhere else... On Tue, Aug 1, 2017 at 11:19 PM, Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript> wrote:

I am a bit reminded of Sean Spicer they re not my words! but clearly I am missing something. And indeed I am: In the code snippet, I wrote correctly import CodeWorld, but in the text I had input CodeWorld. I probably did write LaTeX before writing the lecture notes. Well, glad to have that sorted out. I fixed the mistake and wrote back:

From: Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript>
To: Sebastian R. < @gmail.com>
Betreff: Re: Errors Dear Sebastian, nobody is fighting, and I see the mistake now: The problem is not that the line is not in the code, the problem is that there is a typo in the line and I wrote input instead of import . Thanks for the report, although you did turn it into quite a riddle a simple you wrote import when it should have been import would have been a better user of both our time. Regards, Joachim Am Donnerstag, den 03.08.2017, 13:32 +1000 schrieb Sebastian R.:

(And it seems I now made the inverse typo, writing import instead of input . Anyways, I did not think of this any more until a few days later, when I found this nice message in my mailbox:

From: Sebastian R. < @gmail.com>
To: Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript>
Subject: Re: Errors

a simple you wrote import when it should have been import would have been a better user of both our time.

We're both programmers. How about I cut ALL of the unnecessary garbage and just tell you to s/import/input/ on that last quotation (the thing immediately before this paragraph, in case you didn't know). I blatantly quoted the error, like this:

In your introduction note you have written:

n contrast to a classical intro into Haskell, we do not start with numbers, booleans, tuples, lists and strings, but we start with pictures. These are of course library-defined (hence the input CodeWorld) and not part of the language . But that does not make them less interesting, and in fact, even the basic boolean type is library defined it just happens to be the standard library.

Howeverm there is no input CodeWorld in the code above.

Since that apparently wasn't clear enough, in my second email to you I had to highlight it like so:

You might note that I quoted this precisely from the lectures... and so I repeat myself... this came from your lectures; they're not my words!

In contrast to a classical intro into Haskell, we do not start with numbers, booleans, tuples, lists and strings, but we start with pictures. These are of course library-defined (hence the input CodeWorld) and not part of the language . But that does not make them less interesting, and in fact, even the basic boolean type is library defined it just happens to be the standard library.

This time around, I've highlighted the issue. I hope that made it easier for you to spot...

I'm not sure if you're memeing at me or not now, but it seems either your reading comprehension, or your logical deduction skills might be substandard. Unfortunately, there isn't much either of us can do about that, so I'm happy to accept that some people will be so stupid; after all, it's to be expected and if we don't accept that which is to be expected then we live our lives in denial. Happy to wrap up this discusson here, Seb... On Fri, Aug 4, 2017 at 12:22 AM, Joachim Breitner <noscript>joachim at cis dot upenn dot edu</noscript> wrote:

Well, I chose to be amused by this, and I am sharing my amusement with you.

Previously we talked about all the different kinds of descriptors which USB devices use to communicate their capability. This is important stuff because to write any useful USB device firmware we need to be able to determine how to populate our descriptors. However, having that data on the device is entirely worthless without an understanding of how it gets from the device to the host so that it can be acted upon. To understand that, let's look at the USB wire protocol.

Note, I'll again be talking mostly about USB2.0 low- and full-speed. I believe that high speed is approximately the same but with faster wires, except not quite that simple.

Down to the wire I don't intend to talk about the actual electrical signalling, though it's not un-reasonable for you to know that USB is a pair of wires forming a differentially signalled bidirectional serial communications link. The host is responsible for managing all the framing and timing on the link, and for formatting the communications into packets. There are a number of packet types which can appear on the USB link:

Packet type	Purpose
Token Packet	When the host wishes to send a message to the Control endpoint to configure the device, read data IN, or write data OUT, it uses this to start the transaction.
Data(0/1) Packet	Following a Setup, In, or Out token, a Data packet is a transfer of data (in either direction). The 0 and 1 alternate to provide a measure of confidence against lost packets.
Handshake Packet	Following a data packet of some kind, the other end may ACK the packet (all was well), NAK the packet (report that the device cannot, temporarily, send/receive data, or that an interrupt endpoint isn't triggered), or STALL the bus in which case the host needs to intervene.
Start of Frame	Every 1ms (full-speed) the host will send a SOF packet which carries a frame number. This can be used to help keep time on very simple devices. It also divides the bus into frames within which bandwidth is allocated.

As an example, when the host wishes to perform a control transfer, the following packets are transacted in turn:

1. Setup Token - The host addresses the device and endpoint (OUT0)
2. Data0 Packet - The host transmits a GET_DESCRIPTOR for the device descriptor
3. Ack Packet - The device acknowledges receipt of the request

This marks the end of the first transaction. The device decodes the GET_DESCRIPTOR request and prepares the device descriptor for transmission. The transmission occurs as the next transaction on the bus. In this example, we're assuming 8 byte maximum transmission sizes, for illustrative purposes.

1. In Token - The host addresses the device and endpoint (IN0)
2. Data1 Packet - The device transmits the first 8 bytes of the descriptor
3. Ack Packet - The host acknowledges the data packet
4. In Token - The host addresses the device and endpoint (IN0)
5. Data0 Packet - The device transmits the remaining 4 bytes of the descriptor (padded)
6. Ack Packet - The host acknowledges the data packet

The second transaction is now complete, and the host has all the data it needs to proceed. Finally a status transaction occurs in which:

1. Out Token - The host addresses the device and endpoint (OUT0)
2. Data1 Packet - The host transmits a 0 byte data packet to indicate successful completion
3. Ack Packet - The device acknowledges the completion, indicating its own satisfaction

And thus ends the full control transaction in which the host retrieves the device descriptor. From a high level, we need only consider the activity which occurs at the point of the acknowledgement packets. In the above example:

1. On the first ACK the device prepares IN0 to transmit the descriptor, readying whatever low level device stack there is with a pointer to the descriptor and its length in bytes.
2. On the second ACK the low levels are still thinking.
3. On the third ACK the transmission from IN0 is complete and the endpoint no longer expects to transfer data.
4. On the fourth ACK the control transaction is entirely complete.

Thinking at the low levels of the control interface Before we can build a high level USB stack, we need to consider the activity which might occur at the lower levels. At the low levels, particularly of the device control interface, work has to be done at each and every packet. The hardware likely deals with the token packet for us, leaving the data packets for us to process, and the resultant handshake packets will be likely handled by the hardware in response to our processing the data packets. Since every control transaction is initiated by a setup token, let's look at the setup requests which can come our way...

Setup Packet (Data) Format

Field Name	Byte start	Byte length	Encoding	Meaning
bmRequestType	0	1	Bitmap	Describes the kind of request, and the target of it. See below.
bRequest	1	1	Code	The request code itself, meanings of the rest of the fields vary by bRequest
wValue	2	2	Number	A 16 bit value whose meaning varies by request type
wIndex	4	2	Number	A 16 bit value whose meaning varies by request type but typically encodes an interface number or endpoint.
wLength	6	2	Number	A 16 bit value indicating the length of the transfer to come.

Since bRequest is essentially a switch against which multiple kinds of setup packet are selected between, here's the meanings of a few...

GET_DESCRIPTOR (Device) setup packet

Field Name	Value	Meaning
bmRequestType	0x08	Data direction is IN (from device to host), recipient is the device
bRequest	0x06	GET_DESCRIPTOR (in this instance, the device descriptor is requested)
wValue	0x0001	This means the device descriptor
wIndex	0x0000	Irrelevant, there's only 1 device descriptor anyway
wLength	12	This is the length of a device descriptor (12 bytes)

SET_ADDRESS to set a device's USB address

Field Name	Value	Meaning
bmRequestType	0x00	Data direction is OUT (from host to device), recipient is the device
bRequest	0x05	SET_ADDRESS (Set the device's USB address)
wValue	0x00nn	The address for the device to adopt (max 127)
wIndex	0x0000	Irrelevant for address setting
wLength	0	There's no data transfer expected for this setup operation

Most hardware blocks will implement an interrupt at the point that the Data packet following the Setup packet has been receive. This is typically called receiving a 'Setup' packet and then it's up to the device stack low levels to determine what to do and dispatch a handler. Otherwise an interrupt will fire for the IN or OUT tokens and if the endpoint is zero, the low level stack will handle it once more. One final thing worth noting about SET_ADDRESS is that it doesn't take effect until the completion of the zero-length "status" transaction following the setup transaction. As such, the status request from the host will still be sent to address zero (the default for new devices).

A very basic early "packet trace" This is an example, and is not guaranteed to be the packet sequence in all cases. It's a good indication of the relative complexity involved in getting a fresh USB device onto the bus though... When a device first attaches to the bus, the bus is in RESET state and so the first event a device sees is a RESET which causes it to set its address to zero, clear any endpoints, clear the configuration, and become ready for control transfers. Shortly after this, the device will become suspended. Next, the host kicks in and sends a port reset of around 30ms. After this, the host is ready to interrogate the device. The host sends a GET_DESCRIPTOR to the device, whose address at this point is zero. Using the information it receives from this, it can set up the host-side memory buffers since the device descriptor contains the maximum transfer size which the device supports. The host is now ready to actually 'address' the device, and so it sends another reset to the device, again around 30ms in length. The host sends a SET_ADDRESS control request to the device, telling it that its new address is nn. Once the acknowledgement has been sent from the host for the zero-data status update from the device, the device sets its internal address to the value supplied in the request. From now on, the device shall respond only to requests to nn rather than to zero. At this point, the host will begin interrogating further descriptors, looking at the configuration descriptors and the strings, to build its host-side representation of the device. These will be GET_DESCRIPTOR and GET_STRING_DESCRIPTOR requests and may continue for some time. Once the host has satisfied itself that it knows everything it needs to about the device, it will issue a SET_CONFIGURATION request which basically starts everything up in the device. Once the configuration is set, interrupt endpoints will be polled, bulk traffic will be transferred, Isochronous streams begin to run, etc.

Okay, but how do we make this concrete? So far, everything we've spoken about has been fairly abstract, or at least "soft". But to transfer data over USB does require some hardware. (Okay, okay, we could do it all virtualised, but there's no fun in that). The hardware I'm going to be using for the duration of this series is the STM32 on the blue-pill development board. This is a very simple development board which does (in theory at least) support USB device mode. If we view the schematic for the blue-pill, we can see a very "lightweight" USB interface which has a pullup resistor for D+. This is the way that a device signals to the host that it is present, and that it wants to speak at full-speed. If the pullup were on D- then it would be a low-speed device. High speed devices need a little more complexity which I'm not going to go into for today. The USB lines connect to pins PA11 and PA12 which are the USB pins on the STM32 on the board. Since USB is quite finicky, the STM32 doesn't let you remap that function elsewhere, so this is all looking quite good for us so far. The specific STM32 on the blue-pill is the STM32F103C8T6. By viewing its product page on ST's website we can find the reference manual for the part. Jumping to section 23 we learn that this STM32 supports full-speed USB2.0 which is convenient given the past article and a half. We also learn it supports up to eight endpoints active at any one time, and offers double-buffering for our bulk and isochronous transfers. It has some internal memory for packet buffering, so it won't use our RAM bandwidth while performing transfers, which is lovely. I'm not going to distill the rest of that section here, because there's a large amount of data which explains how the USB macrocell operates. However useful things to note are:

• How IN OUT and SETUP transfers work.
• How the endpoint buffer memory is configured.
• That all bus-powered devices MUST respond to suspend/resume properly
• That the hardware will prioritise endpoint interrupts for us so that we only need deal with the most pressing item at any given time.
• There is an 'Enable Function' bit in the address register which must be set or we won't see any transactions at all.
• How the endpoint registers signal events to the device firmware.

Next time, we're going to begin the process of writing a very hacky setup routine to try and initialise the USB device macrocell so that we can see incoming transactions through the ITM. It should be quite exciting, but given how complex this will be for me to learn, it might be a little while before it comes through.

I've made several changes to the Weblate Docker container which are worth mentioning today. First of all if you are still using nijel/weblate, you should switch to weblate/weblate. They both currently share same configuration, but it might happen that some future updates will go to the weblate owned container only. Now back to the container changes. Since beginning we were using Django built in server. That's fine for development purposes, but it really doesn't work that well in production as it can handle only one request at time. Therefore we've switched to more robust approach using nginx + uwsgi + supervisor. Thanks to this, the docker-compose no longer needs separate nginx server as everything is now sanely handled within the weblate container itself.

Filed under: Debian English Gammu phpMyAdmin SUSE Weblate

After fours years, I will again make it to DebConf, I'm looking forward to meet many great people, so if you want to meet and happen to be in Montreal next week come and say hello to me :-). It seems I've settled down on four year schedule - I've attended DebConf09 and DebConf13 so far. Let's see if next one will come in 2021 or earlier.

Filed under: Debian English Gammu phpMyAdmin Weblate

Here's what happened in the Reproducible Builds effort between Sunday July 23 and Saturday July 29 2017:

• Mattia posted an extensive status update from the Debian Reproducible Builds project to the debian-devel-announce mailing list. There were subsequent discussions on Hacker News and Reddit.
• A NASA presentation on "Real system failures" contained an interesting slide pertaining to reproducible builds.
• A 2015 article on Reproducible Builds ("How Debian Is Trying to Shut Down the CIA and Make Software Trustworthy Again") resurfaced in popularity, possibly due to the recent Wikileaks "Vault 7" exposure of malware whose effects can be ameliorated or detected by adopting Reproducible Builds.

Toolchain development and fixes

• Chris Lamb sent an experimental patch to apt to make the output of apt-ftparchive reproducible. Thanks to David Kalnischkies for reworking the result. (#869557)

Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #869578 filed against gdmap.
  • #869580 filed against teg.
  • #869583 filed against gnome-specimen.
  • #869884 filed against chemical-mime-data.
  • #870047 filed against imagemagick.
  • #870068 filed against kde4libs.
• Chris Lamb:
  • #869516 filed against libcdio.
  • A previous pull request against wheel was merged upstream.
• Helmut Grohne:
  • #869584 filed against fontconfig.
  • #869588 filed against libcap2.

Reviews of unreproducible packages 4 package reviews have been added, 2 have been updated and 24 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Aaron M. Ucko (1)
• Adrian Bunk (35)
• Helmut Grohne (4)
• Stefan Tatschner (1)

diffoscope development

• Juliana Oliveira Rodrigues:
  • Refactor XML comparator to use File.recognizes logic
  • Fix test_ico_image for identify >= 6.9.8
  • Fix test_no_android_manifest from test_apk
  • Fix test_android_manifest from test_apk
  • Fix test_xml_expected_diff output encoding to UTF-8
• Ximin Luo:
  • Improve method for detecting APKs
  • Factor common logic from various comparators into File.recognizes

Misc. This week's edition was written by Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

I m currently doing some embedded work on ARM systems. Having a virtual ARM environment is of course helpful. For the i586 class embedded systems that I run it s very easy to setup a virtual environment, I just have a chroot run from systemd-nspawn with the --personality=x86 option. I run it on my laptop for my own development and on a server my client owns so that they can deal with the hit by a bus scenario. I also occasionally run KVM virtual machines to test the boot image of i586 embedded systems (they use GRUB etc and are just like any other 32bit Intel system). ARM systems have a different boot setup, there is a uBoot loader that is fairly tightly coupled with the kernel. ARM systems also tend to have more unusual hardware choices. While the i586 embedded systems I support turned out to work well with standard Debian kernels (even though the reference OS for the hardware has a custom kernel) the ARM systems need a special kernel. I spent a reasonable amount of time playing with QEMU and was unable to make it boot from a uBoot ARM image. The Google searches I performed didn t turn up anything that helped me. If anyone has good references for getting QEMU to work for an ARM system image on an AMD64 platform then please let me know in the comments. While I am currently surviving without that facility it would be a handy thing to have if it was relatively easy to do (my client isn t going to pay me to spend a week working on this and I m not inclined to devote that much of my hobby time to it). QEMU for Process Emulation I ve given up on emulating an entire system and now I m using a chroot environment with systemd-nspawn. The package qemu-user-static has staticly linked programs for emulating various CPUs on a per-process basis. You can run this as /usr/bin/qemu-arm-static ./staticly-linked-arm-program . The Debian package qemu-user-static uses the binfmt_misc support in the kernel to automatically run /usr/bin/qemu-arm-static when an ARM binary is executed. So if you have copied the image of an ARM system to /chroot/arm you can run the following commands like the following to enter the chroot: cp /usr/bin/qemu-arm-static /chroot/arm/usr/bin/qemu-arm-static
chroot /chroot/arm bin/bash Then you can create a full virtual environment with /usr/bin/systemd-nspawn -D /chroot/arm if you have systemd-container installed. Selecting the CPU Type There is a huge range of ARM CPUs with different capabilities. How this compares to the range of x86 and AMD64 CPUs depends on how you are counting (the i5 system I m using now has 76 CPU capability flags). The default CPU type for qemu-arm-static is armv7l and I need to emulate a system with a armv5tejl. Setting the environment variable QEMU_CPU=pxa250 gives me armv5tel emulation. The ARM Architecture Wikipedia page [2] says that in armv5tejl the T stands for Thumb instructions (which I don t think Debian uses), the E stands for DSP enhancements (which probably isn t relevant for me as I m only doing integer maths), the J stands for supporting special Java instructions (which I definitely don t need) and I m still trying to work out what L means (comments appreciated). So it seems clear that the armv5tel emulation provided by QEMU_CPU=pxa250 will do everything I need for building and testing ARM embedded software. The issue is how to enable it. For a user shell I can just put export QEMU_CPU=pxa250 in .login or something, but I want to emulate an entire system (cron jobs, ssh logins, etc). I ve filed Debian bug #870329 requesting a configuration file for this [1]. If I put such a configuration file in the chroot everything would work as desired. To get things working in the meantime I wrote the below wrapper for /usr/bin/qemu-arm-static that calls /usr/bin/qemu-arm-static.orig (the renamed version of the original program). It s ugly (I would use a config file if I needed to support more than one type of CPU) but it works. #include <stdio.h>
#include <unistd.h>
#include <stdlib.h> int main(int argc, char **argv)

if(setenv("QEMU_CPU", "pxa250", 1))

printf("Can't set $QEMU_CPU\n");
return 1;

execv("/usr/bin/qemu-arm-static.orig", argv);
printf("Can't execute \"%s\" because of qemu failure\n", argv[0]);
return 1;

• [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=870329
• [2] https://en.wikipedia.org/wiki/ARM_architecture

Debian Long Term Support (LTS) This is my monthly working on Debian LTS. This time I worked on various hairy issues surrounding ca-certificates, unattended-upgrades, apache2 regressions, libmtp, tcpdump and ipsec-tools.

ca-certificates updates I've been working on the removal of the Wosign and StartCom certificates (Debian bug #858539) and, in general, the synchronisation of ca-certificates across suites (Debian bug #867461) since at least last march. I have made an attempt at summarizing the issue which led to a productive discussion and it seems that, in the end, the maintainer will take care of synchronizing information across suites. Guido was right in again raising the question of synchronizing NSS across all suites (Debian bug #824872) which itself raised the other question of how to test reverse dependencies. This brings me back to Debian bug #817286 which, basically proposed the idea of having "proposed updates" for security issues. The problem is while we can upload test packages to stable proposed-updates, we can't do the same in LTS because the suite is closed and we operate only on security packages. This issue came up before in other security upload and we need to think better about how to solve this.

unattended-upgrades Speaking of security upgrades brings me to the question of a bug (Debian bug #867169) that was filed against the wheezy version of unattended-upgrades, which showed that the package simply stopped working since the latest stable release, because wheezy became "oldoldstable". I first suggested using the "codename" but that appears to have been introduced only after wheezy. In the end, I proposed a simple update that would fix the configuration files and uploaded this as DLA-1032-1. This is thankfully fixed in later releases and will not require such hackery when jessie becomes LTS as well.

libmtp Next up is the work on the libmtp vulnerabilities (CVE-2017-9831 and CVE-2017-9832). As I described in my announcement, the work to backport the patch was huge, as upstream basically backported a whole library from the gphoto2 package to fix those issues (and probably many more). The lack of a test suite made it difficult to trust my own work, but given that I had no (negative) feedback, I figured it was okay to simply upload the result and that became DLA-1029-1.

tcpdump I then looked at reproducing CVE-2017-11108, a heap overflow triggered tcpdump would parse specifically STP packets. In Debian bug #867718, I described how to reproduce the issue across all suites and opened an issue upstream, given that the upstream maintainers hadn't responded responded in weeks according to notes in the RedHat Bugzilla issue. I eventually worked on a patch which I shared upstream, but that was rejected as they were already working on it in their embargoed repository. I can explain this confusion and duplication of work with:

1. the original submitter didn't really contact security@tcpdump.org
2. he did and they didn't reply, being just too busy
3. they replied and he didn't relay that information back

I think #2 is most likely: the tcpdump.org folks are probably very busy with tons of reports like this. Still, I should probably have contacted security@tcpdump.org directly before starting my work, even though no harm was done because I didn't divulge issues that were already public. Since then, tcpdump has released 4.9.1 which fixes the issue, but then new CVEs came out that will require more work and probably another release. People looking into this issue must be certain to coordinate with the tcpdump security team before fixing the actual issues.

ipsec-tools Another package that didn't quite have a working solution is the ipsec-tools suite, in which the racoon daemon was vulnerable to a remotely-triggered DOS attack (CVE-2016-10396). I reviewed and fixed the upstream patch which introduced a regression. Unfortunately, there is no test suite or proof of concept to control the results. The reality is that ipsec-tools is really old, and should maybe simply be removed from Debian, in favor of strongswan. Upstream hasn't done a release in years and various distributions have patched up forks of those to keep it alive... I was happy, however, to know that a maintainer will take care of updating the various suites, including LTS, with my improved patch. So this fixes the issue for now, but I would strongly encourage users to switch away from ipsec-tools in the future.

apache2 Finally, I was bitten by the old DLA-841-1 upload I did all the way back in February, as it introduced a regression (Debian bug #858373). It turns out it was possible to segfault Apache workers with a trivial HTTP request, in certain (rather exotic, I might add) configurations (ErrorDocument 400 directive pointing to a cgid script in worker mode). Still, it was a serious regression and I found a part of the nasty long patch we worked on back then that was faulty, and introduced a small fix to correct that. The proposed package unfortunately didn't yield any feedback, and I can only assume it will work okay for people. The result is the DLA-841-2 upload which fixes the regression. I unfortunately didn't have time to work on the remaining CVEs affecting apache2 in LTS at the time of writing.

Triage I also did some miscellaneous triage by filing Debian bug #867477 for poppler in an effort to document better the pending issue. Next up was some minor work on eglibc issues. CVE-2017-8804 has a patch, but it's been disputed. since the main victim of this and the core of the vulnerability (rpcbind) has already been fixed, I am not sure this vulnerability is still a thing in LTS at all. I also looked at CVE-2014-9984, but the code is so different in wheezy that I wonder if LTS is affected at all. Unfortunately, the eglibc gymnastics are a little beyond me and I do not feel confident enough to just push those issues aside for now and let them open for others to look at.

Other free software work And of course, there's my usual monthly volunteer work. My ratio is a little better this time, having reached an about even ratio between paid and volunteer work, whereas this was 60% volunteer work in march.

Announcing ecdysis I recently published ecdysis, a set of template and code samples that I frequently reuse across project. This is probably the least pronounceable project name I have ever chosen, but this is somewhat on purpose. The goal of this project is not collaboration or to become a library: it's just a personal project which I share with the world as a curiosity. To quote the README file:

The name comes from what snakes and other animals do to "create a new snake": they shed their skin. This is not so appropriate for snakes, as it's just a way to rejuvenate their skin, but is especially relevant for anthropods since then "ecdysis" may be associated with a metamorphosis:

Ecdysis is the moulting of the cuticle in many invertebrates of the clade Ecdysozoa. Since the cuticle of these animals typically forms a largely inelastic exoskeleton, it is shed during growth and a new, larger covering is formed. The remnants of the old, empty exoskeleton are called exuviae. Wikipedia

So this project is metamorphosed into others when the documentation templates, code examples and so on are reused elsewhere. For that reason, the license is an unusally liberal (for me) MIT/Expat license. The name also has the nice property of being absolutely unpronounceable, which makes it unlikely to be copied but easy to search online.

It was an interesting exercise to go back into older projects and factor out interesting code. The process is not complete yet, as there are older projects I'm still curious in reviewing. A bunch of that code could also be factored into upstream project and maybe even the Python standard library. In short, this is stuff I keep on forgetting how to do: a proper setup.py config, some fancy argparse extensions and so on. Instead of having to remember where I had written that clever piece of code, I now shove it in the crazy chaotic project where I can find it again in the future.

Beets experiments Since I started using Subsonic (or Libresonic) to manage the music on my phone, album covers are suddenly way more interesting. But my collection so far has had limited album covers: my other media player (gmpc) would download those on the fly on its own and store them in its own database - not on the filesystem. I guess this could be considered to be a limitation of Subsonic, but I actually appreciate the separation of duty here. Garbage in, garbage out: the quality of Subsonic's rendering depends largely on how well setup your library and tags are. It turns out there is an amazing tool called beets to do exactly that kind of stuff. I originally discarded that "media library management system for obsessive-compulsive [OC] music geeks", trying to convince myself i was not an "OC music geek". Turns out I am. Oh well. Thanks to beets, I was able to download album covers for a lot of the albums in my collection. The only covers that are missing now are albums that are not correctly tagged and that beets couldn't automatically fix up. I still need to go through those and fix all those tags, but the first run did an impressive job at getting album covers. Then I got the next crazy idea: after a camping trip where we forgot (again) the lyrics to Georges Brassens, I figured I could start putting some lyrics on my ebook reader. "How hard can that be?" of course, being the start of another crazy project. A pull request and 3 days later, I had something that could turn a beets lyrics database into a Sphinx document which, in turn, can be turned into an ePUB. In the process, I probably got blocked from MusixMatch a hundred times, but it's done. Phew! The resulting e-book is about 8000 pages long, but is still surprisingly responsive. In the process, I also happened to do a partial benchmark of Python's bloom filter libraries. The biggest surprise there was the performance of the set builtin: for small items, it is basically as fast as a bloom filter. Of course, when the item size grows larger, its memory usage explodes, but in this case it turned out to be sufficient and bloom filter completely overkill and confusing. Oh, and thanks to those efforts, I got admitted in the beetbox organization on GitHub! I am not sure what I will do with that newfound power: I was just scratching an itch, really. But hopefully I'll be able to help here and there in the future as well.

Debian package maintenance I did some normal upkeep on a bunch of my packages this month, that were long overdue:

• uploaded slop 6.3.47-1: major new upstream release
• uploaded an NMU for maim 5.4.64-1.1: maim was broken by the slop release
• uploaded pv 1.6.6-1: new upstream release
• uploaded kedpm 1.0+deb8u1 to jessie (oldstable): one last security fix (Debian bug #860817, CVE-2017-8296) for that derelict password manager
• uploaded charybdis 3.5.5-1: new minor upstream release, with optional support for mbedtls
• filed Debian bug #866786 against cryptsetup to make the remote initramfs SSH-based unlocking support multiple devices: thanks to the maintainer, this now works flawlessly in buster and may be backported to stretch
• expanded on Debian bug #805414 against gdm3 and Debian bug #845938 against pulseaudio, because I had trouble connecting my computer to this new Bluetooth speaker. turns out this is a known issue in Pulseaudio: whereas it releases ALSA devices, it doesn't release Bluetooth devices properly. Documented this more clearly in the wiki page
• filed Debian bug #866790 regarding old stray Apparmor profiles that were lying around my system after an upgrade, which got me interested in Debian bug #830502 in turn
• filed Debian bug #868728 against cups regarding a weird behavior I had interacting with a network printer. turns out the other workstation was misconfigured... why are printers still so hard?
• filed Debian bug #870102 to automate sbuild schroots upgrades
• after playing around with rash tried to complete the packaging (Debian bug #754972) of percol with this pull request upstream. this ended up to be way too much overhead and I reverted to my old normal history habits.

I decided to try packaging Mesos for Debian/Stretch. I had a spare system with a i7-930 CPU, 48G of RAM, and SSDs to use for building. The i7-930 isn t really fast by today s standards, but 48G of RAM and SSD storage mean that overall it s a decent build system faster than most systems I run (for myself and for clients) and probably faster than most systems used by Debian Developers for build purposes. There s a github issue about the lack of an upstream package for Debian/Stretch [1]. That upstream issue could probably be worked around by adding Jessie sources to the APT sources.list file, but a package for Stretch is what is needed anyway. Here is the documentation on building for Debian [2]. The list of packages it gives as build dependencies is incomplete, it also needs zlib1g-dev libapr1-dev libcurl4-nss-dev openjdk-8-jdk maven libsasl2-dev libsvn-dev. So BUILDING this software requires Java + Maven, Ruby, and Python along with autoconf, libtool, and all the usual Unix build tools. It also requires the FPM (Fucking Package Management) tool, I take the choice of name as an indication of the professionalism of the author. Building the software on my i7 system took 79 minutes which includes 76 minutes of CPU time (I didn t use the -j option to make). At the end of the build it turned out that I had mistakenly failed to install the Fucking Package Management gem and it aborted. At this stage I gave up on Mesos, the pain involved exceeds my interest in trying it out. How to do it Better One of the aims of Free Software is that bugs are more likely to get solved if many people look at them. There aren t many people who will devote 76 minutes of CPU time on a moderately fast system to investigate a single bug. To deal with this software should be prepared as components. An example of this is the SE Linux project which has 13 source modules in the latest release [3]. Of those 13 only 5 are really required. So anyone who wants to start on SE Linux from source (without considering a distribution like Debian or Fedora that has it packaged) can build the 5 most important ones. Also anyone who has an issue with SE Linux on their system can find the one source package that is relevant and study it with a short compile time. As an aside I ve been working on SE Linux since long before it was split into so many separate source packages and know the code well, but I still find the separation convenient I rarely need to work on more than a small subset of the code at one time. The requirement of Java, Ruby, and Python to build Mesos could be partly due to language interfaces to call Mesos interfaces from Ruby and Python. Ohe solution to that is to have the C libraries and header files to call Mesos and have separate packages that depend on those libraries and headers to provide the bindings for other languages. Another solution is to have autoconf detect that some languages aren t installed and just not try to compile bindings for them (this is one of the purposes of autoconf). The use of a tool like Fucking Package Management means that you don t get help from experts in the various distributions in making better packages. When there is a FOSS project with a debian subdirectory that makes barely functional packages then you will be likely to have an experienced Debian Developer offer a patch to improve it (I ve offered patches for such things on many occasions). When there is a FOSS project that uses a tool that is never used by Debian developers (or developers of Fedora and other distributions) then the only patches you will get will be from inexperienced people. A software build process should not download anything from the Internet. The source archive should contain everything that is needed and there should be dependencies for external software. Any downloads from the Internet need to be protected from MITM attacks which means that a responsible software developer has to read through the build system and make sure that appropriate PGP signature checks etc are performed. It could be that the files that the Mesos build downloaded from the Apache site had appropriate PGP checks performed but it would take me extra time and effort to verify this and I can t distribute software without being sure of this. Also reproducible builds are one of the latest things we aim for in the Debian project, this means we can t just download files from web sites because the next build might get a different version. Finally the fpm (Fucking Package Management) tool is a Ruby Gem that has to be installed with the gem install command. Any time you specify a gem install command you should include the -v option to ensure that everyone is using the same version of that gem, otherwise there is no guarantee that people who follow your documentation will get the same results. Also a quick Google search didn t indicate whether gem install checks PGP keys or verifies data integrity in other ways. If I m going to compile software for other people to use I m concerned about getting unexpected results with such things. A Google search indicates that Ruby people were worried about such things in 2013 but doesn t indicate whether they solved the problem properly.

• [1] https://github.com/mesosphere/mesos-deb-packaging/issues/109
• [2] https://github.com/mesosphere/mesos-deb-packaging
• [3] https://github.com/SELinuxProject/selinux/wiki/Releases

Lintian is static analysis tool for Debian packages, reporting on various errors, omissions and quality-assurance issues to the maintainer. I seem to have found myself hacking on it a bit more recently (see my previous installment). In particular, here's the code of mine which made for a total of 20 bugs closed that made it into the recent 2.5.52 release:

---

---

---

Is it possible to do software development, mathematical or not, as mathematician in academics? This is a question I was asking myself recently a lot, seeing my own development from logician at a state university getting rid of foreigners to software developer. And then, a friend pointed me to this very depressing document: The origins of SageMath by William Stein, the main developer of SageMath. And I realized that it seems to be a global phenomenon that mathematicians who are interested in software development have to leave academics. What a sad affair. SageMath has a clear mission:

Creating a viable free open source alternative to Magma, Maple, Mathematica and Matlab.

All the Ma -software packages are commercial, and expensive. On the other hand they often have very good algorithms implemented. The Sage developers invested lots of time, energy, and brain power to develop excellent algorithm in an open source project for the mathematical researcher, but this investment wasn t honored in academic life. To quote from the presentation:

Issues with software dev in academia

• Hard money for software development is virtually nonexistent: I can t think of anyone I know who got tenured based on his or her software.
• Researchers on soft money are systematically discriminated against in favor of tenure-track and tenured faculty.
• Researchers are increasingly evaluated solely on bibliometric counts rather than an informed assessment of their overall portfolio of papers, code, software, industry engagement, or student supervision.

The origins of SageMath, p.31

I can fully agree to this. Both from my own experience as well as from those around me. The presentation slides are full of other examples, from the developers of NumPy, Jupyter, as well as statements by Stephen Wolfram from Mathematica about this issue. A textbook how to not setup academia. My assumption was that this hits only on non-tenured staff, the academic precariat. It is shocking to see that even William Stein with a tenure position is leaving academics. It seems the times are not ready

Every great open source math library is built on the ashes of someone s academic career.
The origins of SageMath, p.32

Here's what happened in the Reproducible Builds effort between Sunday July 16 and Saturday July 22 2017: Toolchain development Bernhard M. Wiedemann wrote a tool to automatically run through different sources of non-determinism, and report which of these caused irreproducibility. Dan Kegel's patches to fpm were merged. Bugs filed Patches submitted upstream:

• Bernhard M. Wiedemann:
  • Sort file lists:
    • eric5 merged
    • libsass-python merged
    • tcl merged
    • python3.6 in progress
    • drbd
    • blobwars
  • Instead of fixing ordering issues in a custom .pak archive (same as blobwars above), we allow to install individual data files to avoid the issue:
    • edgar avoid sort
  • Omit the build date entirely:
    • fence-agents
  • SOURCE_DATE_EPOCH support:
    • criu, merged
    • dapl, merged
    • shorewall, merged
    • youtube-dl in progress
    • automake
    • crosstool-ng
    • docker
    • drbd
    • drbd-utils
    • getdp
    • infinipath-psm
    • opa-fm
    • opa-psm2
    • texinfo
  • geany/glfw unknown

Patches filed in Debian:

• Adrian Bunk:
  • #868599 filed against ocaml-curses.
  • #868609 filed against le.
  • #868612 filed against mixxx.
  • #868855 filed against softhsm2.
  • #868858 filed against gwc.
  • #869086 filed against dsniff.
• Chris Lamb:
  • #868790 filed against castle-game-engine, forwarded upstream.
  • #868843 filed against xorg-server, forwarded upstream.
  • #869516 filed against libcdio.
• Drew Parsons:
  • #868505 filed against sdpa.
• Lucas Nussbaum:
  • #868904 filed against gwc.
  • #868927 filed against python-pybedtools.
• Sascha Steinbiss:
  • #868772 filed against ragel.

Reviews of unreproducible packages 73 package reviews have been added, 44 have been updated and 50 have been removed in this week, adding to our knowledge about identified issues. No issue types were updated. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (106)
• Daniel Stender (1)
• Drew Parsons (1)
• F lix Sipma (1)
• Lucas Nussbaum (25)

diffoscope development

• Juliana Rodrigues:
  • Add new XML comparator. (Closes: #866120)
• Guangyuan Yang:
  • Fix 2 cases in test_device on FreeBSD
• Chris Lamb:
  • comparators.xml: Fix EPUB "missing file" tests; they ship a META-INF/container.xml file.
  • comparators.sqlite: Simplify file detection in Sqlite3Database.RE_FILE_TYPE
  • Style and attribution fixes to XML comparator and comparators.directory
• Ximin Luo:
  • main, logging: restore old logger settings to avoid pytest vomiting in certain situations
  • comparators/directory: Fix #868534 by expecting less strict test output

reprotest development

• Ximin Luo:
  • Use autopkgtest upstream paths, makes things easier to import
  • Add script for importing autopkgtest code, and import autopkgtest 4.4

Ximin also restarted the discussion with autopkgtest-devel about code reuse for reprotest. Santiago Torres began a series of patches to make reprotest more distro-agnostic, with the aim of making it usable on Arch Linux. Ximin reviewed these patches. Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.