Weblate 2.16 has been released today while I'm at DebConf17. There are quite some performance improvements (and more of that is scheduled for 2.17), new file formats support and various other improvements. Full list of changes:

• Various performance improvements.
• Added support for nested JSON format.
• Added support for WebExtension JSON format.
• Fixed git exporter authentication.
• Improved CSV import in certain situations.
• Improved look of Other translations widget.
• The max-length checks is now enforcing length of text in form.
• Make the commit_pending age configurable per component.
• Various user interface cleanups.
• Fixed component/project/sitewide search for translations.

If you are upgrading from older version, please follow our upgrading instructions. You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects. Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English SUSE Weblate

mksh R56 was released with experimental fixes for the history no longer persisted when HISTFILE near-full and interactive shell cannot wait on coprocess by PID issues (I hope they do not introduce any regressioins) and otherwise as a bugfix release. You might wish to know the $EDITOR selection mechanism in dot.mkshrc changed. Some more alias characters are allowed again, and POSIX character classes (for ASCII, and EBCDIC, only) appeared by popular vote. mksh now has a FAQ; enjoy. Do feel free to contribute (answers, too, of course). The jupp text editor has also received a new release; asides from being much smaller, and updated (mksh too, btw) to Unicode 10, and some segfault fixes, it features falling back to using /dev/tty if stdin or stdout is not a terminal (for use on GNU with find xargs jupp, since they don t have our xargs(1) -o option yet), a new command to exit nonzero (sometimes, utilities invoking the generic visual editor need this), and presentation mode . Presentation mode, crediting Natureshadow, is basically putting your slides as (UTF-8, with fancy stuff inside) plaintext files into one directory, with sorting names (so e.g. zero-padded slide numbers as filenames), presenting them with jupp * in a fullscreen xterm. You d hit F6 to switch to one-file view first, then present by using F8 to go forward (F7 to go backward), and, for demonstrations, F9 to pipe the entire slide through an external command (could be just sh ) offering the previous one as default. Simple yet powerful; I imagine Sven Guckes would love it, were he not such a vim user. The new release is offered as source tarball (as usual) and in distribution packages, but also, again, a Win32 version as PKZIP archive (right-click on setup.inf and hit I nstall to install it). Note that this comes with its own (thankfully local) version of the Cygwin32 library (compatible down to Windows 95, apparently), so if you have Cygwin installed yourself you re better off compiling it there and using your own version instead. I ve also released a new DOS version of 2.8 with no code patches but an updated jupprc; the binary (self-extracting LHarc archive) this time comes with all resource files, not just jupp s. Today, the jupprc drop-in file for JOE 3.7 got a matching update (and some fixes for bugs discovered during that) and I added a new one for JOE 4.4 (the former being in Debian wheezy, the latter in jessie, stretch and buster/sid). It s a bit rudimentary (the new shell window functionality is absent) but, mostly, gives the desired jupp feeling, more so than just using stock jstar would.

• source tarballs
• Win32 binaries
• DOS binaries
• drop-in jupprc for JOE 2.8, or to update jupp 2.8
• drop-in jupprc for JOE 3.7
• drop-in jupprc for JOE 4.4

CVS ability to commit to multiple branches of a file at the same time, therefore grouping the commit (by commitid at least, unsure if cvsps et al. can be persuaded to recognise it). If you don t know what cvs(GNU) is: it is a proper (although not distributed) version control system and the best for centralised tasks. (For decentral tasks, abusing git as pseudo-VCS has won by popularity vote; take this as a comparison.) If desired, I can make these new versions available in my WTF APT repository on request. (Debian buster/sid users: please change https to http there, the site is only available with TLSv1.0 as it doesn t require bank-level security.) I d welcome it very much if people using an OS which does not yet carry either to package it there. Message me when one more is added, too In unrelated news I uploaded MuseScore 2.1 to Debian unstable, mostly because the maintainers are busy (though I could comaintain it if needed, I d just need help with the C++ and CMake details). Bonus side effect is that I can now build 2.2~ test versions with patches of mine added I plan to produce to fix some issues (and submit upstream) (read more )

reading up on rapidjson. I was reading the docs for rapidjson performance and I like that source buffer is destroyed for performance. I was wrinting JSON parser myself and performance bottleneck seems to be copying and constructing objects.

Previously we talked about all the different kinds of descriptors which USB devices use to communicate their capability. This is important stuff because to write any useful USB device firmware we need to be able to determine how to populate our descriptors. However, having that data on the device is entirely worthless without an understanding of how it gets from the device to the host so that it can be acted upon. To understand that, let's look at the USB wire protocol.

Note, I'll again be talking mostly about USB2.0 low- and full-speed. I believe that high speed is approximately the same but with faster wires, except not quite that simple.

Down to the wire I don't intend to talk about the actual electrical signalling, though it's not un-reasonable for you to know that USB is a pair of wires forming a differentially signalled bidirectional serial communications link. The host is responsible for managing all the framing and timing on the link, and for formatting the communications into packets. There are a number of packet types which can appear on the USB link:

Packet type	Purpose
Token Packet	When the host wishes to send a message to the Control endpoint to configure the device, read data IN, or write data OUT, it uses this to start the transaction.
Data(0/1) Packet	Following a Setup, In, or Out token, a Data packet is a transfer of data (in either direction). The 0 and 1 alternate to provide a measure of confidence against lost packets.
Handshake Packet	Following a data packet of some kind, the other end may ACK the packet (all was well), NAK the packet (report that the device cannot, temporarily, send/receive data, or that an interrupt endpoint isn't triggered), or STALL the bus in which case the host needs to intervene.
Start of Frame	Every 1ms (full-speed) the host will send a SOF packet which carries a frame number. This can be used to help keep time on very simple devices. It also divides the bus into frames within which bandwidth is allocated.

As an example, when the host wishes to perform a control transfer, the following packets are transacted in turn:

1. Setup Token - The host addresses the device and endpoint (OUT0)
2. Data0 Packet - The host transmits a GET_DESCRIPTOR for the device descriptor
3. Ack Packet - The device acknowledges receipt of the request

This marks the end of the first transaction. The device decodes the GET_DESCRIPTOR request and prepares the device descriptor for transmission. The transmission occurs as the next transaction on the bus. In this example, we're assuming 8 byte maximum transmission sizes, for illustrative purposes.

1. In Token - The host addresses the device and endpoint (IN0)
2. Data1 Packet - The device transmits the first 8 bytes of the descriptor
3. Ack Packet - The host acknowledges the data packet
4. In Token - The host addresses the device and endpoint (IN0)
5. Data0 Packet - The device transmits the remaining 4 bytes of the descriptor (padded)
6. Ack Packet - The host acknowledges the data packet

The second transaction is now complete, and the host has all the data it needs to proceed. Finally a status transaction occurs in which:

1. Out Token - The host addresses the device and endpoint (OUT0)
2. Data1 Packet - The host transmits a 0 byte data packet to indicate successful completion
3. Ack Packet - The device acknowledges the completion, indicating its own satisfaction

And thus ends the full control transaction in which the host retrieves the device descriptor. From a high level, we need only consider the activity which occurs at the point of the acknowledgement packets. In the above example:

1. On the first ACK the device prepares IN0 to transmit the descriptor, readying whatever low level device stack there is with a pointer to the descriptor and its length in bytes.
2. On the second ACK the low levels are still thinking.
3. On the third ACK the transmission from IN0 is complete and the endpoint no longer expects to transfer data.
4. On the fourth ACK the control transaction is entirely complete.

Thinking at the low levels of the control interface Before we can build a high level USB stack, we need to consider the activity which might occur at the lower levels. At the low levels, particularly of the device control interface, work has to be done at each and every packet. The hardware likely deals with the token packet for us, leaving the data packets for us to process, and the resultant handshake packets will be likely handled by the hardware in response to our processing the data packets. Since every control transaction is initiated by a setup token, let's look at the setup requests which can come our way...

Setup Packet (Data) Format

Field Name	Byte start	Byte length	Encoding	Meaning
bmRequestType	0	1	Bitmap	Describes the kind of request, and the target of it. See below.
bRequest	1	1	Code	The request code itself, meanings of the rest of the fields vary by bRequest
wValue	2	2	Number	A 16 bit value whose meaning varies by request type
wIndex	4	2	Number	A 16 bit value whose meaning varies by request type but typically encodes an interface number or endpoint.
wLength	6	2	Number	A 16 bit value indicating the length of the transfer to come.

Since bRequest is essentially a switch against which multiple kinds of setup packet are selected between, here's the meanings of a few...

GET_DESCRIPTOR (Device) setup packet

Field Name	Value	Meaning
bmRequestType	0x08	Data direction is IN (from device to host), recipient is the device
bRequest	0x06	GET_DESCRIPTOR (in this instance, the device descriptor is requested)
wValue	0x0001	This means the device descriptor
wIndex	0x0000	Irrelevant, there's only 1 device descriptor anyway
wLength	12	This is the length of a device descriptor (12 bytes)

SET_ADDRESS to set a device's USB address

Field Name	Value	Meaning
bmRequestType	0x00	Data direction is OUT (from host to device), recipient is the device
bRequest	0x05	SET_ADDRESS (Set the device's USB address)
wValue	0x00nn	The address for the device to adopt (max 127)
wIndex	0x0000	Irrelevant for address setting
wLength	0	There's no data transfer expected for this setup operation

Most hardware blocks will implement an interrupt at the point that the Data packet following the Setup packet has been receive. This is typically called receiving a 'Setup' packet and then it's up to the device stack low levels to determine what to do and dispatch a handler. Otherwise an interrupt will fire for the IN or OUT tokens and if the endpoint is zero, the low level stack will handle it once more. One final thing worth noting about SET_ADDRESS is that it doesn't take effect until the completion of the zero-length "status" transaction following the setup transaction. As such, the status request from the host will still be sent to address zero (the default for new devices).

A very basic early "packet trace" This is an example, and is not guaranteed to be the packet sequence in all cases. It's a good indication of the relative complexity involved in getting a fresh USB device onto the bus though... When a device first attaches to the bus, the bus is in RESET state and so the first event a device sees is a RESET which causes it to set its address to zero, clear any endpoints, clear the configuration, and become ready for control transfers. Shortly after this, the device will become suspended. Next, the host kicks in and sends a port reset of around 30ms. After this, the host is ready to interrogate the device. The host sends a GET_DESCRIPTOR to the device, whose address at this point is zero. Using the information it receives from this, it can set up the host-side memory buffers since the device descriptor contains the maximum transfer size which the device supports. The host is now ready to actually 'address' the device, and so it sends another reset to the device, again around 30ms in length. The host sends a SET_ADDRESS control request to the device, telling it that its new address is nn. Once the acknowledgement has been sent from the host for the zero-data status update from the device, the device sets its internal address to the value supplied in the request. From now on, the device shall respond only to requests to nn rather than to zero. At this point, the host will begin interrogating further descriptors, looking at the configuration descriptors and the strings, to build its host-side representation of the device. These will be GET_DESCRIPTOR and GET_STRING_DESCRIPTOR requests and may continue for some time. Once the host has satisfied itself that it knows everything it needs to about the device, it will issue a SET_CONFIGURATION request which basically starts everything up in the device. Once the configuration is set, interrupt endpoints will be polled, bulk traffic will be transferred, Isochronous streams begin to run, etc.

Okay, but how do we make this concrete? So far, everything we've spoken about has been fairly abstract, or at least "soft". But to transfer data over USB does require some hardware. (Okay, okay, we could do it all virtualised, but there's no fun in that). The hardware I'm going to be using for the duration of this series is the STM32 on the blue-pill development board. This is a very simple development board which does (in theory at least) support USB device mode. If we view the schematic for the blue-pill, we can see a very "lightweight" USB interface which has a pullup resistor for D+. This is the way that a device signals to the host that it is present, and that it wants to speak at full-speed. If the pullup were on D- then it would be a low-speed device. High speed devices need a little more complexity which I'm not going to go into for today. The USB lines connect to pins PA11 and PA12 which are the USB pins on the STM32 on the board. Since USB is quite finicky, the STM32 doesn't let you remap that function elsewhere, so this is all looking quite good for us so far. The specific STM32 on the blue-pill is the STM32F103C8T6. By viewing its product page on ST's website we can find the reference manual for the part. Jumping to section 23 we learn that this STM32 supports full-speed USB2.0 which is convenient given the past article and a half. We also learn it supports up to eight endpoints active at any one time, and offers double-buffering for our bulk and isochronous transfers. It has some internal memory for packet buffering, so it won't use our RAM bandwidth while performing transfers, which is lovely. I'm not going to distill the rest of that section here, because there's a large amount of data which explains how the USB macrocell operates. However useful things to note are:

• How IN OUT and SETUP transfers work.
• How the endpoint buffer memory is configured.
• That all bus-powered devices MUST respond to suspend/resume properly
• That the hardware will prioritise endpoint interrupts for us so that we only need deal with the most pressing item at any given time.
• There is an 'Enable Function' bit in the address register which must be set or we won't see any transactions at all.
• How the endpoint registers signal events to the device firmware.

Next time, we're going to begin the process of writing a very hacky setup routine to try and initialise the USB device macrocell so that we can see incoming transactions through the ITM. It should be quite exciting, but given how complex this will be for me to learn, it might be a little while before it comes through.

In the coming weeks and months I intend to write a mini series about packaging Java software for Debian. The following article basically starts in the middle of this journey because the PDFsam upgrade is still fresh in my mind. It requires some preexisting knowledge about build tools like Maven and some Java terminology. But do not fear. Hopefully it will make sense in the end when all pieces fall into place. A month ago I decided to upgrade PDFsam, a Java application to split, merge, extract, mix and rotate PDF documents. The current version 1.1.4 is already seven years old and uses Ant as its build system. Unfortunately up to now nobody was interested enough to invest the time to upgrade it to the latest version. A quick internet search unveils that the current sources can be found on github.com. Another brief look reveals we are dealing with a Maven project here because we can find a pom.xml file in the root directory and there is no sign of Ant s typical build.xml file anymore. Here are some general tips how to proceed from this point by using the PDFsam upgrade as an example. Find out how many new dependencies you really need The pom.xml file declares its dependencies in the <dependencies> section. It is good practice to inspect the pom.xml file and determine how much work will be required to upgrade the package. A seasoned Java packager will quickly find common dependencies like Hibernate or the Apache Commons libraries. Fortunately for you they are already packaged in Debian because a lot of projects depend on them. If you are unsure what is and what is not packaged for Debian, tracker.debian.org and codesearch.debian.net are useful tools to search for those packages. If in doubt just ask on debian-java@lists.debian.org. There is no automagical tool (yet) to find out what dependencies are really new (we talk about mh_make soon) but if you use the aforementioned tools and websites you will notice that in June 2017 one could not find the following artifacts: fontawesomefx, eventstudio, sejda-* and jackson-jr-objects. There are also jdepend and testFx but notice they are marked as <scope>test</scope> meaning they are only required if you would like to run upstream s test suite as well. For the sake of simplicity, it is best to ignore them for now and to focus on packaging only dependencies which are really needed to compile the application. Test dependencies can always be added later. This pom.xml investigation leads us to the following conclusion: PDFsam depends on Sejda, a PDF library. Basically Sejda is the product of a major refactoring that happened years ago and allows upstream to develop PDFsam faster and in multiple directions. For Debian packagers it is quite clear now that the upgrade of PDFsam is in reality more like packaging a completely new application. The inspection of Sejda s pom.xml file (another Maven project) reveals we also have to package imgscalr, Twelvemonkeys and SAMBox. We continue with these pom.xml analyses and end up with these new source packages: jackson-jr, libimgscalr-java, libsambox-java, libsejda-java, libsejda-injector-java, libsejda-io-java, libsejda-eventstudio-java, libtwelvemonkeys-java, fontawesomefx and libpdfbox2-java. Later I discovered that gettext-maven-plugin was also required. This was not obvious at first glance if you only check the pom.xml in the root directory but PDFsam and Sejda are multi-module projects! In this case every subdirectory (module) contains another pom.xml with additional information, so ideally you should check those too before you decide to start with your packaging. But don t worry it is often possible to ignore modules with a simple ignore rule inside your debian/*.poms file. The package will have less functionality but it can be still useful if you only need a subset of the modules. Of course in this case ignoring the gettext-maven-plugin artifact would result in a runtime error. C est la vie. A brief remark about Java package names: Java library packages must be named like libXXX-java. This is important for binary packages to avoid naming collisions. We are more tolerant when it comes to source package names but in general we recommend to use the exact same name as for the binary package. There are exceptions like prefixing source packages with their well known project name like jackson-XXX or jboss-XXX but this should only be used when there are already existing packages that use such a naming scheme. If in doubt, talk to us. mh_make or how to quickly generate an initial debian directory Packaging a Maven library is usually not very difficult even if it consists of multiple modules. The tricky part is to get the maven.rules, maven.IgnoreRules and your *.poms file right but debian/rules often only consists of a single dh line and the rest is finding the build-dependencies and adding them to debian/control. A small tool called mh_make, which is included in maven-debian-helper, can lend you a helping hand. The tool is not perfect yet. It requires that most build-dependencies are already installed on your local system, otherwise it won t create the initial debian directory and will only produce some unfinished (but in some cases still useful) files. A rule of thumb is to start with a package that does not depend on any other new dependency and requires the fewest build-dependencies. I have chosen libtwelvemonkeys-java because it was the simplest package and met the aforementioned criteria. Here is how mh_make looks like in action. (The animated GIF was created with Byzanz) First of all download the release tarball, unpack it and run mh_make inside the root directory. Ok, what is happening here? First you can choose a source and binary package name. Then disable the tests and don t run javadoc to create the documentation. This will simplify things a little. Tests and javadoc settings can be added later. Choose the version you want to package and then you can basically follow the default recommendations and confirm them by hitting the Enter key. Throughout the project we choose to transform the upstream version with the symbolic debian version. Remember that Java/Maven is version-centric. This will ensure that our Maven dependencies are always satisfied later and we can simply upgrade our Maven libraries and don t have to change the versions by hand in various pom.xml files; maven-debian-helper will automatically transform them for us to debian . Enable all modules. If you choose not to, you can select each module individually. Note that later on some of the required build-dependencies cannot be found because they are either not installed (libjmagick6-java) or they cannot be found in Debian s Maven repository under /usr/share/maven-repo. You can fix this by entering a substitution rule or, as I did in this case, you can just ignore these artifacts for now. They will be added to maven.IgnoreRules. In order to successfully compile your program you have to remove them from this file later again, create the correct substitution rule in maven.rules and add the missing build-dependencies to debian/control. For now we just want to quickly create our initial debian directory. If everything went as planned a complete debian directory should be visible in your root directory. The only thing left is to fix the substitution rule for the Servlet API 3.1. Add libservlet3.1-java to Build-Depends and the following rule to maven.rules: javax.servlet s/servlet-api/javax.servlet-api/ * s/.*/3.1/ * *
s/javax.servlet/javax.servlet.jsp/ s/jsp-api/javax.servlet.jsp-api/ * s/.*/2.3/ * * The maven.rules file consists of multiple rows separated by six columns. The values represent groupId, artifactId, type, version number and two fields which I never use. You can just use an asterisk to match any value. Every value can be substituted. This is necessary when the value of upstream s pom.xml file differs from Debian s system packages. This happens frequently for API packages which are uploaded to Maven Central multiple times under a different groupId/artifactId but provide the same features. In this case the Twelvemonkeys pom requires an older API version but Debian is already at version 3.1. Note that we require a strict version number in this case because libservlet3.1-java does not use a symbolic debian version since we provide more than one Servlet API in the archive and this measure prevents conflicts. Thanks for reading this far. More articles about Java packaging will follow in the near future and hopefully they will clarify some terms and topics which could only be briefly mentioned in this post.

before

and after

Here is my monthly update covering what I have been doing in the free software world during July 2017 (previous month):

• Updated travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds:
  • Moved the default mirror from ftp.de.debian.org to deb.debian.org. [ ]
  • Create a sensible debian/changelog file if one does not exist. [ ]
• Updated django-slack, my library to easily post messages to the Slack group-messaging utility:
  • Merged a PR to clarify the error message when a channel could not be found. [ ]
  • Reviewed and merged a suggestion to add a TestBackend. [ ]
• Added Pascal support to Louis Taylor's anyprint hack to add support for "print" statements from other languages into Python. [ ]
• Filed a PR against Julien Danjou's daiquiri Python logging helper to clarify an issue in the documentation. [ ]
• Merged a PR to Strava Enhancement Suite my Chrome extension that improves and fixes annoyances in the web interface of the Strava cycling and running tracker to remove Zwift activities with maps. [ ]
• Submitted a pull request for Redis key-value database store to fix a spelling mistake in a binary. [ ]
• Sent patches upstream to the authors of the OpenSVC cloud engine and the Argyll Color Management System to fix some "1204" typos.
• Fixed a number of Python and deployment issues in my stravabot IRC bot. [ ]
• Correct a "1204" typo in Facebook's RocksDB key-value store. [ ]
• Corrected =+ typos in the Calibre e-book reader software. [ ]
• Filed a PR against the diaspy Python interface to the DIASPORA social network to correct the number of seconds in a day. [ ]
• Sent a pull request to remedy a =+ typo in sparqlwrapper, a SPARQL endpoint interface for Python. [ ]
• Filed a PR against Postfix Admin to fix some =+ typos. [ ]
• Fixed a "1042" typo in ImageJ, a Java image processing library. [ ]
• On a less-serious note, I filed an issue for Brad Abraham's bot for the Reddit sub-reddit to add some missing "hit the gym" advice. [ ]

I also blogged about my recent lintian hacking and installation-birthday package.

---

Yesterday I uploaded the first update of the TeX Live packages in Debian after TeX Live 2017 has entered Debian/unstable. The packages should by now have reached most mirrors. Nothing spectacular here besides a lot of updates and new packages. If I have to pick one update it would be the one of algorithm2e, a package that has seen lots of use and some bugs due to two years of inactivity. Good to see a new release. Enjoy. New packages algolrevived, invoice2, jfmutil, maker, marginfit, pst-geometrictools, pst-rputover, pxufont, shobhika, tikzcodeblocks, zebra-goodies. Updated packages acmart, adobemapping, algorithm2e, arabluatex, archaeologie, babel, babel-french, bangorexam, beamer, beebe, biblatex-gb7714-2015, bibleref, br-lex, bxjscls, combofont, computational-complexity, dozenal, draftfigure, elzcards, embrac, esami, factura, fancyhdr, fei, fithesis, fmtcount, fontspec, fonttable, forest, fvextra, genealogytree, gotoh, GS1, l3build, l3experimental, l3kernel, l3packages, latexindent, limap, luapackageloader, lwarp, mcf2graph, microtype, minted, mptopdf, pdfpages, polynom, powerdot, probsoln, pxbase, pxchfon, pythontex, reledmac, siunitx, struktex, tcolorbox, tetex, texdirflatten, uowthesistitlepage, uptex-fonts, xcharter.

long time no blog post. & the stretch release happened without many RC bug fixes from me; in practice, the auto-removals are faster & more convenient. what I nevertheless did in the last months was to fix RC bugs in pkg-perl packages (it still surprises me how fast rotting & constantly moving code is); prepare RC bug fixes for jessie (also for pkg-perl packages); & in the last weeks provide patches & carry out NMUs for perl packages as part of the ongoing perl 5.26 transition.

• #783656 libhtml-microformats-perl: "libhtml-microformats-perl: missing dependency on libmodule-pluggable-perl"
  fix in jessie (pkg-perl)
• #788008 libcgi-application-plugin-anytemplate-perl: "libcgi-application-plugin-anytemplate-perl: missing dependency on libclone-perl"
  fix in jessie (pkg-perl)
• #788350 libhttp-proxy-perl: "libhttp-proxy-perl: FTBFS - proxy tests"
  fix in jessie (pkg-perl)
• #808454 src:libdata-faker-perl: "libdata-faker-perl: FTBFS under some locales (eg. fr_CH.UTF-8)"
  fix in jessie (pkg-perl)
• #824843 libsys-syscall-perl: "libsys-syscall-perl: FTBFS on arm64: test suite failures"
  fix in jessie (pkg-perl)
• #824936 libsys-syscall-perl: "libsys-syscall-perl: FTBFS on mips*: test failures"
  fix in jessie (pkg-perl)
• #825012 libalgorithm-permute-perl: "libalgorithm-permute-perl: FTBFS with Perl 5.24: undefined symbol: PUSHBLOCK"
  upload new upstream release (pkg-perl)
• #826136 libsys-syscall-perl: "libsys-syscall-perl FTBFS on hppa arch (with patch)"
  fix in jessie (pkg-perl)
• #826471 intltool: "intltool: Unescaped left brace in regex is deprecated at /usr/bin/intltool-update line 1070"
  extend existing patch, uploaded by maintainer
• #826502 quilt: "quilt: FTBFS with perl 5.26: Unescaped left brace in regex"
  apply patch from ntyni (pkg-perl)
• #826505 swissknife: "swissknife: FTBFS with perl 5.26: Unescaped left brace in regex"
  add patches from ntyni, upload to DELAYED/5
• #839208 libio-compress-perl: "libio-compress-perl: uninstallable, current version superseded by Perl 5.24.1"
  upload newer upstream release (pkg-perl)
• #839218 nama: "nama: FTBFS because of perl's lack of stack reference counting"
  apply patch from Balint Reczey (pkg-perl)
• #848060 src:libx11-protocol-other-perl: "libx11-protocol-other-perl: FTBFS randomly (failing tests)"
  fix in jessie (pkg-perl)
• #855920 src:fail2ban: "fail2ban: FTBFS: test_rewrite_file: AssertionError: False is not true"
  try to reproduce, later propose different fix
• #855951 src:libsecret: "libsecret FTBFS with test failures on many architectures"
  try to reproduce
• #856133 src:shiboken: "shiboken FTBFS on i386/armel/armhf: other_collector_external_operator test failed"
  try to reproduce
• #857087 nsscache: "nsscache: /usr/share/nsscache not installed"
  triaging, later fixed by maintainer
• #858501 libmojomojo-perl: "libmojomojo-perl: broken symlink: /usr/share/perl5/MojoMojo/root/static/js/jquery.autocomplete.js -> ../../../../../javascript/jquery-ui/ui/jquery.ui.autocomplete.min.js"
  fix symlink (pkg-perl)
• #858509 cl-csv-data-table: "cl-csv-data-table: broken symlink: /usr/share/common-lisp/systems/cl-csv-data-table.asd -> ../source/cl-csv-data-table/cl-csv-data-table.asd"
  propose a patch
• #859539 filezilla: "filezilla: Filezilla crashes at startup"
  simply close the bug :)
• #859963 src:mimetic: "mimetic FTBFS on architectures where char is unsigned"
  add patch to make variable signed
• #860142 libgeo-ip-perl: "libgeo-ip-perl: should recommend geoip-database and geoip-database-extra"
  analyse, lower severity (pkg-perl)
• #863049 release.debian.org: "jessie-pu: package shutter/0.92-0.1+deb8u2"
  fix in jessie (pkg-perl)
• #865045 xmltv: "xmltv: FTBFS with Perl 5.26: t/test_filters.t failure"
  sponsor maintainer upload
• #865224 src:uwsgi: "uwsgi: ftbfs with multiple supported python3 versions"
  adjust build dependencies, upload NMU with maintainer's permission
• #865380 libtest-unixsock-perl: "libtest-unixsock-perl: Build-Conflicts-Indep: libtest-simple-perl (>= 1.3), including Perl 5.26"
  conditionally skip a test (pkg-perl)
• #865888 nagios-plugin-check-multi: "nagios-plugin-check-multi: FTBFS with Perl 5.26: Unescaped left brace in regex is deprecated here"
  update unescaped-left-brace-in-regex.patch, upload to DELAYED/5
• #866317 html2ps: "html2ps: relies on deprecated Perl syntax/features, breaks with 5.26"
  prepare a patch, later do QA upload
• #866934 libhttp-oai-perl: "libhttp-oai-perl: /usr/bin/oai_pmh uses the 'encoding' pragma, breaks with Perl 5.26"
  patch out "use encoding" (pkg-perl)
• #866944 libmecab-perl: "libmecab-perl: FTBFS: no such file or directory: /var/lib/mecab/dic/debian/dicrc"
  build-depend on fixed version (pkg-perl)
• #866981 src:libpgobject-type-datetime-perl: "libpgobject-type-datetime-perl FTBFS: You planned 15 tests but ran 5."
  upload new upstream release (pkg-perl)
• #866984 src:libpgobject-type-json-perl: "libpgobject-type-json-perl FTBFS: You planned 9 tests but ran 5."
  upload new upstream release (pkg-perl)
• #866991 libcpan-meta-perl: "libcpan-meta-perl: uninstallable in unstable"
  fix versioned Breaks/Replaces (pkg-perl)
• #867046 gwhois: "gwhois: fails with perl 5.26: The encoding pragma is no longer supported at /usr/bin/gwhois line 80."
  drop "use encoding", upload to DELAYED/5
• #867208 src:libpgobject-type-bytestring-perl: "libpgobject-type-bytestring-perl FTBFS: Failed 1/5 test programs. 0/8 subtests failed."
  upload new upstream release (pkg-perl)
• #867210 libtext-mecab-perl: "libtext-mecab-perl: FTBFS: test failures: Failed to create mecab instance"
  build-depend on fixed version (pkg-perl)
• #867983 libclass-load-perl: "libclass-load-perl: FTBFS: t/012-without-implementation.t failure"
  upload new upstream release (pkg-perl)
• #867984 libclass-load-xs-perl: "libclass-load-xs-perl: FTBFS: t/012-without-implementation.t failure"
  upload new upstream release (pkg-perl)
• #868069 liburi-namespacemap-perl: "liburi-namespacemap-perl: unbuildable with sbuild"
  fix build dependencies (pkg-perl)
• #868075 libperlx-assert-perl: "libperlx-assert-perl: missing dependency on libkeyword-simple-perl libdevel-declare-perl"
  fix dependencies (pkg-perl)
• #868613 src:liblog-report-lexicon-perl: "liblog-report-lexicon-perl FTBFS: Failed 2/10 test programs. 2/305 subtests failed."
  upload new upstream release (pkg-perl)
• #868888 get-flash-videos: "Fails: Can't locate Term/ProgressBar.pm"
  add missing dependency (pkg-perl)
• #869208 src:libalgorithm-svm-perl: "libalgorithm-svm-perl FTBFS on 32bit: SVM.c: loadable library and perl binaries are mismatched (got handshake key 0x80c0080, needed 0x81c0080)"
  add patch to add ccflags (pkg-perl)
• #869360 slic3r: "slic3r: missing dependency on perlapi-*"
  propose a patch
• #869383 src:clanlib: "clanlib FTBFS with perl 5.26"
  propose a patch
• #869418 src:libtest-taint-perl: "libtest-taint-perl FTBFS on armel: not ok 2 - $ENV TEST_ACTIVE is tainted"
  add a patch (pkg-perl)
• #869429 vile: "vile: Uninstallable after BinNMU / not binNMU-safe"
  some triaging

Here's what happened in the Reproducible Builds effort between Sunday July 9 and Saturday July 15 2017: Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #867771 filed against python-trollius.
  • #867773 filed against relatorio.
  • #867781 filed against gcin.
  • #867890 filed against apache-directory-jdbm.
  • #867900 filed against yapsy.
  • #867906 filed against wcsaxes.
• Chris Lamb:
  • #867753 filed against grunt (forwarded upstream).
  • #867848 filed against gconf (forwarded upstream).
  • #868133 filed against grep.
  • #868321 filed against node-marked-man (forwarded-upstream).
• Bernhard M. Wiedemann:
  • chromium
  • chromium
  • kubernetes
  • qtscriptgenerator
  • crash merged
  • ghc-rpm-macros merged
  • samba
  • htmldoc
  • efl
  • webkitgtk3 merged
  • fence-agents
  • fio merged

Reviews of unreproducible packages 13 package reviews have been added, 12 have been updated and 19 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• build_path_captured_by_valac
• timestamps_in_javascript_generated_by_node_grunt_banner

3 issue types have been updated:

• Add fix for timestamps_embedded_in_manpages_by_node_marked_man toolchain issue.
• Add fix for timestamps_in_javascript_generated_by_node_grunt_banner toolchain issue.
• timestamps_in_javascript_generated_by_node_grunt_banner is actually in src:grunt

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (47)

diffoscope development Version 84 was uploaded to unstable by Mattia Rizzolo. It included contributions already reported from the previous weeks, as well as new ones:

• Ximin Luo:
  • Attempt to fix fsimage test on Jenkins
  • Use a tempdir rather than ./cache for guestfs cache
  • Add more debugging output to test_fsimage

After the release, development continued in git with contributions from:

• Mattia Rizzolo:
  • RequiredToolNotFound.get_package(): just call the new get_package_provider()
  • Add a get_package_provider() function, returning the package name that best matches the system
  • Move from deprecated platform.linux_distribution() to the external package distro

strip-nondeterminism development Versions 0.036-1, 0.037-1 and 0.038-1 were uploaded to unstable by Chris Lamb. They included contributions from:

• Niels Thykier:
  • Add missing use statements in handler modules
  • dh_strip_nondeterminism: Assumes tmpdir() exists
  • File::StripNondeterminism:
    • Apply perltidy
    • Lazy load most handlers
    • Lazy load remaining handlers
• Chris Lamb:
  • Add missing File::Temp imports in JAR and PNG handlers. This appears to have been exposed by lazily-loading handlers in #867982.

reprotest development Development continued in git with contributions from:

• Ximin Luo:
  • presets: use newer flag --no-sign for dpkg-buildpackage
  • Document --diffoscope-args= --exclude-directory-metadata and use it in presets
• Mattia Rizzolo:
  • Bump debhelper compat level to 10.
  • Bump Standards-Version to 4.0.0.

buildinfo.debian.net development

• Chris Lamb:
  • Avoid a race condition between check-and-creation of Buildinfo instances.

tests.reproducible-builds.org

• Mattia Rizzolo:
  • Make database backups quicker to restore by avoiding --column-inserts's pg_dump option.
  • Fixup the deployment scripts after the stretch migration.
  • Fixup Apache redirects that were broken after introducing the buster suite
  • Fixup diffoscope jobs that were not always installing the highest possible version of diffoscope
• Holger Levsen:
  • Add a node health check for a too big jenkins.log.

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

revisiting libjson-spirit. I tried compiling a program that uses libjson-spirit and noticed that it still is broken. New programs compiled against the header does not link with the provided static library. Trying to rebuild it fixes it, but it uses compat version 8, and that needs to be fixed (trivially). hmm... actually code doesn't build anymore and there's multiple new upstream versions. ... and then I noticed that it was a stale copy already removed from Debian repository. What's a good C++ json implementation these days?

Yesterday I uploaded the first packages of TeX Live 2017 to Debian/unstable, meaning that the new release cycle has started. Debian/stretch was released over the weekend, and this opened up unstable for new developments. The upload comprised the following packages: asymptote, cm-super, context, context-modules, texlive-base, texlive-bin, texlive-extra, texlive-extra, texlive-lang, texworks, xindy.
I mentioned already in a previous post the following changes:

• several packages have been merged, some are dropped (eg. texlive-htmlxml) and one new package (texlive-plain-generic) has been added
• luatex got updated to 1.0.4, and is now considered stable
• updmap and fmtutil now require either -sys or -user
• tlmgr got a shell mode (interactive/scripting interface) and a new feature to add arbitrary TEXMF trees (conf auxtrees)

The last two changes are described together with other news (easy TEXMF tree management) in the TeX Live release post. These changes more or less sum up the new infra structure developments in TeX Live 2017. Since the last release to unstable (which happened in 2017-01-23) about half a year of package updates have accumulated, below is an approximate list of updates (not split into new/updated, though). Enjoy the brave new world of TeX Live 2017, and please report bugs to the BTS! Updated/new packages:
academicons, achemso, acmart, acro, actuarialangle, actuarialsymbol, adobemapping, alkalami, amiri, animate, aomart, apa6, apxproof, arabluatex, archaeologie, arsclassica, autoaligne, autobreak, autosp, axodraw2, babel, babel-azerbaijani, babel-english, babel-french, babel-indonesian, babel-japanese, babel-malay, babel-ukrainian, bangorexam, baskervaldx, baskervillef, bchart, beamer, beamerswitch, bgteubner, biblatex-abnt, biblatex-anonymous, biblatex-archaeology, biblatex-arthistory-bonn, biblatex-bookinother, biblatex-caspervector, biblatex-cheatsheet, biblatex-chem, biblatex-chicago, biblatex-claves, biblatex-enc, biblatex-fiwi, biblatex-gb7714-2015, biblatex-gost, biblatex-ieee, biblatex-iso690, biblatex-manuscripts-philology, biblatex-morenames, biblatex-nature, biblatex-opcit-booktitle, biblatex-oxref, biblatex-philosophy, biblatex-publist, biblatex-shortfields, biblatex-subseries, bibtexperllibs, bidi, biochemistry-colors, bookcover, boondox, bredzenie, breqn, bxbase, bxcalc, bxdvidriver, bxjalipsum, bxjaprnind, bxjscls, bxnewfont, bxorigcapt, bxpapersize, bxpdfver, cabin, callouts, chemfig, chemformula, chemmacros, chemschemex, childdoc, circuitikz, cje, cjhebrew, cjk-gs-integrate, cmpj, cochineal, combofont, context, conv-xkv, correctmathalign, covington, cquthesis, crimson, crossrefware, csbulletin, csplain, csquotes, css-colors, cstldoc, ctex, currency, cweb, datetime2-french, datetime2-german, datetime2-romanian, datetime2-ukrainian, dehyph-exptl, disser, docsurvey, dox, draftfigure, drawmatrix, dtk, dviinfox, easyformat, ebproof, elements, endheads, enotez, eqnalign, erewhon, eulerpx, expex, exsheets, factura, facture, fancyhdr, fbb, fei, fetamont, fibeamer, fithesis, fixme, fmtcount, fnspe, fontmfizz, fontools, fonts-churchslavonic, fontspec, footnotehyper, forest, gandhi, genealogytree, glossaries, glossaries-extra, gofonts, gotoh, graphics, graphics-def, graphics-pln, grayhints, gregoriotex, gtrlib-largetrees, gzt, halloweenmath, handout, hang, heuristica, hlist, hobby, hvfloat, hyperref, hyperxmp, ifptex, ijsra, japanese-otf-uptex, jlreq, jmlr, jsclasses, jslectureplanner, karnaugh-map, keyfloat, knowledge, komacv, koma-script, kotex-oblivoir, l3, l3build, ladder, langsci, latex, latex2e, latex2man, latex3, latexbug, latexindent, latexmk, latex-mr, leaflet, leipzig, libertine, libertinegc, libertinus, libertinust1math, lion-msc, lni, longdivision, lshort-chinese, ltb2bib, lualatex-math, lualibs, luamesh, luamplib, luaotfload, luapackageloader, luatexja, luatexko, lwarp, make4ht, marginnote, markdown, mathalfa, mathpunctspace, mathtools, mcexam, mcf2graph, media9, minidocument, modular, montserrat, morewrites, mpostinl, mptrees, mucproc, musixtex, mwcls, mweights, nameauth, newpx, newtx, newtxtt, nfssext-cfr, nlctdoc, novel, numspell, nwejm, oberdiek, ocgx2, oplotsymbl, optidef, oscola, overlays, pagecolor, pdflatexpicscale, pdfpages, pdfx, perfectcut, pgfplots, phonenumbers, phonrule, pkuthss, platex, platex-tools, polski, preview, program, proofread, prooftrees, pst-3dplot, pst-barcode, pst-eucl, pst-func, pst-ode, pst-pdf, pst-plot, pstricks, pstricks-add, pst-solides3d, pst-spinner, pst-tools, pst-tree, pst-vehicle, ptex2pdf, ptex-base, ptex-fontmaps, pxbase, pxchfon, pxrubrica, pythonhighlight, quran, ran_toks, reledmac, repere, resphilosophica, revquantum, rputover, rubik, rutitlepage, sansmathfonts, scratch, seealso, sesstime, siunitx, skdoc, songs, spectralsequences, stackengine, stage, sttools, studenthandouts, svg, tcolorbox, tex4ebook, tex4ht, texosquery, texproposal, thaienum, thalie, thesis-ekf, thuthesis, tikz-kalender, tikzmark, tikz-optics, tikz-palattice, tikzpeople, tikzsymbols, titlepic, tl17, tqft, tracklang, tudscr, tugboat-plain, turabian-formatting, txuprcal, typoaid, udesoftec, uhhassignment, ukrainian, ulthese, unamthesis, unfonts-core, unfonts-extra, unicode-math, uplatex, upmethodology, uptex-base, urcls, variablelm, varsfromjobname, visualtikz, xassoccnt, xcharter, xcntperchap, xecjk, xepersian, xetexko, xevlna, xgreek, xsavebox, xsim, ycbook.

A new package! Or at least new on CRAN as the very initial version 0.1.0 had been available via the ghrr drat for over a year. But now we have version 0.1.1 to announce as a CRAN package. RcppMspPack provides R with MessagePack header files for use via C++ (or C, if you must) packages such as RcppRedis. MessagePack itself is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it is faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves. MessagePack is used by Redis and many other projects, and has bindings to just about any language. To use this package, simply add it to the LinkingTo: field in the DESCRIPTION field of your R package---and the R package infrastructure tools will then know how to set include flags correctly on all architectures supported by R. More information may be on the RcppMsgPack page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I am very happy to report that the Nikita Noark 5 core project tagged its second release today. The free software solution is an implementation of the Norwegian archive standard Noark 5 used by government offices in Norway. These were the changes in version 0.1.1 since version 0.1.0 (from NEWS.md):

• Continued work on the angularjs GUI, including document upload.
• Implemented correspondencepartPerson, correspondencepartUnit and correspondencepartInternal
• Applied for coverity coverage and started submitting code on regualr basis.
• Started fixing bugs reported by coverity
• Corrected and completed HATEOAS links to make sure entire API is available via URLs in _links.
• Corrected all relation URLs to use trailing slash.
• Add initial support for storing data in ElasticSearch.
• Now able to receive and store uploaded files in the archive.
• Changed JSON output for object lists to have relations in _links.
• Improve JSON output for empty object lists.
• Now uses correct MIME type application/vnd.noark5-v4+json.
• Added support for docker container images.
• Added simple API browser implemented in JavaScript/Angular.
• Started on archive client implemented in JavaScript/Angular.
• Started on prototype to show the public mail journal.
• Improved performance by disabling Sprint FileWatcher.
• Added support for 'arkivskaper', 'saksmappe' and 'journalpost'.
• Added support for some metadata codelists.
• Added support for Cross-origin resource sharing (CORS).
• Changed login method from Basic Auth to JSON Web Token (RFC 7519) style.
• Added support for GET-ing ny-* URLs.
• Added support for modifying entities using PUT and eTag.
• Added support for returning XML output on request.
• Removed support for English field and class names, limiting ourself to the official names.
• ...

If this sound interesting to you, please contact us on IRC (#nikita on irc.freenode.net) or email (nikita-noark mailing list).

Here's what happened in the Reproducible Builds effort between Sunday May 28 and Saturday June 3 2017: Past an upcoming events

• On June 9th, Chris Lamb will present at the Hong Kong Open Source Conference 2017 on reproducible builds.
• We restarted our IRC meetings, now with a monthly schedule where we meet on the first Thursday of the month at a varying time. Our next meeting will be on the first 6th of July at 17:00 UTC. At the June meeting we had the following agenda, and if you are interested there is an automated summary and full logs too.
  1. Introductions
  2. Feedback for the reproducible.json spec format - is that suitable for Guix and F-Droid as well? LEDE/OpenWrt? Coreboot?
  3. writing parser + user interface for reproducible.json
  4. tests.r-b.o/Debian once Stretch has been released
  5. Reproducible Builds Summit 2017
  6. Next meeting
  7. Any other business

Documentation updates

• Holger added a link to the blog post with a report about the recent hackathon to our website.

Toolchain development and fixes

• Chris Lamb wrote a proof-of-concept implementation for #863622 ("apt: warn when installing packages that are not reproducible"). Try it, it's fun!
• Russ worked on making the debian-policy package build reproducibly which is a good opportunity to shamelessly plug that #844431 ("debian-policy: Packages should be reproducible") is still a work in progress.

Patches and bugs filed

• Bernhard M. Wiedemann:
  • pycryptopp
  • rpm
  • rpm
  • bam
  • boost/jam
  • premake
  • ruby/rdoc
  • ruby/rdoc
  • sdcc (merged)

4 package reviews have been added, 6 have been updated and 25 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (2)
• Chris Lamb (1)

diffoscope development

• Chris Lamb:
  • Add comparator for Fontconfig cache files.
• anthraxx:
  • tools: extend external Arch Linux tools
• Ximin Luo:
  • Add a --exclude-command CLI for filtering out long-running commands like "readelf --debug-dump=info"
  • Use unicode chars for the +/- controls instead of hacky punctuation
  • Don't show +/- controls for differences with no children
  • Fix create_limited_print_func
  • Add a size() method to Difference and check that self._visuals is empty in get_reverse()
  • Add a reader for the JSON format
  • Allow the "source" param to overridden compare() methods to be given as a positional argument

tests.reproducible-builds.org Mattia Rizzolo:

• packages-tests: go back to the shorter way of specifying JUnit jobs.
• update_jdn: call jenkins-jobs instead of jenkins-job-builder, as that's how it is named in 1.6.1 version.
• Upload jenkins-job-builder 1.6.1-1~bpo8+1 to jessie-backports.

Daniel Kahn Gillmor:

• Contribute a few typo fixes.

Vagrant Cascadian:

• Update README to reflect switch to PostgreSQL (some time ago).
• Add three new boards for armhf tests:
  • ff64a-armhf-rb.debian.net: Firefly-rk3399, Rockchip six-core (Cortex-A72 x2 + Cortex-A53 x4), 2GB RAM, USB-sata (future plans for native sata.)
  • jtx1a-armhf-rb.debian.net: Jetson-tx1, quad-core (big.LITTLE Cortex-A53/A57), ~3.5GB RAM, native SATA ~500GB disk
  • odc2a-armhf-rb.debian.net: Odroic-C2, quad-core (Cortex-A53), ~2GB ram, 128GB eMMC
• Ressurect rpi2c-armhf-rb.debian.net (Raspberry PI 2B, broadcom bcm2836 quad-core (cortex-A7), 1GB RAM) from the dead.

Holger Levsen:

• Configure the rc.local service on all build nodes to only start when the network is actually up, as configuring half of them to run in the future requires networking to determine the real current date. See Running Services After the Network is up if you want to learn more about networking with systemd. Somehow this doesnt work yet on the Ubuntu 16.05 arm64 nodes; help welcome, either on #debian-qa or #debian-reproducible.
• Add rpi2c back to the armhf network.
• Also add ff64a, odc2a & jtx1a to the armhf network.
  • Add pbuilder/schroot-setup jobs for the new nodes.
  • Add 10 new armhf builder jobs.
  • Disable all build jobs on odc2a again as haveged segfaults on 4.12~rc2, which is needed for this board & setup.
• Reproducible_cleanup_nodes.sh: adapt for new build service
• Make performance.html show if build jobs are down due to remote node problems and make code to count enabled jobs more robust.
• reproducible_build_service: Make it cope with disabled workers.
• jenkins-master-wrapper: Fail loudly if remote node is not accessable.
• Increase maximum scheduling queue sizes, to adjust for faster scheduler.
• New script, init_node, to initialize new build nodes (derived from refactored existing script).
• Update INSTALL documentation to reflect that we're testing arm64 now too.

Misc. This week's edition was written by Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Wrote and released installation-birthday. Installing this package will celebrate the anniversary of installing your system by sending you an email via cron(8).
• Fixed an issue in the Django web development framework where you couldn't run the testsuite against a read-only copy of the source code. This was found by the Debian Continuous Integration service. (#26755)
• Provided a pull request for the "wammu" mobile phone manager to ensure the build is reproducible. (#49)

---

I've released version 1.6.0 of Nageru, my live video mixer, together with dependent libraries Movit 1.5.1 and bmusb 0.7.0. The primray new feature this time is integration with CasparCG, the dominating open-source broadcast graphics system, which opens up a whole new world of possibilities for intelligent overlay graphics. (Actually, the feature is a bit more generic than that; any FFmpeg file or stream will do as input. Audio isn't supported yet, though.) You can see a simple HTML5 CasparCG setup in the ultimate tournament stream test we did in April, in preparation of a larger event in September; CasparCG generates a stream with alpha, which is then fed into Nageru and used on top of the three camera sources. Apart from that, there's a new frame analyzer that helps with calibrating your signal chain; there are lots of devices that will happily mess with your signal, and measuring is the first step in counteracting that. (There's also a few input interpretation tweaks that will help most common issues.) 1.6.0 is on its way to Debian experimental, along with its dependencies (stretch will release with Nageru 1.4.2); there are likely to be backports when stretch releases and the backport queue opens up.

When I started debci for Debian CI, I went for the simplest thing that could possibly work. One of the design decisions was to use the filesystem directly for file storage. A large part of the Debian CI data is log files and test artifacts (which are just files), and using the filesystem directly for storage makes it a lot easier to handle it. The rest of the data which is structured (test history and status of packages) is stored as JSON files. Another nice benefit of using the filesystem like this is that I get a sort of REST API for free by just exposing the file storage to the web. For example, getting the latest test status of debci itself on unstable/amd64 is as easy as:

$ curl https://ci.debian.net/data/packages/unstable/amd64/d/debci/latest.json
 
  "run_id": "20170528_173652",
  "package": "debci",
  "version": "1.5.1",
  "date": "2017-05-28 17:43:05",
  "status": "pass",
  "blame": [],
  "previous_status": "pass",
  "duration_seconds": "373",
  "duration_human": "0h 6m 13s",
  "message": "Tests passed, but at least one test skipped",
  "last_pass_version": "1.5.1",
  "last_pass_date": "2017-05-28 17:43:05"
 

Now, nothing in life is without compromises. One big disadvantage of the way debci stored its data is that there were a lot of files, which ends up using a large number of inodes in the filesystem. The current Debian CI master has more than 10 million inodes in its filesystem, and almost all of them were being used. This is clearly unsustainable. You will notice that I said stored, because as of version 1.6, debci now implements a data retention policy: log files and test artifacts will now only be kept for a configurable amount of days (default: 180). So there you have it: effective immediately, Debian CI will not provide logs and test artifacts older than 180 days. If you are reporting bugs based on logs from Debian CI, please don t hotlink the log files. Instead, make sure you download the logs in question and attach them to the bug report, because in 6 months they will be gone.

Here's what happened in the Reproducible Builds effort between Sunday May 14 and Saturday May 20 2017: News and Media coverage

• We've reached 94.0% reproducible packages on testing/amd64! (NB. without build path variation)
• Maria Glukhova was interviewed on It's FOSS about her involvement with Reproducible Builds with respect to Outreachy.

IRC meeting Our next IRC meeting has been scheduled for Thursday June 1 at 16:00 UTC. Packages reviewed and fixed, bugs filed, etc. Bernhard M. Wiedemann:

• boost
• pytsk
• bam
• kakoune
• newsbeuter
• trigger-rally
• firebird
• povray
• zynaddsubfx (fixed)
• scintilla (merged)
• cryptopp (merged)

Chris Lamb:

• #862553 filed against vim-command-t.
• #862588 filed against tkhtml1.
• #862592 filed against taskcoach.
• #862676 filed against mp3fs.
• #862825 filed against golang-github-pkg-profile.
• #863015 filed against jellyfish.
• #863054 filed against doxygen.

Reviews of unreproducible packages 35 package reviews have been added, 28 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• new year_variable_in_documentation_generated_by_doxygen
• jellyfish_creates_nondeterministic_json

diffoscope development

• Mattia Rizzolo:
  • Export JUnit-style test report when building on Jenkins

strip-nondeterminism development

• Chris Lamb:
  • Only print log messages by default if the file was actually modified. (Closes: #863033)

tests.reproducible-builds.org Holger wrote a new systemd-based scheduling system replacing 162 constantly running Jenkins jobs which were slowing down job execution in general:

• Nothing fancy really, just 370 lines of shell code in two scripts, out of these 370 lines 80 are comments and 162 are node defitions for those 162 "jobs".
• Worker logs not yet as good as with Jenkins but usually we dont need realitime log viewing of specific builds. Or rather, its a waste of time to do it. (Actual package build logs remain unchanged.)
• Builds are a lot faster for the fast archs, but not so much difference on armhf.
• Since April 12 for i386 (and a week later for the rest), the images below are ordered with i386 on top, then amd64, armhf and arm64. Except for armhf it's pretty visible when the switch was made.

Misc. This week's edition was written by Chris Lamb, Holver Levsen, Bernhard M. Wiedemann, Vagrant Cascadian and Maria Glukhova & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

We re coming closer to the Debian/stretch stable release and similar to what we had with #newinwheezy and #newinjessie it s time for #newinstretch! Hideki Yamane already started the game by blogging about GitHub s Icon font, fonts-octicons and Arturo Borrero Gonzalez wrote a nice article about nftables in Debian/stretch. One package that isn t new but its tools are used by many of us is util-linux, providing many essential system utilities. We have util-linux v2.25.2 in Debian/jessie and in Debian/stretch there will be util-linux >=v2.29.2. There are many new options available and we also have a few new tools available. Tools that have been taken over from other packages

• last: used to be shipped via sysvinit-utils in Debian/jessie
• lastb: used to be shipped via sysvinit-utils in Debian/jessie
• mesg: used to be shipped via sysvinit-utils in Debian/jessie
• mountpoint: used to be shipped via initscripts in Debian/jessie
• sulogin: used to be shipped via sysvinit-utils in Debian/jessie

New tools

• lsipc: show information on IPC facilities, e.g.:

root@ff2713f55b36:/# lsipc
RESOURCE DESCRIPTION                                              LIMIT USED  USE%
MSGMNI   Number of message queues                                 32000    0 0.00%
MSGMAX   Max size of message (bytes)                               8192    -     -
MSGMNB   Default max size of queue (bytes)                        16384    -     -
SHMMNI   Shared memory segments                                    4096    0 0.00%
SHMALL   Shared memory pages                       18446744073692774399    0 0.00%
SHMMAX   Max size of shared memory segment (bytes) 18446744073692774399    -     -
SHMMIN   Min size of shared memory segment (bytes)                    1    -     -
SEMMNI   Number of semaphore identifiers                          32000    0 0.00%
SEMMNS   Total number of semaphores                          1024000000    0 0.00%
SEMMSL   Max semaphores per semaphore set.                        32000    -     -
SEMOPM   Max number of operations per semop(2)                      500    -     -
SEMVMX   Semaphore max value                                      32767    -     -

• lslogins: display information about known users in the system, e.g.:

root@ff2713f55b36:/# lslogins 
  UID USER     PROC PWD-LOCK PWD-DENY LAST-LOGIN GECOS
    0 root        2        0        1            root
    1 daemon      0        0        1            daemon
    2 bin         0        0        1            bin
    3 sys         0        0        1            sys
    4 sync        0        0        1            sync
    5 games       0        0        1            games
    6 man         0        0        1            man
    7 lp          0        0        1            lp
    8 mail        0        0        1            mail
    9 news        0        0        1            news
   10 uucp        0        0        1            uucp
   13 proxy       0        0        1            proxy
   33 www-data    0        0        1            www-data
   34 backup      0        0        1            backup
   38 list        0        0        1            Mailing List Manager
   39 irc         0        0        1            ircd
   41 gnats       0        0        1            Gnats Bug-Reporting System (admin)
  100 _apt        0        0        1            
65534 nobody      0        0        1            nobody

• lsns: list system namespaces, e.g.:

root@ff2713f55b36:/# lsns
        NS TYPE   NPROCS PID USER COMMAND
4026531835 cgroup      2   1 root bash
4026531837 user        2   1 root bash
4026532473 mnt         2   1 root bash
4026532474 uts         2   1 root bash
4026532475 ipc         2   1 root bash
4026532476 pid         2   1 root bash
4026532478 net         2   1 root bash

• setpriv: run a program with different privilege settings
• zramctl: tool to quickly set up zram device parameters, to reset zram devices, and to query the status of used zram devices

New features/options addpart (show or change the real-time scheduling attributes of a process):

--reload reload prompts on running agetty instances

blkdiscard (discard the content of sectors on a device):

-p, --step <num>    size of the discard iterations within the offset
-z, --zeroout       zero-fill rather than discard

chrt (show or change the real-time scheduling attributes of a process):

-d, --deadline            set policy to SCHED_DEADLINE
-T, --sched-runtime <ns>  runtime parameter for DEADLINE
-P, --sched-period <ns>   period parameter for DEADLINE
-D, --sched-deadline <ns> deadline parameter for DEADLINE

fdformat (do a low-level formatting of a floppy disk):

-f, --from <N>    start at the track N (default 0)
-t, --to <N>      stop at the track N
-r, --repair <N>  try to repair tracks failed during the verification (max N retries)

fdisk (display or manipulate a disk partition table):

-B, --protect-boot            don't erase bootbits when creating a new label
-o, --output <list>           output columns
    --bytes                   print SIZE in bytes rather than in human readable format
-w, --wipe <mode>             wipe signatures (auto, always or never)
-W, --wipe-partitions <mode>  wipe signatures from new partitions (auto, always or never)
New available columns (for -o):
 gpt: Device Start End Sectors Size Type Type-UUID Attrs Name UUID
 dos: Device Start End Sectors Cylinders Size Type Id Attrs Boot End-C/H/S Start-C/H/S
 bsd: Slice Start End Sectors Cylinders Size Type Bsize Cpg Fsize
 sgi: Device Start End Sectors Cylinders Size Type Id Attrs
 sun: Device Start End Sectors Cylinders Size Type Id Flags

findmnt (find a (mounted) filesystem):

-J, --json             use JSON output format
-M, --mountpoint <dir> the mountpoint directory
-x, --verify           verify mount table content (default is fstab)
    --verbose          print more details

flock (manage file locks from shell scripts):

-F, --no-fork            execute command without forking
    --verbose            increase verbosity

getty (open a terminal and set its mode):

--reload               reload prompts on running agetty instances

hwclock (query or set the hardware clock):

--get            read hardware clock and print drift corrected result
--update-drift   update drift factor in /etc/adjtime (requires --set or --systohc)

ldattach (attach a line discipline to a serial line):

-c, --intro-command <string>  intro sent before ldattach
-p, --pause <seconds>         pause between intro and ldattach

logger (enter messages into the system log):

-e, --skip-empty         do not log empty lines when processing files
    --no-act             do everything except the write the log
    --octet-count        use rfc6587 octet counting
-S, --size <size>        maximum size for a single message
    --rfc3164            use the obsolete BSD syslog protocol
    --rfc5424[=<snip>]   use the syslog protocol (the default for remote);
                           <snip> can be notime, or notq, and/or nohost
    --sd-id <id>         rfc5424 structured data ID
    --sd-param <data>    rfc5424 structured data name=value
    --msgid <msgid>      set rfc5424 message id field
    --socket-errors[=<on off auto>] print connection errors when using Unix sockets

losetup (set up and control loop devices):

-L, --nooverlap               avoid possible conflict between devices
    --direct-io[=<on off>]    open backing file with O_DIRECT 
-J, --json                    use JSON --list output format
New available --list column:
DIO  access backing file with direct-io

lsblk (list information about block devices):

-J, --json           use JSON output format
New available columns (for --output):
HOTPLUG  removable or hotplug device (usb, pcmcia, ...)
SUBSYSTEMS  de-duplicated chain of subsystems

lscpu (display information about the CPU architecture):

-y, --physical          print physical instead of logical IDs
New available column:
DRAWER  logical drawer number

lslocks (list local system locks):

-J, --json             use JSON output format
-i, --noinaccessible   ignore locks without read permissions

nsenter (run a program with namespaces of other processes):

-C, --cgroup[=<file>]      enter cgroup namespace
    --preserve-credentials do not touch uids or gids
-Z, --follow-context       set SELinux context according to --target PID

rtcwake (enter a system sleep state until a specified wakeup time):

--date <timestamp>   date time of timestamp to wake
--list-modes         list available modes
-r, --reorder <dev>  fix partitions order (by start offset)

sfdisk (display or manipulate a disk partition table):

New Commands:
-J, --json <dev>                  dump partition table in JSON format
-F, --list-free [<dev> ...]       list unpartitioned free areas of each device
-r, --reorder <dev>               fix partitions order (by start offset)
    --delete <dev> [<part> ...]   delete all or specified partitions
--part-label <dev> <part> [<str>] print or change partition label
--part-type <dev> <part> [<type>] print or change partition type
--part-uuid <dev> <part> [<uuid>] print or change partition uuid
--part-attrs <dev> <part> [<str>] print or change partition attributes
New Options:
-a, --append                   append partitions to existing partition table
-b, --backup                   backup partition table sectors (see -O)
    --bytes                    print SIZE in bytes rather than in human readable format
    --move-data[=<typescript>] move partition data after relocation (requires -N)
    --color[=<when>]           colorize output (auto, always or never)
                               colors are enabled by default
-N, --partno <num>             specify partition number
-n, --no-act                   do everything except write to device
    --no-tell-kernel           do not tell kernel about changes
-O, --backup-file <path>       override default backup file name
-o, --output <list>            output columns
-w, --wipe <mode>              wipe signatures (auto, always or never)
-W, --wipe-partitions <mode>   wipe signatures from new partitions (auto, always or never)
-X, --label <name>             specify label type (dos, gpt, ...)
-Y, --label-nested <name>      specify nested label type (dos, bsd)
Available columns (for -o):
 gpt: Device Start End Sectors Size Type Type-UUID Attrs Name UUID
 dos: Device Start End Sectors Cylinders Size Type Id Attrs Boot End-C/H/S Start-C/H/S
 bsd: Slice Start  End Sectors Cylinders Size Type Bsize Cpg Fsize
 sgi: Device Start End Sectors Cylinders Size Type Id Attrs
 sun: Device Start End Sectors Cylinders Size Type Id Flags

swapon (enable devices and files for paging and swapping):

-o, --options <list>     comma-separated list of swap options
New available columns (for --show):
UUID   swap uuid
LABEL  swap label

unshare (run a program with some namespaces unshared from the parent):

-C, --cgroup[=<file>]                              unshare cgroup namespace
    --propagation slave shared private unchanged   modify mount propagation in mount namespace
-s, --setgroups allow deny                         control the setgroups syscall in user namespaces

Deprecated / removed options sfdisk (display or manipulate a disk partition table):

-c, --id                  change or print partition Id
    --change-id           change Id
    --print-id            print Id
-C, --cylinders <number>  set the number of cylinders to use
-H, --heads <number>      set the number of heads to use
-S, --sectors <number>    set the number of sectors to use
-G, --show-pt-geometry    deprecated, alias to --show-geometry
-L, --Linux               deprecated, only for backward compatibility
-u, --unit S              deprecated, only sector unit is supported