For most packages, the dh short-hand rules (possibly with a few overrides) work great. It can often auto-detect the buildsystem and handle all the trivial parts. With one notably exception: What if you need to compile the upstream code twice (or more) with different flags? This is the case for all source packages building both regular debs and udebs. In that case, you would previously need to override about 5-6 helpers for this to work at all. The five dh_auto_* helpers and usually also dh_install (to call it with different sourcedir for different packages). This gets even more complex if you want to support Build-Profiles such as noudeb and nodoc . The best way to support nodoc in debhelper is to move documentation out of dh_install s config files and use dh_installman, dh_installdocs, and dh_installexamples instead (NB: wait for compat 11 before doing this). This in turn will mean more overrides with sourcedir and -p/-N. And then there is noudeb , which currently requires manual handling in debian/rules. Basically, you need to use make or shell if-statements to conditionally skip the udeb part of the builds. All of this is needlessly complex. Improving the situation In an attempt to make things better, I have made a new prototype feature in debhelper called buildlabels in experimental. The current prototype is designed to deal with part (but not all) of the above problems:

• It will remove the need for littering your rules file for supporting noudeb (and in some cases also other noX profiles).
• It will remove the need for overriding the dh_install* tools just to juggle with sourcedir and -p/-N.

However, it currently not solve the need for overriding the dh_auto_* tools and I am not sure when/if it will. The feature relies on being able to relate packages to a given series of calls to dh_auto_*. In the following example, I will use udebs for the secondary build. However, this feature is not tied to udebs in any way and can be used any source package that needs to do two or more upstream builds for different packages. Assume our example source builds the following binary packages:

• foo
• libfoo1
• libfoo-dev
• foo-udeb
• libfoo1-udeb

And in the rules file, we would have something like:

[...]
override_dh_auto_configure:
    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure -B build-udeb -- --without-feature1 --without-feature2
[...]

What is somewhat obvious to a human is that the first configure line is related to the regular debs and the second configure line is for the udebs. However, debhelper does not know how to infer this and this is where buildlabels come in. With buildlabels, you can let debhelper know which packages and builds that belong together. How to use buildlabels To use buildlabels, you have to do three things:

1. Pick a reasonable label name for the secondary build. In the example, I will use udeb .
2. Add buildlabel=$LABEL to all dh_auto_* calls related to your secondary build.
3. Tag all packages related to my-label with X-DH-Buildlabel: $LABEL in debian/control. (For udeb packages, you may want to add Build-Profiles: <!noudeb> while you are at it).

For the example package, we would change the debian/rules snippet to:

[...]
override_dh_auto_configure:
    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure --buildlabel=udeb -B build-udeb -- --without-feature1 --without-feature2
[...]

(Remember to update *all* calls to dh_auto_* helpers; the above only lists dh_auto_configure to keep the example short.) And then add X-DH-Buildlabel: udeb in the stanzas for foo-udeb + libfoo1-udeb. With those two minor changes:

• debhelper will skip the calls to dh_auto_* with buildlabel=udeb if the udeb packages are skipped.
• dh_auto_install will automatically pick a separate destination directory by default for the udeb build (assuming you do not explicitly override it with destdir).
• dh_install will now automatically pick up files from the destination directory.that dh_auto_install used for the given package (even if you overwrote it with destdir). Note that you have to remove any use of sourcedir first as this disables the auto-detection. This also works for other dh_install* tools supporting sourcedir in compat 11 or later.

Real example Thanks to Michael Biebl, I was able to make an branch in the systemd git repository to play with this feature. Therefore I have an real example to use as a show case. The gist of it is in the following three commits:

• Apply labels to packages and dh_auto_* tools: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=880a80bc392a298bed033c047ccbef65e50acc58
• Clean up most make conditionals: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=96c0ab8b35ce04eab031a4a9ee4c4ef686f76977
• Merge multiple dh_install calls into one: https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=wip-dh-prototype-smarter-multi-builds&id=67994cb5e5b451ebcebb0dccfdd97e606ac4a172
  ( fail-missing was never applied to the udeb before; I kept it that way by passing sourcedir to dh_missing because I was to lazy to review the missing files).

Full branch can be seen at: https://anonscm.debian.org/git/pkg-systemd/systemd.git/log/?h=wip-dh-prototype-smarter-multi-builds Request for comments / call for testing This prototype is now in experimental (debhelper/10.7+exp.buildlabels) and you are very welcome to take it for a spin. Please let me know if you find the idea useful and feel free to file bugs or feature requests. If deemed useful, I will merge into master and include in a future release. If you have any questions or comments about the feature or need help with trying it out, you are also very welcome to mail the debhelper-devel mailing list. Known issues / the fine print:

• It is experimental feature and may change without notice.
• The current prototype may break existing packages as it is not guarded by a compat bump to ease your testing. I am still very curious to hear about any issues you may experience.
• The default build directory is re-used even with different buildlabels, so you still need to use explicit build dirs for buildsystems that prefer building in a separate directory (e.g. meson).
• udebs are not automatically tagged with an udeb buildlabel. This is partly by design as some source packages only build udebs (and no regular debs). If they were automatically tagged, the existing packages would break.
• Label names are used in path names, so you may want to refrain from using too exciting label names.
• It is experimental feature and may change without notice. (Yes, I thought it deserved repeating)

Filed under: Debhelper, Debian

Since debhelper/10.3, there has been a number of performance related changes. The vast majority primarily improves bulk performance or only have visible effects at larger input sizes. Most visible cases are:

• dh + dh_* now scales a lot better for large number of binary packages. Even more so with parallel builds.
• Most dh_* tools are now a lot faster when creating many directories or installing files.
• dh_prep and dh_clean now bulk removals.
• dh_install can now bulk some installations. For a concrete corner-case, libssl-doc went from approximately 11 seconds to less than a second. This optimization is implicitly disabled with exclude (among other).
• dh_installman now scales a lot better with many manpages. Even more so with parallel builds.
• dh_installman has restored its performance under fakeroot (regression since 10.2.2)

For debhelper, this mostly involved:

• avoiding fork+exec of commands for things doable natively in perl. Especially, when each fork+exec only process one file or dir.
• bulking as many files/dirs into the call as possible, where fork+exec is still used.
• caching / memorizing slow calls (e.g. in parts of pkgfile inside Dh_Lib)
• adding an internal API for dh to do bulk check for pkgfiles. This is useful for dh when checking if it should optimize out a helper.
• and, of course, doing things in parallel where trivially possible.

How to take advantage of these improvements in tools that use Dh_Lib:

• If you use install_ file,prog,lib,dir , then it will come out of the box. These functions are available in Debian/stable. On a related note, if you use doit to call install (or mkdir ), then please consider migrating to these functions instead.
• If you need to reset owner+mode (chown 0:0 FILE + chmod MODE FILE), consider using reset_perm_and_owner. This is also available in Debian/stable.
  • CAVEAT: It is not recursive and YMMV if you do not need the chown call (due to fakeroot).
• If you have a lot of items to be processed by a external tool, consider using xargs(). Since 10.5.1, it is now possible to insert the items anywhere in the command rather than just in the end.
• If you need to remove files, consider using the new rm_files function. It removes files and silently ignores if a file does not exist. It is also available since 10.5.1.
• If you need to create symlinks, please consider using make_symlink (available in Debian/stable) or make_symlink_raw_target (since 10.5.1). The former creates policy compliant symlinks (e.g. fixup absolute symlinks that should have been relative). The latter is closer to a ln -s call.
• If you need to rename a file, please consider using rename_path (since 10.5). It behaves mostly like mv -f but requires dest to be a (non-existing) file.
• Have a look at whether on_pkgs_in_parallel() / on_items_in_parallel() would be suitable for enabling parallelization in your tool.
  • The emphasis for these functions is on making parallelization easy to add with minimal code changes. It pre-distributes the items which can lead to unbalanced workloads, where some processes are idle while a few keeps working.

Credits: I would like to thank the following for reporting performance issues, regressions or/and providing patches. The list is in no particular order:

• Helmut Grohne
• Kurt Roeckx
• Gianfranco Costamagna
• Iain Lane
• Sven Joachim
• Adrian Bunk
• Michael Stapelberg

Should I have missed your contribution, please do not hesitate to let me know.
Filed under: Debhelper, Debian

Since debhelper/10.3, there has been a number of performance related changes. The vast majority primarily improves bulk performance or only have visible effects at larger input sizes. Most visible cases are:

• dh + dh_* now scales a lot better for large number of binary packages. Even more so with parallel builds.
• Most dh_* tools are now a lot faster when creating many directories or installing files.
• dh_prep and dh_clean now bulk removals.
• dh_install can now bulk some installations. For a concrete corner-case, libssl-doc went from approximately 11 seconds to less than a second. This optimization is implicitly disabled with exclude (among other).
• dh_installman now scales a lot better with many manpages. Even more so with parallel builds.
• dh_installman has restored its performance under fakeroot (regression since 10.2.2)

For debhelper, this mostly involved:

• avoiding fork+exec of commands for things doable natively in perl. Especially, when each fork+exec only process one file or dir.
• bulking as many files/dirs into the call as possible, where fork+exec is still used.
• caching / memorizing slow calls (e.g. in parts of pkgfile inside Dh_Lib)
• adding an internal API for dh to do bulk check for pkgfiles. This is useful for dh when checking if it should optimize out a helper.
• and, of course, doing things in parallel where trivially possible.

How to take advantage of these improvements in tools that use Dh_Lib:

• If you use install_ file,prog,lib,dir , then it will come out of the box. These functions are available in Debian/stable. On a related note, if you use doit to call install (or mkdir ), then please consider migrating to these functions instead.
• If you need to reset owner+mode (chown 0:0 FILE + chmod MODE FILE), consider using reset_perm_and_owner. This is also available in Debian/stable.
  • CAVEAT: It is not recursive and YMMV if you do not need the chown call (due to fakeroot).
• If you have a lot of items to be processed by a external tool, consider using xargs(). Since 10.5.1, it is now possible to insert the items anywhere in the command rather than just in the end.
• If you need to remove files, consider using the new rm_files function. It removes files and silently ignores if a file does not exist. It is also available since 10.5.1.
• If you need to create symlinks, please consider using make_symlink (available in Debian/stable) or make_symlink_raw_target (since 10.5.1). The former creates policy compliant symlinks (e.g. fixup absolute symlinks that should have been relative). The latter is closer to a ln -s call.
• If you need to rename a file, please consider using rename_path (since 10.5). It behaves mostly like mv -f but requires dest to be a (non-existing) file.
• Have a look at whether on_pkgs_in_parallel() / on_items_in_parallel() would be suitable for enabling parallelization in your tool.
  • The emphasis for these functions is on making parallelization easy to add with minimal code changes. It pre-distributes the items which can lead to unbalanced workloads, where some processes are idle while a few keeps working.

Credits: I would like to thank the following for reporting performance issues, regressions or/and providing patches. The list is in no particular order:

• Helmut Grohne
• Kurt Roeckx
• Gianfranco Costamagna
• Iain Lane
• Sven Joachim
• Adrian Bunk
• Michael Stapelberg

Should I have missed your contribution, please do not hesitate to let me know.
Filed under: Debhelper, Debian

Here's what happened in the Reproducible Builds effort between Sunday July 9 and Saturday July 15 2017: Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #867771 filed against python-trollius.
  • #867773 filed against relatorio.
  • #867781 filed against gcin.
  • #867890 filed against apache-directory-jdbm.
  • #867900 filed against yapsy.
  • #867906 filed against wcsaxes.
• Chris Lamb:
  • #867753 filed against grunt (forwarded upstream).
  • #867848 filed against gconf (forwarded upstream).
  • #868133 filed against grep.
  • #868321 filed against node-marked-man (forwarded-upstream).
• Bernhard M. Wiedemann:
  • chromium
  • chromium
  • kubernetes
  • qtscriptgenerator
  • crash merged
  • ghc-rpm-macros merged
  • samba
  • htmldoc
  • efl
  • webkitgtk3 merged
  • fence-agents
  • fio merged

Reviews of unreproducible packages 13 package reviews have been added, 12 have been updated and 19 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• build_path_captured_by_valac
• timestamps_in_javascript_generated_by_node_grunt_banner

3 issue types have been updated:

• Add fix for timestamps_embedded_in_manpages_by_node_marked_man toolchain issue.
• Add fix for timestamps_in_javascript_generated_by_node_grunt_banner toolchain issue.
• timestamps_in_javascript_generated_by_node_grunt_banner is actually in src:grunt

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (47)

diffoscope development Version 84 was uploaded to unstable by Mattia Rizzolo. It included contributions already reported from the previous weeks, as well as new ones:

• Ximin Luo:
  • Attempt to fix fsimage test on Jenkins
  • Use a tempdir rather than ./cache for guestfs cache
  • Add more debugging output to test_fsimage

After the release, development continued in git with contributions from:

• Mattia Rizzolo:
  • RequiredToolNotFound.get_package(): just call the new get_package_provider()
  • Add a get_package_provider() function, returning the package name that best matches the system
  • Move from deprecated platform.linux_distribution() to the external package distro

strip-nondeterminism development Versions 0.036-1, 0.037-1 and 0.038-1 were uploaded to unstable by Chris Lamb. They included contributions from:

• Niels Thykier:
  • Add missing use statements in handler modules
  • dh_strip_nondeterminism: Assumes tmpdir() exists
  • File::StripNondeterminism:
    • Apply perltidy
    • Lazy load most handlers
    • Lazy load remaining handlers
• Chris Lamb:
  • Add missing File::Temp imports in JAR and PNG handlers. This appears to have been exposed by lazily-loading handlers in #867982.

reprotest development Development continued in git with contributions from:

• Ximin Luo:
  • presets: use newer flag --no-sign for dpkg-buildpackage
  • Document --diffoscope-args= --exclude-directory-metadata and use it in presets
• Mattia Rizzolo:
  • Bump debhelper compat level to 10.
  • Bump Standards-Version to 4.0.0.

buildinfo.debian.net development

• Chris Lamb:
  • Avoid a race condition between check-and-creation of Buildinfo instances.

tests.reproducible-builds.org

• Mattia Rizzolo:
  • Make database backups quicker to restore by avoiding --column-inserts's pg_dump option.
  • Fixup the deployment scripts after the stretch migration.
  • Fixup Apache redirects that were broken after introducing the buster suite
  • Fixup diffoscope jobs that were not always installing the highest possible version of diffoscope
• Holger Levsen:
  • Add a node health check for a too big jenkins.log.

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

For the first time in at least two years (and probably even longer), debhelper with the 10.6.2 upload broke the 1 minute milestone for build time (by mere 2 seconds look for Build needed 00:00:58, [ ] ). Sadly, the result it is not deterministic and the 10.6.3 upload needed 1m + 5s to complete on the buildds. This is not the result of any optimizations I have done in debhelper itself. Instead, it is the result of questionable use of developer time for the sake of meeting an arbitrary milestone. Basically, I made it possible to parallelize more of the debhelper build (10.6.1) and finally made it possible to run the tests in parallel (10.6.2). In 10.6.2, I also made the most of the tests run against all relevant compat levels. Previously, it would only run the tests against one compat level (either the current one or a hard-coded older version). Testing more than one compat turned out to be fairly simple given a proper test library (I wrote a Test::DH module for the occasion). Below is an example, which is the new test case that I wrote for Debian bug #866570.

$ cat t/dh_install/03-866570-dont-install-from-host.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More;
use File::Basename qw(dirname);
use lib dirname(dirname(__FILE__));
use Test::DH;
use File::Path qw(remove_tree make_path);
use Debian::Debhelper::Dh_Lib qw(!dirname);
plan(tests => 1);
each_compat_subtest  
  my ($compat) = @_;
  # #866570 - leading slashes must *not* pull things from the root FS.
  make_path('bin');
  create_empty_file('bin/grep-i-licious');
  ok(run_dh_tool('dh_install', '/bin/grep*'));
  ok(-e "debian/debhelper/bin/grep-i-licious", "#866570 [$ compat ]");
  ok(!-e "debian/debhelper/bin/grep", "#866570 [$ compat ]");
  remove_tree('debian/debhelper', 'debian/tmp');
 ;

I have cheated a bit on the implementation; while the test runs in a temporary directory, the directory is reused between compat levels (accordingly, there is a clean up step at the end of the test). If you want debhelper to maintain this exclusive (and somewhat arbitrary) property (deterministically), you are more than welcome to help me improve the Makefile. I am not sure I can squeeze any more out of it with my (lack of) GNU make skills.
Filed under: Debhelper, Debian

Here's what happened in the Reproducible Builds effort between Sunday June 25 and Saturday July 1 2017: Upcoming and past events Our next IRC meeting is scheduled for July 6th at 17:00 UTC (agenda). Topics to be discussed include an update on our next Summit, a potential NMU campaign, a press release for buster, branding, etc. Toolchain development and fixes

• James McCoy reviewed and merged Ximin Luo's script debpatch into the devscripts Git repository. This is useful for rebasing our patches onto new versions of Debian packages.

Packages fixed and bugs filed

• Adrian Bunk:
  • #866713 filed against debhelper.
• Chris Lamb:
  • #865994 filed against xabacus.
  • #866164 filed against qmidinet.
  • #866169 filed against singularity-container.
  • #866330 filed against cd-hit.

Ximin Luo uploaded dash, sensible-utils and xz-utils to the deferred uploads queue with a delay of 14 days. (We have had patches for these core packages for over a year now and the original maintainers seem inactive so Debian conventions allow for this.) Patches submitted upstream:

• openmpi
• gtk-doc fixed by sorting directory listings
• samba
• TeX
• nedit
• criu
• tvheadend sort
• tvheadend date
• cfengine

Reviews of unreproducible packages 4 package reviews have been added, 4 have been updated and 35 have been removed in this week, adding to our knowledge about identified issues. One issue types has been updated:

• Add upstream URL for random_order_of_pdf_ids_generated_by_latex

One issue type has been added:

• timestamps_in_manpages_added_by_golang_go_flags

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (68)
• Daniel Schepler (1)
• Michael Hudson-Doyle (1)
• Scott Kitterman (6)

diffoscope development

• Daniel Shahaf:
  • Fix markup in the man page synopsis. Thanks to Niels Thykier for the report. (Closes: #866577)
• Mattia Rizzolo:
  • Bump backport version check in debian/rules
• Ximin Luo:
  • Fix a progressbar failure
  • Put the 400MB "fsimage" cache in a more obvious place
  • Fix CI tests under Python 3.6
  • Add a --exclude-directory-metadata option. (Closes: #866241)
  • Raise warning for getfacl. Remove a redundant try-clause
  • Fix recursive indentation of headers
  • Use a loop rather than recursion
  • In html-dir mode, put css/icon in separate files to avoid duplication
  • diffcontrol UI tweaks
  • Split index pages up if they get too big

tests.reproducible-builds.org

• Vagrant Cascadian working on testing Debian:
  • Upgraded the 27 armhf build machines to stretch.
  • Fix mtu check to only display status when eth0 is present.
• Helmut Grohne worked on testing Debian:
  • Limit diffoscope memory usage to 10GB virtual per process. It currently tends to use 50GB virtual, 36GB resident which is bad for everything else sharing the machine. (This is #865660)
• Alexander Couzens working on testing LEDE:
  • Add multiple architectures / targets.
  • Limit LEDE and OpenWrt jobs to only allow one build at the same time using the jenkins build blocker plugin.
  • Move git log -1 > .html to node document environment().
  • Update TODOs.
• Mattia Rizzolo working on testing Debian:
  • Updated the maintenance script for postgresql-9.6.
  • Restored the postgresql database from backup after Holger accidently purged it.
• Holger Levsen working on:
  • Merging all the above commits.
  • Added a check for (known) Jenkins zombie jobs and report them. (This is an long known problem with jenkins; deleted jobs sometimes come back )
  • Upgraded the remaining amd64 nodes to stretch.
  • Accidentially purged postgres-9.4 from jenkins, so we could test our backups
  • Updated our stretch upgrade TODOs.

Misc. This week's edition was written by Chris Lamb, Ximin Luo, Holger Levsen, Bernhard Wiedemann, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Earlier today, I uploaded debhelper version 10.5.1 to unstable. The following are some highlights compared to version 10.2.5:

• debhelper now supports the meson+ninja build system. Kudos to Michael Biebl.
• Better cross building support in the makefile build system (PKG_CONFIG is set to the multi-arched version of pkg-config). Kudos to Helmut Grohne.
• New dh_missing helper to take over dh_install list-missing/ fail-missing while being able to see files installed from other helpers. Kudos to Michael Stapelberg.
• dh_installman now logs what files it has installed so the new dh_missing helper can see them as installed.
• Improve documentation (e.g. compare and contrast the dh_link config file with ln(1) to assist people who are familiar with ln(1))
• Avoid triggering a race-condition with libtool by ensuring that dh_auto_install run make with -j1 when libtool is detected (see Debian bug #861627)
• Optimizations and parallel processing (more on this later)

There are also some changes to the upcoming compat 11

• Use /run as run state dir for autoconf
• dh_installman will now guess the language of a manpage from the path name before using the extension.

Filed under: Debhelper, Debian

Updating Britney often makes our life easier. Like:

• smooth-updates (winter 2011)
• Adam s accidental upgrade of the original auto-hinter[1] (winter 2012)
• A non-leaking installability tester (winter 2013)
• Fix non-determinism of easy-hinted smooth updatable packages (summer 2014)
• Support for non-hinted multi-item migrations (autumn 2015)

Concretely, transitions have become a lot easier. When I joined the release team in the summer 2011, about the worst thing that could happen was discovering that two transitions had become entangled. You would have to wait for everything to be ready to migrate at the same time and then you usually also had to tell Britney what had to migrate together. Today, Britney will often (but not always) de-tangle the transitions on her own and very often figure out how to migrate packages without help. The latter is in fact very visible if you know where to look. Behold, the number of manual easy and hint -hints by RT members per year[2]:

Year   Total   easy   hint
-----+-------+------+-----
2005     53      30    23 
2006    146      74    72
2007     70      40    30
2008    113      68    45
2009    229     171    58
2010    252     159    93
2011    255     118   137
2012     29      21     8
2013     36      30     6
2014     20      20     0
2015     25      17     8
2016     16      11     5
2017      1       1     0

As can be seen, the number of manual hints drop by factor of ~8.8 between 2011 and 2012. Now, I have not actually done a proper statistical test of the data, but I have a hunch that drop was significant (see also [3] for a very short data discussion). In conclusion: Smooth-updates (which was enabled late in 2011) have been a tremendous success. [1] A very surprising side-effect of that commit was that the ( original ) auto-hinter could now solve a complicated haskell transition. Turns out that it works a lot better, when you give correct information! [2] As extracted by the following script and then manually massaged into an ASCII table. Tweak the in-line regex to see different hints.

respighi.d.o$ cd "/home/release/britney/hints" && perl -E '
    my (%years, %hints);
    while(<>)   
        chomp;
        if (m/^\#\s*(\d 4 )(?:-?\d 2 -?\d 2 );/ or m/^\#\s*(?:\d+-\d+-\d+\s*[;:]?\s*)?done\s*[;:]?\s*(\d 4 )(?:-?\d 2 -?\d 2 )/)  
             $year = $1; next;
          
         if (m/^((?:easy hint) .*)/)  
             my $hint = $1; $years $year ++ if defined($year) and not $hints $hint ++;
             next;
          
         if (m/^\s*$/)   $year = undef; next;  
     ;
    for my $year (sort(keys(%years)))   
        my $count = $years $year ;
        print "$year: $count\n"
     ' * OLD/jessie/* OLD/wheezy/* OLD/Lenny/* OLD/*

[3] I should probably mention for good measure that extraction is ignoring all hints where it cannot figure out what year it was from or if it is a duplicate. Notable it is omitting about 100 easy/hint-hints from OLD/Lenny (compared to a grep -c), which I think accounts for the low numbers from 2007 (among other). Furthermore, hints files are not rotated based on year or age, nor am I sure we still have all complete hints files from all members.
Filed under: Debian, Release-Team

The soft freeze has been on going for almost a month now and the full stretch freeze will start tomorrow night (UTC). It has definitely been visible in the number of unblock requests that we have received so far. Fortunately, we are no where near the rate of the jessie freeze. At the moment, all unblock requests are waiting for the submitter (either for a clarification or an upload). Looking at stretch at a glance (items are in no particular order):

• Secure boot support is progressing too slowly.
• The openssl transition is still on-going and badly needs closure.
• The MySQL to MariaDB transition is done.
• The gcc-5 removal almost complete missing only 2 packages.
• About 150 RC bugs (in key packages) need to be handled
• We are missing a stretch-backport suite so DSA can start their upgrade tests
• The release checklist have a few items that could be completed now.

Secure boot support Currently, we are blocked on two items:

• We do not have signing done yet for the boot packages (not even manual signing).
• Our shim is not yet signed, so no hardware would be trusting our boot chain out of the box.

After they are done, we are missing a handful of uploads to provide a signed bootloader etc. plus d-i and some infrastructure bits need to be updated. At the moment, we are waiting for a handful of key people/organisations to move on their part. As such, there is not a lot you can do to assist here (unless you are already involved in the work).
On the flip side, if both of these items are resolved soon, there is a good chance that we can support secure boot in stretch.See bug#820036 and blockers for more information on the remaining items. Where can you help with the release? At the moment, the best you can do is to:

• Test (packages, upgrades, etc.) and report bugs
• File bugs against release-notes for issues that should be documented
• Fix RC bugs (please see the next section)

Release Critical Bug report The UDD bugs interface currently knows about the following release critical bugs:

• In Total: 1148 (Including 193 bugs affecting key packages)
  • Affecting stretch: 294 (key packages: 158)
    That s the number we need to get down to zero before the release. They can be split in two big categories:
    • Affecting stretch and unstable: 232 (key packages: 134)
      Those need someone to find a fix, or to finish the work to upload a fix to unstable:
      • 30 bugs are tagged patch . (key packages: 21)
        Please help by reviewing the patches, and (if you are a DD) by uploading them.
      • 17 bugs are marked as done, but still affect unstable. (key packages: 5)
        This can happen due to missing builds on some architectures, for example. Help investigate!
      • 185 bugs are neither tagged patch, nor marked done. (key packages: 108)
        Help make a first step towards resolution!
    • Affecting stretch only: 62 (key packages: 24)
      Those are already fixed in unstable, but the fix still needs to migrate to stretch. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
      • 36 bugs are in packages that are unblocked by the release team. (key packages: 14)
      • 26 bugs are in packages that are not unblocked. (key packages: 10)

Filed under: Debian, Release-Team

As of today, britney now fetches reports from piuparts.debian.org and uses it as a part of her evaluation for package migration. As with her RC bug check, we are only preventing (known) regressions from migrating. The messages (subject to change) look something like:

• Piuparts tested OK
• Rejected due to piuparts regression
• Ignoring piuparts failure (Not a regression)
• Cannot be tested by piuparts (not a blocker)

If you want to do machine parsing of the Britney excuses, we also provide an excuses.yaml. In there, you are looking for excuses[X].policy_info.piuparts.test-results , which will be one of:

• pass
• regression
• failed
• cannot-be-tested

Enjoy.
Filed under: Debian, Release-Team

Yesterday, I just uploaded apt-file 3.1.2 into unstable, which comes with a few things I would like to highlight.

• We fixed an issue where apt-file would not show top-level files in source packages. (bug#676642). Thanks to Paul Wise for the proposed solution.
• Paul Wise also fixed a bug where apt-file list -I dsc <source-pkg> would fail to list all files in the source package if said file was also in other packages.
• We added filter-suites / filter-origins options that can be used to narrow the search space. Example: apt-file search --filter-suites unstable lintian/checks/

You can also set defaults in the config file if you want to always search in unstable, simply do:

# echo 'apt-file::Search-Filter::Suite "unstable";' >> /etc/apt/apt-file.conf

For the suite filter, either a code name ( sid ) or a suite name ( unstable ) will work. Please note that the filters are case-sensitive suites/code names generally use all lowercase, whereas origins appear to use title-case (i.e. unstable vs. Debian ).
Filed under: apt-file, Debian

Shipping Position Independent Executables and using read-only Global Offset Table was already possible for packages but needed package maintainers to opt-in for each package (see Hardening wiki) using the pie and bindnow Dpkg hardening flags. Many critical packages enabled the extra flags but there are still way more left out according to Lintian hardening-no-bindnow and hardening-no-pie warnings. Now we can change that. We can make those hardening flags the default for every package.
We already have the needed patches for GCC (#835148) and dpkg (#835146, #835149). We already have all packages rebuilt once to test which breaks (Thanks to Lucas Nussbaum!). The Release Team already asked porters if they feel their ports ready for enabling PIE and most ports tentatively opted-in (Thanks to Niels Thykier for pushing this!). What is left is fixing the ~75 open bugs found during the test rebuilds and this is where You can help, too! Please check if your packages are affected or give a helping hand to other maintainers who need it. (See PIEByDefaultTransition wiki for hints on fixing the bugs.) Many thanks to those who already fixed their packages! If we can get past those last bugs we can enable those badly needed security features and make Stretch the most secure release ever!

It is the first of October and that means the transition freeze is roughly one month away (Nov 5th 2016). In other words, this is the final boarding call for transitions . Other milestone dates:

• If you want new packages into Stretch, you have about 3 months (Jan 5th 2017). This includes auto removed packages.
• Automatic updates of existing packages stop in 4 months (Feb 5th 2017)

Filed under: Debian, Release-Team

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Fixed a number of instances in the Django web development framework where methods had mutable defaults arguments such as lists or dictionaries. (#7253)
• Made a large number of improvements to travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every update:
  • Worked with Evgeni Golov to run autopkgtest and autodep8 tests after builds. (#23, #24 & #25)
  • Ensured that all remote branches are created locally so that packages that specify options such as pristine_tar=True in debian/gbp.conf work as expected. (commit)
  • Fixed an issue where checking out other branches did not reliably not work due to Travis cloning the repository with --depth=50. (#21)
  • Updated the documentation after prompting from Valerie Young. (db7238 & 470d24)
• Improved my Chrome extension for the FastMail web interface to collapse whitespace when appending to the subject. (#9)
• Submitted a pull request for the Handbrake video transcoder to make the build reproducible. (#320)
• Updated the configuration for my OpenWRT-based router (a device I use as a range extender and to save tedious reconfiguration of all my wifi devices in new locales) to enter a "fallback" mode if it cannot connect to the client network. This works around a longstanding "wontfix" upstream issue. (diff)
• Updated django-agpl, my reusable Django application to assist in releasing projects under the GNU Affero General Public License:
  • Merged support from defivelo to support Django 1.8. (#1)
  • Improved the code quality & testability throughout the package. (diff)
• Filed in issue in Mailvelope (a browser extension for OpenPGP encryption) to support composing/viewing messages in a fixed-width font. (#422)
• Blogged about how to write your first Lintian check as well as on the progress bar I added to Diffoscope.

---

What happened in the Reproducible Builds effort between Sunday September 11 and Saturday September 17 2016: Toolchain developments Ximin Luo started a new series of tools called (for now) debrepatch, to make it easier to automate checks that our old patches to Debian packages still apply to newer versions of those packages, and still make these reproducible. Ximin Luo updated one of our few remaining patches for dpkg in #787980 to make it cleaner and more minimal. The following tools were fixed to produce reproducible output:

• naturaldocs/1.51-2 by Petter Reinholdtsen, original patch by Chris Lamb.

Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible - in our current test setup - after being fixed:

• elog/3.1.2-1-1 by Roger Kalt, original patch by Reiner Herrmann.
• eyed3/0.6.18-3 by Petter Reinholdtsen, original patch by Chris Lamb.
• frog/0.13.5-1 by Maarten van Gompel, original patch by Chris Lamb.
• gtranslator/2.91.7-3 by Andreas Henriksson, original patch by Reiner Herrmann.
• sozi/12.05-1.1 by Daniel Kahn Gillmor, original patch by Chris Lamb.

The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.)

• evince/3.21.92-1 by Michael Biebl.
• gnome-control-center/1:3.21.92-2 by Rapha l Hertzog.
• libipathverbs/1.3-2 by Ana Beatriz Guerrero Lopez.
• pagekite/0.5.8e-2 by Petter Reinholdtsen.

The following 3 packages were not changed, but have become reproducible due to changes in their build-dependencies: jaxrs-api python-lua zope-mysqlda. Some uploads have addressed some reproducibility issues, but not all of them:

• eurephia/1.1.0-6 by Alberto Gonzalez Iniesta, original patch by Chris Lamb.
• fdroidserver/0.7.0-1 by Hans-Christoph Steiner, original patch by Chris Lamb.
• mini-buildd/1.0.18 by Stephan S rken.
• nbc/1.2.1.r4+dfsg-3 by Petter Reinholdtsen, original patch by Chris Lamb.
• ncurses/6.0+20160910-1 by Sven Joachim, #818067 by Niels Thykier.
• python-kinterbasdb/3.3.0-4 by Santiago Vila, original patch by Chris Lamb.
• snapper/0.3.3-1 Hideki Yamane, original patch by Sascha Steinbiss.

Patches submitted that have not made their way to the archive yet:

• #838188 filed against ocaml by Johannes Schauer.

Reviews of unreproducible packages 462 package reviews have been added, 524 have been updated and 166 have been removed in this week, adding to our knowledge about identified issues. 25 issue types have been updated:

• Added a new annotation for issues called "fix-deterministic" to help us update package reviews more easily. This indicates whether we expect that an issue would always happen on Jenkins; i.e. if there is a successful build, then we know the issue is fixed for that package and can update our notes.
• Added random_order_in_sisu_javax_inject_named and too_much_input_for_diff.
• Removed timestamps_in_manpages_generated_by_ronn.
• Updated timestamps_in_allegro_dat_files. Additionally, 21 issues were marked with "fix-deterministic".

Weekly QA work FTBFS bugs have been reported by:

• Chris Lamb (10)
• Filip Pytloun (1)
• Santiago Vila (1)

diffoscope development A new version of diffoscope 60 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Mattia Rizzolo:
  • Various packaging and testing improvements.
• HW42:
  • minor wording fixes
• Reiner Herrmann:
  • minor wording fixes

It also included from changes previous weeks; see either the changes or commits linked above, or previous blog posts 72 71 70. strip-nondeterminism development New versions of strip-nondeterminism 0.027-1 and 0.028-1 were uploaded to unstable by Chris Lamb. It included contributions from:

• Chris Lamb:
  • Testing improvements, including better handling of timezones.

disorderfs development A new version of disorderfs 0.5.1 was uploaded to unstable by Chris Lamb. It included contributions from:

• Andrew Ayer and Chris Lamb:
  • Support relative paths for ROOTDIR; it no longer needs to be an absolute path.
• Chris Lamb:
  • Print the behaviour (shuffle/reverse/sort) on startup to stdout.

It also included from changes previous weeks; see either the changes or commits linked above, or previous blog posts 70. Misc. This week's edition was written by Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

We have been making a lot of minor changes to lintian.d.o and the underlying report framework. Most of them were hardly noticeable to the naked. In fact, I probably would not have spotted any of them, if I had not been involved in writing them. Nonetheless, I felt like sharing them, so here goes. User visible changes:

• The generated reports are now in HTML5 rather than XHTML. [commit:9652c2e]
• Add alt texts to graphs for [commit:83434ec]
• Wrap front page index in a <nav> and reword link text so they can stand alone [commit:13993c8]
• Fix index of maintainer names, so characters outside [A-Z] always go at the end plus add a link directly to that section. [commit:70a24ed, commit:94eb258]
• Put the banner links in a <nav> [commit:baa77fc]
• Always create an index of packages on the individual maintainer pages [commit:124f9a8]
• Wrap the above-mentioned indices in a <nav> tag [commit:2ecea9b]
• Make the links in these indices self-contained [commit:bfc5292, commit:1043d4d]
• Fix mojibake in maintainer names [commit:e907388]

In case you were wondering, the section title is partly a pun as half of these changes were intended to assist visually impaired users. They were triggered by me running into Sam Hartmann at DebConf16, where I asked him about how easy Debian s websites were for blind people. Allegedly, we are generally doing quite good in his opinion (with one exception, for which Sam filed Bug#830213), which was a positive surprise for me. On a related note: Thanks Luke Faraone and Asheesh Laroia for getting helping me started on these changes. Reporting framework / Internal changes:

• Harness supports a no generate reports to make report generation optional. [commit:21fab0b]
• Reduced memory requirements for reports generation by ~15% [commit:534d169]
• Refactored reporting frontends to share the same bootstrap code as lintian/lintian-info ( dplint ) [commit:7d9a7d1, commit:97cf104, plus a bunch of refactoring leading up to those commits]
• Add sub-resource integrity fields in <link> tags [commit:27a367d]
• Add a Content-Security-Policy (via a <meta http-equiv>-tag) [commit:b022a39, commit:30bdc90]
• Support mirrors using Packages.xz files and without SHA1 fields [commit:d9cf314, commit:cb05f1d]
• Shuffle the lintian backlog to randomize the workload [commit:9b8873f]
• Stop lintian when the dead-line/run-time limit is exceeded [commit:7032be5, commit:7b3190d, commit:62532f8]

With the last change + the no generate reports option, we were able to schedule lintian more frequently. Originally, lintian only ran once a day. With the no generate reports , we added a second run and with the last changes, we bumped it to 4 times a day. Unsurprisingly, it means that we are now reprocessing the archive a lot faster than previously. All of the above is basically the all the note-worthy changes on the Lintian reporting framework since the Partial rewrite of lintian s reporting setup (~1 years ago).
Filed under: Debian, Lintian

Today, debhelper 10 was uploaded to unstable and is coming to a mirror near you really soon now . The actual changes between version 9.20160814 and version 10 are rather modest. However, it does mark the completion of debhelper compat 10, which has been under way since early 2012. Some highlights from compat 10 include:

• The dh sequence in compat 10 automatically regenerate autotools files via dh_autoreconf
• The dh sequence in compat 10 includes the dh-systemd debhelper utilities
• dh sequencer based packages now defaults to building in parallel (i.e. parallel is default in compat 10)
• dh_installdeb now properly shell-escapes maintscript arguments.

For the full list of changes in compat 10, please review the contents of the debhelper(7) manpage. Beyond that, you may also want to upgrade your lintian to 2.5.47 as it is the first version that knows that compat 10 is stable.
Filed under: Debhelper, Debian

Thanks to Lucas Nussbaum, we now have a UDD script to filter/select key packages. Some example use cases: Which key packages used compat 4?

# Data file compat-4-packages (one *source* package per line)
$ curl --silent --data-binary @compat-4-packages \
  https://udd.debian.org/cgi-bin/select-key-packages.cgi
alsamixergui
apg
[...]
sgml-base
wwwconfig-common

Also useful for things like bug#830997, which was my excuse for requesting this. Is package foo a key package (yet)?

$ is-key-pkg()   
 RES=$(echo "$1"   curl --silent --data-binary @- \
    https://udd.debian.org/cgi-bin/select-key-packages.cgi)
 if [ "$RES" ]; then
   echo yes
 else
   echo no
 fi
 
$ is-key-pkg bash
yes
$ is-key-pkg mscgen
no
$ is-key-pkg NotAPackage
no

Above shell snippets might need tweaking for better error handling, etc. Once again, thanks to Lucas for the server-side UDD script.
Filed under: Debian

Today, we have completed our first Britney run with mips64el enabled in testing. At the current time, the set of packages in mips64el are not very connected (and you probably cannot even install build-essential yet[1]). Hopefully this will change over the next few days. For now, Britney does not enforce installability of packages on mips64el in general, so do not expect the architecture to be stable at the moment. Cheat sheet for package maintainers:

• Issues with builds (only) on mips64el are not blockers for testing migration (nor RC yet).
  • Such bugs should be filed as important for now (unless they also affect a release architecture, in which case you should still make it an RC bug)
• Your package be an older version on mips64el compared to other architectures.
• Britney may decide to break your package on mips64el if it means something else can migrate on a release architecture.

We will slowly remove these special cases for mips64el as it matures in testing. [1] Update on this: mips64el currently does not have a libc library yet, so build-essential is definitely not installable at the moment. It will hopefully migrate very soon.
Filed under: Debian, Release-Team

Debian packages can run arbitrary code via maintainer scripts (sometimes shortened into maintscripts ) during installation/removal etc. While they certainly have their use cases, their failure modes causes exciting bugs like fails to install or the dreaded fails to remove . They also have other undesirable effects such as:

• Bugs in/Updates to auto-generated snippets require a rebuild of all packages (not to mention the obvious code-duplication in all packages).
• In case of circular dependencies[1] all having postinst scripts, dpkg will have to guess which package to configure first.
• They require forking a shell at least once for each maintscript.
• They complicate the implementations of e.g. detached chroot creation.

Accordingly, I think we should aim for a more declarative packaging style. To help facilitate this, I have implemented 3 tracking tags in Lintian.

• no-ctrl-scripts in 2.5.44
• ctrl-script in 2.5.44
• debhelper-autoscript-in-maintainer-scripts in 2.5.45

With these, we were able to learn that 73.5% of all packages do not have any of these scripts. But I can now also produce a list of helpers that insert the most maintainer script snippets. The current top 15 is:

1. dhpython with 3775 instances
   • This is an umbrella for all helpers using dh-python s python module, see #827774.
2. dh_installmenu with 1861 instances
   • This will presumably drop as menu files should be replaced with desktop files.
3. dh_makeshlibs with 1396 remaining instances
   • This should be in rapid decline since we decided to replace its scripts with a dpkg trigger. Unfortunately, it can take quite a while until all of them have been rebuilt with a new version of debhelper
4. dh_installinit with 1224 instances
5. dh_python2 with 1168 instances
6. dh_installdebconf with 772 instances
7. dh_installdeb with 754 instances
   • These are the dpkg-maintscript-helper snippets for rm_conffile , mv_conffile etc. Hopefully in the near future, dpkg will support these directly.
8. dh_systemd_enable with 447 instances
9. dh_installemacsen with 179 instances
10. dh_icons with 165 instances
11. dh_installtex with 137 instances
12. dh_apache2 with 117 instances
13. dh_installudev with 98 instances
14. dh_installxfonts with 87 instances
15. dh_systemd_start with 79 instances

With this list, it seems to me that some obvious focus areas would be:

• Replacing the python scripts (I presume it is the byte-code handling, but I have not looked at this at all)
• Migrating away from menu files
• Support enabling + starting/stopping/restarting a service declaratively.
  • This might have a hidden requirement on declaratively creating service users if we want these packages to become truly maintscript-less .

Eventually we will also have to dig through all the manual maintainer scripts. But I think we got plenty to start with. [1] For some, circular dependencies in itself is an issue. I can certainly appreciate them as being suboptimal, but most of the issues we have are probably caused by insufficient tooling rather than a theoretical issue (that is, if we remove all postinst scripts).
Filed under: Debhelper, Debian, Lintian