I have been fiddling with setting up both iptables and tor on my local machine. Most of it was fairly easy to do, once I dedicated the time to actually do it. Configuring both at the same time also made things easier for me, but YMMV. Regardless, it did take quite a while researching, tweaking and testing most of that time was spent on the iptables front for me. I ended up doing this incrementally. The major 5 steps I went through were:

1. Created a basic incoming (INPUT) firewall enforcing
2. Installed tor + torsocks and aliased a few commands to run with torsocks
3. Created a basic outgoing (OUTPUT) firewall permissive
4. Make the outgoing firewall enforcing
5. Migrate the majority of programs and services to use tor.

Some of these overlapped time-wise and I certainly revisited the configuration a couple of times. A couple of things, that I learned:

• You probably want to have a look at netstat --listen -put --numeric when you write your INPUT firewall.
• The tor developers have tried a lot to make things easy. It is scary how often torsocks program [args] just works(tm).
  • That said, it does not always work.
• Tor and iptables (OUTPUT) can have a synergy effect on each other.
  • Notably, when it is easier to just torsocks a program than adding the necessary iptables rules.
• Writing iptables rules become a lot easier once:
  • You learn how to iptables s LOG rule
  • You use sensible-editor + iptables-restore or something like puppet s firewall module

Filed under: Debian

After 3 months of installing an automatic decrufter in DAK, it:

• has removed 689 cruft items from unstable and experimental
  • average removal rate being just shy of 230 cruft items/month
• has become the top 11th remover .
• is expected to become top 10 in 6 days from now and top 9 in 10 days.
  • This is assuming a continued average removal rate of 7.6 cruft items per day

On a related note, the FTP masters have removed 28861 items between 2001 and now. The average being 2061 items a year (not accounting for the current year still being open). Though, intriguingly, in 2013 and 2014 the FTP masters removed 3394 and 3342 items. With the (albeit limited) stats from the auto-decrufter, we can estimate that about 2700 of those were cruft items. One could certainly also check the removal messages and check for the common tags used in cruft removals. I leave that as an exercise to the curious readers, who are not satisfied with my estimate. :)
Filed under: Debian, Release-Team

Thanks to hard work of Adam, Julien, Jonathan, Matthias, Scott, Simon and many others, the GCC-5/libstdc++ transition has progressed to a state, where we are ready to migrate the bulk of it to testing. It should be a mostly smooth ride. However, there will a few packages that are going to be uninstallable in testing for a few days and some packages will be temporarily removed from testing. If APT is unable to provide you with an upgrade for all of your packages, please try again in a few days. We apologise for the inconvenience. Currently, we expect about 36 binary packages to become temporarily uninstallable on amd64 and 34 on i386. This involves Britney accepting at least 4800 change items on testing (some of these are removals). Many thanks to Julien for providing a proposed set of hints and Adam extending them. Update: We now got a list of the packages being removed and a list of packages becoming uninstallable. It will be available on debian-devel within 20 minutes from now.
Filed under: Debian

So, yesterday, I unbroke dak twice even! It is of course slightly less awesome that one of the broken parts was in code written by yours truly. Anyhow: Unbreaking the dak auto-decrufter You may remember the auto-decrufter, which I added to dak. As a safety measure, it bails out when in doubt about which removal breaks what package. Turns out it was often in doubt, because the code had a bug. Of course, nothing that could not be solved with a patch. Thanks to Ansgar for merging this. :) Unbreaking dak generate-releases As a part of migrating apt-file to use APTs new acquire system (from experimental), I learned APT really likes having checksums for everything. Now including checksums for both the compressed file and the uncompressed file. Sadly, dak had optimised out the uncompressed checksums for Contents files, but even after removing that optimisation (and Ganneff unbreaking my dinstall breakage) some Contents files still did not have an checksum for the uncompressed Contents file. After some sophisticated debugging (read: printf-debugging ), I finally discovered the issue and submitted a patch. Thanks to Ansgar and Ganneff for merging (and fixing my dinstall breakage).
Filed under: Debian

The other day, I wrote about our recent performance tuning in lintian. Among other things, we reduced the memory usage by ~33%. The effect was also reproducible on libreoffice (4.2.5-1 plus its 170-ish binaries, arch amd64), which started at ~515 MB and was reduced to ~342 MB. So this is pretty great in its own right But at this point, I have seen what was in Pandora s box . By which, I mean the two magical numbers 1.7kB per file and 2.2kB per directory in the package (add +250-300 bytes per entry in binary packages). This is before even looking at data from file(1), readelf, etc. Just the raw index of the package. Depending on your point of view, 1.7-2.2kB might not sound like a lot. But for the lintian source with ~1 500 directories and ~3 300 non-directories, this sums up to about 6.57MB out of the (then) usage at 12.53MB. With the recent changes, it dropped to about 1.05kB for files and 1.5kB for dirs. But even then, the index is still 4.92MB (out of 8.48MB). This begs the question, what do you get for 1.05kB in perl? The following is a dump of the fields and their size in perl for a given entry:

lintian/vendors/ubuntu/main/data/changes-file/known-dists: 1077.00 B
  _path_info: 24.00 B
  date: 44.00 B
  group: 42.00 B
  name: 123.00 B
  owner: 42.00 B
  parent_dir: 24.00 B
  size: 42.00 B
  time: 42.00 B
  (overhead): 694.00 B

With time, date, owner and group being fixed sized strings (at most 15 characters). The size and _path_info fields being integers, parent_dir a reference (nulled). Finally, the name being a variable length string. Summed the values take less than half of the total object size. The remainder of ~700 bytes is just overhead . Time for another clean up:

• The ownership fields are usually always root/root (0/0). So let s just omit them when they satisfy said assumption. [f627ef8]
  • This is especially true for source packages where lintian ignores the actual value and just uses root/root .
• The Lintian::Path API has always had a cop-out on the size field for non-files and it happens to be 0 for these. Let s omit the field if the value was zero and save 0.17MB on lintian. [5cd2c2b]
  • Bonus: Turns out we can save 18 bytes per non-zero size by insisting on the value being an int.
• Unsurprisingly, the date and time fields can trivially be merged into one. In fact, that makes time redundant as nothing outside Lintian::Path used its value. So say goodbye to time and good day to 0.36MB more memory. [f1a7826]

Which leaves us now with:

lintian/vendors/ubuntu/main/data/changes-file/known-dists: 698.00 B
  _path_info: 24.00 B
  date_time: 56.00 B
  name: 123.00 B
  parent_dir: 24.00 B
  size: 24.00 B
  (overhead): 447.00 B

Still a ~64% overhead, but at least we reduced the total size by 380 bytes (585 bytes for entries in binary packages). With these changes, the memory used for the lintian source index is now down to 3.62MB. This brings the total usage down to 7.01MB, which is a reduction to 56% of the original usage (a.k.a. the-almost-but-not-quite-50%-reduction ). But at least the results also carried over to libreoffice, which is now down to 284.83 MB (55% of original). The chromium-browser (source-only, version 32.0.1700.123-2) is down to 111.22MB from 179.44MB (61% of original, better results expected if processed with binaries). In closing, Lintian 2.5.34 will use slightly less memory than 2.5.33.
Filed under: Debian, Lintian

For quite a while, Lintian has been able to create performance logs (--perf-debug --perf-output perf.log) that help diagnose where lintian spends most of its runtime. I decided to make lintian output these logs on lintian.debian.org to help us spot performance issues, though I have not been very good at analysing them regularly. At the beginning of the month, I finally got around to look a bit into one of them. My findings on IRC triggered Mattia Rizzolo to create this following graph. It shows the accumulated runtime of each check/collection measured in seconds. Find these findings, we set out to solve some of the issues. This lead to the following changes in 2.5.33 (in no particular order):

• Increased buffer size in check/cruft.pm to reduce overhead [bc8b3e5] (S)
• Reduced overhead in strings(1) extraction [b058fef] (P)
• Reduced overhead in spell-checking [b824170] (S)
  • Also improves the performance of spellintian!
• Removed a high overhead check that did not work [2c7b922] (P)

Legend: S: run single threaded (1:1 performance improvement). P: run in parallel. Overall, I doubt the changes will give a revolutionary change in speed, but it should improve the 3rd, 4th and 5th slowest parts in Lintian. Beyond runtime performance, we got a few memory optimisations in the pipeline for Lintian 2.5.34:

• Remove member from non-dir nodes in the Lintian path graph (2%) [6365635]
• Remove two fields at the price of computing them as needed (~5%) [a696197 + 8dacc8e]
• Merge 4 fields into 1 (~8%) [5d49cd2 + fb074e4]
• Share some memory between package-based caches (18%) [ffc7174]

Combined these 6 commits reduce memory consumption in caches by ~33% compared to 2.5.33, when lintian processes itself. In absolute numbers, we are talking about a drop from 12.53MB to 8.48MB. The mileages can certainly vary depending on the package (mscgen only saw an ~25% improvement). Nevertheless, I was happy to list #715035 as being closed in 2.5.34. :)
Filed under: Debian, Lintian

Debian is undertaking a huge effort to develop a reproducible builds system. I'd like to thank you for that. This could be Debian's most important project, with how badly computer security has been going.

PerniciousPunk in Reddit's Ask me anything! to Neil McGovern, DPL. What happened in the reproducible builds effort this week: Toolchain fixes More tools are getting patched to use the value of the SOURCE_DATE_EPOCH environment variable as the current time:

• libxslt in #791815 (Dhole),
• texlive-bin via upstream (#792202) (akira),
• sphinx via upstream (Dmitry Shachnev).

In the reproducible experimental toolchain which have been uploaded:

• doxygen to support SOURCE_DATE_EPOCH (akira),
• debhelper now set SOURCE_DATE_EPOCH to the time of the latest debian/changelog entry when exporting build flags, patch sent as #791823 (Dhole),
• epydoc to support SOURCE_DATE_EPOCH (Reiner Herrmann),
• texlive-bin (akira) and libxslt (Dhole) with the aforementioned support for SOURCE_DATE_EPOCH.

Johannes Schauer followed up on making sbuild build path deterministic with several ideas. Packages fixed The following 311 packages became reproducible due to changes in their build dependencies : 4ti2, alot, angband, appstream-glib, argvalidate, armada-backlight, ascii, ask, astroquery, atheist, aubio, autorevision, awesome-extra, bibtool, boot-info-script, bpython, brian, btrfs-tools, bugs-everywhere, capnproto, cbm, ccfits, cddlib, cflow, cfourcc, cgit, chaussette, checkbox-ng, cinnamon-settings-daemon, clfswm, clipper, compton, cppcheck, crmsh, cupt, cutechess, d-itg, dahdi-tools, dapl, darnwdl, dbusada, debian-security-support, debomatic, dime, dipy, dnsruby, doctrine, drmips, dsc-statistics, dune-common, dune-istl, dune-localfunctions, easytag, ent, epr-api, esajpip, eyed3, fastjet, fatresize, fflas-ffpack, flann, flex, flint, fltk1.3, fonts-dustin, fonts-play, fonts-uralic, freecontact, freedoom, gap-guava, gap-scscp, genometools, geogebra, git-reintegrate, git-remote-bzr, git-remote-hg, gitmagic, givaro, gnash, gocr, gorm.app, gprbuild, grapefruit, greed, gtkspellmm, gummiboot, gyp, heat-cfntools, herold, htp, httpfs2, i3status, imagetooth, imapcopy, imaprowl, irker, jansson, jmapviewer, jsdoc-toolkit, jwm, katarakt, khronos-opencl-man, khronos-opengl-man4, lastpass-cli, lava-coordinator, lava-tool, lavapdu, letterize, lhapdf, libam7xxx, libburn, libccrtp, libclaw, libcommoncpp2, libdaemon, libdbusmenu-qt, libdc0, libevhtp, libexosip2, libfreenect, libgwenhywfar, libhmsbeagle, libitpp, libldm, libmodbus, libmtp, libmwaw, libnfo, libpam-abl, libphysfs, libplayer, libqb, libsecret, libserial, libsidplayfp, libtime-y2038-perl, libxr, lift, linbox, linthesia, livestreamer, lizardfs, lmdb, log4c, logbook, lrslib, lvtk, m-tx, mailman-api, matroxset, miniupnpd, mknbi, monkeysign, mpi4py, mpmath, mpqc, mpris-remote, musicbrainzngs, network-manager, nifticlib, obfsproxy, ogre-1.9, opal, openchange, opensc, packaging-tutorial, padevchooser, pajeng, paprefs, pavumeter, pcl, pdmenu, pepper, perroquet, pgrouting, pixz, pngcheck, po4a, powerline, probabel, profitbricks-client, prosody, pstreams, pyacidobasic, pyepr, pymilter, pytest, python-amqp, python-apt, python-carrot, python-django, python-ethtool, python-mock, python-odf, python-pathtools, python-pskc, python-psutil, python-pypump, python-repoze.tm2, python-repoze.what, qdjango, qpid-proton, qsapecng, radare2, reclass, repsnapper, resource-agents, rgain, rttool, ruby-aggregate, ruby-albino, ruby-archive-tar-minitar, ruby-bcat, ruby-blankslate, ruby-coffee-script, ruby-colored, ruby-dbd-mysql, ruby-dbd-odbc, ruby-dbd-pg, ruby-dbd-sqlite3, ruby-dbi, ruby-dirty-memoize, ruby-encryptor, ruby-erubis, ruby-fast-xs, ruby-fusefs, ruby-gd, ruby-git, ruby-globalhotkeys, ruby-god, ruby-hike, ruby-hmac, ruby-integration, ruby-jnunemaker-matchy, ruby-memoize, ruby-merb-core, ruby-merb-haml, ruby-merb-helpers, ruby-metaid, ruby-mina, ruby-net-irc, ruby-net-netrc, ruby-odbc, ruby-ole, ruby-packet, ruby-parseconfig, ruby-platform, ruby-plist, ruby-popen4, ruby-rchardet, ruby-romkan, ruby-ronn, ruby-rubyforge, ruby-rubytorrent, ruby-samuel, ruby-shoulda-matchers, ruby-sourcify, ruby-test-spec, ruby-validatable, ruby-wirble, ruby-xml-simple, ruby-zoom, rumor, rurple-ng, ryu, sam2p, scikit-learn, serd, shellex, shorewall-doc, shunit2, simbody, simplejson, smcroute, soqt, sord, spacezero, spamassassin-heatu, spamprobe, sphinxcontrib-youtube, splitpatch, sratom, stompserver, syncevolution, tgt, ticgit, tinyproxy, tor, tox, transmissionrpc, tweeper, udpcast, units-filter, viennacl, visp, vite, vmfs-tools, waffle, waitress, wavtool-pl, webkit2pdf, wfmath, wit, wreport, x11proto-input, xbae, xdg-utils, xdotool, xsystem35, yapsy, yaz. Please note that some packages in the above list are falsely reproducible. In the experimental toolchain, debhelper exported TZ=UTC and this made packages capturing the current date (without the time) reproducible in the current test environment. The following packages became reproducible after getting fixed:

• 389-admin/1.1.42-1 uploaded by Timo Aaltonen, original patch by Chris Lamb.
• aroarfw/0.1~beta5-2 uploaded by Patrick Matth i, original patch by akira.
• clfft/2.4-2 by Ghislain Antony Vaillant.
• custom-tab-width/1.0.1-3 by Daniel Kahn Gillmor.
• debirf/0.35 uploaded by Daniel Kahn Gillmor, original patch by Chris Lamb.
• enigmail/2:1.8.2-3 by Daniel Kahn Gillmor.
• flashproxy/1.7-4 by Ximin Luo.
• getdns/0.2.0-2 by Daniel Kahn Gillmor.
• glfw3/3.1.1-1 by James Cowgill, reported by akira.
• gpgme1.0/1.5.5-3 by Daniel Kahn Gillmor.
• jzmq/3.1.0-4 by Jan Niehusmann.
• libcommons-cli-java/1.3.1-1 by tony mancill.
• liblo/0.28-5 uploaded by Felipe Sateler, original patch by akira.
• libmoe/1.5.8-2 uploaded by TANIGUCHI Takaki, original patch by Chris Lamb.
• libnet-interface-perl/1.012-2 uploaded by gregor herrmann, original patch by Chris Lamb.
• libxmlenc-java/0.52+dfsg-4 by Emmanuel Bourg.
• lintian/2.5.33 by Niels Thykier.
• litecoin/0.10.2.2-1 uploaded by Dmitry Smirnov, original patch by Chris Lamb.
• node-ws/0.7.2+ds1.349b7460-1 by Ximin Luo.
• pyevolve/0.6~rc1+svn398+dfsg-7 uploaded by Christian Kastner, original patch by Juan Picca.
• sendmail/8.14.9-3 by Andreas Beckmann.
• uthash/1.9.9.1+git20150507-2 by Ilias Tsitsimpis, report by Jakub Wilk.

Ben Hutchings upstreamed several patches to fix Linux reproducibility issues which were quickly merged. Some uploads fixed some reproducibility issues but not all of them:

• apt-dater/1.0.2-1 uploaded by Patrick Matth i, original patch by Dhole, fixed upstream by Thomas Liske.
• argyll/1.7.0+repack-4 by J rg Frings-F rst.
• grads/2:2.0.2-4 by Alastair McKinstry.
• kfreebsd-10 by Steven Chamberlain.
• mariadb-10.0/10.0.20-2 by Otto Kek l inen.
• xtel/3.3.0-18 uploaded by Samuel Thibault, original patch by Dhole.

Uploads that should fix packages not in main:

• fglrx-driver/1:15.7-1 uploaded by Patrick Matth i, fixed by Andreas Beckmann.
• pycuda/2015.1.2-1 uploaded by Tomasz Rybak, patch by Juan Picca.

Patches submitted which have not made their way to the archive yet:

• #787675 on ricochet by Daniel Kahn Gillmor: use the debian/changelog date in the manpage.
• #791648 on fish by Chris Lamb: sort header files in documentation.
• #791691 on fritzing by Chris Lamb: sort the documentation author list using the C locale.
• #791834 on bitcoin by Reiner Herrmann: sort sources files using the C locale.
• #791845 on yacas by Reiner Herrmann: sort hints using the C locale.
• #791851 on scowl by Reiner Herrmann: sort dictionary files using the C locale.
• #791913 on ceph by Chris Lamb: sort configuration file names.
• #791923 on alpine by Chris Lamb: use debian/changelog date as build date and use debian as the builder hostname.
• #791960 on leveldb by Reiner Herrmann: sort sources files using th e C locale.
• #792054 on ben by Reiner Herrmann: use debian/changelog date as bui ld date.
• #792056 on val-and-rick by Reiner Herrmann: sort sources by filenames.

reproducible.debian.net A new package set has been added for lua maintainers. (h01ger) tracker.debian.org now only shows reproducibility issues for unstable. Holger and Mattia worked on several bugfixes and enhancements: finished initial test setup for NetBSD, rewriting more shell scripts in Python, saving UDD requests, and more debbindiff development Reiner Herrmann fixed text comparison of files with different encoding. Documentation update Juan Picca added to the commands needed for a local test chroot installation of the locales-all package. Package reviews 286 obsolete reviews have been removed, 278 added and 243 updated this week. 43 new bugs for packages failing to build from sources have been filled by Chris West (Faux), Mattia Rizzolo, and h01ger. The following new issues have been added: timestamps_in_manpages_generated_by_ronn, timestamps_in_documentation_generated_by_org_mode, and timestamps_in_pdf_generated_by_matplotlib. Misc. Reiner Herrmann has submitted patches for OpenWrt. Chris Lamb cleaned up some code and removed cruft in the misc.git repository. Mattia Rizzolo updated the prebuilder script to match what is currently done on reproducible.debian.net.

Debian now have over 22 000 source packages and 45 500 binary packages. To counter that, the FTP masters and I have created a dak tool to automatically remove packages from unstable! This is also much more efficient than only removing them from testing! :) The primary goal of the auto-decrufter is to remove a regular manual work flow from the FTP masters. Namely, the removal of the common cases of cruft, such as Not Built from Source (NBS) and Newer Version In Unstable (NVIU). With the auto-decrufter in place, such cruft will be automatically removed when there are no reverse dependencies left on any architecture and nothing Build-Depends on it any more. Despite the implication in the opening of this post, this will in fact not substantially reduce the numbers of packages in unstable. :) Nevertheless, it is still very useful for the FTP masters, the release team and packaging Debian contributors. The reason why the release team benefits greatly from this tool, is that almost every transition generates one piece of NBS -cruft. Said piece of cruft currently must be removed from unstable before the transition can progress into its final phase. Until recently that removal has been 100% manual and done by the FTP masters. The restrictions on auto-decrufter means that we will still need manual decrufts. Notably, the release team will often complete transitions even when some reverse dependencies remain on non-release architectures. Nevertheless, it is definitely an improvement. Omelettes and eggs: As an old saying goes You cannot make an omelette without breaking eggs . Less so when the only test suite is production. So here are some of the broken eggs caused by implementation of the auto-decrufter:

• About 30 minutes of dak rm (without no-action) would unconditionally crash.
• A broken dinstall when dak auto-decruft was run without dry-run for the first time.
• A boolean condition inversion causing removals to remove the override for partial removals (and retain it for full removals).
  • Side-effect, this broke Britney a couple of times because dak now produced some unexpected Packages files for unstable.
• Not to mention the single digit bug closure bug.

Of the 3, the boolean inversion was no doubt the worst. By the time we had it fixed, at least 50 (unique) binary packages had lost their override . Fortunately, it was possible to locate these issues using a database query and they have now been fixed. Before I write any more non-trivial patches for dak, I will probably invest some time setting up a basic test framework for dak first.
Filed under: Debian, Release-Team

What happened about the reproducible builds effort for this week: Presentations On May 26th,Holger Levsen presented reproducible builds in Debian at CCC Berlin for the Datengarten 52. The presentation was in German and the slides in English. Audio and video recordings are available. Toolchain fixes

• Dmitry Shachnev uploaded pyqt5/5.4.1+dfsg-3 which makes pyuic output imports in stable order. Original patch by Reiner Herrmann.
• Lunar uploaded mozilla-devscripts/0.41 which uses the UTC timezone when calling zip or unzip.

Niels Thykier fixed the experimental support for the automatic creation of debug packages in debhelper that being tested as part of the reproducible toolchain. Lunar added to the reproducible build version of dpkg the normalization of permissions for files in control.tar. The patch has also been submitted based on the main branch. Daniel Kahn Gillmor proposed a patch to add support for externally-supplying build date to help2man. This sparkled a discussion about agreeing on a common name for an environment variable to hold the date that should be used. It seems opinions are converging on using SOURCE_DATE_UTC which would hold a ISO-8601 formatted date in UTC) (e.g. 2015-06-05T01:08:20Z). Kudos to Daniel, Brendan O'Dea, Ximin Luo for pushing this forward. Lunar proposed a patch to Tar upstream adding a --clamp-mtime option as a generic solution for timestamp variations in tarballs which might also be useful for dpkg. The option changes the behavior of --mtime to only use the time specified if the file mtime is newer than the given time. So far, upstream is not convinced that it would make a worthwhile addition to Tar, though. Daniel Kahn Gillmor reached out to the libburnia project to ask for help on how to make ISO created with xorriso reproducible. We should reward Thomas Schmitt with a model upstream trophy as he went through a thorough analysis of possible sources of variations and ways to improve the situation. Most of what is missing with the current version in Debian is available in the latest upstream version, but libisoburn in Debian needs help. Daniel backported the missing option for version 1.3.2-1.1. akira submitted a new issue to Doxygen upstream regarding the timestamps added to the generated manpages. Packages fixed The following 49 packages became reproducible due to changes in their build dependencies: activemq-protobuf, bnfc, bridge-method-injector, commons-exec, console-data, djinn, github-backup, haskell-authenticate-oauth, haskell-authenticate, haskell-blaze-builder, haskell-blaze-textual, haskell-bloomfilter, haskell-brainfuck, haskell-hspec-discover, haskell-pretty-show, haskell-unlambda, haskell-x509-util, haskelldb-hdbc-odbc, haskelldb-hdbc-postgresql, haskelldb-hdbc-sqlite3, hasktags, hedgewars, hscolour, https-everywhere, java-comment-preprocessor, jffi, jgit, jnr-ffi, jnr-netdb, jsoup, lhs2tex, libcolor-calc-perl, libfile-changenotify-perl, libpdl-io-hdf5-perl, libsvn-notify-mirror-perl, localizer, maven-enforcer, pyotherside, python-xlrd, python-xstatic-angular-bootstrap, rt-extension-calendar, ruby-builder, ruby-em-hiredis, ruby-redcloth, shellcheck, sisu-plexus, tomcat-maven-plugin, v4l2loopback, vim-latexsuite. The following packages became reproducible after getting fixed:

• afterstep/2.2.12-6 uploaded by Robert Luberda, original patch by Juan Picca.
• birdfont/2.8.0-2 by Hideki Yamane.
• bzr/2.6.0+bzr6602-1 by Jelmer Vernooij.
• cassiopee/1.0.3+dfsg-2 uploaded by Olivier Sallou, original patch by akira.
• dhcp-helper/1.1-3 by Simon Kelley.
• elog/3.1.0-2-1 by Roger Kalt.
• flashcache/3.1.3+git20150513-1 uploaded by Liang Guo, original patch by Reiner Herrmann.
• fonts-kiloji/1:2.1.0-21 by Hideki Yamane.
• inspircd/2.0.20-2 by Guillaume Delacour.
• jd/1:2.8.9-150226-3 by Hideki Yamane.
• jruby-joni/2.1.6-2 by Hideki Yamane.
• libaqbanking/5.6.0beta-1 by Micha Lenk.
• libencode-hanextra-perl/0.23-4 fixed and uploaded by Niko Tyni.
• libencode-jis2k-perl/0.02-3 by Niko Tyni.
• libjide-oss-java/3.6.9+dfsg-1 by Markus Koschany.
• librpc-xml-perl/0.78-3 upload by Niko Tyni, original patch by Reiner Herrmann.
• libterm-readkey-perl/2.32-2 by Niko Tyni.
• mkgmap-splitter/0.0.0+svn421-1 by Bas Couwenberg.
• mod-authn-webid/0~20110301-3 by Clint Adams, original patch by Chris Lamb.
• ossim/1.8.16-4 uploaded by Bas Couwenberg, original patch by Juan Picca.
• psfex/3.17.1+dfsg-2 by Ole Streicher.
• socket-wrapper/1.1.3-2 uploaded by Laszlo Boszormenyi, original patch by Jelmer Vernooij.
• stiff/2.4.0-2 by Ole Streicher.
• testng/6.9.4-2 by Eugene Zhukov.
• wcwidth/0.1.4-2 by Sebastian Ramacher.

Some uploads fixed some reproducibility issues but not all of them:

• ant/1.9.5-1 by Emmanuel Bourg.
• gcin/2.8.3+dfsg1-2 by ChangZhuo Chen.
• grcompiler/4.2-5 by Hideki Yamane.
• python2.7/2.7.10-2 uploaded by Matthias Klose, based on a patch by Lunar.
• python3.4/3.4.3-7 uploaded by Matthias Klose, based on a patch by Lunar.
• python3.5/3.5.0~b2-1 uploaded by Matthias Klose, based on a patch by Lunar.
• wml/2.0.12ds1-9 by Axel Beckert.

Patches submitted which did not make their way to the archive yet:

• #787327 on vim by Reiner Herrmann: set a constant user and set modified-by option.
• #787650 on lush by Daniel Kahn Gillmor: remove __DATE__ and __TIME__ macros from source.
• #787669 on cloc by Daniel Kahn Gillmor: use time of latest debian/changelog entry in manpage.
• #787675 on ricochet by Daniel Kahn Gillmor: patch configure to allow an external build date and set it to the time of latest debian/changelog entry.
• #787804 on clipper by akira: set HTML_TIMESTAMP to NO in Doxygen configuration.
• #787829 on colobot by akira: set HTML_TIMESTAMP to NO in Doxygen configuration.
• #787865 on fastjet by akira: call Doxygen with HTML_TIMESTAMP set to NO.
• #787916 on cal3d by akira: remove $datetime from the file api_footer.html.
• #787918 on dime by akira: remove $datetime from the file footer.html.

Daniel Kahn Gilmor also started discussions for emacs24 and the unsorted lists in generated .el files, the recording of a PID number in lush, and the reproducibility of ISO images in grub2. reproducible.debian.net Notifications are now sent when the build environment for a package has changed between two builds. This is a first step before automatically building the package once more. (Holger Levsen) jenkins.debian.net was upgraded to Debian Jessie. (Holger Levsen) A new variation is now being tested: $PATH. The second build will be done with a /i/capture/the/path added. (Holger Levsen) Holger Levsen with the help of Alexander Couzens wrote extra job to test the reproducibility of coreboot. Thanks James McCoy for helping with certificate issues. Mattia Rizollo made some more internal improvements. strip-nondeterminism development Andrew Ayer released strip-nondeterminism/0.008-1. This new version fixes the gzip handler so that it now skip adding a predetermined timestamp when there was none. Holger Levsen sponsored the upload. Documentation update The pages about timestamps in manpages generated by Doxygen, GHC .hi files, and Jar files have been updated to reflect their status in upstream. Markus Koschany documented an easy way to prevent Doxygen to write timestamps in HTML output. Package reviews 83 obsolete reviews have been removed, 71 added and 48 updated this week. Meetings A meeting was held on 2015-06-03. Minutes and full logs are available. It was agreed to hold such a meeting every two weeks for the time being. The time of the next meeting should be announced soon.

Call for Proposals Deadline The deadline for submitting proposals is approaching, with only 12 days left to submit your event by June 15th. Events submitted after that date might not be part of the official DebConf schedule. We are very excited about the upcoming conference, and we would like to encourage you to send your proposals. It s an important part of the conference to hear and discuss new ideas. If you have something that you d like to present but you have not submitted your event yet, please don t wait until the last minute! Check out the proposal submission guide and submit your event. If you have already submitted your event, do take this opportunity to login to summit and review it, expanding the event description to be more descriptive and appealing to the attendees if necessary. Second Batch of Approved talks We are happy to announce the following talks that are already approved:

• What s new in the Linux kernel (Ben Hutchings)
• PPAs - what s next? (Neil McGovern)
• Let s get ready to Go (Margarita Manterola)
• Debian dependency resolution in polynomial time (Niels Thykier)
• A vision of backups in Debian (Lars Wirzenius)

Please hurry up and share your ideas with us. Propose your event before the deadline is reached. Looking forward to see you on Heidelberg, The DebConf content Team

DebConf15 will be held in Heidelberg, Germany from the 15th to the 22nd of August, 2015. The clock is ticking and our annual conference is approaching. There are less than three months to go, and the Call for Proposals period closes in only a few weeks. This year, we are encouraging people to submit half-length 20-minute events, to allow attendees to have a broader view of the many things that go on in the project in the limited amount of time that we have. To make sure that your proposal is part of the official DebConf schedule you should submit it before June 15th. If you have already sent your proposal, please log in to summit and make sure to improve your description and title. This will help us fit the talks into tracks, and devise a cohesive schedule. For more details on how to submit a proposal see: http://debconf15.debconf.org/proposals.xhtml. Approved Talks We have processed the proposals submitted up to now, and we are proud to announce the first batch of approved talks. Some of them:

• This APT has Super Cow Powers (David Kalnischkies)
• AppStream, Limba, XdgApp: Past, present and future (Matthias Klumpp)
• Onwards to Stretch (and other items from the Release Team) (Niels Thykier for the Release Team)
• GnuPG in Debian report (Daniel Kahn Gillmor)
• Stretching out for trustworthy reproducible builds - creating bit by bit identical binaries (Holger Levsen & Lunar)
• Debian sysadmin (and infrastructure) from an outsider/newcomer perspective (Donald Norwood)
• The Debian Long Term Support Team: Past, Present and Future (Rapha l Hertzog & Holger Levsen)

If you have already submitted your event and haven t heard from us yet, don t panic! We will contact you shortly. We would really like to hear about new ideas, teams and projects related to Debian, so do not hesitate to submit yours. See you in Heidelberg,
DebConf Team

Debian Jessie has been released on April 25th, 2015. This has opened the Stretch development cycle. Reactions to the idea of making Debian build reproducibly have been pretty enthusiastic. As the pace is now likely to be even faster, let's see if we can keep everyone up-to-date on the developments. Before the release of Jessie The story goes back a long way but a formal announcement to the project has only been sent in February 2015. Since then, too much work has happened to make a complete report, but to give some highlights:

• New variations are now tested: umask, kernel version, domain name, and timezone. We might only be missing CPU type and current date now.
• Many improvements to the test system on jenkins.debian.net and the pages showing the results.
• Now not only packages from unstable are tested but also those in testing and experimental.
• When rescheduling packages for testing, the build products can be kept and the IRC channel gets a notification when its over.
• binutils version 2.25-6 is now built with the --enable-deterministic-archives flag. Making ar, strip and others create deterministic static libraries.
• Number of identified issues has grown from about 80 to 123 today.

Lunar did a pretty improvised lightning talk during the Mini-DebConf in Lyon. This past week It seems changes were pilling behind the curtains given the amount of activity that happened in just one week. Toolchain fixes

• Niels Thykier uploaded debhelper/9.20150501 which includes fixes to dh_makeshlibs (#774100), dh_icons (#774102), dh_usrlocal (#775020). Patches written by Lunar.
• Helmut Grohne uploaded doxygen/1.8.9.1-3 which will not generate timestamps in HTML by default. Kudos to akira for bringing the issue upstream.
• Kenneth J. Pronovici uploaded epydoc/3.0.1+dfsg-6 adding a --no-include-build-time option. Patch by Jelmer Vernooij.
• David Pr vot uploaded php-apigen/2.8.1+dfsg-2 which now has reproducible output.
• C dric Boutillier uploaded ruby-prawn/2.0.1+dfsg-1 which now produce a deterministic output when using gradients. Patch by Lunar.
• Jelmer Vernooij uploaded samba/2:4.1.17+dfsg-4 which contains a patch by Matthieu Patou making the output of pidl (from libparse-pidl-perl) reproducible.
• Dmitry Shachnev uploaded sphinx/1.3.1-1 in experimental which should produce deterministic output. The original patch from Chris Lamb has inspired the upstream fix.
• gregor herrmann uploaded libextutils-depends-perl/0.404-1 which makes ExtUtils::Depends output deterministic. Original patch by Reiner Herrmann.
• Niko Tyni uploaded perl/5.20.2-4 which makes the output of Pod::Man reproducible. Nice team work visible on #780259.

We also rebased the experimental version of debhelper twice to merge the latest set of changes. Lunar submitted a patch to add a -creation-date to genisoimage. Reiner Herrmann opened #783938 to request making -notimestamp the default behavior for javadoc. Juan Picca submitted a patch to add a --use-date flag to texi2html. Packages fixed The following packages became reproducible due to changes of their build dependencies: apport, batctl, cil, commons-math3, devscripts, disruptor, ehcache, ftphs, gtk2hs-buildtools, haskell-abstract-deque, haskell-abstract-par, haskell-acid-state, haskell-adjunctions, haskell-aeson, haskell-aeson-pretty, haskell-alut, haskell-ansi-terminal, haskell-async, haskell-attoparsec, haskell-augeas, haskell-auto-update, haskell-binary-conduit, haskell-hscurses, jsch, ledgersmb, libapache2-mod-auth-mellon, libarchive-tar-wrapper-perl, libbusiness-onlinepayment-payflowpro-perl, libcapture-tiny-perl, libchi-perl, libcommons-codec-java, libconfig-model-itself-perl, libconfig-model-tester-perl, libcpan-perl-releases-perl, libcrypt-unixcrypt-perl, libdatetime-timezone-perl, libdbd-firebird-perl, libdbix-class-resultset-recursiveupdate-perl, libdbix-profile-perl, libdevel-cover-perl, libdevel-ptkdb-perl, libfile-tail-perl, libfinance-quote-perl, libformat-human-bytes-perl, libgtk2-perl, libhibernate-validator-java, libimage-exiftool-perl, libjson-perl, liblinux-prctl-perl, liblog-any-perl, libmail-imapclient-perl, libmocked-perl, libmodule-build-xsutil-perl, libmodule-extractuse-perl, libmodule-signature-perl, libmoosex-simpleconfig-perl, libmoox-handlesvia-perl, libnet-frame-layer-ipv6-perl, libnet-openssh-perl, libnumber-format-perl, libobject-id-perl, libpackage-pkg-perl, libpdf-fdf-simple-perl, libpod-webserver-perl, libpoe-component-pubsub-perl, libregexp-grammars-perl, libreply-perl, libscalar-defer-perl, libsereal-encoder-perl, libspreadsheet-read-perl, libspring-java, libsql-abstract-more-perl, libsvn-class-perl, libtemplate-plugin-gravatar-perl, libterm-progressbar-perl, libterm-shellui-perl, libtest-dir-perl, libtest-log4perl-perl, libtext-context-eitherside-perl, libtime-warp-perl, libtree-simple-perl, libwww-shorten-simple-perl, libwx-perl-processstream-perl, libxml-filter-xslt-perl, libxml-writer-string-perl, libyaml-tiny-perl, mupen64plus-core, nmap, openssl, pkg-perl-tools, quodlibet, r-cran-rjags, r-cran-rjson, r-cran-sn, r-cran-statmod, ruby-nokogiri, sezpoz, skksearch, slurm-llnl, stellarium. The following packages became reproducible after getting fixed:

• berkeley-abc/1.01+20141105hg5b5af75+dfsg-3 uploaded by Ruben Undheim, original patch by Johann Klammer.
• binutils/2.25-7 by Matthias Klose.
• bitmap-mule/8.5+0.20030825.0433-16 uploaded by Tatsuya Kinoshita, original patch by Chris Lamb.
• blobby/1.0-2 by Felix Geyer.
• codelite/7.0+dfsg-2 by James Cowgill.
• conkeror/1.0~~pre-1+git150409-1 by Axel Beckert.
• convmv/1.15-1 by Christian Perrier.
• cpl/6.6~b-1 by Ole Streicher.
• deheader/1.1-2 by Reiner Herrmann.
• dict-foldoc/20150318-1 uploaded by Iustin Pop, original patch by Chris Lamb.
• ding/1.8-1 by Roland Rosenfeld.
• doc-rfc/20150425-1 by Iustin Pop.
• doxygen/1.8.9.1-3 by Helmut Grohne.
• eureka/1.07-1 by Fabian Greffrath.
• eximdoc4/4.85-2 uploaded by Andreas Metzler, original patch by Chris Lamb.
• flashproxy/1.7-3 by Ximin Luo.
• heimdal/1.6~rc2+dfsg-10 by Jelmer Vernooij.
• init-system-helpers/1.23 by Martin Pitt original patch by Lunar.
• ldb/2:1.1.20-2 by Jelmer Vernooij.
• libalien-wxwidgets-perl/0.67+dfsg-1 uploaded by gregor herrmann, original patch by Chris Lamb.
• libcairo-perl/1.105-1 by intrigeri.
• libclass-methodmaker-perl/2.24-1 uploaded by gregor herrmann, original patch by Chris Lamb.
• libcommon-sense-perl/3.73-3 uploaded by gregor herrmann, original patch by Chris Lamb.
• libelixirfm-perl/1.1.976-4 uploaded by gregor herrmann, original patch by Chris Lamb.
• libgnome2-perl/1.045-3 by intrigeri.
• libjs-jcrop/0.9.12+dfsg-2 uploaded by David Pr vot, original patch by Chris Lamb.
• liblas/1.8.0-2 by Bas Couwenberg.
• libopengl-perl/0.6704+dfsg-1 uploaded by gregor herrmann, original patch by Chris Lamb.
• libparse-recdescent-perl/1.967009+dfsg-2 uploaded by gregor herrmann, original patch by Reiner Herrmann.
• librasterlite/1.1g-5 by Bas Couwenberg.
• libterm-size-perl-perl/0.029-2 uploaded by gregor herrmann, original patch by Chris Lamb.
• libvdpau/1.1-1 by Andreas Beckmann.
• libxml-sax-expatxs-perl/1.33-2 uploaded by gregor herrmann, original patch by Chris Lamb.
• lynx-cur/2.8.9dev4-2 by Axel Beckert.
• mapcache/1.2.1-3 by Bas Couwenberg.
• mdbtools/0.7.1-4 by Jean-Michel Nirgal Vourg re.
• mopidy uploaded by Stein Magnus Jodal, fixed by upstream.
• objenesis/2.1-1 by Markus Koschany.
• opendkim/2.10.1-2 uploaded by Scott Kitterman, original patch by Reiner Herrmann.
• pktools/2.6.3-1 by Bas Couwenberg.
• posh/0.12.4 uploaded by Clint Adams, original patch by Chris Lamb.
• prboom-plus/2:2.5.1.4~svn4425+dfsg1-1 by Fabian Greffrath.
• qlandkartegt/1.8.1+ds-1 by Bas Couwenberg.
• readosm/1.0.0d-1~exp2 by Bas Couwenberg.
• robocode/1.9.2.4-1 by Markus Koschany.
• ruby-prawn/2.0.1+dfsg-1 uploaded by C dric Boutillier, original patch by Lunar.
• sane-backends/1.0.25+git20150425-1 by J rg Frings-F rst.
• scoop/0.7.1-3 by Daniel Stender.
• springlobby/0.218-1 by Markus Koschany.
• subvertpy/0.9.2-1 by Jelmer Vernooij.
• t-prot/3.4-2 by Axel Beckert.
• talloc/2.1.2-3 by Jelmer Vernooij.
• tdb/1.3.4-2 by Jelmer Vernooij, patch now applied by upstream.
• tk-html3/3.0~fossil20110109-5 by Ole Streicher.
• tox/1.9.2-2 uploaded by Barry Warsaw original patch by Reiner Herrmann.
• trove3/3.0.3-2 by Erich Schubert.
• txt2man/1.5.6-2 uploaded by Joao Eriberto Mota Filho, initial patch by Jonathan Wiltshire.
• units/2.11-2 by Stephen Kitt.
• win32-loader/0.7.10 upload by Didier Raboud, original patch by Lunar.
• zec/0.12-3 uploaded by Clint Adams, original patch by Chris Lamb.
• zomg/0.8-1 uploaded by Clint Adams, original patch by Chris Lamb.

Some uploads fixed some reproducibility issues but not all of them:

• ada-reference-manual/1:2012.2-4 by Nicolas Boulenguez.
• adacontrol/1.16r11-3 by Nicolas Boulenguez.
• aspectj/1.8.5-1 by Emmanuel Bourg.
• binutils-m68hc1x/1:2.18-4 uploaded by Riku Voipio, original patch by Chris Lamb.
• debianutils/4.5) uploaded by Clint Adams, original patch by Lunar.
• ioquake3/1.36+u20150412+dfsg1-1 by Simon McVittie.
• libsdl1.2/1.2.15-11 uploaded by Manuel A. Fernandez Montecelo, original patch by Chris Lamb; currently FTBFS.
• libsdl2/2.0.2+dfsg1-7 by Manuel A. Fernandez Montecelo, original patch by Chris Lamb; currently FTBFS.
• nedit/1:5.6a-2 by Paul Gevers.
• openarena-085-data/0.8.5split-4 by Simon McVittie.
• openarena-data/0.8.5split-4 by Simon McVittie.
• openarena/0.8.8-12 by Simon McVittie.
• openchange/1:2.2-6 by Jelmer Vernooij.
• puredata/0.46.6-1 by IOhannes m zm lnig.
• pyexiv2/0.3.2-8 by Michal iha ; currently FTBFS.
• quakespasm/0.90.0-2 by Stephen Kitt.
• sed/4.2.2-5 by Clint Adams, patch by Lunar.
• v4l2loopback/0.8.0-5 uploaded by IOhannes m zm lnig, original patch by Chris Lamb.
• vim/2:7.4.712-1 uploaded by James McCoy, original patch by Reiner Herrmann.
• zookeeper/3.4.6-4 by Emmanuel Bourg.

Patches submitted which did not make their way to the archive yet:

• #783882 on jodconverter by Reiner Herrmann: tell javadoc to stop writing timestamps.
• #783885 on bind9 by Reiner Herrmann: pass -DNO_VERSION_DATE to build system.
• #783688 on serverstats by Chris Lamb: fix permissions.
• #783453 on cdbackup by Chris Lamb: remove build date from version string in binary.
• #783478 on texi2html by Juan Picca: pass --use-date to texi2html.
• #783933 on imagemagick by Reiner Herrmann: remove timestamps from PNG and use deterministic package build date.
• #783515 on memtest86+ by Lunar: fix embedded mtimes and pass -creation-date to genisoimage.
• #783558 on websvn by Chris Lamb: fix permissions.

Improvements to reproducible.debian.net Mattia Rizzolo has been working on compressing logs using gzip to save disk space. The web server would uncompress them on-the-fly for clients which does not accept gzip content. Mattia Rizzolo worked on a new page listing various breakage: missing or bad debbindiff output, missing build logs, unavailable build dependencies. Holger Levsen added a new execution environment to run debbindiff using dependencies from testing. This is required for packages built with GHC as the compiler only understands interfaces built by the same version. debbindiff development Version 17 has been uploaded to unstable. It now supports comparing ISO9660 images, dictzip files and should compare identical files much faster. Documentation update Various small updates and fixes to the pages about PDF produced by LaTeX, DVI produced by LaTeX, static libraries, Javadoc, PE binaries, and Epydoc. Package reviews Known issues have been tagged when known to be deterministic as some might unfortunately not show up on every single build. For example, two new issues have been identified by building with one timezone in April and one in May. RD and help2man add current month and year to the documentation they are producing. 1162 packages have been removed and 774 have been added in the past week. Most of them are the work of proper automated investigation done by Chris West. Summer of code Finally, we learned that both akira and Dhole were accepted for this Google Summer of Code. Let's welcome them! They have until May 25th before coding officialy begins. Now is the good time to help them feel more comfortable by sharing all these little bits of knowledge on how Debian works.

It was quite an experience to partake in the Jessie release and also a rather long Saturday . This post is mostly a time line of how I spent my release day with doing the actual release. I have glossed over some details the post is long enough without these. :) We started out at 8 (UTC) with a final dinstall run, which took nearly 2 hours. It was going to take longer, but we decided to skip the synchronisation to coccia.debian.org (the server hosting the DD-accessible mirror of release.debian.org). The release itself started with the FTP masters renaming the aliases of Squeeze, Wheezy and Jessie to oldoldstable, oldstable and stable respectively. While they worked, the release team reviewed and double checked their work. After an hour (~11), the FTP masters reported that the stable releases were ready for the final review and the SRMs signed the relevant Release files. Then the FTP masters pushed the stable releases to our CD build server, where Steve McIntyre started building the installation images. While Steve started with the CDs, the FTP masters and the release team continued with creating a suite for Stretch. On the FTP/release side, we finished shortly before 12:30. At this point, our last ETA from Steve suggested that the installation media would take another 11 and a half hours to complete. We could have opened for mirror synchronisation then, but we decided to wait for the installation media. At 12:30, there was a long intermission for the release team in the release process. That was an excellent time to improve some of our tools, but that is for another post. :) We slowly started to resume around 22:20, where we tried to figure out when to open for the mirror synchronisation to time it with the installation media. We agreed to start the mirror sync at 23:00 despite the installation media not being completely done then. They followed half an hour later, when Steve reported that the last CD was complete. At this point, all that was left was to update the website and send out the press announcement. Sadly, we were hit by some (minor) issues then. First, I had underestimated the work involved in updating the website. Secondly, we had no one online at the time to trigger an out of band rebuild of the website. Steve and I spent an hour and a half solving website issues (like arm64 and ppc64el not being listed as a part of the release). Unsurprisingly, I decided to expand our the release checklist to be slightly more verbose on this particular topic. My Saturday had passed its 16th hour, when I thought we had fixed all the website issues (of course, I would be wrong) and we would now just be waiting for the an automatic rebuild. I was tempted to just punt it and go to bed, when Paul Wise rejoined us at about 01:25. He quickly got up to speed and offered to take care of the rest. An offer I thankfully accepted and I checked out 15 minutes later at 01:40 UTC. That more or less covers the Jessie release day from my PoV. After a bit of reflection inside the release team, we have found several points where we can improve the process. This part certainly deserves its own post as well, which will also give us some time to flesh out some of the ideas a bit more. :)
Filed under: Debian, Release-Team

Indeed, we settled on a release date for Jessie and pretty quick too. I sent out a poll on the 28th of March and yesterday, it was clear that the 25th of April was our release date. :) With that said, we still have some items left that needs to be done.

• Finishing the release notes. This is mostly pending myself and a few others.
• Translation of the release-notes. I sent out a heads up earlier today about what sections I believe to be done.
• The d-i team got another release planned as well.
• All the RC bugs you can manage to fix before the 18th of April. :)

Filed under: Debian, Release-Team

Earlier today, I posted a mail to debian-devel about how approximately 25 RC bugs affecting Jessie have been unblocked. As mentioned, I planned to age some of them. The expected result is that about 18 of them will migrate tonight and the remaining 7 of them will migrate tomorrow night. After that, there are no more RC bugs waiting for the RT to unblock them! The only remaining item on the list is cgmanager, for which we are requesting a t-p-u (maintainer already contacted about it). If you want a release sooner, please have a look at the list of remaining RC bugs or/and start testing upgrades. In other news, the glibc regression got fixed. The new version of glibc has already been approved by us. It is now waiting for the debian-installer team to testing it and approve it.
Filed under: Debian, Release-Team

I had the mixed pleasure of doing a partial rewrite of lintian s reporting framework. It started as a problem with generating the graphs, which turned out to be not enough memory . On the plus side, I am actually quite pleased with the end result. I managed to scope-creep myself quite a bit and I ended up getting rid of a lot of old issues. The major changes in summary:

• A lot of logic was moved out of harness, meaning it is now closer to becoming a simple dumb task scheduler. With the logic being moved out in separate processes, harness now hogs vastly less memory that I cannot convince perl to release to the OS. On lilburn.debian.org vastly less is on the order of reducing 700ish MB to 32 MB .
• All important metadata was moved into the harness state-cache , which is a simple YAML file. This means that Lintian laboratory is no longer a data store. This change causes a lot of very positive side effects.
• With all metadata now stored in a single file, we can now do atomic updates of the data store. That said, this change itself does not enable us to run multiple lintian s in parallel.
• As the lintian laboratory is no longer a data store, we can now do our processing in throw away laboratories like the regular lintian user does. As the permanent laboratory is the primary source of failure, this removes an entire class of possible problems.

There are also some nice minor features :

• Packages can now be up to date in the generated reports. Previously, they would always be listed as out of date even if they were up to date. This is the only end user/website-visitor visible change in all of this (besides the graphs are now working again \o/).
• The size of the harness work list is no longer based on the number of changes to the archive.
• The size of the harness work list can now be changed with a command line option and is no longer hard coded to 1024. However, the time limit remains hard coded for now.
• The full run (and clean run ) now simply marks everything out-of-date and processes its (new) backlog over the next (many) harness runs. Accordingly, a full-run no longer causes lintian to run 5-7 days on lilburn.d.o before getting an update to the website. Instead we now get incremental updates.
• The harness.log now features status updates from lintian as they happen with processed X successfully or error processing Y plus a little wall time benchmark. With this little feature I filed no less than 3 bugs against lintian 2 of which are fixed in git. The last remains unfixed but can only be triggered in Debian stable.
• It is now possible with throw-away labs to terminate the lintian part of a reporting run early with minimal lost processing. Since the lintian-harness is regular fed status updates from lintian, we can now mark successfully completed entries as done even if lintian does not complete its work list. Caveat: There may be minor inaccuracies in the generated report for the particular package lintian was processing when it was interrupted. This will fix itself when the package is reprocessed again.
• It is now vastly easier to collect new meta data to be used in the reports. Previously, they had to be included in the laboratory and extracted from there. Now, we just have to fit it into a YAML file. In fact, I have been considering to add the wall time and make a top X slowest page.
• It is now possible to generate the html pages with only a state-cache and the lintian.log file. Previously, it also required a populated lintian laboratory.

As you can probably tell, I am quite pleased with the end result. The reporting framework lacks behind in development, since it just sits there and takes care of itself . Also with the complete lack of testing, it also suffers from the if it is not broken, then do not fix it paradigm (because we will not notice if we broke until it is too late). Of course, I managed to break the setup a couple of times in the process. However, a bonus feature of the reporting setup is that if you break it, it simply leaves an outdated report on the website. Anyway, enjoy. :)
Filed under: Debian, Lintian

Here is a slightly overdue status on Jessie.

• We are not ready to set a date on a Jessie release yet. Even if we were, it would be unlikely that said date would be in January. Accordingly, it is safe to assume that the rapid automatic removals clause of the freeze policy will be applied before our release. If your package (or a package you use) depend on any package in this list, then they are at risk.
• It is unclear to me whether our CD1s are able to contain all the necessary packages. This was a major issue for Wheezy. So far I have only done some minor prodding here. However, I really want this item done soon as it can easily take a month or more to resolve this.
• While the release notes have been improved quite a bit, I am certain that there are more cases we can cover. As an example, I am looking for input on #773386.
• Despite the declining number of RC bugs, we still have some particular unhealthy ones left. E.g. #759530. There is also a view of some of the known unfixed Jessie blockers.
• We have had a number of issues with APT and dpkg (e.g. trigger cycles, APT breaking under puppet) that caused upgrade failures or were a severe regression. Most of these have been resolved now. There are some trigger issues left and I have pushed for a fixed dpkg at the cost of possibly removing some packages. See #772866 (among others).

Stricter freeze policy per January 5th The next timed change of the freeze policy will apply per January 5th. After that date, we will only accept RC bugs fixes. Which means it is final chance for translation updates. More on RC bugs In absolute numbers, the RC bugs have declined quite well. We are below 150 now. We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month. However, we still need the final push here. Debian installer release pending Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow. Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support. Pending Debian 7.8 release While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend. Want to help?

• File bugs against release-notes (source at [RN source]) and installation-guide for missing or outdated documentation. Patches and drafts especially welcome.
• Fix RC bugs especially the known Jessie blockers.
• Test upgrade paths and installation media. Although in both cases, you may want to wait a bit (for the new dpkg to migrate and for the new debian-installer release respectively).
• Consider offering your help to teams such as the CD team or debian-installer team.

Thank you, [RN source]: https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ svn co https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ Git Repo: http://anonscm.debian.org/cgit/users/jcristau/release-notes.git/
Filed under: Debian, Release-Team

Here is a slightly overdue status on Jessie.

• We are not ready to set a date on a Jessie release yet. Even if we were, it would be unlikely that said date would be in January. Accordingly, it is safe to assume that the rapid automatic removals clause of the freeze policy will be applied before our release. If your package (or a package you use) depend on any package in this list, then they are at risk.
• It is unclear to me whether our CD1s are able to contain all the necessary packages. This was a major issue for Wheezy. So far I have only done some minor prodding here. However, I really want this item done soon as it can easily take a month or more to resolve this.
• While the release notes have been improved quite a bit, I am certain that there are more cases we can cover. As an example, I am looking for input on #773386.
• Despite the declining number of RC bugs, we still have some particular unhealthy ones left. E.g. #759530. There is also a view of some of the known unfixed Jessie blockers.
• We have had a number of issues with APT and dpkg (e.g. trigger cycles, APT breaking under puppet) that caused upgrade failures or were a severe regression. Most of these have been resolved now. There are some trigger issues left and I have pushed for a fixed dpkg at the cost of possibly removing some packages. See #772866 (among others).

Stricter freeze policy per January 5th The next timed change of the freeze policy will apply per January 5th. After that date, we will only accept RC bugs fixes. Which means it is final chance for translation updates. More on RC bugs In absolute numbers, the RC bugs have declined quite well. We are below 150 now. We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month. However, we still need the final push here. Debian installer release pending Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow. Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support. Pending Debian 7.8 release While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend. Want to help?

• File bugs against release-notes (source at [RN source]) and installation-guide for missing or outdated documentation. Patches and drafts especially welcome.
• Fix RC bugs especially the known Jessie blockers.
• Test upgrade paths and installation media. Although in both cases, you may want to wait a bit (for the new dpkg to migrate and for the new debian-installer release respectively).
• Consider offering your help to teams such as the CD team or debian-installer team.

Thank you, [RN source]: https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ svn co https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ Git Repo: http://anonscm.debian.org/cgit/users/jcristau/release-notes.git/
Filed under: Debian, Release-Team

In the last 24 hours, the number of RC bugs currently affecting Jessie was reduced to just under half of the same number for Wheezy. There are still a lot of bugs to be fixed, but please keep up the good work. :)

Page 1 of 5 To be honest, I do not know how many mails it shows per page (though I assume it is a fixed number). So for comparison, I found the month on debian-devel@l.d.o with the highest volume in the past two years: May 2013 with Page 1 of 4 . I hope you will please forgive us, if we are a bit terse in our replies or slow to reply. We simply got a lot to deal with. :)