Search Results: "Mattia Rizzolo"

23 November 2015

Lunar: Reproducible builds: week 30 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Mattia Rizzolo uploaded a version of perl to the reproducible repository including the patch written by Niko Tyni to add support for SOURCE_DATE_EPOCH in Pod::Man. Dhole sent an updated version of his patch adding support for SOURCE_DATE_EPOCH in GCC to the upstream mailing list. Several comments have been made in response which have been quickly addressed by Dhole. Dhole also forwarded his patch adding support for SOURCE_DATE_EPOCH in libxslt upstream. Packages fixed The following packages have become reproducible due to changes in their build dependencies: antlr3/3.5.2-3, clusterssh, cme, libdatetime-set-perl, libgraphviz-perl, liblingua-translit-perl, libparse-cpan-packages-perl, libsgmls-perl, license-reconcile, maven-bundle-plugin/2.4.0-2, siggen, stunnel4, systemd, x11proto-kb. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: reproducible.debian.net Vagrant Cascadian has set up a new armhf node using a Raspberry Pi 2. It should soon be added to the Jenkins infrastructure. diffoscope development diffoscope version 42 was release on November 20th. It adds a missing dependency on python3-pkg-resources and to prevent similar regression another autopkgtest to ensure that the command line is functional when Recommends are not installed. Two more encoding related problems have been fixed (#804061, #805418). A missing Build-Depends has also been added on binutils-multiarch to make the test suite pass on architectures other than amd64. Package reviews 180 reviews have been removed, 268 added and 59 updated this week. 70 new fail to build from source bugs have been reported by Chris West, Chris Lamb and Niko Tyni. New issue this week: randomness_in_ocaml_preprocessed_files. Misc. Jim MacArthur started to work on a system to rebuild and compare packages built on reproducible.debian.net using .buildinfo and snapshot.debian.org. On December 1-3rd 2015, a meeting of about 40 participants from 18 different free software projects will be held in Athens, Greece with the intent of improving the collaboration between projects, helping new efforts to be started, and brainstorming on end-user aspects of reproducible builds.

15 November 2015

Lunar: Reproducible builds: week 29 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Emmanuel Bourg uploaded eigenbase-resgen/1.3.0.13768-2 which uses of the scm-safe comment style by default to make them deterministic. Mattia Rizzolo started a new thread on debian-devel to ask a wider audience for issues about the -Wdate-time compile time flag. When enabled, GCC and clang print warnings when __DATE__, __TIME__, or __TIMESTAMP__ are used. Having the flag set by default would prompt maintainers to remove these source of unreproducibility from the sources. Packages fixed The following packages have become reproducible due to changes in their build dependencies: bmake, cyrus-imapd-2.4, drobo-utils, eigenbase-farrago, fhist, fstrcmp, git-dpm, intercal, libexplain, libtemplates-parser, mcl, openimageio, pcal, powstatd, ruby-aggregate, ruby-archive-tar-minitar, ruby-bert, ruby-dbd-odbc, ruby-dbd-pg, ruby-extendmatrix, ruby-rack-mobile-detect, ruby-remcached, ruby-stomp, ruby-test-declarative, ruby-wirble, vtprint. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net The fifth and sixth armhf build nodes have been set up, resulting in five more builder jobs for armhf. More than 10,000 packages have now been identified as reproducible with the reproducible toolchain on armhf. (Vagrant Cascadian, h01ger) Helmut Grohne and Mattia Rizzolo now have root access on all 12 build nodes used by reproducible.debian.net and jenkins.debian.net. (h01ger) reproducible-builds.org is now linked from all package pages and the reproducible.debian.net dashboard. (h01ger) profitbricks-build5-amd64 and profitbricks-build6-amd64, responsible for running amd64 tests now run 398.26 days in the future. This means that one of the two builds that are being compared will be run on a different minute, hour, day, month, and year. This is not yet the case for armhf. FreeBSD tests are also done with 398.26 days difference. (h01ger) The design of the Arch Linux test page has been greatly improved. (Levente Polyak) diffoscope development Three releases of diffoscope happened this week numbered 39 to 41. It includes support for EPUB files (Reiner Herrmann) and Free Pascal unit files, usually having .ppu as extension (Paul Gevers). The rest of the changes were mostly targetting at making it easier to run diffoscope on other systems. The tlsh, rpm, and debian modules are now all optional. The test suite will properly skip tests that need optional tools or modules when they are not available. As a result, diffosope is now available on PyPI and thanks to the work of Levente Polyak in Arch Linux. Getting these versions in Debian was a bit cumbersome. Version 39 was uploaded with an expired key (according to the keyring on ftp.debian.org which will hopefully be updated soon) which is currently handled by keeping the files in the queue without REJECTing them. This prevented any other Debian Developpers to upload the same version. Version 40 was uploaded as a source-only upload but failed to build from source which had the undesirable side effect of removing the previous version from unstable. The package faild to build from source because it was built passing -I to debbuild. This excluded the ELF object files and static archives used by the test suite from the archive, preventing the test suite to work correctly. Hopefully, in a nearby future it will be possible to implement a sanity check to prevent such mistakes in the future. It has also been identified that ppudump outputs time in the system timezone without considering the TZ environment variable. Zachary Vance and Paul Gevers raised the issue on the appropriate channels. strip-nondeterminism development Chris Lamb released strip-nondeterminism version 0.014-1 which disables stripping Mono binaries as it is too aggressive and the source of the problem is being worked on by Mono upstream. Package reviews 133 reviews have been removed, 115 added and 103 updated this week. Chris West and Chris Lamb reported 57 new FTBFS bugs. Misc. The video of h01ger and Chris Lamb's talk at MiniDebConf Cambridge is now available. h01ger gave a talk at CCC Hamburg on November 13th, which was well received and sparked some interest among Gentoo folks. Slides and video should be available shortly. Frederick Kautz has started to revive Dhiru Kholia's work on testing Fedora packages. Your editor wish to once again thank #debian-reproducible regulars for reviewing these reports weeks after weeks.

2 November 2015

Lunar: Reproducible builds: week 27 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Packages fixed The following packages became reproducible due to changes in their build dependencies: maven-plugin-tools, norwegian, ocaml-melt, python-biom-format, rivet. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: The following package is currently failing to build from source but should now be reproducible: Patches submitted which have not made their way to the archive yet: reproducible.debian.net A quick update on current statistics: testing is at 85% of packages tested reproducible with our modified packages, unstable on armhf caught up with amd64 with 80%. The schroot name used for running diffoscope when testing OpenWrt, NetBSD, Coreboot, and Arch Linux has been fixed. (h01ger, Mattia Rizzolo) Documentation update Paul Gevers documented timestamps in unit files created by the Free Pascal Compiler. reproducible-builds.org is now live. It contains a comprehensive documentation on all aspects that have been identified so far of what we call reproducible builds . It makes room for pointers to projects working on reproducible builds, news, dedicated tools, and community events. Package reviews 206 reviews have been removed, 171 added and 196 updated this week. Chris Lamb reported 28 failing to build from source issues. New issues identified this week: timestamps_in_pdf_content, different_encoding_in_html_by_docbook_xsl, timestamps_in_ppu_generated_by_fpc, method_may_never_be_called_in_documentation_generated_by_javadoc. Misc. Andrei Borzenkov has proposed a fix for uninitialized memory in GRUB's mkimage. Uninitialized memory is one source of hard to track down reproducibility errors. Holger Levsen presented the efforts on reproduible builds at Festival de Software Libre in Puerto Vallarta, Mexico.

27 October 2015

Lunar: Reproducible builds: week 26 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Mattia Rizzolo created a bug report to continue the discussion on storing cryptographic checksums of the installed .deb in dpkg database. This follows the discussion that happened in June and is a pre-requisite to add checksums to .buildinfo files. Niko Tyni identified why the Vala compiler would generate code in varying order. A better patch than his initial attempt still needs to be written. Packages fixed The following 15 packages became reproducible due to changes in their build dependencies: alt-ergo, approx, bin-prot, caml2html, coinst, dokujclient, libapreq2, mwparserfromhell, ocsigenserver, python-cryptography, python-watchdog, slurm-llnl, tyxml, unison2.40.102, yojson. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: reproducible.debian.net pbuilder has been updated to version 0.219~bpo8+1 on all eight build nodes. (Mattia Rizzolo, h01ger) Packages that FTBFS but for which no open bugs have been recorded are now tested again after 3 days. Likewise for depwait packages. (h01ger) Out of disk situations will not cause IRC notifications anymore. (h01ger) Documentation update Lunar continued to work on writing documentation for the future reproducible-builds.org website. Package reviews 44 reviews have been removed, 81 added and 48 updated this week. Chris West and Chris Lamb identified 70 fail to build from source issues. Misc. h01ger presented the project in Mexico City at the 3er Congreso de Seguridad de la Informaci n where it became clear that we lack academic papers related to reproducible builds. Bryan has been doing hard work to improve reproducibility for OpenWrt. He wrote a report linking to the patches and test results he published.

18 October 2015

Lunar: Reproducible builds: week 25 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Niko Tyni wrote a new patch adding support for SOURCE_DATE_EPOCH in Pod::Man. This would complement or replace the previously implemented POD_MAN_DATE environment variable in a more generic way. Niko Tyni proposed a fix to prevent mtime variation in directories due to debhelper usage of cp --parents -p. Packages fixed The following 119 packages became reproducible due to changes in their build dependencies: aac-tactics, aafigure, apgdiff, bin-prot, boxbackup, calendar, camlmix, cconv, cdist, cl-asdf, cli-common, cluster-glue, cppo, cvs, esdl, ess, faucc, fauhdlc, fbcat, flex-old, freetennis, ftgl, gap, ghc, git-cola, globus-authz-callout-error, globus-authz, globus-callout, globus-common, globus-ftp-client, globus-ftp-control, globus-gass-cache, globus-gass-copy, globus-gass-transfer, globus-gram-client, globus-gram-job-manager-callout-error, globus-gram-protocol, globus-gridmap-callout-error, globus-gsi-callback, globus-gsi-cert-utils, globus-gsi-credential, globus-gsi-openssl-error, globus-gsi-proxy-core, globus-gsi-proxy-ssl, globus-gsi-sysconfig, globus-gss-assist, globus-gssapi-error, globus-gssapi-gsi, globus-net-manager, globus-openssl-module, globus-rsl, globus-scheduler-event-generator, globus-xio-gridftp-driver, globus-xio-gsi-driver, globus-xio, gnome-control-center, grml2usb, grub, guilt, hgview, htmlcxx, hwloc, imms, kde-l10n, keystone, kimwitu++, kimwitu-doc, kmod, krb5, laby, ledger, libcrypto++, libopendbx, libsyncml, libwps, lprng-doc, madwimax, maria, mediawiki-math, menhir, misery, monotone-viz, morse, mpfr4, obus, ocaml-csv, ocaml-reins, ocamldsort, ocp-indent, openscenegraph, opensp, optcomp, opus, otags, pa-bench, pa-ounit, pa-test, parmap, pcaputils, perl-cross-debian, prooftree, pyfits, pywavelets, pywbem, rpy, signify, siscone, swtchart, tipa, typerep, tyxml, unison2.32.52, unison2.40.102, unison, uuidm, variantslib, zipios++, zlibc, zope-maildrophost. The following packages became reproducible after getting fixed: Packages which could not be tested: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Lunar reported that test strings depend on default character encoding of the build system in ongl. reproducible.debian.net The 189 packages composing the Arch Linux core repository are now being tested. No packages are currently reproducible, but most of the time the difference is limited to metadata. This has already gained some interest in the Arch Linux community. An explicit log message is now visible when a build has been killed due to the 12 hours timeout. (h01ger) Remote build setup has been made more robust and self maintenance has been further improved. (h01ger) The minimum age for rescheduling of already tested amd64 packages has been lowered from 14 to 7 days, thanks to the increase of hardware resources sponsored by ProfitBricks last week. (h01ger) diffoscope development diffoscope version 37 has been released on October 15th. It adds support for two new file formats (CBFS images and Debian .dsc files). After proposing the required changes to TLSH, fuzzy hashes are now computed incrementally. This will avoid reading entire files in memory which caused problems for large packages. New tests have been added for the command-line interface. More character encoding issues have been fixed. Malformed md5sums will now be compared as binary files instead of making diffoscope crash amongst several other minor fixes. Version 38 was released two days later to fix the versioned dependency on python3-tlsh. strip-nondeterminism development strip-nondeterminism version 0.013-1 has been uploaded to the archive. It fixes an issue with nonconformant PNG files with trailing garbage reported by Roland Rosenfeld. disorderfs development disorderfs version 0.4.1-1 is a stop-gap release that will disable lock propagation, unless --share-locks=yes is specified, as it still is affected by unidentified issues. Documentation update Lunar has been busy creating a proper website for reproducible-builds.org that would be a common location for news, documentation, and tools for all free software projects working on reproducible builds. It's not yet ready to be published, but it's surely getting there. Homepage of the future reproducible-builds.org website  Who's involved?  page of the future reproducible-builds.org website Package reviews 103 reviews have been removed, 394 added and 29 updated this week. 72 FTBFS issues were reported by Chris West and Niko Tyni. New issues: random_order_in_static_libraries, random_order_in_md5sums.

4 October 2015

Lunar: Reproducible builds: week 23 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Andreas Metzler uploaded autogen/1:5.18.6-1 in experimental with several patches for reproducibility issues written by Valentin Lorentz. Groovy upstream has merged a change proposed by Emmanuel Bourg to remove timestamps generated by groovydoc. Ben Hutchings submitted a patch to add support for SOURCE_DATE_EPOCH in linux-kbuild as an alternate way to specify the build timestamp. Reiner Herrman has sent a patch adding support for SOURCE_DATE_EPOCH in docbook-utils. Packages fixed The following packages became reproducible due to changes in their build dependencies: commons-csv. fest-reflect, sunxi-tools, xfce4-terminal, The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Tomasz Rybak uploaded pycuda/2015.1.3-1 which should fix reproducibility issues. The package has not been tested as it is in contrib. akira found an embedded code copy of texi2html in fftw. reproducible.debian.net Email notifications are now only sent once a day per package, instead of on each status change. (h01ger) disorderfs has been temporarily disabled to see if it had any impact on the disk space issues. (h01ger) When running out of disk space, build nodes will now automatically detect the problem. This means test results will not be recorded as FTBFS and the problem will be reported to Jenkins maintainers. (h01ger) The navigation menu of package pages has been improved. (h01ger) The two amd64 builders now use two different kernel versions: 3.16 from stable and 4.1 from backports on the other. (h01ger) We now graph the number of packages which needs to be fixed. (h01ger) Munin now creates graphs on how many builds were performed by build nodes (example). (h01ger) A migration plan has been agreed with DSA on how to turn Jenkins into an official Debian service. A backport of jenkins-job-builder for Jessie is currently missing. (h01ger) Package reviews 119 reviews have been removed, 103 added and 45 updated this week. 16 fail to build from source issues were reported by Chris Lamb and Mattia Rizzolo. New issue this week: timestamps_in_manpages_generated_by_docbook_utils. Misc. Allan McRae has submitted a patch to make ArchLinux pacman record a .BUILDINFO file.

21 September 2015

Lunar: Reproducible builds: week 21 in Stretch cycle

If you see someone on the Debian ReproducibleBuilds project, buy him/her a beer. This work is awesome. What happened in the reproducible builds effort this week: Media coverage Nathan Willis covered our DebConf15 status update in Linux Weekly News. Access to non-LWN subscribers will be given on Thursday 24th. Linux Journal published a more general piece last Tuesday. Unexpected praise for reproducible builds appeared this week in the form of several iOS applications identified as including spyware. The malware was undetected by Apple screening. This actually happened because application developers had simply downloaded a trojaned version of XCode through an unofficial source. While reproducible builds can't really help users of non-free software, this is exactly the kind of attacks that we are trying to prevent in our systems. Toolchain fixes Niko Tyni wrote and uploaded a better patch for the source order problem in libmodule-build-perl. Tristan Seligmann identified how the code generated by python-cffi could be emitted in random order in some cases. Upstream has already fixed the problem. Packages fixed The following 24 packages became reproducible due to changes in their build dependencies: apache-curator, checkbox-ng, gant, gnome-clocks, hawtjni, jackrabbit, jersey1, libjsr305-java, mathjax-docs, mlpy, moap, octave-geometry, paste, pdf.js, pyinotify, pytango, python-asyncssh, python-mock, python-openid, python-repoze.who, shadow, swift, tcpwatch-httpproxy, transfig. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net Tests for Coreboot, OpenWrt, NetBSD, and FreeBSD now runs weekly (instead of monthly). diffoscope development Python 3 offers new features (namely yield from and concurrent.futures) that could help implement parallel processing. The clear separation of bytes and unicode strings is also likely to reduce encoding related issues. Mattia Rizolo thus kicked the effort of porting diffoscope to Python 3. tlsh was the only dependency missing a Python 3 module. This got quickly fixed by a new upload. The rest of the code has been moved to the point where only incompatibilities between Python 2.7 and Pyhon 3.4 had to be changed. The commit stream still require some cleanups but all tests are now passing under Python 3. Documentation update The documentation on how to assemble the weekly reports has been updated. (Lunar) The example on how to use SOURCE_DATE_EPOCH with CMake has been improved. (Ben Beockel, Daniel Kahn Gillmor) The solution for timestamps in man pages generated by Sphinx now uses SOURCE_DATE_EPOCH. (Mattia Rizzolo) Package reviews 45 reviews have been removed, 141 added and 62 updated this week. 67 new FTBFS reports have been filled by Chris Lamb, Niko Tyni, and Lisandro Dami n Nicanor P rez Meyer. New issues added this week: randomness_in_r_rdb_rds_databases, python-ply_compiled_parse_tables. Misc. The prebuilder script is now properly testing umask variations again. Santiago Villa started a discussion on debian-devel on how binNMUs would work for reproducible builds.

6 September 2015

Lunar: Reproducible builds: week 19 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Dmitry Shachnev uploaded sphinx/1.3.1-6 with improved patches from Val Lorentz. Chris Lamb submitted a patch for ibus-table which makes the output of ibus-table-createdb deterministic. Niko Tyni wrote a patch to make libmodule-build-perl linking order deterministic. Santiago Vila has been leading discussions on the best way to fix timestamps coming from Gettext POT files. Packages fixed The following 35 packages became reproducible due to changes in their build dependencies: apache-log4j2, dctrl-tools, dms, gitit, gnubik, isrcsubmit, mailutils, normaliz, oaklisp, octave-fpl, octave-specfun, octave-vrml, opencolorio, openvdb, pescetti, php-guzzlehttp, proofgeneral, pyblosxom, pyopencl, pyqi, python-expyriment, python-flask-httpauth, python-mzml, python-simpy, python-tidylib, reactive-streams, scmxx, shared-mime-info, sikuli, siproxd, srtp, tachyon, tcltk-defaults, urjtag, velvet. The following packages became reproducible after getting fixed: The package is not in yet in unstable, but linux/4.2-1~exp1 is now reproducible! Kudos to Ben Hutchings, and most fixes are already merged upstream. Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net Some bugs that prevented packages to build successfully in the remote builders have been fixed. (h01ger) Two more amd64 build jobs have been removed from the Jenkins host in favor of six more on the new remote nodes. (h01ger) The munin graphs currently looks fine, so more amd64 jobs will probably be added in the next week. diffoscope development Version 32 of diffoscope has been released on September 3rd with the following new features: It also fixes many bugs. Head over to the changelog for the full list. Version 33 was released the day after to fix a bug introduced in the packaging. Documentation update Chris Lamb blessed the SOURCE_DATE_EPOCH specification with the version number 1.0 . Lunar documented how the .file assembler directive can help with random filenames in debug symbols. Package reviews 235 reviews have been removed, 84 added and 277 updated this week. 29 new FTBFS bugs were filled by Chris Lamb, Chris West (Faux), Daniel Stender, and Niko Tyni. New issues identified this week: random_order_in_ibus_table_createdb_output, random_order_in_antlr_output, nondetermistic_link_order_in_module_build, and timestamps_in_tex_documents. Misc. Thanks to Dhole and Thomas Vincent, the talk held at DebConf15 now has subtitles! Void Linux started to merge changes to make packages produced by xbps reproducible.

25 August 2015

Lunar: Reproducible builds: week 17 in Stretch cycle

A good amount of the Debian reproducible builds team had the chance to enjoy face-to-face interactions during DebConf15.
Names in red and blue were all present at DebConf15
Picture of the  reproducible builds  talk during DebConf15
Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian. A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive. Picture of the  reproducible builds  roundtable during DebConf15 Toolchain fixes Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2html SOURCE_DATE_EPOCH aware. St phane Glondu reported that dpkg-source would not respect tarball permissions when unpacking under a umask of 002. After hours of iterative testing during the DebConf workshop, Sandro Knau created a test case showing how pdflatex output can be non-deterministic with some PNG files. Packages fixed The following 65 packages became reproducible due to changes in their build dependencies: alacarte, arbtt, bullet, ccfits, commons-daemon, crack-attack, d-conf, ejabberd-contrib, erlang-bear, erlang-cherly, erlang-cowlib, erlang-folsom, erlang-goldrush, erlang-ibrowse, erlang-jiffy, erlang-lager, erlang-lhttpc, erlang-meck, erlang-p1-cache-tab, erlang-p1-iconv, erlang-p1-logger, erlang-p1-mysql, erlang-p1-pam, erlang-p1-pgsql, erlang-p1-sip, erlang-p1-stringprep, erlang-p1-stun, erlang-p1-tls, erlang-p1-utils, erlang-p1-xml, erlang-p1-yaml, erlang-p1-zlib, erlang-ranch, erlang-redis-client, erlang-uuid, freecontact, givaro, glade, gnome-shell, gupnp, gvfs, htseq, jags, jana, knot, libconfig, libkolab, libmatio, libvsqlitepp, mpmath, octave-zenity, openigtlink, paman, pisa, pynifti, qof, ruby-blankslate, ruby-xml-simple, timingframework, trace-cmd, tsung, wings3d, xdg-user-dirs, xz-utils, zpspell. The following packages became reproducible after getting fixed: Uploads that might have fixed reproducibility issues: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: St phane Glondu reported two issues regarding embedded build date in omake and cduce. Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar. Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found. reproducible.debian.net Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays. armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger) The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger) mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger) Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output. Connections to UDD have been made more robust. (Mattia Rizzolo) diffoscope development diffoscope version 31 was released on August 21st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep. New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo). jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb. strip-nondeterminism development Andrew Ayer released strip-nondeterminism version 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added. Testing directory ordering issues: disorderfs During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random. Documentation update Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C. Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it. Package reviews 44 reviews have been removed, 192 added and 77 updated this week. New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex. 117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni. Misc. Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source! Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.

16 August 2015

Lunar: Reproducible builds: week 16 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Valentin Lorentz sent a patch for ispell to initialize memory structures before dumping their content. In our experimental repository, qt4-x11 has been rebased on the latest version (Dhole), as was doxygen (akira). Packages fixed The following packages became reproducible due to changes in their build dependencies: backup-manager, cheese, coinor-csdp, coinor-dylp, ebook-speaker, freefem, indent, libjbcrypt-java, qtquick1-opensource-src, ruby-coffee-script, ruby-distribution, schroot, twittering-mode. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: akira found another embedded code copy of texi2html in maxima. reproducible.debian.net Work on testing several architectures has continued. (Mattia/h01ger) Package reviews 29 reviews have been removed, 187 added and 34 updated this week. 172 new FTBFS reports were filled, 137 solely by Chris West (Faux). josch spent time investigating the issue with fonts in PDF files. Chris Lamb documented the issue affecting documentation generated by ocamldoc. Misc. Lunar presented a general Reproducible builds HOWTO talk at the Chaos Communication Camp 2015 in Germany on August 13th. Recordings are already available, as well as slides and script. h01ger and Lunar also used CCCamp15 as an opportunity to have discussions with members of several different projects about reproducible builds. Good news should be coming soon.

10 August 2015

Lunar: Reproducible builds: week 15 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Guillem Jover uploaded dpkg/1.18.2 which makes dependency comparisons deep by comparing not only the first dependency alternative, to get them sorted in a reproducible way. Original patch by Chris Lamb. Dhole updated the patch adding support for SOURCE_DATE_EPOCH in gettext. A modified package is in the experimental reproducible repository. Valentin Lorentz submitted a patch adding support for SOURCE_DATE_EPOCH to ocamldoc. Valentin Lorentz also opened a bug about the inability to set an arbitrary RNG seed for ocamlopt which would be a way to fix an issue affecting many OCaml packages. Dhole submitted a patch adding support for SOURCE_DATE_EPOCH in qhelpgenerator. A modified package has been sent to the experimental repository as well. Several packages have been updated for the experimental toolchain: doxygen (akira), and dpkg (h01ger). Also, h01ger has built and uploaded all experimental packages having arch:any packages for armhf: dpkg, gettext, doxygen, fontforge, libxslt and texlive-bin. We are now providing our toolchain for armhf and amd64. Packages fixed As you might have noticed, Debian sid is currently largely uninstallable, due to the GCC 5 transition, which also can be see in our reproducibility test setup. Please help! The following packages became reproducible due to changes in their build dependencies: glosstex, indent, ktikz, liblouis, libmicrohttpd, linkchecker, multiboot, qterm, rrep, trueprint, twittering-mode. The following packages became reproducible after getting fixed: Patches submitted which have not made their way to the archive yet: Lunar reported an issue on an unstable ABI from a generated header in icedove reminding of an issue affecting libical-dev. The bug has since been fixed by Carsten Schoenert. akira identified an unreferenced embeded code copy (causing unreproducibility!) in gperf. reproducible.debian.net The scheduler has temporarily been changed to not schedule any already tested packages for sid and experimental, due to the the GCC 5 transitions, which are well visible in our graphs now. On the plus side this has caused our stretch testing to catch up (and improve stats). (h01ger) depwait packages (packages where the Build-Depends cannot be satisfied) are now listed in the last 24h and last 48h pages (Mattia Rizzolo) Two new amd64 build nodes (with 8 cores and 32 GB RAM each) have been added, kindly sponsored by Profitbricks. (h01ger) The 4 armhf (setup last week by Vagrant Cascadian) and 2 amd64 build nodes have been made available to Jenkins. Remote job scheduling has been implemented and 35 new jobs have been added for pbuilder and schroot creation and maintenance of the nodes. (h01ger) The manual scheduler gained a flag (-a/--architecture) to select which arch to schedule in. (Mattia Rizzolo) armhf will only be testing stretch for now, due to limited hardware ressources. (h01ger) The page listing maintainers of unreproducible packages gained internal anchors. As an example, one can now link to unreproducible orphaned packages. (Mattia Rizzolo) Packages with a bug tagged pending are marked using a new symbol: a brown P (Mattia Rizzolo) diffoscope development debbindiff is now called diffoscope! It also has a website at diffoscope.org. The name was changed to better reflect that it became a general purpose tool, capable of comparing many different archive formats, or directories. Version 29 is the renaming release. Amongst a couple of other cosmetic changes a favicon showing the new logo has been added to the generated HTML reports. Version 30 replaces the file matching algorithm for files listed in .changes to a smarter one that removes only the version number. It also fixes a bug where squashfs directories were being extracted even if their content was being compared at a later stage. It also fixes an issue with the test suite that was detected by debci. Documentation update More rationale have been added for supporting SOURCE_DATE_EPOCH The unfinished Reproducible Builds HOWTO is now visible on the web, feedback and patches most welcome. Package reviews 261 obsolete reviews have been removed, 73 added and 145 updated this week.

3 August 2015

Lunar: Reproducible builds: week 14 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes akira submitted a patch to make cdbs export SOURCE_DATE_EPOCH. She uploded a package with the enhancement to the experimental reproducible repository. Packages fixed The following 15 packages became reproducible due to changes in their build dependencies: dracut, editorconfig-core, elasticsearch, fish, libftdi1, liblouisxml, mk-configure, nanoc, octave-bim, octave-data-smoothing, octave-financial, octave-ga, octave-missing-functions, octave-secs1d, octave-splines, valgrind. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: In contrib, Dmitry Smirnov improved libdvd-pkg with 1.3.99-1-1. Patches submitted which have not made their way to the archive yet: reproducible.debian.net Four armhf build hosts were provided by Vagrant Cascadian and have been configured to be used by jenkins.debian.net. Work on including armhf builds in the reproducible.debian.net webpages has begun. So far the repository comparison page just shows us which armhf binary packages are currently missing in our repo. (h01ger) The scheduler has been changed to re-schedule more packages from stretch than sid, as the gcc5 transition has started This mostly affects build log age. (h01ger) A new depwait status has been introduced for packages which can't be built because of missing build dependencies. (Mattia Rizzolo) debbindiff development Finally, on August 31st, Lunar released debbindiff 27 containing a complete overhaul of the code for the comparison stage. The new architecture is more versatile and extensible while minimizing code duplication. libarchive is now used to handle cpio archives and iso9660 images through the newly packaged python-libarchive-c. This should also help support a couple other archive formats in the future. Symlinks and devices are now properly compared. Text files are compared as Unicode after being decoded, and encoding differences are reported. Support for Sqlite3 and Mono/.NET executables has been added. Thanks to Valentin Lorentz, the test suite should now run on more systems. A small defiency in unquashfs has been identified in the process. A long standing optimization is now performed on Debian package: based on the content of the md5sums control file, we skip comparing files with matching hashes. This makes debbindiff usable on packages with many files. Fuzzy-matching is now performed for files in the same container (like a tarball) to handle renames. Also, for Debian .changes, listed files are now compared without looking the embedded version number. This makes debbindiff a lot more useful when comparing different versions of the same package. Based on the rearchitecturing work has been done to allow parallel processing. The branch now seems to work most of the time. More test needs to be done before it can be merged. The current fuzzy-matching algorithm, ssdeep, has showed disappointing results. One important use case is being able to properly compare debug symbols. Their path is made using the Build ID. As this identifier is made with a checksum of the binary content, finding things like CPP macros is much easier when a diff of the debug symbols is available. Good news is that TLSH, another fuzzy-matching algorithm, has been tested with much better results. A package is waiting in NEW and the code is ready for it to become available. A follow-up release 28 was made on August 2nd fixing content label used for gzip2, bzip2 and xz files and an error on text files only differing in their encoding. It also contains a small code improvement on how comments on Difference object are handled. This is the last release name debbindiff. A new name has been chosen to better reflect that it is not a Debian specific tool. Stay tuned! Documentation update Valentin Lorentz updated the patch submission template to suggest to write the kind of issue in the bug subject. Small progress have been made on the Reproducible Builds HOWTO while preparing the related CCCamp15 talk. Package reviews 235 obsolete reviews have been removed, 47 added and 113 updated this week. 42 reports for packages failing to build from source have been made by Chris West (Faux). New issue added this week: haskell_devscripts_locale_substvars. Misc. Valentin Lorentz wrote a script to report packages tested as unreproducible installed on a system. We encourage everyone to run it on their systems and give feedback!

28 July 2015

Norbert Preining: ePub editor Sigil landed in Debian

Long long time ago I wanted to have Sigil, an epub editor, to appear in Debian. There was a packaging wishlist bug from back in 2010 with intermittent activities. But thanks to concerted effort, especially by Mattia Rizzolo and Don Armstrong, packaging progressed to a state that I could sponsor the upload to experimental about 4 months ago. And yesterday, after long waiting, finally Sigil passed the watchful eyes of the Debian ftp-masters and has entered Debian/experimental. sigil-debian I have already updated the packaging for the latest version 0.8.7, which will be included in Debian/sid rather soon. Thanks again especially Mattia for his great work.
Email this to someonePrint this pageShare on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInFlattr the author

26 July 2015

Lunar: Reproducible builds: week 12 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Eric Dorlan uploaded automake-1.15/1:1.15-2 which makes the output of mdate-sh deterministic. Original patch by Reiner Herrmann. Kenneth J. Pronovici uploaded epydoc/3.0.1+dfsg-8 which now honors SOURCE_DATE_EPOCH. Original patch by Reiner Herrmann. Chris Lamb submitted a patch to dh-python to make the order of the generated maintainer scripts deterministic. Chris also offered a fix for a source of non-determinism in dpkg-shlibdeps when packages have alternative dependencies. Dhole provided a patch to add support for SOURCE_DATE_EPOCH to gettext. Packages fixed The following 78 packages became reproducible in our setup due to changes in their build dependencies: chemical-mime-data, clojure-contrib, cobertura-maven-plugin, cpm, davical, debian-security-support, dfc, diction, dvdwizard, galternatives, gentlyweb-utils, gifticlib, gmtkbabel, gnuplot-mode, gplanarity, gpodder, gtg-trace, gyoto, highlight.js, htp, ibus-table, impressive, jags, jansi-native, jnr-constants, jthread, jwm, khronos-api, latex-coffee-stains, latex-make, latex2rtf, latexdiff, libcrcutil, libdc0, libdc1394-22, libidn2-0, libint, libjava-jdbc-clojure, libkryo-java, libphone-ui-shr, libpicocontainer-java, libraw1394, librostlab-blast, librostlab, libshevek, libstxxl, libtools-logging-clojure, libtools-macro-clojure, litl, londonlaw, ltsp, macsyfinder, mapnik, maven-compiler-plugin, mc, microdc2, miniupnpd, monajat, navit, pdmenu, pirl, plm, scikit-learn, snp-sites, sra-sdk, sunpinyin, tilda, vdr-plugin-dvd, vdr-plugin-epgsearch, vdr-plugin-remote, vdr-plugin-spider, vdr-plugin-streamdev, vdr-plugin-sudoku, vdr-plugin-xineliboutput, veromix, voxbo, xaos, xbae. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net The statistics on the main page of reproducible.debian.net are now updated every five minutes. A random unreviewed package is suggested in the look at a package form on every build. (h01ger) A new package set based new on the Core Internet Infrastructure census has been added. (h01ger) Testing of FreeBSD has started, though no results yet. More details have been posted to the freebsd-hackers mailing list. The build is run on a new virtual machine running FreeBSD 10.1 with 3 cores and 6 GB of RAM, also sponsored by Profitbricks. strip-nondeterminism development Andrew Ayer released version 0.009 of strip-nondeterminism. The new version will strip locales from Javadoc, include the name of files causing errors, and ignore unhandled (but rare) zip64 archives. debbindiff development Lunar continued its major refactoring to enhance code reuse and pave the way to fuzzy-matching and parallel processing. Most file comparators have now been converted to the new class hierarchy. In order to support for archive formats, work has started on packaging Python bindings for libarchive. While getting support for more archive formats with a common interface is very nice, libarchive is a stream oriented library and might have bad performance with how debbindiff currently works. Time will tell if better solutions need to be found. Documentation update Lunar started a Reproducible builds HOWTO intended to explain the different aspects of making software build reproducibly to the different audiences that might have to get involved like software authors, producers of binary packages, and distributors. Package reviews 17 obsolete reviews have been removed, 212 added and 46 updated this week. 15 new bugs for packages failing to build from sources have been reported by Chris West (Faux), and Mattia Rizzolo. Presentations Lunar presented Debian efforts and some recipes on making software build reproducibly at Libre Software Meeting 2015. Slides and a video recording are available. Misc. h01ger, dkg, and Lunar attended a Core Infrastructure Initiative meeting. The progress and tools mode for the Debian efforts were shown. Several discussions also helped getting a better understanding of the needs of other free software projects regarding reproducible builds. The idea of a global append only log, similar to the logs used for Certificate Transparency, came up on multiple occasions. Using such append only logs for keeping records of sources and build results has gotten the name Binary Transparency Logs . They would at least help identifying a compromised software signing key. Whether the benefits in using such logs justify the costs need more research.

18 July 2015

Niels Thykier: Performance tuning of lintian

For quite a while, Lintian has been able to create performance logs (--perf-debug --perf-output perf.log) that help diagnose where lintian spends most of its runtime. I decided to make lintian output these logs on lintian.debian.org to help us spot performance issues, though I have not been very good at analysing them regularly. At the beginning of the month, I finally got around to look a bit into one of them. My findings on IRC triggered Mattia Rizzolo to create this following graph. It shows the accumulated runtime of each check/collection measured in seconds. Find these findings, we set out to solve some of the issues. This lead to the following changes in 2.5.33 (in no particular order): Legend: S: run single threaded (1:1 performance improvement). P: run in parallel. Overall, I doubt the changes will give a revolutionary change in speed, but it should improve the 3rd, 4th and 5th slowest parts in Lintian. Beyond runtime performance, we got a few memory optimisations in the pipeline for Lintian 2.5.34: Combined these 6 commits reduce memory consumption in caches by ~33% compared to 2.5.33, when lintian processes itself. In absolute numbers, we are talking about a drop from 12.53MB to 8.48MB. The mileages can certainly vary depending on the package (mscgen only saw an ~25% improvement). Nevertheless, I was happy to list #715035 as being closed in 2.5.34. :)
Filed under: Debian, Lintian

12 July 2015

Lunar: Reproducible builds: week 11 in Stretch cycle

Debian is undertaking a huge effort to develop a reproducible builds system. I'd like to thank you for that. This could be Debian's most important project, with how badly computer security has been going.

PerniciousPunk in Reddit's Ask me anything! to Neil McGovern, DPL. What happened in the reproducible builds effort this week: Toolchain fixes More tools are getting patched to use the value of the SOURCE_DATE_EPOCH environment variable as the current time:

In the reproducible experimental toolchain which have been uploaded: Johannes Schauer followed up on making sbuild build path deterministic with several ideas. Packages fixed The following 311 packages became reproducible due to changes in their build dependencies : 4ti2, alot, angband, appstream-glib, argvalidate, armada-backlight, ascii, ask, astroquery, atheist, aubio, autorevision, awesome-extra, bibtool, boot-info-script, bpython, brian, btrfs-tools, bugs-everywhere, capnproto, cbm, ccfits, cddlib, cflow, cfourcc, cgit, chaussette, checkbox-ng, cinnamon-settings-daemon, clfswm, clipper, compton, cppcheck, crmsh, cupt, cutechess, d-itg, dahdi-tools, dapl, darnwdl, dbusada, debian-security-support, debomatic, dime, dipy, dnsruby, doctrine, drmips, dsc-statistics, dune-common, dune-istl, dune-localfunctions, easytag, ent, epr-api, esajpip, eyed3, fastjet, fatresize, fflas-ffpack, flann, flex, flint, fltk1.3, fonts-dustin, fonts-play, fonts-uralic, freecontact, freedoom, gap-guava, gap-scscp, genometools, geogebra, git-reintegrate, git-remote-bzr, git-remote-hg, gitmagic, givaro, gnash, gocr, gorm.app, gprbuild, grapefruit, greed, gtkspellmm, gummiboot, gyp, heat-cfntools, herold, htp, httpfs2, i3status, imagetooth, imapcopy, imaprowl, irker, jansson, jmapviewer, jsdoc-toolkit, jwm, katarakt, khronos-opencl-man, khronos-opengl-man4, lastpass-cli, lava-coordinator, lava-tool, lavapdu, letterize, lhapdf, libam7xxx, libburn, libccrtp, libclaw, libcommoncpp2, libdaemon, libdbusmenu-qt, libdc0, libevhtp, libexosip2, libfreenect, libgwenhywfar, libhmsbeagle, libitpp, libldm, libmodbus, libmtp, libmwaw, libnfo, libpam-abl, libphysfs, libplayer, libqb, libsecret, libserial, libsidplayfp, libtime-y2038-perl, libxr, lift, linbox, linthesia, livestreamer, lizardfs, lmdb, log4c, logbook, lrslib, lvtk, m-tx, mailman-api, matroxset, miniupnpd, mknbi, monkeysign, mpi4py, mpmath, mpqc, mpris-remote, musicbrainzngs, network-manager, nifticlib, obfsproxy, ogre-1.9, opal, openchange, opensc, packaging-tutorial, padevchooser, pajeng, paprefs, pavumeter, pcl, pdmenu, pepper, perroquet, pgrouting, pixz, pngcheck, po4a, powerline, probabel, profitbricks-client, prosody, pstreams, pyacidobasic, pyepr, pymilter, pytest, python-amqp, python-apt, python-carrot, python-django, python-ethtool, python-mock, python-odf, python-pathtools, python-pskc, python-psutil, python-pypump, python-repoze.tm2, python-repoze.what, qdjango, qpid-proton, qsapecng, radare2, reclass, repsnapper, resource-agents, rgain, rttool, ruby-aggregate, ruby-albino, ruby-archive-tar-minitar, ruby-bcat, ruby-blankslate, ruby-coffee-script, ruby-colored, ruby-dbd-mysql, ruby-dbd-odbc, ruby-dbd-pg, ruby-dbd-sqlite3, ruby-dbi, ruby-dirty-memoize, ruby-encryptor, ruby-erubis, ruby-fast-xs, ruby-fusefs, ruby-gd, ruby-git, ruby-globalhotkeys, ruby-god, ruby-hike, ruby-hmac, ruby-integration, ruby-jnunemaker-matchy, ruby-memoize, ruby-merb-core, ruby-merb-haml, ruby-merb-helpers, ruby-metaid, ruby-mina, ruby-net-irc, ruby-net-netrc, ruby-odbc, ruby-ole, ruby-packet, ruby-parseconfig, ruby-platform, ruby-plist, ruby-popen4, ruby-rchardet, ruby-romkan, ruby-ronn, ruby-rubyforge, ruby-rubytorrent, ruby-samuel, ruby-shoulda-matchers, ruby-sourcify, ruby-test-spec, ruby-validatable, ruby-wirble, ruby-xml-simple, ruby-zoom, rumor, rurple-ng, ryu, sam2p, scikit-learn, serd, shellex, shorewall-doc, shunit2, simbody, simplejson, smcroute, soqt, sord, spacezero, spamassassin-heatu, spamprobe, sphinxcontrib-youtube, splitpatch, sratom, stompserver, syncevolution, tgt, ticgit, tinyproxy, tor, tox, transmissionrpc, tweeper, udpcast, units-filter, viennacl, visp, vite, vmfs-tools, waffle, waitress, wavtool-pl, webkit2pdf, wfmath, wit, wreport, x11proto-input, xbae, xdg-utils, xdotool, xsystem35, yapsy, yaz. Please note that some packages in the above list are falsely reproducible. In the experimental toolchain, debhelper exported TZ=UTC and this made packages capturing the current date (without the time) reproducible in the current test environment. The following packages became reproducible after getting fixed: Ben Hutchings upstreamed several patches to fix Linux reproducibility issues which were quickly merged. Some uploads fixed some reproducibility issues but not all of them: Uploads that should fix packages not in main: Patches submitted which have not made their way to the archive yet: reproducible.debian.net A new package set has been added for lua maintainers. (h01ger) tracker.debian.org now only shows reproducibility issues for unstable. Holger and Mattia worked on several bugfixes and enhancements: finished initial test setup for NetBSD, rewriting more shell scripts in Python, saving UDD requests, and more debbindiff development Reiner Herrmann fixed text comparison of files with different encoding. Documentation update Juan Picca added to the commands needed for a local test chroot installation of the locales-all package. Package reviews 286 obsolete reviews have been removed, 278 added and 243 updated this week. 43 new bugs for packages failing to build from sources have been filled by Chris West (Faux), Mattia Rizzolo, and h01ger. The following new issues have been added: timestamps_in_manpages_generated_by_ronn, timestamps_in_documentation_generated_by_org_mode, and timestamps_in_pdf_generated_by_matplotlib. Misc. Reiner Herrmann has submitted patches for OpenWrt. Chris Lamb cleaned up some code and removed cruft in the misc.git repository. Mattia Rizzolo updated the prebuilder script to match what is currently done on reproducible.debian.net.

7 July 2015

Lunar: Reproducible builds: week 10 in Stretch cycle

What happened about the reproducible builds effort this week: Media coverage Daniel Stender published an English translation of the article which originally appeared in Linux Magazin in Admin Magazine. Toolchain fixes Fixes landed in the Debian archive: Lunar submitted to Debian the patch already sent upstream adding a --clamp-mtime option to tar. Patches have been submitted to add support for SOURCE_DATE_EPOCH to txt2man (Reiner Herrmann), epydoc (Reiner Herrmann), GCC (Dhole), and Doxygen (akira). Dhole uploaded a new experimental debhelper to the reproducible repository which exports SOURCE_DATE_EPOCH. As part of the experiment, the patch also sets TZ to UTC which should help with most timezone issues. It might still be problematic for some packages which would change their settings based on this. Mattia Rizzolo sent upstream a patch originally written by Lunar to make the generate-id() function be deterministic in libxslt. While that patch was quickly rejected by upstream, Andrew Ayer came up with a much better one which sadly could have some performance impact. Daniel Veillard replied with another patch that should be deterministic in most cases without needing extra data structures. It's impact is currently being investigated by retesting packages on reproducible.debian.net. akira added a new option to sbuild for configuring the path in which packages are built. This will be needed for the srebuild script. Niko Tyni asked Perl upstream about it using the __DATE__ and __TIME__ C processor macros. Packages fixed The following 143 packages became reproducible due to changes in their build dependencies: alot, argvalidate, astroquery, blender, bpython, brian, calibre, cfourcc, chaussette, checkbox-ng, cloc, configshell, daisy-player, dipy, dnsruby, dput-ng, dsc-statistics, eliom, emacspeak, freeipmi, geant321, gpick, grapefruit, heat-cfntools, imagetooth, jansson, jmapviewer, lava-tool, libhtml-lint-perl, libtime-y2038-perl, lift, lua-ldoc, luarocks, mailman-api, matroxset, maven-hpi-plugin, mknbi, mpi4py, mpmath, msnlib, munkres, musicbrainzngs, nova, pecomato, pgrouting, pngcheck, powerline, profitbricks-client, pyepr, pylibssh2, pylogsparser, pystemmer, pytest, python-amqp, python-apt, python-carrot, python-crypto, python-darts.lib.utils.lru, python-demgengeo, python-graph, python-mock, python-musicbrainz2, python-pathtools, python-pskc, python-psutil, python-pypump, python-repoze.sphinx.autointerface, python-repoze.tm2, python-repoze.what-plugins, python-repoze.what, python-repoze.who-plugins, python-xstatic-term.js, reclass, resource-agents, rgain, rttool, ruby-aggregate, ruby-archive-tar-minitar, ruby-bcat, ruby-blankslate, ruby-coffee-script, ruby-colored, ruby-dbd-mysql, ruby-dbd-odbc, ruby-dbd-pg, ruby-dbd-sqlite3, ruby-dbi, ruby-dirty-memoize, ruby-encryptor, ruby-erubis, ruby-fast-xs, ruby-fusefs, ruby-gd, ruby-git, ruby-globalhotkeys, ruby-god, ruby-hike, ruby-hmac, ruby-integration, ruby-ipaddress, ruby-jnunemaker-matchy, ruby-memoize, ruby-merb-core, ruby-merb-haml, ruby-merb-helpers, ruby-metaid, ruby-mina, ruby-net-irc, ruby-net-netrc, ruby-odbc, ruby-packet, ruby-parseconfig, ruby-platform, ruby-plist, ruby-popen4, ruby-rchardet, ruby-romkan, ruby-rubyforge, ruby-rubytorrent, ruby-samuel, ruby-shoulda-matchers, ruby-sourcify, ruby-test-spec, ruby-validatable, ruby-wirble, ruby-xml-simple, ruby-zoom, ryu, simplejson, spamassassin-heatu, speaklater, stompserver, syncevolution, syncmaildir, thin, ticgit, tox, transmissionrpc, vdr-plugin-xine, waitress, whereami, xlsx2csv, zathura. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net A new package set for the X Strike Force has been added. (h01ger) Bugs tagged with locale are now visible in the statistics. (h01ger) Some work has been done add tests for NetBSD. (h01ger) Many changes by Mattia Rizzolo have been merged on the whole infrastructure: debbindiff development Version 26 has been released on June 28th fixing the comparison of files of unknown format. (Lunar) A missing dependency identified in python-rpm affecting debbindiff installation without recommended packages was promptly fixed by Michal iha . Lunar also started a massive code rearchitecture to enhance code reuse and enable new features. Nothing visible yet, though. Documentation update josch and Mattia Rizzolo documented how to reschedule packages from Alioth. Package reviews 142 obsolete reviews have been removed, 344 added and 107 updated this week. Chris West (Faux) filled 13 new bugs for packages failing to build from sources. The following new issues have been added: snapshot_placeholder_replaced_with_timestamp_in_pom_properties, different_encoding, timestamps_in_documentation_generated_by_org_mode and timestamps_in_pdf_generated_by_matplotlib.

29 June 2015

Lunar: Reproducible builds: week 9 in Stretch cycle

What happened about the reproducible builds effort this week: Toolchain fixes Norbert Preining uploaded texinfo/6.0.0.dfsg.1-2 which makes texinfo indices reproducible. Original patch by Chris Lamb. Lunar submitted recently rebased patches to make the file order of files inside .deb stable. akira filled #789843 to make tex4ht stop printing timestamps in its HTML output by default. Dhole wrote a patch for xutils-dev to prevent timestamps when creating gzip compresed files. Reiner Herrmann sent a follow-up patch for wheel to use UTC as timezone when outputing timestamps. Mattia Rizzolo started a discussion regarding the failure to build from source of subversion when -Wdate-time is added to CPPFLAGS which happens when asking dpkg-buildflags to use the reproducible profile. SWIG errors out because it doesn't recognize the aforementioned flag. Trying to get the .buildinfo specification to more definitive state, Lunar started a discussion on storing the checksums of the binary package used in dpkg status database. akira discovered while proposing a fix for simgrid that CMake internal command to create tarballs would record a timestamp in the gzip header. A way to prevent it is to use the GZIP environment variable to ask gzip not to store timestamps, but this will soon become unsupported. It's up for discussion if the best place to fix the problem would be to fix it for all CMake users at once. Infrastructure-related work Andreas Henriksson did a delayed NMU upload of pbuilder which adds minimal support for build profiles and includes several fixes from Mattia Rizzolo affecting reproducibility tests. Neils Thykier uploaded lintian which both raises the severity of package-contains-timestamped-gzip and avoids false positives for this tag (thanks to Tomasz Buchert). Petter Reinholdtsen filled #789761 suggesting that how-can-i-help should prompt its users about fixing reproducibility issues. Packages fixed The following packages became reproducible due to changes in their build dependencies: autorun4linuxcd, libwildmagic, lifelines, plexus-i18n, texlive-base, texlive-extra, texlive-lang. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Untested uploaded as they are not in main: Patches submitted which have not made their way to the archive yet: debbindiff development debbindiff/23 includes a few bugfixes by Helmut Grohne that result in a significant speedup (especially on larger files). It used to exhibit the quadratic time string concatenation antipattern. Version 24 was released on June 23rd in a hurry to fix an undefined variable introduced in the previous version. (Reiner Herrmann) debbindiff now has a test suite! It is written using the PyTest framework (thanks Isis Lovecruft for the suggestion). The current focus has been on the comparators, and we are now at 93% of code coverage for these modules. Several problems were identified and fixed in the process: paths appearing in output of javap, readelf, objdump, zipinfo, unsqusahfs; useless MD5 checksum and last modified date in javap output; bad handling of charsets in PO files; the destination path for gzip compressed files not ending in .gz; only metadata of cpio archives were actually compared. stat output was further trimmed to make directory comparison more useful. Having the test suite enabled a refactoring of how comparators were written, switching from a forest of differences to a single tree. This helped removing dust from the oldest parts of the code. Together with some other small changes, version 25 was released on June 27th. A follow up release was made the next day to fix a hole in the test suite and the resulting unidentified leftover from the comparator refactoring. (Lunar) Documentation update Ximin Luo improved code examples for some proposed environment variables for reference timestamps. Dhole added an example on how to fix timestamps C pre-processor macros by adding a way to set the build date externally. akira documented her fix for tex4ht timestamps. Package reviews 94 obsolete reviews have been removed, 330 added and 153 updated this week. Hats off for Chris West (Faux) who investigated many fail to build from source issues and reported the relevant bugs. Slight improvements were made to the scripts for editing the review database, edit-notes and clean-notes. (Mattia Rizzolo) Meetings A meeting was held on June 23rd. Minutes are available. The next meeting will happen on Tuesday 2015-07-07 at 17:00 UTC. Misc. The Linux Foundation announced that it was funding the work of Lunar and h01ger on reproducible builds in Debian and other distributions. This was further relayed in a Bits from Debian blog post.

22 June 2015

Lunar: Reproducible builds: week 8 in Stretch cycle

What happened about the reproducible builds effort this week: Toolchain fixes Andreas Henriksson has improved Johannes Schauer initial patch for pbuilder adding support for build profiles. Packages fixed The following 12 packages became reproducible due to changes in their build dependencies: collabtive, eric, file-rc, form-history-control, freehep-chartableconverter-plugin , jenkins-winstone, junit, librelaxng-datatype-java, libwildmagic, lightbeam, puppet-lint, tabble. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net Bugs with the ftbfs usertag are now visible on the bug graphs. This explain the recent spike. (h01ger) Andreas Beckmann suggested a way to test building packages using the funny paths that one can get when they contain the full Debian package version string. debbindiff development Lunar started an important refactoring introducing abstactions for containers and files in order to make file type identification more flexible, enabling fuzzy matching, and allowing parallel processing. Documentation update Ximin Luo detailed the proposal to standardize environment variables to pass a reference source date to tools that needs one (e.g. documentation generator). Package reviews 41 obsolete reviews have been removed, 168 added and 36 updated this week. Some more issues affecting packages failing to build from source have been identified. Meetings Minutes have been posted for Tuesday June 16th meeting. The next meeting is scheduled Tuesday June 23rd at 17:00 UTC. Presentations Lunar presented the project in French during Pas Sage en Seine in Paris. Video and slides are available.

20 June 2015

Lunar: Reproducible builds: week 4 in Stretch cycle

What happened about the reproducible builds effort for this week: Toolchain fixes Lunar rebased our custom dpkg on the new release, removing a now unneeded patch identified by Guillem Jover. An extra sort in the buildinfo generator prevented a stable order and was quickly fixed once identified. Mattia Rizzolo also rebased our custom debhelper on the latest release. Packages fixed The following 30 packages became reproducible due to changes in their build dependencies: animal-sniffer, asciidoctor, autodock-vina, camping, cookie-monster, downthemall, flashblock, gamera, httpcomponents-core, https-finder, icedove-l10n, istack-commons, jdeb, libmodule-build-perl, libur-perl, livehttpheaders, maven-dependency-plugin, maven-ejb-plugin, mozilla-noscript, nosquint, requestpolicy, ruby-benchmark-ips, ruby-benchmark-suite, ruby-expression-parser, ruby-github-markup, ruby-http-connection, ruby-settingslogic, ruby-uuidtools, webkit2gtk, wot. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Also, the following bugs have been reported: reproducible.debian.net Holger Levsen made several small bug fixes and a few more visible changes: strip-nondeterminism Version 0.007-1 of strip-nondeterminism the tool to post-process various file formats to normalize them has been uploaded by Holger Levsen. Version 0.006-1 was already in the reproducible repository, the new version mainly improve the detection of Maven's pom.properties files. debbindiff development At the request of Emmanuel Bourg, Reiner Herrmann added a comparator for Java .class files. Documentation update Christoph Berg created a new page for the timestamps in manpages created by Doxygen. Package reviews 93 obsolete reviews have been removed, 76 added and 43 updated this week. New identified issues: timestamps in manpages generated by Doxygen, modification time differences in files extracted by unzip, tstamp task used in Ant build.xml, timestamps in documentation generated by ASDocGen. The description for build id related issues has been clarified. Meetings Holger Levsen announced a first meeting on Wednesday, June 3rd, 2015, 19:00 UTC. The agenda is amendable on the wiki. Misc. Lunar worked on a proof-of-concept script to import the build environment found in .buildinfo files to UDD. Lucas Nussbaum has positively reviewed the proposed schema. Holger Levsen cleaned up various experimental toolchain repositories, marking merged brances as such.

Next.

Previous.