As web search engines and IRC seems to be of no help, maybe someone here has a helpful idea. I have some service written in python that comes with a .service file for systemd. I now want to build&install a working service file from the software's setup.py. I can override the build/build_py commands of setuptools, however that way I still lack knowledge wrt. the bindir/prefix where my service script will be installed. Solution Turns out, if you override the install command (not the install_data!), you will have self.root and self.install_scripts (and lots of other self.install_*). As a result, you can read the template and write the desired output file after calling super's run method. The fix was inspired by GateOne (which, however doesn't get the --root parameter right, you need to strip self.root from the beginning of the path to actually make that work as intended). As suggested on IRC, the snippet (and my software) no use pkg-config to get at the systemd path as well. This is a nice improvement orthogonal to the original problem. The implementation here follows bley.

def systemd_unit_path():
    try:
        command = ["pkg-config", "--variable=systemdsystemunitdir", "systemd"]
        path = subprocess.check_output(command, stderr=subprocess.STDOUT)
        return path.decode().replace('\n', '')
    except (subprocess.CalledProcessError, OSError):
        return "/lib/systemd/system"
class my_install(install):
    _servicefiles = [
        'foo/bar.service',
        ]
    def run(self):
        install.run(self)
        if not self.dry_run:
            bindir = self.install_scripts
            if bindir.startswith(self.root):
                bindir = bindir[len(self.root):]
            systemddir = "%s%s" % (self.root, systemd_unit_path())
            for servicefile in self._servicefiles:
                service = os.path.split(servicefile)[1]
                self.announce("Creating %s" % os.path.join(systemddir, service),
                              level=2)
                with open(servicefile) as servicefd:
                    servicedata = servicefd.read()
                with open(os.path.join(systemddir, service), "w") as servicefd:
                    servicefd.write(servicedata.replace("%BINDIR%", bindir))

Comments, suggestions and improvements, of course, welcome!

What happened in the Reproducible Builds effort between Sunday October 9 and Saturday October 15 2016: Media coverage

• despinosa wrote a blog post on Vala and reproducibility
• h01ger and lynxis gave a talk called "From Reproducible Debian builds to Reproducible OpenWrt, LEDE" (video, slides) at the OpenWrt Summit 2016 held in Berlin, together with ELCE, held by the Linux Foundation.
• A discussion on debian-devel@ resulted in a nice quotable comment from Paul Wise: "(Reproducible) builds from source (with continuous rechecking) is the only way to have enough confidence that a Debian user has the freedoms promised to them by the Debian social contract."
• Chris Lamb will present a talk at Software Freedom Kosovo on reproducible builds on Saturday 22nd October.

Documentation update After discussions with HW42, Steven Chamberlain, Vagrant Cascadian, Daniel Shahaf, Christopher Berg, Daniel Kahn Gillmor and others, Ximin Luo has started writing up more concrete and detailed design plans for setting SOURCE_ROOT_DIR for reproducible debugging symbols, buildinfo security semantics and buildinfo security infrastructure. Toolchain development and fixes Dmitry Shachnev noted that our patch for #831779 has been temporarily rejected by docutils upstream; we are trying to persuade them again. Tony Mancill uploaded javatools/0.59 to unstable containing original patch by Chris Lamb. This fixed an issue where documentation Recommends: substvars would not be reproducible. Ximin Luo filed bug 77985 to GCC as a pre-requisite for future patches to make debugging symbols reproducible. Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible - in our current test setup - after being fixed:

• cobbler/2.6.6+dfsg1-13 by Thomas Goirand, original patch by Chris Lamb.
• collectd/5.6.1-1 by Marc Fournier.
• fonts-tiresias/0.1-3 by G rkan Myczko, original patch by Chris Lamb.
• fntsample/4.0-2 by , original patch by Chris Lamb.
• fpga-icestorm/0~20160913git266e758-2 by Ruben Undheim, original patch by Chris Lamb.
• frog/0.13.5-1 by Maarten van Gompel, original patch by Chris Lamb.
• lambda-align/1.0.0-2 by Sascha Steinbiss, original patch by Chris Lamb.
• pleiades/1.7.0-2 by Hideki Yamane, original patch by Chris Lamb.
• sweethome3d/5.2+dfsg-1 by Markus Koschany, original fix by Gabriele Giacone.
• trac-subtickets/0.2.0-2 by W. Martin Borgert.

The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.)

• aodh/3.0.0-2 by Thomas Goirand.
• eog-plugins/3.16.5-1 by Michael Biebl.
• flam3/3.0.1-5 by Daniele Adriana Goulart Lopes.
• hyphy/2.2.7+dfsg-1 by Andreas Tille.
• libbson/1.4.1-1 by A. Jesse Jiryu Davis.
• libmongoc/1.4.1-1 by A. Jesse Jiryu Davis.
• lxc/1:2.0.5-1 by Evgeni Golov.
• spice-gtk/0.33-1 by Liang Guo.
• spice-vdagent/0.17.0-1 by Liang Guo.
• tnef/1.4.12-1 by Kevin Coyner.

Some uploads have addressed some reproducibility issues, but not all of them:

• chktex/1.7.6-1 by Thorsten Alteholz, original patch by Sascha Steinbiss.
• dbus/1.10.12-1 by Simon McVittie.
• doomsday/1.15.8-3 by Markus Koschany, #839338 by Lucas Nussbaum.
• emacs25/25.1+1-1 by Rob Browning.
• gpgme1.0/1.7.0-3 by Daniel Kahn Gillmor.
• monkeysign/2.2.0 by Antoine Beaupr .
• python-attrs/16.2.0-1 by Tristan Seligmann, original patch by Chris Lamb.
• shotwell/0.24.0-1 by J rg Frings-F rst, original patch by Alexis Bienven e.
• supple/1.0.6-2 by Daniel Silverstone.
• why/2.36-1 by Ralf Treinen, original patch by Valentin Lorentz.

Some uploads have addressed nearly all reproducibility issues, except for build path issues:

• palo/1.96 by Helge Deller, #778437 by Chris Lamb.
• rbdoom3bfg/1.1.0~preview3+dfsg+git20160807-1 by Tobias Frost.
• singular/4.0.3-p3+ds-1 by Jerome Benoit.
• varnish/5.0.0-3 by Stig Sandbeck Mathisen, original patch by Chris Lamb.
• yaml-cpp/0.5.2-4 by Paul Novotny, original patch by Reiner Herrmann.

Patches submitted that have not made their way to the archive yet:

• #840741 filed against http-icons by Chris Lamb.
• #840177 filed against qconf by Chris Lamb.
• #840845 filed against python-pygraphviz by Chris Lamb.
• #840346 filed against qjoypad by Chris Lamb.

Reviews of unreproducible packages 101 package reviews have been added, 49 have been updated and 4 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated:

• Added max_output_size_reached, ftbfs_due_to_jenkins_semaphore_setup, and build_id_differences_only.

Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by:

• Anders Kaseorg (1)
• Chris Lamb (18)

tests.reproducible-builds.org Debian:

• h01ger has turned off the "Scheduled in testing+unstable+experimental" regular IRC notifications and turned them into emails to those running jenkins.d.n.
• Re-add opi2a armhf node and 3 new builder jobs for a total of 60 build jobs for armhf. (h01ger and vagrant)
• vagrant suggested to add a variation of init systems effecting the build, and h01ger added it to the TODO list.
• Steven Chamberlain submitted a patch so that now all buildinfo files are collected (unsigned yet) at submit@buildinfo.kfreebsd.eu.
• Holger enabled CPU type variation (Intel Haswell or AMD Opteron 62xx) for i386. Thanks to Profitbricks.com for their great and continued support!

Openwrt/LEDE/NetBSD/coreboot/Fedora/archlinux:

• Increase memory on the 2 build nodes from 12 to 16gb, thanks to profitbricks.com

Misc. We are running a poll to find a good time for an IRC meeting. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

Here is my monthly update covering what I have been doing in the free software world (previous month):

• Fixed a number of instances in the Django web development framework where methods had mutable defaults arguments such as lists or dictionaries. (#7253)
• Made a large number of improvements to travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every update:
  • Worked with Evgeni Golov to run autopkgtest and autodep8 tests after builds. (#23, #24 & #25)
  • Ensured that all remote branches are created locally so that packages that specify options such as pristine_tar=True in debian/gbp.conf work as expected. (commit)
  • Fixed an issue where checking out other branches did not reliably not work due to Travis cloning the repository with --depth=50. (#21)
  • Updated the documentation after prompting from Valerie Young. (db7238 & 470d24)
• Improved my Chrome extension for the FastMail web interface to collapse whitespace when appending to the subject. (#9)
• Submitted a pull request for the Handbrake video transcoder to make the build reproducible. (#320)
• Updated the configuration for my OpenWRT-based router (a device I use as a range extender and to save tedious reconfiguration of all my wifi devices in new locales) to enter a "fallback" mode if it cannot connect to the client network. This works around a longstanding "wontfix" upstream issue. (diff)
• Updated django-agpl, my reusable Django application to assist in releasing projects under the GNU Affero General Public License:
  • Merged support from defivelo to support Django 1.8. (#1)
  • Improved the code quality & testability throughout the package. (diff)
• Filed in issue in Mailvelope (a browser extension for OpenPGP encryption) to support composing/viewing messages in a fixed-width font. (#422)
• Blogged about how to write your first Lintian check as well as on the progress bar I added to Diffoscope.

---

I am happy to announce the release of OASIS v0.4.7. OASIS is a tool to help OCaml developers to integrate configure, build and install systems in their projects. It should help to create standard entry points in the source code build system, allowing external tools to analyse projects easily. This tool is freely inspired by Cabal which is the same kind of tool for Haskell. You can find the new release here and the changelog here. More information about OASIS in general on the OASIS website. Pull request for inclusion in OPAM is pending. Here is a quick summary of the important changes:

• Drop support for OASISFormat 0.2 and 0.1.
• New plugin "omake" to support build, doc and install actions.
• Improve automatic tests (Travis CI and AppVeyor)
• Trim down the dependencies (removed ocaml-gettext, camlp4, ocaml-data-notation)

Features:

• findlib_directory (beta): to install libraries in sub-directories of findlib.
• findlib_extra_files (beta): to install extra files with ocamlfind.
• source_patterns (alpha): to provide module to source file mapping.

This version contains a lot of changes and is the achievement of a huge amount of work. The addition of OMake as a plugin is a huge progress. The overall work has been targeted at making OASIS more library like. This is still a work in progress but we made some clear improvement by getting rid of various side effect (like the requirement of using "chdir" to handle the "-C", which leads to propage ~ctxt everywhere and design OASISFileSystem). I would like to thanks again the contributor for this release: Spiros Eliopoulos, Paul Snively, Jeremie Dimino, Christopher Zimmermann, Christophe Troestler, Max Mouratov, Jacques-Pascal Deplaix, Geoff Shannon, Simon Cruanes, Vladimir Brankov, Gabriel Radanne, Evgenii Lepikhin, Petter Urkedal, Gerd Stolpmann and Anton Bachin.

What happened in the Reproducible Builds effort between April 24th and 30th 2016. Media coverage Reproducible builds were mentioned explicitly in two talks at the Mini-DebConf in Vienna:

• Martin Michlmayr had a talk in which he presented an overview about innovations and changes in Debian in the last years. Martin expressed his disappointment that there was no talk from us in Vienna (we'll fix this at DebConf16 in Cape Town) and described the reproducible builds work as "a real innovation". His talk is very much worth seeing, whatever your current perspective, it might change your view on Debian.
• Ben Hutchings explains how Secure Boot will use signed kernels via separate signature packages and how this was designed with reproducible builds in mind.

Aspiration together with the OTF CommunityLab released their report about the Reproducible Builds summit in December 2015 in Athens. Toolchain fixes Now that the GCC development window has been opened again, the SOURCE_DATE_EPOCH patch by Dhole and Matthias Klose to address the issue timestamps_from_cpp_macros (__DATE__ / __TIME__) has been applied upstream and will be released with GCC 7. Following that Matthias Klose also has uploaded gcc-5/5.3.1-17 and gcc-6/6.1.1-1 to unstable with a backport of that SOURCE_DATE_EPOCH patch. Emmanuel Bourg uploaded maven/3.3.9-4, which uses SOURCE_DATE_EPOCH for the maven.build.timestamp. (SOURCE_DATE_EPOCH specification) Other upstream changes Alexis Bienven e submitted a patch to Sphinx which extends SOURCE_DATE_EPOCH support for copyright years in generated documentation. Packages fixed The following 12 packages have become reproducible due to changes in their build dependencies: hhvm jcsp libfann libflexdock-java libjcommon-java libswingx1-java mobile-atlas-creator not-yet-commons-ssl plexus-utils squareness svnclientadapter The following packages have became reproducible after being fixed:

• anope/2.0.3-2 by Dominic Hargreaves, original patch by Alexis Bienven e.
• apparmor-profiles-extra/1.7 by intrigeri, patch by Felix Geyer.
• basket/2.10~beta+git20160425.b77687f-1 by Luigi Toscano, original patch by Alexis Bienven e.
• bind-dyndb-ldap/8.0-2 by Timo Aaltonen.
• bliss/0.73-1 by Jerome Benoit.
• gap-alnuth/3.0.0-3 by Bill Allombert.
• gap-radiroot/2.7-2 by Bill Allombert.
• genometools/1.5.8+ds-3 by Sascha Steinbiss.
• gprbuild/2015-3 by Nicolas Boulenguez.
• gtkhash/0.7.0-3 by M nica Ram rez Arceda.
• josm/0.0.svn10161+dfsg-1 by Bas Couwenberg.
• libjgoodies-animation-java/1.4.3-1 by Markus Koschany.
• ltrsift/1.0.2-7 by Sascha Steinbiss.
• mutt/1.6.0-1 by Matteo F. Vescovi, original patch by Daniel Shahaf.
• netsniff-ng/0.6.1-1 by Kartik Mistry, original patch by Reiner Herrmann.
• netrw/1.3.2-3 by Giovanni Mascellani.
• openthesaurus/20160424-3 by Rene Engelhard, original patch by Dhole.
• php-pear/1:1.10.1+submodules+notgz-8 by Mathieu Parent.
• samba/2:4.4.2+dfsg-2 by Jelmer Vernoo .
• swift/2.7.0-3 by Ond ej Nov .
• tp-smapi/0.42-1 by Evgeni Golov, original patch by Chris Lamb.
• vifm/0.8.1a-0.1 by Ond ej Nov .
• xfonts-mplus/1:2.2.4-2 by Hideki Yamane, original patch by Chris Lamb.
• xpa/2.1.17-2 by Ole Streicher, original patch by Alexis Bienven e.

Some uploads have fixed some reproducibility issues, but not all of them:

• camitk/4.0.0~beta-1 by Emmanuel Promayon, original patch by Maria Valentina Marin.
• elastix/4.8-8 by Gert Wollny.
• freefem++/3.46+dfsg1-1 by Dimitrios Eftaxiopoulos, original patch by Alexis Bienven e.
• grib-api/1.15.0-1 by Alastair McKinstry, original patch by Santiago Vila.
• isorelax/20041111-9 by Emmanuel Bourg.
• libical/2.0.0-0.1 by Andreas Henriksson, original patch by Chris Lamb.
• sawfish/1:1.11.90-1 by Jose M Calhariz, original patch by Alexis Bienven e.
• singular/4.0.3-p1+ds-1 by Jerome Benoit.
• syslinux/3:6.03+dfsg-12 by Mattia Rizzolo, original patch by Reiner Herrmann.
• transdecoder/2.0.1+dfsg-3 by Michael R. Crusoe, original patch by Dhole.

Patches submitted that have not made their way to the archive yet:

• #822566 against stk by Alexis Bienven e: sort lists of object files for reproducible linking order.
• #822948 against shotwell by Alexis Bienven e: normalize tarball permissions and use locale/timezone-independent modification time.
• #822963 against htop by Alexis Bienven e: use SOURCE_DATE_EPOCH for embedded copyright year, which has before already been applied in git and upstream.

Package reviews 95 reviews have been added, 15 have been updated and 129 have been removed in this week. 22 FTBFS bugs have been reported by Chris Lamb and Martin Michlmayr. diffoscope development

• diffoscope 52~bpo8+1 has been uploaded to jessie-backports by Mattia Rizzolo, where it is currently waiting for NEW-approval.
• Support for the deb(5) format (uncompressed data.tar/control.tar, control.tar.xz) (Closes: #818414) has been completed by Reiner Herrmann in git.

strip-nondeterminism development

• Support for EPUB documents has been added (to the development version in git) by Holger Levsen, to address the timestamps_in_epub issue.

tests.reproducible-builds.org

• To monitor the uptodateness of diffoscope everywhere, tests checking this on FreeBSD, NetBSD & MacPorts were added. Tests on other distributions will be added once the relevant bugs in whohas are fixed in jessie. (h01ger)

Misc. Amongst the 29 interns who will work on Debian through GSoC and Outreachy there are four who will be contributing to Reproducible Builds for Debian and Free Software. We are very glad to welcome ceridwen, Satyam Zode, Scarlett Clark and Valerie Young and look forward to working together with them the coming months (and maybe beyond)! This week's edition was written by Reiner Herrmann and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

What happened in the reproducible builds effort between April 10th and April 16th 2016: Toolchain fixes

• Roland Rosenfeld uploaded transfig/1:3.2.5.e-6 which honors SOURCE_DATE_EPOCH. Original patch by Alexis Bienven e.
• Bill Allombert uploaded gap/4r8p3-2 which makes convert.pl honor SOURCE_DATE_EPOCH. Original patch by Jerome Benoit, duplicate patch by Dhole.
• Emmanuel Bourg uploaded ant/1.9.7-1 which makes the Javadoc task use UTF-8 as the default encoding if none was specified and SOURCE_DATE_EPOCH is set.

Antoine Beaupr suggested that gitpkg stops recording timestamps when creating upstream archives. Antoine Beaupr also pointed out that git-buildpackage diverges from the default gzip settings which is a problem for reproducibly recreating released tarballs which were made using the defaults. Alexis Bienven e submitted a patch extending sphinx SOURCE_DATE_EPOCH support to copyright year. Packages fixed The following packages have become reproducible due to changes in their build dependencies: atinject-jsr330, avis, brailleutils, charactermanaj, classycle, commons-io, commons-javaflow, commons-jci, gap-radiroot, jebl2, jetty, libcommons-el-java, libcommons-jxpath-java, libjackson-json-java, libjogl2-java, libmicroba-java, libproxool-java, libregexp-java, mobile-atlas-creator, octave-econometrics, octave-linear-algebra, octave-odepkg, octave-optiminterp, rapidsvn, remotetea, ruby-rinku, tachyon, xhtmlrenderer. The following packages became reproducible after getting fixed:

• autopkgtest/3.20.3 uploaded by Martin Pitt, original patch by Alexis Bienven e.
• fop/1:2.1-3 uploaded by Mathieu Malaterre, original patch by Alexis Bienven e.
• fwupdate/0.5-4 by Mario Limonciello.
• gem/1:0.93.3-10 by IOhannes m zm lnig.
• imview/1.1.9c-16 by Anton Gladky.
• libappindicator/0.4.92-4 by Adam Borowski.
• libjlha-java/0.0.20050504-9 by Emmanuel Bourg.
• lwjgl/2.9.3+dfsg-1 by Markus Koschany.
• lxc/1:2.0.0~rc15-1 uploaded by Evgeni Golov, original patch by Reiner Herrmann.
• manpages-de/1.12-1 uploaded by Tobias Quathamer, original patch by Reiner Herrmann.
• pd-mrpeach/0.1~svn17615-1 by IOhannes m zm lnig.
• pyhoca-gui/0.5.0.6-1 uploaded by Mike Gabriel, original patch by Chris Lamb.
• r-cran-spatstat/1.45-0-1 by Andreas Tille.
• s5/1.1.dfsg.2-6 uploaded by Peter Pentchev, original patch by Juan Picca.
• samba/2:4.3.8+dfsg-1 by Jelmer Vernoo .
• sim4/0.0.20121010-3 uploaded by Andreas Tille, original patch by Alexis Bienven e.
• svxlink/15.11-1 uploaded by Felix Lechner, fixed upstream.
• tinc/1.0.28-1 by Guus Sliepen, fixed upstream.
• ucblogo/6.0+dfsg-1 by Reiner Herrmann.

Some uploads fixed some reproducibility issues, but not all of them:

• mm-common/0.9.10-1 by Michael Biebl.

Patches submitted which have not made their way to the archive yet:

• #820603 on viking by Alexis Bienven e: fix icon headers inclusion order.
• #820661 on nullmailer by Alexis Bienven e: fix the order in which files are included in the static archive.
• #820668 on sawfish by Alexis Bienven e: fix file ordering in theme archives, strip hostname and username from the config.h file, and honour SOURCE_DATE_EPOCH when creating the config.h file.
• #820740 on bless by Alexis Bienven e: always use /bin/sh as shell.
• #820742 on gmic by Alexis Bienven e: strip the build date from help messages.
• #820809 on wsdl4j by Alexis Bienven e: use a plain text representation of the copyright character.
• #820815 on freefem++ by Alexis Bienven e: fix the order in which files are included in the .edp files, and honour SOURCE_DATE_EPOCH when using the build date.
• #820869 on pyexiv2 by Alexis Bienven e: honour the SOURCE_DATE_EPOCH environment variable through the ustrftime function, to get a reproducible copyright year.
• #820932 on fim by Alexis Bienven e: fix the order in which files are joined in header files, strip the build date from fim binary, make the embeded vim2html script honour SOURCE_DATE_EPOCH variable when building the documentation, and force language to be English when using bison to make a grammar that is going to be parsed using English keywords.
• #820990 on grib-api by Santiago Vila: always call dh-buildinfo.

diffoscope development Zbigniew J drzejewski-Szmek noted in #820631 that diffoscope doesn't work properly when a file contains several cpio archives. Package reviews 21 reviews have been added, 14 updated and 22 removed in this week. New issue found: timestamps_in_htm_by_gap. Chris Lamb reported 10 new FTBFS issues. Misc. The video and the slides from the talk "Reproducible builds ecosystem" at LibrePlanet 2016 have been published now. This week's edition was written by Lunar and Holger Levsen. h01ger automated the maintenance and publishing of this weekly newsletter via git.

A good amount of the Debian reproducible builds team had the chance to enjoy face-to-face interactions during DebConf15.

Names in red and blue were all present at DebConf15

Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian. A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive. Toolchain fixes

• Kenneth J. Pronovici uploaded epydoc/3.0.1+dfsg-12 which makes class and modules ordering predictable (#795835) and fixes __repr__ so memory addresses don't appear in docs (#795826). Patches by Val Lorentz.
• Sergei Golovan uploaded erlang/1:18.0-dfsg-2 which adds support for SOURCE_DATE_EPOCH to erlc. Patch by Chris West (Faux) and Chris Lamb.
• Dmitry Shachnev uploaded sphinx/1.3.1-5 which make grammar, inventory, and JavaScript locales generation deterministic. Original patch by Val Lorentz.
• St phane Glondu uploaded ocaml/4.02.3-2 to experimental, making startup files and native packed libraries deterministic. The patch adds deterministic .file to the assembler output.
• Enrico Tassi uploaded lua-ldoc/1.4.3-3 which now pass the -d option to txt2man and add the --date option to override the current date.

Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2html SOURCE_DATE_EPOCH aware.

• #796330 on d-shlibs by Reiner Herrmann: sort with LC_ALL set to C.

St phane Glondu reported that dpkg-source would not respect tarball permissions when unpacking under a umask of 002. After hours of iterative testing during the DebConf workshop, Sandro Knau created a test case showing how pdflatex output can be non-deterministic with some PNG files. Packages fixed The following 65 packages became reproducible due to changes in their build dependencies: alacarte, arbtt, bullet, ccfits, commons-daemon, crack-attack, d-conf, ejabberd-contrib, erlang-bear, erlang-cherly, erlang-cowlib, erlang-folsom, erlang-goldrush, erlang-ibrowse, erlang-jiffy, erlang-lager, erlang-lhttpc, erlang-meck, erlang-p1-cache-tab, erlang-p1-iconv, erlang-p1-logger, erlang-p1-mysql, erlang-p1-pam, erlang-p1-pgsql, erlang-p1-sip, erlang-p1-stringprep, erlang-p1-stun, erlang-p1-tls, erlang-p1-utils, erlang-p1-xml, erlang-p1-yaml, erlang-p1-zlib, erlang-ranch, erlang-redis-client, erlang-uuid, freecontact, givaro, glade, gnome-shell, gupnp, gvfs, htseq, jags, jana, knot, libconfig, libkolab, libmatio, libvsqlitepp, mpmath, octave-zenity, openigtlink, paman, pisa, pynifti, qof, ruby-blankslate, ruby-xml-simple, timingframework, trace-cmd, tsung, wings3d, xdg-user-dirs, xz-utils, zpspell. The following packages became reproducible after getting fixed:

• apr/1.5.2-3 by Stefan Fritsch.
• aprx/2.08.svn593+dfsg-2 uploaded by Colin Tuckley, original patch by Chris Lamb.
• blkreplay/1.0-3 uploaded by Andrew Shadura, patch by Dhole.
• cal3d/0.11.0-6 uploaded by Manuel A. Fernandez Montecelo, patch by akira.
• cgsi-gsoap/1.3.8-1 by Mattias Ellert.
• eyefiserver/2.4+dfsg-1 by Jean-Michel Vourg re.
• gnujump/1.0.8-3 uploaded by Evgeni Golov, original patch by Chris Lamb.
• hkgerman/1:2-29 by Roland Rosenfeld.
• jove/4.16.0.73-4 by Cord Beermann.
• libevhtp/1.2.10-3 by Vincent Bernat.
• libmkdoc-xml-perl/0.75-4 uploaded by gregor herrmann, original patch by Niko Tyni.
• libparse-debianchangelog-perl/1.2.0-6 by Niko Tyni.
• librostlab-blast/1.0.1-5 uploaded by Andreas Tille, original patch by Chris Lamb.
• libxray-absorption-perl/3.0.1-3 uploaded by gregor herrmann, original patch by Niko Tyni.
• lua-penlight/1.3.2-2 by Enrico Tassi.
• mosquitto/1.4.3-1 by Roger A. Light.
• nagios-plugins-contrib/15.20150818 by Jan Wagner and Bernd Zeimetz.
• nn/6.7.3-10 uploaded by Cord Beermann, original patch by Chris Lamb.
• pybik/2.1-1 by B. Clausius.
• pyepr/0.9.3-1 uploaded by Antonio Valentino, original patch by Juan Picca.
• python-xlrd/0.9.4-1 by Vincent Bernat.
• transmissionrpc/0.11-2 uploaded by Vincent Bernat, original patch by Juan Picca.
• unoconv/0.7-1.1 sponsored by Vincent Bernat, fix by Dhole.
• vim-latexsuite/20141116.812-1 uploaded by Johann Felix Soden, original patch by Chris Lamb.
• volk/1.0.2-2 by A. Maitland Bottoms.
• xbmc/2:13.2+dfsg1-5 by Balint Reczey.
• xdotool/1:3.20150503.1-2 uploaded by Daniel Kahn Gillmor, initial patch by Chris Lamb.
• xfig/1:3.2.5.c-5 by Roland Rosenfeld.
• xfireworks/1.3-10 uploaded by Yukiharu YABUKI, original patch by Chris Lamb.
• xul-ext-monkeysphere/0.8-2 uploaded by Daniel Kahn Gillmor, original patch by Dhole.

Uploads that might have fixed reproducibility issues:

• brian/1.4.1-3 uploaded by Yaroslav Halchenko, original patch by Juan Picca.
• opennebula/4.12.3+dfsg-1 by Dmitry Smirnov.
• pcsx2/1.3.1-1008-g9f291a6+dfsg-1 by Miguel A. Col n V lez.
• webassets/3:0.11-1 uploaded by Agustin Henze, original patch by Reiner Herrmann.

Some uploads fixed some reproducibility issues but not all of them:

• apache2/2.4.16-3 uploaded by Stefan Fritsch, original patch by Jean-Michel Vourg re.
• gerris/20131206+dfsg-6 uploaded by Anton Gladky, original patch by Reiner Herrmann.
• kodi/15.1+dfsg1-1 by Balint Reczey.
• zshdb/0.05+git20101031-4 uploaded by Iain R. Learmonth, original patch by Chris Lamb.

Patches submitted which have not made their way to the archive yet:

• #795861 on fakeroot by Val Lorentz: set the mtime of all files to the time of the last debian/changelog entry.
• #795870 on fatresize by Chris Lamb: set build date to the time of the latest debian/changelog entry.
• #795945 on projectl by Reiner Herrmann: sort with LC_ALL set to C.
• #795977 on dahdi-tools by Dhole: set the timezone to UTC before calling asciidoc.
• #795981 on x11proto-input by Dhole: set the timezone to UTC before calling asciidoc.
• #795983 on dbusada by Dhole: set the timezone to UTC before calling asciidoc.
• #795984 on postgresql-plproxy by Dhole: set the timezone to UTC before calling asciidoc.
• #795985 on xorg by Dhole: set the timezone to UTC before calling asciidoc.
• #795987 on pngcheck by Dhole: set the date in the man pages to the latest debian/changelog entry.
• #795997 on python-babel by Val Lorentz: make build timestamp independent from the timezone and remove the name of the build system locale from the documentation.
• #796092 on a7xpg by Reiner Herrmann: sort with LC_ALL set to C.
• #796212 on bittornado by Chris Lamb: remove umask-varying permissions.
• #796251 on liblucy-perl by Niko Tyni: generate lib/Lucy.xs in a deterministic order.
• #796271 on tcsh by Reiner Herrmann: sort with LC_ALL set to C.
• #796275 on hspell by Reiner Herrmann: remove timestamp from aff files generated by mk_he_affix.
• #796324 on fftw3 by Reiner Herrmann: remove date from documentation files.
• #796335 on nasm by Val Lorentz: remove extra timestamps from the build system.
• #796360 on libical by Chris Lamb: removes randomess caused Perl in generated icalderivedvalue.c.
• #796375 on wcd by Dhole: set the date in the man pages to the latest debian/changelog entry.
• #796376 on mapivi by Dhole: set the date in the man pages to the latest debian/changelog entry.
• #796527 on vserver-debiantools by Dhole: set the date in the man pages to the latest debian/changelog entry.

St phane Glondu reported two issues regarding embedded build date in omake and cduce. Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar. Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found. reproducible.debian.net Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays. armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger) The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger) mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger) Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output. Connections to UDD have been made more robust. (Mattia Rizzolo) diffoscope development diffoscope version 31 was released on August 21^st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep. New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo). jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb. strip-nondeterminism development Andrew Ayer released strip-nondeterminism version 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added. Testing directory ordering issues: disorderfs During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random. Documentation update Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C. Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it. Package reviews 44 reviews have been removed, 192 added and 77 updated this week. New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex. 117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni. Misc. Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source! Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.

On 22nd of October 2004 an event called OS04 took place in Seifenfabrik Graz/Austria and it marked the first official release of the Grml project. Grml was initially started by myself in 2003 I registered the domain on September 16, 2003 (so technically it would be 11 years already :)). It started with a boot-disk, first created by hand and then based on yard. On 4th of October 2004 we had a first presentation of grml 0.09 Codename Bughunter at Kunstlabor in Graz. I managed to talk a good friend and fellow student Martin Hecher into joining me. Soon after Michael Gebetsroither and Andreas Gredler joined and throughout the upcoming years further team members (Nico Golde, Daniel K. Gebhart, Mario Lang, Gerfried Fuchs, Matthias Kopfermann, Wolfgang Scheicher, Julius Plenz, Tobias Klauser, Marcel Wichern, Alexander Wirt, Timo Boettcher, Ulrich Dangel, Frank Terbeck, Alexander Steinb ck, Christian Hofstaedtler) and contributors (Hermann Thomas, Andreas Krennmair, Sven Guckes, Jogi Hofm ller, Moritz Augsburger, ) joined our efforts. Back in those days most efforts went into hardware detection, loading and setting up the according drivers and configurations, packaging software and fighting bugs with lots of reboots (working on our custom /linuxrc for the initrd wasn t always fun). Throughout the years virtualization became more broadly available, which is especially great for most of the testing you need to do when working on your own (meta) distribution. Once upon a time udev became available and solved most of the hardware detection issues for us. Nowadays X.org doesn t even need a xorg.conf file anymore (at least by default). We have to acknowledge that Linux grew up over the years quite a bit (and I m wondering how we ll look back at the systemd discussions in a few years). By having Debian Developers within the team we managed to push quite some work of us back to Debian (the distribution Grml was and still is based on), years before the Debian Derivatives initiative appeared. We never stopped contributing to Debian though and we also still benefit from the Debian Derivatives initiative, like sharing issues and ideas on DebConf meetings. On 28th of May 2009 I myself became an official Debian Developer. Over the years we moved from private self-hosted infrastructure to company-sponsored systems, migrated from Subversion (brr) to Mercurial (2006) to Git (2008). Our Zsh-related work became widely known as grml-zshrc. jenkins.grml.org managed to become a continuous integration/deployment/delivery home e.g. for the dpkg, fai, initramfs-tools, screen and zsh Debian packages. The underlying software for creating Debian packages in a CI/CD way became its own project known as jenkins-debian-glue in August 2011. In 2006 I started grml-debootstrap, which grew into a reliable method for installing plain Debian (nowadays even supporting installation as VM, and one of my customers does tens of deployments per day with grml-debootstrap in a fully automated fashion). So one of the biggest achievements of Grml is from my point of view that it managed to grow several active and successful sub-projects under its umbrella. Nowadays the Grml team consists of 3 Debian Developers Alexander Wirt (formorer), Evgeni Golov (Zhenech) and myself. We couldn t talk Frank Terbeck (ft) into becoming a DM/DD (yet?), but he s an active part of our Grml team nonetheless and does a terrific job with maintaining grml-zshrc as well as helping out in Debian s Zsh packaging (and being a Zsh upstream committer at the same time makes all of that even better :)). My personal conclusion for 10 years of Grml? Back in the days when I was a student Grml was my main personal pet and hobby. Grml grew into an open source project which wasn t known just in Graz/Austria, but especially throughout the German system administration scene. Since 2008 I m working self-employed and mainly working on open source stuff, so I m kind of living a dream, which I didn t even have when I started with Grml in 2003. Nowadays with running my own business and having my own family it s getting harder for me to consider it still a hobby though, instead it s more integrated and part of my business which I personally consider both good and bad at the same time (for various reasons). Thanks so much to anyone of you, who was (and possibly still is) part of the Grml journey! Let s hope for another 10 successful years! Thanks to Max Amanshauser and Christian Hofstaedtler for reading drafts of this.

This weekend, Bernd Zeimetz organized a BSP at the offices of conova in Salzburg, Austria. Three days of discussions, bugfixes, sparc removals and a lot of fun and laughter. We squashed a total of 87 bugs: 66 bugs affecting Jessie/Sid were closed, 9 downgraded and 8 closed via removals. As people tend to care about (old)stable, 3 bugs were fixed in Wheezy and one in Squeeze. These numbers might be not totaly correct, as were kinda creative at counting Marga promised a talk about an introduction to properly counting bugs using the Haus vom Nikolaus algorithm to the base of 7 . Speaking of numbers, I touched the following bugs (not all RC):

• #741806: pygresql: FTBFS: pgmodule.c:32:22: fatal error: postgres.h: No such file or directory
  Uploaded an NMU with a patch. The bug was introduced by the recent PostgreSQL development package reorganisation.
• #744229: qpdfview: FTBFS synctex/synctex_parser.c:275:20: fatal error: zlib.h: No such file or directory
  Talked to the maintainer, explaining the importance of the upload and verifying his fix.
• #744300: pexpect: missing dependency on dh-python
  Downgraded to wishlist after verifying the build dependency is only needed when building for Wheezy backports.
• #744917: luajit: FTBFS when /sbin is not in $PATH
  Uploaded an NMU with a patch, which later was canceled due to a maintainer upload with a slightly different fix.
• #742943: nagios-plugins-contrib: check_raid: wants mpt-statusd / mptctl
  Analyzed the situation, verified the status with the latest upstream version of the ckeck and commented on the bug.
• #732110: nagios-plugins-contrib: check_rbl error when nameserver available only in IPv6
  Verify that the bug is fixed in the latest release and mark it as done.
• #684726: nagios-plugins-contrib: RFP: check-v46 Icinga / Nagios plugin for dual stacked (IPv4 / IPv6) hosts
  Mark bug as done, the changelog was missing a proper Closes tag.
• #661167: nagios-plugins-contrib: please include nagios-check-printer-status
  Mark bug as done, the changelog was missing a proper Closes tag.
• #745895: nagios-plugins-contrib: does not compile against Varnish 4.0.0
  Write a patch for supporting the Varnish 3 and 4 APIs at the same time. Also proposed the patch upstream.
• #744922: nagios-plugins-contrib: check_packages: check for security updates broken
  Forward our security_updates_critical patch and Felix fixes to it upstream, then updating check_packages to the latest upstream version.
• #744248: nagios-plugins-contrib: check_cert_expire: support configurable warn/crit times
  Forward Helmut s patch upstream, then updating check_cert_expire to the latest upstream version.
• #745691: django-classy-tags: FTBFS: Sphinx documentation not found
  Analyze the issue and the fix proposed in SVN, comment on the bug.
• #713876: thinkfan: [PATCH] bugfix: use $MAINPID for ExecReload
  Prepare and upload of the latest upstream release, which includes Michael s patch.
• #713878: thinkfan: [PATCH] use dh-systemd for proper systemd-related maintscripts
  Apply Michael s patch to the Debian packaging.
• #728087: thinkfan: Document how to start thinkfan with systemd
  Apply Michael s patch to the Debian packaging.
• #742515: blktap-dkms: blktapblktap kernel module failed to build
  Upload an NMU with a patch based on the upstream fix.
• #745598: libkolab: FTBFS in dh_python2 (missing Build-Conflicts?)
  Upload an NMU with a patch against libkolab s cmake rules, tightening the search for Python to 2.7.
• #745599: libkolabxml: FTBFS with undefined reference to symbol _ZTVN5boost6detail16thread_data_baseE
  Upload an NMU with a patch against libkolabxml s cmake rules, properly linking the tests to the Boost libraries.
• #746160: libcolabxml: FTBFS when both python2 and python3 development headers are installed
  Filling the bug while working on #745599, then uploading an NMU with a patch against libkolabxml s cmake rules, tightening the search for Python to 2.7.
• #714045: blcr-dkms: blcr module is not built on kernel 3.9.1
  Checking the status of the bug upstream, and marking it as forwarded.
• #653404: hdapsd: init.d status support
  Added Peter s patch to the Debian packaging, upload yet pending.
• #702199: hdapsd: Typo in package description
  Fixed typo in the description, upload yet pending.
• #745219: crmsh: should depends on python-yaml
  Verifying the bug with Stefan Bauer and sponsoring his NMU.
• #741600: 389-ds-base: CVE-2014-0132
  Sponsoring the NMU for Tobias Frost.

A couple of (non-free) pictures are available at Uwe s salzburg-cityguide.at. Thanks again to Bernd for organizing and conova and credativ for sponsoring!

0 comment(s) this blog is flattr enabled

Let s assume you are a sysadmin and have to debug a daemon giving bad performance on one machine, but not on the other. Of course, you did not setup either machine, have only basic knowledge of the said daemon and would really love to watch that awesome piece of cinematographic art with a bunch of friends and a couple of beers. So it s like every day, right? The problem with understanding running setups is that you often have to read configuration files. And when reading one is not enough, you have to compare two or more of them. Suddenly, a wild problem occurs: order and indentation do not matter (unless they do), comments are often just beautiful noise and why the hell did that guy smoke/drink/eat while explicitly setting ALL THE OPTIONS to their defaults before actually setting them as he wanted. If you are using diff(1), you probably love to read a lot of differences, which are none in reality. Want an example? [foo]
bar = bar
foo = foo and # settings for foo
[foo]
# foo is best
foo = foo
# bar is ok here, FIXME?
bar = bar and [foo]
foo = x
bar = x
[foo]
foo = foo
bar = bar are actually the same, at least for some parsers. XTaran suggested using something like wdiff or dwdiff, which often helps, but not in the above case. Others suggested vimdiff, which is nice, but not really helpful here either. As there is a problem, and I love to solve these, I started a small new project: cfgdiff. It tries to parse two given files and give a diff of the content after normalizing it (merging duplicate keys, sorting keys, ignoring comments and blank lines, you name it). Currently it can parse various INI files, JSON, YAML and XML. That s probably not enough to be the single diff tool for configuration files, but it is quite a nice start. And you can extend it, of course ;)

0 comment(s) this blog is flattr enabled

When you run Puppet, it is very important to monitor whether all nodes have an uptodate catalog and did not miss the last year of changes because of a typo in a manifest or a broken cron-script. The most common solution to this is a script that checks /var/lib/puppet/state/last_run_summary.yaml on each node. While this is nice and easy in a small setup, it can get a bit messy in a bigger environment as you have to do an NRPE call for every node (or integrate the check as a local check into check_mk). Given a slightly bigger Puppet environment, I guess you already have PuppetDB running. Bonuspoints if you already let it save the reports of the nodes via reports = store,puppetdb. Given a central knowledgebase about your Puppet environment one could ask PuppetDB about the last node runs, right? I did not find any such script on the web, so I wrote my own: check_puppetdb_nodes. The script requires a recent (1.5) PuppetDB and a couple of Perl modules (JSON, LWP, Date::Parse, Nagios::Plugin) installed. When run, the script will contact the PuppetDB via HTTP on localhost:8080 (obviously configurable via -H and -p, HTTPS is available via -s) and ask for a list of nodes from the /nodes endpoint of the API. PuppetDB will answer with a list of all nodes, their catalog timestamps and whether the node is deactivated. Based on this result, check_puppetdb_nodes will check the last catalog run of all not deactivated nodes and issue a WARNING notification if there was none in the last 2 hours (-w) or a CRITICAL notification if there was none for 24 hours (-c). As a fresh catalog does not mean that the node was able to apply it, check_puppetdb_nodes will also query the /event-counts endpoint for each node and verify that the node did not report any failures in the last run (for this feature to work, you need reports stored in PuppetDB). You can modify the thresholds for the number of failures that trigger a WARNING/CRITICAL with -W and -C, but I think 1 is quite a reasonable default for a CRITICAL in this case. Using check_puppetdb_nodes you can monitor the health of ALL your Puppet nodes with a singe NRPE call. Or even with zero, if your monitoring host can access PuppetDB directly.

0 comment(s) this blog is flattr enabled

Dear all, we are proud to announce the 3.0 release of the Shogun Machine-Learning Toolbox. This release features the incredible projects of our 8 hard-working Google Summer of Code students. In addition, you get other cool new features as well as lots of internal improvements, bugfixes, and documentation improvements. To speak in numbers, we got more than 2000 commits changing almost 400000 lines in more than 7000 files and increased the number of unit tests from 50 to 600. This is the largest release that Shogun ever had! Please visit http://shogun-toolbox.org/ to obtain Shogun. News Here is a brief description of what is new, starting with the GSoC projects, which deserve most fame:

• Gaussian Process classification by Roman Votjakov
• Structured Output Learning of graph models by Shell Hu
• Estimators for log-determinants of large sparse matrices by Soumyajit De
• Feature Hashing and random kitchen sinks by Evangelos Anagnostopoulos
• Independent Component Analysis by Kevin Hughes
• A web-based demo framework by Liu Zhengyang
• Metric learning with large margin nearest neighbours by Fernando Iglesias
• Native support for various popular file formats by Evgeniy Andreev

Screenshots Everyone likes screenshots. Well, we have got something better! All of the above projects (and more) are now documented in the form of IPython notebooks, combining machine learning fundamentals, code, and plots. Those are a great looking way that we chose to document our framework from now on. Have a look at them and feel free to submit your use case as a notebook! The web-demo framework has been integrated into our website, go check them out. Other changes We finally moved to the Shogun build process to CMake. Through GSoC, added a general clone and equals methods to all Shogun objects, and added automagic unit-testing for serialisation and clone/equals for all classes. Other new features include multiclass LDA, and probability outputs for multiclass SVMs. For the full list, see the NEWS. Workshop Videos and slides In case you missed the first Shogun workshop that we organised in Berlin last July, all of the talks have been put online. Shogun in the Cloud As setting up the right environment for shogun and installing it was always one of the biggest problems for the users (hence the switching to CMake), we have created a sandbox where you can try out shogun on your own without installing shogun on your system! Basically it's a web-service which give you access to your own ipython notebook server with all the shogun notebooks. Of course you are more than welcome to create and share your own notebooks using this service! *NOTE*: This is a courtesy service created by Shogun Toolbox developers, hence if you like it please consider some form of donation to the project so that we can keep up this service running for you. Try shogun in the cloud. Thanks The release has been made possible by the hard work of all of our GSoC students, see list above. Thanks also to Thoralf Klein and Bj rn Esser for the load of great contributions. Last but not least, thanks to all the people who use Shogun and provide feedback. S ren Sonnenburg on behalf of the Shogun team (+ Viktor Gal, Sergey Lisitsyn, Heiko Strathmann and Fernando Iglesias)

Some days ago I got myself a new shiny Samsung 840 Pro 256GB SSD for my laptop. The old 80GB Intel was just too damn small. Instead of just doing a pvmove from the old to the new, I decided to set up the system from scratch. That is an awesome way to get rid of old and unused stuff or at least move it to some lower class storage (read: backup). One of the things I did not bother to copy from the old disk were my ~/Debian, ~/Grml and ~/Devel folders. I mean, hey, it s all in some kind of VCS, right? I can just clone it new, if I really want. Neither I copied much of my dotfiles, these are neatly gitted with the help of RichiH s awesome vcsh and a bit of human brains (no private keys on GitHub, yada yada). After cloning a couple of my personal repos from GitHub to ~/Devel, I realized I was doing a pretty dumb job, a machine could do for me. As I already was using Joey s mr for my vcsh repositories, generating a mr config and letting mr do the actual job was the most natural thing to do. So was using Python Requests and GitHub s JSON API. And here is Mister Hubert, aka mrhub: github.com/evgeni/MisterHubert. Just call it with your GitHub username and you get a nice mr config dumped to stdout. Same applies for organizations.

• Authentication for private repos? (-p)
• Other clone mechanisms? (-c)
• A help function? (-h)
• Other features?

As usual, I hope this is useful :)

0 comment(s) this blog is flattr enabled

Sorry Bryan,
I can show you plenty of hardware that is perfectly 64 bit capable but probably never will run Ubuntu and/or Unity. First, what is 64 bit for you? Looking at ubuntu.com/download and getting images from there, one gets the impression, that 64 bit is amd64 (also called x86_64). If one digs deeper to cdimage.ubuntu.com, one will find non-Intel images too: PowerPC and amrhf. As the PowerPC images are said to boot on G3 and G4 PowerPCs, these are 32 bit. Armhf is 32 bit too (arm64/aarch64 support in Linux is just evolving). So yes, if 64 bit means amd64, I do have hardware that can run Unity. But you asked if I have hardware that is 64 bit capable and can run Ubuntu/Unity, so may I apply my definiton of 64 bit here? I have an old Sun Netra T1-200 (500MHz UltraSPARC IIe) running Debian s sparc port, which has a 64 bit kernel and 32 bit userland. Unity? No wai. I do not own any ia64 or s390/s390x machines, but I am sure people do. And guess what, no Unity there either :) Sorry for ranting like this, but 64 bit really just means that the CPU can handle 64 bit big addresses etc. End even then, it not always will do so ;)

0 comment(s) this blog is flattr enabled

You may not know this, but I am a huge PowerDNS fan. This may be because it is so simple to use, supports different databases as backends or maybe just because I do not like BIND, pick one. I also happen to live in Germany where ISPs usually do not give static IP-addresses to private customers. Unless you pay extra or limit yourself to a bunch of providers that do good service but rely on old (DSL) technology, limiting you to some 16MBit/s down and 1MBit/s up. Luckily my ISP does not force the IP-address change, but it does happen from time to time (once in a couple of month usually). To access the machine(s) at home while on a non-IPv6-capable connection, I have been using my old (old, old, old) DynDNS.com account and pointing a CNAME from under die-welt.net to it. Some time ago, DynDNS.com started supporting AAAA records in their zones and I was happy: no need to type hostname.ipv6.kerker.die-welt.net to connect via v6 just let the application decide. Well, yes, almost. It s just DynDNS.com resets the AAAA record when you update the A record with ddclient and there is currently no IPv6 support in any of the DynDNS.com clients for Linux. So I end up with no AAAA record and am not as happy as I should be. Last Friday I got a mail from DynDNS:

Starting now, if you would like to maintain your free Dyn account, you must now log into your account once a month. Failure to do so will result in expiration and loss of your hostname. Note that using an update client will no longer suffice for this monthly login. You will still continue to get email alerts every 30 days if your email address is current.
Yes, thank you very much

Given that I have enough nameservers under my control and love hacking, I started writing an own dynamic DNS service. Actually you cannot call it a service. Or dynamic. But it s my own, and it does DNS: powerdyn. It is actually just a script, that can update DNS records in SQL (from which PowerDNS serves the zones). When you design such a service , you first think about user authentication and proper information transport. The machine that runs my PowerDNS database is reachable via SSH, so let s use SSH for that. You do not only get user authentication, server authentication and properly crypted data transport, you also do not have to try hard to find out the IP-address you want to update the hostname to, just use $SSH_CLIENT from your environment. If you expected further explanation what has to be done next: sorry, we re done. We have the user (or hostname) by looking at the SSH credentials, and we have the IP-address to update it to if the data in the database is outdated. The only thing missing is some execution daemon or cron(8). :) The machine at home has the following cron entry now:

*/5 * * * * ssh -4 -T -i /home/evgeni/.ssh/powerdyn_rsa powerdyn@ssh.die-welt.net

This connects to the machine with the database via v4 (my IPv6 address does not change) and that s all.
As an alternative, one can add the ssh call in /etc/network/if-up.d/, /etc/ppp/ip-up.d/ or /etc/ppp/ipv6-up.d (depending on your setup) to be executed every time the connection goes up. The machine with the database has the following authorized_keys entry for the powerdyn user:

no-agent-forwarding,no-port-forwarding,no-pty,no-X11-forwarding,no-user-rc,\ 
command="/home/powerdyn/powerdyn/powerdyn dorei.kerker.die-welt.net" ssh-rsa AAAA... evgeni@dorei

By forcing the command, the user has no way to get the database-credentials the script uses to write to the database and neither cannot update a different host. That seems secure enough for me. It won t scale for a setup as DynDNS.com and the user-management sucks (you even have to create the entries in the database first, the script can only update them), but it works fine for me and I bet it would for others too :) Update: included suggestions by XX and Helmut from the comments.

3 comment(s) this blog is flattr enabled

TL;DR: fqdn, "jabber.die-welt.net" . So, how many servers do you have, that are still running Squeeze? I count one, mostly because I did not figure out a proper upgrade path from OpenVZ to something else yet, but this is a different story. This post is about the upgrade of my communication machine, dengon.die-welt.net. It runs my private XMPP and IRC servers. I upgraded it to Wheezy, checked that my irssi and my BitlBee still could connect and left for work. There I noticed, that Pidgin could only connect to one of the two XMPP accounts I have on that server. sargentd@jabber.die-welt.net worked just fine, while evgeni@golov.de failed to connect. ejabberd was logging a failed authentication:
I(<0.1604.0>:ejabberd_c2s:802) : ( socket_state,tls, tlssock,#Port<0.5130>,#Port<0.5132> ,<0.1603.0> ) Failed authentication for evgeni@golov.de

While Pidgin was just throwing Not authorized errors. I checked the password in Pidgin (even if it did not change). I tried different (new) accounts: anything@jabber.die-welt.net worked, nothing@golov.de did not and somethingdifferent@jabber.<censored>.de worked too. So where was the difference between the three vhosts? jabber.die-welt.net and jabber.<censored>.de point directly (A/CNAME) to dengon.die-welt.net. golov.de has SRV records for XMPP pointing to jabber.die-welt.net. Let s ask Google about ejabberd pidgin srv . There are some bugs. But they are marked as fixed in Wheezy. Mhh Let s read again Okay, I have to set fqdn, "<my_srv_record_name>" . when this does not match my hostname. Edit /etc/ejabberd/ejabberd.cfg, add fqdn, "jabber.die-welt.net" . (do not forget the dot at the end) and restart the ejabberd. Pidgin can connect again. Yeah.

0 comment(s) this blog is flattr enabled

So you probably heard that I have that little new project of mine: QiFi the pure JavaScript WiFi QR Code Generator. It s been running pretty well and people even seem to like it. One of its (unannounced) features is a pretty clean stylesheet that is used for printing. When you print the result will be just the SSID and the QR code, so you can put that piece of paper everywhere you like. That works (I tested!) fine on Iceweasel/Firefox 10.0.12 and Chromium 25.0. Today I tried to do the same in Opera 12.14 and it failed terribly: the SSID was there, the QR code not. And here my journey begins First I suspected the CSS I used was fishy, so I kicked all the CSS involved and retried: still no QR code in the print-out. So maybe it s the QR code library I use that produces a weird canvas? Nope, the examples on http://diveintohtml5.info/canvas.html and http://devfiles.myopera.com/articles/649/example5.html don t print either. Uhm, let s Google for opera canvas print And oh boy I should not have done that. It seems it s a bug in Opera. And the proposed solution is to use canvas.toDataURL() to render the canvas as an image and load the image instead of the canvas. I almost went that way. But I felt that urge need to read the docs before. So I opened http://www.w3.org/html/wg/drafts/html/master/embedded-content-0.html#dom-canvas-todataurl and https://developer.mozilla.org/en-US/docs/DOM/HTMLCanvasElement and started puking:

When trying to use types other than image/png , authors can check if the image was really returned in the requested format by checking to see if the returned string starts with one of the exact strings data:image/png, or data:image/png; . If it does, the image is PNG, and thus the requested type was not supported. (The one exception to this is if the canvas has either no height or no width, in which case the result might simply be data:, .)

If the type requested is not image/png, and the returned value starts with data:image/png, then the requested type is not supported.

Really? I have to check the returned STRING to know if there was an error? Go home HTML5, you re drunk! Okay, okay. No canvas rendered to images then. Let s just render the QR code as a <table> instead of a <canvas> when the browser looks like Opera. There is nothing one could do wrong with tables, right? But let s test with the basic example first: Yes, this is 2013. Yes, this is Opera 12.14. Yes, the rendering of a fucking HTML table is wrong. Needles to say, Iceweasel and Chromium render the example just fine. I bet even a recent Internet Explorer would That said, there is no bugfixworkaround for Opera I want to implement. If you use Opera, I feel sorry for you. But that s all. Update: before someone cries ZOMG! BUG PLZ!!! , I filled this as DSK-383716 at Opera.

0 comment(s) this blog is flattr enabled

Some time ago, the QR Code Generator WiFi Access made quite some noise on the mighty Internet. Sure, it is cool to be able to share your WiFi-access with someone by just showing him a QR code he can scan on his phone and the phone will auto-connect to the WiFi. But I get a strange feeling telling someone I do not know my WiFi credentials. No, I do not mean my guests, I know them. I mean that shiny web-service that will generate a QR code for me. The geek in you will now say: So? Open up a terminal, install qrencode, pipe it the string WIFI:S:<SSID>;T:<WPA WEP >;P:<password>;; and you got our QR code . Yeah, that works. But was it one or two semicolons at the end? And was it really just WPA even if my WiFi uses WPA2? Oh and how do I encode that umlaut again? I do not want to remember this. Thus, without too much rumble, may I present you: QiFi the pure JS WiFi QR Code Generator. QiFi is a QR code generator for WiFi access in pure JavaScript. It will generate the QR code on your machine, in your browser, not leaking your precious credentials to anyone (but your guests). Don t trust me? Read the code. Fork the code. Host the code yourself. I hope you will find QiFi at least slightly useful ;-)

2 comment(s) this blog is flattr enabled

The summer came finally to an end and (yes in Berlin we still had 20 C end of October), unfortunately, so did GSoC with it. This has been the second time for SHOGUN to be in GSoC. For those unfamiliar with SHOGUN - it is a very versatile machine learning toolbox that enables unified large-scale learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis. I again played the role of an org admin and co-mentor this year and would like to take the opportunity to summarize enhancements to the toolbox and my GSoC experience: In contrast to last year, we required code-contributions in the application phase of GSoC already, i.e., a (small) patch was mandatory for your application to be considered. This reduced the number of applications we received: 48 proposals from 38 students instead of 70 proposals from about 60 students last year but also increased the overall quality of the applications. In the end we were very happy to get 8 very talented students and have the opportunity of boosting the project thanks to their hard and awesome work. Thanks to google for sponsoring three more students compared to last GSoC. Still we gave one slot back to the pool for good to the octave project (They used it very wisely and octave will have a just-in-time compiler now, which will benefit us all!). SHOGUN 2.0.0 is the new release of the toolbox including of course all the new features that the students have implemented in their projects. On the one hand, modules that were already in SHOGUN have been extended or improved. For example, Jacob Walker has implemented Gaussian Processes (GPs) improving the usability of SHOGUN for regression problems. A framework for multiclass learning by Chiyuan Zhang including state-of-the-art methods in this area such as Error-Correcting Output Coding (ECOC) and ShareBoost, among others. In addition, Evgeniy Andreev has made very important improvements w.r.t. the accessibility of SHOGUN. Thanks to his work with SWIG director classes, now it is possible to use python for prototyping and make use of that code with the same flexibility as if it had been written in the C++ core of the project. On the other hand, completely new frameworks and other functionalities have been added to the project as well. This is the case of multitask learning and domain adaptation algorithms written by Sergey Lisitsyn and the kernel two-sample or dependence test by Heiko Strathmann. Viktor Gal has introduced latent SVMs to SHOGUN and, finally, two students have worked in the new structured output learning framework. Fernando Iglesias made the design of this framework introducing the structured output machines into SHOGUN while Michal Uricar has implemented several bundle methods to solve the optimization problem of the structured output SVM. It has been very fun and interesting how the work done in different projects has been put together very early, even during the GSoC period. Only to show an example of this dealing with the generic structured output framework and the improvements in the accessibility. It is possible to make use of the SWIG directors to implement the application specific mechanisms of a structured learning problem instance in python and then use the rest of the framework (written in C++) to solve this new problem. Students! You all did a great job and I am more than amazed what you all have achieved. Thank you very much and I hope some of you will stick around. Besides all these improvements it has been particularly challenging for me as org admin to scale the project. While I could still be deeply involved in each and every part of the project last GSoC, this was no longer possible this year. Learning to trust that your mentors are doing the job is something that didn't come easy to me. Having had about monthly all-hands meetings did help and so did monitoring the happiness of the students. I am glad that it all worked out nicely this year too. Again, I would like to mention that SHOGUN improved a lot code-base/code-quality wise. Students gave very constructive feedback about our (lack) of proper Vector/Matrix/String/Sparse Matrix types. We now have all these implemented doing automagic memory garbage collection behind scenes. We have started to transition to use Eigen3 as our matrix library of choice, which made quite a number of algorithms much easier to implement. We generalized the Label framework (CLabels) to be tractable for not just classification and regression but multitask and structured output learning. Finally, we have had quite a number of infrastructure improvements. Thanks to GSoC money we have a dedicated server for running the buildbot/buildslaves and website. The ML Group at TU Berlin does sponsor virtual machines for building SHOGUN on Debian and Cygwin. Viktor Gal stepped up providing buildslaves for Ubuntu and FreeBSD. Gunnar Raetschs group is supporting redhat based build tests. We have Travis CI running testing pull requests for breakage even before merges. Code quality is now monitored utilizing LLVMs scan-build. Bernard Hernandez appeared and wrote a fancy new website for SHOGUN. A more detailed description of the achievements of each of the students follows:

• Kernel Two-sample/Dependence test
  • Student: Heiko Strathmann
  • Mentor: Arthur Gretton, Soeren Sonnenburg
  Heiko Strathmann, mentored by Arthur Gretton, worked on a framework for kernel-based statistical hypothesis testing. Statistical tests to determine whether two random variables are are equal/different or are statistically independent are an important tool in data-analysis. However, when data are high-dimensional or in non-numerical form (documents, graphs), classical methods fail. Heiko implemented recently developed kernel-based generalisations of classical tests which overcome these issues by representing distributions in high dimensional so-called reproducing kernel Hilbert spaces. By doing so, theoretically any pair samples can be distinguished. Implemented methods include two-sample testing with the Maximum Mean Discrepancy (MMD) and independence testing using the Hilbert Schmidt Independence Criterion (HSIC). Both methods come in different flavours regarding computational costs and test constructions. For two-sample testing with the MMD, a linear time streaming version is included that can handle arbitrary amounts of data. All methods are integrated into a newly written flexible framework for statistical testing which will be extended in the future. A book-style tutorial with descriptions of algorithms and instructions how to use them is also included.
• Implement multitask and domain adaptation algorithms
  • Student: Sergey Lisityn
  • Mentor: Christian Widmer
  Like Heiko, Sergey Lisitsyn did participate in the GSoC programme for the second time. This year he focused on implementing multitask learning algorithms. Multitask learning is a modern approach to machine learning that learns a problem together with other related problems at the same time using a shared representation. This approach often leads to a better model for the main task, because it allows the learner to use the commonality among the tasks. During the summer Sergey has ported a few algorithms from the SLEP and MALSAR libraries with further extensions and improvements. Namely, L12 group tree and L1q group multitask logistic regression and least squares regression, trace-norm multitask logistic regression, clustered multitask logistic regression, basic group and group tree lasso logistic regression. All the implemented algorithms use COFFIN framework for flexible and efficient learning and some of the algorithms were implemented efficiently utilizing the Eigen3 library.
• Implementation of / Integration via existing GPL code of latent SVMs.
  • Student: Viktor Gal
  • Mentor: Alexander Binder
  A generic latent SVM and additionally a latent structured output SVM has been implemented. This machine learning algorithm is widely used in computer vision, namely in object detection. Other useful application fields are: motif finding in DNA sequences, noun phrase coreference, i.e. provide a clustering of the nouns such that each cluster refers to a single object. It is based on defining a general latent feature Psi(x,y,h) depending on input variable x, output variable y and latent variable h. Deriving a class from the base class allows the user to implement additional structural knowledge for efficient maximization of the latent variable or alternative ways of computation or on-the-fly loading of latent features Psi as a function of the input, output and latent variables.
• Bundle method solver for structured output learning
  • Student: Michal Uricar
  • Mentor: Vojtech Franc
  We have implemented two generic solvers for supervised learning of structured output (SO) classifiers. First, we implemented the current state-of-the-art Bundle Method for Regularized Risk Minimization (BMRM) [Teo et al. 2010]. Second, we implemented a novel variant of the classical Bundle Method (BM) [Lamarechal 1978] which achieves the same precise solution as the BMRM but in time up to an order of magnitude shorter [Uricar et al. 2012]. Among the main practical benefits of the implemented solvers belong their modularity and proven convergence guarantees. For training a particular SO classifier it suffices to provide the solver with a routine evaluating the application specific risk function. This feature is invaluable for designers who can concentrate on tuning the classification model instead of spending time on developing new optimization solvers. The convergence guarantees remove the uncertainty inherent in use of on-line approximate solvers. The implemented solvers have been integrated to the structured output framework of the SHOGUN toolbox and they have been tested on real-life data.
• Built generic structured output learning framework
  • Student: Fernando Jose Iglesias Garcia
  • Mentor: Nico Goernitz
  During GSoC 2012 Fernando implemented a generic framework for structured output (SO) problems. Structured output learning deals with problems where the prediction made is represented by an object with complex structure, e.g. a graph, a tree or a sequence. SHOGUN's SO framework is flexible and easy to extend [1]. Fernando implemented a na ve cutting plane algorithm for the SVM approach to SO [2]. In addition, he coded a case of use of the framework for labelled sequence learning, the so-called Hidden Markov SVM. The HM-SVM can be applied to solve problems in different fields such as gene prediction in bioinformatics or speech to text in pattern recognition. [1] Class diagram of the SO framework.
  [2] Support Vector Machine Learning for Interdependent and Structured Output Spaces.
• Improving accessibility to shogun
  • Student: Evgeniy Andreev
  • Mentor: Soeren Sonnenburg
  During the latest google summer of code Evgeniy has improved the Python modular interface. He has added new SWIG-based feature - director classes, enabling users to extend SHOGUN classes with python code and made SHOGUN python 3 ready. Evgeniy has also added python's protocols for most usable arrays (like vectors, matrices, features) which makes possible to work with Shogun data structures just like with numpy arrays with no copy at all. For example one can now modify SHOGUN's RealFeatures in place or use.
• Implement Gaussian Processes and regression techniques
  • Student: Jacob Walker
  • Mentor: Oliver Stegle
  Jacob implemented Gaussian Process Regression in the Shogun Machine Learning Toolbox. He wrote a complete implementation of basic GPR as well as approximation methods such as the FITC (First Independent Training Conditional) method and the Laplacian approximation method. Users can utilize this feature to analyze large scale data in a variety of fields. Scientists can also build on this implementation to conduct research extending the capability and applicability of Gaussian Process Regression.
• Build generic multiclass learning framework
  • Student: Chiyuan Zhang
  • Mentor: Cheng Soon Ong
  Chiyuan defined a new multiclass learning framework within SHOGUN. He re-organized previous multiclass learning components and refactored the CMachine hierarchy in SHOGUN. Then he generalized the existing one-vs-one and one-vs-rest multiclass learning scheme to general ECOC encoding and decoding strategies. Beside this, several specific multiclass learning algorithms are added, including ShareBoost with feature selection ability, Conditional Probability Tree with online learning ability, and Relaxed Tree.