Search Results: "mez"

21 March 2016

Lunar: Reproducible builds: week 47 in Stretch cycle

What happened in the reproducible builds effort between March 13th and March 19th 2016:

Toolchain fixes
  • Petter Reinholdtsen uploaded naturaldocs/1.51-1.1 which makes the output reproducible. Original patch by Chris Lamb.
  • Damyan Ivanov uploaded libpdf-api2-perl/2.025-2 which will make internal font ID reproducible.
  • Christian Hofstaedtler uploaded ruby2.3/2.3.0-5 which sets gzip embedded mtime field to fixed value for rdoc-generated compressed javascript data.

Packages fixed The following packages have become reproducible due to changes in their build dependencies: diction, doublecmd, ruby-hiredis, vdr-plugin-epgsearch. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet:
  • #818128 on nethack by Reiner Herrmann: implement support for SOURCE_DATE_EPOCH, set LC_ALL to C, and ensure deterministic build order when running parallel builds.
  • #818111 on debian-keyring by Satyam Zode: fix the order of files in md5sums.
  • #818067 on ncurses by Niels Thykier: strip trailing whitespaces introduced when using dash as system shell.
  • #818230 on aircrack-ng by Reiner Herrmann: build assembly code as a separate .o file.
  • #818419 on mutt by Daniel Shahaf: use C locale when listing files to be put in README.Patches.
  • #818430 on ruby-coveralls by Dhole: ensure UTC is used as the timezone when generating the documentation.
  • #818686 on littlewizard by Reiner Herrmann: use the C locale in the script for iterating over the files.
  • #818704 on strigi by Reiner Herrmann: sort keys when traversing hashes in makecode.pl.

Package reviews 44 reviews have been removed, 40 added and 5 updated in the previous week. Chris Lamb has reported 16 FTBFS.

7 March 2016

Laura Arjona Reina: More involved in the Debian Publicity team

More than 6months since I am Debian Developer and I m learning new things every day and trying to organize my time better to accomplish all the things I would like to do.
In September I became publicity delegate (and we joined Press+Publicity into the Publicity team). We continued discussing how the team works, and hopefully revitalizing it.
I became more active in the IRC channel, installed quassel in my home server and then I could read all the backlog too.
I ve tried to contribute more to Debian Project News (less than I d like), to bits.debian.org and to the social networks (Pump.io and GNU Social).
Since November I ve been failing to call for meeting but finally on 2016/02/29 we had it. I hope to resume semi-regular meetings each 1-2 months.
At the beginning of the year I helped to handle all the news and announcements about the loss of Ian Murdock. It has been hard work but I feel that I could turn some of the sadness into something useful for others. There is still work to be done: parse the condolence emails and setup a website to publish them.
About bits.debian.org, DPN and announcements, I ve learned how to perform the parts of the work that require membership permissions (building the blog, actually publishing the announcement, sending the DPN/announcement mail to the corresponding lists). Not without mistakes, but I guess no matter how much you care, sometimes things happen, and then you learn and then things get better and you too.
Some things I discovered Some clarifications
  • I ve written hard word 4 times in this post (well, 5 now!) but that does not mean I don t want to do those tasks. I feel going out of my comfort zone and that s ok, needed to learn and experience. I m having a good time in Debian in the last months, as always!
  • We re 4 people delegated and more team members and contributors, so anybody could think: Publicity is well covered, let s go to do other things in Debian . It s perfectly ok if you want to do other things in Debian, but please consider combining your contributions with some minutes for the publicity team. Aiming to be the universal operating system, our community and target audience is big and diverse and we d like to show that diversity to the world. The more we are, the merrier!
Comments?
You can comment about this blog post in this pump.io thread.

Filed under: My experiences and opinion Tagged: Communities, Contributing to libre software, Debian, Developer motivations, English, Free Software

4 March 2016

Enrico Zini: Praise of component reuse

I farm bits and pieces out to the guys who are much more brilliant than I am. I say, "build me a laser", this. "Design me a molecular analyzer", that. They do, and I just stick 'em together. (Seth Brundle, "The Fly")
When I decided to try and turn siterefactor into staticsite, I decided that I would go ahead only for as long as it could be done with minimal work, writing code in the most straightforward way on top of existing and stable components. I am pleased by how far that went. Python-Markdown It works fast enough, already comes with extensions for most of what I needed, and can be extended in several ways. One of the extension methods is a hook for manipulating the ElementTree of the rendered document before serializing it to HTML, which made it really easy to go and process internal links in all <a href= and <img src= attributes. To tell an internal link from an external link I just use the standard python urlparse and see if the link has a scheme or a netloc component. If it does not, and if it has a path, then it is an internal link. This also means that I do not need to invent new Markdown syntax for internal references, avoiding the need for remembering things like [text]( < relref "blog/post.md" > ) or [text]( filename /blog/post.md). In staticsite, it's just [text](/blog/post.md) or [text](post.md) if the post is nearby. This feels nicely clean to me: if I wanted to implement fancy markdown features, I could do it as Python-Markdown extensions and submit them upstream. If I wanted to implement fancy interlinking features, I could do it with a special url scheme in links. For example, it would be straigtforward to implement a ssite: url scheme that expanded the url with elements from staticsite's settings using a call to python's string.format (ssite: SETTING_NAME /bar maybe?), except I do not currently see any use cases for extending internal linking from what it is now. Jinja2 Jina2 is a template engine that I already knew, it is widely used, powerful and pleasant to use, both on the templating side and on the API's side. It is not HTML specific, so I can also use it to generate Atom, RSS2, "dynamic" site content, and even new site Markdown pages. Implementing RSS and Atom feeds was just a matter of writing and testing these Jinja2 macros and then reusing them anywhere. toml, yaml, json No need to implement my own front matter parsing. Also, reusing the same syntax as Hugo allows me to just link to its documentation. python-slugify I found python-slugify so I did not bother writing a slug-generating function. As a side effect, now things works better than I would even have thought to implement, including transliteration of non-ascii characters:
$ ./ssite new example --noedit --title "Cos  parl  Enrico"
/enrico-dev/staticsite/example/site/blog/2016/cosi-parlo-enrico.md
(I just filed an RFP) python-livereload Implementing ssite serve which monitors the file system and autoreloads when content changes and renders everything on the fly, took about an hour. Most of that hour went into implementing rendering pages on demand. Then I discovered that it autoreloads even when I edit staticsite's source code. Then I discovered that it communicates with the browser and even automatically triggers a page refresh. I can keep vim on half my screen and a browser in the other half, and I get live preview for free every time I save, without ever leaving the editor. Bootstrap I already use Bootstrap at work, so creating the default theme templates with it took about 10 minutes. This morning I tried looking at my website using my mobile phone, and I pleasantly saw it automatically turning into a working mobile version of itself. Pygments Python-Markdown uses Pygments for syntax highlighting, and it can be themed just by loading a .css. So, without me really doing anything, even staticsite's syntax highligthing is themable, and there's even a nice page with a list of themes to choose from. Everything else... Command line parsing? Straight argparse. Logging? python's logging support. Copying static resource files? shutil.copy2. Parsing dates? dateutil.parser. Timing execution? time.perf_counter. Timezone handling? pytz. Building the command to run an editor? string.format. Matching site pages? fnmatch.translate. ...and then some. If I ever decide to implement incremental rendering, how do I implement tracking which source files have changed? Well, for example, how about just asking git?

3 March 2016

Ben Hutchings: Last week 'flu by

Last Monday, I began a new project at Codethink - working on kernel support for the next generation of an interactive system fitted in some high-performance cars. I travelled up to Manchester the night before, and on Monday I met Codethink's team (most of whom I was familiar with). My first chore was to set up VPN access to the development resources (source control, wiki, etc.). I sandboxed the proprietary VPN client in a VM with a systemd unit to run it at boot, so I can control it by starting and stopping that VM. I then set to work on unpacking and exploring the SoC vendor's evaluation module (EVM), starting by looking at serial output - of which there was none. Nothing on the LCD panel or network port either. A frustrating day. On Tuesday, the architect from our immediate customer, already a friend of mine, joined us in Manchester. As for the EVM, without any obvious change in cabling it was now reliably spitting out the expected u-boot and kernel boot messages, unfortunately ending with errors from the touchscreen driver intermingled with messages from various graphics drivers. I looked at the u-boot environment to see how the kernel parameters were controlled and started tweaking them with the aim of working around the errors. None of this worked. I looked for more documentation on the EVM, but found none beyond the quick start guide. Had I been given the wrong kernel or device tree? They matched the available downloads though the device tree looked very different from the upstream version. I started trying to edit the vendor's device tree to remove the troublesome devices before realising that the decompiled tree was full of absolute cross-references that would be invalid after recompiling. So, a second day of frustration. In the evening I started to feel a cold coming on. I slept poorly, and just dragged myself out of bed in time for 11:00 checkout. In the office, I remembered that u-boot had commands for manipulating device trees in memory. I tried stepping through the boot script and then removing device tree nodes for the touchscreen and graphics hardware before booting the kernel. Success! I got a login prompt where of course username:root, password:[empty] worked. (It doesn't run sshd, but does run telnetd.) Then it was time for a team lunch at the excellent Samsi, though I hesitated to go as I was feeling quite poorly. Shortly after lunch I found myself falling asleep at my desk in the middle of reading documentation. I had to take a nap on the sofa. I got a little more work done that day, but I got progressively feebler. The train journey home was thankfully uneventful. I arrived home at midnight and soon went to sleep. I spent most of Thursday in a high fever, dreaming of abstract entities and their relationships that I struggled to name, but in retrospect might correspond to various kinds of software component. It was incredibly tedious. I had no appetite and little ability to process the real world. Nattie tended to me as she could. I called my mum to cancel her visit planned for the next day. On Friday my fever started to recede and I drank some soup. I was able to look after myself to some extent, though I sorely missed Nattie when she went out. I spent most of the day watching comedy and music docs on the BBC iPlayer, something far too passive for me to do normally. On Saturday morning I felt well enough to tackle the meze platter that Nattie brought back from her previous evening's outing. It tasted pretty damn good. I felt better hour by hour, and in the evening we both went out to a group dinner in a pub. Hot food smelled and tasted even more amazing! I was able to start catching up on email and some real work, preparing kernel stable updates. I even migrated DNS service for decadent.org.uk late that night, driven by anger at the incompetence of my previous provider that had prevented my mails reaching bugs.debian.org. Despite the awfulness of Thursday and the symptoms I still have (tickly throat and runny nose), this bout of the 'flu has been a positive experience for me, but I'll explain why that is in a subsequent entry.

21 February 2016

Lunar: Reproducible builds: week 43 in Stretch cycle

What happened in the reproducible builds effort between February 14th and February 20th 2016:

Toolchain fixes Yaroslav Halchenko uploaded cython/0.23.4+git4-g7eed8d8-1 which makes its output deterministic. Original patch by Chris Lamb. Didier Raboud uploaded pyppd/1.0.2-3 to experimental which now serialize PPD deterministically. Lunar submitted two patches for lcms to add a way for clients to set the creation date/time in profile headers and initialize all bytes when writing named colors.

Packages fixed The following packages have become reproducible due to changes in their build dependencies: dbconfig-common, dctrl-tools, dvdwizard, ekg2, expeyes, galternatives, gpodder, icewm, latex-mk, libiio, lives, navit, po4a, tasksel, tilda, vdr-plugin-infosatepg, xaos. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Unknown status:
  • tomcat7/7.0.68-1 by Emmanuel Bourg (test suite fails in test environment).
Patches submitted which have not made their way to the archive yet:
  • #814840 on tor by Petter Reinholdtsen: use the UTC timezone when calling asciidoc.
  • #815082 on arachne-pnr by Dhole: use the C locale to format the changelog date.
  • #815192 on manpages-de by Reiner Herrmann: tell grep to always treat the input as text so that it works with non-UTF-8 locales.
  • #815193 on razorqt by Reiner Herrmann: tell grep to always treat the input as text so that it works with non-UTF-8 locales.
  • #815250 on jacal by Reiner Herrmann: use the C locale to format the build date.
  • #815252 on colord by Lunar: remove extra timestamps when generating CMF and spectra and implement support for SOURCE_DATE_EPOCH.

reproducible.debian.net Two new package sets have been added: freedombox and freedombox_build-depends. (h01ger)

diffoscope development diffoscope version 49 was released on February 17th. It continues to improve handling of debug symbols for ELF files. Their content will now be compared separately to make them more readable. The search for matching debug packages is more efficient by looking only for .deb files in the same parent directory. Alongside more bug fixes, support for ICC profiles has been added, and libarchive is now also used to read metadata for ar archives.

strip-nondeterminism development Reiner Herrmann added support to normalize Gettext .mo files.

Package reviews 170 reviews have been removed, 172 added and 54 updated in the previous week. 34 new FTBFS bugs have been opened by Chris Lamb, h01ger and Reiner Herrmann. New issues added this week: lxqt_translate_desktop_binary_file_matched_under_certain_locales, timestamps_in_manpages_generated_by_autogen. Improvements to the prebuilder script: avoid ccache, skip disorderfs hook if device nodes cannot be created, compatibility with grsec trusted path execution (Reiner Herrmann), code cleanup (Esa Peuha).

Misc. Steven Chamberlain highlighted reproducibility problems due to differences in how Linux and FreeBSD handle permissions for symlinks. Some possible ways forward have been discussed on the reproducible-builds mailing list. Bernhard M. Wiedemann reported on some reproducibility tests made on OpenSuse mentioning the growing support for SOURCE_DATE_EPOCH. If you are eligible for Outreachy or Google Summer of Code, consider spending the summer working on reproducible builds!

30 January 2016

Simon McVittie: GNOME Developer Experience hackfest: xdg-app + Debian

Over the last few days I've been at the GNOME Developer Experience hackfest in Brussels, looking into xdg-app and how best to use it in Debian and Debian derivatives. xdg-app is basically a way to run "non-core" software on Linux distributions, analogous to apps on Android and iOS. It doesn't replace distributions like Debian or packaging systems, but it adds a layer above them. It's mostly aimed towards third-party apps obtained from somewhere that isn't your distribution vendor, aiming to address a few long-standing problems in that space: To address the first point, each application uses a specified "runtime", which is available as /usr inside its sandbox. This can be used to run application bundles with multiple, potentially incompatible sets of dependencies within the same desktop environment. A runtime can be updated within its branch - for instance, if an application uses the "GNOME 3.18" runtime (consisting of a basic Linux system, the GNOME 3.18 libraries, other related libraries like Mesa, and their recursive dependencies like libjpeg), it can expect to see minor-version updates from GNOME 3.18.x (including any security updates that might be necessary for the bundled libraries), but not a jump to GNOME 3.20. To address the second issue, the plan is for application bundles to be available as a single file, containing metadata (such as the runtime to use), the app itself, and any dependencies that are not available in the runtime (which the app vendor is responsible for updating if necessary). However, the primary way to distribute and upgrade runtimes and applications is to package them as OSTree repositories, which provide a git-like content-addressed filesystem, with efficient updates using binary deltas. The resulting files are hard-linked into place. To address the last point, application bundles run partially isolated from the wider system, using containerization techniques such as namespaces to prevent direct access to system resources. Resources from outside the sandbox can be accessed via "portal" services, which are responsible for access control; for example, the Documents portal (the only one, so far) displays an "Open" dialog outside the sandbox, then allows the application to access only the selected file. xdg-app for Debian One thing I've been doing at this hackfest is improving the existing Debian/Ubuntu packaging for xdg-app (and its dependencies ostree and libgsystem), aiming to get it into a state where I can upload it to Debian experimental. Because xdg-app aims to be a general freedesktop project, I'm currently intending to make it part of the "Utopia" packaging team alongside projects like D-Bus and polkit, but I'm open to suggestions if people want to co-maintain it elsewhere. In the process of updating xdg-app, I sent various patches to Alex, mostly fixing build and test issues, which are in the new 0.4.8 release. I'd appreciate co-maintainers and further testing for this stuff, particularly ostree: ostree is primarily a whole-OS deployment technology, which isn't a use-case that I've tested, and in particular ostree-grub2 probably doesn't work yet. Source code: Binaries (no trust path, so only use these if you have a test VM): The "Hello, World" of xdg-apps Another thing I set out to do here was to make a runtime and an app out of Debian packages. Most of the test applications in and around GNOME use the "freedesktop" or "GNOME" runtimes, which consist of a Yocto base system and lots of RPMs, are rebuilt from first principles on-demand, and are extensive and capable enough that they make it somewhat non-obvious what's in an app or a runtime. So, here's a step-by-step route through xdg-app, first using typical GNOME instructions, but then using the simplest GUI app I could find - xvt, a small xterm clone. I'm using a Debian testing (stretch) x86_64 virtual machine for all this. xdg-app currently requires systemd-logind to put users and apps in cgroups, either with systemd as pid 1 (systemd-sysv) or systemd-shim and cgmanager; I used the default systemd-sysv. In principle it could work with plain cgmanager, but nobody has contributed that support yet. Demonstrating an existing xdg-app Debian's kernel is currently patched to be able to allow unprivileged users to create user namespaces, but make it runtime-configurable, because there have been various security issues in that feature, making it a security risk for a typical machine (and particularly a server). Hopefully unprivileged user namespaces will soon be secure enough that we can enable them by default, but for now, we have to do one of three things to let xdg-app use them: First, we'll need a runtime. The standard xdg-app tutorial would tell you to download the "GNOME Platform" version 3.18. To do that, you'd add a remote, which is a bit like a git remote, and a bit like an apt repository:
$ wget http://sdk.gnome.org/keys/gnome-sdk.gpg
$ xdg-app remote-add --user --gpg-import=gnome-sdk.gpg gnome \
    http://sdk.gnome.org/repo/
(I'm ignoring considerations like trust paths and security here, for brevity; in real life, you'd want to obtain the signing key via https and/or have a trust path to it, just like you would for a secure-apt signing key.) You can list what's available in a remote:
$ xdg-app remote-ls --user gnome
...
org.freedesktop.Platform
...
org.freedesktop.Platform.Locale.cy
...
org.freedesktop.Sdk
...
org.gnome.Platform
...
The Platform runtimes are what we want here: they are collections of runtime libraries with which you can run an application. The Sdk runtimes add development tools, header files, etc. to be able to compile apps that will be compatible with the Platform. For now, all we want is the GNOME 3.18 platform:
$ xdg-app install --user gnome org.gnome.Platform 3.18
Next, we can install an app that uses it, from Alex Larsson's nightly builds of a subset of GNOME. The server they're on doesn't have a great deal of bandwidth, so be nice :-)
$ wget http://209.132.179.2/keys/nightly.gpg
$ xdg-app remote-add --user --gpg-import=nightly.gpg nightly \
    http://209.132.179.2/repo/
$ xdg-app install --user nightly org.mypaint.MypaintDevel
We now have one app, and the runtime it needs:
$ xdg-app list
org.mypaint.MypaintDevel
$ xdg-app run org.mypaint.MypaintDevel
[you see a GUI window]
Digression: what's in a runtime? Behind the scenes, xdg-app runtimes and apps are both OSTree trees. This means the ostree tool, from the package of the same name, can be used to inspect them.
$ sudo apt install ostree
$ ostree refs --repo ~/.local/share/xdg-app/repo
gnome:runtime/org.gnome.Platform/x86_64/3.18
nightly:app/org.mypaint.MypaintDevel/x86_64/master
A "ref" has roughly the same meaning as in git: something like a branch or a tag. ostree can list the directory tree that it represents:
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    runtime/org.gnome.Platform/x86_64/3.18
d00755 0 0      0 /
-00644 0 0    493 /metadata
d00755 0 0      0 /files
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    runtime/org.gnome.Platform/x86_64/3.18 /files
d00755 0 0      0 /files
l00777 0 0      0 /files/local -> ../var/usrlocal
l00777 0 0      0 /files/sbin -> bin
d00755 0 0      0 /files/bin
d00755 0 0      0 /files/cache
d00755 0 0      0 /files/etc
d00755 0 0      0 /files/games
d00755 0 0      0 /files/include
d00755 0 0      0 /files/lib
d00755 0 0      0 /files/lib64
d00755 0 0      0 /files/libexec
d00755 0 0      0 /files/share
d00755 0 0      0 /files/src
You can see that /files in a runtime is basically a copy of /usr. This is not coincidental: the runtime's /files gets mounted at /usr inside the xdg-app container. There is also some metadata, which is in the ini-like syntax seen in .desktop files:
$ ostree cat --repo ~/.local/share/xdg-app/repo \
    runtime/org.gnome.Platform/x86_64/3.18 /metadata
[Runtime]
name=org.gnome.Platform/x86_64/3.16
runtime=org.gnome.Platform/x86_64/3.16
sdk=org.gnome.Sdk/x86_64/3.16
[Extension org.freedesktop.Platform.GL]
version=1.2
directory=lib/GL
[Extension org.freedesktop.Platform.Timezones]
version=1.2
directory=share/zoneinfo
[Extension org.gnome.Platform.Locale]
directory=share/runtime/locale
subdirectories=true
[Environment]
GI_TYPELIB_PATH=/app/lib/girepository-1.0
GST_PLUGIN_PATH=/app/lib/gstreamer-1.0
LD_LIBRARY_PATH=/app/lib:/usr/lib/GL
Looking at an app, the situation is fairly similar:
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master
d00755 0 0      0 /
-00644 0 0    258 /metadata
d00755 0 0      0 /export
d00755 0 0      0 /files
This time, /files maps to what will become /app for the application, which was compiled with --prefix=/app:
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /files
d00755 0 0      0 /files
-00644 0 0   4599 /files/manifest.json
d00755 0 0      0 /files/bin
d00755 0 0      0 /files/lib
d00755 0 0      0 /files/share
There is also a /export directory, which is made visible to the host system so that the contained app can appear as a "first-class citizen" in menus:
$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /export
d00755 0 0      0 /export
d00755 0 0      0 /export/share
user@debian:~$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /export/share
d00755 0 0      0 /export/share
d00755 0 0      0 /export/share/app-info
d00755 0 0      0 /export/share/applications
d00755 0 0      0 /export/share/icons
user@debian:~$ ostree ls --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /export/share/applications
d00755 0 0      0 /export/share/applications
-00644 0 0    715 /export/share/applications/org.mypaint.MypaintDevel.desktop
$ ostree cat --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master \
    /export/share/applications/org.mypaint.MypaintDevel.desktop
[Desktop Entry]
Version=1.0
Name=(Nightly) MyPaint
TryExec=mypaint
Exec=mypaint %f
Comment=Painting program for digital artists
...
Comment[zh_HK]= 
GenericName=Raster Graphics Editor
GenericName[fr]= diteur d'Image Matricielle
MimeType=image/openraster;image/png;image/jpeg;
Type=Application
Icon=org.mypaint.MypaintDevel
StartupNotify=true
Categories=Graphics;GTK;2DGraphics;RasterGraphics;
Terminal=false
Again, there's some metadata:
$ ostree cat --repo ~/.local/share/xdg-app/repo \
    app/org.mypaint.MypaintDevel/x86_64/master /metadata
[Application]
name=org.mypaint.MypaintDevel
runtime=org.gnome.Platform/x86_64/3.18
sdk=org.gnome.Sdk/x86_64/3.18
command=mypaint
[Context]
shared=ipc;
sockets=x11;pulseaudio;
filesystems=host;
[Extension org.mypaint.MypaintDevel.Debug]
directory=lib/debug
Building a runtime, probably the wrong way The way in which the reference/demo runtimes and containers are generated is... involved. As far as I can tell, there's a base OS built using Yocto, and the actual GNOME bits come from RPMs. However, we don't need to go that far to get a working runtime. In preparing this runtime I'm probably completely ignoring some best-practices and tools - but it works, so it's good enough. First we'll need a repository:
$ sudo install -d -o$(id -nu) /srv/xdg-apps
$ ostree init --repo /srv/xdg-apps
I'm just keeping this local for this demonstration, but you could rsync it to a web server's exported directory or something - a lot like a git repository, it's just a collection of files. We want everything in /usr because that's what xdg-app expects, hence usrmerge:
$ sudo mount -t tmpfs -o mode=0755 tmpfs /mnt
$ sudo debootstrap --arch=amd64 --include=libx11-6,usrmerge \
    --variant=minbase stretch /mnt http://192.168.122.1:3142/debian
$ sudo mkdir /mnt/runtime
$ sudo mv /mnt/usr /mnt/runtime/files
This obviously has a lot of stuff in it that we don't need - most obviously init, apt and dpkg - but it's Good Enough . We will also need some metadata. This is sufficient:
$ sudo sh -c 'cat > /mnt/runtime/metadata'
[Runtime]
name=org.debian.Debootstrap/x86_64/8.20160130
runtime=org.debian.Debootstrap/x86_64/8.20160130
That's a runtime. We can commit it to ostree, and generate xdg-app metadata:
$ ostree commit --repo /srv/xdg-apps \
    --branch runtime/org.debian.Debootstrap/x86_64/8.20160130 \
    /mnt/runtime
$ fakeroot ostree commit --repo /srv/xdg-apps \
    --branch runtime/org.debian.Debootstrap/x86_64/8.20160130
$ fakeroot xdg-app build-update-repo /srv/xdg-apps
(I'm not sure why ostree and xdg-app report "Operation not permitted" when we aren't root or fakeroot - feedback welcome.) build-update-repo would presumably also be the right place to GPG-sign your repository, if you were doing that. We can add that as another xdg-app remote:
$ xdg-app remote-add --user --no-gpg-verify local file:///srv/xdg-apps
$ xdg-app remote-ls --user local
org.debian.Debootstrap
Building an app, probably the wrong way The right way to build an app is to build a "SDK" runtime - similar to that platform runtime, but with development files and tools - and recompile the app and any missing libraries with ./configure --prefix=/app && make && make install. I'm not going to do that, because simplicity is nice, and I'm reasonably sure xvt doesn't actually hard-code /usr into the binary:
$ install -d xvt-app/files/bin
$ sudo apt-get --download-only install xvt
$ dpkg-deb --fsys-tarfile /var/cache/apt/archives/xvt_2.1-20.1_amd64.deb \
      tar -xvf - ./usr/bin/xvt
./usr/
./usr/bin/
./usr/bin/xvt
...
$ mv usr xvt-app/files
Again, we'll need metadata, and it's much simpler than the more production-quality GNOME nightly builds:
$ cat > xvt-app/metadata
[Application]
name=org.debian.packages.xvt
runtime=org.debian.Debootstrap/x86_64/8.20160130
command=xvt
[Context]
sockets=x11;
$ fakeroot ostree commit --repo /srv/xdg-apps \
    --branch app/org.debian.packages.xvt/x86_64/2.1-20.1 xvt-app
$ fakeroot xdg-app build-update-repo /srv/xdg-apps
Updating appstream branch
No appstream data for runtime/org.debian.Debootstrap/x86_64/8.20160130
No appstream data for app/org.debian.packages.xvt/x86_64/2.1-20.1
Updating summary
$ xdg-app remote-ls --user local
org.debian.Debootstrap
org.debian.packages.xvt
The obligatory screenshot OK, good, now we can install it:
$ xdg-app install --user local org.debian.Debootstrap 8.20160130
$ xdg-app install --user local org.debian.packages.xvt 2.1-20.1
$ xdg-app run --branch=2.1-20.1 org.debian.packages.xvt
xvt in a container and you can play around with the shell in the xvt and see what you can and can't do in the container. I'm sure there were better ways to do most of this, but I think there's value in having such a simplistic demo to go alongside the various GNOMEish apps.
Acknowledgements: Thanks to all those!

3 December 2015

Dirk Eddelbuettel: RcppCCTZ 0.0.2 -- now with Solaris support

Following on yesterday's announcement of RcppCCTZ, what is the only thing better than another date, time, or timezones library package? One that works on Solaris too :) Bradley White from CCTZ upstream spotted the failed compilation on the machine in Oxford and suggested a quick fix. Jeroen quickly tested what I had put into a branch, and there we have it: version 0.0.2 which now builds everywhere. Changes (for both releases) are summarized here:
Changes in version 0.0.2 (2015-12-02)
  • Additional #ifdef statements suggested by Bradley White in CCTZ ticket #5 permitting compilation on Solaris with thanks to Jeroen for testing our branch.
Changes in version 0.0.1 (2015-12-01)
  • Initial CRAN upload.
  • Package is functional and provides examples.
We now also have a diff to the previous version thanks to CRANberries. More details, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

2 December 2015

Dirk Eddelbuettel: RcppCCTZ 0.0.1

A new package! A couple of weeks ago folks at Google released CCTZ: a C++ library for translating between absolute and civil times using the rules of a time zone. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo. And as the world needs nothing more than additional date, time, or timezones libraries, I started to quickly create a basic R wrapper package. This stalled as the original version of CCTZ used an __int128 for extended precision. That is not only not portable, but also prohibits compilation on baby computers still running a 32 OS (such as my trusted X1 laptop merrily plugging along with 4 gb of ram under Linux). Hence I filed an issue ticket which, lo and behold, got resolved two days ago. And so now we have a shiny new RcppCCTZ package on CRAN in a very basic version 0.0.1. It happily runs all the original examples from CCTZ as e.g. this one:
// from examples/hello.cc
//
// [[Rcpp::export]]
int helloMoon()  
    cctz::TimeZone syd;
    if (!cctz::LoadTimeZone("Australia/Sydney", &syd)) return -1;
    // Neil Armstrong first walks on the moon
    const auto tp1 = cctz::MakeTime(1969, 7, 21, 12, 56, 0, syd);
    const std::string s = cctz::Format("%F %T %z", tp1, syd);
    Rcpp::Rcout << s << "\n";
    cctz::TimeZone nyc;
    cctz::LoadTimeZone("America/New_York", &nyc);
    const auto tp2 = cctz::MakeTime(1969, 7, 20, 22, 56, 0, nyc);
    return tp2 == tp1 ? 0 : 1;
 
which results in
R> library(RcppCCTZ)
R> helloMoon()
1969-07-21 12:56:00 +1000
[1] 0
R>
indicating that the two civil times in fact correspond to the same absolute times when Armstrong walked on the moon. If you want to learn more about CCTZ, there was a corresponding talk CppCon (Youtube, Slides). I hope this provides a starting point for some new interesting computation on time from R. Collaboration welcome via the RcppCCTZ GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

15 November 2015

Lunar: Reproducible builds: week 29 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Emmanuel Bourg uploaded eigenbase-resgen/1.3.0.13768-2 which uses of the scm-safe comment style by default to make them deterministic. Mattia Rizzolo started a new thread on debian-devel to ask a wider audience for issues about the -Wdate-time compile time flag. When enabled, GCC and clang print warnings when __DATE__, __TIME__, or __TIMESTAMP__ are used. Having the flag set by default would prompt maintainers to remove these source of unreproducibility from the sources. Packages fixed The following packages have become reproducible due to changes in their build dependencies: bmake, cyrus-imapd-2.4, drobo-utils, eigenbase-farrago, fhist, fstrcmp, git-dpm, intercal, libexplain, libtemplates-parser, mcl, openimageio, pcal, powstatd, ruby-aggregate, ruby-archive-tar-minitar, ruby-bert, ruby-dbd-odbc, ruby-dbd-pg, ruby-extendmatrix, ruby-rack-mobile-detect, ruby-remcached, ruby-stomp, ruby-test-declarative, ruby-wirble, vtprint. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net The fifth and sixth armhf build nodes have been set up, resulting in five more builder jobs for armhf. More than 10,000 packages have now been identified as reproducible with the reproducible toolchain on armhf. (Vagrant Cascadian, h01ger) Helmut Grohne and Mattia Rizzolo now have root access on all 12 build nodes used by reproducible.debian.net and jenkins.debian.net. (h01ger) reproducible-builds.org is now linked from all package pages and the reproducible.debian.net dashboard. (h01ger) profitbricks-build5-amd64 and profitbricks-build6-amd64, responsible for running amd64 tests now run 398.26 days in the future. This means that one of the two builds that are being compared will be run on a different minute, hour, day, month, and year. This is not yet the case for armhf. FreeBSD tests are also done with 398.26 days difference. (h01ger) The design of the Arch Linux test page has been greatly improved. (Levente Polyak) diffoscope development Three releases of diffoscope happened this week numbered 39 to 41. It includes support for EPUB files (Reiner Herrmann) and Free Pascal unit files, usually having .ppu as extension (Paul Gevers). The rest of the changes were mostly targetting at making it easier to run diffoscope on other systems. The tlsh, rpm, and debian modules are now all optional. The test suite will properly skip tests that need optional tools or modules when they are not available. As a result, diffosope is now available on PyPI and thanks to the work of Levente Polyak in Arch Linux. Getting these versions in Debian was a bit cumbersome. Version 39 was uploaded with an expired key (according to the keyring on ftp.debian.org which will hopefully be updated soon) which is currently handled by keeping the files in the queue without REJECTing them. This prevented any other Debian Developpers to upload the same version. Version 40 was uploaded as a source-only upload but failed to build from source which had the undesirable side effect of removing the previous version from unstable. The package faild to build from source because it was built passing -I to debbuild. This excluded the ELF object files and static archives used by the test suite from the archive, preventing the test suite to work correctly. Hopefully, in a nearby future it will be possible to implement a sanity check to prevent such mistakes in the future. It has also been identified that ppudump outputs time in the system timezone without considering the TZ environment variable. Zachary Vance and Paul Gevers raised the issue on the appropriate channels. strip-nondeterminism development Chris Lamb released strip-nondeterminism version 0.014-1 which disables stripping Mono binaries as it is too aggressive and the source of the problem is being worked on by Mono upstream. Package reviews 133 reviews have been removed, 115 added and 103 updated this week. Chris West and Chris Lamb reported 57 new FTBFS bugs. Misc. The video of h01ger and Chris Lamb's talk at MiniDebConf Cambridge is now available. h01ger gave a talk at CCC Hamburg on November 13th, which was well received and sparked some interest among Gentoo folks. Slides and video should be available shortly. Frederick Kautz has started to revive Dhiru Kholia's work on testing Fedora packages. Your editor wish to once again thank #debian-reproducible regulars for reviewing these reports weeks after weeks.

12 October 2015

Michal &#268;iha&#345;: Gammu 1.36.6

Gammu 1.36.6 has been just released. New bugfix release with lot of improvements in the documentation. Full list of changes: You can download it from http://wammu.eu/download/. Support this program by donations http://wammu.eu/donate/.

Filed under: English Gammu Wammu 0 comments

7 October 2015

Emanuele Rocca: systemd is your friend

Today I want to talk a bit about some cool features of systemd, the default Debian init system since the release of Jessie. Ubuntu has also adopted systemd in 15.04, meaning that you are going to find it literally everywhere.
Logging The component responsible for logging in systemd is called journal. It collects and stores logs in a structured, indexed journal (hence the name). The journal can replace traditional syslog daemons such as rsyslog and syslog-ng, or work together with them. By default Debian keeps on using rsyslog, but if you don't need to ship logs to a centralized server (or do other fancy things) it is possible to stop using rsyslog right now and rely on systemd-journal instead. The obvious question is: why would anybody use a binary format for logs instead of a bunch of tried and true plain-text files? As it turns out, there are quite a lot of good reasons to do so. The killer features of systemd-journald for me are:
  • Index tons of logs while being able to search with good performance: O(log(n)) instead of O(n) which is what you get with text files
  • No need to worry about log rotation anymore, in any shape or form
The last point in particular is really critical in my opinion. Traditional log rotation implementations rely on cron jobs to check how much disk space is used by logs, compressing/removing old files. Log rotation is usually: 1) annoying to configure; 2) hard to get right; 3) prone to DoS attacks. With journald, there is pretty much nothing to configure. Log rotation is built into the daemon disk space allocation logic itself. This also allows to avoid vulnerability windows due to time-based rotation, which is what you get with logrotate and friends. Enough high-level discussions though, here is how to use the journal! Check if you already have the directory /var/log/journal, otherwise create it (as root). Then restart systemd-journald as follows: sudo systemctl restart systemd-journald You can get all messages produced since the last boot with journalctl -b. All messages produced today can get extracted using journalctl --since=today. Want to get all logs related to ssh? Try with journalctl _SYSTEMD_UNIT=ssh.service. There are many more filtering options available, you can read all about them with man journalctl. journald's configuration file is /etc/systemd/journald.conf. Two of the most interesting options are SystemMaxUse and SystemKeepFree, which can be used to change the amount of disk space dedicated to logging. They default to 10% and 15% of the /var filesystem respectively. Here is a little cheatsheet:
journalctl -b                # Show all messages since last boot
journalctl -f                # Tail your logs
journalctl --since=yesterday # Show all messages produced since yesterday
journalctl -pcrit            # Filter messages by priority
journalctl /bin/su           # Filter messages by program
journalctl --disk-usage      # The amount of space in use for journaling
Further reading:
Containers A relatively little known component of systemd is systemd-nspawn. It is a small, straightforward container manager. If you don't already have a chroot somewhere, here is how to create a basic Debian Jessie chroot under /srv/chroots/jessie:
$ debootstrap jessie /srv/chroots/jessie http://http.debian.net/debian/
With systemd-nspawn you can easily run a shell inside the chroot:
$ sudo systemd-nspawn -D /srv/chroots/jessie
Spawning container jessie on /srv/chroots/jessie.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
root@jessie:~#
Done. Everything works out of the box: no need for you to mount /dev, /run and friends, systemd-nspawn took care of that. Networking also works. If you want to actually boot the system, just add the -b switch to the previous command:
$ sudo systemd-nspawn -b -D /srv/chroots/jessie
Spawning container jessie on /srv/chroots/jessie.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.
Welcome to Debian GNU/Linux jessie/sid!
Set hostname to <orion>.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Swap.
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on Syslog Socket.
         Mounting POSIX Message Queue File System...
         Mounting Huge Pages File System...
         Mounting FUSE Control File System...
         Starting Copy rules generated while the root was ro...
         Starting Journal Service...
[  OK  ] Started Journal Service.
[  OK  ] Reached target Slices.
         Starting Remount Root and Kernel File Systems...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Started Copy rules generated while the root was ro.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Reached target Remote File Systems.
         Starting Trigger Flushing of Journal to Persistent Storage...
[  OK  ] Started Load/Save Random Seed.
         Starting LSB: Raise network interfaces....
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Trigger Flushing of Journal to Persistent Storage.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started LSB: Raise network interfaces..
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
         Starting /etc/rc.local Compatibility...
         Starting Login Service...
         Starting LSB: Regular background program processing daemon...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting System Logging Service...
[  OK  ] Started System Logging Service.
         Starting Permit User Sessions...
[  OK  ] Started /etc/rc.local Compatibility.
[  OK  ] Started LSB: Regular background program processing daemon.
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
         Starting Console Getty...
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Login Service.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Update UTMP about System Runlevel Changes.
Debian GNU/Linux jessie/sid orion console
orion login:
That's it! Just one command to start a shell in your chroot or boot the container, again zero configuration needed. Finally, systemd provides a command called machinectl that allows you to introspect and control your container:
$ sudo machinectl status jessie
jessie
           Since: Wed 2015-10-07 11:22:56 CEST; 55min ago
          Leader: 32468 (systemd)
         Service: nspawn; class container
            Root: /srv/chroots/jessie
         Address: fe80::8e70:5aff:fe81:2290
                  192.168.122.1
                  192.168.1.13
              OS: Debian GNU/Linux jessie/sid
            Unit: machine-jessie.scope
                   32468 /lib/systemd/systemd
                   system.slice
                     dbus.service
                       32534 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile -...
                     cron.service
                       32539 /usr/sbin/cron
                     systemd-journald.service
                       32487 /lib/systemd/systemd-journald
                     systemd-logind.service
                       32532 /lib/systemd/systemd-logind
                     console-getty.service
                       32544 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt102
                     rsyslog.service
                       32540 /usr/sbin/rsyslogd -n
With machinectl you can also reboot, poweroff, terminate your containers and more. There are so many things to learn about systemd and containers! Here are some references. This stuff is pretty exciting. Now that all major distributions use systemd by default, we can expect to have access to tools like journalctl and systemd-nspawn everywhere!

4 October 2015

Lunar: Reproducible builds: week 23 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Andreas Metzler uploaded autogen/1:5.18.6-1 in experimental with several patches for reproducibility issues written by Valentin Lorentz. Groovy upstream has merged a change proposed by Emmanuel Bourg to remove timestamps generated by groovydoc. Ben Hutchings submitted a patch to add support for SOURCE_DATE_EPOCH in linux-kbuild as an alternate way to specify the build timestamp. Reiner Herrman has sent a patch adding support for SOURCE_DATE_EPOCH in docbook-utils. Packages fixed The following packages became reproducible due to changes in their build dependencies: commons-csv. fest-reflect, sunxi-tools, xfce4-terminal, The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Tomasz Rybak uploaded pycuda/2015.1.3-1 which should fix reproducibility issues. The package has not been tested as it is in contrib. akira found an embedded code copy of texi2html in fftw. reproducible.debian.net Email notifications are now only sent once a day per package, instead of on each status change. (h01ger) disorderfs has been temporarily disabled to see if it had any impact on the disk space issues. (h01ger) When running out of disk space, build nodes will now automatically detect the problem. This means test results will not be recorded as FTBFS and the problem will be reported to Jenkins maintainers. (h01ger) The navigation menu of package pages has been improved. (h01ger) The two amd64 builders now use two different kernel versions: 3.16 from stable and 4.1 from backports on the other. (h01ger) We now graph the number of packages which needs to be fixed. (h01ger) Munin now creates graphs on how many builds were performed by build nodes (example). (h01ger) A migration plan has been agreed with DSA on how to turn Jenkins into an official Debian service. A backport of jenkins-job-builder for Jessie is currently missing. (h01ger) Package reviews 119 reviews have been removed, 103 added and 45 updated this week. 16 fail to build from source issues were reported by Chris Lamb and Mattia Rizzolo. New issue this week: timestamps_in_manpages_generated_by_docbook_utils. Misc. Allan McRae has submitted a patch to make ArchLinux pacman record a .BUILDINFO file.

29 September 2015

Norbert Preining: Multi-boot stick update: TAILS 1.6, SysresCD 4.6.0, GParted 0.23, Debian 8.2

Updates for my multi-boot/multi-purpose USB stick: All components have been updated to the latest versions and I have confirmed that all of them still boot properly although changes in the grub.cfg file are necessary. So going through these explanations one will end up with a usable USB stick that can boot you into TAILS, System Rescue CD, GNU Parted Live CD, GRML, and also can boot into an installation of Debian 8.2 Jessie installation. All this while still being able to use the USB stick as normal media. multi-usb-disk Since there have been a lot of updates, and also changes in the setup and grub config file, I include the full procedure here, that is, merging and updating these previous posts: USB stick with Tails and SystemRescueCD, Tails 1.2.1, Debian jessie installer, System Rescue CD on USB, USB stick update: TAILS 1.4, GParted 0.22, SysResCD 4.5.2, Debian Jessie, and USB stick update: Debian is back, plus GRML. Let us repeat some things from the original post concerning the wishlist and the main players: I have a long wishlist of items a boot stick should fulfill Requirements A USB stick, the iso images of TAILS 1.6, SystemRescueCD 4.6.0, GParted Lice CD 0.23.0, GRML 2014.11, and some tool to access iso images, for example ISOmaster (often available from your friendly Linux distribution). I assume that you have already an USB stick prepared as described previously. If this is not the case, please go there and follow the section on preparing your usb stick. Three types of boot options We will employ three different approaches to boot special systems: the one is directly from an iso image (easiest, simple to update), the other via extraction of the necessary kernels and images (bit painful, needs some handwork), and the last one is a mixture necessary to get Debian booting (most painful, needs additional downloads and handwork). At the moment we have the following status with respect to boot methods: Booting from ISO image Grub has gained quite some time ago the ability to boot directly from an ISO image. In this case the iso image is mounted via loopback, and the kernel and initrd are specified relatively to the iso image root. This system makes it extremely easy to update the respective boot option: just drop the new iso image onto the USB stick, and update the isofile setting. One could even use some -latest method, but I prefer to keep the exact name. For both SystemRescueCD, GNOME Partition Live CD, and GRML, just drop the iso files into /boot/iso/, in my case /boot/iso/systemrescuecd-x86-4.6.0.iso and /boot/iso/gparted-live-0.23.0-1-i586.iso. After that, entries like the following have to be added to grub.cfg. For the full list see grub.cfg:
submenu "System Rescue CD 4.6.0 (via ISO) ---> "  
  set isofile="/boot/iso/systemrescuecd-x86-4.6.0.iso"
  menuentry "SystemRescueCd (64bit, default boot options)"  
        set gfxpayload=keep
        loopback loop (hd0,1)$isofile
        linux   (loop)/isolinux/rescue64 isoloop=$isofile
        initrd  (loop)/isolinux/initram.igz
   
  ...
 
 
submenu "GNU/Gnome Parted Live CD 0.23.0 (via ISO) ---> "  
  set isofile="/boot/iso/gparted-live-0.23.0-1-i586.iso"
  menuentry "GParted Live (Default settings)" 
    loopback loop (hd0,1)$isofile
    linux (loop)/live/vmlinuz boot=live union=overlay username=user config components quiet noswap noeject  ip= net.ifnames=0 nosplash findiso=$isofile
    initrd (loop)/live/initrd.img
   
  ...
 
 
submenu "GRML 2014.11 ---> "  
  menuentry "Grml Rescue System 64bit"  
        iso_path="/boot/iso/grml64-full_2014.11.iso"
        export iso_path
        loopback loop (hd0,1)$iso_path
        set root=(loop)
        kernelopts=" ssh=foobarbaz toram  "
        export kernelopts
        configfile /boot/grub/loopback.cfg
    
 
Note the added isoloop=$isofile and findiso=$isofile that helps the installer find the iso images. Booting via extraction of kernels and images This is a bit more tedious, but still not too bad. Installation of TAILS files Assuming you have access to the files on the TAILS CD via the directory ~/tails, execute the following commands:
mkdir -p /usbstick/boot/tails
cp -a ~/tails/live/* /usbstick/boot/tails/
The grub.cfg entries look now similar to the following:
submenu "TAILS Environment 1.6 ---> "  
  menuentry "Tails64 Live System"  
        linux   /boot/tails/vmlinuz2 boot=live live-media-path=/boot/tails config live-media=removable nopersistent noprompt timezone=Etc/UTC block.events_dfl_poll_msecs=1000 splash noautologin module=Tails
        initrd  /boot/tails/initrd2.img
 
   
  ...
 
The important part here is the live-media-path=/boot/tails, otherwise TAILS will not find the correct files for booting. The rest of the information was extracted from the boot setup of TAILS itself. Mixture of iso image and extraction Debian jessie As mentioned in the previous post, booting Debian/Jessie installation images via any method laid out above didn t work, since the iso images is never found. It turned out that the current installer iso images do not contain the iso-scan package, which is responsible for searching and loading of iso images. But with a small trick one can overcome this: One needs to replace the initrd that is on the ISO image with one that contains the iso-scan package. And we do not need to create these initrd by ourselves, but simply use the ones from hd-media type installer. I downloaded the following four gzipped initrds from one of the Debian mirrors: i386/initrd text mode, i386/initrd gui mode, amd64/initrd text mode, amd64/initrd gui mode, and put them into the USB stick s boot/debian/install.386, boot/debian/install.386/gtk, boot/debian/install.amd, boot/debian/install.amd/gtk, respectively. Finally, I added entries similar to this one (rest see the grub.cfg file):
submenu "Debian 8.2 Jessie NetInstall ---> "  
    set isofile="/boot/iso/firmware-8.2.0-amd64-i386-netinst.iso"
    menuentry '64 bit Install'  
        set background_color=black
        loopback loop (hd0,1)$isofile
        linux    (loop)/install.amd/vmlinuz iso-scan/ask_second_pass=true iso-scan/filename=$isofile vga=788 -- quiet 
        initrd   /boot/debian/install.amd/initrd.gz
     
    ...
 
Again an important point, don t forget the two kernel command line options: iso-scan/ask_second_pass=true iso-scan/filename=$isofile, otherwise you probably will have to make the installer scan all disks and drives completely, which might take ages. Current status of USB stick Just to make sure, the usb stick should contain at the current stage the following files:
/boot/
    iso/
        firmware-8.2.0-amd64-i386-netinst.iso
        gparted-live-0.23.0-1-i586.iso
        grml64-full_2014.11.iso
        systemrescuecd-x86-4.6.0.iso
    tails/
        vmlinuz Tails.module initrd.img ....
    grub/
        fonts/
            lots of files
        locale/
            lots of files
        x86_64-efi/
            lots of files
        font.pf2
        grubenv
        grub.cfg            *this file we create in the next step!!*
/EFI
    BOOT/
        BOOTX64.EFI
The Grub config file grub.cfg The final step is to provide a grub config file in /usbstick/boot/grub/grub.cfg. I created one by looking at the isoboot.cfg files both in the SystemRescueCD, TAILS iso images, GParted iso image, and the Debian/Jessie image, and converting them to grub syntax. Excerpts have been shown above in the various sections. I spare you all the details, grab a copy here: grub.cfg Conclusion That s it. Now you can anonymously provide data about your evil government, rescue your friends computer, fix a forgotten Windows password, and above all, install a proper free operating system. If you have any comments, improvements or suggestions, please drop me a comment. I hope this helps a few people getting a decent USB boot stick running. Enjoy.

25 August 2015

Lunar: Reproducible builds: week 17 in Stretch cycle

A good amount of the Debian reproducible builds team had the chance to enjoy face-to-face interactions during DebConf15.
Names in red and blue were all present at DebConf15
Picture of the  reproducible builds  talk during DebConf15
Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian. A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive. Picture of the  reproducible builds  roundtable during DebConf15 Toolchain fixes Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2html SOURCE_DATE_EPOCH aware. St phane Glondu reported that dpkg-source would not respect tarball permissions when unpacking under a umask of 002. After hours of iterative testing during the DebConf workshop, Sandro Knau created a test case showing how pdflatex output can be non-deterministic with some PNG files. Packages fixed The following 65 packages became reproducible due to changes in their build dependencies: alacarte, arbtt, bullet, ccfits, commons-daemon, crack-attack, d-conf, ejabberd-contrib, erlang-bear, erlang-cherly, erlang-cowlib, erlang-folsom, erlang-goldrush, erlang-ibrowse, erlang-jiffy, erlang-lager, erlang-lhttpc, erlang-meck, erlang-p1-cache-tab, erlang-p1-iconv, erlang-p1-logger, erlang-p1-mysql, erlang-p1-pam, erlang-p1-pgsql, erlang-p1-sip, erlang-p1-stringprep, erlang-p1-stun, erlang-p1-tls, erlang-p1-utils, erlang-p1-xml, erlang-p1-yaml, erlang-p1-zlib, erlang-ranch, erlang-redis-client, erlang-uuid, freecontact, givaro, glade, gnome-shell, gupnp, gvfs, htseq, jags, jana, knot, libconfig, libkolab, libmatio, libvsqlitepp, mpmath, octave-zenity, openigtlink, paman, pisa, pynifti, qof, ruby-blankslate, ruby-xml-simple, timingframework, trace-cmd, tsung, wings3d, xdg-user-dirs, xz-utils, zpspell. The following packages became reproducible after getting fixed: Uploads that might have fixed reproducibility issues: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: St phane Glondu reported two issues regarding embedded build date in omake and cduce. Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar. Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found. reproducible.debian.net Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays. armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger) The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger) mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger) Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output. Connections to UDD have been made more robust. (Mattia Rizzolo) diffoscope development diffoscope version 31 was released on August 21st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep. New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo). jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb. strip-nondeterminism development Andrew Ayer released strip-nondeterminism version 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added. Testing directory ordering issues: disorderfs During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random. Documentation update Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C. Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it. Package reviews 44 reviews have been removed, 192 added and 77 updated this week. New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex. 117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni. Misc. Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source! Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.

21 August 2015

Simon Kainz: DUCK challenge: Final week

Well, here are the stats for the final week of the DUCK challenge as well as DebConf15: So we had 21 packages fixed and uploaded by 14 different uploaders. People were really working hard on this during DebConf. A big "Thank You" to you!! Since the start of this challenge, a total of 89 packages, were fixed. Here is a quick overview:
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
# Packages 10 15 10 14 10 9 21
Total 10 25 35 49 59 68 89
Thank you all for participating - either on purpose or "accidentially": Some people were really surprised as i sneaked up on them at DebConf15, confronting them with a green lighter! I just tried to put even more fun into Debian, i hope this worked out Pevious articles are here: Week 1, Week 2, Week 3, Week 4, Week 5,Week 6.

10 August 2015

Lunar: Reproducible builds: week 15 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Guillem Jover uploaded dpkg/1.18.2 which makes dependency comparisons deep by comparing not only the first dependency alternative, to get them sorted in a reproducible way. Original patch by Chris Lamb. Dhole updated the patch adding support for SOURCE_DATE_EPOCH in gettext. A modified package is in the experimental reproducible repository. Valentin Lorentz submitted a patch adding support for SOURCE_DATE_EPOCH to ocamldoc. Valentin Lorentz also opened a bug about the inability to set an arbitrary RNG seed for ocamlopt which would be a way to fix an issue affecting many OCaml packages. Dhole submitted a patch adding support for SOURCE_DATE_EPOCH in qhelpgenerator. A modified package has been sent to the experimental repository as well. Several packages have been updated for the experimental toolchain: doxygen (akira), and dpkg (h01ger). Also, h01ger has built and uploaded all experimental packages having arch:any packages for armhf: dpkg, gettext, doxygen, fontforge, libxslt and texlive-bin. We are now providing our toolchain for armhf and amd64. Packages fixed As you might have noticed, Debian sid is currently largely uninstallable, due to the GCC 5 transition, which also can be see in our reproducibility test setup. Please help! The following packages became reproducible due to changes in their build dependencies: glosstex, indent, ktikz, liblouis, libmicrohttpd, linkchecker, multiboot, qterm, rrep, trueprint, twittering-mode. The following packages became reproducible after getting fixed: Patches submitted which have not made their way to the archive yet: Lunar reported an issue on an unstable ABI from a generated header in icedove reminding of an issue affecting libical-dev. The bug has since been fixed by Carsten Schoenert. akira identified an unreferenced embeded code copy (causing unreproducibility!) in gperf. reproducible.debian.net The scheduler has temporarily been changed to not schedule any already tested packages for sid and experimental, due to the the GCC 5 transitions, which are well visible in our graphs now. On the plus side this has caused our stretch testing to catch up (and improve stats). (h01ger) depwait packages (packages where the Build-Depends cannot be satisfied) are now listed in the last 24h and last 48h pages (Mattia Rizzolo) Two new amd64 build nodes (with 8 cores and 32 GB RAM each) have been added, kindly sponsored by Profitbricks. (h01ger) The 4 armhf (setup last week by Vagrant Cascadian) and 2 amd64 build nodes have been made available to Jenkins. Remote job scheduling has been implemented and 35 new jobs have been added for pbuilder and schroot creation and maintenance of the nodes. (h01ger) The manual scheduler gained a flag (-a/--architecture) to select which arch to schedule in. (Mattia Rizzolo) armhf will only be testing stretch for now, due to limited hardware ressources. (h01ger) The page listing maintainers of unreproducible packages gained internal anchors. As an example, one can now link to unreproducible orphaned packages. (Mattia Rizzolo) Packages with a bug tagged pending are marked using a new symbol: a brown P (Mattia Rizzolo) diffoscope development debbindiff is now called diffoscope! It also has a website at diffoscope.org. The name was changed to better reflect that it became a general purpose tool, capable of comparing many different archive formats, or directories. Version 29 is the renaming release. Amongst a couple of other cosmetic changes a favicon showing the new logo has been added to the generated HTML reports. Version 30 replaces the file matching algorithm for files listed in .changes to a smarter one that removes only the version number. It also fixes a bug where squashfs directories were being extracted even if their content was being compared at a later stage. It also fixes an issue with the test suite that was detected by debci. Documentation update More rationale have been added for supporting SOURCE_DATE_EPOCH The unfinished Reproducible Builds HOWTO is now visible on the web, feedback and patches most welcome. Package reviews 261 obsolete reviews have been removed, 73 added and 145 updated this week.

6 August 2015

Eduard Sanou: Reproducible builds on Debian for GSoC 2015, 1st update

This is the second blog post I m writing about my experiences contributing to Debian for Google Summer of Code 2015 (check my first post)

Status update

First month It s been two months and a few days since the GSoC started. During the first month I worked on fixing specific packages, mainly concerning issues with timestamps, which is a very common source of unreproducibility. In many cases, during the build process files are compressed into gzip or zip archives, which store the creation time of files in the metadata. This can lead to unreproducible results when there is timezone variation between builds (easily fixed setting the timezone to UTC before the compression happens). In some cases the compressed files are generated during the build, and thus add build times in the metadata of compressed files (in this case the creation date of the files needs to be normalized somehow). As explained in my application, I finished exams on the end of June, that s why I chose to work on small fixes first, so that I could make the most out of my time between studying and finishing university projects and reports. I m happy with my first month, as I have worked as originally planned. Actually, my estimation of the number of bugs I could submit every week was surpassed in reality!

Second month Once the university was over, I started dedicating myself fully to the project. This allowed me to start working on toolchain fixes, following my original plan on working with timestamp related issues. In particular I have been working a lot in implementing a proposal for deterministic timestamps that appeared in the reproducible builds project. The idea is to define an environment variable called SOURCE_DATE_EPOCH that contains a known timestamp in Unix epoch format. With this variable exported, tools that would embed the localtime in their generated or processed files, can use this externally supplied date. This would happen only if SOURCE_DATE_EPOCH is exported, so the behaviour of the tool wouldn t change if the variable is not set. The first package I patched to implement this behaviour was gcc. The reason behind this is that there are about 420 unreproducible packages due to using the __DATE__, __TIME__ and __TIMESTAMP__ C macros. My patch changes the behavior of the macros __DATE__ and __TIME__ if SOURCE_DATE_EPOCH is exported. I submitted this patch to the gcc-patches list. Even though there was some interesting discussions in the list, the patch has not been accepted yet. Seeing how the reproducible builds idea is gaining momentum and becoming widespread, I m positive that at some point the gcc people will be more receptive for such patch. The second work with SOURCE_DATE_EPOCH was in debhelper; I patched this building tool to export the variable with the latest debian/changelog entry timestamp. With this patch, all the tools that run under dh will be able to use it to embed deterministic timestamps. Unfortunately some parts of the build process of some packages don t happen under debhelper, so the variable needs to be exported in a different way. Having submitted the debhelper patch allowed many packages to become reproducible after the tools that embedded timestamps were patched to honour SOURCE_DATE_EPOCH. As of today, the toolchain packages I have patched to do that are: gcc, libxslt, gettext, ghostscript and qt4-x11 (qhelpgenerator). I have also continued working on fixing individual packages affected by timestamps, random orderings (such as the ones from listing hash keys) and locale depending orderings; I have tagged packages in our infrastructure to note what kind of issue makes them unreproducible; I have updated some parts of the Reproducible Builds Wiki.

Impressions about reproducible builds The work I did during the first month felt a bit tedious sometimes: it didn t require much creativity or thinking as most of the fixes where quite mechanical, following a recipe. After I became free from university duties, I started looking into less trivial issues, which require more deep investigation and feel more rewarding once they are fixed. I also worked on toolchain fixes, which need more work and need more care. Fixing toolchain packages feels particularly rewarding because they can cause many packages to become reproducible at once. There is a very active community in the reproducible builds project! It s great to see so many people contributing to this project spending so many hours and putting so much effort. I ve felt very welcome from the beginning and I have gotten kind replies and helpful answers to all the questions and doubts I ve had, both from my mentor and from the other people in the project. I want to add that I m still amazed by the awesome infrastructure set up for the reproducible builds project. The team is using a Jenkins machine to continuously build packages to check for reproducibility, with irc notifications for changes, and also with a really useful web interface to list all the packages organized by issues that allows exploring them individually with all the available information. Also not only the infrastructure is used to build Debian amd64 packages, but also FreeBSD, NetBSD, OpenWRT, coreboot and lately Debian armhf with the help of a few new arm nodes.

Impressions about working on a free software project This was my first time working on a community driven free software project and I ve learned so many things. Something I learned during the first days, which is even more present in such wide project like the reproducible builds, is that contributing is not just writing patches for the features you want; you also need to convince the maintainer of the package to accept the patch! I was a bit surprised at the beginning because even if this is a Debian project, that aims to make changes to the whole distribution, decisions are not absolute for the whole Debian project. After taking decisions within the reproducible builds teams on how to approach things, we need to convince the rest of the Debian developers (mainly maintainers) to follow them, as they are allowed to disagree. So it is required to work together for solutions that makes everyone comfortable, usually with discussions on mailing lists or irc. There is another fact that I wasn t expecting before getting involved in this project. The kind of teamwork I have done previously involves having a leader who decides how stuff is done, who takes decisions when needed and oversees the whole project. There seems to be a different philosophy in Debian. Instead of having a leader, everyone tries their best, trying to convince the others that their solution is good, often by showing an implementation of the solution and providing proof that it works, rather than trying to get the solution accepted before coding it. Also, solutions and ideas are valued for their quality rather than by the position of the person submitting them, and there is no hierarchy within the group: all comments and advices are taken equally, valuing their usefulness regardless of who gives them. Usually there are no votes when deciding things. Members try their best on their approaches, trying to convince others as best as they can. And even if someone disagrees they may end up accepting the solution if they don t manage to convince the original proposer of doing things differently. The idea is to spend more time working and coding than arguing and deciding on the way to do things. So far I ve seen this approach to be very efficient :) I ve been told by my mentor that for difficult cases there exist a Debian committee that helps mediate on disagreements, but that is only used as a last option, probably when the discussion gets heated up.

Personal experience Overall I m very happy to finally having set my foot in the free software community, where I m able to contribute to the kind of software I have been using for years. The sense of community in Debian is really big, and everyone is invited to participate. I think that Google is doing an awesome job with the Google Summer of Code, not only because it gives a lot to free software but because it helps students to join the free software world as contributors (which is something that may be difficult to get into when you don t know how to begin, as it happened to myself for some time). I plan to continue contributing to the free software world, and I d encourage anyone to find projects to get involved and contribute as well! Happy hacking to everyone!

3 August 2015

Lunar: Reproducible builds: week 14 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes akira submitted a patch to make cdbs export SOURCE_DATE_EPOCH. She uploded a package with the enhancement to the experimental reproducible repository. Packages fixed The following 15 packages became reproducible due to changes in their build dependencies: dracut, editorconfig-core, elasticsearch, fish, libftdi1, liblouisxml, mk-configure, nanoc, octave-bim, octave-data-smoothing, octave-financial, octave-ga, octave-missing-functions, octave-secs1d, octave-splines, valgrind. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: In contrib, Dmitry Smirnov improved libdvd-pkg with 1.3.99-1-1. Patches submitted which have not made their way to the archive yet: reproducible.debian.net Four armhf build hosts were provided by Vagrant Cascadian and have been configured to be used by jenkins.debian.net. Work on including armhf builds in the reproducible.debian.net webpages has begun. So far the repository comparison page just shows us which armhf binary packages are currently missing in our repo. (h01ger) The scheduler has been changed to re-schedule more packages from stretch than sid, as the gcc5 transition has started This mostly affects build log age. (h01ger) A new depwait status has been introduced for packages which can't be built because of missing build dependencies. (Mattia Rizzolo) debbindiff development Finally, on August 31st, Lunar released debbindiff 27 containing a complete overhaul of the code for the comparison stage. The new architecture is more versatile and extensible while minimizing code duplication. libarchive is now used to handle cpio archives and iso9660 images through the newly packaged python-libarchive-c. This should also help support a couple other archive formats in the future. Symlinks and devices are now properly compared. Text files are compared as Unicode after being decoded, and encoding differences are reported. Support for Sqlite3 and Mono/.NET executables has been added. Thanks to Valentin Lorentz, the test suite should now run on more systems. A small defiency in unquashfs has been identified in the process. A long standing optimization is now performed on Debian package: based on the content of the md5sums control file, we skip comparing files with matching hashes. This makes debbindiff usable on packages with many files. Fuzzy-matching is now performed for files in the same container (like a tarball) to handle renames. Also, for Debian .changes, listed files are now compared without looking the embedded version number. This makes debbindiff a lot more useful when comparing different versions of the same package. Based on the rearchitecturing work has been done to allow parallel processing. The branch now seems to work most of the time. More test needs to be done before it can be merged. The current fuzzy-matching algorithm, ssdeep, has showed disappointing results. One important use case is being able to properly compare debug symbols. Their path is made using the Build ID. As this identifier is made with a checksum of the binary content, finding things like CPP macros is much easier when a diff of the debug symbols is available. Good news is that TLSH, another fuzzy-matching algorithm, has been tested with much better results. A package is waiting in NEW and the code is ready for it to become available. A follow-up release 28 was made on August 2nd fixing content label used for gzip2, bzip2 and xz files and an error on text files only differing in their encoding. It also contains a small code improvement on how comments on Difference object are handled. This is the last release name debbindiff. A new name has been chosen to better reflect that it is not a Debian specific tool. Stay tuned! Documentation update Valentin Lorentz updated the patch submission template to suggest to write the kind of issue in the bug subject. Small progress have been made on the Reproducible Builds HOWTO while preparing the related CCCamp15 talk. Package reviews 235 obsolete reviews have been removed, 47 added and 113 updated this week. 42 reports for packages failing to build from source have been made by Chris West (Faux). New issue added this week: haskell_devscripts_locale_substvars. Misc. Valentin Lorentz wrote a script to report packages tested as unreproducible installed on a system. We encourage everyone to run it on their systems and give feedback!

26 July 2015

Lunar: Reproducible builds: week 13 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes akira uploaded a new version of doxygen in the experimental reproducible repository incorporating upstream patch for SOURCE_DATE_EPOCH, and now producing timezone independent timestamps. Dhole updated Peter De Wachter's patch on ghostscript to use SOURCE_DATE_EPOCH and use UTC as a timezone. A modified package is now being experimented. Packages fixed The following 14 packages became reproducible due to changes in their build dependencies: bino, cfengine2, fwknop, gnome-software, jnr-constants, libextractor, libgtop2, maven-compiler-plugin, mk-configure, nanoc, octave-splines, octave-symbolic, riece, vdr-plugin-infosatepg. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net Packages identified as failing to build from source with no bugs filed and older than 10 days are scheduled more often now (except in experimental). (h01ger) Package reviews 178 obsolete reviews have been removed, 59 added and 122 updated this week. New issue identified this week: random_order_in_ruby_rdoc_indices. 18 new bugs for packages failing to build from sources have been reported by Chris West (Faux), and h01ger.

7 July 2015

Lunar: Reproducible builds: week 10 in Stretch cycle

What happened about the reproducible builds effort this week: Media coverage Daniel Stender published an English translation of the article which originally appeared in Linux Magazin in Admin Magazine. Toolchain fixes Fixes landed in the Debian archive: Lunar submitted to Debian the patch already sent upstream adding a --clamp-mtime option to tar. Patches have been submitted to add support for SOURCE_DATE_EPOCH to txt2man (Reiner Herrmann), epydoc (Reiner Herrmann), GCC (Dhole), and Doxygen (akira). Dhole uploaded a new experimental debhelper to the reproducible repository which exports SOURCE_DATE_EPOCH. As part of the experiment, the patch also sets TZ to UTC which should help with most timezone issues. It might still be problematic for some packages which would change their settings based on this. Mattia Rizzolo sent upstream a patch originally written by Lunar to make the generate-id() function be deterministic in libxslt. While that patch was quickly rejected by upstream, Andrew Ayer came up with a much better one which sadly could have some performance impact. Daniel Veillard replied with another patch that should be deterministic in most cases without needing extra data structures. It's impact is currently being investigated by retesting packages on reproducible.debian.net. akira added a new option to sbuild for configuring the path in which packages are built. This will be needed for the srebuild script. Niko Tyni asked Perl upstream about it using the __DATE__ and __TIME__ C processor macros. Packages fixed The following 143 packages became reproducible due to changes in their build dependencies: alot, argvalidate, astroquery, blender, bpython, brian, calibre, cfourcc, chaussette, checkbox-ng, cloc, configshell, daisy-player, dipy, dnsruby, dput-ng, dsc-statistics, eliom, emacspeak, freeipmi, geant321, gpick, grapefruit, heat-cfntools, imagetooth, jansson, jmapviewer, lava-tool, libhtml-lint-perl, libtime-y2038-perl, lift, lua-ldoc, luarocks, mailman-api, matroxset, maven-hpi-plugin, mknbi, mpi4py, mpmath, msnlib, munkres, musicbrainzngs, nova, pecomato, pgrouting, pngcheck, powerline, profitbricks-client, pyepr, pylibssh2, pylogsparser, pystemmer, pytest, python-amqp, python-apt, python-carrot, python-crypto, python-darts.lib.utils.lru, python-demgengeo, python-graph, python-mock, python-musicbrainz2, python-pathtools, python-pskc, python-psutil, python-pypump, python-repoze.sphinx.autointerface, python-repoze.tm2, python-repoze.what-plugins, python-repoze.what, python-repoze.who-plugins, python-xstatic-term.js, reclass, resource-agents, rgain, rttool, ruby-aggregate, ruby-archive-tar-minitar, ruby-bcat, ruby-blankslate, ruby-coffee-script, ruby-colored, ruby-dbd-mysql, ruby-dbd-odbc, ruby-dbd-pg, ruby-dbd-sqlite3, ruby-dbi, ruby-dirty-memoize, ruby-encryptor, ruby-erubis, ruby-fast-xs, ruby-fusefs, ruby-gd, ruby-git, ruby-globalhotkeys, ruby-god, ruby-hike, ruby-hmac, ruby-integration, ruby-ipaddress, ruby-jnunemaker-matchy, ruby-memoize, ruby-merb-core, ruby-merb-haml, ruby-merb-helpers, ruby-metaid, ruby-mina, ruby-net-irc, ruby-net-netrc, ruby-odbc, ruby-packet, ruby-parseconfig, ruby-platform, ruby-plist, ruby-popen4, ruby-rchardet, ruby-romkan, ruby-rubyforge, ruby-rubytorrent, ruby-samuel, ruby-shoulda-matchers, ruby-sourcify, ruby-test-spec, ruby-validatable, ruby-wirble, ruby-xml-simple, ruby-zoom, ryu, simplejson, spamassassin-heatu, speaklater, stompserver, syncevolution, syncmaildir, thin, ticgit, tox, transmissionrpc, vdr-plugin-xine, waitress, whereami, xlsx2csv, zathura. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net A new package set for the X Strike Force has been added. (h01ger) Bugs tagged with locale are now visible in the statistics. (h01ger) Some work has been done add tests for NetBSD. (h01ger) Many changes by Mattia Rizzolo have been merged on the whole infrastructure: debbindiff development Version 26 has been released on June 28th fixing the comparison of files of unknown format. (Lunar) A missing dependency identified in python-rpm affecting debbindiff installation without recommended packages was promptly fixed by Michal iha . Lunar also started a massive code rearchitecture to enhance code reuse and enable new features. Nothing visible yet, though. Documentation update josch and Mattia Rizzolo documented how to reschedule packages from Alioth. Package reviews 142 obsolete reviews have been removed, 344 added and 107 updated this week. Chris West (Faux) filled 13 new bugs for packages failing to build from sources. The following new issues have been added: snapshot_placeholder_replaced_with_timestamp_in_pom_properties, different_encoding, timestamps_in_documentation_generated_by_org_mode and timestamps_in_pdf_generated_by_matplotlib.

Next.

Previous.