Search Results: "rmh"

6 June 2017

Reproducible builds folks: Reproducible Builds: week 110 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday May 28 and Saturday June 3 2017: Past an upcoming events Documentation updates Toolchain development and fixes Patches and bugs filed 4 package reviews have been added, 6 have been updated and 25 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development tests.reproducible-builds.org Mattia Rizzolo: Daniel Kahn Gillmor: Vagrant Cascadian: Holger Levsen: Misc. This week's edition was written by Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

23 May 2017

Reproducible builds folks: Reproducible Builds: week 108 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday May 14 and Saturday May 20 2017: News and Media coverage IRC meeting Our next IRC meeting has been scheduled for Thursday June 1 at 16:00 UTC. Packages reviewed and fixed, bugs filed, etc. Bernhard M. Wiedemann: Chris Lamb: Reviews of unreproducible packages 35 package reviews have been added, 28 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added: diffoscope development strip-nondeterminism development tests.reproducible-builds.org Holger wrote a new systemd-based scheduling system replacing 162 constantly running Jenkins jobs which were slowing down job execution in general: Misc. This week's edition was written by Chris Lamb, Holver Levsen, Bernhard M. Wiedemann, Vagrant Cascadian and Maria Glukhova & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Tianon Gravi: Debuerreotype

Following in the footsteps of one of my favorite Debian Developers, Chris Lamb / lamby (who is quite prolific in the reproducible builds effort within Debian), I ve started a new project based on snapshot.debian.org (time-based snapshots of the Debian archive) and some of lamby s work for creating reproducible Debian (debootstrap) rootfs tarballs. The project is named Debuerreotype as an homage to the photography roots of the word snapshot and the daguerreotype process which was an early method of taking photographs. The essential goal is to create photographs of a minimal Debian rootfs, so the name seemed appropriate (even if it s a bit on the mouthful side). The end-goal is to create and release Debian rootfs tarballs for a given point-in-time (especially for use in Docker) which should be fully reproducible, and thus improve confidence in the provenance of the Debian Docker base images. For more information about reproducibility and why it matters, see reproducible-builds.org, which has more thorough explanations of the why and how and links to other important work such as the reproducible builds effort in Debian (for Debian package builds). In order to verify that the tool actually works as intended, I ran builds against seven explicit architectures (amd64, arm64, armel, armhf, i386, ppc64el, s390x) and eight explicit suites (oldstable, stable, testing, unstable, wheezy, jessie, stretch, sid). I used a timestamp value of 2017-05-16T00:00:00Z, and skipped combinations that don t exist (such as wheezy on arm64) or aren t supported anymore (such as wheezy on s390x). I ran the scripts repeatedly over several days, using diffoscope to compare the results. While doing said testing, I ran across #857803, and added a workaround. There s also a minor outstanding issue with wheezy s reproducibility that I haven t had a chance to dig deep very deeply into yet (but it s pretty benign and Wheezy s LTS support window ends 2018-05-31, so I m not too stressed about it). I ve also packaged the tool for Debian, and submitted it into the NEW queue, so hopefully the FTP Masters will look favorably upon this being a tool that s available to install from the Debian archive as well. Anyhow, please give it a try, have fun, and as always, report bugs!

9 May 2017

Reproducible builds folks: Reproducible Builds: week 106 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday April 30 and Saturday May 6 2017: Past and upcoming events Between May 5th-7th the Reproducible Builds Hackathon 2017 took place in Hamburg, Germany. On May 6th Mattia Rizzolo gave a talk on Reproducible Builds at DUCC-IT 17 in Vicenza, Italy. On May 13th Chris Lamb will give a talk on Reproducible Builds at OSCAL 2017 in Tirana, Albania. Media coverage Toolchain development and fixes Packages reviewed and fixed, and bugs filed Chris Lamb: Reviews of unreproducible packages 93 package reviews have been added, 12 have been updated and 98 have been removed in this week, adding to our knowledge about identified issues. The following issues have been added: 2 issue types have been updated: The following issues have been removed: Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development strip-nondeterminism development
This week's edition was written by Chris Lamb, Holger Levsen and Ximin Luo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

6 May 2017

Dirk Eddelbuettel: x13binary 1.1.39-1

The US Census Bureau released a new build 1.1.39 of their X-13ARIMA-SEATS program, released as binary and source. So Christoph and went to work and updated our x13binary package on CRAN. The x13binary package takes the pain out of installing X-13ARIMA-SEATS by making it a fully resolved CRAN dependency. For example, if you install the excellent seasonal package by Christoph, then X-13ARIMA-SEATS will get pulled in via the x13binary package and things just work: Depend on x13binary and on all relevant OSs supported by R, you should have an X-13ARIMA-SEATS binary installed which will be called seamlessly by the higher-level packages such as seasonal or gunsales. So now the full power of the what is likely the world's most sophisticated deseasonalization and forecasting package is now at your fingertips and the R prompt, just like any other of the 10,500+ CRAN packages. Not many packaging changes in this release besides updating the underlying builds, but we switched our versioning scheme to reflect that our releases are driven by the US Census Bureau releases. But thanks to an initial contribution by David Schaub we now support the 'armhf' architecture common on Chromebooks running Linux. Courtesy of CRANberries, there is also a diffstat report for this release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

3 May 2017

Reproducible builds folks: Reproducible Builds: week 105 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday April 23 and Saturday April 29 2017: Past and upcoming events On April 26th Chris Lamb gave a talk at foss-north 2017 in Gothenburg, Sweden on Reproducible Builds. Between May 5th-7th the Reproducible Builds Hackathon 2017 will take place in Hamburg, Germany. Then on May 26th Bernhard M. Wiedemann will give a talk titled reproducible builds in openSUSE (2017) at the openSUSE Conference 2017 in N rnberg, Germany. Media coverage Already on April 19th Sylvain Beucler wrote a yet another follow-up post Practical basics of reproducible builds 3, after part 1 and part 2 of his series. Toolchain development and fixes Michael Woerister of the Rust project has implemented file maps that affect all path-related compiler information, including "error messages, metadata, debuginfo, and the file!() macro alike". Ximin Luo with support from some other Rust developers and contributors helped steer the final result into something that was compatible with reproducible builds. Many thanks to all involved, especially for the patience of discussing this over several months. Ximin wrote a first-attempt patch to fix R build-path issues. It made 460/477 R packages reproducible, but also caused 3 of these to FTBFS. See randomness_in_r_rdb_rds_databases for details. Bugs filed and patches sent upstream Chris Lamb: Bernhard M. Wiedemann filed a number of patches upstream: Reviews of unreproducible packages 102 package reviews have been added, 64 have been updated and 24 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated: Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development diffoscope 82 was uploaded to experimental by Chris Lamb. It included contributions from: Changes from previous weeks that were also released with 82: Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

11 April 2017

Reproducible builds folks: Reproducible Builds: week 102 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday April 2 and Saturday April 8 2017: Media coverage Toolchain development and fixes Reviews of unreproducible packages 27 package reviews have been added, 14 have been updated and 17 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

1 April 2017

Bits from Debian: Unknown parallel universe uses Debian

This post was an April Fools' Day joke. The space agencies running the International Space Station (ISS) reported that a laptop accidentally threw to space as waste in 2013 from the International State Station may have connected with a parallel Universe. This laptop was running Debian 6 and the ISS engineers managed to track its travel through the outer space. In early January, the laptop signal was lost but recovered back two weeks later in the same place. ISS engineers suspect that the laptop may had met and crossed a wormhole arriving a parallel Universe from where "somebody" sent it back later. Eventually the laptop was recovered and in an first analysis the ISS engineers found that the laptop have a dual boot: a partition running the Debian installation made by them and a second partition running what seems to be a Debian fork or derivative totally unknown until now. The engineers have been in contact with the Debian Project in the last weeks and a Debian group formed with delegates from different Debian teams have begun to study this new Debian derivative system. From the early results of this research, we can proudly say that somebody (or a group of beings) in a parallel universe understand Earth computers, and Debian, enough to: The work towards knowing better this new Universe and find a way to communicate with them has just began; all the Debian users and contributors are invited to join the effort to study the operating system found. We want to prepare our Community and our Universe to live and work peacefully and respectfully with the parallel Universe communities, in the true spirit of Free Software. In the following weeks a General Resolution will be proposed for updating our motto to "the multiversal operating system".

14 March 2017

Reproducible builds folks: Reproducible Builds: week 98 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday March 5 and Saturday March 11 2017: Upcoming events Reproducible Builds Hackathon Hamburg The Reproducible Builds Hamburg Hackathon 2017, or RB-HH-2017 for short, is a 3 day hacking event taking place in the CCC Hamburg Hackerspace located inside the Frappant, which is collective art space located in a historical monument in Hamburg, Germany. The aim of the hackathon is to spent some days working on Reproducible Builds in every distribution and project. The event is open to anybody interested on working on Reproducible Builds issues in any distro or project, with or without prio experience! Packages filed Chris Lamb: Toolchain development Reviews of unreproducible packages 39 package reviews have been added, 7 have been updated and 9 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added: Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: buildinfo.debian.net development reproducible-website development tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

10 March 2017

Reproducible builds folks: Reproducible Builds: week 97 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday February 26 and Saturday March 4 2017: Upcoming Events Ed Maste will present Reproducible Builds in FreeBSD at AsiaBSDCon 2017. Ximin Luo will present Reproducible builds, its uses and the future at Open Source Days in Copenhagen on March 18. Holger Levsen will give a talk at the German Unix User Group's "Fr hjahrsfachgespr ch" in Darmstadt, Germany, about Reproducible Builds everywhere on March 23. Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th. Media coverage Aspiration Tech published a very detailed report on our Reproducible Builds World Summit 2016 in Berlin. Reproducible work in other projects Duncan published a very thorough post on the Rust Programming Language Forum about reproducible builds in the Rust compiler and toolchain. In particular, he produced a table recording the reproducibility of different build products under different individual variations, totalling 187 build+variation combinations. Packages reviewed and fixed, and bugs filed Chris Lamb: Dhole: Reviews of unreproducible packages 60 package reviews have been added, 8 have been updated and 13 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been added: Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development diffoscope 78 was uploaded to unstable and jessie-backports by Mattia Rizzolo. It included contributions from: In addition, the following changes were made on the experimental branch: reproducible-website development tests.reproducible-builds.org Misc. This week's edition was written by Ximin Luo, Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

5 March 2017

Julien Viard de Galbert: Raspberry Pi 3 as desktop computer

For about six months I ve been using a Raspberry Pi 3 as my desktop computer at home. The overall experience is fine, but I had to do a few adjustments.
First was to use KeePass, the second to compile gcc for cross-compilation (ie use buildroot).
KeePass I m using KeePass + KeeFox to maintain my passwords on the various websites (and avoid reusing the same everywhere).
For this to work on the Raspberry Pi, one need to use mono from Xamarin: 
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb http://download.mono-project.com/repo/debian wheezy main"   sudo tee /etc/apt/sources.list.d/mono-xamarin.list
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install mono-runtime
The install instruction comes from mono-project and the initial pointer was found on raspberrypi forums, stackoverflow and Benny Michielsen s blog.
And for some plugin to work I think I had to apt-get install mono-complete. Compiling gcc Using the Raspberry Pi 3, I recovered an old project based on buildroot for the raspberry pi 2. And just for building the tool-chain I had a few issues. First the compilation would stop during mnp compilation: 
 /usr/bin/gcc -std=gnu99 -c -DHAVE_CONFIG_H -I. -I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_divrem_1 -O2 -Wa,--noexecstack tmp-divrem_1.s -fPIC -DPIC -o .libs/divrem_1.o
tmp-divrem_1.s: Assembler messages:
tmp-divrem_1.s:129: Error: selected processor does not support ARM mode  mls r1,r4,r8,r11'
tmp-divrem_1.s:145: Error: selected processor does not support ARM mode  mls r1,r4,r8,r11'
tmp-divrem_1.s:158: Error: selected processor does not support ARM mode  mls r1,r4,r8,r11'
tmp-divrem_1.s:175: Error: selected processor does not support ARM mode  mls r1,r4,r3,r8'
tmp-divrem_1.s:209: Error: selected processor does not support ARM mode  mls r11,r4,r12,r3'
Makefile:768: recipe for target 'divrem_1.lo' failed
make[]: *** [divrem_1.lo] Error 1
I Googled the error and found this post on the Raspberry Pi forum not really helpful
But I finally found an explanation on Jan Hrach s page on the subject.
The raspbian distribution is still optimized for the first Raspberry Pi so basically the compiler is limited to the old raspberypi instructions. While I was compiling gcc for a Raspberry Pi 2 so needed the extra ones. The proposed solution is to basically update raspbian to debian proper. While this is a neat idea, I still wanted to get some raspbian specific packages (like the kernel) but wanted to be sure that everything else comes from debian. So I did some apt pinning. First I experienced that pinning is not sufficient so when updating source.list with plain debian Jessie, make sure to add theses lines before the raspbian lines: 
# add official debian jessie (real armhf gcc)
deb http://ftp.fr.debian.org/debian/ jessie main contrib non-free
deb-src http://ftp.fr.debian.org/debian/ jessie main
deb http://security.debian.org/ jessie/updates main
deb-src http://security.debian.org/ jessie/updates main
deb http://ftp.fr.debian.org/debian/ jessie-updates main
deb-src http://ftp.fr.debian.org/debian/ jessie-updates main
Then run the following to get the debian gpg keys, but don t yet upgrade your system: 
apt update
apt install debian-archive-keyring
Now, let s add the pinning.
First if you were using APT::Default-Release "stable"; in your apt.conf (as I did) remove it. It does not mix well with fine grained pinning we will then implement. Then, fill your /etc/apt/preferences file with the following: 
# Debian
Package: *
Pin: release o=Debian,a=stable,n=jessie
Pin-Priority: 700
# Raspbian
Package: *
Pin: release o=Raspbian,a=stable,n=jessie
Pin-Priority: 600
Package: *
Pin: release o=Raspberry Pi Foundation,a=stable,n=jessie
Pin-Priority: 600
# Mono
Package: *
Pin: release v=7.0,o=Xamarin,a=stable,n=wheezy,l=Xamarin-Stable,c=main
Pin-Priority: 800
Note: You can use apt-cache policy (no parameter) to debug pinning.
The pinning above is mainly based on the origin field of the repositories (o=)
Finally you can upgrade your system: 
apt update 
apt-cache policy gcc 
rm /var/cache/apt/archives/* 
apt upgrade 
apt-cache policy gcc
Note: Removing the cache ensure we download the packages from debian as raspbian is using the exact same naming but we now they are not compiled with a real armhf tool-chain. Second issue with gcc The build stopped on recipe for target 's-attrtab' failed. There are many references on the web, that one was easy, it just need more memory, so I added some swap on the external SSD I was already using to work on buildroot. Conclusion That s it for today, not bad considering my last post was more that 3 years ago

21 February 2017

Reproducible builds folks: Reproducible Builds: week 95 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday February 12 and Saturday February 18 2017: Upcoming Events The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd. Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th. Toolchain development and fixes Ximin Luo posted a preliminary spec for BUILD_PATH_PREFIX_MAP, bringing together work and research from previous weeks. Ximin refactored and consolidated much of our existing documentation on both SOURCE_DATE_EPOCH and BUILD_PATH_PREFIX_MAP into one unified page, Standard Environment Variables, with extended discussion on related solutions and how these all fit into people's ideas of what reproducible builds should look like in the long term. The specific pages for each variable still remain, at Timestamps Proposal and Build Path Proposal, only without content that was previously duplicated on both pages. Ximin filed #855282 against devscripts for debsign(1) to support buildinfo files, and wrote an initial series of patches for it with some further additions from Guillem Jover. Packages reviewed and fixed, and bugs filed Chris Lamb: Reviews of unreproducible packages 35 package reviews have been added, 1 have been updated and 17 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been added: Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by: diffoscope development diffoscope 77 was uploaded to unstable by Mattia Rizzolo. It included contributions from: strip-nondeterminism development strip-nondeterminism 0.031-1 was uploaded to unstable by Chris Lamb. It included contributions from: strip-nondeterminism 0.031-1~bpo8+1 was uploaded to jessie-backports by Mattia. tests.reproducible-builds.org Misc. This week's edition was written by Ximin Luo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

11 February 2017

Reproducible builds folks: Reproducible Builds: week 93 in Stretch cycle

Here's what happened in the Reproducible Builds effort between Sunday January 29 and Saturday February 4 2017: Media coverage Dennis Gilmore and Holger Levsen presented "Reproducible Builds and Fedora" (Video, Slides) at Devconf.cz on February 27th 2017. On February 1st, stretch/armhf reached 90% reproducible packages in our test framework, so that now all four tested architectures are 90% reproducible in stretch. Yay! For armhf this means 22472 reproducible source packages (in main); for amd64, arm64 and i386 these figures are 23363, 23062 and 22607 respectively. Chris Lamb appeared on the Changelog podcast to talk about reproducible builds: Holger Levsen pitched Reproducible Builds and our need for a logo in the "Open Source Design" room at FOSDEM 2017 (Video, 09:36 - 12:00). Upcoming Events Reproducible work in other projects We learned that the "slightly more secure" Heads firmware (a Coreboot payload) is now reproducibly built regardless of host system or build directory. A picture says more than a thousand words: reproducible heads build on two machines Docker started preliminary work on making image builds reproducible. Toolchain development and fixes Ximin Luo continued to write code and test cases for the BUILD_PATH_PREFIX_MAP environment variable. He also did extensive research on cross-platform and cross-language issues with enviroment variables, filesystem paths, and character encodings, and started preparing a draft specification document to describe all of this. Chris Lamb asked CPython to implement an environment variable PYTHONREVERSEDICTKEYORDER to add an an option to reverse iteration order of items in a dict. However this was rejected because they are planning to formally fix this order in the next language version. Bernhard Wiedemann and Florian Festi added support for our SOURCE_DATE_EPOCH environment variable, to the RPM Package Manager. James McCoy uploaded devscripts 2.17.1 with a change from Guillem Jover for dscverify(1), adding support for .buildinfo files. (Closes: #852801) Piotr O arowski uploaded dh-python 2.20170125 with a change from Chris Lamb for a patch to fix #835805. Chris Lamb added documentation to diffoscope, strip-nondeterminism, disorderfs, reprotest and trydiffoscope about uploading signed tarballs when releasing. He also added a link to these on our website's tools page. Packages reviewed and bugs filed Bugs filed: Reviews of unreproducible packages 83 package reviews have been added, 86 have been updated and 276 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated: Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by: diffoscope development Work on the next version (71) continued in git this week: reproducible-website development Daniel Shahaf added more notes on our "How to chair a meeting" document. tests.reproducible-builds.org Holger unblacklisted pspp and tiledarray. If you think further packages should also be unblacklisted (possibly only on some architectures), please tell us. Misc. This week's edition was written by Ximin Luo, Holger Levsen and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

1 February 2017

Wouter Verhelst: Resetting a Raspberry Pi to default using Ansible

I got myself a bunch of Raspberry Pi 3s a short while ago: Stack of Raspberry Pis ...the idea being that I can use those to run a few experiments on. After the experiment, I would need to be able to somehow remove the stuff that I put on them again. The most obvious way to do that is to reimage them after every experiment; but the one downside of the multi-pi cases that I've put them in is that removing the micro-SD cards from the Pis without removing them from the case, while possible, is somewhat fiddly. That, plus the fact that I cared more about price than about speed when I got the micro-SD cards makes "image all the cards of these Raspberry Pis" something I don't want to do every day. So I needed a different way to reset them to a clean state, and I decided to use ansible to get it done. It required a bit of experimentation, but eventually I came up with this: 
---
- name: install package-list fact
  hosts: all
  remote_user: root
  tasks:
  - name: install libjson-perl
    apt:
      pkg: libjson-perl
      state: latest
  - name: create ansible directory
    file:
      path: /etc/ansible
      state: directory
  - name: create facts directory
    file:
      path: /etc/ansible/facts.d
      state: directory
  - name: copy fact into ansible facts directory
    copy:
      dest: /etc/ansible/facts.d/allowclean.fact
      src: allowclean.fact
      mode: 0755
- name: remove extra stuff
  hosts: pis
  remote_user: root
  tasks:
  - apt: pkg=  item   state=absent purge=yes
    with_items: "  ansible_local.allowclean.cleanable_packages  "
- name: remove package facts
  hosts: pis
  remote_user: root
  tasks:
  - file:
      path: /etc/ansible/facts.d
      state: absent
  - apt:
      pkg: libjson-perl
      state: absent
      autoremove: yes
      purge: yes
This ansible playbook has three plays. The first play installs a custom ansible fact (written in perl) that emits a json file which defines cleanable_packages as a list of packages to remove. The second uses that fact to actually remove those packages; and the third removes the fact (and the libjson-perl library we used to produce the JSON). Which means, really, that all the hard work is done inside the allowclean.fact file. That looks like this: 
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
open PACKAGES, "dpkg -l ";
my @packages;
my %whitelist = (
    ...
);
while(<PACKAGES>)  
    next unless /^ii\s+(\S+)/;
    my $pack = $1;
    next if(exists($whitelist $pack ));
    if($pack =~ /(.*):armhf/)  
        $pack = $1;
     
    push @packages, $1;
 
my %hash;
$hash cleanable_packages  = \@packages;
print to_json(\%hash);
Basically, we create a whitelist, and compare the output of dpkg -l against that whitelist. If a certain package is in the whitelist, then we don't add it to our "packages" list; if it isn't, we do. The whitelist is just a list of package => 1 tuples. We just need the hash keys and don't care about the data, really; I could equally well have made it package => undef, I suppose, but whatever. Creating the whitelist is not that hard. I did it by setting up one raspberry pi to the minimum that I wanted, running 
dpkg -l awk '/^ii/ print "\t\""$2"\" => 1," ' > packagelist
and then adding the contents of that packagelist file in place of the ... in the above perl script. Now whenever we apply the ansible playbook above, all packages (except for those in the whitelist) are removed from the system. Doing the same for things like users and files is left as an exercise to the reader. Side note: ... is a valid perl construct. You can write it, and the compiler won't complain. You can't run it, though.

11 January 2017

Reproducible builds folks: Reproducible Builds: week 89 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday January 1 and Saturday January 7 2017: GSoC and Outreachy updates Toolchain development Packages reviewed and fixed, and bugs filed Chris Lamb: Dhole: Reviews of unreproducible packages 13 package reviews have been added, 4 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added/updated: Upstreaming of reproducibility fixes Merged: Opened: Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by: diffoscope development diffoscope 67 was uploaded to unstable by Chris Lamb. It included contributions from : 
[ Chris Lamb ]
* Optimisations:
  - Avoid multiple iterations over archive by unpacking once for an ~8X
    runtime optimisation.
  - Avoid unnecessary splitting and interpolating for a ~20X optimisation
    when writing --text output.
  - Avoid expensive diff regex parsing until we need it, speeding up diff
    parsing by 2X.
  - Alias expensive Config() in diff parsing lookup for a 10% optimisation.
* Progress bar:
  - Show filenames, ELF sections, etc. in progress bar.
  - Emit JSON on the the status file descriptor output instead of a custom
    format.
* Logging:
  - Use more-Pythonic logging functions and output based on __name__, etc.
  - Use Debian-style "I:", "D:" log level format modifier.
  - Only print milliseconds in output, not microseconds.
  - Print version in debug output so that saved debug outputs can standalone
    as bug reports.
* Profiling:
  - Also report the total number of method calls, not just the total time.
  - Report on the total wall clock taken to execute diffoscope, including
    cleanup.
* Tidying:
  - Rename "NonExisting" -> "Missing".
  - Entirely rework diffoscope.comparators module, splitting as many separate
    concerns into a different utility package, tidying imports, etc.
  - Split diffoscope.difference into diffoscope.diff, etc.
  - Update file references in debian/copyright post module reorganisation.
  - Many other cleanups, etc.
* Misc:
  - Clarify comment regarding why we call python3(1) directly. Thanks to J r my
    Bobbio <lunar@debian.org>.
  - Raise a clearer error if trying to use --html-dir on a file.
  - Fix --output-empty when files are identical and no outputs specified.
[ Reiner Herrmann ]
* Extend .apk recognition regex to also match zip archives (Closes: #849638)
[ Mattia Rizzolo ]
* Follow the rename of the Debian package "python-jsbeautifier" to
  "jsbeautifier".
[ siamezzze ]
* Fixed no newline being classified as order-like difference.
reprotest development reprotest 0.5 was uploaded to unstable by Chris Lamb. It included contributions from: 
[ Ximin Luo ]
* Stop advertising variations that we're not actually varying.
  That is: domain_host, shell, user_group.
* Fix auto-presets in the case of a file in the current directory.
* Allow disabling build-path variations. (Closes: #833284)
* Add a faketime variation, with NO_FAKE_STAT=1 to avoid messing with
  various buildsystems. This is on by default; if it causes your builds
  to mess up please do file a bug report.
* Add a --store-dir option to save artifacts.
Other contributions (not yet uploaded): reproducible-builds.org website development tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen and Vagrant Cascadian, reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

3 January 2017

Reproducible builds folks: Reproducible Builds: week 88 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday December 25 and Saturday December 31 2016: Media coverage Reproducible bugs filed Chris West: Chris Lamb: Rob Browning: Reviews of unreproducible packages 7 package reviews have been added, 12 have been updated and 14 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated: Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by: diffoscope development strip-nondeterminism development try.diffoscope.org development tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen and was reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

1 January 2017

Joey Hess: p2p dreams

In one of the good parts of the very mixed bag that is "Lo and Behold: Reveries of the Connected World", Werner Herzog asks his interviewees what the Internet might dream of, if it could dream. The best answer he gets is along the lines of: The Internet of before dreamed a dream of the World Wide Web. It dreamed some nodes were servers, and some were clients. And that dream became current reality, because that's the essence of the Internet. Three years ago, it seemed like perhaps another dream was developing post-Snowden, of dissolving the distinction between clients and servers, connecting peer-to-peer using addresses that are also cryptographic public keys, so authentication and encryption and authorization are built in. Telehash is one hopeful attempt at this, others include snow, cjdns, i2p, etc. So far, none of them seem to have developed into a widely used network, although any of them still might get there. There are a lot of technical challenges due to the current Internet dream/nightmare, where the peers on the edges have multiple barriers to connecting to other peers. But, one project has developed something similar to the new dream, almost as a side effect of its main goals: Tor's onion services. I'd wanted to use such a thing in git-annex, for peer-to-peer sharing and syncing of git-annex repositories. On November 13th, I started building it, using Tor, and I'm releasing it concurrently with this blog post.
git-annex's Tor support replaces its old hack of tunneling git protocol over XMPP. That hack was unreliable (it needed a TCP on top of XMPP layer) but worse, the XMPP server could see all the data being transferred. And, there are fewer large XMPP servers these days, so fewer users could use it at all. If you were using XMPP with git-annex, you'll need to switch to either Tor or a server accessed via ssh.
Now git-annex can serve a repository as a Tor onion service, and that can then be accessed as a git remote, using an url like tor-annex::tungqmfb62z3qirc.onion:42913. All the regular git, and git-annex commands, can be used with such a remote. Tor has a lot of goals for protecting anonymity and privacy. But the important things for this project are just that it has end-to-end encryption, with addresses that are public keys, and allows P2P connections. Building an anonymous file exchange on top of Tor is not my goal -- if you want that, you probably don't want to be exchanging git histories that record every edit to the file and expose your real name by default. Building this was not without its difficulties. Tor onion services were originally intended to run hidden websites, not to connect peers to peers, and this kind of shows.. Tor does not cater to end users setting up lots of Onion services. Either root has to edit the torrc file, or the Tor control port can be used to ask it to set one up. But, the control port is not enabled by default, so you still need to su to root to enable it. Also, it's difficult to find a place to put the hidden service's unix socket file that's writable by a non-root user. So I had to code around this, with a git annex enable-tor that su's to root and sets it all up for you.
One interesting detail about the implementation of the P2P protocol in git-annex is that it uses two Free monads to build up actions. There's a Net monad which can be used to send and receive protocol messages, and a Local monad which allows only the necessary modifications to files on disk. Interpreters for Free monad actions can chose exactly which actions to allow for security reasons. For example, before a peer has authenticated, the P2P protocol is being run by an interpreter that refuses to run any Local actions whatsoever. Other interpreters for the Net monad could be used to support other network transports than Tor.
When two peers are connected over Tor, one knows it's talking to the owner of a particular onion address, but the other peer knows nothing about who's talking to it, by design. This makes authentication harder than it would be in a P2P system with a design like Telehash. So git-annex does its own authentication on top of Tor. With authentication, users would need to exchange absurdly long addresses (over 150 characters) to connect their repositories. One very convenient thing about using XMPP was that a user would have connections to their friend's accounts, so it was easy to share with them. Exchanging long addresses is too hard. This is where Magic Wormhole saved the day. It's a very elegant way to get any two peers in touch with each other, and the users only have to exchange a short code phrase, like "2-mango-delight", which can only be used once. Magic Wormhole makes some security tradeoffs for this simplicity. It's got vulnerabilities to DOS attacks, and its MITM resistance could be improved. But I'm lucky it came along just in time. So, it takes only installing Tor and Magic Wormhole, running two git-annex commands, and exchanging short code phrases with a friend, perhaps over the phone or in an encrypted email, to get your git-annex repositories connected and syncing over Tor. See the documentation for details. Also, the git-annex webapp allows setting the same thing up point-and-click style. The Tor project blog has throughout December been featuring all kinds of projects that are using Tor. Consider this a late bonus addition to that. ;) I hope that Tor onion services will continue to develop to make them easier to use for peer-to-peer systems. We can still dream a better Internet.
This work was made possible by all my supporters on Patreon.

30 December 2016

Antoine Beaupr : My free software activities, November and December 2016

Debian Long Term Support (LTS) Those were 8th and 9th months working on Debian LTS started by Raphael Hertzog at Freexian. I had trouble resuming work in November as I had taken a long break during the month and started looking at issues only during the last week of November.

Imagemagick, again I have, again, spent a significant amount of time fighting the ImageMagick (IM) codebase. About 15 more vulnerabilities were found since the last upload, which resulted in DLA-756-1. In the advisory, I unfortunately forgot to mention CVE-2016-8677 and CVE-2016-9559, something that was noticed by my colleague Roberto after the upload... More details about the upload are available in the announcement. When you consider that I worked on IM back in october, which lead to an upload near the end of November covering around 80 more vulnerabilities, it doesn't look good for the project at all. Of the 15 vulnerabilities I worked on, only 6 had CVEs assigned and I had to request CVEs for the other 9 vulnerabilities plus 11 more that were still unassigned. This lead to the assignment of 25 distinct CVE identifiers as a lot of issues were found to be distinct enough to warrant their own CVEs. One could also question how many of those issues affect the fork, Graphicsmagick. A lot of the vulnerabilities were found through fuzzing searches that may not have been tested on Graphicsmagick. It seems clear to me that a public corpus of test data should be available to test regressions and cross-project vulnerabilities. It's already hard enough to track issues withing IM itself, I can't imagine what it would be for the fork to keep track of those issues, especially since upstream doesn't systematically request CVEs for issues that they find, a questionable practice considering the number of issues we all need to keep track of.

Nagios I have also worked on the Nagios package and produced DLA 751-1 which fixed two fairly major issues (CVE-2016-9565 and CVE-2016-9566) that could allow remote root access under certain conditions. Fortunately, the restricted permissions setup by default in the Debian package made both exploits limited to information disclosure and privilege escalation if the debug log is enabled. This says a lot about how proper Debian packaging can help in limiting the attack surface of certain vulnerabilities. It was also "interesting" to have to re-learn dpatch to add patches to the package: I regret not converting it to quilt, as the operation is simple and quilt is so much easier to use. People new to Debian packaging may be curious to learn about the staggering number of patching systems historically used in Debian. On that topic, I started a conversation about how much we want to reuse existing frameworks when we work on those odd packages, and the feedback was interesting. Basically, the answer is "it depends"...

NSS I had already worked on the package in November and continued the work in December. Most of the work was done by Raphael, which fixed a lot of issues with the test suite. I tried to wrap this up by fixing CVE-2016-9074, the build on armel and the test suite. Unfortunately, I had to stop again because I ran out of hours and the fips test suite was still failing, but fortunately Raphael was able to complete the work with DLA-759-1. As things stand now, the package is in better shape than in other suites as tests (Debian bug #806639) and autopkgtest (Debian bug #806207) are still not shipped in the sid or stable releases.

Other work For the second time, I forgot to formally assign myself a package before working on it, which meant that I wasted part of my hours working on the monit package. Those hours, of course, were not counted in my regular hours. I still spent some time reviewing mejo's patch to ensure it was done properly and it turned out we both made similar patches working independently, always a good sign. As I reported in my preliminary November report, I have also triaged issues in libxml2, ntp, openssl and tiff. Finally, I should mention my short review of the phpMyAdmin upload, among the many posts i sent to the LTS mailing list.

Other free software work One reason why I had so much trouble getting paid work done in November is that I was busy with unpaid work...

manpages.debian.org A major time hole for me was trying to tackle the manpages.debian.org service, which had been offline since August. After a thorough evaluation of the available codebases, I figured the problem space wasn't so hard and it was worth trying to do an implementation from scratch. The result is a tool called debmans. It took, obviously, way longer than I expected, as I experimented with Python libraries I had been keeping an eye on for a while. For the commanline interface, I used the click library, which is really a breeze to use, but a bit heavy for smaller scripts. For a web search service prototype, I looked at flask, which was also very interesting, as it is light and simple enough to use that I could get started quickly. It also, surprisingly, fares pretty well in the global TechEmpower benchmarking tests. Those interested in those tools may want to look at the source code, in particular the main command (using an interesting pattern itself, __main__.py) and the search prototype. Debmans is the first project for which I have tried the CII Best Practices Badge program, an interesting questionnaire to review best practices in software engineering. It is an excellent checklist I recommend every project manager and programmer to get familiar with. I still need to complete my work on Debmans: as I write this, I couldn't get access to the new server the DSA team setup for this purpose. It was a bit of a frustrating experience to wait for all the bits to get into place while I had a product ready to test. In the end, the existing manpages.d.o maintainer decided to deploy the existing codebase on the new server while the necessary dependencies are installed and accesses are granted. There's obviously still a bunch of work to be done for this to be running in production so I have postponed all this work to January. My hope is that this tool can be reused by other distributions, but after talking with Ubuntu folks, I am not holding my breath: it seems everyone has something that is "good enough" and that they don't want to break it...

Monkeysign I spent a good chunk of time giving a kick in the Monkeysign project, with the 2.2.2 release, which features contributions from two other developers, which may be a record for a single release. I am especially happy to have adopted a new code of conduct - it has been an interesting process to adapt the code of conduct for such a relatively small project. Monkeysign is becoming a bit of a template on how to do things properly for my Python projects: documentation on readthedocs.org including a code of conduct, support and contribution information, and so on. Even though the code now looks a bit old to me and I am embarrassed to read certain parts, I still think it is a solid project that is useful for a lot of people. I would love to have more time to spend on it.

LWN publishing As you may have noticed if you follow this blog, I have started publishing articles for the LWN magazine, filed here under the lwn tag. It is a way for me to actually get paid for some of my blogging work that used to be done for free. Reports like this one, for example, take up a significant amount of my time and are done without being paid. Converting parts of this work into paid work is part of my recent effort to reduce the amount of time I spend on the computer. An funny note: I always found the layout of the site to be a bit odd, until I looked at my articles posted there in a different web browser, which didn't have my normal ad blocker configuration. It turns out LWN uses ads, and Google ones too, which surprised me. I definitely didn't want to publish my work under banner ads, and will never do so on this blog. But it seems fair that, since I get paid for this work, there is some sort of revenue stream associated with it. If you prefer to see my work without ads, you can wait for it to be published here or become a subscriber which allows you to get rid of the ads on the site. My experience with LWN is great: they're great folks, and very supportive. It's my first experience with a real editor and it really pushed me in improving my writing to make better articles that I normally would here. Thanks to the LWN folks for their support! Expect more of those quality articles in 2017.

Debian packaging I have added a few packages to the Debian archive:
  • magic-wormhole: easy file-transfer tool, co-maintained with Jamie Rollins
  • slop: screenshot tool
  • xininfo: utility used by teiler
  • teiler (currently in NEW): GUI for screenshot and screencast tools
I have also updated sopel and atheme-services.

Other work Against my better judgment, I worked again on the borg project. This time I tried to improve the documentation, after a friend asked me for help on "how to make a quick backup". I realized I didn't have any good primer to send regular, non-sysadmin users to and figured that, instead of writing a new one, I could improve the upstream documentation instead. I generated a surprising 18 commits of documentation during that time, mainly to fix display issues and streamline the documentation. My final attempt at refactoring the docs eventually failed, unfortunately, again reminding me of the difficulty I have in collaborating on that project. I am not sure I succeeded in making the project more attractive to non-technical users, but maybe that's okay too: borg is a fairly advanced project and not currently aimed at such a public. This is yet another project I am thinking of creating: a metabackup program like backupninja that would implement the vision created by liw in his A vision for backups in Debian post, which was discarded by the Borg project. Github also tells me that I have opened 19 issues in 14 different repositories in November. I would like to particularly bring your attention to the linkchecker project which seems to be dead upstream and for which I am looking for collaborators in order to create a healthy fork. Finally, I started on reviving the stressant project and changing all my passwords, stay tuned for more!

21 November 2016

Reproducible builds folks: Reproducible Builds: week 82 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 13 and Saturday November 19 2016: Media coverage Elsewhere in Debian Documentation update Packages reviewed and fixed, and bugs filed Reviews of unreproducible packages 43 package reviews have been added, 4 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated: 4 issue types have been added: Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by: strip-nondeterminism development disorderfs development debrebuild development debrebuild is new tool proposed by HW42 and josch (see #774415: "From srebuild sbuild-wrapper to debrebuild"). debrepatch development debrepatch is a set of scripts that we're currently developing to make it easier to track unapplied patches. We have a lot of those and we're not always sure if they still work. The plan is to set up jobs to automatically apply old reproducibility patches to newer versions of packages and notify the right people if they don't apply and/or no longer make the package reproducible. debpatch is a component of debrepatch that applies debdiffs to Debian source packages. In other words, it is to debdiff(1) what patch(1) is to diff(1). It is a general tool that is not specific to Reproducible Builds. This week, Ximin Luo worked on making it more "production-ready" and will soon submit it for inclusion in devscripts. reprotest development Ximin Luo significantly improved reprotest, adding presets and auto-detection of which preset to use. One can now run e.g. reprotest auto . or reprotest auto $pkg_$ver.dsc instead of the long command lines that were needed before. He also made it easier to set up build dependencies inside the virtual server and made it possible to specify pre-build dependencies that reprotest itself needs to set up the variations. Previously one had to manually edit the virtual server to do that, which was not very usable to humans without an in-depth knowledge of the building process. These changes will be tested some more and then released in the near future as reprotest 0.4. tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen, Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

17 November 2016

Uwe Kleine-K nig: Installing Debian Stretch on a Turris Omnia

Recently I got "my" Turris Omnia and it didn't take long to replace the original firmware with Debian. If you want to reproduce, here is what you have to do: Open the case of the Turris Omnia, connect the hacker pack (or an RS232-to-TTL adapter) to access the U-Boot prompt (see Turris Omnia: How to use the "Hacker pack"). Then download the installer and device tree: 
# cd /srv/tftp
# wget https://d-i.debian.org/daily-images/armhf/daily/netboot/vmlinuz
# wget https://d-i.debian.org/daily-images/armhf/daily/netboot/initrd.gz
# wget https://www.kleine-koenig.org/tmp/armada-385-turris-omnia.dtb
(The latter is not included yet in Debian, but I'm working on that.) and after connecting the Turris Omnia's WAN to a dhcp managed network start it in U-Boot: 
dhcp
setenv serverip 192.168.1.17
tftpboot 0x01000000 vmlinuz
tftpboot 0x02000000 armada-385-turris-omnia.dtb
tftpboot 0x03000000 initrd.gz
bootz 0x01000000 0x03000000:$filesize 0x02000000
With 192.168.1.17 being the IPv4 of the machine you have the tftp server running. I suggest to use btrfs as rootfs because that works well with U-Boot. Before finishing the installation put the dtb in the rootfs as /boot/dtb. To then boot into Debian do in U-Boot: 
setenv mmcboot=btrload mmc 0 0x01000000 boot/vmlinuz\; btrload mmc 0 0x02000000 boot/dtb\; btrload mmc 0 0x03000000 boot/initrd.img\; bootz 0x01000000 0x03000000:$filesize 0x02000000
setenv bootargs console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=/dev/mmcblk0p1 rootflags=commit=5 rw
saveenv
boot
Known issues: If you have problems, don't hesitate to contact me. Also check the Debian Wiki for further details.

Next.

Previous.