Search Results: "josch"

02 February 2017

Paul Wise: FLOSS Activities January 2017

Changes

Issues

Review

Administration
  • Debian: reboot 1 non-responsive VM, redirect 2 users to support channels, redirect 1 contributor to xkb upstream, redirect 1 potential contributor, redirect 1 bug reporter to mirror team, ping 7 folks about restarting processes with upgraded libs, manually restart the sectracker process due to upgraded libs, restart the package tracker process due to upgraded libs, investigate failures connecting to the XMPP service, investigate /dev/shm issue on abel.d.o, clean up after rename of the fedmsg group.
  • Debian mentors: lintian/security updates & reboot
  • Debian packages: deploy 2 contributions to the live server
  • Debian wiki: unblacklist 1 IP address, whitelist 10 email addresses, disable 18 accounts with bouncing email, update email for 2 accounts with bouncing email, reported 1 Debian member as MIA, redirect 1 user to support channels, add 4 domains to the whitelist.
  • Reproducible builds: rescheduled Debian pyxplot:amd64/unstable for themill.
  • Openmoko: security updates & reboots.

Debian derivatives
  • Send the annual activity ping mail.
  • Happy new year messages on IRC, forward to the list.
  • Note that SerbianLinux does not provide source packages.
  • Expand URL shortener on SerbianLinux page.
  • Invite PelicanHPC, Netrunner, DietPi, Hamara Linux (on IRC), BitKey to the census.
  • Add research publications link to the census template
  • Fix Symbiosis sources.list
  • Enquired about SalentOS downtime
  • Fixed and removed some 404 BlankOn links (blog, English homepage)
  • Fixed changes to AstraLinux sources.list
  • Welcome Netrunner to the census

Sponsors I renewed my support of Software Freedom Conservancy. The openchange 1:2.2-6+deb8u1 upload was sponsored by my employer. All other work was done on a volunteer basis.

18 December 2016

Johannes Schauer: Looking for self-hosted filesharing software

The owncloud package was removed from Debian unstable and testing. I am thus now looking for an alternative. Unfortunately, finding such replacement seems to be harder than I initially thought, even though I only use a very small subset of what owncloud provides. What I require is some software which allows me to:
  1. upload a directory of files of any type to my server (no "distributed" filesharing where I have to stay online with my laptop)
  2. share the content of that directory via HTTP (no requirement to install any additional software other than a web browser)
  3. let the share-links be private (no possibility to infer the location of other shares)
  4. allow users to browse that directory (image thumbnails or a photo gallery would be nice)
  5. allow me to allow anonymous users to upload their own content into that directory (also only requiring their web browser)
  6. already in Debian or easy to package and maintain due to low complexity (I don't have enough time to become the next "owncloud maintainer")
I thought this was a pretty simple task to solve but I am unable to find any software that fits above criteria. The below table shows the result of my research of what's currently available. The columns mark whether the respective software fulfills one of the six criteria from above.
Software 123456
owncloud
sparkleshare
dvcs-autosync
git annex assistant
syncthing
pydio
seafile
sandstorm.io
ipfs
bozon
droppy
Pydio, seafile and sandstorm.io look promising but they seem to be beasts similar in complexity to owncloud as they bring features like version tracking, office integration, wikis, synchronization across multiple devices or online editing of files which are features that I do not need. I would already be very happy if there was a script which would make it easy to create a hard-to-guess symlink to a directory with data tracked by git annex under my www-root and then generate some static HTML to provide a thumbnails view or a photo gallery. Unfortunately, even that solution would not be sufficient as it would still disallow public upload by anybody whom I would give the link to... If you know some software that meets my criteria or would like to submit corrections to above table, please shoot an email to josch@debian.org. Thanks!

21 November 2016

Reproducible builds folks: Reproducible Builds: week 82 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 13 and Saturday November 19 2016: Media coverage Elsewhere in Debian Documentation update Packages reviewed and fixed, and bugs filed Reviews of unreproducible packages 43 package reviews have been added, 4 have been updated and 12 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated: 4 issue types have been added: Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by: strip-nondeterminism development disorderfs development debrebuild development debrebuild is new tool proposed by HW42 and josch (see #774415: "From srebuild sbuild-wrapper to debrebuild"). debrepatch development debrepatch is a set of scripts that we're currently developing to make it easier to track unapplied patches. We have a lot of those and we're not always sure if they still work. The plan is to set up jobs to automatically apply old reproducibility patches to newer versions of packages and notify the right people if they don't apply and/or no longer make the package reproducible. debpatch is a component of debrepatch that applies debdiffs to Debian source packages. In other words, it is to debdiff(1) what patch(1) is to diff(1). It is a general tool that is not specific to Reproducible Builds. This week, Ximin Luo worked on making it more "production-ready" and will soon submit it for inclusion in devscripts. reprotest development Ximin Luo significantly improved reprotest, adding presets and auto-detection of which preset to use. One can now run e.g. reprotest auto . or reprotest auto $pkg_$ver.dsc instead of the long command lines that were needed before. He also made it easier to set up build dependencies inside the virtual server and made it possible to specify pre-build dependencies that reprotest itself needs to set up the variations. Previously one had to manually edit the virtual server to do that, which was not very usable to humans without an in-depth knowledge of the building process. These changes will be tested some more and then released in the near future as reprotest 0.4. tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen, Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

14 February 2016

Lunar: Reproducible builds: week 42 in Stretch cycle

What happened in the reproducible builds effort between February 7th and February 13th 2016:

Toolchain fixes
  • James McCoy uploaded devscripts/2.16.1 which makes dcmd supports .buildinfo files. Original patch by josch.
  • Lisandro Dami n Nicanor P rez Meyer uploaded qt4-x11/4:4.8.7+dfsg-6 which make files created by qch reproducible by using a fixed date instead of the current time. Original patch by Dhole.
Norbert Preining rejected the patch submitted by Reiner Herrmann to make the CreationDate not appear in comments of DVI / PS files produced by TeX. He also mentioned that some timestamps can be replaced by using the -output-comment option and that the next version of pdftex will have patches inspired by reproducible build to mitigate the effects (see SOURCE_DATE_EPOCH patches) .

Packages fixed The following packages have become reproducible due to changes in their build dependencies: abntex, apt-dpkg-ref, arduino, c++-annotations, cfi, chaksem, clif, cppreference-doc, dejagnu, derivations, ecasound, fdutils, gnash, gnu-standards, gnuift, gsequencer, gss, gstreamer0.10, gstreamer1.0, harden-doc, haskell98-report, iproute2, java-policy, libbluray, libmodbus, lizardfs, mclibs, moon-buggy, nurpawiki, php-sasl, shishi, stealth, xmltex, xsom. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet:
  • #813944 on cvm by Reiner Herrmann: remove gzip headers, fix permissions of some directories and the order of the md5sums.
  • #814019 on latexdiff by Reiner Herrmann: remove the current build date from documentation.
  • #814214 on rocksdb by Chris Lamb: add support for SOURCE_DATE_EPOCH.

reproducible.debian.net A new armhf build node has been added (thanks to Vagrant Cascadian) and integrated into the Jenkins setup for 4 new armhf builder jobs. (h01ger) All packages for Debian testing (Stretch) have been tested on armhf in just 42 days. It took 114 days to get the same point for unstable back when the armhf test infrastructure was much smaller. Package sets have been enabled for testing on armhf. (h01ger) Packages producing architecture-independent ( Arch:all ) binary packages together with architecture dependent packages targeted for specific architectures will now only be tested on matching architectures. (Steven Chamberlain, h01ger) As the Jenkins setup is now made of 252 different jobs, the overview has been split into 11 different smalller views. (h01ger)

Package reviews 222 reviews have been removed, 110 added and 50 updated in the previous week. 35 FTBFS reports were made by Chris Lamb, Danny Edel, and Niko Tyni.

Misc. The recordings of Ludovic Court s' talk at FOSDEM 16 about reproducible builds and GNU Guix is now available. One can also have a look at slides from Fabian Keil's talk about ElecrtroBSD and Baptiste Daroussin's talk about FreeBSD packages.

04 November 2015

Johannes Schauer: Let's Encrypt with Pound on Debian

TLDR: mister-muffin.de (and all its subdomains), bootstrap.debian.net and binarycontrol.debian.net are now finally signed by "Let's Encrypt Authority X1" \o/ I just tried out the letsencrypt client Debian packages prepared by Harlan Lieberman-Berg which can be found here: My server setup uses Pound as a reverse proxy in front of a number of LXC based containers running the actual services. Furthermore, letsencrypt only supports Nginx and Apache for now, so I had to manually setup things anyways. Here is how. After installing the Debian packages I built from above git repositories, I ran the following commands:
$ mkdir -p letsencrypt/etc letsencrypt/lib letsencrypt/log
$ letsencrypt certonly --authenticator manual --agree-dev-preview \
    --server https://acme-v01.api.letsencrypt.org/directory --text \
    --config-dir letsencrypt/etc --logs-dir letsencrypt/log \
    --work-dir letsencrypt/lib --email josch@mister-muffin.de \
    --domains mister-muffin.de --domains blog.mister-muffin.de \
    --domains [...]
I created the letsencrypt directory structure to be able to run letsencrypt as a normal user. Otherwise, running this command would require access to /etc/letsencrypt and others. Having to set this up and pass all these parameters is a bit bothersome but there is an upstream issue about making this easier when using the "certonly" option which in princible should not require superuser privileges. The --server option is necessary for now because "Let's Encrypt" is still in beta and one needs to register for it. Without the --server option one will get an untrusted certificate from the "happy hacker fake CA". The letsencrypt program will then ask me for my agreement to the Terms of Service and then, for each domain I specified with the --domains option present me the token content and the location under each domain where it expects to find this content, respectively. This looks like this each time:
-------------------------------------------------------------------------------
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running letsencrypt in manual mode on a machine that is
not your server, please ensure you're okay with that.
Are you OK with your IP being logged?
-------------------------------------------------------------------------------
(Y)es/(N)o: Y
Make sure your web server displays the following content at
http://mister-muffin.de/.well-known/acme-challenge/XXXX before continuing:
 "header":  "alg": "RS256", "jwk":  "e": "AQAB", "kty": "RSA", "n": "YYYY" , "payload": "ZZZZ", "signature": "QQQQ" 
Content-Type header MUST be set to application/jose+json.
If you don't have HTTP server configured, you can run the following
command on the target server (as root):
mkdir -p /tmp/letsencrypt/public_html/.well-known/acme-challenge
cd /tmp/letsencrypt/public_html
echo -n ' "header":  "alg": "RS256", "jwk":  "e": "AQAB", "kty": "RSA", "n": "YYYY" , "payload": "ZZZZ", "signature": "QQQQ" ' > .well-known/acme-challenge/XXXX
# run only once per server:
$(command -v python2   command -v python2.7   command -v python2.6) -c \
"import BaseHTTPServer, SimpleHTTPServer; \
SimpleHTTPServer.SimpleHTTPRequestHandler.extensions_map =  '': 'application/jose+json' ; \
s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \
s.serve_forever()" 
Press ENTER to continue
For brevity I replaced any large base64 encoded chunks of the messages with YYYY, ZZZZ and QQQQ. The token location is abbreviated with XXXX. After temporarily stopping Pound on my webserver I created the directory /tmp/letsencrypt/public_html/.well-known/acme-challenge and then opened two shells on my server, both at /tmp/letsencrypt/public_html. In one, I kept a tiny HTTP server running (like the suggested Python SimpleHTTPServer which will also work if one has Python installed). In the other I copy pasted the echo line that the letsencrypt program suggested me to run. I had to copypaste that echo command for each domain I wanted to verify. This could easily be automated, so I filed an issue about this with upstream. It seems that the letsencrypt servers query each of these tokens twice: once directly each time after having hit enter after seeing the message above and another time once all tokens are in place. At the end of this ordeal I get:
2015-11-04 11:12:18,409:WARNING:letsencrypt.client:Non-standard path(s), might not work with crontab installed by your operating system package manager
IMPORTANT NOTES:
 - If you lose your account credentials, you can recover through
   e-mails sent to josch@mister-muffin.de.
 - Congratulations! Your certificate and chain have been saved at
   letsencrypt/etc/live/mister-muffin.de/fullchain.pem. Your cert will
   expire on 2016-02-02. To obtain a new version of the certificate in
   the future, simply run Let's Encrypt again.
 - Your account credentials have been saved in your Let's Encrypt
   configuration directory at letsencrypt/etc. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Let's
   Encrypt so making regular backups of this folder is ideal.
I can now scp the content of letsencrypt/etc/live/mister-muffin.de/* to my server. Unfortunately, Pound (and also my ejabberd XMPP server) requires the private key to be in the same file as the certificate and the chain, so on the server I also had to do:
cat /etc/ssl/private/privkey.pem /etc/ssl/private/fullchain.pem > /etc/ssl/private/private_fullchain.pem
And edit the Pound config to use /etc/ssl/private/private_fullchain.pem. But that's all, folks! EDIT It seems that manually copying over the echo commands as I described above is not necessary. Instead of using the certonly plugin, I can use the webroot plugin. That plugin takes the --webroot-path option and will copy the tokens to there. Since my webroot is on a remote machine, I could just mount it locally via sshfs and pass the mountpoint as --webroot-path. That I didn't realize that the webroot plugin does what I want (and not the certonly plugin) can easily be explained by the only documentation of the webroot plugin in the help output and the man page generated from it being "Webroot Authenticator" which is not very helpful. Another user seems to have run into similar problems. Better documenting the plugins so that these situations can be prevented in the future is tracked in this upstream bug.

25 October 2015

Johannes Schauer: unshare without superuser privileges

TLDR: With the help of Helmut Grohne I finally figured out most of the bits necessary to unshare everything without becoming root (though one might say that this is still cheated because the suid root tools newuidmap and newgidmap are used). I wrote a Perl script which documents how this is done in practice. This script is nearly equivalent to using the existing commands lxc-usernsexec [opts] -- unshare [opts] -- COMMAND except that these two together cannot be used to mount a new proc. Apart from this problem, this Perl script might also be useful by itself because it is architecture independent and easily inspectable for the curious mind without resorting to sources.debian.net (it is heavily documented at nearly 2 lines of comments per line of code on average). It can be retrieved here at https://gitlab.mister-muffin.de/josch/user-unshare/blob/master/user-unshare Long story: Nearly two years after my last last rant about everything needing superuser privileges in Linux, I'm still interested in techniques that let me do more things without becoming root. Helmut Grohne had told me for a while about unshare(), or user namespaces as the right way to have things like chroot without root. There are also reports of LXC containers working without root privileges but they are hard to come by. A couple of days ago I had some time again, so Helmut helped me to get through the major blockers that were so far stopping me from using unshare in a meaningful way without executing everything with sudo. My main motivation at that point was to let dpkg-buildpackage when executed by sbuild be run with an unshared network namespace and thus without network access (except for the loopback interface) because like pbuilder I wanted sbuild to enforce the rule not to access any remote resources during the build. After several evenings of investigating and doctoring at the Perl script I mentioned initially, I came to the conclusion that the only place that can unshare the network namespace without disrupting anything is schroot itself. This is because unsharing inside the chroot will fail because dpkg-buildpackage is run with non-root privileges and thus the user namespace has to be unshared. But this then will destroy all ownership information. But even if that wasn't the case, the chroot itself is unlikely to have (and also should not) tools like ip or newuidmap and newgidmap installed. Unsharing the schroot call itself also will not work. Again we first need to unshare the user namespace and then schroot will complain about wrong ownership of its configuration file /etc/schroot/schroot.conf. Luckily, when contacting Roger Leigh about this wishlist feature in bug#802849 I was told that this was already implemented in its git master \o/. So this particular problem seems to be taken care of and once the next schroot release happens, sbuild will make use of it and have unshare --net capabilities just like pbuilder already had since last year. With the sbuild case taken care of, the rest of this post will introduce the Perl script I wrote. The name user-unshare is really arbitrary. I just needed some identifier for the git repository and a filename. The most important discovery I made was, that Debian disables unprivileged user namespaces by default with the patch add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by-default.patch to the Linux kernel. To enable it, one has to first either do
echo 1   sudo tee /proc/sys/kernel/unprivileged_userns_clone > /dev/null
or
sudo sysctl -w kernel.unprivileged_userns_clone=1
The tool tries to be like unshare(1) but with the power of lxc-usernsexec(1) to map more than one id into the new user namespace by using the programs newgidmap and newuidmap. Or in other words: This tool tries to be like lxc-usernsexec(1) but with the power of unshare(1) to unshare more than just the user and mount namespaces. It is nearly equal to calling:
lxc-usernsexec [opts] -- unshare [opts] -- COMMAND
Its main reason of existence are: I hoped that systemd-nspawn could do what I wanted but it seems that its requirement for being run as root will not change any time soon Another tool in Debian that offers to do chroot without superuser privileges is linux-user-chroot but that one cheats by being suid root. Had I found lxc-usernsexec earlier I would've probably not written this. But after I found it I happily used it to get an even better understanding of the matter and further improve the comments in my code. I started writing my own tool in Perl because that's the language sbuild was written in and as mentioned initially, I intended to use this script with sbuild. Now that the sbuild problem is taken care of, this is not so important anymore but I like if I can read the code of simple programs I run directly from /usr/bin without having to retrieve the source code first or use sources.debian.net. The only thing I wasn't able to figure out is how to properly mount proc into my new mount namespace. I found a workaround that works by first mounting a new proc to /proc and then bind-mounting /proc to whatever new location for proc is requested. I didn't figure out how to do this without mounting to /proc first partly also because this doesn't work at all when using lxc-usernsexec and unshare together. In this respect, this perl script is a bit more powerful than those two tools together. I suppose that the reason is that unshare wasn't written with having being called without superuser privileges in mind. If you have an idea what could be wrong, the code has a big FIXME about this issue. Finally, here a demonstration of what my script can do. Because of the /proc bug, lxc-usernsexec and unshare together are not able to do this but it might also be that I'm just not using these tools in the right way. The following will give you an interactive shell in an environment created from one of my sbuild chroot tarballs:
$ mkdir -p /tmp/buildroot/proc
$ ./user-unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net \
    --uts --mount --fork -- sh -c 'ip link set lo up && ip addr && \
    hostname hoothoot-chroot && \
    tar -C /tmp/buildroot -xf /srv/chroot/unstable-amd64.tar.gz; \
    /usr/sbin/chroot /tmp/buildroot /sbin/runuser -s /bin/bash - josch && \
    umount /tmp/buildroot/proc && rm -rf /tmp/buildroot'
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ whoami
josch
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ hostname
hoothoot-chroot
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ ls -lha /proc   head
total 0
dr-xr-xr-x 218 nobody nogroup    0 Oct 25 19:06 .
drwxr-xr-x  22 root   root     440 Oct  1 08:42 ..
dr-xr-xr-x   9 root   root       0 Oct 25 19:06 1
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 15
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 16
dr-xr-xr-x   9 root   root       0 Oct 25 19:06 7
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 8
dr-xr-xr-x   4 nobody nogroup    0 Oct 25 19:06 acpi
dr-xr-xr-x   6 nobody nogroup    0 Oct 25 19:06 asound
Of course instead of running this long command we can also instead write a small shell script and execute that instead. The following does the same things as the long command above but adds some comments for further explanation:
#!/bin/sh

set -exu

# I'm using /tmp because I have it mounted as a tmpfs
rootdir="/tmp/buildroot"

# bring the loopback interface up
ip link set lo up

# show that the loopback interface is really up
ip addr

# make use of the UTS namespace being unshared
hostname hoothoot-chroot

# extract the chroot tarball. This must be done inside the user namespace for
# the file permissions to be correct.
#
# tar will fail to call mknod and to change the permissions of /proc but we are
# ignoring that
tar -C "$rootdir" -xf /srv/chroot/unstable-amd64.tar.gz true

# run chroot and inside, immediately drop permissions to the user "josch" and
# start an interactive shell
/usr/sbin/chroot "$rootdir" /sbin/runuser -s /bin/bash - josch

# unmount /proc and remove the temporary directory
umount "$rootdir/proc"
rm -rf "$rootdir"
and then:
$ mkdir -p /tmp/buildroot/proc
$ ./user-unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net --uts --mount --fork -- ./chroot.sh
As mentioned in the beginning, the tool is nearly equivalent to calling lxc-usernsexec [opts] -- unshare [opts] -- COMMAND but because of the problem with mounting proc (mentioned earlier), lxc-usernsexec and unshare cannot be used with above example. If one tries anyways one will only get:
$ lxc-usernsexec -m b:0:1000:1 -m b:1:558752:1 -- unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net --uts --mount --fork -- ./chroot.sh
unshare: mount /tmp/buildroot/proc failed: Invalid argument
I'd be interested in finding out why that is and how to fix it.

14 October 2015

Lunar: Reproducible builds: week 24 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Scott Kitterman fixed an issue with non-deterministic Depends generated by dh-python identified by Santiago Vila and Chris Lamb. Lunar updated the patch against dpkg which makes the order of files in control.tar.gz deterministic using the new --sort=name option available in GNU Tar 1.28. josch released sbuild version 0.66.0-1 with several fixes and improvements. The most notable one for reproducible builds is the new --build-path option and $build_path configuration variable added by akira which allows to explicitly chose a given build path. Reiner Herrmann wrote a new patch for dh-systemd to sort the list of unit files in the generated maintainer scripts. Packages fixed The following packages became reproducible due to changes in their build dependencies: aoeui, apron, camlmix, cudf, findlib, glpk-java, hawtjni, haxe, java-atk-wrapper, llvm-py, misery, mtasc, ocamldsort, optcomp, spamoracle. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Untested Patches submitted which have not made their way to the archive yet: reproducible.debian.net ProfitBricks once again increased their support for reproducible builds in Debian and in other free software projects by adding 58 new cores and 138 GiB of RAM to the already existing setup. Two new amd64 build nodes and 16 new amd64 build jobs have been added which doubles the build capacity per day and allows us to spot many kind of problems earlier. The size of the tmpfs where builds are performed has also been increased from 70 to 200 GiB on all amd64 build nodes. Huge thanks! When examining a package, a link now points to a table listing all previous recorded tests for the same package. (Mattia) The menu on the package pages has also been improved. (h01ger) Packages in the depwait state are now rescheduled automatically after five days. (h01ger) Links to documentation and other projects being tested have been made more visible on the landing page. (h01ger) To reduce noise on the team IRC channel five different types of notifications have been turned into mail notifications. The remaining ones have been shortened and the status changes have been limited to unstable and experimental. (h01ger) Maintainer notifications about status changes in a package will only be sent out once per day, and not on each status change. (h01ger) diffoscope development Some more experiments of concurrent processing have been made. None were good and reliable enough to be shared, though. Package reviews 48 reviews have been removed, 189 added and 23 updated this week. 9 FTBFS bugs were reported by Chris Lamb. Misc. h01ger met with Levente Polyak to discuss testing Arch Linux on Debian continuous test system with an easily extensible framework. The idea is to also allow testing of other distributions, and provide a nice package based view like the one for Debian.

04 October 2015

Johannes Schauer: new sbuild release 0.66.0

I just released sbuild 0.66.0-1 into unstable. It fixes a whopping 30 bugs! Thus, I'd like to use this platform to: And a super big thank you to Roger Leigh who, despite having resigned from Debian, was always available to give extremely helpful hints, tips, opinion and guidance with respect to sbuild development. Thank you! Here is a list of the major changes since the last release:

16 August 2015

Lunar: Reproducible builds: week 16 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Valentin Lorentz sent a patch for ispell to initialize memory structures before dumping their content. In our experimental repository, qt4-x11 has been rebased on the latest version (Dhole), as was doxygen (akira). Packages fixed The following packages became reproducible due to changes in their build dependencies: backup-manager, cheese, coinor-csdp, coinor-dylp, ebook-speaker, freefem, indent, libjbcrypt-java, qtquick1-opensource-src, ruby-coffee-script, ruby-distribution, schroot, twittering-mode. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: akira found another embedded code copy of texi2html in maxima. reproducible.debian.net Work on testing several architectures has continued. (Mattia/h01ger) Package reviews 29 reviews have been removed, 187 added and 34 updated this week. 172 new FTBFS reports were filled, 137 solely by Chris West (Faux). josch spent time investigating the issue with fonts in PDF files. Chris Lamb documented the issue affecting documentation generated by ocamldoc. Misc. Lunar presented a general Reproducible builds HOWTO talk at the Chaos Communication Camp 2015 in Germany on August 13th. Recordings are already available, as well as slides and script. h01ger and Lunar also used CCCamp15 as an opportunity to have discussions with members of several different projects about reproducible builds. Good news should be coming soon.

15 August 2015

Matthieu Caneill: A one-liner to catch'em all!

I wrote a Bash one-liner to open the source code (in Debsources) of any file on your system (if it belongs to a Debian package). It will simply retrieve the associated package and point your default browser to its source code. Add this somewhere in your $PATH, and name this file debsrc:
#!/bin/bash
function debsrc  
    readlink -f $1   xargs dpkg-query --search   awk -F ": " ' print $1 '   xargs apt-cache showsrc   grep-dctrl -s 'Package' -n ''   awk -F " " ' print "http://sources.debian.net/src/"$1"/latest/" '   xargs x-www-browser
 
CMD="$1"
debsrc $ CMD 
And try something like debsrc /usr/share/doc/acpi/AUTHORS. Enjoy! Update: improved the one-liner thanks to josch's advice.

07 July 2015

Lunar: Reproducible builds: week 10 in Stretch cycle

What happened about the reproducible builds effort this week: Media coverage Daniel Stender published an English translation of the article which originally appeared in Linux Magazin in Admin Magazine. Toolchain fixes Fixes landed in the Debian archive: Lunar submitted to Debian the patch already sent upstream adding a --clamp-mtime option to tar. Patches have been submitted to add support for SOURCE_DATE_EPOCH to txt2man (Reiner Herrmann), epydoc (Reiner Herrmann), GCC (Dhole), and Doxygen (akira). Dhole uploaded a new experimental debhelper to the reproducible repository which exports SOURCE_DATE_EPOCH. As part of the experiment, the patch also sets TZ to UTC which should help with most timezone issues. It might still be problematic for some packages which would change their settings based on this. Mattia Rizzolo sent upstream a patch originally written by Lunar to make the generate-id() function be deterministic in libxslt. While that patch was quickly rejected by upstream, Andrew Ayer came up with a much better one which sadly could have some performance impact. Daniel Veillard replied with another patch that should be deterministic in most cases without needing extra data structures. It's impact is currently being investigated by retesting packages on reproducible.debian.net. akira added a new option to sbuild for configuring the path in which packages are built. This will be needed for the srebuild script. Niko Tyni asked Perl upstream about it using the __DATE__ and __TIME__ C processor macros. Packages fixed The following 143 packages became reproducible due to changes in their build dependencies: alot, argvalidate, astroquery, blender, bpython, brian, calibre, cfourcc, chaussette, checkbox-ng, cloc, configshell, daisy-player, dipy, dnsruby, dput-ng, dsc-statistics, eliom, emacspeak, freeipmi, geant321, gpick, grapefruit, heat-cfntools, imagetooth, jansson, jmapviewer, lava-tool, libhtml-lint-perl, libtime-y2038-perl, lift, lua-ldoc, luarocks, mailman-api, matroxset, maven-hpi-plugin, mknbi, mpi4py, mpmath, msnlib, munkres, musicbrainzngs, nova, pecomato, pgrouting, pngcheck, powerline, profitbricks-client, pyepr, pylibssh2, pylogsparser, pystemmer, pytest, python-amqp, python-apt, python-carrot, python-crypto, python-darts.lib.utils.lru, python-demgengeo, python-graph, python-mock, python-musicbrainz2, python-pathtools, python-pskc, python-psutil, python-pypump, python-repoze.sphinx.autointerface, python-repoze.tm2, python-repoze.what-plugins, python-repoze.what, python-repoze.who-plugins, python-xstatic-term.js, reclass, resource-agents, rgain, rttool, ruby-aggregate, ruby-archive-tar-minitar, ruby-bcat, ruby-blankslate, ruby-coffee-script, ruby-colored, ruby-dbd-mysql, ruby-dbd-odbc, ruby-dbd-pg, ruby-dbd-sqlite3, ruby-dbi, ruby-dirty-memoize, ruby-encryptor, ruby-erubis, ruby-fast-xs, ruby-fusefs, ruby-gd, ruby-git, ruby-globalhotkeys, ruby-god, ruby-hike, ruby-hmac, ruby-integration, ruby-ipaddress, ruby-jnunemaker-matchy, ruby-memoize, ruby-merb-core, ruby-merb-haml, ruby-merb-helpers, ruby-metaid, ruby-mina, ruby-net-irc, ruby-net-netrc, ruby-odbc, ruby-packet, ruby-parseconfig, ruby-platform, ruby-plist, ruby-popen4, ruby-rchardet, ruby-romkan, ruby-rubyforge, ruby-rubytorrent, ruby-samuel, ruby-shoulda-matchers, ruby-sourcify, ruby-test-spec, ruby-validatable, ruby-wirble, ruby-xml-simple, ruby-zoom, ryu, simplejson, spamassassin-heatu, speaklater, stompserver, syncevolution, syncmaildir, thin, ticgit, tox, transmissionrpc, vdr-plugin-xine, waitress, whereami, xlsx2csv, zathura. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net A new package set for the X Strike Force has been added. (h01ger) Bugs tagged with locale are now visible in the statistics. (h01ger) Some work has been done add tests for NetBSD. (h01ger) Many changes by Mattia Rizzolo have been merged on the whole infrastructure: debbindiff development Version 26 has been released on June 28th fixing the comparison of files of unknown format. (Lunar) A missing dependency identified in python-rpm affecting debbindiff installation without recommended packages was promptly fixed by Michal iha . Lunar also started a massive code rearchitecture to enhance code reuse and enable new features. Nothing visible yet, though. Documentation update josch and Mattia Rizzolo documented how to reschedule packages from Alioth. Package reviews 142 obsolete reviews have been removed, 344 added and 107 updated this week. Chris West (Faux) filled 13 new bugs for packages failing to build from sources. The following new issues have been added: snapshot_placeholder_replaced_with_timestamp_in_pom_properties, different_encoding, timestamps_in_documentation_generated_by_org_mode and timestamps_in_pdf_generated_by_matplotlib.

17 May 2015

Lunar: Reproducible builds: week 2 in Stretch cycle

What happened about the reproducible builds effort for this week: Media coverage Debian's effort on reproducible builds has been covered in the June 2015 issue of Linux Magazin in Germany. Cover of Linux Magazin June 2015 Article about reproducible builds in Linux Magazin June 2015 Toolchain fixes josch rebased the experimental version of debhelper on 9.20150507. Packages fixed The following 515 packages became reproducible due to changes of their build dependencies: airport-utils, airspy-host, all-in-one-sidebar, ampache, aptfs, arpack, asciio, aspell-kk, asused, balance, batmand, binutils-avr, bioperl, bpm-tools, c2050, cakephp-instaweb, carton, cbp2make, checkbot, checksecurity, chemeq, chronicle, cube2-data, cucumber, darkstat, debci, desktop-file-utils, dh-linktree, django-pagination, dosbox, eekboek, emboss-explorer, encfs, exabgp, fbasics, fife, fonts-lexi-saebom, gdnsd, glances, gnome-clocks, gunicorn, haproxy, haskell-aws, haskell-base-unicode-symbols, haskell-base64-bytestring, haskell-basic-prelude, haskell-binary-shared, haskell-binary, haskell-bitarray, haskell-bool-extras, haskell-boolean, haskell-boomerang, haskell-bytestring-lexing, haskell-bytestring-mmap, haskell-config-value, haskell-mueval, haskell-tasty-kat, itk3, jnr-constants, jshon, kalternatives, kdepim-runtime, kdevplatform, kwalletcli, lemonldap-ng, libalgorithm-combinatorics-perl, libalgorithm-diff-xs-perl, libany-uri-escape-perl, libanyevent-http-scopedclient-perl, libanyevent-perl, libanyevent-processor-perl, libapache-session-wrapper-perl, libapache-sessionx-perl, libapp-options-perl, libarch-perl, libarchive-peek-perl, libaudio-flac-header-perl, libaudio-wav-perl, libaudio-wma-perl, libauth-yubikey-decrypter-perl, libauthen-krb5-simple-perl, libauthen-simple-perl, libautobox-dump-perl, libb-keywords-perl, libbarcode-code128-perl, libbio-das-lite-perl, libbio-mage-perl, libbrowser-open-perl, libbusiness-creditcard-perl, libbusiness-edifact-interchange-perl, libbusiness-isbn-data-perl, libbusiness-tax-vat-validation-perl, libcache-historical-perl, libcache-memcached-perl, libcairo-gobject-perl, libcarp-always-perl, libcarp-fix-1-25-perl, libcatalyst-action-serialize-data-serializer-perl, libcatalyst-controller-formbuilder-perl, libcatalyst-dispatchtype-regex-perl, libcatalyst-plugin-authentication-perl, libcatalyst-plugin-authorization-acl-perl, libcatalyst-plugin-session-store-cache-perl, libcatalyst-plugin-session-store-fastmmap-perl, libcatalyst-plugin-static-simple-perl, libcatalyst-view-gd-perl, libcgi-application-dispatch-perl, libcgi-application-plugin-authentication-perl, libcgi-application-plugin-logdispatch-perl, libcgi-application-plugin-session-perl, libcgi-application-server-perl, libcgi-compile-perl, libcgi-xmlform-perl, libclass-accessor-classy-perl, libclass-accessor-lvalue-perl, libclass-accessor-perl, libclass-c3-adopt-next-perl, libclass-dbi-plugin-type-perl, libclass-field-perl, libclass-handle-perl, libclass-load-perl, libclass-ooorno-perl, libclass-prototyped-perl, libclass-returnvalue-perl, libclass-singleton-perl, libclass-std-fast-perl, libclone-perl, libconfig-auto-perl, libconfig-jfdi-perl, libconfig-simple-perl, libconvert-basen-perl, libconvert-ber-perl, libcpan-checksums-perl, libcpanplus-dist-build-perl, libcriticism-perl, libcrypt-cracklib-perl, libcrypt-dh-gmp-perl, libcrypt-mysql-perl, libcrypt-passwdmd5-perl, libcrypt-simple-perl, libcss-packer-perl, libcss-tiny-perl, libcurses-widgets-perl, libdaemon-control-perl, libdancer-plugin-database-perl, libdancer-session-cookie-perl, libdancer2-plugin-database-perl, libdata-format-html-perl, libdata-uuid-libuuid-perl, libdata-validate-domain-perl, libdate-jd-perl, libdate-simple-perl, libdatetime-astro-sunrise-perl, libdatetime-event-cron-perl, libdatetime-format-dbi-perl, libdatetime-format-epoch-perl, libdatetime-format-mail-perl, libdatetime-tiny-perl, libdatrie, libdb-file-lock-perl, libdbd-firebird-perl, libdbix-abstract-perl, libdbix-class-datetime-epoch-perl, libdbix-class-dynamicdefault-perl, libdbix-class-introspectablem2m-perl, libdbix-class-timestamp-perl, libdbix-connector-perl, libdbix-oo-perl, libdbix-searchbuilder-perl, libdbix-xml-rdb-perl, libdevel-stacktrace-ashtml-perl, libdigest-hmac-perl, libdist-zilla-plugin-emailnotify-perl, libemail-date-format-perl, libemail-mime-perl, libemail-received-perl, libemail-sender-perl, libemail-simple-perl, libencode-detect-perl, libexporter-tidy-perl, libextutils-cchecker-perl, libextutils-installpaths-perl, libextutils-libbuilder-perl, libextutils-makemaker-cpanfile-perl, libextutils-typemap-perl, libfile-counterfile-perl, libfile-pushd-perl, libfile-read-perl, libfile-touch-perl, libfile-type-perl, libfinance-bank-ie-permanenttsb-perl, libfont-freetype-perl, libfrontier-rpc-perl, libgd-securityimage-perl, libgeo-coordinates-utm-perl, libgit-pureperl-perl, libgnome2-canvas-perl, libgnome2-wnck-perl, libgraph-readwrite-perl, libgraphics-colornames-www-perl, libgssapi-perl, libgtk2-appindicator-perl, libgtk2-gladexml-simple-perl, libgtk2-notify-perl, libhash-asobject-perl, libhash-moreutils-perl, libhtml-calendarmonthsimple-perl, libhtml-display-perl, libhtml-fillinform-perl, libhtml-form-perl, libhtml-formhandler-model-dbic-perl, libhtml-html5-entities-perl, libhtml-linkextractor-perl, libhtml-tableextract-perl, libhtml-widget-perl, libhtml-widgets-selectlayers-perl, libhtml-wikiconverter-mediawiki-perl, libhttp-async-perl, libhttp-body-perl, libhttp-date-perl, libimage-imlib2-perl, libimdb-film-perl, libimport-into-perl, libindirect-perl, libio-bufferedselect-perl, libio-compress-lzma-perl, libio-compress-perl, libio-handle-util-perl, libio-interface-perl, libio-multiplex-perl, libio-socket-inet6-perl, libipc-system-simple-perl, libiptables-chainmgr-perl, libjoda-time-java, libjsr305-java, libkiokudb-perl, liblemonldap-ng-cli-perl, liblexical-var-perl, liblingua-en-fathom-perl, liblinux-dvb-perl, liblocales-perl, liblog-dispatch-configurator-any-perl, liblog-log4perl-perl, liblog-report-lexicon-perl, liblwp-mediatypes-perl, liblwp-protocol-https-perl, liblwpx-paranoidagent-perl, libmail-sendeasy-perl, libmarc-xml-perl, libmason-plugin-routersimple-perl, libmasonx-processdir-perl, libmath-base85-perl, libmath-basecalc-perl, libmath-basecnv-perl, libmath-bigint-perl, libmath-convexhull-perl, libmath-gmp-perl, libmath-gradient-perl, libmath-random-isaac-perl, libmath-random-oo-perl, libmath-random-tt800-perl, libmath-tamuanova-perl, libmemoize-expirelru-perl, libmemoize-memcached-perl, libmime-base32-perl, libmime-lite-tt-perl, libmixin-extrafields-param-perl, libmock-quick-perl, libmodule-cpanfile-perl, libmodule-load-conditional-perl, libmodule-starter-pbp-perl, libmodule-util-perl, libmodule-versions-report-perl, libmongodbx-class-perl, libmoo-perl, libmoosex-app-cmd-perl, libmoosex-attributehelpers-perl, libmoosex-blessed-reconstruct-perl, libmoosex-insideout-perl, libmoosex-relatedclassroles-perl, libmoosex-role-timer-perl, libmoosex-role-withoverloading-perl, libmoosex-storage-perl, libmoosex-types-common-perl, libmoosex-types-uri-perl, libmoox-singleton-perl, libmoox-types-mooselike-numeric-perl, libmousex-foreign-perl, libmp3-tag-perl, libmysql-diff-perl, libnamespace-clean-perl, libnet-bonjour-perl, libnet-cli-interact-perl, libnet-daap-dmap-perl, libnet-dbus-glib-perl, libnet-dns-perl, libnet-frame-perl, libnet-google-authsub-perl, libnet-https-any-perl, libnet-https-nb-perl, libnet-idn-encode-perl, libnet-idn-nameprep-perl, libnet-imap-client-perl, libnet-irc-perl, libnet-mac-vendor-perl, libnet-openid-server-perl, libnet-smtp-ssl-perl, libnet-smtp-tls-perl, libnet-smtpauth-perl, libnet-snpp-perl, libnet-sslglue-perl, libnet-telnet-perl, libnhgri-blastall-perl, libnumber-range-perl, libobject-signature-perl, libogg-vorbis-header-pureperl-perl, libopenoffice-oodoc-perl, libparse-cpan-packages-perl, libparse-debian-packages-perl, libparse-fixedlength-perl, libparse-syslog-perl, libparse-win32registry-perl, libpdf-create-perl, libpdf-report-perl, libperl-destruct-level-perl, libperl-metrics-simple-perl, libperl-minimumversion-perl, libperl6-slurp-perl, libpgobject-simple-perl, libplack-middleware-fixmissingbodyinredirect-perl, libplack-test-externalserver-perl, libplucene-perl, libpod-tests-perl, libpoe-component-client-ping-perl, libpoe-component-jabber-perl, libpoe-component-resolver-perl, libpoe-component-server-soap-perl, libpoe-component-syndicator-perl, libposix-strftime-compiler-perl, libposix-strptime-perl, libpostscript-simple-perl, libproc-processtable-perl, libprotocol-osc-perl, librcs-perl, libreadonly-xs-perl, libreturn-multilevel-perl, librivescript-perl, librouter-simple-perl, librrd-simple-perl, libsafe-isa-perl, libscope-guard-perl, libsemver-perl, libset-tiny-perl, libsharyanto-file-util-perl, libshell-command-perl, libsnmp-info-perl, libsoap-lite-perl, libstat-lsmode-perl, libstatistics-online-perl, libstring-compare-constanttime-perl, libstring-format-perl, libstring-toidentifier-en-perl, libstring-tt-perl, libsub-recursive-perl, libsvg-tt-graph-perl, libsvn-notify-perl, libswish-api-common-perl, libtap-formatter-junit-perl, libtap-harness-archive-perl, libtemplate-plugin-number-format-perl, libtemplate-plugin-yaml-perl, libtemplate-tiny-perl, libtenjin-perl, libterm-visual-perl, libtest-block-perl, libtest-carp-perl, libtest-classapi-perl, libtest-cmd-perl, libtest-consistentversion-perl, libtest-data-perl, libtest-databaserow-perl, libtest-differences-perl, libtest-file-sharedir-perl, libtest-hasversion-perl, libtest-kwalitee-perl, libtest-lectrotest-perl, libtest-module-used-perl, libtest-object-perl, libtest-perl-critic-perl, libtest-pod-coverage-perl, libtest-script-perl, libtest-script-run-perl, libtest-spelling-perl, libtest-strict-perl, libtest-synopsis-perl, libtest-trap-perl, libtest-unit-perl, libtest-utf8-perl, libtest-without-module-perl, libtest-www-selenium-perl, libtest-xml-simple-perl, libtest-yaml-perl, libtex-encode-perl, libtext-bibtex-perl, libtext-csv-encoded-perl, libtext-csv-perl, libtext-dhcpleases-perl, libtext-diff-perl, libtext-quoted-perl, libtext-trac-perl, libtext-vfile-asdata-perl, libthai, libthread-conveyor-perl, libthread-sigmask-perl, libtie-cphash-perl, libtie-ical-perl, libtime-stopwatch-perl, libtk-dirselect-perl, libtk-pod-perl, libtorrent, libturpial, libunicode-japanese-perl, libunicode-maputf8-perl, libunicode-stringprep-perl, libuniversal-isa-perl, libuniversal-moniker-perl, liburi-encode-perl, libvi-quickfix-perl, libvideo-capture-v4l-perl, libvideo-fourcc-info-perl, libwiki-toolkit-plugin-rss-reader-perl, libwww-mechanize-formfiller-perl, libwww-mechanize-gzip-perl, libwww-mechanize-perl, libwww-opensearch-perl, libx11-freedesktop-desktopentry-perl, libxc, libxml-dtdparser-perl, libxml-easy-perl, libxml-handler-trees-perl, libxml-libxml-iterator-perl, libxml-libxslt-perl, libxml-rss-perl, libxml-validator-schema-perl, libxml-xpathengine-perl, libxml-xql-perl, llvm-py, madbomber, makefs, mdpress, media-player-info, meta-kde-telepathy, metamonger, mmm-mode, mupen64plus-audio-sdl, mupen64plus-rsp-hle, mupen64plus-ui-console, mupen64plus-video-z64, mussort, newpid, node-formidable, node-github-url-from-git, node-transformers, nsnake, odin, otcl, parsley, pax, pcsc-perl, pd-purepd, pen, prank, proj, proot, puppet-module-puppetlabs-postgresql, python-async, python-pysnmp4, qrencode, r-bioc-graph, r-bioc-hypergraph, r-bioc-iranges, r-bioc-xvector, r-cran-pscl, rbenv, rlinetd, rs, ruby-ascii85, ruby-cutest, ruby-ejs, ruby-factory-girl, ruby-hdfeos5, ruby-kpeg, ruby-libxml, ruby-password, ruby-zip-zip, sdl-sound1.2, stterm, systemd, taktuk, tcc, tryton-modules-account-invoice, ttf-summersby, tupi, tuxpuck, unknown-horizons, unsafe-mock, vcheck, versiontools, vim-addon-manager, vlfeat, vsearch, xacobeo, xen-tools, yubikey-personalization-gui, yubikey-personalization. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: reproducible.debian.net Alioth now hosts a script that can be used to redo builds and test for a package. This was preliminary done manually through requests over the IRC channel. This should reduce the number of interruptions for jenkins' maintainers The graph of the oldest build per day has been fixed. Maintainance scripts will not error out when they are no files to remove. Holger Levsen started work on being able to test variations of CPU features and build date (as in build in another month of 1984) by using virtual machines. debbindiff development Version 18 has been released. It will uses proper comparators for pk3 and info files. Tar member names are now assumed to be UTF-8 encoded. The limit for the maximum number of different lines has been removed. Let's see on reproducible.debian.net how it goes for pathological cases. It's now possible to specify both --html and --text output. When neither of them is specified, the default will be to print a text report on the standard output (thanks to Paul Wise for the suggestion). Documentation update Nicolas Boulenguez investigated Ada libraries. Package reviews 451 obsolete reviews have been removed and 156 added this week. New identified issues: running kernel version getting captured, random filenames in GHC debug symbols, and timestamps in headers generated by qdbusxml2cpp. Misc. Holger Levsen went to re:publica and talked about reproducible builds to developers and users there. Holger also had a chance to meet FreeBSD developers and discuss the status of FreeBSD. Investigations have started on how it could be made part of our current test system. Laurent Guerby gave Lunar access to systems in the GCC Compile Farm. Hopefully access to these powerful machines will help to fix packages for GCC, Iceweasel, and similar packages requiring long build times.

04 February 2015

Johannes Schauer: I became a Debian Developer

Thanks to akira for the confetti to celebrate the occasion!

30 November 2014

Johannes Schauer: simple email setup

I was unable to find a good place that describes how to create a simple self-hosted email setup. The most surprising discovery was, how much already works after:
apt-get install postfix dovecot-imapd
Right after having finished the installation I was able to receive email (but only in in /var/mail in mbox format) and send email (bot not from any other host). So while I expected a pretty complex setup, it turned out to boil down to just adjusting some configuration parameters.

Postfix The two interesting files to configure postfix are /etc/postfix/main.cf and /etc/postfix/master.cf. A commented version of the former exists in /usr/share/postfix/main.cf.dist. Alternatively, there is the ~600k word strong man page postconf(5). The latter file is documented in master(5).

/etc/postfix/main.cf I changed the following in my main.cf
@@ -37,3 +37,9 @@
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
+
+home_mailbox = Mail/
+smtpd_recipient_restrictions = permit_mynetworks reject_unauth_destination permit_sasl_authenticated
+smtpd_sasl_type = dovecot
+smtpd_sasl_path = private/auth
+smtp_helo_name = my.reverse.dns.name.com
At this point, also make sure that the parameters smtpd_tls_cert_file and smtpd_tls_key_file point to the right certificate and private key file. So either change these values or replace the content of /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-cert-snakeoil.key. The home_mailbox parameter sets the default path for incoming mail. Since there is no leading slash, this puts mail into $HOME/Mail for each user. The trailing slash is important as it specifies qmail-style delivery'' which means maildir. The default of the smtpd_recipient_restrictions parameter is permit_mynetworks reject_unauth_destination so this just adds the permit_sasl_authenticated option. This is necessary to allow users to send email when they successfully verified their login through dovecot. The dovecot login verification is activated through the smtpd_sasl_type and smtpd_sasl_path parameters. I found it necessary to set the smtp_helo_name parameter to the reverse DNS of my server. This was necessary because many other email servers would only accept email from a server with a valid reverse DNS entry. My hosting provider charges USD 7.50 per month to change the default reverse DNS name, so the easy solution is, to instead just adjust the name announced in the SMTP helo.

/etc/postfix/master.cf The file master.cf is used to enable the submission service. The following diff just removes the comment character from the appropriate section.
@@ -13,12 +13,12 @@
#smtpd pass - - - - - smtpd
#dnsblog unix - - - - 0 dnsblog
#tlsproxy unix - - - - 0 tlsproxy
-#submission inet n - - - - smtpd
-# -o syslog_name=postfix/submission
-# -o smtpd_tls_security_level=encrypt
-# -o smtpd_sasl_auth_enable=yes
-# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
-# -o milter_macro_daemon_name=ORIGINATING
+submission inet n - - - - smtpd
+ -o syslog_name=postfix/submission
+ -o smtpd_tls_security_level=encrypt
+ -o smtpd_sasl_auth_enable=yes
+ -o smtpd_client_restrictions=permit_sasl_authenticated,reject
+ -o milter_macro_daemon_name=ORIGINATING
#smtps inet n - - - - smtpd
# -o syslog_name=postfix/smtps
# -o smtpd_tls_wrappermode=yes

Dovecot Since above configuration changes made postfix store email in a different location and format than the default, dovecot has to be informed about these changes as well. This is done in /etc/dovecot/conf.d/10-mail.conf. The second configuration change enables postfix to authenticate users through dovecot in /etc/dovecot/conf.d/10-master.conf. For SSL one should look into /etc/dovecot/conf.d/10-ssl.conf and either adapt the parameters ssl_cert and ssl_key or store the correct certificate and private key in /etc/dovecot/dovecot.pem and /etc/dovecot/private/dovecot.pem, respectively. The dovecot-core package (which dovecot-imapd depends on) ships tons of documentation. The file /usr/share/doc/dovecot-core/dovecot/documentation.txt.gz gives an overview of what resources are available. The path /usr/share/doc/dovecot-core/dovecot/wiki contains a snapshot of the dovecot wiki at http://wiki2.dovecot.org/. The example configurations seem to be the same files as in /etc/ which are already well commented.

/etc/dovecot/conf.d/10-mail.conf The following diff changes the default email location in /var/mail to a maildir in ~/Mail as configured for postfix above.
@@ -27,7 +27,7 @@
#
# <doc/wiki/MailLocation.txt>
#
-mail_location = mbox:~/mail:INBOX=/var/mail/%u
+mail_location = maildir:~/Mail

# If you need to set multiple mailbox locations or want to change default
# namespace settings, you can do it by defining namespace sections.

/etc/dovecot/conf.d/10-master.conf And this enables the authentication socket for postfix:
@@ -93,9 +93,11 @@


# Postfix smtp-auth
- #unix_listener /var/spool/postfix/private/auth
- # mode = 0666
- #
+ unix_listener /var/spool/postfix/private/auth
+ mode = 0660
+ user = postfix
+ group = postfix
+

# Auth process is run as this user.
#user = $default_internal_user

Aliases Now Email will automatically put into the '~/Mail' directory of the receiver. So a user has to be created for whom one wants to receive mail...
$ adduser josch
...and any aliases for it to be configured in /etc/aliases.
@@ -1,2 +1,4 @@
-# See man 5 aliases for format
-postmaster: root
+root: josch
+postmaster: josch
+hostmaster: josch
+webmaster: josch
After editing /etc/aliases, the command
$ newaliases
has to be run. More can be read in the aliases(5) man page.

Finishing up Everything is done and now postfix and dovecot have to be informed about the changes. There are many ways to do that. Either restart the services, reboot or just do:
$ postfix reload
$ doveadm reload
Have fun!

07 November 2014

Johannes Schauer: automatically suspending cpu hungry applications

TLDR: Using the awesome window manager: how to automatically send SIGSTOP and SIGCONT to application windows when they get unfocused or focused, respectively, to let the application not waste CPU cycles when not in use. I don't require any fancy looking GUI, so my desktop runs no full-blown desktop environment like Gnome or KDE but instead only awesome as a light-weight window manager. Usually, the only application windows I have open are rxvt-unicode as my terminal emulator and firefox/iceweasel with the pentadactyl extension as my browser. Thus, I would expect that CPU usage of my idle system would be pretty much zero but instead firefox decides to constantly eat 10-15%. Probably to update some GIF animations or JavaScript (or nowadays even HTML5 video animations). But I don't need it to do that when I'm not currently looking at my browser window. Disabling all JavaScript is no option because some websites that I need for uni or work are just completely broken without JavaScript, so I have to enable it for those websites. Solution: send SIGSTOP when my firefox window looses focus and send SIGCONT once it gains focus again. The following addition to my /etc/xdg/awesome/rc.lua does the trick:
local capi =   timer = timer  
client.add_signal("focus", function(c)
if c.class == "Iceweasel" then
awful.util.spawn("kill -CONT " .. c.pid)
end
end)
client.add_signal("unfocus", function(c)
if c.class == "Iceweasel" then
local timer_stop = capi.timer timeout = 10
local send_sigstop = function ()
timer_stop:stop()
if client.focus.pid ~= c.pid then
awful.util.spawn("kill -STOP " .. c.pid)
end
end
timer_stop:add_signal("timeout", send_sigstop)
timer_stop:start()
end
end)
Since I'm running Debian, the class is "Iceweasel" and not "Firefox". When the window gains focus, a SIGCONT is sent immediately. I'm executing kill because I don't know how to send UNIX signals from lua directly. When the window looses focus, then the SIGSTOP signal is only sent after a 10 second timeout. This is done for several reasons: With this change, when I now open htop, the process consuming most CPU resources is htop itself. Success! Another cool advantage is, that firefox can now be moved completely into swap space in case I run otherwise memory hungry applications without ever requiring any memory from swap until I really use it again. I haven't encountered any disadvantages of this setup yet. If 10 seconds prove to be too short to copy and paste I can easily extend this delay. Even clicking on links in my terminal works flawlessly - the new tab will just only load once firefox gets focused again. EDIT: thanks to Helmut Grohne for suggesting to compare the pid instead of the raw client instance to prevent misbehaviour when firefox opens additional windows like the preferences dialog.

29 July 2014

Johannes Schauer: bootstrap.debian.net temporarily not updated

I'll be moving places twice within the next month and as I'm hosting the machine that generates the data, I'll temporarily suspend the bootstrap.debian.net service until maybe around September. Until then, bootstrap.debian.net will not be updated and retain the status as of 2014-07-28. Sorry if that causes any inconvenience. You can write to me if you need help with manually generating the data bootstrap.debian.net provided.

05 June 2014

Johannes Schauer: botch updates

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was four months ago and about several incremental updates. This post will be of similar nature. The most interesting news is probably the additional data that bootstrap.debian.net now provides. This is listed in the next section. All subsequent sections then list the changes under the hood that made the additions to bootstrap.debian.net possible.

bootstrap.debian.net The bootstrap.debian.net service used to have botch as a git submodule but now runs botch from its Debian package. This at least proves that the botch Debian package is mature enough to do useful stuff with it. In addition to the bootstrapping results by architecture, bootstrap.debian.net now also hosts the following additional services: Further improvements concern how dependency cycles are now presented in the html overviews. While before, vertices in a cycle where separated by commas as if they were simple package lists, vertices are now connected by unicode arrows. Dashed arrows indicate build dependencies while solid arrows indicate builds-from relationships. For what it's worth, installation set vertices now contain their installation set in their title attribute.

Debian package Botch has long depended on features of an unreleased version of dose3 which in turn depended on an unrelease version of libcudf. Both projects have recently made new releases so that I was now able to drop the dose3 git submodule and rely on the host system's dose3 version instead. This also made it possible to create a Debian package of botch which currently sits at Debian mentors. Writing the package also finally made me create a usable install target in the Makefile as well as adding stubs for the manpages of the 44 applications that botch currently ships. The actual content of these manpages still has to be written. The only documentation botch currently ships in the botch-doc package is an offline version of the wiki on gitorious. The new page ExamplesGraphs even includes pictures.

Cross By default, botch analyzes the native bootstrapping phase. That is, assume that the initial set of Essential:yes and build-essential packages magically exists and find out how to bootstrap the rest from there through native compilation. But part of the bootstrapping problem is also to create the set of Essential:yes and build-essential packages from nothing via cross compilation. Botch is unable to analyze the cross phase because too many packages cannot satisfy their crossbuild dependencies due to multiarch conflicts. This problem is only about the dependency metadata and not about whether a given source package actually crosscompiles fine in practice. Helmut Grohne has done great work with rebootstrap which is regularly run by jenkins.debian.net. He convinced me that we need an overview of what packages are blocking the analysis of the cross case and that it was useful to have a crossbuild order even if that was a fake order just to have a rough overview of the current situation in Debian Sid. I wrote a couple of scripts which would run dose-builddebcheck on a repository, analyze which packages fail to satisfy their crossbuild dependencies and why, fix those cases by adjusting package metadata accordingly and repeat until all relevant source packages satisfy their crossbuild dependencies. The result of this can then be used to identify the packages that need to be modified as well as to generate a crossbuild order. The fixes to the metadata are done in an automatic fashion and do not necessarily reflect the real fix that would solve the problem. Nevertheless, I ended up agreeing that it is better to have a slightly wrong overview than no overview at all.

Minimizing the dependency graph size Installation sets in the dependency graph are calculated independent from each other. If two binary packages provide A, then dependencies on A in different installation sets might choose different binary packages as providers of A. The same holds for disjunctive dependencies. If a package depends on A C and another package depends on C A then there is no coordination to choose C so to minimize the overall amount of vertices in the graph. I implemented two methods to minimize the impact of cases where the dependency solver has multiple options to satisfy a dependency through Provides and dependency disjunctions. The first method is inspired by Helmut Grohne. An algorithm goes through all disjunctive binary dependencies and removes all virtual packages, leaving only real packages. Of the remaining real packages, the first one is selected. For build dependencies, the algorithm drops all but the first package in every disjunction. This is also what sbuild does. Unfortunately this solution produces an unsatisfiable dependency situation in most cases. This is because oftentimes it is necessary to select the virtual disjunctive dependency because of a conflict relationship introduced by another package. The second method involves aspcud, a cudf solver which can optimize a solution by a criteria. This solution is based on an idea by Pietro Abate who implemented the basis for this idea back in 2012. In contrast to a usual cudf problem, binary packages now also depend on the source packages they build from. If we now ask aspcud to find an installation set for one of the base source packages (I chose src:build-essential) then it will return an installation set that includes source packages. As an optimization criteria the number of source packages in the installation set is minimized. This solution would be flawless if there were no conflicts between binary packages. Due to conflicts not all binary packages that must be coinstallable for this strategy to work can be coinstalled. The quick and dirty solution is to remove all conflicts before passing the cudf universe to aspcud. But this also means that the solution does sometimes not work in practice.

Test cases Botch now finally has a test target in its Makefile. The test target tests two code paths of the native.sh script and the cross.sh script. Running these two scripts covers testing most parts of botch. Given that I did lots of refactoring in the past weeks, the test cases greatly helped to assure that I didnt break anything in the process. I also added autopkgtests to the Debian packaging which test the same things as the test target but naturally run the installed version of botch instead. The autopkgtests were a great help in weeding out some lasts bugs which made botch depend on being executed from its source directory.

Python 3 Reading the suggestions in the Debian python policy I evaluated the possibility to use Python 3 for the Python scripts in botch. While I was at it I added transparent decompression for gzip, bz2 and xz based on the file magic, replaced python-apt with python-debian because of bug#748922 and added argparse argument parsing to all scripts. Unfortunately I had to find out that Python 3 support does not yet seem to be possible for botch for the following reasons:
  • no soap module for Python 3 in Debian (needed for bts access)
  • hash randomization is turned on by default in Python 3 and therefore the graph output of networkx is not deterministic anymore (bug#749710)
Thus I settled for changing the code such that it would be compatible with Python 2 as well as with Python 3. Because of the changed string handling and sys.stdout properties in Python 3 this proved to be tricky. On the other hand this showed me bugs in my code where I was wrongly relying on deterministic dictionary key traversal.

03 April 2014

Johannes Schauer: mapbender - maps for long-distance travels

Back in 2007 I stumbled over the "Plus Fours Routefinder", an invention of the 1920's. It's worn on the wrist and allows the user to scroll through a map of the route they planned to take, rolled up on little wooden rollers. At that point I thought: that's awesome for long trips where you either dont want to take electronics with you or where you are without any electricity for a long time. And creating such rollable custom maps of your route automatically using openstreetmap data should be a breeze! Nevertheless it seems nobody picked up the idea. Years passed and in a few weeks I'll go on a biking trip along the Weser, a river in nothern Germany. For my last multi-day trip (which was through the Odenwald, an area in southern Germany) I printed a big map from openstreetmap data which contained the whole route. Openstreetmap data is fantastic for this because in contrast to commercial maps it doesnt only allow you to just print the area you need but also allows you to highlight your planned route and objects you would probably not find in most commercial maps like for example supermarkets to stock up on supplies or bicycle repair shops. Unfortunately such big maps have the disadvantage that to show everything in the amount of detail that you want along your route, they have to be pretty huge and thus easily become an inconvenience because the local plotter can't handle paper as large as DIN A0 or because it's a pain to repeatedly fold and unfold the whole thing every time you want to look at it. Strong winds are also no fun with a huge sheet of paper in your hands. One solution would be to print DIN A4 sized map regions in the desired scale. But that has the disadvantage that either you find yourself going back and forth between subsequent pages because you happen to be right at the border between two pages or you have to print sufficiently large overlaps, resulting in many duplicate map pieces and more pages of paper than you would like to carry with you. It was then that I remembered the "Plus Fours Routefinder" concept. Given a predefined route it only shows what's important to you: all things close to the route you plan to travel along. Since it's a long continuous roll of paper there is no problem with folding because as you travel along the route you unroll one end and roll up the other. And because it's a long continuous map there is also no need for flipping pages or large overlap regions. There is not even the problem of not finding a big enough sheet of paper because multiple DIN A4 sheets can easily be glued together at their ends to form a long roll. On the left you see the route we want to take: the bicycle route along the Weser river. If I wanted to print that map on a scale that allows me to see objects in sufficient detail along our route, then I would also see objects in Hamburg (upper right corner) in the same amount of detail. Clearly a waste of ink and paper as the route is never even close to Hamburg.
As the first step, a smooth approximation of the route has to be found. It seems that the best way to do that is to calculate a B-Spline curve approximating the input data with a given smoothness. On the right you can see the approximated curve with a smoothing value of 6. The curve is sampled into 20 linear segments. I calculated the B-Spline using the FITPACK library to which scipy offers a Python binding.
The next step is to expand each of the line segments into quadrilaterals. The distance between the vertices of the quadrilaterals and the ends of the line segment they belong to is the same along the whole path and obviously has to be big enough such that every point along the route falls into one quadrilateral. In this example, I draw only 20 quadrilaterals for visual clarity. In practice one wants many more for a smoother approximation.
Using a simple transform, each point of the original map and the original path in each quadrilateral is then mapped to a point inside the corresponding "straight" rectangle. Each target rectangle has the height of the line segment it corresponds to. It can be seen that while the large scale curvature of the path is lost in the result, fine details remain perfectly visible. The assumption here is, that while travelling a path several hundred kilometers long, it does not matter that large scale curvature that one is not able to perceive anyways is not preserved. The transformation is done on a Mercator projection of the map itself as well as the data of the path. Therefore, this method probably doesnt work if you plan to travel to one of the poles. Currently I transform openstreetmap bitmap data. This is not quite optimal as it leads to text on the map being distorted. It would be just as easy to apply the necessary transformations to raw openstreetmap XML data but unfortunately I didnt find a way to render the resulting transformed map data as a raster image without setting up a database. I would've thought that it would be possible to have a standalone program reading openstreetmap XML and dumping out raster or svg images without a round trip through a database. Furthermore, tilemill, one of the programs that seem to be one of the least hasslesome to set up and produce raster images is stuck in an ITP and the existing packaging attempt fails to produce a non-empty binary package. Since I have no clue about nodejs packaging, I wrote about this to the pkg-javascript-devel list. Maybe I can find a kind soul to help me with it.
Here a side by side overview that doesnt include the underlying map data but only the path. It shows how small details are preserved in the transformation. The code that produced the images in this post is very crude, unoptimized and kinda messy. If you dont care, then it can be accessed here

06 February 2014

Johannes Schauer: botch updates

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was three months ago with the announcement of bootstrap.debian.net. Since then a number of things happened, so I thought an update was due.

New graphs for port metrics By default, a dependency graph is created by arbitrarily choosing an installation set for binary package installation or source package compilation. Installation set vertices and source vertices are connected according to this arbitrary selection. Niels Thykier approached me at Debconf13 about the possibility of using this graph to create a metric which would be able to tell for each source package, how many other source packages would become uncompilable or how many binary packages would become uninstallable, if that source package was removed from the archive. This could help deciding about the importance of packages. More about this can be found at the thread on debian-devel. For botch, this meant that two new graph graphs can now be generated. Instead of picking an arbitrary installation set for compiling a source package or installing a binary package, botch can now create a minimum graph which is created by letting dose3 calculate strong dependencies and a maximum graph by using the dependency closure.

Build profile syntax in dpkg With dpkg 1.17.2 we now have experimental build profile support in unstable. The syntax which ended up being added was:
Build-Depends: large (>= 1.0), small <!profile.stage1>
But until packages with that syntax can hit the archive, a few more tools need to understand the syntax. The patch we have for sbuild is very simple because sbuild relies on libdpkg for dependency parsing. We have a patch for apt too, but we have to rebase it for the current apt version and have to adapt it so that it works exactly like the functionality dpkg implements. But before we can do that we have to decide how to handle mixed positive and negative qualifiers or whether to remove this feature altogether because it causes too much confusion. The relevant thread on debian-dpkg starts here.

Update to latest dose3 git Botch heavily depends on libdose3 and unfortunately requires features which are only available in the current git HEAD. The latest version packaged in Debian is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also relies on unreleased features from libcudf. On top of that, the GraphML output of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the git. For now everything is set up as git submodules but this is the major blocker preventing any packaging of botch as a Debian package. Hopefully new releases will be done soon for all involved components.

Writing and reading GraphML Botch is a collection of several utilities which are connected together in a shell script. The advantage of this is, that one does not need to understand or hack the OCaml code to use botch for different purposes. In theory it also allows to insert 3rd party tools into a pipe to further modify the data. Until recently this ability was seriously hampered by the fact that many botch tools communicated with each other through marshaled OCaml binary files which prevent everything which is not written in OCaml from modifying them. The data that was passed around like this were the dependency graphs and I initially implemented it like that because I didnt want to write a GraphML parser. I now ended up writing an xmlm based GraphML parser so as of now, botch only reads and writes ASCII text files in XML (for the graphs) and in rfc822 packages format (for Packages and Sources files) which can both easily be modified by 3rd party tools. The ./tools directory contains many Python scripts using the networkx module to modify GraphML and the apt_pkg module to modify rfc822 files.

Splitting of tools To further increase the ability to modify program execution without having to know OCaml, I split up some big tools into multiple smaller ones. Some of the smaller tools are now even written in Python which is probably much more hackable for the general crowd. I converted those tools to Python which did not need any dose3 functionality and which were simple enough so that writing them didnt take much time. I could convert more tools but that might introduce bugs and takes time which I currently dont have much of (who does?).

Gzip instead of bz2 Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed Packages and Sources files anymore but uses xz instead. This is awesome for must purposes but unfortunately I had to discover that there exist no OCaml bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself or anybody else finds some time to write a libxz OCaml binding.

Self hosting Fedora Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I reproduced his steps for the Debian dependency graph and it turns out that they are a little bit bigger. I'm exchanging emails with Harald Hoyer because it might not be too hard to use botch for rpm based distributions as well because dose3 supports rpm. The article also made me aware of the tred tool which is part of graphviz and allows to calculate the transitive reduction of a graph. This can help making horrible situations much better.

Dose3 bugs I planned to generate such simplified graphs for the neighborhood of each source package on bootstrap.debian.net but then binutils stopped building binutils-gold and instead provided binutils-gold while libc6-dev breaks binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug and thus bootstrap.debian.net will not generate any new results until this is fixed in dose3. Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar while bar is fully virtual and are Multi-Arch:same. Binaries of different architecture with this property can currently not be co-installed with dose3. Unfortunately linux-libc-dev has this property and thus botch cannot be used to analyze cross builds until that bug is fixed in dose3. I hope I get some free time soon to be able to look at these dose3 issues myself.

More documentation Since I started to like the current set of tools and how they work together I ended up writing over 2600 words of documentation in the past few days. You can start setting up and running botch by reading the first steps and get more detailed information by reading about the 28 tools that botch makes use of as of now. All existing articles, thesis and talks are linked from the wiki home.

11 January 2014

Johannes Schauer: Why do I need superuser privileges when I just want to write to a regular file

I have written a number of scripts to create Debian foreign architecture (mostly armel and armhf) rootfs images for SD cards or NAND flashing. I started with putting Debian on my Openmoko gta01 and gta02 and continued with devices like the qi nanonote, a marvel kirkwood based device, the Always Innovating Touchbook (close to the Beagleboard), the Notion Ink Adam and most recently the Golden Delicious gta04. Once it has been manufactured, I will surely also get my hands dirty with the Neo900 whose creators are currently looking for potential donors/customers to increase the size of the first batch and get the price per unit further down. Creating a Debian rootfs disk image for all these devices basically follows the same steps:
  1. create an disk image file, partition it, format the partitions and mount the / partition into a directory
  2. use debootstrap or multistrap to extract a selection of armel or armhf packages into the directory
  3. copy over /usr/bin/qemu-arm-static for qemu user mode emulation
  4. chroot into the directory to execute package maintainer scripts with dpkg --configure -a
  5. copy the disk image onto the sd card
It was not long until I started wondering why I had to run all of the above steps with superuser privileges even though everything except the final step (which I will not cover here) was in principle nothing else than writing some magic bytes to files I had write access to (the disk image file) in some more or less fancy ways. So I tried using fakeroot+fakechroot and after some initial troubles I managed to build a foreign architecture rootfs without needing root priviliges for steps two, three and four. I wrote about my solution which still included some workarounds in another article here. These workarounds were soon not needed anymore as upstream fixed the outstanding issues. As a result I wrote the polystrap tool which combines multistrap, fakeroot, fakechroot and qemu user mode emulation. Recently I managed to integrate proot support in a separate branch of polystrap. Last year I got the LEGO ev3 robot for christmas and since it runs Linux I also wanted to put Debian on it by following the instructions given by the ev3dev project. Even though ev3dev calls itself a "distribution" it only deviates from pure Debian by its kernel, some configuration options and its initial package selection. Otherwise it's vanilla Debian. The project also supplies some multistrap based scripts which create the rootfs and then partition and populate an SD card. All of this is of course done as the superuser. While the creation of the file/directory structure of the foreign Debian armel rootfs can by now easily be done without superuser priviliges by running multistrap under fakeroot/fakechroot/proot, creating the SD card image still seems to be a bit more tricky. While it is no problem to write a partition table to a regular file, it turned out to be tricky to mount these partition because tools like kpartx and losetup require superuser permissions. Tools like mkfs.ext3 and fuse-ext2 which otherwise would be able to work on a regular file without superuser privileges do not seem to allow to specify the required offsets that the partitions have within the disk image. With fuseloop there exists a tool which allows to "loop-mount" parts of a file in userspace to a new file and thus allows tools like mkfs.ext3 and fuse-ext2 to work as they normally do. But fuseloop is not packaged for Debian yet and thus also not in the current Debian stable. An obvious workaround would be to create and fill each partition in a separate file and concatenate them together. But why do I have to write my data twice just because I do not want to become the superuser? Even worse: because parted refuses to write a partition table to a file which is too small to hold the specified partitions, one spends twice the disk space of the final image: the image with the partition table plus the image with the main partition's content. So lets summarize: a bootable foreign architecture SD card disk image is nothing else than a regular file representing the contents of the SD card as a block device. This disk image is created in my home directory and given enough free disk space there is nothing stopping me from writing any possible permutation of bits to that file. Obviously I'm interested in a permutation representing a valid partition table and file systems with sensible content. Why do I need superuser privileges to generate such a sensible permutation of bits? Gladly it seems that the (at least in my opinion) hardest part of faking chroot and executing foreign architecture package maintainer scripts is already possible without superuser privileges by using fakeroot and fakechroot or proot together with qemu user mode emulation. But then there is still the blocker of creating the disk image itself through some user mode loop mounting of a filesystem occupying a virtual "partition" in the disk image. Why has all this only become available so very recently and still requires a number of workarounds to fully work in userspace? There exists a surprising amount of scripts which wrap debootstrap/multistrap. Most of them require superuser privileges. Does everybody just accept that they have to put a sudo in front of every invocation and hope for the best? While this might be okay for well tested code like debootstrap and multistrap the countless wrapper scripts might accidentally (be it a bug in the code or a typo in the given command line arguments) write to your primary hard disk instead of your SD card. Such behavior can easily be mitigated by not executing any such script with superuser privileges in the first place. Operations like loop mounting affect the whole system. Why do I have to touch anything outside of my home directory (/dev/loop in this case) to populate a file in it with some meaningful bits? Virtualization is no option because every virtualization solution again requires root privileges. One might argue that a number of solutions just require some initial setup by root to then later be used by a regular user (for example /etc/fstab configuration or the schroot approach). But then again: why do I have to write anything outside of my home directory (even if it is only once) to be able to write something meaningful to a file in it? The latter approach also does not work if one cannot become root in the first place or is limited by a virtualized environment. Imagine you are trying to build a Debian rootfs on a machine where you just have a regular user account. Or a situation I was recently in: I had a virtual server which denied me operations like loop mounting. Given all these downsides, why is it still so common to just assume that one is able and willing to use sudo and be done with it in most cases? I really wonder why technologies like fakeroot and fakechroot have only been developed this late. Has this problem not been around since the earliest days of Linux/Unix? Am I missing something and rambling around for nothing? Is this idea a lost cause or something that is worth spending time and energy on to extend and fix the required tools?

Next.