Search Results: "lucas"

2 May 2016

Vincent Bernat: Pragmatic Debian packaging

While the creation of Debian packages is abundantly documented, most tutorials are targeted to packages implementing the Debian policy. Moreover, Debian packaging has a reputation of being unnecessarily difficult1 and many people prefer to use less constrained tools2 like fpm or CheckInstall. However, I would like to show how building Debian packages with the official tools can become straightforward if you bend some rules:
  1. No source package will be generated. Packages will be built directly from a checkout of a VCS repository.
  2. Additional dependencies can be downloaded during build. Packaging individually each dependency is a painstaking work, notably when you have to deal with some fast-paced ecosystems like Java, Javascript and Go.
  3. The produced packages may bundle dependencies. This is likely to raise some concerns about security and long-term maintenance, but this is a common trade-off in many ecosystems, notably Java, Javascript and Go.

Pragmatic packages 101 In the Debian archive, you have two kinds of packages: the source packages and the binary packages. Each binary package is built from a source package. You need a name for each package. As stated in the introduction, we won t generate a source package but we will work with its unpacked form which is any source tree containing a debian/ directory. In our examples, we will start with a source tree containing only a debian/ directory but you are free to include this debian/ directory into an existing project. As an example, we will package memcached, a distributed memory cache. There are four files to create:
  • debian/compat,
  • debian/changelog,
  • debian/control, and
  • debian/rules.
The first one is easy. Just put 9 in it:
echo 9 > debian/compat
The second one has the following content:
memcached (0-0) UNRELEASED; urgency=medium
  * Fake entry
 -- Happy Packager <happy@example.com>  Tue, 19 Apr 2016 22:27:05 +0200
The only important information is the name of the source package, memcached, on the first line. Everything else can be left as is as it won t influence the generated binary packages.

The control file debian/control describes the metadata of both the source package and the generated binary packages. We have to write a block for each of them.
Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>
Package: memcached
Architecture: any
Description: high-performance memory object caching system
The source package is called memcached. We have to use the same name as in debian/changelog. We generate only one binary package: memcached. In the remaining of the example, when you see memcached, this is the name of a binary package. The Architecture field should be set to either any or all. Use all exclusively if the package contains only arch-independent files. In doubt, just stick to any. The Description field contains a short description of the binary package.

The build recipe The last mandatory file is debian/rules. It s the recipe of the package. We need to retrieve memcached, build it and install its file tree in debian/memcached/. It looks like this:
#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@
override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
    ./configure --prefix=/usr
    make
    make install DESTDIR=debian/memcached
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)
The empty targets override_dh_auto_clean, override_dh_auto_test and override_dh_auto_build keep debhelper from being too smart. The override_dh_gencontrol target sets the package version3 without updating debian/changelog. If you ignore the slight boilerplate, the recipe is quite similar to what you would have done with fpm:
DISTRIBUTION=$(lsb_release -sr)
VERSION=1.4.25
PACKAGEVERSION=$ VERSION -0~$ DISTRIBUTION 0
TARBALL=memcached-$ VERSION .tar.gz
URL=http://www.memcached.org/files/$ TARBALL 
wget -N --progress=dot:mega $ URL 
tar --strip-components=1 -xf $ TARBALL 
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir
# Build the final package
fpm -s dir -t deb \
    -n memcached \
    -v $ PACKAGEVERSION  \
    -C /tmp/installdir \
    --description "high-performance memory object caching system"
You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 102 At this point, we can iterate and add several improvements to our memcached package. None of those are mandatory but they are usually worth the additional effort.

Build dependencies Our initial build recipe only work when several packages are installed, like wget and libevent-dev. They are not present on all Debian systems. You can easily express that you need them by adding a Build-Depends section for the source package in debian/control:
Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev
Always specify the debhelper (>= 9) dependency as we heavily rely on it. We don t require make or a C compiler because it is assumed that the build-essential meta-package is installed and it pulls those. dpkg-buildpackage will complain if the dependencies are not met. If you want to install those packages from your CI system, you can use the following command4:
mk-build-deps \
    -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
    -i -r debian/control
You may also want to investigate pbuilder or sbuild, two tools to build Debian packages in a clean isolated environment.

Runtime dependencies If the resulting package is installed on a freshly installed machine, it won t work because it will be missing libevent, a required library for memcached. You can express the dependencies needed by each binary package by adding a Depends field. Moreover, for dynamic libraries, you can automatically get the right dependencies by using some substitution variables:
Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends 
The resulting package will contain the following information:
$ dpkg -I ../memcached_1.4.25-0\~unstable0_amd64.deb   grep Depends
 Depends: libc6 (>= 2.17), libevent-2.0-5 (>= 2.0.10-stable)

Integration with init system Most packaged daemons come with some integration with the init system. This integration ensures the daemon will be started on boot and restarted on upgrade. For Debian-based distributions, there are several init systems available. The most prominent ones are:
  • System-V init is the historical init system. More modern inits are able to reuse scripts written for this init, so this is a safe common denominator for packaged daemons.
  • Upstart is the less-historical init system for Ubuntu (used in Ubuntu 14.10 and previous releases).
  • systemd is the default init system for Debian since Jessie and for Ubuntu since 15.04.
Writing a correct script for the System-V init is error-prone. Therefore, I usually prefer to provide a native configuration file for the default init system of the targeted distribution (Upstart and systemd).

System-V If you want to provide a System-V init script, have a look at /etc/init.d/skeleton on the most ancient distribution you want to target and adapt it5. Put the result in debian/memcached.init. It will be installed at the right place, invoked on install, upgrade and removal. On Debian-based systems, many init scripts allow user customizations by providing a /etc/default/memcached file. You can ship one by putting its content in debian/memcached.default.

Upstart Providing an Upstart job is similar: put it in debian/memcached.upstart. For example:
description "memcached daemon"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
respawn limit 5 60
expect daemon
script
  . /etc/default/memcached
  exec memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
end script
When writing an Upstart job, the most important directive is expect. Be sure to get it right. Here, we use expect daemon and memcached is started with the -d flag.

systemd Providing a systemd unit is a bit more complex. The content of the file should go in debian/memcached.service. For example:
[Unit]
Description=memcached daemon
After=network.target
[Service]
Type=forking
EnvironmentFile=/etc/default/memcached
ExecStart=/usr/bin/memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure
[Install]
WantedBy=multi-user.target
We reuse /etc/default/memcached even if it is not considered a good practice with systemd6. Like for Upstart, the directive Type is quite important. We used forking as memcached is started with the -d flag. You also need to add a build-dependency to dh-systemd in debian/control:
Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev,
               dh-systemd
And you need to modify the default rule in debian/rules:
%:
    dh $@ --with systemd
The extra complexity is a bit unfortunate but systemd integration is not part of debhelper7. Without those additional modifications, the unit will get installed but you won t get a proper integration and the service won t be enabled on install or boot.

Dedicated user Many daemons don t need to run as root and it is a good practice to ship a dedicated user. In the case of memcached, we can provide a _memcached user8. Add a debian/memcached.postinst file with the following content:
#!/bin/sh
set -e
case "$1" in
    configure)
        adduser --system --disabled-password --disabled-login --home /var/empty \
                --no-create-home --quiet --force-badname --group _memcached
        ;;
esac
#DEBHELPER#
exit 0
There is no cleanup of the user when the package is removed for two reasons:
  1. Less stuff to write.
  2. The user could still own some files.
The utility adduser will do the right thing whatever the requested user already exists or not. You need to add it as a dependency in debian/control:
Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends , adduser
The #DEBHELPER# marker is important as it will be replaced by some code to handle the service configuration files (or some other stuff). You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 103 It is possible to leverage debhelper to reduce the recipe size and to make it more declarative. This section is quite optional and it requires understanding a bit more how a Debian package is built. Feel free to skip it.

The big picture There are four steps to build a regular Debian package:
  1. debian/rules clean should clean the source tree to make it pristine.
  2. debian/rules build should trigger the build. For an autoconf-based software, like memcached, this step should execute something like ./configure && make.
  3. debian/rules install should install the file tree of each binary package. For an autoconf-based software, this step should execute make install DESTDIR=debian/memcached.
  4. debian/rules binary will pack the different file trees into binary packages.
You don t directly write each of those targets. Instead, you let dh, a component of debhelper, do most of the work. The following debian/rules file should do almost everything correctly with many source packages:
#!/usr/bin/make -f
%:
    dh $@
For each of the four targets described above, you can run dh with --no-act to see what it would do. For example:
$ dh build --no-act
   dh_testdir
   dh_update_autotools_config
   dh_auto_configure
   dh_auto_build
   dh_auto_test
Each of those helpers has a manual page. Helpers starting with dh_auto_ are a bit magic . For example, dh_auto_configure will try to automatically configure a package prior to building: it will detect the build system and invoke ./configure, cmake or Makefile.PL. If one of the helpers do not do the right thing, you can replace it by using an override target:
override_dh_auto_configure:
    ./configure --with-some-grog
Those helpers are also configurable, so you can just alter a bit their behaviour by invoking them with additional options:
override_dh_auto_configure:
    dh_auto_configure -- --with-some-grog
This way, ./configure will be called with your custom flag but also with a lot of default flags like --prefix=/usr for better integration. In the initial memcached example, we overrode all those magic targets. dh_auto_clean, dh_auto_configure and dh_auto_build are converted to no-ops to avoid any unexpected behaviour. dh_auto_install is hijacked to do all the build process. Additionally, we modified the behavior of the dh_gencontrol helper by forcing the version number instead of using the one from debian/changelog.

Automatic builds As memcached is an autoconf-enabled package, dh knows how to build it: ./configure && make && make install. Therefore, we can let it handle most of the work with this debian/rules file:
#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@ --with systemd
override_dh_auto_clean:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
override_dh_auto_test:
    # Don't run the whitespace test
    rm t/whitespace.t
    dh_auto_test
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)
The dh_auto_clean target is hijacked to download and setup the source tree9. We don t override the dh_auto_configure step, so dh will execute the ./configure script with the appropriate options. We don t override the dh_auto_build step either: dh will execute make. dh_auto_test is invoked after the build and it will run the memcached test suite. We need to override it because one of the test is complaining about odd whitespaces in the debian/ directory. We suppress this rogue test and let dh_auto_test executes the test suite. dh_auto_install is not overriden either, so dh will execute some variant of make install. To get a better sense of the difference, here is a diff:
--- memcached-intermediate/debian/rules 2016-04-30 14:02:37.425593362 +0200
+++ memcached/debian/rules  2016-05-01 14:55:15.815063835 +0200
@@ -12,10 +12,9 @@
 override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
-   ./configure --prefix=/usr
-   make
-   make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+   # Don't run the whitespace test
+   rm t/whitespace.t
+   dh_auto_test
It is up to you to decide if dh can do some work for you, but you could try to start from a minimal debian/rules and only override some targets.

Install additional files While make install installed the essential files for memcached, you may want to put additional files in the binary package. You could use cp in your build recipe, but you can also declare them:
  • files listed in debian/memcached.docs will be copied to /usr/share/doc/memcached by dh_installdocs,
  • files listed in debian/memcached.examples will be copied to /usr/share/doc/memcached/examples by dh_installexamples,
  • files listed in debian/memcached.manpages will be copied to the appropriate subdirectory of /usr/share/man by dh_installman,
Here is an example using wildcards for debian/memcached.docs:
doc/*.txt
If you need to copy some files to an arbitrary location, you can list them along with their destination directories in debian/memcached.install and dh_install will take care of the copy. Here is an example:
scripts/memcached-tool usr/bin
Using those files make the build process more declarative. It is a matter of taste and you are free to use cp in debian/rules instead. You can review the whole package tree on GitHub.

Other examples The GitHub repository contains some additional examples. They all follow the same scheme:
  • dh_auto_clean is hijacked to download and setup the source tree
  • dh_gencontrol is modified to use a computed version
Notably, you ll find daemons in Java, Go, Python and Node.js. The goal of those examples is to demonstrate that using Debian tools to build Debian packages can be straightforward. Hope this helps.

  1. People may remember the time before debhelper 7.0.50 (circa 2009) where debian/rules was a daunting beast. However, nowaday, the boilerplate is quite reduced.
  2. The complexity is not the only reason. Those alternative tools enable the creation of RPM packages, something that Debian tools obviously don t.
  3. There are many ways to version a package. Again, if you want to be pragmatic, the proposed solution should be good enough for Ubuntu. On Debian, it doesn t cover upgrade from one distribution version to another, but we assume that nowadays, systems get reinstalled instead of being upgraded.
  4. You also need to install devscripts and equivs package.
  5. It s also possible to use a script provided by upstream. However, there is no such thing as an init script that works on all distributions. Compare the proposed with the skeleton, check if it is using start-stop-daemon and if it sources /lib/lsb/init-functions before considering it. If it seems to fit, you can install it yourself in debian/memcached/etc/init.d/. debhelper will ensure its proper integration.
  6. Instead, a user wanting to customize the options is expected to edit the unit with systemctl edit.
  7. See #822670
  8. The Debian Policy doesn t provide any hint for the naming convention of those system users. A common usage is to prefix the daemon name with an underscore (like _memcached). Another common usage is to use Debian- as a prefix. The main drawback of the latest solution is that the name is likely to be replaced by the UID in ps and top because of its length.
  9. We could call dh_auto_clean at the end of the target to let it invoke make clean. However, it is assumed that a fresh checkout is used before each build.

24 April 2016

Bits from Debian: Debian welcomes its 2016 summer interns

GSoC 2016 logo Outreachy logo We're excited to announce that Debian has selected 29 interns to work with us this summer: 4 in Outreachy, and 25 in the Google Summer of Code. Here is the list of projects and the interns who will work on them: Android SDK tools in Debian: APT - dpkg communications rework: Continuous Integration for Debian-Med packages: Extending the Debian Developer Horizon: Improving and extending AppRecommender: Improving the debsources frontend: Improving voice, video and chat communication with Free Software: MIPS and MIPSEL ports improvements: Reproducible Builds for Debian and Free Software: Support for KLEE in Debile: The Google Summer of Code and Outreachy programs are possible in Debian thanks to the effort of Debian developers and contributors that dedicate part of their free time to mentor students and outreach tasks. Join us and help extend Debian! You can follow the students weekly reports on the debian-outreach mailing-list, chat with us on our IRC channel or on each project's team mailing lists. Congratulations to all of them!

7 April 2016

Petter Reinholdtsen: One in two hundred Debian users using ZFS on Linux?

Just for fun I had a look at the popcon number of ZFS related packages in Debian, and was quite surprised with what I found. I use ZFS myself at home, but did not really expect many others to do so. But I might be wrong. According to the popcon results for spl-linux, there are 1019 Debian installations, or 0.53% of the population, with the package installed. As far as I know the only use of the spl-linux package is as a support library for ZFS on Linux, so I use it here as proxy for measuring the number of ZFS installation on Linux in Debian. In the kFreeBSD variant of Debian the ZFS feature is already available, and there the popcon results for zfsutils show 1625 Debian installations or 0.84% of the population. So I guess I am not alone in using ZFS on Debian. But even though the Debian project leader Lucas Nussbaum announced in April 2015 that the legal obstacles blocking ZFS on Debian were cleared, the package is still not in Debian. The package is again in the NEW queue. Several uploads have been rejected so far because the debian/copyright file was incomplete or wrong, but there is no reason to give up. The current status can be seen on the team status page, and the source code is available on Alioth. As I want ZFS to be included in next version of Debian to make sure my home server can function in the future using only official Debian packages, and the current blocker is to get the debian/copyright file accepted by the FTP masters in Debian, I decided a while back to try to help out the team. This was the background for my blog post about creating, updating and checking debian/copyright semi-automatically, and I used the techniques I explored there to try to find any errors in the copyright file. It is not very easy to check every one of the around 2000 files in the source package, but I hope we this time got it right. If you want to help out, check out the git source and try to find missing entries in the debian/copyright file.

2 March 2016

Antonio Terceiro: Debian Ruby Sprint 2016 - day 2: Japanese cuisine, bug fixes, and Mini Cheese&Wine Party

Day 1 ended with dinner at a Yamato, my preferred Japanese restaurant in the city. Curitiba has a very large Japanese community, and lots of Japanese restaurants. Yamato, however, is the only one were you will stumble upon senior Japanese people, probably first or second generation immigrants, what I guess says something about its authenticity. Right after breaking for lunch, but before actually going out, we made what so far is official group photo (I might try again as the shot was not a really good one). Of course the most interesting part was the actual work that was done, and day 2 list is not less impressive than the day before: On Monday C dric told us that he and Sebastien had brought a bottle of French wine and some smelly French cheeses, and suggested that in the best Debian tradition we should have a Mini Cheese and Wine Party . Sure thing! Luckily there is a farmer s market 2 blocks from home on Tuesdays mornings, where I usually buy my fruits, vegetables, and cheese & friends, so the timing was perfect. I went shopping early in the morning, and bought a few things, and was back before it was the time to go to UTFPR. After the day-long hacking session we stopped by another store nearby to buy a few extra bottles of wine and other snacks. At night, in my place, I ended up playing cheese master. There was enough food that at the end we were all very full. And with the spokesperson task of the day done, off to hacking I am!

2 January 2016

Niels Thykier: dput change-all-of-debian.changes

Lucas Nussbaum recently did a blog post called Debian is still changing . I found it a very welcome continuation of his previous blog post on the same topic. I find the graphs very interesting and was very happy to learn that he included relative graphs this time. Now I can with relatively ease say that 69% of all Debian packages are using a dh-style build (source). We have another 15% using classic debhelper, which means that at least 84% of all packages uses debhelper directly. Assuming all CDBS based packages rely on the debhelper class , we are at 99%! The latter is certainly an assumption, although I suspect it is probably pretty accurate[1]. Now, it is very cute to have world dominance , but that is not my primary interest in these numbers. Instead, we can use these numbers to determine that: Such as automatic dbgsym packages, indexable build-id from dbg(sym) packages via Packages files[2], and replacing maintscripts with ldconfig triggers. All of these changes happen to be changes that could be trivially deployed with very little risk and very high efficiency[3]. Notably, none of them required a compat bump (or a new debhelper tool). Of course, I do not intend to say that every change can (or should) be deployed via debhelper and much less into an existing dh_cmd -tool. Notably, dh_strip is reaching its breaking point for content. And if we were to require a compat bump for your change, we can now at least see the adoption rate via lintian. :) Nevertheless, it is nice to know that (politics aside) there is some agility in the Debian build system! :) [1] I would very much love to see numbers to (dis)prove my assumption about CDBS + debhelper. In fact, an absolute number of packages not using debhelper (indirectly) in Debian would be very intriguing. [2] New fields by default end up the Packages file. See e.g. the Packages.xz file on the debug mirror or your apt-cache via:
apt-cache show mscgen-dbgsym   grep ^Build-Ids
The latter assumes that you have the debug mirror in your sources list. [3] Efficiency here being features people rarely override/disable.
Filed under: Debhelper, Debian

31 December 2015

Rapha&#235;l Hertzog: My Free Software Activities in December 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 21.25 hours on Debian LTS. During this time I worked on the following things: Distro Tracker I put a big focus on tracker.debian.org work this month. I completed the switch of the mail interface from packages.qa.debian.org to tracker.debian.org and I announced the change on debian-devel-announce. The changes resulted in a few problems that I quickly fixed (like #807073) and some other failures seen only by me and that were generated by weird spam messages (did you know that a subject can t have a newline character but that it can be encoded and folded over multiple lines?). Related to that I fixed some services so that they send their mails to tracker.debian.org directly instead of relying on the old emails (they get forwarded for now but it would be nice to be able to get rid of that forward). I updated (with the help of Lucas Nussbaum) the service that forwards the Launchpad bugs to the tracker, I sent a patch to update the @packages.debian.org aliases (not yet applied), I updated the configuration of all git commit notice scripts in the Alioth collab-maint and python-modules project (many remain to be done). I asked Ubuntu s Merge-O-Matic to use the new emails as well (see LP 1525497). DAK and the Debian BTS still have to be updated, as of yet nobody reacted to my announce last but not least I updated many wiki pages which duplicated the instructions to setup the commit notice sent to the PTS. While on a good track I opted to tackle the long-standing RC bug that was plaguing tracker.debian.org (#789183), so I updated the codebase to rely on Twitter s bootstrap v4 instead of v2. I had to switch to something else for the icons since glyphicons is no longer provided as part of bootstrap and the actual license for the standalone version was not suitable for use. I opted for Github s Octicons. I made numerous little improvements while doing that (closing some bugs in the process) and I believe that the result is more pleasant to use. I also did a lot of bug triage and fixed a few small issues like the incomplete architecture list (#793547), or fixing a page used only by people with javascript disabled that was not working. Or the invalid links for packages still using CVS (ugh, see #561228). Misc packaging Django. After having added DEP-8 tests (as part of my LTS work, see above), I discovered that the current version in unstable did not pass its test suite so I filed the issue upstream (ticket 26016) and added the corresponding patch. And I encouraged others to update python-bcrypt in Debian to a newer version that would have worked with Django 1.9 (see #803096). I also fixed another small issue in Django (see ticket 26017 with my pull request that got accepted). I asked the release managers to consider accepting the latest 1.7.x version in jessie (see #807654) but I have gotten zero answer so far. And I m not the only one waiting an answer. It s a bit of a sad situation we still have a few weeks until the next point release but for once I do it in advance and I would love to have timely feedback. Last but not least, I started the maintaining the current LTS release (1.8.x) in jessie-backports. Tryton. I upgraded to Tryton 3.8 and discovered an issue that I filed in #806781. I sponsored 5 new tryton modules for Matthias Behrle (who is DM) as well as one security upload (for CVE-2015-0861). Debian Handbook. I uploaded a new version to Debian Unstable and requested (to the release managers) the permission to upload a backport of it to jessie so that jessie has a version of the package that documents jessie and not wheezy contrary to my other Django request, this one should be non-controversial but I also have had zero answer so far, see #807515. Misc. I filed #808583 when sbuild stopped working with Perl 5.22. I handled #807860 on publican, I found the corresponding upstream ticket and discovered a work around with the help of upstream (see here). Kali related work I reported a bug to #debian-apt about apt miscalculating download size (ending up with 18 EB!) which resulted in a fix here in version 1.1.4. Installing a meta-package that needed more than 2GB was no longer possible without this fix and we have a kali-linux-all metapackage in that situation that gets regularly installed in a Jenkins test. I added captcha support to Distro Tracker and enabled this feature on pkg.kali.org. I filed #808863 against uhd-host because it was not possible to install the package in a systemd-nspawn s managed chroot where /proc is read-only. And we started using this to test dist-upgrade from one version of Kali to the next Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

28 December 2015

Hideki Yamane: How about "grooming" outdated packages?

Lucas's good article: DEBIAN PACKAGES WITH /OUTDATED/ PACKAGING STYLE

Okay, then what's next? Action.

How about do "grooming" for those packages with contributors? Well experienced DDs become mentors and help to make those packages modern and put it to repo.


Packages become simple and modern, contributors can get packaging experience.
Isn't it a good idea? (Meow!)

26 December 2015

Lucas Nussbaum: Debian packages with /outdated/ packaging style

(This is just a copy of this debian-devel@ email) Following my blog post yesterday with graphs about Debian packaging evolution, I prepared lists of packages for each kind of outdatedness. Of course not all practices highlighted below are deprecated, and there are good reasons to continue to do some of them. But still, given that they all represent a clear minority of packages, I thought that it would be useful to list the related packages. (I honestly didn t know if some of my packages would show up in the lists!) The lists are available at https://people.debian.org/~lucas/qa-20151226/ I also pushed them to alioth, so you can either do:
ssh people.debian.org 'grep -A 10 YOURNAME ~lucas/public_html/qa-20151226/*ddlist'
or:
ssh alioth.debian.org 'grep -A 10 YOURNAME ~lucas/qa-20151226/*ddlist' the meaning of the lists is: If you don t understand why your package is listed, you can have a look at allpackages-20151226.yaml that provides more details. If you still don t understand, just ask me. Excluding duplicates, a total of 5469 packages are listed. The dd-list output for the merged list is also available (which isn t very useful, except to know if you are listed).

25 December 2015

Lucas Nussbaum: Debian is still changing

Here is an update to the usual graphs generated from snapshot.d.o. See my previous blog post for the background info. In all graphs, it s easy to see the effect of the Jessie freeze (and the previous freezes since 2005, too). Team maintenance comaint-2015 It s interesting to see that, while the number of team-maintained packages increases, the number of packages that aren t co-maintained is very stable. Maintenance using a VCS vcs-2015 Git is the clear winner now, with the migration rate increasing recently. Packaging helpers helpers-2015 As usual, the number of packages using CDBS is quite stable. The number of packages using traditional debhelper might soon be lower than those using CDBS. Patch systems and packaging formats formats-patches-2015 3.0 is the clear winner, even if we still have 3000+ packages using 1.0, and ~1000 of those modifying files directly. The other patch systems have basically disappeared.
So, all those graphs are kind-of boring now. Any good ideas of additional things to track, that be can identified reliably by looking at source packages? For those interested, below are links to the graphs with percentages of packages. comaint-percent-2015 formats-patches-percent-2015 vcs-percent-2015 helpers-percent-2015

1 September 2015

Rapha&#235;l Hertzog: My Free Software Activities in August 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 6.5 hours on Debian LTS. In that time I did the following: Apart from that, I also gave a talk about Debian LTS at DebConf 15 in Heidelberg and also coordinated a work session to discuss our plans for Wheezy. Have a look at the video recordings: DebConf 15 I attended DebConf 15 with great pleasure after having missed DebConf 14 last year. While I did not do lots of work there, I participated in many discussions and I certainly came back with a renewed motivation to work on Debian. That s always good. :-) For the concrete work I did during DebConf, I can only claim two schroot uploads to fix the lack of support of the new overlay filesystem that replaces aufs in the official Debian kernel, and some Distro Tracker work (fixing an issue that some people had when they were logged in via Debian s SSO). While the numerous discussions I had during DebConf can t be qualified as work , they certainly contribute to build up work plans for the future: As a Kali developer, I attended multiple sessions related to derivatives (notably the Debian Derivatives Panel). I was also interested by the Debian in the corporate IT BoF led by Michael Meskes (Credativ s CEO). He pointed out a number of problems that corporate users might have when they first consider using Debian and we will try to do something about this. Expect further news and discussions on the topic. Martin Kraff, Luca Filipozzi, and me had a discussion with the Debian Project Leader (Neil) about how to revive/transform the Debian s Partner program. Nothing is fleshed out yet, but at least the process initiated by the former DPL (Lucas) is again moving forward. Other Debian work Sponsorship. I sponsored an NMU of pep8 by Daniel Stender as it was a requirement for prospector which I also sponsored since all the required dependencies are now available in Debian. \o/ Packaging. I NMUed libxml2 2.9.2+really2.9.1+dfsg1-0.1 fixing 3 security issues and a RC bug that was breaking publican. Since there s no upstream fix for more than 8 months, I went back to the former version 2.9.1. It s in line with the new requirement of release managers a package in unstable should migrate to testing reasonably quickly, it s not acceptable to keep it unfixed for months. With this annoying bug fixed, I could again upload a new upstream release of publican so I prepared and uploaded 4.3.2-1. It was my first source only upload. This release was more work than I expected and I filed no less than 3 bug to upstream (new bash-completion install path, request to provide sources of a minified javascript file, drop a .po file for an invalid language code). GPG issues with smartcard. Back from DebConf, when I wanted to sign some key, I stumbled again upon the problem which makes it impossible for me to use my two smartcards one after the other without first deleting the stubs for the private key. It s not a new issue but I decided that it was time to report it upstream, so I did it: #2079 on bugs.gnupg.org. Some research helped me to find a way to work-around the problem. Later in the month, after a dist-upgrade and a reboot, I was no longer able to use my smartcard as a SSH authentication key again it was already reported but there was no clear analysis, so I tried to do my own one and added the results of my investigation in #795368. It looks like the culprit is pinentry-gnome3 not working when started by the gpg-agent which is started before the DBUS session. Simple fix is to restart the gpg-agent in the session but I have no idea yet of what the proper fix should be (letting systemd manage the graphical user session and start gpg-agent would be my first answer, but that doesn t solve the issue for users of other init systems so it s not satisfying). Distro Tracker. I merged two patches from Orestis Ioannou fixing some bugs tagged newcomer. There are more such bugs (I even filed two: #797096 and #797223), go grab them and do a first contribution to Distro Tracker like Orestis just did! I also merged a change from Christophe Siraut who presented Distro Tracker at DebConf. I implemented in Distro Tracker the new authentication based on SSL client certificates that was recently announced by Enrico Zini. It s working nice, and this authentication scheme is far easier to support. Good job, Enrico! tracker.debian.org broke during DebConf, it stopped being updated with new data. I tracked this down to a problem in the archive (see #796892). Apparently Ansgar Burchardt changed the set of compression tools used on some jessie repositorie, replacing bz2 by xz. He dropped the old Packages.bz2 but missed some Sources.bz2 which were thus stale and APT reported Hashsum mismatch on the uncompressed content. Misc. I pushed some small improvement to my Salt formulas: schroot-formula and sbuild-formula. They will now auto-detect which overlay filesystem is available with the current kernel (previously aufs was hardcoded). Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

28 August 2015

Lucas Nussbaum: DebConf 15

I attended DebConf 15 last week. After being on semi-vacation from Debian for the last few months, recovering after the end of my second DPL term, it was great to be active again, talk to many people, and go back to doing technical work. Unfortunately, I caught the debbug quite early in the week, so I was not able to make it as intense as I wanted, but it was great nevertheless. I still managed to do quite a lot: DC15 was a great DebConf, probably one of the two bests I ve attended so far. I m now looking forward to DC16 in Cape Town!

21 August 2015

Simon Kainz: DUCK challenge: Final week

Well, here are the stats for the final week of the DUCK challenge as well as DebConf15: So we had 21 packages fixed and uploaded by 14 different uploaders. People were really working hard on this during DebConf. A big "Thank You" to you!! Since the start of this challenge, a total of 89 packages, were fixed. Here is a quick overview:
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
# Packages 10 15 10 14 10 9 21
Total 10 25 35 49 59 68 89
Thank you all for participating - either on purpose or "accidentially": Some people were really surprised as i sneaked up on them at DebConf15, confronting them with a green lighter! I just tried to put even more fun into Debian, i hope this worked out Pevious articles are here: Week 1, Week 2, Week 3, Week 4, Week 5,Week 6.

11 August 2015

Jonathan McDowell: Programming the FST-01 (gnuk) with a Bus Pirate + OpenOCD

Last year at DebConf14 Lucas authorized the purchase of a handful of gnuk devices, one of which I obtained. At the time it only supported 2048 bit RSA keys. I took a look at what might be involved in adding 4096 bit support during DebConf and managed to brick my device several times in doing so. Thankfully gniibe was on hand with his STLinkV2 to help me recover. However subsequently I was loathe to experiment further at home until I had a suitable programmer. As it is this year has been busy and the 1.1.x release train is supposed to have 4K RSA (as well as ECC) support. DebConf15 is coming up and I felt I should finally sort out playing with the device properly. I still didn t have a suitable programmer. Or did I? Could my trusty Bus Pirate help? The FST-01 has an STM32F103TB on it. There is an exposed SWD port. I found a few projects that claimed to do SWD with a Bus Pirate - Will Donnelly has a much cloned Python project, the MC HCK project have a programmer in Ruby and there s LibSWD though that s targeted to smarter programmers. None of them worked for me; I could get the Python bits as far as correctly doing the ID of the device, but not reading the option bytes or successfully flashing (though I did manage an erase). Enter the old favourite, OpenOCD. This already has SWD support and there s an outstanding commit request to add Bus Pirate support. NodoNogard has a post on using the ST-Link/V2 with OpenOCD and the FST-01 which provided some useful pointers. I grabbed the patch from Gerrit, applied it to OpenOCD git and built an openocd.cfg that contained:
source [find interface/buspirate.cfg]
buspirate_port /dev/ttyUSB0
buspirate_vreg 1
buspirate_mode normal
transport select swd
source [find target/stm32f1x.cfg]
My BP has the Seeed Studio probe cable, so my hookups look like this: Bus Pirate + FST-01 SWD connection That s BP MOSI (grey) to SWD IO, BP CLK (purple) to SWD CLK, BP 3.3V (red) to FST-01 PWR and BP GND (brown) to FST-01 GND. Once that was done I fired up OpenOCD in one terminal and did the following in another:
$ telnet localhost 4444
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
> reset halt
target state: halted
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc
Info : device id = 0x20036410
Info : SWD IDCODE 0x1ba01477
Error: Failed to read memory at 0x1ffff7e2
Warn : STM32 flash size failed, probe inaccurate - assuming 128k flash
Info : flash size = 128kbytes
> stm32f1x unlock 0
Device Security Bit Set
stm32x unlocked.
INFO: a reset or power cycle is required for the new settings to take effect.
> reset halt
target state: halted
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc
> flash write_image erase /home/noodles/checkouts/gnuk/src/build/gnuk.elf
auto erase enabled
wrote 109568 bytes from file /home/noodles/checkouts/gnuk/src/build/gnuk.elf in 95.055603s (1.126 KiB/s)
> stm32f1x lock 0
stm32x locked
> reset halt
target state: halted
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x08000280 msp: 0x20005000
Then it was a matter of disconnecting the gnuk from the BP, plugging it into my USB port and seeing it come up successfully:
usb 1-2: new full-speed USB device number 11 using xhci_hcd
usb 1-2: New USB device found, idVendor=234b, idProduct=0000
usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-2: Product: Gnuk Token
usb 1-2: Manufacturer: Free Software Initiative of Japan
usb 1-2: SerialNumber: FSIJ-1.1.7-87063020
usb 1-2: ep 0x82 - rounding interval to 1024 microframes, ep desc says 2040 microframes
More once I actually have a 4K key loaded on it.

7 August 2015

Christoph Egger: Systemd pitfalls

logind hangs If you just updated systemd and ssh to that host seems to hang, that's just a known Bug (Debian Bug). Don't panic. Wait for the logind timeout and restart logind. restart and stop;start One thing that confused me several times and still confuses people is systemctl restart doing more than systemctl stop ; systemctl start. You will notice the difference once you have a failed service. A restart will try to start the service again. Both stop and start however will just ignore it. Rumors have it this has changed post jessie however. sysvinit-wrapped services and stop While there are certainly bugs with sysvinit services in general (I found myself several times without a local resolver as unbound failed to be started, haven't managed to debug further), the stop behavior of wrapped services is just broken. systemctl stop will block until the sysv initscript finished. It will even note the result of the action in its state. However systemctl will return with exitcode 0 and not output anything on stdout/stderr. This has been reported as Debian Bug. zsh helper I found the following zshrc snipped quite helpful in dealing with non-reported systemctl failures. On root shells it will display a list of failed services as part of the prompt. This will give proper feedback whether your systemctl stop failed, it will give feedback if you still have type=simple services and if the sysv-init script or wrapper is broken.
precmd ()  
    if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
    then
      use_systemd=true
      systemd_failed=" systemctl --state=failed   grep failed   cut -d \  -f 2   tr '\n' ' ' "
    fi
 
if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
then
  PROMPT=$'% $fg[red]>>  $systemd_failed$reset_color% \n'
else
  PROMPT=""
fi
PROMPT+=whateveryourpromptis
zsh completion Speaking of zsh, there's one problem that bothers me a lot and I don't have any solution for. Tab-completing the service name for service is blazing fast. Tab-completing the service name for systemctl restart takes ages. People traced down to truckloads of dbus communication for the completion but no further fix is known (to me). type=simple services As described in length by Lucas Nussbaum type=simple services are actively harmful. Proper type=forking daemons are strictly superior (they provide feedback of finished initialization and success thereof) and type=notify services are so simple there's no excuse for not using them even for private one-off hacks. Even if you're language doesn't provide libsystemd-daemon bindings:
(defun sd-notify (event-string)
  (let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :datagram))
        (name (posix-getenv "NOTIFY_SOCKET"))
        (bufferlen (length event-string)))
    (when name
      (sb-bsd-sockets:socket-connect socket name)
      (sb-bsd-sockets:socket-send socket event-string bufferlen))))
This is a stable API guaranteed to not break in the future and implemented in less than ten lines of code with just basic socket functions. And if your language has support it becomes actually trivial:
    try:
        import systemd
        systemd.daemon.notify("READY=1")
    except ImportError:
        pass
Note that in both cases there is no drawback at all on systemd-free setups. It has the overhead of checking the process' environment for NOTIFY_SOCKET or for the systemd package and behaves like a simple service otherwise. Actually the idea of separating the technical aspect (daemonizing) from the semantic aspect of signalizing "initialization finished, everything's fine" is a pretty good idea and hopefully has the potential to reduce the number of services signalizing the "everything's fine" too early. It could even be ported to non-systemd init systems easily given the API.

Christoph Egger: Systemd pitfalls

logind hangs If you just updated systemd and ssh to that host seems to hang, that's just a known Bug (Debian Bug #770135). Don't panic. Wait for the logind timeout and restart logind. restart and stop;start One thing that confused me several times and still confuses people is systemctl restart doing more than systemctl stop ; systemctl start. You will notice the difference once you have a failed service. A restart will try to start the service again. Both stop and start however will just ignore it. Rumors have it this has changed post jessie however. sysvinit-wrapped services and stop While there are certainly bugs with sysvinit services in general (I found myself several times without a local resolver as unbound failed to be started, haven't managed to debug further), the stop behavior of wrapped services is just broken. systemctl stop will block until the sysv initscript finished. It will even note the result of the action in its state. However systemctl will return with exitcode 0 and not output anything on stdout/stderr. This has been reported as Debian Bug #792045. zsh helper I found the following zshrc snipped quite helpful in dealing with non-reported systemctl failures. On root shells it will display a list of failed services as part of the prompt. This will give proper feedback whether your systemctl stop failed, it will give feedback if you still have type=simple services and if the sysv-init script or wrapper is broken.
precmd ()  
    if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
    then
      use_systemd=true
      systemd_failed=" systemctl --state=failed   grep failed   cut -d \  -f 2   tr '\n' ' ' "
    fi
 
if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
then
  PROMPT=$'% $fg[red]>>  $systemd_failed$reset_color% \n'
else
  PROMPT=""
fi
PROMPT+=whateveryourpromptis
zsh completion Speaking of zsh, there's one problem that bothers me a lot and I don't have any solution for. Tab-completing the service name for service is blazing fast. Tab-completing the service name for systemctl restart takes ages. People traced down to truckloads of dbus communication for the completion but no further fix is known (to me). type=simple services As described in length by Lucas Nussbaum type=simple services are actively harmful. Proper type=forking daemons are strictly superior (they provide feedback of finished initialization and success thereof) and type=notify services are so simple there's no excuse for not using them even for private one-off hacks. Even if you're language doesn't provide libsystemd-daemon bindings:
(defun sd-notify (event-string)
  (let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :datagram))
        (name (posix-getenv "NOTIFY_SOCKET"))
        (bufferlen (length event-string)))
    (when name
      (sb-bsd-sockets:socket-connect socket name)
      (sb-bsd-sockets:socket-send socket event-string bufferlen))))
This is a stable API guaranteed to not break in the future and implemented in less than ten lines of code with just basic socket functions. And if your language has support it becomes actually trivial:
    try:
        import systemd
        systemd.daemon.notify("READY=1")
    except ImportError:
        pass
Note that in both cases there is no drawback at all on systemd-free setups. It has the overhead of checking the process' environment for NOTIFY_SOCKET or for the systemd package and behaves like a simple service otherwise. Actually the idea of separating the technical aspect (daemonizing) from the semantic aspect of signalizing "initialization finished, everything's fine" is a pretty good idea and hopefully has the potential to reduce the number of services signalizing the "everything's fine" too early. It could even be ported to non-systemd init systems easily given the API.

5 August 2015

Antonio Terceiro: Elixir in Debian, MiniDebconf at FISL, and Debian CI updates

In June I started keeping track of my Debian activities, and this is my July update. Elixir in Debian Elixir is a functional language built on top of the Erlang virtual machine. If features imutable data structures, interesting concurrency primitives, and everything else that Erlang does, but with a syntax inspired by Ruby what makes it much more aproachable in my opinion. Those interested in Elixir for Debian are encouraged to hang around in #debian-elixir on the OFTC IRC servers. There are still a lot of things to figure out, for example how packaging Elixir libraries and applications is going to work. MiniDebconf at FISL, and beyond I helped organize a MiniDebconf at this year s FISL, in Porto Alegre on the 10th of July. The whole program was targetted at getting more people to participate in Debian, so there were talks about translation, packaging, and a few other more specific topics. I myself gave two talks: one about Debian basics, What is Debian, and how it works , and second one on packaging the free software web , which I will also give at Debconf15 later this month. The recordings are available (all talks in Portuguese) at the Debian video archive thanks to Holger Levsen. We are also organizing a new MiniDebconf in October as part of the Latinoware schedule. Ruby We are in the middle of a transition to switch to Ruby 2.2 as default in Debian unstable, and we are almost there. The Ruby transition is now on hold while GCC 5 one is going on, but will be picked up as soon as were are done with GCC 5. ruby-defaults has been uploaded to experimental for those that want to try having Ruby 2.2 as default before that change hits unstable. I myself have been using Ruby 2.2 as default for several weeks without any problem so far, including using vagrant on a daily basis and doing all my development on sid with it. I started taking notes about Ruby interpreter transitions work to make sure that knowledge is registered. I have uploaded minor security updates of both ruby2.1 and ruby2.2 to unstable. They both reached testing earlier today. I have also fixed another bug in redmine, which I hope to get into stable as well as soon as possible. gem2deb has seen several improvements through versions 0.19, 0.20, 0.20.1 and 0.20.2. I have updated a few packages: Two NEW packages, ruby-rack-contrib and ruby-grape-logging ,were ACCEPTED into the Debian archive. Kudos to the ftp-master team who are doing an awesome job reviewing new packages really fast. Debian Continuous Integration This month I have made good progress with the changes needed to make debci work as a distributed system with one master/scheduler node and as many worker nodes (running tests) as possible. While doing my tests, I have submitted a patch to lxc and updated autodep8 in unstable. At some point I plan to upload both autodep8 and autopkgtest to jessie-backports. Sponsoring I have sponsored a few packages:

31 July 2015

Simon Kainz: DUCK challenge: week 4

The DUCK challenge is making a quite stable progress: in the last 4 weeks there were approximately 12.25 packages fixed and uploaded per week. In the current week the following packages were fixed and uploaded into unstable: So we had 14 packages fixed and uploaded by 10 different uploaders. A big "Thank You" to you!! Since the start of this challenge, a total of 49 packages, uploaded by 31 different persons were fixed. Here is a quick overview:
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
# Packages 10 15 10 14 - - -
Total 10 25 35 49 - - -
The list of the fixed and updated packages is availabe here. I will try to update this ~daily. If I missed one of your uploads, please drop me a line. DebConf15 is approaching quite fast, so please get involved: The DUCK Challenge is running until end of DebConf15! Pevious articles are here: Week 1, Week 2, Week 3.

2 July 2015

Antonio Terceiro: Upgrades to Jessie, Ruby 2.2 transition, and chef update

Last month I started to track all the small Debian-related things that I do. My initial motivation was to be concious about how often I spend short periods of time working on Debian. Sometimes it s during lunch breaks, weekends, first thing in the morning before regular work, after I am done for the day with regular work, or even during regular work, since I do have the chance of doing Debian work as part of my regular work occasionally. Now that I have this information, I need to do something with it. So this is probably the first of monthly updates I will post about my Debian work. Hopefully it won t be the last. Upgrades to Jessie I (finally) upgraded my two servers to Jessie. The first one, my home server, is a Utilite which is a quite nice ARM box. It is silent and consumes very little power. The only problem I had with it is that the vendor-provided kernel is too old, so I couldn t upgrade udev, and therefore couldn t switch to systemd. I had to force systemv for now, until I can manage to upgrade the kernel and configure uboot to properly boot the official Debian kernel. On my VPS things are way better. I was able to upgrade nicely, and it is now running a stock Jessie system. fixed https on ci.debian.net pabs had let me know on IRC of an issue with the TLS certificate for ci.debian.net, which took me a few iterations to get right. It was missing the intermediate certificates, and is now fixed. You can now enjoy Debian CI under https . Ruby 2.2 transition I was able to start the Ruby 2.2 transition, which has the goal of switch to Ruby 2.2 on unstable. The first step was updating ruby-defaults adding support to build Ruby packgaes for both Ruby 2.1 and Ruby 2.2. This was followed by updates to gem2deb (0.18, 0.18.1, 0.18.2, and 0.18.3) and rubygems-integration . At this point, after a few rebuild requests only 50 out of 137 packages need to be looked at; some of them just use the default Ruby, so a rebuild once we switch the default will be enough to make it use Ruby 2.2, while others, specially Ruby libraries, will still need porting work or other fixes. Updated the Chef stack Bringing chef to the very latest upstream release into unstable was quite some work. I had to update: In the middle I also had to package a new dependency, ruby-ffi-yajl, which was very quickly ACCEPTED thanks to the awesome work of the ftp-master team. Random bits

20 June 2015

Lunar: Reproducible builds: week 4 in Stretch cycle

What happened about the reproducible builds effort for this week: Toolchain fixes Lunar rebased our custom dpkg on the new release, removing a now unneeded patch identified by Guillem Jover. An extra sort in the buildinfo generator prevented a stable order and was quickly fixed once identified. Mattia Rizzolo also rebased our custom debhelper on the latest release. Packages fixed The following 30 packages became reproducible due to changes in their build dependencies: animal-sniffer, asciidoctor, autodock-vina, camping, cookie-monster, downthemall, flashblock, gamera, httpcomponents-core, https-finder, icedove-l10n, istack-commons, jdeb, libmodule-build-perl, libur-perl, livehttpheaders, maven-dependency-plugin, maven-ejb-plugin, mozilla-noscript, nosquint, requestpolicy, ruby-benchmark-ips, ruby-benchmark-suite, ruby-expression-parser, ruby-github-markup, ruby-http-connection, ruby-settingslogic, ruby-uuidtools, webkit2gtk, wot. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Also, the following bugs have been reported: reproducible.debian.net Holger Levsen made several small bug fixes and a few more visible changes: strip-nondeterminism Version 0.007-1 of strip-nondeterminism the tool to post-process various file formats to normalize them has been uploaded by Holger Levsen. Version 0.006-1 was already in the reproducible repository, the new version mainly improve the detection of Maven's pom.properties files. debbindiff development At the request of Emmanuel Bourg, Reiner Herrmann added a comparator for Java .class files. Documentation update Christoph Berg created a new page for the timestamps in manpages created by Doxygen. Package reviews 93 obsolete reviews have been removed, 76 added and 43 updated this week. New identified issues: timestamps in manpages generated by Doxygen, modification time differences in files extracted by unzip, tstamp task used in Ant build.xml, timestamps in documentation generated by ASDocGen. The description for build id related issues has been clarified. Meetings Holger Levsen announced a first meeting on Wednesday, June 3rd, 2015, 19:00 UTC. The agenda is amendable on the wiki. Misc. Lunar worked on a proof-of-concept script to import the build environment found in .buildinfo files to UDD. Lucas Nussbaum has positively reviewed the proposed schema. Holger Levsen cleaned up various experimental toolchain repositories, marking merged brances as such.

13 May 2015

Lucas Nussbaum: systemd: Type=simple and avoiding forking considered harmful?

(This came up in a discussion on debian-user-french@l.d.o) When converting from sysvinit scripts to systemd init files, the default practice seems to be to start services without forking, and to use Type=simple in the service description. What Type=simple does is, well, simple. from systemd.service(5):

If set to simple (the default value if neither Type= nor BusName= are specified), it is expected that the process configured with ExecStart= is the main process of the service. In this mode, if the process offers functionality to other processes on the system, its communication channels should be installed before the daemon is started up (e.g. sockets set up by systemd, via socket activation), as systemd will immediately proceed starting follow-up units. In other words, systemd just runs the command described in ExecStart=, and it s done: it considers the service is started. Unfortunately, this causes a regression compared to the sysvinit behaviour, as described in #778913: if there s a configuration error, the process will start and exit almost immediately. But from systemd s point-of-view, the service will have been started successfully, and the error only shows in the logs:

root@debian:~# systemctl start ssh
root@debian:~# echo $?
0
root@debian:~# systemctl status ssh
  ssh.service - OpenBSD Secure Shell server
 Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
 Active: failed (Result: start-limit) since mer. 2015-05-13 09:32:16 CEST; 7s ago
 Process: 2522 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
 Main PID: 2522 (code=exited, status=255)
mai 13 09:32:16 debian systemd[1]: ssh.service: main process exited, code=exited, status=255/n/a
mai 13 09:32:16 debian systemd[1]: Unit ssh.service entered failed state.
mai 13 09:32:16 debian systemd[1]: ssh.service start request repeated too quickly, refusing to start.
mai 13 09:32:16 debian systemd[1]: Failed to start OpenBSD Secure Shell server.
mai 13 09:32:16 debian systemd[1]: Unit ssh.service entered failed state.
With sysvinit, this error is detected before the fork(), so it shows during startup:
root@debian:~# service ssh start
 [....] Starting OpenBSD Secure Shell server: sshd/etc/ssh/sshd_config: line 4: Bad configuration option: blah
 /etc/ssh/sshd_config: terminating, 1 bad configuration options
 failed!
 root@debian:~#
It s not trivial to fix that. The implicit behaviour of sysvinit is that fork() sort-of signals the end of service initialization. The systemd way to do that would be to use Type=notify, and have the service signals that it s ready using systemd-notify(1) or sd_notify(3) (or to use socket activation, but that s another story). However that requires changes to the service. Returning to the sysvinit behaviour by using Type=forking would help, but is not really a solution: but what if some of the initialization happens *after* the fork? This is actually the case for sshd, where the socket is bound after the fork (see strace -f -e trace=process,network /usr/sbin/sshd), so if another process is listening on port 22 and preventing sshd to successfully start, it would not be detected. I wonder if systemd shouldn t do more to detect problems during services initialization, as the transition to proper notification using sd_notify will likely take some time. A possibility would be to wait 100 or 200ms after the start to ensure that the service doesn t exit almost immediately. But that s not really a solution for several obvious reasons. A more hackish, but still less dirty solution could be to poll the state of processes inside the cgroup, and assume that the service is started only when all processes are sleeping. Still, that wouldn t be entirely satisfying

Next.

Previous.