Page 1 of 5 To be honest, I do not know how many mails it shows per page (though I assume it is a fixed number). So for comparison, I found the month on debian-devel@l.d.o with the highest volume in the past two years: May 2013 with Page 1 of 4 . I hope you will please forgive us, if we are a bit terse in our replies or slow to reply. We simply got a lot to deal with. :)

At the start of this week, I wrote that we had 58 open unblock requests open (of which 25 were tagged moreinfo). Thanks to an extra effort from the Release Team, we now down to 25 open unblocks of which 18 are tagged moreinfo. We have now resolved 442 unblock requests (out of a total of 467). The rate has also declined to an average of ~18 new unblock requests a day (over 26 days) and our closing rated increased to ~17. With all of this awesomeness, some of us are now more than ready to have a well-deserved weekend to recharge our batteries. Meanwhile, feel free to keep the RC bug fixes flowing into unstable.

The release team receives an extreme amount of unblock requests right now. For the past 22 days[1], we have been receiving no less than 408 unblock/ageing requests. That is an average of ~18.5/day. In the same period, the release team have closed 350 unblocks requests, averaging 15.9/day. This number does not account for number of unblocks, we add without a request, when we happen to spot when we look at the list of RC bugs[2]. Nor does it account for unblock requests currently tagged moreinfo , of which there are currently 25. All in all, it has been 3 intensive weeks for the release team. I am truly proud of my fellow team members for keeping up with this for so long! Also a thanks to the non-RT members, who help us by triaging and reviewing the unblock requests! It is much appreciated. :) Random bonus info:

• d (our diffing tool) finally got colordiff support during the Release Sprint last week. Prior to that, we got black n white diffs!
  • ssh coccia.debian.org -t /srv/release.debian.org/tools/scripts/d <srcpkg>
  • Though coccia.debian.org do not have colordiff installed right now. I have filed a request to have it installed.
• The release team have about 132 (active) unblock hints deployed right now in our hint files.

[1] We started receiving some in the 10 days before the freeze as people realised that their uploads would need an unblock to make it into Jessie. [2] Related topics: what is adsb? (the answer being: Our top hinter for Wheezy)

The release team are doing a sprint right now up the mini-DebConf in Cambridge, kindly hosted by ARM. We have a fairly large agenda of around 10 items, ranging from simple things like determine the name for the next release to reviewing the list of RC bugs affecting Jessie. Personally, I am very pleased with our progress. We managed to finish 8 of items on the first day. Furthermore, we spent several hours on the RC bug list and keeping up with the unblock requests. There is also live status of the release team, courtesy of Jonathan Wiltshire. Mind you it is manually updated. We will announce the results of our sprint Sunday morning in our Release update talk. The announcement will also be posted to debian-devel-announce like our freeze announcement. Update: fix link to the live status

The upcoming version of Lintian will feature a new set of API that attempts to promote safer code. It is hardly a ground-breaking discovery , just a much needed feature. The primary reason for this API is that writing safe and correct code is simply too complicated that people get it wrong (including yours truly on occasion). The second reason is that I feel it is a waste having to repeat myself when reviewing patches for Lintian. Fortunately, the kind of issues this kind of mistake creates are usually minor information leaks, often with no chance of exploiting it remotely without the owner reviewing the output first[0]. Part of the complexity of writing correct code originates from the fact that Lintian must assume Debian packages to be hostile until otherwise proven[1]. Consider a simplified case where we want to read a file (e.g. the copyright file):

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  # BAD: This is an example of doing it wrong
  open(my $fd, '<', $info->unpacked($filename));
  ...;
  close($fd);
  return;
 

This has two trivial vulnerabilities[2].

1. Any part of the path (usr,usr/share, ) can be asymlink to somewhere else like /
   1. Problem: Access to potentially any file on the system with the credentials of the user running Lintian. But even then, Lintian generally never write to those files and the user has to (usually manually) disclose the report before any information leak can be completed.
2. The target path can point to a non-file.
   1. Problem: Minor inconvenience by DoS of Lintian. Examples include a named pipe, where Lintian will get stuck until a signal kills it.

Of course, we can do this right[3]:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
use Lintian::Util qw(is_ancestor_of);
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $root = $info->unpacked
  my $path = $info->unpacked($filename);
  if ( -f $path and is_ancestor_of($root, $path))  
    open(my $fd, '<', $path);
    ...;
    close($fd);
   
  return;
 

Where is_ancestor_of is the only available utility to assist you currently. It hides away some 10-12 lines of code to resolve the two paths and correctly asserting that $path is (an ancestor of) $root. Prior to Lintian 2.5.12, you would have to do that ancestor check by hand in each and every check[4]. In the new version, the correct code would look something like this:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $path = $info->index_resolved_path($filename);
  if ($path and $path->is_open_ok)  
    my $fd = $path->open;
    ...;
    close($fd);
   
  return;
 

Now, you may wonder how that promotes safer code. At first glance, the checking code is not a lot simpler than the previous correct example. However, the new code has the advantage of being safer even if you forget the checks. The reasons are:

1. The return value is entirely based on the file index of the package (think: tar vtf data.tar.gz). At no point does it use the file system to resolve the path. Whether your malicious package trigger an undef warning based on the return value of index_resolved_index leaks nothing about the host machine.
   1. However, it does take safe symlinks into account and resolves them for you. If you ask for foo/bar and foo is a symlink to baz and baz/bar exists in the package, you will get baz/bar . If baz/bar happens to be a symlink, then it is resolved as well.
   2. Bonus: You are much more likely to trigger the undef warning during regular testing, since it also happens if the file is simply missing.
2. If you attempt to call $path->open without calling $path->is_open_ok first, Lintian can now validate the call for you and stop it on unsafe actions.

It also has the advantage of centralising the code for asserting safe access, so bugs in it only needs to be fixed in one place. Of course, it is still possible to write unsafe code. But at least, the new API is safer by default and (hopefully) more convenient to use. [0] Lintian.debian.org being the primary exception here. [1] This is in contrast to e.g. piuparts, which very much trusts its input packages by handing the package root access (albeit chroot ed, but still). [2] And also a bug. Not all binary packages have a copyright instead some while have a symlink to another package. [3] The code is hand-typed into the blog without prior testing (not even compile testing it). The code may be subject to typos, brown-paper-bag bugs etc. which are all disclaimed (of course). [4] Fun fact, our documented example for doing it correctly prior to implementing is_ancestor_of was in fact not correct. It used the root path in a regex (without quoting the root path) fortunately, it just broke lintian when your TMPDIR / LINTIAN_LAB contained certain regex meta-characters (which is pretty rare).

As mentioned by Rapha l Hertzog, I have been spending some time on improving Britney2. Just the other day I submitted a second branch for review that I expect to merge early next week. I also got another set patches coming up soon. Currently, none of them are really user visible, so unless you are hosting your own version of Britney, these patches are probably not all that interesting to you. The highlights:

1. Reduce the need for backtracking by finding semantically equivalent packages.
2. Avoid needing to set up a backtrack point in some cases.
   • This has the side-effect of eliminating some O(e^n) runtime cases.
3. Optimise installability testing of packages affected by a hinted migration.
   • This has the side-effect of avoiding some O(e^n) runtime cases when the post-hint state does not trigger said behaviour.
   • There is a follow-up patch for this one coming in the third series to fix a possible bug for a corner-case (causing a valid hint to be incorrectly rejected when it removed an uninstallable package).
4. Reduce the number of affected packages to test when migrating items by using knowledge about semantically equivalent packages.
   • In some cases, Britney can now do free migrations when all binaries being updated replace semantically equivalent packages.
5. (Merge pending) Avoid many redundant calls to sort_actions() , which exhibits at least O(n^2) runtime in some cases.
   • For the dataset Rapha l submitted, this patch shaves off over 30 minutes runtime. In the particular case, each call to sort_actions takes 3+ minutes and it was called at least 10 times, where it was not needed.
   • That said, sort_actions have a vastly lower runtime in the runs for Debian (and presumably also Ubuntu, since no one complained from their side so far).

The results so far: After the first patch series was merged, the Kali dataset (from Rapha l) could be processed in only ~2 hours. With the second patch series merged, the dataset will drop by another 30-50 minutes (most of which are thanks to the change mentioned in highlight #5). The third patch series currently do not have any mention-worthy performance related changes. It will probably be limited to bug fixes and some refactoring. Reflections: The 3 first highlights only affects the new installability tester meaning that the Britney2 instances at Ubuntu and Tanglu should be mostly unaffected by the O(n^2) runtime. Although those cases will probably just fail with several AIEEE s. :) The 5th highlight should equally interesting to all Britney2 instances though. For me, the most interesting part is that we have never observed the O(n^2) behaviour in a daily sid -> testing run. The dataset from Rapha l was basically a stable -> testing/sid run, which is a case I do not think we have ever done before. Despite our current updates, there is still room for improvements on that particular use case. In particular, I was a bit disheartened at how poorly our auto hinter(s) performed on this dataset. Combined they only assisted with the migration of something like 28 items . For comparison, the main run migrated ~7100 items and 9220 items were unable to migrate. Furthermore, the Original auto hinter spend the better part of 15 minutes computing hints at least it results in 10 items migrating. Links to the patches:

• First series (already merged)
• Second series (pending, to be merged Monday, the 11th of August)
• Third series (under development. Warning: the branch will be rebased if/as needed)

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (548.59 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Distro Tracker Now that tracker.debian.org is live, people reported bugs (on the new tracker.debian.org pseudo-package that I requested) faster than I could fix them. Still I spent many, many hours on this project, reviewing submitted patches (thanks to Christophe Siraut, Joseph Herlant, Dimitri John Ledkov, Vincent Bernat, James McCoy, Andrew Starr-Bochicchio who all submitted some patches!), fixing bugs, making sure the code works with Django 1.7, and started the same with Python 3. I added a tox.ini so that I can easily run the test suite in all 4 supported environments (created by tox as virtualenv with the combinations of Django 1.6/1.7 and Python 2.7/3.4). Over the month, the git repository has seen 73 commits, we fixed 16 bugs and other issues that were only reported over IRC in #debian-qa. With the help of Enrico Zini and Martin Zobel, we enabled the possibility to login via sso.debian.org (Debian s official SSO) so that Debian developers don t even have to explicitly create their account. As usual more help is needed and I ll gladly answer your questions and review your patches. Misc packaging work Publican. I pushed a new upstream release of publican and dropped a useless build-dependency that was plagued by a difficult to fix RC bug (#749357 for the curious, I tried to investigate but it needs major work for make 4.x compatibility). GNOME 3.12. With gnome-shell 3.12 hitting unstable, I had to update gnome-shell-timer (and filed an upstream ticket at the same time), a GNOME Shell extension to start some run-down counters. Django 1.7. I packaged python-django 1.7 release candidate 1 in experimental (found a small bug, submitted a ticket with a patch that got quickly merged) and filed 85 bugs against all the reverse dependencies to ask their maintainers to test their package with Django 1.7 (that we want to upload before the freeze obviously). We identified a pain point in upgrade for packages using South and tried to discuss it with upstream, but after closer investigation, none of the packages are really affected. But the problem can hit administrators of non-packaged Django applications. Misc stuff. I filed a few bugs (#754282 against git-import-orig uscan, #756319 against wnpp to see if someone would be willing to package loomio), reviewed an updated package for django-ratelimit in #755611, made a non-maintainer upload of mairix (without prior notice) to update the package to a new upstream release and bring it to modern packaging norms (Mako failed to make an upload in 4 years so I just went ahead and did what I would have done if it were mine). Kali work resulting in Debian contributions Kali wants to switch from being based on stable to being based on testing so I did try to setup britney to manage a new kali-rolling repository and encountered some problems that I reported to debian-release. Niels Thykier has been very helpful and even managed to improve britney thanks to the very specific problem that the kali setup triggered. Since we use reprepro, I did write some Python wrapper to transform the HeidiResult file in a set of reprepro commands but at the same time I filed #756399 to request proper support of heidi files in reprepro. While analyzing britney s excuses file, I also noticed that the Kali mirrors contains many source packages that are useless because they only concern architectures that we don t host (and I filed #756523 against reprepro). While trying to build a live image of kali-rolling, I noticed that libdb5.1 and db5.1-util were still marked as priority standard when in fact Debian already switched to db5.3 and thus should only be optional (I filed #756623 against ftp.debian.org). When doing some upgrade tests from kali (wheezy based) to kali-rolling (jessie based) I noticed some problems that were also affecting Debian Jessie. I filed #756629 against libfile-fcntllock-perl (with a patch), and also #756618 against texlive-base (missing Replaces header). I also pinged Colin Watson on #734946 because I got a spurious base-passwd prompt during upgrade (that was triggered because schroot copied my unstable s /etc/passwd file in the kali chroot and the package noticed a difference on the shell of all system users). Thanks See you next month for a new summary of my activities.

One comment Liked this article? Click here. My blog is Flattr-enabled.

I just did some Debian package review in a somewhat unusual way, and I wanted to share that. I'm hoping other Debian developers (and other free software contributors) that need to review others' contributions can learn something from this, and that I can use this blog post as a way to find out if other people are doing something similar. It was pretty exciting! At the end of it, I joined #debian-mentors to talk about how my cool process. Someone summarized it very accurately:

<sney> it almost sounds like you're working to replace yourself with automation

Context about alpine in Debian (Skip to "Package review, with automation" if you're familiar with Debian.) I'm the maintainer of alpine in Debian. There are quite a few problems with the alpine package in Debian right now, the biggest of which are:

• We're one version behind -- 2.11 is the latest available, but 2.10 is the newest that we have in Debian.

• The packaging uses a decreasingly-popular packaging helper, cdbs, about which I happen to know less than the dh-style helper (aka dh7).

• There are lots of bugs filed, and I don't respond in a timely fashion.

This doesn't change my deep love for alpine -- I've had that for about half my life now, and so far, I don't see it going away. A month or so ago, I got a friendly private message from Unit193, saying he had converted the package to the dh style, and also packaged the newer version. They wanted to know if they should clean this up into something high-enough quality to land in Debian. (In Debian, we have a common situation where enthusiastic users update or create a new package, and aren't yet Debian Developers, so they don't have permission to upload that directly to the "Debian archive", which is the Debian equivalent of git master. Package "sponsorship" is how we handle that -- a Debian Developer reviews the package, makes sure it is of high quality, and uploads it to the Debian archive along with the Debian Developer's OpenPGP signature, so the archive processing tools know to trust it.) On Friday evening, I had a spare moment, so I sent a private message to Unit193 apologizing for not getting back to them in a reasonable amount of time. Having another person help maintain is a pretty exciting prospect, and I wanted to treat that enthusiasm with the respect it deserves, or at least apologize when I haven't. I was surprised to see a reply within a few minutes. At that point, I thought: I wasn't planning on doing any package review this weekend, but if they're online and I'm online... might as well!

Package review, with automation Unit193 and I popped into ##alpine on irc.freenode.net, and I started reading through their packaging changes, asking questions. As I asked questions, I wondered -- how will I know if they are going to fix the issues I'm raising? Luckily, Unit193 wanted to use git to track the packaging, and we settled on using git-buildpackage, a tool that was fairly new to both of us. I thought, I might as well have some executable documentation so I don't forget how to use it. ("Executable documentation" is Asheesh-speak for a shell script.) One thing I knew was that I'd have to test the package in a pbuilder, or other pristine build environment. But all I had on me at the moment was my work laptop, which didn't have one set up. Then I had a bright idea: I could use Travis-CI, a public continuous integration service, to check Unit193's packaging. If I had any concerns, I could add them to the shell script and then point at the build log and say, "This needs to be fixed." Then there would be great clarity about the problems. Some wonderful things about Travis-CI:

• They give you root access on an Ubuntu Precise (10.04) virtual machine.
• Their build hosts are well-connected to the Internet, which means fast downloads in e.g. pbuilder.
• They will let you run a build for up to 50 minutes, for free.
• Build just means "command" or "set of commands," so you can just write a shell script and they will run it.
• Travis-CI will watch a github.com repository, if you like. This means you can 'git commit --allow-empty' then 'git push' and ask it to re-run your script.

Since Unit193's packaging was in git (but not on github), I created a git repo containing the same contents, where I would experiment with fixes for packaging problems I found. It'd be up to Unit193 to fix the problems in the Alioth packaging. This way, I would be providing advice, and Unit193 would have an opportunity to ask questions, so it would be more like mentorship and less like me fixing things. We did a few rounds of feedback this way, and got the packaging to higher and higher quality. Every time Unit193 made a fix and pushed it, I would re-run the auto-build, and see if the problems I spotted had gone away. While the auto-build runs, I can focus on conversing with my mentee about problems or just generally chatting. Chatting is valuable community-building! It's extremely nice that I can do that while waiting on the build, knowing that I don't have to read it carefully -- I can just wait a few minutes, then see if it's done, and see if it's red or green. Having the mentee around while I'm reviewing it means that I can use the time I'm waiting on builds as fun free software social time. (Contrast this with asynchronous review, where, all alone, I would wait for a build to finish, then write up an email at the end of it all.) This kind of mentorship + chatting was spread out over Friday night, Saturday night, and Sunday morning. By the end of it, we had a superb package that I'm excited to sign and push into Debian when I'm next near my OpenPGP key.

Implementation details You can see the final shell script here:

• do.sh

And you can see the various builds here:

• builds

The shell script:

• Alternates between the Alioth packaging vs. my fork of it. (This way, I can test packaging changes/suggestions.)
• Disables ccache in pbuilder, due to a permissions problem with ccache/pbuilder/travis-ci, and I didn't need ccache anyway.
• Handles 'git dch' slightly wrong. I need to figure that out.
• Optionally passes --git-ignore-new to git-buildpackage, which was required initially, but should not be required by the time the package is ready. (This is an example of a thing I might forget to remark upon to my mentee.)
• Plays games with git branches so that git-buildpackage on Travis-CI can find the pristine-tar branch.
• Tries to use cdn.debian.net as its mirror, but on Saturday ran into problems with whicever mirror that is, so it falls back to mirror.mit.edu in case that fails.
• Contains a GPG homedir, and imports the Debian archive key, so that it can get past Ubuntu-Debian pbuilder trust issues.

I also had a local shell script that would run, effectively:

• git commit --allow-empty -m 'Trigger build'
• git push

This was needed since I was basically using Travis-CI as remote shell service -- moreover, the scripts Travis-CI runs are in a different repo (travis-debcheck) than the software I'm actually testing (collab-maint/alpine.git). Unit193 and I had a technical disagreement at one point, and I realized that rather than discuss it, I could just ask Travis-CI to test which one of us was right. At one point in the revisions, the binary package build failed to build on Precise Pangolin (the Ubuntu release that the Travis-CI worker is running), and Unit193 said that it was probably due to a problem with building on Ubuntu. So I added a configuration option to build just the source package in Ubuntu, keeping the binary package test-build within the Debian sid pbuilder, although I believed that there was actually a problem with the packaging. This way, I could modify the script so that I could demonstrate the problem could be reproduced in a sid pbuilder. Of course, by the time I got that far, Unit193 had figured out that it was indeed a packaging bug. I also created an option to SKIP_PBUILDER; initially, I wanted to get quick automated feedback on the quality of the source package without waiting for pbuilder to create the chroot and for the test build to happen. You might notice that the script is not very secure -- Niels Thykier already did! That's fine by me; it's only Travis-CI's machines that could be worsened by that insecurity, and really, they already gave me a root shell with no password. (This might sound dismissive of Travis-CI -- I don't mean it to be! I just mean that their security model already presumably involves throwing away the environment in which my code is executing, and I enjoy taking advantage of that.) Since the Travis virtual machine is Ubuntu, and we want to run the latest version of lintian (a Debian packaging "lint" checker), we run lintian within the Debian sid pbuilder. To do that, I use the glorious "B90lintian" sample pbuilder hook script, which comes bundled with pbuilder in /usr/share/doc/pbuilder/. The full build, which includes creating a sid pbuilder from scratch, takes merely 7-10 minutes. I personally find this kind of shockingly speedy -- in 2005, when I first got involved, doing a pbuilder build seemed like it would take forever. Now, a random free shell service on the Internet will create a pbuilder, and do a test build within it, in about 5 minutes.

Package review, without automation I've done package review for other mentees in the past. I tend to do it in a very bursty fashion -- one weekend day or one weeknight I decide sure, it's a good day to read Debian packages and provide feedback. Usually we do it asynchronously on the following protocol:

1. I dig up an email from someone who needed review.
2. I read through the packaging files, doing a variety of checks as they occur to me.
3. If I find problems, I write an email about them to the mentee. If not, success! I sign and upload the package.

There are some problems with the above:

• The burstiness means that if someone fixes the issues, I might not have time to re-review for another month or longer.
• The absence of an exhaustive list of things to look for means that I could fail to provide that feedback in the first round of review, leading to a longer wait time.
• The person receiving the email might not understand my comments, which could interact really badly with the burstiness.

I did this for Simon Fondrie-Teitler's python-pypump package recently. We followed the above protocol. I wrote a long email to Simon, where I remarked on various good and bad points of the packaging. It was part commentary, part terminal transcript -- I use the terminal transcripts to explain what I mean. This is part of the email I sent:

I got an error in the man page generation phase -- because at 
build-time, I don't have requests-oauthlib:
make[2]: Leaving directory  /tmp/python-pypump-0.5-1+dfsg/docs'
help2man --no-info \
	-n 'sets up an environment and oauth tokens and allows for interactive testing' \
        --version-string=0.5.1 /tmp/python-pypump-0.5-1+dfsg/pypump-shell > /tmp/python-pypump-0.5-1+dfsg/debian/pypump-shell.1
help2man: can't get  --help' info from /tmp/python-pypump-0.5-1+dfsg/pypump-shell
Try  --no-discard-stderr' if option outputs to stderr
make[1]: *** [override_dh_auto_build] Error 1
This seems to be because:
   python-pypump-0.5-1+dfsg  ./pypump-shell 
Traceback (most recent call last):
  File "./pypump-shell", line 26, in <module>
    from pypump import PyPump, Client
  File "/tmp/python-pypump-0.5-1+dfsg/pypump/__init__.py", line 19, in <module>
    from pypump.pypump import PyPump, WebPump
  File "/tmp/python-pypump-0.5-1+dfsg/pypump/pypump.py", line 28, in <module>
    from six.moves.urllib import parse
ImportError: No module named urllib
$ ./pypump-shell 
Traceback (most recent call last):
  File "./pypump-shell", line 26, in <module>
    from pypump import PyPump, Client
  File "/tmp/python-pypump-0.5-1+dfsg/pypump/__init__.py", line 19, in <module>
    from pypump.pypump import PyPump, WebPump
  File "/tmp/python-pypump-0.5-1+dfsg/pypump/pypump.py", line 29, in <module>
    from requests_oauthlib import OAuth1
ImportError: No module named requests_oauthlib

The deeper problem was a missing build-dependency, and I explained that in my email. But the meta problem is that Simon didn't try building this in a pbuilder, or otherwise clean build environment. Simon fixed these problems, and submitted a fresh package to Paul Tagliamonte and myself. It happened to have some typos in the names of the new build dependencies. Paul reviewed the fixed package, noticed the typos, fixed them, and uploaded it. Simon had forgotten to do a test build the second time, too, which is an understandable human failure. There was a two-day delay between Simon's fixed resubmission, and Paul signing+uploading the fixed result. In a pedagogical sense, there's something disappointing about that exchange for me: Paul fixed an error Simon introduced, so we're not teaching Simon to take total responsibility for his packages in Debian, nor to understand the Debian system as well as he could. (Luckily, I think Simon already understands the importance of taking responsibility! In this case, it's just a hypothetical in this case.)

For the future The next time I review a package, I'm going to try to do something similar to my Travis-CI hack. It would be nice to have the do.sh script be a little more abstract; I imagine that as I try to use it for a different package, I'll discover the right abstractions. I'd love it if Travis-CI did not require the git repositories to be on GitHub. I'd also like if the .travis.yml file could be in a different path. If so, we could create debian/travis-configuration (or something) and keep the packaging files nice and separate from the upstream source. I'd also love to hear about other people's feedback. Are you doing something similar? Do you want to be? Would you have done some of this differently? Leave a comment here, or ping me (paulproteus) on #debian-mentors on irc.debian.org (aka irc.oftc.net). I'll leave you with some conversation from #debian-mentors:

<paulproteus> The automation here, I think, is really interesting.
<paulproteus> What I really want is for mentees to show up to me and say "Here is my package + build log with pbuilder, can you sponsor it please?"
<Unit193> Oooooh!
-*- Unit193 gets ideas.
<paulproteus> Although the irony is that I actually like the community-building and relationship-building nature of having these things be conversations.
<bremner> how will this brave new world cope with licensing issues?
<paulproteus> bremner: It's not a replacement for actual review, just a tool-assist.
<paulproteus> bremner: You might be relieved to know that much of Unit193's and my back and forth related to get-orig-source and licensing. (-:
<bremner> I didn't doubt you ;).
<paulproteus> If necessary I can just be a highly productive reviewer, but I would prefer to figure out some way that I can get other non-paulproteus people to get a similar assist.
<paulproteus> I think the current blocker is "omg travis why are you bound to githubbbbbbbb" which is a reasonable concern.

Some jenkins.debian.net activities Yesterday and today I've spent some time time on jenkins.debian.net, so that it's finally only available via https, or so I thought. I quickly learned that it's better still serving via http, but only when queried from localhost Second, I've moved the git repo to QA group namespace on alioth and set up IRC notifications using the wonderful KGB client. There is no mailinglist which receives commit diffs yet, but thats also because it's mostly still me working on this. Though since yesterday there are now seven committers in the git history, which is two more than before: Niels Thykier submitted job configuratios for running the lintian testsuite on stable, testing and sid on every commit to the lintian git repo. The test suite passes nicely on testing and unstable but fails for stable as it needs backports and we need to make the job to install those. Gabriele Giacone submitted several patches so that soon hurd wgnu/hurd can be tested too. Currently this fails for two reasons, both of which are very temporarily: the hurd .iso image which was here this morning is gone now, and secondly I haven't installed (the locally build packages for) qemu and qemu-kvm packages with the fixes for #742386 as that .iso ain't there so this is pointless atm But I'm very happy to have a hopefully soon working job which will regularily test installation of Debian hurd and I do hope that some Debian kfreebsd enthusiasts will pick up and fix the existing jobs for kfreebsd tests. (Hint hint hint! I'm also totally happy to help with that!) This reminds me to mention one last bit: Thanks to Paul Wise, jenkins.d.n bugs are now tracked in the BTS among other Debian QA group bugs!

The next version of Lintian, 2.5.22, is long overdue mostly because 2.5.21 FTBFS in sid. Besides fixing test failures, 2.5.22 also fixes:

• False-negative for most/all JAR related tags caused by file(1) changing its output for these files.
• A performance regression in checks/cruft.pm. For large source packages, this could severely affect performance of Lintian. This is #738342.
• A regression where lintian would not parse options appearing after file arguments.
• Some false-positives in the GFDL with invariants check.

Besides bug fixes, we also added a couple of new features/nice changes:

• The color option now defaults to auto instead of never .
• The homepage now links to the generated API-docs and thus, also the developer tutorial.
  • Patches extending/improving either/both are more than welcome.
• The test-suite runner now caches test artifacts meaning that subsequent test runs become vastly faster.
• The test-suite runner can now run tests under Devel::Cover , so we can now collect statistics about how much of the code is covered by the test suite.
• The html_reports tool (used e.g. on lintian.d.o) has been optimised. In particular, it only calls gnuplot twice, instead of once per data file (of which there was basically one per tag).
• The html-pages generated by the html_reports tool now uses content -based names for resources (e.g. images and the .css file). This allows them to be cached vastly longer, since any content changes will rename the resource and force browser to refetch them.
  • All resources handled this way are now stored in resources/ . As a side-effect, this means that images files are moved from images/ to resources/ .
  • A notable exception to this is the auto-generated (.svg) graphs.

We just need to do some final tweaks and get the tag descriptions reviewed before we are ready to release the next version.

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was three months ago with the announcement of bootstrap.debian.net. Since then a number of things happened, so I thought an update was due.

New graphs for port metrics By default, a dependency graph is created by arbitrarily choosing an installation set for binary package installation or source package compilation. Installation set vertices and source vertices are connected according to this arbitrary selection. Niels Thykier approached me at Debconf13 about the possibility of using this graph to create a metric which would be able to tell for each source package, how many other source packages would become uncompilable or how many binary packages would become uninstallable, if that source package was removed from the archive. This could help deciding about the importance of packages. More about this can be found at the thread on debian-devel. For botch, this meant that two new graph graphs can now be generated. Instead of picking an arbitrary installation set for compiling a source package or installing a binary package, botch can now create a minimum graph which is created by letting dose3 calculate strong dependencies and a maximum graph by using the dependency closure.

Build profile syntax in dpkg With dpkg 1.17.2 we now have experimental build profile support in unstable. The syntax which ended up being added was:

Build-Depends: large (>= 1.0), small <!profile.stage1>

But until packages with that syntax can hit the archive, a few more tools need to understand the syntax. The patch we have for sbuild is very simple because sbuild relies on libdpkg for dependency parsing. We have a patch for apt too, but we have to rebase it for the current apt version and have to adapt it so that it works exactly like the functionality dpkg implements. But before we can do that we have to decide how to handle mixed positive and negative qualifiers or whether to remove this feature altogether because it causes too much confusion. The relevant thread on debian-dpkg starts here.

Update to latest dose3 git Botch heavily depends on libdose3 and unfortunately requires features which are only available in the current git HEAD. The latest version packaged in Debian is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also relies on unreleased features from libcudf. On top of that, the GraphML output of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the git. For now everything is set up as git submodules but this is the major blocker preventing any packaging of botch as a Debian package. Hopefully new releases will be done soon for all involved components.

Writing and reading GraphML Botch is a collection of several utilities which are connected together in a shell script. The advantage of this is, that one does not need to understand or hack the OCaml code to use botch for different purposes. In theory it also allows to insert 3rd party tools into a pipe to further modify the data. Until recently this ability was seriously hampered by the fact that many botch tools communicated with each other through marshaled OCaml binary files which prevent everything which is not written in OCaml from modifying them. The data that was passed around like this were the dependency graphs and I initially implemented it like that because I didnt want to write a GraphML parser. I now ended up writing an xmlm based GraphML parser so as of now, botch only reads and writes ASCII text files in XML (for the graphs) and in rfc822 packages format (for Packages and Sources files) which can both easily be modified by 3rd party tools. The ./tools directory contains many Python scripts using the networkx module to modify GraphML and the apt_pkg module to modify rfc822 files.

Splitting of tools To further increase the ability to modify program execution without having to know OCaml, I split up some big tools into multiple smaller ones. Some of the smaller tools are now even written in Python which is probably much more hackable for the general crowd. I converted those tools to Python which did not need any dose3 functionality and which were simple enough so that writing them didnt take much time. I could convert more tools but that might introduce bugs and takes time which I currently dont have much of (who does?).

Gzip instead of bz2 Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed Packages and Sources files anymore but uses xz instead. This is awesome for must purposes but unfortunately I had to discover that there exist no OCaml bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself or anybody else finds some time to write a libxz OCaml binding.

Self hosting Fedora Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I reproduced his steps for the Debian dependency graph and it turns out that they are a little bit bigger. I'm exchanging emails with Harald Hoyer because it might not be too hard to use botch for rpm based distributions as well because dose3 supports rpm. The article also made me aware of the tred tool which is part of graphviz and allows to calculate the transitive reduction of a graph. This can help making horrible situations much better.

Dose3 bugs I planned to generate such simplified graphs for the neighborhood of each source package on bootstrap.debian.net but then binutils stopped building binutils-gold and instead provided binutils-gold while libc6-dev breaks binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug and thus bootstrap.debian.net will not generate any new results until this is fixed in dose3. Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar while bar is fully virtual and are Multi-Arch:same. Binaries of different architecture with this property can currently not be co-installed with dose3. Unfortunately linux-libc-dev has this property and thus botch cannot be used to analyze cross builds until that bug is fixed in dose3. I hope I get some free time soon to be able to look at these dose3 issues myself.

More documentation Since I started to like the current set of tools and how they work together I ended up writing over 2600 words of documentation in the past few days. You can start setting up and running botch by reading the first steps and get more detailed information by reading about the 28 tools that botch makes use of as of now. All existing articles, thesis and talks are linked from the wiki home.

Today, we held an one-hour IRC-meeting to debate the status of the current architectures in sid. During that one hour, we brought up all 13 architectures. We made a decision on 9 of the architectures. The remaining 4 will be revisited within 2 months. As for the actual results, we will announce these on debian-devel-announce with our next bits mail. I suspect the announcement to happen within a couple of days. I am quite pleased that we managed to keep the meeting to one hour. Especially considering that it was planned less than a week ago. One of the things, that I believe helped us, was that we had summarised the status for each architecture prior to the meeting. In my view, the summary allowed us to finish at least half of the architectures within their allocated 5 minutes. In fact, we had 20 minutes for the last 2 architectures partly because amd64 + i386 combined took about a minute. A second thing that helped us stay on time was cutting debates short. Obviously, with less than 5 minutes per architecture, there was little time for debate and, I admit, that was by design. This meant that we had to defer 4 decisions for later. We are currently waiting for information, which we hope to have soon. More on that in the announcement later.
I would like to thank the attendees for contributing to a productive and short meeting. It was a pleasure and I hope we can repeat the success next time.

I am very pleased to say that RC bug counter for Jessie finally dropped below 500 RC bugs. For comparison, we broke the 700 RC bug curve in the start of November, so that is about 200 RC bugs in about 7-8 weeks (or about 25-28 RC bugs per week). Today, we have about 162 RC bugs on the auto-removal list. Though, I suspect many of the affected packages have reverse dependencies, so their removal from testing may be up 30 days away. Nevertheless, by this time in January, we could be looking at no more than 350 RC bugs left for Jessie.

Yesterday, I have managed to finish up an old pet peeve of mine. I wrote a series of patches to have harness reprocess all packages, which were last checked by an older version of Lintian than the current. It is not completed to the level I would like, but it is good enough for now. The implementation works by processing up to 1024 groups after each daily run. It also have a timer to skip this, if the run has taken more than 4 hours. Packages to be reprocessed are sorted by the version they were last processed should a new version of Lintian be upload before all packages have been reprocessed, the oldest results are refreshed first. A rough estimate suggests than in about 3-4 weeks, the entire archive will have been reprocessed with Lintian 2.5.20. It is by no means faster than a full run (which is about a week), but it is automatic and removes the human factor in keeping Lintian results up to date.

I have decided to step down as main maintainer of Lintian and pass the baton to Bastien Roucari s. This is actually fairly old news, since I announced this almost a month ago. However, I was not very vocal about it until now (so do not be surprised if you had not heard of this before). In the past month, I never once regretted having stepped down. If anything, I should probably have done it a bit earlier. Beyond the lack of motivation, I also realised that I had become an all talk and no do -maintainer. The kind that I have been advocating against publicly. This may sound weird to a lot of people, who knows me as the Lintian guy or Mr. Lintian (or whatever Lintian-related nickname people gave me). But the fact is that Bastien has done more for Lintian in the past month than I have in the past two. Despite stepping down as main developer, I have not left Lintian completely. I am still around for helping/explaining, uploading and keeping lintian.debian.org running.

It looks like we have dropped (or will drop) below 700 RC bugs affecting Jessie today[1]. On a related note, I got my eye on libfso-glib + fso-gsmd, samba + -samba4, mono and haskell. By my count, each of the four groups will (once they migrate) bring at least 4 RC bug fixes with them to testing. Besides that, we still have ~97 RC bugs in the autoremoval queue[2], so we could be looking at 600 RC bugs in a week or two. Please keep up the steady flow of RC bug fixes; but also, please make sure your packages will actually migrate. If you need help interpreting the excuses page, do not hesistate to ask I will not bite. If you want to help, please consider having a look at the cleaner or the Release Team -view on the UDD bug tracker. Sadly not everything there contains actionable problems[3]. [1] http://bugs.debian.org/release-critical/ says 700 while the IRC bot in #debian-devel-changes says we are at 692. [2] http://udd.debian.org/bugs.cgi?release=jessie_and_sid&merged=ign&autoremovals=only&fnewerval=7&flastmodval=7&rc=1&sortby=id&sorto=asc dd-list-like: http://udd.debian.org/cgi-bin/autoremovals.cgi [3] E.g. the kfreebsd-8 bugs appear because kfreebsd-8 is listed in Built-Using for d-i, but the package has already been removed from testing. So there is nothing to do with that (except possibly waiting for a new version of d-i).

Early this month, the first batch of packages were automatically removed from testing due to RC bugs. Personally, I find that the results stand out pretty well on the RC bug graph[1]. There has been a very rapid decline in the number of RC bugs affecting testing a week or two ago, we had over 1000 RC bugs affecting testing and now it has dropped to something like 760. There has also been quite a few RC bug fixes in this time period. I spent a bit of time finding and aging some of the RC bug fixes in sid, so they would migrate to testing faster[2]. On the first batch, we saw something like 40 RC bugs disappear and we still have more RC bugs fixes in the sid->testing pipeline. While looking for packages I could age, I also noticed that we have quite a few RC bug fixes blocked by out of date binaries (which is usually either build issues or uncoordinated transitions). I have been filing a couple of RC bugs for these, but it does not scale do this kind of work manually. If you have uploaded a package recently, please take the 5 minutes to check that it actually built and it has no out of date binaries issues in its excuses. [1] http://bugs.debian.org/release-critical/ [2] The aging was equivalent to an urgency=medium, so reduced the requirement from 10 days to 5 days.

Today, I released Lintian 2.5.19. It is mostly a bug-fix release, but it also features a new tag. The new tag, homepage-in-binary-package, is emitted for packages where the source package has no Homepage-field, but one of its binaries do. The problem is that some tools (e.g. the PTS) only processes the data from the .dsc files (or Sources indices from the mirrors) and would in these cases miss the Homepage of the page. We now also carry a work around for a bug in tool, t1disasm, where it ends up in an infinite recursion (i.e. until the C call stack overflows). That issue is filed as #724571. The work around is to prevent Lintian from exiting with a failure (which causes packages to be auto-rejected when uploaded to Debian archive). These problems are currently reduced to a warning on stderr. In other (Lintian related news), the full run was completed yesterday after only 4 days (rather than the 8, I expected). While that is a pleasant surprise, I cannot find any changes in Lintian since the last run that would explain this difference. My current conjecture is that our last pseudo full run (when we added amd64) included more processing since the amd64 packages would not have any pre-collected information. With the follow up run to catch packages uploaded after the full run, Lintian spent a little more than 3 and half days (~86.5 hours) processing about 20700 package groups. This gives an average of about 240 groups an hour (or ~4 a minute). Though, remember these numbers include some data having been pre-collected from previous run on the majority of packages.

Today I had to pleasure of releasing Lintian 2.5.18.1. It is a minor bug fix release on top of Lintian 2.5.18 (which I also released today ). When I was about to deploy the new version of Lintian on lintian.debian.org, I noticed a couple of errors in the log file that smelled of bugs. Indeed, when I investigated them I found two bugs, which were both introduced in Lintian 2.5.17.

Due to another bug in 2.5.17 and the some of the false-positives fixes, we are now doing a full run on lintian.debian.org. After having run for 7 hours, Lintian passed cipux . My rough estimate suggests that it will take 8 days at the current rate. For comparision, when we added amd64, Lintian spent 5 days processing all the new binaries (plus their source and the old binaries built from those source packages).

Of course estimate is rather rough. In particular, I am not sure that the heavy duty groups (e.g. linux and libreoffice) are evenly spread out among the letters of the alphabeth. Anyhow, there will be no updates on lintian.debian.org for the next week or so.

I have just uploaded Lintian 2.5.18 to unstable. While fixing 22 bugs, it only features 5 new tags.

• debian-changelog-has-wrong-weekday
• debian-rules-missing-good-practice-target-dfsg
• empty-udeb-package
• file-name-in-PATH-is-not-ASCII
• misplaced-extra-member-in-deb

The release also include fixes to some false-positives, such as python:any dependencies triggering python-script-but-no-python-dep, a rewritten README file. We also included a patch to make Lintian accept the Original-Maintainer field by default for non-Debian vendors (even if they do not have a profile and Lintian ends up loading the debian/main profile). We also added support for running Lintian directly from a git checkout or source tree without setting LINTIAN_ROOT (or passing root). Since that was the primary use-case for root that option has now been deprecated. I also had lintian and lintian-info require the include-dir and [no-]user-dir options as the first if given at all. I would like to thank Bastien Roucari s, Gaudenz Steinlin, Gunnar Wolf, J r my Bobbio and Lucas Nussbaum for contributing to Lintian and the many who submitted reports or suggestions for improvements. I would also like to thank Brian hugmeir Fraser, who assisted me in identifying and working around a bug in Perl s glob function when run under threads (filed upstream as RT#119897).