Search Results: "nthykier"

19 February 2015

Niels Thykier: Partial rewrite of lintian s reporting setup

I had the mixed pleasure of doing a partial rewrite of lintian s reporting framework. It started as a problem with generating the graphs, which turned out to be not enough memory . On the plus side, I am actually quite pleased with the end result. I managed to scope-creep myself quite a bit and I ended up getting rid of a lot of old issues. The major changes in summary: There are also some nice minor features : As you can probably tell, I am quite pleased with the end result. The reporting framework lacks behind in development, since it just sits there and takes care of itself . Also with the complete lack of testing, it also suffers from the if it is not broken, then do not fix it paradigm (because we will not notice if we broke until it is too late). Of course, I managed to break the setup a couple of times in the process. However, a bonus feature of the reporting setup is that if you break it, it simply leaves an outdated report on the website. Anyway, enjoy. :)
Filed under: Debian, Lintian

30 December 2014

Niels Thykier: Status on Jessie (December 2014)

Here is a slightly overdue status on Jessie. Stricter freeze policy per January 5th The next timed change of the freeze policy will apply per January 5th. After that date, we will only accept RC bugs fixes. Which means it is final chance for translation updates. More on RC bugs In absolute numbers, the RC bugs have declined quite well. We are below 150 now. We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month. However, we still need the final push here. Debian installer release pending Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow. Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support. Pending Debian 7.8 release While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend. Want to help? Thank you, [RN source]: https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ svn co https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ Git Repo: http://anonscm.debian.org/cgit/users/jcristau/release-notes.git/
Filed under: Debian, Release-Team

Niels Thykier: Status on Jessie (December 2014)

Here is a slightly overdue status on Jessie. Stricter freeze policy per January 5th The next timed change of the freeze policy will apply per January 5th. After that date, we will only accept RC bugs fixes. Which means it is final chance for translation updates. More on RC bugs In absolute numbers, the RC bugs have declined quite well. We are below 150 now. We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month. However, we still need the final push here. Debian installer release pending Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow. Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support. Pending Debian 7.8 release While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend. Want to help? Thank you, [RN source]: https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ svn co https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ Git Repo: http://anonscm.debian.org/cgit/users/jcristau/release-notes.git/
Filed under: Debian, Release-Team

8 December 2014

Niels Thykier: Jessie has half the number of RC bugs compared to Wheezy

In the last 24 hours, the number of RC bugs currently affecting Jessie was reduced to just under half of the same number for Wheezy. There are still a lot of bugs to be fixed, but please keep up the good work. :)

30 November 2014

Thorsten Alteholz: My Debian Activities in November 2014

FTP assistant In contrast to the last month, this month has been rather quiet and I really liked that :-) . The stress has moved to the next team. So all in all I marked 101 packages for accept and had to reject 27 packages. As I mostly reviewed really new packages, I didn t have to file any RC bug this month. Squeeze LTS This was my fifth month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. This month I got assigned a workload of 14.25h and I spent these hours to upload new versions of: I also uploaded [DLA 85-1] libxml-security-java security update, but as nobody of the LTS sponsors had any interest in this package, I did this in my spare time. A package with security in its name should not be affected by security issues. This month my failure of the month has been the binutils package. Although the security team prepared the way for finding the correct patches for all those CVEs, I somehow managed to not find them. This is embarassing I am also a bit disappointed by current LTS users. All important packages have been made available for testing before uploading them to the archive. Apart from some brave fellow DDs, no other feedback was reported on debian-lts. Complaints arrived only when the packages have been finally uploaded. Do admins have enough time nowadays and don t need to use some kind of testbed? Times are changing Other packages This month I even found some time to sponsor uploads, so please welcome a new version of fastaq in experimental and patiently wait for aegaen and kmc to pass NEW. At this point I also want to mention the Debian Med Advent Calendar, which was announced in this email and already mentioned by Andreas in his latest Debian Med bits. Everybody is invited to take care of as much as possible poor souls. Support If you would like to support my Debian work you could either be part of the Freexian initiative (see above) or consider to send some bitcoins to 1JHnNpbgzxkoNexeXsTUGS6qUp5P88vHej. Contact me at donation@alteholz.eu if you prefer another way to donate. Every kind of support is most appreciated.

27 November 2014

Niels Thykier: Volume of debian-release mailing list

Page 1 of 5 To be honest, I do not know how many mails it shows per page (though I assume it is a fixed number). So for comparison, I found the month on debian-devel@l.d.o with the highest volume in the past two years: May 2013 with Page 1 of 4 . I hope you will please forgive us, if we are a bit terse in our replies or slow to reply. We simply got a lot to deal with. :)

21 November 2014

Niels Thykier: Release Team unblock queue flushed

At the start of this week, I wrote that we had 58 open unblock requests open (of which 25 were tagged moreinfo). Thanks to an extra effort from the Release Team, we now down to 25 open unblocks of which 18 are tagged moreinfo. We have now resolved 442 unblock requests (out of a total of 467). The rate has also declined to an average of ~18 new unblock requests a day (over 26 days) and our closing rated increased to ~17. With all of this awesomeness, some of us are now more than ready to have a well-deserved weekend to recharge our batteries. Meanwhile, feel free to keep the RC bug fixes flowing into unstable.

17 November 2014

Niels Thykier: The first 12 days and 408 unblock requests into the Jessie freeze

The release team receives an extreme amount of unblock requests right now. For the past 22 days[1], we have been receiving no less than 408 unblock/ageing requests. That is an average of ~18.5/day. In the same period, the release team have closed 350 unblocks requests, averaging 15.9/day. This number does not account for number of unblocks, we add without a request, when we happen to spot when we look at the list of RC bugs[2]. Nor does it account for unblock requests currently tagged moreinfo , of which there are currently 25. All in all, it has been 3 intensive weeks for the release team. I am truly proud of my fellow team members for keeping up with this for so long! Also a thanks to the non-RT members, who help us by triaging and reviewing the unblock requests! It is much appreciated. :) Random bonus info: [1] We started receiving some in the 10 days before the freeze as people realised that their uploads would need an unblock to make it into Jessie. [2] Related topics: what is adsb? (the answer being: Our top hinter for Wheezy)

7 November 2014

Niels Thykier: Release sprint Preparing for Jessie

The release team are doing a sprint right now up the mini-DebConf in Cambridge, kindly hosted by ARM. We have a fairly large agenda of around 10 items, ranging from simple things like determine the name for the next release to reviewing the list of RC bugs affecting Jessie. Personally, I am very pleased with our progress. We managed to finish 8 of items on the first day. Furthermore, we spent several hours on the RC bug list and keeping up with the unblock requests. There is also live status of the release team, courtesy of Jonathan Wiltshire. Mind you it is manually updated. We will announce the results of our sprint Sunday morning in our Release update talk. The announcement will also be posted to debian-devel-announce like our freeze announcement. Update: fix link to the live status

27 September 2014

Niels Thykier: Lintian Upcoming API making it easier to write correct and safe code

The upcoming version of Lintian will feature a new set of API that attempts to promote safer code. It is hardly a ground-breaking discovery , just a much needed feature. The primary reason for this API is that writing safe and correct code is simply too complicated that people get it wrong (including yours truly on occasion). The second reason is that I feel it is a waste having to repeat myself when reviewing patches for Lintian. Fortunately, the kind of issues this kind of mistake creates are usually minor information leaks, often with no chance of exploiting it remotely without the owner reviewing the output first[0]. Part of the complexity of writing correct code originates from the fact that Lintian must assume Debian packages to be hostile until otherwise proven[1]. Consider a simplified case where we want to read a file (e.g. the copyright file):
package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  # BAD: This is an example of doing it wrong
  open(my $fd, '<', $info->unpacked($filename));
  ...;
  close($fd);
  return;
 
This has two trivial vulnerabilities[2].
  1. Any part of the path (usr,usr/share, ) can be asymlink to somewhere else like /
    1. Problem: Access to potentially any file on the system with the credentials of the user running Lintian. But even then, Lintian generally never write to those files and the user has to (usually manually) disclose the report before any information leak can be completed.
  2. The target path can point to a non-file.
    1. Problem: Minor inconvenience by DoS of Lintian. Examples include a named pipe, where Lintian will get stuck until a signal kills it.

Of course, we can do this right[3]:
package Lintian::cpy_check;
use strict; use warnings; use autodie;
use Lintian::Util qw(is_ancestor_of);
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $root = $info->unpacked
  my $path = $info->unpacked($filename);
  if ( -f $path and is_ancestor_of($root, $path))  
    open(my $fd, '<', $path);
    ...;
    close($fd);
   
  return;
 
Where is_ancestor_of is the only available utility to assist you currently. It hides away some 10-12 lines of code to resolve the two paths and correctly asserting that $path is (an ancestor of) $root. Prior to Lintian 2.5.12, you would have to do that ancestor check by hand in each and every check[4]. In the new version, the correct code would look something like this:
package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run  
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $path = $info->index_resolved_path($filename);
  if ($path and $path->is_open_ok)  
    my $fd = $path->open;
    ...;
    close($fd);
   
  return;
 
Now, you may wonder how that promotes safer code. At first glance, the checking code is not a lot simpler than the previous correct example. However, the new code has the advantage of being safer even if you forget the checks. The reasons are:
  1. The return value is entirely based on the file index of the package (think: tar vtf data.tar.gz). At no point does it use the file system to resolve the path. Whether your malicious package trigger an undef warning based on the return value of index_resolved_index leaks nothing about the host machine.
    1. However, it does take safe symlinks into account and resolves them for you. If you ask for foo/bar and foo is a symlink to baz and baz/bar exists in the package, you will get baz/bar . If baz/bar happens to be a symlink, then it is resolved as well.
    2. Bonus: You are much more likely to trigger the undef warning during regular testing, since it also happens if the file is simply missing.
  2. If you attempt to call $path->open without calling $path->is_open_ok first, Lintian can now validate the call for you and stop it on unsafe actions.
It also has the advantage of centralising the code for asserting safe access, so bugs in it only needs to be fixed in one place. Of course, it is still possible to write unsafe code. But at least, the new API is safer by default and (hopefully) more convenient to use. [0] Lintian.debian.org being the primary exception here. [1] This is in contrast to e.g. piuparts, which very much trusts its input packages by handing the package root access (albeit chroot ed, but still). [2] And also a bug. Not all binary packages have a copyright instead some while have a symlink to another package. [3] The code is hand-typed into the blog without prior testing (not even compile testing it). The code may be subject to typos, brown-paper-bag bugs etc. which are all disclaimed (of course). [4] Fun fact, our documented example for doing it correctly prior to implementing is_ancestor_of was in fact not correct. It used the root path in a regex (without quoting the root path) fortunately, it just broke lintian when your TMPDIR / LINTIAN_LAB contained certain regex meta-characters (which is pretty rare).

7 August 2014

Niels Thykier: Recent improvements to Britney2

As mentioned by Rapha l Hertzog, I have been spending some time on improving Britney2. Just the other day I submitted a second branch for review that I expect to merge early next week. I also got another set patches coming up soon. Currently, none of them are really user visible, so unless you are hosting your own version of Britney, these patches are probably not all that interesting to you. The highlights:
  1. Reduce the need for backtracking by finding semantically equivalent packages.
  2. Avoid needing to set up a backtrack point in some cases.
    • This has the side-effect of eliminating some O(e^n) runtime cases.
  3. Optimise installability testing of packages affected by a hinted migration.
    • This has the side-effect of avoiding some O(e^n) runtime cases when the post-hint state does not trigger said behaviour.
    • There is a follow-up patch for this one coming in the third series to fix a possible bug for a corner-case (causing a valid hint to be incorrectly rejected when it removed an uninstallable package).
  4. Reduce the number of affected packages to test when migrating items by using knowledge about semantically equivalent packages.
    • In some cases, Britney can now do free migrations when all binaries being updated replace semantically equivalent packages.
  5. (Merge pending) Avoid many redundant calls to sort_actions() , which exhibits at least O(n^2) runtime in some cases.
    • For the dataset Rapha l submitted, this patch shaves off over 30 minutes runtime. In the particular case, each call to sort_actions takes 3+ minutes and it was called at least 10 times, where it was not needed.
    • That said, sort_actions have a vastly lower runtime in the runs for Debian (and presumably also Ubuntu, since no one complained from their side so far).
The results so far: After the first patch series was merged, the Kali dataset (from Rapha l) could be processed in only ~2 hours. With the second patch series merged, the dataset will drop by another 30-50 minutes (most of which are thanks to the change mentioned in highlight #5). The third patch series currently do not have any mention-worthy performance related changes. It will probably be limited to bug fixes and some refactoring. Reflections: The 3 first highlights only affects the new installability tester meaning that the Britney2 instances at Ubuntu and Tanglu should be mostly unaffected by the O(n^2) runtime. Although those cases will probably just fail with several AIEEE s. :) The 5th highlight should equally interesting to all Britney2 instances though. For me, the most interesting part is that we have never observed the O(n^2) behaviour in a daily sid -> testing run. The dataset from Rapha l was basically a stable -> testing/sid run, which is a case I do not think we have ever done before. Despite our current updates, there is still room for improvements on that particular use case. In particular, I was a bit disheartened at how poorly our auto hinter(s) performed on this dataset. Combined they only assisted with the migration of something like 28 items . For comparison, the main run migrated ~7100 items and 9220 items were unable to migrate. Furthermore, the Original auto hinter spend the better part of 15 minutes computing hints at least it results in 10 items migrating. Links to the patches:

23 March 2014

Niels Thykier: Upcoming changes to Lintian (in 2.5.22)

The next version of Lintian, 2.5.22, is long overdue mostly because 2.5.21 FTBFS in sid. Besides fixing test failures, 2.5.22 also fixes: Besides bug fixes, we also added a couple of new features/nice changes: We just need to do some final tweaks and get the tag descriptions reviewed before we are ready to release the next version.

26 January 2014

Niels Thykier: Release architecture meeting 2014-01-26

Today, we held an one-hour IRC-meeting to debate the status of the current architectures in sid. During that one hour, we brought up all 13 architectures. We made a decision on 9 of the architectures. The remaining 4 will be revisited within 2 months. As for the actual results, we will announce these on debian-devel-announce with our next bits mail. I suspect the announcement to happen within a couple of days. I am quite pleased that we managed to keep the meeting to one hour. Especially considering that it was planned less than a week ago. One of the things, that I believe helped us, was that we had summarised the status for each architecture prior to the meeting. In my view, the summary allowed us to finish at least half of the architectures within their allocated 5 minutes. In fact, we had 20 minutes for the last 2 architectures partly because amd64 + i386 combined took about a minute. A second thing that helped us stay on time was cutting debates short. Obviously, with less than 5 minutes per architecture, there was little time for debate and, I admit, that was by design. This meant that we had to defer 4 decisions for later. We are currently waiting for information, which we hope to have soon. More on that in the announcement later.
I would like to thank the attendees for contributing to a productive and short meeting. It was a pleasure and I hope we can repeat the success next time. :)

28 December 2013

Richard Hartmann: Release Critical Bug report for Week 52

I had been pondering to do an end-of-year bug stat post. Niels Thykier forced my hand, so here goes :) The UDD bugs interface currently knows about the following release critical bugs: Graphical overview of bug stats thanks to azhag:

26 December 2013

Niels Thykier: Jessie finally has less than 500 RC bugs

I am very pleased to say that RC bug counter for Jessie finally dropped below 500 RC bugs. For comparison, we broke the 700 RC bug curve in the start of November, so that is about 200 RC bugs in about 7-8 weeks (or about 25-28 RC bugs per week). Today, we have about 162 RC bugs on the auto-removal list. Though, I suspect many of the affected packages have reverse dependencies, so their removal from testing may be up 30 days away. Nevertheless, by this time in January, we could be looking at no more than 350 RC bugs left for Jessie.

23 December 2013

Niels Thykier: Automated reprocessing of packages on lintian.debian.org

Yesterday, I have managed to finish up an old pet peeve of mine. I wrote a series of patches to have harness reprocess all packages, which were last checked by an older version of Lintian than the current. It is not completed to the level I would like, but it is good enough for now. The implementation works by processing up to 1024 groups after each daily run. It also have a timer to skip this, if the run has taken more than 4 hours. Packages to be reprocessed are sorted by the version they were last processed should a new version of Lintian be upload before all packages have been reprocessed, the oldest results are refreshed first. :) A rough estimate suggests than in about 3-4 weeks, the entire archive will have been reprocessed with Lintian 2.5.20. It is by no means faster than a full run (which is about a week), but it is automatic and removes the human factor in keeping Lintian results up to date.

21 December 2013

Niels Thykier: Getting out of the way

I have decided to step down as main maintainer of Lintian and pass the baton to Bastien Roucari s. This is actually fairly old news, since I announced this almost a month ago. However, I was not very vocal about it until now (so do not be surprised if you had not heard of this before). In the past month, I never once regretted having stepped down. If anything, I should probably have done it a bit earlier. Beyond the lack of motivation, I also realised that I had become an all talk and no do -maintainer. The kind that I have been advocating against publicly. This may sound weird to a lot of people, who knows me as the Lintian guy or Mr. Lintian (or whatever Lintian-related nickname people gave me). But the fact is that Bastien has done more for Lintian in the past month than I have in the past two. Despite stepping down as main developer, I have not left Lintian completely. I am still around for helping/explaining, uploading and keeping lintian.debian.org running.

3 November 2013

Niels Thykier: Breaking the 700 RC bug threshold for Jessie

It looks like we have dropped (or will drop) below 700 RC bugs affecting Jessie today[1]. :) On a related note, I got my eye on libfso-glib + fso-gsmd, samba + -samba4, mono and haskell. By my count, each of the four groups will (once they migrate) bring at least 4 RC bug fixes with them to testing. Besides that, we still have ~97 RC bugs in the autoremoval queue[2], so we could be looking at 600 RC bugs in a week or two. Please keep up the steady flow of RC bug fixes; but also, please make sure your packages will actually migrate. If you need help interpreting the excuses page, do not hesistate to ask I will not bite. If you want to help, please consider having a look at the cleaner or the Release Team -view on the UDD bug tracker. Sadly not everything there contains actionable problems[3]. [1] http://bugs.debian.org/release-critical/ says 700 while the IRC bot in #debian-devel-changes says we are at 692. [2] http://udd.debian.org/bugs.cgi?release=jessie_and_sid&merged=ign&autoremovals=only&fnewerval=7&flastmodval=7&rc=1&sortby=id&sorto=asc dd-list-like: http://udd.debian.org/cgi-bin/autoremovals.cgi [3] E.g. the kfreebsd-8 bugs appear because kfreebsd-8 is listed in Built-Using for d-i, but the package has already been removed from testing. So there is nothing to do with that (except possibly waiting for a new version of d-i).

26 October 2013

Niels Thykier: Breaking the RC bug curve

Early this month, the first batch of packages were automatically removed from testing due to RC bugs. Personally, I find that the results stand out pretty well on the RC bug graph[1]. There has been a very rapid decline in the number of RC bugs affecting testing a week or two ago, we had over 1000 RC bugs affecting testing and now it has dropped to something like 760. There has also been quite a few RC bug fixes in this time period. I spent a bit of time finding and aging some of the RC bug fixes in sid, so they would migrate to testing faster[2]. On the first batch, we saw something like 40 RC bugs disappear and we still have more RC bugs fixes in the sid->testing pipeline. While looking for packages I could age, I also noticed that we have quite a few RC bug fixes blocked by out of date binaries (which is usually either build issues or uncoordinated transitions). I have been filing a couple of RC bugs for these, but it does not scale do this kind of work manually. If you have uploaded a package recently, please take the 5 minutes to check that it actually built and it has no out of date binaries issues in its excuses. [1] http://bugs.debian.org/release-critical/ [2] The aging was equivalent to an urgency=medium, so reduced the requirement from 10 days to 5 days.

7 October 2013

Gregor Herrmann: using lintian profiles for fun & profit

since lintian gained support for vendor profiles, or at least since it has a nice documentation about writing checks & even a tutorial (the latter only local/in the package), I thought about writing some checks in a specific 'pkg-perl' profile. after talking to niels at debconf13, I tried in late august, & realized that at least simple checks are fun to write & really not difficult; luckily niels also reviewed my first attempts. & axel soon joined the fun & added some more checks. so here we are: the new pkg-perl-tools package also ships a lintian vendor profile 'pkg-perl' with a couple of simple but still useful checks that help to enforce our group policy. if you want to use them, install the package and check /usr/share/doc/pkg-perl-tools/README.lintian, if you want to look at the checks, you can also clone the git repo or inspect it in gitweb (directory lintian).

Next.

Previous.