Search Results: "Frank Lichtenheld"

28 December 2008

Russ Allbery: Lintian 2.1.3

Over the last couple of days, I've found some time to work on Lintian for the first time in quite a while. It's been a good feeling both for getting back to a project that I've done very little on for several months and finally feeling energized about a coding project. I think after a week of vacation, I'm starting to recover my energy. I uploaded Lintian 2.1.3 earlier today and am currently doing a fresh archive-wide run for lintian.debian.org. This version has several new checks for watch files from Raphael Geissert, several fixes for the lintian.d.o output, a few other random bug fixes, and an attempt to tackle unversioned shared libraries in a more comprehensive fashion. Shared libraries with an SONAME that doesn't contain a version number can't be represented in the shlibs system (but can in the symbols system), so Lintian always reports them as missing from shlibs when they're installed in public directories. That's not really the right tag, since they don't belong in shlibs. The more basic problem is that, except for a few special cases mostly coming from libc, they really don't belong in public directories. With no versioning in the SONAME, we can't handle changes to the ABI without completely changing the library name and can't use the shlibs system to manage them. Most of the libraries in this situation are actually private libraries for one particular application. As Policy already says, those libraries shouidn't be installed in public directories like /usr/lib; they should be installed in private subdirectories of /usr/lib for that application. The biggest group of offenders here currently seems to be KDE; for some reason, unless I'm missing something, this seems to be a semi-standard KDE way of doing things. This version of Lintian will hopefully handle this better. Such libraries won't be expected to be in shlibs files and won't get erroneous tags about not being listed there. We'll also do a better job of distinguishing between versioned shared libraries and shared library names that just contain a hyphen. Such unversioned libraries will get a separate tag pointing at the portion of Policy that says to put private libraries and plugins in a subdirectory instead of /usr/lib. Thanks as always to Frank Lichtenheld, Adam D. Barratt, and Jord Polo for all the hard work they've been putting into Lintian. It's gotten much better over the few months that I've not been able to work on it, which is making me more excited to get back to it and do more.

21 September 2008

Frank Lichtenheld: Lintian 2.0.0~rc2 in experimental

For the impatient reader: New lintian in experimental, please test and give feedback. You will miss most changes though unless you read the rest of the post (Hint, Hint ;)) During the past week I've uploaded new lintian versions to experimental which we designated to be release candidates for 2.0.0. Code-wise the changes are not that much more intrusive than for many of our past releases, but they change the way lintian classifies tags in a fundamental way, thanks due to the hard work of Jord Polo in his Google Summer of Code project (mentored by Marc Brockschmidt). Lintian Tag Classification, old and new Previously lintian classified tags only in one dimension, in the categories "Info", "Warning", and "Error". While this worked reasonably well, the difference between the categories was not very well defined. The general idea was that everything violating a "must" in Debian Policy or endangering the building or usage of the package should be an "Error", i.e. something very similar to the definition of RC bugs (except that not all "must"s in Policy are deemed worthy of filing RC bugs). Some errors were downgraded to "Warning" or even "Info" though on the basis that their detection was too prone to false positives. Due to this it was a long existing desire to split the classification of tags into two dimensions, one for the impact/importance of the tag, and one for the certainty of its correct detection. This should make it easier for people to interpret and/or filter the output. At various points in the last few years people began to work on this but quickly gave up, usually overwhelmed by the sheer number of tags (728 in 2.0.0~rc2) to classify anew and to make sure that the old and new categorisation could exist side-by-side (because breaking backwards compatibility was not really feasible). Finally this year Jord Polo decided to tackle this task as a Google Summer of Code project, with great success. Tags are now classified in two dimensions "Severity" (with the possible values wishlist, minor, normal, important, serious, which are intentionally very close to the available severities in the Debian bug tracking system), and "Certainty" (possible values: wild-guess, possible, certain). A third classification by "Source" (i.e. Policy, Developers Reference, ...) is planned but not yet fully implemented. For backwards compatibility there is a mapping of these new classification to the old ones (which lead to a few reclassifications of tags). The default output of lintian is unchanged. The new output formats that support the classification are still experimental (see below). How to use it You can specify exactly which levels of Severity and Certainty you want to have displayed with the new --display-level (-L) option. Please see the manual page for the details, but to give you an idea, the default behaviour (i.e. "show warnings and errors" in the "old" vocabulary) is equivalent to specifying
-L ">=important" -L "+>=normal/possible" -L +minor/certain
And to get a report with only severe tags we're very certain of, you could use
-L ">=important/certain"
which will only display tags that have severity "important" or "serious" and a certainty of certain. There is also the (intentionally undocumented) option --exp-output which allows you to play with some experiments we're doing with the output format. --exp-output format=letterqualifier will give you an output very similar to the "classic" one, but with additional information about severity and certainty. --exp-output format=colons gives you a colon-separated format which includes all the possible information lintian currently has available during tag output and which should be easily machine-consumable. Note that these formats are experimental and might be changed at any point without notice. If you're interested in using alternative formats for lintian output, please join the mailing list and talk to us about it. Etc. Other changes include the usual share of bug fixes and of course: New tags Credits This lintian release is brought to you by (sorted by number of changesets):

14 September 2008

Frank Lichtenheld: Backport of git 1.6 to etch

Since I can't upload git 1.6 from experimental to backports.org (because that is only for packages available in testing), I've made a backport to etch available at people.debian.org. If you have any issues with that that don't affect the version in experimental, feel free to drop me a mail.

11 September 2008

Enrico Zini: Undoable apt-get build-dep

Undoable apt-get build-dep If I do apt-get build-dep foobar, all the dependencies will be installed, but they will not be marked as autoinstalled: if I then do not need to build the package anymore, I have a hard time finding out what to remove. What I would like is a script that turns build dependencies of a package into a metapackage, so that I can later remove the metapackage and apt-get autoclean will remove everything from the system. It turns out that it already exists: it's called mk-build-deps and is part of devscripts:
# apt-get install equivs
$ mk-build-deps lxlauncher

That will give you lxlauncher-build-deps_0.2-2_all.deb. It also works on arbitrary control files. There is also a repository of such build-deps packages maintained by Frank Lichtenheld.

21 August 2008

Frank Lichtenheld: Source Views

(This is a more detailled and hopefully better structured explanation of the topic of my Lightning Talk during Debconf) With the ever growing number of packages in our archive we need ever better methods to structure, sort, and search this collection. While projects like debtags try to improve the search in packages, my idea is about improving the way the results are displayed. There is one relation between packages that is currently never strongly featured when presenting package meta-data: the source ↔ binary relation. Usually package managers operate either on binary packages or source packages, but seldomly do they expose the fact that one is generated from the other. In the trivial case of one source package producing one binary package there is probably nothing to gain. But there are a large number of source packages where this isn't true (at the moment about 29% of all the source packages in main build more than one binary package, together producing about 61% of all binary packages). I have the idea (which so far is nothing more) that if we could present packages through their source packages that this could drastically reduce the number of packages to choose from and make some common search actions more effective. This is not by any meaning a trivial task, though, since on the other hand not all source packages necessarily produce binary packages that have much to do with each other, and even for the ones that do, the exact relationship between the binary packages can vary greatly. Also some source packages are really belonging together but are split to make maintenance easier (e.g. KDE and X.org). These splits might not necessarily make any sense to an user. Some ideas what to implement in terms of code to make it easier to let the user benefit from the fact that multiple binaries have the same source:
Add support for a Description field for source packages.
For bonus points implement automatic substvars $ source:Description , $ source:Short-Description , and $ source:Long-Description that make it trivial to reuse (parts of) that description in the descriptions of the binary packages.
Add some way to identify the "main" package built from a common source.
Identify source package usage patterns.
There are a lot of source packages that share common patterns about what they build. Example patterns include:
Application
Main package + -data package + -doc package + ...
Library
-dev package + shlib package + -doc package + ...
Application + Library
Collection of Applications
etc.
I guess many of us already look out for these patterns when searching for packages but identifying them explicetly might make this easier and allow more users to benefit from this information.

Frank Lichtenheld: RC bugs in etch (Starting Stable QA, Part 1)

As everyone can see in the graph of release critical bugs, the number of RC bugs in etch had constantly risen since the release (which was already noted in some other blogs) up until last week. Since I suspected that this number included a lot of false positives, I began to triage this list last week during Debconf. I started with identifying all bugs that were filed against the version in etch, but really only meant the package in testing/unstable (but can't be interpreted correctly with version tracking alone since the version in stable, testing, and unstable is actually identical). There are a lot of reasons why an RC bug might be valid for testing/unstable, but not for stable, even though they share the same package:
Library transitions.
While most cases of library transitions can be handled with targeted binNMUs these days, there remain a few cases where a sourceful upload is needed, e.g. for renamed -dev packages and necessary source changes to adapt to new APIs.
Build-Problems caused by toolchain changes.
Newer gcc versions are often more strict about what constructs they allow. This can cause build problems in a lot of packages once the default version used is increased. Other packages that regulary cause new build problems are linux-libc-dev and dpkg-dev. Since we only require that a package is buildable in the release it is contained in, these are not valid RC bugs for stable.
Changes in the RC policy of the release team.
One example during the lenny release cycle was the use of invoke-rc.d in maintainer scripts to (re-)start daemons. Bugs were filed against a lot of packages not using invoke-rc.d before the etch release but they were only declared to be of RC severity after etch.
etc.
So as the first part of my triaging efforts I tried to identify these cases. I also looked for bugs that were listed as affecting etch due to incomplete version information (i.e. if there is no version given, the bug is assumed to affect all versions). As you can see from the bump in the graph I was able to identify about 200 bugs that met one of these conditions. We should avoid in the future to let the number of false positives grow to such large numbers. Everyone can help with that:
Maintainers
If you get bugs reported without a version number and you can verify that this bug was not present in an older version of the package, add correct "found" information. If you get a bug reported against a version that is both in stable and in testing/unstable, but not valid for stable, tag the bug appropriatly. I would recommend to err on the side of false positives though instead of on the side of false negatives!
Mass-bug filers
Most mass-bug filing that involve RC bugs should use tags to avoid creating false positives.
QA Group
Bugs filed about the proposed orphaning or removal of packages should usually be tagged, since only in very few cases these actually warrant any changes in stable.
At this point you probably ask yourself: What are the appropriate tags? This is much less clear than one might hope, since the involved tags (<suite>) changed/lost their meaning somewhat with the introduction of version tracking. Following my question about the appropriate tags on debian-release the release team and Don Armstrong for the debbugs maintainers seem to have agreed on their new meaning: A bug with suite tags affects the intersection of the set of suites indicated by its version information and the set of suites indicated by its suite tags. Next post: What about the other 600 RC bugs in stable?

7 August 2008

Frank Lichtenheld: Moving a mailing list from SF.net to another Mailman instance

I had to move a mailing list from SourceForge to a self-hosted Mailman instance while preserving all the user options. Since one has no shell access to these SF's Mailman I decided to extract the information from the Web-Interface, which sadly enough is no easy task. There is no complete list available but only chunked by starting letter and in groups of 30 addresses or less. Since the mailing list in question had about 1500 subscribers manual transcription was really no option. So I wrote a small script that automatically extracts all the information and outputs it in a CSV-like format. I've also hacked Mailman's add_members script to set all these options from this format. In case someone finds this useful, both are available on my git server under free licenses:
grab-subscribers.pl
Extract subscriber information from Mailman Web-Interface with LWP.
add_members
Use the information extracted by grab-subscribers.pl to populate a Mailman mailing list.
DISCLAIMER: This is hacked together really quickly and was used exactly once. Don't expect too much.

26 July 2008

Philipp Kern: Stable Point Release: Etch 4.0r4 (aka etchnhalf)

Another point release for Etch has been done; now it's the time for the CD team to roll out new images after the next mirror pulse. The official announcements (prepared by Alexander Reichle-Schmehl, thanks!) will follow shortly afterwards. FTP master of the day was Joerg Jaspert, who did his first point release since Woody, as he told us on IRC. We appreciate your work and you spending your time that shortly before going to Argentina. This point release includes the etchnhalf update introducing a new kernel image (based on 2.6.24) and some driver updates. Additionally the infamous openssl hole will be fixed for good, even for new installs. Again I want to present you a list of people who contributed to this release. It cannot be complete as I got the information out of the Changed-by fields of the uploads. From the Release Team we had dann frazier (who drove the important kernel part of etchnhalf), Luk Claes, Neil McGovern, Andreas Barth, Martin Zobel-Helas and me working on it. ;-)

13 June 2008

Frank Lichtenheld: packages.ubuntu.com moved to Canonical machine

packages.ubuntu.com is now hosted on a server provided by Canonical. This will hopefully greatly improve performance and reliability, since my own server was increasingly swamped and had repeated problems with its hard disks. Many thanks to Chris Jones for handling the move.

2 June 2008

Frank Lichtenheld: Just booked my flights

I'm going to DebConf8, edition 2008 of the annual Debian 
          developers meeting I really hope the sponsoring situation still improves, but I've decided to go either way. Judging from the last years, it's worth it :)

13 May 2008

Frank Lichtenheld: Introducing archive.debian.net

Just the other day I was wondering about what release of Debian a specific version of a package was in, and I knew it was older than oldstable. But searching the Packages files of archive.debian.org was tedious and I thought that there has to be better way. So I quickly set up a packages.debian.org instance for archive.debian.org at archive.debian.net I guess this isn't really interesting for the day-to-day use, but maybe someone can ocassionally profit from it. Caveats: Feel free to help me improve that site, preferably by sending patches against the archive-master branch of git://git.debian.org/git/webwml/packages.git.

6 May 2008

Russ Allbery: Debian status

Tonight, I converted the XML-Security-C package from Stanford's internal Subversion repository to Git and packaged 1.4.0, but unfortunately I can't upload yet because Xalan needs to be updated to build against the latest Xerces-C. (I'd NMU, but I should not be taking on more things right now.) This is the first of the three current Shibboleth packages, and I hope to tackle OpenSAML and Shibboleth SP tomorrow or Wednesday. An Alioth project for Shibboleth packaging now exists and the pkg-shibboleth-devel mailing list is being created. I've requested Git space on git.debian.org, and as soon as that's set up, I'll push the three existing repositories. Shibboleth 2.0 packaging will add at least three more packages (OpenSAML 2.0, Shibboleth SP 2.0, and another XML support library), possibly four if we have to package log4shib, but I still hope to use the standard log4cpp. And then there will be another couple of packages once the group packages the Shibboleth IdP and WAYF, which I hope someone with more Java experience can tackle. One of the main goals for getting this set up is so that people other than me can do more of the work so that I can work on other things. Like, for example, krb5 packaging. Bastian Blank started using the Kerberos server packages and posted a whole flurry of bugs, a couple of which are (somewhat arguably) RC, and I'm trying to absorb that and fix the ones I can ASAP. OpenLDAP is in desperate need of attention, but I don't feel as much responsibility for that. I said going in that I'd only have time to work on it in fits and starts, and just because no one else has any time to work on it either doesn't change that. However, I do need to find time to post an RFH, since it really needs an active maintainer, and I'll at least find time to upload the upcoming 2.4.9 release before lenny freezes more, since the current version has a ton of bugs. Frank Lichtenheld has been doing a lot of the work on Lintian lately, which is wonderful, but I still have at least 10 bugs and patches that I want to deal with as soon as possible for it and get another upload out. We're well over my magic threshold of 100 bugs, and a lot of the stuff that's in the BTS just needs to be reviewed and committed. And then there's Policy, which is just about ready for an upload, and where I owe several responses on bugs and discussion threads. It's in the best shape right now of the stuff on my hot list, but there really should be a new upload in May, and preferrably in the first half. The current goal is to get the two other RC bugs against the Shibboleth packages dealt with in the next couple of days, then do a krb5 upload before the end of the week, and then Policy and Lintian are back at the top of the list, insofar as work lets me concentrate on anything else. BTW, I have to say, the more I use Git, the more I'm becoming a convert. I still think I'd rather use quilt if I have complex merges. Ironic, given that that's supposed to be Git's strength, but I have fifteen years of experience merging patches and I really understand how quilt works. That may mean I'll change my mind when I become even more familiar with Git. But for the typical package with a small number of divergences from upstream, I'm getting the hang of managing those on branches and I think I like it. I've yet to see a good reason to use rebase instead of merge, though, at least as part of a Debian packaging workflow. So either I'm still not getting something, or many of the people using Git are way more obsessive than I am about their branch merge graphs (which is weird, because I tend to be remarkably obsessive about things like that). I suppose if I ever get to using git bisect, that may change my mind.

22 February 2008

Frank Lichtenheld: Obligatory FOSDEM post

I'm going to FOSDEM, the Free and Open Source Software Developers' European Meeting I will see you all at my talk, right? ;)

6 February 2008

Frank Lichtenheld: More updates about packages. debian.org,ubuntu.com

31 January 2008

Frank Lichtenheld: Some packages. debian.org,ubuntu.com updates

9 September 2007

Frank Lichtenheld: Introducing sourcedeps.d.n

I get regulary annoyed by the fact that I find random -dev packages on my main system and can't figure out which package's build-depends I satisfied by installing them. Also they don't disappear if the package changes its build-depends. So I decided to solve this problem. Since I'm not really a C/C++ hacker and didn't want to learn hacking APT for this I choose to solve this problem with brute force and Perl ;-) The result is sourcedeps.debian.net, a APT archive which contains one binary package for each source package. The following mapping was done:
     source package name => binary package name, but with appended '-build-depends'
     Build-Depends       => Depends
     Build-Depends-Indep => Recommends
     Build-Conflicts     => Conflicts
     Binary              => Suggests
     Binary              => Provides (with appended '-build-depends')
If any of the Build-Depends fields contains arch limiters, arch-dependent packages will be created, otherwise one arch-independent package. This allows easy installation of build-depends by installing the corresponding meta package, tracking them, and removing them automatically in case you deinstall the meta package. It only works for known architectures though and it requires creating around 60,000 binary packages. It also doesn't allow tracking build-dependencies for more than one version of a package (e.g. unstable and experimental). You can use this by adding something like
deb http://sourcedeps.debian.net/ sid main contrib non-free
to your sources.list. The archive is signed with the following key, available from a keyserver near you and signed by me:
pub   1024D/ED505694 2007-09-08 [expires: 2008-09-07]
      Key fingerprint = 4ECF DF07 F419 0B5B 45C4  51D0 00E9 C47B ED50 5694
uid                  SourceDeps.Debian.Net Archive Key 
Comments welcome.

2 September 2007

Joerg Jaspert: Upgrading packages.debian.org

Finally we (Frank Lichtenheld, myself) took the time and upgraded the host for packages.d.o to etch. This was absolutely needed, as pdo was always running with a very high load for the machine, and the new code needed some etch-specific packages/package versions. So, today we went, turned off old pdo and upgraded it. It got a new kernel and lots of new package (versions) from etch, including apache2-mpm-worker. It is now accessible for everyone again, and its running the new code. Which does look nice, is way better and also much faster. Uses libapache2-mod-perl2, database files (instead of textfiles and grep) and a nice new layout. And possibly much more changes, but Frank will send a mail about it later. (At the time of writing this blog pdo is still generating its initial database files, so not everything working atm, but when that first run is done it will be MUCH faster in the future…) Say thanks to Frank for his work on pdo! Update: Thanks to the new and fast code (a full update now only takes real 40m26.415s, user 16m36.538s and sys 0m50.055s instead of hours), packages.d.o will now update twice a day, with a push directly after the mirror update has happened. Which should greatly help to reduce the “packages.d.o is out of date” symptom…

Nico Golde: packages.debian.org relaunch

Thanks to Joerg Jaspert and Frank Lichtenheld, the new packages.debian.org site looks way better and also seems to be really faster!

10 June 2007

Frank Lichtenheld: Experimental tabbing on packages.d.o

Too make the pages for single packages on packages.debian.org less loaded with information I experimented today with changing the layout to a "tabbed" one, so that information is spreaded over several sub pages. Currently this is implemented with Javascript, but if people really like it I would probably implement it on the server side too. Please check it out and tell me what you think.

2 June 2007

Frank Lichtenheld: packages.debian.org status and development

Copy of a mail I sent to debian-www earlier today. I don't think it warrants posting to -devel-announce, but I post it here to make it visible to people not usually following debian-www. Since I seem to sense an increased stream of offers to help out with packages.d.o coming my way (but maybe that is just wishful thinking ;) I wanted to give a short update on the development of the current code and the status of the infrastructure so that nobody can claim he wanted to help but failed due to lack of information.

Next.