Search Results: "reg"

30 January 2012

Colin Watson: APT resolver bugs

I've managed to go for eleven years working on Debian and nearly eight on Ubuntu without ever needing to teach myself how APT's resolver works. I get the impression that there's a certain mystique about it in general (alternatively, I'm just the last person to figure this out). Recently, though, I had a couple of Ubuntu upgrade bugs to fix that turned out to be bugs in the resolver, and I thought it might be interesting to walk through the process of fixing them based on the Debug::pkgProblemResolver=true log files. Breakage with Breaks The first was Ubuntu bug #922485 (apt.log). To understand the log, you first need to know that APT makes up to ten passes of the resolver to attempt to fix broken dependencies by upgrading, removing, or holding back packages; if there are still broken packages after this point, it's generally because it's got itself stuck in some kind of loop, and it bails out rather than carrying on forever. The current pass number is shown in each "Investigating" log entry, so they start with "Investigating (0)" and carry on up to at most "Investigating (9)". Any packages that you see still being investigated on the tenth pass are probably something to do with whatever's going wrong. In this case, most packages have been resolved by the end of the fourth pass, but xserver-xorg-core is causing some trouble. (Not a particular surprise, as it's an important package with lots of relationships.) We can see that each breakage is:
Broken xserver-xorg-core:i386 Breaks on xserver-xorg-video-6 [ i386 ] < none > ( none )
This is a Breaks (a relatively new package relationship type introduced a few years ago as a sort of weaker form of Conflicts) on a virtual package, which means that in order to unpack xserver-xorg-core each package that provides xserver-xorg-video-6 must be deconfigured. Much like Conflicts, APT responds to this by upgrading providing packages to versions that don't provide the offending virtual package if it can, and otherwise removing them. We can see it doing just that in the log (some lines omitted):
Investigating (0) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-tseng:i386
Investigating (1) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-i740:i386
Investigating (2) xserver-xorg-core [ i386 ] < 2:1.7.6-2ubuntu7.10 -> 2:1.11.3-0ubuntu8 > ( x11 )
  Fixing xserver-xorg-core:i386 via remove of xserver-xorg-video-nv:i386
OK, so that makes sense - presumably upgrading those packages didn't help at the time. But look at the pass numbers. Rather than just fixing all the packages that provide xserver-xorg-video-6 in a single pass, which it would be perfectly able to do, it only fixes one per pass. This means that if a package Breaks a virtual package which is provided by more than ten installed packages, the resolver will fail to handle that situation. On inspection of the code, this was being handled correctly for Conflicts by carrying on through the list of possible targets for the dependency relation in that case, but apparently when Breaks support was implemented in APT this case was overlooked. The fix is to carry on through the list of possible targets for any "negative" dependency relation, not just Conflicts, and I've filed a patch as Debian bug #657695. My cup overfloweth The second bug I looked at was Ubuntu bug #917173 (apt.log). Just as in the previous case, we can see the resolver "running out of time" by reaching the end of the tenth pass with some dependencies still broken. This one is a lot less obvious, though. The last few entries clearly indicate that the resolver is stuck in a loop:
Investigating (8) dpkg [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( admin )
Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils ) (< 1.15.8)
  Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
  Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
Investigating (8) dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils )
Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl ) (= 1.16.1.2ubuntu5)
  Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
  Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
Investigating (9) dpkg [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( admin )
Broken dpkg:i386 Breaks on dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils ) (< 1.15.8)
  Considering dpkg-dev:i386 29 as a solution to dpkg:i386 7205
  Upgrading dpkg-dev:i386 due to Breaks field in dpkg:i386
Investigating (9) dpkg-dev [ i386 ] < 1.15.5.6ubuntu4.5 -> 1.16.1.2ubuntu5 > ( utils )
Broken dpkg-dev:i386 Depends on libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl ) (= 1.16.1.2ubuntu5)
  Considering libdpkg-perl:i386 12 as a solution to dpkg-dev:i386 29
  Holding Back dpkg-dev:i386 rather than change libdpkg-perl:i386
The new version of dpkg requires upgrading dpkg-dev, but it can't because of something wrong with libdpkg-perl. Following the breadcrumb trail back through the log, we find:
Investigating (1) libdpkg-perl [ i386 ] < none -> 1.16.1.2ubuntu5 > ( perl )
Broken libdpkg-perl:i386 Depends on perl [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
  Considering perl:i386 1472 as a solution to libdpkg-perl:i386 12
  Holding Back libdpkg-perl:i386 rather than change perl:i386
Investigating (1) perl [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
Broken perl:i386 Depends on perl-base [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl ) (= 5.14.2-6ubuntu1)
  Considering perl-base:i386 5806 as a solution to perl:i386 1472
  Removing perl:i386 rather than change perl-base:i386
Investigating (1) perl-base [ i386 ] < 5.10.1-8ubuntu2.1 -> 5.14.2-6ubuntu1 > ( perl )
Broken perl-base:i386 PreDepends on libc6 [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs ) (>= 2.11)
  Considering libc6:i386 -17473 as a solution to perl-base:i386 5806
  Added libc6:i386 to the remove list
Investigating (0) libc6 [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs )
Broken libc6:i386 Depends on libc-bin [ i386 ] < 2.11.1-0ubuntu7.8 -> 2.13-24ubuntu2 > ( libs ) (= 2.11.1-0ubuntu7.8)
  Considering libc-bin:i386 10358 as a solution to libc6:i386 -17473
  Removing libc6:i386 rather than change libc-bin:i386
So ultimately the problem is something to do with libc6; but what? As Steve Langasek said in the bug, libc6's dependencies have been very carefully structured, and surely we would have seen some hint of it elsewhere if they were wrong. At this point ideally I wanted to break out GDB or at the very least experiment a bit with apt-get, but due to some tedious local problems I hadn't been able to restore the apt-clone state file for this bug onto my system so that I could attack it directly. So I fell back on the last refuge of the frustrated debugger and sat and thought about it for a bit. Eventually I noticed something. The numbers after the package names in the third line of each of these log entries are "scores": roughly, the more important a package is, the higher its score should be. The function that calculates these is pkgProblemResolver::MakeScores() in apt-pkg/algorithms.cc. Reading this, I noticed that the various values added up to make each score are almost all provably positive, for example:
         Scores[I->ID] += abs(OldScores[D.ParentPkg()->ID]);
The only exceptions are an initial -1 or -2 points for Priority: optional or Priority: extra packages respectively, or some values that could theoretically be configured to be negative but weren't in this case. OK. So how come libc6 has such a huge negative score of -17473, when one would normally expect it to be an extremely powerful package with a large positive score? Oh. This is computer programming, not mathematics ... and each score is stored in a signed short, so in a sufficiently large upgrade all those bonus points add up to something larger than 32767 and everything goes haywire. Bingo. Make it an int instead - the number of installed packages is going to be on the order of tens of thousands at most, so it's not as though it'll make a substantial difference to the amount of memory used - and chances are everything will be fine. I've filed a patch as Debian bug #657732. I'd expected this to be a pretty challenging pair of bugs. While I certainly haven't lost any respect for the APT maintainers for dealing with this stuff regularly, it wasn't as bad as I thought. I'd expected to have to figure out how to retune some slightly out-of-balance heuristics and not really know whether I'd broken anything else in the process; but in the end both patches were very straightforward.

29 January 2012

Gregor Herrmann: RC bugs 2012/04

good news: from looking through RC bugs in the BTS, it seems that more & more people are starting to join the RCBW effort!

& here's my usual list for the past week:

28 January 2012

Russ Allbery: Summary of C++ symbols experience

This is a follow-up to my previous post about the way that Debian's library symbols analysis support interacts with C++ libraries, based on further debian-devel discussion. The primary outcome of that discussion was to point me at the pkg-kde-tools utilities for handling maintenance of these files, which solved a bunch of my problems. As before, this was also posted on debian-devel, which is where I expect most of the discussion to be. First of all, for those who haven't explored the pkg-kde-tools infrastructure for this, it looks like the effective process goes something like this:
  1. Do a local build of the package and then generate an initial symbols template for that architecture. So, for example, I did:
        fakeroot debian/rules build install
        pkgkde-gensymbols -plibxml-security-c16 -v1.6.1 -Osymbols.i386 \
            -edebian/libxml-security-c16/usr/lib/*/libxml-security-c.so.*.*
        pkgkde-symbolshelper create -o debian/libxml-security-c16.symbols \
            -v 1.6.1 symbols.i386
    
    This generates an initial symbols file that will work for i386.
  2. Build and upload this version of the package. Now wait for all the buildds to fail (because they will, on probably nearly every other architecture than your local one).
  3. Download all of the build logs and feed them back into pkgkde-symbolshelper so that it can adjust your symbols template for all of the architecture variations. So, for example, I did:
        pkgkde-getbuildlogs
        pkgkde-symbolshelper batchpatch -v 1.6.1 *_unstable_logs/*.build
    
    This will generate rather nice annotated symbols files with appropriate arch lists and using subst tags where appropriately (such as when an argument to a function is a size_t).
  4. Modify your package to use pkg-kde-tools during the build, because subst tags are a really nice way to handle a lot of C++ symbol patterns but #533916 against dpkg-dev is wontfix, so you need the pkg-kde-tools replacement for dpkg-gensymbols. I added a dependency on pkg-kde-tools and then added the (undocumented) pkgkde_symbolshelper add-on to my dh --with flag.
  5. Build and upload the package again. Now it will hopefully build on all architectures.
This is, as you might expect, a fair bit of work, but it does work, and what comes out the other end is a symbols file that works across the current buildd architectures and detects any mistakes by upstream that change the symbols exported. However, it does have some problems, some obvious, and some more subtle. Here's a list of the issues that I see:
  1. It feels like this symbols file is still likely to be fragile. While pkg-kde-tools detects a lot of template instances and marks them optional, I was still seeing a lot of "leaked" symbols from other C++ libraries that my package build-depends on, and which I suspect may also disappear. There's also the problem of optimizers eliminating some things that are inlined; for example, that appears to be the variation that my package sees on arm. And for a substantial C++ library, the symbols file is HUGE. Over 12,000 lines for one of my libraries. It's kind of scary to think of how many fragility landmines are buried under there.
  2. By the nature of this process, it's very sensitive to the current supported architectures. I expect any package with a symbols template built this way stands a good chance of FTBFS on a new architecture. For example, my symbols file has a lot of !armel !armhf patterns, which don't appear to be anything unique to arm except that the C++ compiler behaves slightly differently there due to inlining, and those are the only two architectures in the current set with that pattern.
  3. The lack of dpkg-gensymbols support for subst is very unfortunate, since it's used by the pkg-kde-tools utilities and it's the Right Thing To Do for symbols that take, for example, size_t as an argument. I would strongly encourage the dpkg maintainers to accept the work in #533916 except maybe for the vt work, since the alternative is to generate regexes that are significantly worse at capturing mistakes (since they're going to accept either int or long, rather than only whatever one corresponds to size_t, for example), or to embed a lot of specific architecture restrictions listing, for instance, all 64-bit architectures and guarantee FTBFS for every new architecture.
  4. It's a little hard on the infrastructure, since adding a symbols file requires uploading a package that's guaranteed to fail to build almost everywhere just so that you can get all the build logs to feed into the machinery. I always feel guilty about doing that.
  5. It's still not clear that the benefit is worth the amount of effort, since I expect most C++ libraries to require frequent SONAME changes anyway, which means that the long-term binary compatibility angle of symbols is probably futile. Mostly, all of this is to give you a tool to do ABI checking, which will catch some mistakes but not all of them. And I'm guessing it's going to be substantially more work than maintaining symbols for a C library if one actually does it properly and checks every missing symbol and change in new builds.
I'm going to try this for the Shibboleth packages for a while since I want to see what it's like across a new upstream release, but I'm still very much not convinced that this is a useful use of a packager's time.

27 January 2012

Lior Kaplan: PHP 5.4 @ Zend

As a Linux integration guy at Zend, most of my time is spent in compiling PHP related code or dealing with the variety of Linux distributions we support. With the coming release for PHP 5.4, we (at Zend) had some interesting stuff going on. As part of the RC phase, I got to check the status of the 50-60 PHP extensions we provide, especially the PECL extensions which have different release cycles than the main PHP. With minor versions, this usually doesn t really matter, but for major versions this means that some extensions need a little bit of love and fixes to work well with the new PHP version. This of course with the help of our developers. The changes are usually one-liners due to a variable type change, or finding commits in the extension s SCM and applying/back-porting it to the current versions (e.g. pecl memcache). Our policy regarding the patches we have, is that they should at least be sent to upstream (a core member or a bug report, e.g. #55703). I think I m in the best position to enforce that patch policy, so in a few recent cases, I found myself asking one of our developers if the patch he sent me was already accepted upstream before willing to take it into the build process (in this case they are used temporarily, till we ll work with the next RC or final release).
While most people build PHP as a final and standalone product, we also test it against ZendServer (or the other way around, depends on your POV). This helps to discover problems related to and implications of the changes done in the major version. During the PHP 5.4 RC cycle (which is still not finished), we had, more than a few times, that an internal problem discovered led to debugging, code reviewing and sending feedback (and patches when relevant) to the PHP project. Providing fixes for the issues found, helps having PHP in a better shape for release. At least for me, that s one of the fun parts of work getting the chance to contribute back to the community (or at least making sure others do, as I don t write the patches myself).

Filed under: PHP

26 January 2012

MJ Ray: Phones, Privacy and Co-ops

And now a slightly longer than usual rant: The problem with the o2 network disclosing mobile browsers phone numbers that I repeated 2 days ago (and it appeared on our co-op website) snowballed yesterday to the point that it was on the short bulletins from ITN, BBC, IRN and probably many more. And then o2 fixed it. Good! The reply claims that it s only since 10th January which is rather at odds with other claims that it has been happening since at least March 2010 in some situations. I started buying from o2 in December. I was using Three, but their network where I stay in Norfolk isn t reliable and you can t just buy a device in a shop for The Phone Co-op. The dongle from o2 is a recent Huawei USB device that just worked in debian and was fairly easy for me to get working in Ubuntu. There s space in it for a memory card, so maybe I could boot from it but that s an idea for later. The o2 deal is OK but not great, and the included wifi is nowhere near as good as it looked: when it says it includes BT Openzone that doesn t include any of the BT Openzone-H hotspots that are much more common. You re only allowed to register one device for wifi, so no using your phone, tablet and laptop at different times! I can t believe it s legal to advertise that as unlimited wifi , but o2 is still a better offer than access to BT Openzone-H hotspots at 39/month (yes, that s the price for wifi-only ). Ultimately, I think the problem is that there s a rubbish choice of mobile (wifi or 3G) internet access providers in the UK. It s a completely and utterly failed market, so you need to use Virtual Private Networks and similar tricks to protect yourself from the dysfunctional networks. My VPN meant my mobile number was safe: how about yours? As luck would have it, I had already proposed a resolution about protecting customer privacy to The Phone Co-op (affiliate link) for our AGM on Saturday 4 February (if you re a member, let me know). We were trying to find a compromise wording and I don t think this little o2 scandal has hurt my proposal at all! At least the phone co-op s mobile service is based on Orange s network, which wasn t affected. How does your network perform? There s an Internet Service Provider evilness test which might tell you.

Joey Hess: announcing github-backup

Partly as a followup to a Github survey, and partly because I had a free evening and the need to write more haskell code, any haskell code, I present to you, github-backup. github-backup is a simple tool you run in a git repository you cloned from Github. It backs up everything Github knows about the repository, including other forks, issues, comments, milestones, pull requests, and watchers. This is all stored in the repository, as regular files, on a "github" branch. Available in Cabal now, in Debian maybe if someone packages haskell-github.

24 January 2012

Gunnar Wolf: BugCon friends, are you trying to scare away 50%+ of the target audience?

You are scaring away much more than that. I just came across an invitation for BugCon 2012. BugCon is a Mexican conference devoted to computer security I cannot comment on its level or value because, although it's a topic that has long interested me, I must recognize each day I feel less of an expert, nowadays finding myself at the level of a "sysadmin who tries not to be too dumb for his own job security". Oh, and also because it would be completely off-topic for this post. If you look at Vendetta's (the main organizer) blog post, it will probably give you the impression that the conference is just an excuse for the afterparty: Lets go see some b00bs! Do you think your fellow female hackers will have any interest in joining a bunch of sex-starved, hormone-infested teenagers who only want to pwn a website and grab more pr0n? Do you think females will feel welcome (or even mildly safe) between you? I would not think so. And I also think you are alienating any professional who might have any interest in joining your community, be it as a member, as a mentor, or whatnot. I cannot right now do a coherent post on this topic, but I can reference you to what I have seen (and read) over the last almost 10 years, when the issue was first brought up to our attention. I am very glad to see that, at least in the Free Software area, there has been a real change of mindset. I hope you are in time to think about it and rectify. Oh, and not the description of an incident, but a very interesting and thoughtful take on this: [pdf] Interesting analysis by Hannah Wallach on the numbers and motivations of women in Free Software groups. I don't know if Hannah has published this in article form, but many interesting points can be understood by looking at the presentation. My good friend Vendetta: I don't mean this post (longer than what I originally intended) as a way to say you and the conference you are organizing for the third year (IIRC) already is unprofessional or targetted to pimply teenagers. I know the work you have put in it. I hope you see the points I'm trying to drive You are of course free to have whatever afterparty you have. But, if as the main organizer, you are giving the images of nice chicks at Hooters more weight and relevance than to the conference itself... you are doing yourself a disservice. I hope you can rectify it, and make BugCon attractive to hacker women as well.

Lior Kaplan: RTL status for Libre Office 3.5.0

As LibreOffice is approaching its final 3.5.0 release, I d like to sum up the RTL status for RC1. So far, 6 RTL related bugs were resolved in the 3.5 cycle (#32530, #34222, #40950, #43790, #43793, #44078), and a few minor issues reported directly to the developer s mailing list got quick responses. Most importantly, the new features of page break and header/footer not only support RTL but actually looks good. During the LibreOffice conference I was suggested to help with these features, providing feedback, and I m glade the needed attention was given to it. Besides that, a few l10n and translation issues were solved in the process of doing the Hebrew translation (which also reflects on other RTL languages). At a few cases, these issue because a general l10n issues which affects all the languages. In general, I found the core developers responsive to mails about RTL support. I m sure the talk about RTL problems during the conference helped, as well as being more active in the project and having more personal acquaintance with the developers. That s being said, RTL support for LibreOffice still has problems, which I hope will be pushed during the 3.5.x cycles (full list at Bug 43808, the rtl meta bug). As to get some focus regarding was is to be done, I m listing the top problems:
  1. #44657 RTL UI: Horizontal scrollbar in calc main window is broken
  2. #33302 brackets inverted in rtl text (mac only)
  3. #37692 RTL list numbering reverses its direction
  4. #42070 - RTL support in broken in presenter Console extension
  5. #32531 - Incorrect cursor key movement between table cells of different directionality
  6. #104515 - RTL UI: moving active embedded object to the left moves it to the right (reported for OO.org, but verified in LibO)
  7. #37128 - Writer saves text alignment of RTL paragraph not according to the ODF specification
I hoped to have the first two done for 3.5.0, but didn t succeed in getting them fixed. Will keep trying
Filed under: i18n & l10n, Israeli Community, LibreOffice

23 January 2012

Philipp Kern: Call for testing: Upcoming Squeeze point release 6.0.4

Adam sent a new call for testing for the next point release of Debian Squeeze. Please test the packages in squeeze-proposed-updates on some machines running squeeze if possible, so that we don't screw up your production machines with bad updates in a week. The point release is scheduled for January 28th, i.e. next Saturday. Don't forget to copy the debian-release mailing list when you encounter regressions. Thanks for your efforts.

If you want to receive these notices by mail, please subscribe to the debian-stable-announce mailing list.

22 January 2012

Gregor Herrmann: RC bugs 2012/03

here's the list of RC bugs I've worked on during the last week:

Jon Dowland: The Crying Tree

Somebody bought The Crying Tree by Naseem Rakha for Sarah as a present and she asked me to read it so we could talk about it, so I did. It's a novel about a family tragedy, Southern-state U.S.A. and the death penalty. A little boy is shot dead, a man is arrested for the crime and faces the death penalty. The book chronicles what happens to the family and to the killer during the 19 years or so that passes before the execution takes place. The book could be accused of having an anti-capital punishment agenda, but I have no problem with that. At least until about half-way through the book, the plot and happenings follow the synopsis with little in the way of surprise, but then there's a twist. I felt that the twist diminished the message of the book. Without trying to spoil what happens, I felt that you didn't need the core details of the story to be varied or expanded upon or U-turned to push that agenda. In my opinion, it's perfectly possible for someone to construct a story about a killer who faces death, who was unambiguously guilty of the crime they are charged with, and still leave the reader feeling remorse for the killer, and questioning the justice meted out. In fact I thought Rakha was doing a fine job of just that. However, a twist (or twists) occur, and the facts are called into question, and so you ask even further questions about the practise of killing criminals, in my view redundantly. It's a good book and worth a read, regardless of what your position is on the issue. It does a great job of describing the unbearable pain of losing a child, but also of the power of forgiveness for the victims.

21 January 2012

Lior Kaplan: PHP 5.4 @ Debian

PHP 5.4.0 is around the corner, with RC6 released this weekend. With the courtesy of Ond ej Sur it s already available in experimental. Earlier this week, Raphael Geissert, tested some PHP extensions with the new version and reported 16 bugs (severity: important) against those who failed to build with PHP 5.4. As they will become RC bugs when 5.4.0 will enter unstable (probably around mid February), I preferred to handle them sooner than later. The result is 3 patches sent (one already uploaded) and 5 NMUs done. During the process I poked through a lot of upstream SCMs. In the way I found out a trivial change was done in PECL s SVN two years ago for all the extensions located there, which got me suspicious regarding some extensions we have. For non PECL hosted extensions, I had to track if and when a fix was done and use it for building or do a trivial fix myself. Packages using the 3.0 (quilt) source format were extremely easy to fix, and I was quite happy it generally went fast. In general I m a passive subscriber to the Debian PHP Maintainers mailing list, and finally could help actually and not just reply emails. For me it s also doing some Debian work after a few months of focusing on the coming LibreOffice 3.5 release. update [26/1/2012]: From the 16 issues, only 2 aren t fixed already or will be on the next upstream release. As their upstreams are dead, this is in the hands or the debian people, at least till they ll FTBFS on unstable (instead of experimental).
Filed under: Debian GNU/Linux, PHP

Andrew Pollock: [debian] Bits from the ISC DHCP Maintainer

I really should write these a bit more often. Wow, I can't believe it was over 4 years ago that I started having occasional face to face meetings with the ISC DHCP folks. The entire ISC DHCP team (of 5) was in town for an all-hands meeting, and Larissa Shapiro, the Product Manager for DHCP (and BIND) suggested it would be a good opportunity for another catch up. Given the current (bad) state of DHCP 4.2 in unstable, I thought this was an excellent idea, and so we all had lunch on Tuesday. I pretty much set the agenda, and it was The general state of 4.2.2 in Debian In a nutshell, it's a bit of a mess. We've got release critical bugs, build failures, the whole cat and kaboodle. It makes me very sad, because 4.2.2 was the first 4.2 series that I had a chance to upload, and I was very excited to do so, because it contains the hotly desired LDAP patches merged upstream. Unfortunately, it's also got the beginnings of the BIND/DHCP merger that's going to be BIND 10, and that is all a bit of a mess. It's directly responsible for the kFreeBSD FTBFS and the introduction of the RFCs, which are both keeping 4.2.2 out of testing. I gave the ISC folks a high-level overview of how Debian development works, and the normal progression of packages from unstable to testing to stable, and the release process and whatnot, and impressed upon them the implications of the current release critical bugs. I also showed them how Ubuntu development fitted into the picture. Finally, I showed them the popcon statistics for DHCP. I think they found it useful. FTBFS issues on kFreeBSD This was a good segue to #643569. The issue is actually with the embedded BIND sources. I'd already forwarded this bug upstream when it first happened, but I don't know what had happened to it. They seemed to act as if this was the first they'd heard of it. I'm hoping that they can get this fixed in 4.2.3, which is due around the end of the quarter. Embedded BIND sources Since we were already talking about an issue caused by the embedded BIND sources, we moved on to talking about #645760 and the existence of the embedded BIND sources in general. It should be pretty straightforward for them to strip the RFCs out of the source. They've already done it in the past for the DHCP sources, so I'm also hopeful that this will get resolved in 4.2.3. The issue of the embedded BIND sources is apparently a bit more complicated, although the day before our meeting, Michael Gilbert filed #643569 and #645760, so I hope that the ISC folks can take a look at these patches and see if it's feasible to adopt them. Patches for GNU/Hurd Finally we talked about #616290, which I know is near and dear to the GNU/Hurd porters' hearts. We probably spent the most time talking about this. The DHCP developers have concerns about accepting a patch for an OS that they do absolutely no testing on, and also questioned the viability of the OS in general. They stressed that they're fairly thin in numbers relative to what they have on their plate to achieve this year, and so pushed back pretty firmly on accepting the current patch. I relayed the frustration that the Hurd folks were having about a lack of dialogue around the patch (most of the interaction has been via an ISC support person). There was actually a bit of a split between the developers, with one of them appreciating that the Hurd was unlikely to go anywhere as a platform without a working DHCP client, so in some regards, they were condemning the platform by taking the position they were taking. They're going to go away and take another look at the patch and try to come back with some actionable feedback on what needs to change to make it more acceptable to them, so we'll see what comes of this. I'm not particularly optimistic that anything acceptable to the GNU/Hurd folks is likely to happen any time soon, but maybe if the patch gets cleaned up a bit more, I'll just bite the bullet and start applying it to the Debian package. BIND 10 One of the guys is more involved in BIND 10 than DHCP, and asked if I could help out with the packaging of a build dependency for BIND 10. It seemed like #578387 was languishing so I offered to pick it up. I've not packaged a library before, mainly because the library packaging guide has scared me off it (I feel I lack the deep C fu that seems necessary), but I figured that this would be a good learning opportunity, so I'm going to dive in.

18 January 2012

Matthew Garrett: Why UEFI secure boot is difficult for Linux

I wrote about the technical details of supporting the UEFI secure boot specification with Linux. Despite me pretty clearly saying that this was ignoring issues of licensing and key distribution and the like, people are now using it to claim that Linux could support secure boot with minimal effort. In a sense, they're right. The technical implementation details are fairly straightforward. But they're not the difficult bit.
Secure boot requires that all code that can touch hardware be trustedRight now, if you can run unstrusted code before the OS then you can subvert the OS. Secure boot gives you a mechanism for making sure you only run trusted code, which protects against that. So your UEFI drivers have to be signed, your bootloader has to be signed, and your bootloader must only load a signed kernel. If you've only booted trusted code then you know that your OS is safe. But, unlike trusted boot, secure boot provides no way for you to know that only trusted code was executed. That has to be ensured by OS policy.

This doesn't sound like too much of a problem. But it is. Imagine we have a signed Linux bootloader and a signed Linux kernel, and that these signatures are made with a globally trusted key. These will boot on any hardware using secure boot. Now imagine that an attacker writes a kernel module that sets up a fake UEFI environment, stops the kernel from running code and then executes the Windows bootloader - kind of like kexec, but a little more fiddly. The Windows bootloader believes that it's performing a secure boot, but in fact everything that it believes is trustworthy is the attacker's fake UEFI code. The user's encryption passphrase is logged, the Windows kernel is modified to hide the Linux code and despite everything looking fine your credit card details are on their way to China. In this scenario, the signed Linux kernel is simply used as a malware loader. The only sign that anything is wrong is that boot will be slightly slowed down.

Signing the kernel isn't enough. Signed Linux kernels must refuse to load any unsigned kernel modules. Virtualbox on Linux? Dead. Nvidia binary driver on Linux? Dead. All out of tree kernel modules? Utterly, utterly dead. Building an updated driver locally? Not going to happen. That's going to make some people fairly unhappy.

(The same applies to Windows, of course. Windows 7 allows you to disable driver integrity checks. Windows 8 will have to forbid that when the system's using secure boot)
LicensingGPLv3 has various requirements for signing keys to be available. Microsoft's new requirement that systems support the installation of user keys would let users boot their own modified bootloaders, so that may end up being sufficient to satisfy the license. But we're then beholden on Microsoft - if they remove that requirement then users lose that freedom, and suddenly we're in an awkward licensing situation. There are ongoing conversations about exactly what we're able to do here, but it's not a solved problem.
Key distributionThe UEFI spec doesn't describe or mandate a central certifying authority. Microsoft require that everyone carry their key. We could generate our own, but we have much less sway with vendors. There's no way to guarantee that all hardware vendors will generate our key. And, obviously, if we generate a key, we can't just hand the private half out to others. That means that it becomes impossible for people to produce derivative versions of Linux distributions without getting their own key. The kind of identity verification that would be required for getting such a key is likely to be expensive, and also fairly likely to require that the distribution have a legally registered company in order to facilitate the identity verification. Think Extended Validation certificates, not Startssl Free. Hobbyist Linux distributions will be a thing of the past.
Doesn't custom mode fix this?Microsoft's certification requirements now state that all systems must support a custom mode, implying that it will be possible for a user to install their own keys. Linux vendors would then be able to ship with their own keys on the install media and impose their own policies. Everyone's happy. It's not really good enough, though. People have spent incredible amounts of time and effort making it easy to install Linux by doing little more than putting a CD in a drive. Asking them to go into the firmware and reconfigure things adds an extra barrier that restricts the ability to install Linux to more technically skilled users. And it's even worse than that. This is the full description of the requirement for custom mode:
  1. It shall be possible for a physically present user to use the Custom Mode firmware setup option to modify the contents of the Secure Boot signature databases and the PK.
  2. If the user ends up deleting the PK then, upon exiting the Custom Mode firmware setup, the system will be operating in Setup Mode with Secure Boot turned off.
  3. The firmware setup shall indicate if Secure Boot is turned on, and if it is operated in Standard or Custom Mode. The firmware setup must provide an option to return from Custom to Standard Mode which restores the factory defaults.

There's a few things missing from this, namely:

So no, custom mode doesn't make everything ok. Custom mode with a mandated UI and a documented key format would be much closer, but it wouldn't solve the problem of unattended automated installs.
SummaryWe can write the code required to support secure boot on Linux in a minimal amount of time - in fact, most of it's now done. But significant practical problems remain, and so far we have no workable solutions for any of them.

comment count unavailable comments

16 January 2012

Eddy Petri&#537;or: What's common between Windows 7 and GNOME 3 / gnome-shell?

Update: I managed to make sound work. For some weird reason, a mute switch option of some the many (and who knows how useful) switches of my sound card was enabled. Now the damn thing works. Did I mention that since I did the upgrade all my sound cards (I have a USB sound card, too) have listed as available inputs all the inputs of my internal sound card (mic, front mic, line in, CD, etc.) in Audacity? That makes for a very confusing and loooong sound input sources list! The upside is that I can finally record clips from televisions that do not provide such a feature and FastVideoDownload doesn't handle.

I also seem to have found a possible fix for the caps-ctrl issue in Xfce4 (obviously, setting "-option ctrl:swapcap" in ~/.Xkbmap, instead of that Alt modifier).




As I said in my previous post, I will tell you what do GNOME 3 and Windows 7 have in common.

Before everything else, I want to make it clear that when I am saying GNOME 3, I am referring to Debian Wheezy's GNOME 3, since I recently upgraded from Squeeze on my laptop. I'll probably drop a line or two about that, too.

First, I'll tell you about the (boring, probably for many) experience with Windows 7. As I said before, my new job requires me to use a Windows machine, so up until a few months ago I was using Windows XP with some additional software and tweaks to make it usable. Then came the Windows 7 upgrade . I am using quotes since the more appropriate term would be fresh installation on a new partition , not even close to what Debian users are used to call an upgrade.

So after a fresh Windows 7 installation, my first shock was the fact there was NO Quick Launch*. Some of you might be laughing, but I had never used Windows 7 up until then, just saw it on a laptop of a friend of mine (Ovidiu, one of the guys with whom I am doing this podcast, went to Denkfest with, and made these interviews). That was the first shock. Initial discussions about this with Windows users lead me to believe Quick Launch was dead and for some unexplained reason, I believed them. Later, much later, a week ago, to be precise, I found out that you can bring back the Quick Launch through some convoluted way**. Up until that point I had to have some icons pinned to the task bar, but some others on the desktop (and I hate that) because some of them, like Cygwin, if pinned, would start a cmd console, since Win 7 pins the process, not the starting script.

Among other things which broke in Win 7 and used to work fine in XP, the Virtual Dimension application which provides me with a virtual desktop, was the first one which was broken. I have been using a liniar 4 desktops-wide virtual desktop for over 5 years and I am worthless and inefficient if all my apps are on the same desktop. Mail application is always on the first desktop, work and file managers are on the second, the third is for extras and multimedia editing while the fourth is my gateway to the internet, containing the browser, instant messenger, or whatever.

The shortcuts I use to get to the various desktops are Win+1 ... Win+4 keyboard shortcuts, but the M$ Evil Empire decided that those shortcuts are going to start or bring foreward the first, second and so on applications pinned on the task bar. And you can't change those shortcuts***. Nor is disabling just those possible since they are all disabled through a huge switch which disables ALL Win+x keyboard shortcuts, among which Win+E (file expolrer) and Win+D (Show Desktop) were also. Luckly, Win+L (lock screen) was not disabled. So I disabled al those Win+ shortcuts, since I need virtual desktops.

Now, imagine if I had to start a Cygwin console and I had all sorts of apps open! Win+D was disabled, so I had to minimize the apps covering the desktop shortcut for Cygwin, click on the icon to start it, bring back the minimized windows and go on with my work. What a waste of clicks, mouse movement, energy and time, just because some dudes thought a Quick Launch-like feature was useless****.


You might wonder already what do those '*' sings mean. Well, sadly, that's what GNOME 3 / gnome-shell and Windows 7 have in common.

Gnome 3 was a shock for me. An empty desktop right after upgrade. No panels, no shortcuts*, no power indicators, no wicd indicator, no virtual desktops, no desktop icons, (I have a few dirs and docs there). Sounds like an Evil Empire decision, doesn't it?

Luckly I have been using Tilda as my always-ready console and I could fire up iceweasel from the console in order to understand where my panel disappeared.

I then realised that the upgrade brought me Network Manager, that app which wicd replaced. As a consequence, I had no working wlan since Network Manager made sure to mess up with the network manager I chose.

After looking through the documentation of Network Manager and realising I either had it set up to leave wlan0 alone or I didn't understood NM's documentation, I simply stopped the service, which let Wicd its job flawlessly.

The first thing I searched was Gnome 3 panel or something of that sort and I was confronted with the obvious option to appeal to the Forced Fallback Mode which was disabled. I figured I either had an old version, or Debian disabled this feature (hoping they provided an alternative). There was also the option to conform to this convoluted way of working** with Actions and such uselessness like that. I still wonder, what is the purpose of the Favourites bar on the left side, since it's accessible only after wasting a lot of mouse movement and time? For Joe's Pesci sake, I use focus under mouse just to avoid needless mouse and keyboard manipulation. Why? Why? WHY would I want every time I need to start or SWITCH to another application to move the mouse to the upper-left corner then take my hands off the mouse to type, move the mouse downwards or move across the whole width of the screen to get to my beloved virtual desktops and pick the app I want?

Making a long story short, after even trying XFCE4 (which for some unknown reason resets almost immediately my keyboard layout to the default layout with the Caps on Caps, instead of my preferred and set Ctrl on Caps - yes, it's global), I managed to find GNOME Shell Frippery** which made the experience better.

Later I found out that GNOME 3's file manager, Nautilus, has decided that an up on level button is useless, since the default is to use that uncopy-pastable button location bar instead of a sane text location bar. And it seems the GNOME developers decided this*** and I should conform to it.

To add insult to injury, those icons on my old panel are apparently useless**** and even in the fallback version I can't get them back. Or so the GNOME developers decided.

At some point this sunday, don't know how or why this change happened, producing sound was impossible. I know the problem is pulseaudio since when I kill the pulseaudio daemon from the console I can play audio. BTW, great timing, just when I needed sound the most, before releasing episode 32 of our podcast (yay, I reaslised that xfce just decided to reset my caps to be caps, after setting to ctrl a few minutes ago).

I know I praised pulseaudio when I first tried it, but failing to make it work out of the box or after some tinkering is a deal breaker for me, so I removed it. Now I find it that is a default in GNOME, yet all it manages to do is prevent audio from working. At least on my machine.

Other problems? Gnome Power Manager manages to hang and block my session, GNOME managed somehow to fail to start at some point. Yeah, and that sound problem which I didn't fix yet, didn't went away after removing all the pulseaudio packages which could be removed (e.g.: ryhtmbox depends on libpulse0, same do some other apps like audacity, so I couldn't remove all pulse related packages).

I got involved with Debian and GNU/Linux because it was tweakable and customisable, didn't use to force all sorts of option on me and now I find with its increasing popularity it becomes more and more like a product of a corporation which decides to change some things just to change and totally disregadring user experience and uses.

So, in the light of all of these problems I think it's time to probably consider trying KDE. Is it any good lately?

Yves-Alexis Perez: Advocating people for hardware sponsoring

Our Dear Project Leader, Stefano Zacchiroli, regularly mentions the fact that there's an amount of Debian money available for hardware sponsoring of Debian developers, but it seems that not much people benefit from it. Each time I saw one of this reminder, I wonder if I should apply, and the anser is usually no. The fact is that I don't think any new laptop or desktop to do my Debian stuff, and the last time I bought a box (my x201s last summer) it was not really specifically for Debian tasks so I didn't dare to ask (not to mention the fact I bought it because I did have the money to do so). And I think this is mostly the problem. I might be wrong, but I think that most people which could benefit from this just don't dare asking or don't estimate themselves eligible for it. When I saw Ben Hutchings post, where the first thing he says is about how hardware is expensive, I thought hey, he should get some Debian money for buying new hardware: building kernel is really time consuming and having multiple powerful cores, more ram and fast disks/SSDs really helps . Turns out that Ben just didn't really want to spend too much money there, but the case still stands. We also see from time to time people saying they'll be offline for a while because of broken laptop or something like that. Once again, maybe those people wouldn't mind some help from the Debian project, and maybe they just don't think about asking, or they don't dare. So thinking about it a bit more, I think I wouldn't dare asking money for myself, but maybe I could dare asking money for other people (this is a bit like the flattr posts by Rapha l Hertzog, where he incited people to give money to projects he liked). If I'm not alone in this case, maybe those Debian developers who think some of their peers would benefit some hardware could drop them a mail with leader@ on copy, to propose just that. No need for huge publicity on that (in order to not embarass people), though the transparency rules still apply when it comes to Debian money.
What do you think? It's not really a formal proposal (thus the post on my blog and not a mail to -project), but if that fits you, then nobody prevents you to start yourself. And if you consider it a bad idea, well, nothing forces you to do anything.

Rapha&#235;l Hertzog: Review of my Debian related goals for 2011

Last year I shared my Debian related goals for 2011 . I tend to put more goals than what I can reasonably complete alone and this year was no exception. Let s have a look.
  1. Translate my Debian book into English: PARTLY DONE
    It took more time than expected to prepare and to run the fundraising campaign but it has been successful and the translation is happening right now.
  2. Finish multiarch support in dpkg: DONE BUT NOT ENTIRELY MERGED YET
    Yes, multiarch support was already in the pipe last year in January. I completed the development between January and April (it was sponsored by Linaro) and since then it has mostly been waiting on Guillem to review it, tweak it, and integrate it.
  3. Make deb files use XZ compression by default: TRIED BUT ABANDONED
    After discussing the issue with Colin Watson and Joey Hess during debconf, I came to the conclusion that it was not really desirable at this point. The objections were that debian-installer was not ready for it and that it adds a new dependency on xz for debootstrap to work on non-Debian systems. I believe that the debian-installer side is no longer a problem since unxz is built in busybox-udeb (since version 1:1.19.3-2). For the other side, there s not much to do except ensuring that xz is portable to all the other OS we care about. DAK has been updated too (see #556407).
  4. Be more reactive to review/merge dpkg patches: PARTLY DONE
    I don t think we had any patch that received zero input. We still have a backlog of patches, and the situation is far from ideal but the situation improved.
  5. Implement the rolling distribution proposed as part of the CUT project and try to improve the release process: NOT DONE
    We had a BoF during debconf, we discussed it at length on debian-devel, but in the end we did nothing out of it. Except Josselin Mouette who wrote a proof of concept for his idea. For me testing is already what people are expecting from a rolling distribution. It s just a matter of documenting how to effectively use testing, and of some marketing by defining rolling as alias to testing.
  6. Work more regularly on the developers-reference: PARTLY DONE
    I did contribute some new material to the document but not as much as I could have hoped. On the other hand, I have been rather reactive to ensure that sane patches got merged. We need more people writing new parts and updating the existing content.
  7. Write a 10-lesson course called Smart Package Management : NOT DONE
  8. Create an information product (most likely an ebook or an online training) and sell it on my blog: NOT DONE
    This was supposed to happen after the translation of the Debian Administrator s Handbook. Since the translation is not yet over, I did not start to work on this yet.
  9. By the end of the year, have at least 1/3 of my time funded by donations and/or earnings of my information products: NOT REACHED
    My target was rather aggressive with 700 each month, and given that I did not manage to complete any information product, I m already very pleased to have achieved a mean amount of 204 of donations each month (min: 91 , max: 364 ). It s more than two times better than in 2010. Thank you! Note that those figures do not take into account the revenues of the fundraising of the Debian Administrator s Handbook since they will be used for its translation.
That makes quite a lot of red (for things that I did not achieve) on the other hand I completed projects that I did not foresee and did not plan. For instance improving dpkg-buildflags and then merging Kees Cook work on hardened build flags was an important step for Debian. This was waiting for so long already

2 comments Liked this article? Click here. My blog is Flattr-enabled.

15 January 2012

Roland Mas: Looking for the ultimate distributed filesystem

(This is not quite a Dear lazyweb post. If anything, it's Dear lazyweb, I've done my homework, now what? ) I'm looking for the ultimate distributed filesystem. Something that's simple to use, redundant, fault-tolerant, and still smart enough to avoid the more obvious performance chokepoints. Ideally, it should work for all of the following sets of files: The requirements on the ultimate distributed filesystem (which I'll call UDFS for short, otherwise you'll get bored and go look at pictures of kittens) are as follows: Quite a few constraints, eh? I have a feeling they are not mutually incompatible though, so I had a look at several candidates. I started with Wikipedia, and I followed the links to Tahoe-LAFS, XtreemFS and Ceph. The following is my evaluation of these candidates based on much reading of docs and websites and wikis, some questioning on IRC, and very little testing. The overall picture seems full of good things scattered across different solutions, but unfortunately none of the existing ones seems to address the whole problem; at least, not my whole problem. It would be good if each focused on one layer and did that layer well, but that seems not to be the case either, so they can't be combined to get the best of all worlds. It may be that I'm missing something, or that I failed to read some docs properly, or that I misunderstood the docs, or that the docs themselves are simply lacking; but my ideal UDFS currently doesn't seem to exist as a turn-key solution. However, the main pieces are available, and implementing the remaining parts may be doable. My humble idea of a way forward would be based on Tahoe-LAFS, with the following three changes: Also nice to have would be a way to work with multiple introducer nodes, but that seems to be in the works already. This would be pretty damn close to my UDFS; read/write performance would certainly be far from what can be obtained on native filesystems stored on local disks, but my use cases involve reasonably small files for which instant access is not compulsory, and the filesystem cache would probably absorb most of the access times. In case anyone is looking for ideas of things to do in their spare time, here are rough sketches of other possible UDFS implementations I thought of. These are wild ideas, and I'm not even sure they could be doable in practice: Such is the state of my research so far. I would welcome feedback, pointers to things I neglected to read, corrections for things I misread or misunderstood, comments on the ideas and so on. I'll probably post an update if my search goes significantly forward. Update: I've already received two pieces of feedback, including a lengthy one with corrections about Tahoe-LAFS. For the sake of fairness, I'll solicit (and wait for) the same from the other candidates I looked at. I was also pointed at HekaFS, GlusterFS and git-annex, which I'll have to look at in more details. Other suggestions are still welcome, but the more I get, the more the full update will be delayed. Thanks already!

Gregor Herrmann: RC bugs 2012/02

while working on RC bugs, I noticed that some DDs sent patches to the BTS in the last weeks (yeah!) but did not upload the fixes in NMUs. short request: could you please either state the reason for not uploading, or just go ahead and dput/dupload the fixed packages?

& here's my list:

14 January 2012

Lior Kaplan: Israel to recognize software patents

The Israeli patent registrar have reverted previous ruling regarding patents on software and published a draft for the procedures to accept such patents. The procedures are open to public comments for the next 30 days. Two years ago the patent registrar have started to consider revising the the issue and got many position papers (e.g. Hamakor s papter). At the end he decided not to accept patents on software. De facto, at least one such patent (US patent 7,596,609) was accepted to be valid in Israel after that ruling. Now that the rulings is reverted, I guess will see many more software patents granted here. I wonder about the implications on the local software market hopefully it won t stagnate due to patent wars such as the ones we see every now and then in the US.
Filed under: Software Patents

Next.

Previous.