procps-ng version 3.3.7 was released today. It has some new and interesting features in the top program that Jim has been busy working on. There is a new filter feature which can exclude fields that match a value for example. The remainder of the changes are small bug fixes and getting the compile warnings count down with -Wall enabled. The library revision was updated but this did not involve an API or ABI change. procps-ng can be downloaded off the sourceforge page which has the current and previous releases stored there. Alternatively you can visit our gitorious page if git fetch is more your thing. Debian packages will be going into experimental until the freeze is over and we get things unblocked.

kadmin-remctl is a remctl wrapper around the kadmin protocol that we use at Stanford to provide an easier API to Kerberos administration (for Java applications in particular, but also somewhat for humans) and to use a finer-grained ACL than Kerberos kadmin provides. This package is still kind of a mess, with some Stanford-specific bits and a whole lot of messy Perl code, but we had some serious bugs that I needed to deal with. So this is mostly a bug-fix release without the much-delayed cleanup. (More on that in a moment.) The primary problem this release addresses is a very annoying network problem that we've been having at Stanford. Some interaction between our new OpenFlow fabric and our current firewalls causes some number of TCP connections to just be dropped under heavy load. Since we do a lot of kadmin queries (although we're working on various caching strategies to cut down on that), kadmin sees this a lot, which results in failures that cause user-noticable problems in various applications. After some experimentation, though, we found that most of the failures were on the original connection, so catching connection failure and retrying works around most of the problem (at the cost of a delay). That was only the beginning of the odyssey, however. When I added the retry code, I found that we were improperly configuring Heimdal::Kadm5 to actually throw exceptions. And then, even after fixing that, I found that error messages were still printed to standard error. It turns out that the underlying Heimdal libraries print warnings to standard error by default unless the application directs the log messages somewhere else, and Heimdal::Kadm5 doesn't have a look available to do that. So I ended up adding a bunch of ugly code to close standard error while doing the initial connection. This release also cleans up a bunch of the error handling, including going back to the behavior that the code always should have had: check_passwd returns a non-zero status if the password is rejected. I lost an argument with a co-worker about that years ago, but they've since left and I still think that's the right thing to do. The new code also uses IPC::Run to run commands, which makes for much shorter and saner code, and improves error reporting when trying to change the password of a disabled account with Heimdal's kpasswd. It also strips whitespace from the username in passwd_change. At some point, I'm going to rewrite this software completely, but what that really needs is a much saner underlying Kerberos module. One of the problems with all existing Kerberos Perl modules is that they all use various hacks to try to build with different Kerberos libraries within the ExtUtils::MakeMaker framework, which is totally inadequate. (That's also why there aren't any Perl modules linked with the server version of the kadmin libraries, so they're all painfully slow and require a keytab even when running directly on the KDC.) I think the answer is a brand new Perl module that incorporates my portability layer on top of Kerberos libraries and a configure script that does proper probing for Kerberos library functionality, and then integrates that Autoconf probing into a Module::Build infrastructure. There are some bits on CPAN that one could start with, but not all the proper glue, so this is a bit of a project. I also have to figure out how to link different extensions in the project with different libraries, since having separate distributions for the server and client kadmin modules (which share 95% of their code) is dumb, but they have to be separately linked with the appropriate library. Sometime, when I have some time, I'll do that, and then rebuild a remctl interface on top of that infrastructure with a much better protocol. In the meantime, you can get the latest version of kadmin-remctl from the kadmin-remctl distribution page.

While working on the XLcloud project (HPC on cloud) it appeared clear to us that OpenStack was missing a critical component towards resource reservations. A capacity leasing service is something really needed by service providers, especially in the context of cloud platforms dedicated to HPC style workload. Instead of building something really specific, the decision has been made to build a new standalone OpenStack components aiming to provide this kind of functionnality to OpenStack. In the spirit of others OpenStack components, it will be extensible to fullfil a large panel of needs around this problematic. The project is named Climate, and is hosted on StackForge. It will follow the standard OpenStack development modal. This service will be able to handle a calendar of reservations for various resources, based on various criteria. The project is still at its early design stage, and we plan to have a unconference session during the next OpenStack summit in Portland to present our plans and ideas for the future!

The MIT graphic identity website provides downloadable stationery templates for letterhead and envelopes. They provide both Microsoft Word and LaTeX templates. But although they provide both black and white and color templates for Word, they only provide the monochrome templates for LaTeX. When writing cover letters for the job market this year, I was not particularly interested in compromising on color and was completely unwilling to compromise on TeX. As a result, I ended up modifying each of the three templates to include color. In the process, I fixed a few bugs and documented one tricky issue. I ve published a git repository with my changes. It includes branches for each version of three of the old black and white templates as well as my my three new color templates. I hope others at MIT find it useful. I ve tried to keep the changes minimal. I ve emailed the folks at MIT Communication Production Services to see if they want to publish my modified versions. Until then, anyone interested can help themselves to the git repository. LaTeX user that you are, you probably prefer that anyway.

We were lazy and wrote a simple PostgreSQL monitoring check in shell instead of using some proper language. The code looked about this:

out=$(psql -tAc "SELECT some_stuff, t > now() - '1 day'::interval FROM some_table" some_db 2>&1)
case $out in
    *t) echo "OK: $out" ;;
    *) echo "NOT OK: $out" ;;
esac

If the string ends with 't', all is well, if it ends with 'f' or someting else, something is wrong. Unfortunately, this didn't go that well: OK: psql: FATAL: database "some_db" does not exist

But you can t just tar.gz up the bare repositories on the server and hope for the best. Maybe a given repository will be in a valid state; maybe it won t.

-- Jeff Mitchell in a followup to the recent KDE near git disaster This was a surprising statement to me. I seem to remember that one of (many) selling points for git talked about back in the day was that it avoided the problem that making a simple cp (or backup) of a repository could lead to an inconsistent result. A problem that subversion repositories had, and required annoying commands to work around. (svnadmin $something -- iirc the backend FSFS fixed or avoided most of this issue.) This prompted me to check how I handle it in ikiwiki-hosting. I must have anticipated a problem at some point, since ikisite backup takes care to lock the git repository in a way that prevents eg, incoming pushes while a backup is running. Probably, like the KDE developers, I was simply exercising reasonable caution. The following analysis has probably been written up before (train; limited network availability; can't check), but here are some scenarios to consider:

• A non-bare repository has two parts that can clearly get out of sync during a backup: The work tree and the .git directory.
  • The .git directory will likely be backed up first, since getdirent will typically return it first, since it gets created first . If a change is made to the work tree during that backup, and committed while the work tree is being backed up, the backup won't include that commit -- which is no particular problem and would not be surprising upon restore. Make commit again and get on with life.
  • However, if (part of) the work tree is backed up before .git, then any changes that are committed to git during the backup would not be reflected in the restored work tree, and git diff would show a reversion of those changes. After restore, care would need to be taken to reset the work tree (without losing any legitimate uncommitted changes).
• A non-bare repository can also become broken in other ways if just the wrong state is snapshotted. For example, if a commit is in progress during a backup, .git/index.lock may exist, and prevent future commits from happening, until it's deleted. These problems can also occur if the machine dies at just the right time during a commit. Git tells you how to recover. (git could go further to avoid these problems than it does; for example it could check if .git/index.lock is actually locked using fcntl. Something I do in git-annex to make the .git/annex/index.lock file crash safe.)
• A bare repository could be receiving a push (or a non-bare repository a pull) while the backup occurs. These are fairly similar cases, with the main difference being that a non-bare repository has the reflog, which can be used to recover from some inconsist states that could be backed up. Let's concentrate on pushes to bare repositories.
  • A pack could be in the process of being uploaded during a backup. The KDE developers apparently worried that this could result in a corrupt or inconsistent repository, but TTBOMK it cannot; git transfers the pack to a temp file and atomically renames it into place once the transfer is complete. A backup may include an excess temp file, but this can also happen if the system goes down while a push is in progress. Git cleans these things up.
  • A push first transfers the .git/objects, and then updates .git/refs. A backup might first back up the refs, and then the objects. In this case, it would lose the record that refs were pushed. After being restored, any push from another repository would update the refs, even using the objects that did get backed up. So git recovers from this, and it's not really a concern.
  • Perhaps a backup chooses to first back up the objects, and then the refs. In this case, it could back up a newly changed ref, without having backed up the referenced objects (because they arrived after the backup had finished with the objects). When this happens, your bare repository is inconsistent; you have to somehow hunt down the correct ref for the objects you do have. This is a bad failure mode. git could improve this, perhaps, by maintaining a reflog for bare repositories. (Update: core.logAllRefUpdates can be set to true for bare repositories, but is disabled by default.)
• A "backup" of a git repository can consist of other clones of it. Which do not include .git/hooks/ scripts, .git/config settings, and potentially other valuable information, that strangely, we do not check into revision control despite having this nice revision control system available. This is the most likely failure mode with "git backups". :P

I think that it's important git support naive backups of git repositories as well as possible, because that's probably how most backups of git repositories are made. We don't all have time to carefully tune our backup systems to do something special around our git repositories to ensure we get them in a consistent state like the KDE project did, and as their experience shows, even if we do it, we can easily introduce other, unanticipated problems. Can anyone else think of any other failure modes like these, or find holes in my slightly rushed analysis?

---

PS: git-annex is itself entirely crash-safe, to the best of my abilities, and also safe for naive backups. But inherits any problems with naive backups of git repositories.

So. following up on my previous post about implementing a libravatar server, this will be an announcement of a quick Qt5 based client library for libravatar so that everyone can put avatars everywhere! Get it while it is hot on
git.kde.org There isn t much screenshots to show off, given it is effectively just converting a string with an email into a URL for a image. And thanks to Martin Sandsmark for coming up with such a great name. Now I expect him to show off what it can be used for. Another thanks goes to Andreas Hartmetz for going thru the code & api.

there are not many RC bugs left (yeah!), this also means that my work in this area was rather minimal last week:

• #702486 libnet-twitter-perl: "libnet-twitter-perl: version 4.x required for new Twitter API"
  ack removal from testing and upload new upstream release to unstable (pkg-perl)
• #702523 libnet-twitter-lite-perl: "libnet-twitter-lite-perl: needs update for new Twitter API"
  ack removal from testing, new upstream release already in experimentalunstable (pkg-perl)
• #703094 owncloud: "owncloud: multiple vulnerabilities (oC-SA-2013-009, oC-SA-2013-010)"
  upload package with patches from last week to DELAYED/3
• #703496 license-reconcile: "license-reconcile: missing dependency on libuniversal-require-perl"
  add missing dependency (pkg-perl)

A few days ago, during a discussion in EFN about interesting books to read about copyright and the data retention directive, a suggestion to read the 1968 short story Kod mus by Tore ge Bringsv rd came up. The text was only available in old paper books, and thus not easily available for current and future generations. Some of the people participating in the discussion contacted the author, and reported back 2013-03-19 that the author was OK with releasing the short story using a Creative Commons license. The text was quickly scanned and OCR-ed, and we were ready to start on the editing and typesetting. As I already had some experience formatting text in my project to provide a Norwegian version of the Free Culture book by Lawrence Lessig, I chipped in and set up a DocBook processing framework to generate PDF, HTML and EPUB version of the short story. The tools to transform DocBook to different formats are already in my Linux distribution of choice, Debian, so all I had to do was to use the dblatex, dbtoepub and xmlto tools to do the conversion. After a few days, we decided to replace dblatex with xsltproc/fop (aka docbook-xsl), to get the copyright information to show up in the PDF and to get a nicer <variablelist> typesetting, but that is just a minor technical detail. There were a few challenges, of course. We want to typeset the short story to look like the original, and that require fairly good control over the layout. The original short story have three parts/scenes separated by a single horizontally centred star (*), and the paragraphs do not contain only flowing text, but dialogs and text that started on a new line in the middle of the paragraph. I initially solved the first challenge by using a paragraph with a single star in it, ie <para>*</para>, but it made sure a placeholder indicated where the scene shifted. This did not look too good without the centring. The next approach was to create a new preprocessor directive <?newscene?>, mapping to "<hr/>" for HTML and "<fo:block text-align="center"><fo:leader leader-pattern="rule" rule-thickness="0.5pt"/></fo:block>" for FO/PDF output (did not try to implement this in dblatex, as we had switched at this time). The HTML XSL file looked like this:

<?xml version='1.0'?> 
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version='1.0'>
  <xsl:template match="processing-instruction('newscene')">
    <hr/>
  </xsl:template>
</xsl:stylesheet> 

And the FO/PDF XSL file looked like this:

<?xml version='1.0'?> 
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version='1.0'>
  <xsl:template match="processing-instruction('newscene')">
    <fo:block text-align="center">
      <fo:leader leader-pattern="rule" rule-thickness="0.5pt"/>
    </fo:block>
  </xsl:template>
</xsl:stylesheet> 

Finally, I came across the <bridgehead> tag, which seem to be a good fit for the task at hand, and I replaced <?newscene?> with <bridgehead>*</bridgehead>. It isn't centred, but we can fix it with some XSL rule if the current visual layout isn't enough. I did not find a good DocBook compliant way to solve the linebreak/paragraph challenge, so I ended up creating a new processor directive <?linebreak?>, mapping to <br/> in HTML, and <fo:block/> in FO/PDF. I suspect there are better ways to do this, and welcome ideas and patches on github. The HTML XSL file now look like this:

<?xml version='1.0'?> 
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version='1.0'>
  <xsl:template match="processing-instruction('linebreak)">
    <br/>
  </xsl:template>
</xsl:stylesheet> 

And the FO/PDF XSL file looked like this:

<?xml version='1.0'?> 
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version='1.0'
  xmlns:fo="http://www.w3.org/1999/XSL/Format">
  <xsl:template match="processing-instruction('linebreak)">
    <fo:block/>
  </xsl:template>
</xsl:stylesheet> 

One unsolved challenge is our wish to expose different ISBN numbers per publication format, while keeping all of them in some conditional structure in the DocBook source. No idea how to do this, so we ended up listing all the ISBN numbers next to their format in the colophon page. If you want to check out the finished result, check out the source repository at github (future/new/official repository). We expect it to be ready and announced in a few days.

I ve been using this process for debugging the installer while launched from a Debian Live desktop. Having a full live environment at your disposal during install is a boon to debugging because you don t have to work within the constraints normally imposed upon you by the limited debian-installer environment. Caveat: running debian-installer from the debian-installer-launcher still has one or two unresolved issues, so unless you re specifically working on live integration issues like I am, compare with a conventional d-i image install to gain the added confidence the install is the same in both environments. There are no longer active maintainers working on debian-installer-launcher and live-installer, and I think it s really important for wheezy to release with solid support for live installs, so I ve been using this setup to try to solve #702335. So far, progress is slow because I m new to this. If anyone is interested in collaborating in this work, give me a shout. I am SynrG @ irc.oftc.net and can be found on #debian-live and #debian-boot if you would like to drop in. Overview:

1. Use images/lxde-desktop config from live-images.git and build your own image with live-build >=3.0.1-1. Just add -b hdd in auto/config before starting your build, as having a read/writable image is handy for this exercise.
2. Boot the live medium. You may prefer to do this in a VM for convenience, having prepared a blank virtual disk image as your target in advance.
3. Make any small changes you want to the installer before launching it. A quick hack I ve used is to sudo vi /usr/sbin/debian-installer-launcher and near the end on the line after prepare right before run , open a new line and add bash so an interactive shell will be opened after the installer is extracted but before it is launched. For larger changes (e.g. inclusion of updated udebs, etc.) add them to your live image configuration before building as described in live-manual.
4. Open two root terminals and do debian-installer-launcher -t debug in one and tail -f /lib/live/installer/var/log/syslog in the other. If you ve added the hack from step 3, don t forget to exit the interactive shell after making changes. Now you can watch debug output as you go through the steps of the installer. If you like, you may append additional boot prompt parameters (e.g. preseeds) to the debian-installer-launcher command.
5. During the install, you may modify other parts of the unpacked installer under /lib/live/installer (e.g. to add set -x to some scripts) prior to executing steps that would call them.

So via hackernews I recently learned about fight code, and my efforts have been fun. Currently my little robot is ranked ~400, but it seems to jump around a fair bit. Otherwise I've done little coding recently:

• slaughter has had its code moved to github.
• The bookmark code got one patch and has had some interest

I'm pondering libpcap a little, for work purposes. There is a plan to write a deamon which will count incoming SYN packets, per-IP, and drop access to clients that make "too many" requests "too quickly". This plan is a simple anti-DoS tool which might or might not work in the real world. We do have a couple of clients that seem to be DoS magnets and this is better than using grep + sort against apache access logs. For cases where a small number of source IPs make many-many requests it will help. For the class of attacks where a huge botnet has members making only a couple of requests each it won't do anything useful. We'll see how it turns out.

The 2nd Shogi Denousen has began. This is the first cut-throat match between top-notch professional Shogi players(yes, there are such people) and the best crop of computer Shogi engines, 5 on 5. Shogi is a distant cousin of Chess. Unlike Chess, you may re-use captured pieces anywhere on the board anytime. It brings quite lot of additional complexities, and even after the defeat of Garry Kasparov in 1997, many considered that (at least top-level) human Shogi players have a great lead on computer Shogi engines. The situation had changed dramatically when a newcomer Shogi engine called Bonanza, developed by a Japanese chemist Kunihito Hoki, won the 16th World Computer Shogi Championship. Bonanza appeared totally out of the blue Hoki incorporated some new ideas developed in the field of computer Chess, and Bonanza beated existing engines with no mercy. Bonanza could even corner some professional Shogi players in 2007. Later Hoki released(but not strictly open-source) the source code of Bonanza, and the standard of computer Shogi has risen considerably since then. Finally, in the 1st Denousen last year, Kunio Yonenaga, long retired but possibly one of the greatest Shogi players in the history, was defeated by Bonkras, a clustered version of Bonanza developed by Eiki Itoh of Fujitsu. The first match of Denousen this year was held yesterday between Kouru Abe, an 18-year old prodigy from Aomori, and Shueso, which finished in 5th at the 22nd World Computer Shogi Championship. I hoped a close game, but Abe could beat Shueso quite easily. Shueso somehow could not bring its ability into full play, to my great disappointment. Next weekend(Mar. 30), we will see the second match between Shinichi Sato, another young pro, and Ponanza, developed based on Bonanza by Issei Yamamoto of The University of Tokyo. BTW, Debian already has the package of GPS Shogi, which won the 22nd World Computer Shogi Championship and considered by many the strongest Shogi engine available now (there is also gnushogi in Debian, but gnushogi is quite weak). You may have fun with

    $ xshogi -fsp gpsshogi

Unfortunately, we don t have good modern GUI for Shogi yet

A new relase 0.10.3 of Rcpp is now on CRAN and in Debian. This is the fourth release in the 0.10.* series, and further extends and solidifies the excellent Rcpp attributes. A few other bugs were fixed as well, and support for wide character strings has been added. We once again tested this fairly rigorously by checking against 86 of the 100 CRAN packages depending on Rcpp. All of these passed. So we do not expect any issues with dependent packages, but one never knows. The complete NEWS entry for 0.10.3 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.10.3 (2013-03-23)

• Changes in R code:
  • Prevent build failures on Windowsn when Rcpp is installed in a library path with spaces (transform paths in the same manner that R does before passing them to the build system).
• Changes in Rcpp attributes:
  • Rcpp modules can now be used with sourceCpp
  • Standalone roxygen chunks (e.g. to document a class) are now transposed into RcppExports.R
  • Added Rcpp::plugins attribute for binding directly to inline plugins. Plugins can be registered using the new registerPlugin function.
  • Added built-in cpp11 plugin for specifying the use of C++11 in a translation unit
  • Merge existing values of build related environment variables for sourceCpp
  • Add global package include file to RcppExports.cpp if it exists
  • Stop with an error if the file name passed to sourceCpp has spaces in it
  • Return invisibly from void functions
  • Ensure that line comments invalidate block comments when parsing for attributes
  • Eliminated spurious empty hello world function definition in Rcpp.package.skeleton
• Changes in Rcpp API:
  • The very central use of R API R_PreserveObject and R_ReleaseObject has been replaced by a new system based on the functions Rcpp_PreserveObject, Rcpp_ReleaseObject and Rcpp_ReplaceObject which shows better performance and is implemented using a generic vector treated as a stack instead of a pairlist in the R implementation. However, as this preserve / release code is still a little rough at the edges, a new #define is used (in config.h) to disable it for now.
  • Platform-dependent code in Timer.cpp now recognises a few more BSD variants thanks to contributed defined() test suggestions
  • Support for wide character strings has been added throughout the API. In particular String, CharacterVector, wrap and as are aware of wide character strings

Thanks to CRANberries, you can also look at a diff to the previous release 0.10.2. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Hello To check the validity of Debian dependencies in Debian package, Config::Model queries a remote web server to get the list of package version known to Debian. The first version of this check did sequential requests: when the cache was not very fresh, I had to wait more than 60s before getting results for complex package like padre. That was very frustrating (but less frustrating than checking package names manually) For the following version, I addedhacked AnyEvent in Dpkg model to run parrallel queries. This went much faster but gaves weird results: before getting a response from Debian server, the packages were flagged as unknown. To get consistent results, running twice cme check dpkg was required. So, at new year, I ve decided to bite the bullet and implement correctly value checks with asynchronous queries to remote server. This new feature is now ready and will be delivered in Config::Model 2.030. cme will now return consistent results. This new release is mostly backward compatible. You may notice some quirks with some other modules based on Config::Model:

• Current Config::Model::Dpkg will returns unconsistent results as usual
• Tk UI will not show correct package status (i.e. packages may be wrongly shown as unknown)
• config-model-edit process must be killed if it refuses to exit.
• Tk UI non-regression tests will block

These quirks will disappear once these modules are updated. This should not be long since all the updates are ready in github. All the best
Tagged: Config::Model, debian, dpkg, Perl

Am I the only one who logically inferred this from the Ubuntu for Android's tagline "One device to rule them all"?

Following up on using Debian on the WeTab with GNOME Shell I figured accelerated video using the built in CrystalHD to save battery power and CPU cycles would be nice to have. There's even a Debian package available but not for gstreamer 1.0 (which is used by GNOME 3.6) so I had a look at gstreamer's plugin writers guide and porting guide and updated the driver. Packages are here until there's a new upstream version or Debian package release. This blog is flattr enabled.

Let's see if we will have 82 open bugs next week ;) Current trend points towards week 16/17. The UDD bugs interface currently knows about the following release critical bugs:

• In Total: 718
  • Affecting wheezy: 93 That's the number we need to get down to zero before the release. They can be split in two big categories:
    • Affecting wheezy and unstable: 47 Those need someone to find a fix, or to finish the work to upload a fix to unstable:
      • 16 bugs are tagged 'patch'. Please help by reviewing the patches, and (if you are a DD) by uploading them.
      • 9 bugs are marked as done, but still affect unstable. This can happen due to missing builds on some architectures, for example. Help investigate!
      • 27 bugs are neither tagged patch, nor marked done. Help make a first step towards resolution!
    • Affecting wheezy only: 46 Those are already fixed in unstable, but the fix still needs to migrate to wheezy. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
      • 17 bugs are in packages that are unblocked by the release team.
      • 29 bugs are in packages that are not unblocked.

How do we compare to the Squeeze release cycle?

Week	Squeeze	Wheezy	Diff
43	284 (213+71)	468 (332+136)	+184 (+119/+65)
44	261 (201+60)	408 (265+143)	+147 (+64/+83)
45	261 (205+56)	425 (291+134)	+164 (+86/+78)
46	271 (200+71)	401 (258+143)	+130 (+58/+72)
47	283 (209+74)	366 (221+145)	+83 (+12/+71)
48	256 (177+79)	378 (230+148)	+122 (+53/+69)
49	256 (180+76)	360 (216+155)	+104 (+36/+79)
50	204 (148+56)	339 (195+144)	+135 (+47/+90)
51	178 (124+54)	323 (190+133)	+145 (+66/+79)
52	115 (78+37)	289 (190+99)	+174 (+112/+62)
1	93 (60+33)	287 (171+116)	+194 (+111/+83)
2	82 (46+36)	271 (162+109)	+189 (+116/+73)
3	25 (15+10)	249 (165+84)	+224 (+150/+74)
4	14 (8+6)	244 (176+68)	+230 (+168/+62)
5	2 (0+2)	224 (132+92)	+222 (+132/+90)
6	release!	212 (129+83)	+212 (+129/+83)
7	release+1	194 (128+66)	+194 (+128/+66)
8	release+2	206 (144+62)	+206 (+144/+62)
9	release+3	174 (105+69)	+174 (+105/+69)
10	release+4	120 (72+48)	+120 (+72/+48)
11	release+5	115 (74+41)	+115 (+74/+41)
12	release+6	93 (47+46)	+93 (+47/+46)
13	release+7
14	release+8
15	release+9
16	release+10
17	release+11
18	release+12

Graphical overview of bug stats thanks to azhag:

Update: This isn't actually that much better than letting them access the private key, since nothing is stopping the user from running their own SSH agent, which can be run under strace. A better solution is in the works. Thanks Timo Juhani Lindfors and Bob Proulx for both pointing this out. At work, we have a shared SSH key between the different people manning the support queue. So far, this has just been a file in a directory where everybody could read it and people would sudo to the support user and then run SSH. This has bugged me a fair bit, since there was nothing stopping a person from making a copy of the key onto their laptop, except policy. Thanks to a tip, I got around to implementing this and figured writing up how to do it would be useful. First, you need a directory readable by root only, I use /var/local/support-ssh here. The other bits you need are a small sudo snippet and a profile.d script. My sudo snippet looks like:

Defaults!/usr/bin/ssh-add env_keep += "SSH_AUTH_SOCK"
%support ALL=(root)  NOPASSWD: /usr/bin/ssh-add /var/local/support-ssh/id_rsa

Everybody in group support can run ssh-add as root. The profile.d goes in /etc/profile.d/support.sh and looks like:

if [ -n "$(groups   grep -E "(^  )support(  $)")" ]; then
    export SSH_AUTH_ENV="$HOME/.ssh/agent-env"
    if [ -f "$SSH_AUTH_ENV" ]; then
        . "$SSH_AUTH_ENV"
    fi
    ssh-add -l >/dev/null 2>&1
    if [ $? = 2 ]; then
        mkdir -p "$HOME/.ssh"
        rm -f "$SSH_AUTH_ENV"
        ssh-agent > "$SSH_AUTH_ENV"
        . "$SSH_AUTH_ENV"
    fi
    sudo ssh-add /var/local/support-ssh/id_rsa
fi

The key is unavailable for the user in question because ssh-add is sgid and so runs with group ssh and the process is only debuggable for root. The only thing missing is there's no way to have the agent prompt to use a key and I would like it to die or at least unload keys when the last session for a user is closed, but that doesn't seem trivial to do.