Ldap seems like a good database to store accounts in: Lots of tools can use it, it's easily integrated into pam, and lots more. But one sometimes just hits borders in the data model of openldap: There are no semantics available. So, let me give you an example. Take e.g. the dnsZoneEntry attribute in the debian.org-ldap. Actually, one wants to allow anyone to write into that attribute in his own entry. But - and that's a big but - one wants to have two additional checks before writing: One is that one cannot claim any dns entry already used by someone else. And the other is that the format needs to comply to specific standards. Now the question is, is there some way to use ldap with more semantics? Or does one need to write ones own backend to do that (and in all the non-plain-db*-backends, it is specified that one needs to write his own authorization code as well). Or some other recommended way to do that? (Or is the only existing incarnation of that ActiveDirectory?) Update: Thanks to Faidon Liambotis I know now about the overlays in openldap. The standard overlays unique and constraint will probably solve the case above - of course, I have a few more complex cases, but that are at least good starters.

I've just finished reading "Don't You Have Time to Think", a collection of letters written to and from Richard P. Feynman. A birthday present from my wishlist. Previously I've read the collection of letters to/from Tolkien. (Several times actually. Very nice collection!) It suddenly struck me that over my lifetime I've probably written <200 letters to people. When I was young I had a couple of pen pals, and when I was entering university I was involved with a couple of play by mail games which involved writing random letters involving strategy & etc. Personal letters though? I've written very few, and I think they've mostly consisted of letters to my partner/partners of the time. (For example Megan went home for a few months at the end of a university year about two months after I initially met her. So there were many letters back and forth. Recently she spent two months working in the USA; counting eggs and avoiding alligators so again there was a flurry of written letters, maybe 20 total during the duration of her trip.) I guess that most of my (hand)written messages to people have been in the form of postcards whilst on holiday. A long time ago I offered to mail postcards to Debian developers. I know I sent at least two, and I received at least one back - but the thing I remember most was exchanging addresses with Amayita and getting into character set issues. Her emails, containing her Spanish address, were difficult to understand as my mutt/console refused to display the foreign character set properly. I can't recall whether she did ultimately receive a card from me, but I'm sure she'll remind me if she did. Anyway I have no resolution, intention, or expectation that I will suddenly start writing more physical mails to people. But I think it almost counts as something we do less of these days. The telephone and internet have become the norm. In some ways this is fantastic. In others it less good. On the left my handwriting is so bad that maybe this isn't necessarily a problem. ObQuote: E.T.

Because my diploma exams are getting nearer, I have to reduce my contribution to the Debian project for the next two months (up to mid/end of May). There is fortunately currently not much to do for me and also no need to NMU my packages. But please read the following notes, if you (still) think there is a reason for a NMU: GCC 4.3 transition Done. All related packages have been fixed. My sponsor just needs to upload psicode to fix the last outstanding bug. PS: Also the build-twice-in-a-row release goal will be reached by the upload of the apbs update by my sponsor. gfortran transition Almost done. mpqc already reached the buildds. ghemical has been uploaded, but waits for mopac7 and libghemical, which are in NEW because of the new library names. apbs and psicode already have been transitioned. Other bugs Ok, I’ve closed almost all relevant bugs in my packages. There are a few outstanding bugs that should be examined for Lenny and can be fixed in NMUs - preferable via delayed NMU queue and an information to the debichem group:

#420795: chemical-mime-data: Unknown media type in type ‘chemical/x-*’

Upstream bug tracker for shared-mime-info contains some more information, how to deal with this. Patches/ideas welcome.

#438694: gabedit: Crashes when loading any XYZ format file

Looks tricky. Maybe an OpenGL issue related to some video card drivers. Not sure.

#442223: openbabel: some connect information lost when convert from pdb to txyz

Nothing examined yet to solve this bug.

#464867: extrema: conflicts with psi3 package

Hope, my team will cover this asap.

#465723: mopac7 — please do not use g2c.

There is currently a workaround using f2c to solve this bug. But MOPAC7 has been written in FORTRAN and we can use gfortran and drop f2c completely. See the report for more information. This is something I would like to see in Lenny.

For the xmlto package it was planned to add the dblatex toolchain for PDF/PS/DVI creation because passivetex will very probably never have a comeback in Debian and docbook-xsl/fop is not fully main and cannot create DVI output. In the case you want to help, feel free to add this feature by NMU to experimental (maybe with version 0.0.21~dblatexX.Y). A first patch is available in bug report #416622. New upstream releases It is possible that there will be a new upstream release for docbook-xsl in the near future. DO NOT consider uploading it by NMU if you did not test it! This upcoming release ships manpage stylesheets, that have been rewritten in large parts. So regressions can be expected. Fortunately Michael Smith and me agreed, that we should test it carefully, before the release is done. Unfortunately I will not have the time to do it. So if you consider uploading it, please get in contact with Michael (xmldoc) and offer your help testing it. He added (and I hope, many Debian packagers and packaging groups will like that) some kind of regressions tests. These tests build the manpages from several Debian packages: e.g. samba, apt, aptitude, git, fglrx and some more. There are further some new supported features, that should be added to the manpage example shipped with the Debian package. I also planned to split the source package (atm, docbook-xsl and docbook-xsl-doc tarballs are merged together for Debian) as soon as docbook-xsl-ns is packaged - which I wanted to do with the next upstream release. So consider all of the above notes if you plan an NMU to upload a new upstream release and in question, mail me and wait for my answer (which will need some days). In all other cases, new upstream releases should be easier to handle. ITPs All my ITPs are delayed. However if you plan to still upload one of these packages, you should consider the following notes:

docbook-xsl-ns

See the above note about docbook-xsl and the source package split.

html-xml-utils

Already in a good shape, but I sent upstream some patches to improve the manpages. Version 4.6 of the html-xml-utils should contain these patches and this version should be uploaded into Debian. With this version, Bert Bos will probably also decide, which prefix will be used for the binaries (they have very generic names atm; the prefix will probably be hx- or hxu-). Waiting for this releaes therefor is IMHO a good choice.

DocBook 5

I already found a problem that need to be addressed in the package: Where to install its catalog? You will see, that there is already a catalog file in /usr/share/xml/docbook/schema/dtd/ shipped by the docbook-xml. Now I’m not sure, what to do here. Upstream ships one catalog file for all supported schemas (RNG, DTD, W3C schema and schematron), so maybe the catalog file can go into /usr/share/xml/docbook/schema/. Really not sure atm. The answer to this question might affect the Debian XML policy and should be made carefully.

QA work/fixes Several lintian warnings have already been fixed in the SVN repositories used for my packages but have not yet been uploaded. I really only plan to reduce my work during the preparation and the exams. So whatever you do, please try to make it easy for me to continue my packaging work after the exams :)

I've been extremely busy with labwork lately, trying to get a second paper ready for submission in the coming months. It's taken an obvious toll on my free software work, but that's life I guess. The upside is that the other night I realized why a year and half's worth of experiments kept turning up with negative results that seemed to contradict our other results. It turns out that this experiment was conceptually flawed, and even though I wasted a year and a half on this I did learn something even from those negative results that should go in to the final paper. I'm actually relieved, because even though I don't have all the data yet I can really see how the paper will end up, so I can start writing soon. Oftentimes knowing why something is wrong so you can fix it is even more important than being right.

I set up an ikiwiki to help organize my lab data a few weeks ago, and it's been a huge win. Joey really did a fantastic job on the program and I can't imagine using a wiki that doesn't share a very similar design for any of my personal stuff. I showed my boss some of my data using it and now he wants me to set up a wiki for the lab. He actually wants me to show off my wiki to give my lab members a feel for the idea. Since ikiwiki doesn't seem to be appropriate for my lab members, I'm going to have to look in to the giant list of wikis to figure out the exact right one for my lab's requirements. This is something I never expected to be doing in grad school.

Because there's no way my brain can go without doing a little bit of programming, I finally sat down and wrote myself a small shell script that lets me grep through pdf's, which is something I couldn't find googling around. If you have a lot of pdf's like I do and don't something like beagle or tracker churning away in the background constantly just so you can search them, this is very helpful. It's obviously very crude, but it gets the job done nicely. It relies on the pdftotext program; hopefully it'll be of use to someone.

I've been digging in to my copy of UNIX Power Tools lately, trying to pick up new UNIX and vi arcana, which is something I haven't made a real serious effort to do in a long time. It's been surprisingly fun to try and replace old habits with things that I know are better but don't use. A good example is that I usually will fire off a new subshell in vim when I should really just suspend vim and use my original shell to save time. Re-examining the basics of how I use the system has been a lot of fun, and has made me a lot more comfortable with the simple day to day tasks.

Before I start on this, note that SPI's next board meeting is next Wednesday at 1900 UTC on irc.oftc.net #spi and logs should appear here and on spi-general less than a day after. I expect the agenda will appear here. Last Tuesday morning, someone asked about SPI holding copyrights. On Tuesday afternoon, I answered with the policy and the only practice example I found. SPI's policy is fine. SPI's practice seemed unclear. On Wednesday morning, I clarified that reply to one board member and made a suggestion to one who seemed to want to ask a lawyer. On Sunday afternoon, after a staggering number of posts which weren't really getting anywhere, I tried to drag it back to the practice example a bit rudely, but that's the level the discussion had fallen to by then. It didn't go down well. It seems rude is fine from other people, but not me. Damn baggage. There was a sequence of posts telling me that SPI members weren't even allowed to ask about this task. I replied to one on Tuesday morning and took the useful bits off to debian vote. Yesterday, I lost my cool, really, and posted too much, replying to all the stupid posts suggesting that you can only ask if you wear the right hat, and one or two particularly ugly personal attacks. Sorry about that. Meanwhile, over on IRC, a draft solution to the problem has been developed/found. So all's well in the end. I give big thanks to Jimmy, Sam and Lucas in particular and a big apology for responding to the mailing list abuse.

As I mentioned in an earlier article, one of the goals for the Android port of Hecl is to make it as easy as it is to create Java ME applications:

1. Write the script.
2. Wrap it up and send it to the phone.

No "compiling" (well, none that's obvious or explicitly asked of the user, at least, as we'll see) or fiddling with build files or writing new Java classes or any of that. We're still not there yet 100%, but I did managed to hack together a system that, from an original Hecl.apk, creates a new application that can be installed on the emulator. This is tricky, because Android looks at the uniqueness of the package and class utilized for the application, meaning you can't just rename Hecl.apk to Foo.apk and install it as a separate app. The emulator will overwrite your old copy of Hecl. Here's what we go through to make this work. Be forewarned - big, ugly hacks that are not for the faint of heart follow. The command line isn't exactly beautiful either, but it beats all the individual steps that it replaces, as we'll see.

java -jar /home/davidw/workshop/hecl/jars/AndroidBuilder.jar -android \
    /opt/android-sdk_m5-rc15_linux-x86/ -class Example -label "Hecl Example" \
    -package my.example -permissions ACCESS_LOCATION,ACCESS_GPS

1. The first thing that needs to be done is to replace the AndroidManifest.xml that is contained within our "template" application, that resides in Hecl.apk, a copy of which is contained within the AndroidBuilder jar file. Since the compiled version of the xml file is a binary format that hasn't been reverse engineered or documented yet, the best way to do this was to simply:
2. Write out a new AndroidManifest.xml in our temporary work directory that uses our provided package name, label and class.
3. Using the aapt command in the SDK, create a new, empty apk file, specifying the newly created xml file as the manifest file to use.
4. Now we can extract the compiled version of the xml file, for later use.
5. At this point, we extract a version of Hecl.jar contained within AndroidBuilder.jar to the temp directory.
6. Now we need to generate some Java classes that correspond to the new package and class names that we've passed to AndroidBuilder. Ideally, I would have done this with some tools to write the byte code out directly, since the classes are so simple: all they do is subclass the Hecl and SubHecl classes so that methods fall through to those.
7. But instead, I just write out the .java code and compile it.
8. The two new classes get stashed inside the copy of Hecl.jar that we're working with.
9. We now use the dx command to compile the entire jar into the classes.dex file that Android uses.
10. Since Hecl.apk is just a zip file, we can replace the AndroidManifest.xml file and classes.dex file with the new files we have created.
11. At this point, we rename the whole works, and move it out of the temporary working directory to the current directory.

Phew! That's a lot of ugly work. It's obvious that a lot of this should and could be handled in a less "brute force" way, but for now it gets the job done. The only thing left to do is replace the Hecl script, which is very easy:

zip -r Example.apk res/raw/script.hcl

This makes it possible to whip out new Android Hecl scripts without carrying around a lot of project infrastructure - all you really need is AndroidBuilder.jar and your script. Naturally, all of this is very much a work in progress, so it may not work as advertised, and, given the mess that we go through, bugs are likely as well. Do report them if you find any, and have fun! In the future, it will be necessary to make it so that it's possible to add resources to the apk file with zip, and have them be available to Hecl applications, which was actually the focus of the previous article.

PGAPack is rather nice and fairly small library for 'parallel' optimisation via generic algorithms using the MPI message passing protocol. PGAPack 1.0 was written by David Levine while doing graduate work during the mid-1990s at Argonne Labs / University of Chicago. PGAPack has also been in or around Debian for a rather long time, but it suffered from benign neglect in the last few years. Some of this came to the fore in this bugreport which lead to my offer to the then-maintainer Andreas to help on the relicencing request. After all, Argonne Labs is just a few miles from where I live, and I had already spent a little bit of time polishing and upgrading the package for my own exploratory use. So I called Rusty Lust, head of the Mathematics and Computer Science section at Argonne to try to sort this out. He was sympathetic and put me in email contact with David Levine. As we are all somewhat busy, this dragged on for a little longer than we thought --- but as of today, about and a half years later, we have a new and shiny PGAPack 1.1 release, or around twelve years after the initial 1.0 version came out. I have done a fair amount of polishing: there are now two library packages for serial use (i.e. for debugging) as well as parallel use via MPI. We use Open MPI where available and LAM where not. All open Debian bugs have been addressed. One minor issue in the postscript documentation remains as David can no longer locate his LaTeX sources; I may just have to extract the text and re-latex this from scratch to update it. One day. Anyway, for full reference, the changelog entry is below. The package is currently in the NEW queue (as the new sub-package require manual inspection and approval) but should hit mirrors in a couple of days. My thanks to the two previous Debian maintainers; to Rusty Lusk for helping with the from the end MSC department at Argonne Labs and for suggesting the rather liberal and easy MPICH2 license (and he happens to be one of the MPICH2 authors); and of course to David Levine for writing PGAPack in the first place, for agreeing to relicense it and giving valuable feedback on my repackaging of what is now version 1.1 on the MCS ftp server at Argonne --- this library has held up really well over the years; let's hope it will find more good use going forward.

pgapack (1.1-1) unstable; urgency=low
  * Really good news:  The MCS divsion of Argonne National Laboratories has
    agreed to relicense pgapack using the MPICH2 license. So pgapack
    is now Free Software and can move into Debian's main archive!
  
    Our thanks go to David Levine and Rusty Lusk to make this possible.
  
  * New maintainer, following Andreas' offer dated 2006-10-04 in #379388
  * debian/control: Change section to math		(Closes: #379388)
  
  * Added new brinary packages libpgapack-mpi1 and libpgapack-serial1
  * The MPI package is configured using Open MPI where available and LAM
    where not. 
  * debian/control: Changed Build-Depends: to use OpenMPI where available, 
    and LAM otherwise.
 
  * Finally acknowledges old NMUs 		(Closes: #379168,#359549)
  * source/integer.c: Apply patch for one-off error 	(Closes: #333381)
  * source/report.c:  Do not unconditionally print at generation 1
  
  * debian/rules: Remove a bashism 			(Closes: #379168)
  * debian/rules: Install examples directly 		(Closes: #134331)
  * debian/control: libpgapack-lam1 Depends on lam4 	(Closes: #60376)
  * debian/rules: Rewritten using debhelper
  * debian/control: Added Build-Depends: section for debhelper
  * No longer install mpi.h in /usr/include		(Closes: #404027)
    
  * debian/control: Updated Standards-Version: to current version
  * man/man1/PGAGetCharacterAllele.1: fix whatis entry 		(lintian)
  
 -- Dirk Eddelbuettel   Mon, 10 Mar 2008 18:03:34 -0500

Quite a lot people are unhappy with how the package descriptions are translated. Different teams handle it differently, but the approach the German "team" chose is quite unfriendly from a quality point of view: The webinterface for it doesn't require any authentication at all, leading technically to anonymous translations all over the place. The so-called "review" process consists of the same not-existing authentication, leading to a situation where unknown people can put in whatever they like and have other (or potentially the same) unknown people acknowledge that. The language team has actively chosen that way because it was said that bad translations simply won't happen and that the review (three people opening the page and clicking onto a button) will not let that happen. Well, it happened. And is happening all over the place. Things like "Gedultsspiel" and "Murmelirrsinn" are pretty tough and almost hiding translations from "counting pipe" to "Zählrohr" and "villages" to "Orte" (and no, those aren't the only examples that accumulated over the last months). As this all happens anonymously one can't even get a message to the people submitting (extremely) low quality translations, helping them to improve their skills so they won't do the same mistake in future translations; meaning things are hard to improve. I am usual an advocate of translating stuff, did put a lot of effort into that area—but the total lack of quality in not only a small and tiny bit here but a much broader area is why I suggest to everyone (at least from a German language point of view): Put APT::Acquire::Translation "none"; into your /etc/apt/apt.conf file and don't get annoyed by them. When quantity is the only thing that counts people wanting to have quality are simply ignored with their mails on the lists.

I have been maintaining several minor sites hosted at Dreamhost for about a year. And since over one month ago, my personal website is with them as well. And I must say, I am very pleased with them. No, not (well, not only) because they run Debian on their servers, nor because they are probably the cheapest game in town (I paid something like US$200 for a basically unlimited package , for three years), but because of their degree of responsability and personal service.
Responsability? Aren't they well-known for their network outages? Why, yes, of course - Today's example is paramount: Somebody edited the wrong firewall entry, and all of Dreamhost became unavailable. In general terms, Dreamhost has a great blog-like structured page where they inform customers of every network or server problem they have - No, you don't have to dig in to understand why your site is down: They bring it up to you. Upfront. And in a familiar, very non-formal style.
Whenever I have submitted an issue to their request tracker, I get prompt reply. Does it always solve the situation? no, by far. I'm often told to, basically, go screw myself if I really need such feature... But they are straightforward with that, they are good, nice BOFHs (if such thing ever existed), and they don't present you with corporate-minded studies backing up their solution. Yes, I know that in their servers, it's plainly their way or the highway. But hey, that's what I paid for, right?
That is what wins my heart. Yes, Dreamhost is no good for many, many tasks - including, for example, anything that requires a real RDBMS (forgodssake, they offer MySQL but not PostgreSQL, damnit! WTF!?), nor any legendary five-nines reliability. But they are great for the vast majority of the Internet sites' needs. They even exceed what a simple person like me would ever dream of.
So, my hat off to you guys. Again.
(No, and I'm not getting paid or discounted on services because of this blog post. Although maybe I should! ;-) )

Since my last example of fixing a bug received some interesting feedback (although I notice no upload of the package in question ..) we'll have another go. Looking over my ~/.bash_history file one command I use multiple times a day is make. Happily GNU make has at least one interesting bug open:

• #419190 - make: double free or corruption

I verified this bug by saving the Makefile in the report and running make:

skx@gold:~$ make
make: file.c:84: lookup_file: Assertion  *name != '\0'' failed.
Aborted

(OK so this isn't a segfault; but an assertion failure is just as bad. Honest!) So I downloaded the source to make, and rebuilt it. This left me with a binary with debugging symbols. The execution was much more interesting this time round:

skx@gold:~$ ./make
*** glibc detected ***
  /home/skx/./make: double free or corruption (fasttop): 0x00000000006327b0 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b273dbdd8a8]
/lib/libc.so.6(cfree+0x76)[0x2b273dbdf9b6]
/home/skx/./make[0x4120a5]
/home/skx/./make[0x4068ee]
/home/skx/./make[0x406fb2]
...
[snip mucho texto]

And once I'd allowed core-file creation ("ulimit -c 9999999") I found I had a core file to help debugging. Running the unstripped version under gdb showed this:

(gdb) up
#5  0x00000000004120a5 in multi_glob (chain=0x1c, size=40) at read.c:3106
3106			    free (memname);

So it seems likely that this free is causing the abort. There are two simple things to do here:

• Comment out the free() call - to see if the crash goes away (!)
• Understand the code to see why this pointer might be causing us pain.

To get started I did the first of these: Commenting out the free() call did indeed fix the problem, or at least mask it (at the cost of a memory leak):

skx@gold:~$ ./make
make: *** No rule to make target  Erreur_Lexicale.o', needed by  compilateur'.  Stop.

So, now we need to go back to read.c and see why that free was causing problems. The function containing the free() is "multi_glob". It has scary pointer magic in it, and it took me a lot of tracing to determine the source of the bug. In short we need to change this:

free (memname);

To this:

free (memname);
memname = 0;

Otherwise the memory is freed multiple times, (once each time through the loop in that function. See the source for details). Patch mailed.

Often if you see a block diagram like this, what comes to mind is a compatability layer in between a program and several operating systems. Generally something that's general-purpose like java, or a web browser, or a widget toolkit.

-------------
             
             
-------------
             
-------------
             
-------------

(Generally it's drawn up all purty, but I'm lame.) But lately I've seen and written a lot of code where the diagram is more complex:

-------------
             
   program   
             
             
-------------
  V  C  S    
-------------
     OS      
-------------

Sometimes the program code is littered with multiple switch statements, as in debcheckout, debcommit, and etckeeper.

case "$vcs" in
git)
svn)
hg)
esac

Sometimes it pushes the VCS-specific code into modules.

use IkiWiki::Rcs::$rcs;
rcs_commit();

But if it does, these modules are specific to that one program. This isn't a general-purpose library. dpkg source v3 doesn't need to use the VCS is the same way as ikiwiki, and even ikiwiki's rcs_commit is very specific to ikiwiki, in its error handling, conflict resolution, locking, etc. pristine-tar injects and extracts data directly from git, using low-level git plumbing, in a way that probably can't be done at all with other VCSes. But even as I was adding that low-level, very git-specific code into pristine-tar, I found myself writing it with future portability in mind.

if ($vcs eq 'git')  
    # git write-tree, git update-index, and other fun
 
else  
    die "unsupported vcs $vcs";
 

When Avery Pennarun talks about git being the next unix, he's talking about programs like these, that build on top of a VCS. But if git is the next unix, then so is mercurial, so is darcs, so is bzr, so too even svn (unless it's Windows?). In other words, we're back to the days when every program had to be ported to a bunch of incompatible and not-quite-compatible operating systems. Back to the unix wars. In Elija's discussion of the "limbo" VCS state he gives several great examples of how multiple VCSes that each seem on the surface to offer similar commands like "$vcs add" and "$vcs commit" can behave very differently.

echo 1 > foo
$vcs add foo
echo 2 > foo
$vcs commit

What was committed, "1" or "2"? Depends on which $vcs you use. Compare with unix where open(2) always opens a file, perhaps with different options, or different handling of large files, but portably enough that you generally don't need to worry about it. Even if you're porting to Windows, you can probably get away with a compatability layer to call whatever baroque monstrosity Windows uses to open a file, and emulate open(2) close enough to not have to worry about it most of the time. A thin compatability layer that calls "$vcs add" isn't very useful for a program that builds on top of multiple VCSes. mr is essentially such a thin compatability layer; it manages to be useful only by being an interface for humans, who can deal with different limbo implementations and other quirks. The VCSes are to some degree converging, but so far it's mostly a surface convergence, with commands that only look the same. Where will things go from here?

• Maybe there will be a standardisation effort like POSIX for VCSes. Though it seems harder; VCSes have a wider interface than just syscalls and filesystems, so there's more scope for incompatibility.
• Or will complicated compatability code like cygwin be developed, to let a program that was written for git use bzr instead, carefully hiding all the differences?
• Another option would be that one VCS wins. So far I'm seeing some consolidation, but little indication that one VCS will emerge as the choice for everyone.
• Maybe the VCSes might begin to support each other's repositories. If a program only supports git, and you use svn, that's fine, if git can transparently access the svn repo.
• Or will we go on for decades spending a lot of time on portability code?

Riku Voipio just sent a "bits from armel porters" in debian-devel-announce (sorry, no link: I'm offline while writing this). I just want to point out that the success of armel becoming an official architecture is also due to the very clever and efficient way the porters interacted with maintainers. As a very small example, Riku recently sent a bug report about samba for armel-related issues and this bug report was full of useful and precise information. Even without Steve Langasek endorsing the fixes, I would have applied the suggested changes even though I don't understand a single bit of it: just because it was so obvious that Riku knew what he was talking about. As Riku points in his announce mail: this is a great succes and this also demonstrate that good interaction between package maintainers in Debian can still happen, despite some general perception that maintainers are "grumpy".

Continuing from where we left off… The lower bound for me becoming a DD was 8th Feb ‘98 when I applied; for comparison, the upper bound as best I can make out was 23rd Feb, when I would have received this mail through the debian-private list:

Resent-Date: 23 Feb 1998 18:18:57 -0000
From: Martin Schulze 
To: Debian Private 
Subject: New accepted maintainers
Hi folks,
I wish you a pleasant beginning of the week.  Here are the first good
news of the week (probably).
This is the weekly progress report about new-maintainers.  These people
have been accepted as new maintainer for Debian GNU/Linux within the
last week.
[...]
Anthony Towns <ajt@debian.org>
    Anthony is going to package the personal proxy from
    distributed.net - we don't have the source... He may adopt the
    transproxy package, too.
Regards,
        Joey

I never did adopt transproxy – apparently Adam Heath started fixing bugs in it a few days later anyway, and it was later taken over by Bernd Eckenfels (ifconfig upstream!) who’s maintained it ever since. Obviously I did do other things instead, which brings us back to where we left off…

---

• Crypto in main: It’s getting hard to remember the days when strong crypto software was classed as munitions and even getting a web browser to do SSL properly was a legal challenge; but for a long while that’s how it was, and as a result we had to keep all our security tools hosted separately to our main, US-based archive. The laws for that changed in early 2000, introducing a new exemption to the prohibition for software whose source code was available royalty-free (I think they called it “public source”). Unfortunately there were still drawbacks: it wouldn’t apply to some stuff in non-free, it wasn’t clear how far you had to go with the source code (is libssl’s source enough, or do you need the source for the apps that use ssl too?), and the biggest problem was you had to notify the export bureau and the NSA for everything you wanted to distribute. And all of that’s couched in the sort of confusing language you’d expect to find in, well, export regulations. So we ended up getting nowhere on merging crypto into the main archive for a long while because we couldn’t be confident of what we actually needed to do. We’d more or less already missed the opportunity to do the merge for the potato (2.2) release because it was frozen more or less exactly when the new regulations came out, and according to my mail archives Ben Collins (DPL at the time) got a team organised to review the legal issues by mid-May 2001. By July 2001, though, we still weren’t really getting anywhere; the lawyer we were talking to was looking at the ENC exception (which was for closed source stuff, and requires an active review by the BXA), instead of the TSU exception (which just requires notification). While it wasn’t terrible legal advice – following the ENC exception had a lower risk of misunderstandings on our part leading to problems later – it wasn’t useful in achieving our aims, ie, getting crypto stuff into main with a minimal level of ongoing effort, as well as a minimal level of risk. So moving onto August, we’d moved on from the lawyer we’d been using for this and other issues, to talking to a lawyer who specialised in export issues (and whose time was being covered by HP). That actually got us somewhere, with the end results being a matter of public record – we got the advice we wanted by November, we got a new incoming system so we could make sure we didn’t publish anything we hadn’t advised the NSA and BXA about, and most of all, we got cryptographic software in main for woody’s release. ssh at standard priority, apt by default checking archive signatures before downloading code to run as root, iceweasel coping with https:// urls out of the box all suddenly became possible, rather than artificially blocked due to pointless legal constraints. Yay! The way we notified BXA/NSA was somewhat interesting – we ended up notifying once for each package, when it passed through NEW: which is easy to automate, and limits the amount of software we block from review by other developers as best we can. OTOH, it also means a lot of notifications: the initial notification was a huge box of paper that got faxed off (there should be pictures somewhere on the net, but I don’t know where), and the vague hope that the ongoing reporting would not only comply with the regs, but manage to be something of a denial-of-service by compliance. As it turns out it apparently got noticed, with the following mail arriving in September 2005:

  From: Encryption 
  Date: Fri, 23 Sep 2005 12:51:02 -0400
  Subject: Addition to Debian Source Code updates
  Dear Mr. Collins,
  Your diligent reporting of updates to the Debian source code to the Bureau
  of Industry and Security has been an excellent example of perspicacity.
  However, you no longer need to do so per the following update to the EAR:
  [...]
  

  (The mails have always been sent with Ben's name attached) That mail was followed by another suggesting a phone conversation; unfortunately nobody seemed to notice the mail, at least until a year later (to the day!) when Joerg pointed out he was getting bounces form the BXA email address we'd been using... (Since then, based on the above mail, we've been recording notifications, but not passing them on; I sent a reply to the mails above indicating that, but didn't receive a response. There's been about 5000 notifications since then)
• Security and the architecture explosion: with crypto-in-main out of the way, along with lots of RC bug squashing, woody was looking good to come out in April or May, but unfortunately that all fell apart because of a lack of automation for handling security updates. Taking a step back, woody managed to introduce five new architectures compared to the previous release -- HP's pets, hppa and ia64; the IBM mainfram arch, s390, and the SGI/embedded mips and mipsel chips. That came just short of doubling our supported architectures (which had been i386, m68k, alpha, arm, sparc and powerpc), and was working really impressively well: testing was watching that they kept in sync, and the buildd network was keeping them all up to date with the latest source uploads, and the toolchain maintainers were making sure new kernels and compilers didn't cause massive breakages. It was great fun to be a part of, and I'm not sure how much is just blurry memories, but from here, it sure feels like it was a golden age on that score... Anyway, all the extra architectures freaked the security team out, given they were already having to worry about ssh'ing to half a dozen machines and manually do builds, and fix them when they broke, and to deal with that there'd been plans afoot for "rbuilder" to get rolled out, which never actually happened. All of which means it escalated from being the security team's problem, to a problem for everyone, particularly the release manager... The result of which meant knuckling down and coming up with a new security build infrastructure, which involved doing a dak install for the security archive, then building on the changes just rolled out for crypto-in-main to allow for packages to be autobuilt before actually being included in the archive, then updating official buildds for all the various architectures to target stable-security and testing-security and prioritise them and upload to the right place. But as stressful and last minute as that was, at least it worked, and woody made it out the door on July 20th 2002 -- and it's a good thing it worked, given woody ended up getting security support until June 30th 2006 -- the usual year after it's successor is released -- for four years support. (The same rule applied to sarge has resulted in two years and ten months' security support; eighteen month release cycles in general should result in two years and six months of support) Since then, with security managed by dak, there's been a few more bits I've been involved in. The most interesting of which (to my mind) was splitting the security upload queue from being completely secret and restricted to folks who are allowed to access updates prior to public announcement, into separate secret and non-secret queues. That was December 2005, and since then we've gone from having Joey Schulze basically being the sole person doing security updates to two separate teams, with ten new members. It's not without it's problems -- particularly for packages that get (mis)classified as NEW, or that get lost on their way for inclusion in the next stable point update, but it's nevertheless lightyears from where we were in April 2002, or even November 2005.
• Playing other people for suckers: with woody released, the whole "redo to release process" thing was essentially done -- it had been implemented, it had worked all the way to getting a release out, and in a fair few respects had improved things. Instead of the eight months of hand-approved uploads for the potato freeze, things had gone it automatically, albeit with gradually stricter constraints, right up until April; and even when the security problems delayed things, the manual updates only lasted a bit under three months. So at that point I was pretty happy, and looking forward to making things work a bit better and basically passing the buck to someone else. The first bit of buck passing that worked exceptionally well was jumping on Colin Watson, who'd been offering some debbugs patches on IRC, and after a delightfully brief discussion with some of the other debbugs folks, got getting him added to the group and able to make changes directly on the site. His advogato post the next day was:

  Yesterday I was invited onto the group that administers the Debian bug tracking system. I can't decide whether to be intimidated or excited. There's a great deal to do: my first couple of major projects look set to be integrating some of the existing QA interfaces to release-critical bug tracking, and developing a tool to help us edit spam out of bug histories.

  There's a lot to be said for grabbing people at the point where they're still intimidated by what they're joining; sure there's a definite risk that they'll break something you'll have to clean up, but that's pretty easy to deal with, and well worth all the extra excitement and energy that comes with a little bit of nerves. I did a talk on debbugs a few years later, at the 2005 DebConf, at which point we managed to entice Pascal Hakim and Don Armstrong into signing up, if I'm remembering right. At the same time Colin rolled out his implementation of version tracking support for debbugs (which I'd specced up a while ago, and helped design, but cannily avoided actually coding). Shortly after, Pasc put together bug subscriptions, and these days Don's pretty much the primary maintainer of debbugs. How cool is that? Passing on release managership was a bit less fortuitous though; it's a big job, and it's not something that you can offer "patches" for, generally. So after a bit of cogitating on just how to do it, I went with an open recruitment call in March 2003, and then a bunch of practical assignments to get the volunteers used to what RM work actually entailed and to see whether they actually like it. The recruitment call was mostly focussed on all the negatives, half to avoid people volunteering to be RM for the power (to decide which release policy wins, to tell maintainers what to do, to drop packages, or to just decide what name/number the next release gets) instead of wanting to actually do the work to get the next release out and keep developers happy; and the other half was the theory that if you make a job look hard, technical, but not terribly risky, it starts looking interesting to the right kind of people. The assignments generally were two parters -- first to introduce some stuff about release management that they mightn't be familiar with, and the second a bunch of problems that needed resolving that'd use those skills. The first set was RC bug fixing, followed by working out why packages aren't getting into testing, followed by dealing with packages with mutual dependencies and removing packages. At some point that evolved into the testing scripts accepting "hints" from the release team as to what packages to try in combination, or which packages to force out of testing. That turned out pretty successful, with Steve Langasek, Colin Watson and Joey Hess becoming official release assistants in August, and Colin and Steve replacing me as RM in mid-2004. It was also pretty satisfying to see Andi Barth do some more release assistant recruiting in 2005 following much the same model, and indeed using much of my mail verbatim. You can't really ask for better than that, I figure.

Geez, this was meant to be briefer...

1. Bzr Exactly 2 years ago I started using bzr. I wanted to start using one of the new distributed VCS for my small projects, and I chose bzr. Not having the time/energy to do an exhaustive comparison among them and try them all to see which one I liked most, I read some articles about them, and decided that bzr would be the most appropriate for me, and that I’d stick to it if I was satisfied enough. (JFTR, at that time it was very important for me the ability to serve branches over plain HTTP, and bzr was the only one in which HTTP was a first-class citizen.) I’ve found bzr easy to use and to learn, powerful enough, customizable enough via plugins, and in general, not requiring a whole lot of attention to get things done. Also, not slow enough as to abandon it. 2. Git Yesterday somebody posted in #bzr a link to this post by Elijah Newren, where one comment said:

There’s one thing I keep telling people though: Learning git is like learning vi — it’s different from the VCS/text editors you know on a fundamental level. But once you’ve overcome that problem, you’ll not want to go back. Ever.

Which pretty much matched the idea I had of git. In the summer after my first year at Uni, I told myself: “OK, let’s learn vim.” And I think it’s one of the decisions that has paid off the most. Like learning to touch type. Reading the above quoted paragraph made more urgent the doubts I had been sporadically having during the past year: maybe I would have been better served by choosing Git instead of Bzr in the first place. What would be my life like if I had stayed with joe? 3. More about Git So yesterday I decided to spend a couple hours (or three) getting the feel of it, trying to find cool stuff. Until now, whenever I had had the need to use git, I just sticked to the bare minimum, without diving into it. I’m going to mention only two examples of the cool stuff I found. And I’ll confirm that, in my opinion, the comparison to vi really holds.

1. commands have lots of options. Tons. And some are incredibly cool, for example git log -S. It allows you to show entries whose diff contains the given string (or pattern, with --pickaxe-regex). I was missing bzr’s log --message, but it was obviously there, as log --grep.
2. Elijah has another entry that talks about what he dubs the “libmo”, and how it’s different between git and the rest, because of the “index is not automatic” schema. In particular, that entry opened my eyes about one very useful use of the index: putting there parts of an upcoming commit that are known to be correct, and that need no further inspection. Doing that with git add -p makes those parts no longer appear in your git diff, which is... whoa. I like to review before committing a lot. However, with other VCSen I always find myself reviewing/glancing at the same parts over and over. So having a way to review what’s pending or WIP is something to die for. And I know even before getting addicted to it!

4. More about Bzr I’m not sure I want to completely switch from bzr to git, not yet anyway. However, I think I’ll try it for some new projects, or migrate some minor ones. Before I can migrate everything I’ll need a bzr frontend for git-fast-import. One of the bzr authors, Ian Clatworthy, has written bzr-fastimport, and he tells me a simple exporter would not be very hard to write, and that he may even put some work into it. Also, I’m (still?) not very sold with some of the git standard practice of throwing away parts of your history regularly. We’ll see about that. Finally, one feature of bzr I would love to see in git is indented formatting for non-mainline revisions in log output (for git, in --topo-order mode only). This is most needed when you’re not rewriting your history for submission to mainline, I’ll reckon. Compare —date-order to —topo-sort to —topo-sort with —indent. Update: For this —indent thing, one could use tig, which after pressing g offers a sort of tree view in a curses-based interfaces. Somebody had mentioned this on IRC, and Peter Baumann kindly mentioned again over e-mail after reading this entry.

I found a few people asking how to automate scripts using ssh with the -f option but not many solutions. Th -f option makes the ssh process fork into the background once started, but since it doesn't give you it's PID it becomes hard to kill it once the script ends. The solution is to setup a control channel with -M. This allows you to communicate to the ssh process and close it down when your process finishes. For example, here is how I open a VNC connection to my desktop at work, which involves going through a corporate ssh firewall box.

I had a call last week from one of our customers that a self-managed server had been attacked and email wasn't running smoothly on it any more. The exploited web app (old version of mambo IIRC) had been repaired and there didn't appear to be a shell-gaining exploit, so what was wrong? Looking at the mail queue gave a clue. I couldn't see anything at first. Nothing. No output. Eventually, qmail-qstat returned and reported over 50,000 emails in the queue, which is pretty high for that webserver. Poking around the mailserver found lots of examples of a few types of spam. I stopped outgoing mail so I could clean up. I don't use qmail much. It hasn't been free software for long and the default upstream packaging (which was being used on that server) isn't very friendly IMO. So, I had no idea how to remove messages from a qmail queue, like exim -Mrm or postsuper -d. I looked at the qmail man pages and the qmail binaries. Nothing. I looked at the qmail FAQ. Nothing. I asked a LUG IRC channel. Something, but no answer. With some web searches, I found mailRemove.py - why is this not in qmail and not even a FAQ? No clear contact details on the FAQ, either.

So, as I previously mentioned I want to be able to tag messages in Mutt. There exist folder-based solutions already, using the X-Label header. There doesn't appear to be any existing solution allowing you to view all messages with a given tag across mailboxes. So I wrote a simple shell script to create virtual mailboxes, such as ~/Maildir/tags-debian for all messages with a debian tag, using hardlinks. My conclusion is that this solution will not work properly in practise, primarily because of deficiencies in mutt. The simple case works just fine. I add a tag to a message, and later when the indexing job runs the virtual folder is created. I can open it and work on it just fine. So where's the problem? Well in my case I tend to tag messages with a label such as "todo". Once I've done whatever I was supposed to I can remove the tag. Using this hardlinking scheme I cannot remove the tag(s) in the virtual folder - I have to remove it in the original message which is a real pain. Why? Well quite simply mutt will not let me work on my virtual message without destroying the hardlink.. If I use the edit function, for example, I am presented with a copy of the mail for editing - and the hardlink is replaced when that copy is saved. Even the edit-label patch which allows you to edit the X-Label header from within mutt ends up replacing the hardlink with a new file! So whats the solution? Well I guess I want to be able to run an external command against a message in mutt - passing the filename of the Maildir message as an argument. That way I can edit the live file. Right now I don't believe that is possible, but I'd love to be told different. If anybody has any solutions of editing, or even just deleting, a header from a message within mutt - in such a way that the hardlink isn't destroyed please do let me know. Simple reproducer:

mkdir -p ~/Maildir/.foo/cur
mkdir    ~/Maildir/.foo/new
mkdir    ~/Maildir/.foo/tmp
cp 'validmessage' ~/Maildir/
ln validmessage ~/Maildir/.foo/cur

Now edit the message - start mutt open the message in the index and press 'e' - the hardlink is now gone. Replaced by a new file with the contents, so the original mail message is unchanged. Update: I've got an "edit-inplace" primitive working, via the very hacky header-fu patch. It is not complete, but it demonstrates that it can be done. My world is now complete.