I've been using GMail ever since they decided to allow users to download their mail using POP. I was happy reading my email as usual in mutt. The only problem was that I had to log onto the website once in a while to clear out my spam folder and fish out the false positives.

However, as the amount of spam I receive daily started growing exponentially (in part thanks to my involvement in Debian), I became more and more dissatisfied with this solution. I had to login several times a day just to keep the levels of spam down to a manageable level. I tried emailing the GMail support team a few times without success.

The solution came just as I was looking at switching to a different email provider: IMAP access.

Using fetchmail, I can now download my received and sent mail using POP and then download (and purge) the contents of my spam folder. Here are the relevant lines of my ~/.fetchmailrc:

poll pop.gmail.com protocol pop3 user "fmarier@gmail.com" is "francois" password "" fetchall ssl
poll imap.gmail.com protocol imap user "fmarier@gmail.com" is "francois" password "" folder "[Gmail]/Spam" fetchall ssl

I can then run all of my mail through my own SpamAssassin filter (to which I added these excellent custom rules) and sort my mail in one of three folders:

• definitely spam (conveniently located in /dev/null) for a score greater than 12
• probably spam (I need to check those manually once in a while)
• not spam for messages with a score less than 5

This is done using procmail with the following bit in my ~/.procmailrc:

# Use spamassassin to check for spam
# (only check messages 400kB so that it doesn't choke) :0fw:/etc/mail/spamc.lock
* 409600
/usr/bin/spamc

# Throw away messages with a score of > 12.0
:0:
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*
/dev/null

:0:
* ^X-Spam-Status: Yes
/home/francois/mail/spam

In this post, Steven Hanley wonders how to keep /var/cache/apt/archives empty on a machine that has a full mirror of debian on it. That’s actually the wrong question. The right question is ‘Why is /var/cache/apt/archives even relevant on a machine with a full mirror of debian?’ /etc/apt/sources.list has always supported file:/ URLs. for example, sources.list on my local debian mirror looks like this:

deb file:/home/ftp/debian unstable main contrib non-free
deb file:/home/ftp/debian-multimedia/ unstable main

With a file URL, apt-get doesn’t need to download any packages, and thus doesn’t need to cache anything in /var/cache/apt/archives. No need to keep it empty, it just is empty. In a related post, Andrew Pollock suggests adding DPkg::Post-Invoke “apt-get clean”; ; to /etc/apt/apt.conf. That would be fine if you ONLY ever ran “apt-get update ; apt-get dist-upgrade”. It’s not fine if you run “apt-get -d” to download packages first, then install a few packages with “apt-get install package”, and then run “apt-get dist-upgrade”. If you try that, then the DPKG::Post-Invoke rule will clean out /var/cache/apt/archives when you install the individual package(s). When you then run the dist-upgrade, apt-get will have to download all the packages again. Better just to get into the habit of running apt-get clean or autoclean occasionally when you want to reclaim some disk space. Syndicated from Craig Sanders' Errata: Miscellaneous Thoughts Keeping apt archives empty…or not.

It seems existing versions of gcc don't warn when you use an assignment as a truth value as the first operand to the conditional operator. For example: Presumably you meant ==, but unfortunately that does not warn with gcc (maybe it will one day). We can use the tree dumping mechanisms (previously mentioned here) to confirm our problem. For the following function gcc creates: As you can see, this isn't the result we wanted. This can be quite nasty if you use the conditional operator in a hidden fashion; for example behind an assert. A common idiom is: You'll currently get no warning you've slipped up and used the assert wrong. Moral of the story: a quick audit of your code might turn up some surprising uses! However, in this case, the Intel compiler picks this one up: Although it's often not trivial to move to another compiler, as a rule I would recommend trying alternative compilers for your code as it's a great sanity check.

Till today I thought make was a very simple concept, but looking at other people's debian/rules files I start to lose that faith. So let's begin with some basics (as I guess many reading this are maintainers for Debian packages, and you might need some of this knowledge): As you will already know, the most important part of a Makefile is a rule. Each rule is there to produce something and has prerequisites, i.e. things that have to be done before. So far so simple. When you think about this way, the first pitfall is already no pitfall anymore:

   build: patch build-indep
   build-indep: build-indep-stamp
   build-indep-stamp:
   	$(MAKE) doc
   	touch build-indep-stamp
   

There are two mistakes in this. First of all, the clean is only called on ./debian/rules build, but not on ./debian/rules build-indep. And then patch is only called by the build target. But what you really want is that the source is patched before you do something, so it is something do be done before build-indep-stamp. The pitfall with this error is that you will not see it most of the time. As make usually processes targets in the order it finds them, it usually runs patch first. Except when you have multiple processors and tell make to make sure of them (and even then there is a chance it might work as the command to run first is fast enough) or if someone trusting on what he learned about make does call ./debian/rules build-indep-stamp build. I guess a reason one sees this so often is the next pitfall. Most likely previous the following was tried:

   build-indep: build-indep-stamp
   build-indep-stamp: patch
   	$(MAKE) doc
   	touch build-indep-stamp
   patch: patch-stamp
   patch-stamp:
   	whatever
   	touch patch-stamp
   

The pitfall with this is some new concept. The problem is that the patch rule is phony. A rule is called phony when it does not produce the target it claims to do. The classical example is a clean target. You do not want that a clean target creates a file clean and the next time it is called says "already everything done". Targets get phony by telling make it is (via .PHONY:) or by just not producing the file it tells to produce or (suprising to many, it seems) by having a phony target as prerequisite. In the above example the patch target gets implicitly phony, as it does not produce a file called patch. Thus after having build the source and calling binary to create packages, this will most likely in some way depend on build-indep-stamp. But when make thus looks at build-indep-stamp to decide if it already there and even though it sees the file produces there with the touch command, it cannot determine if that is up to date. It depends on patch and there is no file called this, thus make must assume it is not up to date, thus build-indep-stamp has become effectively also phony, in the sense of having to be called every time it is depended on. (In case you have not noticed, the fix would have been to make build-indep-stamp depend on patch-stamp instead or to go without patch-stamp and make patch the file to be generated). Thus, you can put as many -stamp files as you want in a row, as long as you have a single phony prerequisite in them, it is all void.

Brian Aker dropped by and replied to my previous essay by making the following comment:

I believe you are hitting the nail on the organic vs nonorganic open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution but I don t believe either are really optimized at this point. Optimization to me would lead to a system of less priests and more inclusion.

I made an initial reply as comment, and then decided it was so long that I should promote it to a top-level post. I assume that when Brian talks about “organic open source” what he means is what I was calling an “open source development community”. Some googling turned up the following definition from Mozilla Firefox’s organic software page: “Our most well-known product, Firefox, is created by an international movement of thousands, only a small percentage of whom are actual employees.” This puts it in contrast with “non-organic” software, where all or nearly all of the developers are employed by one company. (And anyone who proves talented at adding features to that source base soon gets a job offer by that one company. By that definition we can certainly see projects like Wine, Mysql, Ghostscript (at one time), and others as fitting into that model, and being quite successful. There’s nothing really wrong with the non-organic software model, although many of them have struggled to make enough money when competing with pure proprietary softare competitors, with MySQL perhaps being the exception which proves the rule. In most of these cases, though, the project started more as an organic open source, and then transitioned into the non-organic model when there was a desire to monetize the project and/or when the open source programmers decided that it would be nice if they could turn their avocation into a vocation, and let their hobby put food on the family table. Solaris, of course, is doing something else quite different, though. They are trying to make the transition from a proprietary customer/supplier relationship to trying to develop an Open Source community and what John s candidate statement pointed out is that they weren t really interested in creating an organic open source developer community at all, but they wanted the fruits of an open source community with plenty of application developers, end-users, etc., all participating in that community. We don t have a lot of precedent for projects who try to go in this direction, but I suspect they are skipping a step when they try to go to the end step without bothering to try to make themselves open to outside developers. And by continuing to act like a corporation, they end up shooting themselves in the foot. For example, the OpenSolaris license still prohibits people from publishing benchmarks or comparisons with other operating systems. Very common in closed-source operating systems and databases, but it discourages people from even trying to make things better, both within and outside of the Open Solaris core team. Instead, they respond to posts like David Miller s with Have you ever kissed a girl? . (Thanks, Simon, for that quote; I had seen it before, but not for a while, and it pretty well sums up the sheer arrogance of the Open Solaris development team.) So while Linux may not be completely optimized in terms of less priests and more inclusion, at least over 1200 developers contributed to 2.6.25 during its development cycle. Compared to that, Open Solaris is positively dominated by high priests and with a you may not touch the holy-of-holies attitude; heck, they won t even allow you to compare them to other religions without branding you a heretic and suing you for licensing violations!

Brian Aker dropped by and replied to my previous essay by making the following comment:

I believe you are hitting the nail on the organic vs nonorganic open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution but I don t believe either are really optimized at this point. Optimization to me would lead to a system of less priests and more inclusion.

I made an initial reply as comment, and then decided it was so long that I should promote it to a top-level post. I assume that when Brian talks about “organic open source” what he means is what I was calling an “open source development community”. Some googling turned up the following definition from Mozilla Firefox’s organic software page: “Our most well-known product, Firefox, is created by an international movement of thousands, only a small percentage of whom are actual employees.” This puts it in contrast with “non-organic” software, where all or nearly all of the developers are employed by one company. (And anyone who proves talented at adding features to that source base soon gets a job offer by that one company. By that definition we can certainly see projects like Wine, Mysql, Ghostscript (at one time), and others as fitting into that model, and being quite successful. There’s nothing really wrong with the non-organic software model, although many of them have struggled to make enough money when competing with pure proprietary softare competitors, with MySQL perhaps being the exception which proves the rule. In most of these cases, though, the project started more as an organic open source, and then transitioned into the non-organic model when there was a desire to monetize the project and/or when the open source programmers decided that it would be nice if they could turn their avocation into a vocation, and let their hobby put food on the family table. Solaris, of course, is doing something else quite different, though. They are trying to make the transition from a proprietary customer/supplier relationship to trying to develop an Open Source community and what John s candidate statement pointed out is that they weren t really interested in creating an organic open source developer community at all, but they wanted the fruits of an open source community with plenty of application developers, end-users, etc., all participating in that community. We don t have a lot of precedent for projects who try to go in this direction, but I suspect they are skipping a step when they try to go to the end step without bothering to try to make themselves open to outside developers. And by continuing to act like a corporation, they end up shooting themselves in the foot. For example, the OpenSolaris license still prohibits people from publishing benchmarks or comparisons with other operating systems. Very common in closed-source operating systems and databases, but it discourages people from even trying to make things better, both within and outside of the Open Solaris core team. Instead, they respond to posts like David Miller s with Have you ever kissed a girl? . (Thanks, Simon, for that quote; I had seen it before, but not for a while, and it pretty well sums up the sheer arrogance of the Open Solaris development team.) So while Linux may not be completely optimized in terms of less priests and more inclusion, at least over 1200 developers contributed to 2.6.25 during its development cycle. Compared to that, Open Solaris is positively dominated by high priests and with a you may not touch the holy-of-holies attitude; heck, they won t even allow you to compare them to other religions without branding you a heretic and suing you for licensing violations!

I've implemented last night's design, in debhelper 7.0.0. Currently fully supported are packages that use the autotools, or perl's MakeMaker, or python setup.py, or a simple Makefile. The other classes of packages will be added as bugs are filed about them. I've uploaded a few of my own packages converted to use debhelper 7 with minimised rules files, so there are a range of examples in incoming:

• mr uses a very simple makefile, and a 3 line debian/rules
• aalib uses automake and autoconf, and is a library package of average complexity. The rules file is minimised where possible using dh, but still does complex things in its build and install targets. It's also a good example of how to set up a -dbg package when using dh, and of how to set up /usr/share/doc symlinks between packages when using dh.
• libaudio-mpd-perl is an example of a perl module package using dh. It shows how to override dh default behavior in two ways: Skipping running dh_auto_test, and passing a changelog filename to dh_installchangelogs.

Early adopters, have at it!

anjuta is my favourite IDE for C work - the configurable symbol browser/highlighter, the ability to right click on a function name or typedef and go straight to that part of the relevant source file somewhere else in the project (or outside), the debug support, mostly sane syntax highlighting (i.e. when it works it is perfect, when it doesn't work it doesn't try and make things worse). geany is very competent too, although possibly better suited to small tasks. (The compile and build menu options in geany don't always use the existing Makefile.)

Recently anjuta has developed an annoying habit of complaining about filetypes it claims not to understand (e.g. configure.in ac, Makefile.am - quite important files in the general C workload), geany (my favourite text editor, especially for perl) does those well but doesn't seem to support highlighting ChangeLog or debian/changelog - a task left to gedit but I can't use gedit for source code because the highlighting and general support of normal source code is truly awful, especially for perl.

(I got comments last time I rated geany higher than gedit - it's a question of focus. gedit cannot do everything - at least not well - but it tries anyway and therefore fails badly. Geany has far, far superior source code highlighting and an incredibly useful symbol browser because it has a more "source code" focus. I'm not going to go scrabbling around in gedit to create a customised highlighting scheme and only have two or three colours for perl is completely unacceptable in gedit (and changing colour scheme as no effect), as is the lack of a symbol browser.) Besides, geany loads and runs much faster than gedit.

Compare for yourself: Geany:
geany.png>

or gedit:
gedit.png

(inlined images removed - originals behind the links are quite large)

I mean, WTF? No separate colour to distinguish interpreter keywords or function names from plain text strings in gedit?? Duh? Badly broken regexp highlighting that highlights only the $ at the end? The entire file is like that - there are three colours in all. 99.9% is puce, variable names are olive and characters like line endings or tab markers are blue. Yes, I can change those colours but what's the point of changing everything that is now in puce into blue when what I need is some parts to be blue, some parts to be light blue, some parts to be red and some parts to be yellow. The gedit perl highlighting has to be the worst I've seen in any editor. It's almost easier in monochrome. Perl is by far the worst but gedit C highlighting isn't much better.

Compare with geany where keywords are always identifiable, regexps are marked up clearly, no silly marking of line ending characters inside a string, sane marking up of pod content and HERE documents and variable highlighting - and it never gets confused.

Needless to say, vi (or more precisely vim) does all the highlighting with aplomb, even though asking for a symbol browser in vim is a bit excessive, even for me.

vi.png

Why can't I have a single GUI that does highlighting as well as vim ? Geany is the closest of the three - lacking only ChangeLog and debian/changelog (and debian/control) highlighting, anjuta is almost there (once that annoying bug is fixed - already filed by someone else) but gedit? What? Has groundhog day been changed to 1st April somewhere? They're having a laugh, must be.

simple case Imagine a very simple debian/rules, that should work for most packages w/o modification:

#!/usr/bin/make -f
build:
    dh build
clean:
    dh clean
binary-arch:
    dh binary-arch
binary-indep:
    dh binary-indep
binary: build binary-arch binary-indep

absurdly simple case

AKA debian/rules zen guru level.
#!/usr/bin/make -f
%:
    dh %@

the complex cases Often you'll want to handle configuring by hand. (And maybe, run automake too.)

build:
    dh build --before configure
    #aclocal
    #autoheader
    #automake --add-missing
    #autoconf
    ./configure --kitchen-sink=yes
    dh build --after configure

Here's how to skip two steps in a row (configure and build):

build:
    dh build --before configure
    ./mondoconfig
    make universe-explode-in-delight GROOVINESS=420
    dh build --after build

Another common case is wanting to run some code manually after a particular debhelper command is run.

binary: build
    dh binary --until dh_fixperms
    # dh_fixperms has run, now override it for one program
    chmod 4755 debian/foo/usr/bin/foo
    # and continue
    dh binary

Suppose you hate dh_strip, and just don't want it to run. (This might be a contrived example; if people frequently don't want to run a few commands in a set, it may be better to have -X exclude them.)

binary: build
    dh binary --before dh_strip
    dh binary --after dh_strip

Maybe you want to run a command out of order. This should be no problem:

binary: build
    dh binary --until dh_fixperms
    dh_install foo usr/bin
    dh binary --after dh_fixperms

A separate install target can be added if you like. And maybe you want to run make install by hand in it. (This is tricky because dh binary needs to avoid running dh_auto_install which was skipped over in the install target.)

install: build
    dh install --before install
    $(MAKE) install PREFIX=debian/tmp
    dh install --after install
binary: install
    dh binary

Once you have an install target, you can do this to run some commands manually after the install target runs, and before the remaining steps are run to build a binary.

binary: install
    dh_strip -X foo
    dh_fixperms -X bar
    dh binary --remaining

If the --remaining was left off, dh binary would only run commands in its list after dh_fixperms. With --remaining, it runs commands that haven't run before, avoiding running dh_strip and dh_fixperms again. A more elaborate way to do the same thing, without needing a separate install target is this.

binary: build
    dh binary --before dh_strip
    dh_strip -X foo
    dh binary --after dh_strip --before dh_fixperms
    dh_fixperms -X bar
    dh binary --after dh_fixperms

the dh_auto_* commands dh_auto_* are thin wrappers, with enough intelligence to handle 90% of cases (autofoo, perl/python packages, etc) automatically. These will rarely be run by hand, instead if they don't do what is needed, they can just be skipped.

• dh_auto_configure: Figures out how to run autofoo, ./configure, Makefile.PL, or whatever.
• dh_auto_build: runs make
• dh_auto_test: runs make test or the like (may have a config file specifying if test failures should fail the build).
• dh_auto_clean: runs "make clean" etc
• dh_auto_install: runs "make install" or similar

the dh command dh runs a set of other debhelper commands in order (and prints their names as it does so). The set that it runs is determined by its parameter. The sets of commands are:

• build: dh_auto_configure, dh_auto_build, dh_auto_test
• clean: dh_testdir, dh_testroot(?), dh_auto_clean, and finally dh_clean.
• install: Everything in the build list, plus dh_installdirs, dh_install, dh_auto_install (and probably dh_install* too)
• binary: Everything in the install list, and all debhelper commands used in binary targets.
• binary-arch: Like binary, but runs commands with -s.
• binary-indep: Like binary, but does not run dhstrip or dhshlibdeps or dh_makeshlibs, and runs commands with -i.

Possibly dh does smart optimisations to avoid running lesser-used commands if they will do nothing. (Or not.) Each debhelper command will record when it's successfully run in debian/package.log.debhelper. (Which dh_clean deletes.) So dh can tell which commands have already been run, for which packages, and skip those. Options passed to dh are passed on to each command it runs, which is useful for common options like -v, -X, and the like. For that matter, you can use it for uncommon options as well, since commands will ignore options that don't apply to them. dh has a few options of its own:

• --until cmd: Run commands until and including cmd, then stop.
• --before cmd: Run commands before cmd, then stop.
• --after cmd: Run commands that are listed after cmd.
• --remaining: Run all listed commands that have yet to be run.

cmd can be a full name of a debhelper command, or a substring. It'll first search for a command exactly matching the name, to avoid any ambiguity. For example, "--before dh_installdeb" does not refer to dh_installdebconf. Otoh, "--before build" is much easier to type, and clearer, than "--before dhautobuild". If there are multiple substring matches, the last in the command list is taken. So "--before install" stops before dh_auto_install, and not before the earlier dh_install. Each time dh is run, it examines the log, and finds the last logged command that is in its list of commands to run. It then continues with the following command in its list. (Or the one after --after, if that was specified. Or if --remaining was specified, start at the first command in the list that is not in the log.) So... If you run debian/rules build twice, and it just has "dh build" in it, the second run will be a no-op. The same as if the build target touched a build-stamp. This is generally considered a feature, so I'll consider it a feature in dh too. (To force a rebuild, run dh_clean to remove the log.) If you run debian/rules binary, and a build hasn't happened yet, it will run the build steps first. Even if the rules file is a bit lame (or over-golfed), and doesn't have the binary target explicitly depend on the build target. Ditto for the install target. If a rules file runs a command out of order, it'll need to use --after to get dh back on track. If you manually run a debhelper command at the command line, running debian/rules binary will skip forward until after that command. Similarly, if debian/rules binary is interrupted (or fails), running it again will resume after the last debhelper command run. This is a bit of a behavior change, and could be considered a bug, but is easy enough to work around by running debian/rules clean binary. (Or by manually deleting the log.)

I originally had the idea for a list to keep track of stuff that doesn't work with Linux something like 12 years ago. I built an initial version, that proved reasonably popular at the time, but then the place it was hosted had some problems, and I was busy with other things. I finally got around to building a new version four or five years ago, and it's been doing ok since then, but with a baby on the way, I've been thinking of things to get rid of, and this is not something I do active work on, so maybe it would be best to find a new home for the site, located at http://www.leenooks.com. Here's the catch:

• Whoever gets it has to promise to maintain it as essentially what it is today - you have to continue to provide information about hardware that does not function with Linux, or only functions with proprietary drivers.
• It does generate some money, that I put towards the server where it's hosted. I wouldn't mind selling it for something, which will hopefully encourage only those truly interested to apply. It could probably generate more money if it were plastered with a bunch of ugly ads, or, hopefully, if more time and effort were put into it than I have at the moment.

It currently runs on my own server, and uses DedaWiki, an open source, Rails-based Wiki, as well as PostgreSQL. You would have to relocate it to your own server. So, if you're interested, contact me: davidw@dedasys.com. If I don't find the right person, no worries; I'll keep maintaining it for a while yet.

[Warning, long post ahead. If you aren’t interested in Debian or it’s internals or my work - skip it. :) ] So as you might have read here, I got the delegation of the debadmin AKA FTPMaster group lately. I’ve got some pressure to not accept this delegation, but the response to my blog post about it had been extremely positive, everyone said I should not deny it. Knowing to have such a support within Debian is something I really appreciate, making it worth to spent time on my various jobs. Now, lets take this and write a little about what happened since I gained that extra group:

• Aj resigned from anything that requires additional group privileges. (As a DD run id ajt on a debian.org machine if you want to verify this).

Work I did since then:

• One of my first action was to finally finish the release transitions support, something which was waiting for it’s final merge for a long time. The release team was waiting for this and did ping us (ftpteam) multiple times, but now it is there. You can read more about this feature in my list mail and my initial blog post. (Oh, and something I have missed about everywhere I wrote about this: This whole transitions thing is based on an idea from Marga!)
• Added a mapping in dak, requested by Luk
• Merged a patch from another ftpteam member, Thomas Viehmann, which fixes daks handling of Format 3.0 (something) uploads. Currently it rejects them, without the merged patch it just crashed.
• Changed our daily cronscript to no longer VACUUM/VACUUM analyze projectb in every run. Postgres >= 8.1 has autovacuum, and its turned on, so we should be fine without extra runs of it. (And while i write daily cronscript or cron.daily - I mean dinstall, right now run twice a day).
• Fixed a breakage in code introduced by Debian maintainers which led to process-accepted (one main part of the daily cronscript) breaking. It was triggered by some octave-$FOO package upload, where the ###censored### did add someone twice into the Uploaders: field. Not helpful, but now we ignore this case and just warn us.
• Added a script to expire old database dumps. We do dump our projectb (the main archive database) twice in every run (beginning/end), which made 4 dumps a day. While old dumps seemed to have been removed on an irregular schedule - the dumps took about 71Gb in total. Which pretty much filled up merkels /org, as they get synced there. Now, my script (reworked version of a script from Florian Reitmeir) works with an “incremental deletion” scheme:

  RULES = [ ‘days’:14, ‘interval’:0 , ‘days’:31, ‘interval’:1 , ‘days’:365, ‘interval’:31 , ‘days’:3650, ‘interval’:365 , ]

  That means we now keep all files from the last 14 days. Then keep one per week. Then one per month and finally one per year for up to ten years. All the rest gets deleted. Space usage of our dumps now: 4.8Gb. Thats 66Gb gained. Source is in dak bzr if you want it, MIT licensed. If you improvie it further I love to see patches.
• Changed the way the projectb copy on merkel gets synced. Up to now it was run from a cronjob which syncs a number of files people want to see on merkel, including the database dumps, then reloads the database using the latest dump. This was run at a “random” time, scheduled to be shortly after dinstall. Problem with such a setup is that it will miss extra dinstall runs and also might import old data, as the timing of the cronjob meant it runs while dinstall on ftp-master is still doing its work. So bad bad. It is now done using the ssh trigger feature I described a while ago. So our daily cronscript now tells merkel when it should sync its database again.
• Coordinated with the qa group we now also push them, when the sync on merkel is done, so they can kick their cronjobs to reread projectb and just run whenever they get told that something changed. Yet another bad timing eliminated. (Well, will be done today). Oh, if you have something that depends on information from projectb / ftp-master and try to schedule it to run after dinstall - talk to me, we might just push you too.
• Started talking with the release team how britney, the script which manages our testing distribution, should be handled in the future. It currently is run fully under control of us ftpmasters (and reading hint files from the release team). Ideally this gets changed so that the release team has the control of the code and whatever gets run with it and then, in some way, maybe using ssh triggers, tells us to import a new dataset for testing. That needs some thinking, as obviously it should be limited to testing and we should make sure it doesn’t break the archive, but I think that is doable.
• Prepared a patch that sanity checks the Provides: field, something we currently don’t do. Mailed our internal team address about 3 days ago, to let others look at it, no response yet.

The above should nicely sum up my work in the past few days. Most of it pretty trivial, but still things that have to be done. I plan to keep doing things, slowly going from the trivial stuff to more complicated ones, but there is lots of stuff ahead. Some ideas, and this is a very incomplete list just from my head, there are tons of things that one could do, include

• Finally allowing dpkg-sig support in .deb files. I once, years ago, made a patch for it, which never got applied. (Mostly my fault here). Need to rewrite that to fit the actual code.
• Add a few more sections to our archive. There are multiple bugs asking for them, and some others also come to mind. Like java, ruby, octave, haskell, debug to name a few, but there might be more needed. (Yes, longterm all sections should die and be replaced with debtags. I just don’t think debtags is there yet?!)
• Maybe finally drop m68k and add other architectures to the archive, like some $foo*bsd one. (Thats for after lenny of course, at the time arm also gets dropped).
• Merge a patch I got from Myon which will keep uploaders in projectb.
• Enable bzip2 support for sources, not only for the binary packages. I have a patch ready, it’s not yet merged. Needs to test if anything breaks in stable if we apply it, before we can think about it. Worst case it has to wait until we are running lenny. (The archive runs on a machine with stable, so allowed features always have to be supported in stable).
• Process the currently waiting requests for pseudo-packages and either reject or approve them. After thats done - talk to the BTS maintainers to hand the management of pseudo-packages over to them, as 99% of the usage of those is in the BTS.
• Get dak to optionally mail the sponsor of a package too, if not already in the list of those getting a dak mail. Which isn’t easy. One proposal is to just look at the gpg keyid and get the Debian uid from it, send mail there. I don’t want this, as people are, unfortunately, able to disable mail receiving for their @debian.org address. Which would give us lots of bounces. No way we like that. Similar problems with “Just take one userid that looks like a mail address from their key”.
• Improve various parts run by our daily cronjob, as that one takes ages (about an hour today). Speed improvements are always nice there.
• Think about having more dinstall runs a day. Like - every 2 or 4 or 6 hours could be nice. But that needs discussion within the ftpteam and also with the mirror people before it can be done. But 6 hours seems to be something nice to have, more might be also ok.
• Coordinate with DSA to enable the continous AKA WAL archiving for our database.
• Make merkels sync be more instant, ie. no longer rely on a push of a database dump whenever dinstall run, but something thats near realtime. Having changes done on ries immediately synced to merkel.
• More (active) ftp-team member, maybe?!

Now, do you have any time to kill? Do you want to help? Some things you could do:

• Go look at our bug page and start there. Someone, I think Don, already started using user categories. It would be nice if someone is willing to keep up with the bugs and put them in categories. And continue to do that in the future.
• Generally help with our bug page. One nice task some people do, and possibly more could also do, is retitling bugs. Especially removal bugs tend to have a wrong title. The format is pretty specific…
• Feel free to submit patches to me. I promise to look at them without too much delay and at least give you a hint what might be wrong or could be done better. And if it makes sense, doesn’t break the archive, works, and is something dak should have - I will also merge it. Of course trivial patches are way easier in that regard, non-trivial ones should go via our ftpmaster address first, to leave others a chance to comment on. Bonus points if you provide bzr trees one can merge from, and the bzr tree you should work with is http://ftp-master.debian.org/bzr/ftpmaster-dak/ - patches/bzr trees against outdated versions won’t have much luck to get looked at. :)
• Keep me happy. However you do it, just don’t hug me. :)

An issue that causes ongoing discussion is what is the purpose of a Planet installation such as Planet Debian [1]. The discussion usually seems to take the less effective form of what is “appropriate” content for the Planet or what is considered to be “abuse” of the Planet. Of course it’s impossible to get anything other than a rough idea of what is appropriate is the purpose is not defined, and abuse can only be measured on the most basic technical criteria. My personal use of Planet Debian and Planet Linux Australia [2] is to learn technical things related to Linux (how to use new programs, tricks and techniques, etc), to learn news related to Linux, and to read personal news about friends and colleagues. I think that most people have some desire to read posts of a similar nature (I have received a complaint that my blog has too many technical posts and not enough personal posts), but some people want to have a Planet with only technical articles. In a quick search of some planets the nearest I found to a stated purpose of a Planet installation was from the Wiki to document Planet Ubuntu [3] which says ‘Subscribed feeds ought to be at least occasionally relevant to Ubuntu, although the only hard and fast rule is “don’t annoy people”‘. Planet Perl [4] has an interesting approach, they claim to filter on Perl related keywords, I initially interpreted this to mean that if you are on their list of blogs and you write a post which seems to refer to Perl then it will appear - but a quick browse of the Planet shows some posts which don’t appear to match any Perl keywords. Gentoo has implemented a reasonable system, they have a Universe [5] configuration which has all blog posts by all Gentoo bloggers as well as a Planet installation which only has Gentoo related posts. It seems to me that the a reasonable purpose for Planet Debian would be to have blog feeds which are occasionally specific to Debian and often relevant to Debian. Personal blog posts would be encouraged (but not required). Posts which are incomprehensible or have nothing to say (EG posts which link to another post for the sole purpose of agreeing or disagreeing) would be strongly discouraged and it would be encouraged to make links-posts rare. Having two installations of the Planet software, one for posts which are specific to Debian (or maybe to Debian or Linux) and one for all posts by people who are involved with Debian would be the best option. Then people who only want to read the technical posts can do so, but other people can read the full list. Most blog servers support feeds based on tag or category (my blog already provides a feed of Debian-specific posts). If we were going to have a separate Planet installation for only technical posts then I expect that many bloggers would have to create a new tag for such posts (for example my posts related to Debian are in the categories Benchmark, Linux, MTA, Security, Unix-tips, and Xen) and the tag Debian is applied to only a small portion of such posts. But it would be easy to create a new tag for technical posts. Ubuntu is also the only organisation I’ve found to specify conditions upon which blogs might be removed from the feed, they say: We reserve the right to remove any feed that is inaccessible, flooding the page, or otherwise interfering with the operation of the Planet. We also have the right to move clearly offensive content or content that could trigger legal action. That is reasonable, although it would be good to have a definition for “flooding the page” (I suggest “having an average of more than two posts per day appear over the period of a week or having posts reappear due to changing timestamps”). Also the “could trigger legal action” part is a minor concern - product reviews are often really useful content on a Planet… Some time ago my blog was removed from Planet Fedora for some reason. I was disappointed that the person who made that change didn’t have the courtesy to inform me of the reason for their action and by the fact that there is no apparent way of contacting the person who runs the Planet to ask them about it. Needless to say this did not encourage me to write further posts about Fedora. If a blog has to be removed from a feed due to technical reasons then the correct thing to do is to inform the blogger of why it’s removed and what needs to be fixed before it can be added again. If a blog is not meeting the content criteria then I expect that in most cases the blogger could be convinced to write more content that matches the criteria and tag it appropriately. Having criteria for some aspects of blog quality and encouraging the bloggers to meet the criteria can only improve the overall quality. Currently there is a Planet installation on debian.net being recommended which is based on Planet Debian, but with some blogs removed (with no information available publicly or on debian-private as to what the criteria are for removing the blogs in question). It seems to me that if it’s worth using Debian resources to duplicate the Planet Debian then it should be done in a way that benefits readers (EG by going to the Planet vs Universe model that Ubuntu follows), and that if blogs are going to be removed from the feed then there should be criteria for the removal so that anyone who wants their blog to be syndicated can make whatever changes might be necessary.

• [1] http://planet.debian.org/


• [2] http://planet.linux.org.au/


• [3] https://wiki.ubuntu.com/PlanetUbuntu


• [4] http://planet.perl.org/


• [5] http://planet.gentoo.org/universe/

Share This

OK I'm done with this now, the sift utility has been released. I think that is a large overlap with imapfilter; but I win because I can write simple rules, rather than any actual code, to perform jobs.   In other news I flew my kite today, and I still like eating Pies: Thank God reading Debian Planet isn't mandatory. ObQuote: The Godfather

I started physiotherapy this past week. The first clinic I went to on Wednesday (Mountain View Physical Therapy) turned into an interesting disaster. When we walked into the place, the first impressions were that it was noisy and dirty. There was someone sitting upright in the waiting area asleep. The place is one big room, with the massage tables in the middle of the room and exercise equipment all around the outside. None of the staff looked really happy to be there. I met my therapist who, it turns out, is the director of the clinic. She was truly excellent - good enough that I would've overlooked everything else, except for the fact that she doesn't have 3 days a week free to see me. More weeks, I would've been lucky to get her more than once per week.

So for Friday, we found a different clinic. Orthosport. It was such a difference from the previous place, there's no good comparison. The receptionist was friendly, the therapist very professional, and the place really tidy. Even the assistants were very friendly and checked in on a regular basis.

But in between those two days, I got sick again. I had some sort of stomach flu or drug reaction the previous week, and it seemed to be coming back. After two days of throwing up, I made an appointment with the after-hours clinic today to get more anti-nausea drugs. They've also taken some more blood and want to do some tests next week to make sure that everything's still doing okay.

Another bit of love and affection goes to my new bank, RBC Centura. Not only have they done lovely things for bill payments, and sending money for free between my Canadian and American accounts, they also have a "Bank by Mail" program, where I can just stick cheques in an envelope that's already postage paid, and they deposit them for me. No need to find a machine of the right persuasion to do deposits. We still have a couple tasks to do before we can close out the HSBC account, but we're almost there.

Leif is pushing out new teeth. He'll be up to 9 when he's done.

So I've written a quick hack. A client-side filter/utility program for working against IMAP servers. Consider it a general purpose system which is similar to Procmail, but applied after your remote machine has already done the sorting. Here's a flavour:

<GMail>
  username somebody.like.me
  password yeah.right
</Gmail>
<Folders>
  <livejournal>
        unread exec /usr/local/bin/notify "Livejournal Comment"
        mark read
  </livejournal>
  <inbox>
        mark read
  </inbox>
</Folder>

What does that do? It first of all logs into GMail with the given username and password, then selects two folders:

=livejournal/

For each unread message in the folder it runs the specified command with STDIN being the message body. Then it marks each new message as "read".

=inbox/

This simple rule just marks all messages as read.

Why? Well I have a bunch of folders on a bunch of gmail accounts and I don't pay attention to them - but some, specific, mails should result in an SMS being sent to me ... so I need to do something clever. I'm sure with a bit of effort this could be made IMAP-server independent, and could have a more flexible matching system. The simplicity right now comes about primarily because i dont want to parse a config file. Anyway, suggestions for potential features are welcome. It does what I need as-is, even if it isn't pretty. ObQuote: Lost Boys

New developer accounts have been created moments ago, “finally”. Congratulations to all new DDs, with a special note for Aur lien G R ME (ag) and Cyril BRULEBOIS (KiBi). All of this made possible by Sam, our best DPL to date. Also, don’t listen to the fucktards going around telling “OMG DEBIAN IS DYING!!11!1!”. They’re just that, fucktards.

Patrick Winnertz writes about the demotivating effect of unreasonable delays on joining the Debian project [1]. While I agree that things need to be improved in terms of getting people in the project in a timely manner (the suggestion of providing assistants seems good), I don’t think that anyone has a good reason for being demotivated because of this. I first applied to join Debian in late 1998 or some time in 1999. At the time part of the process of joining was to receive a phone call. At the time I was living in a hotel and they refused to call me on such a line. I could have easily camped out in the hallway of a hotel (the cheap London hotels often had a pay-phone in the hall and no phones in the rooms) and pretended that it was my own phone with an unlisted number. Unless they refused to allow people with unlisted numbers to join (which seems unlikely) then I could have joined then. So it seems that at the time I could only join Debian if I was prepared to lie about my ownership of a phone line. I wasn’t overly bothered by this - there has never been a shortage of free software projects that need contributions of code. By late 2000 the rules had changed and I joined without needing a phone call. In the mean time I had forked the Bonnie storage benchmark program to form my own project Bonnie++ [2], created Postal - mail server benchmark suite [3] and worked on many other things as well. I have sympathy for the people who apply to become Debian Developers and who have to wait a long time, I’ve been in the same situation myself. But there are plenty of things that you can do in the mean time. Some of the things that you can do are upstream development work, filing bug reports, submiting patches that fix bugs, and writing documentation (all forms including blog posts). Also when projects aren’t yet in Debian it often happens that someone creates unofficial packages, the person who does this doesn’t need to be a DD. Producing back-ported packages for new versions of programs that are in a stable release can also be done by people who are not DDs. Unofficial and back-ported packages provide less benefit for the project as a whole but considerable benefit for the people who want to use them. There is a lot of work that can be done to fulfill clause 4 of the Debian Social Contract [4] (Our priorities are our users and free software) which doesn’t require being a Debian developer. It seems to me that if you have the right approach to this and maintain the perspective that Debian is one part of the free software community (and not necessarily the biggest or most significant) then a delay in your application to become a DD won’t be particularly demotivating.

• [1] http://www.der-winnie.de/posts/Demotivaton_of_new_prospective_Developers/


• [2] http://www.coker.com.au/bonnie++/


• [3] http://www.coker.com.au/postal/


• [4] http://www.debian.org/social_contract

Share This

Another zero-day announcement of a Software in the Public Interest online meeting has been posted. The announcement mentions "one motion has been raised" but the agenda doesn't include any motions, so I'm not sure what. Also, the agenda lists "Debian logo licence" as up for discussion but I thought last month's meeting resolved that. I watch SPI fairly closely and I'm confused. How about the rest of you? SPI isn't alone in this: many software organisations seem to suffer from similar problems. For example, I ranted in passing about Free Software Manchester yesterday, who just posted the notes from their own zero-day meeting and I've suggested possible ways of reforming debian's currently-stalled New Maintainer process more than once. Are free software users particularly bad at the basics of running an interest society (like welcoming and expiring members, calling meetings, publishing routine communications, and so on), have I been spoiled by cooperatives with their friendly Member Services departments or secretariats, or what? Is this why so many free software orgs seem to include self-perpetuating leadership groups? Is this a serious problem if, as reported, Software Development is a Team Sport [etbe]? Are there fully-working free software mass participation groups out there? I feel a lot of these problems are caused by attempting to order our inherently entropy-filled world completely and insisting everything follows petty rules, such as refusing to answer a question because the "wrong" member asked it. The world will not become less random just because hackers try to impose arbitrary rules. Sometimes it's good to put down minimum standards (because calling zero-day meetings is a mostly-avoidable way of excluding some members) but it will always be a poor alternative to trying to do the best you can for others. How do we get past this? My pro-cooperation-and-better-business platform for SPI board went pretty badly and I've had some anti-cooperative flames back from someone starting another free software group this month, so I don't think I can fix these existing organisations any time soon. About 1 in 6 people in the UK are members of a cooperative, so even if that is reflected among hackers (and I think it's lower), all of them would not be enough to reform much. One of the most common memes in free software is "show us the code" and the few other free software cooperatives I've seen have mostly failed, with a few surviving but hitting a size limit. As a result, I'm currently negotiating the start of a new free software cooperative. So far, I'm really happy with how that's going. Many cooperators learn at the feet of large consumer cooperatives like the Cooperative Group who run training courses for new members about putting cooperative values and principles into practice, which we smaller groups couldn't run ourselves. As a result, most cooperators already know how to work well together. Should large software societies like SPI try commissioning similar courses?

Albert writes about software development and how much teamwork is used [1]. He makes an interesting clash of analogies by suggesting that it’s not a “team sport” because “its not like commercial fishing where many hands are used to pull in the net at the same time“. I think that software development for any non-trivial project is a team sport. You don’t have the same level of direct coordination as required for pulling in a net (or the rugby scrum [2] to use a sporting analogy), but that doesn’t stop it being a team effort. Some parts of team development projects are like a relay event, in corporate environments the work is done in parallel simply because everyone is working the same hours but in free software development projects the work is often serialised. I think that it’s often more effective to serialise some work, if someone is actively working on one sections of code it may save time to avoid working in that area until they are finished. There is little benefit in writing code to old interfaces. Some parts of team projects have specialised skill areas (EG debugging, skills in particular programming languages, and graphical design). Soccer is one sport where different rules apply to different players (the goal keeper can use their hands). In ice-hockey the protective clothing used by the goal keeper is considerably different from that used by other players. In most team sports where the aim is to put a ball through a goal at one end (EG basketball and all versions of football) there seems to be some degree of specialisation, some players are dedicated to scoring goals while others are dedicated to defense. The fielding team in cricket has every player assigned to a different part of the field - with slight differences in the skills required. Then there is the issue of large projects such as Linux distributions. It seems to me that a Linux distribution will always comprise multiple teams as well as some individual projects. Maybe we could consider Linux distributions (and distributions of the other free OSs) to be similar to countries that compete in the Olympics. The culture of the Free Software (or Open Source if that’s your idea) community can be compared to the Olympic Spirit. Of course the Olympic idea that people should come together in peace for the Olympic Games and that it’s about honor not money is pretty much dead. Maybe the Free Software development processes should be compared to an ideal of what sporting contests would be if there weren’t unreasonable amounts of money (and therefore corruption) involved. Of course no analogy is perfect and there are many ways in which this one breaks down. One of which is the cooperation between distributions. There is a lot of private discussion between developers of various distributions and upstream developers about how to plan new features. It’s not uncommon for developers to announce certain development decisions as soon as they are made to help other distributions make decisions - for a developer in a distribution project if there is an issue which doesn’t matter much to you or your users then it’s often good to strive for compatibility with other distributions. When users advocate new features or changes they sometimes try multiple distributions. It’s not uncommon for a feature request to be rejected by one distribution and then accepted by another. Once a feature is included in a major distribution the upstream developer is more likely to accept it due to it’s wide testing. Then when the feature is in upstream it’s almost certain to be included in all other distributions. I often recommend that when someone disagrees with one of their bugs being closed as “not a bug” that they try reproducing it in another distribution and reporting it there. As a side note, the criteria for reporting a bug in any free software distribution is that you can describe it in a way that allows other people to reproduce it - whether it’s a bug that afflicts you every day or whether you installed the distribution for the sole purpose of reporting the bug in a new forum is not relevant. As a general rule I recommend that you not have the same bug report open in more than one distribution at any time (if you notice a bug reported in multiple distributions then please add a note to each bug report so that work can be coordinated). As a general rule the only situation where I will open the same bug in multiple forums is if I have been told that the responsible person or people in one forum are unwilling or unable to fix it. Finally, the people who consider that they don’t need to be a team player because they do their coding alone might want to consider Qmail. Dan Bernstein is a great coder and Qmail is by most metrics a fine piece of software, in terms of security Qmail is as good as it gets! If Dan was more of a team player then I believe that his mail server would have been much more successful (in terms of the number of sites using it). However I do understand his desire to have a great deal of control over his software.

• [1] http://www.docunext.com/blog/2008/04/13/the-people-who-live-on-planet-debian/


• [2] http://en.wikipedia.org/wiki/Scrum_(rugby)

Share This

I don't think any of these are mine, but this needs to be published to the general public (at least until it rests on AlumNit).

Poo Alliterations
- The Earl of Excrement
- The Duke of Dung
- The Prince of Poop
- The Sultan of Shit
- The Father of Feces
- The Matriarch of Merde
- The Steward of Stink
- The Dignitary of Dangleberries
- The Grand Poohbah of Grand Poo
- The Tyrant of Turd
- The Shogun of Scheisse
- The Baron of Bowels
- The Cardinal of Crap
- The Director of Dump
- The Surveyor of Stool
- The Lord of the Log
- The Crown Prince of Cow Patties
- The Royal Archbishop of Road Apples
- The Duchess of Droppings
- The Princess of Poodie
- The Legionnaire of Loaf
- The Supervisor of Stinkers
- The Pharaoh of Floaters
- The Magnate of Manure
- The Tycoon of TCPDumps
- The Czar of Czesspools
- The Mogul of Bowel Movements
- The Ruler of the Runs
- The Dean of Diarrhea
- The Royal Ringleader of Rump Raisins
- The Denizen of Doo-doo
- The Emperor of Evacuations
- The Headmaster of Horrible Odors
- The Titan of the Trots
- The King of Klingons
- The Big Boss of Bum Beans
- The Taskmaster of Turtleheads
- The Supervising Sovereign of Sewer Serpents
- The Colonel of Colon Cobras
- The Nobleman of Number Two
- The Top Dog of Toilet Twinkies
- The Kingpin of Keester Cakes
- The Shepherd of Sea Pickles
- The Head Navigator of Heinie Nuggets
- Rectal Representative
- The Commander of Cow Pie
- The First Mate of Fecal Matter
- The Creator of Cloth Touchers
- The Protector of Porcelan
- The Theologian of the Throne
- The God of Grunts
- The Small Insignificant Yodeling President of Shit in your Pants
- The Diaper Destroyer
- The Loader of Loincloths
- The Pontiff of Plum Pudding
- The Daiymo of Dysentery
- The Master of Montezuma's revenge
- The Squire of Squirts
- The [Weaver] of [WvLog]s
- The Primary Contact of Pudding Cookies
- The Best Producer of Brown Puddles
- The Senior Partner of Steaming Piles
- The Fine Baker of Fudge Brownies
- The Sheriff of Stench
- The Don of the Deuce
- The Torrid Putrid Tainted Pervert of Two-Ply Toilet Paper
- The Sacred Monarch of Skid Marks
- The Main Scientist of Microfeceological Studies
- The G.I. Joe of Gastro Intestinal Journeys
- The Premier of Polyps
- The Brigadier General of Bedpan Gold
- The Plunger Pilot
- The Specialist of Splashers
- The Foreman of Fake Farts
- The Ambassador's Under-Secretary of Astounding Unearthly Scents
- The Old Man and the Sea of Odiferous Monsters in the Sink
- The Bafflingly Blunt and Blasphemous Bearer of Big Bad Beastly Booty Burdens
- The Crisp Pungence of Chain Pooping
- The Goddess of Groundhogging
- The Burly Bobby's Brief Battle with Brim Blasting Bowel Bombs
- The Nasty Neighbour's Noisy and Nefarious Neglect of Normal Nasal Niceties
- The Judge Jury and Executor of Jolly Jumper Excrement
- The Forwarder of the Forwarded
- The Wizard of the Water-closet
- The Grunting Lobber of Great Logs
- The Annoyingly Arrogant Architect of Awful and Abhorrent yet Amazingly Aerodynamic Anal Atrocities
- The Dark, Dangerous and Delusional Dealer of Death-Defying Doses of Disturbingly Decadent Dung
- The Shakespeare of Sinfully Smelly and Supremely Scary Scripture with Soggy Stains and Screaming Splashes
- The Wise but Weakened Warlord Whose Willful Wrath Was Wholly Wrought With Wicked and Wretched Winds from Within
- The Languid and Lackluster Legend of the Little Log from the Loo and its Lavish Life of Luxury in a Lasagna
- The Kahuna of Kaka
- The Emergency Evacuator of Enraged Entrails
- The Squire of Sultry Stinknuggets
- The Deadening Discomfort of a Dearth of Dumps
- The Far From Favourable Fragrance of Four Fragments of Fantastically Fresh Feces
- The Principal of Prodigiously Putrid Pellets
- The Markedly Morose Maintenance of Maniacally Mounded Midden Movements and Mostly Meaty Meals Mysteriously Made Mushy
- The Crass and Cranky Coroner Connecting Clues Concerning a Cornucopia of Casualties Caused by a Crude Cluster of Completely Contemptible Colon Concoctions
- The Thinking Thoughts of a Thinker
- The Horrible, Hubristic Heathen Hoarding Hoary Humps of Hideousness Hailing from His Huge Hairy Heinie in His Harmfully Heady Hovel
- Inane Writings on Irritated Wreathing of the Incubator of Wrath
- That Special Numbness from Sniffing Noodled Stink Nougat
- The Moral Sanctity of the Mystery Shit
- The Ornery Opressor's Opulent Outpost Overseeing the Ordinary Occupants' Outrageous and Ongoing Observance Of an Occasional but Offensive Orgy Of Offal
- The Rector of Relief
- The Rectal Regurgitator
- The Shit Squat
- The Smelly Surprise
- The World Wide Wisdom
- The Slayer Supreme of System Shock