Chris reports back to Debian duty: Since one week I'm back from NYC where I enjoyed a very nice Debconf. It was my second one after last year in C ceres. When I unpacked all my luggage - three weeks in the US need some stuff - I found these cute green guys between my clothes. Did they break their UFO and use my airplane to travel with me? Anyway. Thanks to all who made Debconf10 in New York possible. It was a great conference! And last but not least: "Happy Squeeze Freeze"' I too enjoyed DC10 a lot.

(this is a post in two parts) At the BoF of GPG Key Management: Best Practices, dkg had two immediate recommendations for improving WoT resilience:

• Regularly refresh your local keyring from the key servers;
• If you have spare space and transit, provide a key server to take part in the pool.

Regular Refreshes This is important because unless keys are refreshed, changed expirations and (more importantly) revocations on keys are never made known to you. Making regular refreshes is easy, though I was surprised to hear people had experienced difficulty with it it s a simple entry to cron, like: 0 0 * * 1 /usr/bin/gpg --refresh-keys to run at midnight on a Sunday, or any other convenient time. Append > /dev/null to make it silent unless there are errors. Provide a key server This is nearly as trivial a refreshing the keyring, but it does take some disk space (on my machine, 4G for the decompressed dump from another server, 7.4G for the database itself and a few hundred MB for other stuff. Supposedly the dump can go after a full database build, but as I m not constrained I haven t bothered). The sks package in Lenny and Squeeze manages the database and answers to clients. However, take care to follow README.Debian carefully, since if the permissions on /var/lib/sks or /var/log/sks aren t set right it will silently fail and much head-scratching will ensue. Building the database for the first time takes some hours even on a reasonably quick machine, and takes a lot of CPU time, so it s probably best done overnight. Once sks is running, it needs to catch up from the date of the key dump to now (which is why a recent dump is always a good start). To do that, it should be listed in the pool.sks-keyservers.net and therefore needs gossip peers. http://code.google.com/p/sks-keyserver/wiki/Peering is an excellent guide, though the version of sks in Lenny does not support multiple listening addresses for either the recon or hkp processes. When a small enough delta between the server and its peers is reached, it s included in the DNS pool for use by other clients.

Thanks to the sterling work of aurel32, debian-cd now supports creating CDs for the kfreebsd-amd64 and kfreebsd-i386 ports. For now, only the daily "sid" builds will be available - see http://cdimage.debian.org/cdimage/daily-builds/sid_d-i/arch-latest/. Once there's been another alpha/beta release of d-i then we'll get squeeze daily builds and weekly builds too.

A DebConf doesn't just happen in a few months. It's not even a year, it's closer to two or two and a half. I've only been through two cycles, and I think this process has happened for the last three or four DebConfs. This post will talk about a "brief" summary of the general DebConf timeline, with elaboration on all the bits later. This post describes the preparation of DebConfN in year 20XX.
At DebConfN-2 (mid 20XX-2), there is a talk where interested groups can describe their location and how it would be suitable for a DebConf. This consists of a short presentation and discussion, but not many details by this point. No decision is made here.Throughout late 20XX-2, teams develop their bid as best they can. The last two years there has been a nominal deadline of December 31 20XX-2 for submitting a bid, but like all DebConf deadlines it's more for guidance than a real deadline. There are various checklists and templates to provide guidance here, which I'll describe later.Early 20XX-1 brings a lot of work to finalize bids, and then the actual selection process. There is a burst of mailing list traffic where people ask questions and help each other refine their bids. There are several meetings where bid teams give their status and people can ask questions. In February or March, there is a selection meeting.After the decision meeting, there is a big round of cheers but people soon refocus on DebConf N-1 since it is more urgent. Ideally, most of the people involved in DebConfN help with DebConfN-1 both because DebConfN-1 needs help, but also to learn how to make DebConfN go well. The DebConfN local team keeps working in the background doing whatever is needed to prepare for their conference.At DebConfN-1, there is a talk where the DebConfN team presents their progress to the DebConf attendees and tries to build support.Late 20XX-1 brings the end of the previous conference and time for N's localteam to really start getting stuff done. Throughout the last part of the year, we try to work out enough details for registration to begin: final conference dates, locations, accommodation, etc.Registration opens in early 20XX. The exact dates for opening/sponsorship deadline/reconfirmation/etc all depend on how early in the year the conference will be. There are nominal deadlines published, but they usually slip. With our limited volunteer base, it's hard to make things go according to schedule, but things always work out.Around this time, the fundraising team really starts working hard. Also, the travel sponsorship team needs to meet to decide who gets funded. The talks team needs to issue a call for events, read, and select them. The video team needs to make sure that they will be ready for the DebCamp setup period. There are many more things I'm forgetting, too.In the months before arrival, it becomes kind of crazy from a local perspective. There are many things to get organized before people arrive, mainly focusing on food, accommodation, and venue space logistics.Once DebCamp starts, things get even more complicated, but we have a flood of experienced DebConfers from around the world come in, and they all know a bit and can help out. DebCamp is effectively the DebConf preparation week to the organizers. DebCamp becomes DebConf, and much bigger, once all attendees arrive.Eventually, people leave and everyone goes home. This doesn't mean local stuff is over - I'm still engaged in DebConf10 wrap-up, making sure everyone gets paid and everything is returned.Mid to late 20XX brings a brief period of rest for most people.There is one last thing to do, though: the DebConf final report. This describes the events of DebConf, and is very useful for getting sponsors. It is also a valuable reference for future years looking back on the lessons learned. You can browse past final reports in our archive . Because everyone is tired, this too frequently gets pushed off until early 20XX+1, when DebConfN+1's fundraising team needs it. A frantic rush finishes up the missing pieces and it's done.
So, there you have it: a rough outline of the timeline of a DebConf. It extends from mid 20XX-2 to early 20XX+1, more than two and a half years. This post nowhere near describes all the details that go into DebConf, almost any of the paragraphs above could be made it's own post. Stay tuned for them.Tags: debconf, debian

As you may remember, our co-op is working on various RFID (radio tags instead of barcodes, basically) extensions for Koha. One of the main ethical concerns about RFID is privacy if done wrong, it could become quite easy to see what books from which libraries someone has in their bag, without their consent and without physical contact. If borrower cards themselves become RFID-enabled, you might even obtain their personal data pretty easily, although I m not aware of any libraries in practice who put any personal data onto the borrower card tags yet. On 12 May 2009, the European Commission Recommended privacy and data protection in all RFID applications. That was followed by EU Mandate M436 for the European Standards Organisations (ESO) to develop standards to support that recommandation. Phase 1 started last March and has delivered a document RFID-DTR07044v006-draft (PDF). Phase 2 will build on that to produce formal standards for signage, privacy impact assessments and so on. EDItEUR are replying to that draft on behalf of the library sector and have sent me their draft response RFID_LibrarySectorCommentsDTR-07044-1 as a PDF. It s worth opening those two side-by-side to read through. If you can help improve the combined response, please send feedback to the email address in the Library Sector PDF by 6 September. If you d like to make your own response, you have until 15 September to reply to the ESO PDF using their template. By the way, does the ESO PDF look jaggy to anyone else? It looks like ghostscript can t anti-alias the text.

My project. This part of this post assumes that the reader has read my last post. The Summer of Code is over. It was great to spend time working on GHC and getting money for it. Although the implementation is not mature enough to be included in the repository, I m happy with the state it is now. I think it s a good start, and I plan to keep working on it. It s good to see how my motivation increased now that the program is over, and that I m free to not work on it if I want to. So I m going to take a look at it again, as soon as I do the stuff I was postergating during the program. I ve created a wiki page for it, and my plans are to implement and measure the Remove partial lists patch, and then to debug Allocate in lines in minor GCs . Any help is very welcome.

I was thinking about this the other day, and I realized I can't particularly think of a label to attach to this concept. Anyone out there got a name for this? This could be due to my lack of formal computer science training, so I thought I'd ask, at the risk of looking stupid (I'm having an off day, so that's a very real possibility). If you have a series of functions, say: a(x), which in turn calls b(x), which calls c(x), and so on, and you end up with a faily long call chain, out to say m(x), and you subsequently decide that m(x) needs an additional piece of information that is contained in a local variable in the body of the a(x) function - how do you get it to m(x) ?

• You could simply make it a global variable! That's easy and generally a pretty bad idea.
• You could modify every function signature between a(x) and m(x) to look like b(x, y), c(x, y). That's a lot of work though. And what happens if something *else* needs to get passed around?
• In an object oriented language, you could make x an object, and attach y to it, so that you mostly pass it around just as before.
• Along the same lines, you could make x a hash table, so that you can look arguments up by their name, to have some idea what they are.
• You could just make x a list, and remember what position everything is in. That's not so nice though if you have to revisit the code and you didn't document where everything is, and why it's there.
• Other things? You could make a(x)'s scope visible to m(x), I suppose, although that sounds like it's a fairly involved, and potentially hairy answer.

It seems the answers are basically either 1) add the parameter to all the functions and everything that calls them, or 2) use some kind of composite type to stash the one or more values in.
It seems that this is a reasonable common thing to have to do, so it must have a name. How well does your language handle that kind of refactoring?

Quite a few things have happened recently in the Debian/ARM land

ARM and Canonical have generously provided us with bunch of fast armel machines. Four of these are now as buildd's, bringing the total of armel buildd's to 7. In other words, armel should no longer lag behind other architectures when building unstable packages.

One of machines, abel.debian.org has been setup as a porter box. It is faster (around 3x) than the older porterbox (agricola). All Debian Developers have access to the porterboxes.

The new buildds have allowed us to enable more suites. Thanks to Philipp Kerns work, armel builds now experimental, lenny-volatile, lenny-backports as well as unstable/non-free. Especially if you are using stable Debian, access to backports and volatile should make life happier :)

Finally, the next big thing is Hardfloat ARM port, effort being lead by Konstantinos Margaritis. This doesn't mean that the armel port is going away. Majority of ARM cpus sold are still without FPU, so the softloaf port (armel) will still have a long life ahead. Meanwhile, the armhf port will provide a more optimal platform for people with bleeding edge ARM cores (ARMv7 + vfp). Some people have been unhappy with the new proposed new port, and various alternatives have been proposed. However, armhf is currently the only solution being actively worked on.

Update: thank canonical too

The unofficial rc bug tracker currently knows about the following bugs:

In Total:	401
Affecting Squeeze:	302
Squeeze only:	83
Remaining to be fixed in Squeeze:	219

Of these 219 bugs, the following tags are set:

Pending in Squeeze:	18
Patched in Squeeze:	43
Duplicates in Squeeze:	21
Contrib or non-free in Squeeze:	15
Claimed in Squeeze:	2
Delayed in Squeeze:	13
Can fixed in a security Update:	10
Otherwise fixed in Squeeze:	19

Ignoring all the above (multiple tags possible), 128 bugs need to be fixed by Debian Contributers to get Debian 6.0 Squeeze released. However, with the view of the Release Managers, 243 need to be dealt with for the release to happen. Please see my older post for an explanation of the different numbers.

People often want to know what an application looks like before they install it. This is one of the reason why Christoph Haas started with the screenshots.debian.net service. People can upload screenshots for their favorite applications for others to take a look at. Finally, as convenience feature, they are now added on packages.debian.org. When you click on the screenshot of a package you can see the list of all of the available ones. In the case when no screenshot is available yet the page will show a placeholder image which gives you a convenience link to the screenshots page of the package where you can submit one for the benefit of all. Please make sure that you are following the guidelines for submissions, otherwise your upload might get rejected. On a not totally unrelated topic, the packages.ubuntu.com site finally will also show you packages in maverick. It doesn't show screenshots yet if you think adding support for screenshots in there would be a good thing too, please let me know! Enjoy!

/debian permanent link Comments: 5

At DebConf10, there was a BOF entitled "DebConf & Debian", discussing about the similarities and differences of Debian and DebConf. At times this became a little bit heated as people wanted more integration and transparency.I don't think the differences are that great. I don't even see them as separate. I see DebConf as a Debian team, one that has to release every year on a short time frame. Things are rushed, things are not perfect, but we welcome anyone who wants to help.It's my goal to start a series of posts on DebConf, in the spirit of other teams documenting themselves and calling for contributors. I hope to let people know what's the current state of things, how you can get involved.If you want to get involved, join #debconf-team on OFTC, or subscribe to the debconf-team mailing list at http://lists.debconf.org/ . There are many people who would like to help you get involved. Right now (post-debconf) is a bit of an idle time, but one of the few times that big changes can be made. DebConf is a do-ocracy, if you'd like there to be a change, the best way is to help implement it. There may be some resistance, but we (at least I) will help to find a solution.These posts are my take on things, I can't claim to speak for the DebConf team. I don't know all of the history of DebConf, and I may get things wrong. I'll run these posts by the rest of the DebConf team for suggestions/corrections, but there may still be things wrong. You should consider anything useful to be thanks to the DebConf team, and anything wrong to be my mistake.I've only been involved in DebConf for two years. I first got involved as part of DebConf10, but spent most of 2009 working on global management things for DebConf9. This year, I spent more time with local on-the-ground tasks. I'd like to keep being involved next year - and hopefully keep improving, with your help.Tags: debconf, debian

I do love a bit of Geek Art and I loved what one of our LUGgers (whose name escapes me!) bought with them tonight s LUG Meeting A great addition to any condiment cupboard!

Debian package building scripts consist of several inter-related tools, which are documented well with manpages, but not in the code itself. Should you have some free time I urge you to take a peak at the source code of dpkg-buildpackage, debsign, and debuild. They work. They work well, but they're not shining examples of clear code are they? Given that they work, and that they are complex I'm not sure I'm interested in improving things, but I do have a personal desire to run something like:

debuild -sa --output=/tmp/results/

What would that do? Instead of placing the resulting package in the parent directory, which is what we have by default, it would place them in the named directory. However looking over the code I see that there are too many places where "../$file" is hardwired. e.g. Take a look at dpkg-buildpackage and you see:

my $chg = "../$pva.changes";
..
open OUT, '>', $chg or syserr(_g('write changes file'));

If you change that to the simple code below it works - but suddenly debsign fails to find the file:

my $dir = $ENV 'OUTPUT'    "..";
my $chg = $dir . "/" . $pva . ".changes";

Ugh. I guess I can come up with some hack to index files in the parent, run a build process, and move the only new files into place. But that's sleazy at best, and still doesn't solve the problem that we have tools which we rely upon which are .. unfriendly to changes.

I thought I could handle this via:

debuild \
 --post-dpkg-buildpackage-hook="move-changes-contents /tmp/output/ ../%p_%v_ dpkg-architecture -qDEB_HOST_ARCH .changes" \
 --no-tgz-check -sa -us -uc

(Here "move-changes-contents" would read the given .changes file and move both it and the contents listed in it to the specified directory.) Unfortunately %v corresponds to the package version - which isn't the same as the version in the .changes filename. (i.e. epochs are stripped out.)

ObQuote: I am the servant of the power behind the Nothing - The Neverending Story.

When Lars and I started work on Branchable, the Ikiwiki-based wiki and blog hosting service that we first discussed a year ago, and launched two weeks ago, we didn't know what software to use to build it. Probably, we assumed, we'd need a relational database. (Or maybe Couchdb, we wondered?) To manage the sysadmin jobs of keeping a slew of Ikiwiki sites running, we investigated using Puppet or Chef. (We worried that scaling Puppet or Chef to handle an arbitrary number of ikiwiki sites could be a problem.) For the user interface frontend, we assumed we'd be writing code in some kind of web application framework. Nearly all of that conventional wisdom turned out to be false. Instead, all the frontend code for Branchable runs as part of Ikiwiki. www.branchable.com is just another Branchable site, though one with some custom plugins that can make other sites and display a user's control panel. Other plugins handle things like the DNS setup interface. Those plugins are part of the first public site hosted on Branchable, ikiwiki-hosting.branchable.com. That wiki's git repository holds all the code, as well as the wiki for documentation and bug tracking. Also included in ikiwiki-hosting are some backend programs, like ikisite, which handles the high-level site management (creation, deletion, branching, etc), and is what we used instead of Chef. (Though we may use Chef for server deployment later.) Rather than use a relational database, we did out own little noSQL thing, by using a special, locked-down wiki as our database. That wiki is so special-purpose that it is not even accessible via the web! But it's managed by Branchable just like any other site. Deciding to not use a database was a watershed moment, because at that point we were fully dogfooding everything for Branchable. It's worked out well so far, possibly because our database needs are minimal. We have over a dozen other Ikiwiki sites in Branchable that we use either internally, or for other special purposes. Like this parked site that is branched every day to refresh sandbox.branchable.com. So, no database, no Puppet, lots of dogfooding. Did we use a web app framework? Not really. The closest we came is when developing Branchable highlighted that Ikiwiki needed a more user-friendly openid login interface, so I went off and adapted one, and of course added it into Ikiwiki proper. And that uses the JQuery framework. (I might write sometime about all the other Ikiwiki improvements that Branchable spurred.) We've done a lot of eating our own dogfood at Branchable, but I think it's all been pragmatic, not due to Not Invented Here syndrome. And the result is that Branchable is increasingly flexible in what its sites can do, and runs on a very simple architecture, with no odd special-cases for database servers, etc. It's just a bunch of Ikiwiki sites that work together.

I have some mp3 files in my collection. Some years ago, back in the times, when I was a Windows-User, I used "The Godfather", to keep the chaos under control. This software, although not Open Source, is really good at what it does. And it does almost everything to organize mp3s. From (mass-)tagging to auto-renaming to auto-sorting your mp3s. But this wouldn't be a blog entry by me if this became... a praise to a windows software so..

When I became a daily Linux User I searched for alternatives. With GUI or without a GUI, but I did not really find an application that suited my needs. As far as tagging and renaming was concerned it wasn't that hard. There are really excellent command line tools (lltag, id3, etc.) and rumours have been heard that there are also GUI tools.
But what I still did not find is a simple, yet flexible, tool to sort a huge collection of mp3s into a flexible structure on the filesystem like "The Godfather" was able to do since.. ehh.. a lot of years.

So the day before yesterday I finally decided to write one on my own and came up with an ~ 280 lines perl script (POD-documentation included) which does exactly what I want and is simple.
It does no tagging.
It does no renaming.
All it does is sorting mp3s into a given template-based directory hierarchy based on their ID3 tag.

At this point a warning is due. I'm sharing that script with you under the terms of the GPL. BUT it still needs testing and therefore you probably do not want to use it without its safety measures (e.g. dry-run or copy instead of move) to avoid data loss. And notice that although it has been written with portability in mind, it has only been tested on a Debian GNU/Linux system.

The script is hosted at github. Here (or raw to download it directly).

After I've talked to a co-worker about the script, he told me that arename would do what I want. So you probably don't want to use my script, because arename is probably more tested and much more sophisticated. But on the other hand I had a quick look at the arename manpage and its so utterly feature-loaded, that it cannot avoid a certain complexicity. My tool is simple.
And there is another advantage. If you want arename to handle a certain amount of mp3s you have to use your shell magic to find the files and pipe it to arename. My script finds all mp3s recursively from where you let it start (default is $PWD so be careful) and will happily move it into a hierarchy under a directory you ask it to. For the simple job of sorting mp3s its probably easier to use.

(Oh and if its worth nothing, it might still be of use as a simple programming example, how one could solve this problem in Perl)

I ve decided to try using local DNS caching again, using the simple dnsmasq. The problem is that, even including

prepend domain-name-servers 127.0.0.1;

in /etc/dhcp3/dhclient.conf, NetworkManager will overwrite /etc/resolv.conf, and remove

nameserver 127.0.0.1

from there. I found out in a comment here, that I could create a script in /etc/NetworkManager/dispatcher.d/ to make this line be included in the file. The script is simple

#!/bin/sh
set -e
cp /etc/resolv.conf /etc/resolv.conf.tmp
echo nameserver 127.0.0.1 > /etc/resolv.conf
cat /etc/resolv.conf.tmp >> /etc/resolv.conf
rm /etc/resolv.conf.tmp

and it s working .

At some point I had to debug an issue with some code I worked on in the past. It was using OpenLDAP to connect to an Active Directory server to get some information. At some point I got a report that the authentication failed with an ugly error.

Username is stored. Authenticating as domain\user.
Enter password:
ldap_search_ext: Operations error (1)
additional info: 00000000: LdapErr: DSID-0C090627, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, vece

I looked for the meaning of that error and after some staring at different possible explanations, I ended up on a page dealing with some python-ldap code. My code was in C, but it was clear it was the underlying library issuing the error message, so I looked at what it said:

Normally, this error indicates that you're attempting to bind anonymously, which Active Directory (sensibly) doesn't allow by default. We were supplying credentials to bind, though, and changing the base DN on the search to a sub-OU was all that was necessary to get the search to work. It turns out that python-ldap was binding anonymously, so the error was only sort of a red herring.

This was really strange because the authentication was actually done, as it was obvious from the messages and the traffic (analyzed with wireshark). Later in that post there were some hints that indicated that parts of the data might be stored on another server and the suggested fix was to instruct the library not to try chase referrals.

ldap.set_option(ldap.OPT_REFERRALS, 0)

I tried to see what was going on with our server using a ldapsearch command and at the end of the output there were some referrals specified.

[..]
# search reference
# refldap://ForestDnsZones.domeniu.ro/DC=ForestDnsZones,DC=domeniu,DC=ro

# search reference
# refldap://DomainDnsZones.domeniu.ro/DC=DomainDnsZones,DC=domeniu,DC=ro

# search reference
# refldap://domeniu.ro/CN=Configuration,DC=domeniu,DC=ro

# search result

# numResponses: 5
# numEntries: 1
# numReferences: 3

Bingo! So, after looking at the man page, I added this bit of code:

+ /* do not chase referrals */
+ if (ldap_set_option (ld,LDAP_OPT_REFERRALS,LDAP_OPT_OFF)!=LDAP_SUCCESS)  
+  ldap_perror(ld,"ldap_set_option");
+  return NULL;
+  
+

And then it worked. I hope this helps others that might be in the same situation as I was.

Having swapped details with many, many people at Debconf, and then been away for a week after that, I found myself with an overflowing mailbox and a long task of open mail, provide pass-phrase, pipe to gpg import . I wanted a way to batch-import all these signatures (there are three times as many, because my key has three UIDs) in one or two goes, and tidy up the stragglers later. David Bremner wrote a small Perl script to do this from an mbox file, but I wanted to work in pure shell and with mutt. Just shoving the mbox at gpg resulted in it decrypting one message, then bailing at the fact the IDEA plugin is not present. Here was my eventual workflow, which only requires you to provide the pass-phrase once:

1. create a maildir, either with maildir-make or a directory with cur, new and tmp directories nested inside;
2. mark all relevant messages as read, and save them to here (it doesn t matter if others get caught up in it);
3. now change to the maildir /cur directory, and run the following bash (disclaimer: totally untested and used at your own risk):
   
for a in ls ; do mv $a $a.gpg; done
gpg --decrypt-files *.gpg
rm *.gpg
gpg --import *
rm *

I expect there are better/quicker/safer ways to do it, but this worked well for me at midnight on a Monday evening. 19/08/10: Yes, it turns out I am a numpty, and Mutt can handle this all by itself with Ctrl-K and a tagged list. This is still quite handy when the private key is not on the machine you re using to read mail, though. Thanks for the corrections.

I m back. Where have I been? I was in Portishead, Plymouth and Taunton in Co-operatives Fortnight back in late June/early July. I seemed to spend July flat out with work trying to catch up again, then I fell ill just as I caught up (soon after cycling a 40-mile round trip to Portishead, but that wasn t the cause) and spent the rest catching up again. The last two weeks, I ve been busy travelling around internationally (some work, some sight-seeing), then catching up again. Of course, this has meant that other members of our co-op have been helping my clients more than usual, so haven t written here either. But now I m back on schedule again, clearing the backlog and writing a few things here. I ve quite a stack of things to write here. I ll pace myself and I ve some news to post first. This week is getting back into Koha again. It s good to be back on track.

This LWN article is perhaps a bit confused, but it does point out a truth: On modern hardware, with fast, local storage, and large, often static files, rsync is often unncessarily slow. That's probably true to some extent when using rsync across a LAN, but it's especially true when using it to copy files locally. rsync runs as a client/server pair, and both processes MD5 checksum the files as they are being transferred. That's nice, but it's slow too. Funny thing about rsync is that probably 50% of uses of it don't involve the core feature that it was written to provide: Updating files by transferring differences. Which, other than ensuring data validity, is the only reason it needs slow checksums. Instead, rsync is often chosen because of all the other awesome features that were glommed onto it over the past several decades. Compared with rsync, cp is pretty laughable -- it can't even exclude files from being copied by patterns. And unlike rsync, cp command lines are not often developed by repeated trial and error -- cp does not recover well from being ctrl-c'd in the middle, while rsync does. These kinds of things make a lot of us reach for rsync first, even if the situation does not involve incremental file changes. In most any situation, one of rsync's 120-some options is sure to be just what you need... So lots of scripts use rsync to synchronise directories, but the amount of speedup obtained by using the rsync algorithm is often low. (Correction: rsync turns off the delta-transfer algorythm by default for local to local transfers. It still does md5 checksums however.) Since rsync is reasonably fast, we generally don't care that it does these checksums that probably on average slow it down. But if larger files, like videos, are involved, this starts to change.. When rsync is run on a typical home NAS, with a slow (arm) CPU, the picture changes entirely -- now the checksum overhead is unbearable, while the IO overhead is minimal. So, here's local-rsync. It takes all the same options as rsync, with the caveat that SOURCE and DEST must be the first two options, and must be local directories.

% local-rsync huge/ /media/usb/huge/ -a -v --exclude '*~' --delete --max-delete=100

It speeds up rsync in these types of situations, by querying it to find what files need to be updated, and updating them the brute force way, with cp. At the end, rsync is run, to take care of the non-brute force stuff (like deletions and file permissions). On a fast CPU, local-rsync will probably speed up rsync of large, static files by a factor of 2. On a slow CPU, local-rsync is so fast, and rsync so slow, that I have not bothered to benchmark. :) This is just a hack (easily replaced by a --no-checksum option in rsync, of course). But I think it illustrates some interesting things about how a program's underlying assumptions about its environment can change over time, and how free software programs can accrete value until the original differentiating reason for their existence is not the most important thing about them anymore.