I think most people reading my blog would agree that backups are extremely important. So much important data is on computers these days: family photos, emails, financial records. So I take backups seriously.
A little while back, I purchased two identical 400GB external hard disks. One is kept at home, and the other at a safe deposit box in a bank in a different town. Every week or two, I swap drives, so that neither one ever becomes too dated. This process is relatively inexpensive (safe deposit boxes big enough to hold the drive go for $25/year), and works well.
I have been using
rdiff-backup to make these backups for several years now. (Since at least 2004, when I submitted a patch to make it record all metadata on MacOS X). rdiff-backup is quite nice. It is designed for storage to a hard disk. It stores on the disk a current filesystem mirror along with some metadata files that include permissions information. History is achieved by storing compressed rdiff (rsync) deltas going backwards in time. So restoring "most recent" files is a simple copy plus application of metadata, and restoring older files means reversing history. rdiff-backup does both automatically.
This is a nice system and has served me well for quite some time. But it has its drawbacks. One is that you always have to have the current image, uncompressed, which uses up lots of space. Another is that you can't encrypt these backups with something like gpg for storage on a potentially untrusted hosting service (say, rsync.net). Also, when your backup disk fills up, it takes forever to figure out what to delete, since rdiff-backup --list-increment-sizes must stat tens of thousands of files. So I went looking for alternatives.
The author of rdiff-backup actually wrote one, called
duplicity. Duplicity works by, essentially, storing a tarball full backup with its rdiff signature, then storing tarballs of rdiff deltas going forward in time. The reason rdiff-backup must have the full mirror is that it must generate rdiff deltas "backwards", which requires the full prior file available. Duplicity works around this.
However, the problem with duplicity is that if the full backup gets lost or corrupted, nothing newer than it can be restored. You must make new full backups periodically so that you can remove the old history. The other big problem with duplicity is that it doesn't grok hard links at all. That makes it unsuitable for backing up /sbin, /bin, /usr, and my /home, in which I frequently use hard links for preparing CD images, linking DVCS branches, etc.
So I went off searching out other projects and thinking about the problem myself.
One potential solution is to simply store tarballs and rdiff deltas going forward. That would require performing an entire full backup every day, which probably isn't a problem for me now, but I worry about the load that will place on my hard disks and the additional power it would consume to process all that data.
So what other projects are out there? Two caught my attention. The first is
Box Backup. It is similar in concept to rdiff-backup. It has its own archive format, and otherwise operates on a similar principle to rdiff-backup. It stores the most recent data in its archive format, compressed, along with the signatures for it. Then it generates reverse deltas similar to rdiff-backup. It supports encryption out of the box, too. It sounded like a perfect solution. Then I realized it
doesn't store hard links, device entries, etc., and has a design flaw that causes it to miss some changes to config files in /etc on Gentoo. That's a real bummer, because it sounded so nice otherwise. But I just can't trust my system to a program where I have to be careful not to use certain OS features because they won't be backed up right.
The other interesting one is
dar, the Disk ARchive tool, described by its author as the great grandson of tar -- and a pretty legitimate claim at that. Traditionally, if you are going to back up a Unix box, you have to choose between two not-quite-perfect options. You could use something like tar, which backs up all your permissions, special files, hard links, etc, but doesn't support random access. So to extract just one file, tar will read through the 5GB before it in the archive. Or you could use zip, which doesn't handle all the special stuff, but does support random access. Over the years, many backup systems have improved upon this in various ways. Bacula, for instance, is incredibly fast for tapes as it creates new tape "files" every so often and stores the precise tape location of each file in its database.
But none seem quite as nice as dar for disk backups. In addition to supporting all the special stuff out there, dar sports built-in compression and encryption. Unlike tar, compression is applied per-file, and encryption is applied per 10K block, which is really slick. This allows you to extract one file without having to decrypt and decompress the entire archive. dar also maintains a catalog which permits random access, has built-in support for splitting archives across removable media like CD-Rs, has a nice incremental backup feature, and sports a host of tools for tweaking archives -- removing files from them, changing compression schemes, etc.
But dar does not use binary deltas. I thought this would be quite space-inefficient, so I decided I would put it to the test, against a real-world scenario that would probably be pretty much a worst case scenario for it and a best case for rdiff-backup.
I track Debian sid and haven't updated my home box in quite some time. I have over 1GB of .debs downloaded which represent updates. Many of these updates are going to touch tons of files in /usr, though often making small changes, or even none at all. Sounds like rdiff-backup heaven, right?
I ran rdiff-backup to a clean area before applying any updates, and used dar to create a full backup file of the same data. Then I ran apt-get upgrade, and made incrementals with both rdiff-backup and dar. Finally I ran apt-get dist-upgrade, and did the same thing. So I have three backups with each system.
Let's look at how rdiff-backup did first.
According to rdiff-backup --list-increment-sizes, my /usr backup looks like this:
Time Size Cumulative size
-----------------------------------------------------------------------------
Sun Apr 13 18:37:56 2008 5.15 GB 5.15 GB (current mirror)
Sun Apr 13 08:51:30 2008 405 MB 5.54 GB
Sun Apr 13 03:08:07 2008 471 MB 6.00 GB
So what we see here is that we're using 5.15GB for the mirror of the current state of /usr. The delta between the old state of /usr and the state after apt-get upgrade was 471MB, and the delta representing dist-upgrade was 405MB, for total disk consumption of 6GB.
But if I run du -s over the /usr storage area in rdiff, it says that 7.0GB was used. du -s --apparent-size shows 6.1GB. The difference is that all the tens of thousands of files each waste some space at the end of their blocks, and that adds up to an entire gigabyte. rdiff-backup effectively consumed 7.0GB of space.
Now, for dar:
-rw-r--r-- 1 root root 2.3G Apr 12 22:47 usr-l00.1.dar
-rw-r--r-- 1 root root 826M Apr 13 11:34 usr-l01.1.dar
-rw-r--r-- 1 root root 411M Apr 13 19:05 usr-l02.1.dar
This was using bzip2 compression, and backed up the exact same files and data that rdiff-backup did. The initial mirror was 2.3GB, much smaller than the 5.1GB that rdiff-backup consumes. The apt-get upgrade differential was 826MB compared to the 471MB in rdiff-backup -- not really a surprise. But the dist-upgrade differential -- still a pathologically bad case for dar, but less so -- was only 6MB larger than the 405MB rdiff-backup case. And the total actual disk consumption of dar was only 3.5GB -- half the 7.0GB rdiff-backup claimed!
I still expect that, over an extended time, rdiff-backup could chip away at dar's lead... or maybe not, if lots of small files change.
But this was a completely unexpected result. I am definitely going to give dar a closer look.
Also, before I started all this, I converted my external hard disk from ext3 to XFS because of ext3's terrible performance with rdiff-backup.
Sunday was the 100th anniversary of the modern marathon: It was during the London
2008 Olympics that the marathon was run for the first time for the 26 miles
385 yard, or 42.185 km, distance. And it so happens that I was there to run
the 2008 edition of the London
Marathon.
Pretty nice weather at the start and finish: sunny, not too hot, occassional
clouds. But being London, we still managed to get drenched for about 30
minutes. Overall, the conditions were good -- or else Martin Lel would
have had a hard time for a
new course record.
The course is pretty nice, right from the start in Greenwhich all the way to
the spectacular finish in Westminster. Crowd support was good too, if only a
little uneven. But the second half of the race, and particularly the last
miles in the back from Canary Whard through the City to Westminster past
Parliament and Buckingham Palace were awesome. Lots of people, lots of noise.
Oh, and I even saw the leaders in a group of five, including Lel and Hall
as the course had parallel 'out' and 'back' tracks around miles 13 for me and
22 for them.
I had planned to take it easy and not try to run too hard, and aimed for a
time of around 3:25 and then finished at 3:24:41. A little less 'even' than
I had hoped, but still a very satisfying result. And for once the legs aren't
all that shot afterwards :)
And the whole weekend was nice as I got to stay with friends in the southwest
of London. Staying up late on both ends of the trip suppressed the jet lag
fairly well. At least that's what I keep telling myself.
We are at T-48h of 
And some of the infrastructure crew:
See you all there!
I sadly agree with 
The delays in granting accounts have other effects on Debian as a whole, besides possibly dissuading people from joining.
Even if i don't like talking about this in public, 
It seems spring has finally arrived in northern Germany and I'm laying in bed, trying to get rid of a flu. So in case you are waiting for something from me, you will have to wait a bit longer :-( 
It has now been more than 4 months since the last Debian developer account was created. 18 contributors have been through all steps, and are simply waiting for this simple administrative task to be done.
We are sending a terrible message to potential contributors. We have strong requirements on the technical level of our developers. During the new maintainer process, we ask them to answer about 80 questions about Debian. We ask them to do grunt work. We review their reports twice (New Maintainers’ Front Desk, then Debian Account Manager). But even after we are totally satisfied about what they did, even after they became more qualified than many of our current DDs, we ask them to wait for months, so that the only person allowed to create accounts can finally do his “job”.
It discourages the contributors currently in the NM process. I’ve seen several signs of frustration, or even depression. Some of them reduce their involvement in Debian, so we lose them before they even became Debian developers. Some of them consider resigning from the NM process. We should all feel guilty about that.
But it also discourages people from joining Debian. Instead, they go to other more welcoming projects, which is totally understandable. Debian isn’t the only distribution with developers from the community those days. There’s Gentoo, Ubuntu, Fedora, openSUSE. Some of those have nice programs for new contributors, like “school” sessions. Sure, Debian is the “biggest” distribution without a company behind it. But is independance worth all the trouble?
Of course, we have Debian Maintainers. DM is great for people who want to work on their packages. But, when we are trying to release lenny, we need more: people who are going to go through RC bugs, submitting patches. Who are going to do QA work. In short: people who care more about the whole distribution.
Can we afford not recruiting anybody? Can we run Debian with the current manpower? I don’t think so. There are more than 550 RC bugs in lenny, many packages are currently blocked from migrating to lenny because they are RC-buggy, and many packages are orphaned or neglected. There are also a lot of bugs which haven’t been filed yet (I asked for help with running piuparts, which would probably result in 100-200 new RC bugs, but nobody had time to help). Most of the work that needs to be done is not rocket science. We could use a lot more manpower. Currently, the same small set of developers is doing most of the grunt work. They will get tired too.
So, what can we do, as simple developers? There’s no magic solution, but we can try a few things.
No, the king id not dead. Still, congrats to 
So the question really comes down to asking the user sensible questions that they a) know how to answer and b) care about.
The point I made to Richard on IRC is that this is actually a difficult and multi-faceted problem and just hiding these problems won’t help in solving them well, so my take is that PackageKit should allow asking of questions, probably via a dbus interface, have a debconf frontend that channels questions by this channel, and then let the Ubuntu and Debian guys get on the case of forming projects and policy to help fix the problem at source.
Update 2:Richard pointed me to 
Habits control our choice sometimes more than we would like to admit…
New mobile phone
Since about two weeks ago I’ve a new mobile phone. The Nokia E51 will
replace my slowly dying Nokia 6130i.
I knew I needed a new mobile phone when my 6310i started to turn off
itself shortly after I turned it on. I needed up to about ten times
switching it on to make it stay on. Sometimes it already switched
itself off before I could enter the PIN. Looks like a loose contact,
but I never figured out where it is.
Although I know about Nokia’s behaviour in Germany, I still had to buy
a Nokia, because after using a 6130 (the GSM 1800 only clone of 6110
and 6150), a 6210i and the already mentioned 6130i over the last
decade, I got so used to how Nokia mobile phones are navigated and how
you type with Nokia phones (blank on 0, point and comma on 1, case
changing on #), everything else (especially those with blank on 0 and
case changing on *) would be worse than the half-dead mobile phone,
I’m currently using.
Spoilt for choice
So which Nokia? For a long time I refused to buy a mobile phone with a
camera or radio in it. But since the E70 was no more available (and is
said to have quite buggy software) and the E61 has been replaced with
the E61i, there are no more smartphones without a camera, at least not
from Nokia. But 
