I think most people reading my blog would agree that backups are extremely important. So much important data is on computers these days: family photos, emails, financial records. So I take backups seriously.
A little while back, I purchased two identical 400GB external hard disks. One is kept at home, and the other at a safe deposit box in a bank in a different town. Every week or two, I swap drives, so that neither one ever becomes too dated. This process is relatively inexpensive (safe deposit boxes big enough to hold the drive go for $25/year), and works well.
I have been using
rdiff-backup to make these backups for several years now. (Since at least 2004, when I submitted a patch to make it record all metadata on MacOS X). rdiff-backup is quite nice. It is designed for storage to a hard disk. It stores on the disk a current filesystem mirror along with some metadata files that include permissions information. History is achieved by storing compressed rdiff (rsync) deltas going backwards in time. So restoring "most recent" files is a simple copy plus application of metadata, and restoring older files means reversing history. rdiff-backup does both automatically.
This is a nice system and has served me well for quite some time. But it has its drawbacks. One is that you always have to have the current image, uncompressed, which uses up lots of space. Another is that you can't encrypt these backups with something like gpg for storage on a potentially untrusted hosting service (say, rsync.net). Also, when your backup disk fills up, it takes forever to figure out what to delete, since rdiff-backup --list-increment-sizes must stat tens of thousands of files. So I went looking for alternatives.
The author of rdiff-backup actually wrote one, called
duplicity. Duplicity works by, essentially, storing a tarball full backup with its rdiff signature, then storing tarballs of rdiff deltas going forward in time. The reason rdiff-backup must have the full mirror is that it must generate rdiff deltas "backwards", which requires the full prior file available. Duplicity works around this.
However, the problem with duplicity is that if the full backup gets lost or corrupted, nothing newer than it can be restored. You must make new full backups periodically so that you can remove the old history. The other big problem with duplicity is that it doesn't grok hard links at all. That makes it unsuitable for backing up /sbin, /bin, /usr, and my /home, in which I frequently use hard links for preparing CD images, linking DVCS branches, etc.
So I went off searching out other projects and thinking about the problem myself.
One potential solution is to simply store tarballs and rdiff deltas going forward. That would require performing an entire full backup every day, which probably isn't a problem for me now, but I worry about the load that will place on my hard disks and the additional power it would consume to process all that data.
So what other projects are out there? Two caught my attention. The first is
Box Backup. It is similar in concept to rdiff-backup. It has its own archive format, and otherwise operates on a similar principle to rdiff-backup. It stores the most recent data in its archive format, compressed, along with the signatures for it. Then it generates reverse deltas similar to rdiff-backup. It supports encryption out of the box, too. It sounded like a perfect solution. Then I realized it
doesn't store hard links, device entries, etc., and has a design flaw that causes it to miss some changes to config files in /etc on Gentoo. That's a real bummer, because it sounded so nice otherwise. But I just can't trust my system to a program where I have to be careful not to use certain OS features because they won't be backed up right.
The other interesting one is
dar, the Disk ARchive tool, described by its author as the great grandson of tar -- and a pretty legitimate claim at that. Traditionally, if you are going to back up a Unix box, you have to choose between two not-quite-perfect options. You could use something like tar, which backs up all your permissions, special files, hard links, etc, but doesn't support random access. So to extract just one file, tar will read through the 5GB before it in the archive. Or you could use zip, which doesn't handle all the special stuff, but does support random access. Over the years, many backup systems have improved upon this in various ways. Bacula, for instance, is incredibly fast for tapes as it creates new tape "files" every so often and stores the precise tape location of each file in its database.
But none seem quite as nice as dar for disk backups. In addition to supporting all the special stuff out there, dar sports built-in compression and encryption. Unlike tar, compression is applied per-file, and encryption is applied per 10K block, which is really slick. This allows you to extract one file without having to decrypt and decompress the entire archive. dar also maintains a catalog which permits random access, has built-in support for splitting archives across removable media like CD-Rs, has a nice incremental backup feature, and sports a host of tools for tweaking archives -- removing files from them, changing compression schemes, etc.
But dar does not use binary deltas. I thought this would be quite space-inefficient, so I decided I would put it to the test, against a real-world scenario that would probably be pretty much a worst case scenario for it and a best case for rdiff-backup.
I track Debian sid and haven't updated my home box in quite some time. I have over 1GB of .debs downloaded which represent updates. Many of these updates are going to touch tons of files in /usr, though often making small changes, or even none at all. Sounds like rdiff-backup heaven, right?
I ran rdiff-backup to a clean area before applying any updates, and used dar to create a full backup file of the same data. Then I ran apt-get upgrade, and made incrementals with both rdiff-backup and dar. Finally I ran apt-get dist-upgrade, and did the same thing. So I have three backups with each system.
Let's look at how rdiff-backup did first.
According to rdiff-backup --list-increment-sizes, my /usr backup looks like this:
Time Size Cumulative size
-----------------------------------------------------------------------------
Sun Apr 13 18:37:56 2008 5.15 GB 5.15 GB (current mirror)
Sun Apr 13 08:51:30 2008 405 MB 5.54 GB
Sun Apr 13 03:08:07 2008 471 MB 6.00 GB
So what we see here is that we're using 5.15GB for the mirror of the current state of /usr. The delta between the old state of /usr and the state after apt-get upgrade was 471MB, and the delta representing dist-upgrade was 405MB, for total disk consumption of 6GB.
But if I run du -s over the /usr storage area in rdiff, it says that 7.0GB was used. du -s --apparent-size shows 6.1GB. The difference is that all the tens of thousands of files each waste some space at the end of their blocks, and that adds up to an entire gigabyte. rdiff-backup effectively consumed 7.0GB of space.
Now, for dar:
-rw-r--r-- 1 root root 2.3G Apr 12 22:47 usr-l00.1.dar
-rw-r--r-- 1 root root 826M Apr 13 11:34 usr-l01.1.dar
-rw-r--r-- 1 root root 411M Apr 13 19:05 usr-l02.1.dar
This was using bzip2 compression, and backed up the exact same files and data that rdiff-backup did. The initial mirror was 2.3GB, much smaller than the 5.1GB that rdiff-backup consumes. The apt-get upgrade differential was 826MB compared to the 471MB in rdiff-backup -- not really a surprise. But the dist-upgrade differential -- still a pathologically bad case for dar, but less so -- was only 6MB larger than the 405MB rdiff-backup case. And the total actual disk consumption of dar was only 3.5GB -- half the 7.0GB rdiff-backup claimed!
I still expect that, over an extended time, rdiff-backup could chip away at dar's lead... or maybe not, if lots of small files change.
But this was a completely unexpected result. I am definitely going to give dar a closer look.
Also, before I started all this, I converted my external hard disk from ext3 to XFS because of ext3's terrible performance with rdiff-backup.
I spend too much of my time trying to talk with
people who don't want to discuss, yet somehow
I won't stop.
I keep hoping that things like
suggesting good advice on meeting scheduling
will avoid them repeating old mistakes.
The most extreme life-and-death example
is probably trying to help with
Kewstoke Toll Road, where people still speed and someone crashed off again last night
(although I don't know what caused last night's crash - could have been a simple accident).
Of course, it's better to talk with people who
have asked questions and want to hear the
answer.
I'm currently involved in several groups
like that and it makes me
much happier [4HWW].
I've even
made a confidentiality agreement [Network Blogging article]
about one group because I really like
the organiser and want to help them,
but I've yet to see
changes happen because of it and that's
probably about all I can write here, which
does rather suck.
But talking of changes that make me happy,
I spotted that
Bristol Wireless
has now gone further than
TTLLP
by deciding to change people away from Microsoft Windows when they find it:-
Last month we had a 
It happens that when I get burnt out by Debian (I might blog about that later, but then I just might not) I am forced to face the real world just to learn that Debian was not so bad to begin with. Oh, the irony! I need to smoke more so that I spend less time involved in this crap nobody can change. It is hard enough to change Debian, I don't even want to think about the rest.
It has now been more than 4 months since the last Debian developer account was created. 18 contributors have been through all steps, and are simply waiting for this simple administrative task to be done.
We are sending a terrible message to potential contributors. We have strong requirements on the technical level of our developers. During the new maintainer process, we ask them to answer about 80 questions about Debian. We ask them to do grunt work. We review their reports twice (New Maintainers’ Front Desk, then Debian Account Manager). But even after we are totally satisfied about what they did, even after they became more qualified than many of our current DDs, we ask them to wait for months, so that the only person allowed to create accounts can finally do his “job”.
It discourages the contributors currently in the NM process. I’ve seen several signs of frustration, or even depression. Some of them reduce their involvement in Debian, so we lose them before they even became Debian developers. Some of them consider resigning from the NM process. We should all feel guilty about that.
But it also discourages people from joining Debian. Instead, they go to other more welcoming projects, which is totally understandable. Debian isn’t the only distribution with developers from the community those days. There’s Gentoo, Ubuntu, Fedora, openSUSE. Some of those have nice programs for new contributors, like “school” sessions. Sure, Debian is the “biggest” distribution without a company behind it. But is independance worth all the trouble?
Of course, we have Debian Maintainers. DM is great for people who want to work on their packages. But, when we are trying to release lenny, we need more: people who are going to go through RC bugs, submitting patches. Who are going to do QA work. In short: people who care more about the whole distribution.
Can we afford not recruiting anybody? Can we run Debian with the current manpower? I don’t think so. There are more than 550 RC bugs in lenny, many packages are currently blocked from migrating to lenny because they are RC-buggy, and many packages are orphaned or neglected. There are also a lot of bugs which haven’t been filed yet (I asked for help with running piuparts, which would probably result in 100-200 new RC bugs, but nobody had time to help). Most of the work that needs to be done is not rocket science. We could use a lot more manpower. Currently, the same small set of developers is doing most of the grunt work. They will get tired too.
So, what can we do, as simple developers? There’s no magic solution, but we can try a few things.
I got my 
So the question really comes down to asking the user sensible questions that they a) know how to answer and b) care about.
The point I made to Richard on IRC is that this is actually a difficult and multi-faceted problem and just hiding these problems won’t help in solving them well, so my take is that PackageKit should allow asking of questions, probably via a dbus interface, have a debconf frontend that channels questions by this channel, and then let the Ubuntu and Debian guys get on the case of forming projects and policy to help fix the problem at source.
Update 2:Richard pointed me to 
I'm still following The Lost Ring, but it got a little too intense
today when someone who was not very familiar with what was going on was
recruited to pick up an envolope hidden in an umbrella stand at a gallery
in Hong Kong became disturbed by the contents and destroyed it. All we're
left with is some fuzzy pictures.
This fragment contains a code. If we can figure out enough of the code, it
can be used to download a digital copy of the paper that got torn up. It's
a kind of checksum, was written with a typewriter (that writes capital
letters a bit higher than the rest of the line, and tends to fade bottoms
of numbers), and will look something like
"
Habits control our choice sometimes more than we would like to admit…
New mobile phone
Since about two weeks ago I’ve a new mobile phone. The Nokia E51 will
replace my slowly dying Nokia 6130i.
I knew I needed a new mobile phone when my 6310i started to turn off
itself shortly after I turned it on. I needed up to about ten times
switching it on to make it stay on. Sometimes it already switched
itself off before I could enter the PIN. Looks like a loose contact,
but I never figured out where it is.
Although I know about Nokia’s behaviour in Germany, I still had to buy
a Nokia, because after using a 6130 (the GSM 1800 only clone of 6110
and 6150), a 6210i and the already mentioned 6130i over the last
decade, I got so used to how Nokia mobile phones are navigated and how
you type with Nokia phones (blank on 0, point and comma on 1, case
changing on #), everything else (especially those with blank on 0 and
case changing on *) would be worse than the half-dead mobile phone,
I’m currently using.
Spoilt for choice
So which Nokia? For a long time I refused to buy a mobile phone with a
camera or radio in it. But since the E70 was no more available (and is
said to have quite buggy software) and the E61 has been replaced with
the E61i, there are no more smartphones without a camera, at least not
from Nokia. But 
Just now, a movie was on TV. This was a non-fiction film, but it was
still a story, rather than just a simple documentary or something; it
only wasn't scripted or anything similar. The film is about a group of
shepherds in Mongolia, who not only own sheep, but also camels, for
transport. One of these camels gave birth to a rare white camel;
however, the mother does not accept the young for some reason, despite
the shepherds' best efforts.
Eventually, however, they did succeed in calming the mother, and
persuading her to accept the young. What really blew me away, however,
is how they did that:
By using music. Incredible.
So, next time you need to calm down a camel, here is the
procedure:
It seems I've been elected to be DPL for the next year, starting
this coming Thursday (18th April) according to Manoj. Thanks to the
people that voted for me, and also to the other two candidates -
better luck next time to Marc and Raphael. And (of course) thanks to
Sam as well for his excellent efforts over the last year.
I hope to do a good job for you all this year, and I have a few
plans on what I'm going to do. More concrete details once I'm actually
officially in charge... :-)
In mid-February, I started the lenny l10n NMU campaign, to tackle down
pending debconf l10n bug reports.