Theodore Ts'o: Wanted: Incremental Backup Solutions that Use a Database
There are a number of very popular Open Source backup solutions that use a very clever hack of using hard link trees to maintain incremental backups. The advantage of such schemes is that they are very easy to code up, and it allows you to easily browse incremental backups by simply using cd and ls .
The disadvantage of such schemes is that it creates very large number of directories blocks which must be validated by an fsck operation. As I ve discussed previously, this causes e2fsck to consume a vast amount of memory; sometimes more than can be supported by 32-bit systems. Another problem which has recently been brought home to me, is how much time it can take to fsck such file systems.
This shouldn t have come as a surprise. Replicating the directory hierarchy for each incremental backup is perhaps the most inefficient way you could come up with for storing information for an incremental backup. The filenames are replicated in each directory hierarchy, and even if an entire subtree hasn t changed, the directories associated with that subtree must be replicated for each snapshot. As a result, each incremental snapshot results in a large number of additional directory blocks which must be tracked by the filesystem and checked by fsck. For example, in one very extreme case, a user reported to me that their backup filesystem contained 88 million inodes, of which 77 million of them were directories. Even if we assume that every directory was only a single block long, that still means that during e2fsck s pass 2 processing, 77 million times 4k, or 308 gigabytes, worth of directory blocks must be read into memory by e2fsck s pass 2. Worse yet, these 308 GB of directory blocks are scattered all across the filesystem, which means the time to simply read all of the directory blocks so they can be validated will take a very, very, long time indeed.
The real right way to do implement tracking incremental backups is to use a database, since it can much more efficiently store and organize the information of what file is located where, for each incremental snapshot. If the user wants to browse an incremental snapshot via cd and ls , this could be done via a synthetic FUSE filesysem. There s a reason why all of the industrial-strength, enterprise-class backup systems use real databases; it s the same reason why enterprise class databases use their own data files, and not try to store relational tables in a filesystem, even if the filesystem supports b-tree directories and efficient storage of small files. Purpose written-and-opimzied solutions can be far more efficient than general purpose tools.
So, can anyone recommend OSS backup solutions which track incremental backups using some kind of database? It doesn t really matter whether it s MySQL, Postgresql, SQLite, so long as it s not using/abusing a file systems hard links to create directory trees for each snapshot. For bonus points, there would be a way to browse a particular incremental snapshot via a FUSE interface. What suggestions can you give me, so I can pass it on to ext3/4 users who are running into problems with backup solutions such as Amanda, Backup-PC, dirvish, etc.?
Update: A feature which a number of these hard-link tools do right is that they do de-duplication; that is, they will create hard links between multiple files that have the same contents, even if they are located in different directories with different names (or they have been renamed or their directory structure reorganized since they were originally backed up) and even if the two files are from different clients. This basically means the database needs to include a checksum of the file, and a way to look up to see if the contents of that file have already been backed up, perhaps with a different name. Unfortunately, it looks like many of the tape-based tools, such as Bacula, assume that tape is cheap, so they don t have these sorts of de-duplication features. Does anyone know of non-hard-link backup tools that do de-dup and which are also Open Source?
I'm not sure how you can pre-announce something in a way that cannot be later faked.
The best I can imagine is you write it in a text file and post the size / hash of the file.
This is just to keep a reference (so I don t have to google it again :-) ): One can create a quick and dirty
As some people complained that Gammu does not build under
There's a lot of folklore in the Debian community about how apt works.
Unfortunately, a lot of it is also just plain wrong. For instance, I
often hear people confidently say things like 

Lessons learned: using the phrase third, final, and slightly
desperate is an excellent way to motivate people into reacting to
your Call for Talks. I've even had to reject some people now,
unfortunately.
Maybe next year, I'll use that phrase in my first call for
talks.
Then again, maybe not.
Anyway, of course that does mean that there are no slots
available anymore. If you wanted to hold a talk in the Debian devroom,
you've had plenty of opportunity and will now have to wait until next
year.
There are still some minor loose ends that need to be filled in (of
course there are always people who will wait until the very
last moment to send mails they need to send... including me), but once
that's done, I'll announce the schedule on the -events-eu mailinglist,
and probably here too. And if you're worried that the word 'desperate'
perhaps got us some boring talks in the end, don't be; the talks that
are on the schedule are, I hope, all very interesting. Watch this
space!
So, things left to do, in no particular order:

Yipee!
Just uploaded some very rough 


This code allows unattended ssh(1) processes, like the ones started by cron(8), to access the SSH keys stored by a running ssh-agent(1) (which will probably have been started in a screen(1) instance).
Be sure to understand the security implications of persistent ssh-agent processes and passwordless keys.

It's a new year, so why not a new