I ll make 2 presentations about Config::Model applications. One on configuration upgrades. The other one is more presenting ideas on sharing packaging work between distributions. Note that I ll make these presentations quite late Sunday and I ll have to leave about 30mns after the last presentation. So if you already have questions about Config::Model, it would be better to contact me before the presentations. See you there

For those of you who follow me through anything other than Planet Debian (who will surely tolerate this repetition): Interesting but frightful things are happening in North African countries. We saw what happened in Tunisia some days ago. Now, Egypt is a way larger, way more populous... And, dare I say, way more important country. Also, it is a country which borders Israel, where I lived in for ~18 months, and to which although I am today unaffiliated is very important for me. And what happens in Egypt will surely shape the whole mid-East. So, here goes a copy of Mohammed Sameer's post Not just a link to it, as he might have to take it down. Of course, Mohammed, if you in any way think I should delete this, please please tell me, and I will comply immediately.

Egypt has been fighting for freedom already for 3 days now. The whole country has experienced a large number of street demonstrations and protests within the last 3 days to be continued also on Friday, the 28th of January and afterwards. The protests' main goal is to oust dictator Hosni Mubarak's regime, which has been in power for almost 30 years. We want him out. A massacre has happened in Suez. Police used live ammunition and tear gas. There are unconfirmed rumors that the army might interfere. Even a bigger protest is supposed to take place a few hours from now. After the Prayer on Friday. The internet has been shutdown completely. Egypt is no longer online since Friday, the 28th of January 00:45 AM. Text messages to cell phones have been cut off too and all cell phones services will be following. No one knows exactly the intentions of the regime but it doesn't sound good. Please help us.
Please blog about it in English and in all languages.
Please spread the news everywhere.
Please talk to media.
Please petition your government if that will help. If there's anything that you can do, please do it and help us save the country and the people. More news from twitter #jan25.

...From what I read from Israeli sources, what worries them and many people around the world is that, although Egypt is formally a democratic country, the ruling party has seen towards having basically no oposition Besides the stubborn Islamic Brotherhood (very akin to Hamas). Now, the Islamic Brotherhood (as well as Hamas, and as well as most other Islamic regimes I know off, although I might be way mistaken) have got huge popular support because... They are true to their beliefs. In short, they are not corrupt Something that cannot be said about many political organizations around the globe. They are true to their word. And although many of us shiver at thinking of their word getting more power... It is only the Egyptians who should have a say on who rules Egypt. I honestly hope the Egyptian people get their long-owed self-determination. Of course, I hope Tunisians also get it, and not just a mirage of it. And every people, including us poor Mexicans who have neither had a chance to rule our destiny. And I hope it is a change for good, for tolerance, for peaceful coexistance with the neighbours, even if they are in many ways the rivals. [Update] I just found out that, as expected, the news have reached Slashdot: Egypt shuts off all Internet access. I'm sure you will find lots of more information there.

I love revision control. I love the ability to track changes over time, whether that be to see why I changed something in the past, or to see why a particular thing has stopped working, or to see if a particular thing is fixed in a more recent version than the one I'm using. However I have a few opinions about the use of revision control that are obviously not shared by other people. Here are a few of them:

• One change per changeset. The only argument I can see against this is laziness. Changesets are cheap. Checking in multiple things in a single go makes it hard to work out exactly which piece of code fixes which problem. I'm fine with a big initial drops of code if logically it all needs to go together, but changesets that bundle up half a dozen different fixes piss me off.
• Descriptive changeset comments. Don't make me guess what you changed. Tell me. Bug numbers are not sufficient (though including them is really helpful).
• Comments in the changeset, not per file. I've only seen this with BitKeeper; you can have per file comments and then an overall changeset comment. At first I thought this was quite neat, because you can explain each part of a change. Now it just annoys me, because I want the relevant detail in one place rather than having to drill down to a per file level to figure out what's going on.
• The tree should always compile. There are people I respect who are all for checking in all the time throughout development no matter what the status. I have to disagree, at least for anything that's available to other people. The tree should always compile. This avoids pissing off your coworkers (especially if they're in a different timezone) and means you can do things like git bisect more easily. Plus it shows you've at least done minimal testing.
• Don't hide your tree. I like centralised locations for master trees. It means I can make an educated guess about where to look first for information about changes. Trees that live in obscure network shares or, worse, someone's home directory aren't helpful. While I may not always agree with the choice of VCS for the centralised service as long as it's actually fit for purpose I think it makes much more sense to use it than go off on a separate path that's less obvious for others to find.

Just a few days after OpenSSH 5.7 was released upstream, our (Debian s as well as Ubuntu s) tireless OpenSSH and GRUB maintainer Colin Watson uploaded a first package of OpenSSH 5.7 to Ubuntu Natty and to Debian Experimental. Besides the obvious new thing, the implementation of Elliptic Curve Cryptography which promises better speed and shorter keys while staying at the same level of security, one other item of his changelog entry stuck out and caught my attention:

• scp(1): Add a new -3 option to scp: Copies between two remote hosts are transferred through the local host.

That s something I always wondered why it didn t just work . While it still doesn t seem to detect such a situation by default, it s now at least possible to copy stuff from on remote box to another without ugly port forwarding and tunneling hacks. Further cool stuff in the changelog:

• sftp(1)/sftp-server(8): add a protocol extension to support a hard link operation. It is available through the ln command in the client. The old ln behaviour of creating a symlink is available using its -s option or through the preexisting symlink command.

Colin++

Egypt has been fighting for freedom already for 3 days now.

The whole country has experienced a large number of street demonstrations and protests within the last 3 days to be continued also on Friday, the 28th of January and afterwards.

The protests' main goal is to oust dictator Hosni Mubarak's regime, which has been in power for almost 30 years.

We want him out.

A massacre has happened in Suez. Police used live ammunition and tear gas. There are unconfirmed rumors that the army might interfere.

Even a bigger protest is supposed to take place a few hours from now. After the Prayer on Friday.

The internet has been shutdown completely. Egypt is no longer online since Friday, the 28th of January 00:45 AM.

Text messages to cell phones have been cut off too and all cell phones services will be following. No one knows exactly the intentions of the regime but it doesn't sound good.

Please help us.
Please blog about it in English and in all languages.
Please spread the news everywhere.
Please talk to media.
Please petition your government if that will help.

If there's anything that you can do, please do it and help us save the country and the people.

More news from twitter #jan25. read more

Are you graduating this year? Or recently graduated? Do you want to work in Open Source? Do you have the right to work in the UK? Do you want to work with some of the best minds in the field: Chris Lord, of Happy Wombats fame; Damien Lespiau, the Clutter GST mastermind; Emmanuele Bassi, our Clutter super-hero; Jussi Kukkonen, who puts the clue in Geoclue; Ross Burton, our EDS magician; Srini Ragavan, Evolution shiny-thing maker; Thomas Wood, of the MX and control-center massive; Tomas Frydrych, our Antarctic naming scheme generator and of course pippin. Interested? Take a look at our job entry. I should be around as FOSDEM so feel free to corner me to talk.

I've (again) changed schedule of what to implement and I've spend fourth Hackweek 6 day by doing more cleanup and documentation work than real coding. The major new feature is support for direct installation of the applet to the phone, so the installation is now easier. There were minor improvements on the both sides to allow support for calls and deleting of SMS messages. Adding calendar and todo entries is still left for future (hopefully tomorrow). I've done (more or less) final cleanup of my changes in the applet itself and I've sent changes back upstream hoping for merging them (I'd give you link to the mail, but the archive does not work). Rest of the time was spent collecting feedback from users testing the code (surprisingly there are already few brave users who gave it a try). It resulted in few bug fixes and improved documentation. Last but not least - all code is is available in our Git repository and check documentation for information how to set it up.

Filed under: English Gammu Suse 0 comments Flattr this!

(Update: Link to Tomeu s blog post, repost for planet.gnome.org) Last week I was in Prague to attend the GNOME/Python 2011 Hackfest for gobject-introspection, to which Tomeu Vizoso kindly invited me after I started working with PyGI some months ago. It happened at a place called brmlab which was quite the right environment for a bunch of 9 hackers: Some comfy couches and chairs, soldering irons, lots of old TV tubes, chips, and other electronics, a big Pirate flag, really good Wifi, plenty of Club Mate and Coke supplies, and not putting unnecessary effort into mundane things like wallpapers. It was really nice to get to know the upstream experts John (J5) Palmieri and Tomeu Vizoso (check out Tomeu s blog post for his summary and some really nice photos). When sitting together in a room, fully focussing on this area for a full week, it s so much easier to just ask them about something and getting things done and into upstream than on IRC or bugzilla, where you don t know each other personally. I certainly learned a lot this week (and not only how great Czech beer tastes )! So what did I do? Application porting After already having ported four Ubuntu PyGTK applications to GI before (apport, jockey, aptdaemon, and language-selector),
my main goal and occupation during this week was to start porting a bigger PyGTK application. I picked system-config-printer, as it s two magnitudes bigger than the previous projects, exercises quite a lot more of the GTK GI bindings, and thus also exposes a lot more GTK annotation and pygobject bugs. This resulted in a new pygi s-c-p branch which has the first 100 rounds of test, break, fix iterations. It now at least starts, and you can do a number of things with it, but a lot of functionality is still broken. As a kind of finger exercise and also to check for how well pygi-convert works for small projects now, I also ported computer-janitor. This went really well (I had it working after about 30 minutes), and also led me to finally fixing the unicode vs. str mess for GtkTreeView that you got so far with Python 2.x. pygobject and GTK fixes Porting system-config-printer and computer-janitor uncovered a lot of opportunities to improve pygi-convert.sh, a big perl -e kind of script to do the mechanical grunt work of the porting process. It doesn t fix up changed signatures (such as adding missing arguments which were default arguments in PyGTK, or the ubiquitous user_data argument for signal handlers), but at least it gets a lot of namespaces, method, and constant names right. I also fixed three annotation fixes in GTK+. We also collaboratively reviewed and tested Pavel s annotation branch which helped to fix tons of problems, especially after Steve Fr cinaux s excellent reference leak fix, so if you play around with current pygobject git head, you really also have to use the current GTK+ git head. Speaking of which, if you want to port applications and always stay on top of the pygobject/GTK development without having to clutter your package system with make install s of those, it works very well to have this in your ~/.bashrc:

export GI_TYPELIB_PATH=$HOME/projects/gtk/gtk:$HOME/projects/gtk/gdk
export PYTHONPATH=$HOME/projects/pygobject

Better GVariant/GDBus support The GNOME world is moving from the old dbus-glib python bindings to GDBus, which is integrated into GLib. However, dbus-python exposed a really nice and convenient way of doing D-Bus calls, while using GDBus from Python was hideously complicated, especially for nontrivial arguments with empty or nested arrays:

from gi.repository import Gio, GLib
from gi._gi import variant_type_from_string
d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)
vb = GLib.VariantBuilder()
vb.init(variant_type_from_string('r'))
vb.add_value(GLib.Variant('s', 'test'))
vb.add_value(GLib.Variant('u', 1))
vb.add_value(GLib.Variant('s', 'gtk-ok'))
vb.add_value(GLib.Variant('s', 'Hello World!'))
vb.add_value(GLib.Variant('s', 'Subtext'))
# add an empty array
eavb = GLib.VariantBuilder()
eavb.init(variant_type_from_string('as'))
vb.add_value(eavb.end())
# add an empty dict
eavb = GLib.VariantBuilder()
eavb.init(variant_type_from_string('a sv '))
vb.add_value(eavb.end())
vb.add_value(GLib.Variant('i', 10000))
args = vb.end()
result = notify.call_sync('Notify', args, 0, -1, None)
id = result.get_child_value(0).get_uint32()
print id

So I went to making the GLib.Variant constructor work properly with nested types and boxed variants, adding Pythonic GVariant iterators and indexing (so that you can treat GVariant dictionaries/arrays/tuples just like their Python equivalents), and finally a Variant.unpack() method for converting the return value of a D-Bus call back into a native Python data type. This looks a lot friendlier now:

from gi.repository import Gio, GLib
d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)
args = GLib.Variant('(susssasa sv i)', ('test', 1, 'gtk-ok', 'Hello World!',
    'Subtext', [],  , 10000))
result = notify.call_sync('Notify', args, 0, -1, None)
id = result.unpack()[0]
print id

I also prepared another patch in GNOME#640181 which will provide the icing on the cake, i. e. handle the variant building/unpacking transparently and make the explicit call_sync() unnecessary:

from gi.repository import Gio, GLib
d = Gio.bus_get_sync(Gio.BusType.SESSION, None)
notify = Gio.DBusProxy.new_sync(d, 0, None, 'org.freedesktop.Notifications',
    '/org/freedesktop/Notifications', 'org.freedesktop.Notifications', None)
result = notify.Notify('(susssasa sv i)', 'test', 1, 'gtk-ok', 'Hello World!',
            'Subtext', [],  , 10000)
print result[0]

I hope that I can get this reviewed and land this soon. Thanks to our sponsors! Many thanks to the GNOME Foundation and Collabora for sponsoring this event!

Third Hackweek 6 day went a bit differently than I planned, but there is still lot of progress. I've added some more calls to the Series60 remote protocol, so that some things now work much faster (eg. getting number of contacts, calendar entries). The applet now also has a function to toggle Bluetooth so it does not have to be done elsewhere in phone menus. On Gammu side I focused on messaging support, so now Gammu can list messages in the phone or send new message. Modification of calendar entries is left for tomorrow, hopefully I will get to it and the support will be more or less complete and I will have Friday for final cleanup. And as usual - all code is is available in our Git repository and check documentation for information how to set it up.

Filed under: English Gammu Suse 2 comments Flattr this!

Like last year, here s a list of Debian related projects that I d like to tackle this year. I might not do them all, but I like to write them down, I am more productive when I have concrete objectives.

1. Translate my Debian book into English.
   I will run a fundraising campaign with Ulule.com and if enough people are interested, I will spend a few months with Roland Mas to translate the book. Hopefully this project can be completed until the summer.
2. Finish multiarch support in dpkg.
   I m working on this with Guillem Jover right now, thanks to Linaro who is sponsoring my work.
3. Make deb files use XZ compression by default.
   It s a simple change in dpkg-deb and it would literally save gigabytes of space on mirrors. It s particularly important so that single CD/DVD can have a more complete set of software. #556407 (on DAK) needs to be fixed first though and a project-wide discussion is in order. Some archs might want to have a different default.
4. Be more reactive to review/merge dpkg patches.
   While we welcome help, we have a backlog of patches sitting in the BTS and it happened several times that we failed to review/merge submitted branches in a decent time. It s very frustrating for the contributor. I already tried to improve the situation by creating the Review/Merge Queue but nobody stepped up to help review and update the patches.
   As I am getting more familiar with the C part of dpkg, I hope to be able to jump in when Guillem doesn t have the time or the interest.
5. Implement the rolling distribution proposed as part of the CUT project and try to improve the release process.
   I really want the new rolling distribution but it will inevitably have an impact on the release process. It can t be done as a standalone project. I would also like to see progress in the way we deal with transitions (see discussion here).
6. Work more regularly on the developers-reference.
   Hopefully I will be able to combine this with my blog writing activities, i.e. write blog articles on the topics that the developers-reference shall cover and then adapt my articles with some docbook markup.

To the above list, I shall add a few supplementary goals related to funding my Debian work:

1. Write a 10-lesson course called Smart Package Management .
   It will delivered by email to my newsletter subscribers.
2. Create an information product (most likely an ebook or an online training) and sell it on my blog.
   The precise topic has not yet been defined although I have a few ideas. Is there something that you would like me to teach you? Leave your suggestions in a comment.
3. By the end of the year, have at least 1/3 of my time funded by donations and/or earnings of my information products.
   More concretely it means 700 each month or a 9-fold increase compared to the current income level (around 80 /month mostly via Flattr).

That makes up lots of challenges for this year. You can help me reach those goals, join my Debian Supporters Guild and you ll be informed every time that I start something new or when you can help in specific ways.

6 comments Liked this article? Click here. My blog is Flattr-enabled.

The easiest way to compress the data that is being served to the visitors of your web application is to make use of mod_deflate. Once you have enabled that module and provided it with a suitable configuration file, it will compress all releant files on the fly as it is serving them.

Given that I was already going to minify my Javascript and CSS files ahead of time (i.e. not using mod_pagespeed), I figured that there must be a way for me to serve gzipped files directly.

"Compiling" Static FilesI decided to treat my web application like a c program. After all, it starts as readable source code and ends up as an unreadable binary file.

So I created a Makefile to minify and compress all CSS and Javascript files using YUI Compressor and gzip:

all: build

build:
     find static/css -type f -name "[^.]*.css" -execdir yui-compressor -o  .css   \;
     find static/js -type f -name "[^.]*.js"  -execdir yui-compressor -o  .js   \;
     cd static/css && for f in *.css.css ; do gzip -c $$f >  basename $$f .css .gz ; done
     cd static/js && for f in *.js.js ; do gzip -c $$f >  basename $$f .js .gz ; done

clean:
     find static/css -name "*.css.css" -delete
     find static/js -name "*.js.js" -delete
     find static/css -name "*.css.gz" -delete
     find static/js -name "*.js.gz" -delete
     find -name "*.pyc" -delete

This leaves the original files intact and adds minified .css.css and .js.js files as well as minified and compressed .css.gz and .js.gz files.

How browsers advertise gzip supportThe nice thing about serving compressed content to browsers is that browsers that support receiving gzipped content (almost all of them nowadays) include the following HTTP header in their requests:

Accept-Encoding = gzip,deflate

(Incidently, if you want to test what non-gzipped enable browsers see, just browse to about:config and remove what's in the network.http.accept-encoding variable.)

Serving compressed files to clientsTo serve different files to different browsers, all that's needed is to enable Multiviews in our Apache configuration (as suggested on the Apache mailing list):

<Directory /var/www/static/css>
 AddEncoding gzip gz
 ForceType text/css
 Options +Multiviews
 SetEnv force-no-vary
 Header set Cache-Control "private"
</Directory>

<Directory /var/www/static/js>
 AddEncoding gzip gz
 ForceType text/javascript
 Options +Multiviews
 SetEnv force-no-vary
 Header set Cache-Control "private"
</Directory>

The ForceType directive is there to force the mimetype (as described in this solution) and to make sure that browsers (including Firefox) don't download the files to disk.

As for the SetEnv directive, it turns out that on Internet Explorer, most files with a Vary header (added by Apache) are not cached and so we must make sure it gets stripped out before the response goes out.

Finally, the Cache-Control headers are set to private to prevent intermediate/transparent proxies from caching our CSS and Javascript files, while allowing browsers to do so. If intermediate proxies start caching compressed content, they may incorrectly serve it to clients without gzip support.

update-apt-xapian-index on other distros I've drafted a little HOWTO on using apt-xapian-index on non-Debian distributions. The procedure has been tried on Mageia with some success, and there's no reason it wouldn't work everywhere else: the index itself does not depend on anything distro-specific.

I have just sent a mail to the Debian/Ruby Extras mailing list stating that I will also give up most of my Ruby packaging efforts and reduce my involvement in the team. I am following Lucas Nussbaum in his decision. Although our reasons are slightly different, I subscribe most of what he says in his blog post. Besides that, I haven t done much Ruby programming in the past years, making me lose the motivation for the huge load of packages I am responsible for and thus subconsciously perceive that as a burden, as I feel guilty not having/making time to work on them. So, what does this mean? I will handover (co)maintainership of over 50 libraries. Many of which have dead upstream, unfortunately. Also will I be looking for someone to take over the administrative tasks of heading the team such as mailing list moderation and Alioth repository access. I will keep maintaining some programs that I still use, such as Camping, and am approachable for (mentor) uploads. Meanwhile I will continue my work on my PhD thesis, which is progressing well and should lead to my defence (hopefully) somewhere around September this year. That said, I will try to keep tinkering with Ruby and other F/OSS projects. Oh, and also to get more motivated and some inspiration

When my wife bought her HP 2605dn color laser, she had also ordered a set of replacement cartridges because the originals would supposedly not last very long. It turned out that they would last over two years, though. Now, the black cartridge was empty. The replacements are original HP components, but recycled/refilled second hand from a different source. I put in the black one, and the trouble started with the new drum having a big scratch that would appear in every print. After two years, returning the cartridge was not an option. As the old drum was still good, the idea was to put the new toner into the old cartridge. A bit of googling (and visual inspection) quickly revealed that you are not supposed to do this, the device doesn't have any facilities to access anything. Some post did indicate though that the cartridge actually consists of two parts that are only held together by two pivot pins and two small springs. If you pull the two parts apart a bit, you can feel the axis the parts are moving around. So, I would try to use the old drum part with the new toner container part. Removing the springs is easy (also removing the flap covering the drum, along with a tiny third spring). The two pivot pins are pretty inaccessible, but I managed to pull out both using a pointy tong. One pin is made of metal, if you need more space you can drill a small hole right next to it. The other pin is made of plastic with a small hole in it, in one case I succeeded by screwing in a little screw and pulling at it. Reassembling the parts is not so hard. I put the "new" cartridge into the printer, waited for the initialization to finish, only to find out the printer would still think the cartridge was empty and should be replaced. Duh. I then discovered a small chip attached to one corner of the drum unit, probably carrying product ID and some serial number. Luckily, it was easy to be removed and exchanged with the other drum unit. Still, the printer thought the cartridge was empty. Some (windows) driver re-installations later it was clear that there is no "ignore fill level and print anyway" option. (Cups didn't print either.) The HP Support Forum has a long Ink cartridge empty but it's not thread with lots of complaints but few answers. (Their problem is mostly with new HP cartridges becoming "empty" after a few days.) Someone suggests trying to cold reset the printer. For the 2605dn, that's holding the left and right buttons while powering on. I removed the cartridges for doing so. When I had put them back in, the printer asked me to confirm that non-HP material was installed, and subsequently didn't display any fill level for the black cartridge anymore, just " ? " in the display. Printing works now. Yay. (Lesson learned: I still hate dealing with printers.)

I've spent second day of Hackweek 6 on more different tasks. First I tried to add some improvements to Series60 applet. First things to fix were crashes on requesting non existing contact or calendar entry. After that I started to improve user experience a bit, for example by making the menu work and contain some useful things. After that I've added new functionality to capture screenshots of the device. So you can see how the applet actually looks like: On the Gammu side, I've implemented reading of contacts and todos. No write support yet, that will be done tomorrow. And of course counterpart for the screenshot function. I've also spent some time on writing documentation, so now there is some documentation on setting up the S60 connection or the new gammu screenshot command. All code is is available in our Git repository.

Filed under: English Gammu Suse 0 comments Flattr this!

rdiff-backup vs. ZFS As I ve been writing about backups, I ve gone ahead and run some tests with rdiff-backup. I have been using rdiff-backup personally for many years now probably since 2002, when I packaged it up for Debian. It s a nice, stable system, but I always like to look at other options for things every so often. rdiff-backup stores an uncompressed current mirror of the filesystem, similar to rsync. History is achieved by the use of compressed backwards binary deltas generated by rdiff (using the rsync algorithm). So, you can restore the current copy very easily a simple cp will do if you don t need to preserve permissions. rdiff-backup restores previous copies by applying all necessary binary deltas to generate the previous version. Things I like about rdiff-backup:

1. Bandwidth-efficient
2. Reasonably space-efficient, especially where history is concerned
3. Easily scriptable and nice CLI
4. Unlike tools such as duplicity, there is no need to periodically run full backups old backups can be deleted without impacting the ability to restore more current backups

Things I don t like about it:

1. Speed. It can be really slow. Deleting 3 months worth of old history takes hours. It has to unlink vast numbers of files and that s pretty much it, but it does it really slowly. Restores, backups, etc. are all slow as well. Even just getting a list of your increment sizes so you d know how much space would be saved can take a very long time.
2. The current backup copy is stored without any kind of compression, which is not at all space-efficient
3. It creates vast numbers of little files that take forever to delete or summarize

So I thought I would examine how efficient ZFS would be. I wrote a script that would replay the rdiff-backup history first it would rsync the current copy onto the ZFS filesystem and make a ZFS snapshot. Then each previous version was processed by my script (rdiff-backup s files are sufficiently standard that a shell script can process them), and a ZFS snapshot created after each. This lets me directly compare the space used by rdiff-backup to that used by ZFS using actual history. I enabled gzip-3 compression and block dedup in ZFS. My backups were nearly 1TB in size and the amount of space I had available for ZFS was roughly 600GB, so I couldn t test all of them. As it happened, I tested the ones that were the worst-case scenario for ZFS: my photos, music collection, etc. These files had very little duplication and very little compressibility. Plus a backup of my regular server that was reasonably compressible. The total size of the data backed up with rdiff-backup was 583 GB. With ZFS, this came to 498GB. My dedup ratio on this was only 1.05 (meaning 5% or 25GB saved). The compression ratio was 1.12 (60GB saved). The combined ratio was 1.17 (85GB saved). Interestingly 498 + 85 = 583. Remember that the data under test here was mostly a worst-case scenario for ZFS. It would probably have done better had I had the time to throw the rest of my dataset at it (such as the 60GB backup of my iPod, which would have mostly deduplicated with the backup of my music server). One problem with ZFS is that dedup is very memory-hungry. This is common knowledge and it is advertised that you need to use roughly 2GB of RAM per TB of disk when using dedup. I don t have quite that much to dedicate to it, so ZFS got VERY slow and thrashed the disk a lot after the ARC grew to about 300MB. I found some tweakables in zfsrc and the zfs command that let me tweak the ARC cache to grow bigger. But the machine in question only has 2GB RAM, and is doing lots of other things as well, so this barely improved anything. Note that this dedup RAM requirement is not out of line with what is expected from these sorts of solutions. Even if I got absolutely stellar dedup ratio of 2:1, that would get me at most 1TB. The cost of buying a 1TB disk is less than the cost of upgrading my system to 4GB RAM, so dedup isn t worth it here. I think the lesson is: think carefully about where dedup makes sense. If you re storing a bunch of nearly-identical virtual machine images the sort of canonical use case for this go for it. A general fileserver well, maybe you should just add more disk instead of more RAM. Then that raises the question: if I don t need dedup from ZFS, do I bother with it at all, or just use ext4 and LVM snapshots? I think ZFS still makes sense, given its built-in support for compression and very fast snapshots LVM snapshots are known to cause serious degradation to write performance once enabled, which ZFS doesn t. So I plan to switch my backups to use ZFS. A few observations on this:

1. Some testing suggests that the time to delete a few months of old snapshots will be a minute or two with ZFS compared to hours with rdiff-backup.
2. ZFS has shown itself to be more space-efficient than rdiff-backup, even without dedup enabled.
3. There are clear performance and convenience wins with ZFS.
Backup Scripts So now comes the question of backup scripts. rsync is obviously a pretty nice choice here and if used with inplace perhaps even will play friendly with ZFS snapshots even if dedup is off. But let s say I m backing up a few machines at home, or perhaps dozens at work. There is a need to automate all of this. Specifically, there s a need to:
1. Provide scheduling, making sure that we don t hammer the server with 30 clients all at once
2. Provide for run before jobs to do things like snapshot databases
3. Be silent on success and scream loudly via emails to administrators on any kind of error and keep backing up other systems when there is an error
4. Create snapshots and provide an automated way to remove old snapshots (or mount them for reading, as ZFS-fuse doesn t support the .zfs snapshot directory yet)
To date I haven t found anything that looks suitable. I found a shell script system called rsbackup that does a large part of this, but something about using a script whose homepage is a forum makes me less than 100% confident. On the securing the backups front, rsync comes with a good-looking rrsync script (inexplicably installed under /usr/share/doc/rsync/scripts instead of /usr/bin on Debian) that can help secure the SSH authorization. GNU rush also looks like a useful restricted shell.

The Free Software Foundation is in the last week of its annual fundraiser and has still has a bit of ground to make up. The FSF needs members and donations to merely sustain its basic activity protecting free software and engaging in minimal outreach. So as I've done in the last couple years, I've written a fundraising appeal for the organization. That why today my face is plastered, Jimmy Wales style, all over the FSF website. (For the record, the last bit was not my idea and I find it a little embarrassing.) My appeal this year begins with my feeling that as our lives become increasingly mediated by technology, the question of who controls technology becomes another way of asking "Who controls us?" Read the appeal for more. That statement, like much of the FSF's work, is not going to get very far with most users today. Most people don't think of technology as the source of control, power, or autonomy. They don't understand why software should be free or even what that might mean. Making this argument widely is something the FSF works hard to do. But supporting this work is a bit of a chicken-egg problem. The users we most need to reach have never heard our message and it can take some time to turn even the most receptive into supporters. For those of us that do care about technology and the power it has over its users, it's hugely important that we take action to support software freedom and organizations like the FSF that play a key place in this struggle. If we don't, nobody else will. The pool of people who might support the FSF today is small -- there are currently about 3,000 members -- but the stakes are huge: A choice between a world of DRM, locked down mobile phones, and non-free network services on one hand, and the possibility of a world where there are free alternatives on the other. The FSF won't win this fight alone, but it plays a critical role in it. The FSF is a small, humble, and responsible non-profit and a little money can have a big impact. If you are not an FSF associate member, now would be a great time to join. Membership is $120 per year ($60 for students) and payable monthly. In preparing this appeal, I realized that my membership had lapsed a couple months ago and that my partner Mika was still donating at the reduced student price. We've personally fixed both these things and made an addition donation. If you are a member, you can check your status or make a donation. If you are not yet, Please consider joining us.

There are certain people who are NIH-prone. Usually I consider this insane, but sometimes I sympathize. Certainly I have pulled a NIH once or thrice before. Maybe I'm one of those old curmudgeons who incorrectly remembers the past, but I have the feeling that software used to work a whole lot better. Admittedly it did less, but it was also less prone to weird semi-failures. I am probably repressing a bunch of kernel oopses in the '90s. Now though, I can waste a week and a half being miserable as SIP clients fail on me erratically, because I hate Twinkle and no one thought to tell me that Twinkle is really the only SIP client that works. The interoperability of CalDAV and GroupDAV servers and clients seems to be abysmal. The same even appears to be true of plain ol' WebDAV. There are multiple forks of Markdown. I blame John Gruber for not maintaining a strong-enough central standard. No working LUKS-encrypted live images appear to exist in the wild. The only reliable and useful backup software for my needs was abandoned in 2008. Everything in Rails is bloated and crazy, except for Redmine. How is Redmine able to transcend? I do not understand. There is no way you can run an Android phone on only free software. freeswitch is on crack.

Emacs snapshot packages for Ubuntu are now available, thanks to Dimitri Fontaine.

In its Fast distributions and slow servers article, Joe Zonker Brockmeier explains the difficulties that Fedora has to use its own distribution on their infrastructure and echoes some questions they faced:

Why wouldn t a Linux distribution project wish to showcase the distribution by hosting its infrastructure using its own system?

I completely share the underlying assumption. Eating its own dog food is very important if you want to build a Linux distribution and claim with some confidence that it s of quality and usable. Debian does quite well nowadays in that respect. There were times where the mailing list server was using Qmail (non-free at that time, and thus not part of Debian) but that s long gone. We have also seen our build infrastructure relying on software that was not public and not packaged in Debian, but that also is history. The Debian System Administration team (DSA) maintains more than 140 servers running Debian. They mostly run the stable version of Debian but a few test machines are already running Squeeze since a few months (qa.debian.org for example). The Debian admins want to ensure that the next stable release of Debian still has everything they need and that the software they use still work as expected. Big kudos to the DSA team for this choice! I hope we ll be able to continue to live up to those standards for a long time to come. PS: If you want to learn more about the setup that the DSA team uses, head to dsa.debian.org. You ll find all their repositories and some of their internal documentation.

Subscribe to this blog by RSS, by email or on Facebook.

12 comments Liked this article? Click here. My blog is Flattr-enabled.