My plea has been answered by a DVFS project specification. For some strange reason, my edits do not show up in the freedesktop.org wiki page and the blog of its creator is not reachable, so I post it here. I am very interested to help develop this tool and then use it in Simple Backup Suit (sbackup). An additional feature that I would love to see is automatic splitting of a file into several chunks. In an ideal work it would be possible for me to get (in Python) a filehandle-like object that would compress the data (gzip/bzip/..), encrypt it (with simple encr. or gpg style), transfer it over network (ssh/ftp/dav/smb/…) and then split into 100 Mb pieces on the remote server (to overcome possible max filesize problems with 10,50 or 100 Gb backups). And have the freedom of flexibility of activation of each element of this chain when creating the file handle. Also the interface must react in exactly the same manner regardless or the backed being user (for example opening non-existing files must always rise the same error).

A couple of days ago, I replied to Madduck's post regarding unhappiness with XFS. Ben replied with a trackback as well: He's also moving away from XFS. And... Well, I noticed the unavoidable: XFS, in the end, _did_ eat my data.
I use a chroot in my desktop machine as a backups server, for two other machines. The backups are done three times a week. And last time, they failed. Why? Because /dev/hdd2 is not mounted. Why?
Oh, nice, great and stinky shit. Ok, lets fix it...
Fortunately, that's only some backups and a very untweaked chroot, made easily with debootstrap... But still, that's something I'll have to work on this evening. Sigh, as I didn't have enough pending work already :-/

I went up today to visit Kasper Souren, AKA Guaka, who's visiting wt:Montreal for a few weeks. Kasper and I have been in touch for several years -- he's been involved with Wikitravel almost as long as anyone -- but we just met in person a couple of weeks ago at Wikimania. He's just returned from South America, and will soon depart for the West Coast of North America and then off to Mexico. He's quite a traveller. Guaka's here to help out with the CouchSurfing Project, an excellent wt:hospitality exchange program that lots of people say good things about. There are apparently bunches of CouchSurfers in Montreal right now helping out with technical issues, content, support, and other great Website-errific stuff. Of course, the local CS community is practicing the ethic and sharing their homes with visitors. I'm not 100% sure how it works, CouchSurfing runs on a very collective model, and a lot of people are devoting their time to this get-together. I went to visit this afternoon, partly to catch up with Guaka, and partly to visit the collective. We talked a lot about the difference between Wikitravel and hospitality exchange -- how anonymous the sharing is for an information-oriented project like a wiki, versus the very intimate person-to-person nature of home sharing. I also talked for a while with Casey Fenton, the founder of CouchSurfing and a very nice guy. We all discussed how to let CouchSurfers copy and distribute Wikitravel guides, possibly customizing them for CS needs, and also for linking back to WT to update and improve the data on our guides. I'm pretty jammed on the whole CS thing -- I'm going to sign up quite soon. Should be fun. tags: couchsurfing wikitravel guaka casey fenton hospitality exchange

DekiWiki I'm interested to see how this OpenGarden thing works out for MindTouch. I saw a story about their Wiki product on Port 25, which I have to admit I read pretty regularly. Every once in a while there's something I should probably know. OpenGarden is MindTouch's Open Source initiative, including a wiki engine called DekiWiki, built on the .NET technology, and a wp:REST-based Web app framework called Dream. Apparently the whole thing runs fine on top of Mono, which means theoretically I could install it. But I haven't really done a lot of Mono web stuff before -- no idea how well it works. Also, have you noticed how people involved with Mono say "MOH-NOH", like the actual original Spanish pronunciation? Unlike everyone else, who calls it "MAH-NOH", like mononucleosis? That's a weird thing. OpenGarden has a MediaWiki conversion tool, which is almost a necessity for any wiki engine trying to compete in 2006. They might want to try lobbying those guys at the Mono Project... who appear to be using MW as their main site engine! Anyways, I'm going to give this DekiWiki a try. Who knows? It could be fun. I don't know about their overreaching claims for the GPL, but it can't hurt for a local install, right? I seem to remember there was a Scheme language plugin for .NET somewhere, too... By the way, I'm still bullish on JAMWiki, the MediaWiki-workalike using Java Web Services. But I like to experiment with new software, too. tags: wiki engine mono open garden mindtouch dekiwiki jamwiki wiki

SocialText Speaking of commercial Open Source Wiki software, I was really glad to see that SocialText has released a FLOSS version of their flagship product. SocialText has been such a good corporate citizen in the wiki and FLOSS world. Not only were they instrumental in the launch of BarCamp, but ST employs some of the smartest wiki minds in the world today (like Seb and Sunir). And, of course, Ross Mayfield's blog is a great clearinghouse for wiki business info. ST has made a business model in training companies in the WikiWay. I think that makes a lot of sense, and they show a lot of commitment to the idea by demonstrations like this. tags: socialtext open source wiki ross mayfield

Fired from Technorati for my UFO beliefs So, I was having some random and unpleasant 500 Server Error errors with this blog recently, and apparently Technorati has given up on my site in frustration. No matter how often I do an update, and then ping, it still doesn't refresh my recent articles. I'll have to figure out some tricks to get back in Technorati's good graces. I like the service a lot. tags: technorati 500 ufo beliefs

Both of these come via Newshog; the first is a very interesting article from NBC talking about the ”unprecedented cooperation and coordination” between US and UK officials; unprecedented in this case appears to mean ”arguing all the way to the police station”, but this is just according to one unnamed UK official, so possibly a pinch of salt required. The article goes on to mention in passing the arrest of the ringleader, Rashid Rauf, in Pakistan, apparently causing another disagreement between the US and the UK over choice of jurisdiction - which some are linking to the possibility of torture. There are some fantastic quotes in the NBC article, regarding the timing and actual preparedness of the “attack”, especially that ”the suspects in Britain had obtained at least some of the materials for the explosive but had not yet actually prepared or mixed it.” and that ”the suspects had not yet purchased any airline tickets. In fact, some did not even have passports.”. All this added up begins to bring serious doubt as to the actual danger represented by the terrorists, and also raises the question of what a more “hands-off” - not arresting the suspects, but allowing them to continue their preparations longer - approach to monitoring and intelligence gathering would have resulted in. The second is interesting in its own right - Max Hastings - friend of Lady Thatcher, dyed in the wool Conservatist - writing for the Guardian - left/liberal and proud of the fact - leads his article with “George Bush sometimes sounds more like the Mahdi, preaching jihad against infidels, than the leader of a western democracy” (edited to link to Wikipedia’s conservative party page.)

Migration complete! I now have two distinct servers/hosts:

• A dedicated host: AMD 1600 + 1Gb memory + 40Gb disk. (Donated)
  • Runs www.debian-administration.org + the matching planet (Apache + Perl + MySQL + memcached)..
  • Other than SSH no other services are running.
• A 10Gb Xen instance with 256Mb memory
  • Runs multiple virtual hosts; Apache + Exim4.
  • Holds my CVS Repositories.

Both servers are backed up nightly to a “backup server” hosted in my house (Hostname: itchy.my.flat) via rsnapshot. An interesting few days getting the split to occur nicely, which involved some DNS + Domain Registrar juggling due to obscure reasons. Now that the split has been made I can start working on better caching, and performance tweaks on the d-a.org site. I’m currently investigating the use of fast_cgi, since I think that the code should be simple to modify for this. Otherwise I’m now back in action. I have pending releases of several packages to upload, and a new upstream release of GNUMP3d to prepare.

I decided that before my socket 939 ASUS A8V Deluxe motherboard becomes obsolete, I should do an upgrade to dual-core. It so happened that my dealer had an Athlon64 X2 4800+ returned to him the other week, so I went for it. It's the fastest that the board supports and comes with juicy 2x 1MB L2 cache (which apparently got axed by AMD now as part of the price-battle with Intel). My only concern was about power/heat/noise because it's the 110W model, but to my delight it appears to be running cooler and quieter than the (130nm) 3500+ I had before. Cool! ;-) While I was at it I also doubled the RAM to 2GB and bought two SATA hard disks and a swappable tray system for them.

The latter was mainly so I could do some tests in regards to #380703. The outcome is that restore works fine with SATA disks on above motherboard using its Via SATA controller (haven't tried the Promise one yet). So far, so good - or rather bad, because I can't produce the problem.

At least I do have a fix for #379966 which basically entices the introduction of a new function mr_stresc() to properly escape things when using system() or popen(). It's gone through a few revisions both by the submitter and upstream (Thanks Steve & Bruno!) and should be in upstream soon.

For the time being, I have just uploaded mindi-1.09-1 and mondo-2.09-1. They are purely new upstream versions and don't close any of the open bugs - hopefully, this will follow soon. Testing was done on an etch system using a SATA disk and a sid system using PATA. The archiving mode was NFS both times and the kernel used was also the same, i.e. 2.6.16-2-amd64-k8-smp (2.6.16-17).

I am experiencing problems with 2.6.17 kernels and creating ISO images on NFS mounts using mkisofs. mksiofs would just hang sooner or later whilst writing the image file. With 2.6.16 there is no such issue. And neither cp nor dd have this problem with 2.6.17. Weird. I'll have to do some more testing and probably will bring it up on the debian-kernel list before I file a bug report.

To keep you reading all the way through, the best one's at the end.
1. I'm striking back out on my own againAfter a couple of years working full time in an office, I've decided it's the right time to take my own business, the Useful Information Company, further on. As the rest of this post illustrates, we'll be involved in web technology conferences, our own products, and Rails-based software development. I'm always looking for collaborators, comment and customers, so get in touch if any of this lot grabs you.
2. I'm organising an amazing web developer event in London, late OctoberUseful's first venture is XTech WebDev London. This is the first of a new type of event for XTech: a focused one-day training event for web developers. The idea is to focus on the best practice in web development that emerges from the cutting-edge work demonstrated in the regular XTech conference.This year, that means: Ajax, web standards, open data, REST and Ruby on Rails. Any web developer wanting inspiration and a view of the future should come along. London, UK. October 31, 2006. You can reserve your place now. (We're right-priced for your training budget, and start late enough that folks could make it from all over the UK and beyond without staying a night.)
3. The pace is hotting up for XTech 2007 in ParisMeanwhile, I'm busy putting together the call for participation and programme committee for XTech 2007, in Paris, France. Our theme is "The Ubiquitous Web." As keynotes, I'm pleased to welcome Adam Greenfield, Matt Webb and Jack Schulze.
4. I'll be off travelling shortly and reconnect with old and new friends Though a little daunted by the new face of air travel, I'm heading out to O'Reilly in Sebastopol at the end of this month to take part in FooCamp. One of my main aims is to seek out the ingredients required for a successful professional online community. In September I'll be putting in some time at EuroOSCON. Both visits will let me hook up with friends, many of whom I've not seen for a couple of years.
5. I'm on the programme committee for O'Reilly's Emerging Technologies 2007 conferenceThis one has me interested and engaged. Rael Dornfest is due to unveil the call for participation in a couple of weeks' time, and I'll give my take on the conference then. The show itself is March 26-29, 2007.
6. I have an exciting startup under the covers, in the area of conference and event managementOne of the things I've done a lot of over the last half-decade is organise, coax and bully people into presenting at conferences. I'm pouring what I've learned, and more besides, into a great product. You'll soon be bored of me talking about it.
7. I'm involved in taking Pharmalicensing into an exciting new phasePharmalicensing is an online exchange for intellectual property in the biotech and pharmaceutical world. I was part of its foundation in 1998, and am still involved. I will be part of a newly energized team of people pushing it forward into 2007. 8. I'm having a bit of a clean-out of projectsAlas, with so much going on, I need to cut out a few existing commitments. One of these things will be Monopod, my Mono-based podcast client for Linux. If I can't find anybody interested in its maintenance, I'll just hang the source code out there for anyone to pick up.
9. I want to get more energy put into DOAPOne of my long term projects I definitely don't want to cut out is DOAP, my project to describe software projects in RDF. It's had a slow but sure take-up over the last few years, and is finding its way into some significant projects. However, I do need a bit more energy for it, and one or two collaborators would certainly help. 10. Around Christmas-time, I'm going to become a father to twinsI've still not recovered from the shock. Rachael and I are going to be the proud, stunned, parents of a couple of bright shiny bundles of Human 2.0. Completely awesome, and I've absolutely no idea what to do! For now, we're just concentrating on throwing enough junk out to make room for them.

Daniel, if you can somehow build a card that's at r2xx level (note the current state of the art is r5xx), not to mention be register-compatible, you are nothing short of a genius, and ATI and NVIDIA will be begging you to let them pay you a number of your choice.

It's not going to happen anytime soon. They're not worried about that.

Daniel Burrows writes "and the built-in graphics card (usually Intel) on the motherboard is just fine for that." I would agree with you, if I had an built-in graphics card with Intel. That is in fact what I'm aiming for for my next laptop. But actually none of the PCs I have in my house has Intel graphics, unfortunately. Intel graphics is pretty much restricted to Core CPUs and Laptops. Which sucks. I wish there would be more "budget" systems with Intel graphics. As you mentioned I don't need high end graphics anyway. I actually don't need high end CPU anyway (power consumption is more interesting, and the value/price ratio). I wish I could have bought a system with Intel graphics, but that would have been twice the price. I wish we had some standard like VESA, just a bit more modern. Like having sane refresh rates for non-TFT screens and maybe some extra acceleration. Oh, and I actually doubt that ATI would give away much secrets if it would allow distributing the existing opensource X drivers. It's not as if the specs of their CPU would be a huge surprise to Nvidia. Some actually say nvidia builds the better graphics cards. And Intel does even give away source code. We're not talking about chip design here, or driver optimizations. Just the plain registers and ports. heck, ATI, Nvidia and Intel could probably even agree on a standard here, except it would mean they'd need to change their drivers and hardware for the next generation with really no visible benefit to them (only to us, we could maybe run the basics with the same driver).

Are there any cell phone interface design principles? (Google’s answer is inconclusive.) I would think a multi-billion dollar industry would have developed some best practices at this point, but it seems like each new phone is designed by monkeys jumping on typewriters. Maybe eventually they’ll get it right, but I’d rather not wait until infinity. A persistent flaw that has always puzzled me across nearly all models is the “shut down sound” feature. If you are shutting off your cell phone, do you really need a distinctive musical reminder? Did the designers of this feature ever consider that, in many instances, the user is shutting off the cell phone because they are in a silent theatre or a business meeting? Who are these designers, anyway? Of course you can always turn the volume down first (or probably better — just silence the phone and keep it on), but that is not going to be the first inclination for most users. I was reminded of this flaw last night at a Counterpoint Concert that was being recorded for VPR radio broadcast. After the chorus director reminded people to turn off their cell phones for the recording, a symphony of shut down sounds followed shortly thereafter. Cell phone gripes aside, the concert was great — particularly the performance of They Called Her Moses. On a related note, see this petition regarding cell phone user interfaces and software. Joel on Software also has some nice observations on this topic.

Maj went out for her birthday party with a passel of ladies, so I stayed home and ate pizza and watched Twelve Monkeys with the baby. We had a good time, but as the baby is approaching 1 years old, my going-to-sleep tricks are becoming somewhat outdated. My main way to put the baby to sleep is to walk her from the front door of our apartment all the way back to the laundry room, and then turn around and walk out to the front door again. I sing during the entire process, mostly Neil Young and Johnny Cash tunes, but also a mix of 80s pop, reggae, songs, and goofy children's songs. Really, it's anything I can remember the words to, or that I can bridge over with some ad-libbing. This worked pretty well when she was teensy-tiny, and it's still the best way for me to get her to sleep. But as she's approaching 15kg, it's getting to be a serious workout for me to slog up and down the hallway 50 times belting out Take the Skinheads Bowling at high volume. We have a few baby books on how to get a child to sleep easier, but I've been taking the path of least resistance. I'll probably have to take the plunge and get her to fall asleep in bed soon -- which will probably mean some angry baby moments. Still, putting the baby to sleep is one of the most satisfying things in my life right now -- I highly recommend it. tags: baby amitajune sleep neil young singing

Programmer hierarchy Luke Welling's programmer hierarchy, based on The Brunching Shuttlecocks' classic Geek Hierarchy, is humorous and accurate. tags: programming languages haha

Tag improvements So, I've made some improvements to the tagging system on this blog. Instead of doing a full-text search at runtime to lookup which pages contain which tag, I instead keep a map of tags to page names in a separate db, and then look up in that index at runtime. The bummer is that I'm not figuring out a way to keep that db up-to-date easily. The best action, of course, would be to have the tags db updated at write time, but I can't figure out how to make that work. I tried it with a write macro, but you can't get the title of the current page with a write macro, unfortunately. I might just add a hook to the commit action -- that's about all I'm coming up with right now. It's annoying that Wiliki doesn't have hooks enabled anywhere. Gar. tags: wiliki tagging optimization

Apparently the government agencies in UK that are managing the current security craze have little to no idea how much their super-paranoid security policies acctualy cost. It appears that direct losses from one day of air traffic chaos costs airlines 175 million pounds. Everyone who has their flight delayed for more then an hour for whatever reason need simply to sue the airlines and further the UK government according the EU regulations. Each person delayed for just 4 hours can claim as much as 600 euros of compensations. I wonder what compensation must a person delayed 24+ hours get, considering missed meeting, needed hotel bookings and other expenses. Now if all passengers and all airlines united and sued the government for the compensations (those can be in billions, considering that no sane business person would fly without their mobile and notebook and no mother would want to fly without toys to keep their children happy, so that will cause huge decreese in flying), then the government will think twice before disturbing lives of tens of thousands of people even when all suspects have been arrested. And we do not even know if there was any terror plot. Last time there was a terror craze in the UK, a road was closed and isolated, a house was raided and then disassembled and a man was shot, but in the end nothing was found. Who will foot the bill this time? The news say that people will not receive any compensation even if they were insured, because “it was an extreme event”. WTF?! That is what regulations and insurance are for!!! And the minister is saying that they think that most restrictions will stay forever. No carry-on baggage? That is ridiculose. SUE THE UK GOVERNMENT!

No, I'm not dead, just very very busy... There have been quite a few interesting new Free Software projects and news popping up recently (or I only recently noticed them):

• The GPLv3 is back in the news: Stallman, Torvalds, Moglen share views on DRM and GPLv3. Very interesting read!
• Ditto for the next-generation Creative Commons licenses: Creative Commons Version 3.0 - Public Discussion Launched. See also Evan Prodromou's Debian Creative Commons Workgroup report.
• On to more practical things: I have put quite some time into trying out and working on LinuxBIOS, a free replacement of traditional, closed-source, proprietary BIOSes. More details on that later... If you want to check it out, and/or help (patches are always welcome), have a look at the wiki or the mailing list.
• The Call for nominations for the 2006 FSF Award for the Advancement of Free Software has been posted. Who do you think should get the award this year?
• Intel has released Free Software drivers (2D and 3D!) for their Intel® 965 Express Chipset family graphics controller. The code is licensed partly under the MIT license, partly the GPL. Thanks Intel!
• Apple has released some more code as Free Software: Apple Opens Up: Kernel, Mac OS Forge, iCal Server, Bonjour, Launchd. Thanks Apple! They switched from the APSL 2.0 to the Apache license 2.0 (at least for some of the code), not sure why. They're both considered "Open Source", and both are GPL-incompatible.
• GNU Classpath 0.92 "Bling! Bling!" has been released, with full support for 2D graphics. One step closer to a free Java environment (not that I would recommend anybody to write stuff in Java if you have a choice)...

Yesterday I was back again at the Parish to continue working on arete, the newly-installed Ubuntu Dapper machine. I was back because I had to continue migrating my father’s files from Windows to the new setup, since he wanted to try out Scribus as a replacement for his MS Publisher system for writing the parish newsletter. Anyhow, this reason, coupled with my own urge to seriously try out Ubuntu Dapper (I was running Kubuntu Dapper, doh) made me spend the whole day trying out something new. As I was looking at Dad’s documents folder, I realized that I had a problem: there are so many files here, some of them sizing up from tens to hundreds of megs (actually, stuff worked on thru the Gimp,) and knowing well that disk drive space was at a premium (arete had an 80G drive, yes, but nevertheless its already cramped,) I simply couldn’t accept copying the files from the XP’s NTFS C:\ to arete’s ext3 /home. Moving wasn’t an option either; although Dad would very much like to try out Ubuntu, he still had some projects (especially the newletter) that needed, uhm, proprietary software to work (sigh.) Thus, remembering that Linux already had some way of reading NTFS partitions, I promptly Googled about it. I also knew that the Linux-NTFS guys (who made the ntfsprogs suite and kernel module) were already working on getting full read/write NTFS support on Linux, but it didn’t occur to me that they were able to get a working implementation done so soon… Enter ntfs-3g: the third generation Linux-NTFS driver. Weighing in at around 550K, this seems to be the one thing that I need for my problem. From the looks of it, ntfs-3g can both read from and write to NTFS partitions, allowing one to create, delete, and update files and directories on the system; it can even allow access to certain metadata that even Windows XP hides from its users. Furthermore, its built from the ground up as a filesystem-in-userspace driver, using the Linux fuse subsystem. It certainly looks very attractive indeed. So, being the sysadmin for arete that I was, I quickly decided to give ntfs-3g a try. After all, it was a simple matter of grabbing the tarball, then reading the README, then doing the three-step tango of ./configure; make; sudo make install, then calling ntfs-3g /dev/hda1 /media/hda1 -o umask=007 as root. Right? Wrong. I quickly ran into a trio of subproblems: first, I needed fuse (>= 2.5.0); the one Dapper had on its official archives was 2.4.2 . So, naturally, I had to make a quick backport of the fuse source from Edgy (2.5.3), but I at this point I ran into the second problem: for Eggy’s fuse to get busted, I needed the new makedev (2.3.1) as well. Forgetting to do apt-get -f install for the moment, I grabbed Edgy’s makedev and proceeded backporting it, which ran me against the last problem: I also needed the newest sysvinit (2.86.ds1). In short, I needed to do a trio of backports first before being able to build ntfs-3g. My own fuse was almost ready to explode. Fortunately, everything went fine after the backports (thank goodness for my being a Debian package maintainer and MOTU,) so I didn’t break my own fuse after all. Soon after, I was reading and writing stuff to and from /media/hda1 without so much as batting an eyelash. Dad was definitely pleased, since that meant that he no longer had to dread the thought of having to maintain two separate document trees and cause him pain and gnashing of teeth. I was glad too, since I no longer had to dread the thought of having to hack on this today, causing me to disrupt my Summer of Code schedule. But, being the packager that I was, I realized that I could make the world a better(?) place by turning ntfs-3g from a regular Joe source to a real Debian/Ubuntu package. While I was happily waiting for ntfs-3g’s build to complete, I happened to pass by a certain HOWTO on the Web using pre-built fuse and ntfs-3g Ubuntu packages. I grabbed those packages, and took a look at them, and while they were pretty much useful, the ntfs-3g package certainly did not look to pass the gold(?) standards of Debian and Ubuntu: the ntfs-3g package was a single binary package, shipping both a shared library and its headers, coupled with the working binaries. It had outdated FSF addresses too, which my bugging Lintian pointed out to me. With some more extra time to spare on arete, I decided to go ahead and build proper a proper source package for ntfs-3g, producing the three binary packages ntfs-3g (containing ntfs-3g and mount.ntfs-3g,) libntfs-3g0 (depended on by the former package,) and libntfs-3g-dev (containing the include headers.) CDBS and debhelper definitely made this easy pickings, and made my life with ntfs-3g easier. And, I’m going to make everyone else’s life with ntfs-3g easier, by making this package available for nit-picking. I won’t pretend this to be getting into the official Debian and/or Ubuntu archives for the moment, since the upstream authors have explicitly stated that ntfs-3g will be eventually merged into the existing ntfsprogs suite, thereby making my packaging of ntfs-3g obsolete sometime soon; furthermore, the binary packages up there are built against Debian unstable and Ubuntu Edgy, so anyone looking for a Dapper solution can read this post first to avoid reinventing the wheel. But if anyone needs a good solution for NTFS read/write right now (or, if somebody else has time to kill in testing ntfs-3g,) then here’s a good one for you. Caveat emptor!  Ok, next up: finishing my SoC work…

It is a common belief that Ubuntu provides newer software than Debian. This is of course true when Ubuntu is compared to Debian stable. This is also true for specific software packages, like Gnome. But how does it compare in general ? At the time of the Dapper release, here is how Ubuntu Dapper and Debian Unstable compared wrt source package versions :

In Ubuntu, but not in Debian: 1795
In Debian, but not in Ubuntu: 588
Exact same version: 4361
Same upstream version: 2963
… and same Debian version, but Ubuntu has local changes: 731
… “Debian version” newer in Debian: 2197
… “Debian version” newer in Ubuntu: 35
Different upstream version: 2166
… newer in Debian: 1955
… newer in Ubuntu: 211

So Debian easily wins. But this comparison is unfair, because Debian unstable is a moving target : of course, it’s newer than Ubuntu ! What about Debian testing ?

In Ubuntu, but not in Debian: 2137
In Debian, but not in Ubuntu: 514
Exact same version: 4529
Same upstream version: 2652
… and same Debian version, but Ubuntu has local changes: 775
… newer in Debian: 1816
… newer in Ubuntu: 61
Different upstream version: 1967
… newer in Debian: 1666
… newer in Ubuntu: 301

Yes, Debian still wins. Okay, but Ubuntu Dapper had a particularly long release cycle. What about Ubuntu Breezy, and Debian Testing at the time of the Breezy release ?

In Ubuntu, but not in Debian: 2462
In Debian, but not in Ubuntu: 384
Exact same version: 4586
Same upstream version: 2253
… and same Debian version, but Ubuntu has local changes: 981
… newer in Debian: 1172
… newer in Ubuntu: 100
Different upstream version: 1375
… newer in Debian: 1020
… newer in Ubuntu: 355

Debian testing wins again. Notes:

• Statistics might be biased because of source package name changes.
• Ubuntu doesn’t automatically remove packages which were removed from Debian, which partially explains the high number of packages in Ubuntu but not in Debian.
• OK, there’s no Debian unstable/Ubuntu edgy comparison. I believe that not many people use Ubuntu development versions (except a few weeks before release), and that Ubuntu dev versions are usually not considered usable. OTOH, many people use Debian testing or Debian unstable.
• The quick and dirty script used to generate the stats is available.

Update: Yes, Scott, I am comparing stable releases on one side, with development versions on the other side. However, I think that it is widely known that Debian testing is usable outside of the Debian development community. I know many people using Debian testing who aren’t Debian developers. And many Ubuntu users, if they weren’t using Ubuntu, but Debian, would probably use Debian testing, not stable. Regarding Ubuntu Edgy, its use is currently strictly limited to the Ubuntu development community, and I don’t know anybody not interested in Ubuntu development running Ubuntu Edgy (that’s also because Ubuntu is released every 6 months, not 18 months, of course). That’s why it’s irrelevant to compare with it.
Also, after a quick poll on IRC, I discovered that many Ubuntu users thought that Ubuntu stable releases had newer software than Debian Testing (that was my exact question). The only point of this blog entry is to prove them wrong. Update 2 : I’ll be on holidays for the next few days, so comments are put in the moderation queue until I come back.

Lately I have been trying out a number of distributed version control systems (VCS or SCM).

One of my tests was a real problem: I wanted to track the Linux 2.6.16.x kernel tree, apply the Xen patches to it, and pull only specific patches (for the qla2xxx driver) from 2.6.17.x into this local branch. I wanted also to be able to upgrade to 2.6.17.x later (once Xen supports it) and have the version control system properly track which patches I already have.

But before going on, let's establish what it means to be an ideal distributed VCS:

• 1. The fundamental method of collaboration must be a branch. A checkout should mean creating a local branch on which a person can commit and work without having to involve the server. An update from some central server should take the form of a merge from that branch to the local branch.


• 2. Branching should be cheap. It should be easy to create a local branch, the operation to do so should be fast, and it shouldn't take an inordinate amount of space. It should also be as easy as possible to branch from a remote repository.


• 3. Merging between branches is intelligent. It should be easy to merge another branch with your own. The VCS should know which changesets from the other branch are already on yours, and should not attempt to merge changesets that you have already merged previously.


• 4. Inividial changesets should be mergeable without bringing across the whole history. You should be able to bring across the minimum number of changesets necessary to effect a specific change. This corresponds to my test case above. Future merges from the whole branch should, of course, recognize that these changesets are present already.


• 5. Branching preserves full history. A branch should be a first-class copy of a repository, even if the repository is remote. It should contain the full history of the branch it was made from, including diffs for each individual changeset and full commit logs, unless otherwise requested by the user.


• 6. Merging preserves full history. A merge from one branch to another should also preserve full history. Changesets merged to the local branch should retain the individual, distinct diffs and commit logs for each changeset.

There are also some things that we would generally want:

• 7. It is possible to commit, branch, merge, and work with history offline.


• 8. The program is fast enough for general-purpose use.

Evaluation

Let's look at some common VCSs against these criteria. I'll talk about Arch (tla, baz, etc), bzr (bazaar-ng), Darcs, Git, Mercurial (hg), and Subversion (svn) for reference.

1. The fundamental method of collaboration must be a branch
All of the tools pass this test except for svn.

2. Branching should be cheap
Everyone except svn generally does this reasonably well.

The tla interface for Arch had a pretty terrible interface for this, so it took awhile simply due to all the typing involved. That's better these days.

Darcs supports hardlinking of history to other local repositories and will do this automatically by default. Git also supports that, but defaults to not doing it, or you can store a path to look in for changesets that aren't in the current repo. I believe Mercurial also can use hardlinks, though I didn't personally verify that. bzr appears to have some features in this area, but not hardlinks, and the features were too complex (or poorly documented) to learn about quickly.

svn does not support branching across repositories, so doesn't really pass this test. Branches within a repository are not directly supported either, but are conventionally simulated by doing a low-cost copy into a specially-named area in the repository.

3. Merging between branches is intelligent
Arch was one of the early ones to work on this problem. It works reasonably well in most situations, but breaks in spectacular and unintelligble ways in some other situations.

When asked to merge one branch to another, Darcs will simply merge in any patches from the source branch onto the destination which the destination doesn't already have. This goes farther than any of the other systems, which generally store a "head" pointer for each branch that shows how far you've gone. (Arch is closer to darcs here, though ironically bzr is more like the other systems)

Merging between branches in svn is really poor, and has no support for recognizing changesets that have been applied both places, resulting in conflicts in many development models.

4. Inividial changesets should be mergeable without bringing across the whole history
Darcs is really the only one that can do this right. I was really surprised that nobody else could, since it is such a useful and vital feature for me.

Both bzr and git have a cherry-pick mode that simulates this, but really these commands just get a diff from the specific changeset requested, then apply the diff as with patch. So you really get a different changeset committed, which can really complicate your history later -- AND lead to potential conflicts in future merges. bzr works around some of the conflict problems because on a merge, it will silently ignore patches that attempt to perform an operation that has already occured. But that leads to even more confusing results, as the merge of the patch is recorded for a commit that didn't actually merge it. (That could even be a commit that doesn't modify the source.) Sounds like a nightmare for later.

Arch has some support for it, but in my experience, actually using this support tends to get it really confused when you do merges later.

Neither Mercurial nor svn have any support for this at all.

5. Branching preserves full history
git, darcs, and Mercurial get this right. Making a branch from one of these repos will give you full history, including individual diffs and commit logs for each changeset.

Arch and bzr preserve commit logs but not the individual changesets on a new branch. I was particularly surprised at this shortcoming with bzr, but sure enough, a standard bzr merge from a remote branch commited three original changesets into one and did not preserve the individual history on the one commit.

svn doesn't support cross-repo branching at all.

6. Merging preserves full history
Again, darcs, git, and Mercurial get this right (I haven't tested this in Mercurial, so I'm not 100% sure).

Arch and bzr have the same problem of preserving commit logs, but not individual changesets. A merge from one branch to another in Arch or bzr simply commits one big changeset on the target that represents all the changesets pulled in from the source. So you lose the distinctness of each individual changeset. This can result in the uncomfortable situation of being unable to re-create full history without access to dozens of repositories on the 'net.

Subversion has no support for merging across repositories, and its support for merging across simulated local branches isn't all that great, either.

7. It is possible to commit, branch, merge, and work with history offline
Everyone except Subversion does a good job of this.

8. The program is fast enough for general-purpose use
All tools here are probably fast enough for most people's projects. Subversion can be annoying at times because many more svn commands hit the network than those from others.

In my experience, Arch was the slowest. Though it was still fine for most work, it really bogged down with the Linux kernel. bzr was next, somewhere between arch and darcs. bzr commands "felt" sluggish, but I haven't used it enough to really see how it scales.

Darcs is the next. It used to be pretty slow, but has been improving rapidly since 1.0.0 was released. It now scales up to a kernel-sized project very well, and is quite usable and reasonably responsive for such a thing. The two main things that slow it down are very large files (10MB or above) and conflicts during a merge.

Mercurial and git appear to be fastest and pretty similar in performance.

All of these tools perform best with periodic manual (or scheduled cron jobs) intervention -- once a month to once a year, depending on your project's size. Arch users have typically created a new repo each year. Darcs users periodically tag things (if things are tagged as part of normal work, no extra work is needed here) and can create checkpoints to speed checkouts over the net. git and Mercurial also use a form of checkpoints. (not sure about bzr)

Subversion works so differently from the others that it's hard to compare. (For one, a checkout doesn't bring down any history.)

Conclusions
I was surprised by a few things.

First, that only one system actually got #4 (merging individual changesets) right. Second, that if you had to pick losers among VCSs, it seems to be Arch and bzr -- the lack of history in branching and merging is a really big issue, and they don't seem to have any compelling features that git, darcs, or Mercurial lack. #4 was a unique feature to Darcs a few years ago, but I figured it surely would have been cloned by all the other new VCS projects that have popped up since. It seems that people have realized it is important, and have added token workaround support for it, but not real working support.

On the other hand, it was interesting to see how VCS projects have copied from each other. Everyone (except tla) seems to use a command-line syntax similar to CVS. The influence of tla Arch is, of course, plainly visible in baz and bzr, but you can also see pieces of it in all the other projects. I was also interested to see the Darcs notion of patch dependencies was visible (albeit in a more limited fashion) in bzr, git, and Mercurial.

So, I will be staying with Darcs. It seems to really take the idea of distributed VCS and run with it. Nobody else seems to have quite gotten the merging thing right yet -- and if you are going to make it difficult to do anything but merge everything up to point x from someone's branch, I just don't see how your tool is as useful as Darcs. But I am glad to see ideas from different projects percolating across and getting reused -- this is certainly good for the community.

Updates / Corrections
I got an e-mail explaining how to get the individual patch diffs out of bzr. This will work only for "regular", non-cherry-picked merges, and requires some manual effort.

You'll need to run bzr log, and find the patch IDs (these are the long hex numbers on the "merged:" line) of the changeset you're interested in, plus the changeset immediately before it on the same branch (which may not be on the same patch and may not be obvious at all on busy projects.) Then, run bzr diff -r revid:old-revid-string..new-revid-string.

I think this procedure really stinks, though, since it requires people to manually find previous commits from the same branch in the log.

I must be very mean but the story of madduck about XFS made me laugh. Let me explain: I exactly had the same kind of problem on my workstation 2 years ago. A single hard reboot used to totally fuck up some random files (which were opened in read-only mode before the crash, like libraries) on my file system. Since that day I regularly spite on XFS, and I am quite amused that more than 2 years later, XFS is still totally crap! I don't understand how a so broken file system can be still in the Linux kernel. Even without power outage, I used to have dead lock problems by simply moving a (big) file and opening it at the same time. Even if I heard that this bug was fixed some versions later, I totally stopped to use that filesystem in favor of ext3, which is really the more stable filesystem I have ever seen (maybe after ext2 :-)).

Not mine, I got it via Boing-Boing but I had to link to it in light of the stupid new air travel regulations.