Since the "World's First Motherboard Using LinuxBIOS Released" hype at the beginning of this year (which was incorrect btw; it was not the first supported desktop board, there were many others before), LinuxBIOS hasn't been in the news very much. That doesn't mean that there was no progress, however. We've been working hard behind the scenes to improve the LinuxBIOS code, add support for new chipsets and boards, and advance the upcoming next-generation LinuxBIOSv3 version which will brings lots of great improvements in various areas. Here's a random collection of stuff that happened in the last few months. New chipsets:

• AMD K8 / NVIDIA MCP55, contributed by Yinghai Lu of AMD
• VIA VT82C686A/B southbridge, contributed by Corey Osgood
• AMD Geode LX / CS5536, contributed by Marc Jones and Jordan Crouse of AMD
• Intel 810 northbridge, contributed by Corey Osgood
• AMD K8 / VIA K8T890 / VT8237R, contributed by Rudolf Marek / Corey Osgood
• AMD K8 / SiS761GX / SiS966(L), contributed by Morgan Tsai of SiS

New mainboards:

• Sun Ultra40, contributed by Ronald G. Minnich (LinuxBIOS project founder)
• K9SD Master-S2R (MS-9185), contributed by Bingxun Shi of MSI
• K9SD Master Series (MS-9282), contributed by Bingxun Shi of MSI
• GIGABYTE GA-M57SLI-S4, contributed by Yinghai Lu of AMD
• NVIDIA l1_2pvv, contributed by Yinghai Lu of AMD
• Supermicro H8DMR, contributed by Yinghai Lu of AMD
• Tyan S2912, contributed by Yinghai Lu of AMD
• Tyan S1846, contributed by myself
• AMD Norwich (AMD Geode LX reference platform), contributed by Marc Jones and Jordan Crouse of AMD
• IGEL Winnet III thin client, contributed by myself
• ASUS A8N-E, contributed by Phillip Degler
• IEI JUKI-511P, contributed by Nikolay Petukhov
• IEI ROCKY-512, contributed by Nikolay Petukhov
• AMD DB800 (a.k.a. Salsa), contributed by Marc Jones and Jordan Crouse of AMD
• ASUS MEW-VM, contributed by Corey Osgood
• Artec Group DBE61, contributed by Marc Jones and Jordan Crouse of AMD
• PC Engines ALIX.1C, contributed by Ronald G. Minnich
• MSI MS-6178, contributed by myself
• MSI MS-7260 (K9N Neo), contributed by myself
• IGEL-316 thin client, contributed by J rgen Beisert
• AXUS TC320 thin client, contributed by J rgen Beisert
• GIGABYTE GA-2761GXDK (Churchill), contributed by Morgan Tsai of SiS
• And a bunch of older Intel 440BX based boards, contributed by myself with some help by testers via IRC: ASUS P2B/P2B-F/P3B-F, A-Trend ATC-6220, AZZA PT-6IBD, Biostar M6TBA, Compaq Deskpro EN SFF P600, GIGABYTE GA-6BXC
• ASUS A8V-E SE, contributed by Rudolf Marek

Note that not all of these may be 100% supported, some may still be work in progress with some TODO items left... Check the LinuxBIOS wiki or ask on the mailing list for details. The future Most work will probably go into LinuxBIOSv3 in the future, in order to make it suitable for productive use.
Of course, work on new chipsets and boards will continue, too. For example the VIA CN700 chipset (plus Jetway J7F2WE board using it) is being worked on right now, probably also several others I don't know about. Call for board testers If you're interesting in trying out LinuxBIOS, please check the list of supported motherboards. If your board is not listed there, but the chipset is already supported we can probably add support for your board relatively easy with some testing help from you. Please contact us on IRC or preferrably on the mailing list if you want to help get your board supported! An (incomplete) list of good candidate boards for future support is available in the wiki. Thanks! We're very grateful for the many contributors who have helped us with testing and fixing existing code, or who even contributed code for new chipsets and motherboards. Thanks a lot! Many thanks especially to all hardware vendors who have been supporting us or even actively contributed by submitting code for their chipsets or boards (recently or in the past), including AMD, SiS, VIA, MSI, Tyan, Artec Group, and many others. Your efforts are very appreciated. Thanks!

I have often posted on the accuracy of my mail filtering mechanisms on the mailing lists (I have not had a false positive in years, and I stash all discards/rejects locally, and do spot checks frequently; and I went through 6 months of exhaustive checks when I put this system in place). False negatives are down to about 3-4 a month (0.019%). Yes, that is right: I am claiming that my classification correctness record is 99.92 (99.98% accuracy for messages my classifiers are sure about). Incorrectly classified unsure ham is about 3-4(0.019%) a month; incorrectly classified unsure Spam is roughly the same, perhaps a little higher. Adding these to the incorrect classification, my best estimate of not confidently classified mail is 0.076%, based on the last 60 days of data (which is what gets you the 99.92%). I get unsure/retrain messages at the rate of about 20 a day (about 3.2% of non-spam email) about 2/3 rds of which are classified correctly; but either SA and crm114 disagree, or crm114 is unsure. So I have to look at about 20 messages a day to see if a ham message slipped in there; and train my filters based on these; and the process is highly automated (just uses my brain as a classifier). The mail statistics can be seen on my mail server. Oh, my filtering front end also switches between reject/discard and turns grey listing on and off based on whether or not the mail is coming from mailing lists/newsletters I have authorized; mimedefang-filter However, all these numbers are manually gathered, and I still have not gotten around to automating my setup s overall accuracy, but now I have some figures on one of the two classifies in my system. Here is the data from CRM114. I ll update the numbers below via cron. First, some context: when training CRM114 using the mailtrainer command, one can specify to leave out a certain percentage of the training set in the learn phase, and run a second pass over the mails so skipped to test the accuracy of the training. The way you do this is by specifying a regular expression to match the file names. Since my training set has message numbers, it was simple to use the least significant two digits as a regexp; but I did not like the idea of always leaving out the same messages. So I now generate two sets of numbers for every training run, and leave out messages with those two trailing digits, in effect reserving 2% of all mails for the accuracy run. An interesting thing to note is the assymetry in the accuracy: CRM114 has never identified a Spam message incorrectly. This is because the training mechanism is skewed towards letting a few spam messages slip through, rather than let a good message slip into the spam folder. I like that. So, here are the accuracy numbers for CRM114; adding in Spamassassin into the mix only improves the numbers. Also, I have always felt that a freshly learned css file is somewhat brittle in the sense that if one trains an unsure message, and then tried to TUNE (Train Until No Errors) the css file, a large number of runs through the training set are needed until the thing stabilizes. So it is as if the learning done initially was minimalistic, and adding the information for the new unsure message required all kinds of tweaking. After a while TOEing (Training on Errors) and TUNEing, this brittleness seems to get hammered out of the CSS files. I also expect to see accuracy rise as the css files get less brittle The table below starts with data from a newly minted .css file. As you can see, the accuracy numbers are trending up, and already are nearly up to the values observed on my production system.

Finally after fixing a lot of code (errors caused specially because of the different behaviour used on shell scripts) here are some of the first outputs generated by the scripts.

This post should give a better idea of how DeBaBaReTools work.

---

Test packages:

• kcheckgmail (one for etch an other for sarge; with an accidental duplicate .orig in the .dsc files)
  
• zend-framework
  
• php-htscanner
  

$ ls /home/raphael/Deb/repository/pub/UploadQueue/
kcheckgmail_0.5.7.2-0+etch1.diff.gz php5-htscanner_0.8.1-1_i386.deb
kcheckgmail_0.5.7.2-0+etch1.dsc php-htscanner_0.8.1-1.diff.gz
kcheckgmail_0.5.7.2-0+etch1_i386.changes php-htscanner_0.8.1-1.dsc
kcheckgmail_0.5.7.2-0+etch1_i386.deb php-htscanner_0.8.1-1_i386.changes
kcheckgmail_0.5.7.2-0+etch1_source.changes php-htscanner_0.8.1-1_source.changes
kcheckgmail_0.5.7.2-0+sarge1.diff.gz php-htscanner_0.8.1.orig.tar.gz
kcheckgmail_0.5.7.2-0+sarge1.dsc zend-framework_1.0.2-1.diff.gz
kcheckgmail_0.5.7.2-0+sarge1_i386.changes zend-framework_1.0.2-1.dsc
kcheckgmail_0.5.7.2-0+sarge1_i386.deb zend-framework_1.0.2-1_source.changes
kcheckgmail_0.5.7.2-0+sarge1_source.changes zend-framework_1.0.2.orig.tar.gz
kcheckgmail_0.5.7.2.orig.tar.gz

I've included the already built binary packages (built by a previous, functional, version of wannaBuild) because I'm recreating my repository from scratch.

So let's run processIncoming:

$ DEBUG=1 EXTRA_PROBE_DIRS=$PWD/backends VERBOSE=1 ./processIncoming 2> /tmp/log
processIncoming v0.1 started.
Processing /home/raphael/Deb/repository/pub/UploadQueue/php-htscanner_0.8.1-1_i386.changes...
Processing /home/raphael/Deb/repository/pub/UploadQueue/zend-framework_1.0.2-1_source.changes...
Processing /home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+sarge1_i386.changes...
Processing /home/raphael/Deb/repository/pub/UploadQueue/php-htscanner_0.8.1-1_source.changes...
Processing /home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+sarge1_source.changes...
Processing /home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+etch1_source.changes...
Processing /home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+etch1_i386.changes...
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+etch1.diff.gz" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+etch1.dsc" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+etch1_i386.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+etch1_i386.deb" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+etch1_source.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+sarge1.diff.gz" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+sarge1.dsc" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+sarge1_i386.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+sarge1_i386.deb" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2-0+sarge1_source.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/kcheckgmail_0.5.7.2.orig.tar.gz" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/php5-htscanner_0.8.1-1_i386.deb" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/php-htscanner_0.8.1-1.diff.gz" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/php-htscanner_0.8.1-1.dsc" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/php-htscanner_0.8.1-1_i386.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/php-htscanner_0.8.1-1_source.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/php-htscanner_0.8.1.orig.tar.gz" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/zend-framework_1.0.2-1.diff.gz" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/zend-framework_1.0.2-1.dsc" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/zend-framework_1.0.2-1_source.changes" to "/home/raphael/Deb/repository/incoming/accepted/"
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/zend-framework_1.0.2.orig.tar.gz" to "/home/raphael/Deb/repository/incoming/accepted/"

The DEBUG=1 var makes the scripts (it works on all the scripts of DeBaBaReTools) set -x.

$ cat /tmp/log wc -l
2542

:)

Now, let's see what happens when an unsigned file is in the upload queue.
For this I've copied zend-framework_1.0.2-1_source.changes again to UploadQueue and stripped the signature information (note that it is only the .changes file):

$ DEBUG=1 EXTRA_PROBE_DIRS=$PWD/backends VERBOSE=1 ./processIncoming 2> /tmp/log
processIncoming v0.1 started.
Processing /home/raphael/Deb/repository/pub/UploadQueue/zend-framework_1.0.2-1_source.changes...
gpgv: no valid OpenPGP data found.
gpgv: the signature could not be verified.
Please remember that the signature file (.sig or .asc)
should be the first file given on the command line.
mv'ing "/home/raphael/Deb/repository/pub/UploadQueue/zend-framework_1.0.2-1_source.changes" to "/home/raphael/Deb/repository/incoming/rejected/"
processIncoming v0.1 stopped.

$ wc -l < /tmp/log
374

$ ls /home/raphael/Deb/repository/incoming/rejected
zend-framework_1.0.2-1_source.changes

Nice :D

So, let's continue with needsBuild:

$ DEBUG=1 EXTRA_PROBE_DIRS=$PWD/backends VERBOSE=1 ./needsBuild 2> /tmp/log
needsBuild v0.2 started.
Architectures: i386 amd64 all
Now processing /home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+etch1_i386.changes
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+etch1_i386.deb" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+etch1_i386.changes" to "/home/raphael/Deb/repository/incoming"
Now processing /home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+etch1_source.changes
NOT Adding kcheckgmail to be built in i386 for the etch distribution (already listed)
Adding kcheckgmail to be built in amd64 for the etch distribution
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+etch1.dsc" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2.orig.tar.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+etch1.diff.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+etch1_source.changes" to "/home/raphael/Deb/repository/incoming"
Now processing /home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+sarge1_i386.changes
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+sarge1_i386.deb" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+sarge1_i386.changes" to "/home/raphael/Deb/repository/incoming"
Now processing /home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+sarge1_source.changes
NOT Adding kcheckgmail to be built in i386 for the sarge distribution (already listed)
Adding kcheckgmail to be built in amd64 for the sarge distribution
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+sarge1.dsc" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+sarge1.diff.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/kcheckgmail_0.5.7.2-0+sarge1_source.changes" to "/home/raphael/Deb/repository/incoming"
Now processing /home/raphael/Deb/repository/incoming/accepted/php-htscanner_0.8.1-1_i386.changes
mv'ing "/home/raphael/Deb/repository/incoming/accepted/php5-htscanner_0.8.1-1_i386.deb" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/php-htscanner_0.8.1-1_i386.changes" to "/home/raphael/Deb/repository/incoming"
Now processing /home/raphael/Deb/repository/incoming/accepted/php-htscanner_0.8.1-1_source.changes
NOT Adding php-htscanner to be built in i386 for the sid distribution (already listed)
Adding php-htscanner to be built in amd64 for the sid distribution
mv'ing "/home/raphael/Deb/repository/incoming/accepted/php-htscanner_0.8.1-1.dsc" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/php-htscanner_0.8.1.orig.tar.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/php-htscanner_0.8.1-1.diff.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/php-htscanner_0.8.1-1_source.changes" to "/home/raphael/Deb/repository/incoming"
Now processing /home/raphael/Deb/repository/incoming/accepted/zend-framework_1.0.2-1_source.changes
Adding zend-framework to be built in all for the sid distribution
mv'ing "/home/raphael/Deb/repository/incoming/accepted/zend-framework_1.0.2-1.dsc" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/zend-framework_1.0.2.orig.tar.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/zend-framework_1.0.2-1.diff.gz" to "/home/raphael/Deb/repository/incoming"
mv'ing "/home/raphael/Deb/repository/incoming/accepted/zend-framework_1.0.2-1_source.changes" to "/home/raphael/Deb/repository/incoming"
needsBuild v0.2 stopped.

$ wc -l < /tmp/log
1983

And a quick look at sid's needsBuild list:

$ cat /home/raphael/Deb/repository/build/data/needsBuild.sid
php-htscanner_0.8.1-1 i386 built
php-htscanner_0.8.1-1 amd64 needsBuild
zend-framework_1.0.2-1 all needsBuild

So, next time processIncoming is run all those files will be installed in the repository :yes:

(of course, I first have to fix my reprepro setup :s)

Reading MJ Ray's post on planet.debian.org and the interesting news of how spamers "crack" CAPTCHA's I keep thinking solutions like the one offered by projecthoneypot.org might be the feature of anti-spam technologies.

---

Please let me know if you see other successful ways to reduce spam or encourage good comments.

I would say using projecthoneypot's http:BL API could help (even tough I haven't actually tried it).

But the statistics and available IP data give a good impression of the system :D

I've even suggested to collaborate and use http:BL at b.d.o but at the moment there has been no reply.

In my first post about my current problems with Debian, I noticed that there doesn't seem to be a common vision of Debian's future. My last post was about Debian's current state of moving from stable release to stable release without actual aims.

Today I would like to rant about our lack of enthusiasm. Could you name the biggest new feature of Debian Etch? I think the graphical installer is very cool, it made the installation look a lot friendlier and made translations to languages not based on the Latin alphabet easier. But what else happened in Etch? We added SecureApt, which might be cool if you are a hardcore hacker, but leaves most people bored. The same is true for using UTF8 for new installations and the addition of udev to the default install. If you don't know where I got this impressive list of thing - look at the Debian Wiki.

Most of the things listed on that page miss the bling we see in other distributions. Ubuntu 7.10 introduced Compiz bling while Fedora 7 had full NetworkManager support and tried to "just work" [tm]. Yeah, Debian isn't that far away from such things, whenever we release, such things are also picked up. But somehow it doesn't look like people would get out and party because we include the newest and coolest feature - no, Debian Developers prefer to discuss such things as the default MTA.

Of course, we need to discuss the default MTA - and exim4 might not be perfect for the job - but we should also start to think about advertising the features that impress non-geeks, so that we can go out and say "Hey, lenny came out and now we have a full dependency based init system and a smaller MTA and our kernel is now completely free of binary blobs" to our geek friends, while our non-geek friends can hear about the new easy-to-use and beautiful desktop environments and configuration applications that allow them to just install and use their Debian, without moving to a command line once.

We are half-way there. Debian is cool. Debian becomes better every day (*cough* please look away from the RC bug graph, thanks). We should start telling people about it and have fun doing so.

On Friday 2007-11-02, I watched Larry Lessig give a presentation at the University of Washington entitled Is Google (2008) Microsoft (1998)? Since it was Lessig, the talk was articulate and thought-provoking, and he used his slides very well (unlike many presenters who just read bullet points).

The argument he made is that Google is like Microsoft in some ways (and so are other companies like Facebook). In his analysis, the problem with Microsoft was that so many companies made themselves Microsoft-dependent, which both forced other companies to follow suit (due to network effects) and gave Microsoft leverage they could use against everyone in their ecosystem (e.g., Netscape). According to Lessig, Microsoft was not evil to use this leverage, but you have to recognize that Microsoft will do what is best for Microsoft, and this may not be what is best for you. He then suggested that we should be aware of the potential for the same thing to happen; in support of this point, he quoted excerpts from the terms of service from the APIs of Google and Facebook, which contain typical statements like we reserve the right to terminate your use of this service for any reason or for no reason, and asked: What if Microsoft had written this license agreement?

But it seemed to me that this wasn't his main point. He was using this observation as a springboard to talk about issues revolving around competition and public policy. Specifically, he feels that we should work towards a more just model of interaction between companies and users whose work makes the company more valuable (by contributing to a software ecosystem, by posting content on the Web site, etc). But the correct way to do this (he says) is not via competition, after-the-fact litigation, or voluntary codes of conduct: competition may not be around to enforce good behavior (as we've seen with Microsoft), litigation may be too late and ineffective (again, see Microsoft), and voluntary codes of conduct will disappear as soon as they get in the way of the bottom line. The correct approach, he argued, is across-the-board regulation so that all the companies have to play by the same rules.

From there he went on to talk about governmental corruption. He said (quoting Robert Reich's book Supercapitalism) that the reason we consistently look for solutions in the market and in voluntary compliance is that our governmental system is broken and does not effectively regulate corporations in the public interest. But Lessig is optimistic that we can change things (he joked that his publisher was unhappy with this point of view, because his brand has been built on pessimism). In his view, the politicians in Washington, by and large, want to be honest and do good, but they aren't able to within the current political system. For instance, on the few occassions that he managed to get access to lawmakers to discuss copyright issues, it was often the first time they had heard that there was more than one side to the argument. He thinks we need a national political movement that will shame politicians into being less corrupt.

This is all a simplification of what he said, of course; unfortunately, I wasn't able to find a recording of this talk on Lessig's Web site, or I'd point you at primary source material. I don't know if this is because he chose not to post it, or because it just hasn't been made available yet.

My view

In general, I thought that the speech was very interesting and well thought out. But as much as I'd like to buy into his optimism that we can fix our system, I don't think it's well-founded, for two reasons.

First, he argued that competition will resolve some of these problems (that was before and after he said we can't rely on competition, which I found confusing and may indicate that I misunderstood something). His evidence for this seemed to be that when Microsoft was dominating the world, users responded by reducing their dependence on Microsoft products and by switching to alternatives such as Linux. But, in fact, although more people may be using Linux than in the past, virtually everyone is still only using Windows, it's still almost impossible to find a programming job that isn't Windows-only, and Microsoft is still raking in money hand over fist; indeed, they're doing so well that they're planning to expand their workforce by a third. To take just one data point of many: I ride the bus to Seattle and back, and I regularly see other people using laptops as they commute. Virtually all the laptops run Windows. Occasionally I'll see a Mac, and every few months I'll see a guy running Linux on his laptop; I say "a guy" because it's the same guy every single time. So by my estimation, use of Linux on laptops in buses on the 545 route from Redmond to Seattle is way below 1%, with Macintoshes somewhere around 5%.

To broaden my point, I think Lessig has fallen into a trap that highly intelligent people, particularly those in academia, tend to fall into. In academia, you are surrounded by people who value learning and thinking deeply about the world. These are people who will give serious and unbiased consideration to questions like what is the social effect of my acceptance of the Facebook Terms of Service? and If I become dependent on this service, will that possibly affect me in five years if the company decides to act against my interests? The problem with this is that these people are not representative of the population at large. The population at large thinks why is this check-box getting in the way of me sending pictures to my friends? And I have no doubt that any kid who, e.g., refuses to sign up for Facebook as a protest against their TOS will be roundly mocked in their social circles for being a weird antisocial nonconformist. (of course, this may not apply to the small minority of people like myself whose social circles consist of weird antisocial nonconformists)

Second, Lessig's optimism about the political system seemed to be based on his observation that the people in government are, by and large, not venally corrupt: they want to do good, but the system is constructed in a way that makes this impossible. To me, that's a tremendously disheartening statement. If our problem were simply a few, or even many, corrupt politicians, this would be a solvable problem: even nowadays, Americans have some ability to choose who gets elected, and I believe that a sufficiently well-coordinated campaign could replace bad politicians with good ones.

But if the problem is that the system is corrupt even when the individuals are honest, it's a much deeper problem and frankly one that I don't know how to solve. While individuals have some direct control over who gets elected, we have no way to directly change the system that produces the corruption; the levers to do this are in the hands of the politicians elected through that broken system. It's unlikely that the politicians themselves will fix the problem, because they've benefited deeply from it: if the system is left as it is, well over 90% of Congress members will keep
their jobs in the next election cycle, and any effective change seems likely (by definition) to disrupt this cushy arrangement.

On the other hand, running new candidates for office will not fix the problem. After all, if the problem is that the system is corrupt even when the participants are honest, then putting more honest people into government will have no effect. In order to get elected, these new politicians will have to become just as corrupt as the old ones, because that's how the system is set up. So we might change the faces, but we'll have the same old problems.

I recall two specific concrete proposals that Lessig made to make the system more responsive. First, he suggested shaming politicians who engage in corrupt behavior -- presumably taking large donations from interested parties. I don't see how this will help. Americans generally have a very low regard for the political class as it is; will pointing out particular examples of politicians taking money really make a difference when it comes to votes? And if it doesn't, will it really produce any change in their behavior? Politicians are motivated by votes the same way corporations are motivated by money; unless they stand a chance of losing their job (which is virtually impossible in any event), there's no reason for them to change how they behave.

His second proposal was for a system of contributions where it's impossible for any individual to prove they had donated a particular amount of money to a particular candidate. Even assuming it could be implemented perfectly, I don't think this will really solve the problem, for two reasons. First, it's not generally a secret what moneyed interests want. Politicians who want to attract large donations can just take positions that they know will appeal to donors of that sort. The donors themselves can make this easier on the candidates by, e.g., posting public position statements on issues of the day. Secondly, and this is far more insidious, even if politicians honestly represent their positions, then because success in running for office is so tied to the amount of money the candidate can raise, only candidates who hold opinions favorable to large corporations and wealthy individuals will be able to get elected. In fact, for all I know this is what happens already!

But with all that said, it was a really interesting talk and Lessig tied together some things you might not think would go together. If a video or Flash presentation becomes available, I would recommend watching through it. And hopefully my pessimistic predictions will be wrong; it's certainly happened before.

From Debian I downloaded, installed and toyed around with Liquidsoap. It took me a few moments to understand the language, but once I did, I got very enthusiastic about it. I always wanted something more than just streaming a set for my friends now and then and this tool allows me to created a quite contrived webradio setup. More about that later… To Debian unstable I finally uploaded Camping, a small Ruby web framework for MCV type applications. It is in NEW now. I’ve been writing (or at least started writing) quite a few apps based on Camping the past few months. Primarily because it is fun, but also because I like its flexibility and strength very much (see also some examples). Once Camping is in Debian, I’d like to put more Camping apps out in the open.

Hello World ok, second night of my birhday is over, went again to the pub with some of my very important people. The day today was quite different - not only joy, fun, and … Today also other facts were playing an important role - Remember people who can’t be here tonight (dead), lost love and new targets. Nevertheless, I’m quite happy and enjoy the last hours of this night. Regards, Mario (with different emotions)

I'm just too bored that everytime I try to write a blog post I end up closing the tab and get back working on something.

---

Not even the fact that DeBaBaReTools is *almost* functional now (at least on my local PC, it will be for the rest of the world as soon as I commit the changes) except for all the hardcoded stuff in wannaBuild.

And not even the fact that there's a new DSA (weasel) stops my boredom (maybe if he could take care of a RT... that would let me spend some time spaming the BTS with over 800 bug reports from DEHS :D).

There seems to have been a lot of talk recently about using Open Source software for social networking, which has been something that has been at the back of my mind for quite a while now. If you could take a web site and using the presence information and IM features from jabber, you already have the beginnings of a social networking or community site. The main advantage I can see is that anyone could start their own social networking site, rather than joining one of the larger existing (closed source) sites or basing a community around a web forum (which I tend to avoid if at all possible). Now if I can just properly gather my thoughts on this I would start some coding.

There seems to have been a lot of talk recently about using Open Source software for social networking, which has been something that has been at the back of my mind for quite a while now. If you could take a web site and using the presence information and IM features from jabber, you already have the beginnings of a social networking or community site. The main advantage I can see is that anyone could start their own social networking site, rather than joining one of the larger existing (closed source) sites or basing a community around a web forum (which I tend to avoid if at all possible). Now if I can just properly gather my thoughts on this I would start some coding.

Two nights ago I was having dinner with a colleague who recounted a truly horrible experience he had with the modern-day American health care “system.” While in Montana over the summer, he got bit by a bat and well, as a result, he and his entire family had to get rabies shots. These shots are hard to get and priceeeeeeeeeeeeeeeeeeey. The final bill was $ 20,000 and his NYU based health insurance, United Health Care is claiming they are not responsible!### Like getting bit by a bat and the possibility of rabies are not bad enough. Isn’t the point of health insurance to take care of the unexpected?? Someone also pointed me to The Daily Kos’ wrangle with BCBS of California and well, they picked the wrong guy to mess with given there are so many many many many eyeballs tuned to his blog, everyday (just check out the number of comments). I wonder if the public relations director will contact him as they did with me after I blogged about my trials and tribulations. I certainly don’t have the audience and readers he does, so I am always surprised they ever contacted me in the first place. The Daily Kos also provided a link for a site that I was thinking of creating if my problems with BCBS were not solved (glad to see someone else has) called Sick of Blue Cross. I personally like the name I came up with a little better Blue Cross No Shield but the important thing is that a site like that exists and it is very good site, design and content wise. There should be a site created for every health insurance company and they should federate and pool resources to gain more visiblity. While I am not hopeful that the health insurance industry will change within, I think mounting and steady consumer pressure can do a lot to force the industry to change their awful ways. And while I think the bad publicity is a key part to putting pressure on the health insurance to change, I actually think that the power of the consumers also can come from elsewhere, and this elsewhere is overwhelming the administrative channels within and outside of the health insurance. The gist is if they send us a lot of (useless) paperwork, send them A LOT of paperwork back. On the Daily Kos, a lot of people noted that things got moving when they complained to the Department of Banking and Insurance. While my own complaint to that department did not produce any immediate fruit, it did get the ball moving for the appeal process, which eventually panned out for me. That is, I learned a lot and started to get decent customer service. Doors opened. Now, it took me weeks to find out that I could complain to the Department of Banking and Insurance and I am sure many find themselves in that situation of walking in a no man’s land of confusion and frustration. If more and more people know that you can file a complaint to these Departments (and why are doctor’s offices not relaying this information??!!), they and the health insurance companies will get inundated with that which they love to inflict on us, paperwork. This may slow things down for you but at a certain point, state governments will get the message that something is deeply wrong. So, people MUST use these channels and overwhelm these agencies with complaints, faxes, emails as this is a powerful and very material message that something is deeply wrong and broken and must be fixed. Also, send faxes and letters to various departments in your health insurance company (appeals, customer service, medical director’s office, public relations office…) get people’s email’s and send them emails. Inundate them with a barrage of administrative paperwork. If they are not being helpful, send a fax or two a day concerning your case until you get clear answers. While individual faxes won’t make much of a difference, imagine if there are 300 a day coming in or 3000…. This can work with the numbers are there and I have a feeling that the numbers Are There. So spread the word for housebreaking your health insurance and push the paperwork back on them.

Two weeks from now, the NW-European Programming Contest (NWERC) of the ACM ICPC will have started here in Utrecht. Things are going well - the problem set is ready - but, as always, a lot of small things are still to do: ordering t-shirts and balloons, finalizing food and buildings, etc. However, I think if we would halt all preparations now we would at least have a basic contest in two weeks. Registration is almost over, and the list of teams shows that we have participants from nine countries representing 26 institutions distributed over 51 teams.

Switching from aterm to xterm I found it annoying that xterm didn’t select a full url when i did a double-click somewhere on it, like aterm did. Now, doing a little search on it I found something way stronger. Take a url like http://foo.bar/baz for the following example. If you now put something like the following into your .Xdefaults (and load it with xrdb -load .Xdefaults), you get xterms selection in a more useful way. A double click on foo will select foo.bar, clicking three times it selects the full url (well, better speak is to say it selects everything separated by whitespace). Click four times and it selects everything up to the start of the prompt (default prompt that is). 5 clicks selects the whole line.

XTerm*on2Clicks:  regex [^/@ \n]+
XTerm*on3Clicks:  regex [^ \n]+
XTerm*on4Clicks:  regex [^#$]+
XTerm*on5Clicks:  line

John Goerzen blogs about his experience of using vim for about 8 months. Having been an emacs user a few years back, I understand both editors on a basic level, and can relate to some of the things he says. I have to disagree with one particular bit, though:

It starts faster. I'm not sure if that really was true even when I switched, but it certainly isn't true on any of my machines today. Both Vim and Emacs have had major version upgrades (v7 and v22, respectively) since I started using Vim. People seem to say that Emacs 22 feels faster, though I don't know if that's true. The startup times of the two, if they're different, are imperceptible.

Sorry, but that just isn't true. Using my very non-scientific method of measuring startup times ("click on the icon, watch the clock, and start counting until the editor is ready to use"), I can tell that gvim on my hardware takes less than a second to start, whereas emacs requires approximately 5 seconds to do the same. For reference, this is on my laptop, a 1.3Ghz PowerPC G4. In any case, I'm not a fan of either editor. There are things that vi does well, and there are things that emacs does well. On emacs' end, I like M-x gdb, I slightly prefer its M-x compile over vim's :make, and I just love psgml. I've also occasionally used gnus for news reading (never for mail, though). On vim's end, I find regex search & replace easier (as you mention in your blog post), and I prefer its syntax highlighting (I've seen emacs get confused because the first line in the output window started in the middle of a string. Such things will never happen to vim: if it gets confused once on a given file, it will always be confused on the same file. While annoying, at least you know it's there). All in all, I don't think it matters much. I've seen people who use an editor as an operating system, both in the vim and in the emacs camp, by having a ~/.vimrc or ~/.emacs that's larger than my collection of pictures[1]. I'm not one of them; to me, an editor that doesn't properly work in its defaults shouldn't exist. [1] okay, that's an exaggeration. Still.

I have just read an interesting article titled Why Crunch Mode Doesn’t Work [1] which documents the research on efficiency vs amount of time spent working (and by inference amount of time spent on leisure activities and sleep). It shows that a 40 hour working week was chosen by people who run factories (such as Henry Ford) not due to being nice for the workers but due to the costs of inefficient work practices and errors that damage products and equipment. Now these results can only be an indication of what works best by today’s standards. The military research is good but only military organisations get to control workers to that degree (few organisations try to control how much sleep their workers get or are even legally permitted to do so), companies can only give their employees appropriate amounts of spare time to get enough sleep and hope for the best. Much of the research dates from 80+ years ago. I suspect that modern living conditions where every house has electric lights and entertainment devices such as a TV to encourage staying awake longer during the night will change things, as would ubiquitous personal transport by car. It could be that for modern factory workers the optimum amount of work is not 40 hours a week, it could be as little as 30 or as much as 50 (at a guess). Also the type of work being done certainly changes things. The article notes that mental tasks are affected more than physical tasks by lack of sleep (in terms of the consequences of being over-tired), but no mention is made about whether the optimum working hours change. If the optimum amount of work in a factory is 40 hours per week might the optimum for a highly intellectual task such as computer programming be less, perhaps 35 or 30? The next factor is the issue of team-work. In an assembly-line it’s impossible to have one person finish work early while the rest keep working, so the limit will be based on the worker who can handle the least hours. Determining which individuals will work more slowly when they work longer hours is possible (but it would be illegal to refuse to hire such people in many jurisdictions) and determining which individuals might be more likely to cause industrial accidents may be impossible. So it seems to me that the potential for each employee to work their optimal hours is much greater in the computer industry than in most sectors. I have heard a single anecdote of an employee who determined that their best efficiency came from 5 hours work a day and arranged with their manager to work 25 hours a week, apart from that I have not heard any reports of anyone trying to tailor the working hours to the worker. Some obvious differences in capacity for working long hours without losing productivity seem related to age and general health, obligations outside work (EG looking after children or sick relatives), and enjoyment of work (the greater the amount of work time that can be regarded as “fun” the less requirement there would be for recreation time outside work). It seems likely to me that parts of the computer industry that are closely related to free software development could have longer hours worked due to the overlap between recreation and paid work. If the amount of time spent working was to vary according to the capacity of each worker then the company structures for management and pay would need to change. Probably the first step towards this would be to try to pay employees according to the amount of work that they do, one problem with this is the fact that managers are traditionally considered to be superior to workers and therefore inherently worthy of more pay. As long as the pay of engineers is restricted to less than the pay of middle-managers the range between the lowest and highest salaries among programmers is going to be a factor of at most five or six, while the productivity difference between the least and most skilled programmers will be a factor of 20 for some boring work and more than 10,000 for more challenging work (assuming that the junior programmer can even understand the task). I don’t expect that a skillful programmer will get a salary of $10,000,000 any time soon (even though it would be a bargain compared to the number of junior programmers needed to do the same work), but a salary in excess of $250,000 would be reasonable. If pay was based on the quality and quantity of work done (which as the article mentions is difficult to assess) then workers would have an incentive to do what is necessary to improve their work - and with some guidance from HR could adjust their working hours accordingly. Another factor that needs to be considered is that ideally the number of working hours would vary according to the life situation of the worker. Having a child probably decreases the work capacity for the next 8 years or so. These are just some ideas, please read the article for the background research. I’m going to bed now. ;)

• [1] http://www.igda.org/articles/erobinson_crunch.php

Share This

I have ordered my new MacBook today. The model silently got an update yesterday and that was what I was waiting for. While my “old” PowerBook (4 years old) is still working fine, it has a broken hinge and unexplainable display-startup problems. Combined with the fact that it always connected to all my peripheral devices, external drives and a second screen, it is less then ideal to just pick it up and go. (Although we have XRandr 1.2 now, the whole desktop environment is not integrated with these features yet). Moreover, because I somehow ended up in a 5 year cycle of needing a computer system update and something portable at the same time, I always bought a heavy laptop. This time, I want to break this cycle, and buy a light, portable laptop and maybe later some workstation. And why Apple or a MacBook… well, I have good experiences with it, well-designed and thought-through, and if you look for portable laptops, 1000 , with a 12-14” screen, quality hardware and good specs, the MacBook is on of the few options that is left, actually.

I was quoted in a recent Needham Times article regarding my complaints with the T:

But Adam Rosi-Kessel, a Roslindale resident who takes the Needham Line to his job in the financial district, said 15- to 20-minute delays have kept him from getting to work before 9 a.m. since May. Now Rosi-Kessel, a lawyer who biked to work when he lived in Jamaica Plain, is thinking about trading in his Charlie Card again. It used to be, living here in Roslindale, the train was always the fastest way to get in, he said. Now, it s slowed down to the point where it s starting to get competitive with biking again.

I’m glad they got my biking quote. Here’s the part I don’t understand (emphasis added):

Massachusetts Bay Transportation Authority spokeswoman Lydia Rivera said Needham Line delays have been caused by construction work and speed restrictions on tracks that don t go anywhere near Needham. In order to maximize train use, the T often transfers trains between tracks, rather than leaving underused trains empty in the MBTA yard. The method helps the T get trains where they re needed most, but it also means a delay on one track can slow service on a seemingly unrelated track. It s a Catch-22, Rivera said.

I don’t think the T spokesperson knows what a “Catch-22″ is. I refer those interested to the Wikipedia article on the subject, but it is nicely summarized there as “heads I win, tails you lose.” The T’s problem is not a logical paradox, but some combination of inadequate resources and poor organization. Can someone please help them out? Also on Boston local topics: I’ve received several unattributed mailings lately related to the city council election that do not facially promote any particular candidate. Today’s mailing attacked Stephen Murphy for repeatedly trying (and failing) to win some other office or get some other job than City Councilor. The return address was 31 Milk Street, which is the address of many different businesses. A few days ago, I received another unattributed mailing bemoaning how long it has been since an at-large city councilor came from the Parkway Area (my neighborhood), but not mentioning any candidate in particular. My guess is these mailings are all meant to support John Connolly, a West Roxbury resident, attorney, and ostensibly good guy. I was feeling pretty happy about the possibility of Connolly replacing Murphy on the Council (I have to admit some unfair prejudice in that the only house in our neighborhood I’ve ever seen prominently posting a sign in support of Bush also features a Murphy billboard). But these questionable campaign tactics are giving me pause. Does anyone know anything more about this? In any case, here are my endorsements for next week’s election, notwithstanding the concern outlined above: (1) Felix Arroyo, (2) John Connolly, and (3) Sam Yoon. I have no pick for a fourth candidate. I’ll also throw in a vote for Matt Geary. I really wouldn’t want the socialists running the City, but a broader spectrum of political opinion within the council wouldn’t hurt. Incidentally, it is remarkably hard to find any information summarizing all of the candidates’ positions and records. It’s almost as if the election isn’t even happening. (A commenter properly points me to Brighton Centered as a good resource.) 11/3/07 Update: More discussion here. Technorati Tags: Boston, City Council, Elections, Felix Arroyo, John Connolly, Sam Yoon, Stephen Murphy, Politics, MBTA, Transportation, Roslindale, Needham, Matt Geary

Share This

Other Unix systems apparently calculate the load average differently to Linux. According to the Wikipedia page about Load(computing) [1] most Unix systems calculate it based on the average number of processes that are using a CPU or available for scheduling on a CPU while Linux also includes the count of processes that are blocked on disk IO (uninterruptible sleep). There are three load average numbers, the first is for the past minute, the second is for the past 5 minutes, and the third is for the past 15 minutes. In most cases you will only be interested in the first number. What is a good load average depends on the hardware. For a system with a single CPU core a load average of 1 or greater from CPU use will indicate that some processes may perform badly due to lack of CPU time - although a long-running background process with a high “nice” value can increase the load average without interfering with system performance in most cases. As a general rule if you want snappy performance then the load average component from CPU use should be less than the number of CPU cores (not hyper-threads). For example a system with two dual-core CPUs can be expected to perform really well with a load average of 3.5 from CPU use but might perform badly with a load average of 5. The component of the load average that is due to disk IO is much more difficult to interpret in a sensible manner. A common situation is to have the load average increased by a NFS server with a network problem. A user accesses a file on the NFS server and gets no response (thus giving a load average of 1), they then open another session and use “ls” to inspect the state of the file - ls is blocked and gives a system load average of 2. A single user may launch 5 or more processes before they realise that they are not going to succeed. If there are 20 active users on a multi-user system then a load average of 100 from a single NFS server that has a network problem is not uncommon. While this is happening the system will perform very well for all tasks that don’t involve the NFS server, the processes that are blocked on disk IO can be paged out so they don’t use any RAM or CPU time. For regular disk IO you can have load average incremented by 1 for each non-RAID disk without any significant performance problems. For example if you have two users who each have a separate disk for their home directory (not uncommon with certain systems where performance is required and cooperation between users is low) then each could have a single process performing disk IO at maximum speed with no performance problems for the entire system. A system which has four CPU cores and two hard drives used for separate tasks could have a load average slightly below 6 and the performance for all operations would be quite good if there were four processes performing CPU intensive tasks and two processes doing disk intensive tasks on different disks. The same system with six CPU intensive programs would under-perform (each process would on average get 2/3 of a CPU), and if it had six disk intensive tasks that all use the same disk then performance would be terrible (especially if one of the six was an interactive task). The fact that a single load average number can either mean that the system is busy but performing well, under a bit of load, or totally overloaded means that the load average number is of limited utility in diagnosing performance problems. It is useful as a quick measure, if your server usually has a load average of 0.5 and it suddenly gets a load average of 10 then you know that something is wrong. Then the typical procedure for diagnosing it starts with either running “ps aux grep D” (to get a list of D state processes - processes that are blocked on disk IO) or running top to see the percentages of CPU time idle and in IO-wait states. Cpu(s): 15.0%us, 35.1%sy,  0.0%ni, 49.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7331 rjc      25  0  2868  640  312 R  100  0.0  0:21.57 gzip Above is a section of the output of top showing a system running gzip -9 < /dev/urandom > /dev/null. Gzip is using one CPU core (100% CPU means 100% of one core - a multi-threaded program can use more than one core and therefore more than 100% CPU) and the overall system statistics indicate 49.9% idle (the other core is almost entirely idle). Cpu(s):  1.3%us,  3.2%sy,  0.0%ni, 50.7%id, 44.4%wa,  0.0%hi,  0.3%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7425 rjc      17  0  4036  872  588 R    4  0.1  0:00.20 find Above is a section of the output of top showing the same system running find /. The system is registering 44% IO wait and 50.7% idle. The IO wait is the percentage of time that CPU core is waiting on IO, so 44% of the total system CPU time (or 88% of one CPU core) is idle while the system is waiting for disk IO to complete. A common mistake is to think that if the IO was faster then more CPU time would be used, in this case with the find program using 4% of one CPU core if all the IO was instantaneous (EG in cache) then the command would complete 25 times faster with 100% CPU use. But if the disk IO performance was doubled (a realistic possibility given that the system has a pair of cheap SATA disks in a RAID-1) then find would probably use 8% of CPU time. Really the only use for load average is for getting an instant feel for whether there are any performance problems related to CPU use or disk IO. If you know what the normal number is then a significant change will stand out. Dr. Neil Gunther has written some interesting documentation on the topic [2], which goes into more technical detail including kernel algorithms used for calculating the load average. My aim in this post is to educate Unix operators as to the basics of the load average. His book The Practical Performance Analyst gives some useful insights into the field. One thing I learned from his book is the basics of queueing theory. One important aspect of this is that as the rate at which work arrives approaches the rate at which work can be done the queue length starts to increase exponentially, and if work keeps arriving at the same rate when the queue is full and the system can’t perform the work fast enough the queue will grow without end. This means that as the load average approaches the theoretical maximum the probability of the system dramatically increasing it’s load average increases. A machine that’s bottlenecked on disk IO for a task where there is a huge number of independent clients (such as a large web server) may have it’s load average jump from 3 to 100 in a matter of one minute. Of course this won’t mean that you actually need to be able to serve 30 times the normal load, merely slightly more than the normal load to keep the queues short. I recommend reading the book, he explains it much better than I do.

• [1] http://en.wikipedia.org/wiki/Load_(computing)


• [2] http://www.teamquest.com/resources/gunther/display/5/index.htm

Share This

I'd been using Emacs for quite awhile, and about 8 months ago I decided I would try using Vim. I'd only used vi for system emergency work, but knew a number of people that swore by it for regular work. So I decided I would learn Vim and use it for my regular work. I figure that with things like this, I don't get a real feel for how well they work unless I use them for all my work. So I haven't really opened Emacs at all in the past 8 months.

Yesterday I finally decided that Vim was not living up to my expectations and I'm in the process of switching back to Emacs. I thought I ought to write down why I'm doing that, for my own future reference... and since nobody has ever written about Emacs vs. Vim, I might as well post it where everyone can see it.

So here we are.

Original Reasons for Using Vim

It would lead to more comfortable typing. Lots of Vim users mention that you don't have to hold down keys while hitting other keys as much in Vim as in Emacs, and that the movement keys are all on the home row. That's true, but I didn't find it to be that big of an improvement, since Esc is a farther reach than anything in Emacs, and let me tell you, you're hitting Esc all the time in Vim. I found that removing the armrests from my chair made my hands happier than Vim ever did, and swapping Ctrl and CapsLock in Emacs will probably help there too.

It starts faster. I'm not sure if that really was true even when I switched, but it certainly isn't true on any of my machines today. Both Vim and Emacs have had major version upgrades (v7 and v22, respectively) since I started using Vim. People seem to say that Emacs 22 feels faster, though I don't know if that's true. The startup times of the two, if they're different, are imperceptible.

Vim would use less RAM. Frankly, these days, both Emacs and Vim are way down on the list of things that use up RAM. Heck, kmail has 141MB resident, and each of its two IMAP processes is using more than 30MB. Emacs in X right after start has 16MB resident, 10MB of which is shared, and 25MB VSS. gvim right after start has 8MB resident, 5MB of which is shared, and a 43MB VSS. Emacs tends to use fewer processes for things that vim. So they're not all that different, and Emacs could come out smaller in certain situations. But the difference is irrelevant on today's machines, and modern Gnome and KDE apps are many times larger than both of them.

It will make me more comfortable in rescue environments where I have only traditional vi available. Actually, the vi on AIX is so different from modern Vim that this didn't really help.

It would make me more productive. There are some editing commands that did, but as you'll see below, it was more than balanced out by other problems.

Things I Liked about Vim

The commands dt, dT, df, dF. Wonderful little things those. Emacs now has M-z (Zap), which is similar to df but can actually go to other lines (a nice addition). And there are easy ways to bind keys to the others as well, though that doesn't make it a pervasive convention like it is in Vim.

Antialiased fonts. It's crazy that Emacs doesn't have this yet. But not a showstopper; I still like good ole 10x20 just fine.

Regexp search-and-replace. Emacs actually has this now, and maybe it had it back then too. M-C-%. Apparently in Emacs22 the replacement expression can also have lisp code in it, which sounds really slick but I can't see myself using it regularly.

Annoying Things in Vim

Syntax highlighting. The syntax highlighting for most languages in Vim felt like it was about as smart as it was in Emacs about 10 years ago. Strings like "Hello!\"" (in languages where \" inserts a literal ") often confused it. Sometimes quotes within comments confused it. Sometimes it would be confused permanently. Other times, just until I scrolled around in the file or reloaded it.

Indentation. This is much more annoying than the syntax highlighting, really. In many languages -- and especially the two modes I've used most recently, XML and Haskell -- it really, really stinks. The indentation there isn't aware of syntax, or not very much. Sometimes it is smart enough to know that if an XML line starts with </ that it moves left and if it starts with an opening tag, that the next line moves right. But it's not smart enough to do this reliably. Not only that, but indentation is not handled with consistent configuration between languages. And even though Vim ships with a ton of language modes, the central docs only cover indentation for C.

I've asked Vim experts about this, and have tried all sorts of various tweaks, have read through Vim indentation mode source files, etc. There is just no way to get it anywhere near the intelligence of Emacs for most languages, short of writing my own mode, it appears. This is even worse because when using the backspace key in insert mode, for awhile it deletes individual spaces, and then all of a sudden deletes a big chunk of whitespace back to the beginning of the line. (And no, the insertion of Tab characters is disabled.) Indentation is my #1 complaint about Vim, and something that shows no progress towards being fixed any time soon.

And forget about anything like Emacs M-x reindent-region. This is a syntax-aware indenter. You can write out an entire source file with no indentation whatsoever, and it will indent the entire thing according to the indentation rules you've defined and the syntax of the language you're using. The best I've seen in Vim are commands that add or remove space at the beginning of every line in a region.

In short, Emacs seems to "understand" the file format on a much deeper level than Vim, and can automate things to a much better extent because of it.

Too many things disrupt the paste buffer. I can use Y or y to yank some text in Vim, and it's really, really easy to overwrite that buffer with other things. Yes, I know that I can yank it into a named buffer, but that's inconvenient and I don't usually know in advance that I'll have that need. In Emacs, only C-k and other "large area" commands disrupt it.

Vim doesn't like you having lots of files open at once. It's surprisingly convoluted to do this. If you use the basic documented command to edit another file, :e, it closes the file you're working on. The normal way to open multiple files at once is to use split windows. Well, I don't like split windows all that well, and often just want to make a quick change in one file -- in full screen -- and then go back to another. Even though I use set hidden in my ~/.vimrc, it still is annoying and more convoluted than it should be.

Vim can't create new top-level X windows. In Emacs, I can press C-x 5 2, and poof, I have a second Emacs window in X, and it's tied to the same editing session and Emacs process. Not a new process, with a different set of files, its own buffers, etc. The same process, same set of files. Just like a split window, but with a new top-level X window instead. gvim simply has no way to do that. This is also a large annoyance.

gqap stinks. This has burned me more than once. I'll be editing an XML document, and insert some text in the middle of a paragraph. Now I have a really wide line. So I type gqap to reformat the paragraph. My cursor is near the bottom of the screen, so I don't really see much past the current line. I then save the document and exit. Later I discover that vim considered the entire rest of the document part of the single paragraph, and removed all the different indentation levels at </para> and the like, so it's completely messed up. Emacs is smart enough to know what is a paragraph in XML mode, and M-q does the right thing. Oh, and Emacs reindent-region can fix the Vim gqap-induced mess.