I have often posted on the accuracy of my mail filtering mechanisms on the mailing lists (I have not had a false positive in years, and I stash all discards/rejects locally, and do spot checks frequently; and I went through 6 months of exhaustive checks when I put this system in place). False negatives are down to about 3-4 a month (0.019%). Yes, that is right: I am claiming that my classification correctness record is 99.92 (99.98% accuracy for messages my classifiers are sure about). Incorrectly classified unsure ham is about 3-4(0.019%) a month; incorrectly classified unsure Spam is roughly the same, perhaps a little higher. Adding these to the incorrect classification, my best estimate of not confidently classified mail is 0.076%, based on the last 60 days of data (which is what gets you the 99.92%). I get unsure/retrain messages at the rate of about 20 a day (about 3.2% of non-spam email) about 2/3 rds of which are classified correctly; but either SA and crm114 disagree, or crm114 is unsure. So I have to look at about 20 messages a day to see if a ham message slipped in there; and train my filters based on these; and the process is highly automated (just uses my brain as a classifier). The mail statistics can be seen on my mail server. Oh, my filtering front end also switches between reject/discard and turns grey listing on and off based on whether or not the mail is coming from mailing lists/newsletters I have authorized; mimedefang-filter However, all these numbers are manually gathered, and I still have not gotten around to automating my setup s overall accuracy, but now I have some figures on one of the two classifies in my system. Here is the data from CRM114. I ll update the numbers below via cron. First, some context: when training CRM114 using the mailtrainer command, one can specify to leave out a certain percentage of the training set in the learn phase, and run a second pass over the mails so skipped to test the accuracy of the training. The way you do this is by specifying a regular expression to match the file names. Since my training set has message numbers, it was simple to use the least significant two digits as a regexp; but I did not like the idea of always leaving out the same messages. So I now generate two sets of numbers for every training run, and leave out messages with those two trailing digits, in effect reserving 2% of all mails for the accuracy run. An interesting thing to note is the assymetry in the accuracy: CRM114 has never identified a Spam message incorrectly. This is because the training mechanism is skewed towards letting a few spam messages slip through, rather than let a good message slip into the spam folder. I like that. So, here are the accuracy numbers for CRM114; adding in Spamassassin into the mix only improves the numbers. Also, I have always felt that a freshly learned css file is somewhat brittle in the sense that if one trains an unsure message, and then tried to TUNE (Train Until No Errors) the css file, a large number of runs through the training set are needed until the thing stabilizes. So it is as if the learning done initially was minimalistic, and adding the information for the new unsure message required all kinds of tweaking. After a while TOEing (Training on Errors) and TUNEing, this brittleness seems to get hammered out of the CSS files. I also expect to see accuracy rise as the css files get less brittle The table below starts with data from a newly minted .css file. As you can see, the accuracy numbers are trending up, and already are nearly up to the values observed on my production system.

These questions came in as comments this week, but I think they're worth a whole post. Peter Swan asked:

"This may seem like a dumb question, but I have not seen anywhere the answer to it. Can a satellite dish be mounted in a loft space or must it be mounted outside?"

There are few dumb questions. DIY dish info is still rather hard to find. You can install a dish behind a window and I've read about special tiles which allow satellite frequencies through on analoguesat's pages about hiding dishes but I don't know anyone using them. A more usual solution is either a hidden location such as a flat roof with baffle boards (dishes only need to see the Clarke belt, which is an elevation of about 20 degrees in England) or to use a non-dish antenna like the digiglobe or the LX2000 pipe. I think it's a really good idea to put the dish somewhere that you can fix it easily. Even the best dish seems to age eventually. I suspect some "professional" installers put the dishes at tops of walls to encourage repair call-outs. Marcus asked:

"Is it possible to receive British satellite television in other countries? I am living in Finland and I was considering purchasing a sky system from the UK and installing it over here. Would it work? Also would I be able to connect a sky set-top box to a different dish?"

Yes, you can get British TV in other countries. Getting external TV like BBC World is easy, but BBC UK services are on Astra 2D, which is focused on the UK. Coverage maps are available on many sites and it looks like Finland isn't well-covered. You'd need a colossal dish. You can connect Sky boxes to other dishes. Sky dishes and Sky LNBs need to go together, sometimes called Skyware, but dishes and boxes are interchangeable. A Sky system will work in Finland, but will only receive services on the other beams like 2A and I doubt Sky will register you at a non-UK address (anyone know for sure?). Also, Sky boxes are horribly crippled for anything other than Sky services and hopefully BBC Freesat will make BBC UK services easier-to-use with standard boxes.

On Friday 2007-11-02, I watched Larry Lessig give a presentation at the University of Washington entitled Is Google (2008) Microsoft (1998)? Since it was Lessig, the talk was articulate and thought-provoking, and he used his slides very well (unlike many presenters who just read bullet points).

The argument he made is that Google is like Microsoft in some ways (and so are other companies like Facebook). In his analysis, the problem with Microsoft was that so many companies made themselves Microsoft-dependent, which both forced other companies to follow suit (due to network effects) and gave Microsoft leverage they could use against everyone in their ecosystem (e.g., Netscape). According to Lessig, Microsoft was not evil to use this leverage, but you have to recognize that Microsoft will do what is best for Microsoft, and this may not be what is best for you. He then suggested that we should be aware of the potential for the same thing to happen; in support of this point, he quoted excerpts from the terms of service from the APIs of Google and Facebook, which contain typical statements like we reserve the right to terminate your use of this service for any reason or for no reason, and asked: What if Microsoft had written this license agreement?

But it seemed to me that this wasn't his main point. He was using this observation as a springboard to talk about issues revolving around competition and public policy. Specifically, he feels that we should work towards a more just model of interaction between companies and users whose work makes the company more valuable (by contributing to a software ecosystem, by posting content on the Web site, etc). But the correct way to do this (he says) is not via competition, after-the-fact litigation, or voluntary codes of conduct: competition may not be around to enforce good behavior (as we've seen with Microsoft), litigation may be too late and ineffective (again, see Microsoft), and voluntary codes of conduct will disappear as soon as they get in the way of the bottom line. The correct approach, he argued, is across-the-board regulation so that all the companies have to play by the same rules.

From there he went on to talk about governmental corruption. He said (quoting Robert Reich's book Supercapitalism) that the reason we consistently look for solutions in the market and in voluntary compliance is that our governmental system is broken and does not effectively regulate corporations in the public interest. But Lessig is optimistic that we can change things (he joked that his publisher was unhappy with this point of view, because his brand has been built on pessimism). In his view, the politicians in Washington, by and large, want to be honest and do good, but they aren't able to within the current political system. For instance, on the few occassions that he managed to get access to lawmakers to discuss copyright issues, it was often the first time they had heard that there was more than one side to the argument. He thinks we need a national political movement that will shame politicians into being less corrupt.

This is all a simplification of what he said, of course; unfortunately, I wasn't able to find a recording of this talk on Lessig's Web site, or I'd point you at primary source material. I don't know if this is because he chose not to post it, or because it just hasn't been made available yet.

My view

In general, I thought that the speech was very interesting and well thought out. But as much as I'd like to buy into his optimism that we can fix our system, I don't think it's well-founded, for two reasons.

First, he argued that competition will resolve some of these problems (that was before and after he said we can't rely on competition, which I found confusing and may indicate that I misunderstood something). His evidence for this seemed to be that when Microsoft was dominating the world, users responded by reducing their dependence on Microsoft products and by switching to alternatives such as Linux. But, in fact, although more people may be using Linux than in the past, virtually everyone is still only using Windows, it's still almost impossible to find a programming job that isn't Windows-only, and Microsoft is still raking in money hand over fist; indeed, they're doing so well that they're planning to expand their workforce by a third. To take just one data point of many: I ride the bus to Seattle and back, and I regularly see other people using laptops as they commute. Virtually all the laptops run Windows. Occasionally I'll see a Mac, and every few months I'll see a guy running Linux on his laptop; I say "a guy" because it's the same guy every single time. So by my estimation, use of Linux on laptops in buses on the 545 route from Redmond to Seattle is way below 1%, with Macintoshes somewhere around 5%.

To broaden my point, I think Lessig has fallen into a trap that highly intelligent people, particularly those in academia, tend to fall into. In academia, you are surrounded by people who value learning and thinking deeply about the world. These are people who will give serious and unbiased consideration to questions like what is the social effect of my acceptance of the Facebook Terms of Service? and If I become dependent on this service, will that possibly affect me in five years if the company decides to act against my interests? The problem with this is that these people are not representative of the population at large. The population at large thinks why is this check-box getting in the way of me sending pictures to my friends? And I have no doubt that any kid who, e.g., refuses to sign up for Facebook as a protest against their TOS will be roundly mocked in their social circles for being a weird antisocial nonconformist. (of course, this may not apply to the small minority of people like myself whose social circles consist of weird antisocial nonconformists)

Second, Lessig's optimism about the political system seemed to be based on his observation that the people in government are, by and large, not venally corrupt: they want to do good, but the system is constructed in a way that makes this impossible. To me, that's a tremendously disheartening statement. If our problem were simply a few, or even many, corrupt politicians, this would be a solvable problem: even nowadays, Americans have some ability to choose who gets elected, and I believe that a sufficiently well-coordinated campaign could replace bad politicians with good ones.

But if the problem is that the system is corrupt even when the individuals are honest, it's a much deeper problem and frankly one that I don't know how to solve. While individuals have some direct control over who gets elected, we have no way to directly change the system that produces the corruption; the levers to do this are in the hands of the politicians elected through that broken system. It's unlikely that the politicians themselves will fix the problem, because they've benefited deeply from it: if the system is left as it is, well over 90% of Congress members will keep
their jobs in the next election cycle, and any effective change seems likely (by definition) to disrupt this cushy arrangement.

On the other hand, running new candidates for office will not fix the problem. After all, if the problem is that the system is corrupt even when the participants are honest, then putting more honest people into government will have no effect. In order to get elected, these new politicians will have to become just as corrupt as the old ones, because that's how the system is set up. So we might change the faces, but we'll have the same old problems.

I recall two specific concrete proposals that Lessig made to make the system more responsive. First, he suggested shaming politicians who engage in corrupt behavior -- presumably taking large donations from interested parties. I don't see how this will help. Americans generally have a very low regard for the political class as it is; will pointing out particular examples of politicians taking money really make a difference when it comes to votes? And if it doesn't, will it really produce any change in their behavior? Politicians are motivated by votes the same way corporations are motivated by money; unless they stand a chance of losing their job (which is virtually impossible in any event), there's no reason for them to change how they behave.

His second proposal was for a system of contributions where it's impossible for any individual to prove they had donated a particular amount of money to a particular candidate. Even assuming it could be implemented perfectly, I don't think this will really solve the problem, for two reasons. First, it's not generally a secret what moneyed interests want. Politicians who want to attract large donations can just take positions that they know will appeal to donors of that sort. The donors themselves can make this easier on the candidates by, e.g., posting public position statements on issues of the day. Secondly, and this is far more insidious, even if politicians honestly represent their positions, then because success in running for office is so tied to the amount of money the candidate can raise, only candidates who hold opinions favorable to large corporations and wealthy individuals will be able to get elected. In fact, for all I know this is what happens already!

But with all that said, it was a really interesting talk and Lessig tied together some things you might not think would go together. If a video or Flash presentation becomes available, I would recommend watching through it. And hopefully my pessimistic predictions will be wrong; it's certainly happened before.

• A regular expression IDE for Emacs
• 2007-11-04: Sinfest
  Sinfest does Superman.
• Tog on Keyboard Shortcuts vs. Using the Mouse
  Not sure I believe it.
• Power Cords: The Bane of the Industrial Designer's Existence

Hello World ok, second night of my birhday is over, went again to the pub with some of my very important people. The day today was quite different - not only joy, fun, and … Today also other facts were playing an important role - Remember people who can’t be here tonight (dead), lost love and new targets. Nevertheless, I’m quite happy and enjoy the last hours of this night. Regards, Mario (with different emotions)

I have been rather quiet during the last few months. Not really because I wasn t doing anything, but because most of what I have been doing was done by private e-mail. Apparently people are expecting the DPL to disappear around the middle of their term, because I did not get many complaints. But yes, I know that sucked. And well, I now have a reason to regain motivation. \o/ DSA++ \o/ I am very, very pleased to let you know that this morning Phil Hands (fil) added Peter Palfrader (weasel) to the adm group. Which means that the DSA team now has a new member! The first since, wow, quite some time. I am grateful to everyone who made this possible, for bearing with my insistance, for making concessions, for being patient, and I hope this is only the first step to bigger and better teams in Debian. Of course the path is still long but I must admit this is truly refreshing. I wish great success to Peter for the thankless job that awaits him. Nexenta After meeting Michael Man and several Sun people at Debconf and seeing Michael s great talk I got really convinced that the technology present in OpenSolaris could benefit Debian in many ways. And the shortest path to there seemed to be Nexenta. I am currently discussing the possibility of having a Nexenta machine accessible to Debian developers to port and test their packages. I also suggested the Nexenta people to submit patches directly to Debian and try to join friendly packaging teams. Please be nice to them! Conferences Last month I attended Encuentro Linux 2007 in Arica, Chile where I gave a talk about the history and organisation of Debian and how to help and become a member of the project. Due to various issues (such as my passport being stolen when I was 2000 km away from the conference) I did not have the time to give my other planned talk about attracting developers who do not speak English to Debian and FLOSS projects. Fortunately I was able to discuss it with members of the local communities. My goal is to understand why such a large Spanish-speaking area has so few Debian developers, and what we can do about that. One of my observations was that two very important documents (the Debian Policy and the Debian Developer s Reference) were not translated into Spanish, while for instance there is a French translation of the latter. Any volunteers?

It is nice to read the New York Times and get some good news. Nadia Abu El-Haj was granted tenure after a bitter public battle opposing her appointment. As someone who has taken graduate courses with her in Chicago and who was advised by her, I know Columbia and Barnard made the right choice.

Two weeks from now, the NW-European Programming Contest (NWERC) of the ACM ICPC will have started here in Utrecht. Things are going well - the problem set is ready - but, as always, a lot of small things are still to do: ordering t-shirts and balloons, finalizing food and buildings, etc. However, I think if we would halt all preparations now we would at least have a basic contest in two weeks. Registration is almost over, and the list of teams shows that we have participants from nine countries representing 26 institutions distributed over 51 teams.

Switching from aterm to xterm I found it annoying that xterm didn’t select a full url when i did a double-click somewhere on it, like aterm did. Now, doing a little search on it I found something way stronger. Take a url like http://foo.bar/baz for the following example. If you now put something like the following into your .Xdefaults (and load it with xrdb -load .Xdefaults), you get xterms selection in a more useful way. A double click on foo will select foo.bar, clicking three times it selects the full url (well, better speak is to say it selects everything separated by whitespace). Click four times and it selects everything up to the start of the prompt (default prompt that is). 5 clicks selects the whole line.

XTerm*on2Clicks:  regex [^/@ \n]+
XTerm*on3Clicks:  regex [^ \n]+
XTerm*on4Clicks:  regex [^#$]+
XTerm*on5Clicks:  line

So, I just came back home (quite drunk), had a very nice evening with my girlfriend and then went to the pub where I got quite a lot of free beer after 00:00, I’m finally 18 now! That does mean I’m no more underage! Regards, Mario

John Goerzen blogs about his experience of using vim for about 8 months. Having been an emacs user a few years back, I understand both editors on a basic level, and can relate to some of the things he says. I have to disagree with one particular bit, though:

It starts faster. I'm not sure if that really was true even when I switched, but it certainly isn't true on any of my machines today. Both Vim and Emacs have had major version upgrades (v7 and v22, respectively) since I started using Vim. People seem to say that Emacs 22 feels faster, though I don't know if that's true. The startup times of the two, if they're different, are imperceptible.

Sorry, but that just isn't true. Using my very non-scientific method of measuring startup times ("click on the icon, watch the clock, and start counting until the editor is ready to use"), I can tell that gvim on my hardware takes less than a second to start, whereas emacs requires approximately 5 seconds to do the same. For reference, this is on my laptop, a 1.3Ghz PowerPC G4. In any case, I'm not a fan of either editor. There are things that vi does well, and there are things that emacs does well. On emacs' end, I like M-x gdb, I slightly prefer its M-x compile over vim's :make, and I just love psgml. I've also occasionally used gnus for news reading (never for mail, though). On vim's end, I find regex search & replace easier (as you mention in your blog post), and I prefer its syntax highlighting (I've seen emacs get confused because the first line in the output window started in the middle of a string. Such things will never happen to vim: if it gets confused once on a given file, it will always be confused on the same file. While annoying, at least you know it's there). All in all, I don't think it matters much. I've seen people who use an editor as an operating system, both in the vim and in the emacs camp, by having a ~/.vimrc or ~/.emacs that's larger than my collection of pictures[1]. I'm not one of them; to me, an editor that doesn't properly work in its defaults shouldn't exist. [1] okay, that's an exaggeration. Still.

I have just read an interesting article titled Why Crunch Mode Doesn’t Work [1] which documents the research on efficiency vs amount of time spent working (and by inference amount of time spent on leisure activities and sleep). It shows that a 40 hour working week was chosen by people who run factories (such as Henry Ford) not due to being nice for the workers but due to the costs of inefficient work practices and errors that damage products and equipment. Now these results can only be an indication of what works best by today’s standards. The military research is good but only military organisations get to control workers to that degree (few organisations try to control how much sleep their workers get or are even legally permitted to do so), companies can only give their employees appropriate amounts of spare time to get enough sleep and hope for the best. Much of the research dates from 80+ years ago. I suspect that modern living conditions where every house has electric lights and entertainment devices such as a TV to encourage staying awake longer during the night will change things, as would ubiquitous personal transport by car. It could be that for modern factory workers the optimum amount of work is not 40 hours a week, it could be as little as 30 or as much as 50 (at a guess). Also the type of work being done certainly changes things. The article notes that mental tasks are affected more than physical tasks by lack of sleep (in terms of the consequences of being over-tired), but no mention is made about whether the optimum working hours change. If the optimum amount of work in a factory is 40 hours per week might the optimum for a highly intellectual task such as computer programming be less, perhaps 35 or 30? The next factor is the issue of team-work. In an assembly-line it’s impossible to have one person finish work early while the rest keep working, so the limit will be based on the worker who can handle the least hours. Determining which individuals will work more slowly when they work longer hours is possible (but it would be illegal to refuse to hire such people in many jurisdictions) and determining which individuals might be more likely to cause industrial accidents may be impossible. So it seems to me that the potential for each employee to work their optimal hours is much greater in the computer industry than in most sectors. I have heard a single anecdote of an employee who determined that their best efficiency came from 5 hours work a day and arranged with their manager to work 25 hours a week, apart from that I have not heard any reports of anyone trying to tailor the working hours to the worker. Some obvious differences in capacity for working long hours without losing productivity seem related to age and general health, obligations outside work (EG looking after children or sick relatives), and enjoyment of work (the greater the amount of work time that can be regarded as “fun” the less requirement there would be for recreation time outside work). It seems likely to me that parts of the computer industry that are closely related to free software development could have longer hours worked due to the overlap between recreation and paid work. If the amount of time spent working was to vary according to the capacity of each worker then the company structures for management and pay would need to change. Probably the first step towards this would be to try to pay employees according to the amount of work that they do, one problem with this is the fact that managers are traditionally considered to be superior to workers and therefore inherently worthy of more pay. As long as the pay of engineers is restricted to less than the pay of middle-managers the range between the lowest and highest salaries among programmers is going to be a factor of at most five or six, while the productivity difference between the least and most skilled programmers will be a factor of 20 for some boring work and more than 10,000 for more challenging work (assuming that the junior programmer can even understand the task). I don’t expect that a skillful programmer will get a salary of $10,000,000 any time soon (even though it would be a bargain compared to the number of junior programmers needed to do the same work), but a salary in excess of $250,000 would be reasonable. If pay was based on the quality and quantity of work done (which as the article mentions is difficult to assess) then workers would have an incentive to do what is necessary to improve their work - and with some guidance from HR could adjust their working hours accordingly. Another factor that needs to be considered is that ideally the number of working hours would vary according to the life situation of the worker. Having a child probably decreases the work capacity for the next 8 years or so. These are just some ideas, please read the article for the background research. I’m going to bed now. ;)

• [1] http://www.igda.org/articles/erobinson_crunch.php

Share This

I was quoted in a recent Needham Times article regarding my complaints with the T:

But Adam Rosi-Kessel, a Roslindale resident who takes the Needham Line to his job in the financial district, said 15- to 20-minute delays have kept him from getting to work before 9 a.m. since May. Now Rosi-Kessel, a lawyer who biked to work when he lived in Jamaica Plain, is thinking about trading in his Charlie Card again. It used to be, living here in Roslindale, the train was always the fastest way to get in, he said. Now, it s slowed down to the point where it s starting to get competitive with biking again.

I’m glad they got my biking quote. Here’s the part I don’t understand (emphasis added):

Massachusetts Bay Transportation Authority spokeswoman Lydia Rivera said Needham Line delays have been caused by construction work and speed restrictions on tracks that don t go anywhere near Needham. In order to maximize train use, the T often transfers trains between tracks, rather than leaving underused trains empty in the MBTA yard. The method helps the T get trains where they re needed most, but it also means a delay on one track can slow service on a seemingly unrelated track. It s a Catch-22, Rivera said.

I don’t think the T spokesperson knows what a “Catch-22″ is. I refer those interested to the Wikipedia article on the subject, but it is nicely summarized there as “heads I win, tails you lose.” The T’s problem is not a logical paradox, but some combination of inadequate resources and poor organization. Can someone please help them out? Also on Boston local topics: I’ve received several unattributed mailings lately related to the city council election that do not facially promote any particular candidate. Today’s mailing attacked Stephen Murphy for repeatedly trying (and failing) to win some other office or get some other job than City Councilor. The return address was 31 Milk Street, which is the address of many different businesses. A few days ago, I received another unattributed mailing bemoaning how long it has been since an at-large city councilor came from the Parkway Area (my neighborhood), but not mentioning any candidate in particular. My guess is these mailings are all meant to support John Connolly, a West Roxbury resident, attorney, and ostensibly good guy. I was feeling pretty happy about the possibility of Connolly replacing Murphy on the Council (I have to admit some unfair prejudice in that the only house in our neighborhood I’ve ever seen prominently posting a sign in support of Bush also features a Murphy billboard). But these questionable campaign tactics are giving me pause. Does anyone know anything more about this? In any case, here are my endorsements for next week’s election, notwithstanding the concern outlined above: (1) Felix Arroyo, (2) John Connolly, and (3) Sam Yoon. I have no pick for a fourth candidate. I’ll also throw in a vote for Matt Geary. I really wouldn’t want the socialists running the City, but a broader spectrum of political opinion within the council wouldn’t hurt. Incidentally, it is remarkably hard to find any information summarizing all of the candidates’ positions and records. It’s almost as if the election isn’t even happening. (A commenter properly points me to Brighton Centered as a good resource.) 11/3/07 Update: More discussion here. Technorati Tags: Boston, City Council, Elections, Felix Arroyo, John Connolly, Sam Yoon, Stephen Murphy, Politics, MBTA, Transportation, Roslindale, Needham, Matt Geary

Share This

Other Unix systems apparently calculate the load average differently to Linux. According to the Wikipedia page about Load(computing) [1] most Unix systems calculate it based on the average number of processes that are using a CPU or available for scheduling on a CPU while Linux also includes the count of processes that are blocked on disk IO (uninterruptible sleep). There are three load average numbers, the first is for the past minute, the second is for the past 5 minutes, and the third is for the past 15 minutes. In most cases you will only be interested in the first number. What is a good load average depends on the hardware. For a system with a single CPU core a load average of 1 or greater from CPU use will indicate that some processes may perform badly due to lack of CPU time - although a long-running background process with a high “nice” value can increase the load average without interfering with system performance in most cases. As a general rule if you want snappy performance then the load average component from CPU use should be less than the number of CPU cores (not hyper-threads). For example a system with two dual-core CPUs can be expected to perform really well with a load average of 3.5 from CPU use but might perform badly with a load average of 5. The component of the load average that is due to disk IO is much more difficult to interpret in a sensible manner. A common situation is to have the load average increased by a NFS server with a network problem. A user accesses a file on the NFS server and gets no response (thus giving a load average of 1), they then open another session and use “ls” to inspect the state of the file - ls is blocked and gives a system load average of 2. A single user may launch 5 or more processes before they realise that they are not going to succeed. If there are 20 active users on a multi-user system then a load average of 100 from a single NFS server that has a network problem is not uncommon. While this is happening the system will perform very well for all tasks that don’t involve the NFS server, the processes that are blocked on disk IO can be paged out so they don’t use any RAM or CPU time. For regular disk IO you can have load average incremented by 1 for each non-RAID disk without any significant performance problems. For example if you have two users who each have a separate disk for their home directory (not uncommon with certain systems where performance is required and cooperation between users is low) then each could have a single process performing disk IO at maximum speed with no performance problems for the entire system. A system which has four CPU cores and two hard drives used for separate tasks could have a load average slightly below 6 and the performance for all operations would be quite good if there were four processes performing CPU intensive tasks and two processes doing disk intensive tasks on different disks. The same system with six CPU intensive programs would under-perform (each process would on average get 2/3 of a CPU), and if it had six disk intensive tasks that all use the same disk then performance would be terrible (especially if one of the six was an interactive task). The fact that a single load average number can either mean that the system is busy but performing well, under a bit of load, or totally overloaded means that the load average number is of limited utility in diagnosing performance problems. It is useful as a quick measure, if your server usually has a load average of 0.5 and it suddenly gets a load average of 10 then you know that something is wrong. Then the typical procedure for diagnosing it starts with either running “ps aux grep D” (to get a list of D state processes - processes that are blocked on disk IO) or running top to see the percentages of CPU time idle and in IO-wait states. Cpu(s): 15.0%us, 35.1%sy,  0.0%ni, 49.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7331 rjc      25  0  2868  640  312 R  100  0.0  0:21.57 gzip Above is a section of the output of top showing a system running gzip -9 < /dev/urandom > /dev/null. Gzip is using one CPU core (100% CPU means 100% of one core - a multi-threaded program can use more than one core and therefore more than 100% CPU) and the overall system statistics indicate 49.9% idle (the other core is almost entirely idle). Cpu(s):  1.3%us,  3.2%sy,  0.0%ni, 50.7%id, 44.4%wa,  0.0%hi,  0.3%si,  0.0%st   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7425 rjc      17  0  4036  872  588 R    4  0.1  0:00.20 find Above is a section of the output of top showing the same system running find /. The system is registering 44% IO wait and 50.7% idle. The IO wait is the percentage of time that CPU core is waiting on IO, so 44% of the total system CPU time (or 88% of one CPU core) is idle while the system is waiting for disk IO to complete. A common mistake is to think that if the IO was faster then more CPU time would be used, in this case with the find program using 4% of one CPU core if all the IO was instantaneous (EG in cache) then the command would complete 25 times faster with 100% CPU use. But if the disk IO performance was doubled (a realistic possibility given that the system has a pair of cheap SATA disks in a RAID-1) then find would probably use 8% of CPU time. Really the only use for load average is for getting an instant feel for whether there are any performance problems related to CPU use or disk IO. If you know what the normal number is then a significant change will stand out. Dr. Neil Gunther has written some interesting documentation on the topic [2], which goes into more technical detail including kernel algorithms used for calculating the load average. My aim in this post is to educate Unix operators as to the basics of the load average. His book The Practical Performance Analyst gives some useful insights into the field. One thing I learned from his book is the basics of queueing theory. One important aspect of this is that as the rate at which work arrives approaches the rate at which work can be done the queue length starts to increase exponentially, and if work keeps arriving at the same rate when the queue is full and the system can’t perform the work fast enough the queue will grow without end. This means that as the load average approaches the theoretical maximum the probability of the system dramatically increasing it’s load average increases. A machine that’s bottlenecked on disk IO for a task where there is a huge number of independent clients (such as a large web server) may have it’s load average jump from 3 to 100 in a matter of one minute. Of course this won’t mean that you actually need to be able to serve 30 times the normal load, merely slightly more than the normal load to keep the queues short. I recommend reading the book, he explains it much better than I do.

• [1] http://en.wikipedia.org/wiki/Load_(computing)


• [2] http://www.teamquest.com/resources/gunther/display/5/index.htm

Share This

I'd been using Emacs for quite awhile, and about 8 months ago I decided I would try using Vim. I'd only used vi for system emergency work, but knew a number of people that swore by it for regular work. So I decided I would learn Vim and use it for my regular work. I figure that with things like this, I don't get a real feel for how well they work unless I use them for all my work. So I haven't really opened Emacs at all in the past 8 months.

Yesterday I finally decided that Vim was not living up to my expectations and I'm in the process of switching back to Emacs. I thought I ought to write down why I'm doing that, for my own future reference... and since nobody has ever written about Emacs vs. Vim, I might as well post it where everyone can see it.

So here we are.

Original Reasons for Using Vim

It would lead to more comfortable typing. Lots of Vim users mention that you don't have to hold down keys while hitting other keys as much in Vim as in Emacs, and that the movement keys are all on the home row. That's true, but I didn't find it to be that big of an improvement, since Esc is a farther reach than anything in Emacs, and let me tell you, you're hitting Esc all the time in Vim. I found that removing the armrests from my chair made my hands happier than Vim ever did, and swapping Ctrl and CapsLock in Emacs will probably help there too.

It starts faster. I'm not sure if that really was true even when I switched, but it certainly isn't true on any of my machines today. Both Vim and Emacs have had major version upgrades (v7 and v22, respectively) since I started using Vim. People seem to say that Emacs 22 feels faster, though I don't know if that's true. The startup times of the two, if they're different, are imperceptible.

Vim would use less RAM. Frankly, these days, both Emacs and Vim are way down on the list of things that use up RAM. Heck, kmail has 141MB resident, and each of its two IMAP processes is using more than 30MB. Emacs in X right after start has 16MB resident, 10MB of which is shared, and 25MB VSS. gvim right after start has 8MB resident, 5MB of which is shared, and a 43MB VSS. Emacs tends to use fewer processes for things that vim. So they're not all that different, and Emacs could come out smaller in certain situations. But the difference is irrelevant on today's machines, and modern Gnome and KDE apps are many times larger than both of them.

It will make me more comfortable in rescue environments where I have only traditional vi available. Actually, the vi on AIX is so different from modern Vim that this didn't really help.

It would make me more productive. There are some editing commands that did, but as you'll see below, it was more than balanced out by other problems.

Things I Liked about Vim

The commands dt, dT, df, dF. Wonderful little things those. Emacs now has M-z (Zap), which is similar to df but can actually go to other lines (a nice addition). And there are easy ways to bind keys to the others as well, though that doesn't make it a pervasive convention like it is in Vim.

Antialiased fonts. It's crazy that Emacs doesn't have this yet. But not a showstopper; I still like good ole 10x20 just fine.

Regexp search-and-replace. Emacs actually has this now, and maybe it had it back then too. M-C-%. Apparently in Emacs22 the replacement expression can also have lisp code in it, which sounds really slick but I can't see myself using it regularly.

Annoying Things in Vim

Syntax highlighting. The syntax highlighting for most languages in Vim felt like it was about as smart as it was in Emacs about 10 years ago. Strings like "Hello!\"" (in languages where \" inserts a literal ") often confused it. Sometimes quotes within comments confused it. Sometimes it would be confused permanently. Other times, just until I scrolled around in the file or reloaded it.

Indentation. This is much more annoying than the syntax highlighting, really. In many languages -- and especially the two modes I've used most recently, XML and Haskell -- it really, really stinks. The indentation there isn't aware of syntax, or not very much. Sometimes it is smart enough to know that if an XML line starts with </ that it moves left and if it starts with an opening tag, that the next line moves right. But it's not smart enough to do this reliably. Not only that, but indentation is not handled with consistent configuration between languages. And even though Vim ships with a ton of language modes, the central docs only cover indentation for C.

I've asked Vim experts about this, and have tried all sorts of various tweaks, have read through Vim indentation mode source files, etc. There is just no way to get it anywhere near the intelligence of Emacs for most languages, short of writing my own mode, it appears. This is even worse because when using the backspace key in insert mode, for awhile it deletes individual spaces, and then all of a sudden deletes a big chunk of whitespace back to the beginning of the line. (And no, the insertion of Tab characters is disabled.) Indentation is my #1 complaint about Vim, and something that shows no progress towards being fixed any time soon.

And forget about anything like Emacs M-x reindent-region. This is a syntax-aware indenter. You can write out an entire source file with no indentation whatsoever, and it will indent the entire thing according to the indentation rules you've defined and the syntax of the language you're using. The best I've seen in Vim are commands that add or remove space at the beginning of every line in a region.

In short, Emacs seems to "understand" the file format on a much deeper level than Vim, and can automate things to a much better extent because of it.

Too many things disrupt the paste buffer. I can use Y or y to yank some text in Vim, and it's really, really easy to overwrite that buffer with other things. Yes, I know that I can yank it into a named buffer, but that's inconvenient and I don't usually know in advance that I'll have that need. In Emacs, only C-k and other "large area" commands disrupt it.

Vim doesn't like you having lots of files open at once. It's surprisingly convoluted to do this. If you use the basic documented command to edit another file, :e, it closes the file you're working on. The normal way to open multiple files at once is to use split windows. Well, I don't like split windows all that well, and often just want to make a quick change in one file -- in full screen -- and then go back to another. Even though I use set hidden in my ~/.vimrc, it still is annoying and more convoluted than it should be.

Vim can't create new top-level X windows. In Emacs, I can press C-x 5 2, and poof, I have a second Emacs window in X, and it's tied to the same editing session and Emacs process. Not a new process, with a different set of files, its own buffers, etc. The same process, same set of files. Just like a split window, but with a new top-level X window instead. gvim simply has no way to do that. This is also a large annoyance.

gqap stinks. This has burned me more than once. I'll be editing an XML document, and insert some text in the middle of a paragraph. Now I have a really wide line. So I type gqap to reformat the paragraph. My cursor is near the bottom of the screen, so I don't really see much past the current line. I then save the document and exit. Later I discover that vim considered the entire rest of the document part of the single paragraph, and removed all the different indentation levels at </para> and the like, so it's completely messed up. Emacs is smart enough to know what is a paragraph in XML mode, and M-q does the right thing. Oh, and Emacs reindent-region can fix the Vim gqap-induced mess.

* They know what I wanted or want. Before marriage, it was spams regarding some pills and nowadays it is about lottery or prize of $50,000,00.00 (Note: they care about cents!). or some lonely son/daughter of former dictator of some unknown African country.

Amusing tidbits you pick up in the mailing industry...

No, you can't mail running chainsaws. At least I'm pretty sure about that; I don't know a specific regulation against them, but USPS takes a dim view of anything that puts its employees and machinery at risk.

On the other hand, you can mail live scorpions if you package them appropriately. Note that, however, that this is limited to the US; apparently doing so in Australia is not a good idea. The same page also gives the rules for mailing baby chicks (which must be not more than 24 hours old) as well as adult chickens (Express Mail only, please; and no refund if the chicken dies in transit unless the container was damaged).

Finally this testing series is stabilizing and there are no known regressions. The latest big regression with encoding PDU data has been just fixed. Full list of changes:

• Fixed wrong PDU encoding (bug #412).
• Provide API for cleaning SMS backup structure.
• Added testing of SMS PDU encoding.

You can download from usual place: http://cihar.com/gammu/ I hope this will be really the last one before 1.14.0.

I have nothing to add to this article on The Register. NP: Red Snapper: Bogeyman