I have been in the beautifully dramatic city of San Francisco for a few days now to catch up with old friends, meet new ones, and present at the AAA meetings, probably the single largest gathering of anthropologists on Planet Earth. I love seeing the friends, I love going to (some) of the panels, but I (totally) loathe trying to say something of substance in 15 minutes. A lot of my work in mired in esoterica (legally, technically, and culturally) and I am giving my short talk on protests against Scientology, the Lulz, etc. etc. and well, if you are in the know, I think I can pull it off in 15 minutes. If you are not in the know, it may strike as completely odd, esoteric, opaque but at least I can say I did it for the Lulz

I have been in the beautifully dramatic city of San Francisco for a few days now to catch up with old friends, meet new ones, and present at the AAA meetings, probably the single largest gathering of anthropologists on Planet Earth. I love seeing the friends, I love going to (some) of the panels, but I (totally) loathe trying to say something of substance in 15 minutes. A lot of my work in mired in esoterica (legally, technically, and culturally) and I am giving my short talk on protests against Scientology, the Lulz, etc. etc. and well, if you are in the know, I think I can pull it off in 15 minutes. If you are not in the know, it may strike as completely odd, esoteric, opaque but at least I can say I did it for the Lulz

A few years ago, not long after we got married, it was getting close to Christmas — the first Christmas we’d spent together. We set out one Saturday morning in Indianapolis to find a Christmas tree. We both have allergies, so a real tree was out of the question. An artificial tree it is. We went to the mall. We went to Sears, to Target, and to any number of other places that I’m sure I’m forgetting. We saw trees that spun, trees that had red needles, trees with fiber optics in them, trees with some incredibly fake-looking snow, pre-decorated “generic” trees, and all sorts of things that just screamed “tacky.” What we didn’t find were trees that looked like… trees. I think there were a few Terah would have been OK with, until I had to veto them on the grounds that they were too fake, too short, to skinny, or whatever. So I thought I’d check the Internet. I came upon Tree Classics, and we ordered a 7.5ft Evergreen Supreme. We (or, well, *I*) got some good-natured teasing for being crazy enough to order a tree over the Internet. This was before people were quite used to the idea that you can buy anything online — and, as far as I know, Tree Classics was the only company (except perhaps for Sears) selling trees online back then. But it arrived and we put it up and liked it. It looked like a tree, it was nice and full. And it shipped direct from the manufacturing plant in Illinois. The downside to that particular tree was that it took several hours to assemble each year. Over the years, whatever unlucky family members happened to be visiting our house over Thanksgiving would get a nice Thanksgiving dinner — and then a pile of tree branches to shape. We started getting more haphazard over the years, and this year, decided that our now 7-year-old tree had survived three moves and a toddler, and we could do with a new, easier-to-assemble model. So I hopped online again. We didn’t even bother with the big box stores this year. There are more places selling Christmas trees these days, but we wound up buying another tree from Tree Classics. They had basically the same types of trees as other places, but a wider selection and generally better prices. Though their website isn’t quite as fancy. We got a 7.5ft Kensington Spruce this year. It arrived Monday and we set to assembling it as soon as we could. Our Christmas tree has had this tradition of getting put up on about Dec. 23 and then staying up until about Valentine’s Day because we are great at procrastinating with the long job of assembling it (and the almost-as-long job of disassembling it.) We got a pre-lit tree (standard incandescent white light bulbs, of course) this year. It came in about 4, rather than several dozen, sections. The injection-molded branch ends are amazingly realistic. We got it assembled in record time. But it didn’t look very full. The injection-molded ends seemed all squished together, and despite following the instructions, didn’t really bend out like we thought they should. So Terah called Tree Classics today. The woman that answered gave her a tip on how to do it (”splay” them out, side to side, then shape them up and down after that). She’s got it started, and we’ll probably finish out the shaping tomorrow. Tree Classics had also exchanged a few emails with me prior to our purchase to help answer questions about whether their current trees are easier to assemble than their prior ones, and questions about the appearance of their different standard and LED pre-lit options. Overall, we’ve been happy with them and will order from them again in the future, I’m sure.

Yesterday I wrote that I'd ordered a new ASUS EEE PC, the 901 model, and today it arrived. The machine is gorgeous though I suspect in the long term I'll regret ordering the white model. This entry was written on the device, slowly due to my fat fingers, with its new Debian Lenny operating system. To save time I didn't even use the default system, I just immediately rebooted it into the installer via a 2Gb USB stick. Unfortunately I had to run through the installation twice, because I made some bad partitioning decisions and decided to fix them rather than live with them. Happily everything on the device appears to work perfectly, although there were a few hiccups along the way. The only niggle is that suspend to RAM seems a little flaky; 50% of the time I try to resume and just get a blank window not my X.org desktop. Happily suspend-to-disk works perfectly, and the bootup/restoration process looks very pretty with the splashy package installed. There are times when I really love using Debian, and this is definitely one of them. Together we've produced an operating system which just works on an amazing array of devices and systems and gets better and better as time goes on! For example in the past I'd always regarded Network Manager as "that thing we remove to stop breaking our system" - but now I see it working correctly with no effort on my part at all. Amazing! I guess I'll be returning my Nokia internet tablet in the near future. This device is bigger, but much more capable and versatile. ObFilm: St Trinians (the recent remake; not too bad. Bonus points for the Shampoo cover).

--3177-- VALGRIND INTERNAL ERROR: Valgrind received a signal 11 (SIGSEGV) - exiting
--3177-- si_code=80;  Faulting address: 0x0;  sp: 0x402A9FD40
valgrind: the 'impossible' happened:
   Killed by fatal signal
   ==3177==    at 0x3801FDEA: unlinkBlock (m_mallocfree.c:190)
   ==3177==    by 0x38020CAE: vgPlain_arena_malloc (m_mallocfree.c:1055)
   ==3177==    by 0x38035516: vgPlain_cli_malloc (replacemalloc_core.c:101)
   ==3177==    by 0x380022F5: vgMemCheck_malloc (mc_malloc_wrappers.c:182)
   ==3177==    by 0x38035BA7: do_client_request (scheduler.c:1158)
   ==3177==    by 0x380372B1: vgPlain_scheduler (scheduler.c:869)
   ==3177==    by 0x38051B59: run_a_thread_NORETURN (syswrap-linux.c:87)

This code has already killed one debugger (Intel DB) and sent strace into an infinite loop of segfaults. <Sigh />. Tags debugging, programming, impossible

I just read the news that PC Magazine is being canceled. It’s not exactly a shock, given the state of technical magazines right now. I haven’t read one of those in years, since they turned to be more of a consumer than a technical publication. But I hope I am not the only one out there that remembers PC Magazine from the mid to late 1980s. I had two favorite parts in each issue: the programming example, and the “Abort, Retry, Fail” page at the back of the magazine. The programming example was usually some sort of DOS (or, on occasion, OS/2) utility. It was usually written in assembly, and would be accompanied by a BASIC program you could type in to get the resulting binary, as assemblers weren’t readily available. The BASIC program was line after line of decimal numbers that would decode them and write out the resulting binary — sort of a primitive uuencode for paper. Trying to type those in gave me some serious eyestrain on more than one occasion. By now, I forget what most of those utilities did, but I remember one: BatchMan. It was a collection of tools for use in DOS batch files, and could do things like display output in color or even — yes — play monophonic music. It came with an example that displayed some lyrics about batch programming on-screen, set to what I later realized was the Batman theme. Geek nirvana, right? But Batchman was too big to publish the source code, or the BASIC decoder, in print. It might have been one of those things that eventually led me to a CompuServe account. PC Magazine had some deal with CompuServe that you could get their utilities for free, or reduced cost — I forget. CompuServe was probably where I sent my first email, from my account which was 71510,1421 — comma and all. In later years, you could pay a small fee to send email to the Internet, and I had the amazingly attractive email address of 71510.1421@cis.compuserve.com. Take that, gmail. PC Magazine eventually stopped running utilities that taught people about assembly or batch programming and shifted more to the genre of Windows screensavers. They stopped their articles about how hard disks work and what SCSI is all about, and instead have cover stories like “Vista made easy!” I am, sadly, not making this up. Gone are the days of investigating alternative operating systems like OS/2. It appears that “Abort, Retry, Fail” is gone, too. It was a one-page thing at the back of each magazine that featured braindead error messages and funny stories about people that did things like FAX an image of a floppy disk to a remote office — before such stories were cliche. Sort of like DailyWTF these days. The sad truth is that the people that would FAX an image of a floppy are probably the ones that are reading PC Magazine today. I still have a bunch of PC Magazine issues — the good ones — in my parents’ basement. I also still have my floppies with the utilities on them somewhere. One day, when I get some time — I’m estimating this will be about when Jacob goes to college — I’ll go back and take another look at them.

As reported elsewhere, a small victory against the spammers had some impact. Going from 1.5 to 0.5 emails received per minute on this little server. The impact on spam that made it through all the filters is not noticeable, but I don't do statistics on that number. (I assume McColo ranges were blocked by blacklists anyway, so the drop should be mostly for the unfiltered spamtrap accounts. Anybody has opinions about Knujon?)

Australia has slow Internet access when compared to other first-world countries. The costs of hosting servers are larger and the cost of residential access is greater with smaller limits. I read news reports with people in other countries complaining about having their home net connection restricted after they transfer 300G in one month, I have two net connections at the moment and the big (expensive) one allows me 25G of downloads per month. I use Internode, here are their current prices [1] (which are typical for Australia - they weren’t the cheapest last time I compared but they offer a good service and I am quite happy with them). Most people in Australia don’t want to pay $70 per month for net access, I believe that the plans which have limits of 10G of download or less are considerably more popular. Last time I investigated hosting servers in Australia I found that it would be totally impractical. The prices offered for limits such as 10G per month (for a server!) were comparable to prices offered by Linode [2] (and other ISPs in the US) for hundreds of gigs of transfer per month. I have recently configured a DomU at Linode for a client, Linode conveniently offers a choice of server rooms around the US so I chose a server room that was in the same region as my client’s other servers - giving 7 hops according to traceroute and a ping time as low as 2.5ms! Currently I am hosting www.coker.com.au and my blog in Germany thanks to the generosity of a German friend. An amount of bandwidth that would be rather expensive for hosting in Australia is by German standards unused capacity in a standard hosting plan. So I get to host my blog in Germany with higher speeds than my previous Australian hosting (which was bottlenecked due to overuse of it’s capacity) and no bandwidth quotas that I am likely to hit in the near future. This also allows me to do new and bigger things, for example one of my future plans is to assemble a collection of Xen images of SE Linux installations - that will be a set of archives that are about 100MB in size. Even when using bittorrent transferring 100MB files from a server in Australia becomes unusable. Most Australians who access my blog and have reasonably fast net connections (cable or ADSL2+) will notice a performance improvement. Australians who use modems might notice a performance drop due to longer latencies of connections to Germany (an increase of about 350ms in ping times). But if I could have had a fast cheap server in Australia then all Australians would have benefited. People who access my blog and my web site from Europe (and to a slightly lesser extent from the US) should notice a massive performance increase, particularly when I start hosting big files. It seems to me that the disadvantages of hosting in Australia due to bandwidth costs are hurting the country in many ways. For example I run servers in the US (both physical and Xen DomUs) for clients. My clients pay the US companies for managing the servers, these companies employ skilled staff in the US (who pay US income tax). It seems that the career opportunities for system administrators in the US and Europe are better than for Australia - which is why so many Australians choose to work in the US and Europe. Not only does this cost the country the tax money that they might pay if employed here, but it also costs the training of other people. It is impossible to estimate the cost of having some of the most skilled and dedicated people (the ones who desire the career opportunities that they can’t get at home) working in another country, contributing to users’ groups and professional societies, and sharing their skills with citizens of the country where they work. Companies based in Europe and the US have an advantage in that they can pay for hosting in their own currency and not be subject to currency variations. People who run Australian based companies that rent servers in the US get anxious whenever the US dollar goes up in value. To quickly investigate the hosting options chosen for various blogs I used the command “traceroute -T -p80” to do SYN traces to port 80 for some of the blogs syndicated on Planet Linux Australia [3]. Of the blogs I checked there were 13 hosted in Australia, 11 hosted independently in the US, and 5 hosted with major US based blog hosting services (Wordpress.com, Blogspot, and LiveJournal). While this is a small fraction of the blogs syndicated on that Planet, and blog hosting is also a small fraction of the overall Internet traffic, I think it does give an indication of what choices people are making in terms of hosting. Currently the Australian government is planning to censor the Internet with the aim of stopping child porn. Their general plan is to spend huge amounts of money filtering HTTP traffic in the hope that pedophiles don’t realise that they can use encrypted email, HTTPS, or even a VPN to transfer files without them getting blocked. If someone wanted to bring serious amounts of data to Australia, getting a tourist to bring back a few terabyte hard disks in their luggage would probably be the easiest and cheapest way to do it. Posting DVDs is also a viable option. Given that the Internet censorship plan is doomed to failure, it would be best if they could spend the money on something useful. Getting a better Internet infrastructure in the country would be one option to consider. The cost of Internet connection to other countries is determined by the cost of the international cables - which can not be upgraded quickly or cheaply. But even within Australia bandwidth is not as cheap as it could be. If the Telstra monopoly on the local loop was broken and the highest possible ADSL speeds were offered to everyone then it would be a good start towards improving Australia’s Internet access. Australia and NZ seem to have a unique position on the Internet in terms of being first-world countries that are a long way from the nearest net connections and which therefore have slow net access to the rest of the world. It seems that the development of Content Delivery Network [4] technology could potentially provide more benefits for Australia than for most countries. CDN enabling some common applications (such as Wordpress) would not require a huge investment but has the potential to decrease international data transfer while improving the performance for everyone. For example if I could have a Wordpress slave server in Australia which directed all writes to my server in Germany and have my DNS server return an IP address for the server which matches the region where the request came from then I could give better performance to the 7% of my blog readers who appear to reside in Australia while decreasing International data transfer by about 300MB per month.

• [1] http://www.internode.on.net/residential/internet/home_adsl/nakedextreme/pricing/


• [2] http://www.linode.com/


• [3] http://planet.linux.org.au/


• [4] http://en.wikipedia.org/wiki/Content_Delivery_Network

The mobile phone I travel with is pretty good and I use JabberMixClient to instant message from it. Version 2.1 finally got rid of the avatar-replacement bug, so I only wanted to solve my inability to make an SSL connection to our servers. (The linked email is slightly wrong: it’s signed by our own Certificate Authority, not self-signed.) The answer was to install TTLLP’s Certificate to the phone, as suggested on Sony Ericsson’s developer site. After a few unsuccessful atttempts, I corrected my two mistakes: firstly, I needed to install our CA certificate, not the server certificate; secondly, use PEM format, like on tjworld’s k800i. The exact command was obexftp -b $PHONE -p ca.pem and then the phone prompted me whether I wanted to install it. Now I can jabber from the phone without the phone company eavesdropping and without having to MidpSSH to a server. Can’t I?

I love collecting quotes but I file most of them away in some file on my computer or distribute them in class. I thought I might start sharing some of my favorite passages/quotes. This one, which I read 3 years ago, reminded me of the craft-like spirit in Debian and Free Software in general. I like how the author brings together the themes of labor, time, quality, mistakes, intimacy and tools all in one swoop.

I refuse to be in such a hurry that I squeeze the aesthetic value out of everything just to gain a few minutes of time time which will then just be filled with more rushing and more mass-produced, soulless junk. In the drive to achieve instant gratification, we have spent a century trying to shorten the learning curve and eliminate the chance of error in every human activity. There is much good in this, but something has been lost in the process. The Galoots are the guardians of that which was almost lost: the challenge of trying to master a skill that can never be fully mastered, the creative freedom that comes from intimacy with a medium as complex as wood, the sense of self-sufficiency that comes with knowing that you can make a useful object with tools so simple that you can make the tools too, and the peaceful mediation of trying to bring eye, hand, and wood together into harmony through finesse and understanding rather than brute force.

Quoted from Hohn (2005), The Romance with Rust, a Harper’s article on tool collectors. A Galoot is a person with a deep passion for old hand tools, their ways, workings, and their history.

I wrote last month about my failed first attempt to achieve a "level 3" high power rocketry certification. Last week, I learned that one of the commercial altimeters I flew for control of the ejection sequence has a firmware bug, which can cause premature ejection of the main parachute on flights above 10k feet! So it may in fact be the case that everything I did was perfect and I was simply the victim of a bug in software... how ironic! Yesterday, on my second certification attempt, I was successful! The rocket I flew this time was based on a Polecat Aerospace ten inch Goblin kit. I incorporated several modifications in my build, including additional fiberglass and carbon fiber reinforcement, and a payload bay in the nose cone. Once again, my wife Karen graced me with a parachute sewn from the Team Vatsaas design, this time a slightly larger one in burgundy and black rip-stop nylon. She incorporated several design tweaks based on her experiences sewing the first one which we'll try to write up at some point, that allowed her to sew this one in less time. The motor selected was again an Aerotech M1297W reload. The launch went perfectly, and the ascent was impressive. At apogee, the nose cone separated as planned and the drogue parachute deployed. Unfortunately, when the backup charge fired two seconds later, the main parachute also deployed. That wasn't intended... the main was supposed to deploy much lower, at 1300 feet. Luckily, the winds were light enough that the rocket touched down only about a mile downrange from the launch rail, well within the waiver distance, and was easily recovered without damage. There are three things I'll consider changing before flying this rocket again. First, the chute size calculator used to design the main parachute seems way off. The actual descent rate was around 32 feet per second, while our goal was 20. Other than that, the parachute performed admirably! Second, the premature deployment of the main at apogee could be cured by replacing the paper taped over that parachute bay with another piston capping the bay and held in with a pair of shear pins. Finally, the hard touch down snapped some of the nylon wire ties holding the batteries in place in the avionics bay. While no damage was done, and reducing the descent rate might prevent it happening again, improving the battery mounting would be an easy upgrade. For more information, see my project page for this rocket. James Russell took some pictures at the launch, including a great one of the rocket under boost just leaving the launch rail. And Jeff Lane captured video of the ascent and descent. It was a great flight and a great day, and represents a major milestone in our hobby rocket activities!

For various reasons I've recently been thinking about forums. Many technical users dislike forums, because they are things that are hard to follow. Even with RSS feeds & etc you need to keep a login and remember to return to see if your post(s) have been answered. However non-technical users love forums, and from a community-building perspective they're very cheap and easy. Particularly if you manage to appoint moderators from within the comunity. I currently find myself in a position where I'd kinda like to have a forum package. Something that I can integrate into existing site easily. Unfortunately most of the "best" forum packages are PHP-based, and have their own complex login, group, and admin facilities. That makes it hard to update them to authenticate against my existing MySQL table(s). (We'll leave my PHP-allergy in the background) So, once more, I've been contemplating the bad route; create my own forum software. I'm well aware that down that path lies badness madnesss. Let us recap. What is a forum?

• A forum is an online site.
• With a coarse list of topics.
• Inside each topic is a list of threads.
• Each thread is comprised of a number of (threaded) messages.

Sound familiar? It should if you use email:

• ~/Maildir contains storage for a collection of mailboxes.
• Each mailbox is a course list of topic-specific discussion.
• Each topic is comprised of a number of (threaded) messages.

So, the unthinkable, could we convert (bi/uni-directinally?) from a Maildir hierarchy to an online forum? Would that make sense? On the face of it. Yes. There are implementation details - the forum index would be essentially a list of Maildir folders (perhaps "~/Maildir/topic1/.title" would be require to give it a pretty name). Each thread topic would be a rendered display of the messages in the folder. So, what are the drawbacks? Well reading Maildir folders gives us threading, and subjects, bodies, etc. But it does mean a fair bit of overhead parsing messages. (Times like this I remember Hughe. Every time we've gotten together for beer & geekery the topic of an extensible perl-based IMAP server comes up. I'm sure it should be written ..) I'll wrap this up now. I'm sure I've made the point. There are some details which have impact - Should the forum accept new posts online? Or only via gated email-delivery? Will it work? Should it be Maildir, or IMAP based? Still at least filtering your SPAM would be easy ;) More questions. Some questions have no answers. Some answers we ignore because we don't like. I need to sleep. ObFilm: 007: Quantum of Solace Bad film. Don't waste your pennies.

Today, the NYT has an interesting piece on the declining numbers of women in the field of computer science. Ultimately the article presents a bleak picture but does not give a firm sense of why this is so (I think because it is so hard to answer). I do agree that if girls are not hopping on the computer at a young age and are not using it as a tool for production (they do use it a social tool), they are always going to have trouble catching up to men. Many CS majors, not to mention most hackers, develop quite intimate relations with the computer from a young age and thus have a level of comfort and expertise they have is nothing short of astonishing. If women are not developing that expertise as children but only much later in life, there will always be two classes of citizens in computing. Men, in other words, are native speakers, while women learn computing as a second language. The comment I agree with less is the following:

Ms. Cassell identifies another explanation for the drop in interest, which is linked to the pejorative figure of the nerd or geek. She said that this school of thought was: Girls and young women don t want to be that person.

It seems to be that in fact in the 80s and prior to that, the only word in town to describe computer folks were nerds. But geek arose to take its place and in part to take away the pejorative sting. Geek is cool. Nerd is not. And geek is now associated with all sorts of computer cultures in a way that it was not before. So it seems to be that more than ever, there is a positive geeky association with computing so in fact this would open the doors to more people than before.

A few years ago, I posted a story about my frustrations with Blue Cross Blue Shield. They were not coughing up the dough for a 4000 dollar bill and for the life of me, I could not get them to do it and I could not even get in touch with people in the organization to help me. After that post circulated, the Public Relations director wrote me and this helped me get the access I needed to eventually get all the money (it did take almost 3 years though and I should write about that but later). Though the ethical stakes and scenario are totally different, I have had equal problems contacting someone, a live body, a person who might reply to an email, a person who might return a phone call, from the American Anthropological Association. Next week is their Annual Meeting in San Francisco, CA and I am giving a presentation. I simply want to inquire whether there is A/V equipment in the room for if there is not, I need to do some serious shuffling of my talk because it is currently so dependent on the audio visual components. Given that I coughed up $ 400 bucks (and some change) to be a member and to attend the zoo that is this conference, given that it costs money to fly there, given that it is a member supported organization, one would think that one could just get a simple answer to a simple question. I realize they are not staffed to field endless inquiries but it is a conference they are putting on and thus, I do expect an answer or basic information about the A/V equipment (which we also requested when we signed up) especially after 1.5 months of trying. Anyhow, I am frustrated and I realize this is a really silly and small annoyance. But if no one ever calls them on it, there is no way for them to every change their archaic and pathetic ways.

One of the most interesting books I’ve read in the past years is without doubt G del, Escher, Bach by Douglas Hofstadter. There he introduces TNT and asks the reader to translate the statement “b is a power of 2″ into TNT. My proposed solution, which I wrote down two years ago, is:
∼∃a: <∼∃c: a=(c+c) ∧ ∀d: b=(SSa⋅d)> This is based on the following equation:
u ⋅ n u=2k+1 : k, n ∈ ℕ = ℕ∖ 2ⁿ n ∈ ℕ
Which means that every natural number except powers of 2 can be written as a product of an odd natural number u and a natural number n. This can be easily understood. Just take the prime factors of an arbitrary natural number. If one of these prime factors is odd, the number can be written as a product of this odd prime factor and the product of the remaining prime factors (which can be even or odd). If there is no odd prime factor, all prime factors must be even and since 2 is the only even prime number the original number must be a power of 2. I’m wondering if my solution is correct and how other people have written this statement in TNT.

My macbook reinstalled, and emacs. Lenny new install, with emacs22. muse-el not working for me today, dunno why. wl is telling me 'Cannot open load file wl'. I need to debug some more to find out why. I found that the new classic MacBook is very different from the MacBook I bought 2 years ago; it has a different sound engine, and different Infrared (does it have anymore?), and different keyboard.

One month with Emacs and counting - Part 2 A while ago I've blogged about my switch from Vim to Emacs, promising a blog post series, quite a mouthful nevertheless, it's time to continue the series. The first part was about why I think that nowadays Emacs is ready to be switching to. This second part is about flawed Vim design choices which substantially contributed to my choice. (Of course all this post is under a IMO-/rant-disclaimer, it can't be otherwise, so take it cum grano salis) Flawed Vim design choices Vim started as a wonderful editor, following the path of Vi, which in turn has got right a good deal of design choices. One is the often debated modality. In spite of being a HCI PITA in general, modality is not a problem at all for software you are using every single day (and the editor is the 2nd program I use the most, with the terminal being the 1st), because you learn it as a habit. In the case of Vim, modality is good because it let you use powerful yet simple motions in command mode, which can be easily combined with operator-pending commands. This can be considered yet another incarnation of the UNIX philosophy, casted inside the editor itself. ... which brings us to the good old joke which describes Emacs as a great operating system, lacking only a decent editor. In fact, Vim is no longer the nice Unix philosophy player which it used to be, and Emacs plays much more nicely with external tools than Vim. As arguments, let's consider a few (not so) recent Vim evolutions:

• Spell checking. Starting with version 7, Vim has got support for on-the-fly spell-checking of text buffers. It is a super-nice feature, which I've advertised in the past, because spell checking is needed in many tasks we daily do with editors (mail/doc writing, code commenting), and having a single interface to do it comes handy. Unfortunately, how that is implemented in Vim is definitely not along the lines of the UNIX philosophy. We have countless different FOOspell implementations whereas Vim implemented yet another one, without relying on any pre-existing implementation. Even more so, the file format of spell files is basically ad-hoc, and not sharable with other tools. This is already a PITA for packaging (let's add to Debian another 20 binary packages vim-spellfiles-FOO), but is even worst than that: the spell files must be generated in ad-hoc ways which are seriously prone to failures (guess why in Debian we don't have yet the vim-spellfiles-FOO packages which used to exist in experimental a long time ago?). What would have been the UNIX-friendly way of achieving the same result? Well, obviously spawning FOOspell and interact with its process from the editor. Keep that in mind.
• Internal grep. A feature of Vim I've always loved is :grep (yet another instance of quick-fixing, which is also used for jumping to compile-time errors). In its original incarnation it was as simple as invoking grep and parse its output. KISS. Then, Vim started to have a problem: grep -r can take ages and in the meantime the editor was stuck. Hence, starting with Vim 7, vimgrep has born: an internal re-implementation of grep. OK, it is more portable (but how many of us do care? on how many system you use you are missing grep? damn, I used to install it also on Windoze!), and the editor retain control. Control that is exploited just to show a nice progress bar of what is being scanned ..., while the editor is still stuck. Well, to me it looks like the problem would have been solved much more nicely by adding the ability to interact with an external process (deja vu?). That process can then be grep -r. Full stop.
• Lack of debugging support (on purpose). Long standing missing feature of Vim: you can not "easily" drive a debugger from Vim, because a firm choice in Vim is to ... (OK, I confess, I'm starting to beat a dead horse) ... not have support for interaction with external processes. I found such a choice quite dumb: to me it is evident that it induces a trade-off between being feature-complete wrt programmer needs (as available external tools get available to face those needs) and not "re-implementing the wheel" inside the editor. Going back to the lack of debugging support in Vim, that has spawned projects like GdbVim, Clewn, Bram's Agide, and my own tiny teeny incarnation specific to ocamldebug called WOWcamldebug. Every single project of that list acts as an intermediary between Vim and an external process (a debugger of some sort), whereas that could have been implemented by an editor plugin, provided that the editor offer, guess what, support for interaction with external processes.
• Top-level. Similar to the debugging support issue, there is the interpreter issue, namely how to evaluate "phrases" of your interpreted language of choice directly from the editor. Nowadays a lot of languages have the so called top-levels which evaluate language phrases interactively (Python, Ruby, OCaml, many Lisps, ...) and show evaluation results. That ability can speed up testing considerably, but can become cumbersome to use for large code snippets without editor support (good luck with copy and paste). Yes, Vim has support for that, but the implementation choice is close to be ridicolous: only if you have linked Vim together the interpreted for a given language, then you will be able to interactive evaluate phrases of that language. What if you program in Python, Ruby, and Perl? Well, you need to link all of them. What if you also program in OCaml? Then you're screwed, because Vim currently does not support linking with it. (Yes, I know that there are other reasons for doing that linking, they are discussed below.) Do I need to tell you which feature will be enough to address top-level support? interaction with extern.. OK, I'll drop it, you know that already.

Well, I acknowledge that Emacs got this choice right, and that it's an important one: Emacs supports interaction with external processes, and plugin authors have been happy to exploit it to create really cool stuff (sticking to the UNIX philosophy). Please welcome: Flyspell, M-x grep (shameless smartass plug: same number of keystroke of :grep), Grand Unified Debugger, a sampling of language-specific major modes with top-level support. (But I recommend having a look at least once at the coolness of other stuff built-on top of external process support, like Flymake, mentioned by dancer not a long ago.)

• Its own scripting language. I've bored you enough with the external process topic, hence here is the second one: the saga of Vim and its scripting language, i.e., the programming language using which you can customize the editor. Note that the choice of scripting language impacts on 2 different targets: final users in need of fire and forget automations (when they start to get to complex to be implemented on top of, say, macros), and developer of editor extensions such as addons. The impact of scripting language on extension developer is crucial for the evolution of an editor aiming at being powerful. If the scripting language and the API towards the editor is flexible enough, then a lot of cool extensions and features will be developed by third party authors, otherwise the burden stays on editor upstream author.
  1. In its phase 1 (my term, Vim < 7.0) Vim pretended that it needed only a minimal scripting language (Vim script) and that everything else can be developed using external programming languages, linked in the editor. As a consequence Vim script was just a tiny teeny procedural language with editor interfacing capabilities, most notably lacking any "decent" data structures (no lists, no dictionaries, ...). In that phase, if you were a user it was fair enough. You just had to chose your external programming language (at compile time ...), and you were able to write your quick hacks with it. But if you were willing to develop an extension you were screwed, because you had either to impose your external programming language to the final user (that is what gave born to absurdities like vim-full, look at the deps!) or to enjoy a good deal of masochism to explode your 30 lines of Python extension into 200 lines of Vim script to make your Vim extension more "portable".
  2. In its phase 2 (Vim >= 7.0) it has been realized that something were wrong in the initial choices about Vim script, and data structures (lists, dictionaries, function references!) have been added to it, relieving a bit the pain of programming portable extensions. Nevertheless, and probably due to how it started, Vim script is far from being a programming language one can enjoy programming in. Consider for example the standard library of the language. It started as a random collection of unrelated functions being added on a "as-needed" basis, ... and continued along that path! Now the API reference lists together totally unrelated functions, organized inconsistently, and making really painful finding what you need (assuming it exists).

Even in this respect, I acknowledge that Emacs got this right. It has chosen a single, now full-fledged, programming language (Elisp), which is offered both to the final user (fact: the average Emacs user knows much better how to program her own editor than the average Vim user) and to extensions developer. The standard library of the language is organized with a consistent naming (even though not always intuitive) and documented in an organized manner. You might not like the language, but at least is has been taken seriously and managed as such. Finally, being a Lisp dialect, you might need what you learn with Elisp elsewhere. Vim script is too committing: it is useful only inside Vim. I believe that contributed significantly in its scarce diffusion among Vim users. The fact that Elisp was born together with Emacs and that it is also used internally for the editor implementation is not relevant here: I do care about what is offered to me as an user and as an extension developer. What I see on the Emacs side in this respect is much better that what I see on the Vim side. Moreover, from the point of view of the users, is the difference between what is inside the editor and what is outside (i.e., the extensions) that relevant? I think it is not, and that brings me to the final Vim choice which I consider flawed and which has the potential of seriously limit its future evolutions:

• Editor as a distribution. Emacs is managed as a intermediate software distribution layer, similarly to what happens in other softwares where many third party authors cooperates (e.g., TeXLive). The distribution architecture has advantages and contributes to the longevity and potentialities of a software project. What does this mean concretely in the Vim vs Emacs dispute? For example it means that Emacs get more feature-complete each passing release, still preserving uniformity in documentation and keybindings. On the contrary Vim sports many addons, which are very likely to step on each other feet when enabled simultaneously. Actually, that is one of the reason which brought me to the creation of vim-addons: you cannot enable all the content of vim-scripts together due to various kind of conflicts. As we know well in Debian, distributions have a fundamental role in blending together lightly-coupled software components, that's role is missing in Vim evolutionary path and I find that worrisome.

Where to from here in this blog post series? For sure some tips on how to migrate for hard-core Vim users, then we'll see ... If you did enjoy the read please let me know commenting in the discussion page (as you did in the last post, thanks!).

I hadn’t noticed until today when Jordi Guti rrez Hermoso pointed it out to me: We already have AGPL’ed stuff in Debian, and not only in sid but it will also be shipped in Lenny. We have discussed that license in debian-legal during August and September this year, and in November its usage in client code has also being clarified when the FSF updated their FAQ to address that question, but we never reached a definitive conclussion. I also filed a bug against FTP Master asking them to clarify the situation of AGPL licensed stuff in Debian, which still remains unanswered. In any case, debian-legal is just a mailing list for discussion, and does not have the capacity to take any final decisions (that’s FTP Master’s task) but it would have been nice if someone had pointed it out there.

NYC Info Law:

Know everything (nearby). This is a compendium of news and New York City events of interest to IP, tech, privacy, communications, and legal-minded folk.

One of the speakers at OSCon this year — I forget which one — made a point that ran something like this, heavily paraphrased:

Education used to be an end in itself, not a means. It wasn’t about having a high-paying career. It was about knowing the world, about having knowledge and wisdom for its own sake. It was, quite bluntly, the accumulation of useless knowledge by the elite — those that could afford to spend time on such things, knowing that useless knowledge has a way of becoming useful in the most unexpected of ways. How fortunate we are to live in an age where the accumulation of useless knowledge is available to so many, and how sad it is that so few take advantage of it.

What a powerful statement, and it rings true to me. I remember in high school, when people from the local liberal arts college would come and talk. They’d talk about the value throughout a lifetime of knowledge in a broad range of disciplines: English, history, political science, religion, science, and the arts. They’d talk about how their graduates went on to lead distinguished lives, how this broad core of knowledge serves a person well through life. I guess I didn’t believe them, because due to their lack of a computer science major, I went elsewhere. That local school may not have been the best choice for me for other reasons, but as I look back on it, I think they had a much stronger message than I realized back then. Here I am, just two math classes, one computer science class, and one biology class away from a degree. Yet I have had not one class covering the history of east Asia, not one class on different world cultures or religions, and only a very basic understanding of one foreign language (German). This hits me in the face almost every day. Yesterday I was wondering about the history of slavery and racism in Europe. Today I’m curious about China’s history as an economic powerhouse. Last week I was curious about Roman law and daily life. The fact is, everything from philosophy to calculus is screamingly relevant to daily, modern life. We hear talk of “an American revolution” in Washington, of a shift of power in the Senate. It seems we forget that the notion of a Senate is considerably older than the United States is — and that we have such a thing because our founders were aware of this. Macroeconomic theory is thrust in our faces on an almost daily basis these days, yet I’ve never had a class on economics at all. We might feel fear of terrorist attacks, or see our fellow citizens lash out at “the Arabs.” Our own short memories fail to remind us of the light in which we are seen, fail to put the really quite minor terrorist threat in context of what London or Dresden endured in World War II. We demand our government to make us safer, and our government responds by making us less safe but making us *feel* safer at airports. In my own field, I see some universities buckling to pressure from Business to turn out large numbers of mediocre programmers that know the Java or .NET standard library well, but have no sense of the theory behind computer science, and would be utterly lost if asked to, say, write a recursive QuickSort. I find myself almost completely baffled that some companies that want to hire the world’s best programmers are only looking for people that are already fluent in $LANGUAGE — not ones that are good programmers, and so well-versed in computer science that they can easily pick up any language. I think there is a lot to the argument that a good, broad, classical education can serve a person well in any career. I wish I had realized that a little earlier.