Listmaster Thomas Viehmann wrote (IMHO) a very good summary for a policy on how to proceed with spam in the public archives of the Debian mailing lists. Some of them might need further discussions. Please help the listmasters and write your own ideas and opinions about his proposal. Also many thanks to Frans Pop, who first drafted some of them listmaster-internally quite a while ago. listmaster

Switching from aterm to xterm I found it annoying that xterm didn’t select a full url when i did a double-click somewhere on it, like aterm did. Now, doing a little search on it I found something way stronger. Take a url like http://foo.bar/baz for the following example. If you now put something like the following into your .Xdefaults (and load it with xrdb -load .Xdefaults), you get xterms selection in a more useful way. A double click on foo will select foo.bar, clicking three times it selects the full url (well, better speak is to say it selects everything separated by whitespace). Click four times and it selects everything up to the start of the prompt (default prompt that is). 5 clicks selects the whole line.

XTerm*on2Clicks:  regex [^/@ \n]+
XTerm*on3Clicks:  regex [^ \n]+
XTerm*on4Clicks:  regex [^#$]+
XTerm*on5Clicks:  line

Recently, GozRita unveiled the names of our two Falleretes majors for 2008's Falles festivities. All the free newspapers did some extensive coverage, with reports on who they are and what they do. Qu Valencia interviewed the little Fallera major, and then posted this:

Little Victoria learns Valencian So, Victoria Bl zquez speaks English and Valencian "nearly perfectly". Great! I think having newspapers treat Valencian as if it were just another foreign language that students are forced to learn is a great example of the dark future our language will face in just a few generations.

(wow, three blog posts — albeit short ones — in one day) Some people appear to believe that having a letter in the Granuiad is sad and I should be shamed or something. Honestly, I thought the email was going to be in some online forum thingy, not actually printed, but can I help it if even my brief flippant replies are insightful? Anyway, you can't shame me with newspaper clippings, I've had a like-cures-like homeopathic treatment against that.

I have just read an interesting article titled Why Crunch Mode Doesn’t Work [1] which documents the research on efficiency vs amount of time spent working (and by inference amount of time spent on leisure activities and sleep). It shows that a 40 hour working week was chosen by people who run factories (such as Henry Ford) not due to being nice for the workers but due to the costs of inefficient work practices and errors that damage products and equipment. Now these results can only be an indication of what works best by today’s standards. The military research is good but only military organisations get to control workers to that degree (few organisations try to control how much sleep their workers get or are even legally permitted to do so), companies can only give their employees appropriate amounts of spare time to get enough sleep and hope for the best. Much of the research dates from 80+ years ago. I suspect that modern living conditions where every house has electric lights and entertainment devices such as a TV to encourage staying awake longer during the night will change things, as would ubiquitous personal transport by car. It could be that for modern factory workers the optimum amount of work is not 40 hours a week, it could be as little as 30 or as much as 50 (at a guess). Also the type of work being done certainly changes things. The article notes that mental tasks are affected more than physical tasks by lack of sleep (in terms of the consequences of being over-tired), but no mention is made about whether the optimum working hours change. If the optimum amount of work in a factory is 40 hours per week might the optimum for a highly intellectual task such as computer programming be less, perhaps 35 or 30? The next factor is the issue of team-work. In an assembly-line it’s impossible to have one person finish work early while the rest keep working, so the limit will be based on the worker who can handle the least hours. Determining which individuals will work more slowly when they work longer hours is possible (but it would be illegal to refuse to hire such people in many jurisdictions) and determining which individuals might be more likely to cause industrial accidents may be impossible. So it seems to me that the potential for each employee to work their optimal hours is much greater in the computer industry than in most sectors. I have heard a single anecdote of an employee who determined that their best efficiency came from 5 hours work a day and arranged with their manager to work 25 hours a week, apart from that I have not heard any reports of anyone trying to tailor the working hours to the worker. Some obvious differences in capacity for working long hours without losing productivity seem related to age and general health, obligations outside work (EG looking after children or sick relatives), and enjoyment of work (the greater the amount of work time that can be regarded as “fun” the less requirement there would be for recreation time outside work). It seems likely to me that parts of the computer industry that are closely related to free software development could have longer hours worked due to the overlap between recreation and paid work. If the amount of time spent working was to vary according to the capacity of each worker then the company structures for management and pay would need to change. Probably the first step towards this would be to try to pay employees according to the amount of work that they do, one problem with this is the fact that managers are traditionally considered to be superior to workers and therefore inherently worthy of more pay. As long as the pay of engineers is restricted to less than the pay of middle-managers the range between the lowest and highest salaries among programmers is going to be a factor of at most five or six, while the productivity difference between the least and most skilled programmers will be a factor of 20 for some boring work and more than 10,000 for more challenging work (assuming that the junior programmer can even understand the task). I don’t expect that a skillful programmer will get a salary of $10,000,000 any time soon (even though it would be a bargain compared to the number of junior programmers needed to do the same work), but a salary in excess of $250,000 would be reasonable. If pay was based on the quality and quantity of work done (which as the article mentions is difficult to assess) then workers would have an incentive to do what is necessary to improve their work - and with some guidance from HR could adjust their working hours accordingly. Another factor that needs to be considered is that ideally the number of working hours would vary according to the life situation of the worker. Having a child probably decreases the work capacity for the next 8 years or so. These are just some ideas, please read the article for the background research. I’m going to bed now. ;)

• [1] http://www.igda.org/articles/erobinson_crunch.php

Share This

I'd been using Emacs for quite awhile, and about 8 months ago I decided I would try using Vim. I'd only used vi for system emergency work, but knew a number of people that swore by it for regular work. So I decided I would learn Vim and use it for my regular work. I figure that with things like this, I don't get a real feel for how well they work unless I use them for all my work. So I haven't really opened Emacs at all in the past 8 months.

Yesterday I finally decided that Vim was not living up to my expectations and I'm in the process of switching back to Emacs. I thought I ought to write down why I'm doing that, for my own future reference... and since nobody has ever written about Emacs vs. Vim, I might as well post it where everyone can see it.

So here we are.

Original Reasons for Using Vim

It would lead to more comfortable typing. Lots of Vim users mention that you don't have to hold down keys while hitting other keys as much in Vim as in Emacs, and that the movement keys are all on the home row. That's true, but I didn't find it to be that big of an improvement, since Esc is a farther reach than anything in Emacs, and let me tell you, you're hitting Esc all the time in Vim. I found that removing the armrests from my chair made my hands happier than Vim ever did, and swapping Ctrl and CapsLock in Emacs will probably help there too.

It starts faster. I'm not sure if that really was true even when I switched, but it certainly isn't true on any of my machines today. Both Vim and Emacs have had major version upgrades (v7 and v22, respectively) since I started using Vim. People seem to say that Emacs 22 feels faster, though I don't know if that's true. The startup times of the two, if they're different, are imperceptible.

Vim would use less RAM. Frankly, these days, both Emacs and Vim are way down on the list of things that use up RAM. Heck, kmail has 141MB resident, and each of its two IMAP processes is using more than 30MB. Emacs in X right after start has 16MB resident, 10MB of which is shared, and 25MB VSS. gvim right after start has 8MB resident, 5MB of which is shared, and a 43MB VSS. Emacs tends to use fewer processes for things that vim. So they're not all that different, and Emacs could come out smaller in certain situations. But the difference is irrelevant on today's machines, and modern Gnome and KDE apps are many times larger than both of them.

It will make me more comfortable in rescue environments where I have only traditional vi available. Actually, the vi on AIX is so different from modern Vim that this didn't really help.

It would make me more productive. There are some editing commands that did, but as you'll see below, it was more than balanced out by other problems.

Things I Liked about Vim

The commands dt, dT, df, dF. Wonderful little things those. Emacs now has M-z (Zap), which is similar to df but can actually go to other lines (a nice addition). And there are easy ways to bind keys to the others as well, though that doesn't make it a pervasive convention like it is in Vim.

Antialiased fonts. It's crazy that Emacs doesn't have this yet. But not a showstopper; I still like good ole 10x20 just fine.

Regexp search-and-replace. Emacs actually has this now, and maybe it had it back then too. M-C-%. Apparently in Emacs22 the replacement expression can also have lisp code in it, which sounds really slick but I can't see myself using it regularly.

Annoying Things in Vim

Syntax highlighting. The syntax highlighting for most languages in Vim felt like it was about as smart as it was in Emacs about 10 years ago. Strings like "Hello!\"" (in languages where \" inserts a literal ") often confused it. Sometimes quotes within comments confused it. Sometimes it would be confused permanently. Other times, just until I scrolled around in the file or reloaded it.

Indentation. This is much more annoying than the syntax highlighting, really. In many languages -- and especially the two modes I've used most recently, XML and Haskell -- it really, really stinks. The indentation there isn't aware of syntax, or not very much. Sometimes it is smart enough to know that if an XML line starts with </ that it moves left and if it starts with an opening tag, that the next line moves right. But it's not smart enough to do this reliably. Not only that, but indentation is not handled with consistent configuration between languages. And even though Vim ships with a ton of language modes, the central docs only cover indentation for C.

I've asked Vim experts about this, and have tried all sorts of various tweaks, have read through Vim indentation mode source files, etc. There is just no way to get it anywhere near the intelligence of Emacs for most languages, short of writing my own mode, it appears. This is even worse because when using the backspace key in insert mode, for awhile it deletes individual spaces, and then all of a sudden deletes a big chunk of whitespace back to the beginning of the line. (And no, the insertion of Tab characters is disabled.) Indentation is my #1 complaint about Vim, and something that shows no progress towards being fixed any time soon.

And forget about anything like Emacs M-x reindent-region. This is a syntax-aware indenter. You can write out an entire source file with no indentation whatsoever, and it will indent the entire thing according to the indentation rules you've defined and the syntax of the language you're using. The best I've seen in Vim are commands that add or remove space at the beginning of every line in a region.

In short, Emacs seems to "understand" the file format on a much deeper level than Vim, and can automate things to a much better extent because of it.

Too many things disrupt the paste buffer. I can use Y or y to yank some text in Vim, and it's really, really easy to overwrite that buffer with other things. Yes, I know that I can yank it into a named buffer, but that's inconvenient and I don't usually know in advance that I'll have that need. In Emacs, only C-k and other "large area" commands disrupt it.

Vim doesn't like you having lots of files open at once. It's surprisingly convoluted to do this. If you use the basic documented command to edit another file, :e, it closes the file you're working on. The normal way to open multiple files at once is to use split windows. Well, I don't like split windows all that well, and often just want to make a quick change in one file -- in full screen -- and then go back to another. Even though I use set hidden in my ~/.vimrc, it still is annoying and more convoluted than it should be.

Vim can't create new top-level X windows. In Emacs, I can press C-x 5 2, and poof, I have a second Emacs window in X, and it's tied to the same editing session and Emacs process. Not a new process, with a different set of files, its own buffers, etc. The same process, same set of files. Just like a split window, but with a new top-level X window instead. gvim simply has no way to do that. This is also a large annoyance.

gqap stinks. This has burned me more than once. I'll be editing an XML document, and insert some text in the middle of a paragraph. Now I have a really wide line. So I type gqap to reformat the paragraph. My cursor is near the bottom of the screen, so I don't really see much past the current line. I then save the document and exit. Later I discover that vim considered the entire rest of the document part of the single paragraph, and removed all the different indentation levels at </para> and the like, so it's completely messed up. Emacs is smart enough to know what is a paragraph in XML mode, and M-q does the right thing. Oh, and Emacs reindent-region can fix the Vim gqap-induced mess.

This weekend I finally finished the Linux Kernel Driver DataBase . Really I started this project in 2000, but in 2001 after some flames in the LKML (for CML2, the configuration engine used by my project) I put the project in a long hibernation. Now I had some time to finish it.

LKDDb is a database of hardware (and protocols), kernel configuration items and associated kernel driver files. Actually it has nearly 6000 entries (see the statistics). The database is generated automagically, quite fast: it take 2 minutes to scan the whole kernel source, and to interpret it. Unfortunately it is not complete and not accurate (I'm not sure if there is some black-list in kernel, but my scanner cannot distinguish the black-list with the supported hardware list).

Now I'm looking for some application of the database. In 2001 it seemed that there was some interest, but let see.

As a prototype, I dis-hibernate also the AutoKernConf, an automagical kernel configuration. The old version was a more powerful, so I need to forward-port some old features. BTW I changed an other time the project name: autoconfig was too similar to existing projects, kautoconfig, kernfig, kernautoconf were other names, but I like better the new name, and I can use the automagical word in the description ;-)

Do you see other uses of the database?

* They know what I wanted or want. Before marriage, it was spams regarding some pills and nowadays it is about lottery or prize of $50,000,00.00 (Note: they care about cents!). or some lonely son/daughter of former dictator of some unknown African country.

I hate spam! Which probably puts me in the same camp as 99.99999% of the world. The other 1 in 10 million are, of course, the spammers, who seem to take the space invaders approach to sending e-mail: we'll keep sending you more until you die. A few years ago I used to only receive perhaps 1 every 100 seconds, which was pretty annoying, but Spamassassin was quite able to filter out 99% of those and let through about 1-2 each day, which I could deal with. My spam levels increased to maybe 1 every 20 seconds, and late in 2005 I implemented a second layer of spam filtering on my laptop using DSpam. This worked quite effectively, but DSpam is really not the tool for the job - it's much more appropriate as a company-wide antispam solution, and potentially as a replacement for Spamassassin. It drove me nuts on my laptop because it's resource usage slowed down the interactive response. When I got my new laptop at the beginning of the year I decided against continuing with my rather baroque mail setup and to leave the spam filtering on the server. What I didn't realise is that my spam rate had increased again to around 1 every 8 seconds, and it has been slowly driving me to distraction ever since. It seems to have cranked up another notch recently, to perhaps 1 every 3 seconds now, so that 1% making its way through Spamassassin was getting to a very annoying several hundred each day. The longer I took to resolve it, the more time I would be wasting dealing with it every day. What I chose to apply on this occasion was CRM114, which I had some vague idea might be able to help. I was fairly impressed by the relatively simple install, but what completely blew me away was the speed with which it was able to learn to be useful. Starting from scratch, it seems to be correctly classifying over 90% of my incoming mail after about 12 hours of training, on a total of only 75 'Unsure' messages. Even after only an hour it was getting over 50% (I'll describe my actual CRM114 installation process in a comment below). So far there have been no false positives. Now that CRM114 is installed I will be able to look into some of it's other mail classifying features too, and I'm really looking forward to that too.

Two years after writing about what I found out when I was looking why my HDDs kept breaking down, I started to receive emails about this post. It seems that it was mentioned in an Ubuntu bug report and this morning I noticed that exactly this bug report was mentioned on Slashdot, so I can consider my post to be slashdotted (indirectly). For the past week, I’m counting 15,000 hits and it is still rising fast. It’s a good thing that I use a static HTML journal, so that my journal-hosting-pentium-233MHz can handle this. It’s also a good thing that the issue finally gets some attention.

Perhaps my favorite article in Wikipedia, the List of homophonous phrases, was deleted from Wikipedia in mid-August. Those arguing for deletion claimed that it was original research and were, I suppose, correct in that designation. But that doesn't make me happy to see it go. I asked an admin to move the list into a temporary home in my userspace until I can find a better home for it. Please help me find one and I'll redirect. It's the latest in several unfortunate deletions I witnessed recently. I created a humorous "undeletionist" barnstar to give to an admin for undeleting some humorous project pages in Wikipedia (which have since been redeleted, but copied first). My barnstar was also deleted. In retaliation for this all, and in good fun, I proposed the Association of Deletionist Wikipedians for deletion. The delete tag stuck around for a week until it was deleted under suspicion that it might be a joke. True enough. On the other hand, I have similar suspicions about the subject of the article.

Some people around Free Software tend to be too enthusiastic and very critical, both with themselves and with other people. Whenever Linux or Free Software appear in the media, people who wrote the articles are severely and merciless attacked, and the criticism is not always constructive. It might be hard to properly talk about these topics in a non-technical magazine or newspaper targeted towards the mainstream population, not only because they have to explain it in an understandable way for non-initiated people, but also because they’re usually very limited in space. Thus, many times the articles are inaccurate or incomplete, not always being a journalist’s fault, but also of the conditions in which the article has to be published, as well as the time they have to develop it. Someone, a journalist, told me this last weekend that she tries to avoid writing stuff about Linux herself due to the harsh attacks she’s subjected to from the Linux and Free Software activists as soon as she publishes anything on that topic. I kind of understand that. It might be the same than when you write a program and release it as Free Software, and then some people get too demanding and aggressive towards you for not having done a perfect program and submitting to their wishes, instead of thinking about the effort invested in what is done. I seriously think that we might be losing a lot of power and strength if we attack so much the media that support us than they fear us enough to avoid these topics. We’re on the same team, and it wouldn’t be so bad to be a little understanding and supporting of them, and rely on constructive and friendly criticism instead of treating them as if they were the enemy. They also have feelings, you know.

Sometimes, I think people go out of their way to create key mappings that are silly, stupid, and whatever.

• Why is irssi's "Ctrl-K" a key combination to delete a whole line, when "Ctrl-J", with J being right next to K, is a key combination to confirm the same line?
• Why is mutt's "x" a command to leave a folder without saving changes, while "c" is a command to leave a folder with saving changes?
• Why did the mac designers position the "eject" key right above backspace—the second most often used key on a keyboard?

Please, pretty please, with sugar on top: make sure accidentally mishitting a key in any application doesn't perform the opposite of what one would assume it would do. Thanks.

I finally managed to resend the signatures to the few people I decided to send them a while back after debconf7.

I actually resent all the signatures I thought I should sign (if I didn't socialize at all with you during debconf or before you shouldn't receive a signature from me).

So, please:

• sorry, if you get my signatures again; if so, ignore
• don't be mad if you didn't receive a signed key from me, I probably don't consider I know you enough to do that yet ;-)
  

Now I can cross one more item on my long todo list. Yay!

This message has emerged thanks to: caff, dato, python's smtplib and rfc822, vi, gpg, exim, linksys, dell, todo(the application from openhand) and blogger :-)

apt-xapian-index: smart way of querying tags I've recently posted:

• an introduction of apt-xapian-index;
• an example of how to query it;
• a way to add simple result filters to the query;
• a way to suggest keywords and tags to use to improve the query.
• a way to search for similar packages.
• a way to implement an adaptive cutoff on result quality.

Note that I've rewritten all the old posts to only show the main code snippets: if you were put off by the large lumps of code, you may want to give it another go. Today I'll show how to implement a really good way of searching for Debtags tags. When I say really good, I mean the sort of good that after you run it you wonder how could it possibly manage to do it. The idea is simple: you run a package search, but instead of showing the resulting packages, you ask Xapian to suggest tags like we saw in axi-query-expand.py. For extra points, I'll use an adaptive cutoff in chosing the packages that go in the rset. So, let's ask the user to enter some keywords to look for tags, and use them to run a normal package query: Now, instead of showing the results of the query, we ask Xapian what are the tags in the index that are most relevant to this search. First, we pick some representative packages for the expand: Then we define the filter that only keeps tags: Then we print the tags: That's it. We turned a package search into a tag search, and this allows us to search for tags using keywords that are not present in the tag descriptions at all:

$ ./axi-query-tags.py explore the dungeons
27.50 game::rpg:rogue
26.14 use::gameplaying
17.53 game::rpg
10.27 uitoolkit::ncurses
...
$ ./axi-query-tags.py total world domination
7.55 use::gameplaying
5.68 x11::application
5.35 interface::x11
5.05 game::strategy
...

You can use the wsvn interface to get to the full source code and the module it uses. You can see a similar technique working in the Debtags tag editor: enter a package, then choose "Available tags: search".

I have just read Career Development for Geeks [1] by Erik de Castro Lopo [2]. It makes some interesting points about a traditional approach to an IT career. The path I followed for most of my career (after I had a few years experience) was to work as a contractor and happily leave jobs without having anything else lined up. Erik suggests getting Engineers rather than managers to give references, it’s an interesting idea. Engineers can give better references about quality of work, while managers can speak on behalf of their employer (in theory). In practice a sane manager won’t give a bad review for legal reasons so the value of a reference from a manager is probably limited. Of course one problem with reviews is that I have never heard of a recruiting agent or employer actually verifying the ID of a reference. I could be listed on a friend’s CV as a senior manager in a multi-national company (which doesn’t have a major Australian presence) and give a good review and it seems unlikely that the typical recruiter would work it out. For someone with plenty of spare time and no significant assets (no risk of being sued) it could be entertaining to apply for a bunch of positions that they are not qualified for with friends using pre-paid mobile phones to give references. This could be done as a documentary on corporate hiring practices, or to simply try and get the highest paid job possible. Based on observing some former colleagues it seems that little skill is required to get a job and that when people do really badly they get paid for at least a few months. I am constantly amazed when reading reports about so-called “con artists” who commit crimes for what are often small amounts of money. Getting an income significantly greater than average without knowing anything about how to do the work is very common and is never treated as fraud (the classic example was a former colleague who wanted to write his own encryption algorithm but who didn’t even know about binary and therefore couldn’t use operations such as XOR). Erik’s main suggestion for dealing with recruiting agents is to talk about projet management issues (recruiters don’t understand technology). My way of dealing with them has been to assure them that I know it all and tell them to forward my CV to the client. Another suggestion is to learn new skills and diversify your skills. I don’t supoprt this because I believe that the majority of people who read my blog are significantly more skillful than the typical programmer. If an area of technology starts to go out of fashion then it’s the people with the least skills who suffer the most. If you are good at your work and enjoy it then it shouldn’t matter much if people around you are being laid off. Of course to rely on this you have to be working in a reasonably large field. For example if you develop software in a language which has hundreds of programmers then you may be at risk, but if there are tens of thousands of programmers using the language then you only need to be in the most skillful 10% to be assured of employment for the next decade or two. That said there are benefits to moving around, meeting new people, and working on new contracts. One thing you don’t want is to have 10 years of experience which are the same year repeated 10 times! Update: Here is a paper Erik submitted to OSDC on the same topic [3]. Mostly the advice is the same but with more detail and in a better format for reading.

• [1] http://www.mega-nerd.com/Career-dev/career-dev.pdf


• [2] http://www.mega-nerd.com/erikd/Blog


• [3] http://www.mega-nerd.com/erikd/cddg.pdf

Share This

• Single Threaded Memory Model
• L Root changing IP address on 1st November
• Comic for 26 Oct 2007
  Dilbert garbage man ftw.
• CURVING SPACE
  Love this.
• the discovery of france
  My great-grandparents often spoke in Norman, not French. The language degenerated into a little used patois after WW2.
• Restaurants I will not be eating at in 2008...
  Crazy.
• A Sense of Scale
  "Or to put it another way, turning Central Park into a solar electric generating station, like Kramer Junction (assuming, magically, the same solar conditions as the Mojave Desert), would power approximately 8-10% of Manhattan's households."

apt-xapian-index: adaptive quality cutoff I've recently posted:

• an introduction of apt-xapian-index;
• an example of how to query it;
• a way to add simple result filters to the query;
• a way to suggest keywords and tags to use to improve the query.
• a way to search for similar packages.

Note that I've rewritten all the old posts to only show the main code snippets: if you were put off by the large lumps of code, you may want to give it another go. Today I'll show how to implement an adaptive cutoff to get rid of the worse results. Recall that Xapian shows results by decreasing order of quality, and as we pull more an more results out, we reach a point where the matches are so approximated that they look random. This can be a problem if we want to change the order of the result, for example we may want to sort by package size, or by popcon popularity. There are many scenarios in which a really bad match could end up at the top of the results. For most cases, you just want to say "discard all results whose quality is less than 70%". But sometimes you have queries that OR lots of terms, and even your top result, while still being a very good result, may be below the cutoff you decided. Implementing an adaptive cutoff is extremely simple: first, you get the quality estimate of the top result: Then you tell Xapian that you want a cutoff value that is, for example, 70% of that: Finally, you repeat the query. If you want, you can go for bigger result sets, as the cutoff will make it so that if you have lots of results, they will very likely be all good results: This is it. You can use the wsvn interface to get to the full source code and the module it uses. Next in the series: smart way of querying tags.