I am very saddened by the state of the IT industry. It seems like nearly everyone who works in or around IT is, to put it bluntly, a cowboy. There appears to be very little desire for actually helping people; instead all I see is a very large number of people circling around looking for the next person they can make a quick buck off. In short, most of the IT industry is composed of people who are little better than shysters and conmen. As a case in point, I recently had contact with someone who is trying to get an online store running. They've been trying, unsuccessfully so far, for several months. They've been screwed over by basically every IT-related person they've had contact with so far. Now, after all of their suffering, they've finally gotten in contact with me. Most of their money is gone, and they're deeply behind schedule. So I'm left trying to pull a miracle job in little time and with little money. Yes, I could shoo them away, because they're asking for more than they can afford, but there's a little spark of helpfulness left in me, and I'll try to get them going as best they can with what they've got. If they'd found someone competent right at the beginning, they'd have been fine. They started with sufficient cash and time to produce a good end result. But chancing upon someone decent doesn't have good odds in IT; it's a bit like winning the lottery. This story isn't unique in my personal experience, by any stretch. There seems to be a near-endless procession of non-technical people who come out of their encounters with the IT industry battered and bruised, whether it be on the web, in their business infrastructure, or their personal computing needs. While it's easy to blame the customer for not managing themselves and their computing properly, I don't think it's fair to lay the blame on them -- to me, that's like blaming the patient for dying on the operating table, when the responsibility must lie, prima facie, with the surgeon. It's the surgeon, after all, who is the (presumably) trained and experienced professional in the relationship, and it is his responsibility, primarily, to ensure that everything goes smoothly[1]. Depressingly, I don't see much of a way around this problem. The few IT people who did care could try and group together, build some sort of a rating or "membership implies quality" system, educate the consumer, and try and drive the cowboys out, but guilds and trade associations are frowned upon these days, smacking (as they do) of restraint of trade and collusion. Government regulation never seems to produce anywhere near the desired result, while hoping that the consumer will either rise up and demand quality, or will gain sufficient knowledge to be able to pick a cowboy is a complete non-starter, for the same reason as teaching everyone basic surgical techniques isn't going to improve the quality of hospitals.

---

There's an ad that used to be on Australian TV for a certain bank, where a guy is at a party and he's asked by another of the partygoers what he does for a living. As he replies, "I work in banking", the entire party goes silent and stares at him in horror. It's only when he says, "It's OK, I'm with $BANK_BEING_ADVERTISED" that everyone looks happy again and continues on with their party. Although I've never had that particular experience, I'm almost getting to the point where I'm not happy telling people what I do for a living. This is because in any non-trivial collection of people, there's almost certainly going to be at least one person who has had such a poor experience with the IT industry that they're either going to take it out on me, ask me interminable questions about computers, or will just hate me on principle. It's unfortunate that brothels don't have pianos any more; if I were to say "I play piano in a whorehouse" I'm pretty sure the response would be "lawyer or sysadmin?". As Edsger Dijkstra said back in 2001, "The average customer of the computing industry has been served so poorly that he expects his system to crash all the time, and we witness a massive worldwide distribution of bug-ridden software for which we should be deeply ashamed." Are you?

---

1. That isn't to say that the patient (or client) has no responsibility in ensuring a positive outcome -- if you go bungee jumping the day after open heart surgery, or never send your web developer any content, you're very unlikely to have a positive outcome. However, it seems to me as though the only customer consultation done by most IT "professionals" on taking on a project is a Homeresque "I only have two questions. 'How much?' and 'give it to me'".

I spent two hours answering the phone, ribbing the clients affectionately before sending their calls off to different areas in the firm. It was like I couldn't get enough. I had to stop when Carol came back to her desk and demanded to know what the hell I was doing. I rejected the first fourteen answers that came to mind, and instead gave her a shrug, a wink, and a fingersnap. Then like a Carmelob, I scuttled away and strolled back across town to my office. My dyspeptic boss tapped his foot in a hostile manner. He demanded to know where I had been. I rejected the first three answers that came to mind, and simply responded that I had been answering phones. Also I quit. It seemed the thing to do. Explaining to my colleagues that I had just freed up a bit of time, I returned to Carol's office, where I found her rocking back and forth in the corner. I turned off her light. It seemed the thing to do.

So it’s a few days after Christmas, and like most of us tech-heads, I’ve got a few more gadgets to play with. First up: the Creative Zen 4GB. This one was a little bit of a saga. Last year, we got the kids no-name MP3 players, on the theory that we didn’t want to spend megabucks on something they wouldn’t use. They made valiant attempts to use them, but the little machines just weren’t up to the job. So, it seemed prudent to buy them iPods this year. Well, except for Apple’s attempts to break all non-iTunes iPod software, which had the side effect of making the devices unusable under Linux. Still, this was what they wanted, and they had been good this year, and very patient with my ever-more-convoluted schemes to get the old players working. So, iPod Nano 3Gs for both of them. My heart sank as I watched some of my hard-earned money go to reward such behavior. As part of the deal, I vowed to find a non-Apple player that would be good for when the iPods gave up the ghost or became “uncool”. And my dear wife, upon hearing this, went online, did some research, and bought me the aforementioned Creative Zen 4GB. From a Linux perspective, it’s in the “not quite ready for prime time” mode. Rhythmbox and Banshee are working on support; I tried a prerelease of Rhythmbox, and found its support to be very unstable. The only usable app is Gnomad2, which has a terrible UI and also occasionally crashes, but can manage to upload audio, video, and photos without too much hassle. Still, this is a problem of fine-tuning, and not of a hostile hardware vendor; I’m confident that these devices will be well-supported in the near future. The Zen is picky about what video files it will play, but I managed to figure it out: DivX or XviD video, 320×200 or smaller image size, encoded at a 480 kbit/sec video bitrate or less. Other video files might work, too, but you’ll have to find them on your own. My Zen has a little problem with the button locking feature: after unlocking, the screen comes up to all-white, and you have to power-cycle it to get the display back. I’m assuming this is a firmware bug, as the screen is still visible for a short time after engaging the lock. Other than this, the Zen is a delight, and every bit as functional as the iPod. The other nice gadget: a 24-inch LCD from Envision, bought after Christmas with a combination of gift cards, exchanges, and some of my own money. It was an open-box, and I saved about $80 for that; the only problem turns out to be a single dead pixel in the corner of the screen which is barely visible. It does 1920×1200 in very nice, bright color. Here, too, an improvement on my life only came after some effort. Debian 4.0’s drivers for the Intel graphics chipset are not capable of driving a widescreen LCD; the best I could get was 1600×1200, a normal-width resolution stretched across the wide display. I booted an Ubuntu Gutsy live CD to verify that the problem wasn’t with the monitor, and then set to the task of backporting everything I needed from lenny. Happily, before I started, I found that someone (Holger Levsen, to be exact) had done the work for me. Things are now about 90% there. The new drivers still don’t have everything figured out for running both Compiz desktop effects and XVideo acceleration at the same time, so I’ve had to turn XVideo off. My computer can render video without hardware support, but the quality isn’t as high. But, I have my nice wide screen, with crisp fonts and lots of room. I figure I’ll live with what I have until lenny releases, and then see what progress has been made.

Meet the EeePC Being in Taiwan, we swiftly got hold of an Eeepc. Instead of installing Debian into it, we decided to keep the original system and see how it works. It's a Debian derivative, and the feeling inside a terminal window is quite familiar. The boot is very fast. Two seconds after the video bios quickly shows on the screen, the X cursor appears. It's definitely worth having a look at how this devil boots. The "Asus Launcher" is worth a look. IMHO it's nicer and more useful than the usual launcher menu that we get in Gnome or KDE 3, although it probably only makes sense on a small display. It replaces the desktop background, has tabs, no clutter and allows to launch applications. Turns out it's customisable as well. What's on the system KDE 3.4.2, with some applications renamed so that their names are more human. For example, konsole became console. vim! \o/ But not emacs :) mc! Someone out there wanted to make my life easier. fbreader. I had never heard of it, but it's a very good discovery that I've now started to use it on my laptop as well. Little howtos To get to a terminal, hit Ctrl+T in the file manager, or Ctrl+Alt+T elsewhere. The root password is the same as the user password. To change the system language, I managed with a simple dpkg-reconfigure locales. Ways they simplified the unix system It's single user: I didn't find a way to create multiple users besides the terminal, and the login program does not ask for a username, only for the password. The "win" key has a house painted on it, and it's used as "hide/show all applications" key. When all applications are minimised, the Asus launcher is visible instead of the X background: this behaviour basically turns the key into a sort of "run application" key. The key still works as a kind of shift, although it probably was not intended to. The repository management is interesting. /etc/apt/sources.list contains:

deb http://update.eeepc.asus.com/p701 p701 main
deb http://update.eeepc.asus.com/p701/tw p701 main

which means they have a repository per eepc model and a subrepository per localised version. The "Internet" group of applications has a Wikipedia toplevel application: it's nice to see the ecosystem of free software / free culture coming together to provide a nice user experience. An extra link to the SD card mount point (besides the one in /mount) appears in the home directory automatically when the SD card is inserted. This means that when you do "save as" from all sorts of applications, the SD card is there, easy to reach. This helps if one decides not to use the internal flash for data, and just save everything in the SD card: I like doing this, as it allows me to quickly move the SD card with all the data between the EeePC and other computers. Changes I made so far Activate en_GB.UTF-8 via dpkg-reconfigure locales. Add en_GB.UTF-8 to /etc/scim/global, to get SCIM input methods to work. Little flaws Virtual screens are enabled, so W+arrow switches virtual screen. The feeling you get if you hit W+arrow is that all your applications disappeared. This could be improved by having the vm keep the asus launcher at the bottom of the current virtual screen, instead of just at the bottom of the first screen. Or, to disable virtual screens by default. It is possible to drag the lower panel around, maybe accidentally: that's another of our fancy default "features" that should be disabled by default. It is also possible to remove applets from the applet bar by mistake: for example I wanted to disconenct the wireless, and I instead ended up quitting the wireless applet. Luckily, the next time I started the computer it magically came back. ~/.xsession-errors is continuously getting the useless stdout/stderr debugging flood of GUI apps. Noone bothers usually, except that in this case the file is on flash, where unneeded writes are also very much unwanted. I'm considering symlinking it to /dev/null, but ideally we should get GUI apps to only write out what is really important. Battery charging doesn't show how long it is going to take until the battery is fully charged. No capslock or numlock leds. This probably calls for disabling or remapping of capslock. Numlock is very hard to hit by accident, but capslock is. Random thoughts If you buy an eeepc, I really suggest you think of it a mass consumption appliance and stay on the original OS for a while. Most of what's in here is what we use everyday, just on a different context. Try to use it as an appliance and see if it is perfect, and if it isn't, try to find out what is missing. It is a fantastic way to find out important bits that are missing in Debian as well. Also, if you're used to tailoring everything to yourself before starting to use a Linux system, this is a great way to try the usage experience that we can offer by default. The Firefox welcome page the first time you connect, for example, is surprisingly nice. Everything we know as doable comes a bit as as a surprise because this time someone has done it for us. I wish that that someone can be invited to talk at the next Debconf: the possibility of having a look at the work that has been done in bending Debian to this nice little device is to me one of the most valuable things so far about the eeepc. Help/About KDE/Credits It's reachable by most applications, and says:

The development team would like to thank the following people and organizations for their contributions:

• the Debian Project,
• the GNU Project,
• the KDE Project,
• the Mozilla Project,
• the OpenOffice.org Project,
• the SAMBA Project,
• the X.Org Foundation,

Linus Torvalds and the other Linux kernel developers, and Free software developers around the world.

I'm using an appliance that is thanking me, and others like me: priceless!

Enrico's post about the EeePC made me look closer to this thing. Wow. I actually didn't know that this PC is not only designed to be cheap and small and running Linux....but *also* has a custom Debian distribution pre-installed. Given the various explanations given by Enrico, that seems to be a killer CDD, indeed. It's been a while since I want to install a machine for my father in law, who's 85 without any computing skills. I was about to install a light Debian system on a small laptop but I was really fearing that some quirks we (geeks) never notice would be blockers. Apparently, Asus managed to polish everything to a level that's suitable for the average user (making compromises, of course, such as the user management system but, hey, when one knows about the "user management" in Windows XP Home, that's still a huge improvment). So, the last blocker for /me to buy such a beast is the damn keyboard. I can't imagine giving a machine without our beloved French AZERTY keyboard (even to a non computer-skilled person: we have tons of appliances around here, including the (in)famous Minital that do use this keymap). So, do Asus sell these for exotic countries? :-). Of course, any hack like putting stickers on a crappy QWERTY keyboard is out of question.

While looking at outstanding bugs against my Debian packages, I noticed that an RC bug had crept in against Planner. The bug itself isn't new. In fact, Ubuntu had experienced it already and simply decided to build their package without SQL support to resolve the issue. Planner is a fairly well-matured piece of software, with very few bugs remaining. It remains quite usable in everyday use. However, it has fallen behind in terms of keeping its code up-to-date against the latest versions of some optional dependencies. Tonight, I was just sitting here pondering what to do and it occurred to me that I'm simply no longer interested in maintaining the package. I have done a fairly good job of overhauling the build scripts and of upgrading them for the Python transition but, at this stage, I feel that I've accomplished as much as I could do. Besides, I no longer have the time to maintain so many packages, so I have decided to focus on maintaining those I actually care for and Planner isn't one of them. The package already is assigned to Debian's GNOME team, so no need to actually orphan it, since anyone from the team can pick it up. If you're a member of the GNOME team and feel like taking over, perhaps even submitting patches to upstream to close a few more bugs, go right ahead and jump in.

Using a bluetooth headset for giving presentations While looking for ways of flipping slides while not near the computer, I stumbled on this idea this idea, bought a cheap headset for like 10euros and started playing. Getting the headset to work with alsa It's supposed to just work, but bluez-utils in Debian has audio support disabled. I've rebuild the package with the patch found in the bug report, copied the extra configuration files as I mentioned in a followup to the bug report, and brought my Debian up to speed. Once the patched bluez-utils is in place, this is how to get the headset to work:

1. put the headset into pairing mode
2. hcitool scan (to get the address of the headset)
3. put this in ~/.asoundrc (using the address you got with hcitool scan):

   pcm.bluetooth_hw  
      type bluetooth
      device 00:11:22:33:44:55
    
   ctl.bluetooth  
      type bluetooth
      device 00:11:22:33:44:55
    
   # No idea why this is needed, but it makes aplay complain a bit
   # less, and also it makes aoss work
   pcm.bluetooth  
       type plug
       slave.pcm "bluetooth_hw"
    
   

Now, forget hcitool cc: in order to pair with the headset, the command is:

mplayer -ao alsa:device=bluetooth moo.wav

This will trigger a funky chain of events that will eventually cause the bluetooth applet to ask you for the pin (0000 for the headsets), get things paired and activate super cow powers. Now you have an alsa sound card called 'bluetooth' that works only with some, but not all, alsa applications. mplayer works. aoss works, but not always. aplay seems to work. twinkle won't work; apparently, it's because it tries to open the device multiple times. Feeding synthesised speech to the headset Synthesised speech is done by festival. Festival works with oss. The headset works with alsa. aoss works, but not always. This is the script I'd like to run:

#!/bin/sh
warn()  
    echo -n "Attenzione: "
    if [ $1 == 1 ]
    then
        echo -n "manca un minuto"
    else
        echo -n "mancano $1 minuti"
    fi
    echo " alla fine."
 
(
    for x in 5 4 3 2 1
    do
        warn $x
        sleep 1
    done
)   festival --tts --language italian

By tweaking the sleep times and numbers a bit, it's a useful prototype that sends updates on how much time is left for the presentation. This, under aoss, does not work reliably. An alternative is to replace the festival invocation with this:

echo string   text2wave tmpfile.wav
mplayer -ao alsa:device=bluetooth tmpfile.wav

However, I found no way to tell text2wave to use Italian instead of English. I did manage, however, to get some link between festival and the headphones. At a further stage, I can write a program for the countdown that keeps the audio device open and every once in a while gets a waveform out of text2wave and feeds it to the audio device. Keeping the audio device open seems a good idea, to avoid losing the beginning of the message while the connection is established (happens with my cheap headset) and to avoid the headset to go idle and switch off. Reacting to the buttons on the headset It's supposed to be possible, but I didn't manage to find out how to do this using the normal alsa driver, now that btsco is apparently deprecated. Once I find out how to be notified of button presses, I'll try to see how to send a Page Down keystroke to OpenOffice Impress or xpdf. Voice commands sphinx2-bin seems promising, but since everyone around me is now asleep, playing with voice commands has to be postponed. Further ideas

• IRC link: send me a message on a public IRC channel, and I could head it on the headphone: nice for giving talks that people is watching remotely via streaming.
• Giving the percentage of time passed, instead (or additionally to) the time remaining.
• Linking with the presentation software and also say the number of slides remaining.
• Get the status update being spoken not just at regular intervals, but also when changing slide.
• Get the headset to read slide notes, or only the slide notes marked with a special leading keyword.

I’ve just read an interesting article in the New Yorker titled “Twilight of the Books” [1]. It started with some depressing statistics about literacy. One comment that I couldn’t figure out concerned “the panic that takes hold of humanists when the decline of reading is discussed“. The decline in literacy concerns me because I want to live in a society where everyone can contribute as equals and where all voters have the possibility of becoming well informed about the issues that affect their vote. If literacy declines far enough then fascism seems to be the inevitable result. I’m not sure whether that makes me a humanist according to the author. Below in the second part of this post (not included in the feed) I include my dictionary definition of the word, #4 is the definition most commonly used today (which is a reasonable match for my beliefs). There is some interesting information regarding basic research on the mental abilities of literate and illiterate people which suggests that people who an read are better at handling abstract concepts and hypothetical situations. The article notes that spending time on the net increases academic scores for children. It also makes the interesting claim that visiting pr0n sites increases grades, I find that difficult to believe. I wonder whether there is a difference between readers and writers in this regard. Does the typical serious net user who averages writing more than 3650 messages a year for fun (in addition to whatever email may need to be written for work) plus some blog posts and writing other web pages have different mental abilities to someone who just reads what others write? Does blogging (and other writing on your choice of topic) change your brain? What about programmers, does spending 8 hours a day writing formal rules for hypothetical mathematical situations (which is essentially what programming is about) change your brain? Are programmers going to have greater abilities for managing abstract concepts and hypothetical situations than people who have not had such experience? NB It will be difficult to determine this as people who have such skills will find programming easier and more fun so therefore a career in programming computers is self-selecting for people with such abilities. Another interesting fact is the results of research into the comprehension of a PowerPoint presentation. People who read it without an audio-visual accompanying it learned more and rated it as more interesting. Based on this my plan is that in future when attending conferences I will try and read papers which are presented BEFORE attending the lectures. Unfortunately most conferences which have papers presented seem based on the idea that you give a lecture about a topic as a teaser for your paper and assume that the audience knows nothing about it (conferences which don’t have papers outnumber the ones that do). Most Linux conferences seem based on the idea that almost no-one reads papers at all, they attend lectures and visit web sites and IRC channels afterwards. NB this comment is based on talking to people who attend conferences and my observations on attending conferences - it is not a criticism of any conferences merely an observation on cultural attitudes. Based on this information it seems reasonable to provide copies of lecture notes to the audience before the lecture. For example for a typical talk that I give to a LUG I could blog the lecture notes before the meeting and email the LUG mailing-list with the URL, people who really want to learn would hear me talk AFTER reading the notes and hopefully get the best of both forms of learning. I had been considering producing some YouTube [2] videos with instructions on how to use SE Linux. But it seems that might not be such a good idea (at least not a good enough idea to justify the amount of effort I would have to spend). The author seems to misunderstand YouTube, it is NOT about making the Internet resemble TV (where millions of mindless drones watch content created for the lowest common denominator in the audience). The AtGoogleTalks on YouTube [3] and www.ted.com are examples of high-quality free content (comparable to the best TV documentaries - the type of content which is cited as increasing the academic results of children) and the culture of creation and discussion must be a good thing. Most TV is bad because it’s one-way communication and the quality of the content is low. YouTube videos are widely discussed in email, blogs, and other written forums. The creation of the best (and often most viewed) videos would surely involve some writing. I wonder whether the benefits attributed to reading can be obtained without reading fiction. The nature of work is changing to require an increasing amount of reading and writing. The most menial work requires reading instruction manuals. Most of the best paid jobs involve a lot of reading and writing. If reading fiction isn’t required then it seems that there is little risk in the immediate future.

• [1] http://www.newyorker.com/arts/critics/atlarge/2007/12/24/071224crat_atlarge_crain?currentPage=all


• [2] http://www.youtube.com


• [3] http://au.youtube.com/profile?user=AtGoogleTalks

(more…)

Share This

Today I decided to add some EWMH support to awesome. It now supports a bunch of this extensions quite nicely. However, while reading the spec and writing the code, it appears that this forces a window manager to behave in only one way: have a poor desktop support, and no multi-head/XRandR/Xinerama support at all. The main caveats are that in Xinerama/XRandR mode, you'll have only one root window. And the root window is where you must store the NET_WM X properties So you cannot handle screens in a independant way like awesome does. That's really a shame. There's also a big problem for window managers like awesome which are happy to draw several desktops at the same time. There's no support for stuff like that. So far, I think EWMH is nice but is really too narrow-minded for softwares and people who want to think window management in a different way.

Linus on the value of documentation:

You seem to put a lot of trust in a piece of documentation.

Do you realize how those pieces of paper are written? They are written by people who have absolutely *nothing* to do with the actual implementation, and whose job it is to write documentation. And while the people who actually do the programming etc are supposed to help them, the two parties generally detest each other.

Technical writers hate the "real engineers" for not helping them, and the "real engineers" tend to dislike having to be pestered to explain their stuff and have to read through some document that isn't meant for them, but that they need to sign off on.

In other words: please do *not* expect that the documentation actually matches reality. You seem to think that the documentation came first and/or is quite accurate. That's not at all likely to be true.

(link)

Pelican has decided to abandon his efforts to scale the walls of the legal academy and proposes this simple eight-step program for others who would follow in his footsteps:

1. Don t do interdisciplinary work. The legal academy doesn t know what to make of it unless it is economics.
2. Don t go to a school without a law review or grades. I did and it was a huge problem.
3. Get on law review, clerk, write.
4. Check the faculty listings at most law schools. Go to the most often listed schools: Harvard, Yale, etc. It does matter as law faculty select their own, usually. I was told by a faculty member as a 1L expressing interest in the academy that I should transfer immediately to Harvard if possible. I didn t.
5. If you think you want to be a legal academic, look at what is on the FAR form in your first or second year. Orient your academic career to produce a good looking FAR.
6. Remarkably, the legal academy does not care about your ability to raise research money or bring in grants.
7. Don t publish in interdisciplinary journals. Publish only in law reviews writing only dense and impenetrable texts.
8. Demonstrated impact of your work in policy or law is not relevant.

This critique could be extended broadly to much of the academy (not just law schools). I had once thought I might like to teach law, but now I can only really see myself as a clinician. Two contradictory academic trends:

• The tendency toward dense, impractical writing that only makes sense to other insiders. This is often but not exclusively the influence of postmodernism. Steve recently brought Chomsky’s response to postmodern thought to my attention. It boils down to this: sure, it might be complicated. But there are lots of other fields that are complicated–quantum mechanics, nanotechnology, even game theory–in which an expert can still explain to a bright layperson the contours of the field sufficient for the layperson to get the gist of it, even if they can’t grasp all the details without extensive study. This doesn’t seem to be true of postmodern philosophy. Is the emperor naked? Paul Graham’s How to Do Philosophy reflects a similar attitude with less of a political bent, although my friends pursuing philosophy doctorates claim Graham has got it all wrong.
• The tendency to reduce all disciplines to a science, particular the humanities and social sciences. This can either mean approaching the study with some method that approaches “scientific rigor” or literally basing the analysis in an already recognized science, preferably neuroscience. While I can see some sense in incorporating neuroscience, cognitive science, and certainly psychology into fields such as economics, I don’t think this is a good thing to the extent it overwhelms other modes of analysis, particular for literature. Neuroscientist/author Jonah Lehrer discussed this phenomenon (and its drawbacks) with Christopher Lydon in an episode of the newly-revived Radio Open Source. (Warning to Debian readers: Radio Open Source has very little to do with open source software.)

Technorati Tags: Law, Legal Academy, Postmodernism, Neuroscience, Jonah Lehrer, Paul Graham, Philosophy, Noam Chomsky, Christopher Lydon

Share This

Some time ago I've been working on an AI to classify web pages. I've spidered (and classified!) about half a million web pages I'm using for training.This totals to about 40.000 categories and 100.000 useful tokens. I need 8 bytes of storage - two float values - for each combination, which is a matrix of about 27 Gigabytes. I'm using mmap to map my data files into memory for convenient access. On a 64 bit system, I can keep them mmapped all the time.This results in the following funny top output:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12443 erich     26  10 27.7g 309m  21m R  8.5 32.8  86:01.98 train

So according to top, the process is using 27 GB of virtual memory. That is the data files, actually. The 300 MB resident memory is the training data it's processing. At this point, all my data has been converted to 32-bit integers, so this is already condensed down a bit.It's also obvious that ext3 supports sparse files - running "du" on my database directory currently gives 1.5 GB (it just has started filling the matrix), adding the "apparent size" switch gives 29G.I hope that when the training run has finally completed (ETA is a couple of days), the AI will actually be able to classify webpages somewhat reliably. :-) You can think of it like a huge bayesian spam filter. Except that I've added some boosting that should improve results, some computational improvements for speed and scaled it up to 40.000 categories instead of just "spam".Other uses I'm planning for the AI is automatic email filtering into multiple folders (best to be employed on Google GMail, where an email can have multiple labels, not just one folder), automatic document classification for document and knowledge management systems, etc. pp.[Update: I havn't yet decided if or when I'll release the code as GPL. Right now, the code could require some cleanups. And I'm considering to start a business around this datamining and classification stuff, so I might want to keep it closed for some time at least. The business will most likely be around integrating the code though, so a public release might well be possible.]

One great thing about kids — especially kids who have not been overexposed to TV media — is how easily entertained they are. We recently showed Esther some Looney Tunes cartoons from the 1940’s (on a real TV, no less!) and she couldn’t stop laughing. It is refreshing to see someone spellbound by animated content that didn’t require a rendering farm to create. I’m sure she’ll be jaded soon enough, but we’re trying to drag it out as long as possible by providing only very occasional small doses. This piece on the end of reading as a cultural activity is chock full of interesting statistics. This excerpt, in particular, is relevant here:

In August, scientists at the University of Washington revealed that babies aged between eight and sixteen months know on average six to eight fewer words for every hour of baby DVDs and videos they watch daily. A 2005 study in Northern California found that a television in the bedroom lowered the standardized-test scores of third graders. And the conflict continues throughout a child s development. In 2001, after analyzing data on more than a million students around the world, the researcher Micha Razel found little room for doubt that television worsened performance in reading, science, and math. The relationship wasn t a straight line but an inverted check mark : a small amount of television seemed to benefit children; more hurt. For nine-year-olds, the optimum was two hours a day; for seventeen-year-olds, half an hour. Razel guessed that the younger children were watching educational shows, and, indeed, researchers have shown that a five-year-old boy who watches Sesame Street is likely to have higher grades even in high school. Razel noted, however, that fifty-five per cent of students were exceeding their optimal viewing time by three hours a day, thereby lowering their academic achievement by roughly one grade level.

It certainly doesn’t surprise me that television–even educational TV–is generally not good for kids. The “inverted check mark” bit was a surprise, though. Who would guess that two hours a day is optimal for the nine-year-old brain? On a related note, Kiddie Records Weekly (recommended by about.com here) is an astonishingly extensive source of free children’s books in MP3 format. It’s also an excellent example of the increasing relevance of bittorent in enabling wide economical distribution of legitimate content. The content is primarily (exclusively?) digitized vinyl records from the 1930’s and 1940’s. Fortunately, Alice in Wonderland gets no worse with age. To complete the nostalgia circuit, who can forget “loaf of bread, a container of milk, and a stick of butter” (via sketchelement, quite a while ago): These are good times to be a kid. Or maybe just to be a parent. Technorati Tags: Looney Tunes, Alice in Wonderland, Reading, TV, Cartoons, Sesame Street, Bittorrent

Share This

Full moon is bright in the skies today and tomorrow. Moreover, it will share the skies with Mars, reaching its opposition point. So, for everybody around, just go outside a few hours after sunset and enjoy. Tomorrow (Dec. 24th), some will even enjoy an occultation with the Moon passing in front of Mars. See this page for occultation details and locations. In short: state of Washington, USA, British Columbia (CA) and other parts of Western Canada, Greenland, Scotland, Germany, Czech Republik, Hungary, Serbia, Greece, Turkey, Lebanon, Israel. Then some parts of Armenia, Iran, Uzbekistan, Russia (Siberia) and Aleoutians.

Thursday evening, I joined the inhabitants of the twin town of Valga-Valka, sitting smack on the Estonian-Latvian border, to celebrate their entry into the Schengen treaty. Ever since I first crossed the border there on a roadtrip from Tallinn to Fallingbostel, in year 2001, I knew I would have to return and, sure enough, I briefly passed through during last summer with an Estonian friend, on our way to an acquaintance's birthday party. Still, that told me nothing of the town's life and left me wanting for more. Upon hearing that year 2004 EU accession countries would join Schengen in December 2007, I immediately promised myself to show up and join the crowd. As it turns out, I missed Aleks Tapin of the All About Latvia blog by very little, having I caught his last-minute e-mail the next afternoon. Aleks blogged a great article depicting the Latvian side of life and providing some background info on the town, if you're curious. I arrived late-evening on Thursday via the Tallinn-Viljandi-Valga bus and checked into my hotel, then proceeded to checkpoint Valga II around 23:30, with the intention of grabbing a drink on the Latvian side and returning just before midnight for the celebrations. Hardly anyone was in sight, except for Latvian officer abana who was completing the inspection of a Russian car with three noisy passengers. Upon presenting her with my passport, officer abana cheerfully lead me to the office and slid my passport through a slot to officer Bukss, who was visibly surprised to have any tourist show up on the last day of his job to get their passport stamped. Upon explaining to him the reasons for my presence, he pointed me to a nearby bar where I could have a drink, visibly amused by the situation. I spent the next few minutes in a local casino in the company of three hilarious Latvian truck drivers, Zintars and Aivars, two Latvians with limited English language skills and Yuri, an ethnic Russian living in Ireland who was born stateless on the Latvian side of the town and who later acquired Estonian citizenship by claiming ancestral land on the Estonian side. Upon returning to the border, just minutes before midnight, I found myself in the middle of a huge crowd of villagers, police officers, border guards and politicians from both countries - barely getting noticed by anyone. I handed my passport to an Estonian border guard who emotionally commented to a civilian friend of his nearby that, "Wow! That was the last one!", handing me my passport back just as the midnight bells rang and the mayor of the Estonian side started his speech amidst pyrotechnics lining the road. I then had a chance to chat with the Estonian mayor, who promptly handed me a bilingual certificate, signed by the mayors of both towns, attesting that I was the first to cross the open border, while introducing me to his two young daughters with whom he was about to take a stroll on the Latvian side. The next morning, an even bigger and more symbolic event took place: the demolition of the fence that had been cutting the S prus/Rai a street in half. You can see a picture of what the street looked like before on Aleks' blog article [edit: or on Jens-Olaf's blog article]. After the fence was removed, it instead looked like this: The crowd then proceeded to checkpoint Valga III, where a stage had been setup for the politicians to make their speeches. An interesting fact is that, because of the way the border was drawn, along a creek leading to Pedeli river, an Estonian main road was technically on the Latvian border and thus aptly named "Raja" (border) street. The Estonian mayor commented in his speech that the the brand new supermarket standing behind us was also technically on borderland and could not have possibly been built earlier, weren't it for the Schengen treaty matter-of-factedly eliminating borders between participating EU countries. The speeches were closed by a youth group whose choreography featured six break-dancing boys, dressed as Estonian and Latvian border guards and as border posts. The choreography ended with the teenage border guards carrying away the human posts on their shoulders, just as a choir of young girls replaced them with songs in either languages. Before catching my bus back to Tallinn, I dropped by the Valga tourist info, only to face a nervous-looking Kapo officer in plain clothes. Upon mumbling something about the tourist info and wanting to grab maps, I noticed that a press conference was taking place behind. As he finally let me pass to the tourist info side of the building, a familiar voice said to me, "Hey! Weren't you at our place last summer with Martin Ranna?" Yup, the wife of the acquaintance whose birthday party I had attended last summer is working there and greeted with me refreshments and munchies she had cooked in prevision for the presidential visit! As was my luck, she had one of the commemorative plaques that had been given to the politicians at Valga III on hand: I was glad to see the town becoming one again, even though it is ethnically divided. There are already signs of people on both sides learning one another's languages and shopping on either side of town, not to mention plans of merging the municipal bus operations of the Estonian and Latvian halves of the town, so I'm sure that they'll get there in due course. For me, Schengen brought a much more practical and quite welcome change: the end of messy border crossing stamps that were rapidly filling my almost new passport, every time I visited the head office of our company in Tallinn. While Estonian border guards mostly stamp passports in an orderly fashion, being careful to fit exactly 8 stamps in a single page and to put entry and exit stamps side-by-side, Finnish border guards apparently are incapable of doing so, instead systematically wasting pages by either stamping sloppily in a way that makes it impossible for more than 4 stamps to appear on a given page or by flipping to a brand new page, every time I had to cross the border. In case anyone ever wondered why I'm currently representing Estonian interests abroad, despite living in Finland, this example is one of the many reasons: Finnish authorities routinely display arrogant carelessness towards the population and doubly so towards immigrants. It amounts to an accumulation of idiocy that has costly consequences on people's lives. For some, it's about being denied the public institutions' support when they need it the most and their life forever going downhill thereafter. For me, it has been about countless missed opportunities (jobs, love affairs, travel plans), plus very costly passport and residence permit renewals. Given this, I simply don't see myself ever representing Finnish interests ever again. Faith no more. Besides, the Estonians are fun and easy-going people. Siski ma m tlen, kas ma peaksen n d Eestisse kolima v i?

It took them half a year, but the Belgian federal politicians finally managed to come up with some form of government. I can't say I'm impressed. Well. In a way I am impressed. After trying for 6 months, Leterme still failed to produce some form of government. Since the situation was becoming quite dire, the King gave Verhofstadt the initiative, who then promptly produced a government in only two weeks. Granted, Verhofstadt's mission was not as hard as Leterme's. Verhofstadt was told to form a government only to handle the most urgent matters, and which will be disbanded at Easter; Leterme was asked to form a government to last for four years, which is how long a Belgian federal government usually holds office. And yes, Leterme's mission has most likely been one of the hardest one in the history of Belgian formation talks. But even if all that is true, I'm baffled at the fact that the intention now still is to make Leterme the prime minister at or around Easter. In my opinion, he's had his chance and blew it. Then again, maybe I'm biased. I've never liked Leterme; and when his party CD&V decided to ally themselves with a separatist party, I decided to gracefully extend my antipathy to his entire party.

In a comment on my post about Bad Project Management [1] Don Marti [2] says “the more money you charge, the less of your time people waste, and the more seriously they take you” and “you can affect the client more, whether you’re peddling idiocy or wisdom“. That’s a nice theory, however I don’t recall seeing an example of it anywhere that I have worked. There are two broad categories of employers, large and small. Large companies have HR departments etc who decide on rough pay scales. Once the budget is approved (by senior management and HR) for someone’s contract fees or salary the amount of money seems to be of little concern to the line manager. They have a group of N people reporting to them who are all paid by money that comes from some mysterious place (with no direct connection to the quality or quantity of work done) and therefore the amount of money paid is not very relevant. Often team-leaders don’t even know how much the people who report to them are being paid! Note that this applies to people who have a similar job title, if I am paid twice as much as another sys-admin in a corporate environment I don’t expect to get more attention, but someone who has a more senior job title (or in the case of the UK has been to a more prestigious school or has a better accent) may get more attention. Small companies are rarely in a position to be silly in this way. This doesn’t mean that the most highly paid person gets the most attention, it’s more a matter of making a logical case. Most of the small companies I have worked for have been run in a sensible manner, and my ideas have often received more attention than those of others because I have had a better technical background than my colleagues. However in some cases my suggestions have received less attention because I was receiving a higher pay rate. The reason is that I was being paid a rate that the company could not afford in the long-term and therefore the people who would work there in future years would have to maintain the systems in question which meant that their ability to do such maintenance was often considered to be more important than my ideas about the best way to design the system. I have worked for a small company that had no financial pressure to do sensible things (they had some sweet deals going with major corporations which essentially gave them money for nothing). They had chaotic management which they could get away with because there were no immediate consequences for doing silly things (such as not forcing their clients to have a method of backing up servers that they installed). I can’t claim that Don’s advice in this regard never applies, but my observations of situations where it didn’t apply lead me to believe that it’s applicable to some situations in limited areas and is not a general rule. Don also refers to “a high number that you would quote to a client you didn’t want in order to get them to go away“. This is an issue that causes a lot of disagreement. One consultant I respect advises people to never make such quotes, if you don’t want a job then say that you are not available - one reason is that if a position is so bad that you want the potential client to go away then you don’t want to be tempted to reconsider if they agree to pay you a silly amount of money, another is that it may be considered ethically wrong to make an offer is not genuine. Another consultant I respect claims that his greatest career success was when he turned up to an interview in casual clothes, explained to the potential client that he would never wear a suit and demanded an excessive amount of money. The client agreed to such terms and he then went on to charge other clients similar amounts of money and continued to refuse to wear a suit. I won’t claim that it’s wrong to demand excessive amounts of money to make an unwanted potential client go away and I will not rule out the possibility of doing so at some future time. But I think it should be noted that the practice is controversial and not referred to as something that anyone might do. It is something that I don’t recall doing at any previous time in my career (so I haven’t done it in a formal manner at least - I may have done so at the informal discussion phase and forgotten). Please note that I encourage you to read my previous post with Don’s comment in it’s original context to avoid any risk of misinterpretation. The purpose of this post is not to disagree with Don (I agree with him almost all of the time) but to disagree with some very commonly held beliefs which Don represented in his comment.

• [1] http://etbe.coker.com.au/2007/12/19/bad-project-management/


• [2] http://www.linuxworld.com/community/?q=blog/6

Share This