Given my desire to see the community around Squeezed Books grow, I went ahead and added a creative commons 'attribution/share alike' license to it. I also did this in the spirit of reciprocity towards other people pursuing similar initiatives and using similar licensing. The goal really is to give people an idea of what some of these well known business books are about. Even though the book in question may present a unique and interesting idea, many of them can be summed up pretty quickly. As Phil Greenspun says:

This book illustrates a fault in the publishing industry. If you have a 50-page idea it is too long for a magazine. But it is too short for a book. So if you wanted to get it distributed before the Web came along, you had to drop in words until you reached 200 pages.

I don't really think the web has changed things that much, either. Who wants to simply put up a web page when they could publish a book? The idea isn't necessarily to avoid buying books, either - some of the good ones are really good reads, and a summary just isn't the same thing as the book itself.

It's been a few days since I posted on my blog -- since I left Montreal for Taipei and Wikimania 2007. My time has been upside-down and backwards as I've traveled around to the other side of the clock and a day into the future, and I've just been insanely busy with other work. My flight to Taipei was dreamy. I travelled enough last year to get Air Canada's "silver" frequent-flier level, which earned me a couple of system-wide upgrade certificates. They came in handy, since I bumped up to business class on the leg between Montreal and Vancouver, and then to the "Premier Laurel" class on the EVA Air codeshare between Vancouver and Taipei. That was first class across the Pacific Ocean, and man, it's the only way to fly. Jimmy Wales was on the same plane (but back in business class...), so we got to chat a bit and we shared a taxi to the conference site. Jimmy's really fun to talk to -- it's always nice talking to another entrepreneur who's passionately dedicated to Free and Open Content. (We were in line for customs together, too, and I tried to peek at his passport to find out his date of birth, over which there is some confusion. But he had his thumb over the day (intentionally...?), and all I got was "Aug 1966", which everyone agrees on anyways. Dang! My personal theory is that Jimmy was born on 6 June 1966 and uses 8 Aug 1966 instead to avoid the demonic stigma.) The conference centre we're in is great -- a really big, clean, modern place. It's nice when we're inside, but Taipei in August is muggy and hot like a laundromat in rural Mississippi. I find myself rushing between buildings. tags: wiki wikimania2007 taipei evaair jimmywales travel

IM IN TAIPEI HACKING UR DAYS I came in a couple of days early to participate in the Hacking Days event. This is a pre-conference meetup for MediaWiki developers and those who love them. The meetup was really unstructured and fun because of it, although there were a few points of uncomfortable silence when people stopped talking and we all realized we had 14 more hours of meetings and not much left to say. There's also a real tension between the conference organizers, who've encouraged non-developers to come to Hacking Days, and the core developers who really want to just be left alone and get some work done. This year the 2-day event was split into "Hacking Days" and "Hacking Days Extra", with the first being invite-only and the second being open to all. I think it's a tension that needs to be resolved -- if outsiders aren't invited to Hacking Days, we just need to say that and live with it. We shouldn't let people think they're going to participate and then cut them out. That's a bait-and-switch tactic and it's unfair. I think that there's a need for an open-attendance MediaWiki conference; I also think that MW is a big enough piece of software that third-party conference organizers would be eager to make an MWCon work. And core developers could get hefty fees for coming to talk! tags: wiki mediawiki hackingdays wikimania2007

Announcements I've been having problems sleeping here. I've been falling asleep at the wrong times and waking up at the very wrong time. Last night, I missed the Wikitravel Eat-together because I fell asleep at dinner time and didn't get up for the event. Make mental note: don't miss your favorite wiki's user get-together! Fortunately Jani covered for me and apparently the event was a lot of fun. Speaking of Jani: this morning he announced a new venture that Maj and I have been helping him with. Wikitravel Press is an independent company that will be publishing a line of printed travel guidebooks (yes, paper books! That you can hold in your hand!) based on Wikitravel content. I think it's a great project and I'm really glad we're going to finally be fulfilling this important goal of the Wikitravel project. I also think it's going to be a great way to attract a new round of contributors -- people who are more reliant on their paper guidebooks than to a search engine for learning about their destinations. I think that the idea that they can change and improve the books they have will be a really important incentive for these folks. And like it or not, there are still a lot of people who don't consider digital information "real" in the same way as analog-stored info. So putting Wikitravel guides on paper is like Pinocchio becoming a Real Boy. Thank you, Blue Fairy! More info at the project's announcement and the on-site discussion (Wikitravel:3 August 2007). tags: wikitravel wikitravelpress guidebooks travel announcements

Called out This morning's keynote address by the Wikimedia Foundation chairwoman Florence Devouard was informative and interesting. She even talked about ways to bring money into the Foundation... including selling books based on the site content. I was giggling about this with Jani as she moved on to talking about the attention economy, to hear, "For example, Evan is not paying attention right now and talking to his neighbour." D'oh! Lesson learned: don't mess with User:Anthere! tags: anthere keynote books wikimediafoundation

After many months of few contributions, and lots of spam, I decided to shut down the Windows for Linux Users wiki, and transfer its contents to an article on my new personal web site: http://www.welton.it/articles/windows_for_linux_users.html I will still accept contributions, of course, but it's more efficient to simply apply them myself rather than sort through a lot of spam on a relatively static wiki. The content is based on my experiences with windows several years ago - I realized that I knew very little about using it productively, and so I attempted to share what I learned about turninig it into an environment I could be productive in. Since most people go from Windows to Linux, they're not likely to face those problems, but for those who do, some of the listed tips and tricks are useful. These days, I can happily report that I'm happily back to a windows-free life.

Sometimes in programming one gets to concentrate for a bit on a single, simple, pure program. A command line utility which is completely in batch mode, does not handle any kind of complicated data, does not need to deal with other programs, or the network, or anything nasty like that. Something where one can just look at the one or two interesting parts and not have to worry about anything else. This is the story of one such program. I need a program to generate test data sets for backup performance testing. The program needs to be reasonably efficient, since the simulated backups may be rather large. My first attempt was not fast enough. There were two reasons for this: the way I was generating random binary data, and the way I was generating filenames. The program generates random binary data to simulate large files, such as image, audio, or video files, and thus it needs to be uncompressible with the usual compression algorithms. My first attempt was to generate each output byte with the Python pseudo-random number generator (random.randint). This only results in about 0.1 megabyte/s (on my laptop), so I needed to improve it. After some iteration and advice, I landed on the following: I generate random bytes with the Python PRNG (random.getrandbits), and feed these to an MD5 checksum. After each byte, I get the current digest and append it to the output. This way, for each random byte, I get 16 bytes of output. It isn't completely random, but it satisfies the uncompressibility requirement. This resulted in about 2.5 MB/s. I then tweaked the code a bit, doing things like avoiding unnecessary arithmetic. I further decided to only feed the first few dozen bytes from the PRNG, and the rest as the same constant byte. The initial run of randomness is enough to make sure the output is unique for different files, so that compression doesn't find lots of duplicates of the big files. This resulted in a speed of about 6.6 MB/s on my laptop and about 13 MB/s on my server. This still didn't make the program fast enough, but the remaining problem was not that writing the contents of one file was too slow, but operations on directories were very, very slow. After benchmarking with zero-byte files, turns out that at the slowest, with file counts in the millions, it would create only a few tens of files per second. That's empty files, so only requiring an update of indoes and the creation of a new directory entry. Obviously, the problem was that I was putting too many files in directories, causing the filesystem to spend a lot time scanning them. I was creating up to 250 files per sub-directory, and then creating a new sub-directory. All of these sub-directories were going directly under the user's chosen output directory. To measure what the optimal size of a directory was, and whether to create two levels of sub-directories, I wrote a little benchmark program. From the graphs I deduced that overall, having two levels of directories and up to 1000 files per sub-directory was the best option of those tested. On the other hand, the bigger the files, the less it mattered what the directory structure was. Perhaps the obvious reason for the slowness wasn't that, after all. Then I realized my code was slow because I was doing an os.listdir (listing all files in a directory) for every directory every time I creted a file. Fixing that idiocy made the code so fast that any overhead from the directory operations was no longer relevant. But at least I had some nice graphs. On the server I will be generating my test data the program is now happily generating just over 11 MB/s of test data. I wouldn't mind having it be a lot faster, but it's fast enough. It will take hours to generate hundreds of gigabytes, but I hopefully won't need to do that very often. Thus, further optimization isn't worth my time, right now. The program is genbackupdata, in case anyone else cares about it.

Thanks to Centauri Dreams for the pointer to the ESA DARWIN proposal now here (high-resolution version also here). DARWIN is a space mission to detect the presence of life on other planets by spectroscopy: looking for biomarkers such as the presence of Ozone, water and methane together on terrestrial-sized planets. These are groups of gases that collectively would imply the presence of life. Unfortunately it appears that DARWIN is slipping: its not currently hardware-funded, and has a planned launch-date of 2020, having slipped from 2012. This is when NASAs alternative Terrestrial Planet Finder and Space Interferometry Mission have been defunded indefinitely. At the same time, the French COROT mission has just announced its first month-long series light-curves. COROT hunts for Earth-sized planets: it's a small space telescope that stares at the same patch of sky for long periods, looking for changes in the amount of light from stars that indicate a planet passing in front of a star. It has been much more successful than expected, having already announced its first exoplanet detection in April. NASA is also launching a similar mission, Kepler, in 2008. So we face the prospect that any day now we may know of Earth-sized planets in a Habitable zone, and yet be over a decade from determining if they could contain life. So do we have any other tricks up our sleeves? Other possibilities include the James Webb Space Telescope, the successor to the Hubble. With a launch planned in 2013, it would be able to determine some atmospheric features found in COROT detections, but would be too expensive to do the planet-hunting Darwin could do. Perhaps this could be imroved with occulters such as UMBRAS (thanks to enzo). Other novel detection missions include the Voyager Interstellar Mission (via), reusing exisiting 'end-of-life' spacecraft as telescopes. And finally, could ground-based telescopes do it? tests have just started on a new 10M telescope in the Canaries that could help; in terms of larger scopes, colleagues of mine in NUI Galway have been collaborating on the design of Euro50,a proposed 50 M telescope. Together with new techniques such as spectral deconvolution (via via) may help. Tags astronomy, exoplanets

Fudge Update

Yesterday I blogged about the guy handing out fudge. He saw my post and explained why he was doing it. Today he was around handing out fudge, and I thanked him for his comment. He gave me two pieces of fudge for blogging about it.

Pineapple

At most of the breaks, they have this truly wonderful real (non-canned) pineapple. I think I have eaten more pineapple this week than any other week, ever. Very insidious, O'Reilly.

Conference materials

Slides of talks are available. Apparently keynotes are being posted on Youtube and Google Video, though they haven't provided a specific link AFAICT.

Philip Rosedale, Founder/CEO of Linden Lab

Second Life and the XPrize are two examples of escapism: you can escape to virtual earth, or escape the planet entirely.

In order for Second Life to grow, it has to become profoundly open. It has to be a standardized protocol. They are working on code to let people run their own servers. They are trying to make the server trusted and make it able to be open sourced as well. They see openness as the key way for it to grow.

If you are writing a web app that depends on the network effect -- such as Facebook -- you should open source everything right from the beginning. Not because it's best for humanity, but because you will win.

Jimmy Wales

He's trying to do open source search. Nice idea, could have been summarized in 45 seconds, I think.

Simon Wardli

Was going to talk about Zimki, which was going to open sourced, but they decided not to.

Funny and interesting talk, I sorta forgot to take notes while listening...

Nat Torkington

OSCon program chair.

Funny talk about Linux, an adolescent at 16 years old. Thinking about different languages and how they're doing. How the Linux community is organized: are we fighting battles, and do we need to?

James Larsson

This guy takes old hardware and makes insane devices out of it. 15,000 volts running over a coat hanger used as a "game controller", for instance.

In all, awesome keynotes today.

This makes me smile.

And even the DM GR makes me smile, as I hope it will pass :-) Stefano summarized nicely why I think DMs are a good thing.

And "as usual" Joey came up with a new interesting thought: The interesting thing to consider is, if this vote fails, how can the bug be fixed without another vote?

• Go to nm.debian.org/maintainers.php
• Find your name in the list
• Read upwards
• Be amazed at the amount of people who apparently aren't a DD as long as you are, but whom you thought would be.

No, really. Obviously that only works for Debian Developers. But it's still fun.

Dear Lazyweb, does anyone know a small web calendar which has a web frontend and does not rely on a database server? It should save it’s data directly into .ical (or .ics) files so I can fetch and alter them with a desktop calendar application too. I don’t need fancy features like multi user support or the ability to publish calendars, I just want do see and edit my calendar/todo stuff when I’m not at home and have the .ical saved on a centralized (my) server to make it available for desktop calendar software. For privacy reasons it has to run on my own server.

What’s the single biggest advancement Linux has brought to the industry? It’s an interesting question, and one that in my opinion has a very simple answer: Package management—or, more specifically, the ability to install and upgrade software over the network in a seamlessly integrated fashion—along with the distributed development model package management enabled. It used to be that operating systems were big, monolithic products, and applications were big, monolithic products you put on top of them. If you wanted to deploy, say, a web application, you sourced the middleware stack (which itself was probably several big products too), you sourced the operating system, and you (often painfully) had to integrate the two yourself (or pay a big company lots of money to do it for you). These days, you increasingly just “apt-get install whatever“. In this world, where does the operating system end and the application begin? The line is increasingly blurred—for when applications are deployed using an OS facility, seamlessly integrated with the OS itself, is the result an application or a feature of the operating system? In fact, in a very real way, all software looks to become part of the operating system—or, at least, this has certainly been the trend in the Linux world. What does this shift mean to the industry? For one thing, those of us that build operating systems as monolithic products have to change—it’s what users expect, and with a componentized operating system rather than a wad of stuff, it becomes far easier to push new innovations out into the marketplace and generally evolve the OS over time. Indeed, refashioning Solaris as a “distro” is the essence of Project Indiana—and package management is the key technology that will hold it all together. So, the next time you read about how Project Indiana aims to make Solaris more “Linux-like”, keep in mind that what we’re actually “copying” is the distro model, not Linux itself—which, after all, is a kernel, and has nothing to do with the package management and so forth the distros (you know, like Debian) built above it. This, better than anything else, highlights the opportunity for Solaris: What people really know when they say they know “Linux” is the environment that exists around the Linux kernel—the distro—all of which Solaris can deliver, and more.

Being a good Debian user

Notes: the bulk of this article was written on the 17th; updates were added in later; this article is really long, you might want to read it in more sessions

Debian's arm port uses hardfloat. This means that apps like rrdtool (graph function) take ages to run because there is no floating point unit. In order to make this faster you'll want to use soft-float emulation. But this support needs to be present in all depended upon libs, including libc, which is an ABI incompatible change which would break all binaries. I can't use the yet-in-development-and-unofficial armel port since the machine has to be up and running almost 24/7 and I can't afford downtime.

So what to do?

Well, obviously use another soft-float enable libc, like uclibc, in parallel withe the hrad-float enabled libc. Debian Sid contains uclibc development libs, but building the whole toolchain is a pain. There is no real support in the distro itself and building on the arm machine is kind of out of the question. I tried a little in a debootstrapped chroot on the native machine, but this was a pain and I was stuck tring to figure out how to disable whitles and bells for rrdtool since I didn't needed any of the perl, tcl or python support. So I stopped, went to bed and decided I'll continue the next day (the day I wrote the biggest part of this article). The future looked dark, trying to build a softfloat uclibc rrdtool on a system that wasn't prepared for that and from which softfloat was pulled away somewhere during etch's development stage....

Deciding to go with the competition - native building

From link to link I found out that gentoo has arm-softfloat-linux-uclibc port, among others, so I decided to use that on the target machine (a NSLU2) in a chroot. Well I stumbled on the low computing power, when I tryed to emerge sync the tree. I gave up un that and tried to use emerge-websync. It looked better, but I didn't got too far with this either since the machine ended up being overloaded, with figures varying from 3 to 8 for the load average (all of them).

Deciding to go with the competition - cross building

It was time to bring in the big guns and I decided I was going to crossbuild those binaries, probably statically since it meant lower risk for things to break on the target machine.


Since I observed that gentoo has a nice tool called crossdev which actually uses emerge (the equivalent of dpkg+apt in Debian world) to make the cross chain tool and saw that they have some really good documentation and a development version of an arm-softfloat-linux-uclibc, I decided I could .

So I went on and downloaded the latest stage3 tarball for x86 (so I can have a gentoo system to start with), made all the necessary things and created a gentoo chroot. I emerge sync-ed, updated portage, installed crossdev (following the nice instructions on the gentoo embedded site about the subject). This took a while, but it went like a breeze. I made the necessary arangements to be able to cross build packages (setting SYSROOT, making that nice xmerge script).

As my goal was to make an arm-softfloat-linux-uclibc binary for rrdtool I tried directly to xmerge rrdtool (of course, first just pretended -p).

Finding bugs in the packages

First bug

All went fine until I found freetype which for some weird reason detected the i486-pc-linux-gnu-gcc compiler as i486-pc-linux-gnu (observe the missing trailing -gcc). I looked over the logs and determined that the CC_BUILD environment variable was not set. So exported CC_BUILD=i486-pc-linux-gnu-gcc and xmerged the library.

(While trying to fix this issue I entered #gentoo-embedded on freenode and actually met Yuri Vasilev, whom I had met in Edinburgh, at DebConf. Look for the video about Ligusk if you are interested about his project to make a Debian like distribution based on Gentoo. It was nice to meet a friendly "face" - as much you can say that online.)


I reported the bug (already there is a patch, I haven't tested, but looks ok) and the workaround in the gentoo bugzilla... after creating an account, since I never used that BTS before.

Second bug

Next pain was libart_lgpl whose upstream, for some comfortability reason chose to detect the target type sizes (and basically ignoring inttypes.h) with a small program which was built with the cross compiler which later was supposed to be ran on the host. I don't think is necessary to say that i failed, of course, since the binary was an ARM binary, while the host was an x86 machine. Yupee! not.

Ok, two bugs, no softfloat rrdtool yet. I reported this second bug and tried to work around the issue. Since I didn't really used too much a gentoo system, I wanted to know details and what possibilities I had. I got some really nice tips from "solar" (aka Ned Ludd) on #gentoo-embedded. He pointed out that while building a package one can press CTRL+z, do some stuff, like altering the Makefile.in file in the build directory and then fg back. Ok, cool.

All I need now is somebody with direct access to an arm machine that can compile and run for me the program in question. seanius on #debian-devel was kind enough to do this for me and, to my surprize, I got the same output header file as I got in the host x86 chroot. But that detail is not important, let's just compile that rrdtool. Well, first libart_lgpl...

Ok, I CTRL+Z-ed the build process just after the configure stage ended, cd-ed into the build directory, I tampered with the Makefile.in file and changed that line from

./gen_art_config > gen_art_config.h

into

cp /gen_art_config.h.arm gen_art_config.h

Haha! In your face! Ok, nasty did was done so I fg-ed and went on confident that I will get my hands on that rrdtool very soon. The package build ended succesfully, without any other suprizes, so I went for the graal!


xmerging rrdtool, the bugs don't stop

xmerge -v rrdtool

Bla, bla, bla.... bla, bla, bla, what do you know, there are a few IEEE math function tests in the configuration script of rrdtool. They failed. I don't know why, but I have a feeling that this should be happening on softfloat or even maybe the rrdtool source is not that cross compile aware. I didn't care that much, so I did what I knew best, tampering with an ongoing build.

This time is was a little bit harder to catch since I had to wait for the source to unpack (well, there is a larger frame, but that is the rough timing) and at that moment CTRL+Z the build.

I didn't had much time to waste, so after looking shortly over the tests I realized I should just ignore the error and look for the "failing command" that was interrupting my beloved build. Muhahah!

Thank God for verbose error messages, I could find the place fast and chaged the two "exit 1" statements into two harmless 'echo "Ignoring IEEE math errors"', sic. I had to do this a few times since I got the timping wrong, at some point I got the syntax wrong (at which point I remembered the cp trick I did earlier and saved the modified file in /configure.rrdtool )...

xmerging rrdtool and the 4th bug

After passing past this stage, I got a compilation error which, judging from the command, meant that the linker tried to link the target (arm) binary against the host (x86) libraries.

Oooook, this is not going to stop me! I look at the Makefile.in to try understand why this happened and then realized it wasn't worth it for me to try to fix it the right way. So I cheated once again! Muhaha, I just ran some useful commands which I'll let you feast your eyes to:

ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libart_lgpl_2.so /usr/lib/
ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libpng12.a /usr/lib/
ln -sf /usr/arm-softfloat-linux-uclibc/usr/lib/libfreetype.so /usr/lib/

Ok, good to go. Of course, I had to stop again the build to copy the "fixed" configure script that ignored IEEE math errors, but that is already history.

And, believe it or not, actually the build succeded. I had a cross built rrdtool using shared libraries built against softfloat ucllibc. No guarantees about its correctness, but it built!

This is when I decided to start writing this article.

Still there are things to do

I will have to do a few things before being happy about the victory:

• check that all the resulted binaries are indeed ARM ELF files, and not Intel x86. It appears the shared libs are already.
• done, they are (checked the next day)
• figure out qpkg (is a utility from portage-utils which seems to be a way to make binary packages) to package somehow the result of this work or maybe just ignore it and use plain cp
• see how I can fit together both the softfloat uclibc binaries and the native debian binaries on the same filesystem, including ld.so configuration...
• but I fear this will be a nightmare, so I might decide to go with static compilation of rrdtool, which I don't think it will be quite straight forward given the experiences describe above; there could be also the size issue to take into account, and since the NSLU2 has only 32MB of RAM, this could be problematic
• I think I'll have to use a statically linked binary after all (note added late, the second day)
  
• see if the rrdtool binary really works and if all the work payed off, performance wise - it should, taking into account what I have heard from wookey during the debconf7 talk about the armel (warning, 73MB ogg file) port
• be happy with the solution I got or wait for the Debian armel port to catch up and switch to that asap
• if I'll have to do the switch I'll probably think of ways of migrating the whole system from arm to armel, and not to reinstall everything; this might prove useful for the whole debian armel port, when people are working on migration plans, since arm will be deprecated in favour of armel

It’s a week since Soraya and me were are in Brussels working for JADE - European Confederation of Junior Enterprises, I hope to blog more about that in the next days. This is a nice city, see wikitravel or the wikipedia but I hope to discover more about it with the help of maemo-mapper, that’s why I was looking forward geoinfo about Brussels and I found belgeoblog, where you can download nice POI of Belgium. I wish we could have MetrO public transports database integrated in OSM. By the way, is there any local geek group here in Brussels ? or any Debian fan who would like to share a great belgian beer ? :-)

Not so long time ago i used to be a hardcore photography taker. I would carry a camera everywhere i went, and took hundreds of pictures. I even had a darkroom where to develop my b&w films. But digital came, and it was easier to shoot, check, re-shoot, erase,… you know, all the “benefits” you get with a digital: you care less what you are shooting because it is inexpensive to erase what went not-so-good. It all started again getting into shape last month, before our trip to Spain and France, when i gave my girlfriend my Polaroid camera. She was amazed and bought some film for it. In Barcelona we found a lomography shop, where i purchased a fisheye2 camera. And it finally happened 2 weeks ago when i revisited a flickr.com page of a guy who lives in NY and takes pictures daily. I was amazed, and following some links i ended up learning about Lomography.com and the Lomo LC-A’s. It was monday last week when i purchased a sealed, unused Lomo LC-A for 80e and few hours later i had my first film in several years in the shop for development. Now it is 2 weeks later (almost) and i have purchased few other items: a wide angle for my serviced old Pentax ME-1, few 120 rolls for my Yashica D, some more 35mm rolls for it (i will mod the case with some foam to insert 35mm rolls and take full 3.8×6 takes), an Olympus XA, a Minox 35ML and a couple of old FEDs. All together for less than 350e. My flicker page already reflects the activity i have been enjoying here and there. I admit it needs some cleanup, some pro upgrade and some descriptions added, but with time i will get there.

(no, not the animals, but web crawlers!)For a pet project of mine, I've recently been spidering the web a bit myself. So far, I've processed over 100.000 websites. The machine doing the spidering is an old K6-450, so it's not particularly fast...My spider is downloading the web pages HTML, and eventually some framesets (but at most 1 level deep). It's using text contents, image 'alt' attributes, title and some meta tags. The text contents are tokenized and stemmed.This results in some fun numbers:

• The average web page uses about 194 different words.
• The average token (after stemming!) is 6.8 characters long

Each of the web pages I'm spidering has about 6.4 categories assigned to it. I'll be using this training set to train an AI to classify web sites.(I've also started a web page for the project, but it's still pretty much empty so far, not worth looking.)

Admittedly I’ve not been working much on Webconverger in the past couple of weeks. Sorry about that. However yesterday I started plugging away at a couple of things and this morning I have had a couple of business enquiries! Goes to show you’ve got to show some activity in the project otherwise people won’t contribute. I’ve been dragging my heels the last weeks as I am starting to realise a Webconverger image for each locale is too hard on my computing resources. So I am working on getting all the iceweasel-l10n-*, myspell-* etc. things in one CD. And dropping http://download.webconverger.com/i18n. Users will have to enter in locale definition in manually, else please purchase a customized CD. Ok, I’ve just released 2.26 which is now double the size with all the locales & dictionaries I could find crammed in there. Also I re-enabled printing. Download the Webconverger Web kiosk and tell me what you think. There is a Debian installer mode (for hard drive installs) I am experimenting with, so Webconverger will be less of a LiveCD/USB, and more of a “distribution”.

In answer to zobel's post, here is how to fight efficiently against those nasty spams. Well, there is a wonderful tool, called clamav that you know already for sure. What is less known is that there are people that have had the idea to use clamav to fight spams as well. They provide constantly renewed spam signatures that fight against the jpgs/gifs/... that are too many those days. I use this script twice a day to update my signatures, and it works well. I use this setup on a medium sized mail server with excellent results, here are the numbers for the last 30 days. The mail server had:

 2.357.038 connections attempts
 1.841.425 mails have been greylisted[]
 ---
   510.193 mails have been rejected
   238.869 of those thanks to clamav (~50%) 
 ---
   502.580 mails have been accepted for delivery
 1.564.130 mails have been delivered to users

As you can see, on 4 mails that are considered for delivery (after the greylisting), 1 is rejected thanks to clamav. That's 25% of the incoming mails that get simply dropped, and that has almost 0 false positives^[2]. Another note about greylisting: a quick reader could think that 1.8M - (2.3M - 0.5M - 0.5M) ~= 0.5M of mails are greylisted for nothing. That's not the case at all, the 2.3M are connection attempts. And we have some SMTPs that we talk to a lot (as it's the mail server of the Alumni of my school, we talk to the school MX a lot e.g.) and some of the connections carry up to dozens of mail on a regular basis. Our estimation is that in a regular day, greylisted mails that are submitted again are around the thousands, meaning some dozens of thousands a month, which is ridiculously small. And among them, sadly, most are still spams. These good ratios exists because we use conditional greylisting: we greylist IPs that look suspcicious only. But I already talked about that, and it's not really the matter of this post.

The DDs reading this might know the situation: You are subscribed to a gazillion of mailing lists, and spend quite some time answering questions of people using your packages. That’s fine, service to your users. Occasionally, users take great pains in finding out a personal mail address (for example, by googling, and finding the webmasteridiot mail address on my personal web page) to ask their question in private e-mail. This prevents the answers from showing up in mail archives and deprives the public of a possibility to find a solution to this question themselves in the future. Today, I had this happen to me twice. In the morning, I received a question about exim4 in private e-mail. In German. I investigated the user’s problem (which looked interesting enough), found two minor issues in the Debian exim4 packages and upstream exim behaving not as expected. I fixed the two issues in the package and reported the unexpected behavior of exim to the upstream mailing list. I then reported this - in German - back to the person who asked the question. Time from incoming message to outgoing report was like 80 minutes, most of which I spent with this user’s issue. On my personal time. A short while afterwards, I found out that this person has waited a mere eleven minutes before asking the same question on pkg-exim4-users, this time in English. Had I known before, I would have drafted my answer in English as well, to have the answer publicly visible. By doing this double post, the user has deprived the public of my answer. I’m probably going to stop answering questions asked in private mail. I might draft a boilerplate answer along the lines of “I don’t answer questions asked in private mail unless I am being paid to do so. You might be better off by asking one of the numerous mailing lists that are available on the Internet. I read a lot of them, might see your question, and you might get an answer from me. But otoh, there are many more knowledgeable people on these lists, and you’d want them to see your question as well.” Or I just might hit Delete on questions in my private inbox. The “Delete” approach has become more attractive this afternoon after I pulled the following (anonymized) from my personal Inbox:

Great job breaking exim4 on my system with no easy way to fix it.
$ /etc/init.d/exim4 start
Starting MTA:DEBCONFsomethingDEBCONF found in exim configuration. This
is most probably
caused by you upgrading to exim4 4.67-3 or later without accepting the
suggested conffile changes. Please read
/usr/share/doc/exim4-config/NEWS.Debian.gz for 4.67-2 and 4.67-4
2007-07-17 11:31:57 Exim configuration error in line 25 of
/var/lib/exim4/config.autogenerated.tmp:
  malformed macro definition
Invalid new configfile /var/lib/exim4/config.autogenerated.tmp, not
installing
/var/lib/exim4/config.autogenerated.tmp to
/var/lib/exim4/config.autogenerated
I’ve been running exim4 v4.63 for awhile now and only have you bugger
exim4-config it doesn’t work.

I mean, this user has

• Not read NEWS.Debian.gz before updating
• Declined the configuration file updates that were offered by dpkg during the update.
• Received a clear-text error message explaining the issue and pointing to the docs.
• Vulgarly insulted the maintainer

Sorry, but: How stupid can people be? I think I’m going to stop being responsive to users. They’re not worth the blood pressure.

We went camping this weekend near Nominingue, northeast of Mont Tremblant. It was fun -- we met our friends Jeff and Anne-Marie, who'd biked there from Tremblant during the day. Maj and Amita June and I brought tons of food and big tents and stuff in the car. Our drive up was nice; Amita kept saying, "Camping!" every few minutes to emphasize how excited she was about going camping. She's getting much better about longer road trips. She entertains herself, reading books or telling herself stories or looking out the window and thinking out loud. Occasionally we sing songs or color together. The night was rainy, though, and Amita June slept in a dry but loud tent all night. At about 6AM she woke up angry. "Purple house, NOW!", she said, talking about our house in Montreal, with purple trim. "Shhh, Amita!" Maj said. "We're sleeping in the sleeping bag. We're camping!" "NO CAMPING!" said Amita. So the two of them got up and walked down to Lac Nominingue, while I got some rest. Later that morning, all the people in the campgrounds around us were smiling. "No camping! Camping no!" became a pretty popular refrain. Ah, well. At least they had a sense of humour about it. tags: amitajune laurentides camping travel maj jeff annemarie

Inefficient days Long day today. Brian got back from Halifax (Nova Scotia) on Friday, so we once again have a full house. Which is fun, but it makes it hard to work at home. I went out to Caff ArtJava near our house early this morning, but their le Sans Fil? connection was down. So I came home to pick up my bike to ride to La ka, but the back tire had a hole in it. And I didn't have a pump. Gar. So I spent a couple of hours working out that situation, and by that time I needed to go pick up our moving van. Maj found a great teak "wall unit" on Montreal Craigslist, and we need the shelves for our living room. So I got a van to pick up the pieces -- it was far too big to fit in our car. And, since my friend Niko wanted to go to IKEA, I traded him an afternoon with the van for help with my wall unit. He and Maj went off to IKEA as planned, and me at Amita went to the wading pool at Parc La Fontaine. It was a great time, although it's a long walk. I've had a hard time getting Amita onto the back of my bike -- she got pinched once when we were putting on her helmet, and since then she's wanted nothing to do with helmets, bikes, or nothing. If I force her helmet on and stick her in the seat, she loves it, but it's hard to trick her into it. She screams bloody murder when I try to put the helmet on her. But the pool was fun -- we chased each other up and down through the water. She fell asleep during the walk home, and I managed to get a little work done this afternoon. But Maj and Niko got back around 5:30, and Niko had a comedy show at 7PM and couldn't help me out -- the welcher! So I got Brian to come instead. The furniture was heavy -- really heavy. Teak is a heavy dang wood. Getting it out of the guy's house was really hard, and getting it up to our second-floor living room was Sisyphean. I'm amazed we didn't break any limbs. But at the end of it all, we had a great-looking new living room, and Maj had made a delicious gazpacho with polenta on the side. Yum! Hopefully the rest of the week is going to be more productive that today. Niko and I launch a new site at next week's DemoCampMontreal3 (as he so lugubriously pointed out in this blog post). And another group of partners will be launching kei.ki during Wikimania 2007, which is coming up soon. On top of that, we leave on Thursday for my brother's "commitment ceremony" in San Francisco, announced about 2 weeks in advance. We were lucky we had lots of miles so we could make the trip, but it still really cuts down on our Montreal summer experience. tags: maj furniture craigslist amitajune brian niko teak

(Or the unfortunate story of why I am not yet participating in the Debian games or perl teams.) I'm a big fan of both the Debian games and perl teams. The games team keeps finding new and interesting games and flowing them into Debian. The perl team is responsible for a rather amazing quantity of perl modules. Since I maintain both some games and some perl modules, I've looked at participating in both teams from time to time. I think it's odd that the reason I haven't been able to do so in both cases comes down to something as basic as how they use their subversion repositories.

---

In the case of the perl team, they have a fairly standard use of subversion, as explained in their guide. The problem is repository layout. They chose to use the "package/ trunk,tags,branches " layout rather than the " trunk,tags,branches /package" layout. This means that it's not easy to check out the source for every package in the repo all at once. Having all the sources checked out is a prerequisite for many kinds of work that span all packages. In my work on the d-i packages, which are in a repository using the latter structure, I often grep all the packages for a problem, and fix the problem in every package. To manage this with the debian-perl repo, I'd either have to check out the whole thing, with who knows how many gigabytes of tags, or I'd have to construct and maintain an svn:externals list that checked out the trunk of every package. (Open offer to the perl team: I'd be happy to reorganise your repo for you.)

---

The games team uses the " trunk,tags,branches /package" repo layout, which is nice. But they have a policy of not keeping all upstream files in subversion at all. They prefer to only keep files that are patched in subversion, to save space. That looks on the surface to be a nice space optimisation, especially for space-hungry game data, but from what I tell, what it saves in space, it loses in complexity and maintainer time. And my time is much more valuable than alioth's disk space. It's also very debatable how much space this saves at all for games. I keep uqm and all its data files in one subversion repo. I have 4 years of history of all of upstream's changes to the files in there, and it takes 165 mb of data. Which sounds like a lot, but the files in trunk alone take 170 mb to check out (not counting .svn directories)! And the games team is also storing upstream tarballs in a directory on alioth. One tarball per release, often multiple releases per game, 2.0 gb of tarballs in all. Would it really use much more space in svn? Even if it used 10x the space somehow, is 20 gb even worth worrying about? I think not. (Open offer to the games team: I'd happily do the grunt work of checking in upstream sources for all the games in your repo.) (Update: I realized after posting this that while the space gains on alioth might be illusory, there is certianly a space gain for people checking the whole repo out. What's really needed is a way to keep all the upstream sources in svn, but check them out only when wanted, and check out all the debian directories for cross-game work. Hmm..)

---

I also checked out one other big package repo, collab-maint. The policy for collab-maint's repo uses the same structure as the perl team's repo. However, I doubt I'd ever want to check out all of collab-maint or do group maintenance across it, so that's not a big deal. The documentation also recommends not storing upstream sources in svn, although I've found a lot of packages in the repo that do. I could probably dump in ~100 mb of subversion history for 50-odd packages without anyone caring ... right?

---

It's interesting that all three of these big subversion repos have their own custom policies for how to use their repo. Having different policies for each repo results in more complexity and a higher learning curve. I wager that 99% of all the packages in these repos are pretty stock in their packaging and can be pretty easily understood and worked with by most Debian maintainers. We've have de-facto standards and policies that they all use. Until you get down to the repos, which, despite using subversion each seem to need to go their own way for some reason.

I just realized that tonight is the last minute to place your bet about the day the 500000th Debian bug will be reported. And I didn't even bet myself yet!