Dan has been complaining I haven't been ranting enough lately. I have been waiting for the Scottish elections to sort themselves out, because thats a nice mess and easily due some ranting. However, given the need for a placeholder, I wrote this little moan on the subject of internet shopping. Everyone knows that the future of shopping is online. Every year, more money is spent online. Christmas last year was marked by getting a few parcels a day from amazon. However, I find it fraught with all things of annoying problems. Over the last few days, I have attempted to buy a t-shirt and a bottle of whisky online, while buying a pair of sandals and a book offline. While the wo aren't the same, it was still instructive as to the problems of internet shopping. Taking money from shoppers. To actually take money from people has to be the most important part of running any sort of shop. However, it is someone hard for me, the shopper, to give websites my money. Consider the example of Geekz Shop. To pay them, it appears I need to have a paypal account[0]. On examining the paypal signup page, I see references to the FSA. While its good that Paypal are actually registered and regulated these days, it does strongly suggest that they are a bank, and as such opening a account with them is the same as opening a bank account with HSBC or someone. In other words, I need to open a new bank account just to buy a t-shirt. Thats a lot of hoops to jump though. In their defense, they did offer a collection of other ways to pay when emailed about it, but its still a lot more work than walking into a shop and giving them cash. In more general terms, as a shop, its important to make it as easy as possible for your customers to give you money. Its common to outsource this to someone else, but any general solution serving lots of sites will suck[2]. Of course, doing it yourself means you are stuck dealing with credit cards - which is a pain, or BACS which is a joke. Oh well. The next issue you have is giving people their goods. In a real shop, you hand them their goods and they carry it away (massive generalization). Unless you are selling somthing completely electronic, for example domains, then you have to get the goods to the person. Once again, running your own service is hard - far harder than your own online payment system. However, it and your website is the thing people see of your shop. The classic problem is late delivery. You have to make some sort of claim as to when goods will arrive, but getting it wrong will result in arguments. My best example of this is a christmas present bought off amazon last year - it arrived in January, despite amazon's claim of having it to me before christmas day. The other classic example of getting it wrong is the apple store. It doesn't quote a delivery time - instead it quotes a "ready to ship" time. This is the time needed for apple to put your purchase in a box and stick a label on it. The problem with this is that your poor customer thinks that his new mac is going to show up in a couple of days. It doesn't, so he is massively disapointed. You've disapointed the guy before your product has left your warehouse. Thats one step aay from employing a bouncer to beat your customers when they leave your store. Having thought that delivery problems are an issue, consider people returning stuff. If its hard, they just moan at their credit card company. If your support and sales folk are in a call centre and can't be reached, then they spend the next week explaining to their mates about how dire you are and why they would sooner deal with the devil than you. Oh, and neither email nor rt is a way to deal with this. The final problem with most online shops is usability. Usability is hard - full stop. Its harder to sell something when the customer can't pick it up and feel it. While some shops are quite good, the number that force me to think when filling out the checkout forms or looking for products is amazing. Its like they don't want my custom. From this, I have claimed that a good online store needs a good way to take money off customers, a good way to get goods to customers, a sane way to deal with problems and a usable website (Next week, I'm going to blog about how water is wet). However, lots of sites, including big ones, fail at this. I don't think internet shopping is going away, but I do think that the shops that fail to take money off folk, that have sites that can't be navigated, that give goods to couriers who "lose" them, will go out of business. I hope that most shops steadily improve on these areas, just so I spend less time being pissed off at them BTW: this post was inspired by my mother's attempts to buy photos of my graduation online (no - it hasn't happened yet). The site in question was apparently so confusing that she has given up and has asked me to buy them. Sorry for ranting.

• [0]: Yes, I know it says you don't need one. Try making a purchase. I did mail them about this problem.
• [2]: Show me a payement site that doesn't suck and I'll show you one that you don't use.

Open Tuesday is an attempt to build a regular meeting place for people interested in free software (open source), both technical and business people. The goal is to have an event the first Tuesday every month, and some company or another sponsors it by picking up the bar tab. It's fun. You meet people, and can chat about various things. Unfortunately, it isn't quite regular enough, and the website and mailing list aren't updated reliably enough, or early enough, that an event is happening. It is now Sunday evening, and I still don't know there's an event the day after tomorrow. There doesn't seem to have been one last Tuesday, which is sensible, since the first of May is a big hangover day in Finland; it's also an official holiday. Since it's fun to meet people like that, I am going to see if I can't get a "Free Thursday" event going. This is partly in competition with Open Tuesday, but a friendly kind of competition. I hope Open Tuesday will continue, since it's a different kind of event from Free Thursday. Free Thursday is simply people who want to have a drink with each other. No sponsors, no organization, no prepared talks, no web forums, no web site, just show up and have a good time. You'll have to pay for your own drinks. Free Thursday will happen the second Thursday each month at 18:00. The first event will be Thusday, May 10, at the Teerenpeli bar in Kamppi, Helsinki. In case people want to discuss or announce anything related to Free Thursday, I've set up a mailing list. E-mail free-thursday-subscribe@liw.iki.fi to subscribe, and check the archives if you're not a subscriber. Posting is free to subscribers, everyone else gets moderated; this is for preventing spam, all non-spam will be allowed through. Please come, and please spread the word. Now, if someone will start a "Net Wednesday" event, all will be perfect.

Learn to bake!



Chicks dig it! On popular demand: The (German) recipe. Special hints from the baker: don't make the milk too hot, that'll kill the yeast. Also put the yeast-milk-flour mixture in a sufficiently large dish to avoid extra cleaning work. (A regular coffee cup is too small!)

Originally posted on 2006-04-16. Clarification 2006-04-19: The misfeature discussed below applies to feeds only, as far as I know, not to regular HTML serving. Update 2007-05-06: I have just learned that the option in question is Activate strict RFC2616 RSS-Feed compliance. I have no idea when it was added. While investigating (wearing my hat as the Planet Haskell editor/technician) John Goerzen’s Planet floods, I stumbled on a very strange misfeature in Serendipity. A HTTP/1.1 client can ask a server to send a file over only if the file has been modified since some date. What Serendipity does is send only those entries dated after the provided timestamp. Now, this might seem like a triviality, but it isn’t. The HTTP/1.1 feature is there to make proxy caching work. It is very important for caching that the file sent is the same, regardless whether the request is “since this timestamp”, or a regular request. Serendipity’s behaviour breaks this very important caching invariant! What’s more is that Serendipity’s developers seem rather proud of this misfeature. While the feature this misuse of HTTP/1.1 is intended to create is certainly useful, it should be implemented as a separate mechanism, and not by misusing HTTP/1.1 caching! (A separate protocol is probably necessary, perhaps specifying a URI query syntax for this is enough; obviously, one needs to get the aggregators to support this.) Serendipity will apparently have a configuration option to fix this, but it will be off by default. Serendipity users, upgrade (as soon as it is possible) and mark that option On, to remain compatible with the rest of the Internet.

Today is my one-year blogging anniversary. I started this blog on Journal/15 Flor al CCXIV, kind of trepidatiously and with ambiguous intentions. Now, a year later, I feel much more confident in the format and clear in my purpose. I've varied between an almost maniacally consistent one-post-per-day regimen to a more impressionistic, when-I-feel-like-it once-a-week schedule. I'll admit that I'm happier with the more frequent update schedule; I feel like I deliver more to the reader that way. I think the main thing I've enjoyed from having this blog has been the personal satisfaction in frequent and informal writings on my own schedule and to my own whim. Working with wikis makes a lot of my life about careful collaboration and consensus gathering, and having a place where I can speak out with my own voice on my own thoughts is really refreshing... and addictive. Most of all, I've enjoyed how having this blog has connected me with new people, and with friends I already had, in new and different ways. This year of blogging has been contemporaneous with a deepening of my connection with the Montreal technology community, and I can't help but think there's some causation involved. I think that my blogging is bringing me into a common medium and a common language with a group of people I want to spend time with. I've had some side effects I didn't anticipate, either. My family has started reading this blog, which means they know how I'm doing and what I'm up to more often than when I do once-a-week phone calls. And my more extended family -- aunts, uncles, cousins -- also read regularly. I can only wonder what they think about my complaints over various, say, Scheme module formats, but they seem to like it, too. Even Maj's mother reads it (hi Rita!). I think that m-c's 10 reasons to blog really hold true for me; it's good reading with a year's perspective behind me. I'm looking forward to the next year and what it has in store. tags: blogging me

Great day It was actually a great day for a blogiversary. Last night, Maj went out with her friend Sam for Sam's birthday, so this morning Amita June had a very quiet breakfast and then went out to Best Buy to do some shopping. I just bought a Canon MP600 printer, and toner cartridges weren't included in the package, so I had to go buy more. (Later, while configuring the printer, I found the toner packages in the bottom of the box. D'oh!) Amita fell asleep on the way home, so I just dropped her in bed. Maj and I walked through a brisk but sunny Montreal weekend afternoon down to the me dia offices, where Creacamp was already in full swing. There were lots of old friends and new ones, and the offices were light and airy and nicely connected. There was a talk by Guylaine Couture about her painted books, and by Luce Beaulieu about her awesome codesigner business, POSCH. Both really interesting. Of course my favourite talk was Anne-Marie's, about her great video series Music Spins the World (or La musique spin le monde). It's an incredible first-person series about her travel adventures with various rockstars to obscure and dangerous parts of the globe. A-M is becoming quite the rock diva, but I'm glad she's staying down-to-earth about it. You can check her work on 33mag.tv. Tonight I had a great time, too. Alex Roberts and Jon Phillips of Creative Commons are in town for the awesome Libre Graphics Meeting being held at the cole Polytechnique this weekend. The event seems great -- bringing together developers from all the great Open Source/Open Content graphics projects together in one room to talk. Jon and Alex met me and Niko at Concordia to see the Montreal premiere of the Helvetica movie. Helvetica is a great documentary about the history of this ubiquitous, unassuming font that epitomizes the modernist view of typography and design. The movie gives Helvetica a personality all its own, and in the process explicates the history of graphic design over the last fifty year. It also introduces us to a rogue's gallery of design and typography legends, all of whom are amusingly and passionately opinionated about design in general and Helvetica in particular. It has that characteristic of great documentaries: it illuminates the camera obscura of global culture through the pinhole of its narrow subject, a single font. Incredible. After the movie the four of us went to La Banquise, the landmark 24-hour poutine parlour right near my house. Alex and Jon indulged deeply in the local specialty -- they each got poutines with lots of bacon and ground beef -- and we all chatted about San Francisco, Web culture, Creative Commons and the cast of characters that surround it. A great, fun time. tags: montreal creacamp helvetica banquise jon phillips alex roberts

The URL http://linuxresource.lemonup.com/ currently has a mirror of my blog. Disregarding the DMCA take-down notice I sent them a week ago (which is also mirrored on their own site) they have again copied the content from my site without permission (I only allow non-commercial use). But this time they go even further and claim copyright over my text! This is going way too far. Now I’m going to ask their ISP to deal with them. Update: Their site is now offline. Their ISP acted quite quickly and less than 3.5 hours after my complaint the entire site was offline (not only the section that had my posts). I suspect that it was the fact that they mirrored blog posts such as this one which made it appear to be willfull infringement which got such a fast response - but the only response I got from the ISP was to say that they would do what seemed right and not comment to me about it due to privacy reasons. This is not an ideal outcome. I would much rather have had them respect my license terms without such measures. I only contacted their ISP because the first take-down request took four days to complete (after receiving a response on day 0 so it wasn’t four days of holiday for the operator) and because they then mirrored my site again under a different URL. I am still unsure of whether this was a genuine mistake (as claimed by the operator) due to lack of communication between multiple people involved in running the site, or whether they just didn’t think I would catch them. I don’t have any malice towards the operators of lemonup, I have already offered some suggestions that may help them in future business ventures and would be happy to make some more suggestions if asked. In response to a comment. The traditional meaning of the word pirate is violent acts at sea that don’t have state sponsorship, this usually involves armed robbery but the main criteria is violence without state sponsorship. The slang use means anything which goes against the wishes of a copyright holder.

I just committed to Evince a patch which makes the use of libgnome optional, meaning that Evince is now buildable on a pure GMAE system (obligatory screenshot). As GMAE has GConf and gnome-vfs, and gnome-keyring is trivial enough to leave in, the patch turned out to be quite simple:

• Don't use libgnome to start the help browser, but call Yelp directly
• Don't use libgnome to get an icon for a MIME type, instead copy ten lines of code from GTK+
• Disable use of GnomeProgram and session management

So the only regressions are that the keyboard accelerators are not saved, there is no session management (until session management lands in GTK+), and if gnome-vfs wants a password you don't get a password dialog. In my opinion only the last regression is actually a problem, and I want to fix that at some point. Thanks to the Evince maintainers for their rapid review! NP: Position Correction, TZU

That's a picture of Jacob in his "suit", with his soft shoes. Both came from Indiana.

He's been really growing, and is big. He's about 7 months old now, and is a lot of fun to play with.

Sunday was child dedication day at church, and Jacob was dedicated that day, wearing his suit. (Here is an explanation of Mennonite child dedication if you're curious)

We took him to the front of the church (along with another family who also had a newborn). We promised to nurture him as a child from God, said a few words, and then the pastor -- as he always does -- carried Jacob down the sanctuary aisle.

Jacob watched as the pastor introduced him to the congregation. He said "these will be your Sunday School teachers, mentors, and friends." He pointed out Jacob's grandparents. And he said, "These are the pews your Great Grandpa Alvin sat in. If he were here today, I know he'd be proud of you."

After it was over, we went back to our seats, breathed a sigh of relief that Jacob wasn't screaming during the whole thing. And promptly had to take him to the nursery because he started screaming right about then.

Greg Laughlin from UCO Lick pointed out that there is a transit possible of Gliese 581c on May7. Looking at the ephemeris, its at 17:13 UT, though: not very good for those of us in Ireland. Yesterdays Nature (subscription required) shows that someone is going one better:

On 26 April, just three days after the planet's existence was revealed, Dimitar Sasselov of Harvard University began using the small Canadian space telescope MOST (Microvariability and Oscillations of Stars) to observe Gliese 581. Sasselov predicts that, if the geometry is right, his team should see the star dim on 7 May as the planet passes in front of it.

They reckon on 50/1 against a transit, on geometrical grounds. Good luck to them though. The point of all of this is that the current planetary detections of Gliese 581b and 581c are Doppler radial velocity measurements. While they tell us the period, they only give us a lower bound on the mass : 581c is at least 5 Earth masses, but could be much more (its M Sin(i), where i is the inclination relative to us is unknown). We also don't know the radius of the planet : the figure quoted in the news articles to date is just a guess based on 5 Earth masses and current theory. A transit measurement, where the planet passes in front of the star, could answer these questions. But it occurs to be we could get numbers tonight: there is a transit (alignment) of Gliese 581b tonight at 22:40 UT. Much more favourable to those of us in Europe. A transit of GL 581b, the big neighbour of Gl 581c, would tell is that a transit of 581c is likely, and would give us an accurate mass for 581b, and from this, a measure of the inclination. So we could also calculate the mass of 581b. Unfortunately I don't have a photometer or CCD. Anyone else up for it? Clear Skies over Ireland forecast , too ... Tags astronomy, exoplanets

I’ll be making my first big appearance as a Sun employee next Monday at CommunityOne in San Francisco (this is the day before JavaOne). In addition to participating in the keynote with Tim O’Reilly, Rich Green, and Tim Bray, I’ve put together a track called Linux vs. Solaris? designed to show that the gap between Linux and Solaris isn’t as wide as one might think:

All too often, technologies are pitted against each other in the popular imagination, and Linux and Solaris are no exception—”Linux vs. Solaris” certainly does make a catchy soundbyte. Despite the juxtaposition, Linux and Solaris have much in common—both are open source, have common ancestry, and are similar enough that both users and developers can move back and forth between them with comparative ease. The “vs.” mentality is caused as much by lack of understanding of the “other side” as anything else. In this track, we will focus on the similarities between Linux and Solaris rather than the differences, the goal being to increase understanding of Solaris among Linux users and developers and vice versa. Where we discuss differences, these differences will be expressed in terms of “how we can learn from each other”. We will also discuss the migration up the stack of developer platforms and address the question, “Do operating systems still matter?” The ultimate goal of the track is to change the conversation: Not Linux vs. Solaris, but open vs. closed.

The track includes four sessions. I’ll kick things off by answering the burning question: “What’s a Linux guy doing at Sun?” Next, Jeff Bailey of Canonical and Bart Smaalders of Sun will take us “inside the sausage factory” to show how Ubuntu and Solaris, respectively, get made. Next, Don Kretsch of Sun and Joe Little of Stanford will show us Solaris Express Developer Edition and Nexenta, two very different takes on OpenSolaris. And last but certainly not least, Josh Berkus, Robert Lor, Harpreet Singh, Tim Bray, and Greg Luck will highlight why the OS still very much matters even as software development moves up the stack. Of course, there are a number of other great tracks as well. If operating systems are your bag, you should also check out the OpenSolaris track. Fortunately, my times don’t overlap with Ben Rockwood’s introduction to OpenSolaris, nor will I have to miss the opportunity to see the amazing Bryan Cantrill, Adam Leventhal, and Mike Shapiro educate, entertain, and gesticulate wildly! If you’re in San Francisco or the Bay area or are already planning to be at JavaOne, you should definitely plan to attend CommunityOne too. Register today! (P.S. - It’s free!)

A couple of hours of today went on dealing with the effects of a big spam attack on a managed server. One user had installed the Joomla ExtCal component on their web site, while another reported that email was running slowly. The attacker was exploiting com_extcalendar/admin_events.php to send spam and trying to send 9000 emails meant they hit the maximum permitted sending rate quite quickly. As soon as I spotted it, I shut down outgoing mail, killed the sending process, disabled the exploited site and component, cleaned the spam from the queue and restarted mail. I'm searching the rest of the server for any other com_extcalendar installations and then I should contact the other exploited servers I've seen during this attack. It looks like there's an entire class of evil "Mambots" out there, using exploitable Mambo and Joomla sites to crawl in search of others. I don't think the exploited site's owner will be billed for the clean-up, because I can't find a security advisory for this. Neither the mambo site (whose site search doesn't work for me) nor the joomla site seems clear about security alerts. I didn't find a third-party advisory that was clearly about this (but there were a few maybe-relevant ones). The homepage at Mambo last released in 2005 and doesn't mention this in its release notes, while the homepage listed by Joomla 404s (and I don't enjoy using the Joomla site because, as it says,

"we are aware, and regret, that our website does not comply with many WCAG/508 requirements"

). If you are using com_extcalendar on your site, check whether it's safe, or replace it with a better calendar.

Ok, so LPI will be at Debconf, giving discounted certifications to registered attendees. Is this good or bad? Mario likes the idea, Madduck is in the middle ground, not decided on his stance on this regard, and Joerg basically says it's not worth much to him personally. Actually, I'll quote Madduck, as he has an interesting point:

I am not looking for employment, and if I was, I'd certainly not want to work at a company that thinks certifications are the true proof of capabilities. So I guess that leaves me with a 'no' still.

When confronted with this topic, I always oppose certifications. Why? First of all, I think they are worth very little. I got three or four Brainbench certifications when they were free - And of course, noticed right away that such a simplistic test is worth very little. Of course, LPI is a better established name, and is usually respected - Lets be fair, and talk about LPI together in the line with Cisco's, Microsoft's, Novell's and similar certification programs.
I've worked with several people who have got certified in different technologies, and almost always, this works against them rather than in their favor - Such people usually are blinded to all but their technologies. My most recent experiences are with the network infrastructure people - Cisco people know how to push Cisco, but know very little about protocol details, and cannot recommend a tool that's not madesold by Cisco. Same goes for 3Com. Same goes for everybody else. Although many certification tests include general situations like Solve this real-world problem, they are hampered by the final exam syndrome: The certification candidate spent a couple of nights frantically reading the books, and the material sits eager to jump on his brain lobes. Of course, given a couple of weeks, he has forgotten most of it and confused the rest. No, I don't have hard data to back this up except for my experience - But I have some experience at least. Oh, and of course: This people can quote from memory in inverse alphabetical order each of the command-line options to ls, but might be unable to spit up a clever shell pipeline without sketching it in paper and thinking it over for some minutes.
What will this mean for most of Debconf's target audience? Well, just what Ganneff and Madduck said: Take the test if you want to get a new job more easily - but you should have more confidence in yourself.
Just as a final note: Whenever I've interviewed people to work with me or for people that trust me, from all of the received curricula, I start by throwing out every curriculum that has the certifications earned in a prominent place. People who give too much weight to certifications IMHO tends to be worthless to work with.

A few days ago I wrote my first DMCA take-down notice, I followed the instructions on the Wikipedia page. The reason for this was that someone was mirroring my blog and putting google adverts on the copy. Before I started putting Google adverts on my web sites I wouldn’t have been bothered about this. But now that I’m making a small amount of money from Google advertising I don’t want someone else just mirroring my content and taking the money away from me. The person who managed the site in question took a surprisingly large amount of time to comply with the request (a discussion of several messages plus a couple of reminders over the course of a few days). The most recent news about DMCA abuse is the case of trying to prevent the distribution of a code used for decrypting DVD-HD. It is widely believed that copyright was used to prevent the distribution. Strangely many people who otherwise have a good understanding of technology have been saying “you can’t copyright a number”. What precisely is a program binary if not a long series of numbers (or a single large number depending on how you look at it)? For that matter a JPEG file or the ASCII representation of a book is also either a very large number or a series of small numbers. Also apparently it’s not protected under copyright but under the anti-circumvention clause of the DMCA. If it was a matter of copyright it would not be an issue of whether a number can be copyrighted, but what defines such a number. One criteria for copyright is that it has to be on something non-trivial (EG I couldn’t copyright the use of “a few days ago” as an introduction) so length is a criteria. Another is that it has to be a creative expression (so an encryption key can’t be copyright). However in many jurisdictions there are separate laws regarding distributing passwords without permission, such laws are designed for preventing people from granting unauthorised access to computers but I believe that they can be used more generally (I have been advised that such laws exist in the state of Pennsylvania in the US - I’m not sure what the law is in other regions but expect that something so useful would be copied). Another breaking story is that the RIAA has created an organisation with a US government mandate to collect royalties on ALL music that is played over Internet radio. This includes music for which the copyright owner is not an RIAA member and does not consent to have the royalties applied. You can create your own music, grant free access to everyone out of philanthropy, and then have the RIAA tax the music! It’s unfortunate that only the down-side of this dramatic change in copyright law has been discussed. Compulsory licenses have a lot of potential in other areas of copyright material. Recently people have been complaining that government sponsored scientific research is often only published in journals that cost large amounts of money. Why not have a compulsory license for journals at a fair price that everyone can afford? Software is often unreasonably expensive (Windows Vista with the latest version of MS Office can cost up to twice as much as a new PC), let’s have compulsory licenses for software at a reasonable fee! Software vendors often cease selling old versions of software to force customers to upgrade, a compulsory license scheme would permit us to buy MS-DOS 3.30 at a reasonable price regardless of whether MS wants to sell it. Finally there is at least one evil cult that claims it’s “religious” texts are copyright as a way of preventing the public from seeing what a drug-addled second-rate sci-fi author produces. Let’s have a compulsory license for them so everyone can read them! The only thing that’s wrong with the RIAA scheme is that there is no option for copyright owners to directly license their material to the users (including granting a free license if they so desire). The up-side of this is that it proves beyond all doubt that the RIAA is not representing copyright owners. Update: I initially accepted the claims about the DMCA take-down notices being based on copyright rather than anti-circumvention. Since learning of my mistake I modified this post to reflect the fact that it was not a copyright issue.

When reading this number

09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0

I thought it was the fingerprint of a GPG key.The first time I came across it, it was encoded in a puzzle; not a particularly difficult one (what key is inbetween of F8 and F10 on your keyboard?).Turns out that this number can be used to decrypt most HD-DVDs produced so far.No, you can't really think you have a copyright on this number? You could have it as a registered trademark, I guess. But I think everybody will be using it to refer to you. :-)From a crypto point of view, it was just a matter of time until this number leaked out. And in fact, the HD-DVD encryption was designed to handle this cases (but it wasn't used correctly). Face it: cryptographers have been telling you right from the beginning that your encryption will be broken by it's very nature (remember: you are shipping decryption keys worldwide in your HD-DVD player devices).[Update: others thought it might be an IPv6 address maybe?][Update #2: Google thinks there are about 300k pages with this number. Maybe we can stop this meme now? :-)][Update #3: it was pointed out to me that nobody claimed to have copyright on this number, but that it is a device to cirumvent encryption and thus violates the DMCA. And that this is similar to distributing magic markers that could be used to 'crack' a famous CD protection used by Sony some time ago...]

Just submitted my third-year project. Was probably the longest document I will ever write:

$ wc -w  find ~/local/bzr/3yp/docs/report -name "*.tex"    tail -n1
19234 total
$

(If new readers are picking up the story late, in the final year of the Warwick Computer Science course, students are required to undertake a reasonably sized project. I chose to write a Java compiler in Python.) Actually writing the report was a bit more painful than I thought, dispite having collected and organised most of the data, references and interesting topics well in advance. I’ve nearly overcome some sort of writers-block-cum-nervousness issue though, which is good. I got a few interesting results: My backend system had a peephole optimiser to run on all code regardless of the input language and it seems like Sun’s javac seems to miss out on many really trivial optimisations, probably because the JIT takes care of any performance blackspots. Regardless, I plan to have a go at adding this into OpenJDK after my exams just as an excuse to hack on the Real Java. (I also did some register allocation, also beating javac, althought I didn’t really investigate way) I’ll have to wait for while due to University regulations anyway, but I’m not sure if the project code is really fit for public consumption yet, but I plan to rip out the silly Lisp implementation (and most of the Java one) leaving a rather fast and “enterprise-ready” Brainf*** compiler.

As already announced, we’ve arranged for interested people to take LPI exams during DebConf7. As the announcement says, you need to register for the exams in two places. First go to the LPI system to get an LPI-ID, and then register for this event, so the LPI people know how many attendees to expect, and for which of the 6 possible tests. (Ignore the MySQL ID, no MySQL certifications will be carried out). You will see that there are currently two timeslots to choose from on 19 June, one at 10:00 and one at 14:00. This should be enough for the number of people we expect to show up, but if not we will arrange more slots with the LPI team. As also stated in the announcement, there is a special pack available. If you registered for DebConf7 and also reconfirmed in time, you can take two LPI exams for only 95 EUR, which is less than one exam normally costs. If you haven’t managed to register and reconfirm in time, don’t worry: the LPI exam will still be available at half the normal price. Oh, and someone just asked: you can register for the LPI exams right up to the day they’re held at DebConf. Only the special pack is limited to those who registered and reconfirmed for DebConf in time…

With a few days lag, the slides to my VCFe 8.0 talk Aktuelle, freie Software auf alter Hardware (“Up to date, free software on old hardware”, held in German using Kazehakase and S5) are now online. In comparision to my former talks on that subject (held at some DebianDays), this talk was not Debian focused but focused more on not so well known, but resource-friendly free software as well as focused on an audience which has more knowledge of old hardware than of current software. :-) Additionally, I updated my old blog post about X on my ThinkPad 760ED named bijou so that now also my current XF86Config-4 for Sarge on that box is linked in there. Apropos bijou: I couldn’t recommend Debian 4.0 Etch that much for old computers with not so much memory since especially aptitude has grown much in regards of it’s memory and performance needs. Regarding my experiences with Etch, any computer with less than 50 MB of RAM will start to swap if aptitude is only started on such a box. I’ve looked throough the aptitude documentation, but I haven’t found a way to switch of some of the tables it generates internally. E.g. I have no need for the tag database it always generates. I really would be happy, if someone knows a way to turn even only that feature off. Then I may dist-upgrade bijou to Etch, since I found that dselect is no real alternative to aptitude anymore. Oh yeah, and I of course bought new old hardware at the VCFe: A 386SX Thin Client named Flytech Carry-I 9300 from 1991 with about 200 MB of harddisk and 10 MB of RAM.

Recovering a corrupted FAT partition A friend has seen his FAT partition in his USB key become unmountable. Here's how I recovered it. Analysis:

1. Make a dump: dd if=/dev/sdb1 of=partition.img
2. Have a look. I like mc for this, as it has a nice hex viewer and editor (F3 to view, then F4 for hex view; F2 would then allow hex edit, if needed).
3. Turns out that the initial part of the partition has been completely overwritten by 0xff bytes. After some point (around block 224), the 0xff bytes end and there seems to be a FAT structure (easy to identify because it contains sequences of increasing 16bit values).

It seems that the damage is only limited to filesystem structures. I should be able to recover most data by regenerating the filesystem data and hoping that at least the second FAT is still intact. Solution:

# Make a copy of the partition
cp partition.img partition1.img
# Reformat it: that gives us the bits of filesystem structures we need
mkdosfs partition1.img
# Graft together the two parts
dd if=partition1.img of=partition2.img count=224
dd if=partition.img of=partition2.img seek=224 skip=224
# Tidy up the result, telling dosfsck to use the 2nd FAT.
dosfsck -rw partition2.img
# Mount and check what's available
mount -o loop,ro partition2.img /mnt

Yo! Everything was there again. When I teach about standard Unix commands, it takes a while for people to realise all the powerful things you can easily do with them.

In a comment on my blog post “a Heartbeat developer comments on my blog post” Alan Robertson writes:

---

I got in a hurry on my math because of the emergency. So, there are even more assumptions (errors?) than I documented. In particular, the probability model I gave was for a particular node to fail. So the probability of either of two failing would be double that, and either of three failing would be triple that. Note that the probability of multiple simultaneous failures goes up as a power, but the probability of either of only goes up linearly. I really need to sit down and do the math carefully - but the idea of the simultaneous failures going up as a power is true. And the “any of” probability goes up linearly. That’s also true. This is why people can actually use larger HA clusters ;-). The 5 years figure is the industry standard quoted figure for an average Intel-based server to fail. The four hours to repair is a common high-quality of service response time from a hardware vendor. I admit that’s not the same as actual repair time, but if some “repairs” are just reboots, then it’s not a horrible number to start with - if your vendor has cached some spares nearby. I suppose I should sit down and do the math right, and make a spreadsheet of it. (I wonder if I remember that much math?) I assume disk failures are taken care of by hot swap disks, RAID, etc. and so in effect they “never fail” (at least not totally) so that these failures don’t have to be accounted for by the overall availability model. Here’s an intuitive way of thinking about it “from your gut”… If I took your whole data center and made a cluster out of it, what’s the chance that at least half of your servers would fail at once? Pretty darn small, is the short answer ;-). If it’s not pretty darn small, you need to buy better servers, and IBM has just the servers for you ;-). Or maybe they need to hire a better SysAdmin ;-) If you ask yourself “when is the last time at least half my machines in my data center couldn’t communicate with the other half”, then hopefully that’s also a “pretty darn small” chance too. If not, there are well-known methods for making networks highly reliable too. [I’m still ignoring “catastrophes” that you haven’t accounted for in your HA architecture]. I’m not saying this is free, and it can be pricey. One of my other favorite sayings is “Paranoia is an expensive hobby”. How much do you want to spend? You tell me how much you want to spend, and you can figure out how to spend it. I’ll make a separate comment on quorum models later. It’s getting late here.

---

My only comment in response to that is to say that I still believe the calculations of probability to be correct in my original post and I am interested to see someone prove otherwise. Another comment by Alan:

---

Data corruption, no doubt, is almost always much worse than loss of availability. And some kinds of data corruption are worse than others. For example, mounting a non-clustered shared disk filesystem twice simultaneously is usually much worse, than updating two replicas of the data simultaneously. In the first case, you have to restore to your previous backups and lose all data since then. In the second case, you only lose updates that were made to one of the sides, and you instantly have a working copy of the data which is nearly always much newer than your last backup (with the possibility of recovering them by significant effort). Typically you would only lose a few minutes of updates at worst - and depending on the kind of networking failure, you might not lose anything. Heartbeats certainly aren’t enough. You need to monitor the health of your servers and the health of your applications. Heartbeat monitors applications and can easily be informed of and act on the health of your servers (with release 2 style Linux-HA Heartbeat configurations).

---

Followed by:

---

Since data corruption is so serious, this is why cluster designers worry so much about split-brain, which is managed using the ideas of quorum and it’s sibling fencing. This is all about keeping bad things from happening. This post is really about quorum, since Russ had expressed interest in it. Quorum is the idea that you can uniquely choose a subcluster to represent the whole cluster in those cases where communication failure has caused the cluster to split into separate sub-clusters which cannot properly communicate with each other. In this way, only one of the subclusters continues on, and the others will sit on their hands and do nothing waiting for a person to fix things. Some of the kinds of quorum mentioned below are better than others. But, most importantly, they can be used in combination as described later. The most common kind of quorum is that Russ mentioned in his earlier post - the majority quorum. In this method, for a cluster of n nodes, you grant quorum to a sub-cluster which has more than INT(n/2) members. This means that if you have a 3-node cluster, you have to have two nodes to continue. If you have 4 nodes, you have to have 3 nodes to continue. For 5 nodes, you have to have 3 nodes, and so on. Other basic methods include disk reserve, so that you have reserve a disk to have quorum. In this case, if only one node survives and it can reserve the disk, it continues to run. However, the disk becomes a single point of failure. This may not be a problem if this single disk is required to run any of the cluster services, since they would fail without it anyway. [Heartbeat does not support this method]. An analagous method is to implement a software resource which grants quorum to one subcluster in a fashion analagous to the disk reserve method. This has the advantage of not requiring disk reserves, or a shared disk, but it has the same SPOF disadvantage as the disk reserve method. Heartbeat does support this method using the quorum daemon. It’s incredibly useful for those cases (like split-site clusters) where you cannot use fencing. Another method is to grant quorum to any subcluster which can ping a certain set of nodes, and not grant it to any which can’t access those nodes. This isn’t a wonderful method, and has obvious disadvantages with respect to uniqueness, and single points of failure. (Heartbeat doesn’t yet implement this one). Another method is to grant quorum to any node which is a member of a 2-node cluster. This is better than losing quorum and stopping when one node stops, but obviously completely ignores the uniqueness requirement of quorum. Another method is to ask a human being if you have quorum. This is hardly an ideal circumstance, but useful in some contexts as described below. (Heartbeat doesn’t yet implement this one). Perhaps you say, really the only one of these that’s really good is the first one - the majority vote method. And, I would generally agree with you. But, Heartbeat has the ability to use these in combination which makes some of those methods that seem flaky to be much more reasonable. Heartbeat has the ability to have multiple quorum modules declared, and they’re used in this way: Any module can return HAVEQUORUM, NOQUORUM, or TIE. If they return HAVEQUORUM or NOQUORUM, then no further quorum modules are consulted. However, if they return TIE, then the next quorum module is consulted for its opinion. If the last quorum module returns TIE, it is treated the same as NOQUORUM. This enables you to use one quorum module to break the tie declared by a previous quorum module. You could then use the quorumd to break the tie created by a voting module. Or you could use the quorumd instead of the “two-node” module. Or you could use the “pingable” module instead of the “two-node” module. Or you could at the end always tack on a “human” module, in case all else returns TIE. This is kind of cool, actually. My favorites for next implementation are the pingable and consult human modules. And, of course, if your cluster loses quorum due to real server failures failures, there are always ways to work around it, with a little human intervention. One method is to tell Heartbeat to ignore quorum. Another is to tell Heartbeat to remove certain nodes from the cluster, after you verify that they’re really dead. And, I’m sure that in a pinch, some new methods will be invented. And some of them might actually work ;-).

---

Regarding the quorumd, it seems that this is an extra server that will generally run on another machine separate from the rest of the cluster. So if we had a two-node cluster with a quorumd then it would effectively be a three-node cluster where one node is not configured to run any resources. It seems that the simpler approach in many cases would be to merely have a three-node cluster with resources not configured to run on one of the nodes. For example if I was running a mail server cluster for an ISP I might configure a three node cluster of the two mail server back-end machines and one other machine that is lightly loaded (EG a DNS server) and have it configured not to run MTA resoures on the DNS machine.

This morning I thought I couldn’t get my GPS tracklog from my bike ride through London off my Etrex GPS device as I didn’t have:

1. A serial port on my X40
2. Google plus (20USD for getting data off my GPS!) on my Windows desktop at work

As I left work, a colleague suggested I try an USB serial adaptor that was lying around. He also gave me a CD with the drivers. I said I don’t need drivers! I run Debian, much to my colleague’s bemusement. Anyhoo, the USB/Serial interface works well, it comes up like so:

Apr 30 22:56:53 monty kernel: usbcore: registered new driver usbserial
Apr 30 22:56:53 monty kernel: drivers/usb/serial/usb-serial.c: USB Serial support registered for generic
Apr 30 22:56:53 monty kernel: usbcore: registered new driver usbserial_generic
Apr 30 22:56:53 monty kernel: drivers/usb/serial/usb-serial.c: USB Serial Driver core
Apr 30 22:56:53 monty kernel: drivers/usb/serial/usb-serial.c: USB Serial support registered for pl2303
Apr 30 22:56:53 monty kernel: pl2303 2-1:1.0: pl2303 converter detected
Apr 30 22:56:53 monty kernel: usb 2-1: pl2303 converter now attached to ttyUSB0

Then I ran: gpsbabel -t -i garmin -f /dev/ttyUSB0 -o gpx -F tracklog.gpx To download the track log from the Garmin. Then I converted the GPX into KML with this Gpsvisualizer Web application. Then I load up the KML with Google Earth to see where and when I have been cycling London. Google Earth really needs the package libgl1-mesa-dri installed to get DRI working on my X40 machine. Though I prefer not to use Google Earth. I wish I could load the data with this Google maps animator, though the service doesn’t seem to accepting new accounts. Grrr… Update: Just a couple of comments about riding in London:

• There are more bike paths than I expected
• Traffic was real easy last Sunday
• Overtaking buses is scary, but it’s better than sitting behind them in the fumes and you do tend to go much faster on average than a bus