Search Results: "cas"

14 August 2006

Steve Kemp: Step inside, suprise!

As another reason why I dislike mailing list software consider the following error: 
2006-08-14 21:40:25 1GCk6m-0006ZH-Vu ==  /usr/lib/ecartis/ecartis -s xen-tools-commits
R=vdom_aliases T=address_pipe defer (0):
Child process of address_pipe transport returned 75 (could mean temporary error)
from command: /usr/lib/ecartis/ecartis
What does that mean? I see precisely one hit for this error in Polish. I resorted to debugging via strace: 
strace -ff -o /tmp/foo /var/lib/ecartis/ecartis -s xen-tools-commits
   < /tmp/msg.txt
The error? “Connection refused” when connecting to 127.0.0.1:25. D’oh! It should invoke the local mailer via /usr/ lib sbin /sendmail in the case of socket error. Ideally chosing the local execution in preference to the socket. In conclusion: I believe my mailing lists are fixed, however several messages have been dropped/ignored.

Thom May: MLP-tastic

Both of these come via Newshog; the first is a very interesting article from NBC talking about the ”unprecedented cooperation and coordination” between US and UK officials; unprecedented in this case appears to mean ”arguing all the way to the police station”, but this is just according to one unnamed UK official, so possibly a pinch of salt required. The article goes on to mention in passing the arrest of the ringleader, Rashid Rauf, in Pakistan, apparently causing another disagreement between the US and the UK over choice of jurisdiction - which some are linking to the possibility of torture. There are some fantastic quotes in the NBC article, regarding the timing and actual preparedness of the “attack”, especially that ”the suspects in Britain had obtained at least some of the materials for the explosive but had not yet actually prepared or mixed it.” and that ”the suspects had not yet purchased any airline tickets. In fact, some did not even have passports.”. All this added up begins to bring serious doubt as to the actual danger represented by the terrorists, and also raises the question of what a more “hands-off” - not arresting the suspects, but allowing them to continue their preparations longer - approach to monitoring and intelligence gathering would have resulted in. The second is interesting in its own right - Max Hastings - friend of Lady Thatcher, dyed in the wool Conservatist - writing for the Guardian - left/liberal and proud of the fact - leads his article with “George Bush sometimes sounds more like the Mahdi, preaching jihad against infidels, than the leader of a western democracy” (edited to link to Wikipedia’s conservative party page.)

13 August 2006

Lars Wirzenius: Random thought: Quick! Press 3, click mouse button, press A, press 4, keep mouse button pressed...

I am in a murderous mood. Every time I see movement, my right index finger twitches. My head keeps jerking left, right, up, down. My ankle hurts because I almost twisted it, turning suddenly around when I heard a noise from outside. Today, instead of doing a bit of work that I left undone on Friday, and going to the gym, and making a nice meal, I ate leftovers and played Quake 2. Finished it, in fact. I probably died a hundred times, but eventually I got the bastard. Now I'm frustrated. I don't actually feel anything from the first paragraph. My ankle got hurt because I was trying to catch a bottle that fell off the table. However, I do feel frustrated. As I told before, some time ago I started looking at modern computer games, or rather, modern first person shooters, in single-player mode. I had a friend demo me a representative selection, and then got myself an authentic copy of Quake 2. I've now finished it, and I'm not entirely happy about the experience. The actual playing was OK while it lasted. The game is highly playable, the controls are easy, they quickly feel natural, and the game reacts to them very well. The only thing I never really got the hang of was jumping, but that was rarely necessary. The place with the three stones sliding around in water was quite annoying. Playing Quake 2 consists mainly of two things: shooting everything that moves, and solving small puzzles. The puzzles are mostly of the kind where you need to run around in buildings and twisty little corridors, all alike, and find a button to press. Not highly challenging, but if you miss one, you may have to run around the place an extra time or two, and that takes time, and gets quite boring. The shooting was also a bit of a puzzle, at times, since in order to not get shot yourself, you can't just stand in one place. In the ideal case, you can figure out a way to deal with the opponents from a safe place, throwing grenades around a corner or something like that. Mostly, though, you just shoot. Aim and click. Aim and click. Move about a bit. Aim and click. Aim and click. Move about. Aim and click. Continue until the thing blows up. The computerized opponents do get smarter, and a bit harder to kill, as the game progresses. The final big bad guy is really hard to kill, took me a dozen attempts. That turned out to be surprisingly unsatisfying. There is a storyline in the game, but it was pretty much irrelevant. There are several stages, and at each stage you need to do certain tasks, and as a result, you can go to the next task, or the next stage. All the tasks are "shoot everything" and "press a few buttons in the right order". Oh, sometimes you have to collect some items, too, and take them to places. Now that I've finished the game, I wonder if it was all worth it. I spent quite a number of hours on the game, and what I mainly got out of it was some adrenaline boosts, and those weren't enjoyable. They're addictive, even highly so, and every time I had to stop playing (to do work, or sleep), it required a lot of willpower. I wanted to continue, even if it wasn't fun. I hate being addicted. I can't deny having had a bit of fun. The feeling of being able to do things, to run tirelessly for hours, to overcome obstacles, to vanquish my foe, all that is fun, and gives a certain satisfaction. That satisfaction is feeble and fleeting. As soon as I re-focus my eyes, get up from the couch, and look outside, I realize that it was all for nothing. I haven't really accomplished anything. I'm a hacker. My life revolves around doing things with my brain, and having them affect the real world. I make software that other people use, and their lives get better, even if only a little. Every time I fix a bug in Debian, I improve the world. Heck, even just finding a new bug from piuparts log files gives me a nice, warm feeling in my stomach. I realize people don't play games to improve the world, and that's all right. I don't watch movies or read fiction to improve the world, myself. Computer games are a form of entertainment, and as long as they're fun, and you're enjoying yourself, they're fine. If playing makes you happy, be happy, and I'll be happy for you. I am not judging you in any way, I'm expressing my own feelings. I can't seem to get enjoyment from computer games, or at least I didn't get it from Quake 2. I got an adrenaline addiction, and a high frustration level. I do get distraction, something to take my mind off things, but I can get that from, say, romantic comedies, without any side effects. Thus I conclude that I am better off not playing first person shooter games. I suspect that no computer game is going to be particularly good for me. For many years now, I've played a bit of tetris, or Mahjongg, or a solitaire card game, when I wanted a bit of distraction. Those never were satisfying, either. While I did get distracted, somewhat, for a while, they didn't really work for that, either. Watching Lord Arthur and Miss Mabel get each other works much better. If I solve solitaire, nothing has changed; if I watch An Ideal Husband the nth time, at least I've felt something beautiful. A surrogate emotion is better than an adrenaline addiction.

Edd Dumbill: Ten things you need to know about me

To keep you reading all the way through, the best one's at the end.
1. I'm striking back out on my own againAfter a couple of years working full time in an office, I've decided it's the right time to take my own business, the Useful Information Company, further on. As the rest of this post illustrates, we'll be involved in web technology conferences, our own products, and Rails-based software development. I'm always looking for collaborators, comment and customers, so get in touch if any of this lot grabs you.
2. I'm organising an amazing web developer event in London, late OctoberUseful's first venture is XTech WebDev London. This is the first of a new type of event for XTech: a focused one-day training event for web developers. The idea is to focus on the best practice in web development that emerges from the cutting-edge work demonstrated in the regular XTech conference.This year, that means: Ajax, web standards, open data, REST and Ruby on Rails. Any web developer wanting inspiration and a view of the future should come along. London, UK. October 31, 2006. You can reserve your place now. (We're right-priced for your training budget, and start late enough that folks could make it from all over the UK and beyond without staying a night.)
3. The pace is hotting up for XTech 2007 in ParisMeanwhile, I'm busy putting together the call for participation and programme committee for XTech 2007, in Paris, France. Our theme is "The Ubiquitous Web." As keynotes, I'm pleased to welcome Adam Greenfield, Matt Webb and Jack Schulze.
4. I'll be off travelling shortly and reconnect with old and new friends Though a little daunted by the new face of air travel, I'm heading out to O'Reilly in Sebastopol at the end of this month to take part in FooCamp. One of my main aims is to seek out the ingredients required for a successful professional online community. In September I'll be putting in some time at EuroOSCON. Both visits will let me hook up with friends, many of whom I've not seen for a couple of years.
5. I'm on the programme committee for O'Reilly's Emerging Technologies 2007 conferenceThis one has me interested and engaged. Rael Dornfest is due to unveil the call for participation in a couple of weeks' time, and I'll give my take on the conference then. The show itself is March 26-29, 2007.
6. I have an exciting startup under the covers, in the area of conference and event managementOne of the things I've done a lot of over the last half-decade is organise, coax and bully people into presenting at conferences. I'm pouring what I've learned, and more besides, into a great product. You'll soon be bored of me talking about it.
7. I'm involved in taking Pharmalicensing into an exciting new phasePharmalicensing is an online exchange for intellectual property in the biotech and pharmaceutical world. I was part of its foundation in 1998, and am still involved. I will be part of a newly energized team of people pushing it forward into 2007. 8. I'm having a bit of a clean-out of projectsAlas, with so much going on, I need to cut out a few existing commitments. One of these things will be Monopod, my Mono-based podcast client for Linux. If I can't find anybody interested in its maintenance, I'll just hang the source code out there for anyone to pick up.
9. I want to get more energy put into DOAPOne of my long term projects I definitely don't want to cut out is DOAP, my project to describe software projects in RDF. It's had a slow but sure take-up over the last few years, and is finding its way into some significant projects. However, I do need a bit more energy for it, and one or two collaborators would certainly help. 10. Around Christmas-time, I'm going to become a father to twinsI've still not recovered from the shock. Rachael and I are going to be the proud, stunned, parents of a couple of bright shiny bundles of Human 2.0. Completely awesome, and I've absolutely no idea what to do! For now, we're just concentrating on throwing enough junk out to make room for them.

Martin F. Krafft: Officially commencing my Ph.D. research

My professor, Brian Fitzgerald, identifies a common misconception about the initial phase of Ph.D. research as the drawing of a line to indicate a point in time when the research starts. Undoubtedly, a lot of thinking preceeds any such decision, and that's all part of the research. I didn't argue. Nevertheless, I do consider my research to have officially started with the beginning of this month, especially since I've really taken a turn from my initial plans last year. It's been painful staying away from computers as much as possible while trying to cure my wrist problems with all these ideas and a lot of enthusiasm about getting to work on them. Even though the problems hadn't subsided in February, I decided to get back online while trying to take more care of my body. Since then, I've attended a few conferences, addressed some legal issues, cleaned up my data, and travelled around in Southeast Asia. I could have continued going about, having a good time, and getting stuff done endlessly and I would have never started my research, which is why I decided to come to Limerick and booked a flight to have a fixed date. So far, it's been a lot of literature and papers, but I've also spent a considerable amount of time making plans for how to proceed, and I am really pleased with what I'm currently looking at, even though I really need to figure out how to avoid distractions. And since my wrists and the rest of the body aren't all well yet, I should really be explicit of how I spent the few hours per day that I can be typing. I've started a little web page to document my intentions and progress, and to bridge between the horribly tangled world of academia and the more pragmatic outside world, including the Debian project (which is core to my research). You can also find my current proposal on that page, in case you want to know more than the brief overview reveals. PS: is it really the case that there are no means on Linux to annotate PDFs and highlight text in them???

Clint Adams: Waiting for something to break

Looking out over Toronto from the top of the CN Tower, and still running for public office, I realize that I have taken several actions in my life for which I owe public apologies. Once, I blamed the entire history of Middle East politics on a hapless middle-aged retard just because he said that he liked malls. He is currently awaiting trial at the International Criminal Court in rainy The Hague. I would like to apologize to the people of the Netherlands for my part in inflicting an American on them. When I was 55, I strapped a large snowboard to each foot and sat in the middle of a narrow trail in Vermont, just to be a dick. I would like to apologize to all the skiers who injured themselves fatally in attempts to avoid me. Four hours later, I tried the same on a mountain in Utah. I would like to apologize to the American people for failing to injure any Mormons in that endeavor. In order to console myself, I broke into a beer brewery and upped the alcohol content to 4% by weight, in contravention of state law. I would like to apologize for wasting alcohol on the people of Utah. When I was three, my cousin hung up on an imaginary phone call with Mr. President and proceeded to place an imaginary phone call to Mr. Government. I became so overcome by stress from her antics that I cast away her mock aloe, peddled my tricycle to Boston, and pissed all over the First Church of Christ, Scientist. I would like to apologize for not having waited several years in order to piss on some Scientologists instead. I would like to apologize to Uncle Bill for having dragged his name into this blog entry, especially in such close proximity to the Scientologists, who hate him with aplomb. I would like to apologize for having re-enacted this episode every month since then, despite the protests of Mr. "I-am-not-a-Jew" Schwartz et al. Several years later, I defecated on the streets of Hoboken, Jersey City, Atlantic City, and Camden during rush-hour traffic. I would like to apologize for not doing this in every city in New Jersey (Newark in particular). Last month on a flight to Dar-us-Salaam, I noticed that Steve Martin was sitting across the aisle from me. While he was sleepily drooling into his pillow, I swiped his notebook. I would like to apologize to Steve Martin for plagiarizing this blog entry verbatim. I would also like to apologize for having sex with his wife Karen seventeen times. I would like to apologize to Karen for having sex with her three times without her consent, and I would like to apologize to both Steve and Karen for having sex with their children while pretending they were members of the Plimpton family. I would also like to apologize for having sex with their elderly Hawaiian turtle, whom I mistakenly believed was dead. In 1981, I dined at Sammy's Roumanian Steakhouse while wearing a purple zoot suit with a silver see-through mesh tank top. I would like to apologize to the Romanian people, the denizens of Chrystie Street, the City of New Orleans, the poblano farmers, the shiksa that gave me a lapdance, and the marmot named Stu who saw the pictures on the website of a Jewish dog rapist and promptly choked to death on his own vomit. In 1987, I dined at a Benihana with a girl who would later get tattoos and piercings. We sat at a hibachi with a truckload of Italian tourists who spoke no English at all. I would like to apologize for laughing hysterically at them when one asked, Du sprichst Deutsch? , and for getting blood all over the teppanyaki when I slit their throats after another asked, Quel pays ? On New Year's Eve in 1989, I dined at the same Benihana with a different girl whose command of Japanese was incomplete. When a middle-aged drunken British woman jammed her tongue into my mouth, a Brooklyn school bus driver laughed at me. After leaving the restaurant, he offered me a Werther's Original candy to get the taste of old woman out of my mouth. I would like to apologize for following him home and forcing him to watch the same episode of Full House over and over again for 37 hours. I would also like to apologize to Girl Scout Troop 5166 for calling them a bunch of stupid whores. While they still are a bunch of stupid whores, there was no way I could have known that at the time. Furthermore, when I was growing up, the term stupid whore approximated the use of Khidir beneath Momouteh and Shaka, when the walls fell. Finally, I would like to apologize for making a Star Trek reference when I could have said that thing about porchmonkeys instead. Now on with the campaign!

Andrew Pollock: [tech] More benchmarking

When I did my Red Hat Certified Engineer training, the second time around (when I updated from Red Hat 7.1 to Enterprise Linux 3), the trainer mentioned this Ext3 vs Reiserfs benchmark that Gurulabs performed. Probably because it had such a memorable name, I've remembered it and referred people to it, and referred to it myself, a few times over the years since. So it seemed only fair to try out their benchmarking methods (i.e. use NetApp's postmark with a few different settings) on the uber-disk as well, while I had it idle. So I did. Firstly, I ran it locally on minotaur, the piddling little 866Mhz Pentium III, with 320Mb of RAM (but with 1.5Tb of disk attached directly via USB), and then I ran the exact same tests from teevee, the box destined to become my MythTV server (3.0Ghz Pentium 4, 1Gb of RAM), with the disks mounted with ATA over Ethernet. This was with a JFS filesystem on a single logical volume striped across four physical volumes. I ran the tests three times, just like the Gurulabs benchmark, first time around, it was with: The idea being that this will generate around 150Mb of data, which should all fit in the RAM (of either system) The second time around was with: The idea being that this, generating 1400Mb of data, will exceed the RAM on both systems (but not uniformally, obviously). The final run was a total thrash with: This one causes about 19Gb of data to be written, and absolutely hammers the living daylights out of the disks (or the network, and the disks, in the case of ATAoE). So, now for the results:
Test case 1, with locally attached disks
Time:
        22 seconds total
        21 seconds of transactions (2380 per second)
Files:
        27170 created (1235 per second)
                Creation alone: 2000 files (2000 per second)
                Mixed with transactions: 25170 files (1198 per second)
        24986 read (1189 per second)
        24697 appended (1176 per second)
        27170 deleted (1235 per second)
                Deletion alone: 2340 files (2340 per second)
                Mixed with transactions: 24830 files (1182 per second)
Data:
        149.53 megabytes read (6.80 megabytes per second)
        161.85 megabytes written (7.36 megabytes per second)
Test case 1, with ATAoE attached disks
Time:
        27 seconds total
        25 seconds of transactions (2000 per second)
Files:
        27170 created (1006 per second)
                Creation alone: 2000 files (2000 per second)
                Mixed with transactions: 25170 files (1006 per second)
        24986 read (999 per second)
        24697 appended (987 per second)
        27170 deleted (1006 per second)
                Deletion alone: 2340 files (2340 per second)
                Mixed with transactions: 24830 files (993 per second)
Data:
        149.53 megabytes read (5.54 megabytes per second)
        161.85 megabytes written (5.99 megabytes per second)
Findings
A degradation in performance, sure, but not a huge one. Graph of the results
Test case 2, with locally attached disks
Time:
        218 seconds total
        209 seconds of transactions (239 per second)
Files:
        27144 created (124 per second)
                Creation alone: 2000 files (400 per second)
                Mixed with transactions: 25144 files (120 per second)
        24830 read (118 per second)
        25094 appended (120 per second)
        27144 deleted (124 per second)
                Deletion alone: 2288 files (572 per second)
                Mixed with transactions: 24856 files (118 per second)
Data:
        1405.24 megabytes read (6.45 megabytes per second)
        1535.91 megabytes written (7.05 megabytes per second)
Test case 2, with ATAoE attached disks
Time:
        165 seconds total
        158 seconds of transactions (316 per second)
Files:
        27144 created (164 per second)
                Creation alone: 2000 files (1000 per second)
                Mixed with transactions: 25144 files (159 per second)
        24830 read (157 per second)
        25094 appended (158 per second)
        27144 deleted (164 per second)
                Deletion alone: 2288 files (457 per second)
                Mixed with transactions: 24856 files (157 per second)
Data:
        1405.24 megabytes read (8.52 megabytes per second)
        1535.91 megabytes written (9.31 megabytes per second)
Findings
Whoa, who'd have thought this? Better performance over the wire? I'm blaming the fact that minotaur is a bit of a weakling compared to teevee, and teevee had a better caching advantage with more RAM. Mind you, you'd expect to see the same for the first test as well, unless size is a factor. Inconclusive results I guess. Graph of the results
Test case 3, with locally attached disks
Time:
        7356 seconds total
        7310 seconds of transactions (27 per second)
Files:
        104099 created (14 per second)
                Creation alone: 4000 files (117 per second)
                Mixed with transactions: 100099 files (13 per second)
        100246 read (13 per second)
        99560 appended (13 per second)
        104099 deleted (14 per second)
                Deletion alone: 4198 files (349 per second)
                Mixed with transactions: 99901 files (13 per second)
Data:
        18995.06 megabytes read (2.58 megabytes per second)
        19738.85 megabytes written (2.68 megabytes per second)
Test case 3, with ATAoE attached disks
Time:
        12320 seconds total
        12111 seconds of transactions (16 per second)
Files:
        104099 created (8 per second)
                Creation alone: 4000 files (32 per second)
                Mixed with transactions: 100099 files (8 per second)
        100246 read (8 per second)
        99560 appended (8 per second)
        104099 deleted (8 per second)
                Deletion alone: 4198 files (49 per second)
                Mixed with transactions: 99901 files (8 per second)
Data:
        18995.06 megabytes read (1.54 megabytes per second)
        19738.85 megabytes written (1.60 megabytes per second)
Findings
So presumably the network latency made its presence felt with data volumes of this magnitude. Graph of the results

12 August 2006

Martin F. Krafft: The relation between cipher blocks and physical blocks

Many people have responded to my post on encrypted filesystems and power losses; thank you! To clear up the confusion, I decided to write a new entry rather than updating the previous one... I learnt two important things about encrypted block devices, which make me conclude that the addition of encryption does not put your filesystem at jeopardy any more than if it lived directly on the medium. First of all, AES works on blocks of size 128 bits, not 64 bytes, but that's mostly a cosmetic issue with respect to the original blog post. What's more important is that a power loss while writing out a cipher block effectively is the same as a power loss while writing out a physical block to the storage medium: the cipher block is usually smaller than the disk block, and an incompletely written disk block is considered to be invalid. So any incomplete cipher block results in an incomplete disk block, which in turn causes the filesystem to consider the block invalid and try to recover from the journal. What confused me for a long time was the use of chaining block ciphers (see the excellent Wikipedia article for more info). Essentially, block chaining adds security to encryption by making each (cipher) block depend on its predecessor. Great, I thought, so if the first block on disk is corrupted, it'll percolate all the way through the disk and render it all unusable... Well, fortunately, this is not the case... by that logic, a write to the first block would require all remaining blocks on the disk to be recalculated. It is true that cipher blocks are chained, but only within a physical block. This means that a read error on the medium may corrupt a cipher block, which then renders all following cipher blocks contained in the physical block unusable (for they cannot be decrypted anymore). But since the filesystem would already consider the entire block invalid, this does not add to problem. PS: Thanks to Peter Samuelson for (passively) leading me to the light.

Adam Rosi-Kessel: Cell Phone Interface Design Principles

Are there any cell phone interface design principles? (Google’s answer is inconclusive.) I would think a multi-billion dollar industry would have developed some best practices at this point, but it seems like each new phone is designed by monkeys jumping on typewriters. Maybe eventually they’ll get it right, but I’d rather not wait until infinity. A persistent flaw that has always puzzled me across nearly all models is the “shut down sound” feature. If you are shutting off your cell phone, do you really need a distinctive musical reminder? Did the designers of this feature ever consider that, in many instances, the user is shutting off the cell phone because they are in a silent theatre or a business meeting? Who are these designers, anyway? Of course you can always turn the volume down first (or probably better — just silence the phone and keep it on), but that is not going to be the first inclination for most users. I was reminded of this flaw last night at a Counterpoint Concert that was being recorded for VPR radio broadcast. After the chorus director reminded people to turn off their cell phones for the recording, a symphony of shut down sounds followed shortly thereafter. Cell phone gripes aside, the concert was great — particularly the performance of They Called Her Moses. On a related note, see this petition regarding cell phone user interfaces and software. Joel on Software also has some nice observations on this topic.

Evan Prodromou: 25 Thermidor CCXIV

Maj went out for her birthday party with a passel of ladies, so I stayed home and ate pizza and watched Twelve Monkeys with the baby. We had a good time, but as the baby is approaching 1 years old, my going-to-sleep tricks are becoming somewhat outdated. My main way to put the baby to sleep is to walk her from the front door of our apartment all the way back to the laundry room, and then turn around and walk out to the front door again. I sing during the entire process, mostly Neil Young and Johnny Cash tunes, but also a mix of 80s pop, reggae, songs, and goofy children's songs. Really, it's anything I can remember the words to, or that I can bridge over with some ad-libbing. This worked pretty well when she was teensy-tiny, and it's still the best way for me to get her to sleep. But as she's approaching 15kg, it's getting to be a serious workout for me to slog up and down the hallway 50 times belting out Take the Skinheads Bowling at high volume. We have a few baby books on how to get a child to sleep easier, but I've been taking the path of least resistance. I'll probably have to take the plunge and get her to fall asleep in bed soon -- which will probably mean some angry baby moments. Still, putting the baby to sleep is one of the most satisfying things in my life right now -- I highly recommend it. tags:

Programmer hierarchy Luke Welling's programmer hierarchy, based on The Brunching Shuttlecocks' classic Geek Hierarchy, is humorous and accurate. tags:

Tag improvements So, I've made some improvements to the tagging system on this blog. Instead of doing a full-text search at runtime to lookup which pages contain which tag, I instead keep a map of tags to page names in a separate db, and then look up in that index at runtime. The bummer is that I'm not figuring out a way to keep that db up-to-date easily. The best action, of course, would be to have the tags db updated at write time, but I can't figure out how to make that work. I tried it with a write macro, but you can't get the title of the current page with a write macro, unfortunately. I might just add a hook to the commit action -- that's about all I'm coming up with right now. It's annoying that Wiliki doesn't have hooks enabled anywhere. Gar. tags:

Steve Kemp: I ll be so alone without you

My migration to the new Xen host is 75% complete. DNS changes appear to have finished and I’ve moved all the websites that I intended to move successfully. (Including databases & etc.) I’ve got three domains to move still, but I’m being very careful with them since they tie into “local” things like CVS repositories, or mailing lists + archive creation. The website migrations appear to have been a success, although I only noticed a couple of problems just now. In all cases my problems were: I’ve tested all the areas I can think of testing now. (e.g. “find /home/www -name ‘*.cg’ -print”; then executing each one in turn!) The next job will be the mail setup. My plan:

Lars Wirzenius: Debian Inquirer: BTS almost out of bug numbers, sources reveal

Debian development expected to halt! In a shocking revelation our reporter exposes a conspiracy to cover up the biggest risk to Debian development, ever! #debian-devel, the secret hangout of Debian movers and shakers, today casually discussed the possibility of the Debian bug tracking system running out of new bug numbers. This was the first hint of the impeding doom. When word leaked to our investigative reporter, he quickly dug into the case. Despite public protestations and secret threats, the truth was soon outed! Debian users will soon be unable to report new bugs. The bug tracking system uses a limited, fixed size integer for doing calculations on bug numbers, and the project is soon reaching its limits. With no new bugs to fix, Debian development can't continue. "We're already at six digits, there's only three more to go", says Lars Wirzenius, the Inquirer's resident technological expert. One wonders where the official annoucement is. Once again the Debian Inquirer delivers the scoop!

11 August 2006

Evan Prodromou: 24 Thermidor CCXIV

I said a few days ago that Ross Mayfield had one of the best blogs about wikis and the wiki business. But Eugene Eric Kim gives him a run for his money. EEK Speaks covers in detail a lot of the public and private aspects of wiki going on right now. Of course, I wouldn't paint EEK and Ross as rivals; after all, they won the Web 1.0 pitch contest at Wikimania together. tags:

Victory is mine (I think) For the last few weeks, some of the more advanced features of this wiki have been blowing up with a 500 Internal Server Error, to my great chagrin. I had switched from using CGI to FastCGI for WiLiKi and I'd seen a real improvement in the responsiveness of the site, but these occasional errors were becoming more frequent. I finally hunkered down to figure out the problem. Annoyingly, there was no error output in the Apache error logs, so I guessed that the problem was sooooo bad that the process didn't even get a chance to scream on stderr before dying. I had all kinds of theories about this. Most of the problems had to do with full-text searches of the database, so I guessed that the underlying cause had to do with I/O and concurrency. Gauche has a nice logging facility, so I threw in a lot of logging printf statements (or their equivalent) and isolated the problem to a few calls into Gauche's dbm libraries. Aha! The problem had to be concurrent DB access. I layered on a big slathery scoop of mutexes and locking and other such good concurrency stuff and... the problem was still there. D'oh! So much for my diagnostic skills. I decided to take a step back. After ascertaining for myself that any writes to stderr from within a FastCGI program should indeed show up in the error log -- by reading and by writing a couple of sample programs -- I turned to the debugging tool of last resort: strace. strace logs all system calls by a program, which can be a lot of information. I couldn't, of course, just start my FastCGI program, so I had to strace the entire Apache process, with lots of command-line flags to divvy up the output on a per-process level. Once I had that going, I could kind of see what was going on. My full-text searches were starting up fine, but about halfway through the search they'd get a SIGTERM and terminate. Holy moly! That wasn't an error in my program -- someone was murdering my poor little FastCGI processes. At first I suspected fratricide -- that the processes were terminating each other when they needed the database resource -- but I realized right away how dumb that was. That only left one suspect -- the parent! I dug a little bit into the fcgid documentation, and sure enough, if a process went too long without giving any output, fcgid would terminate it. I upped the time limit for this wait, and now all my search processes are working fine. I'm going to need to start doing something smarter with these full-text searches. I think doing some indexing at write time is the best bet -- that should probably speed stuff up quite well. But for the time being the processes will run, if slowly. The only bummer is now I have to go dig out all the mutex-y lock-y stuff I wrote earlier. tags:

Scott James Remnant: Newest Software

I’m confused by your blog, Lucas There seems to be no point to it; you’re comparing the stable releases of one distribution against the unstable archive of another one with a completely different release schedule. Of course the software is going to be out of date! In theory, there’s only one time when both Ubuntu and Debian are in sync (for main, at least); that’s Ubuntu’s Upstream Version Freeze. You won’t compare edgy because “not many people use Ubuntu development versions”, I would argue that this is not true and that the development community at least have an edgy machine around by now. And I would argue that the same is true of Debian, it’s the development community that use Debian testing or unstable. A sensible comparison can only be performed between the stable releases, and then you find it’s quite boring. There’s a linear map between the versions of software and the date of release (e.g. hoary has later software than sarge which has later software than warty).

Lucas Nussbaum: Ubuntu or Debian, which one has the newest software ?

It is a common belief that Ubuntu provides newer software than Debian. This is of course true when Ubuntu is compared to Debian stable. This is also true for specific software packages, like Gnome. But how does it compare in general ? At the time of the Dapper release, here is how Ubuntu Dapper and Debian Unstable compared wrt source package versions :
In Ubuntu, but not in Debian: 1795
In Debian, but not in Ubuntu: 588
Exact same version: 4361
Same upstream version: 2963
… and same Debian version, but Ubuntu has local changes: 731
… “Debian version” newer in Debian: 2197
… “Debian version” newer in Ubuntu: 35
Different upstream version: 2166
… newer in Debian: 1955
… newer in Ubuntu: 211
So Debian easily wins. But this comparison is unfair, because Debian unstable is a moving target : of course, it’s newer than Ubuntu ! What about Debian testing ?
In Ubuntu, but not in Debian: 2137
In Debian, but not in Ubuntu: 514
Exact same version: 4529
Same upstream version: 2652
… and same Debian version, but Ubuntu has local changes: 775
… newer in Debian: 1816
… newer in Ubuntu: 61
Different upstream version: 1967
… newer in Debian: 1666
… newer in Ubuntu: 301
Yes, Debian still wins. Okay, but Ubuntu Dapper had a particularly long release cycle. What about Ubuntu Breezy, and Debian Testing at the time of the Breezy release ?
In Ubuntu, but not in Debian: 2462
In Debian, but not in Ubuntu: 384
Exact same version: 4586
Same upstream version: 2253
… and same Debian version, but Ubuntu has local changes: 981
… newer in Debian: 1172
… newer in Ubuntu: 100
Different upstream version: 1375
… newer in Debian: 1020
… newer in Ubuntu: 355
Debian testing wins again. Notes: Update: Yes, Scott, I am comparing stable releases on one side, with development versions on the other side. However, I think that it is widely known that Debian testing is usable outside of the Debian development community. I know many people using Debian testing who aren’t Debian developers. And many Ubuntu users, if they weren’t using Ubuntu, but Debian, would probably use Debian testing, not stable. Regarding Ubuntu Edgy, its use is currently strictly limited to the Ubuntu development community, and I don’t know anybody not interested in Ubuntu development running Ubuntu Edgy (that’s also because Ubuntu is released every 6 months, not 18 months, of course). That’s why it’s irrelevant to compare with it.
Also, after a quick poll on IRC, I discovered that many Ubuntu users thought that Ubuntu stable releases had newer software than Debian Testing (that was my exact question). The only point of this blog entry is to prove them wrong. Update 2 : I’ll be on holidays for the next few days, so comments are put in the moderation queue until I come back.

Aigars Mahinovs: British rural public transport

I am writing a collective article about all thing I hate (and a few I like) in UK, but this is too important to let it slide until then - I hate British public transport.
Background information: I live in Cranfield village (ZIP code MK43 0JN, look it up) right in the middle between Milton Keynes and Bedford. Last bus from Milton Keynes to Cranfield leaves MK at 19:20. Last bus from Bedford to Cranfield leaves at 22:40. None of them is late enough for my taste, but most importantly none of them is synchronised with other transport, like trains from London.
The Story: today the sponsors of my Masters project were meeting with a technical evangelist from Google and invited me along to a pub in London. I planed my journey via Bedford. The meeting went well, the time flew by and I found myself taking a Midlands express back to Bedford that left London’s St. Pancras station at 22.00. Needless to say I missed the last bus to Cranfield by 3 minutes. Wonderful.
Of course the good thing was that I had done this walk once before a few weeks ago, when I was returning from getting my new laptop from Newcastle. Then it was a backbreaking 4 hour walk along a highway with a 10+ kg backpack and almost no water. This time I was more prepared - I knew the names of the villages in the area and knew the directions better and I had less weight with me. I also discovered a number of shops and take-away places that were still open at that hour (a rare sight here) and manged to get a sandwich and a bottle of water for the trip. I took a few safe shortcuts and in the end came home in just over 3 hours.
But still that does not change the fact - British public transport sucks, especially in rural areas.

John Goerzen: Whose Distributed VCS Is The Most Distributed?

Lately I have been trying out a number of distributed version control systems (VCS or SCM).

One of my tests was a real problem: I wanted to track the Linux 2.6.16.x kernel tree, apply the Xen patches to it, and pull only specific patches (for the qla2xxx driver) from 2.6.17.x into this local branch. I wanted also to be able to upgrade to 2.6.17.x later (once Xen supports it) and have the version control system properly track which patches I already have.

But before going on, let's establish what it means to be an ideal distributed VCS:


There are also some things that we would generally want:



Evaluation

Let's look at some common VCSs against these criteria. I'll talk about Arch (tla, baz, etc), bzr (bazaar-ng), Darcs, Git, Mercurial (hg), and Subversion (svn) for reference.

1. The fundamental method of collaboration must be a branch
All of the tools pass this test except for svn.

2. Branching should be cheap
Everyone except svn generally does this reasonably well.

The tla interface for Arch had a pretty terrible interface for this, so it took awhile simply due to all the typing involved. That's better these days.

Darcs supports hardlinking of history to other local repositories and will do this automatically by default. Git also supports that, but defaults to not doing it, or you can store a path to look in for changesets that aren't in the current repo. I believe Mercurial also can use hardlinks, though I didn't personally verify that. bzr appears to have some features in this area, but not hardlinks, and the features were too complex (or poorly documented) to learn about quickly.

svn does not support branching across repositories, so doesn't really pass this test. Branches within a repository are not directly supported either, but are conventionally simulated by doing a low-cost copy into a specially-named area in the repository.

3. Merging between branches is intelligent
Arch was one of the early ones to work on this problem. It works reasonably well in most situations, but breaks in spectacular and unintelligble ways in some other situations.

When asked to merge one branch to another, Darcs will simply merge in any patches from the source branch onto the destination which the destination doesn't already have. This goes farther than any of the other systems, which generally store a "head" pointer for each branch that shows how far you've gone. (Arch is closer to darcs here, though ironically bzr is more like the other systems)

Merging between branches in svn is really poor, and has no support for recognizing changesets that have been applied both places, resulting in conflicts in many development models.

4. Inividial changesets should be mergeable without bringing across the whole history
Darcs is really the only one that can do this right. I was really surprised that nobody else could, since it is such a useful and vital feature for me.

Both bzr and git have a cherry-pick mode that simulates this, but really these commands just get a diff from the specific changeset requested, then apply the diff as with patch. So you really get a different changeset committed, which can really complicate your history later -- AND lead to potential conflicts in future merges. bzr works around some of the conflict problems because on a merge, it will silently ignore patches that attempt to perform an operation that has already occured. But that leads to even more confusing results, as the merge of the patch is recorded for a commit that didn't actually merge it. (That could even be a commit that doesn't modify the source.) Sounds like a nightmare for later.

Arch has some support for it, but in my experience, actually using this support tends to get it really confused when you do merges later.

Neither Mercurial nor svn have any support for this at all.

5. Branching preserves full history
git, darcs, and Mercurial get this right. Making a branch from one of these repos will give you full history, including individual diffs and commit logs for each changeset.

Arch and bzr preserve commit logs but not the individual changesets on a new branch. I was particularly surprised at this shortcoming with bzr, but sure enough, a standard bzr merge from a remote branch commited three original changesets into one and did not preserve the individual history on the one commit.

svn doesn't support cross-repo branching at all.

6. Merging preserves full history
Again, darcs, git, and Mercurial get this right (I haven't tested this in Mercurial, so I'm not 100% sure).

Arch and bzr have the same problem of preserving commit logs, but not individual changesets. A merge from one branch to another in Arch or bzr simply commits one big changeset on the target that represents all the changesets pulled in from the source. So you lose the distinctness of each individual changeset. This can result in the uncomfortable situation of being unable to re-create full history without access to dozens of repositories on the 'net.

Subversion has no support for merging across repositories, and its support for merging across simulated local branches isn't all that great, either.

7. It is possible to commit, branch, merge, and work with history offline
Everyone except Subversion does a good job of this.

8. The program is fast enough for general-purpose use
All tools here are probably fast enough for most people's projects. Subversion can be annoying at times because many more svn commands hit the network than those from others.

In my experience, Arch was the slowest. Though it was still fine for most work, it really bogged down with the Linux kernel. bzr was next, somewhere between arch and darcs. bzr commands "felt" sluggish, but I haven't used it enough to really see how it scales.

Darcs is the next. It used to be pretty slow, but has been improving rapidly since 1.0.0 was released. It now scales up to a kernel-sized project very well, and is quite usable and reasonably responsive for such a thing. The two main things that slow it down are very large files (10MB or above) and conflicts during a merge.

Mercurial and git appear to be fastest and pretty similar in performance.

All of these tools perform best with periodic manual (or scheduled cron jobs) intervention -- once a month to once a year, depending on your project's size. Arch users have typically created a new repo each year. Darcs users periodically tag things (if things are tagged as part of normal work, no extra work is needed here) and can create checkpoints to speed checkouts over the net. git and Mercurial also use a form of checkpoints. (not sure about bzr)

Subversion works so differently from the others that it's hard to compare. (For one, a checkout doesn't bring down any history.)

Conclusions
I was surprised by a few things.

First, that only one system actually got #4 (merging individual changesets) right. Second, that if you had to pick losers among VCSs, it seems to be Arch and bzr -- the lack of history in branching and merging is a really big issue, and they don't seem to have any compelling features that git, darcs, or Mercurial lack. #4 was a unique feature to Darcs a few years ago, but I figured it surely would have been cloned by all the other new VCS projects that have popped up since. It seems that people have realized it is important, and have added token workaround support for it, but not real working support.

On the other hand, it was interesting to see how VCS projects have copied from each other. Everyone (except tla) seems to use a command-line syntax similar to CVS. The influence of tla Arch is, of course, plainly visible in baz and bzr, but you can also see pieces of it in all the other projects. I was also interested to see the Darcs notion of patch dependencies was visible (albeit in a more limited fashion) in bzr, git, and Mercurial.

So, I will be staying with Darcs. It seems to really take the idea of distributed VCS and run with it. Nobody else seems to have quite gotten the merging thing right yet -- and if you are going to make it difficult to do anything but merge everything up to point x from someone's branch, I just don't see how your tool is as useful as Darcs. But I am glad to see ideas from different projects percolating across and getting reused -- this is certainly good for the community.

Updates / Corrections
I got an e-mail explaining how to get the individual patch diffs out of bzr. This will work only for "regular", non-cherry-picked merges, and requires some manual effort.

You'll need to run bzr log, and find the patch IDs (these are the long hex numbers on the "merged:" line) of the changeset you're interested in, plus the changeset immediately before it on the same branch (which may not be on the same patch and may not be obvious at all on busy projects.) Then, run bzr diff -r revid:old-revid-string..new-revid-string.

I think this procedure really stinks, though, since it requires people to manually find previous commits from the same branch in the log.

10 August 2006

Martin F. Krafft: Blackouts and encrypted filesystems

After some recent filesystem fun, I now have a laptop with encrypted partitions, using dm-crypt and LUKS, conveniently installed with partman-crypto. Pre-Scriptum: I have updated my previous post about XFS troubles in response to Ingo's statements. Just in case you care... The following post poses a question and a meta-question: As I was cycling home today, I thought about what would happen if my laptop lost power now in the middle of a write operation. Sure, ext3's journal would make sure the filesystem was consistent, but what about the actual data on the disk? Journaling works by writing a change to the journal, then writing it to disk, then deleting the enty from the journal. If the filesystem is mounted and it has entries in the journal, those are "played back", meaning they just get written to disk again, overwriting potentially incomplete data written as the power was lost. Yes, this is simplified, but it'll do for now. With encrypted filesystems, there's an important layer between the filesystem and the physical storage: the encryption algorithm. An encryption algorithm like AES works on blocks, by default 64 bytes in size. 64 bytes are not a whole lot, but given the nature of hard drives, it's not possible to write 64 bytes to the drive atomically [*]. Thus, conceivably, it could happen that your power cuts out after only 32 bytes have been written. I don't believe dm-crypt does journaling for the encrypted data it writes, so effectively, a power loss on systems with encrypted filesystems almost certainly means data loss? Please correct me if I'm wrong! And the meta-question would be: where would I go to discuss such issues? I've abused debian-security previously, the issue seems too general for the dm-crypt mailing list, there is cryptography t metzdowd.com but somehow this does not seem to be a crypto problem... so where would you take this? Let me know...
[*]

In a computer architecture course in college, the prof raised the interesting point that between the loss of power and the actual disappearance of all current in the circuits, several million operations are possible on modern machines. I have not yet heard of any computer or hardware device making use of those precious cycles.

Update: actually, several manufacturers are making use of the precious cycles. See for instance this post, which describes some stuff done by SGI. Hanspeter Kunz also adds that harddrives use this time to park their heads.

Martin F. Krafft: Through with XFS

I am through with XFS, once and for all. Well, at least for laptops. I still think it's a good filesystem when you can ensure that the power never goes, and your hardware is reliable, but it's just not adequate for laptops or even desktops. I ran into some serious problems a while ago, but managed to recover. Two nights ago, however, three XFS filesystems on my laptop decided to blow up and left my system thoroughly broken. I guess as the hibernate maintainer, I should really start doing my tests somewhere else than my main system... It all started out with a dist-upgrade and this output: 
dpkg: error processing /var/cache/apt/archives/dpkg_1.13.22_i386.deb
(--unpack): unable to make backup link of
'./usr/share/man/man1/dpkg-deb.1.gz' before installing new version: Unknown
error 990
Looking at /usr/share/man/man1, I started to anticipate the apocalypse: 
# ls -l /usr/share/man/man1
total 7956
?????????? ? ? ?    ? ? ? 7zr.1.gz
?????????? ? ? ?    ? ? ? 822-date.1.gz
?????????? ? ? ?    ? ? ? CA.pl.1ssl.gz
?????????? ? ? ?    ? ? ? Defoma::Common.1.gz
So I look at the log, and amidst kernel oops notices, there's this lovely cookie: 
Filesystem "hda6": Corruption of in-memory data detected.  Shutting down
filesystem: hda6
Filesystem hda6 is /usr, so at that time I figured "it could have been worse", booted to single user, and remade the filesystem with the intention to simply reinstall all packages... when I found /var/lib/dpkg/info to be in similar condition. The rest of /var seemed fine, but I resolved then that there was no hope in reviving this system. Fortunately I brought an external drive that had just enough free space to hold my /home and some other stuff, but since USB is really slow when it comes to shifting large amounts of data, I decided to do something productive in the mean time and to answer some outstanding mails. It wasn't difficult to get SSH back up, so I started to work on a remote machine and used the time efficiently. Some time later, though, I got confused in the mist of screen sessions and was browsing my home directory on the laptop, thinking I was elsewhere (my home directories are mostly synchronised), when I noticed a directory in similar condition as the above. Oh shit. Imagine my pain and fear as I first thought my remote machine was also dying, imagine the sigh when I found out I was on the local filesystem, and imagine the shock when I realised that /home was also affected by the XFS breakage... A glance around /var confirmed that the XFS breakage was actually spreading and had now affected three filesystems on this machine. Fortunately, by that time, I had copied everything to the external drive, and decided to put my laptop and myself to sleep. I woke the next morning to the task of reinstalling the thing and decided to be optimistic about it. After all, a reinstall would mean I could finally try partman-crypto and encrypt my laptop's data to protect against leaking sensitive stuff in the case of loss or theft of my laptop. The installation was not as painless as I had hoped, but that was mainly because I ran into a known problem with the graphical installer and partman-crypto, which does not allow to set up volumes with random encryption keys (e.g. swap; see the forthcoming announcement for the beta3 release of the installer), and a bunch of smaller bugs. I had to restart the installation with the traditional frontend to get what I wanted, but other than that, I was very impressed with what our installer development team has accomplished! And a special round of gratitude to Frans Pop for not losing his patience while helping me on several occasions throughout the process. Now, 24 hours after the incident, I am back to normal with a fresh laptop and no data lost (except for one directory which I pulled from a mirrored remote machine; it had no local modifications (so why did XFS screw it up anyway?)). The fonts are all jaggy, so there's something I have to figure out. All things considered, I am sad to have lost 24 hours, but I can also relax more now, without fear of further XFS breakage or loss of private data. Update: Oh, and despite this, I did choose ext3 for all my laptop's filesystems. JFS was really cumbersome and slow last time I tried it, and I surely would not touch RazorFS after experiencing serious data loss on numerous occasions. Update: Two responses so far. Full ack for Julien (except for him laughing at me), Ingo's post warrants a reply though. First, ext3 is also journaled, and if you're about to say "yeah, but it's a hack on top of ext2, well... ext2 is damn mature, and journaling isn't really rocket science, so that "hack" isn't going to be too complicated. In fact, I like the idea of journaling being an option rather than a built-in feature. Second, of course you're supposed to keep backups. But since you keep backups, my top requirement of a filesystem is not "how to get the data back", but "how to ensure it does not break. If it breaks, I can reinstall and restore from backup, but that's a certain amount of time lost. If it doesn't break, well, that's like stealing a little something back from death then, isn't it? Third, I do follow the linux-xfs mailing list, but so what? I did not have write cache enabled, and I was running the 2.6.17.7 kernel at the time of the mishap. Lastly, you point to "excellent tools" to recover the filesystem. I am not sure how excellent xfs_repair really is when it reports "bad magic number 0x0 on dir inode 4696727" during the run, claims to have fixed it, I mount the filesystem, unmount it, run xfs_repair again and get the same message. No filesystem is perfect, and as we know from Biella's problems (among many others, ext3 is no exception. But we did get her data off! So then it's really an open field again, crap filesystem against crap filesystem. I guess at this point it helps to know that ext3 actually follows VFS semantics, while on XFS, a completed sync() syscall does not actually mean it has written the data to disk (see e.g. #317479). And then there are bugs like #239111... ext3 it is for now. If that let's me down, I'll try JFS. If that fails and noone has actually implemented a proper filesystem, I might have a go myself. Haha. Update: Alceste Scalas adds:
Ingo is right when he says that every filesystem has bugs --- but bugs apart, the design of Ext3 (i.e. its physical-block journaling) makes it a far more reliable choice for desktop and laptop PCs, expecially for people without an UPS. An Ext3 filesystem could only crash because of a bug or an hardware failure, while an XFS filesystem can be trashed even without bugs or hardware failures, due to the unavoidable consequences of a power loss on PC-class hardware.
He also alerted me to this mailing list post, which compares data=ordered journaling of ext3 (which almost noone does for performance reasons) with XFS and RazorFS. Update: You may also be interested in this post. Update: Otavio Salvador points me to this FAQ entry, which SGI must have added very lately. It explains how to deal with the directory corruption that was part of my problem. I guess I would have liked to know earlier, but I consider the outcome with dm-crypt + ext3 a win anyhow.

Wouter Verhelst: Re: SDL

I received a number of comments on my previous blog post. I was going to reply to each of them separately, but then decided that this was worthy of a blog post of its own. Two people posted comments telling me to check whether I had OpenGL acceleration switched on. In case you guys missed it, Freetennis is a sprite-based and essentially 2D game. It does not use OpenGL, by the looks of it. Neither does starfighter, powermanga, or a number of other 2D games I tried. Yet they all require the full 1.3Ghz of my processor, and some of them, including freetennis, think even that isn't enough. I received some other claim that SDL by itself performs pretty well. While this may be true, the fact remains that many simple games that should work well on older hardware seem to require a lot of CPU time, and that they all seem to use SDL. If it's not SDL at fault, then it's probably the SDL documentation which doesn't stress the need to KISS enough. Or so. A final remark compared the situation to Quake2 on his X40. I don't know what you've been doing, but I'd been happily playing Quake2 on my 650Mhz PentiumIII until that one got migrated to become my parent's computer. Without hardware acceleration. And, luckily, without SDL, too.

Next.

Previous.