I started the grep-dctrl rewrite last spring. I soon had most of the code implemented, and I asked for testers on -devel. There were no debs at that time, and so it probably was not much of a surprise that nobody volunteered. I got discouraged and abandoned the project for several months. In August, just before my vacation at work ended, I decided to polish the code, and I did. I prepared debs and was ready to submit them to unstable. (At that time I was under the impression that experimental was the wrong place for any grep-dctrl stuff.) I felt it was necessary to warn users about the code, since it was sure to contain nasty surprises. I considered sending a mail to some lists: -devel-announce would have reached my audience, but the message would have been inappropriate there; I feared it would not be noticed in -devel. I considered a message at installation time,either with debconf or without. However, debconf messages are seen by system administrators and not users, so that was not an attractive option. I finally decided on that annoying banner:

==========================================================================
                                  NOTE                                    
 grep-dctrl has been rewritten from scratch.  Although this does add new  
 features, regressions are certainly possible.  Please watch for them and 
 report them to the BTS.                                                  
==========================================================================
(The above annoying banner will not be shown to you again, unless you
request it with the -B switch.  It will also be removed entirely soon.)

I took great care to make the banner as unobtrusive as possible, still taking in account that it had to be prominent to have any effect. I added logic that would suppress the banner after a user had seen it once (using a dotfile in home directories, ~/.grep-dctrl-banner-shown; if you find one in your home directory, you can now safely remove it). I did not print it to stdout (it would clobber the output) nor to stderr (it might be redirected, which would have been counterproductive); rather, I used /dev/tty. After all, many people use grep-dctrl in scripts without realizing it; especially they would need to know that there was a weak link there. Nevertheless, I expected to get flamed for this. To my considerable surprise, I got no reactions - aside from several good and substantial bug reports. The banner stayed in longer than I intended, mainly because I was exhausted by my teaching duties during the autumn. I finally removed it last week, in 1.103. And then, several days after the banner had been removed, I get a bug report (which, strangely, is still open even though I sent a closing message almost 24 hours ago). That's when I decided to post this entry.

I've found a couple of issures with my SELinux backports. One is actually a bug present in sysvinit and cron (at least): The debian/rules makefiles uses dpkg-architecture to detect a Linux system (as opposed to GNU hurd or FreeBSD or Solaris) since SELinux only works on Linux. This code queries the DEB_HOST_ARCH_OS variable via dpkg-architecture, but this variable isn't available on sarge (yet)... When building with either my dpkg backport or by using code similar to the following:

dpkg-architecture -qDEB_HOST_ARCH_OS   dpkg-architecture -qDEB_HOST_GNU_SYSTEM

the packages will actually have SELinux support enabled. I've already filed a bug against sysvinit, but should I also file bugs against logrotate and cron, who apparently use the same code to detect SELinux? (openssh and coreutils have better fallback code than suggested above in place)

...people imitate you when you were trying to demonstrate the sillyness of doing so in the first place.

I went to the local Java Users Group meeting last Saturday to check it out. I hope to give a presentation on Hecl sometime soon, so I figured it was fair to go once or twice before showing up to recount about what I've been hacking on. The meeting started off with presentations on first Java Server Faces, then Tapestery. Lots of "then include this, configure that XML file, add these". The final presentation was two guys who had volunteered to redo the user group web site. After poking around at a few different things, they built it with Ruby on Rails, and gave a presentation on their experience. I was once again surprised at the uptake of that system in the Java world. The response amongst the people with whom I chatted afterwards was that it's really Rails itself that's so attractive. I see Tcl and Tk as being a lot faster than Java for GUI work, but perhaps Java folks are used to using some kind of GUI builder for that, so the difference is less pronounced... and Java's uptake has probably been greatest in the web world, so it stands to reason that something like Rails would really garner a lot of attention.

Apathy

I’m sleeping a lot more than I should be which is making me pretty apathetic at the moment. Even checking email is a real effort. Let alone replying.

Last Post

As previously mentioned I’m utterly unfranchised with WordPress. This will be my last post using it. All future updates will be made using Chronicle, a simple Perl-based application I’ve been working on for a while. (The best part is the name!) So far I’ve got code to :

• Import old wordpress entries into a new database.
• Add new posts.
• Show archives.
• Do searching.
• Add new entries.

TODO:

• Import old comments.
• Add code to handle displaying/editting/deleting comments.
• Add some mod_rewrite magic so that all my old links still work.

The application is pretty nice, comprised of several perl modules and built using the CGI::Application base-class. I realise this contradicts the apathetic stanza above …

Thanks to Andrew Pollock, I now know how to write "ba onnette" in English. "now know how": nice....probably wrong but nice. Andrew, continue with your life experience of weird things happening to Australian people left in Californian jungle.

Since interviewing the creator of Ruby on Rails, David Heinemeier Hansson, I've been thinking a little more about what he calls "opinionated software." That handy little phrase seems to sum up what attracts me to a lot of technology, at all levels. What I like Free software--it's not really economics that drew me to free software, but the principles and aims. When you release or use free software, you're exercising your right to free speech and adding your voice to an argument. Desktop environments--GNOME 2.0: The move from GNOME 1.0 to version 2.0 was highly opinionated (Mac users, think OS 9 to OS X). Complexity of user interface was thrown out in favour of sensible defaults. Web standards--CSS, RDF, XHTML: not only is there a right way to do things, but it turns out to be better too! I don't automatically fall in love with everything that emerges from the W3C, but some things are so obviously the right way it's impossible not to love them--warts and all. Muine music player--this player throws out the paraphernalia of iTunes clones and gets you close to the music. Queue, play. What I want to do. Epiphany web browser--a simple text bar does for bookmark lookup, search and URLs. The bookmarks work like del.icio.us tags. Despite the lure of Firefox extensions, simplicity and efficiency keeps me with Epiphany. Make friends and enemies The thing about opinions is that people are inclined to disagree with them. Opinions won't please everybody and, being opinionated, you will never carry everybody with you. But opinions give you something with which to scythe through indifference. In both my writing and software development work taking a particular stance enables me to create interest, and gives direction to my projects. I tend to get bored easily, and if I'm bored, then my readers and users will never make it. Being opinionated in software development is easier when there's a strong leader, or it's a one-person project. It's more of a dilemma if you're running a free software project with multiple contributors. Such projects need such contributions, but run a risk of losing focus and style. I think that there we need to be brave again. If being opinionated means losing potential users, it also means losing potential contributors. Opinions help in dealing with users, too. It's obvious that the mere act of listening to users isn't good enough. They often won't say what they mean, or their requests mask a deeper issue. How often have you given your boss exactly what he asked for, to find it wasn't what he wanted? A combination of reading-between-the-lines and having opinions helps cut a way through the noise of user feedback. Stirring the pot Here are a few wishes for some free software projects that I want to see succeed further, but which really need to get opinionated to be effective. These projects aren't failing, but they're not reaching the potential I know is there. Each of them are infrastructural in some way, and that's probably where the temptation to avoid being opinionated is strongest. But here's the genius: the rewards of being opinionated here are high. Ruby on Rails is the existence proof.

• Beagle, live search technology for GNOME. Needs to decide where its primary use will be, and get on and hit the 1.0 point for that use only. Choose one or two neat applications for integration, and be so utterly cool that every other app author flocks to integrate too.
• Synchronization. First MultiSync, then OpenSync, have rumbled on for ages, without ever being truly effective for any particular device. My take: choose a few "must-have" target devices--say, iPods and Nokia phones--and make them work completely. Nearly-but-not-quite pleases nobody.
• Mono. Mono's great, but where's the story? There's loads of potential in the people and the ideas behind the project, but playing catchup with Microsoft isn't very inspiring. Early projects like Muine helped Mono's cachet here, but we need more. My personal favourite option would have been a completely free alternative to ASP.NET for web applications. (I've seen a project called MonoRail, but don't know if it's the one...)

What's better? A handful of users with the "right" idea about how fundamentally important you are to everything, or loads of users who only see a portion of the capabilities of a project? Copy the experts It may seem like cheating to copy Apple's lead, but a lot of what they've done in OS X is to take a stand on what's useful and not. On many occasions this has inconvenienced people, but it's not eroded that special lure that Apple products hold. There's not harm in asking "What have Apple done?" with some of our projects. Another poster-child for being opinionated is Ubuntu Linux. Technically, it's only marginally better than other Linux distributions. Strategically, it's miles ahead. Its founder, Mark Shuttleworth, had very solid opinions about where Ubuntu should go, and pursued them. Over time, these have pervaded the product itself, but at the start it was the message and the style that made the difference. Opinions not expressed are rarely effective.

This posting is part of my Finnish criminal court series. See the introductory essay for more information and an overview of the Finnish criminal court system. The following is a fictional account of a criminal case, one of the most frequent ones, written in the form of a play script. A PDF is also available. There is also an OpenOffice-generated XHTML renderition, which may be more pleasant to read than the version below (but lacks most of its logical markup). DRIVING WHILE INTOXICATED
A glimpse of Finnish criminal justice
Part One Dramatis personae
Presiding judge
Prosecutor
Defendant Extras
Three lay judges
Court reporter Sets
Courtroom
Courthouse lobby DISCLAIMER This is a work of fiction. Although this play is based in general terms on real life, all the specific characters and events portrayed or described in this play are fictitious, and any resemblance to real people or events is purely coincidental. The verdict presented is fictitious, and while it is based on the author's understanding of the law, it may be erroneus. No legal advice should be inferred from this work of fiction. Scene One: And so it begins The courtroom is about the size of a large school classroom, and shaped as a rectangle in terms of floor layout. The judges' bench, a long wooden table, lays near one of the short walls; the space between the bench and the wall is enough to give the judges and the court clerks comfortable room to sit and to move behind the sitting judges. At the center of the bench is the seat of the presiding judge, who has easy access to a telephone, a public address system and a panel for controlling the LED display visible in the lobby above the courtroom door. To his or her right is the court reporter's seat in front of a PC and in control of recording equipment. To the reporter's right there is room for one additional individual. To the presiding judge's left, there are three chairs for the lay judges. If one were to stand directly opposite the presiding judge, on the other side of the bench, one would find an open Bible. In front of the bench, disregarding the Bible, there is some room, approximately two meters or so. Along each long wall there is a small table, and those two tables face each other, and are therefore turned sideways from the bench's point of view. By the desk on the bench's right is the seat of the prosecutor, and the other, facing desk is reserved for the witness being examined. Facing the bench, separated from it by that empty space, are two rows of tables with chairs. Beyond them there are two rows of seats for the audience without tables; the farther row is right at the other short wall. The long wall to the judges' right contains one double door leading to the court building lobby, and the other long wall contains two sets of double doors at each end of the wall, leading to a part of the building not accessible to the public. We'll call the door closer to the bench the judges' door. The PRESIDING JUDGE is a 50-something man dressed formally in black and blue. The three LAY JUDGES are more casually but still essentially formally clothed. LAY JUDGE ONE is a sixty-something male, LAY JUDGE TWO is a female in her fourties and LAY JUDGE THREE is a female just under thirty. The COURT REPORTER is a female in her fifties, also dressed formally. It is 9:00. The four JUDGES and the COURT REPORTER are sitting on the bench. Each JUDGE has papers in front of him or her, the PRESIDING JUDGE's stack being by far the thickest. The PROSECUTOR, fourty-something man, enters through the judges' door, carrying law books and a large stack of paper. He sits down at his desk. PRESIDING JUDGE (Nodding to the prosecutor.) Okay. Let's start. (Enters codes on the LED machine. Then picks up the public address microphone.) The criminal case 'prosecutor against Matti Meik l inen' begins in room three. Parties to room three. Scene Two: Main hearing The DEFENDANT enters through the double door from the lobby. He feels a little lost. PRESIDING JUDGE (To the defendant:) Matti Vihtori Meik l inen? (DEFENDANT nods.) Please, take a seat in the front row. (He waits for the defendant to comply.) This case will be tried as follows. First, the prosecutor will read the charge. You will then respond by telling us if the charge is correct or not, that is, if you in fact did commit the acts specified in the charges, and if you did not, or the charge is only partially correct, you should state briefly why the charge is in error. The prosecutor will then present the case, he will describe us his understanding of the events in question. When he is finished, you will again have opportunity to state if the prosecutor is correct or not, and if not, in what way is he wrong. If there is disagreement between you and the prosecutor, we will hear the evidence. After that, the prosecutor will state his or her concluding opinion of the case as presented here. Again, you will have the opportunity to respond. We will conclude with discussion of what punishment, if any, would be appropriate. The court will then discuss the case alone, and will finally announce the verdict. Is this understood? (DEFENDANT nods.) Okay. Prosecutor? PROSECUTOR (Reads the charge from a paper everyone has in front of his or her.) I demand punishment to Matti Vihtori Meik l inen for driving while intoxicated. Matti Vihtori Meik l inen had on the first of May this year operated his car while his exhaled air contained 0,597 milligrams of alcohol per liter. Matti Vihtori Meik l inen's driving license must be suspended. PRESIDING JUDGE (To the defendant.) Is the charge correct? DEFENDANT Yes, it is. PRESIDING JUDGE Okay. The prosecutor will now tell us the circumstances and events in detail. PROSECUTOR Thank you. It was to my understanding late evening on May day, and the defendant had come from a bar and was apparently trying to move his car to a cost-free parking space. A police patrol happened to observe this and the defendant had apparently been a little shaky in his walking, so the officers gave him a breathalyzer test, which showed significant alcohol content. The defendant was brought in the station where a precision test was performed. The first reading was 0,597 and the second reading was 0,617; the mean of these two is 0,607, and after performing the confidence subtraction, the reading 0,597 milligrams per liter was obtained. PRESIDING JUDGE (To DEFENDANT) Well then, is what the prosecutor just said true? DEFENDANT (Speaking in a colloquial manner and very somberly. He is ashamed and wishes this would be over soon.) Yeah, it is. PRESIDING JUDGE All right. It seems to me that there are no points of disagreement, so we don't need to hear witnesses. (The PROSECUTOR nods.) Okay, it's time for the concluding statements. I think we will hear consequence opinions at the same time. Prosecutor? PROSECUTOR Thank you. I don't think there is much to say. The case has been shown beyond reasonable doubt, in my opinion. We have the precision test readings and the defendant's admission. As to the penalty... the amount of alcohol we are talking about is just below the limit for serious intoxication, so this is no small deal. On the other hand, the circumstances were not very dangerous. I think suspended imprisonment could well be considered, though a stiff fine would probably be more appropriate. The driving ban must be fairly long, obviously. PRESIDING JUDGE All right. (To the defendant.) Do you have anything to say before we adjourn to consider the case? DEFENDANT Well, I suppose it's clear that something is going to be given to me. I prefer a fine, naturally. PRESIDING JUDGE Naturally. The court will consider all options. Do you have any special requests concerning a driving ban? DEFENDANT As short as possible, of course. PRESIDING JUDGE Do you have a particular need for a car, do you need it for work, for example? DEFENDANT No. PRESIDING JUDGE Okay. Now, your income. We have here the last completed tax report for you, it says you earned 500 euros a month and had no children. Has this changed? DEFENDANT No change. PRESIDING JUDGE Okay, thank you. If you will then go outside to wait while we consider this case, thank you. The DEFENDANT exits to the lobby, the PROSECUTOR exits through the judges' door. Scene Three: Deliberation We are now in the lobby outside the courtroom. There are many doors that lead to many courtrooms. Above each door, attached to the wall, is a display showing the current case number; there is also a light saying do not enter, which is lit for most of the rooms, including Three. Also there is a numeric display which is out at the moment for most of the rooms. For some rooms, they indicate when the session will resume. Each courtroom door also has a paper pasted to the door frame listing the cases for the day. The door leading outside is separated from the lobby by a routine security check area, and one of the courthouse police guards stands there waiting for people to enter. Some other guards and janitors sit in their glass booth. There are many people sitting on the chairs, waiting for their turn to enter one of the courtrooms. DEFENDANT enters from the Courtroom Three door and sits down, waiting. It takes about fifteen minutes. During this time, announcements from other courtrooms could be occasionally heard, and people would enter or exit other courtrooms. Then... PRESIDING JUDGE (Over the PA) Verdict in the case 'prosecutor against Matti Meik l inen' in Courtroom 3. DEFENDANT exits through the Courtroom Three door. Scene Four: The Verdict The DEFENDANT enters from the lobby and sits down. After a minute, the PROSECUTOR rushes through the judges' door. PRESIDING JUDGE (Announcing to all people present.) The District Court has examined this case and has decided that Matti Vihtori Meik l inen did in fact commit the crime he was accused of in the way specified in the charge. The justification is the combination of the measured alcohol content in Meik l inen's breath and his own admission of guilt. Matti Vihtori Meik l inen is therefore sentenced to pay sixty-five day-fines, that is three hundred and ninety euros. Penal code, Chapter 23, Section 3. His driving license is suspended until the first of November this year. Road Traffic Act Section 75 Paragraph 1. One who is dissatisfied with this verdict can appeal it to the Vaasa Apellate Court within thirty days. Dissatisfaction must be declared, however, either right now to me or within a week to the Office of this District Court. (To DEFENDANT) Come and take the fine ticket and written appeal instructions (DEFENDANT takes those from the COURT REPORTER.) This hearing is concluded. The DEFENDANT leaves, as the court and the prosecutor prepare for the next case. TO BE CONTINUED
(with the next case)

When I want to upload new packages to my sarge SELinux repository on alioth, it usually takes me around 10 tries. Something is seriously wrong with LDAP on that box - it just goes totally crazy. This almost unbearable... Maybe I should move the repository to people.debian.org instead?

I spent the whole weekend looking for a DRM-encumbered Sony CD so that I could check for myself whether my code was really being redistributed without permission. I eventually found one: Suspicious Activity?, by The Bad Plus. A rather enjoyable post-modern jazz album. I am rather lucky, it could have been C line Dion. As expected, installing the CD s custom player also installed a stealth aries.sys driver along with a few other interesting files such as $sys$DRMServer.exe, all hidden in a $sys$filesystem directory. I rapidly got rid of this cruft and started studying the really important file, ECDPlayerControl.ocx. It is the file Sebastian Porst and Matti Nikki found to be containing parts of mpglib, LAME, faad2, VLC... Evidence that the DRMS code comes from VLC There are of course obvious similarities between some functions and structures of ECDPlayerControl.ocx and the ones in VLC s source. However, given how precise the implementation of a cryptographic protocol needs to be, the possibility that two separate implementations show many similarities must not be dismissed. The data structures and constant tables especially are likely to be identical. I think however that I found strong enough evidence that this is not the case here. The original drms.c was written by Jon on 01/05/04. I then hacked on that file on 01/18/04 and reorganised it, also replacing the reverse-engineered MD5 and AES functions with clean implementations, then Jon added DRMSv2 support on 05/05/04 and I reorganised the code again on 05/08/04. Some of these changes were specific to VLC (not in that they made the code unusable outside of VLC, but rather that they made it possible to use an external MD5 hash provider, for instance), so they were the bits I was looking for in ECDPlayerControl.ocx. For instance, Jon s original code had the following instruction:

p_acei[ 4 ] = 0x5476212A;

Since all bytes of this long word are comprised between 0x20 and 0x7f, it was later rewritten like this:

char p_secret1[] = "Tv!*";

This change makes the code more human-readable because many other secrets in the Apple DRMS protocol are ASCII strings. Also, I know how to look for binaries that use the string "Tv!*", but it takes me some more time to think of a way to find binaries that use 0x5476212A. And the Sony rootkit uses the following (only relevant lines shown):

 .text:100883B8   mov   cl, byte ptr ds:xxxxx+4 ; 0
 .text:100883C2   mov   eax, dword ptr ds:xxxxx ; "Tv!*"
 .text:100883C8   mov   [esp+70h+yyyyy], eax
 .text:100883DC   mov   [esp+7Ch+zzzzz], cl

The use of xxxxx+4 shows that ECDPlayerControl.ocx uses a string instead of a 32 bits integer to store the secret, disregarding the fact that the 5th character of the string (a null char) is never used. Another example: in this commit I reorganised the DoExtShuffle() function, merging FourthPass() and FifthPass() and discarding a structure allocated on the stack. What is important here is that DoExtShuffle() was built by a complex combination of choices, the main purpose of which being to reduce the size of the C functions:

 static void DoExtShuffle( uint32_t * p_bordel )
  
     uint32_t i_ret;
     i_ret = FirstPass( p_bordel );
     SecondPass( p_bordel, i_ret );
     ThirdPass( p_bordel );
     FourthPass( p_bordel );
  

Note how i_ret is not reused and could be merged into the p_bordel array, or maybe SecondPass() merged into FirstPass(). This choice is definitely a legacy of how the code evolved. And the Sony equivalent: a perfect match.

 .text:10089F3C xxxxx:
 .text:10089F3C   mov   ecx, esi
 .text:10089F3E   call  yyyyy
 .text:10089F43   push  eax
 .text:10089F44   mov   eax, esi
 .text:10089F46   call  zzzzz
 .text:10089F4B   add   esp, 4
 .text:10089F4E   call  ttttt
 .text:10089F53   call  uuuuu

After having studied the Sony code for a while, I have of course gathered dozens of such examples. But I also discovered a few new things. Where does the DRMS code come from? The code undoubtedly comes originally from VLC, but it has traveled a lot. I would be surpried if it came directly from VLC, as no other part of VLC is included with the rootkit. And although there are parts of FAAC in the Sony code and FAAC includes VLC s drms.c, the version currently in the FAAC CVS is horribly outdated. Sebastian Porst also noticed a few obvious differences between the Sony code and the version of drms.c present in VLC. The most obvious being this one:

 if( p_shuffle->i_version == 0x01000300 )
  
     DoExtShuffle( p_bordel );
  

Which became:

 .text:10089F2E   cmp   eax, 1000300h
 .text:10089F33   jz    short DoExtShuffle
 .text:10089F35   cmp   eax, 1000400h
 .text:10089F3A   jnz   short skip
 .text:10089F3C DoExtShuffle:
 .text:10089F3C   ...
 .text:10089F58 skip:
 .text:10089F58   ...

That 0x01000400 is a version check for 4th generation iPods firmwares. I also discovered at least 40 KB of new lookup tables that seem to be used for more buffer shuffling. At first I thought it was simply F4I s DRM implementation, but the shuffling calls are nested with iPod hardware information retrieval from our DRMS code, so they re really part of drms.c. Sony distributing a DRMSv3 descrambler? The only explanation I can think of is that someone took the VLC code, added support for an upcoming Apple DRMS format version and redistributed the software without letting the VLC authors know, probably violating the GPL (I only say that because I have not found the software yet, but I feel pretty safe in saying it). Since it is based on GPL software, I was already entitled to ask Sony for the source code of ECDPlayerControl.ocx. But as the copyright holder of a significant part of the software they counterfeited, maybe I have a greater chance of being heard. Not that it would be difficult to reverse-engineer it yet another time, but it would be awesome if the next opensource iTunes Music Store file player was contributed by Sony!

WARNING: nothing in this post has been implemented or tested, except maybe at a minimal proof-of-concept level ("it compiles!"). It's just some musings on how you can make the computer do your work for you.

I was reading a defense of Hungarian notation when I started to get a familiar notion tickling the back of my brain. The author was explaining that the original idea behind Hungarian was not to replicate type information in variable names, but rather to include some sort of usage information: one example would be giving a string value holding a value that is untrusted the name utName. While it's more reasonable than lpschwzMyVariable, this suggesion made me suspicious that the author was really hacking around suckiness in the C language.

Tagging values with usWhatever and then verifying that you never mix usBlah with Blah isn't just tedious and error-prone, it smells a lot like doing work by hand that the computer ought to do for you. It seemed to me that it ought to be possible to "tag" unsafe values at the type level and let the language's type-checker sort things out. The simple approach is of course to define something like this:

newtype Unsafe t = Unsafe t

NB: I'm using Haskell because it was the first language that sprang to mind when I thought "type system hack".

This newtype declaration lets you attach an Unsafe tag to any type. Presumably the type signatures of routines that return untrusted data (such as a hypothetical getHTMLFormValue or a routine to retrieve previously entered form values from a database, if they aren't sanitized first) should be modified; for instance, if the signature of getHTMLFormValue was

getHTMLFormValue :: HTMLForm -> String -> String

it should instead read

getHTMLFormValue :: HTMLForm -> String -> Unsafe String

To create a new Unsafe value you simply apply the Unsafe constructor to the value:

getHTMLFormValue = Unsafe (oldGetHTMLFormValue)

To unwrap Unsafes you use pattern-matching:

let (Unsafe unwrapped) = wrapped in ...

So, this sort of works, but it's unsatisfying on several levels:

• It allows the creation of types such as Unsafe Unsafe Integer and even Unsafe [Unsafe Integer]. If a value is already unsafe, how can it be "more unsafe"? And what in the world is an unsafe list, or an unsafe list of unsafe values?
  
  Unfortunately, this is probably a limitation that any scheme that relies on type-tagging will have. You can get around it by either explicitly defining each new unsafe type:
  
  

  newtype UnsafeString = Unsafe String

  
  
  This is arguably a good idea anyway, since virtually all unsafe values (in the sense the previously linked article mentiones) are likely to be strings, and as I note later, it's not clear how to "sanitize" a non-string. OTOH, placing unnecessary type restrictions feels un-Haskellish to me, and it seems likely that you can trust the programmer not to pile up Unsafes.
  
  
• This scheme makes it very tempting to strip off the unsafeness of a value and lose the fact that you did so.
  
  Since the only way to strip off unsafeness is via pattern-matching, it seems to me that it would be tempting to pattern-match away the unsafeness in function declarations:
  
  

  foo (Unsafe x) = ...

  
  
  This may be just fine, but my preference would be to forbid pattern-matching altogether (by not exporting the constructor directly to other modules; just export a normal function that constructs an Unsafe value) and instead providing two routines to remove unsafety:
  
  

  escapeData :: Unsafe t -> t
  unsafeExtractRawData :: Unsafe t -> t

  
  
  The first routine encodes the data into a "safe" form, while the second simply fetches the wrapped value:
  
  

  unsafeExtractRawData (Unsafe x) = x
  
  foo x = doSomeStuffWith (unsafeExtractRawData x)

  
  
  The rationale here is twofold: first, by giving the dangerous extraction routine a scary and hard-to-type name, you remind yourself to think twice before invoking it (cf unsafePerformIO); second, you can easily audit all the locations where the unsafety is "stripped away" with a textual search.
  
  There are some issues involving the safe encoding -- you probably ought to make a typeclass of things that can be safely encoded from an Unsafe value; not too hard and this entry is already too long.
  
  
• The final issue involves manipulation of Unsafe values. So far you can pack and unpack Unsafes; however, this seems like it would get a bit annoying if you had to do any significant operations on them. For instance, searching in an unsafe string for a substring seems like it would be a pain and a bit error-prone (since you'd have to remember to wrap the string back up in an Unsafe). You might think that ideally, all normal String operations should work directly on an Unsafe String or pairs of Unsafe Strings -- you just shouldn't be able to convert between the two.
  
  Unfortunately, I can't see any way of doing everything that you'd want to here. For some types you can. For instance, the following definitions let you compare values wrapped in Unsafe in the usual way:
  
  

  instance (Eq t) => Eq (Unsafe t) where
      (==) (Unsafe v1) (Unsafe v2)   = v1 == v2
      (/=) (Unsafe v1) (Unsafe v2)   = v1 /= v2

  
  
  This admits (Unsafe t) to the Eq class, and looks reasonable enough (assuming that it's safe to compare unsafe values, of course). However, for probably the most important Unsafe values -- Unsafe Strings -- I can't see how to do this. The problem is that String is not a typeclass, it's a synonym for [Char], a list of characters. The important string operations are either defined directly on the String type, or generically on [t]; i.e., "a list of anything". Because there is no typeclass, you simply can't overload these operations to apply to unsafe strings as well as regular strings.
  
  One way around this is to provide an "unwrapping combinator":
  
  

  liftUnsafe :: (t -> t) -> Unsafe t -> Unsafe t
  liftUnsafe f (Unsafe v) = Unsafe (f v)

  
  
  This would automatically convert string operations to unsafe-string operations; not ideal, but reasonable enough. Operators are still a little annoying, but you can mangle their names:
  
  

  (++u) :: Unsafe [t] -> Unsafe [t] -> Unsafe [t]
  (++u) = liftUnsafe (++)

  
  
  So you have to use ++u instead of ++ to concatenate unsafe lists; not too bad.
  

While it may not be perfect -- it only makes sure that you don't accidentally violate safety, and doesn't stop you from deliberately doing something dumb -- this approach feels a lot safer to me than ad-hoc mangling of variable names. I used Haskell because you can do type hacks fairly easily in it, but I don't see why you couldn't play a similar game (albeit much less conveniently) in C++, or even in a dynamic language like Python. Of course, Python would give you runtime errors about safety violations instead of static ones, but that still beats getting runtime security holes.

I would add as a hypothesis that it should be possible to express just about any variable-name-mangling technique whose goal is to "make wrong code look wrong" in the type system, provided your type system doesn't suck; and that doing so is preferable to doing it manually, since it means that a lot of repetitive error-checking can be handled by the computer. And since the computer is better than you are at verifying formal invariants, forcing it to handle them whenever possible is always a good idea.

According to Jeff, I’ve become a smelly Nokia contractor (the reason he stayed with Luis instead of at Mako’s place in Boston, although I’m sure he was referring to Rob Taylor causing the smell). Part of this entails exchanging project schedules with various managers, where time is often talked about in terms of numbered weeks of the year, like: Foo task will be completed by week 42 of 2005. We ran into some problems with varying definitions of these numbers when exchanging schedules with people running on different platforms. Then we realised that even the software we’re using seems to have different ideas about what’s going on too…

• Outlook and Evolution seem to agree, and define the week as starting on Monday, and the first day of the year is always week 1, even if this results in a truncated week. Hence, week 1 of 2006 starts and ends on Sunday 1st January, and then week 2 begins on Monday 2nd, etc. It’s my belief that this is the correct behaviour as defined by some ISO standard somewhere which governs week numbers.
• GtkCalendar gets the week number correct, in that the first day of the year is always in week 1, but varies its definition of the first day of the week depending on your locale. This is presumably so that the display of the weeks is correct, but using it to number the weeks results in bogus week numbers. For me in LOCALE=en_GB it claims that week 1 of 2006 starts on Sunday 1st January, and ends on Saturday 7th January, making my week numbers one lower than the rest of the world.
• Planner uses GtkCalendar for inputting dates, so gets it wrong there as described, but seems to get it wrong in its Gantt chart view in a different way. It has the week starting on Monday, but claims that January 1st 2006 is the last day of week 52 of 2005. This results in week numbers that, match those GtkCalendar give me on weekdays, but differ over weekends, and are still totally bogus when I try and talk about project plans with my managers.

The result is a whole world of pain, and at best causes extreme confusion when we provide documents referring to both dates and week numbers which are inconsistent with each other in their minds, and at worst makes our managers think we’ll have things done a week sooner than we do. I’m not using Evolution at the moment for calendaring stuff, so I can manage if I remember to add one to all my 2006 week numbers when interacting with managers based on GtkCalendar and Planner, but this all seems to be horribly broken. I really don’t want to have to do this for the whole year.

Ray Ozzie: “As an industry, we have simply not designed our calendaring and directory software and services for this ‘mesh’ model. The websites, services and servers we build seem to all want to be the ‘owner’ and ‘publisher’; it s really inconsistent with the model that made email so successful, and the loosely-coupled nature of the web.” Great stuff. Can’t wait to see what comes of this. Personally, I think synchronization is going to be one of the key platform features of Web 2.0. As I’ve said before, I simply don’t buy that we’re just around the corner from ubiquitious network connectivity, and absent that, we’re going to have to deal effectively with the disconnected operation problem.

Note: When reading this entry, please set your date to 2005-11-10. Delays in the system prevented me from posting this earlier.
I have lived together with Nadezhda for almost ten years - And this month (on November 1st, to be precise), it is ten years we first met on one of the few BBSes we had in Mexico by then - Thanks a lot, Mag, for setting it up - you changed my life in more ways I'd ever have thought!
Over the time, of course, our relation has changed a lot - the first step was telling to other people about each other's existence. The second was accepting we were not together only in between (long story... But I was about to leave the country for good. And I did. But then she followed. And we came back together). Then, little by little, we started settling down, we became a formal couple in the eyes of our families and friends.
We always thought social conventions were silly. We laughed about the prospect of marriage, of having to accept in front of others what we felt towards each other - But anyway, some months ago, we started talking it would be nice to make out of our nice, private relationship something serious and formal to commemorate our ten year anniversary.
Not much was told about it for some months... Until by mid-October we realized we would not make it in time if we didn't hurry - So we went to the Registro Civil at Coyoac n and asked for an appointment with the judge. November 1st is an official holiday, so it was not possible to make it right on the same date - We got the appointment for November 9.
Some people might be asking here why weren't they told about this - Well, first of all, as you can see, we didn't have much time to plan this. Second, we decided to take this formal step, but to have it privately. We went only with my father, my brother, Nadezhda's mother, sisters, our nephews and one long-time friend. We were summoned at 10:30AM, had to wait for the judge to marry some other couples until around 12:00PM, had some nice words from him for ~15 minutes, and went for a nice breakfast in a nearby restaurant with the whole gang.
Oh! Something interesting: We actually didn't know when we were formally getting married. We were called some 30 minutes before the official words to sign some papers, which the judge would later, and privately, certify as valid. The judge just talks, he does not sign anything with us present... But, well, it is a nice moment ;-)
And... Well, here is the proof. It took some days to receive the document, it took me some more days to finally post about it - But finally, our acquintances have been informed of our formal step towards a life together :)

I'm not usually a grand fan of the latest and greatest fads'' in programming. The only C# I know is the black key next to D#, and I loathe coffee. An IDE in the hands of a poor programmer will still result in crap code. While I do think that agile methods (such as XP) can do well in certain development environments, I certainly don't subscribe to the notion that XP can solve all the world's software production ills. And so on. I do know quite a number of programming languages, and tend to pick up new ones out of intellectual curiousity (although Ada did put a severe dent in that habit) or because a bit of research tells me that the language might be particularly suitable for a particular task (although I've since learned that the actual purpose of PHP isn't dynamic websites so much as bringing the security levels of Linux boxes in line with Windows-led industry standards). I also learn languages because I want to hack on projects written in them, which is motivational, but not a particularly good way to get a deep understanding of a language. Why all of this introspection in a blog? So you have the background to understand how everything about Ruby is different for me, including how I'm getting started using it. I have no interest in hacking on a Ruby-implemented project, and -- although Rails proponents give me an impression similar to that of a horny teenager with a shiny new unlimited broadband account -- I don't have any particular use for Ruby at the moment (I want to write less webapps, FFS, not more!). So, why the hell did I fork out $90 hard-earned South Pacific pesos for a copy of "Programming Ruby: The Pragmatic Programmers' Guide"? A couple of reasons:

• I'll admit it, Ruby caught my attention because of the hype surrounding it. What made this hype different was that it was coming from people who I knew as being (a) intelligent, and (b) usually immune to the typical hype machines. This suggested to me that the hype about Ruby might be, at the very least, a different sort of marketing bollocks, if not evidence of actual utility. Worth a look, at any rate.
• I had previously read a fantastic book on the general concepts of programming, entitled "The Pragmatic Programmer: From Journeyman to Master", by Dave Thomas and Andy Hunt. I had enjoyed the style of the book, and respected their ideas, and so when they were advocating Ruby as "the language that made programming fun again", and they had a complete tutorial and reference book out on the topic at hand, it seemed natural that I would look at their Ruby book as my "book of first refusal", as it were.

I'm glad I got "Programming Ruby". The book is superbly organised, reads very easily, and gives you all of the love as and when you need it. Surprisingly for me and a technical book, I read the first section of the book "cover to cover". By the end of it, I came away with a real (if naturally superficial) understanding of the language, and a feeling that I was ready to tackle the world in Ruby. It is a sad state of affairs when the first piece of new programming you can sink your teeth into is some four days after finishing your initial studies into a language, but this afternoon I wrote my first Ruby program in anger. It's quite simple -- just take stdin and rewrite it all on stdout with the lines in a random order. But I wrote it in a new programming language, in about 3 minutes, with nothing more than ri (Ruby's answer to firing up python and typing help(xyzzy), more or less) and "Programming Ruby". I think I also produced a suitably reasonable result:

#!/usr/bin/env ruby
lines = []
while line = gets() do
	lines << line
end
	
while ! lines.empty? do
	idx = rand(lines.length)
	puts lines[idx]
	lines.delete_at(idx)
end

(Any Ruby afficionados out there, feel free to e-mail me and tell me what egregious sins I've committed). I think it looks reasonably neat and clean, it's certainly simple, and the use of real Object Orientation makes for a certain readability (I think the method call lines.empty? is particularly cute, although I did initially consider the use of a question mark in a method name to be a crime against humanity). So, will I be using Ruby more in the future? I think so. There are some features in Ruby which I'm hoping to get a lot more cosy with -- the idea of "blocks" looks particularly interesting, and I don't think the utility of languages that have been designed, top-to-bottom, to really live the OO ideal has been explored enough (you SmallTalk bigots can stop glaring at me now -- really, I mean it -- stop it!). It's far too early to tell whether Ruby will become my "go to" language for random development tasks, but anything that stops the nightmares (where I start thinking about using PHP as a general scripting language) cannot be all bad. And, if I continue to make new webapps, Rails will probably come in handy. Update: HTML <pre> tags don't like having < inside them unescaped. Also, Philipp Kern pointed out that the Ruby documentation browser is called ri (not ir as I originally wrote). Philipp was also kind enough to provide a much shorter version of my original script, which shows the power of Ruby so much better:

 lines = []
 STDIN.each_line    line  lines << line  
 lines.sort_by   rand  .each    line  puts line  

Now tell me that isn't a language worth using!

The photo is from yesterday, but it is still new for most of you so the photo-a-day is still going strong for me :)
Today was spend in a very productive fashion - bugfixing. I had a lot of feedback for SBackup over last couple of months - it was time to fix some of the bugs people were reporting. Therefore version 0.9 of SBackup will be hitting you nearest mirror as we speak (I mean, I am uploading it now :)).
This realease is mostly a hey-this-thing-is-still-alive-and-kicking-ouch release and it also includes a TODO file with my plans for its future. I expect more feedback both on features and bugfixes for the stable 1.0 branch and for the planned 2.0 rewrite.
Also I spent couple of hours reviewing Wiki engines with an idea to put one on SourceForge to provide documentation for SBackup - I found no good candidates. I wanted a simple wiki, preferably in python, with MySQL database (SourceForge mounts web directories read-only). Not good. Now I am considering making a page in Wikipedia and using that as a documentation page. Scream at me violently if have any other solutions :)

Where do I start? It's been an eventful couple of days... The flight to LA was uneventful. Qantas is certainly orders of magnitude more pleasant to fly with than United, even though we were seated in a bulkhead area. I had a 13-month old baby seated next to me (no, Mikal was on the other side), and he was about as well behaved as you could possibly ask a 13-month old baby to behave. Mikal also behaved himself. I had the intention of trying to get something approximating 8 hours sleep, as we arrived at LAX at 7am local time, but I only managed about 4 hours. I did spend about 8 hours with a mask on and my brain in idle, so I arrived at LAX being mostly coherent. Entering the US via LAX was a much less traumatic experience than I was expecting. The E-3 visa was accepted fine, and I now have an I-94 stapled into my passport. Quarantine involved getting looked up and down by a guy in a booth, who waved me through. We had 3 hours to spare (or about 2 by the time we cleared all the immigration/customs queues) and then hopped on a 50 minute American Airlines flight to San Francisco, hired a car and drove to Santa Clara. The accommodation we're staying in for a month is very spiffy indeed. It's a largish gated community of units. Mine's got a ground floor entry, but everything else (except the garage) is upstairs. When Sarah gets here, we'll put up some photos. The weather is lovely as well. Our first outing was to the Social Security office to attempt to get social security numbers. This wasn't as successful as I'd hoped, as we were too freshly in the country to show up on their systems, so we left with letters saying we'd get one within four weeks, when our details had been verified. Yesterday evening, we went for a general recce of the surrounding area, and checked out a supermarket. That was less disconcerting than I expected, although I don't think it was a terribly huge supermarket. Having a GPS certainly raises one's confidence when navigating significantly. Last night I discovered that all the keys in my apartment were not created equal, and that the two loose in the lock box on the front door were the only two that opened the unit, and the two sets on key rings on the coffee table only opened the amenities and my mailbox. So I got to spend a bit of time sitting on my front doorstep until the nice man from Synergy Relocations came out to let me in (except his key didn't work either, so he got to make conversation with a legal alien while we waited for someone else to come out with a key). Today, we drove around to see a shopping mall, and the first set of shops we found happened to be around the corner from an enormous (by my standards) apartment complex, which had some signage implying that it had vacancies, so we decided to get the low down on how renting worked before we did our rental tours. Casey drove us around in a golf buggy and gave a tour of what two different floor plans looked liked. It seems that in this part, it's standard for an apartment to include the refrigerator, dishwasher, microwave, washing machine and clothes dryer, which is good to know. Let me just say that if this is what living in an apartment complex is like, it's more like living in a resort. The place had pools everywhere, tennis courts, two gyms, a community centre, it was just unreal. It was also cheaper than I was expecting. We then went to a massive shopping mall to do some window shopping. Discovered Wetzel's Pretzels. Hope they're not too fattening... Tomorrow, I'm going to avail myself to the complex's gym, and generally get ready for the week ahead. Future posts will be in a new category, which I won't be feeding to Planet Debian as not to bore people. I will inflict Planet Linux Australia readers with them until someone asks me to stop. If you'd like a custom feed that excludes my American adventures, please let me know. I wonder if adjusting Blosxom's idea of timezone buggers up existing posts and aggregators like Planet in general?

In another installation of my pointing to Cringely's columns, let me mention yesterday's column speculating about Google. Choice quotes:

[...]
Google hired a pair of very bright industrial designers to figure out how to cram the greatest number of CPUs, the most storage, memory and power support into a 20- or 40-foot box. We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage that can be dropped-off overnight by a tractor-trailer rig. The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid.
While Google could put these containers anywhere, it makes the most sense to place them at Internet peering points, of which there are about 300 worldwide.
Two years ago Google had one data center. Today they are reported to have 64. Two years from now, they will have 300-plus.
[...] There will be the Internet, and then there will be the Google Internet, superimposed on top. We'll use it without even knowing.
[...] All this is based, of course, on Google's proven network and hardware expertise. Have you seen Google's Search Appliance? They ship you a 1U prebuilt server. You connect it to your network, fill out a simple configuration screen, and it scans and indexes your web site (or sites) for you. Google monitors and manages it remotely, and sucks up the data and adds it to theirs. You just plug the thing in and turn it on. It just works. You need do nothing else to keep it running. Google understands how to do this stuff. Microsoft definitely does not.
[...] Microsoft can't compete. Yahoo probably can't compete. Sun and IBM are like remora, along for the ride. And what does it all cost, maybe $1 billion? That's less than Microsoft spends on legal settlements each year.
Game over. And yet next week I'll take it one more step.

Fact? Fiction? Feasible? As they say, read the whole thing.