I just finished rewriting an old firewall tool from Perl to Python. Since it's in Python and about firewalling I dubbed it Pyroman. I investigated a dozen of firewall-tools before, including shorewall and firehol. Each had it's stength and its weaknesses. After writing iptables rules in a shell script every now and then for a more complex project (with like 6 networks of which 3 are bridged together) and two dozen of differently configured hosts, NATs, VPN, everything. My predecessor had written a shell script to configure the firewall, but this was really bad to maintain. So I ended up writing a perl application to generate the rules from a modular configuration (read: usually one file per host, containing a perl hashmap) Well, after happily using this script for two years, I dedcided it's about time to rewrite it and document it extensively. I chose python for the rewrite. You can get the result here: Pyroman 0.1.1. The good:

• Written in pretty python
• Extensively documented (Python docstrings)
• Much faster than sh+awk based solutions
• Really easy syntax to add hosts, nats
• You can add custom iptables rules when needed
• Designed for complex networks
• Lots of verification checks done before execution
• Designed to use the same configuration files on multiple hosts (e.g. failover firewalls or the destination host itself; it will detect if you are talking about a local or a remote host
• It will report file name and line number on parser errors, verification errors and execution errors.
• If any rule fails to setup, a rollback will occur, restoring your previous firewall
• (will likely have auto-rollback if you lock yourself out in the next version with a simple timer)
• Extensive configuration example

The bad:

• Not yet tested in production
• Doesn't completely hide iptables complexity (some core config files are just containing iptables rules, but why invent a new syntax?)
• Only iptables, no TC/Shaping, no IPsec, proxy arp setup, VPN, ifconfig (I use other tools for that, e.g. heartbeat)

To tease you a little more into testing, here's an example host configuration: ("dmz" is an interface alias - where the web server is connected to -, as are "INT", "DMZ" and "ANY" for clients on these interfaces)

"""
A really simple webserver configuration.
These examples are just boring... ;-)
But without NAT they would be even more boring. ;-)
"""
# web server
add_host(
        name="web",
        ip="10.100.1.2",
        iface="dmz"
)
# offering, well, web service.
allow(
        client="ANY DMZ INT",
        server="web",
        service="www ssh ping"
)
# internal hosts may access FTP, too
allow(
        client="INT",
        server="web",
        service="ftp"
)
# setup NAT
add_nat(
        client="ANY INT",
        server="web",
        ip="12.34.56.80"
)

(Yes, this is a python script. No, you probably won't care to write your configuration in a programming language, will you?)

Wikipedia wasn't really "new". Wikis have been around before. There were a dozen of encyclopedia attempts around (although maybe not using a wiki, and usually more focussed around a specific field of knowledge), there were huge collaboration applications. I think that Wikipedia just offered the right options for the users at the very right moment, when this "revolution" - people not using the web as a way to 'download' content as much, but to actually publish their very own stuff, taking over content production - was just kicking off, without being too fragmented yet. Today, you'll find dozen of "encyclopedias" for specific parts, who somewhat try to copy the success of Wikipedia, but which maybe was so successful because it didn't put up much restrictions on what it was to be used for... As for the "not new" part - take for example a look at DMOZ.org, which is for example used by Google as datasource for Directory.google.com. This has been around for years (since 1995?), 5 Million verified Links in there, 70k editors just to verify and organize the links (many just taking care of a small part like the links for their home town), and everybody can submit new links. I started being an editor when it had around 2 Million links, and "timed out" due to inactivity when it reached 3 Million. I have the impression (partially of course to the success of search robots like google, who have also "ruined" the directory business for e.g. Yahoo) that growth slowed down horribly the last years a lot. Although the level you could contribute wasn't that different from what e.g. Wikipedia offers. On the one hand it's a shame that great projects like DMOZ get so little attention, but on the other hand e.g. Wikipedia is fantastic, and I'm happy that many people have realized that they can be an active part of the web, not just a content consumer. That there is more to the web than to share music and access pr0n^Wcommercial websites.

Yet another thread on keyring updates in Debian has gotten me thinking about key signatures, what they really represent, and other associated topics. For those of you who might not know what I'm talking about, a brief primer. Before someone can be admitted as a Debian Developer, they have to have an OpenPGP key (practically, a GPG key) signed by a current Debian Developer. That is, someone who is currently a DD has looked at a photo ID, compared the photo to the face offering a key fingerprint, and compared the name on the ID to the name on the key. If they all match, we have a signature. The other qualification for becoming a Developer is going through a process designed to determine whether you have the required competencies. Since this testing process is conducted electronically, not face-to-face, the primary means of identifying the person behind the answers is via the OpenPGP key they used throughout the process. So, what do we have out of this process? Two things, that I can determine:

• The signature says that someone who Debian already trusts (the current Developer) affirms that the name on the key belongs to someone who has the same name.
• The holder of the relevant key has passed the necessary competence checks. Effectively, the New Maintainer process certifies that a particular key is competent, to all intents and purposes. This is important, because it's that key that gives you all of the "privileges" associated with being a DD -- voting, package upload, etc. Your identity is effectively that key.

So, the problem at issue is this -- what do you do when you lose control of that key? You lose it, or it gets compromised, or it expires (whoops!), or whatever. For all intents and purposes, when you lose control of your key, you no longer have an identity, in Debian terms. Generating a new key (and hence identity) is straightforward -- a quick invocation of gpg and you're away. Unfortunately, all that gets you is a pile of bits. There's nothing that intrinsically links you to that key. That's what other people's signatures on your key is for. Then, it comes time to get Debian to accept your new key as belonging to you. This is a lot harder than trying to convince the RTA (DMV.en_AU('New South Wales')) to give you a replacement drivers' licence. Why should it be? You're doing effectively the same thing -- "I am $FIRSTNAME $LASTNAME, give me my (licence upload privileges)". The difference is that, in the Debian case, all of our identity is effectively tied up in the key we use to go through NM. The person behind the key is somehow secondary to that whole process. So when the key completes NM, we record it in the "people who can upload" keyring. When a new key (which is prima facie completely untrusted) should be put into the keyring, we need to ensure that it is qualified. How can that key prove qualification? Well, we can send the new key through NM (dear ghods, no!) or refer to the source document (as it were) -- the human being. We're looking for consistency between the person between the two keys. That's trickier than it looks at first glance. Why? The big one is the non-uniqueness of $FIRSTNAME $LASTNAME (or even $FIRSTNAME $INITIAL $LASTNAME, for those so blessed). Google for yourself one day, and you'll see what I'm talking about. Anyone who has the same name as you could, at first instance, pretend to be "you" (for some values of, anyway). It might seem fairly unlikely, but there are two DDs with easily-mistakable names, out of a pool of 1,000 or so. The probabilities are not in our favour on this one. Then, of course, there's fake IDs, which are a whole other ballgame. How does the government get around the problem of name clashes and identity-theft-by-ID? By using other metadata about you to more uniquely identify you. Your date of birth, street address, perhaps even a general physical description -- all of these can help to disambiguate all of the identically named people out there. Debian can't use that, though, since that metadata isn't initially collected, and so it can't be rechecked later. This does suggest a bit of a solution, though. I said that Debian doesn't collect that info, but that's not entirely true. There is an optional repository of Developer contact details stored for other DDs to access, so we've got something to work from. The Developer updates their own details (whatever they feel comfortable entering). But, of course, people signing your key need to know about all that before they sign the key, so they can check (for example) your residential address and date of birth on your photo ID. You might ask "but how is the info the developer themselves entered accurate?", and that's a good question. The answer is that the legitimate developer entered that info, so if someone with the same name matches that data, then it's reasonable that it's still the same physical person. It's not a real good solution, though. Not all developers enter full metadata (and I can understand why they don't want to). People also often forget to update their details, too, which could quickly cause all sorts of problems. What is a good solution, then? We could always go with a bit of biometrics -- if there was any solution that was sufficiently cheap, reliable, and distributed enough to be used to identify 1,000 developers distributed across the globe to a reasonable standard of reliability. I'll wait for a minute while you finish laughing and wipe your eyes dry. I have to say, I'm stumped. I'd hoped, while writing this, that by the time I was finished I would have worked something out. But nope, it's a problem that's way beyond me. But I think I've got a much clearer handle on the problem space, so with a bit of luck, my subconscious (or someone else's) will eventually produce a sensible answer.

This entry is mostly for the benefit of Martin-Éric Racine, who is having trouble with Blosxom and dates. I'm forced to use this arcane method of communication, since Martin-Éric threatens to charge for any incoming email. I would like to direct your attention to Blosxom plugins for indexing. They allow, for example, to preserve entry dates even when the entries are modified later.

I claimed yesterday that the council of minister looks very much like an upper house of a bicameral parliament. It is somewhat unusual in that the members are not chosen for a certain term but serve by virtue of holding an office in a member state, but it is not a decisive unusuality. Martin-Éric Racine countered: No, it is not. The council is bureaucracy overruling the Parliament, thus invalidating the very principle of democracy. We have seen this in action with the new EU Patents law: the Parliament had passed several amendments to the original proposal, yet the Council ended up ignoring those democraticly voted amendments and passing an even worse legislation than the original draft. This is clearly different than what happens between a higher and lower chambers each examining new legislation before it is ratified. Wouter Verhelst already said most of what I would have said myself; the bottom line is that Martin-Éric is mistaken about the powers of the council of ministers. Does he by any chance confuse the council of ministers and the commission? It would explain his calling the council a "bureaucracy", which it really is not. The power balance between houses in bicameral parliaments varies. My claim cannot be dismissed just because the two organs do not work exactly like the English parliament, for example. An important thing is that the lower house is almost always directly elected and the upper house comprises often representatives of areas or member states. Finally, I would like to note that the bicamerality of the EU parliament system is independent of my claim that the EU is a country; if a non-country can have a parliament, it certainly can have a bicameral parliament. So you don't have to rebut it just because you don't believe in my claim about the countriness of EU. Oh, BTW, Martin-Éric, please don't mess with your posting dates. It makes your postings to appear more recent in PD than they actually are and to float to the top, where they obscure legitimately new postings by others. For example, your response to my previous posting appears much later in PD than Wouter's reply to it.

Date: Mon, 17 Apr 2000 03:07:13 +0300
From: Antti-Juhani Kaijanaho <gaia@iki.fi>
To: "Debian developers' misc list" <debian-devel@lists.debian.org>
Subject: RFC/ITP: Constitution of Finland

On IRC I proposed packaging the Constitution of Finland.  Among other
things, it would be an example of a good Constitution :-)  The response
on IRC was positive; what do people think here?

Since Finnish laws are specifically not protected by copyright (see
Finnish copyright law), the law is free and can go to main.  There is
an English translation at http://www.om.fi/perustuslaki/3313.htm.
I am researching its copyright status.

Assuming people are okay with this, and the copyright issues around the
English translation are resolved, I will package Constitution of Finland
to woody.

-- 
%%% Antti-Juhani Kaijanaho % gaia@iki.fi % http://www.iki.fi/gaia/ %%%

                    I'm moving IRL on May 2, 2000.
               New contact information on the home page

Do you remember this infamous thread? :) For some reason, it has almost completely been censored off the Debian mailing list archives. I digged my own archive and put the thread online in a mailbox. Have fun with nostalgia and AJK-bashing :) Maybe this will help to reestablish the thread in the archives; who knows?

Just uploaded to experimental a new version of grep-dctrl, 2.3. It can do numeric comparisons:

ajk@kukkaruukku[01:02:34]:dctrl-tools$ grep-available -FInstalled-Size --gt 100000 -sPackage
Package: openoffice.org-bin
Package: atlc-examples
Package: xlibmesa-dri-dbg
Package: vtk-doc
Package: gcc-snapshot
Package: fgfs-base
Package: gcc-experimental
Package: libopenh323-dbg
Package: vegastrike-data

As I start to write this, it's the last minute of my country's 87th birthday (the Finnish Independence Day). It got me thinking about nation-states and independence and whatnot. And I came to a threefold conclusion: First, there is some value in independence. Getting rid of the status of a Grand Duchy in personal union with Russia was certainly a good step for Finland, in light of the rough road the Russians themselves have since travelled. For the same reason, the two wars Finland fought against the USSR were quite worthwhile. Similarly, if Finland were to be threatened by a neighbour again, armed defense would certainly be justifiable. Second, there is no intrinsic value in independence. If a humanly life (including freedom and local government) can be guaranteed even if there is no Finnish state in itself, I see no reason to hold on to the old-fashioned concept of a nation state. Third, I am in favour, as a long-term goal, of a world government. Of course, a single government becomes a single point of failure in case of a coup, for example, but if the global state is arranged properly, the danger of a successful coup is minimal. On the other hand, a world government is in far better position to institute necessary policies to combat the global problems of poverty, climate change and other such issues than the current system of competing and cooperating nation-states. In the current system, the big issues are easily put aside in name of local opportunism. Finland is by no means immune to that problem. The great crises of the 20th Century were its devastating wars. Fortunately, we have overcome them to a sufficient degree. The great crisis of the 21st Century is not terrorism - though it is a symptom. The great crisis is caused by the limits of growth, and we are already feeling quite concretely the nearness of those limits. If something is not done, our current civilization will crash spectacularly. The human race will probably survive - but do we really want to live through another Dark Age? Long Live the Earth!

Weblog Software

I’ve been keeping an online journal over at LiveJournal since 23rd February 2002. To support them I actually bought a “Permanent Account” a while back, an example of putting money behind companies and/or services which are useful - even when it isn’t required. After getting into the swing of things there I started the blog which is used upon Planet Debian and have also been enjoying that. Livejournal is a nice service, and they have great software powering the site. The Livejournal code is available and it is reasonably simple to install. (I’ve setup a few installations of the codebase, and played with them. I admit I’ve never run it in anger). Obviously it is completely overkill to run your own installation of Livejournal - unless, perhaps, you’re a school or a large company. So for my personal blog I use Wordpress. I don’t particularly enjoy using wordpress. To be honest I need only four features/abilities of weblog software:

• The ability to create posts.
• The ability for users to be able to leave comments upon entries.
• The ability to search previous entries.
• The ability to export the blog entries as a feed.

Everything else is utterly overkill. The downside of using Wordpress is threefold:

• Because it is popular there are multiple bots crawling the web and leaving spammy comments.
• Because it uses some funky addressing system any replacement would leave lots of broken links.
• It has historically had a couple of security issues.

I’ve been telling mysefl for the past few months:

• This software has got to go.
• I’ll wait until there is another security hole discovered.

So .. HTML::Template & CGI::Application will at some point come to the rescue. I’m just not sure when. I guess this means I’ve not yet written a bloggin system…. I want to only cover the basics:

• Multiple users.
• Multiple catagories/tags.
• Comments with a sane anti-spam system.

PS - The Sisters of Mercy rock.

This posting is part of my Finnish criminal court series. See the introductory essay for more information and an overview of the Finnish criminal court system. The following is a fictional account of a criminal case, one of the most frequent ones, written in the form of a play script. A PDF is also available. There is also an OpenOffice-generated XHTML renderition, which may be more pleasant to read than the version below (but lacks most of its logical markup). DRIVING WHILE INTOXICATED
A glimpse of Finnish criminal justice
Part One Dramatis personae
Presiding judge
Prosecutor
Defendant Extras
Three lay judges
Court reporter Sets
Courtroom
Courthouse lobby DISCLAIMER This is a work of fiction. Although this play is based in general terms on real life, all the specific characters and events portrayed or described in this play are fictitious, and any resemblance to real people or events is purely coincidental. The verdict presented is fictitious, and while it is based on the author's understanding of the law, it may be erroneus. No legal advice should be inferred from this work of fiction. Scene One: And so it begins The courtroom is about the size of a large school classroom, and shaped as a rectangle in terms of floor layout. The judges' bench, a long wooden table, lays near one of the short walls; the space between the bench and the wall is enough to give the judges and the court clerks comfortable room to sit and to move behind the sitting judges. At the center of the bench is the seat of the presiding judge, who has easy access to a telephone, a public address system and a panel for controlling the LED display visible in the lobby above the courtroom door. To his or her right is the court reporter's seat in front of a PC and in control of recording equipment. To the reporter's right there is room for one additional individual. To the presiding judge's left, there are three chairs for the lay judges. If one were to stand directly opposite the presiding judge, on the other side of the bench, one would find an open Bible. In front of the bench, disregarding the Bible, there is some room, approximately two meters or so. Along each long wall there is a small table, and those two tables face each other, and are therefore turned sideways from the bench's point of view. By the desk on the bench's right is the seat of the prosecutor, and the other, facing desk is reserved for the witness being examined. Facing the bench, separated from it by that empty space, are two rows of tables with chairs. Beyond them there are two rows of seats for the audience without tables; the farther row is right at the other short wall. The long wall to the judges' right contains one double door leading to the court building lobby, and the other long wall contains two sets of double doors at each end of the wall, leading to a part of the building not accessible to the public. We'll call the door closer to the bench the judges' door. The PRESIDING JUDGE is a 50-something man dressed formally in black and blue. The three LAY JUDGES are more casually but still essentially formally clothed. LAY JUDGE ONE is a sixty-something male, LAY JUDGE TWO is a female in her fourties and LAY JUDGE THREE is a female just under thirty. The COURT REPORTER is a female in her fifties, also dressed formally. It is 9:00. The four JUDGES and the COURT REPORTER are sitting on the bench. Each JUDGE has papers in front of him or her, the PRESIDING JUDGE's stack being by far the thickest. The PROSECUTOR, fourty-something man, enters through the judges' door, carrying law books and a large stack of paper. He sits down at his desk. PRESIDING JUDGE (Nodding to the prosecutor.) Okay. Let's start. (Enters codes on the LED machine. Then picks up the public address microphone.) The criminal case 'prosecutor against Matti Meik l inen' begins in room three. Parties to room three. Scene Two: Main hearing The DEFENDANT enters through the double door from the lobby. He feels a little lost. PRESIDING JUDGE (To the defendant:) Matti Vihtori Meik l inen? (DEFENDANT nods.) Please, take a seat in the front row. (He waits for the defendant to comply.) This case will be tried as follows. First, the prosecutor will read the charge. You will then respond by telling us if the charge is correct or not, that is, if you in fact did commit the acts specified in the charges, and if you did not, or the charge is only partially correct, you should state briefly why the charge is in error. The prosecutor will then present the case, he will describe us his understanding of the events in question. When he is finished, you will again have opportunity to state if the prosecutor is correct or not, and if not, in what way is he wrong. If there is disagreement between you and the prosecutor, we will hear the evidence. After that, the prosecutor will state his or her concluding opinion of the case as presented here. Again, you will have the opportunity to respond. We will conclude with discussion of what punishment, if any, would be appropriate. The court will then discuss the case alone, and will finally announce the verdict. Is this understood? (DEFENDANT nods.) Okay. Prosecutor? PROSECUTOR (Reads the charge from a paper everyone has in front of his or her.) I demand punishment to Matti Vihtori Meik l inen for driving while intoxicated. Matti Vihtori Meik l inen had on the first of May this year operated his car while his exhaled air contained 0,597 milligrams of alcohol per liter. Matti Vihtori Meik l inen's driving license must be suspended. PRESIDING JUDGE (To the defendant.) Is the charge correct? DEFENDANT Yes, it is. PRESIDING JUDGE Okay. The prosecutor will now tell us the circumstances and events in detail. PROSECUTOR Thank you. It was to my understanding late evening on May day, and the defendant had come from a bar and was apparently trying to move his car to a cost-free parking space. A police patrol happened to observe this and the defendant had apparently been a little shaky in his walking, so the officers gave him a breathalyzer test, which showed significant alcohol content. The defendant was brought in the station where a precision test was performed. The first reading was 0,597 and the second reading was 0,617; the mean of these two is 0,607, and after performing the confidence subtraction, the reading 0,597 milligrams per liter was obtained. PRESIDING JUDGE (To DEFENDANT) Well then, is what the prosecutor just said true? DEFENDANT (Speaking in a colloquial manner and very somberly. He is ashamed and wishes this would be over soon.) Yeah, it is. PRESIDING JUDGE All right. It seems to me that there are no points of disagreement, so we don't need to hear witnesses. (The PROSECUTOR nods.) Okay, it's time for the concluding statements. I think we will hear consequence opinions at the same time. Prosecutor? PROSECUTOR Thank you. I don't think there is much to say. The case has been shown beyond reasonable doubt, in my opinion. We have the precision test readings and the defendant's admission. As to the penalty... the amount of alcohol we are talking about is just below the limit for serious intoxication, so this is no small deal. On the other hand, the circumstances were not very dangerous. I think suspended imprisonment could well be considered, though a stiff fine would probably be more appropriate. The driving ban must be fairly long, obviously. PRESIDING JUDGE All right. (To the defendant.) Do you have anything to say before we adjourn to consider the case? DEFENDANT Well, I suppose it's clear that something is going to be given to me. I prefer a fine, naturally. PRESIDING JUDGE Naturally. The court will consider all options. Do you have any special requests concerning a driving ban? DEFENDANT As short as possible, of course. PRESIDING JUDGE Do you have a particular need for a car, do you need it for work, for example? DEFENDANT No. PRESIDING JUDGE Okay. Now, your income. We have here the last completed tax report for you, it says you earned 500 euros a month and had no children. Has this changed? DEFENDANT No change. PRESIDING JUDGE Okay, thank you. If you will then go outside to wait while we consider this case, thank you. The DEFENDANT exits to the lobby, the PROSECUTOR exits through the judges' door. Scene Three: Deliberation We are now in the lobby outside the courtroom. There are many doors that lead to many courtrooms. Above each door, attached to the wall, is a display showing the current case number; there is also a light saying do not enter, which is lit for most of the rooms, including Three. Also there is a numeric display which is out at the moment for most of the rooms. For some rooms, they indicate when the session will resume. Each courtroom door also has a paper pasted to the door frame listing the cases for the day. The door leading outside is separated from the lobby by a routine security check area, and one of the courthouse police guards stands there waiting for people to enter. Some other guards and janitors sit in their glass booth. There are many people sitting on the chairs, waiting for their turn to enter one of the courtrooms. DEFENDANT enters from the Courtroom Three door and sits down, waiting. It takes about fifteen minutes. During this time, announcements from other courtrooms could be occasionally heard, and people would enter or exit other courtrooms. Then... PRESIDING JUDGE (Over the PA) Verdict in the case 'prosecutor against Matti Meik l inen' in Courtroom 3. DEFENDANT exits through the Courtroom Three door. Scene Four: The Verdict The DEFENDANT enters from the lobby and sits down. After a minute, the PROSECUTOR rushes through the judges' door. PRESIDING JUDGE (Announcing to all people present.) The District Court has examined this case and has decided that Matti Vihtori Meik l inen did in fact commit the crime he was accused of in the way specified in the charge. The justification is the combination of the measured alcohol content in Meik l inen's breath and his own admission of guilt. Matti Vihtori Meik l inen is therefore sentenced to pay sixty-five day-fines, that is three hundred and ninety euros. Penal code, Chapter 23, Section 3. His driving license is suspended until the first of November this year. Road Traffic Act Section 75 Paragraph 1. One who is dissatisfied with this verdict can appeal it to the Vaasa Apellate Court within thirty days. Dissatisfaction must be declared, however, either right now to me or within a week to the Office of this District Court. (To DEFENDANT) Come and take the fine ticket and written appeal instructions (DEFENDANT takes those from the COURT REPORTER.) This hearing is concluded. The DEFENDANT leaves, as the court and the prosecutor prepare for the next case. TO BE CONTINUED
(with the next case)

I spent the whole weekend looking for a DRM-encumbered Sony CD so that I could check for myself whether my code was really being redistributed without permission. I eventually found one: Suspicious Activity?, by The Bad Plus. A rather enjoyable post-modern jazz album. I am rather lucky, it could have been C line Dion. As expected, installing the CD s custom player also installed a stealth aries.sys driver along with a few other interesting files such as $sys$DRMServer.exe, all hidden in a $sys$filesystem directory. I rapidly got rid of this cruft and started studying the really important file, ECDPlayerControl.ocx. It is the file Sebastian Porst and Matti Nikki found to be containing parts of mpglib, LAME, faad2, VLC... Evidence that the DRMS code comes from VLC There are of course obvious similarities between some functions and structures of ECDPlayerControl.ocx and the ones in VLC s source. However, given how precise the implementation of a cryptographic protocol needs to be, the possibility that two separate implementations show many similarities must not be dismissed. The data structures and constant tables especially are likely to be identical. I think however that I found strong enough evidence that this is not the case here. The original drms.c was written by Jon on 01/05/04. I then hacked on that file on 01/18/04 and reorganised it, also replacing the reverse-engineered MD5 and AES functions with clean implementations, then Jon added DRMSv2 support on 05/05/04 and I reorganised the code again on 05/08/04. Some of these changes were specific to VLC (not in that they made the code unusable outside of VLC, but rather that they made it possible to use an external MD5 hash provider, for instance), so they were the bits I was looking for in ECDPlayerControl.ocx. For instance, Jon s original code had the following instruction:

p_acei[ 4 ] = 0x5476212A;

Since all bytes of this long word are comprised between 0x20 and 0x7f, it was later rewritten like this:

char p_secret1[] = "Tv!*";

This change makes the code more human-readable because many other secrets in the Apple DRMS protocol are ASCII strings. Also, I know how to look for binaries that use the string "Tv!*", but it takes me some more time to think of a way to find binaries that use 0x5476212A. And the Sony rootkit uses the following (only relevant lines shown):

 .text:100883B8   mov   cl, byte ptr ds:xxxxx+4 ; 0
 .text:100883C2   mov   eax, dword ptr ds:xxxxx ; "Tv!*"
 .text:100883C8   mov   [esp+70h+yyyyy], eax
 .text:100883DC   mov   [esp+7Ch+zzzzz], cl

The use of xxxxx+4 shows that ECDPlayerControl.ocx uses a string instead of a 32 bits integer to store the secret, disregarding the fact that the 5th character of the string (a null char) is never used. Another example: in this commit I reorganised the DoExtShuffle() function, merging FourthPass() and FifthPass() and discarding a structure allocated on the stack. What is important here is that DoExtShuffle() was built by a complex combination of choices, the main purpose of which being to reduce the size of the C functions:

 static void DoExtShuffle( uint32_t * p_bordel )
  
     uint32_t i_ret;
     i_ret = FirstPass( p_bordel );
     SecondPass( p_bordel, i_ret );
     ThirdPass( p_bordel );
     FourthPass( p_bordel );
  

Note how i_ret is not reused and could be merged into the p_bordel array, or maybe SecondPass() merged into FirstPass(). This choice is definitely a legacy of how the code evolved. And the Sony equivalent: a perfect match.

 .text:10089F3C xxxxx:
 .text:10089F3C   mov   ecx, esi
 .text:10089F3E   call  yyyyy
 .text:10089F43   push  eax
 .text:10089F44   mov   eax, esi
 .text:10089F46   call  zzzzz
 .text:10089F4B   add   esp, 4
 .text:10089F4E   call  ttttt
 .text:10089F53   call  uuuuu

After having studied the Sony code for a while, I have of course gathered dozens of such examples. But I also discovered a few new things. Where does the DRMS code come from? The code undoubtedly comes originally from VLC, but it has traveled a lot. I would be surpried if it came directly from VLC, as no other part of VLC is included with the rootkit. And although there are parts of FAAC in the Sony code and FAAC includes VLC s drms.c, the version currently in the FAAC CVS is horribly outdated. Sebastian Porst also noticed a few obvious differences between the Sony code and the version of drms.c present in VLC. The most obvious being this one:

 if( p_shuffle->i_version == 0x01000300 )
  
     DoExtShuffle( p_bordel );
  

Which became:

 .text:10089F2E   cmp   eax, 1000300h
 .text:10089F33   jz    short DoExtShuffle
 .text:10089F35   cmp   eax, 1000400h
 .text:10089F3A   jnz   short skip
 .text:10089F3C DoExtShuffle:
 .text:10089F3C   ...
 .text:10089F58 skip:
 .text:10089F58   ...

That 0x01000400 is a version check for 4th generation iPods firmwares. I also discovered at least 40 KB of new lookup tables that seem to be used for more buffer shuffling. At first I thought it was simply F4I s DRM implementation, but the shuffling calls are nested with iPod hardware information retrieval from our DRMS code, so they re really part of drms.c. Sony distributing a DRMSv3 descrambler? The only explanation I can think of is that someone took the VLC code, added support for an upcoming Apple DRMS format version and redistributed the software without letting the VLC authors know, probably violating the GPL (I only say that because I have not found the software yet, but I feel pretty safe in saying it). Since it is based on GPL software, I was already entitled to ask Sony for the source code of ECDPlayerControl.ocx. But as the copyright holder of a significant part of the software they counterfeited, maybe I have a greater chance of being heard. Not that it would be difficult to reverse-engineer it yet another time, but it would be awesome if the next opensource iTunes Music Store file player was contributed by Sony!

WARNING: nothing in this post has been implemented or tested, except maybe at a minimal proof-of-concept level ("it compiles!"). It's just some musings on how you can make the computer do your work for you.

I was reading a defense of Hungarian notation when I started to get a familiar notion tickling the back of my brain. The author was explaining that the original idea behind Hungarian was not to replicate type information in variable names, but rather to include some sort of usage information: one example would be giving a string value holding a value that is untrusted the name utName. While it's more reasonable than lpschwzMyVariable, this suggesion made me suspicious that the author was really hacking around suckiness in the C language.

Tagging values with usWhatever and then verifying that you never mix usBlah with Blah isn't just tedious and error-prone, it smells a lot like doing work by hand that the computer ought to do for you. It seemed to me that it ought to be possible to "tag" unsafe values at the type level and let the language's type-checker sort things out. The simple approach is of course to define something like this:

newtype Unsafe t = Unsafe t

NB: I'm using Haskell because it was the first language that sprang to mind when I thought "type system hack".

This newtype declaration lets you attach an Unsafe tag to any type. Presumably the type signatures of routines that return untrusted data (such as a hypothetical getHTMLFormValue or a routine to retrieve previously entered form values from a database, if they aren't sanitized first) should be modified; for instance, if the signature of getHTMLFormValue was

getHTMLFormValue :: HTMLForm -> String -> String

it should instead read

getHTMLFormValue :: HTMLForm -> String -> Unsafe String

To create a new Unsafe value you simply apply the Unsafe constructor to the value:

getHTMLFormValue = Unsafe (oldGetHTMLFormValue)

To unwrap Unsafes you use pattern-matching:

let (Unsafe unwrapped) = wrapped in ...

So, this sort of works, but it's unsatisfying on several levels:

• It allows the creation of types such as Unsafe Unsafe Integer and even Unsafe [Unsafe Integer]. If a value is already unsafe, how can it be "more unsafe"? And what in the world is an unsafe list, or an unsafe list of unsafe values?
  
  Unfortunately, this is probably a limitation that any scheme that relies on type-tagging will have. You can get around it by either explicitly defining each new unsafe type:
  
  

  newtype UnsafeString = Unsafe String

  
  
  This is arguably a good idea anyway, since virtually all unsafe values (in the sense the previously linked article mentiones) are likely to be strings, and as I note later, it's not clear how to "sanitize" a non-string. OTOH, placing unnecessary type restrictions feels un-Haskellish to me, and it seems likely that you can trust the programmer not to pile up Unsafes.
  
  
• This scheme makes it very tempting to strip off the unsafeness of a value and lose the fact that you did so.
  
  Since the only way to strip off unsafeness is via pattern-matching, it seems to me that it would be tempting to pattern-match away the unsafeness in function declarations:
  
  

  foo (Unsafe x) = ...

  
  
  This may be just fine, but my preference would be to forbid pattern-matching altogether (by not exporting the constructor directly to other modules; just export a normal function that constructs an Unsafe value) and instead providing two routines to remove unsafety:
  
  

  escapeData :: Unsafe t -> t
  unsafeExtractRawData :: Unsafe t -> t

  
  
  The first routine encodes the data into a "safe" form, while the second simply fetches the wrapped value:
  
  

  unsafeExtractRawData (Unsafe x) = x
  
  foo x = doSomeStuffWith (unsafeExtractRawData x)

  
  
  The rationale here is twofold: first, by giving the dangerous extraction routine a scary and hard-to-type name, you remind yourself to think twice before invoking it (cf unsafePerformIO); second, you can easily audit all the locations where the unsafety is "stripped away" with a textual search.
  
  There are some issues involving the safe encoding -- you probably ought to make a typeclass of things that can be safely encoded from an Unsafe value; not too hard and this entry is already too long.
  
  
• The final issue involves manipulation of Unsafe values. So far you can pack and unpack Unsafes; however, this seems like it would get a bit annoying if you had to do any significant operations on them. For instance, searching in an unsafe string for a substring seems like it would be a pain and a bit error-prone (since you'd have to remember to wrap the string back up in an Unsafe). You might think that ideally, all normal String operations should work directly on an Unsafe String or pairs of Unsafe Strings -- you just shouldn't be able to convert between the two.
  
  Unfortunately, I can't see any way of doing everything that you'd want to here. For some types you can. For instance, the following definitions let you compare values wrapped in Unsafe in the usual way:
  
  

  instance (Eq t) => Eq (Unsafe t) where
      (==) (Unsafe v1) (Unsafe v2)   = v1 == v2
      (/=) (Unsafe v1) (Unsafe v2)   = v1 /= v2

  
  
  This admits (Unsafe t) to the Eq class, and looks reasonable enough (assuming that it's safe to compare unsafe values, of course). However, for probably the most important Unsafe values -- Unsafe Strings -- I can't see how to do this. The problem is that String is not a typeclass, it's a synonym for [Char], a list of characters. The important string operations are either defined directly on the String type, or generically on [t]; i.e., "a list of anything". Because there is no typeclass, you simply can't overload these operations to apply to unsafe strings as well as regular strings.
  
  One way around this is to provide an "unwrapping combinator":
  
  

  liftUnsafe :: (t -> t) -> Unsafe t -> Unsafe t
  liftUnsafe f (Unsafe v) = Unsafe (f v)

  
  
  This would automatically convert string operations to unsafe-string operations; not ideal, but reasonable enough. Operators are still a little annoying, but you can mangle their names:
  
  

  (++u) :: Unsafe [t] -> Unsafe [t] -> Unsafe [t]
  (++u) = liftUnsafe (++)

  
  
  So you have to use ++u instead of ++ to concatenate unsafe lists; not too bad.
  

While it may not be perfect -- it only makes sure that you don't accidentally violate safety, and doesn't stop you from deliberately doing something dumb -- this approach feels a lot safer to me than ad-hoc mangling of variable names. I used Haskell because you can do type hacks fairly easily in it, but I don't see why you couldn't play a similar game (albeit much less conveniently) in C++, or even in a dynamic language like Python. Of course, Python would give you runtime errors about safety violations instead of static ones, but that still beats getting runtime security holes.

I would add as a hypothesis that it should be possible to express just about any variable-name-mangling technique whose goal is to "make wrong code look wrong" in the type system, provided your type system doesn't suck; and that doing so is preferable to doing it manually, since it means that a lot of repetitive error-checking can be handled by the computer. And since the computer is better than you are at verifying formal invariants, forcing it to handle them whenever possible is always a good idea.

I have high respect for his writings, all I read were worth it. He just wrote about Web 2.0, and I find this a very interesting read. I agree with him here, especially with the line "Web 2.0 means using the web the way it's meant to be used." Ajax is not fundamentally new. It's just JavaScript. It's still slow, has sometimes usability problems, and often just doesn't work right... And Ajax certainly isn't worth to be called "Web 2.0"... What has been reappearing this year are "third party hacks" on the net. That was also the biggest thing about Google Maps: the API - others could use it (for free) to make other cool stuff with it. This is part of what Graham calls "Democracy" and of what he calls "Don't maltreat users" (especially not "lead users" and developers if you want them to adopt your stuff!) But basically I still don't think we're at anything fundamentally new here. The API stuff for example makes me think of OpenSource vs. Freeware... getting stuff for free is nice, but being able to modify it to exactly fit your needs is priceless. ;-)

Well, there's nothing like posting a really rubbish program to your blog to find out the right way to do it. Firstly, Philipp Kern has come up with another, even shorter version than his version I posted in the update to my original Ruby post, this time blowing me away with the strangeness of explicit casts in Ruby. Apparently, casting an input stream to an array produces an array with one-line-per-element, as in this example:

 STDIN.to_a.sort_by rand .each l puts l 

As Philipp warns, however: "Ruby is full of fun and power; one must watch out that the scripts remain readable". For a fairly toy example, this solution is still fairly readable, however I can see hair growing on this thing if your program grows large. Next, Pierre-Charles David showed his qualifications for becoming a co-author of the next edition of the Pickaxe (AKA "Programming Ruby: The Pragmatic Programmers' Guide"), with his example and explanation. He's kindly given permission for me to reproduce his e-mail in it's entireity, since it's too good to chop up:

Here is the shortest and most readable version of your "randomize stdin" script I can think of:

  #!/usr/bin/env ruby
  print ARGF.readlines.sort_by   rand  
 

ARGF is a special variable representing either stdin or, in the case where file names are passed on the command line, the content of these files (think of Perl's <>). ARGF behaves like an IO object (although I think it is not one). In particular, it responds to the #readlines method which splits the receiver's content into lines and returns an array. In other words,

  lines = ARGF.readlines
 

is the idiomatic way of doing

  lines = []
  while line = gets() do
    lines << line
  end
 

in your example. The next part (sort_by) uses a new form of the sort method, which I think is not documented in the Pickaxe. It implements the Schwartzian transform technique. In essence, a random number is associated to each element of the array, which are then sorted based on this index. As a general style remark, note that although Ruby supports "while" and "for x in y" loops, they are rarely used, in favor of iterators and/or higher order methods. For example, to gather all the input lines in an array without using readlines, one might write

  lines = Array.new
  ARGF.each_line    l  lines << l  
 

or

  lines = ARGF.inject([])    lines, l  lines << l  
 

Hope it helps, and welcome to Ruby!

Pierre, your explanation definitely helped, and I hope it helps others too. The third person to help me out is Decklin Foster, through a post in his blog. What particularly caught me about his solution was the neat demonstration of another feature of Ruby which I think I will really learn to love: the ability to dynamically add methods to an existing class. "Aiee!" I hear all the purists scream -- and I can see the potential for severe misuse, too. But, since every good idea seems to get deeply perverted, we may as well have some real useful features to pervert while we're at it. Decklin has the clever idea of adding methods to the Array class that shuffle the elements of the array in a random fashion. His code is beautiful in it's simplicity:

 class Array
    def swap!(a, b)
        self[a], self[b] = self[b], self[a]
        self
    end
    def shuffle!
        each_index do  i 
            j = i + rand(length - i)
            swap!(i, j)
        end
        self
    end
 end

He then goes on to rewrite my bumbling example in probably the most comprehensible manner I've yet seen:

 STDIN.readlines.shuffle!.each    line  puts line  

Decklin writes, "Now that is cool.". I couldn't have put it better myself. My thanks to Philipp, Pierre-Charles, and Decklin, for helping me along the Ruby path. So, two new Ruby Rules for me to live by:

• Whenever you reach for a while, first take a good look around for any iterator methods which might apply.
• Never dismiss the power of defining your own methods on an existing class if it will improve the comprehension of your program.

Hmm, is it possible that Ruby is nothing more than Functional Programming by Stealth? Aiee!

I saw Ruby the Rival via an article on Lambda the Ultimate, and of course it tickled my interest in the diffusion, rise, and decline of programming languages. The guys they picked to interview have pretty good commentary - I found it thoughtful and insightful. Robert Cooper's comments to the effect that you should do the easy things first and make the complex stuff available are spot on. Simple stuff has got to be simple in order for the language and its libraries and applications to scale down, and without scaling down, you leave yourself vulnerable to someone who is willing and able to get people started with something easier than what you're offering. Bill Venners points out that the Rails folks have done a very good job of marketing what they've done. Agreed. It's been hard to see the opposite effect in the Tcl community, which has some really good technology that has been sold very poorly. Speaking of which, James Duncan Davidson says Ruby doesn't have a "compelling" gui toolkit. It has Tk though, and if they do their homework, will soon have Tile. And that's not bad... maybe it's not the glitziest thing out there, so I could see why it might not appeal to a Mac guy, but on the other hand, there are countless very compelling desktop apps waiting to be written that are interesting to people for what they do, not just because they are slick looking. And frankly, it's not like Java is that hot in the gui space in any case... Mini-Language Only tangentially related to the above, I found this while looking at ofbiz, which looks to be a pretty good system for what it does. However, the "scripting language" doesn't really make sense to my way of doing things. Look at the example towards the end: http://www.ofbiz.org/docs/minilang.html That seems like an awful lot of verbosity for something that doesn't absolutely need to be edited by a computer.

Idle thought of the day: How does licensing interact with standards - how does the presence of a standard affect which license you should choose for an open source project? We know how the GPL and BSD licenses work, and how the differ... how each can work in different situations with different community and business goals. But... let's throw another factor in - an open specification/standard that is well defined and doesn't have a lot of "wiggle room" for real innovation. Should you choose the maximum amount of openness (BSD) because a proprietary implementation doesn't have that much room between you and the "ceiling" (100% spec compliance), so there is no real reason to make a proprietary fork? Or is the whole thing a moot point because most specs are usually big things that leave a lot of room underneath for innovation. It would be ridiculous to reduce Linux or FreeBSD to "just an implementation of POSIX and a few other Unixy specs", for example. The uptake of the standard probably weighs on the equation too. If it's something you just want to see as widely used as possible (say, TCP and IP), the BSD license is a good one to get the code into a lot of people's hands. An already adopted standard in wide use with no free implementation might have other implications, depending on what your goals are - if you did a nice implementation, perhaps you could corner some licensing fees by dual-licensing the code as GPL/proprietary. Perhaps it's not that important a factor in the mix... what are your thoughts?

KDE

I have never once written anything using Qt. Why not? I have no real dislike of the toolkit, license, or implementation language. It just hasn’t ever been necessary or appropriate for me to use it. Generally speaking I only dabble with coding online applications in Perl/PHP (Perl is my preference, sometimes I get suckered into using PHP). Any software which I’ve needed to write with a GUI usually ends up being coded using the raw X11 libraries (xlib, etc). A few times I’ve needed to have something be stable, sharable, and GUI-driven and xlib didn’t cut it I’ve chosen to use the GTK libraries. Mostly that has been down to three reasons:

• Finding small examples of code is very simple; as it is well-used.
  • By contrast many Qt applications are larger pieces of code. Finding examples using the functions you’re interested in is harder.
• I love C++, but generally I develop in Perl, or C.
• I don’t have to worry about the moc preprocessor, writing Makefiles, etc.

I’m sure that sometime in the future I will experiment with Qt, and possibly even install KDE to try it out. But it won’t be soon. (I think that desktop environments are a good thing, and can be attractive, productive, and useful. However I’ve only used Gnome briefly and tend to stick with the more lightweight IceWM environment). Once upon a time I did install and tinker with KDevelop, but nothing much came of it. When I lost my desktop machine due to hardware failure I didn’t feel the need to re-install it. Instead I stuck to using GNU Emacs + Vim for editing code.

Right now I’ve written that I shall go back to shaving my head. To make me look more like my avatar.

Coding Mistakes

I broke my debian administration website yesterday. Predictably enough I’d woken up around 9AM and thought that it would be a good time to make significant refactering changes. Tested them on my laptop, committed them to CVS, then updated the live site. Had a quick play to make sure nothing had obviously broken - but didn’t try to leave a comment on an article. Went to sleep and woke up around 4pm to find 15 emails telling me things were broken. Most mails on the site in a single day! Should be fixed now. But I’ll never again make changes before lunchtime, I’m just not good with the whole sleepy-coding thing.

Things I have never coded: 1

Inspired by Joey Hess’s “Ten years of free software”. Things I have never coded part 1. I have never coded a text editor. *shock* In the past I have written simple wrappers around the various text controls in different environments - for example under Windows I once coded a tabbed text editor called MyNotepad, which featured a novel dockable tree-control to the side which allowed navigation and loading of files easily. When I first started working with Java I coded a similar thing, but with the ability to load external Java .class files as plugins. This editor was called Rosetta, inspired by the band Rosetta Stone (who were in turn inspired by the actual Rosetta stone ) At times I think about text editors. I use them daily, obviously, and I love working with some more than others. My editor is usually GNU Emacs but I’m also partial to Vim. I will use both happily; although neither is pleasant without my personal customizations. Simple text editors are a traditional project for coders, and freshmeat is littered with their discared corpses. Probably the only reason I’ve not started - that and apathy. My ideal editor would be:

• Both curses and x11 based.
• Scriptable (Lua is love?)
• Extensible.
• Portable between (at least) GNU/Linux and Solaris.
• Modeless.