Here’s a guide to supporting Xen servers for people who are not Linux experts. If your job means that you have root access to a Xen server that someone else installed for the purpose of fixing problems when they are not available then this will help you solve some common problems. Xen is a virtualization system that is primarily used for running Linux virtual machines under a Linux host. It is mostly used as a Paravirtualization system in that the virtual machine knows that it is running in a virtual environment - this allows some performance benefits. The host environment is known as Dom0 and root in that domain has the ability to control the other domains (which are known as DomU domains). If you perform an orderly shutdown of the Dom0 (via the shutdown or reboot commands or notification from the UPS of an impending power failure) then when the machine is booted again the DomU’s will be automatically restarted (if the on_reboot setting has the value restart - a common configuration). If you run the command shutdown in a DomU then the domain will be destroyed, and the command reboot will restart the DomU with the same settings - if you want to change the settings for a DomU you need to shut it down and create a new instance. The main sys-admin command related to Xen is xm. Here are the main xm options that are useful in support:

1. xm list provides a list of running domains. For each domain it gives the name of the domain, the ID number, the memory allocated to it, the number of virtual CPUs allocated to it, the state, and the amount of CPU time used in execution. The ID numbers are allocated sequentially, if you reboot a DomU by running the command “reboot” inside it then it will get a new ID number when it re-starts. Many xm operations that may take the name of a domain will also take a Domain ID number. Generally you never use an ID number and ignore it - the only relevant thing about an ID is whether it is 0.

Here is a sample of the output of xm list:
# xm list
Name        ID Mem(MiB) VCPUs State  Time(s)
Domain-0    0    1236    4 r---–  14116.3
wind        13    2999    3 -b----  60114.1
wind-f7    52      519    1 -b----  3329.9 You can see from this output that the domain named wind has 2999M of RAM, 3 virtual CPUs (out of 4 physical CPUs in the machine) and has 60,114 seconds of CPU time used (that is 114 minutes of CPU use - the equivalent of almost two hours for a single CPU). Here are the values you might see in the state field (from the man page xm(1)):

• r - running
  The domain is currently running on a CPU - note that Dom0 will always appear to be running because you are running the xm utility!
  
• b - blocked
  The domain is blocked, and not running or runnable. This can be caused because the domain is waiting on IO (a traditional wait state) or has gone to sleep because there was nothing else for it to do.


• p - paused
  The domain has been paused, usually occurring through the administrator running xm pause. When in a paused state the domain will still consume allocated resources like memory, but will not be eligible for scheduling by the Xen hypervisor.


• c - crashed
  The domain has crashed, which is always a violent ending. Usu ally this state can only occur if the domain has been config ured not to restart on crash. See xmdomain.cfg for more info.


• d - dying
  The domain is in process of dying, but hasn t completely shut down or crashed.

If you see domains that are running which normally aren’t busy then make a note of this. If you see domains that are paused, crashed, or dying then contact the sys-admin. Also know which domains are expected to be running so that if a domain is missing then you will recognise it as a problem!

xm top is similar to the top command in Unix but displays Xen data, by default it displays the same information as xm list but also includes the amount of data read and written from network devices and disks. If your terminal is less than about 145 columns wide the lines will wrap and it will be confusing - stretch the width of your xterm before running it.

If you have multiple network interfaces then you can see the transfer counts for each of them separately by pressing the N key. If you have multiple network interfaces in DomU’s then this can help diagnose some network problems (although you may find that tcpdump is more useful). If you have multiple disk devices in a DomU then you can see their transfer counts separately by pressing the B key. One problem that can be partially diagnosed through this is excessively poor performance. If a DomU is running extremely slowly then it may be impossible to login to diagnose and/or fix the problem (it could take tens of minutes to login), in that case seeing where the disk access is going from outside the DomU can shed some light on the problem. VBD  768 [ 3: 0]
VBD  832 [ 3:40]
VBD 5632 [16: 0]
VBD 5696 [16:40] Above is the identification of the virtual devices /dev/hda and /dev/hdb in a DomU. The numbers inside the brackets are the device node numbers in hexadecimal, so 16:40 means the device 22,64 as a pair of decimal numbers (22*256+64=5696). # ls -l /dev/hd?
brw-r---– 1 root disk  3,  0 Jul 23 17:24 /dev/hda
brw-r---– 1 root disk  3, 64 Jul 23 17:24 /dev/hdb
brw-r---– 1 root disk 22,  0 Jul 23 17:24 /dev/hdc
brw-r---– 1 root disk 22, 64 Jul 23 17:24 /dev/hdd Above is the result of a ls -l on the devices in question from inside the DomU. When I set up a Xen DomU I generally use /dev/hda for the root filesystem and /dev/hdb for swap. So if the machine is performing poorly and /dev/hdb ([3:40]) is being accessed excessively then it indicates that the machine has some memory hungry programs running and is paging heavily.

xm list –long [domain] gives detailed information on all domains, or it can be run with the name of a domain such as xm list –long wind to give the detailed information on only one domain. Generally this is something that you will log to a disk file before restarting domains, in the short-term there is little use for this.

xm console <domain> gives you the console of a domain. If a domain is not working correctly and it is impossible to login via ssh (either due to a network problem or a problem with ssh) then you can access the console (equivalent to a serial-port login on physical hardware) to diagnose the problem. Often the kernel will log messages to the console, such messages will be stored by the Xen system until they are read. If you suspect that there may be many such messages then use script(1) to log the output to disk, if you are unsure then use script to make sure that you don’t miss any data. Even if you don’t understand it the sys-admin probably will!

If the system is half-working then you can login as root to investigate problems. You can escape from the console by pressing CTRL-].

xm dmesg gives Xen logging data comparable to the dmesg command in Linux. If you ever have to reboot the machine (run reboot from Dom0) due to a problem with Xen then you MUST save the output of xm dmesg to a file for later review by the sys-admin.

xm destroy <domain> will kill a specified domain. It’s a last resort for stopping a domain that is not working correctly - it is greatly preferrable to login to the domain via ssh or xm console and give an orderly shutdown.

xm create [-c] <domain> creates a new domain. The configuration for the domain will be taken from a file of the same name in the current directory or in the /etc/xen directory - if /etc/xen is not the current directory when you run xm create then make sure that there is no file-name conflict. You can use this command after destroying a domain or to start a domain that was not previously run.

If you want to change a configuration option of a domain (such as the amount of RAM used) then the usual procedure is to edit the configuration file, run halt or shutdown from within the domain, and then create the domain again with xm create. Note that the -c option is used to attach to the console after starting the domain (you usually want to do this).
I will probably update this post when I get some feedback. I may write more posts of a similar nature if there are requests.

Share This

Joey, I see a huge difference between voting on the current DM proposal and your virtual (sorry, I can’t remember a better word for it) voting on sponsoring uploads. Debian Developers were always allowed to upload any package they saw as fitting into Debian (though there were no guarantees for these packages to actually being included). Also, NMUs are OK if the maintainer agreed. So all they (seemingly) did was putting a different maintainer address into the control file. However, at least in theory, the sponsors were still responsible for their upload if it turned out fishy. The DM proposal (don’t get me wrong there, I generally support the idea, I just don’t like the way it is implemented in the proposal) is shifting this responsibility from the sponsor onto the shoulders of the DM, which is a good thing. However I think it doubles some of the work the NM queue already needs done, so my proposal would be not to have it as a completely seperate path for a contributor, but instead see it integrated into the NM process. Wether it would mean adding a new maintainer to the DM keyring after he had an advocate (which is pretty near to the current proposal) or after an AM was assigned and checked some basic things (which I would prefer) doesn’t make a lot of difference to me.

Share This

Following up to Myon’s post, I also wanted to blog my honest opinion about the GR. Given my current understanding of the GR, I hope to see it fail (and will send my vote shortly after this blog entry I think). The reason is not that I think it shouldn’t be made easier to contribute to Debian (on the contrary). But the GR micro-manages too much and on the other hand doesn’t make the current NM process any easier for DM’s than for any non-DM applicant to the NM process. Also, like some others said before, I seriously doubt that a big part of the current DD-applicants (i.e. those in the NM queue at some level) would prefer DM status over DD status. And the same is true (IMHO) for other people wanting to contribute to Debian. This is a simple thing: If I contribute to something regularly in my spare time (like most contributors in the FL/OSS world do), I also would like to have some influence over its directions. And a DM is missing this influence even more than a DD is. So what should be done instead in my opinion?
I think that a DM like status is fine as part of a (possibly re-designed) NM process. Given the current NM process, I would like people been given a DM like status after they finished the T&S (tasks^IIRC and skills) part if their AM found their skills and knowledge of the policy rules sufficient to give them that status. If that is all they wanted, they can drop from the NM process at that point and stay DMs. If they want to become DDs later on, they can resume the NM process at that very step (or continue right ahead). This would do several things:

• introduce the DM status as an alternative to full DD status
• integrate the DM status closer with the existing NM process
• take some load of the friendly sponsors who upload non-DD packages now, giving them more time to help more new contributors
• give people waiting for DD status some intermediate level at which they already can do more than before their joining of the NM queue

So I really think “more discussion” is the only valid option for me in this GR. If you want to change my mind: Please leave some feedback in my blog. Or give a precise (but brief) description why the current proposal is better than integration with the current NM process in one way or the other. Abbreviations:

• DD = Debian Developer (a full Debian project member, includes voting rights)
• DM = Debian Maintainer (planned to be a somewhat restricted DD, among others without voting rights)
• NM = New Maintainer (should actually be: New Developer)
• GR = General Resolution (a way in Debian to create new rules or to ask the project leader or others in the project to do or refrain from doing something)

Update:
The GR even seems to contradict itself a bit. If the upload rules are applied in an AND fashion, one of these rules is superfluous (the first one), at least if I’m not mistaken:

• the Maintainer: field of the uploaded .changes file corresponds with the owner of the key used (ie, non-developer maintainers may not sponsor uploads)
• the most recent version of the package uploaded to unstable or experimental includes the field “DM-Upload-Allowed: yes” in the source section of its control file
• the most recent version of the package uploaded to unstable or experimental lists the uploader in the Maintainer: or Uploaders: fields (ie, non-developer maintainers cannot NMU or hijack packages)

The first rules also means that no DM could ever do an upload for packages team-maintained in the fashion cyrus-imapd-2.2 is maintained (Maintainer address is the teams mailinglist address, all uploading team members are in the Uploaders field). This is IMHO a serious flaw in the proposal. If the first rule was dumped, all would be well for team maintainers.

Share This

Since march the DSA team uses a request tracker to handle all the requests that they get. Therefore you should no more use the debian-admin@debian.org alias (DSA members only). The debian-admin@lists.debian.org mailing list (DSA + local admins) is also scheduled for removal. Instead you should mail admin@rt.debian.org making sure to put “Debian RT” somewhere in the subject (crude but working spam prevention). By default, your ticket ends up in a private incoming queue. Up to now only DSA members had access to those queues. They moved most tickets in the public queue but it happened several times that this simple administrative burden took several weeks. Since a few days, Matt Taggart and myself have been granted RT accounts with required privileges to handle tickets and move them between the various queues. So from now on, all tickets will be quickly moved in the public queue (when they don’t contain sensitive information of course). We’ll be able to do some triage and ping DSA members for urgent requests. Matt and me have been trying to help out the DSA team for quite some time now, and while we’re not DSA with root rights, we’re in regular contact with the members (except Martin “Joey” Schulze who refuses to join the DSA IRC channel, who doesn’t use the request tracker and who doesn’t read the DSA mails either) and with several local admins. So if you have some troubles interacting with the DSA team, you can try to get in touch with us and we’ll see if we can help you. In too many cases people contact directly individual DSA members (Joey in particular). Please refrain from doing so, other volunteers (who might help you) won’t notice your request. Furthermore systematic request tracker usage helps us identifying what is well handled and what’s not. And it’s not more complicated than a simple email for you. Update: currently local admins are not subscribed to the request tracker. The RT setup will need some further adjustment for that, so the debian-admin@lists.debian.org list can still be useful if you want to reach them.

OK, this is probably my last Tour post for a while... back to business, software and the village after this, but I heard this on the radio over breakfast and I can't see much about it online. One of the most disappointing things for English Tour fans about Moreni's failed dope test [tdfblog] is the departure of his Cofidis team-mate Bradley Wiggins from the race, days before the last (and most Wiggins-winnable) of the individual time trials. In an interview this morning on BBC Radio 4's Today Programme at about 07:45 (in case you can Listen Again to it), there was the question:

"We've had whole teams withdrawing. It's not just the two who test positive. It's the sense that those are the tip of an iceberg, isn't it? The whole team pulls out and you don't really know how many others in that team are affected by the same behaviour."

and UCI president Pat McQuaid answered:

"Well, I mean, the fact that the whole team pulled out is something which the UCI is quite annoyed about, in actual fact, and it was that the organisers asked the teams to pull out. It wasn't the fact that the teams pulled out of their own accord. The organisers said to them, as a result of the positive, that they got the one person and they said they want the whole team out of it. That actually goes against the rules and it shouldn't have happened. Erm, so, from that point of view, you can't put anything on the other riders. You can't say that the other riders are up to anything just because they're not in the race. They were asked by the organisers to get out."

The interview continues to point out how there's much more testing in cycling now than ever before, and more than in any other sport (remember, Puerto implicated footballers and others too), but it's interesting how the whole-team-out approach is criticised for removing the other riders on a team from the race testing! Anyway, by the UCI president's reasoning, Bradley Wiggins should still be racing. Well done to UCI and the tour for catching Moreni and almost-well-done to Moreni for admitting it, but boo Cofidis for pulling the whole team out! (2007-07-26: +1 comment)

After today's keynote session, I met up with Simon Peyton-Jones and Bryan O'Sullivan. A vibrant discussion with these two Haskellers. We had all noticed that Haskell, and FP languages in general, have been doing really well lately and mused about why. Perhaps it's that people are becoming more used to functional approaches to problem solving because popular languages include some of them these days. Or perhaps it's that FP already has tools to solve problems that are suddenly hot, such as parallelization.

In any case, Simon's Haskell talks have really been highlights this week for many.

I had some time to visit the show floor. Stopped by the Debian booth and met Steve Langasek. A couple of Sun guys dropped by to talk about trademarks with OpenJDK. Bart Massey of Portland State Univ. also dropped by. I've conversed with him in email a few times. We talked about what PSU is doing with Haskell, some of the code I've written, and some of the code they've written. They're doing some fun stuff with rocketry and software-defined radio.

Oh, attended Simon's talk on nested data parallelism, which he is working to add to GHC. That sounds like a very interesting topic. Think list comprehensions, but the compiler automatically parallelizes the task for you.

After the conference's events, went out to dinner with some local (and some conference) Haskellers. That was a lot of fun, and Terah and Jacob got to come along too. People are doing some interesting stuff with Haskell.

After my comments about the TV audience figures, Ismael Valladolid Torres commented:

"Audience in Spain will keep on raising while Contador keeps playing such a big race!"

I hope that's still the case after the recent revelations. The evidence suggests that Vinokourov had someone else's blood [tdfblog] and Cofidis's Moreni was using something like a nutsack patch [tdfblog] while Rabobank have fired race leader Rasmussen for telling them he was in Mexico when he was in Italty [tdfblog] so it looks like this year's race is still really unpredictable as the dopes are caught and kicked out. Maybe my hopes for a Valverde (now 6th, 10:18 back) win still stand a chance... (My "serious" prediction, Kl den, left with Vino.)

I believe that an ideal installation process for Linux would have the option of performing a Xen install. The basic functionality of installing the Xen versions of the required packages (the kernel and libc), the Xen hypervisor, and the Xen tools is already done well in Fedora and it’s an option to install them in Debian. But more than that is required. Xen has two options for networking, bridging and routing. The bridging option can be confusing to set up and changing a system from routed to bridged networking once it’s running is a risky process. I have documented the basic requirements for running bridging in a previous post, but it would be better if there was an option to have Xenbr0 as the primary device from the initial install - and there are non-Xen reasons for doing this so it would be a more generally useful feature. Another common requirement for a Xen server is to have a DVD image on the local hard drive for creating new DomU’s. If we are going to need a copy of the DVD on the local hard drive for Xen installation and we need data from the DVD for the Dom0 installation then it makes sense to have one of the early installation tasks (immediately after running mkfs) be to copy the contents of the DVD to the hard drive. Hard drives are significantly faster than DVDs - especially for random access. It would also avoid the not uncommon annoyance of getting part way through an install only to encounter a DVD or CD read error… Here are some reasons for running Xen (or an equivalent technology) when not running more than one DomU:

1. Avoid problems booting. Everyone who has spent any significant amount of time running servers has had problems where machines don’t boot. Even with a capable out of band management option such as the HP ILO it can be unreasonably inconvenient to fix such problems. Separating the base hardware management tasks of the OS from the user process management tasks makes recovery much easier. If a DomU stops booting then it’s easy to mount it on the Dom0 and chroot into it to discovere the problem.


2. Easier upgrades. Often you have users demand that you install software that only works with a newer version of the OS. You can install the new version under a different DomU, test it, and then replace the old DomU when you think it’ll work - this gives a matter of minutes of down-time instead of hours for the upgrade. If the upgrade doesn’t work then you destroy the DomU and create one for the old version. Running two versions of the OS at the same time with NFS shares for the data files is also possible.


3. Security. If a DomU gets cracked the Dom0 will not necessarily be compromised, this puts you in a good position to track down what the attackers have done. You can get a dump of the DomU’s memory to enable experts to examine what the attackers were doing. Reinstalling a DomU to replace data potentially corrupted by an attacker is much easier than reinstalling an entire machine.

Even in situations when reason #2 was the motivation for installing Xen I believe that most systems will want to have a Xen DomU running the same version as the Dom0 for the initial install. Therefore integrating the installation process would make things easier. Among other benefits if you have a server with multiple CPUs (the minimum number seems to be two CPUs on all recent machines) and hardware RAID then doing two installations at the same time is likely to give better performance overall. Also I believe that it will often be the case that the Dom0 will exist purely to support DomU’s, therefore if you only install the Dom0 then you have done less than half the installation! For a manual installation there are some reasons for not doing this all at the same time. Having the sys-admin enter configuration data for some DomU’s at the same time as the Dom0 can get confusing. However for an automated install this would be desirable. I would like to boot from a CD and have the installation process take all configuration from the network (either via NFS or HTTP) and then perform the complete installation of the Dom0 and the DomU’s automatically. Let me know what you think of these ideas, it’s just at the conceptual stage at the moment.

Share This

I am still looking for a solution that allows me to tag my mails similar to what Gmail offers. Having thought about this for quite a bit now, I can see three solutions, implemented at different levels in the mail flow: as filesystem, patch to the IMAP server (Dovecot), or patch to mutt. Since this post is about the filesystem, I'll concentrate on that, but I'll also touch upon the other two solutions briefly.

Brice Goglin just announced that he considers the bug triage task on the XSF bugs nearly completed. This graph shows the result of that giant work. So, please, be impressed in case you never noticed that work and just buy Brice a beer next time you meet him. And then go back cleaning out your own packages' bugs, of course..:-)

The dpkg team has a nice history of changing VCS over time. At the beginning, Ian Jackson simply uploaded new tarballs, then CVS was used during a few years, then Arch got used and up to now Subversion was used. When the subversion repository got created, the arch history has not been integrated as somehow the conversion tools didn’t work. Now we’re likely to move over git for various reasons and we wanted to get back the various bits of history stored in the different VCS. Unfortunately we lost the arch repository. So we have disjoints bits of history and we want to put them all in a single nice git branch… git comes with git-cvsimport, git-archimport and git-svnimport, so converting CVS/SVN/Arch repositories is relatively easy. But you end up with several repositories and several branches. Git comes with a nice feature called “git rebase” which is able to replay history over another branch, but for this to work you need to have a common ancestor in the branch used for the rebase. That’s not the case… so let’s try to create that common ancestor! Extracting the first tree from the newest branch and committing it on top on the oldest branch will give that common ancestor because two identical trees will have the same identifier. Using git_load_dirs you can easily load a tree in your git repository, and “git archive” will let you extract the first tree too. In short, let’s see how I attach the “master” branch of my “git-svn” repository to the “master” branch of my “git-cvs” repository:

$ cd git-svn
$ git-rev-list --all   tail -1
0d6ec86c5d05f7e60a484c68d37fb5fc31146c40
$ git-archive --prefix=dpkg-1.13.11/ 0d6ec86c5d05f7e60a484c68d37fb5fc31146c40   (cd /tmp && tar xf -)
$ cd ../git-cvs
$ git checkout master
$ git_load_dirs -L"Fake commit to link SVN to older CVS history" /tmp/dpkg-1.13.11
[...]
$ git fetch ../git-svn master:svn
$ git checkout svn
$ git rebase master

That’s it, your svn branch now contains the old cvs history. Repeat as many times as necessary…

In a default configuration of Xen there will be a virtual Ethernet device created for each interface which will be associated with a bridge. A previous post documented how to configure a bridge named xenbr0. The basic configuration of Xen that most people use is to have a single virtual Ethernet port for each Xen instance and have them all connected to the one bridge, and then the Dom0 will have an IP address on the bridge interface that is used for routing packets to the outside world. This works really well if you have a subnet that you are using for all Xen DomU IP addresses, if you are using NAT for communication, or if the DomU needs no communication outside the Dom0 and other DomU’s on the same machine (a common case for testing). But if you have a collection of servers that you want to consolidate on a single piece of hardware then you end up using a single sub-net that spans some physical machines, some Xen Dom0’s, and some DomU’s. The solution to this is to use bridged networking. Unfortunately most documentation of bridged networking is really confusing, and non of my google searches turned up the most relevant fact: When setting up a bridge on the local Ethernet you must make your physical ethernet device (eth0 or whatever) be strictly a slave to the bridge and then assign the IP address used for the physical network to the bridge.

ifconfig eth0 up

brctl addif xenbr0 eth0

For example if you have 10.0.0.42 being the IP address used by the Dom0 on the local Ethernet via device eth0 and you want to use bridging for DomU’s then you simply make eth0 owned by xenbr0 (the typical name for the Xen bridge) with the above commands in your script to configure the xenbr0 device. Then treat xenbr0 in the same way that you treated eth0 before enabling bridging. Also there’s nothing stopping you from having one bridge for DomU’s that can talk directly to the physical Ethernet and another for DomU’s that are only to use routed networking, see my previous post about using multiple ethernet devices in Xen for more background information.

Share This

Occasionally, I update my sidebar to reflect the current contents of my iRiver. Most people read this blog through the RSS feed, however, and never see the sidebar. So I thought I’d provide occasional snapshots here. Lately:

• Fred Frith: Cheap at Half the Price and Rivers and Tides Working With Time
  Two very different genre-evading albums (file under — alternative?). Rivers and Tides (the film) left an indelible mark on me, but the soundtrack stands alone. Cheap at the Half the Price is a good example of why we can probably get by for another few centuries just on music that’s already been made.
• Petra Haden: The Who Sell Out
  Occasionally too cute, but mostly brilliant one-woman-voice-only remake of The Who album. This one also gets its own Wikipedia article. I’m also a big fan of her album with Bill Frisell. (File under — jazz?)
• G. Love: Lemonade
  Who would have guessed the G. stands for “Good”? I recently went back and listened G. Love and Special Sauce. This one is much better. I’ll post two short clips when I have a minute.
• Paul McCartney: McCartney and Memory Almost Full
  McCartney has been a solo artist for longer than I’ve been around. His first album includes at least two tracks that rival anything from the Beatles (one of which even has its own Wikipedia entry). His newest album is not quite as good, but really, it’s hard for McCartney to really go wrong. (Typically engrossing New Yorker profile.)
• The White Stripes: Get Behind Me Satan
  Thanks to Steve for reminding me what a great song My Doorbell is.

Share This

I’ve been thinking about what I’m able to write about TiVo without upsetting anyone, and so I’ve been doing a bit of research about what’s already been written. Surprisingly, a lot. So I decided that there really wasn’t a whole lot that I could say that isn’t already somewhere else on the web, and figured I’d just add in some technical explanations to questions that are likely to come up. For example, the remote that we will ship has the thumbs buttons, but the Comcast remote [pdf] doesn’t. If you compare the remotes closely (there’s a picture of the TiVo remote in the article linked above), you’ll see the Comcast remote has two buttons that the TiVo remote doesn’t: Page Up and Page Down. There you go. The TiVo UI uses Channel Up/Channel Down as Page Up/Page Down in lists, so we don’t need separate Page Up/Page Down buttons, and use them for Thumbs buttons instead. Also, to confirm what was written in that article linked to above, we really did rewrite our software in Java. Sort of. As the article states, we run on top of TVWorks’ TV Navigator middleware. It’s not a full OCAP implementation, but rather an implementation of OnRamp To OCAP, which is (from Informitv) not a precursor but “a subset of OCAP so that applications will be forward-compatible with the full OCAP specification.” In other words, we didn’t write the OCAP layer, nor the OS. And the Java-based software so far only runs on the Motorola hardware.

This article in The Age about Mohamed Haneef shows the terrorist threat that we face. The chance that I will be injured by Al Quaeda in any way is quite remote. The chance of being attacked by ASIO is a lot greater. The main benefit of being in a democracy is having a legal system where the defendant is presumed innocent until proven guilty and where they have the right to legal representation. The war in Iraq has not brought the US or Australian system of government to Iraq, instead it is bringing Saddam Hussein’s system of government to Australia and the US. Traditionally under the Australian and US legal systems innocent people are not punished, unlike under Saddam Hussein. Now ASIO has the authority to detain innocent civilians indefinitely if they believe that it helps them in some way - and there is no method of policing ASIO to ensure that even such excuses are met. Traditionally under the Australian and US legal systems everyone who is accused of a crime is entitled to a trial, unlike under Saddam Hussein. Now ASIO and the CIA have been given the authority to punish anyone without a trial. ASIO can also extend the punishment to anyone who might receive evidence of such actions and publish it (I guess that the CIA can do the same). Saddam lost the battle but his legacy is winning the war. For the best definition of Terrorism see Noam Chomsky’s paper. The actions taken by the Australian government against the people of Iraq, foreign citizens in Australia, and almost certainly Australian citizens (it’s not credible to believe that ASIO has such powers and doesn’t use them occasionally on Australians) fits the definition of Terrorism. Elections are coming soon, both in the US and in Australia. Whatever you do, don’t vote for Neo-Cons (Republicans in the US or Liberals in Australia). PS Before anyone suggests that I should worry about ASIO kidnapping me in retaliation for this, I’m sure that they know of the Streisand Effect. I’ll try and avoid any unplanned down-time for my blog after this post goes out to avoid false-alarms… ;) Update: I incorrectly wrote “guilty until proven innocent” above, that is the current Australian government policy not the way it should be.

Share This

Too Many Python Parser Generators Dear LazyWeb, there are too many Python parser generators. There is a Wiki page comparing them, articles at various editions of the Python conference, tons of mailing list posts. I'm so bored about all this amount of information (which I actually did read some months ago) that I'm not even willing to look for the relevant hyperlinks and add them here; that sucks. ( Rant: I frankly have hard time in believing that a language with a respectable standard library as Python has failed so far in establishing a de facto solution for parser/lexer generations. However this seems to be the case. ) Feeling lost, I convinced myself that I wanna go for a Python-Bison glue, so that at least I'll have a grammar in a widespread format that I can reuse in other languages in case of need. But even in this case there seems to be several Python-Bison glues out there ... and almost all seem to be dead projects! Dear LazyWeb, any suggestions about which to choose?

My brother Ted and his partner Nabil had a great commitment ceremony yesterday afternoon in Dolores Park in San Francisco. It was a beautiful day, and they had lots of friends who came out for them. I came up from Los Altos Hills with our parents, and my brothers Nate and Andy came with their friends and/or families, too. Ted and Nabil march to a different drummer, to put it mildly. Their ceremony was remarkably like a Quaker wedding, although I don't think they planned it that way. They had a nice picnic and tea party in the park for a few hours, let the guests get to know each other, and then called everyone together to listen to their promises to each other. (Very sweet, by the way). They then exchanged bejewelled carabiners -- those funky little mountain-climbing metal loops that people also use as keyrings. "We may lose necklaces or rings, but we're not going to lose our keys." The carabiners were decorated with the California state gem, benitoite (Benito-ite), which is rarer than diamonds. Brilliantly, their friends who designed the carabiners also managed to include a USB thumbdrive, on which they're going to copy all the pictures, stories, movies, and other digital paraphernalia from the ceremony itself. Ted and Nabil will be carrying around those pictures as long as they have the carabiners. Clever! After they'd exchanged the carabiners, they hooked them to their belts, and then hooked them together. "And now... we're HITCHED!" Thunderous applause. They then invited up anyone who wanted to make a toast. My brother Nate went first, and girt in a bastard sword (for unclear reasons), he made a great toast. A dozen other people, including my Dad, came up to speak. By then, Ted and Nabil were falling over with fatigue, so we had mercy on them and stopped coming up to say nice things about them. Everyone dug in to champagne and cake, and Amita June and her two cousins Elena and Tessa ran around on the huge playground in the park. Definitely a good, good wedding-like event. tags: ted nabil wedding carabiners commitment ceremony

Long day This morning we had a Skype conference call relatively early in the morning, and then a brunch with Nabil's family (who are visiting from Virginia and Philadelphia). It was really nice -- my folks decided to build a huge patio in their backyard, and it's great for having kids and family running around, as well as for outdoor brunches. (We haven't had a meal inside since we got here, actually.) This afternoon we drove up to Novato to visit Maj's grandmother Jeannette and her Uncle Kevin and Aunt Robin. Also present were Jeannette's lifelong friend Roseanne and Kevin's highschool pal Jim. Amita June loved running around their big backyard and playing with their sweet old dog Holly. She also got a lot of mileage out of their porch swing. Maj and Robin swapped cooking tips and renovation horror stories, and I got to listen to Jeannette and Roseanne tell gambling stories from various Southern California Indian casinos. Jim, meanwhile, was on his cell phone in another corner of the yard, negotiating with another gold prospector with a hand injury to work the friend's claim. Quite an evening. After a dinner of barbecued salmon, caprese salad, and corn-on-the-cob, Kevin and Robin's neighbors came over, and we talked more about statistical analysis, sheep-farming in Marin County and driving through Austria. A long drive back to my folks', but Amita slept the entire time, thankfully. Good for us all. tags: kevin robin novato jeannette indiancasinos porchswing amitajune

Voice jail I had to call Air Canada a few days ago to get our tickets to come out here to California, and I had a terribly frustrating time working my way through their automated systems to get to a human being that I needed to talk to. And I thought to myself, "Why do I hate these automated voice systems so?" I never call a 1-800 number or another voice system if my problem could in any way be handled automatically. I always go to the Web first, and only try to get on the phone if the particular special-purpose request I have hasn't been, or can't be, built into the Web interface. But the economics of call centres makes the company want to shunt me into one or another automated voice systems in hopes that I'll be satisfied and not bother a real human being. My only goal with a phone call is to talk to an intelligent human being who can handle my exceptional situation. But the interface is designed precisely to deny me that goal. Can that interface ever be anything but frustrating? tags: phones voicejail frustration interface

Least likely to succeed Ross Mayfield posted a nice pic of me offering to run his company in exchange for a hot meal. It's a good shot of me, actually. In other linking issues, my Technorati "Authority" value is now 101, putting me squarely into B-list Bloglebrity territory. Hooray for the second string! tags: b-list socialtext technorati rossmayfield bloglebrity

Just because I can't run sa-learn on my 15,000+ messages spam folder. It'll crash due to some hardware problems.
I thought that splitting the mailbox into smaller files will allow me to feed it to sa-learn.
I'm not sure something similar doesn't exist but I wrote mine anyway ;-)

The only problem is it consumes a lot of CPU and RAM, it was killed/crashed multiple times but it worked and allowed me to feed my spam mailbox to spamassassin!

Here it is in case someone needs it: split_mailbox.py. Needs python 2.5
read more

Since I spent quite some time thinking about my SMTP setup earlier today, I finally took the time to hack up a little perl script to parse my mainlog, pick out the IP addresses of hosts that appear to have been spamming me, put them in a database, and modified my exim4.conf to look them up and deny them access if I want to. Those lookups are pretty easy, really. In the first section of your exim4.conf, I have this:

hide pgsql_servers = (/var/run/postgresql/.s.PGSQL.5432)/*/spam/Debian-exim/

with '*' being the password. Except that this is ident authentication, so I don't need a password, but apparently exim still expects something (if I'm not on crack). Oh well. Then, in an acl, I have this:

  deny
    message = "Your IP was used for spamming over the last few days. Please stop the spamming, wait a few days, and try again."
    log_message = "denied access for $sender_host_address: in blacklist database"
    condition = $ lookup pgsql SELECT id FROM spammer WHERE       \
       adres='$sender_host_address' AND lastseen > now() - '10 \
       days'::interval true false 

(with the backslashed lines being one line, really, but I didn't want to fuck up scrolling) You can put this in any ACL you want, really. In my case, since I'm very sure by the nature of the way I gather these IP addresses that they're spammers, it's in acl_smtp_helo. Depending on how you populate your database, you may want to put this in a different place. If you want, you can teergrube them a bit by adding delay=30s. An alternative would be to set up a DNS blacklist; but that would just take too much effort to set up properly, and I just didn't feel like it. This works.

I won’t re-explain in great length why I think it’s good to endorse the concept of Debian Maintainers. I have been enough involved in the debian-vote discussions (and the previous debian-project discussions). I would just like to remind everybody that we elected sam to see changes and progress on many areas. We haven’t seen many results yet, but I know that sam has been working hard and I’ve been helping him as much as I can on the problem of our DSA team (one day I might blog about it or even start a GR if the situation continues to not improve despite the numerous efforts that we’ve put into it). Here we have a concrete proposal for a change, and I believe a change for the better. But just like for every change, people have fears: they fear people who only care about technical excellence and not too much about philosophy, they fear that the quality level will drop, they fear that nobody will care about NM afterwards, etc. It’s legitimate to have concerns, to express them and to discuss them. But we should not let them take us over or we’ll end up abandoning all initiatives that are required to evolve and adjust to our moving environment. We need to encourage people who are ready to try out new things. In cases where it’s not entirely clear how the situation will evolve, it’s better to try out and react accordingly instead of doing nothing and hoping that people will wait us. When we do things while hoping for the best, the worst won’t come up true so easily. Have faith in our future.