Is XFS that bad? Well, at least with default settings, XFS on Debian seems to be blown away by ext3 completely in terms of speed. I don't mind 1.5x slowdown, maybe even 2x, but 20x is a show stopper. I am already using ext3 for any pbuilder builds, because it's a difference to wait for 30s with XFS, compared to 3s with ext3 to extract the base image. And I'll probably switch to ext3 completely, unless someone finds a way how to fix this.

I recently got burned by this when running Sage on my computer, because it compiles a lot of Python files when started for the first time. Normally it should take roughly 15s, but instead it took 6 minutes on my comp and then it triggered a so far undiscovered bug in Sage, that I reported.

Michael Abshoff, the release manager of Sage, suggested that something is FUBAR (Fucked Up Beyond Any Recognition) on my shiny Debian amd64 sid system running on Intel Core Quad, so I said no way, because I really care about this machine, as I use it for larger finite elements calculations and other stuff (like compiling huge deb packages in parallel, like paraview).

So I offered a bet, that I give him an access to this compter, he finds the problem and if it's a problem in my Debian configuration, I'll write to this blog that I am lame, while if it's a problem in Sage, he will write to his blog that he is lame. And I was smiling to myself, how good I am and that I will have some fun too reading planet.sagemath.org with the top post from Michael saying that he is lame.

But then I remembered my old struggle with cowbuilder and XFS and I stopped smiling. See e.g. this wiki I created half a year ago. Something is FUBAR with XFS and Debian. I also asked on the Czech server Root, that is famous for having a lot of experts willing to share their knowledge, and it was quickly revealed, that the problem is with the "nobarrier" option of XFS (my post is here, but it's in Czech).

First, on that amd64 machine, the above problem was fixed after issuing this command:

mount -o remount,rw,nobarrier /dev/sda3 /home/

(notice the "nobarrier" option). You can read some background behind this on the lkml list. Unfortunately, I also have my laptop, and there I already use this "nobarrier" option, and it doesn't help at all. I just created a new ext3 partition and verified that on my laptop, ext3 is around 10x faster than XFS with nobarrier (that was supposed to fix this). I use the latest 2.6.24 kernel from unstable on both.

Time to move from XFS to ext3 on my laptop? Seems like that. I'll leave XFS on the other machine, because I know some other peole have good experience with XFS and the "nobarrier" option seems to fix the problem there.

But as to the bet, yeah, I am lame and I should still learn a lot from Michael. :)

Today's xkcd reminds me that about 15 pounds of grapefruit into this year, I tried to calculate how many I could eat a day, if I were to do were peel and eat grapefruit, all day. I may have been trying to figure out if it is possible to live on a pure grapefruit diet. I remember arriving at a figure around 70. My actual daily consumption at the height of grapefruit season is 3 to 5. It's become my favorite winter fruit and quite puts one off nasty sweet oranges. There's a trick to both the peeling and the eating of grapefruit which the xkcd guy has apparently never learned, that avoids the healhful skin and leaves only the delicious tart flavour droplets behind. I'm available for lessons, the first 2 or 3 weeks that the Texas ruby reds are in season.

---

(Of course as wrong as Randall is about grapefruit, he's 5 million times as wrong about a proper tomato..)

I learned about two bash features I hadn't known about today.

From a colleague, GLOBIGNORE. A colon-separated glob of files to ignore when expanding globs. Helpful behavior when set to "*~" and used with grep.

From the Git FAQ, in a section explaining that it breaks the Git build process, CDPATH. A colon-separated search path to use when you type cd. Possibly useful to refer to subdirs of ~ or other common areas. Seems like it's prone to break a ton of scripts if exported though.

Russ Nelson: The Influence of Government writes:

"Larry Lessig proposes to lessen the influence of money on legislators. Unfortunately for him, he is trying to hide a symptom without curing the disease. The disease is that governments regulate businesses. The symptom is that businesses then have a profit motive in regulating governments. If you want a government which is free of corruption, you have to eliminate the motive for corrupting them."

which looks like utter bovine excrement to me. Corporations - and the overpowering profit motive of some corporations - are artificial things, created purely by regulations. If you free corporations from all regulations, you destroy them. If you only free them from taxes, you remove some of the few restraints on their bad behaviour in pursuit of profit. See Joel Bakan: The Corporation for a better approach. In summary:-

• Relegislate. Base regulations on the precautionary principle for environmental, health and safety issues;
• Set realistic enforcement agency staffing levels and punishments;
• Involve local elected representatives in more of the regulatory system;
• Protect and enhance the role of trade unions and civil society in advising regulators.

Some of those may seem surprising, but if you read the book, you'll see why those conclusions make sense.

For those who are not going to FOSDEM, the DebConf video team aims to provide live streams of all talks in the Debian dev room as an ogg theora stream. I hope it will work out :-)

Currently there is a single server behind that URL. If we see lots of viewers we can add another server with round robin dns easily. Please submit your offer in the teams irc channel or on the mailinglist, if you already operated at streaming server for DebConf7 in Edinburgh that would be best.

And if you do attend FOSDEM please consider helping the video team, there are still about 20 positions to cover (operating the camera, the audio mixer and the video mixer). All those jobs are easy and fun, if you don't have to do them for the whole conference ;-)

I really don’t mind if Debian lags behind with package versions. And I’ve gotten used to running an ancient version of “pgadmin3″ (the GUI client for PostgreSQL). But now that I upgraded a few pgsql servers from 8.2 to 8.3 and started pgadmin3 I was surprised - in a not at all fun way. Several [...]

I’ve a problem for some time: something seem to be changing my cpu frequency behind my back, and I don’t know what is doing it. It happen when the computer is doing cpu intensive mostly non interactive thing, as watching certain .avi with some codec. The change of cpufreq is relay a problem because then the cpu became to slow to decode the film at the speed I need to see it. So, what happen ?
If I use the ondemand governor, it seem that it will eventually believe that I do not demand cpu anymore, slow down to 800Mhz, and one cannot look at the film anymore…
If I use the userspace governor, setting the frequency to the max available value, well, the frequency will eventually change to something smaller. It will even lower the max frequency… Worst still, cpufreq setting won’t change to what I want if I ask for it:

# cpufreq-info && echo && cpufreq-set --max 2GHz && cpufreq-set -f 2GHz && LC_ALL=C cpufreq-info
cpufrequtils 002: cpufreq-info (C) Dominik Brodowski 2004-2006
Report errors and bugs to linux@brodo.de, please.
analyzing CPU 0:
  driver: powernow-k8
  CPUs which need to switch frequency at the same time: 0
  hardware limits: 800 MHz - 2.00 GHz
  available frequency steps: 2.00 GHz, 1.80 GHz, 1.60 GHz, 800 MHz
  available cpufreq governors: userspace, powersave, ondemand, conservative, performance
  current policy: frequency should be within 800 MHz and 1.20 GHz.
                  The governor "userspace" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz (asserted by call to hardware).
cpufrequtils 002: cpufreq-info (C) Dominik Brodowski 2004-2006
Report errors and bugs to linux@brodo.de, please.
analyzing CPU 0:
  driver: powernow-k8
  CPUs which need to switch frequency at the same time: 0
  hardware limits: 800 MHz - 2.00 GHz
  available frequency steps: 2.00 GHz, 1.80 GHz, 1.60 GHz, 800 MHz
  available cpufreq governors: userspace, powersave, ondemand, conservative, performance
  current policy: frequency should be within 800 MHz and 1.20 GHz.
                  The governor "userspace" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz (asserted by call to hardware).

Not only my setting have been changed, But they won’t change back to what I want. I have to wait sometime, before demanding cpufreq-set to change my setting if I want them obeyed.

Finally I found the URL of a device I’ve been hearing rumours about. The HotPlug is a device to allow you to move a computer without turning it off [1]. It is described as being created for “Government/Forensic customers” but is also being advertised for moving servers without powering them down. The primary way that it works is by slightly unplugging the power plug and connecting wires to the active and neutral terminals, then when mains power is no longer connected it supplies power from a UPS. When mains power is re-connected the UPS is cut off. Modern electrical safety standards in most countries require that exposed pins of a power plug (other than the earth) be shielded to prevent metal objects or the fingers of young children from touching live conductors. The image above shows a recent Australian power plug which has the active and neutral pins protected with plastic such that if the plug is slightly removed there will be no access to live conductors. I have photographed it resting on a keyboard so that people who aren’t familiar with Australian plugs can see the approximate scale. I’m not sure exactly when the new safer plugs were introduced, a mobile phone I bought just over three years ago has the old-style plug (no shielding) while most things that I bought since then have it. In any case I expect that a good number of PCs being used by Australian companies have the old style as I expect that some machines with the older plugs haven’t reached their three year tax write-down period. For a device which has a plug with such shielding they sell kits for disassembling the power lead or taking the power point from the wall. I spoke to an an electrician who assured me that he could with a 100% success rate attach to wires within a power cord without any special tools (saving $149 of equipment that the HotPlug people offer). Any of these things will need to be implemented by a qualified electrician to be legal, and any electrician who has been doing the job for a while probably has a lot of experience working before the recent safety concerns about “working live“. The part of the web site which concerns moving servers seems a little weak. It seems to be based on the idea that someone might have servers which don’t have redundant PSUs (IE really cheap machines - maybe re-purposed desktop machines) which have to be moved without any down-time and for which spending $500US on a device to cut the power (plus extra money to pay an electrician to use it) is considered a good investment. The only customers I can imagine for such a device are criminals and cops. I also wonder whether you could get the same result with a simple switch that cuts from one power source to another. I find that it’s not uncommon for brief power fluctuations to cause the lights to flicker but for most desktop machines to not reboot. So obviously the capacitors in the PSU and on the motherboard can keep things running for a small amount of time without mains power. That should be enough for the power to be switched across to another source. It probably wouldn’t be as reliable but a “non-government” organisation which desires the use of such devices probably doesn’t want any evidence that they ever purchased one… Now given that such devices are out there, the question is how to work around them. One thing that they advertise is “mouse jigglers” to prevent screen-lock programs from activating. So an obvious first step is to not allow jiggling to prevent the screen-saver. Forcing people to re-authenticate periodically during their work is not going to impact productivity much (of course the down-side is that it offers more opportunities for shoulder-surfing authentication methods). Once a machine is taken the next step is to delay or prevent an attacker from reading the data. If an attacker has the resources of a major government behind them then they could read the bus of the machine to extract data and maybe isolate the CPU and send memory read commands to system memory to extract all data (including the keys for decrypting the hard drive). The only possible defence against that would be to have multiple machines exchanging encrypted heart-beat packets and configured to immediately shut themselves down if all other machines stop sending packets to them. But if defending against an attacker with more modest resources the shutdown period could be a lot longer (maybe a week without a successful login). Obviously an attacker who gets physical ownership of a running machine will try and crack it. This is where all the OS security features we know can be used to delay them long enough to allow an automated shut-down that will remove the encryption keys from memory.

• [1] http://www.wiebetech.com/products/HotPlug.php

Share This

A nice feature from PostgreSQL are “write ahead log” (WAL) files. Quoting from the PostgreSQL website they describe every change made to the database’s data files. […] if the system crashes, the database can be restored to consistency by “replaying” the log entries made since the last checkpoint. This basically enables online backup of your database cluster. And as you end up having the complete log of all actions within your database(s) you are also able to go back in time to any point you like. I won’t write a complete howto about it, as the PostgreSQL docs are pretty good, just some notes how I did my setup, as a kind of reminder for me when I have to do it again.

---

There must be a place to store WAL files which ends up not being on the same machine, otherwise it doesn’t make much sense. For me that means scp to a different host, but an NFS mount or some storage array or drbd device might also be an option. Much of the following is for the scp method, but easily adaptable to other storage methods. I need a target user on the host I copy files to, and as I did the setup for the pentabarf database running on host skinner, I helpfully named it pentabarf-backup. The copy process can’t be interactive so the postgres user on the database host needs to have a passwordless SSH key to login to the pentabarf-backup user on the backup host. To limit my bad feeling about “passwordless key to login” I restrict the key by adding

from="skinner.debconf.org,72.32.250.224,72.32.79.241",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty 

in front of it while copying it to the pentabarf-backup users .ssh/authorized_keys file. I need a few directories, namely pentabarf-WAL-skinner, pentabarf-tarballs and pg_xlog-skinner for the stuff to backup, created in the home of the pentabarf-backup user. Next step is to tell PostgreSQL our wish to have online backups. Add the following statements to your postgresql.conf, adapting the values to your needs:

archive_command = 'ssh backup test ! -f pentabarf-WAL-skinner/%f && rsync -az %p backup:pentabarf-WAL-skinner/%f'
checkpoint_timeout = 1h
archive_timeout = 12h

The above tells PostGreSQL that i want it to archive the files using ssh to my backup host and transfer files with rsync (via ssh), but only if they do not exist yet. It also tells it to archive the files at least once every 12 hours. The effect is that WAL files get copied over when they are “full” or after 12 hours passed, whatever happens first. Don’t set it too low, as the file will always have the full, fixed size and you waste space transferring mostly-empty files. In case you are concerned with the possibly slow speed of a network link, the PostgreSQL site has the following to say about this: The speed of the archiving command is not important, so long as it can keep up with the average rate at which your server generates WAL data. Normal operation continues even if the archiving process falls a little behind. I haven’t had a problem with the speed yet. A 16MB file every (at least) 12 hours isn’t that bad. Now that we have done all the setup and are able to transfer WAL files, we need a base backup to start with. Issue a SELECT pg_start_backup(‘BASE_BACKUP’); as a database superuser and then backup the database cluster by backing up the whole clusters data/ directory (and every other location if you use tablespaces). When you are done issue SELECT pg_stop_backup(); and you are done with your base backup. You might want to exclude pg_xlog/ to save space, you don’t need it.
As I’m lazy I use a script (written by Peter ‘weasel’ Palfrader), you can get it here, but whatever you take is fine. If you use the script just make sure to adapt the variables in the script to values that fit your system. From now on you should do such base backups in regular intervals, I run them once a week, as recovery does need the last base backup plus all WAL files from then on. Doing regular base backups limits the amount of WAL files you need to keep around / you need to replay.

---

Reading the above you might find that there is a window of up to 12 hours from which I do not have backups. True, and to make this window way smaller, I am using another script (also written by weasel), get it here, together with an init script. This synchronizes the pg_xlog/ directory to my backup host every 5 minutes, so in case of a server crash I can replay everything up to the last 5 minutes. Recovery using WAL files This is pretty simple, and also good documented at the PostgreSQL site again. For my scenario it basically boils down to

1. Stop postgres
2. Move away the cluster.

   mv /var/lib/postgresql/8.2 /var/lib/postgresql/8.2_old

3. Restore the backup dump, be careful to have the file rights correct. For the above described setup that means copy and untar the latest tarballs created weekly.
4. Recreate pg_xlog:

     mkdir /var/lib/postgresql/8.2/main/pg_xlog
     chown postgres:postgres /var/lib/postgresql/8.2/main/pg_xlog
     chmod 40700 /var/lib/postgresql/8.2/main/pg_xlog
     mkdir /var/lib/postgresql/8.2/main/pg_xlog/archive_status
     chown postgres:postgres /var/lib/postgresql/8.2/main/pg_xlog/archive_status
     chmod 40700 /var/lib/postgresql/8.2/main/pg_xlog/archive_status
    

5. Copy possibly unarchived WAL files from /var/lib/postgresql/8.2_old/main/pg_xlog into the newly created pg_xlog.
6. Copy all the WAL segment files that are needed over from the backup host. That should be all the files on the backup host, directory pentabarf-WAL-skinner/. You should be able to limit it by taking all those files >= the latest *.backup file.
7. Create a recovery command file recovery.conf in the cluster data directory. You may also want to temporarily modify pg_hba.conf to prevent ordinary users from connecting until you are sure the recovery has worked. The content of recovery.conf is

         restore_command = 'cp /tmp/pg-WAL/%f "%p"'
        

   Make sure that the file location you specify is the location you put the WAL segment files into in the prior step.
8. Start the server, it will go into recover mode and proceed to read through all the archived WAL files it needs.
9. Look at the database and decide if you like what you see.

Of course the procedure can be slightly adapted, so you end up with a database state from 2 weeks ago. Or 3 days. Or whatever you may need (and have WAL files for). There are many possibilities, and there is a lot of text on the PostgreSQL page that I don’t want to duplicate here, so please read it. Also, I’m not using this as the sole and only way to backup my database(s), they are all still dumped into SQL files every day. This is just additional.

XQuery 1.0 and XSLT 2.0: news and how to use them in Debian (long post about the second generation of the XSL family and how to use the related languages in Debian)

• XPath 2.0, XQuery, et al.
• the above stuff .. in Debian
• links

---

The 2nd generation of the XSL family At the beginning of 2007, W3C released a family of interrelated specification: XPath 2.0, XSLT 2.0, XQuery (1.0). The 3 specifications are based on the very same underlying data model which (finally!) supports typed values and exploits them to some extent, for example permitting some form of static type checking in XPath and XQuery. Types usually come from XML Schema, with both built-in and user-defined types supported. A brief overview of the 3 specifications and what they change in the state of the art follows. XPath 2.0 XPath 2.0 basically consists in XPath 1.0 + 3 new macro features. The first one is a revamp of those language features which are dealing with types, in order to better integrate them with the new typed data model. XPath now has operators to check whether a value belongs to a given type (so yes, type information are kept at run time), and to cast them from one type to another. The basic new type constructor is now the sequence, which replaces the old node-set solving many annoying issues, such as the impossibility of having a node ordering other than document order. The second feature is the improvement of standard library of functions which was ridiculously small in XPath 1.0; it is much better now. Additionally, 2nd generation languages (XSLT 2.0 and XQuery) now support the ability to define functions which will then be visible to inner (XPath) expressions, pushing yet forward the possibilities of plain old XPath. Functions can now also declare types in their signatures (for their arguments and return value) and untyped arguments will be automatically casted to them upon invocation. Hence, even if you are not using an implementation which assigns types to your XML trees, once you "enter" the typed world calling a typed function (almost all standard library functions are decently typed) you will be able to stay there avoiding annoying casts everywhere. Finally XPath 2.0 has turned into a powerful purely functional language and is now powered by constructs like conditionals, for-each loops, existential/universal quantifiers, and existentially-quantified comparison operators for sequences. Here is a complex expression to hwet your appetite (or scare you away ...), comments come as (: smiley faces :)

for $book in /bookshelf//books
return
  if ((every $author in $book/authors/author
       satisfies $author/nativeLang eq "it_IT")
      and $book/lang eq "it_IT")
  then $book
  else ()
(: think about the trouble of writing this in XSLT/XPath 1.0 ... :)

XSLT 2.0 XSLT 2.0 is what I would call a "bug fix release" of XSLT 1.0 + the routinary reworking of the language to deal with typed values, which is not sensibly different than what has been done for XPath 2.0. The fixed "bugs" are several, starting from the annoying issue of result tree fragments. They are basically tree snippets that in XSLT 1.0 you were able to generate for future use. Unfortunately they were not thaaat reusable, given that you were not even able to navigate them with XPath operators! Now the specification is much more clear and distinguishes final result trees from ordinary variables, which can now contain sequences of (navigable) tree nodes. Another important "bug" fixed is the new ability to output multiple documents with a single XSLT stylesheet: it marks the end of stupid extra post processing to be added in pipeline to a XSLT processor. Other minor "bugs" fixed are a limited amount of backtracking capabilities among imported templates, regular expression support directly in the language, and powerful grouping constructs on the lines of SQL's GROUP BY (but much more powerful). Here is a template snippet exploiting the latter feature:

<xsl:for-each-group select="*" group-starting-with="h1">
  <div>
    <xsl:apply-templates select="current-group()" />
  </div>
<xsl:for-each-group>

XQuery XQuery is the end of the chains imposed by XML-based syntaxes. Why the heck one has to use an XML syntax (as in the above snippet) only because she is manipulating XML tress is one of the mysteries of XML technologies which have always been floating around in my head. XQuery is the (supposedly) SQL equivalent for databases of XML documents, but is actually much more than that. I depict it in my head as the XML manipulation language with a syntax I can finally stand. Technically it is XPath 2.0 (say 80% of the whole language) + some extra ingredients (say 20%); so remember that every XPath 2.0 expression is also a XQuery expression. The main extra ingredient is the so called FLWOR expression (to be read: "flower expression", which in addition to the "smiley faces" used for comments gives a "back to 1968"-flavour to the language .... erm FLWOR/flower/flavour, no pun intended). A FLWOR expression is very similar to SQL's SELECT-FROM-WHERE: it lets you generate a tuple stream by iterating on sequences (F: for clauses), binding expression values to names (L: let clauses), filter out tuples which do not satisfy a required condition (W: where clause), order the survived tuples (O: order by clause), and finally return a sequence built using the residual tuple stream (R: return clause). The other interesting extra ingredient is the ability to build the XML snippets you want to manipulate. Within XQuery you do that using plain XML syntax (the only place where a sane-minded programmer actually wants to see it!) which also supports a classical interpolation mechanism to embed expressions which will be evaluated inside XML snippets, and also the other way around. A canonical XQuery example is:

for $t in doc("books.xml")//title,
    $e in doc("reviews.xml")//entry
where $t = $e/title
return <review>  $t, $e/remarks  </review>
(: braces denote the escaping context where XQuery expressions will
   be evaluated inside snippets; plain XML syntax is used for the
   other way around :)

But remember: XQuery for XML is much more than SQL for RDBMS, thanks to the implicit templating mechanism implemented by interpolation, and thanks to several language features fostering modularity (user-defined functions, library modules, XPath 2.0 standard library, ...) you can basically do with it any kind of XML manipulation you can imagine. I don't think I will ever write myself any other single line of XML output in DOM or XSLT ...

---

Cool, how can I use it in Debian? ... if only this stuff were decently supported in the open source world. Last time I checked, the author of most parts of the GNOME toolchain for dealing with XML (libxml2, libxslt, ...) was not intentioned to implement XSLT 2.0, not even mentioning XQuery. This comes as no surprise, the whole GNOME XML toolkit is written in C, and XSLT 2.0 / XQuery have reached a level of complexity and formal specification which usually entails a higher level approach. So on the GNOME side we are stuck. The other open source implementations of XSLT 2.0 / XQuery I'm aware of are Saxon and Galax. Saxon is an XSLT 2.0 and XQuery implementation written in Java, which is unfortunate per se. Additionally, it is also unfortunate that it is only partially open source. Indeed, Saxon is split into SaxonB (for "basic") which is open source under the Mozilla Public License and SaxonSA which is commercial. While SaxonSA is a fully conformant, XML Schema-aware processor, with support for static typing, SaxonB is a basic-conformant processor with no type-aware features and actually much less optimized than SaxonSA. This is annoying. (Much more annoying is the fact that SaxonSA's author is the only editor of the XSLT 2.0 specification and that instead of his mail address the specification includes an URL pointing to the website selling SaxonSA ...) Galax is an open source (IBM CPL / Lucent license) OCaml implementation of XQuery which is not fully conformant to the specification (though it gets quite close) which is type-aware and implements static typing. The only XQuery / XSLT 2.0 implementation available in Debian at the time of writing is SaxonB. The binary package is libsaxonb-java, kudos to Michael Koch and the Debian Java Maintainers for having packaged it (and to have stood some annoying pings of mine bug #408842). To execute XQuery code you just have to aptitude install libsaxonb-java, prepare a query.xq file containing your query, and then execute something like:

CLASSPATH=/usr/share/java/saxonb.jar \
java net.sf.saxon.Query query.xq

note that the information in README.Debian are still referring to old Saxon versions, see bug #465894 which proposes a more up to date README.Debian. Similarly, to perform a XSLT 2.0 transformation you have to do something like:

CLASSPATH=/usr/share/java/saxonb.jar \
java net.sf.saxon.Transform -ext:off -s:input.xml -xsl:style.xsl -o:output.xml

Do not remove the -ext:off flag when processing untrusted stylesheets!, see bug #465885 for the reason. I've written some handier (1-liner) shell script helpers which remove the need of invoking java manually. They are attached to bug #465894 and I've proposed their addition to the saxonb package. Using them the above invocations become:

saxonb-xquery query.xq
saxonb-xslt -ext:off -s:input.xml -xsl:style.xsl -o:output.xml

Galax in Debian ... well, since long time I've been planning to package it (the ITP has been filed some months ago: bug #447984) and the authors sent me a newer version than what is available online to gather feedback before the long overdue final release. Unfortunately I've been lagging behind in finishing the packaging (which is tricky due to the need of binding libraries to several different languages: OCaml is native, but there is also Java for example). Hopefully this post will give me some renewed motivation for finishing the work ...

---

References

• Debian package w3-recs, ships the whole list of W3C Recommendations for offline consultation; it includes all the specifications we have discussed in this post
• Saxon: home page of Saxon, a XSLT 2.0 / XQuery processor written in Java
• Galax: home page of Galax, a XQuery processor written in OCaml
• Debian package libsaxonb-java, ships SaxonB as a Debian package

---

Acknowledgements Thanks to godog for his helpful comments.

---

Update: the helpers I've proposed have been accepted into the official package, I've also made available their manpages. Also the pending changes to README.Debian has been accepted. Kudos again to Michael Koch for his quick feedback on my patches!

As Josep asks for it every now and then, and we really enjoyed this trip, it's time to write about the four days we spent cycling from Ripoll to Girona, through the Ruta del Carrilet, a Catalan via verda similar to the one we completed a year ago. I spent this year's 9 d'Octubre cycling trip with totally different trip mates, not related to my triathlon team. The extensive group was formed by Sabri, Mar, B rbara, Carles, Desi, Adela, N ria, Amador and myself. We started our journey by car from Val ncia to Mollet de Mar, where we parked the cars and take a regional train to Ripoll with our bikes, where the real journey would start.

Arrival at Ripoll When we got to Ripoll, it was quite late and dark, but we managed to find the start of the Ruta del Ferro rail trail which would take us to Sant Joan de les Abadesses. It was quite cold, but especially really humid, so we had to think twice before settling on a place to camp and setup our tents. We had to take special care of covering the bicycles as the air was really wet, and anything we left uncovered would appear soaking next morning. In part, my shoes suffered from this.

Early next morning we had breakfast on the wet grass and then started our way to Sant Joan through a deliciously well equipped and maintained cycling track, which unfortunately only lasts for 12 kilometres through a splendind landscape. After having breakfast in the old railway station in Sant Joan, we abandoned the Ruta del Ferro to take a road to Olot via Sant Pau de Seguries, where we had to climb a small mountain and then descend through the Vall de Bianya until we got to Olot.

Sant Joan de les Abadesses' railway station Asking the locals for the start of the Ruta del Carrilet was fun due to the strong oriental accent in the area and we sometimes wouldn't get a word of what we were told. An old man recommended us finding a big park in the outskirts of the town, where we prepared our entrepans for lunch, and after a little rest, we started cycling up the Carrilet, which would take us to Girona in two stages. This Catalan area is of volcanic origin, and there are several natural parks dedicated to the phenomenon. There are many mountains covered with forests and impossible peak shapes, which due to the time of the year were starting to go from green to brown and yellow, making fantastic colour schemes in the landscape.

Under a green ceiling in Olot Just after the few first kilometres, the railroad track starts to descend most of the way, which makes it easy for people who aren't too trained, and easier to enjoy your way chatting with people while you cycle. As we consumed the daylight, we came across a parish church party with lots of old and young people from the nearby town of Sant Miquel de Pineda. Amador and I were lagging behind a bit at that point, and when we got there, we found the rest of the group were already off their bikes and either having a curious look over the party, or directly following suit and dancing like the others. We had a peek into the small church and also into a small graveyard in the back, where I was surprised to find the enemies of my Catalan grandmother: Fam lia Matabosch , as one of the headstones revealed.

Partying at Sant Miquel de Pineda As it was getting late, we eventually started off again, but luckily B rbara spotted what seemed a good sleeping place for that night. To the right of the trail there was a hermitage (devoted to Santa Cec lia) up on the hill which, for bonus points, had a porch with a recently renewed roof which would help cover ourselves from humidity and rain, as we found soon after starting cooking dinner. While the cooks prepared our soup, some others went down to the Carrilet in order to find out if the next town was close so we could get some driking water for the dinner. Luckily, Sant Feliu was close enough, and we found a bar where we got some water... and two bottles of wine, which made some people back at the hermitage very happy. We were quite tired though, so we eventually went to sleep after reading some stories from a great tale book, with the sound of rain hitting the grass outside the porch. The rain was still with us when we started waking up, so we had breakfast and packed our stuff very slowly, in an attempt to avoid getting wet. But rain didn't stop, so we made a few hacks on our bicycle bagpacks to minimise the amount of rain wetting them. When we were finally ready to set off and had cycled around 3 kilometres under the light rain, it finally stopped raining, making the rest of the journey very pleasant.

Sleeping under the porch of Santa Cec lia's hermitage This part of the route is again quite beautiful, with impressive amounts of green vegetation at both sides of the track. There's a lot of water presence in the land, which makes the type of trees and bushes quite different to the ones we are used to find in the Valencian Country. After going by a few small towns, we arrived in the old railway station of Amer, hometown of the Puigdemonts. Even if it was a bit early, we decided to have lunch there, so a few of us went down to the centre of the town to buy bread and some other details to eat. Being in my friend Josep's town for the first time, I wanted to visit his family's bakery, but unfortunately it is closed on Mondays. We had lunch back at the cute station, and eventually kept going on our way to Girona. Outside Amer, the Carrilet was temporarily cut by the road, and there were two possible alternatives: using the road, with heavy lorry traffic, or diverting through a very steep track with very hard slopes. Most of the group preferred the road, but Mar, Amador and I went up that track, which was hard to climb and really fun to descend. Apparently, Amer locals call those 1.5 kilometres the Tourmalet . It's probably not so bad, though. :)

Amer's railway station Unfortunately, the Ruta del Carrilet starts getting uglier after Amer, and as you approach the more industrialised towns near the capital Girona. The area around Angl s wasn't that fun, with the track continuously being invaded by cars and other vehicles. Soon after we were in the farmland area surrounding Girona and its Devesa. Jonathan was waiting for us in the city, after his two month stay in Scotland, and we were all happy to meet him. We discarded continuing our way to the coast, which was the initial plan, and instead dropped our stuff in the house of one of Adela's friends, and went out to have dinner to a Wok restaurant, where the poor people running it suffered our childish behaviour involving the rotating central dish on our table and custard sucking contests. Jonathan guided us through the Cathedal and city wall areas of the town centre, which are impressive, and we climbed all of the wall towers to have great looks over the dark and quiet city. Eventually, we went to sleep, as Adela had to leave early, and we had to pack to get our train back to Barcelona. Back in Mollet, we cooked our last camping-gaz lunch in the park right next to the station, before noticing the place stinked of dog poo. After the careful operation of filling the three cars with 9 bicycles, we were finally on our way back to Val ncia, completing another great cycling journey. If you like bicycle tourism and can travel to this area in Catalonia, I highly recommend it, as there's many great places to visit, all of them accessible with bikes.

Having lunch in Mollet

after my laptop I tried to get 2.6.24 running on my desktop machine; several attempts (Debian kernels, self-compiled ones with different options) all led to the same results: a machine that feels rather unresponsive & where music has sporadic dropouts.

some searching on the intarweb showed that the new CFS scheduler (cf. Documentation/sched-design-CFS.txt) (at least the enabled-by-default CONFIG_FAIR_USER_SCHED flavour) might have latency problems, in itself or in combination with the ondemand CPU frequency scaling governor or with nice'd processes; after turning off all CONFIG_FAIR_* options in .config, recompiling & rebooting the machine seems to behave normally.

maybe I'm missing some great new features but actually I prefer a working desktop computer :)

update 1: approximately half an hour after the original blog entry I had my first "ion sorbet", i.e. a half-frozen ion3 (the applications are still running, I can switch to a tty but I cannot change/close/whatever ion3's frames). at the moment I have no idea what's going (wr)on(g) here ...

update 2: two more "ion sorbets" today yet; let's see if a new kernel (CONFIG_FAIR_USER_SCHED seems to be fixed in 2.6.24.1) & a new ion3 version help.

update 3: still the same problems (& inbetween some glitches with switching from X to a tty and back; I hope I solved them with some kernel config options); ATM I suspect ion3 to be the culprit; when it hangs it eats almost all the CPU, & strace'ing the pid gives only SIGALRM messages.

update 4: maybe I fixed the problem; by changing from ion3's deprecated statusbar_external.lua to the newer statusd_exec.lua (I just wanted to get rid of the error when starting ion3 ...).

Earlier today I watched the presentation Jonathan Oxer's gave during LCA about Package caching solutions. Although it was certainly an interesting presentation and although I very much agree that my current local mirror is wasting a lot of diskspace and bandwidth, I'm still not going to switch from debmirror to any of the available caching solutions, because (unless I'm really missing something) none of them scratches my itch. My local mirror currently consists of five architectures (i386, amd64, hppa, sparc and s390) and only has unstable and testing. I use it for:

1. (fast and convenient) updating of my systems
2. doing Debian Installer builds and installation tests
3. test builds of installation CDs (using debian-cd)

Now, the last one is somewhat hard (debian-cd uses hardlinks to packages on a local mirror instead of retrieving packages), so let's concentrate on the first two. Caching is great if you have a large number of machines – of the same architecture and that are all likely to need roughly the same packages – sitting behind the proxy: the first one triggers the download of the package and the rest gets it almost instantaneously. It is a lot less great if you have only one, maybe two systems per architecture: most of the time you'll still end up going down that (relatively) slow ADSL connection. An important reason why I have my mirror is so that when I do my daily updates for sid or run an installation test, the packages are already available locally. I really don't want to double or even treble the time needed for installation tests just because some required packages aren't yet available locally and need to be downloaded over that slow line. So, I have my partial mirror. Somewhat tuned (I exclude some ridiculously large debug packages for example, saving about 10GB), but still with a lot of junk^Wpackages on it I'll never ever use, especially for hppa, sparc and s390 as those systems only have fairly basic installations. Getting rid of that would significantly reduce my daily sync and allow me, for example, to also have a mirror of stable and oldstable, keep old versions of packages and probably still save a lot of diskspace. Wishlist What we should have is a hybrid solution: a program that will present itself as a proxy to clients, but is smart enough to pre-fetch new packages that are likely to be needed in a sync run, based on usage date from the the proxy and configuration settings. Some ideas for features/configuration options it could have:

• should support both source and binaries (debs and udebs)
• transparently retrieve packages that are requested but not available locally (just like a regular proxy)
• options to always sync (per architecture):
  • all required and standard packages
  • packages listed in a certain config file
  • packages in a certain section (e.g. udebs in the debian-installer section)
• for packages that have been used (requested by clients) within the last X days: pre-fetch any new version (per architecture)
• options to include security/volatile updates in that
• be smart about ABI changes: when a package name changes because the ABI version changes, also pre-fetch the new package; an alternative solution for that could be:
• for recently used packages: ensure that their dependencies are also pre-fetched; this would also provide support when packages are renamed (dependencies in the transition packages would ensure the new packages are pre-fetched)
• expire packages from the mirror that have not been used for X days
• allow faster expiration of certain types of packages (-dev or -dbg for example)
• options to keep X previous versions that are no longer referenced in the Packages files for Y days

Possibly such an implementation could even be used on some of the lower tier Debian mirrors. Unfortunately, unlike some of my esteemed colleagues, I'm not able to just whip up something like this, so I'm condemned to wait and see if there's someone else who'd like to pick up this idea. I am of course more than willing to help develop this idea further and to test it. Now, if I've totally failed in my research and something like this already exists, a pointer in the right direction would be much appreciated.

Cannot help but laugh and share.
I've been triaging and trying to reproduce some oldish bugs on pkg-perl's packages. Some bugs are no longer there, some have to be forwarded upstream, and so on. Usual tasks, yes.
Until I stumbled with #406227. I just have to laugh and share! Hope nobody feels ashamed - The bug is the result of different people coding maybe under pressure and with quite different mindsets :)
For some reason I fail to understand, the submitter's test case (rot13 implemented over a HTTP proxy) is invoked in the report as ./rot13 2>/dev/null. Of course, when trying to debug a bug report, the first thing to do is not to ignore STDERR. So, off goes the 2>/dev/null. What happens next? 0 gwolf@mosca[2]/tmp$ perl ./rot13 &
[1] 4394
0 gwolf@mosca[3]/tmp$ GET -p http://localhost:8080/ http://www.debian.org/
Can't locate object method "filter" via package "UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)" (perhaps you forgot to load "UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)"?) at /usr/share/perl5/HTTP/Proxy/FilterStack.pm line 126.
500 EOF when chunk header expected WTF... Well, at least the program name gives me a clue... Lets try to "decrypt" the error message... gwolf@mosca[4]/tmp$ echo 'UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)' rot13
HTTP::Proxy::BodyFilter::simple=HASH(0x604160) hrm... How comes the filter is filtering its own code and only then refusing to find itself!? Ok, time to open up the manpage - Remember, I'm only group-maintaining this pacakge, I am not yet at all familiar with it! Ok, so the core of the filter is when the submitter states: my $proxy = new HTTP::Proxy();
$proxy->push_filter(response => new HTTP::Proxy::BodyFilter::simple(sub tr/a-zA-z/n-za-mN-ZA-M/; )); While the manpage states it should be invoked as: my $filter = HTTP::Proxy::BodyFilter::simple->new( sub $ $_[1] =~ s/foo/bar/g; );
$proxy->push_filter( response => $filter ); Of course, once looking at it, the answer is simple: The submitter left out which element to act on in the anonymous function body - The $ $_[1] =~ part. Adding it makes gur svygre jbex nf rkcrpgrq... Err, sorry - makes the filter work as expected. Now, bonus points: For the non-Perlers out ther: How come we get the namespace translated as well? Oh, that's very simple: In Perl, as in Python (and concievably other languages I'm unaware of), the object is passed to any of its methods as the first argument. Functions in Perl receive their arguments via @_ (read: the default array). And, of course, the tr (regex-based transliteration) takes by default the first thing it sees - the object itself. And what happens when you apply a (string-oriented) regex to an object? Of course, it gets stringified - which, by default, in Perl means converting it to the closest possible description: "a hash reference blessed as an object of the class such-and-such at this memory location". That string gets worked on, and we get UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160). This proxy does not die on very-very-short web pages, where the whole content fits on one iteration of the code (although it does not work correctly - the text remains unaltered, of course, as it was not worked on), but if the request spans several chunks, the second time the filter is called, it will be... just gibberish. Oh, and what about the extra $ (...) around $_[1]? Oh, simple: The string is passed as a scalar reference, so it can be modified in place. Yes, it's the Perl way of pass-by-reference instead of pass-by-value (the default behaviour): Of course the parameter is only passed as a value. Only that the value is incidentally a reference - but who cares? ;-) Anyway... Many oddities. I would implement the module in a completely different way, and it looks quite backwardish in my book. But then again, TIMTOWTDI.

I am very fond of Jean-Yves Lefort s link-monitor-applet, which puts little bars in my panel that display the ping time to certain hosts. Very useful, especially since I m often using a shaky wireless connection.With the latest release, though, the author added an icon next to these bars. He argued that the small bars, if there are only few of them, are hard to hit with the mouse, and therefore added this icon as a usability improvement for less advanced users.But for me, this makes the applet almost unusable, as I just don t have the space left for a useless icon. Things are here proably worse because the icon is higher than wide, scales to the maximum size that fits the panel and my panel is vertical and slightly wider than the normal horizontal panel is wide.I asked the author to make this behaviour configurable or to hide the icon when there are more than three hosts to display (with patch included), but he wanted to stick to the icon. So what are my options? I could manually download and extract the source, apply the patch and build the package every time a new version is uploaded to Debian but shouldn t this be easier?If Debian were not 20 but 2 years old I assume that all sources would be in one big git/darcs/hg/whatever repository, with something like the SuSE build service, which would automatically build the packages, and I could make my system automatically track a branch of the real Debian with my patches added. But at the moment, this would be utopical.I could imagine creating something smilar that would track the uploads to debian and, if link-monior-applet is uploaded, tries to automatically build a package with my modification, providing this in an apt repository that has a high apt priority for me. Has anyone done that already, or tried it, or thought about that?

Currently in the US the main political parties are deciding who will contest the next presidential election. Naturally this gets some commentary from all sides. Planet Debian has syndicated two blog posts commenting on these issues, it’s interesting to compare them: First John Goerzen writes a post about an issue he (and almost everyone in the Debian community) considers important - copyright law [1]. He quotes the Randall Munroe who is the author of www.xkcd.com (described by the author as “A webcomic of romance, sarcasm, math, and language“) which is wildly popular in the free software community (I believe that there is a fan-club which has meetings). Randall’s commentary is quite interesting and I recommend reading it [2]. The most noteworthy fact is that Barak Obama has sought advice from Lawrence Lessig - who has done more for the free software community than any other lawyer. John’s post doesn’t contain much original content, but citing a lengthy post from a highly regarded source and quoting a particularly relevant paragraph make it very noteworthy. Next Jaldhar Vyas writes about “the war against Islamic terrorism” [3] and makes the claim “we are finally gaining the upper hand in Iraq“. To consider that claim it’s best to read some expert commentary. William S. Lind [4] is a widely respected conservative who is an expert on military strategy, his comments about stabilising Iraq are always interesting [5]. It’s sufficient to note that William has been predicting failure for the US occupation of Iraq right from the start and that events over the last five years have proven him correct. William S. Lind’s thoughts on Democracy are interesting too [8], I don’t agree with him in this regard (I generally disagree with him on most things other than military matters) but it’s an interesting thought. Jaldhar also claims that “Democrats on the other hand have a closer association with the media who are behind many of the sillier IP laws“, this seems to be a reference the the “liberal media” conspiracy theories. There seems no clear party association with bad IP legislation (both the Democrats and Republicans do silly things in this regard), and debunking the liberal media claims can be done simply by watching some TV (choose a couple of random news shows on different channels). Fox is not the only right-wing news outlet. Jaldhar also has a conspiracy theory about the Supreme Court and the “need to ensure that the court does not veer leftward again as it will inevitably do during a Democratic administration“, not that the Democratic party is particularly “left” by whatever definition you might apply to the word. I prefer the www.politicalcompass.org analysis which shows that all the serious contenders in the Democratic primary are Authoritarian Right [6]. So the result of the next US presidential election will determine how authoritarian and right-wing the government will be, the fact that it will be authoritarian and right-wing is beyond doubt. That said it would be good if the authoritarian right-wing government that we (*) get at least has some decent policies in regard to IP law (which is the hope for Barak Obama). John Nichols has an interesting analysis of Ann Coulter’s support of Hillary Clinton over John McCain [7]. It would be interesting to see Jaldhar’s comments on this issue. (*) I use the term “we” when talking about US politics to acknowledge the fact that the Australian government will take orders from Washington. Some pundits predict that the Greens will be the second major party in Australian politics in a couple of elections time, so maybe in 8 years time we will have an Australian government that represents Australians instead of Americans. But for the entire duration in office of the next US president (be it 4 years or 8) they will be able to get the Australian government to do almost anything that they desire.

• [1] http://changelog.complete.org/posts/683-xkcd-author-endorses-Obama.html


• [2] http://blag.xkcd.com/2008/01/28/obama/


• [3] http://www.braincells.com/debian/index.cgi/search/item=181


• [4] http://en.wikipedia.org/wiki/William_S._Lind


• [5] http://dni2.wordpress.com/2008/01/23/on-war-247-crossing-the-channel/


• [6] http://www.politicalcompass.org/usprimaries2008


• [7] http://www.thenation.com/blogs/thebeat?bid=1&pid=278859


• [8] http://dni2.wordpress.com/2008/01/29/on-war-248-my-masters-voice/

Share This

I would like to suggest to my readers to ask airplane crews for explanations of their rules. If we can get a larger number of people to inquire about the reasons behind the do s and dont s on airplanes, maybe the airline companies will adopt the practice. In the context of a previous post on the lack of explanation of the motivations behind airline rules, I was utterly impressed when the steward on the Air New Zealand flight from Melbourne to Wellington asked me to turn off my music player so that I would hear when they asked us to evacuate the plane or similar. While I doubt that I would continue to listen to The Flower Kings in an exceptional event, his explanation actually got me to turn off the player, which I had previously never done (rebel me!). I know it s a bit ridiculous, but I was previously so set on the idea of small devices like a music player interfering with the airplane instruments that I failed to see this obvious bit of logic. The steward thus gets my best steward award. When you fly, ask the crew about the reasons for the rules they impose on you, the passengers! NP: The Flower Kings: Unfold the Future

New Jersey is one of the states holding presidential primaries on this Super Tuesday and since you asked, I voted for John McCain. The chief reason for my vote is of course the war against Islamic terrorism. Now that we are finally gaining the upper hand in Iraq, we can't afford to elect someone who will fail to see victory through. Mind you this isn't necessarily a reason not to vote Democrat. The anti-war wing of that party has been thoroughly marginalized. Hillary atleast has a commendable reputation for viciousness. Even Obama has made some encouraging remarks (ok that was probably a red herring but I have the audacity to hope you see.) But while the Democrats do not take the anti-war crowd seriously, they do depend on them for a great deal of their funding. A Democratic president might feel the need to throw them some bones inadvertantly screwing up things in the process. On the Republican side, Huckabee might leave tactical planning to Jesus. Romney would toe the line but there is no sign that he has any original thoughts on the matter. While I continue to support the President, it must be admitted that this whole war is stretching out as long as it is because mistakes were made. We need options beyond "staying the course" and "cutting and running" and McCain a military hero and early supporter of the surge could be the one to find them. The other big issue is supreme court appointments. One, possibly two justices may need to be replaced in the next four years. We need to ensure that the court does not veer leftward again as it will inevitably do during a Democratic administration. Here there is a big split in the Conservative establishment. Ideologues consider McCain to be insufficiently pure and worry he will end up appointing another Souter or O'Connor (both appointed by Republicans it should be noted.) Pragmatists point out that McCain is viewed more favorably than Romney or Huckabee amongst the independent and moderate voters who will be crucial to victory and that even the slight chance of a bad choice is better than the certainty of a bad choice if Clinton or Obama get to make it. Social issues are important to many Republican voters and in this area too there is some grumbling about McCain being too liberal. Some have even suggested that if McCain beats Huckabee or Romney, a Democratic victory and four years out of power would be preferable in order to bring the GOP back to the straight and narrow. Although this contingent is loud I frankly don't think they have the influence people think (As evidenced by the actual performance of Huckabee or previous champion Fred Thompson.) When push comes to shove, they will rally behind the party candidate especially if McCain makes a bona fide social conservative (Huckabee himself?) his vice-president. On economic issues, the president's job is to not interfere and let business run itself. I don't see any of the Republican candidates being much different in this regard. Or the Democrats for that matter. [Careful readers will have noticed I have avoided mentioning Ron Paul. Paulestinians, consider this carte blanche to send me your long, rambling, incoherent screeds about how Dr. Paul has the answers to everything. Because God knows I haven't read enough of them lately.] Update: Kevin asks what I think about the candidates in terms of "geek" issues such as net neutrality, DMCA etc. IMO there is not a whole lot of difference between the major candidates on these issues. Republicans one would think would be more beholden to corporations in general. Democrats on the other hand have a closer association with the media who are behind many of the sillier IP laws. Overall, I don't think it ought to be much of a factor in choosing the president even for Free Software types. Rather we should focus on the legislative branch (in Congress as well as the states) which holds the real power here and in the court of public opinion.