XQuery 1.0 and XSLT 2.0: news and how to use them in Debian (long post about the second generation of the XSL family and how to use the related languages in Debian)

• XPath 2.0, XQuery, et al.
• the above stuff .. in Debian
• links

---

The 2nd generation of the XSL family At the beginning of 2007, W3C released a family of interrelated specification: XPath 2.0, XSLT 2.0, XQuery (1.0). The 3 specifications are based on the very same underlying data model which (finally!) supports typed values and exploits them to some extent, for example permitting some form of static type checking in XPath and XQuery. Types usually come from XML Schema, with both built-in and user-defined types supported. A brief overview of the 3 specifications and what they change in the state of the art follows. XPath 2.0 XPath 2.0 basically consists in XPath 1.0 + 3 new macro features. The first one is a revamp of those language features which are dealing with types, in order to better integrate them with the new typed data model. XPath now has operators to check whether a value belongs to a given type (so yes, type information are kept at run time), and to cast them from one type to another. The basic new type constructor is now the sequence, which replaces the old node-set solving many annoying issues, such as the impossibility of having a node ordering other than document order. The second feature is the improvement of standard library of functions which was ridiculously small in XPath 1.0; it is much better now. Additionally, 2nd generation languages (XSLT 2.0 and XQuery) now support the ability to define functions which will then be visible to inner (XPath) expressions, pushing yet forward the possibilities of plain old XPath. Functions can now also declare types in their signatures (for their arguments and return value) and untyped arguments will be automatically casted to them upon invocation. Hence, even if you are not using an implementation which assigns types to your XML trees, once you "enter" the typed world calling a typed function (almost all standard library functions are decently typed) you will be able to stay there avoiding annoying casts everywhere. Finally XPath 2.0 has turned into a powerful purely functional language and is now powered by constructs like conditionals, for-each loops, existential/universal quantifiers, and existentially-quantified comparison operators for sequences. Here is a complex expression to hwet your appetite (or scare you away ...), comments come as (: smiley faces :)

for $book in /bookshelf//books
return
  if ((every $author in $book/authors/author
       satisfies $author/nativeLang eq "it_IT")
      and $book/lang eq "it_IT")
  then $book
  else ()
(: think about the trouble of writing this in XSLT/XPath 1.0 ... :)

XSLT 2.0 XSLT 2.0 is what I would call a "bug fix release" of XSLT 1.0 + the routinary reworking of the language to deal with typed values, which is not sensibly different than what has been done for XPath 2.0. The fixed "bugs" are several, starting from the annoying issue of result tree fragments. They are basically tree snippets that in XSLT 1.0 you were able to generate for future use. Unfortunately they were not thaaat reusable, given that you were not even able to navigate them with XPath operators! Now the specification is much more clear and distinguishes final result trees from ordinary variables, which can now contain sequences of (navigable) tree nodes. Another important "bug" fixed is the new ability to output multiple documents with a single XSLT stylesheet: it marks the end of stupid extra post processing to be added in pipeline to a XSLT processor. Other minor "bugs" fixed are a limited amount of backtracking capabilities among imported templates, regular expression support directly in the language, and powerful grouping constructs on the lines of SQL's GROUP BY (but much more powerful). Here is a template snippet exploiting the latter feature:

<xsl:for-each-group select="*" group-starting-with="h1">
  <div>
    <xsl:apply-templates select="current-group()" />
  </div>
<xsl:for-each-group>

XQuery XQuery is the end of the chains imposed by XML-based syntaxes. Why the heck one has to use an XML syntax (as in the above snippet) only because she is manipulating XML tress is one of the mysteries of XML technologies which have always been floating around in my head. XQuery is the (supposedly) SQL equivalent for databases of XML documents, but is actually much more than that. I depict it in my head as the XML manipulation language with a syntax I can finally stand. Technically it is XPath 2.0 (say 80% of the whole language) + some extra ingredients (say 20%); so remember that every XPath 2.0 expression is also a XQuery expression. The main extra ingredient is the so called FLWOR expression (to be read: "flower expression", which in addition to the "smiley faces" used for comments gives a "back to 1968"-flavour to the language .... erm FLWOR/flower/flavour, no pun intended). A FLWOR expression is very similar to SQL's SELECT-FROM-WHERE: it lets you generate a tuple stream by iterating on sequences (F: for clauses), binding expression values to names (L: let clauses), filter out tuples which do not satisfy a required condition (W: where clause), order the survived tuples (O: order by clause), and finally return a sequence built using the residual tuple stream (R: return clause). The other interesting extra ingredient is the ability to build the XML snippets you want to manipulate. Within XQuery you do that using plain XML syntax (the only place where a sane-minded programmer actually wants to see it!) which also supports a classical interpolation mechanism to embed expressions which will be evaluated inside XML snippets, and also the other way around. A canonical XQuery example is:

for $t in doc("books.xml")//title,
    $e in doc("reviews.xml")//entry
where $t = $e/title
return <review>  $t, $e/remarks  </review>
(: braces denote the escaping context where XQuery expressions will
   be evaluated inside snippets; plain XML syntax is used for the
   other way around :)

But remember: XQuery for XML is much more than SQL for RDBMS, thanks to the implicit templating mechanism implemented by interpolation, and thanks to several language features fostering modularity (user-defined functions, library modules, XPath 2.0 standard library, ...) you can basically do with it any kind of XML manipulation you can imagine. I don't think I will ever write myself any other single line of XML output in DOM or XSLT ...

---

Cool, how can I use it in Debian? ... if only this stuff were decently supported in the open source world. Last time I checked, the author of most parts of the GNOME toolchain for dealing with XML (libxml2, libxslt, ...) was not intentioned to implement XSLT 2.0, not even mentioning XQuery. This comes as no surprise, the whole GNOME XML toolkit is written in C, and XSLT 2.0 / XQuery have reached a level of complexity and formal specification which usually entails a higher level approach. So on the GNOME side we are stuck. The other open source implementations of XSLT 2.0 / XQuery I'm aware of are Saxon and Galax. Saxon is an XSLT 2.0 and XQuery implementation written in Java, which is unfortunate per se. Additionally, it is also unfortunate that it is only partially open source. Indeed, Saxon is split into SaxonB (for "basic") which is open source under the Mozilla Public License and SaxonSA which is commercial. While SaxonSA is a fully conformant, XML Schema-aware processor, with support for static typing, SaxonB is a basic-conformant processor with no type-aware features and actually much less optimized than SaxonSA. This is annoying. (Much more annoying is the fact that SaxonSA's author is the only editor of the XSLT 2.0 specification and that instead of his mail address the specification includes an URL pointing to the website selling SaxonSA ...) Galax is an open source (IBM CPL / Lucent license) OCaml implementation of XQuery which is not fully conformant to the specification (though it gets quite close) which is type-aware and implements static typing. The only XQuery / XSLT 2.0 implementation available in Debian at the time of writing is SaxonB. The binary package is libsaxonb-java, kudos to Michael Koch and the Debian Java Maintainers for having packaged it (and to have stood some annoying pings of mine bug #408842). To execute XQuery code you just have to aptitude install libsaxonb-java, prepare a query.xq file containing your query, and then execute something like:

CLASSPATH=/usr/share/java/saxonb.jar \
java net.sf.saxon.Query query.xq

note that the information in README.Debian are still referring to old Saxon versions, see bug #465894 which proposes a more up to date README.Debian. Similarly, to perform a XSLT 2.0 transformation you have to do something like:

CLASSPATH=/usr/share/java/saxonb.jar \
java net.sf.saxon.Transform -ext:off -s:input.xml -xsl:style.xsl -o:output.xml

Do not remove the -ext:off flag when processing untrusted stylesheets!, see bug #465885 for the reason. I've written some handier (1-liner) shell script helpers which remove the need of invoking java manually. They are attached to bug #465894 and I've proposed their addition to the saxonb package. Using them the above invocations become:

saxonb-xquery query.xq
saxonb-xslt -ext:off -s:input.xml -xsl:style.xsl -o:output.xml

Galax in Debian ... well, since long time I've been planning to package it (the ITP has been filed some months ago: bug #447984) and the authors sent me a newer version than what is available online to gather feedback before the long overdue final release. Unfortunately I've been lagging behind in finishing the packaging (which is tricky due to the need of binding libraries to several different languages: OCaml is native, but there is also Java for example). Hopefully this post will give me some renewed motivation for finishing the work ...

---

References

• Debian package w3-recs, ships the whole list of W3C Recommendations for offline consultation; it includes all the specifications we have discussed in this post
• Saxon: home page of Saxon, a XSLT 2.0 / XQuery processor written in Java
• Galax: home page of Galax, a XQuery processor written in OCaml
• Debian package libsaxonb-java, ships SaxonB as a Debian package

---

Acknowledgements Thanks to godog for his helpful comments.

---

Update: the helpers I've proposed have been accepted into the official package, I've also made available their manpages. Also the pending changes to README.Debian has been accepted. Kudos again to Michael Koch for his quick feedback on my patches!

Bruce Schneier blogs about an issue that, in the context of the current vote craze in the US, seems to be pretty popular: the question of whether or not the US governments should give a drivers' license to illegal immigrants. Bruce argues in favour of giving a drivers' license to illegal immigrants, since requesting that the administration which issues drivers' licenses also check whether or not people are illegal immigrants is going to cost the government a lot of money (both from the extra amount of resources that needs to be spent on doing this job, and the lawsuits to make up for the errors). The fun bit is, I think that they are creating a problem where there is none. By not creating an official, designated, identification system, the US government is forcing people to look for alternatives, effectively giving a drivers' license a double function; that of an ID card, and that of a drivers' license. This can't be a good idea. Often the major argument against issuing ID cards is that it reduces people's privacy. I think this is wrong, for a very simple reason:

The US already has an ID card. It's called a drivers' license.

No, really. For all practical matters, that little piece of paper is an ID card. Better get done with it and issue a real ID card, then.

As Josep asks for it every now and then, and we really enjoyed this trip, it's time to write about the four days we spent cycling from Ripoll to Girona, through the Ruta del Carrilet, a Catalan via verda similar to the one we completed a year ago. I spent this year's 9 d'Octubre cycling trip with totally different trip mates, not related to my triathlon team. The extensive group was formed by Sabri, Mar, B rbara, Carles, Desi, Adela, N ria, Amador and myself. We started our journey by car from Val ncia to Mollet de Mar, where we parked the cars and take a regional train to Ripoll with our bikes, where the real journey would start.

Arrival at Ripoll When we got to Ripoll, it was quite late and dark, but we managed to find the start of the Ruta del Ferro rail trail which would take us to Sant Joan de les Abadesses. It was quite cold, but especially really humid, so we had to think twice before settling on a place to camp and setup our tents. We had to take special care of covering the bicycles as the air was really wet, and anything we left uncovered would appear soaking next morning. In part, my shoes suffered from this.

Early next morning we had breakfast on the wet grass and then started our way to Sant Joan through a deliciously well equipped and maintained cycling track, which unfortunately only lasts for 12 kilometres through a splendind landscape. After having breakfast in the old railway station in Sant Joan, we abandoned the Ruta del Ferro to take a road to Olot via Sant Pau de Seguries, where we had to climb a small mountain and then descend through the Vall de Bianya until we got to Olot.

Sant Joan de les Abadesses' railway station Asking the locals for the start of the Ruta del Carrilet was fun due to the strong oriental accent in the area and we sometimes wouldn't get a word of what we were told. An old man recommended us finding a big park in the outskirts of the town, where we prepared our entrepans for lunch, and after a little rest, we started cycling up the Carrilet, which would take us to Girona in two stages. This Catalan area is of volcanic origin, and there are several natural parks dedicated to the phenomenon. There are many mountains covered with forests and impossible peak shapes, which due to the time of the year were starting to go from green to brown and yellow, making fantastic colour schemes in the landscape.

Under a green ceiling in Olot Just after the few first kilometres, the railroad track starts to descend most of the way, which makes it easy for people who aren't too trained, and easier to enjoy your way chatting with people while you cycle. As we consumed the daylight, we came across a parish church party with lots of old and young people from the nearby town of Sant Miquel de Pineda. Amador and I were lagging behind a bit at that point, and when we got there, we found the rest of the group were already off their bikes and either having a curious look over the party, or directly following suit and dancing like the others. We had a peek into the small church and also into a small graveyard in the back, where I was surprised to find the enemies of my Catalan grandmother: Fam lia Matabosch , as one of the headstones revealed.

Partying at Sant Miquel de Pineda As it was getting late, we eventually started off again, but luckily B rbara spotted what seemed a good sleeping place for that night. To the right of the trail there was a hermitage (devoted to Santa Cec lia) up on the hill which, for bonus points, had a porch with a recently renewed roof which would help cover ourselves from humidity and rain, as we found soon after starting cooking dinner. While the cooks prepared our soup, some others went down to the Carrilet in order to find out if the next town was close so we could get some driking water for the dinner. Luckily, Sant Feliu was close enough, and we found a bar where we got some water... and two bottles of wine, which made some people back at the hermitage very happy. We were quite tired though, so we eventually went to sleep after reading some stories from a great tale book, with the sound of rain hitting the grass outside the porch. The rain was still with us when we started waking up, so we had breakfast and packed our stuff very slowly, in an attempt to avoid getting wet. But rain didn't stop, so we made a few hacks on our bicycle bagpacks to minimise the amount of rain wetting them. When we were finally ready to set off and had cycled around 3 kilometres under the light rain, it finally stopped raining, making the rest of the journey very pleasant.

Sleeping under the porch of Santa Cec lia's hermitage This part of the route is again quite beautiful, with impressive amounts of green vegetation at both sides of the track. There's a lot of water presence in the land, which makes the type of trees and bushes quite different to the ones we are used to find in the Valencian Country. After going by a few small towns, we arrived in the old railway station of Amer, hometown of the Puigdemonts. Even if it was a bit early, we decided to have lunch there, so a few of us went down to the centre of the town to buy bread and some other details to eat. Being in my friend Josep's town for the first time, I wanted to visit his family's bakery, but unfortunately it is closed on Mondays. We had lunch back at the cute station, and eventually kept going on our way to Girona. Outside Amer, the Carrilet was temporarily cut by the road, and there were two possible alternatives: using the road, with heavy lorry traffic, or diverting through a very steep track with very hard slopes. Most of the group preferred the road, but Mar, Amador and I went up that track, which was hard to climb and really fun to descend. Apparently, Amer locals call those 1.5 kilometres the Tourmalet . It's probably not so bad, though. :)

Amer's railway station Unfortunately, the Ruta del Carrilet starts getting uglier after Amer, and as you approach the more industrialised towns near the capital Girona. The area around Angl s wasn't that fun, with the track continuously being invaded by cars and other vehicles. Soon after we were in the farmland area surrounding Girona and its Devesa. Jonathan was waiting for us in the city, after his two month stay in Scotland, and we were all happy to meet him. We discarded continuing our way to the coast, which was the initial plan, and instead dropped our stuff in the house of one of Adela's friends, and went out to have dinner to a Wok restaurant, where the poor people running it suffered our childish behaviour involving the rotating central dish on our table and custard sucking contests. Jonathan guided us through the Cathedal and city wall areas of the town centre, which are impressive, and we climbed all of the wall towers to have great looks over the dark and quiet city. Eventually, we went to sleep, as Adela had to leave early, and we had to pack to get our train back to Barcelona. Back in Mollet, we cooked our last camping-gaz lunch in the park right next to the station, before noticing the place stinked of dog poo. After the careful operation of filling the three cars with 9 bicycles, we were finally on our way back to Val ncia, completing another great cycling journey. If you like bicycle tourism and can travel to this area in Catalonia, I highly recommend it, as there's many great places to visit, all of them accessible with bikes.

Having lunch in Mollet

I enjoyed reading the ArchitectNotes on the Varnish wiki:

Take Squid for instance, a 1975 program if I ever saw one: You tell it how much RAM it can use and how much disk it can use. It will then spend inordinate amounts of time keeping track of what HTTP objects are in RAM and which are on disk and it will move them forth and back depending on traffic patterns.

Well, today computers really only have one kind of storage, and it is usually some sort of disk, the operating system and the virtual memory management hardware has converted the RAM to a cache for the disk storage.

So what happens with squid's elaborate memory management is that it gets into fights with the kernel's elaborate memory management, and like any civil war, that never gets anything done.

Varnish was designed by FreeBSD developer Poul-Henning Kamp. There is something to be said about user-space apps designed by kernel developers: they're always super fast because kernel developers know how to make the most of the system, having spent hours optimizing the kernel itself for some usage patterns.

This is why git (designed by Linus Torvalds, for those of you who've lived under a rock for the past three years) is so much faster than other comparable systems, for example. It uses tricks such as giving the O_NOATIME flag to open() to avoid updating access times on files where it knows that they don't matter, and it's a "huge time-saver". (Of course, you can also just mount your partitions with noatime and enjoy a similar performance boost in all applications!)

wouter@quickstep:~$ uname -a
Linux quickstep 2.6.24-1-mac #1 Fri Feb 1 00:14:53 CET 2008 m68k GNU/Linux

Ph34r my experimental buildd, which will be back... soon. Unfortunately, doing 'apt-get upgrade' from etch-m68k (the last release to support 2.2, which quickstep was still running) to unstable takes a while. Other than that... yeah! The update to 2.6.24 had some issues; e.g., the key map wasn't set correctly; also, 2.2 and 2.6 have different ideas about what the system clock is, so it insisted on doing an fsck at boot time initially, which with 18G hard disks takes quite a while on a 25Mhz processor. Quickstep was one of the last machines that I had which was still running 2.2; unfortunately it took a long time for 2.6 on m68k to become a viable option. Apparently, the issues that existed for Centris 650 (of which quickstep is one) had been fixed a while ago already; I just hadn't noticed. Oops. My final 2.2-running machine now is barok, my IIci. But that machine isn't very fast anyway, so its lack of 2.6 doesn't really hurt all that much. Having said that, Finn Thain told me he'd be working on IIci support soon, so that issue should be dealt with as well. Whee!

I don't think the first trip ever made a mention in either of our blogs. Funny how an exhausting trip tends to stem the care factor to write about it afterwards... Anyway, Sarah was in a fair bit of pain last night when she went to bed, and despite taking 800mg of Ibuprofen, some amount of Oxycodone, and some sleeping tablets, she was up again under 2 hours later, in agony. The main concern was that the staph infection she'd apparently had in the incision a few weeks prior hadn't been kicked fully by the antibiotics, and was coming back. So off we trotted to Stanford's ER, getting there around 1am. Fortunately it was very quiet (not that it matters when you mention the magic words "chest pain") and Sarah got straight in. They did the usual battery of tests and decided to keep her in for observation, so I trundled off home for some sleep at about 5am. They told her she could go home at about 1:30pm. Apparently the problem was muscle spasms, which can happen in some people after having heart surgery (even this long after the surgery). The solution? Valium. The poor girl's going to rattle soon with all these pills. They did some blood work and there's no sign of the staph infection. Whatever pain medication they gave Sarah intravenously today made her extremely nauseous, and she was still concerned about the state of the incision around the area that had been weeping previously, so we trekked back in to Stanford this afternoon to see the cardiologist, who wasn't concerned about the incision at all. In fact, he said the surgical doctor had been a bit generous with pronouncing a staph infection in the incision two weeks previously, saying that the swab they took at the time probably just picked it up off Sarah's skin, where it's quite common to be found. So she may never have had a staph infection in the first place.

The problem with the Internet is that it never goes to sleep. And if you study the Internet as I do, that means spending a lot of time starting at the screen tagging, copying, pasting, downloading, watching, and assessing. I have taken a step or two back away as I have moved from life as a postdoc to working full time but ever since Anonymous declared war on Scientology last month, I have been spending more time online again (more than I want to at least), tracking and following some of their developments, including spending time on some of their IRC channels, downloading lots and lots of videos as well as all sorts of great flyers. Well today there was an IRL action loosely coordinated by Anonymous to protest the CoS and I have to say, I am simultaneously surprised and unsurprised at what I think was a pretty vigorous response and turnout. I was not too surprised as the attack against and criticism of Scientology is one of the older wars spurred on in part by the net and net ethics and it always seems to crop of from time to time. But I think the depth of smack talking I witnessed on IRC, (often clearly youthful in nature as this example reveals—>, [mizzanon] I will have my full license in approx. 3-4 weeks, as soon as I do, i will be driving around my entire region with tons of
anti-propaganda) led me to believe that they could not pull of this global day of action. But they did and perhaps with the exception of New Zealand (where there are probably better things to do anyway), they seemed to garner a fair bit of participants. It will be interesting to see how long they will continue to generate this type of support. The next day of action will be around the middle of March.

Today, I orphaned AUB -- my very first Debian package and the first free software project whose maintenance I took over. I had been helping and doing work in Debian and the free software community for some time but AUB was the first package I uploaded into Debian with my own name in the maintainer field and with my key in the Debian keyring. AUB is a program for working with Usenet binaries. As late as 2002 perhaps, it was a pretty useful tool for a variety of things. Today, however, it doesn't seem to be useful for much more than indiscriminately downloading large amounts of porn and spam. The software is crusty and written in Perl 4 which, today, seems almost unforgivable. Like Usenet, AUB's day has come and gone. During my maintenance of the package, I ended up taking over upstream development and writing and integrating quite a few new features and patches. In fact, there still seem to be a few users! Unfortunately, I am not one them and I officially gave up on upstream maintenance a few months ago and contacted the submitters of all pending bug reports. Today I'm orphaning the Debian package and completely letting go. None of this is particularly noteworthy, I suppose. If AUB was ever important, it hasn't been for a long time. I think its worth mentioning because it's symbolic of the completion of the life cycle of a free software project that was important in my own growth. Languages, the world, and even I have moved on. In the process, I've grown hugely as a developer, programmer, and free software activist and advocate. Minor as it be, AUB played an important role in this whole process. If you are interested in taking over AUB, please let me know. Otherwise, I'm just as happy to let it disappear.

Liberation Dance from the curiously-difficult-to-find album Anthem for the New Nations (yes, that’s CDN$97.18!) by Dollar Brand. Unfortunately, I can only share a taste (mp3, ogg). How music this good could be out of print is beyond me. Technorati Tags: Dollar Brand, Abdullah Ibrahim, Jazz, Anthem for the New Nations, Liberation Dance Update: I added the Google embedded Mp3 player as an experiment and resampled the MP3. Apparently the Google MP3 player can only play files at 44.1 KHz correctly.

Share This

installing Debian on Zonbu mini desktop ... ... it's trivial! Me and godog have bought a Zonbu Mini Desktop to replace our previous noisy caseful server. The supposed duties of such a server are just running some P2P clients and a slimserver for a Squeezebox (so yes, the Zonbu will also need to proxy a very big USB disk). I've discovered the Zonbu mini thanks to a blog post by caesar, and I'm quite happy about the hardware. It's absolutely noiseless, small, eye catching. Though I don't need particular performances, the only aspect that worried my was the I/O on the CF card, which apparently is a non-concern. The only missing bit in the hardware is a video output (only VGA is provided), a must if you want to use the Zonbu as a multimedia station. But this kind of usage wasn't on my plans, so ... FWIW, before buying the Zonbu I've also tried to evaluate the cost of building something similar by myself, but the only cost of the mother board almost dominates the final price of the Zonbu. So apparently it's a good device also price-wise. I haven't even tried the on-board Gentoo-derived installation. Well, in fact I had, but considering that the only monitor I've at home is a B/W 15 years old CRT, my chances to see something vanished at the precise moment the frame buffer started, of course they did not get any better when X started. The minute after that, I've fired up D-I.

---

Debian Installer vs Zonbu mini Installing Debian on the Zonbu mini is actually trivial:

• go for the USB memory stick installation method. The only USB port supporting boot from memory stick is the front one, apparently I was lucky since it is the first one I've tried
• according to another post by caesar at kernel greater than 2.6.22 is needed, so go for the unstable [daily built D-I] (architecture is i386) and install at least Lenny on the target system
• go take a beer, wait a bit, ... your Debian-powered Zonbu Mini Desktop will then be ready

/me happy about the new geeky gadget

---

Sad note: according to godog's tests, the provided wifi (which is optional to buy) works either with ndiswrapper or with a binary kernel module built against Fedora kernel, that sucks.

I don't know why this is appears to be a problem but AFAICT there is no particular impediment in talking to delegates at conferences, no matter what the gender.
Maybe it is just the way that such discussions are begun or the tone being used or maybe it is just that my RealLife(tm) world involves a radically different environment.
Various stats claim that IT is massively dominated by male workers.
Many stats show that pharmacy is massively dominated by female staff.
Maybe it is just an indication that a degree in a course other than IT is more of a benefit than would otherwise appear.
As a single guy, the pharmacy world is little different to IT as far as conversations with colleagues is concerned. To me, that is how it should be.
Maybe I'm just too old to be deemed "a problem".

My Apple Powerbook 5,4 just booted for the first time using GRUB, with no manual intervention, from Apple chime to GDM prompt. This is a great milestone for GRUB2 on powerpc-ieee1275 and OpenFirmware, as until now, multiple problems in the loader would drop you to OF console straight away, although other PowerPC hardware with other firmware implementations did manage to work. Recent fixes by Pavel in the core and some other minor fixes for the userland utils have taken grub2 to the point where it is usable on most PowerPC hardware. On Apple, the only minor issue remaining is grub-probe insisting on (hd) not being a valid device name, so for now I have to trick it into believing it's really (hd1). In parallel, many other GRUB2 improvements haven't stopped hitting CVS in the last months, which have seen how new contributors joined grub-devel and helped GRUB2 get the great momentum it's enjoying right now. Vesa, Robert and Bean have been really active lately, and have fixed long standing issues or written a lot of new code. One of the features GRUB2 acquired recently was image loading for background images. Much more powerful than the implementation in GRUB Legacy, GRUB2 can now read images in multiple formats, can handle up to 24bit colour and render a menu in arbitrary resolutions. The menu can now show UTF-8, and the Debian package will configure a pretty theme that matches the rest of the system if desktop-base is installed:

GRUB2 speaking UTF-8 Catalan Although I'm not sure if GRUB2 is completely up-to-par with GRUB Legacy on i386/amd64, it seems the tricky bits, like video, LVM, RAID and the standard filesystems are supported and working. What GRUB2 needs now, in order to finally replace the aging and upstream-lacking GRUB Legacy you probably have installed, is massive testing. Debian has traditionally been a testbed for GRUB Legacy patches, and is also the platform where GRUB2 is being more widely tested. Having GRUB2 included in lenny's debian-installer would be a great step forward, and by the looks, I think we're well on time to manage this. Replacing GRUB Legacy with GRUB2 is trivial. On PCs, just install the grub-pc package. You'll be offered to keep GRUB Legacy, but with an added menu entry to chainload GRUB2. If you're worried that GRUB2 might fail on your hardware, accept this, and try to load GRUB2 from GRUB. If it works, you then know you can get rid of GRUB Legacy completely and keep GRUB2 in the MBR. On PowerPC-based Macs, you'll have to work around the small issue I mentioned above. Install the grub-ieee1275 package. You also need a very recent powerpc-ibm-utils package, which was just uploaded to unstable. Mount your bootstrap partition, probably /dev/hda2 in /boot/grub, and generate a device.map file with grub-mkdevicemap. Check the contents. If your first device name lacks a drive number such as (hd), it's probably correct, although this will make things fail later. Change it to (hd0) for now. As grub-install relies on grub-probe, you'll have to generate your grub image by hand. Copy all .mod files in /usr/lib/grub/powerpc-ieee1275 to your bootstrap partition, and generate a core.img:

root@powerpc:/boot/grub# grub-mkimage -d . -o /boot/grub/core.img *.mod
root@powerpc:/boot/grub# update-grub
root@powerpc:/boot/grub# nvsetenv boot-device hd:2,core.img

The generated grub.cfg will have references to (hd0,X), which you'll have to correct back to (hd,X) if necessary for your OpenFirmware. After this, you are probably ready to reboot, cross two fingers and get a warm "Welcome to GNU GRUB!" message at boot, which will then be followed by a standard GRUB menu, but on your nice PowerPC box. Unfortunately, the eye candy in the screenshot above isn't available yet in this platform, as it lacks VESA. Does someone in the audience want to contribute a video driver for powerpc? :)

Earlier today I watched the presentation Jonathan Oxer's gave during LCA about Package caching solutions. Although it was certainly an interesting presentation and although I very much agree that my current local mirror is wasting a lot of diskspace and bandwidth, I'm still not going to switch from debmirror to any of the available caching solutions, because (unless I'm really missing something) none of them scratches my itch. My local mirror currently consists of five architectures (i386, amd64, hppa, sparc and s390) and only has unstable and testing. I use it for:

1. (fast and convenient) updating of my systems
2. doing Debian Installer builds and installation tests
3. test builds of installation CDs (using debian-cd)

Now, the last one is somewhat hard (debian-cd uses hardlinks to packages on a local mirror instead of retrieving packages), so let's concentrate on the first two. Caching is great if you have a large number of machines – of the same architecture and that are all likely to need roughly the same packages – sitting behind the proxy: the first one triggers the download of the package and the rest gets it almost instantaneously. It is a lot less great if you have only one, maybe two systems per architecture: most of the time you'll still end up going down that (relatively) slow ADSL connection. An important reason why I have my mirror is so that when I do my daily updates for sid or run an installation test, the packages are already available locally. I really don't want to double or even treble the time needed for installation tests just because some required packages aren't yet available locally and need to be downloaded over that slow line. So, I have my partial mirror. Somewhat tuned (I exclude some ridiculously large debug packages for example, saving about 10GB), but still with a lot of junk^Wpackages on it I'll never ever use, especially for hppa, sparc and s390 as those systems only have fairly basic installations. Getting rid of that would significantly reduce my daily sync and allow me, for example, to also have a mirror of stable and oldstable, keep old versions of packages and probably still save a lot of diskspace. Wishlist What we should have is a hybrid solution: a program that will present itself as a proxy to clients, but is smart enough to pre-fetch new packages that are likely to be needed in a sync run, based on usage date from the the proxy and configuration settings. Some ideas for features/configuration options it could have:

• should support both source and binaries (debs and udebs)
• transparently retrieve packages that are requested but not available locally (just like a regular proxy)
• options to always sync (per architecture):
  • all required and standard packages
  • packages listed in a certain config file
  • packages in a certain section (e.g. udebs in the debian-installer section)
• for packages that have been used (requested by clients) within the last X days: pre-fetch any new version (per architecture)
• options to include security/volatile updates in that
• be smart about ABI changes: when a package name changes because the ABI version changes, also pre-fetch the new package; an alternative solution for that could be:
• for recently used packages: ensure that their dependencies are also pre-fetched; this would also provide support when packages are renamed (dependencies in the transition packages would ensure the new packages are pre-fetched)
• expire packages from the mirror that have not been used for X days
• allow faster expiration of certain types of packages (-dev or -dbg for example)
• options to keep X previous versions that are no longer referenced in the Packages files for Y days

Possibly such an implementation could even be used on some of the lower tier Debian mirrors. Unfortunately, unlike some of my esteemed colleagues, I'm not able to just whip up something like this, so I'm condemned to wait and see if there's someone else who'd like to pick up this idea. I am of course more than willing to help develop this idea further and to test it. Now, if I've totally failed in my research and something like this already exists, a pointer in the right direction would be much appreciated.

So, I'm back from Holidays, to find we now have a new Government in Ireland. (An unexpected choice of government, of which more later.) We were camping in Fouesnant, in Brittany; Family & Friends might like to see the holiday snaps here). Camping in France went well. With three children aged 7, 4 and 2, there is only so much you can expect from a family holiday. Like it or not, you have to mind the toddler fairly consistently, and the older kids would like your full attention too. This is a holiday ''for'' the kids; plan also a break ''from'' the kids later (a long-weekend, with kids at grandparents, for example). This was our first time camping abroad, and its well recommended. While we went touring with a camper van with the two older children two years ago, this is different in that we were at the campsite for almost two weeks: the kids got to meet and play with others in a way they couldn't before. Also, there were good facilities at the site in Fouesnant, and the swimming pools in particular were appreciated (Caveat emptor though: there were no lifeguards or supervision. We went off-peak, and there were apparently two accidents in the time we were there: a broken kneecap and chin when people came down the slides too soon after others and collided). It was the first time the children (a) had pocket money to spend, and (b) were able to walk to a (campsite) shop to spend it. Lessons were learnt in the finiteness of pocket money, especially by Caoimhe . The weather was fairly good: two wet days (though not overly so, compared to what we're used to in Galway); enough to give the 'authentic camping experience'. And for the children especially, just warm enough. Long drives South were not an option with young kids, and we wanted to see Carnac and Brittany in general. We went touring to two places: Mont St. Michel, on the way down, Carnac, and locally around Quimper and Concarneau. At seven, Cormac is beginning to be old enough to enjoy this, but Sadhbh at 2 is too young, and easily aggravated. Caoimhe can appreciate some of it, such as the well recommended Breton museum in Quimper, seeing the local costumes and clothing is just her style at the moment. We're looking forward to going back when the kids are older, or we can enjoy touring on our own. The camp site is dominated by English / Dutch / Irish. It is still term-time in Britain and Ireland, which meant the children on the site was full of 2-4 year-olds, with a shortage of playmates for Cormac. Next time, synchronise with English school times, etc. We went during Cormacs term-time because it was much cheaper (1000 Euro) than high-season. If Cormac was having difficulty in school things would have been different, as it is, we had no qualms in doing so. Things will probably be different next year with Caoimhe also in school at the same time, and we would go later. Suffice it to say I think synchonisation of school timetables is a bad idea, as it has led to these huge price differentials and parents taking their kids from school. As a friend with experience of the French school system points out, they've spent the last 40 years desynchronising their timetables ... So would we go again, or buy our own tents, etc? possibly. The brochure for next year arrived already, with large discounts for booking early ... Tags holiday, family, france, brittany, cormac, coimhe, sadhbh

Well, the people have spoken, now the airwaves are full of puzzled politicans and pundits trying to decide what they said. The outcome was a suprise, especially in Galway West (where I live), for which I'm annoyed; as I said before, a poll would have been useful. Niall O'Brolch in, said in the last poll in March to be likely to take a seat, came fairly off the mark, unfortunately. Quick summary: The Irish Parliament (the D il) has 166 seats, so the majority is 83 (I seat goes to the Chair or Ceann Comhairle). In recent years, majorities of one or two seats are normal, so we all wait for the final seats to be decided before really knowing the outcome: The Bookmakers paid out for Bertie Ahern being the next Taoiseach by 9.30 AM yesterday, though technically the numbers could still add up for the alternative, Enda Kenny. We might not know until next week.

read more »

Tags politics, galway, greens

So a few folks left some interesting comments in response to my link to the article on the rise of the alpha-girl based on the research of Harvard psychologist, Dan Kindlon. My response to both Joe and Karl is that it is worthy to lower the barriers to entry not because girls will change the cultural ecology of geekdom in positive ways (though they may) or because geekdom is inherently “omg totally awesome!!!!!!!!!!!!!!!!!!!!” but for a much more mundane reason: it may allow girls to follow interests and eventually discover a passion. I think it is a great idea if more women were involved in geekdom and open source development not because it is inherently great but because I am sure there a lot of women out there who would enjoy it (and by extension, I am sure a lot who would not). By highlighting this article I am not advocating some forced hacker-female-labor-migration-policy but pointing to the fact that social conditions outside of geekdom play a role in ensuring more participation among women. But let me explain a few things before I argue my position a little further. First a little clarification, especially now that I have read the article. By alpha the author does not mean dominant, aggressive, and totally jerky, which is what might immediately come to mind when you read/hear alpha (especially if one knows any alpha-geeks or alpha-lawyers for that matter, who are much worse, imho). And I certainly don’t want any more alpha-anything (alpha-women, alpha-dogs, alpha-cats, alpha-hamsters ) given our world is already chock-full of alpha-jerks. By using the term, the author basically means confidence, which apparently is on the rise among girls, leaving them much more likely to be go-getters, undaunted by explicit and implicit barriers and obstacles and thus more likely then ever to enter arenas that have been thought of as traditionally male or in fact totally dominated by males. If this trend is indeed in place, I think it is great and I hope that this eventually translates into more girls/women populating traditionally male areas whether it is neurology, movie directing, hacking, or surfing. What I appreciate about the article is that his research can help us question the idea that women are naturally averse to competition due to their estrogenic hormones while boys, brimming with testosterone, just love it and exude it. I never identified with that conventional well-worn script, in part because I guess I am fairly confident and somewhat competitive. Being in academia for the last 10 years, I have witnessed a lot of really confident women that have helped inspire me and keep me going, especially when times got rough. What is fascinating about the article (and by extension probably his book) is that gender parity and equity and changes in the psychological makeup of women have not changed overnight but have taken a much longer time to settle in place. We are only now bearing the fruits of structural and educational changes first instituted decades ago and that tackled some serious forms of discrimination. As a result, we are seeing girls and women donning a deeper confidence that may help them participate with more success in the arenas they want to whether it be sports or in the workplaces (though there are certainly still major barriers and issues, which are addressed in the article). So why would someone want to participate in the world of free software and hacking in particular? My first answer is there may be any number of reasons why anyone, female or male or transgender or whatever, may want to do so for the love of technology, to feel an intense belonging to a community who share your passions and who work together to create something with a lot of value, to enjoy the challenge of learning, to spread freedom to every corner of the earth, etc… It is not that geekdom is inherently fun and exciting but that it will be interesting to some slice of the population men or women (or other) and what is the harm in lowering the barriers to entry, especially if it brings enjoyment and frankly a lot of economic security too? There are a number of girls/women who already find it worthwhile and I am sure many more who would. And the point is not to create some policy to make sure that we shuttle women into hackerdom but as a society we should equip them with the necessary psychological tools so that if they think this is worthwhile, and discover that they love it, etc, they will dare to venture in there and more importantly, stay if they want to. In terms of Karls point that a lot of people are professional geeks in part because they d have trouble being anything else; like being gay, it s not a choice, I think that is far too narrow of an assessment, not to mention an outdated caricature of geeks. While there is certainly a class of socially awkward geeks (if that is what you mean??), I would say they are in the minority though they may certainly stand out precisely because they are the odd ducks and because the stereotype is so entrenched. Many geeks I have met, while they may pretty darn focused on geekdom, also have full and rich lives/personalities that cannot be easily collapsed into one immutable personality type. Yes they may be obsessed with tech but aren’t doctors, academics, musicians, lawyers sort of fanatical too? I spend like my whole week working on academic stuff (part of necessity, part out of love). And at least hacking has way better conferences and economic perks, which may help explain why people stay Finally, I think you assume a little to strongly that we do what we do because we have a pre-formed existing desired to do so. While I think this is the case for many things (and I knew the minute I learned about anthropology, I wanted to be one, which was odd but it proved to be correct), I think desire is also formed as much through experience and hence the importance of exposure to different worlds and experiences. I know that there are many things I could have never imagined I would have loved–karoake and sailing are two things that come to mind—until I tried them about both took some degree of courage. In the later case, it took a lot of guts to move onto some ship at the age of 18 instead of going to college and I am so glad I did. This is an instance where confidence and an initial curiosity led to discovering a love and passion I never knew I had. So if desire can be formed and not just expressed, I think it is key to make sure people have all sorts of opportunities to cultivate the passions they never knew they had.

By using my name in various web services I usually spot bad implementation of displaying name field in their website. I can see that over years the amount of servers which cripple my name to something like iha is getting lower, but there are still lots of them. When I find way how to report it, I usually do so and I always enjoy fields like "How to repeat". This time the guilty one is the MySQL bugtracker. PS: Fortunately all Debian pages seem to catch up this year and my name is no longer crippled in NEW queue or Lintian pages :-).

This year at Linux.Conf.Au there was a student party sponsored by Google. The party was held in a bar and lots of free drinks were provided. This was fine for the university students, but for school kids it was obviously lacking. Some people point out that it’s “quite legal” to run a party that excludes children, the point however is whether it’s desirable to exclude them. Also the concept of a state where laws dictate all aspects of your life to a degree such that obeying the law is the only possibility is fortunately restricted to science fiction. Another common fallacy is when people point out that we should be grateful for Google’s sponsorship. As far as I am aware Google doesn’t insist on any particular conditions for sponsoring a party. If the conference organisers were to request a party at a more child-friendly venue (for example a restaurant - which if licensed could serve alcohol to adults who desire it) then I doubt that Google would refuse. Being grateful for Google’s sponsorship is entirely unrelated to the issue of whether their sponsorship money was spent in the best possible manner. My interest in this topic started at LCA 2007 when I heard complaints from young delegates who were excluded from the Google party. This year the Google party (a different event from the “Google student party”) allowed everyone to attend and issued coloured wrist-bands to indicate whether the person had shown suitable ID to be served alcohol. The Google party was inviting for all and I believe that it was a significant improvement over last year (more attention was paid to serving food than alcohol). I have suggested that at future events some tables be reserved for people who aren’t drinking. As a general rule people enjoy being around people who have consumed a similar type and quantity of mind-altering substances (something I learned from my friends in Amsterdam). There is of course demand for serious drinking, and it seems impossible to satisfy people who want to do serious drinking at the same party as people who won’t or can’t drink at all. If there is not going to be an official party that is suitable for U18s then I’ll arrange it and pay for it myself. The consensus of opinion seems to be that less than six U18s are not worth catering for (one of the common objections to my suggestions is that there may be only four or five U18s). I can pay for a party for that many people which (in terms of food and non-alcoholic drinks) compares well with whatever Google might offer for the drinkers. The rough idea is that U18s will have free food and non-alcoholic drinks. The venue will be some suitable restaurant (is there a Pizza hut or similar near the next LCA?). The party will be open to parents and delegates who are >18 and don’t plan to drink (but I’m sorry I can’t afford to shout you). Opportunities to learn about cool Linux stuff will abound (I expect that a number of knowledgable Linux people who can teach some interesting things will be interested in attending). If I’m paying then children who aren’t delegates will also be welcome to attend, but their parents would have to pay for their food/drink. But this is merely a matter of budget, if it was to be an official event or there were other sponsors then this might not be the case. What I would like right now is expressions of interest from young people who plan to attend the conference and from parents (who plan to attend the conference or whose children will attend). If it looks like there will be a suitably large number of people interested in this then the conference organisers may decide to make it an official event. Also comments from adults who would prefer an alcohol free event (whether it be due to medical reasons, religion, or choice) would be of interest. It’s all about the number of people who will attend.

Share This

In the past I was not blogging small things if they were not worth a full blog entry on their own (or if I did not have the time to make up a full entry of it :). Inspired by the example of Raphaël Hertzog <hertzog@debian.org> with "News for Debian developers" for <debian-devel-announce@lists.debian.org>, I am intending to do the same on a monthly basis about my own Debian related work. This first entry about January 2008 also covers a few things from the last month of 2007. General News

• Trusted Computing
  
  After having taken over some rotting ITP bugs, Debian has now all the important packages to support Trusted Computing on platforms containing a TPM chip. The packages are not in good enough quality for release, but there is still some time left to fix that. Looking forward to make lenny supporting TPM properly out of the box, as well as aiming for making Debian the best TPM enabled distribution available.
  
  Currently, the TSS software stack consists of trousers, tpm-tools, opencryptoki, libengine-tpm-openssl, and ecryptfs-utils.
  
  
• Smartcards
  
  I am using smartcards since a while to hold GnuPG, LUKS, and OpenSSL keys/certificates. Thanks to Jonas Meurer <mejo@debian.org> for applying a fix and a patch of mine to cryptsetup. Together with my uploads of gnupg-pkcs11-scd, pam-pkcs11 and pcsc-omnikey, I can now use Debian out of the box with smartcards for authentification, not relaying on patched or private packages anymore.
  
  The next aim is to integrate smartcard support into debian-installer. It should be possible to use/store keys or certificates for encrypted filesystems directly from/to a smartcard during installation time.
  

Package News

• botan-devel
  
  The Botan library is maintained upstream in a stable release branch and a development release branch. Before, I was uploading as versioned source packages, botan1.4 and botan1.5. To be a bit more consistent, the two source packages are named botan and botan-devel now.
  
  
• gitosis
  
  Tommi Virtanen <tv@eagain.net> has written a marvelous tool to securely manage hosted git repositories, named gitosis. Having always looked for an elegant sollution to maintain git repositories through push/pull over SSH only, this package made an excellent addition to the Debian archive.
  
  
• gnunet
  
  GNUnet, more precisely the gnunet, gnunet-gtk and gnunet-qt packages where mainly maintained the last years by Arnaud Kyheng <Arnaud.Kyheng@free.fr> with me playing co-maintainer.
  
  Arnaud has become busy within the last months, so I took over a more active role on its maintenance. Currently, all gnunet related packages are in sync and in its latest upstream version available in the archive. There is even a new sibling, gnunet-fuse.
  
  
• icedove-l10n
  
  After one and a half year of pushing its maintainer without any effect at all, I finally took over the unmaintained icedove-locales package in October 2007. I immediately changed the package to the proper naming scheme (icedove-l10n-*) which was an outstanding issue since August 2006.
  
  In the last weeks, I have added unofficial localizations for Galizian, Nepali and Ukrainian, now supporting 38 localizations in total. Icedove localizations are now in good shape again, always matching the Icedove version and no longer lagging behind.
  
  
• iceowl-l10n
  
  Following iceweasel-l10n and icedove-l10n, there are finally also localizations for iceowl (Mozilla Sunbird) available. Populating packages for iceowl-extension (Mozilla Lightning) from the same source package is a bit trickier package-wise, but will be available soon too.
  
  
• iceweasel-l10n
  
  In September 2007, Robert Milan <rmh@debian.org> on behalf of the Catalan/Valencian Debian translator team asked me to remove the iceweasel-l10n-roa-es-val package. Unfortunately, this localization is related to a long standing cultural and political dispute between the Catalan speaking Spanish people and the City of Valencia. Additionally, the localization itself is of poor quality. Of course, I removed the package in unstable immediately and prepared an upload for stable at the same time.
  
  As usual, me asking something on debian-release results in first beeing ignored, then denied and third, two month after my request, beeing privately asked to upload nevertheless. This is one of the reasons I sometimes fully understand fellow maintainers not carring about stable at all anymore, it has become indeed too much of a hassle.
  
  
• traceroute
  
  For the last decade, Debian was shipping an implementation of traceroute originating from BSD. After I took over the package in late 2006, it became clear to me that Debian should not maintain its very own version of traceroute. Therefore, I replaced traceroute in August 2007 with an implementation from Dmitry K. Butskoy <Dmitry@Butskoy.name>. Apart from the fact, that this modern implementation is better in every way (faster, cleaner, less bugs), it has an active upstream maintainer and is used by different other Linux distributions too.
  
  
• syslinux
  
  Thanks to the work of Robert Milan <rmh@debian.org>, syslinux has now 64-bit CPU detection in unstable (2:3.55-2) and experimental (2:3.60-2) again.
  
  

Team News

• Debian Forensics
  
  Unfortunately, Debian has at the moment very few forensic related packages in the archive.
  
  Christophe Monniez <dfence.242@gmail.com> maintains a lot of forensic related packages on behalf of his Debian based forensic LiveCD. Beeing interested to see them in Debian, I am mentoring him now how to merge them properly into Debian where possible, and maintain them in the official archive. We have formed an Alioth project for this purpose. Packages are maintained in git, everyone is welcome to join.
  
  
• Debian Website
  
  After having offered my help to maintain the Debian Consultants list, I got added to the webwml team. Doing Debian consultancy myself, this is just another way of helping out Debian from time to time by adding, removing and updating some entries of like-minded people in a list.
  
  

Finally I was able to open the registration for DebConf8 and send the Call for Papers. For the full details please read the whole mail. We had a lot of work in the last few weeks to make this possible, as we set us the goal “We will keep all data entered last year, people won’t have to enter that again”. And we made it! It wasn’t all that easy to do, as we switched from the old 0.2.10 version of Pentabarf to the more recent, completly rewritten 0.3.9 release of it. Which had a completly new database schema, making the database upgrade a fun task, having had a lot of changes for our own data… After Mark and Stephen finished the database it was my task to get all the changes to the interface back which we had done. Which took about a week longer than I expected it, unfortunately delaying the CfP about 4 days. But hey, the new code is way better now. The actual difference between us and plain upstream is about 30% of that with 0.2.10. Which is nice. Of course we still miss a few things, but those aren’t important for the CfP and will be readded soon. Some of those features also will be sent (and most possibly merged) upstream. A big thanks go to Mark (mhy) Hymers and Stephen (sgran) Gran who did the database upgrade (keeping all our own data in it!) and to upstream who did help me out multiple times to get this running. (Yes, this stuff is also the reason why NEW got that huge again. No worries, I will fix that this weekend, shouldn’t take more than one or two “NEW-sessions”).