XQuery 1.0 and XSLT 2.0: news and how to use them in
Debian
(long post about the second generation of the XSL family
and how to use the related languages in Debian)
The 2nd generation of the XSL family
At the beginning of 2007,
W3C
released a
family of
interrelated specification:
XPath 2.0,
XSLT 2.0,
XQuery (1.0). The 3
specifications are based on the very same underlying
data model which
(finally!) supports typed values and exploits them
to some extent, for example permitting some form of static
type checking in XPath and XQuery. Types usually come from
XML Schema, with both
built-in and user-defined types supported.
A brief overview of the 3 specifications and what they change in
the state of the art follows.
XPath 2.0
XPath 2.0 basically consists in
XPath 1.0 + 3 new macro features.
The first one is a revamp of those language features which are
dealing with types, in order to better integrate them with the new
typed data model. XPath now has operators to check whether a value
belongs to a given type (so yes, type information are kept at run
time), and to cast them from one type to another. The basic new
type constructor is now the
sequence, which replaces the
old node-set solving many annoying issues, such as the
impossibility of having a node ordering other than
document
order.
The second feature is the improvement of
standard
library of functions which was ridiculously small in XPath
1.0; it is much better
now. Additionally, 2nd
generation languages (
XSLT 2.0 and
XQuery) now support the ability to define functions
which will then be visible to inner (XPath) expressions, pushing
yet forward the possibilities of plain old XPath. Functions can now
also declare types in their signatures (for their arguments and
return value) and untyped arguments will be automatically casted to
them upon invocation. Hence, even if you are not using an
implementation which assigns types to your XML trees, once you
"enter" the typed world calling a typed function (almost all
standard library functions are decently typed) you will be able to
stay there avoiding annoying casts everywhere.
Finally XPath 2.0 has turned into a powerful
purely
functional language and is now powered by constructs like
conditionals, for-each loops, existential/universal quantifiers,
and existentially-quantified comparison operators for sequences.
Here is a complex expression to hwet your appetite (or scare you
away ...), comments come as
(: smiley faces :)
for $book in /bookshelf//books
return
if ((every $author in $book/authors/author
satisfies $author/nativeLang eq "it_IT")
and $book/lang eq "it_IT")
then $book
else ()
(: think about the trouble of writing this in XSLT/XPath 1.0 ... :)
XSLT 2.0
XSLT 2.0 is what I would call a "bug fix release" of XSLT 1.0 +
the routinary reworking of the language to deal with typed values,
which is not sensibly different than what has been done for XPath
2.0. The fixed "bugs" are several, starting from the annoying issue
of
result tree fragments. They are basically tree snippets
that in XSLT 1.0 you were able to generate for future use.
Unfortunately they were not thaaat reusable, given that you were
not even able to navigate them with XPath operators! Now the
specification is much more clear and distinguishes
final result
trees from ordinary variables, which can now contain sequences
of (navigable) tree nodes.
Another important "bug" fixed is the new ability to
output
multiple documents with a single XSLT stylesheet: it marks the
end of stupid extra post processing to be added in pipeline to a
XSLT processor.
Other minor "bugs" fixed are a limited amount of
backtracking capabilities among imported templates,
regular expression support directly in the language, and powerful
grouping constructs on the lines of SQL's
GROUP
BY (but much more powerful). Here is a template snippet
exploiting the latter feature:
<xsl:for-each-group select="*" group-starting-with="h1">
<div>
<xsl:apply-templates select="current-group()" />
</div>
<xsl:for-each-group>
XQuery
XQuery is the end of the chains imposed by XML-based syntaxes.
Why the heck one has to use an XML syntax (as in the above snippet)
only because she is manipulating XML tress is one of the mysteries
of XML technologies which have always been floating around in my
head.
XQuery is the (supposedly) SQL equivalent for databases of XML
documents, but is actually much more than that. I depict it in my
head as
the XML manipulation language with a syntax I can
finally stand. Technically it is XPath 2.0 (say 80% of the whole
language) + some extra ingredients (say 20%); so remember that
every XPath 2.0 expression is also a XQuery expression.
The main extra ingredient is the so called
FLWOR
expression (to be read: "flower expression", which in addition
to the "smiley faces" used for comments gives a "back to
1968"-flavour to the language .... erm FLWOR/flower/flavour, no pun
intended). A FLWOR expression is very similar to SQL's
SELECT-FROM-WHERE: it lets you generate a
tuple stream by
iterating on sequences (F:
for clauses),
binding expression values to names (L:
let clauses), filter out tuples which do not satisfy a
required condition (W:
where clause),
order the survived tuples (O:
order by
clause), and finally return a sequence built using the residual
tuple stream (R:
return clause).
The other interesting extra ingredient is the ability to build
the
XML snippets you want to manipulate. Within XQuery you
do that using plain XML syntax (the only place where a sane-minded
programmer actually wants to see it!) which also supports a
classical
interpolation mechanism to embed expressions
which will be evaluated inside XML snippets, and also the other way
around. A canonical XQuery example is:
for $t in doc("books.xml")//title,
$e in doc("reviews.xml")//entry
where $t = $e/title
return <review> $t, $e/remarks </review>
(: braces denote the escaping context where XQuery expressions will
be evaluated inside snippets; plain XML syntax is used for the
other way around :)
But remember: XQuery for XML is much more than SQL for RDBMS,
thanks to the implicit templating mechanism implemented by
interpolation, and thanks to several language features fostering
modularity (user-defined functions, library modules, XPath 2.0
standard library, ...) you can basically do with it any kind of XML
manipulation you can imagine.
I don't think I will ever write myself any other single line
of XML output in DOM or XSLT ...
Cool, how can I use it in Debian?
... if only this stuff were decently supported in the open
source world.
Last time I checked, the author of most parts of the GNOME
toolchain for dealing with XML (libxml2, libxslt, ...) was not
intentioned to implement XSLT 2.0, not even mentioning XQuery. This
comes as no surprise, the whole GNOME XML toolkit is written in C,
and XSLT 2.0 / XQuery have reached a level of complexity and formal
specification which usually entails a higher level approach. So on
the GNOME side we are stuck.
The other open source implementations of XSLT 2.0 / XQuery I'm
aware of are
Saxon and
Galax.
Saxon is an XSLT 2.0 and XQuery implementation written in Java,
which is unfortunate per se. Additionally, it is also unfortunate
that it is only
partially open source. Indeed, Saxon is
split into SaxonB (for "basic") which is open source under the
Mozilla Public License and SaxonSA which is commercial. While
SaxonSA is a fully conformant, XML Schema-aware processor, with
support for static typing, SaxonB is a basic-conformant processor
with no type-aware features and actually much less optimized than
SaxonSA. This is annoying. (Much more annoying is the fact that
SaxonSA's author is the only editor of the XSLT 2.0 specification
and that instead of his mail address the specification includes an
URL pointing to the website selling SaxonSA ...)
Galax is an open source (IBM CPL / Lucent license) OCaml
implementation of XQuery which is not fully conformant to the
specification (though it gets quite close) which is type-aware and
implements static typing.
The only XQuery / XSLT 2.0 implementation available
in
Debian at the time of writing is SaxonB. The binary package is
libsaxonb-java,
kudos to Michael Koch and the Debian Java Maintainers for having
packaged it (and to have stood some annoying pings of mine
bug #408842).
To execute XQuery code you just have to
aptitude install
libsaxonb-java, prepare a
query.xq file containing
your query, and then execute something like:
CLASSPATH=/usr/share/java/saxonb.jar \
java net.sf.saxon.Query query.xq
note that the information in README.Debian are still referring
to old Saxon versions, see
bug #465894 which proposes a
more up to date README.Debian.
Similarly, to perform a XSLT 2.0 transformation you have to do
something like:
CLASSPATH=/usr/share/java/saxonb.jar \
java net.sf.saxon.Transform -ext:off -s:input.xml -xsl:style.xsl -o:output.xml
Do not remove the -ext:off flag when processing
untrusted
stylesheets!, see
bug
#465885 for the reason.
I've written some handier (1-liner) shell script helpers which
remove the need of invoking java manually. They are attached to
bug #465894 and I've
proposed their addition to the saxonb package. Using them the above
invocations become:
saxonb-xquery query.xq
saxonb-xslt -ext:off -s:input.xml -xsl:style.xsl -o:output.xml
Galax in Debian ... well, since long time I've been planning to
package it (the ITP has been filed some months ago:
bug #447984) and the authors
sent me a newer version than what is available online to gather
feedback before the long overdue final release. Unfortunately I've
been lagging behind in finishing the packaging (which is tricky due
to the need of binding libraries to several different languages:
OCaml is native, but there is also Java for example). Hopefully
this post will give me some renewed motivation for finishing the
work ...
References
- Debian package w3-recs, ships the whole
list of W3C Recommendations for offline consultation; it includes
all the specifications we have discussed in this post
- Saxon: home page of
Saxon, a XSLT 2.0 / XQuery processor written in Java
- Galax: home page of
Galax, a XQuery processor written in OCaml
- Debian package libsaxonb-java,
ships SaxonB as a Debian package
Acknowledgements
Thanks to
godog for his
helpful comments.
Update: the helpers I've proposed have been
accepted into the official package, I've also made available their
manpages. Also the pending changes to README.Debian has been
accepted. Kudos again to Michael Koch for his quick feedback on my
patches!
So, sometime over the past few weeks I clocked up ten years as a
Debian developer:
Modern electrical safety standards in most countries require that exposed pins of a power plug (other than the earth) be shielded to prevent metal objects or the fingers of young children from touching live conductors. The image above shows a recent Australian power plug which has the active and neutral pins protected with plastic such that if the plug is slightly removed there will be no access to live conductors. I have photographed it resting on a keyboard so that people who aren’t familiar with Australian plugs can see the approximate scale.
I’m not sure exactly when the new safer plugs were introduced, a mobile phone I bought just over three years ago has the old-style plug (no shielding) while most things that I bought since then have it. In any case I expect that a good number of PCs being used by Australian companies have the old style as I expect that some machines with the older plugs haven’t reached their three year tax write-down period.
For a device which has a plug with such shielding they sell kits for disassembling the power lead or taking the power point from the wall. I spoke to an an electrician who assured me that he could with a 100% success rate attach to wires within a power cord without any special tools (saving $149 of equipment that the HotPlug people offer). Any of these things will need to be implemented by a qualified electrician to be legal, and any electrician who has been doing the job for a while probably has a lot of experience working before the recent safety concerns about “working live“.
The part of the web site which concerns moving servers seems a little weak. It seems to be based on the idea that someone might have servers which don’t have redundant PSUs (IE really cheap machines - maybe re-purposed desktop machines) which have to be moved without any down-time and for which spending $500US on a device to cut the power (plus extra money to pay an electrician to use it) is considered a good investment. The only customers I can imagine for such a device are criminals and cops.
I also wonder whether you could get the same result with a simple switch that cuts from one power source to another. I find that it’s not uncommon for brief power fluctuations to cause the lights to flicker but for most desktop machines to not reboot. So obviously the capacitors in the PSU and on the motherboard can keep things running for a small amount of time without mains power. That should be enough for the power to be switched across to another source. It probably wouldn’t be as reliable but a “non-government” organisation which desires the use of such devices probably doesn’t want any evidence that they ever purchased one…
Now given that such devices are out there, the question is how to work around them. One thing that they advertise is “mouse jigglers” to prevent screen-lock programs from activating. So an obvious first step is to not allow jiggling to prevent the screen-saver. Forcing people to re-authenticate periodically during their work is not going to impact productivity much (of course the down-side is that it offers more opportunities for shoulder-surfing authentication methods).
Once a machine is taken the next step is to delay or prevent an attacker from reading the data. If an attacker has the resources of a major government behind them then they could read the bus of the machine to extract data and maybe isolate the CPU and send memory read commands to system memory to extract all data (including the keys for decrypting the hard drive). The only possible defence against that would be to have multiple machines exchanging encrypted heart-beat packets and configured to immediately shut themselves down if all other machines stop sending packets to them. But if defending against an attacker with more modest resources the shutdown period could be a lot longer (maybe a week without a successful login).
Obviously an attacker who gets physical ownership of a running machine will try and crack it. This is where all the OS security features we know can be used to delay them long enough to allow an automated shut-down that will remove the encryption keys from memory.
I was bought a pair of these for Christmas. For the past few years I've
been using a pair of Hitachi TMP028s - cheap clip on headphones that
I've found far better than the popular ear bud style, but nothing
special. The Sennheisers are much better again. They have a neat little
padded cup that means they make a good fit with your ears, the headband
is reasonably adjustable (I've had issues with my glasses with other
headphones) and they come with a carry case and fold flat which means
throwing them in my bag for my train commute much easier. All in all
I've been very impressed in my use over the past month. And then today I
finally put some batteries in to power the noise cancellation function.
Wow. Much more effective than I expected. I was able to knock the volume
on my iAudio down a fair bit and still hear the music better than usual.
The noise of the train was still present, but reduced significantly to
the point where it wasn't an issue at all. Definitely recommended.
I went back to work today after something like a total of 6 weeks off. (2
weeks vacation, 4 weeks unpaid leave).
The month of January went pretty quickly really. The week in hospital with
Sarah was a blur. The week and a half afterwards was also pretty much a
blur. Then we had the week where things went a bit backwards, and then we
had this week, a few days of which were spent up north at an inn in Little
River, Mendocino County. Heck, let's just call the entire month a blur and
be done with it.
Sarah's doing pretty well now. Surgery on the 3rd, discharged on the 9th,
back in the Emergency Room on the 21st with tachycardia. Concerns about the
incision weeping on the 21st, and again on the 28th. General increase in
pain on the 28th, along with a few sleepless nights preceding.
But now, with a doubling of her beta blockers, some antibiotics, pain
killers, anti-inflammatories and sleeping tablets, she's feeling a bit more
human again, if not like a human pill bottle.
The surgical folks want to see her again on Monday to follow up how the
incision is going since they prescribed the antibiotics. The weeping has
calmed down significantly, but hasn't gone away completely.
A couple more months to go before we can call her "recovered".
...install a 
but otherwise, everything is so twisted, so jumbled or so gone, it is beyond stunning. She certainly knew I was around and helping for she got upset when she knew I was leaving, often bursting in tears, asking me to take her to New York. But she just knew me as Gabriella from Nueva York who brought her novio bueno to PR and not as anyone else.
It is certainly not easy to see this, to experience it day in and out but it is what it is and there is no running from it. Logistically it is also beyond hard to take care of my mom. To get her to eat, to calm down after one of her morning fits (she always, always, always had a raging fit in the morning, the afternoon and evenings being far more calm periods), to convince her to wear her dentures (which is the only remedy to stop her from spitting all over the place), to dress her, to deal with her in the middle of the night, often twice, are all extraordinarily challenging. Very quickly, your waking hours fill up with seemingly endless tasks and this I guess is, in some respects, a hidden blessing; there is just a lot less time to ruminate over how all of this, frankly, sucks. It is only later, late at night, early in the morning, or on the plane ride home where the darkness of her life would set deeply into my own.
It may be obvious why this state and experience is tough but there are two parts that I find particularly rough. If my mother’s state is one of near total chaos and entropy, there is one thing that is utterly clear. She knows she is suffering and she wants out. On particularly bad days, she would mutter or sometimes yell loudly that she wants to die and would ask for the medicine that would accomplish this. She would even sometimes call me mala for not giving her this medicine. It is certainly not the case that everyone with Alzheimers goes through what my mother does. Apparently, some experience the illness quietly and peacefully. This is so not the case for my mother, who lived a particularly hard yet vibrant life. Donning a difficult personality, she lived lived as fully as she could and it is clear that as life leaves her, she would rather not be around to experience what is a cruel exorcism of self. Thankfully, she is not constantly or not even mostly in a suicidal state. I think she at times forgets how lousy she feels and rather enjoys the rather limited life she has. And at this point, I just hope that she continues to have calm or even happy moments.
The second surprise concerns my own memories. My mother has had this rare form of Alzheimers, 
DemoCamp CUSEC 2
It starts off just as you'd expect.
As part of 
… or something like that… I planned to write a short note about how to start with using git-svn so I can provide a pointer to some of my colleagues. It turned out that git has too many nice features that you should be aware of. 
Hopefully my notes (now being a reference for myself as well, thx to 