Just an observation on the user interface of mobile phone chargers. My phone runs flat all the time. And it's all UI. The phone manual sayeth: "Do not leave the phone on the charger once it is charged; doing so will reduce battery life." The phone gives no signal when its charged. So I have to stand over the phone when its charging, to ensure I unplug it appropriately. As a result: I don't charge it when I'm in a rush; and it never gets charged when I'm about to do something else - I will forget it. Blech. How much can a 'disconnect when charged' circuit really cost?

In the past few months in the Debian-EeePC team, a number of interesting things have been happening. Progress has been made to ensure the Eee’s drivers get merged upstream. Chris Snook from Red Hat has taken over atl2 upstream and has started merging it with the atl1 driver to make a unified atlx driver that will be suitable for inclusion in the kernel. As well, there is continued progress on the madwifi driver, with a patch now included to support version 2.6.24 of the Linux kernel. In the meantime, the Debian Eee PC Install HowTo has been under constant revision, even gaining recently the beginnings of translations in French and German. ACPI, another important piece of infrastructure for the Eee, is now supported in lenny and sid through Eric Cooper’s eeepc-acpi kernel module. This is a fork of the asus-acpi module renamed so that it won’t conflict with the in-tree version. It turns out that asus-acpi is deprecated, having been replaced by asus-laptop. Eric has been in touch with the asus-laptop maintainer to ensure the Eee-specific bits are merged so that we can eventually retire our forked version. There still remains at the top of our Todo list the issue of ACPI scripts to go with the kernel module. Having at first considered patching acpi-support, we have decided instead to start with Eric’s own scripts which will be packaged shortly for Debian. This gives us more freedom to tinker before considering submitting patches to more general laptop support packages like acpi-support. Finally, after Brendan M. had to send his Eee back to Asus for repairs, work stopped for a while on the custom debian-eeepc installer. Fortunately, he just got his system back from the shop as good as new and has returned with renewed vigor to that task. He has produced a new version of the installer which we are now testing. Thanks to the efforts of numerous users and developers who are being added to our ranks daily, we expect by the time Lenny releases we will be well on our way to providing a pure Debian solution for the Eee. Whether or not everything needed for the Eee is in Lenny at that time remains to be seen. We need to allow for how long it takes to get new drivers into the kernel. But if we miss the release, we will certainly provide backports and look forward to full support in the following release.

Finally I found the URL of a device I’ve been hearing rumours about. The HotPlug is a device to allow you to move a computer without turning it off [1]. It is described as being created for “Government/Forensic customers” but is also being advertised for moving servers without powering them down. The primary way that it works is by slightly unplugging the power plug and connecting wires to the active and neutral terminals, then when mains power is no longer connected it supplies power from a UPS. When mains power is re-connected the UPS is cut off. Modern electrical safety standards in most countries require that exposed pins of a power plug (other than the earth) be shielded to prevent metal objects or the fingers of young children from touching live conductors. The image above shows a recent Australian power plug which has the active and neutral pins protected with plastic such that if the plug is slightly removed there will be no access to live conductors. I have photographed it resting on a keyboard so that people who aren’t familiar with Australian plugs can see the approximate scale. I’m not sure exactly when the new safer plugs were introduced, a mobile phone I bought just over three years ago has the old-style plug (no shielding) while most things that I bought since then have it. In any case I expect that a good number of PCs being used by Australian companies have the old style as I expect that some machines with the older plugs haven’t reached their three year tax write-down period. For a device which has a plug with such shielding they sell kits for disassembling the power lead or taking the power point from the wall. I spoke to an an electrician who assured me that he could with a 100% success rate attach to wires within a power cord without any special tools (saving $149 of equipment that the HotPlug people offer). Any of these things will need to be implemented by a qualified electrician to be legal, and any electrician who has been doing the job for a while probably has a lot of experience working before the recent safety concerns about “working live“. The part of the web site which concerns moving servers seems a little weak. It seems to be based on the idea that someone might have servers which don’t have redundant PSUs (IE really cheap machines - maybe re-purposed desktop machines) which have to be moved without any down-time and for which spending $500US on a device to cut the power (plus extra money to pay an electrician to use it) is considered a good investment. The only customers I can imagine for such a device are criminals and cops. I also wonder whether you could get the same result with a simple switch that cuts from one power source to another. I find that it’s not uncommon for brief power fluctuations to cause the lights to flicker but for most desktop machines to not reboot. So obviously the capacitors in the PSU and on the motherboard can keep things running for a small amount of time without mains power. That should be enough for the power to be switched across to another source. It probably wouldn’t be as reliable but a “non-government” organisation which desires the use of such devices probably doesn’t want any evidence that they ever purchased one… Now given that such devices are out there, the question is how to work around them. One thing that they advertise is “mouse jigglers” to prevent screen-lock programs from activating. So an obvious first step is to not allow jiggling to prevent the screen-saver. Forcing people to re-authenticate periodically during their work is not going to impact productivity much (of course the down-side is that it offers more opportunities for shoulder-surfing authentication methods). Once a machine is taken the next step is to delay or prevent an attacker from reading the data. If an attacker has the resources of a major government behind them then they could read the bus of the machine to extract data and maybe isolate the CPU and send memory read commands to system memory to extract all data (including the keys for decrypting the hard drive). The only possible defence against that would be to have multiple machines exchanging encrypted heart-beat packets and configured to immediately shut themselves down if all other machines stop sending packets to them. But if defending against an attacker with more modest resources the shutdown period could be a lot longer (maybe a week without a successful login). Obviously an attacker who gets physical ownership of a running machine will try and crack it. This is where all the OS security features we know can be used to delay them long enough to allow an automated shut-down that will remove the encryption keys from memory.

• [1] http://www.wiebetech.com/products/HotPlug.php

Share This

A not-for-profit club has asked me to recommend a web-based membership manager that can do two main tasks:-

1. email members when their renewal date is approaching;
2. control access to some web pages.

(In addition, they want a web forum, but that doesn't need to be connected and they're fairly easy to find.) I'm somewhat surprised to say that I haven't found a good free software one. There's CiviCRM, but it's under an activated Affero GPL and I don't see why this non-software club should be a distributor (which is why I don't see activated-Affero software as free software). There's eZpublish (as used by FSFE) but it looks like overkill for this task and will cause more problems than it solves. Other FOSS organisations seem to be using bespoke apps. The VCS FOSS ICT wiki page is pretty bare. What would you recommend? Where should I be looking? Who to ask? If I have to write one, I'll try to make it reusable.

A nice feature from PostgreSQL are “write ahead log” (WAL) files. Quoting from the PostgreSQL website they describe every change made to the database’s data files. […] if the system crashes, the database can be restored to consistency by “replaying” the log entries made since the last checkpoint. This basically enables online backup of your database cluster. And as you end up having the complete log of all actions within your database(s) you are also able to go back in time to any point you like. I won’t write a complete howto about it, as the PostgreSQL docs are pretty good, just some notes how I did my setup, as a kind of reminder for me when I have to do it again.

---

There must be a place to store WAL files which ends up not being on the same machine, otherwise it doesn’t make much sense. For me that means scp to a different host, but an NFS mount or some storage array or drbd device might also be an option. Much of the following is for the scp method, but easily adaptable to other storage methods. I need a target user on the host I copy files to, and as I did the setup for the pentabarf database running on host skinner, I helpfully named it pentabarf-backup. The copy process can’t be interactive so the postgres user on the database host needs to have a passwordless SSH key to login to the pentabarf-backup user on the backup host. To limit my bad feeling about “passwordless key to login” I restrict the key by adding

from="skinner.debconf.org,72.32.250.224,72.32.79.241",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty 

in front of it while copying it to the pentabarf-backup users .ssh/authorized_keys file. I need a few directories, namely pentabarf-WAL-skinner, pentabarf-tarballs and pg_xlog-skinner for the stuff to backup, created in the home of the pentabarf-backup user. Next step is to tell PostgreSQL our wish to have online backups. Add the following statements to your postgresql.conf, adapting the values to your needs:

archive_command = 'ssh backup test ! -f pentabarf-WAL-skinner/%f && rsync -az %p backup:pentabarf-WAL-skinner/%f'
checkpoint_timeout = 1h
archive_timeout = 12h

The above tells PostGreSQL that i want it to archive the files using ssh to my backup host and transfer files with rsync (via ssh), but only if they do not exist yet. It also tells it to archive the files at least once every 12 hours. The effect is that WAL files get copied over when they are “full” or after 12 hours passed, whatever happens first. Don’t set it too low, as the file will always have the full, fixed size and you waste space transferring mostly-empty files. In case you are concerned with the possibly slow speed of a network link, the PostgreSQL site has the following to say about this: The speed of the archiving command is not important, so long as it can keep up with the average rate at which your server generates WAL data. Normal operation continues even if the archiving process falls a little behind. I haven’t had a problem with the speed yet. A 16MB file every (at least) 12 hours isn’t that bad. Now that we have done all the setup and are able to transfer WAL files, we need a base backup to start with. Issue a SELECT pg_start_backup(‘BASE_BACKUP’); as a database superuser and then backup the database cluster by backing up the whole clusters data/ directory (and every other location if you use tablespaces). When you are done issue SELECT pg_stop_backup(); and you are done with your base backup. You might want to exclude pg_xlog/ to save space, you don’t need it.
As I’m lazy I use a script (written by Peter ‘weasel’ Palfrader), you can get it here, but whatever you take is fine. If you use the script just make sure to adapt the variables in the script to values that fit your system. From now on you should do such base backups in regular intervals, I run them once a week, as recovery does need the last base backup plus all WAL files from then on. Doing regular base backups limits the amount of WAL files you need to keep around / you need to replay.

---

Reading the above you might find that there is a window of up to 12 hours from which I do not have backups. True, and to make this window way smaller, I am using another script (also written by weasel), get it here, together with an init script. This synchronizes the pg_xlog/ directory to my backup host every 5 minutes, so in case of a server crash I can replay everything up to the last 5 minutes. Recovery using WAL files This is pretty simple, and also good documented at the PostgreSQL site again. For my scenario it basically boils down to

1. Stop postgres
2. Move away the cluster.

   mv /var/lib/postgresql/8.2 /var/lib/postgresql/8.2_old

3. Restore the backup dump, be careful to have the file rights correct. For the above described setup that means copy and untar the latest tarballs created weekly.
4. Recreate pg_xlog:

     mkdir /var/lib/postgresql/8.2/main/pg_xlog
     chown postgres:postgres /var/lib/postgresql/8.2/main/pg_xlog
     chmod 40700 /var/lib/postgresql/8.2/main/pg_xlog
     mkdir /var/lib/postgresql/8.2/main/pg_xlog/archive_status
     chown postgres:postgres /var/lib/postgresql/8.2/main/pg_xlog/archive_status
     chmod 40700 /var/lib/postgresql/8.2/main/pg_xlog/archive_status
    

5. Copy possibly unarchived WAL files from /var/lib/postgresql/8.2_old/main/pg_xlog into the newly created pg_xlog.
6. Copy all the WAL segment files that are needed over from the backup host. That should be all the files on the backup host, directory pentabarf-WAL-skinner/. You should be able to limit it by taking all those files >= the latest *.backup file.
7. Create a recovery command file recovery.conf in the cluster data directory. You may also want to temporarily modify pg_hba.conf to prevent ordinary users from connecting until you are sure the recovery has worked. The content of recovery.conf is

         restore_command = 'cp /tmp/pg-WAL/%f "%p"'
        

   Make sure that the file location you specify is the location you put the WAL segment files into in the prior step.
8. Start the server, it will go into recover mode and proceed to read through all the archived WAL files it needs.
9. Look at the database and decide if you like what you see.

Of course the procedure can be slightly adapted, so you end up with a database state from 2 weeks ago. Or 3 days. Or whatever you may need (and have WAL files for). There are many possibilities, and there is a lot of text on the PostgreSQL page that I don’t want to duplicate here, so please read it. Also, I’m not using this as the sole and only way to backup my database(s), they are all still dumped into SQL files every day. This is just additional.

I wonder if things could go any worse if we actively tried? So despite going back to see the cardiologist on Monday afternoon, concerned about there possibly being still some sort of infection, and being told there wasn't, on Thursday morning, the top part of the incision was looking swollen. It looked even more so by the evening when I got home from work, and Sarah emailed a photo to the cardiologist, and he agreed it didn't look too crash hot, so he phoned a prescription for another, longer course of antibiotics through to the pharmacy, but too late to get filled last night. By bedtime it was looking even more ready to blow, and sure enough, around 1am (it's always 1am) it erupted everywhere. Fortunately (for me), I missed all the gory details, but I think there was a fair bit of fluid built up in there. So we went back to the clinic this morning, and they decided that they needed to open up part of the incision to let it drain and heal properly. They took another swab to run another culture, and then got a physician's assistant to take to her with a scalpel. I've seen my poor wife go through too many painful things in the last six months. Today's was no better. Most of the problem was the local anaesthetic injections, but watching them open up the incision again wasn't much fun either. No one should have to go through as much shit in a short time as what Sarah has had to. It's just not fair. She now has a gaping hole in her chest with some gauze packed in it, which has to be changed twice a day (by lucky old me). The idea is to let it heal from the bottom up, rather than the top down, as it seems an abscess had formed in the wound, and that was were all the trouble was. So by leaving it all open, it can drain properly, and heal from the bottom up. $DEITY only knows what sort of mess it's going to make of the incision area, but we can deal with that later. They reckon it'll take another 4-6 weeks to heal. I'm really hoping now that with it draining, and another course of antibiotics, Sarah will be in much less pain. I just did the evening gauze change. Fun. The opening in the incision is longer than I remembered them making this morning. It's about an inch long, and a centimetre deep, and there's a small hole in the bottom that presumably leads to the "interior". It's not as bad looking as I thought it might be. Packing the gauze in didn't seem to hurt Sarah too much, which is just as well. This really is the pits. I really hope this is the end of this, and the infection goes away for good, and the incision heals quickly. In terms of recovery, this is a pretty major setback in my opinion. From now on, we won't be relying on the cardiologist for surgical opinions, we'll be insisting a member of the surgical team checks Sarah out.

The X Developers' Conference 2008 will be held at the Googleplex in Mountain View, California from April 16th-18th, 2008. If you're interested in coming, please sign up at the wiki; we're also interested in having people talk if they want to. More details as they come to hand, but start booking your flights now.

XQuery 1.0 and XSLT 2.0: news and how to use them in Debian (long post about the second generation of the XSL family and how to use the related languages in Debian)

• XPath 2.0, XQuery, et al.
• the above stuff .. in Debian
• links

---

The 2nd generation of the XSL family At the beginning of 2007, W3C released a family of interrelated specification: XPath 2.0, XSLT 2.0, XQuery (1.0). The 3 specifications are based on the very same underlying data model which (finally!) supports typed values and exploits them to some extent, for example permitting some form of static type checking in XPath and XQuery. Types usually come from XML Schema, with both built-in and user-defined types supported. A brief overview of the 3 specifications and what they change in the state of the art follows. XPath 2.0 XPath 2.0 basically consists in XPath 1.0 + 3 new macro features. The first one is a revamp of those language features which are dealing with types, in order to better integrate them with the new typed data model. XPath now has operators to check whether a value belongs to a given type (so yes, type information are kept at run time), and to cast them from one type to another. The basic new type constructor is now the sequence, which replaces the old node-set solving many annoying issues, such as the impossibility of having a node ordering other than document order. The second feature is the improvement of standard library of functions which was ridiculously small in XPath 1.0; it is much better now. Additionally, 2nd generation languages (XSLT 2.0 and XQuery) now support the ability to define functions which will then be visible to inner (XPath) expressions, pushing yet forward the possibilities of plain old XPath. Functions can now also declare types in their signatures (for their arguments and return value) and untyped arguments will be automatically casted to them upon invocation. Hence, even if you are not using an implementation which assigns types to your XML trees, once you "enter" the typed world calling a typed function (almost all standard library functions are decently typed) you will be able to stay there avoiding annoying casts everywhere. Finally XPath 2.0 has turned into a powerful purely functional language and is now powered by constructs like conditionals, for-each loops, existential/universal quantifiers, and existentially-quantified comparison operators for sequences. Here is a complex expression to hwet your appetite (or scare you away ...), comments come as (: smiley faces :)

for $book in /bookshelf//books
return
  if ((every $author in $book/authors/author
       satisfies $author/nativeLang eq "it_IT")
      and $book/lang eq "it_IT")
  then $book
  else ()
(: think about the trouble of writing this in XSLT/XPath 1.0 ... :)

XSLT 2.0 XSLT 2.0 is what I would call a "bug fix release" of XSLT 1.0 + the routinary reworking of the language to deal with typed values, which is not sensibly different than what has been done for XPath 2.0. The fixed "bugs" are several, starting from the annoying issue of result tree fragments. They are basically tree snippets that in XSLT 1.0 you were able to generate for future use. Unfortunately they were not thaaat reusable, given that you were not even able to navigate them with XPath operators! Now the specification is much more clear and distinguishes final result trees from ordinary variables, which can now contain sequences of (navigable) tree nodes. Another important "bug" fixed is the new ability to output multiple documents with a single XSLT stylesheet: it marks the end of stupid extra post processing to be added in pipeline to a XSLT processor. Other minor "bugs" fixed are a limited amount of backtracking capabilities among imported templates, regular expression support directly in the language, and powerful grouping constructs on the lines of SQL's GROUP BY (but much more powerful). Here is a template snippet exploiting the latter feature:

<xsl:for-each-group select="*" group-starting-with="h1">
  <div>
    <xsl:apply-templates select="current-group()" />
  </div>
<xsl:for-each-group>

XQuery XQuery is the end of the chains imposed by XML-based syntaxes. Why the heck one has to use an XML syntax (as in the above snippet) only because she is manipulating XML tress is one of the mysteries of XML technologies which have always been floating around in my head. XQuery is the (supposedly) SQL equivalent for databases of XML documents, but is actually much more than that. I depict it in my head as the XML manipulation language with a syntax I can finally stand. Technically it is XPath 2.0 (say 80% of the whole language) + some extra ingredients (say 20%); so remember that every XPath 2.0 expression is also a XQuery expression. The main extra ingredient is the so called FLWOR expression (to be read: "flower expression", which in addition to the "smiley faces" used for comments gives a "back to 1968"-flavour to the language .... erm FLWOR/flower/flavour, no pun intended). A FLWOR expression is very similar to SQL's SELECT-FROM-WHERE: it lets you generate a tuple stream by iterating on sequences (F: for clauses), binding expression values to names (L: let clauses), filter out tuples which do not satisfy a required condition (W: where clause), order the survived tuples (O: order by clause), and finally return a sequence built using the residual tuple stream (R: return clause). The other interesting extra ingredient is the ability to build the XML snippets you want to manipulate. Within XQuery you do that using plain XML syntax (the only place where a sane-minded programmer actually wants to see it!) which also supports a classical interpolation mechanism to embed expressions which will be evaluated inside XML snippets, and also the other way around. A canonical XQuery example is:

for $t in doc("books.xml")//title,
    $e in doc("reviews.xml")//entry
where $t = $e/title
return <review>  $t, $e/remarks  </review>
(: braces denote the escaping context where XQuery expressions will
   be evaluated inside snippets; plain XML syntax is used for the
   other way around :)

But remember: XQuery for XML is much more than SQL for RDBMS, thanks to the implicit templating mechanism implemented by interpolation, and thanks to several language features fostering modularity (user-defined functions, library modules, XPath 2.0 standard library, ...) you can basically do with it any kind of XML manipulation you can imagine. I don't think I will ever write myself any other single line of XML output in DOM or XSLT ...

---

Cool, how can I use it in Debian? ... if only this stuff were decently supported in the open source world. Last time I checked, the author of most parts of the GNOME toolchain for dealing with XML (libxml2, libxslt, ...) was not intentioned to implement XSLT 2.0, not even mentioning XQuery. This comes as no surprise, the whole GNOME XML toolkit is written in C, and XSLT 2.0 / XQuery have reached a level of complexity and formal specification which usually entails a higher level approach. So on the GNOME side we are stuck. The other open source implementations of XSLT 2.0 / XQuery I'm aware of are Saxon and Galax. Saxon is an XSLT 2.0 and XQuery implementation written in Java, which is unfortunate per se. Additionally, it is also unfortunate that it is only partially open source. Indeed, Saxon is split into SaxonB (for "basic") which is open source under the Mozilla Public License and SaxonSA which is commercial. While SaxonSA is a fully conformant, XML Schema-aware processor, with support for static typing, SaxonB is a basic-conformant processor with no type-aware features and actually much less optimized than SaxonSA. This is annoying. (Much more annoying is the fact that SaxonSA's author is the only editor of the XSLT 2.0 specification and that instead of his mail address the specification includes an URL pointing to the website selling SaxonSA ...) Galax is an open source (IBM CPL / Lucent license) OCaml implementation of XQuery which is not fully conformant to the specification (though it gets quite close) which is type-aware and implements static typing. The only XQuery / XSLT 2.0 implementation available in Debian at the time of writing is SaxonB. The binary package is libsaxonb-java, kudos to Michael Koch and the Debian Java Maintainers for having packaged it (and to have stood some annoying pings of mine bug #408842). To execute XQuery code you just have to aptitude install libsaxonb-java, prepare a query.xq file containing your query, and then execute something like:

CLASSPATH=/usr/share/java/saxonb.jar \
java net.sf.saxon.Query query.xq

note that the information in README.Debian are still referring to old Saxon versions, see bug #465894 which proposes a more up to date README.Debian. Similarly, to perform a XSLT 2.0 transformation you have to do something like:

CLASSPATH=/usr/share/java/saxonb.jar \
java net.sf.saxon.Transform -ext:off -s:input.xml -xsl:style.xsl -o:output.xml

Do not remove the -ext:off flag when processing untrusted stylesheets!, see bug #465885 for the reason. I've written some handier (1-liner) shell script helpers which remove the need of invoking java manually. They are attached to bug #465894 and I've proposed their addition to the saxonb package. Using them the above invocations become:

saxonb-xquery query.xq
saxonb-xslt -ext:off -s:input.xml -xsl:style.xsl -o:output.xml

Galax in Debian ... well, since long time I've been planning to package it (the ITP has been filed some months ago: bug #447984) and the authors sent me a newer version than what is available online to gather feedback before the long overdue final release. Unfortunately I've been lagging behind in finishing the packaging (which is tricky due to the need of binding libraries to several different languages: OCaml is native, but there is also Java for example). Hopefully this post will give me some renewed motivation for finishing the work ...

---

References

• Debian package w3-recs, ships the whole list of W3C Recommendations for offline consultation; it includes all the specifications we have discussed in this post
• Saxon: home page of Saxon, a XSLT 2.0 / XQuery processor written in Java
• Galax: home page of Galax, a XQuery processor written in OCaml
• Debian package libsaxonb-java, ships SaxonB as a Debian package

---

Acknowledgements Thanks to godog for his helpful comments.

---

Update: the helpers I've proposed have been accepted into the official package, I've also made available their manpages. Also the pending changes to README.Debian has been accepted. Kudos again to Michael Koch for his quick feedback on my patches!

Recently, Students for Free Culture -- a non-profit organization dear to my heart -- elected its new board. Several months ago, the group voted to hold its elections using the same preferential election method system that Debian uses. To help make their election easier I agreed to support them with a new set of features in Selectricity aimed at more structured organizational decision-making. Currently Selectricity is more geared toward more informal QuickVotes. From a democratic and voting technology perspective, the election was a huge success. With 16 voters and 13 candidates, a traditional plurality or "first past the post" election would have been a poor match for their group -- the 16 first-place votes were very split among the candidates. The results also show one very polarizing candidate who won the plurality but was in the bottom third of most preferential rankings! The use of Selectricity helped SFC select a board who better represented the preference of their group than they would have otherwise. Exciting stuff! You can read more on the Free Culture website or on the Selectricity blog. Thanks are due both to the previous SFC board who took the risk on the technology and to all of the candidates and voters! I'm currently integrating feedback and improvements based on the SFC election and will open the feature up the public in the next couple weeks. If you want hear about this when it happens, you should subscribe to the Selectricity Blog or drop an email to team@selectricity.org.

pbuilder with default settings will fail to build a package depending on sun-java5 or sun-java6, because the license needs to be accepted first. Now some time ago I saw a patch for pbuilder itself to “fix” this, but for the time beeing I used to set DEBIAN_FRONTEND to readline in my pbuilderrc. But today Michael Koch came up with much better suggestions: Preset the debconf value in the CHROOT and save it. It’s pretty easy:

$ sudo pbuilder login --save-after-login
# echo "sun-java5-jdk shared/accepted-sun-dlj-v1-1 boolean true"   debconf-set-selections
# echo "sun-java6-jdk shared/accepted-sun-dlj-v1-1 boolean true"   debconf-set-selections
# exit

and voila, problem solved. Thanks to Michael for the tip.

I've gone over the design for "dammit" again, and I think I can even get away with non-specialised nodes for the various compilation tasks in the work tree, which will make support for additional input languages pluggable, an idea I tossed in the beginning for fear that it might not be doable. So I finally get to solve one thing I found annoying with automake. :-) The other big feature I'm excited about is having two different modes of importing stuff from libraries, with and without reexporting the ABI. Depending on that, any newly built library will incorporate the SONAME of the dependent library into its own, and its development package will depend on the other -dev package (obviously, for that to work, I'll have to add package building to it as well :-P )

Yay! I got my new toy two days ago but I wasn't able to install debian on it booting from USB with several versions, when I almost got it there was some issue with the ethernet driver (with sarge) or that didn't loaded the correct driver for detecting my hard drive or worst, with the hd-media, found an iso but didn't work out! Hehehe Anyway, I installed my system trough PXE with my old notebook serving as gateway so I got it! :D It runs smoothly and is the smaller notebook I could ever wish, plus the battery gives me 12h of energy so w00t!! I installed i386 instead of amd64 because of the java-browser plugin that I will use this semester and to get skype and all that. Is not that I will use larger archives and stuff so I will be ok by the moment. Later I will post the spec details, right now I will keep working on a project for my university. BTW, I installed directly sid :D

So today I'm a little bit lazy and I've got the day off work. As my previous plan suggested I wanted to spend at least some of the day tackling semi-random bugs. Earlier I picked a victim: less. less rocks, and I use it daily. I even wrote an introduction to less once upon a time. So lets take a look at two bugs from the long-neglected pile. These two issues are basically the same:

• gzip: zless seems spew raw gzip to output for empty files
• zless thinks gzipped empty file is binary

They seem like simple ones to fix, with the same root cause. Here's an example if you want to play along at home:

 cp /dev/null testing
 gzip testing
 zless testing.gz

What do you see? I see this:

"testing.gz" may be a binary file.  See it anyway? 

When I select "y" I see the raw binary of the compressed file. So, we can reproduce it. Now to see why it happens. /bin/zless comes from the gzip package and is a simple shell script:

#!/bin/sh
# snipped a lot of text
LESSOPEN=" gzip -cdfq -- %s"; export LESSOPEN
exec less "$@"

So what happens if we run that?

$ LESSOPEN=" gzip -cdfq -- ~/testing.gz" /usr/bin/less ~/testing.gz
"/home/skx/testing.gz" may be a binary file.  See it anyway? 

i.e. it fails in the same way. Interestingly this works just fine:

gzip -cdfq -- ~/testing.gz   less

So we've learnt something interesting and useful. We've learnt that when LESSOPEN is involved we get the bug. Which suggests we should "apt-get source less" and then "rgrep LESSOPEN ~/less-*/". Doing so reveals the following function in filename.c:

	public char *
open_altfile(filename, pf, pfd)
	char *filename;
	int *pf;
	void **pfd;
 
/* code to test whether $LESSOPEN is set, and attempt to run the 
   command if it is */
		/*
		 * Read one char to see if the pipe will produce any data.
		 * If it does, push the char back on the pipe.
		 */
		f = fileno(fd);
		SET_BINARY(f);
		if (read(f, &c, 1) != 1)
		 
			/*
			 * Pipe is empty.  This means there is no alt file.
			 */
			pclose(fd);
			return (NULL);
		 
		ch_ungetchar(c);
		*pfd = (void *) fd;
		*pf = f;
		return (save("-"));

That might not be digestible, but basically less runs the command specified in $LESSOPEN. If it may read a single character of output from that command it replaces the file it was going to read with the output of the command instead! (i.e. Here less failed to read a single character, because our gzipped file was zero-bytes long! So instead it reverted to showing the binary gzipped file.) So we have a solution: If we want this to work we merely remove the "read a single character test". I can't think of circumstance in which that would do the wrong thing, so I've submitted a patch to do that. Bug(s) fixed. Incidentally if you like these kind of "debuggin by example" posts, or hate them, do let me know. So I'll know whether to take notes next time or not..

I've replaced the top banner on my personal site with a sunset panorama taken from the other end of Sand Bay. Yes, it may look green and grassy, but it is Sand Bay. I've added a shared bookmarks RSS to my site, generated from the first subfolder in my Bookmarks Toolbar Folder by a daft shell script. I've put an old paper online. Fighting in the shadows: moles and activists is about how to keep an open group, but minimising the effectiveness of moles:-

"It does not matter whether an activist is a mole or not, as long as they work well for you. The mole may be working against you in secret, but they would probably work against you anyway. To be an effective mole, they have to do some helpful work in order to avoid detection. They are a resource that we would not have access to if they weren't a mole."

Finally, I've attended a few events recently, so the next few posts will probably be mostly conference reports.

Say you're working on a Debian box for which you don't have root access, you need to get some source packages, but there is no deb-src lines (or not the ones you need) in /etc/apt/sources.list. So usually you will have to manually wget the files, after looking the exact location, and all that jazz. Today I found that APT can be configured to use a different sources.list file, and different directories for the package lists. Which is enough to make it work for downloading source packages. What you need to do is: $ mkdir apt
$ cp /etc/apt/sources.list apt/
$ vi apt/sources.list # customise as needed
$ mkdir -p apt/lists/partial
$ cat > apt/apt.conf
Dir::Etc::SourceList "home_dir/apt/sources.list"; # You need to use the full path
Dir::State::Lists "home_dir/apt/lists";
^D
$ export APT_CONFIG=~/apt/apt.conf
$ apt-get update
...
$ apt-get source foo
After downloading the lists, it will complain about it not being able to lock the dpkg directory, but no need to worry. It will work perfectly. Tags: Planet Lugfi, Planet Debian

I enjoyed reading the ArchitectNotes on the Varnish wiki:

Take Squid for instance, a 1975 program if I ever saw one: You tell it how much RAM it can use and how much disk it can use. It will then spend inordinate amounts of time keeping track of what HTTP objects are in RAM and which are on disk and it will move them forth and back depending on traffic patterns.

Well, today computers really only have one kind of storage, and it is usually some sort of disk, the operating system and the virtual memory management hardware has converted the RAM to a cache for the disk storage.

So what happens with squid's elaborate memory management is that it gets into fights with the kernel's elaborate memory management, and like any civil war, that never gets anything done.

Varnish was designed by FreeBSD developer Poul-Henning Kamp. There is something to be said about user-space apps designed by kernel developers: they're always super fast because kernel developers know how to make the most of the system, having spent hours optimizing the kernel itself for some usage patterns.

This is why git (designed by Linus Torvalds, for those of you who've lived under a rock for the past three years) is so much faster than other comparable systems, for example. It uses tricks such as giving the O_NOATIME flag to open() to avoid updating access times on files where it knows that they don't matter, and it's a "huge time-saver". (Of course, you can also just mount your partitions with noatime and enjoy a similar performance boost in all applications!)

OK, I may not have wireless yet on my HP laptop but I have managed to get wireless networking running with the new b43 native kernel driver for this iBook G4 laptop.
0001:10:12.0 Network controller: Broadcom Corporation BCM4306 802.11b/g Wireless LAN Controller (rev 03) (that's the Apple internal card).
Linux fergus 2.6.24-1-powerpc #1 Mon Feb 11 12:53:42 CET 2008 ppc GNU/Linux
linux-image-2.6-powerpc 2.6.24+13
I needed:
b43-fwcutter-011.tar.bz2
broadcom-wl-4.80.53.0.tar.bz2
(Note the versions - older versions don't work, newer versions give errors from the kernel and don't work either.)
www locations and full instructions available at http://linuxwireless.org/en/users/Drivers/b43.
The best bit is that because this is a native driver, network-manager is able to find it and use it so that I don't have to waste time with detailed settings in /etc/network/interfaces etc. The worst bit is that it still needs a second internet connection to download the firmware from Broadcom and compile the firmware cutter from source.
As I've only just got it working, I've yet to test it with suspend etc. but at least I no longer have to use bcm43xx (which is deprecated by b43 anyway).
It also appears that this will work with the HP laptop, once the device in that laptop is recognised by the kernel. Just such a pain that Broadcom don't open up the licence for the firmware so that we can have a sensible package that can be installed ready and working without all the work.

Update - it continues working after suspend.

wouter@quickstep:~$ uname -a
Linux quickstep 2.6.24-1-mac #1 Fri Feb 1 00:14:53 CET 2008 m68k GNU/Linux

Ph34r my experimental buildd, which will be back... soon. Unfortunately, doing 'apt-get upgrade' from etch-m68k (the last release to support 2.2, which quickstep was still running) to unstable takes a while. Other than that... yeah! The update to 2.6.24 had some issues; e.g., the key map wasn't set correctly; also, 2.2 and 2.6 have different ideas about what the system clock is, so it insisted on doing an fsck at boot time initially, which with 18G hard disks takes quite a while on a 25Mhz processor. Quickstep was one of the last machines that I had which was still running 2.2; unfortunately it took a long time for 2.6 on m68k to become a viable option. Apparently, the issues that existed for Centris 650 (of which quickstep is one) had been fixed a while ago already; I just hadn't noticed. Oops. My final 2.2-running machine now is barok, my IIci. But that machine isn't very fast anyway, so its lack of 2.6 doesn't really hurt all that much. Having said that, Finn Thain told me he'd be working on IIci support soon, so that issue should be dealt with as well. Whee!