As part of trying to help with Freedom Box, I gained access to a Dream Plug The Dream Plug is one of the leading possible platforms for Freedom Box. Computationally it s sexy for a low-power device. There s a 1200 mhz arm with 512m of RAM. As a platform to enable people to create novel applications, it s great. It has multiple USB ports, two ethernets, a built-in 802.11B/G access point, bluetooth, audio, ESATA, Micro SD and full-sized SD. So, whether you application needs storage, networking, audio, or some interesting side device, you re covered. With the optional JTAG adapter it s even fairly friendly to developers: there are options for recovering from most failures and full console access. If you re looking at a reference platform that s still nominally embedded, but that allows you to play around thinking about what your application could do, this is a great option. However, especially for Freedom Box, it s important to remember that the reference board a developer wants is not the same as a cost-reduced product on which to actually deploy something for people to buy. The Dream Plug is wrong for every actual deployment I ve imagined. First, it s huge. If you happen to have free power plugs on your wall with lots of space above and below them, it might be an option. If however, you re like everyone I know and have power strips, constrained spacing, etc, then the industrial engineering will disappoint at every turn. I was hoping for something that kind of looked like an Apple Airport Express. It s significantly larger than that in every dimension. Also, the plug is oriented the wrong way for minimizing the fraction of the power strip it takes up. The power supply part of the device can be detached, although even that is way too huge for a power strip, and the cable between the power supply and the computer is fairly short. You can reconfigure the device without its plug nature: running a cord from a normal power outlet to the power supply. But then if you have to detach the power supply for heat management reasons, you get a cord from the outlet to the power supply and another cord from the supply to the device. Add a few USB or ESATA devices and the octopus of cables begins to resemble some arcane mechanism. (So far, no elder gods have appeared though.) The other issue is that you almost never want all the functionality. You pay for the bluetooth, audio and ESATA in terms of cost, heat and space regardless of whether you use them. I don t have a lot of applications that really take advantage of the full array of hardware. The firmware update mechanism is decidedly not targeted at end-users. The version I obtained had a hacked copy of Debian lenny. No mechanism was provided for replacing the image in a safe manner that did not potentially require the optional highly-non-end-user-compatible JTAG board if something got interrupted. You could either unscrew the device and get to the micro SD card containing the image, or run software to replace the image from within Debian lenny. It s possible to configure the boot loader to run an update image off USB or SD until that succeeds, but doing that is also a non-end-user operation. In conclusion, the name is perfect. This is exactly what hackers need to dream about the power of small computers everywhere. However we must not forget that there s a step required to turn dreams into reality. Just as with any fully proprietary product, Freedom Box will require cost reduction steps, semi-custom boards and actual OEMs to truly be usable. The claim in the previous sentence that the Freedom Box may have proprietary elements is disquieting to some. I think we can put together a software stack that is free with the possible exception of some device firmware. However, my suspicion is that anyone who turns that into a fully realized end-user product will add proprietary elements. I suspect some of the results of the cost reduction process such as resulting semi-custom boards will be proprietary. In many cases though, I suspect some proprietary software elements will be introduced.

In the graphics side, the major differences between ARM and X86 systems is that on ARM 3D acceleration is done with OpenGL ES. It is mostly a subset of modern OpenGL used on X86 desktop machines. From QEMU point of view this could mostly be ignored, as OpenGL was mostly used on games and specialist applications. This has now been changing, as desktops and user interfaces have started OpenGL to render graphics. Without acceleration, these user interfaces become slower than slugs crossing a tarpit.

For this reason, MeeGo introduced OpenGL ES acceleration support to QEMU. With the lack of easily available MeeGo QEMU images and test setup, I've created a test setup for Linaro 11.05 image.

Running torus opengl es1 demo from mesa-demos inside QEMU.

I ve gotten fed up with Evolution s bugs and slowness. Even though I experience less problems than some weeks ago since I ve applied a patch related to locks in e-d-s, Evolution has now become really too slow and given that my mail flow is not really diminishing, I ve come to the point when I think about migrating. I ve been a user of Gnus for years also (at home, with much satisfaction, as it runs fine inside a GNU Screen), so I could switch to it with moderate damage. Also, as I m using org-mode, it would be quite complementary of course, staying more in Emacs One nice (and even priceless) feature of Evolution is the vfolders, that allow me to manage the tons of emails in different contexts, wherever that may be located. AFAIK, Gnus doesn t offer any comparable feature. But it seems that the most interesting way to have it is to integrate Gnus with NotMuch. Fortunately, my colleague Roland has written a nice howto explaining (among other details of his mail system setup) how to integrate these together. Now, if I m able to setup a similar NotMuch + Gnus setup for newly received mail, I m left with migrating all my piles of old mail, currently stored in Evolution. It seems that one possible way to do so is through copying Evolution mail folders into a newly setup local IMAP server s Maildir storage. The copy is supposed to preserve some flags like the read/unread status of the important flag. From the first tests I ve made, it s possible to install a local Dovecot IMAPd server, configure it so that the mail is stored in the user s ~/Maildir/ maildir directory, which can then be recognized by Evolution as a target for copying mail. It seems that nomuch can then be configured so that it doesn t flag all new mails as unread, and understangs the imap server flags accordingly (unread, important, and such). Needless to say, I m glad all these useful programs are packaged in Debian I need to make further tests and also test Gnus + Notmuch integration, but having a possible solution to migrate my existing mail looks like a relief. I d be curious to read your alternative ideas for such a migration. Stay tuned for next iterations.

As a Debian/Ubuntu user, you re likely to be exposed at some point to an error reported by dpkg. In a series of articles, I ll explain some of the errors that you might encounter. Some error messages can be confusing at times. Most of the error strings do not appear very often and developers thus tend to use very terse description of the underlying problem. In other cases the architecture of the software makes it difficult to pin-point the real problem because the part that displays the error is several layers above the one that generated the initial error. This is for example the case with this error of dpkg:

Unpacking replacement xulrunner-1.9.2 ...
dpkg-deb (subprocess): data: internal gzip read error: '<fd:0>: too many length or distance symbols'
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing /var/cache/apt/archives/xulrunner-1.9.2_1.9.2.17+build3+nobinonly-0ubuntu1_amd64.deb (--unpack):
 short read on buffer copy for backend dpkg-deb during  ./usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so'

First, the decompression layer discovers something unexpected in the data read in the .deb file and dpkg-deb outputs the error message coming from zlib ( too many length or distance symbols ). This causes the premature end of dpkg-deb --fsys-tarfile that dpkg had executed to extract the .data.tar archive from the deb file. In turn, dpkg informs us that dpkg-deb did not send all the data that were announced (and hence the short read in the error message) and that were meant to be part of the file /usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so . That s all nice but it doesn t help you much in general. What you must understand from the above is that the .deb file is corrupted (sometimes just truncated). In theory it should not happen since APT verifies the checksums of files when they are downloaded. But computers are not infallible and even if the downloaded data was good, it can have been corrupted when stored on disk (for example cheap SSD disks are known to not last very well). Try removing the file (usually with apt-get clean since it s stored in APT s cache) and let APT download it again. Chances are that it will work on the second try. Otherwise consider doing a memory and HDD check as something is probably broken in your computer.

Join my free newsletter and learn more tips for users.

4 comments Liked this article? Click here. My blog is Flattr-enabled.

I sponsored the upload of fonts-anonymous-pro prepared by Rog rio Brito. This fixed-width font is especially designed for people who write code, for instance by making clear differences between characters such as O and 0, or I, l and 1. I fixed a few glitches in various l10n scripts related to D-I (those tthigns that run daily, for which I receive a daily mail). They were the consequence of Alioth migration and I omitted to fix them until now....mumbling every morning about the scripts' noisy output..:-) I worked on only one bug for samba (doh). This was #631118 which I forwarded upstream in order to get it fixed in 3.5 series (even if 3.6 is due out soon and will be what we target for wheezy). I tried to follow the flow of translations/reviews in DDTP for French. The activity is high, here, with about 10 packages translated every day (most often packages I target for the team...by using high-popcon stats). We even had a big bump during the week-end when Georges Khaznadar (who is very active when it comes at educative software) initiated the translation of many edu-related packages (many of which he maintains himself). That's about it. Quiet week, indeed with more time invested in paid work, including during my spare time..:-). And also a week-end in our country house in Normandy, which usually means more gardening than hacking.

The goal of this lab is to setup a site-to-site IPsec VPN. This lab is similar to my first lab using UML. The major differences are:

• We only setup one VPN instead of two.
• Static routing is used in place of BGP for inter-site routing. Moreover, BIRD is used as a routing daemon.
• VPN are using an external network for Internet access.
• Both internal network and external network are redundant using one OSPF instance each.

The lab

The big picture The main point of this architecture is that you get two redundant networks for each site. The external ones are used when hosts need Internet access. The internal ones are used when they want to communicate with each other. Because we don t want hosts from the first site to communicate with hosts from the second site through the external network, we extend the internal network with the help of a site-to-site IPsec VPN. Each set of networks is using two layer-3 networks on top of two layer-2 networks. OSPF is used to provide redundancy and aggregation. For example, R1 has two routes through V1 to access R2 which is on the other site:

# ip route show 192.168.96.0/19
192.168.96.0/19  proto zebra  metric 10000 
        nexthop via 192.168.1.100  dev eth0 weight 1
        nexthop via 192.168.2.100  dev eth1 weight 1

Setting up the lab First, grab the source of the lab:

$ git clone https://vincentbernat@github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-s2s-vpn

You may try to run ./setup directly. It may be safer to build an image of Debian inside inside a directory and use it as a base for the lab.

$ sudo debootstrap sid ../sid-chroot http://ftp.fr.debian.org/debian/
[...]
$ sudo chroot ../sid-chroot /bin/bash
# apt-get install iproute zsh rsyslog less makedev tcpdump
# apt-get install bird quagga racoon
# exit
$ ROOT=../sid-chroot ./setup

You still need a working UML kernel. On Ubuntu, the one provided in user-mode-linux does not contain the aufs.ko module. I did not find a simple workaround. You can just grab the package from Debian and install it on your Ubuntu with dpkg -i.

Use of BIRD With the previous lab, Quagga was used as an OSPF routing daemon. We still use it on R1, R2, E1 and E2. However, Quagga is not able to run multiple instances of OSPF. BIRD can. Therefore, we use it on V1 and V2 because we need two OSPF instances: one for the external network and one for the internal network. Since we also need ECMP, we need at least BIRD 1.3.0. BIRD allows the definition of multiple routing tables. Those tables are independant of the kernel routing tables. Each protocol will maintain its own set of routes and will allow to export (or import) routes from one of the routing table maintained by BIRD. In our setup, we use four protocols and only one routing table (the default one). The ospf EXTERNAL protocol is one of the two OSPF instances we use on V1. We import all the routes learned through the OSPF process into the internal BIRD routing table. We don t export any route from the BIRD routing table to this OSPF instance. Here is the configuration of this protocol:

protocol ospf EXTERNAL  
   ecmp yes;
   import all;
   export none;
   area 0.0.0.0  
      networks  
         203.0.113.0/26;
         203.0.113.64/26;
         203.0.113.150/32;
       ;
      interface "eth2";
      interface "eth3";
      interface "lo"  
        stub yes;
       ;
    ;
 

203.0.113.150/32 is defined on the loopback interface of V1. This is the IP of the local VPN endpoint. The ospf INTERNAL protocol is the second OSPF instance. We import all routes from this protocol to the BIRD routing table. We also want to advertise the route to the remote site. This route is defined in the static STATIC protocol and imported into BIRD routing table.

protocol static STATIC  
   import all;
   export none;
   route 192.168.96.0/19 via "lo";
 

Therefore, ospf INTERNAL protocol export all routes originating from the static STATIC protocol. This is done using filters:

protocol ospf INTERNAL  
   ecmp yes;
   import all;
   export filter  
     if proto = "STATIC" then accept;
     reject;
    ;
   area 0.0.0.0  
      networks  
         192.168.1.0/24;
         192.168.2.0/24;
       ;
      interface "eth0";
      interface "eth1";
    ;
 

The kernel protocol is managing one kernel routing table. By default, it handles the main table. We don t import any route from the kernel. However, we copy routes that come from other protocols, except the static route because we already have a default route and the static route is some kind of blackhole, we don t want to use it for real routing.

protocol kernel  
   persist;
   import none;
   export filter  
     if proto = "EXTERNAL" then accept;
     if proto = "INTERNAL" then accept;
     reject;
    ;
 

The filtering mechanism built inside BIRD is pretty powerful.

Testing After starting the lab, you need to wait a few seconds. You should then be able to ping R2 from R1:

R1# ping -c3 -I 192.168.15.1 192.168.115.1
PING 192.168.115.1 (192.168.115.1) from 192.168.15.1 : 56(84) bytes of data.
Warning: time of day goes back (-5435us), taking countermeasures.
64 bytes from 192.168.115.1: icmp_req=1 ttl=62 time=0.921 ms
64 bytes from 192.168.115.1: icmp_req=2 ttl=62 time=0.569 ms
64 bytes from 192.168.115.1: icmp_req=3 ttl=62 time=0.698 ms
--- 192.168.115.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2009ms
rtt min/avg/max/mdev = 0.569/0.729/0.921/0.147 ms

On V1, you can check BIRD internal routing table using birdc:

> show route 
0.0.0.0/0          multipath [EXTERNAL 19:43] * E2 (150/10/1) [0.0.0.11]
        via 203.0.113.1 on eth2 weight 1
        via 203.0.113.65 on eth3 weight 1
192.168.96.0/19    dev lo [STATIC 19:42] * (200)
192.168.1.0/24     dev eth0 [INTERNAL 19:42] * I (150/10) [0.0.0.100]
192.168.2.0/24     dev eth1 [INTERNAL 19:42] * I (150/10) [0.0.0.100]
192.168.15.0/24    multipath [INTERNAL 19:43] * I (150/20) [0.0.0.10]
        via 192.168.1.10 on eth0 weight 1
        via 192.168.2.10 on eth1 weight 1
203.0.113.0/26     dev eth2 [EXTERNAL 19:42] * I (150/10) [0.0.0.100]
203.0.113.64/26    dev eth3 [EXTERNAL 19:42] * I (150/10) [0.0.0.100]
203.0.113.150/32   dev lo [EXTERNAL 19:42] * I (150/0) [0.0.0.100]

The kernel routing table is similar except it does not contain the static route.

Adding more redundancy The first goal of this lab was to test multiple OSPF instances and ECMP with BIRD. It could be completed:

• Adding a redundant VPN is easy: this is just a matter of duplicating the setup of the existing VPN.
• Each router should be redundant by getting a sibling with the same configuration. In the case of R1 and R2, a VIP could be shared between the router and its sibling with VRRP to allow hosts behind to have a single gateway and not use a routing protocol.
• Because of the use of static routes instead of BGP, there is nothing to detect a VPN failure (which may be problematic if we setup a redundant VPN). Some VPN are able to invalidate a route when the associated VPN is down. This is not our case. Moreover, there may be some duplication in routing. This is not the case here because each site has its own internal network.

The goal of this lab is to setup a site-to-site IPsec VPN. This lab is similar to my first lab using UML. The major differences are:

• We only setup one VPN instead of two.
• Static routing is used in place of BGP for inter-site routing. Moreover, BIRD is used as a routing daemon.
• VPN are using an external network for Internet access.
• Both internal network and external network are redundant using one OSPF instance each.

The lab

The big picture The main point of this architecture is that you get two redundant networks for each site. The external ones are used when hosts need Internet access. The internal ones are used when they want to communicate with each other. Because we don t want hosts from the first site to communicate with hosts from the second site through the external network, we extend the internal network with the help of a site-to-site IPsec VPN. Each set of networks is using two layer-3 networks on top of two layer-2 networks. OSPF is used to provide redundancy and aggregation. For example, R1 has two routes through V1 to access R2 which is on the other site:

# ip route show 192.168.96.0/19
192.168.96.0/19  proto zebra  metric 10000 
        nexthop via 192.168.1.100  dev eth0 weight 1
        nexthop via 192.168.2.100  dev eth1 weight 1

Setting up the lab First, grab the source of the lab:

$ git clone https://vincentbernat@github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-s2s-vpn

You may try to run ./setup directly. It may be safer to build an image of Debian inside inside a directory and use it as a base for the lab.

$ sudo debootstrap sid ../sid-chroot http://ftp.fr.debian.org/debian/
[...]
$ sudo chroot ../sid-chroot /bin/bash
# apt-get install iproute zsh rsyslog less makedev tcpdump
# apt-get install bird quagga racoon
# exit
$ ROOT=../sid-chroot ./setup

You still need a working UML kernel. On Ubuntu, the one provided in user-mode-linux does not contain the aufs.ko module. I did not find a simple workaround. You can just grab the package from Debian and install it on your Ubuntu with dpkg -i.

Use of BIRD With the previous lab, Quagga was used as an OSPF routing daemon. We still use it on R1, R2, E1 and E2. However, Quagga is not able to run multiple instances of OSPF. BIRD can. Therefore, we use it on V1 and V2 because we need two OSPF instances: one for the external network and one for the internal network. Since we also need ECMP, we need at least BIRD 1.3.0. BIRD allows the definition of multiple routing tables. Those tables are independant of the kernel routing tables. Each protocol will maintain its own set of routes and will allow to export (or import) routes from one of the routing table maintained by BIRD. In our setup, we use four protocols and only one routing table (the default one). The ospf EXTERNAL protocol is one of the two OSPF instances we use on V1. We import all the routes learned through the OSPF process into the internal BIRD routing table. We don t export any route from the BIRD routing table to this OSPF instance. Here is the configuration of this protocol:

protocol ospf EXTERNAL  
   ecmp yes;
   import all;
   export none;
   area 0.0.0.0  
      networks  
         203.0.113.0/26;
         203.0.113.64/26;
         203.0.113.150/32;
       ;
      interface "eth2";
      interface "eth3";
      interface "lo"  
        stub yes;
       ;
    ;
 

203.0.113.150/32 is defined on the loopback interface of V1. This is the IP of the local VPN endpoint. The ospf INTERNAL protocol is the second OSPF instance. We import all routes from this protocol to the BIRD routing table. We also want to advertise the route to the remote site. This route is defined in the static STATIC protocol and imported into BIRD routing table.

protocol static STATIC  
   import all;
   export none;
   route 192.168.96.0/19 via "lo";
 

Therefore, ospf INTERNAL protocol export all routes originating from the static STATIC protocol. This is done using filters:

protocol ospf INTERNAL  
   ecmp yes;
   import all;
   export filter  
     if proto = "STATIC" then accept;
     reject;
    ;
   area 0.0.0.0  
      networks  
         192.168.1.0/24;
         192.168.2.0/24;
       ;
      interface "eth0";
      interface "eth1";
    ;
 

The kernel protocol is managing one kernel routing table. By default, it handles the main table. We don t import any route from the kernel. However, we copy routes that come from other protocols, except the static route because we already have a default route and the static route is some kind of blackhole, we don t want to use it for real routing.

protocol kernel  
   persist;
   import none;
   export filter  
     if proto = "EXTERNAL" then accept;
     if proto = "INTERNAL" then accept;
     reject;
    ;
 

The filtering mechanism built inside BIRD is pretty powerful.

Testing After starting the lab, you need to wait a few seconds. You should then be able to ping R2 from R1:

R1# ping -c3 -I 192.168.15.1 192.168.115.1
PING 192.168.115.1 (192.168.115.1) from 192.168.15.1 : 56(84) bytes of data.
Warning: time of day goes back (-5435us), taking countermeasures.
64 bytes from 192.168.115.1: icmp_req=1 ttl=62 time=0.921 ms
64 bytes from 192.168.115.1: icmp_req=2 ttl=62 time=0.569 ms
64 bytes from 192.168.115.1: icmp_req=3 ttl=62 time=0.698 ms
--- 192.168.115.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2009ms
rtt min/avg/max/mdev = 0.569/0.729/0.921/0.147 ms

On V1, you can check BIRD internal routing table using birdc:

> show route 
0.0.0.0/0          multipath [EXTERNAL 19:43] * E2 (150/10/1) [0.0.0.11]
        via 203.0.113.1 on eth2 weight 1
        via 203.0.113.65 on eth3 weight 1
192.168.96.0/19    dev lo [STATIC 19:42] * (200)
192.168.1.0/24     dev eth0 [INTERNAL 19:42] * I (150/10) [0.0.0.100]
192.168.2.0/24     dev eth1 [INTERNAL 19:42] * I (150/10) [0.0.0.100]
192.168.15.0/24    multipath [INTERNAL 19:43] * I (150/20) [0.0.0.10]
        via 192.168.1.10 on eth0 weight 1
        via 192.168.2.10 on eth1 weight 1
203.0.113.0/26     dev eth2 [EXTERNAL 19:42] * I (150/10) [0.0.0.100]
203.0.113.64/26    dev eth3 [EXTERNAL 19:42] * I (150/10) [0.0.0.100]
203.0.113.150/32   dev lo [EXTERNAL 19:42] * I (150/0) [0.0.0.100]

The kernel routing table is similar except it does not contain the static route.

Adding more redundancy The first goal of this lab was to test multiple OSPF instances and ECMP with BIRD. It could be completed:

• Adding a redundant VPN is easy: this is just a matter of duplicating the setup of the existing VPN.
• Each router should be redundant by getting a sibling with the same configuration. In the case of R1 and R2, a VIP could be shared between the router and its sibling with VRRP to allow hosts behind to have a single gateway and not use a routing protocol.
• Because of the use of static routes instead of BGP, there is nothing to detect a VPN failure (which may be problematic if we setup a redundant VPN). Some VPN are able to invalidate a route when the associated VPN is down. This is not our case. Moreover, there may be some duplication in routing. This is not the case here because each site has its own internal network.

I had cause to try and get a core dump from a segfaulting process at work the other day, and I wanted to figure out if fiddling with /etc/security/limits.conf was going to do the trick (I didn't) or if I had to modify the initscript to include a ulimit -c unlimited call. Of course, on a modern system (>= 2.6.24), one would just take a look at /proc/PID/limits and get on with life, but unfortunately the system in question was running 2.6.18, so more fiddling around was required. I'd found something once before that told me how to do it with GDB, but all I could find this time around was a rather over-complicated Knol article, which made a bunch of assumptions (well mainly that the binary in question wasn't stripped). So with some help from the Knol article, I muddled through it. Disclaimer: I don't profess to be an expert on system internals like this, so if this eats your first-born, don't come crying to me. Firstly, you need to know that it's the getrlimit(2) system call that you want to be using, and then you need to figure out the number for the resource limit you want to retrieve. The man page for getrlimit(), tells you it's defined in /usr/include/sys/resource.h, but I've found that the actual useful bits end up being in /usr/include/bits/resource.h I wanted the resource limit for the maximum core dump size, which is RLIMIT_CORE and has a value of 4. Next, you need to know that the getrlimit() system call takes an integer and a pointer to an rlimit structure as arguments. We've just figured out the value for the integer, but we're also going to need to pass a pointer as the second parameter. A pointer to enough memory to hold an rlimit structure. Fortunately, the rlimit structure is pretty simple:

struct rlimit  
    rlim_t rlim_cur;  /* Soft limit */
    rlim_t rlim_max;  /* Hard limit (ceiling for rlim_cur) */
 ;

After a bit of grepping around in /usr/include, I determined that an rlim_t is essentially an unsigned long int, so we need to allocate a pointer big enough to hold two of them. Note that if we had an unstripped binary, we could have saved a lot of faffing around by just going

print sizeof(struct rlimit)

in GDB (assuming that the binary has the getrlimit symbol in it) The sure-fire way of figuring out how much memory we need for this pointer is to go

print sizeof(unsigned long int)

in GDB, and then double that (since we want two of them). On my system an unsigned long int is four bytes, so I'm going to want to allocate enough memory for 8 bytes. Now it's time to attach GDB to the offending process and see what the resource limit currently is. (gdb -p PID)

(gdb) print malloc(8)
$1 = 152186904
(gdb) print getrlimit(4, $1)
$2 = 0
(gdb) x/2xw $1
0x9123018:	0x00000000	0x7fffffff
(gdb) quit

So in this particular case, we've retrieved the soft and hard limits of the RLIMIT_CORE resource limit, and you can see that the soft limit is zero, and the hard limit is unlimited. Note that the getrlimit() function returns an integer as its return code, which is what the $2 = 0 is above. Now it's just a case of altering the the resource limits via the preferred mechanism, restarting the process and then repeating this GDB examination of the process to check they were changed successfully. Circumstances where this can all fall down would appear to be ones where the getrlimit symbol isn't present in the binary, or the binary is compiled as a position-independent executable. I'd think that in the latter case, the system is probably modern enough to be running a kernel that supports directly examining the resource limits via the /proc filesystem.

Due to two totally different bugs in tip22 the d-i netinstall images for IP22 (SGI Indy) and IP32 (SGI O2) won't boot the installer correctly. Tip22 is responsible for merging a minimal loader, kernel and initramfs into a single ELF or ECOFF binary, depending on the architecture. This can then be fetched by the machine's ARC PROM via tftp. A fixed tip22 has been uploaded to unstable and updated images are available from here until we can get this fixed via stable-proposed-updates. This blog is flattr enabled.

While I'm keenly interested in where mobile phone technology is going, I've never been much for living on the cutting edge of new and expensive phones. I originally wrote Hecl for the humble Nokia 3100, and got a Nokia 6210 classic a few years ago. For the first time in... probably something like 10 years, I got something that wasn't a Nokia: a new Samsung phone with Android. I find it interesting because my own, small purchase appears to very closely mirror a broader trend in the market. Since more or less the day it came out, Android jumped out at me as the place to be for an open source guy like me, and indeed, I'm quite happy with the phone, even though it's towards the low end of the Android range. It does everything I need though - music, GPS, photos, email, Kindle, etc... etc.... To tell the truth, even though it's a cheaper phone (around 140 euros) than the Nokia, it does far more. To some degree, that's of course to be expected, because the Nokia is at this point something like 3 years old, but going from the traditional rocker switch to a touch screen is really a night-and-day difference. Many of you are probably saying "duh" at me and wondering what sort of luddite I am, but like I said, I started hacking on Android when it came out, but it took me a while to finally get one. For a few things like making calls, the Nokia experience is in some ways still superior. It's got 'call' buttons that you can hit and be talking to people in short order, but for most of the rest of the applications and software are just night and day better on the Android phone. Gmail and Maps both strike me as something I could use more readily, rather than the hacky, slow Gmail that ships for Symbian. And of course, with a slightly bigger screen, the web is also a bit better. More than anything, it feels a lot more like a 'system' rather than a very, very solid core phone/sms system with some other junk bolted on.

Sam Hartman is a Debian developer since 2000. He has never taken any sort of official role within Debian (that is besides package maintainer), yet I know him for his very thoughtful contributions to discussions both on mailing lists and IRL during Debconf. Until I met him at Debconf, I didn t know that he was blind, and the first reaction was to be impressed because it must be some tremendous effort to read the volume of information that Debian generates on mailing lists. In truth he s at ease with his computer much like I am although he uses it in a completely different way. Read on to learn more, my questions are in bold, the rest is by Sam. Raphael: Who are you? Sam: I m Sam Hartman. I m a 35-year-old software engineer. I am a principal consultant and co-owner of a small consulting company, called Painless Security. I started using Debian in the mid 1990 s around the time of the bo release. I ended up deciding to join the project as a developer in 2000. Raphael: You re blind and yet you re using your computer as effectively as I am. Can you explain us how you setup your computer ? Sam: I gave a talk at Debconf9 on how my computer is set up; you can watch the video for full details. My main laptop runs Debian. I use the gnome-orca package as my primary screen reader. It speaks the Gnome desktop. It does a relatively good job of speaking Iceweasel/Firefox and Libreoffice. While it does speak gnome-terminal, it s not really good enough at speaking terminal programs that I am comfortable using it. So, I run Emacs with the Emacspeak package. Within that, I run the Emacs terminal emulator, and within that, I tend to run Screen. For added fun, I often run additional instances of Emacs within the inner screens. Raphael: Are there important problems in Debian in terms of accessibility to blind people? KDE documentation talks a lot about accessibility, but at least for blind users, the code completely fails to deliver. That means there are a lot of good packages a blind user cannot use. The non-free Adobe Flash player has some accessibility, but it could be a lot better. The free alternatives have none. The free PDF readers have basically no accessibility. You can use pdftotext, but you cannot actually read a PDF in a graphical application. It s way too easy for a misbehaving program to lock up the entire accessibility infrastructure. Gnash is a big culprit here: if my Iceweasel starts Gnash, there s a good chance that either Iceweasel or the entire desktop will appear to hang from an accessibility standpoint. Other programs, including gksu tend to fail in this way. Some of the more dynamic website features like pop up menus or selection lists are really difficult to find and click without causing them to disappear. This gets better and worse over time as the accessibility support in Iceweasel changes and as websites change. Raphael: What s your biggest achievement within Debian? Sam: I decided to be a Debian developer because I thought that in 2000, Debian support for using enterprise security and infrastructure software was lacking. Back then, any software that included crypto functionality was segregated into a special non-us archive. Some of the software was missing; I started by packaging MIT Kerberos for Debian. Other software had security or enterprise features disabled in the packages. At the height of working on this, I was maintaining krb5, some SASL modules, PAM, some PAM modules,OpenAFS, a version of and Ssh with Kerberos support. I was also involved in the effort to resolve the legal issues so that we could move security software into the main archive. I think this work has been a huge success. In fact, it s been such a success that other people are now doing most of the work. I still maintain the krb5 package. When I started I felt like I was pushing against the flow trying to get people to add patches, sometimes even having to fork a package. However, now, I can maintain just one component and there are enough others who shared my original goal that the work continues. These days, I m working on something that s an evolution of this enterprise security work. I m packaging Project Moonshot for Debian and Ubuntu. Project Moonshot is a response to the wide variety of identity management systems such as OpenID, Oauth, SAML, and the like. Moonshot works to create an identity management approach that works well both for the web and for client-server and other applications. The project is a lot of fun, but the role of Debian in the project is also interesting. One of the things we want to show with Moonshot is how our technology can be effective when it is integrated throughout a platform. To really show the potential it needs to work with as many applications in a given platform as possible. The open and collaborative nature of the Debian community makes it possible to introduce a new technology and have that technology evaluated based on its merits. We don t need huge business relationships or marketing deals to integrate our technology: we need some combination of doing the work ourselves and showing others the benefits of working with us. For someone trying to do innovative work, the Debian model is powerful. Raphael: If you could spend all your time on Debian, what would you work on? Sam: I d really love to work with the embedded Debian folks. The vision of a single source base that could be the stack from devices from small embedded devices all the way up to high-end servers is very appealing. Doing that with Debian involves a number of challenges. However with the right people working on meeting these challenges full-time, I think we could offer something promising. I d also love to have the time to contribute to project infrastructure: working on the release team, helping ftpmaster, that sort of thing. However I don t. I m just happy that so much of my consulting practice involves working on open-source software. Raphael: What s the biggest problem of Debian? Sam: There s something really not right about how we transition libraries from unstable to testing. Every time I get involved in a library transition I m shocked at how complicated it is and how disruptive it is both to testing and unstable. We need to look at technology and processes to break up the dependency snarles. For example we don t have good archive tools for keeping old versions of libraries around to ease transitions. If you haven t thought about this issue you ll probably say that I m being overly picky and this can t be the major problem for Debian. However, if you think about how much this impacts our ability to introduce things into unstable around the times of the freeze or about how much it slows the release process, you may begin to appreciate how big of an issue this is. Raphael: Is there someone in Debian that you admire for their contributions? Sam: There are a number of people who have been role models for me over the years. Anthony Towns really helped me understand a lot of what drives free-software projects and what needs to be true for positive motivation. Joey Hess showed us all that sometimes, social problems do have technical solutions. If the tools are so good that doing the right thing is far easier than any other course of action, quality improves.

---

Thank you to Sam for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

I read something about dh_make again a while ago, which made me wonder, besides dh_make, how do people generally bootstrap new Debian packages. I don't think I've ever actually used dh_make for anything but experimenting. For the longest time, copying over the packaging files from some similar package worked pretty well.

Recently, however, I have applied a completely new strategy: I start with nothing, call debuild, fix the error, retry, until it works. This might sound pretty bizarre, but it works pretty well and enhances your understanding of how the packaging tools work.

Here is about how it works:

Start with nothing, call debuild, it complains:

cannot find readable debian/changelog anywhere!

Call dch --create, which says:

Cannot find debian directory!

OK, mkdir debian, call dch --create again, fill in the template. Call debuild again, now it says:

but there's no debian/rules there!

Create a debian/rules, that's easy to do by heart nowadays, at least as a start:

#!/usr/bin/make -f

%:
        dh $@

Call debuild again, now dpkg-buildpackage actually starts, but stops with:

dpkg-source: error: cannot read mypackage/debian/control: No such file or directory

At this point I'm too lazy to figure out what is necessary to put into debian/control, so let's just keep it simple and touch debian/control. At this point dpkg-source gets into a bit of trouble:

Use of uninitialized value $sourcepackage in sprintf at /usr/bin/dpkg-source line 290.

So let's put that in and also follow the subsequent advice to add debian/source/format and the Maintainer and Standards-Version fields. So far we have:

Source: mypackage
Maintainer: Peter Eisentraut <petere@debian.org>
Standards-Version: 3.9.2

The next hint of a message is:

dh: No packages to build.

This means we need a binary package stanza, so I'll just add

Package: mypackage
Architecture: any

Now debhelper complains:

dh: Compatibility levels before 5 are deprecated.

Not sure why it didn't complain about that earlier. Let's stick 8 in there.

At this point I had to do actual work and mess around with debian/rules a bit to get the package to actually build, but a few minutes later I had a functioning provisional package.

The next step is to clean up the warnings from the various dpkg-* workers:

dpkg-gencontrol: warning: missing information for output field Description
dpkg-gencontrol: warning: package mypackage: unused substitution variable $ shlibs:Depends 
dpkg-deb: warning: parsing file 'debian/mypackage/DEBIAN/control' near line 6 package 'mypackage': missing description
dpkg-genchanges: warning: missing Section for binary package mypackage; using '-'
dpkg-genchanges: warning: missing Priority for binary package mypackage; using '-'
dpkg-genchanges: warning: missing Section for source files
dpkg-genchanges: warning: missing Priority for source files

So we add Description, Depends, Section, and Priority.

And finally we have a list of complaints from Lintian to address:

W: mypackage source: debhelper-but-no-misc-depends mypackage
E: mypackage source: package-uses-debhelper-but-lacks-build-depends
W: mypackage source: package-needs-versioned-debhelper-build-depends 8
W: mypackage source: debhelper-overrides-need-versioned-build-depends (>= 7.0.50~)
E: mypackage source: temporary-debhelper-file debhelper.log
E: mypackage: no-copyright-file

The only things I added manually after that were Vcs-*, Homepage, and Enhances.

Now the only things left to do are running the thing through cowbuilder a few times and putting in all the necessary build dependencies, and writing a nice changelog entry.

Note, this method does not replace putting in some thought. But it's an interesting way to get a relatively clean package.

It took a long time to get all the parts of the Xen support into the Linux kernel. While rudimentary Dom0-support was available since 2.6.38, support for device backends were missing. It was possible to replace this backend with a userspace implementation included in qemu, but I never tested that. With Linux 3.0, both the traditional block backend and the network backend are available. They are already enabled in the current 3.0-rc3/-rc4 packages in experimental, so the packages can be used as Dom0 and run guests. Right now the backend modules are not loaded, so this still needs some work. Neither the init scripts loads them, because the names where in flux the last time I laid hand on it, nor does the kernel themself expose enough information to load them via udev. I think using udev to load the modules is the way to go. This step marks the end of a five year journey. Around 2.6.16 the Xen people started to stay really close to Linux upstream. With the 2.6.18 releas this stopped and the tree was pushed in different states into Debian Etch and RHEL 5. After that, Xen upstream ceased work on newer versions completely, only changes to the now old 2.6.18 tree where done. SuSE started a forward port of the old code base to newer kernel versions and Debian Lenny released with such a patched 2.6.26. Around that time, minimal support for DomU on i386 using paravirt showed up and Lenny had two different kernels with Xen support. Since 2.6.28 this support was mature and works rather flawless since. Somehow after that, a new port of the Dom0 support, now using paravirt, showed up. This tree based on 2.6.32 is released with Debian Squeeze. After several more rounds of redefining and polishing it is now mostly merged into the core kernel. I don't know what the future brings. We have two virtualization systems supported by Linux now. The first is KVM that converts the kernel into a hypervisor and runs systems with help of the hardware virtualization. The later one is Xen that runs under a standalone hypervisor and supports both para- and hardware virtualization. Both works, KVM is easier to use and even works on current System z hardware. It can be used by any user with hopefully enough margin of security between them. Xen's home is more suited for servers, where you don't have users at all. Both have advantages and disadvantages, so everyone have to decide what he needs, there is no "one size fits all".

Second speaker at today'sseminar,Vladimir Mojica. He is talking about th legal backing for DRM and TPM as well as laws against circumvention. He quotes USA's DMCA as one of the most complete, advanced and forward-minded laws,inviting the audience to push for such a law here. I hope he gets to my (written)question, as today is a very important day in this regard: The Senate has requested the presidency to reject signing the ACTA treaty!
[Update] The Senate is sending Resolution agreeing to prompt the Federal Executive Power titular to instruct the State Secretaries and other dependencies involved in the Anti-Counterfeiting Trade Agreement not to sign said agreement, as it would hinder free interchange of legal material. It also mentions the ISPs should not become de-facto authorities to define when an intellectual/intangible properties right violation is in course, and that it would be unacceptable for ISPs to monitor the users' activities in the net, or to analize the content of the IP packets. Very good news, in a document very well made, in a precise, legal and coherent(!) way.

When the programming language Python came out while it seemed like an interesting idea, I didn t really see the point. I already know Perl, PHP and C (plus a few others) why learn another language? So until recently, I didn t Recent frustrations with PHP and how loose it is with things like variables made me have another look at Python and what it could do for me. As just learning a language by itself it pretty boring I decided to write some small programs in the language and used the framework Turbogears to do it. First of all, learning the basics of python from perl or PHP is dead simple. The idea of what a language looks like is not too different between these three so it was a snap to write simple stuff in python. So, what does yet-another language give me?

• Variables must be defined before being used, but assigning them defines them too. This is a nice compromise between PHP where you can check undefined variables and C where it all must be defined first, or else.
• try and except. For code that might fail, use try and then catch the error in except. We ve all been guilty of using the @ symbol in PHP to supress stupid error messages, this does it correctly.
• No PHP extract() or compact(). Actually just no extract function is good enough. That function is pure evil.
• Sqlalchemy A proper you look after the database and ill play with objects database extraction layer thingamy. I don t want to play with databases just let me write code and thats what sqlalchemy does well.
• Testing Turbogears uses nosetests which does functional and unit testing. I do get some annoying artifacts based upon testing but generally its all good.
• Python has help() everywhere and pydoc on the command line. Not sure what variable foo can do? Just type help(foo) or even stick it in your script.
• paster! You can run a development site off the command-line using a simple sqlite database. Don t like it, dump the DB and reinstall it with one command.

It s not one-way though, there have been some challenges. Quite likely these are more my deficiencies than the languages.

• Error messages take up a whole screen and often make it difficult to isolate. It s not quite 10-screens-full ala Eiffel but sometimes the error messages tell you something is going wrong, but not what. Mess up your object versus database table references and you get something rather obscure.
• When is an integer not an integer? The answer is, sometimes. Sometimes they work, sometimes they break. It s probably more to do with what pysnmp is doing with them.
• Documentation some is great, some is downright misleading. It comes down to the writer of the documentation for modules. At least it doesn t have well get round to it like some functions in PHP on their website. Documents that say you can directly iterate dictionaries were another dead-end I wandered down. If you are stuck here, add a .items() at the end of it.
• Deployment. It s not just a matter of taring the files up and sticking them into /var/www/sitename and you re done. There are probably good reasons why, but its just annoying.

Despite the minor problems, python really does give you much much more than PHP and some more than Perl. For me and what I am intending on writing it is worth learning yet another language.

Related articles

• A Python programmer s first impression of CoffeeScript (ssokolow.com)
• Guido van Rossum: 21 Years of Python (perspectives.mvdirona.com)
• Which language next? (sfmlcoder.wordpress.com)
• Python vs. Perl vs. C++ Python is Better (strombergers.com)

forked-daapd v0.17 is out; the main purpose of this release is to add support for libav 0.7. There are also a few bugfixes that made it into this release, but nothing earth-shattering. One More Thing You ll notice an extra tarball and tag in this release, both marked 0.17gcd. With this release, the GCD (Grand Central Dispatch / libdispatch) codebase is making its debut. libdispatch replaces libevent and allows for greater concurrency inside forked-daapd. This makes forked-daapd snappier, which shows when using Remote or a SoundBridge. The filescanner also benefits from this; it now scans several directories concurrently, reducing the time needed for a rescan by making better use of the machine resources. Thanks to Mark Heily s efforts on libdispatch/libkqueue/libpthread_workqueue over the past year, we now have a Linux version of libdispatch that works well enough to support forked-daapd. This codebase will become the primary codebase for forked-daapd once the database performance issue will be fixed. After this point, the current libevent 1.4 codebase will be legacy and won t see any further development (we ll discuss bugfixes in due time). This new codebase comes with new requirements:

• the Clang compiler from the LLVM project is required, due to the use of the C Blocks extension that isn t supported by GCC;
• libdispatch, of course, is required;
• libtre is a new dependency for concurrent regexps;
• glibc 2.13+ is required, or any glibc version containing a fix for glibc BZ#10563.

libdispatch and its dependencies are available in Debian unstable; it s best and easier to use whatever is in unstable, instead of mixing and matching the libdispatch/libkqueue/libpthread_workqueue versions until they all work together. Due to platform support in Clang, the GCD codebase can only be built for and will only run on i386 and amd64, as far as Linux is concerned. Other architectures will have to wait until support for them is added to Clang. FreeBSD is untested and will probably require some modifications to work; recent enough versions of FreeBSD have a native pthread_workqueue implementation and do not need libpthread_workqueue. Packages are available in experimental for Debian users; they ll end up in unstable in a few weeks, before the libevent2 migration happens there. Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ ; GPG signatures made with my Debian key F5D65169.

John Atherton sent me a link to cyberunions discussing open-source and worker co-ops. In general, it s a very good show and worth a listen by anyone from the union movement, in my opinion. It s available in Ogg as well as MP3, which I think is also a plus point. There were a couple of points where I was ranting at the radio, though The first was the choice to try Skype alternatives. Leaving aside the danger of playing the alternatives game, I can understand why they chose that (the change of corporate control to Microsoft has been in the headlines recently and the phones affect them but not their listeners or readers), but open internet phones are a bit fiddly because they try to move quite a lot of data (audio) in real-time and a lot of domestic network devices don t behave correctly with the protocols that they use to do it. I agree with their conclusion that the problem is probably networking, but I wonder why they didn t log in to their routers, look at any debugging output and try to find the setting to change? Or try switching on debugging in the software and see what s wrong, or send it to someone who can help? Instead, it seemed like Linphone and Ekiga got blamed for not doing Skype-style router-busting without asking. They don t mention whether they tried switching on the STUN option mentioned in the Linphone FAQ and I don t see a email on linphone-users asking for support but I don t read every mail on that list. Am I being unfair, expecting them to seek support? Maybe but I m not sure. If you got the software from a store, what would you do? Would you throw it away and tell people that the stupid thing doesn t work? Moreover, would you use your podcast to say it doesn t work because users need to be clever enough for that obscure tool? That s a great way to scare people off unnecessarily. Wouldn t you usually think you might be overlooking some detail and ask for support, from its maker or the place where you got it? I think you d ask. I d probably ask, even though I m available for hire to fix FOSS problems and it s not the proudest feeling when I have a script-reading call-centre worker solve my mistake. Of course, with most from-store software, you ve paid an artificially-inflated box price, but the bigger cost is your time and that s the same for downloaded free and open source software, so why give it less of a chance to work? I also think they had easier options, like switching more of their website over to free and open source software, like getting rid of the rubbish comment system which doesn t work if you don t let it run javascript on your browser. I ve had a bit of discussion with them since the show and it seems my first suggestion may no longer work (typical!), so we ll see how that goes. Back to the show: the next segment did a good job of linking FOSS to the co-operative movement. I really think they should call it Free and Open Source Software and not just open source because I feel the freedom should be important to the labour movement. Also, calling the whole thing open source seems like calling all of the political left Revolutionary Socialism : inaccurate at best. I don t often express this objection these days (I ve bigger targets to work on), but it s important when trying to inform new audiences, else they only see part of our rainbow, a less radical part. Pretty interesting to suggest Orbea bikes as Mondragon s most famous brand. I d pick Eroski or Caja Laboral. What about you? Then it moved on to a pretty good description of UK co-ops, mentioning co-ops that I m a member of (the Co-operative Group) or buy from (John Lewis and Suma). I was surprised that it seemed to stop short and didn t mention any of the co-ops that work with free software, or with any software in fact. Of course, I d love software.coop to be mentioned, but there are other interesting and very different ones like BristolWireless.net and OSAlliance.com. Globally, many of the worker tech co-ops are linked through the tech-coop mailing list. This was the second part where I was shouting at the speakers. After all, Debian is a great voluntary project making a fantastic GNU/Linux distribution, but it is not a worker co-op by a long way and I suspect some of my fellow developers would be horrified to hear that suggested. Why not mention some actual tech worker co-ops? Finally, before the listener feedback, suggestions of tools which union activists could use. There was some minor confusion about exactly which version or name was really FOSS and a surprising claim about openoffice into LibreOffice (I thought Oracle were still screwing that up with Apache s help but I may have missed some news), but it was a good list of door-opening suggestions. I hope some activists will take them up. It seems there s an earlier show about open social networks, which is a topic I ll look at after I finish my series of posts about mailing lists. So, please let me know if you have comments on this show, that show, open social networks or mailing lists which you think I might find interesting, or if there s anything above you d like me to expand upon (or correct often, correct ).

I've been getting strange failures on some architectures for xdotool. xdotool is a library and a command-line tool to allow you to inject events into an existing X11 session. I'm trying to understand (or even to reproduce) these errors so i can fix them. The upstream project ships an extensive test suite; this test suite is failing on three architectures: ia64, armel, and mipsel; it passes fine on the other architectures (the hurd-i386 failure is unrelated, and i know how to fix it). The suite is failing on some "typing" tests -- some symbols "typed" are getting dropped on the failing architectures -- but it is not failing in a repeatable fashion. You can see two attempted armel builds failing with different outputs: The first failure shows [ and occasionally < failing under a us,se keymap (that is, after the test-suite's invocation of setxkbmap -option grp:switch,grp:shifts_toggle us,se):

Running test_typing.rb
Setting up keymap on new server as us
Loaded suite test_typing
Started
...........F..
Finished in 19.554214 seconds.
  1) Failure:
test_us_se_symbol_typing(XdotoolTypingTests)
    [test_typing.rb:58:in  _test_typing'
     test_typing.rb:78:in  test_us_se_symbol_typing']:
<" 12345678990-=~ !@\#$%^&*()_+[] ;:\",./<>?:\",./<>?"> expected but was
<" 12345678990-=~ !@\#$%^&*()_+] ;:\",./>?:\",./<>?">.
14 tests, 14 assertions, 1 failures, 0 errors

The second failure, on the same buildd, a day later, shows no failures under us,se, but several failures under other keymaps:

Running test_typing.rb
Setting up keymap on new server as us
Loaded suite test_typing
Started
..F.F.F.......
Finished in 16.784192 seconds.
  1) Failure:
test_de_symbol_typing(XdotoolTypingTests)
    [test_typing.rb:58:in  _test_typing'
     test_typing.rb:118:in  test_de_symbol_typing']:
<" 12345678990-=~ !@\#$%^&*()_+[] ;:\",./<>?:\",./<>?"> expected but was
<" 12345678990-=~ !@\#$%^&*()_+] ;:\",./<>?:\",./<>?">.
  2) Failure:
test_se_symbol_typing(XdotoolTypingTests)
    [test_typing.rb:58:in  _test_typing'
     test_typing.rb:108:in  test_se_symbol_typing']:
<" 12345678990-=~ !@\#$%^&*()_+[] ;:\",./<>?:\",./<>?"> expected but was
<" 12345678990-=~ !@\#$%^&*()_+[] ;:\",./<>?:\",./<>?">.
  3) Failure:
test_se_us_symbol_typing(XdotoolTypingTests)
    [test_typing.rb:58:in  _test_typing'
     test_typing.rb:88:in  test_se_us_symbol_typing']:
<" 12345678990-=~ !@\#$%^&*()_+[] ;:\",./<>?:\",./<>?"> expected but was
<" 12345678990-=~ !@\#$%^&*()_+ ;:\",./>?:\",./<>?">.
14 tests, 14 assertions, 3 failures, 0 errors

I've tried to reproduce on a cowbuilder instance on my own armel machine; I could not reproduce the problem -- the test suites pass for me. I've asked for help on the various buildd lists, and from upstream; no one resolutions have been proposed yet. I'd be grateful for any suggestions or hints of things i might want to look for. It would be a win if i could just reproduce the errors. Of course, one approach would be to disable the test suite as part of the build process, but it has already helped catch a number of other issues with the upstream source. It would be a shame to lose those benefits. Any thoughts?