Not so long ago i posted a plea for people to donate their old hardware. Following my own advice, i donated one laptop which ended up in good hands, i believe ;) Unfortunately i got some people putting comments in my blog asking for a donation to be sent to places like India and Pakistan. And i say unfortunately because i cannot do that: i dont have any more laptops to donate, and even if i had, probably the cost of sending them would be similar to the price one would pay to purchase the same hardware in the country of destination. So No Can Do. I received a proposal to coordinate some donations, but so far i had not had the time to think about it. Too occupied with other things.

I finally decided to publish the few hacks I am using to run Linux on SUN's LDOM.
They are not pretty but they do the job done for me.
I take 0 responsibility for them so use them at your own risk.
Use this HOWTO only if you are dealing to mess with your system. Do *NOT* use for production systems if you do not understand what you are doing Step 1: read all the possible LDOM documentation that SUN made available for you. Step 2: understand that this is all experimental software. Step 3: repeat 1 and 2 a few times. LDOM server in Solaris allows to export 3 kind of block devices (file, partition/slice, entire disk) to the guest and those are seen as virtual disks.
The guest cannot really feel the difference but it is important to understand the way in which the LDOM software plays with them.
I did *not* test the export for partition/slice and my hacks do not know how to cope with it. You have been warned. The problem: LDOM virtual disk daemon mangles partition tables In an attempt to make sure that the partition tables exported to the guests are sane, the daemon ends up trashing valid ones because the check it forces to the devices is too strict and it behaves differently if the device is a file, a partition/slice or an entire disk.
The ldm bind operation is the dangerous one to perform and that can kill your partition table for good.
How ldm bind operation work: - for entire disks: * check device partition table
* fix device partition table if check fails
* lock device
* export device to guest
* store status of device once it is validated and do not check it ever again
- for files: * check device partition table
* fix device partition table if check fails
* export device to guest
Remember that I did *not* test export of a slice and that my hack does not take it into account. You have been warned twice. Limitations of the workaround First of all I assume that you installed LDOM 1.0 in the standard path (/opt/SUNWldm/) otherwise you might need to change the script manually to match an alternate path.
Each guest is using the same kind of devices. The workaround does not yet how to cope with exports of files and partitions to the same guest.
You know how to use dd and some other basic tools.
If you start playing with bind/unbind operations manually and things break, you keep all pieces.
How to This list of operation sounds extremely complex, but it is not and it is required only at the first install time and possibly to update your partition table backups if you decide to change them. The changes to the ldom init script will take to restore the last known to work partion table on each reboot. NOTE: if you fail to follow these instruction and do not use valid configuration entries or backup partitions, LDOM might not start anymore or even core dump. TAKE EXTREME CARE (and learn how to reboot into factory-defaults at SC/ALOM)

1 - get the few files from here. This html file is there too for your convenience.
2 - ldmd_start is the newly modified init script for LDOM. It should be placed in /opt/SUNWldm/bin and made executable. Note that if there is no configuration file or the configuration file is empty nothing will happen and you will be running the exact same script as the original. ldmd_start.orig and .diff are there for your convenience to check that I did not add a root kit ;).
3 - ldom.linux is an example configuration file that you want to place in /etc.
4 - mkdir /.ldom.linux , we will use this directory to store the backup partition tables.
5 - decide what kind of device you want to export.
Follow steps 6 and 7 if you decided to use a full disk as device and skip to 8 if you are using a file 6 - Use the format command from Solaris and make sure to label and format the disk.
7 - Take a copy of the clean partition table.
Example: dd if=/path/to/device of=/.ldom.linux/$guestname-$(basename /path/to/device).format count=1 bs=512
So let say that your guest is called foo1 and you are exporting /dev/rdsk/c1t1d0s2 you will issue the following command:
dd if=/dev/rdsk/c1t1d0s2 of=/.ldom.linux/foo1-c1t1d0s2.format count=1 bs=512
Continue from here if you are exporting a file 8 - Configure your LDOM guest following the LDOM administration guide.
9 - Netboot and install Linux in the guest.
10 - Before rebooting into the installation, halt the guest and unbind it. The unbind is required to unlock the exported device for read.
11 - Take a copy of the partition table that has been created by Linux.
Example: dd if=/path/to/device of=/.ldom.linux/$guestname-$(basename /path/to/device).backup
Following the above example: dd if=/dev/rdsk/c1t1d0s2 of=/.ldom.linux/foo1-c1t1d0s2.backup count=1 bs=512
or
dd if=/path/to/file of=/.ldom.linux/foo1-file.backup count=1 bs=512
12 - Edit the configuration file in /etc that you installed at step 3.
Each line contains:
guest_name disk1 disk2 ...
NOTE: you do not need to add entries for Solaris guests. Only Linux guests are required! Empty lines are skipped, no comments are allowed in the file.
Follow steps 13 to 19 if you are exporting a disk 13 - Restore Solaris partition table:
Example: dd conv=notrunc if=/.ldom.linux/$guest-$(basename /path/to/device).format of=/path/to/device bs=512 count=1
14 - Bind the guest
15 - Unbind the guest again. HINT: this is the most important step here because LDOM disk server now has validated the partition table and it will not check it again.
16 - Restore Linux partition table:
Example: dd conv=notrunc if=/.ldom.linux/$guest-$(basename /path/to/device).backup of=/path/to/device bs=512 count=1
17 - Bind the guest again.
18 - Start the guest.
19 - Enjoy.
Follow steps 20 to 23 if you are using a file 20 - Bind the guest
21 - Restore Linux partition table:
Example: dd conv=notrunc if=/.ldom.linux/$guest-$(basename /path/to/file).backup of=/path/to/file bs=512 count=1
22 - Start the guest.
23 - Enjoy.
This said, the script will allow you to reboot the controller without having to worry about Linux guests and it should ensure that LDOM will always start properly. LDOM stores the status of a guests in the Machine Description and we will use this status to restore the partition tables too. If the guest is in either bind or active status, we will take proper actions to stop/unbind/restore/bind/start. If the guest is in unbind state, we will do nothing because we do not know why is in that state. The script also is not error prone. Failing to stop or unbind the domain, to perform restore partition table operation, might cause LDOM to abort and can require manual fixing but the above information should be enough for any experienced user to recover. Have fun

One problem that I have had in configuring Heartbeat clusters is in performing a STONITH that originates outside the Heartbeat system. STONITH was designed for the Heartbeat system to know when a node is not operating correctly (this can either be determined by the node itself or by other nodes in the network) and then force a hardware reset so that the non-functional node will not interfere with another node that is designated to take over the service. However sometimes code that is called by Heartbeat will have more information about the state of the system than Heartbeat can access. For example if I have a service that accesses a filesystem on an external RAID then it’s common for the RAID to track who is accessing it. In some situations the RAID hardware has the ability to “fence” the access (so that when machine B mounts the filesystem machine A can no longer access it). In other situations the RAID may only be capable of informing the system that another machine is registered as the owner of the device. To solve this problem a machine that is to mount such a device must either prohibit the previous owner from accessing the device (which may be impossible or unreasonably difficult) or reset the previous owner. Until recently I had been doing this by writing some code to extract the STONITH configuration from the CIB and call the stonith utility. The problem with this is that there is no requirement that every node be capable of performing a STONITH on every other node, and that even if every node is are designed to be capable of rebooting every other node a partial failure condition may restrict the set of nodes that are capable of performing a STONITH on the target. Currently the recommended way of doing this is via the test program. Below is an example of the command used to reset the node node-1 with a timeout of 20000ms and the result of it being successfully completed. I have suggested that the Heartbeat developers make an official interface for doing this (rather than a test of the API) and I believe that this is being considered. In the mean time the following is the only way of doing it: # /usr/lib/heartbeat/stonithdtest/apitest 1 node-1 20000 0
optype=1, node_name=node-1, result=0, node_list=node-0

Share This

(Or the unfortunate story of why I am not yet participating in the Debian games or perl teams.) I'm a big fan of both the Debian games and perl teams. The games team keeps finding new and interesting games and flowing them into Debian. The perl team is responsible for a rather amazing quantity of perl modules. Since I maintain both some games and some perl modules, I've looked at participating in both teams from time to time. I think it's odd that the reason I haven't been able to do so in both cases comes down to something as basic as how they use their subversion repositories.

---

In the case of the perl team, they have a fairly standard use of subversion, as explained in their guide. The problem is repository layout. They chose to use the "package/ trunk,tags,branches " layout rather than the " trunk,tags,branches /package" layout. This means that it's not easy to check out the source for every package in the repo all at once. Having all the sources checked out is a prerequisite for many kinds of work that span all packages. In my work on the d-i packages, which are in a repository using the latter structure, I often grep all the packages for a problem, and fix the problem in every package. To manage this with the debian-perl repo, I'd either have to check out the whole thing, with who knows how many gigabytes of tags, or I'd have to construct and maintain an svn:externals list that checked out the trunk of every package. (Open offer to the perl team: I'd be happy to reorganise your repo for you.)

---

The games team uses the " trunk,tags,branches /package" repo layout, which is nice. But they have a policy of not keeping all upstream files in subversion at all. They prefer to only keep files that are patched in subversion, to save space. That looks on the surface to be a nice space optimisation, especially for space-hungry game data, but from what I tell, what it saves in space, it loses in complexity and maintainer time. And my time is much more valuable than alioth's disk space. It's also very debatable how much space this saves at all for games. I keep uqm and all its data files in one subversion repo. I have 4 years of history of all of upstream's changes to the files in there, and it takes 165 mb of data. Which sounds like a lot, but the files in trunk alone take 170 mb to check out (not counting .svn directories)! And the games team is also storing upstream tarballs in a directory on alioth. One tarball per release, often multiple releases per game, 2.0 gb of tarballs in all. Would it really use much more space in svn? Even if it used 10x the space somehow, is 20 gb even worth worrying about? I think not. (Open offer to the games team: I'd happily do the grunt work of checking in upstream sources for all the games in your repo.) (Update: I realized after posting this that while the space gains on alioth might be illusory, there is certianly a space gain for people checking the whole repo out. What's really needed is a way to keep all the upstream sources in svn, but check them out only when wanted, and check out all the debian directories for cross-game work. Hmm..)

---

I also checked out one other big package repo, collab-maint. The policy for collab-maint's repo uses the same structure as the perl team's repo. However, I doubt I'd ever want to check out all of collab-maint or do group maintenance across it, so that's not a big deal. The documentation also recommends not storing upstream sources in svn, although I've found a lot of packages in the repo that do. I could probably dump in ~100 mb of subversion history for 50-odd packages without anyone caring ... right?

---

It's interesting that all three of these big subversion repos have their own custom policies for how to use their repo. Having different policies for each repo results in more complexity and a higher learning curve. I wager that 99% of all the packages in these repos are pretty stock in their packaging and can be pretty easily understood and worked with by most Debian maintainers. We've have de-facto standards and policies that they all use. Until you get down to the repos, which, despite using subversion each seem to need to go their own way for some reason.

For what can be read on Planet SoC, computer problems seems to be the norm between SoC students... I've got my part of this cake today with a power cut and some strange /dev permissions when my PC got back...

Fortunately it was somewhat easy to solve (I believe it was the udev upgrade which did the trick), so I've been able to release btsutils 0.2. Please take a look on the announcement for more details.

Still on the SoC side, thanks to the great work from the Debian ftpmaster team and Lo c Minier, the packages python-btsutils and python-bzutils have made their way through the NEW queue and are on the Debian official archive.

And finally, on Debian side, Lo c Minier also uploaded libgnomedb3 3.0.0-2, a bug fix release for libgnomedb closing two bugs. Thanks, Lo c.

Mark Shuttleworth asks if people are interested in a high-end free-software laptop (it seems that Linspire is leading in the low-end free-software laptop stakes). I am interested in such things. My last couple of laptops have been Thinkpad T series. They are reasonably light (not really heavy), are reasonably fast, have full-size keyboards and reasonable size screens (currently got a 1400×1050 screen on a 3yo laptop). Unfortunately for Mark I’m planning on making my current Thinkpad last for another three years. The idea makes a lot of sense because laptops are not re-purposed very often. It’s quite common for a desktop machine or a server to be re-installed several times over it’s life - and often having significant hardware changes during the process. Laptops are extremely difficult for hardware upgrades to the degree that by the time people desire an upgrade it often makes sense to buy a new one. So having a BIOS that only supports Linux and prevents the machine from ever being used to run a lesser OS is not likely to reduce the utility of the machine. The benefit of better Linux integration is that the greater degree of hardware control would decrease the power use and extend battery life. Maybe in three years time I’ll buy a LinuxBIOS machine second-hand from a Ubuntu user.

Share This

It has been almost two weeks of silence from be because I was enjoying vacation on Sicily. Now I've read almost all mails and getting back to work ;-). There were opened lots of bugs in Gammu bug tracker, so it is first thing I need to look at. Fortunately there were no new bugs on my Debian packages, so at least this part will be easy (and only Sonata package needed upgrade, which has been done right now).

I seriously lack originality on that title above. Nevertheless, that gives some clues on what I’m up to: I rode my first airplane flight ever to Cebu with Jerome last Sunday to meet up with the rest of the pioneering team of Winston’s new startup, Morph Labs. Since it was a Sunday, there wasn’t much talk about work, but we did get to play with some of the new stuff we will be using, like Macbooks, iPhones, Airports, and a Mac Mini. Hint-hint indeed Thus began the first day, on a Monday. We got up to speed (from a 3-to-7 AM hibernate,) configuring the rest of the Macbooks, getting the HSDPA modems to work, laying out an ad-hoc WiFi net for the moment, and starting to plan out the directions the company will take. So far, so good. More to come, on Second Life Morphs. (I seriously need to express myself better.)

There are a gazillion HOWTOs out there for flashing a BIOS image without having to resort to ugly "boot DOS from floppy" or "run Windows *.exe file from BIOS vendor" and other ugly stuff. Unfortunately, the proposed solutions are equally ugly (e.g. creating custom CD-ROMs which contain the "floppy" with DOS/Windows flash tools). Folks, this is so much simpler than you think: The flashrom tool (GPL'd, written for LinuxBIOS purposes, but works perfectly fine with proprietary BIOSes, too) will easily do what you want, on a running Linux system. No floppy crap, no CD-ROM crap, no DOS/Windows crap, no rebooting crap. Install it:

$ apt-get install flashrom

Detect whether flashrom knows about your chipset/mainboard/BIOS chip:

$ flashrom

Read the BIOS image into a file:

$ flashrom -r backup.bin

Write a BIOS image (proprietary or LinuxBIOS) on the ROM chip:

$ flashrom -wv newbios.bin

WARNING: This will overwrite your current BIOS! Make sure you know what you're doing! For the Debian-challenged, flashrom is available in source form too, of course:

$ svn co svn://linuxbios.org/repos/trunk/util/flashrom
$ cd flashrom
$ make

The list of supported chipsets, mainboards, and ROM chips is limited of course, but it's constantly expanding. Contact us on the LinuxBIOS mailing list if you want other hardware supported (or even better: if you have patches!). In many cases adding support for new hardware is pretty easy...

I now have a working system for creating minimal installations of Fedora Core (6), and CentOS (4.5 & 5). I've updated xen-tools to make use of this, via --install-method=rinse - and now I can create new Xen guests :) There were several false starts: <dl>

Have a static list of .rpm names + versions

This is precisely what rpmstrap does, and it is a broken approach. I'm happy to have a static list of packages, but restricting the packages to specific versions is fragile and broken.

Use Yum

yum is used by many RPM based distributions, but unfortunately each distro uses slightly different versions to the one in Debian - so whilst I could run:

mkdir /tmp/fc6
yum -c /tmp/yum.conf --installroot /tmp/fc6 install yum

This results in a system which cannot be updated:

chroot /tmp/fc6 /usr/bin/yum update
[errors about yum database formats]

I suspect this is because the versions of the Berkeley database are different .. but I'm not sure.

Parse Dependencies

This would be a neat approach, however it is difficult. Too difficult. Abandoned after several fruitless hours.

Cheating

I cheated in the end.

• Download a list of packages which are known to be required to produce a working yum, and unpack them.
• Copy those downloaded packages into the yum cache tree.
• Run yum install yum - to make the local Yum + RPM database match what we fetched manually
  • Unfortunately "rpm rebuilddb" couldn't do that for us.

Worst case? I download each package twice. Once for the initial download, and once again when the Yum process runs. I can live with that.

</dl> I'll talk about something else next time. Promise.

A few people commented on my last post asking for details about how I configured exim and dovecot to have the fake scott+canonical account and separate Maildir tree. exim4 router configuration The first key part of the configuration is to configure exim4 to split the local part into a user name and a suffix. This allows a local part such as “scott+canonical” to be split into the user name “scott” and the suffix “+canonical”. This is configured by adding the following two options to the appropriate routers in your exim4.conf:

local_part_suffix = +*
local_part_suffix_optional

The first option defines the suffix; normal practice appears to be to use both “+” and “-“, but since I’d previously used “-” with qmail I opted to use “+” only so that I didn’t confuse myself during the transition. The second option allows the suffix to be optional, so that mail to “scott” is still delivered normally. Now mail to “scott+canonical” will be delivered to the “scott” user. Forwarding configuration The next task is to ensure that mail is actually forwarded to this address; for me this was a configuration performed by the Canonical sysadmins to ensure that my work e-mail is actually delivered to scott+canonical on my own mail server. Filtering configuration Since both personal and work e-mail are now both being delivered to the same user account on my home mail server, I need to filter the mail into separate folders. This can be done by checking the $localpartsuffix variable in Exim filter .forward files, e.g.:

# Exim filter
if $local_part_suffix is "+canonical"
then
    save Maildir/Canonical/
endif

Now incoming work e-mail is filtered into a different mail folder, while personal mail is delivered into the primary one. I’ve used similar filter instructions for mailing lists, mailman messages, Launchpad mails, etc. to filter into appropriate folders. Where the mail is personal, it is filtered into (e.g.) Maildir/.Lists/upstart-devel/ Where the mail is for work, I add /Canonical/ to the path, (e.g.) Maildir/Canonical/.Lists/linux-hotplug-devel/ This means that the Maildir/ directory is my personal INBOX, with sub-folders immediately under that and beginning with a period; and the Maildir/Canonical/ directory is my work INBOX, with sub-folders immediately under that and beginning with a period. This defines two trees in a manner compatible with dovecot. There’s no particular reason that Maildir/Canonical/ has to be under Maildir/, it could have been Maildir-Canonical/ and this would still work. I simply wanted them in one place to ease backups. Dovecot configuration Now we need to configure dovecot to permit login by a fake (“virtual”) user, with a different Maildir tree, so that I can configure them as two separate accounts. The first set of changes is to dovecot.conf to add an additional authentication source. Modify the “auth default” block to add a new “passwd-file” passdb in addition to the “pam” passdb (or whatever your system is using).

passdb passwd-file  
  args = /etc/dovecot/passwd
 

This lets us authenticate virtual users, but we also want to set their attributes, so we can use the same file as a userdb in the same “auth” block.

userdb passwd-file  
  args = /etc/dovecot/passwd
 

Dovecot will now check both PAM and this file for user information. We now simply need to add a line to this file to specify the virtual user and set up the alternate Maildir tree.

scott+canonical:PASSWORD:1000:1000::/home/scott::userdb_mail=maildir:/home/scott/Maildir/Canonical

The format is that of an ordinary passwd file; the first two parts give the passdb authentication credentials and the rest give the userdb information. I’ve set this user to have the same uid, gid and home directory as my real “scott” user. The final part changes the mail environment for this virtual user, instead rooting it at Maildir/Canonical/ Client configuration The mail client will need two accounts adding; one for “scott” and the other for “scott+canonical”. It will see two separate folder trees for each account. An unexpected bonus is that the reply account is now automatically set for me, since I’m replying from the specific account rather than from a single general one.

In a comment on my post about memory errors Chris Samuel referred me to an interesting post on the Beowulf mailing list about memory errors. In that list posting Joe Landman says “it is pretty easy to deduce which chip is problematic (assuming it is ram) based upon the address” and then describes how to use Machine Check Exception (MCE) data from an error detected/corrected by the ECC system. Damn the vendors of motherboards for switching to 8-bit RAM just when it was about to be useful to have 9-bit RAM! 286 class machines had 9 bits of RAM per byte with one bit used for parity. Parity errors were extremely rare, largely due to the fact that memory errors could affect more than one bit at a time and therefore would often give a correct parity - if multiple bit errors were totally random then parity might be expected to pass 50% of the time! The Pentium was the first commonly used CPU to operate with a 64bit memory bus. If it had 9 bits per byte it would have had 72 bit wide memory buses - a Hamming Code could use this to detect and correct single-bit errors, detect all double-bit errors, and detect some errors involving more bits. This would mean that some errors would be recoverable and would display the location of the memory problem instead of being fatal and giving no information. Now it’s become a standard feature in servers to have ECC memory (at significantly greater cost) and most desktop machines don’t have ECC support - I wonder whether this is aimed at price-gouging people who need reliable servers (they can’t use cheap RAM from desktop machines). Unfortunately due to issues of electricity use, noise, and price I have to run all the servers that are most important to me on desktop PC hardware. Is anyone selling ECC RAM in desktop systems? I am particularly interested in machines that are a couple of years old so I can get them cheap at auction…

Share This

On Thursday I bought a $99 (discounted from $199) MP4 player from Aldi (a German supermarket chain that has recently opened up here). The player is a Tevion model M6. By a long way it’s the cheapest and nastiest piece of consumer electronics that I have ever owned. It has very flimsy construction (feels like it will crumple in my pocket - not like a solid iRiver) and poor design all around. The viewing angle of the LCD screen is very small, so if I hold it close to my face then the viewing angle will be wrong for at least one eye. There are two power switches, an electronic one on the top (which is also sometimes used as an escape key for menus that don’t recognise the key labelled as ESC) and a slide switch at the side. When I use the electronic switch to turn the power off the back-light will usually flicker - I guess that they wanted to have an electronic switch and then put a mechanical switch in the design when they couldn’t get it working. The menus are strange, they have a game menu that only has Tetris - why not have a Tetris menu instead? The FM radio function doesn’t seem to work and the option to select a European frequency range is lost when the power is cut, along with all saved station frequencies. I never got around to testing the voice-recording function (one of the reasons for purchasing the device) as it failed in too many other ways. The device has an AVI of Barbie Girl by Aqua and also a MP3 with a text file that has the lyrics for Karaoke, this is probably the only good feature of the device. Unfortunately it appears to have some sort of DRM as it gave a padlock icon and stopped working after I played it a few times (fortunately it’s unable to store settings so a power cycle solved that problem). When I connected it to my PC via USB it showed two devices, /dev/sda and /dev/sdb. /dev/sdb gave an IO error (apparently due to not having an SD memory card installed) and and /dev/sda was not in any format recognised by file -s /dev/sda. I’m going to have to return this, even $99 is too much for a device of such quality. Maybe I’ll buy an iRiver to do this if they sell one without DRM.

Share This

Films One of the very few good things about my partner being away to the US for the next six weeks is that I can catch up on watching films which she doesn't like! Every weekend we go out and spend 10-15 pounds on 2-3 second-hand DVDs from local stores. That gives us something new to watch every week, and is a fun way to spend a day shopping together, in amongst doing other things.This has meant I've watched an awful lot of (awful!) films I'd never have previously considered, but I've also found a few gems I'm glad I got to see. The down-side is that we don't often watch films we've already got, since we don't have the time to do so. Anyway tonight is Aliens & Battle Royale.. Source Scanning A little more progress on that this week, another local root attack, a symlink attack against the Amaya browser and a potential root attack against evms. The EVMS bug is frustrating because the code is contained in a plugin which appears to never get built. I've tried searching the web for more details, but unfortunately I couldn't find anything. So in the interest of demonstration here's how you should not write code which runs as root:

        if(system("lsrsrc -axd IBM.PeerNode > /tmp/rsct_node_info") == -1) 
                LOG_ERROR("ERROR:get_nodes_info() fails ");
         
        

Why is this bad? Because it users a static filename /tmp/rsct_node_info - and that could be a symlink. Consider what happens if a local user were to run:

skx@vain:~$ ln -s /etc/passwd /tmp/rsct_node_info

The EVMS plugin would happily trash the /etc/passwd file, rendering the system broken... If this works on other distros that I couldn't spot don't tell anybody; it'll be our little secret ;) X.org Today I rebooted, to make sure that purging EVMS hadn't screwed with my initrd - and found errors relating to failure to load "keyboard" driver(s). Turns out you need to change:

Section "InputDevice"
        Identifier      "Generic Keyboard"
        Driver          "keyboard"
        Option          "CoreKeyboard"

TO:

Section "InputDevice"
        Identifier      "Generic Keyboard"
        Driver          "kbd"
        Option          "CoreKeyboard"

That took a fair bit of head-scratching.

Like most geeks, I run my own mail server. The burden of administering it is much less than the increased flexibility in filtering incoming mail, let alone dealing with SPAM. My mail server configuration has remained pretty static the entire time, and controversially, I’ve always used qmail. The reason for this dates back to my first sysadmin job in the mid-to-late nineties, and the decision in those days did tend to be sendmail or qmail; with the security conscious choosing the latter. qmail’s delivery system is a little odd, everything on the left hand side up to the first “-” is considered a user, and everything after can be used for filtering. The default local delivery component takes this into account, so e-mail to scott-foo can be filtered by the /home/scott/.qmail-foo file. This gives a pretty natural way to deal with mailing lists; you subscribe with a unique address for that list, and all the mail goes into the right folder automatically. This has served me reasonably well over the years, with heavy patching across the daemon to add features such as LDAP integration, and SPAM filtering that I wanted. Unfortunately it’s been getting to burdensome to maintain. Since it’s not true open source software, and is effectively abandoned upstream, it’s not as up to date as I’d like. SPAM filtering tends to take place in the local delivery loop, rather than at SMTP time; and due to the strange delivery system, it’s unreasonably hard to perform any kind of sender verification or greylisting. Since every special address is filtered differently, it’s quite hard to add common filtering unless it’s pre-planned and you use addresses with a common prefix. The clincher has been dealing with super-sites like Launchpad which send huge amounts of different e-mails to a single address, and no facility to separate that from your published contact address. I needed a better mail server. So I’ve now moved to exim. I was surprised by how quickly I was able to pick it up, I did the migration in two day outages. The first to simply migrate delivery and stash the mail in one big folder, and the second to customise the delivery and filtering to my liking. I’ve also re-subscribed to mailing lists with single addresses again, so now I have a single filter rule which happily can filter Launchpad mails around as well. Happily I’ve been able to make a change I’ve wanted to for a while, home and work e-mail is separated into different Maildir/ trees; and mailing list subscriptions made with the most appropriate address. The magic of dovecot lets me create a fake scott+canonical user that uses the alternate Maildir tree, while still retaining my user permissions, etc. Overall I’m pleased with the new setup, and how the migration went. SpamAssassin needs some tuning as a little SPAM is still getting through, but otherwise it seems to be working well.

The Debian Description Translation Project is a very old project which had many adventures in the past. Indeed, working on packages descriptions translations has been my very first work in Debian l10n back in....long ago. Unfortunately, because of the servers compromise and also because of the lack of support for translated descriptions in the archive and package tools such as APT, the project slowly lost interest from translators. The revival may be dated back in 2005 when Michael Bramer revived the project and even more in 2006 when experimental support was added in APT as well as some support in the archive. While the framework is still quite crude (no offense intended for grisu who did a great job maintaining it) and not very opened to collaborative work and peer review, a few teams did a great job translating the zillions of package descriptions. And we still expect to use Pootle soon. So, as we now have a big enough handicap, it's time for the French team to beat them all in the friendly competitions we all enjoy..:-). For that reason, I began to translate dozens of packages descriptions last week and a few other French translators joined me. I somewhat expect us to slowly put all of our (wo)manpower in this and, therefore, translate as many packages descriptions as possible. The count begins with 1896 active translations, French being ranked 5th after German, Japanese, Czech and Brazilian Portuguese, even with Korean. So, the next target is the Brazilian team. Watch out your back, fellows..:-)

For a long time I've wanted to know how to embed binary blobs into executables. This would be most useful for files like Glade and and UI Manager definitions, which are required for a given program to work at all but either cannot be embedded as a string literal (Glade) or can be but is annoying (UI Manager). I finally asked the Interweb, and Daniel Jacobowitz replied with some pointers. It turns out that doing this is remarkable simple. First, a caveat. This probably requires GNU ld, which may or may not be a deal breaker for many people. First, create a data file. Let's call it foo.txt, and put some text in it.

Hello, World!

Using ld this can be read in as a plain binary blob, and then written as a standard relocatable ELF object.

ld -r -b binary -o foo.o foo.txt

Now we have a standard ELF object with the data and some useful symbols defined. objdump will show you the contents.

$ objdump -x foo.o 
foo.o:     file format elf32-i386
Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .data         0000000d  00000000  00000000  00000034  2**0
                  CONTENTS, ALLOC, LOAD, DATA
SYMBOL TABLE:
00000000 l    d  .data  00000000 .data
0000000d g       .data  00000000 _binary_foo_txt_end
0000000d g       *ABS*  00000000 _binary_foo_txt_size
00000000 g       .data  00000000 _binary_foo_txt_start

Here we see 13 bytes of data, and a symbol which contains the address of the data. This is all we need to access it from a C program.

#include <stdio.h>
extern char _binary_foo_txt_start[];
int main (void)  
  puts (_binary_foo_txt_start);
  return 0;
 

Now if we compile this and link it against the generated object, we'll have a binary.

$ gcc -o test test.c foo.o
$ ./test
Hello, World!

Hooray! One small problem which alert people should have noticed: the string itself is in the .data section, which is read/write. For my use, I want it to be read-only data in the .rodata section so that it isn't copied for every instance of the application. As far as I know, this isn't possible with ld but objcopy will let us rename sections on the fly.

$ objcopy --rename-section .data=.rodata,alloc,load,readonly,data,contents foo.o foo.o
$ objdump  -h foo.o
...
  0 .rodata       0000000d  00000000  00000000  00000034  2**0

Excellent, problem solved. If you want to download this sample, I have a tarball. Many thanks to Daniel Jacobowitz for pointing out how to achieve this. Update: note that any data embedded in the binary like this won't be terminated with a NULL. This is obvious in hindsight, but due to luck my example still worked. There might be a way of asking objcopy to append a 0 to the end of the data, but if not always remember to use the start and end pointers or size instead of just the start, or append a NULL yourself before converting to an ELF. NP: (), Sigur Rós

Status Page for OCaml Transitions Today my spare time (not necessarily driven by me) produced the OCaml package status page. Even if a lot of improvements have already been requested by other fellow caml-ers (i.e. more work to do), I'm kinda happy about the result ... I've even enjoyed using Genshi more than in the past. Unfortunately the page reminds me that there's still a lot TODO before starting to upload OCaml 3.10.0 packages to unstable. Damned status pages!

At the risk of seeming a self-aggrandizing, I wanted to point folks to a nice biographical profile that Linux.com is running about me upon my election my to the FSF board. I'm pretty honored, and excited, by the whole thing. The article talks a little bit about my road to free software and the FSF board in particular and about some of my ideas about the foundation and its work. There are three little footnotes I thought I would add to what I think is a great article:

• The phrase "rebel with rather too many causes" was a phrase originally directed at protest.net -- a event calendaring system for activists that I was briefly involved in over a summer during college. I like the phrase and use it frequently but I didn't want to take credit for it. Google indicates that it originates in NTK #53 I'm not at all surprised.

• My parents worked as doctors in Kenya, Papa New Guinea, and elsewhere before they had children. This probably doesn't matter to anyone else but they worked with an organization that was like MSF in that it was a humanitarian organization that sent physicians around the world but it was not actually MSF as the article states. I don't think MSF had grown beyond French doctors when my parents were practicing overseas.

• When LWN pointed to the article, and in the original was well, there a focus on some comments I make about non-profit organizations. Since in a context of talk about my political work I just want to clarify my comments in a little more depth here. I think that one problem that has stemmed form Open Source's emphasis on businesses and efficiency is that free/open source software people end up making arguments in business terms: you should use application X because it is more efficient and faster. For many of the folks who have built this whole movement though, and for most in the free software camp, it's about freedom, not efficiency. By targeting businesses, we encounter a skeptical audience. More importantly though, we end up making arguments that, while true, are not the ones that motivate us. I think that low-hanging fruit for free software activists might include groups that already support free software ideas of sharing and user empowerment and that are looking for ways to use free software already. Groups we don't need to be afraid about saying "freedom" around. Not so coincidentally, these are sometimes organizations that I have a lot in common with politically. But that's far from always the case. There's a big group of philosophically aligned organizations in the NGO/non-profit community and the problems keeping them away are often technical. This is good news, of course, since solving technical problems is the free software movement's core competency. This is something I've been thinking about for a long time: Debian-NP was one project I helped start that tried to address this issue. Now, many people involved in the FSF, including myself, have political convictions that go beyond software. I do not want these convictions, and my statements about philosophically aligned organization, to be interpreted as call for a political shift in the organization in mainstream political terms. I deeply respect the way that RMS has kept his political opinions separate from the Foundation's. Biella and I have ever written about the importance of this political demarcation to free software's success. It's certainly not something I would want to change.

Last night NBC11 was running a story and an email poll about the impending new legislation requiring that all cats and dogs over 6 months of age be spayed or neutered. Anyway, the legislation seems (surprisingly, in my opinion) divisive within the community. Since Sarah has done a lot of volunteer work at the Palo Alto Animal Services animal shelter in the past, as well as doing a bit of feral cat trap/neuter/release work, she's strongly in favour of the legislation as it'll help reduce the number of unwanted kittens and puppies that get dumped at shelters (or just dumped) every year. Unfortunately, plenty of pet owners seem to feel some need to not have their pet spayed or neutered, even if they have no intention of using the animal to breed, and these people have been kicking up a fair bit of fuss. So, as I was saying, NBC11 ran an email opinion poll, and Sarah wrote in a fairly informed piece about why she felt the way she did, and got an email back this morning for a reporter wanting to interview her, and that happened today. It's funny, Sarah was saying on the way home from work, that in hindsight one of her answers was pretty lame, and sure enough, the one sound bite they decided to use was that one... Oh well, she's had her 30 seconds of fame.