I’ve been installing free software for library cataloguing onto Apples recently. It’s not been great (GNU/Linux is still easier and faster, in my opinion) but it has worked much better than older Macs. As well as making servers that are finally getting easier, Apple lead the way in locked-down portable players, with their iPod and iPhone devices and iTunes service. Quite rightly, free software users campaign against Apple’s use of DRM/TPM (Technical Protection Measures). One of the rallying points has been Defective by Design (DbD), an FSF initiative since 2006. Long-time readers may remember that I took part in a DbD protest at an Apple Store in September 2006:-

“I noticed the huge till queue, which was attractive for three reasons: it had no security (I guess shoplifters rarely queue for the till), most of them are standing around waiting for something to read and they are almost certainly Apple customers, so who better to inform? After I rejoined the back of the queue a few times, a security man was watching it, so I moved off and chatted to people waiting to do hands-on tests on the upper floor for a bit [...] then I moved back to leaflet the till queue again. [...] Then, I got a bit bored and started putting the leaflets into Apple’s product leaflet dispensers (in front of Apple’s - no leaflets were removed), which soon got me rumbled.”

To me, that was fine: I dislike that type of store, which tries to look like public space, but is actually privatised and controlled. My resource use was minimal (one security man escorting me out and some Saturday kid probably “cleaned” the leaflet racks) but I passed information to many Apple customers, who are one of only two groups Apple must keep happy, ultimately. Popey is a bit upset by the latest protest idea, which he describes as:-

“urging people to book slots at the Apple in-store “Genius Bar” to ask probing questions which they already know the answer to about their company policy on DRM and Free software. The ‘Genius Bar’ is an official Apple support avenue for their customers, and is a service provided inside many of their stores.”

It’s also been criticised by neuro and Pete along similar lines. I agree. It’s wasting both side’s resources. Apple don’t really need to keep their staff happy (they can hire new staff) and it offends Apple customers and shareholders. It’s not even efficient: each unit of Apple employee time wasted costs a unit of DbyD worker time and who’s got the most resources there? What are FSF thinking? It’s stupid, even without the bad press it’s had. So, this brings me in a roundabout way to answering

“I am aware of the Linux Foundation and the FSF and others, but how does SPI differentiate itself and its goals from those organizations?”

from why people don’t join SPI. In my opinion, a key difference of Software in the Public Interest is that it is managed by free software developers in a more-or-less democratic fashion. Even though I’ve failed in two elections, I still think it’s better than the alternatives. No project as unpopular as wasting Apple Store helpdesk time would ever get backed by SPI, would it?

I have gotten quite some response to my blog post about the hardware test proposal thingy from Bdale. It would seem I haven't been entirely clear Someone referred me to a page by Kenshi Muto that parses 'lspci' output into a hardware compatibility list. This doesn't help. It ignores x.org stuff (which is important, too) and the page itself clearly says that it cannot guarantee whether a piece of hardware will actually work. For clarity: when I said 'Apparently all the other vendors have it, too', I really meant 'Apparently all the other GNU/Linux vendors have it, too'. People have suggested some cooperation between kernel/xorg upstream, other vendors, and perhaps something like freedesktop.org; and while that may be a good idea in and of itself, in the end Debian will have to provide something which it will call 'official' and which will tell a vendor whether or not Debian is supported on their hardware. This thing may not be fully automated or polished; it may need interactivity; but it must be something which will give a result that a product manager may want to lose some sleep over if it's not good enough. Holger also suggested we try implementing this with Debian Live, rather than debian-installer. This may be a good suggestion; a debian-live image will have a full Debian system available rather than the somewhat limited d-i environment, so writing a test should be easier. Also, I don't even want to think how testing X.org drivers would work out from within d-i. So what's left is a way to figure out how to do such a live system. I'll be posting some suggestion to the debian-devel mailinglist 'soon'.

Some interesting two days have just passed. I took the airplane from Mar del Plata to Buenos Aires on the 17th, slept one night in a hostel on a mattress that felt like stone, and went to the 'Circulo del Officieles de la Mar' (or some such, I don't know spanish) where I had to give a talk. The oral feedback I've gotten so far indicates that people did appreciate it, which is good. I had originally planned to do some sightseeing in Buenos Aires after my talk, but me being tired and not immediately finding anyone else to go with me changed those plans. Instead, I just stayed at the C.O.M., where we played some Mao. Amongst other things. About an hour before I left the place, Holger Levsen gave me a tripod, with the request to take it with me to Belgium, and keep it until FOSDEM. As that would also give me the use of an tripod, and being the photography hobbyist that I am, I was too happy to oblige. There was just one problem: the suitcase that contained my luggage already had a bulge from here to tokio, so adding it was a no-no. No problem, I thought. I'll just keep it in my carry-on luggage. There's some space left there, and I certainly didn't have 5kg of carry-on luggage yet. Problem solved. The 'security' in EZE is a joke; the security officer doesn't even look at the screen while X-raying my luggage. Well done. So I fly from EZE to MAD on the first, 11-and-a-half-hour leg of my flight. Madrid Barajas is a crazy airport, with its own private metro. Somewhere between gates R-something (where IB6844 from EZE arrived) and J40 (where the flight to BRU will depart from), we need to go through a security checkpoint. Strange, since I hadn't had to do that on my way in from BRU to EZE; but of course you have no choice, so I comply. As I've gotten used to by now, what with the mess of cabling in my laptop bag and the other metal objects there, I get a baggage check. The security officer who goes through my very messy bag asks me to open the bag containing the tripod, so he can see it. He then promptly decides that it is a "sharp" object, and that it is "not acceptible". Here the fun begins. Of course, trying to argue with a security guard isn't the best idea if you want to be allowed on any flight, ever, so I don't even try that. I ask him 'can I check that in, instead?' and he tells me how to go back, to the exit, and to the departures hall where I can have that little tripod get checked in. So I go there. The lady at the check-in desk tells me that she cannot check the tripod in, because the flight has already been closed and I'm too late. I tell her I'm from a connecting flight, and that I can't help that. She says she knows. So I ask her what my other options are. She tells me I could try to get on the next flight to Brussels, which is at 20:00 (rather than the 16:20 flight that I was booked on). I consider that an acceptable compromise, so ask her how to get that done, and she refers me to the ticket sales desk, some 10 meters to my left. Unfortunately, the 20:00 flight turns out to be completely booked, and there are no other flights on the same day. The best alternative he can give me is a flight to Amsterdam at 19:20, which would cost 'a little more'. I ask him 'how much more'. He starts looking up things in his computer, and after a few minutes tells me that he cannot refund my ticket for the 16:20 flight and that a flight to amsterdam would cost me 429. At this point I get somewhat angry, and tell him that this is not acceptable. He refers me to the customer service desk right across. The lady at the customer service desk is friendly, but firm: she cannot help me. Either I leave the tripod behind, or I pay whatever the ticket service asks me—she takes my word that it is 429, because she cannot check it, and cannot change it, either. I get more agitated, and ask to see the supervisor. She points me to a lady a little across who can 'call' the supervisor. At this point it is 15:55 As I reach that lady, apparently someone else had just asked for the same thing, and starts arguing with a guy in black uniform, who appears to be the supervisor, about the fact that he's got more than 30kg of luggage which he isn't allowed to take along with him. Dude, not a single airline will let you do that, no matter how much you argue. I let them argue for about 10 minutes, all the while nervously looking at my watch, but then I interrupt with 'please, I don't have much time left before my flight leaves'. That gets his attention. In our two-minute conversation, the guy in black backs up everything the other three people have told me, adding that '429 is reasonable; your luggage is already on the flight, and it will cost Iberia a lot of money to delay the flight and get it off'. I curse. He also seems surprised at the notion that I'm not allowed to take a tripod in my carry-on luggage. Hmm. Figuring I have nothing to loose, I curse one more time, and make a run for it. When I reach the security checkpoint, I ask to be allowed to skip the queue in front of it, on the grounds that I have only 15 minutes left before my flight leaves. Mind you, this is not the same security checkpoint as the one where I was rejected about an hour earlier; this is the security checkpoint between the departures hall and the secure area, whereas the other security checkpoint was one between two different terminals. I am granted my request, and quickly throw everything in the X-ray boxes. They need to go through my bag again. I help them, showing where everything conspicuous is, and opening my flute box so they know what's in there. He looks at everything, and signals that I can move on. Hang on, I think. This can't be right. 'Okay,' I ask? 'Yes,' he says. I run for the nearest info desk, show them my boarding pass, and tell them 'please, my flight leaves in 10 minutes, can you tell me where to go?' They are quick and efficient. Gate J40. I make a run for it again, and just make it in time. Of course, I could forget about my plans of finding a power outlet, plugging in my cell phone and calling my parents, but that's not a big deal—there's a train station at Brussels Airport. So what have we learned? Security on airports is random, arbitrary, non-standardized, and utterly stupid. Items that pose no danger at all—plastic bottles containing more than 100ml of water, small quantities of yoghurt that are over 100grammes but not over 100ml, tripods—are not allowed, while on the other hand I've occasionally gotten through security carrying knives and tools that are clearly forbidden by the rules. We've also learned that the rules very much depend on the particular person checking you. If you're refused for an item that is not a knife or a gun or something else similarly problematic, it may help to just find a different security checkpoint and try again. It's not as if they make any sense, anyway.

While I enjoyed watching some of the DebConf 8 Video Team’s Excellent Service, not all of it was at convenient times for me, being several timezones away, so I used the following script to record some of the streams:-

#!/bin/sh
# Example: recordstream 09:00 09:55 -Otalk.ogv http://server:8000/stream
start=$1
end=$2
shift 2
( echo wget -q "$@" '&'
echo 'echo $! > $HOME/wget.pid' )   at $start
echo 'kill $(<$HOME/wget.pid)'   at $end

I think that’s right, but I might have fluffed a $@ for a $* or similar (I usually use the simpler rc shell but it’s not on that server yet). It uses the oft-forgotten at command for one-off timed scripts, so you’ll need the at package installed. I think it might exploit the fact that players like mplayer, VLC and ffplay don’t mind truncated streams, but it seems to work fairly reliably apart from the timing. In other Debian-related news: Debian GNU/Linux: 15 Years Old and at the Crossroads by Bruce Byfield seems an interesting summary (warning: its feedback form contains an eyetest), but doesn’t mention DebConf apart from getting its name wrong. I had the impression that there were a few “at the crossroads” type of talks at DebConf. I guess we’ll see what concrete results come out of DebConf over the next few weeks.

Besides the slides from the tutorial at UseR! 2008 that were mentioned here previously, I also gave a short talk on scripting with R in high-performance computing using our littler frontend to R. The talk introduces and extends an example related to some of the material from the tutorial itself. The slides from the talk are a little rough as the talk was somewhat ad-hoc: As session chair, I was confronted with a fairly last-minute cancellation and a 15 minute hole, and thought this would make a good little talk. It does show a nice trick for using littler with Open MPI (via snow) under the powerful slurm resource manager and batch/queue engine.

I'm happy that we were able to upload reportbug 3.40 in unstable this night!

Now there is a team to work on reportbug (thanks to Chris to let it go), so more workpower, more thinking heads, more bugs closed ;)

So, since this is the transition upload, please let us know asap if something went wrong; or if you want a new feature, something you feel needs to be change etc etc.

There was even an article on FreeSoftwareMagazine some days ago, so let's spread the word! :D

A large part of the disagreement about the way to manage the policy seems to be based on who will be the primary “owner” of the policy on the machine. This isn’t a problem that only applies to SE Linux, the same issue applies for various types of configuration files and scripts throughout the process of distribution development. Having a range of modules which can be considered configuration data that come from a single source seems to make SE Linux policy unique among other packages. The reasons for packaging all Apache modules in the main package seem a lot clearer. One idea that keeps cropping up is that as the policy is modular it should be included in daemon packages and the person maintaining the distribution package of the policy should maintain it. The reason for this request seems to usually be based on the idea that the person who packages a daemon for a distribution knows more about how it works than anyone else, I believe that this is false in most cases. When I started working on SE Linux I had a reasonable amount of experience in maintaining Debian packages of daemons and server processes, but I had to learn a lot about how things REALLY work to be able to write good policy. Also if we were to have policy modules included in the daemon packages, then those packages would need to be updated whenever there were serious changes to the SE Linux policy. For example Debian/Unstable flip-flopped on MCS support recently, changing the policy packages to re-enable MCS was enough pain, getting 50 daemon packages updated would have been unreasonably painful. Then of course there is the case where two daemons need to communicate, if the interface which is provided with one policy module has to be updated before another module can be updated and they are in separate packages then synchronised updates to two separate packages might be required for a single change to the upstream policy. I believe that the idea of having policy modules owned by the maintainers of the various daemon packages is not viable. I also believe that most people who package daemons would violently oppose the idea of having to package SE Linux policy if they realised what would be required of them. Caleb Case seems to believe that ownership of policy can either be based on the distribution developer or the local sys-admin with apparently little middle-ground [1]. In the section titled “The Evils of Single Policy Packages” he suggests that if an application is upgraded for a security fix, and that upgrade requires a new policy, then it requires a new policy for the entire system if all the policy is in the same package. However the way things currently work is that upgrading a Debian SE Linux policy package does not install any of the new modules. They are stored under /usr/share/selinux/default but the active modules are under /etc/selinux/default/modules/active. An example of just such an upgrade is the Debian Security Advisory DSA-1617-1 for the SE Linux policy for Etch to address the recent BIND issue [2]. In summary the new version of BIND didn’t work well with the SE Linux policy, so an update was released to fix it. When the updated SE Linux policy package is installed it will upgrade the bind.pp module if the previous version of the package was known to have the version of bind.pp that didn’t allow named to bind() to most UDP ports - the other policy modules are not touched. I think that this is great evidence to show that the way things currently work in Debian work well. For the hypothetical case where a user had made local modifications to the bind.pp policy module, they could simply put the policy package on hold - I think it’s safe to assume that anyone who cares about security will read the changelogs for all updates to released versions of Debian, so they would realise the need to do this. Part of Caleb’s argument rests on the supposed need for end users to modify policy packages (IE to build their own packages from modified source). I run many SE Linux machines, and since the release of the “modular” policy (which first appeared in Fedora Core 5, Debian/Etch, and Red Hat Enterprise Linux 5) I have never needed to make such a modification. I modify policy regularly for the benefit of Debian users and have a number of test machines to try it out. But for the machines where I am a sysadmin I just create a local module that permits the access that is needed. The only reason why someone would need to modify an existing module is to remove privileges or to change automatic domain transition rules. Changing automatic domain transitions is a serious change to the policy which is not something that a typical user would want to do - if they were to do such things then they would probably grab the policy source and rebuild all the policy packages. Removing privileges is not something that a typical sysadmin desires, the reference policy is reasonably strict and users generally don’t look for ways to tighten up the policy. In almost all cases it seems best to consider that the policy modules which are shipped by the distribution are owned by the distribution not the sysadmin. The sysadmin will decide which policy modules to load, what roles and levels to assign to users with the semanage tool, and what local additions to add to the policy. For the CentOS systems I run I use the Red Hat policy, I don’t believe that there is a benefit for me to change the policy that Red Hat ships, and I think that for people who have less knowledge about SE Linux policy than me there are more reasons not to change such policy and less reasons to do so. Finally Caleb provides a suggestion for managing policy modules by having sym-links to the modules that you desire. Of course there is nothing preventing the existence of a postfix.pp file on the system provided by a package while there is a local postfix.pp file which is the target of the sym-link (so the sym-link idea does not support the idea of having multiple policy packages). With the way that policy modules can be loaded from any location, the only need for sym-links is if you want to have an automatic upgrade script that can be overridden for some modules. I have no objection to adding such a feature to the Debian policy packages if someone sends me a patch. Caleb also failed to discuss how policy would be initially loaded if packaged on a per-module basis. If for example I had a package selinux-policy-default-postfix which contains the file postfix.pp, how would this package get installed? I am not aware of the Debian package dependencies (or those of any other distribution) being about to represent that the postfix package depends on selinux-policy-default-postfix if and only if the selinux-policy-default package is installed. Please note that I am not suggesting that we add support for such things, a package management system that can solve Sudoku based on package dependency rules is not something that I think would be useful or worth having. As I noted in my previous post about how to package SE Linux policy for distributions [3] the current Debian policy packages have code in the postinst (which I believe originated with Erich Schubert) to load policy modules that match the Debian packages on the system. This means that initially setting up the policy merely requires installing the selinux-policy-default package and rebooting. I am inclined to reject any proposed change which makes the initial install of of the policy more difficult than this. After Debian/Lenny is released I plan to make some changes to the policy. One thing that I want to do is to have a Debconf option to allow users to choose to automatically upgrade their running policy whenever they upgrade the Debian policy package, this would probably only apply to changes within one release (IE it wouldn’t cause an automatic upgrade from Lenny+1 policy to Lenny+2). Another thing I would like to do is to have the policy modules which are currently copied to /etc/selinux/default/modules/active instead be hard linked when the source is a system directory. That would save about 12M of disk space on some of my systems. I’ve taken the unusual step of writing two blog posts in response to Caleb’s post not because I want to criticise him (he has done a lot of good work), but because he is important in the SE Linux community and his post deserves the two hours I have spent writing responses to it. While writing these posts I have noticed a number of issues that can be improved, I invite suggestions from Caleb and others on how to make such improvements.

• [1] http://www.calebcase.com/node/2


• [2] http://lists.debian.org/debian-security-announce/2008/msg00201.html


• [3] http://etbe.coker.com.au/2008/08/18/se-linux-policy-packaging-distribution/

Share This

Anthropology and the social sciences in general are not at the forefront of thinking of open access in publishing but thanfkfully there is a crew of anthropologists who are thinking about such questions and have some insights about the (sorry) direction our national organization has decided to take. Read more ...

What I liked

• Excellent organization, orga team fully deserved all the praise they got.
• Meeting people - it was the first DebConf I attended in person, so it was a pleasure to meet lots of folks whom I’ve only known online before.
• Quite a few interesting talks (videos are available, or will be available shortly, in DebConf8 video archive). Particularly recommended: quality assurance talks/BOFs by Lucas Nussbaum, BTS development progress by Don Armstrong, method diffusion in large volunteer projects by Martin Krafft, talk and debate on OpenSSL security issue by Luciano Bello, various lightning talks.
• Deserving a special mention, a talk by Mark Shuttleworth on Debian and Ubuntu relationship. I thought that he said all the right words and, judging by reactions, attitude towards Ubuntu has changed significantly during the last year, turning from hostility to peaceful cooperation.
• Argentinian food: steaks, empanadas, alfajores.

What I did not like

• People pay much less attention to fixing RC bugs than they used to, the two BSPs organized during DebConf8 attracted only 3-5 people.
• Some nice ideas being thrown around without anyone volunteering to work on them. Not really surprising, given that most of the core teams are short on manpower.
• Almost complete lack of events targeted at engaging users and beginner contributors.
• Quite a few specialized talks, interesting (IMHO) to only a minor percentage of the attendees.
• Lots of discussions with few or no specific decisions or action items.
• Pretty long trips to and from Mar del Plata (about 30 hours door-to-door travel time for me), luggage was lost for 2 days on the way there.

What I did

• Attended about 75% of talks, tried to contribute to discussion in constructive, non-flamy ways .
• Participated in BSPs, contributed debugging information, patches or fixes for about a dozen of release critical bugs, started triaging sparc-specific bugs.
• Started working on updating the Debian kernel handbook for Lenny, about 1/3 done.
• Gave a “Google and Open Source” lightning talk.

Following the recent post about my issues with byte swapping on amd64, I have received several answers.. Since comments were broken, Rupert sent me a good explanation of the cryptic code from OSS4. Let me remember you what it was doing: The code is indeed very simple, but it has to be decomposed carefully.. First, the author assumes that unsigned char = one byte, which seems reasonable. Then, a is initialised at the second character of the two-characters value x, and b on the first of the returned (swapped) value y. Now, the two cryptic lines can be decomposed as follows: *(b++) = *(a--), which literally means "put in b the value from a, and then increase b and decrease a. When executing the second line, a points to the first character from x, and b to the second from y. As Rupert noticed, the second call could simply have been *b = *a, but this wouldn't have been l33t enough :) Now, about the original code, namely (x 8) (x >> 8), MadCoder (salut !), as well as another anonymous comment both suggested that this would be a type error. Indeed, I had noticed the wrong right shift, but no matter how I tried, with or without mask, with unsigned short, short, or uint16_t, as suggested by MadCoder, I always get the same corrupted sound.. So, thanks for your comments, I'm now trying to dig the issue down in the code that makes use of this function. Most likely the issue comes from a wrong transtyping there.. I am also wondering what's the efficiency of the above OSS4's code with regard to (x 8) (x >> 8). Any idea ? PS: Yes, text on comments are mangled a lot. This is a basic safehtml protection, and I don't have the control over it, sorry..

We should coordinating to be able to release a "new" Wiki when Lenny is released. e.g :

1. Choose a License for the Wiki content. This topic has been discussed many times, but was never addressed (searching "wiki license site:lists.debian.org/debian-www" or in "/debian-legal", reveals lots of discussions !)

2. The layout could be refactored.
2.a. Create/adapt the moinmoin theme (CSS) to be more " la Debian" (either www.d.o, or pts, etc...).
2.b. Front page content : which links. The "new" front page is nicer, but the icons are meaning less. also it has too many entries... (compare http://wiki.debian.org/FrontPage?action=recall&rev=488 and current http://wiki.debian.org/FrontPage )
2.c. Front page Layout : Once we have decided the content, pick a layout (what do we want to emphasize : News ? Projects ? Download ? Support ? DDs ? Users ? Communities ? "Get Involved" ? )

3. Coordination : Make a planning until Lenny's release date.Full event details

maemo sdk seems to come with a cdbs, debhelper and dpkg-dev too old to make it easy to build several debian unstable packages like kde4.
maemo seems to come with glibc2.5 and debian/armel kde4 packages depends on 2.7 or greater.
trying to build a newer dpkg in maemo environment requires several perl modules not available - and when buildling those, a circular build dependency is quickly reached. So - dear lazyweb, what does smart man do now?

Some time ago, when I packaged treeline, I wanted to make the package configure the environment so that .trl files would be automatically open with the program. That meant that the package had to configure the MIME types in the desktop to handle that, and assign the new MIME type the selected program. I tried a dozen of things at the time, based on a lot of howto’s that I found, without any success, so I decided to forget about it. This week I started packaging a new program: L ve, a 2D game development framework based in Lua and OpenGL. Many of the games developed for this platform are distributed as a zip file with the extension .love, so I decided I wanted to try again the MIME stuff. After many unsuccessful tries, I found a package that gave me some hints on how to do it: chemical-mime-data. After having a look at it, I was able to prepare a proof of concept that did what I wanted. It is supposed to work, at least, in GNOME, KDE and ROX. In case someone else is interested, here is the code:

• mime-example_0.1-1.dsc
• mime-example_0.1-1.diff.gz
• mime-example_0.1.orig.tar.gz

po4a - or PO for anything. PO is the de facto standard for translations and po4a tries to help maintainers prepare packages for translation.

1. I18n (internationalisation) is making a package accessible for translators and is an upstream/maintainer action. i18n is done once for all translation support and updated when the content changes (typically during the build).


2. l10n (localisation) is using i18n support to create an individual translation for a specific locale. l10n is done many times per package, with individual translations being updated each time the i18n content changes. (Tdebs will support this area but are a separate topic from po4a.) PO files that exist when the POT is updated are also updated, tagging modified strings as "fuzzy".

The natural burden, therefore, falls within l10n but without i18n, l10n is all but impossible (it means having to do the i18n "outside" the package).
I'm using po4a and I've been doing various translation-support stuff now for a few years, so I'm going to add po4a into my blog entries as a series to help maintainers make their own packages i18n aware.
This is particularly important for:

1. all Debian native packages, particularly scripted packages. My example will be based on perlpod documentation often embedded wthin the perl scripts themselves as well as DocBook XML for manpage generation. po4a supports many other formats.


2. All manpages (or other documentation) created by Debian maintainers

To make things easier, I'll be creating a new category in my blog ("i18n") for such entries. This is helpful for me too because the first time I used po4a (current versions of emdebian-tools) I basically got things wrong and made more work for myself than was necessary. This entry primarily concerns providing i18n support from manpage sources and perl scripts. i18n support in compiled packages is a separate issue. The fixes described here will be included in the next version of emdebian-tools (which will be uploaded to Emdebian but not into Debian until after Lenny).
Terminology

• PO - (Portable Object) files contain the POSIX C locale strings abstracted from the "master document" which in many cases is the source file in the relevant package(s), alongside the translated version of that string (if any). Multiple master documents are typically combined into a single collection of translatable strings. PO files are translated into individual locales, one PO file per locale.


• POT - PO Template - the definitive collection of translatable strings from which all PO files are prepared and updated. The POT file is never translated directly, it is maintained, updated and packaged by the maintainer.


• gettext - the binary application that handles the translatable strings. gettext abstracts the strings out of the source files during the package build and inserts the translated strings back into the text actually displayed to the user at runtime. If no suitable PO file exists, the user gets the untranslated string, exactly as it appears in the POT file. If a suitable PO file does exist but the strings have been modified since translation, gettext drops "fuzzy" translations that do not match precisely and the user gets a "mixed" translation containing some translated strings and some untranslated. Most translators try to maintain a 100% translation because the ugliness of the "mixed" output is a partial incentive to keep up with the updates.

Using po4a
po4a needs a config file where the maintainer keeps all metadata relating to the master documents and the relevant formats. po4a then converts the master documents into a collection of strings for translation and puts those strings into a POT file (a PO template file). The maintainer packages the POT file (and optionally sends the POT file to relevant mailing lists seeking translations to be returned as PO files) and keeps the POT file up to date during the build process. With a config file (e.g. called po4a.config), all the maintainer needs to do is call:

po4a po4a.config

po4a then parses the list of files defined in the config file, generates the lists of strings, collates the lists and identifies where the same string is used in multiple files and writes out a POT file containing references to where the strings are used and the unique strings.
po4a config file
Comments start with '#' in the usual way. Instructions are enclosed in square brackets.
po4a_paths defines where the POT file will be put (it can be extended later but I'm keeping this simple). The name of the POT file is important for compiled packages but can just be the name of the source package for scripted packages and/or manpages.

[po4a_paths] po/emdebian-tools.pot

type identifies *each* master document to be processed by po4a. The syntax is well suited to being generated from some other list within the package but I'll give just a snippet of the final output here:

[type: xml] doc/xml/em_autobuild.1.xml

[type: xml] doc/xml/embug.1.xml

[type: pod] emtargetcmp

[type: pod] emdebcheck

That's it. The complexity of my current script was entirely unnecessary.

Also, po4a is more than capable of handling multiple document types within the same source package and the same POT file which is something that I had missed so far. One more hack that can be removed.

All I do now is use xmllint to "clean-up" the DocBook XML and pod2html and some XSLT to generate HTML documentation for the website. The loops to do those tasks already contain the list of files to be processed by po4a, so a few echo commands is all that is needed to create po4a.config at build time and run po4a to generate a combined POT file for all strings in the package.

#!/bin/sh

CONFIG="po4a.config"

echo "[po4a_paths] po/emdebian-tools.pot" > $CONFIG

for file in  ls doc/xml/*.xml ; do

	xmllint --format "$file" --output "$file"

	echo "[type:xml] $file" >> $CONFIG

done

for file in emdebcheck emtargetcmp emprunecross Emdebian/Tools.pm \

 em_installtdeb emrecent emcache; do

	if [ "$file" = "Emdebian/Tools.pm" ]; then

		pod2html -outfile html/EmdebianTools.html -title $file  ../$file

	else

		pod2html -outfile html/$file.html -title $file  ../$file

	fi

	if [ "$file" = "Emdebian/Tools.pm" ]; then

		pod2man ../$file > man/Emdebian::Tools.3

	else

		pod2man ../$file > man/$file.1

	fi

	echo "[type: pod] $file" >> $CONFIG

done

po4a $CONFIG

When configuring servers the trade-offs between RAM and disk are well known. If your storage is a little slow then you can often alleviate the performance problems by installing more RAM for caching and to avoid swapping. If you have more than adequate disk IO capacity then you can over-commit memory and swap out the things that don’t get used much. One that often doesn’t get considered i the trade-off between RAM and CPU. I just migrated a server image from a machine with two P4 CPUs to a DomU on a machine with Opteron CPUs. The P4 system seemed lightly loaded (a maximum of 30% CPU time in use over any 5 minute period) so I figured that if two P4 CPUs are 30% busy then a single Opteron core should do the job. It seems that when running 32bit code, 30% of 2*3.0GHz P4 CPUs is close to the CPU power of one core of an Opteron 2352 (2.1GHz). I’m not sure whether this is due to hyper-threading actually doing some good or inefficiencies in running 32bit code on the Opteron - but the Opteron is not giving the performance I expected in this regard. Now having about 90% of the power of that CPU core in use might not be a problem, except that the load came in bursts. When a burst took the machine to 100% CPU power a server process kept forking off children to answer requests. As all the CPU power was being used it took a long time to answer queries (several seconds) so the queue started growing without end. Eventually there were enough processes running that all memory was used, the machine started thrashing, and eventually the kernel out of memory handler started killing things. I rebooted the DomU with two VCPUs (two Opteron cores) and there was no problem, performance was good, and because the load bursts last less than a minute the load average seems to stay below 1. It seems that the use of virtual machines increases the scope of this problem. The advantage of virtual machines is that you can add extra virtual hardware more easily (up to the limit of the physical hardware of course) - I could give the DomU in question 6 Opteron cores in a matter of minutes if it was necessary. The disadvantage is that the CPU use of other virtual machines can impact the operation. As there seems to be an exponential relationship between the number of CPU cores in a system and the overall price it’s not feasible to just put in 32 core Xen servers. While CPU power has generally been increasing faster than disk performance for a long time (at least the last 20 years) it seems that virtualisation provides a way of using a lot of that CPU power. It is possible to have a 1:1 mapping of real CPUs and VCPUs in the Xen DomU’s, if you were to install 8 DomU’s that each had one VCPU on a server with 8 cores then there would be no competition between DomUs for CPU time - but that would significantly increase the cost of running them (some ISPs offer this for a premium price). In this example, if I had a burst of load for the service in question at the same time as other DomUs were using a lot of CPU time (which is a possibility as the other DomUs are the clients for the service in question) then I might end up with the same problem in spite of having assigned two VCPUs to the DomU. The real solution is to configure the server to limit the number of children that it forks off, the limit can be high enough to guarantee 100% CPU use at times of peak load without being high enough to start swapping. I wonder how this goes with ISPs that offer Xen hosting. It seems that you would only need to have one customer who shares the same Xen server as you experiencing such a situation to cause enough disk IO to cripple the performance that you get.

Share This

Third day of DebConf8 now - and it has been a pretty nice time since I arrived. The local orga team is doing a marvelous job, making this whole conference a really nice experience for about every attendee. Thanks to that we had a lot of time doing actual Debian work, which meant that we got a nice number of changes in dak either done or planned for the future, including a redesign of our database schema. Which is urgently needed if we want to follow some of the points we discussed. Like throwing away apt-ftparchive and generating the Packages files ourselve. Or various other enhancements. So for dak, the changes are summarised into:

• Adapt an old patch for process-new, so that NEW packages only adding new binary components are sorted higher during the processing.
• Imported a patch from Joachim that adds an (effectively) anti-s390-buildd-maintainer option to our cruft-report full mode. Well, actually it lists a set of packages which are marked “not-for-us” on the buildd, but still have a (outdated) binary in unstable, effectively blocking updates of the packages going into testing. Ideally a buildd maintainer would report to the maintainer (through a bug or even just a mail) when he sets not-for-us (which is a short-term-solution, long-term P-a-s should be used), but that doesn’t seem to happen.
• Replaced a 128 line NIH syndrom perl script with one of those new-fangled sarge-and-later options in rsync :)
• Fixed bugs in code-pathes that are nearly never used - but which we managed to trigger twice while fixing a gpl violation (missing source) in our archive.
• Merged a patch that changes NEW handling of packages for stable/oldstable.
• Added signing of the buildd incoming.debian.org directory. Still need to announce this on d-d-a, but buildd maintainers should now be able to enable the gpg verification.
• Automate the i18n package description translation import so we finally get rid of the “one-time-manual-sync-back-in-2006” outdated translations. And have an automated but secure way to get updates.
• Started the layout of the new projectb.

But we did not only do work on dak, we also did some work for the conference. In the days down here we (re)wrote the Pentabarf entrance controller, enabling our frontdesk to check people in, keeping track of various details they need to know about them. We wrote entirely new controllers for Pentabarf, one of it being the part that manages this years Assassins game. Other orga team members (hello triple des :) ) wrote a video and a volunteer controller. I guess after DebConf we will have some fun time to merge our changes back to upstream.

---

Travel in .ar Oh well, as some people know, I stay in Argentina for about 10 days more after DebConf is over (my flight back goes on 27. August, in the evening). Up to now I haven’t decided - or even started planning - what I do here. I should probably look at that soon. Hrm. Anyone out there doing something similar the next days? :) Comments: 0

For some time people have been telling me about the benefits of SpamAssassin (SA). I have installed it once for a client (at their demand and against my recommendation) but was not satisfied with the result (managing the spam folder was too complex for their users). The typical configuration of SA has it run after mail has been accepted by the server. Messages that it regards as spam are put into a spam folder. This means that when someone phones you about some important message you didn’t receive then you have to check that folder. Someone who sends mail to a user who has such a SA configuration can not expect that the message will either be received or rejected (thus giving them a bounce message). Even worse it seems to be quite common for technical users to train the Bayesian part of SA on messages from the spam folder - without reviewing them! Submitting a folder of spam that has been carefully reviewed for Bayesian training can increase the accuracy of classification (including taking account for locality and language differences in spam). Submitting a folder which is not reviewed means that when a false-positive gets into that folder (which will eventually happen) it is used as training for spam recognition thus increasing the incidence of false-positives! Spam has been becoming more of a problem for me recently, on a typical day between 20 and 40 spam messages would get past the array of DNSBL services I use and be re-sent to pass the grey-listing. Also I have been receiving complaints from people who want to send email to me about some of the DNSBL and RHSBL services I use (the rfc-ignorant.org service gets a lot of complaints - there are a huge number of ignorant and lazy people running mail servers). So now I have installed spamassassin-milter to have SA run during the SMTP protocol. Then if SA checks indicate that the message is SPAM my mail server can just reject the message with a 55x which will cause the sending mail server to generate a local bounce (if it’s a legitimate message) or to just be discard it in the case of a spam server. Here is how to set it up on Debian/Lenny and CentOS 5: Install the package yum install spamass-milter or apt-get install spamass-milter spamassassin spamc (spamassassin seems to be installed by default on CentOS). On a Debian system the milter will be setup and running. On CentOS you have to run the following commands:
useradd -m -c "Spamassassin Milter" -s /bin/false spamass-milter
mkdir /var/run/spamass-milter
chown spamass-milter /var/run/spamass-milter
chmod 711 /var/run/spamass-milter
echo SOCKET="/var/run/spamass-milter/spamass.sock" >> /etc/sysconfig/spamass-milter On CentOS edit /etc/init.d/spamass-milter and change the daemon start line to ‘runuser - spamass-milter -s /bin/bash -c "/usr/sbin/spamass-milter -p $SOCKET -f $EXTRA_FLAGS"‘ Then add the following lines below it:
chown postfix:postfix /var/run/spamass-milter/spamass.sock
chmod 660 /var/run/spamass-milter/spamass.sock The spamass-milter program talks to the SpamAssassin daemon spamd. On both Debian and CentOS run the command “useradd -c Spamassassin -m -s /bin/false spamassassin” to create an account for SA. The Debian bug #486914 [1] has a request to have SA not run as root by default. On CentOS it seems that SA wants to use a directory under the spamass-milter home directory, the following commands alllow this. It would be good to have it not do that, or maybe it would be better to have the one Unix account used for SA and the milter.
chmod 711 ~spamass-milter
mkdir ~spamassassin/.spamassassin
chown spamassassin ~spamassassin/.spamassassin On Debian edit the file /etc/default/spamassassin and add “-u spamassassin -g spamassassin” to the OPTIONS line. On CentOS edit the file /etc/sysconfig/spamassassin and add “-u spamassassin -g spamassassin” to the SPAMDOPTIONS line. To enable the daemons, on CentOS you need to run “chkconfig spamass-milter on ; chkconfig spamassassin on“, on Debian edit the file /etc/default/spamassassin and set ENABLED=1. Now start the daemons, on CentOS use the command “service spamassassin start ; service spamass-milter start“, on Debian use the command “/etc/init.d/spamassassin start“. Now you have to edit the mail server configuration, for Postfix on CentOS the command “postconf -e smtpd_milters=unix:/var/run/spamass-milter/spamass.sock” will do it, for Postfix on Debian the command “postconf -e smtpd_milters=unix:/var/spool/postfix/spamass/spamass.sock” will do it. Now restart Postfix and it should be working. For correct operation you need to ensure that the score needed for a bounce is specified as the same number in both the spamass-milter and SA configuration. If you have a lower number for the spamass-milter configuration (as is the default in Debian) then bounces can be generated - you should never generate a bounce for a spam. The config file /etc/default/spamass-milter allows you to specify the score for rejecting mail, I am currently using a score of 5. Any changes to the score need matching changes to /etc/mail/spamassassin/local.cf (which has a default required_score of 5 in Debian). You can grep for “spamd..result..Y” in your mail log to see entries for messages that were rejected. One problem that I have with this configuration on Debian (not on CentOS) is that spamd is logging messages such as “spamd: handle_user unable to find user: ‘russell’“. I don’t want it to look for ~russell when processing mail for russell@coker.com.au because I have a virtual domain set up and the delivery mailbox has a different name. Ideally I could configure it to know the mapping between users and mailboxes (maybe by parsing the /etc/postfix/virtual.db file). But having it simply not attempt to access per-user configuration would be good too. Any suggestions would be appreciated. Now that I have SpamAssassin running it seems that I am getting about 5 spams a day, the difference is significant. The next thing I will do is make some of the DNSBL checks that are prone to false-positives become SpamAssassin scores instead. When I started writing this post I was not planning to compare the sys-admin experiences of CentOS and Debian. But it does seem that there is less work involved in the task of installing Debian packages.

• [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=486914

Share This