I just had this crazy idea about solving snapshot.debian.net's space issue which just might work if done right.

We all know that snapshot.debian.net has ran out of space in early May this year and we all lost a valuable service. Since there are probably many users, developers and maintainers which find this service useful, it might make sense to think that the users themselves could solve the problem.

How? Think about implementing a really distributed storage/file system (not necesarly a real file system) that can rely on cluster nodes that can enter or exit at any time and the nodes are contributed by users, very much alike the GoogleFS file system (or maybe clients in a torrent swarm).


First problems that I can already see:

• there must be some indexing service that should run 24x7 to manage the information about the blocks which are desired.
• some data should always be kept on a trusted machine - the gpg signatures, the Packages files indexing information and meta data
  
• there might be a security risk involved since data would be stored on untrusted machines, but a sha1/md5 check in colaboration with the gpg signature should be enough (which isn't always doable - see large packages like openoffice.org or game data packages); if there are reasons why this might not be secure, maybe we need to rethink a little about the dpkg-sig package and see if we can get all the packages in the archive to be individually signed when entering the archive, or something of that sort
• the meta-data of the FS cluster might be really big, but my gut feeling tells me that it won't be as big as the current snapshot.d.n storage
• when a file is requested, the file needs to be cached on a central server so it can be assembled and its checksum be verified before being delivered to the client requesting it
• people might donate really little space compared to their usage
• some nodes of the network are special and this could lead to the failure of the entire infrastructure in case the special nodes fail; some of these nodes might turn out to need really big muscles
  
• some data might be unavailable at times depending on the clients connected to the network at different points in time
• in an attempt to create more copies of a block in more nodes in the network, a DoS might be triggered if many clients request the same info from a (set of) node(s) containing the rare info - still torrent's way of workig seems to cope quite well with that model
• none of this is implemented and probably I won't do it myself, although it could be a really nice project, even it is doomed to fail from day 1
  

Other ideas:

• maybe it would make more sense for the FS to actually be an ever-growing torrentt which people can connect to via some special client which would allow pushing to the clients in order to store information
• the number of copies of a file (or chunk of a file) in the distributed storage should be higher for more recent packages and lower for older ones (still one copy might be necessary to be stored on safe nodes - aka controled and trusted by debian so the information doesn't get lost - back to our current storage problem)
• probably the entire thing could be built somehow by piggy-backing the current apt-transport-debtorrent implementation or debtorrent, or apt's cache - the idea is that although not all files might be available, some people might still have relatiely recent packages files in their apt cache
• such a system, if done right, might prove to be usable as a live backup system between different nodes in the cluser, so it could be used in other situations by individuals, without the need of a central server - think trackerless trorrents
  

What do other people think?

Some time ago, when I packaged treeline, I wanted to make the package configure the environment so that .trl files would be automatically open with the program. That meant that the package had to configure the MIME types in the desktop to handle that, and assign the new MIME type the selected program. I tried a dozen of things at the time, based on a lot of howto’s that I found, without any success, so I decided to forget about it. This week I started packaging a new program: L ve, a 2D game development framework based in Lua and OpenGL. Many of the games developed for this platform are distributed as a zip file with the extension .love, so I decided I wanted to try again the MIME stuff. After many unsuccessful tries, I found a package that gave me some hints on how to do it: chemical-mime-data. After having a look at it, I was able to prepare a proof of concept that did what I wanted. It is supposed to work, at least, in GNOME, KDE and ROX. In case someone else is interested, here is the code:

• mime-example_0.1-1.dsc
• mime-example_0.1-1.diff.gz
• mime-example_0.1.orig.tar.gz

Although far from winning the war against Azeredo s bill, we finally won a battle. Chamber-of-Deputies IT Committee approved the request [pt-BR] for a public hearing [pt-BR]. Although a small step forward, all people involved with entities and NGOs interested in blocking this bill will have a chance to speak their minds. This comes in a very special timing. A pool was published early this week [pt-BR] run by Vox Populi revealing that more than a half of the Brazilian voters distrust e-voting and about 20% think it s possible to spy on their choices. This is a election year in Brazil, and all this setting can end up helping our case. More than 106-thousand already signed the petition and I think with all those signatures adding to this setting this is going to be a very interesting hearing. At least people supporting this bill will have to stand up and say they are for it And who knows what damage this can cause to them in a election year. Some public hearings are broadcast through Internet. I ll let you know if this one will be.

Unfortunately, Matthew is correct, and it seems that Ryan Farmer is the idiot that signed a few of us up to about 200 mailing lists this morning. Not only do I trust Matthew's analysis (and was watching him go through the various steps on IRC as it happened), but then there were two other bits.

Ryan refused to approve two of my comments I made on his latest blog entry (more than happy to screenshot this). He claimed he too had been spammed, and had to abandon his email address of a few years. Even if you ignore the length of the text blanked out in the title bar not even remotely matching the length of his personal email address, there were only 131 messages in the inbox, of which 127 were unread, and no others but spam visible. He claimed they were archived; I offered to publicly apologise if he'd do a quick screencast showing him logging into that account and showing some messages from the archive to prove the point. Unfortunately he didn't bother approving these comments. (Incidentally, if you had to 'abandon' an email address due to an avalance of newsletters, et al, which took an hour or two to fully unsubscribe from, would you still bother blacking out the address everywhere?)

As per his standard 'insult everyone with random images from the internet and YouTube videos, then wildly claim conspiracies against him' policy, he went on to claim Matthew was conspiring against him, but given the choice between Matthew conspiring and severely irritating a few of his friends in the process, and some scorned 16-year old continuing his track record of acting like an irrational, immature idiot, I know which one I'd pick.

It takes some skill to be banned from all the Ubuntu IRC channels and the Ubuntu forums both, but I guess this morning showed that was completely justified. Apologies to everyone waiting on the announcement of sponsorship for XDS, as well as hotels, but unfortunately some people just can't conceive of having better things to do than acting like chavs. Hopefully IzanbardPrince/TheAlmightyCthulu/etc goes back to impersonating Kevin Carmony, flaming the bejesus out of companies using BIOSes which trip Linux bugs, and other, equally productive, uses of his time.

Last week I needed a way of converting a two-dimensional array, in perl, into a tree-like datastructure.<br /> While seemingly trivial, the convoluted use of references when passing around arrays and hashes between perl function calls -- recursive calls, in this case -- made this a mind-boggling exercise to me. (the fact that I hadn't seriously used perl in the last eight years might have had part in that, too)<br /> <br /> Below the jump is my implementation, debugging info left in and all. If it's useful for someone, that's great; if not, ... well, except for my mental sanity no harm was done in writing this, either :)
Continue reading "Perl hash tree helper"

po4a - or PO for anything. PO is the de facto standard for translations and po4a tries to help maintainers prepare packages for translation.

1. I18n (internationalisation) is making a package accessible for translators and is an upstream/maintainer action. i18n is done once for all translation support and updated when the content changes (typically during the build).


2. l10n (localisation) is using i18n support to create an individual translation for a specific locale. l10n is done many times per package, with individual translations being updated each time the i18n content changes. (Tdebs will support this area but are a separate topic from po4a.) PO files that exist when the POT is updated are also updated, tagging modified strings as "fuzzy".

The natural burden, therefore, falls within l10n but without i18n, l10n is all but impossible (it means having to do the i18n "outside" the package).
I'm using po4a and I've been doing various translation-support stuff now for a few years, so I'm going to add po4a into my blog entries as a series to help maintainers make their own packages i18n aware.
This is particularly important for:

1. all Debian native packages, particularly scripted packages. My example will be based on perlpod documentation often embedded wthin the perl scripts themselves as well as DocBook XML for manpage generation. po4a supports many other formats.


2. All manpages (or other documentation) created by Debian maintainers

To make things easier, I'll be creating a new category in my blog ("i18n") for such entries. This is helpful for me too because the first time I used po4a (current versions of emdebian-tools) I basically got things wrong and made more work for myself than was necessary. This entry primarily concerns providing i18n support from manpage sources and perl scripts. i18n support in compiled packages is a separate issue. The fixes described here will be included in the next version of emdebian-tools (which will be uploaded to Emdebian but not into Debian until after Lenny).
Terminology

• PO - (Portable Object) files contain the POSIX C locale strings abstracted from the "master document" which in many cases is the source file in the relevant package(s), alongside the translated version of that string (if any). Multiple master documents are typically combined into a single collection of translatable strings. PO files are translated into individual locales, one PO file per locale.


• POT - PO Template - the definitive collection of translatable strings from which all PO files are prepared and updated. The POT file is never translated directly, it is maintained, updated and packaged by the maintainer.


• gettext - the binary application that handles the translatable strings. gettext abstracts the strings out of the source files during the package build and inserts the translated strings back into the text actually displayed to the user at runtime. If no suitable PO file exists, the user gets the untranslated string, exactly as it appears in the POT file. If a suitable PO file does exist but the strings have been modified since translation, gettext drops "fuzzy" translations that do not match precisely and the user gets a "mixed" translation containing some translated strings and some untranslated. Most translators try to maintain a 100% translation because the ugliness of the "mixed" output is a partial incentive to keep up with the updates.

Using po4a
po4a needs a config file where the maintainer keeps all metadata relating to the master documents and the relevant formats. po4a then converts the master documents into a collection of strings for translation and puts those strings into a POT file (a PO template file). The maintainer packages the POT file (and optionally sends the POT file to relevant mailing lists seeking translations to be returned as PO files) and keeps the POT file up to date during the build process. With a config file (e.g. called po4a.config), all the maintainer needs to do is call:

po4a po4a.config

po4a then parses the list of files defined in the config file, generates the lists of strings, collates the lists and identifies where the same string is used in multiple files and writes out a POT file containing references to where the strings are used and the unique strings.
po4a config file
Comments start with '#' in the usual way. Instructions are enclosed in square brackets.
po4a_paths defines where the POT file will be put (it can be extended later but I'm keeping this simple). The name of the POT file is important for compiled packages but can just be the name of the source package for scripted packages and/or manpages.

[po4a_paths] po/emdebian-tools.pot

type identifies *each* master document to be processed by po4a. The syntax is well suited to being generated from some other list within the package but I'll give just a snippet of the final output here:

[type: xml] doc/xml/em_autobuild.1.xml

[type: xml] doc/xml/embug.1.xml

[type: pod] emtargetcmp

[type: pod] emdebcheck

That's it. The complexity of my current script was entirely unnecessary.

Also, po4a is more than capable of handling multiple document types within the same source package and the same POT file which is something that I had missed so far. One more hack that can be removed.

All I do now is use xmllint to "clean-up" the DocBook XML and pod2html and some XSLT to generate HTML documentation for the website. The loops to do those tasks already contain the list of files to be processed by po4a, so a few echo commands is all that is needed to create po4a.config at build time and run po4a to generate a combined POT file for all strings in the package.

#!/bin/sh

CONFIG="po4a.config"

echo "[po4a_paths] po/emdebian-tools.pot" > $CONFIG

for file in  ls doc/xml/*.xml ; do

	xmllint --format "$file" --output "$file"

	echo "[type:xml] $file" >> $CONFIG

done

for file in emdebcheck emtargetcmp emprunecross Emdebian/Tools.pm \

 em_installtdeb emrecent emcache; do

	if [ "$file" = "Emdebian/Tools.pm" ]; then

		pod2html -outfile html/EmdebianTools.html -title $file  ../$file

	else

		pod2html -outfile html/$file.html -title $file  ../$file

	fi

	if [ "$file" = "Emdebian/Tools.pm" ]; then

		pod2man ../$file > man/Emdebian::Tools.3

	else

		pod2man ../$file > man/$file.1

	fi

	echo "[type: pod] $file" >> $CONFIG

done

po4a $CONFIG

This is part 1 of a series on articles about the Firewire security issues mentioned below. For many years now, attacks via Firewire / i.LINK / IEEE 1394 have been a known security issue. Basically, if you gain physical access to a PC or laptop which has Firewire ports (or PCMCIA/Cardbus/ExpressCard, more on that later) you can

• read arbitrary RAM contents from the victim's system,
• overwrite arbitrary RAM contents with whatever you want,
• and perform many, many severe attacks based on the two issues above. Examples include grabbing a full RAM dump via Firewire (takes only a few minutes), grabbing ssh-agent keys, grabbing screen contents, modifying screen contents, bypassing login/password screens, and many, many more...

All of this is done by exploiting a "feature" of the Firewire spec (OHCI-1394) (PDF), namely that it allows read/write access to physical memory (via DMA) for external Firewire devices. Worse, as this is DMA, the CPU/OS will not even know what's going on. Even worse, this works regardless of whether you have locked your screen with a password-protected screensaver, or xlock, or vlock, or whatever. As long as the system is running, you're vulnerable. In this article, I intend to give a fairly complete overview of the available papers published on this issue, tools for testing the attacks, as well as mitigation techniques for various OSes. If I'm missing some important papers or tools, please post a comment! Papers, Attacks, and Tools Over the years a number of presentations and papers have been released with information about these Firewire issues. Maximilian Dornseif et. al. The first publication that I know of was done by Maximilian Dornseif, Michael Becher, and Christian Klein. They gave a number of talks on various security conferences on this topic:

• Pacsec 2004: 0wned by an iPod (PDF)
• 21C3: Hacking with Fire / Feuriges Hacken - Spa mit Firewire (PDF)
• Cansecwest 2005: Firewire: all your memory are belong to us (PDF)

They also released a number of tools, Firewire libraries for Mac OS X and Linux, as well as small demo scripts which use those libs:

• pyfw-20041111.tar.gz: Mac OS X Firewire bindings for Python, various demos
• http://c0re.23.nu/c0de/pyfw/: Other versions of the above code
• pyfw_linux.tar.gz: Linux Firewire bindings for Python (can be used with above demo code on Linux)
• ipod-linux-pyfw.tar.bz2: Instructions (+ disk image) on how to turn your iPod into a Firewire attack tool

Adam Boileau In 2006 Adam Boileau (a.k.a. Metlstorm) gave a talk called Hit by a Bus: Physical Access Attacks with Firewire (PDF) at Ruxcon 2006. In 2008 he then released a set of tools:

• pythonraw1394-1.0.tar.gz: Python bindings for libraw1394 (Linux). Tools: businfo, romtool, 1394memimage
• winlockpwn: Python script which can circumvent a locked Windows XP screen (an arbitrary password will log you in)
• bioskbsnarf: Grabs/shows the BIOS keyboard buffer via Firewire (which often contains your BIOS password)

Peter Panholzer As of early 2008 Peter Panholzer from sec-consult.com published a two-page whitepaper which says they were able to run a winlockpwn-like attack on Windows Vista via Firewire. There's not much information in the PDF unfortunately, and no tools were released, as far as I know.

• Whitepaper: Physical Security Attacks on Windows Vista (PDF)
• Bugtraq Thread about the whitepaper

David R. Piegdon The most recent toolset and papers I know of are from David R. Piegdon (a.k.a. IosTrace), who gave a number of talks in 2007/2008 about the issue, and also released a toolset called SEAT1394.

• 2007: OpenChaos: hacking in physically addressable memory (no extra slides)
• Easterhegg 2008: FireWire Hacking: paper (PDF), slides (PDF)
• More papers here...
• http://0null.net/software/SEAT1394/: tools and libraries for pyhsical memory access via Firewire/DMA: 1394csrtool, liblinear, libphysical, debug-sshkey, dump-memory, remote-ps, show-stack, snarf-sshkey, test-all-pagedirs, dmashell, ...

I'll go into much more detail on how the tools are used and what they can do in another follow-up article. Mitigation There are ways to eliminate or at least mitigate these attack vectors. The simplest and most secure way is to not have any Firewire ports installed (don't put Firewire PCI/PCIe cards in your PC, don't use Firewire PCMCIA/Cardbus/ExpressCard cards). Now, if you have a laptop with built-in Firewire ports, you have a problem, of course. In that case you could still physically destroy the port (by opening the laptop and cutting/desoldering stuff, or by putting glue/epoxy in the port in order to prevent any Firewire cables being attached). These are slightly drastic (but effective!) measures. Note: Even if you don't have any Firewire ports, you're not automatically safe and secure. If your laptop has a PCMCIA/Cardbus/ExpressCard slot, an attacker can simply insert a PCMCIA Firewire card (for instance) in that slot. Chances are, that your OS will automatically load the driver for that card and also the Firewire drivers you'll need if you want to use the card for attaching Firewire devices. Game over. Your "secure" laptop is now vulnerable... If you cannot (or don't want to) remove/destroy/disable your Firewire ports, the next best thing is to ensure that nobody except yourself ever gets physical access to your PC/laptop. This is hard to do for a PC, and almost impossible for a laptop, mind you. Finally, there are some software measures you can use to prevent at least physical DMA access for Firewire devices: Mitigation: Linux Pretty much every Linux system with the "old" Firewire drivers loaded (kernel module ohci1394 et. al.) is vulnerable to these issues. Newer kernels now also ship with a new Firewire stack called "juju" (kernel module firewire_ohci et. al.) which may or may not have the same issues (not fully tested by me so far, will report back later). Per default, all recent kernels, e.g. 2.6.26, are vulnerable, but see below. Under Linux, simply using a kernel which doesn't have any Firewire support (neither built-in, nor as a module) is the most secure option. If you must have Firewire support you can load the ohci1394 module with the phys_dma=0 parameter to at least disable physical DMA support:

  $ modprobe ohci1394 phys_dma=0

I have personally tested this on some boxes and I can confirm that it renders the currently published tools useless. As for the new "juju" Firewire stack, I'm not so sure. A few quick tests showed that the currently available tools don't work with the new stack, but you shouldn't feel too secure! AFAIK the new stack does support (or will support soon) physical DMA for Firewire, so it's probably just a matter of adapting the tools a bit (I'll do some testing/research on this later, as time permits). Mitigation: Mac OS On Mac OS you might also be able to completely remove Firewire support from the kernel (but I don't know if/how that can be done, not sure if you can easily recompile Mac OS kernels, and/or if you even have buildable source code and toolchains for that). However, you can at least remove the Firewire support in the default Mac OS installation by unloading AppleFWOHCI.kext:

  $ sudo kextunload /System/Library/Extensions/IOFireWireFamily.kext/Contents/PlugIns/AppleFWOHCI.kext

Thanks to a Daniel Reutter for letting me abuse his MacBook via Firewire and for finding the above kextunload command line. We have successfully tested that after unloading AppleFWOHCI.kext the current tools won't work anymore. The tests were done on a Mac OS X Tiger (I think) with all recent security updates applied. Please leave a comment if you can test other versions of Mac OS X... Mitigation: Windows As for Windows, well, I guess you're screwed. While Windows XP does implement sort of "protection" in that it only allows physical DMA access via Firewire to devices which "deserve it", e.g. iPods (or any other Firewire mass storage device, I guess) this can be easily defeated by having your attack PC/laptop pretend to be an iPod (see the romtool Python script by Adam Boileau). The only remaining option I know of (short of removing/destroying Firewire ports or preventing physical access alltogether) is to disable the Firewire ports/drivers in the device manager (untested by me so far). If you do that, remember to also disable all PCMCIA/Cardbus/ExpressCard controllers, of course (see above). So far I've tested Windows XP SP2/SP3 successfully with Adam Boileau's tools. I haven't yet been able to test Windows 95/98/Vista, if you can verify one of them, please leave a comment. Mitigation: OpenBSD/FreeBSD/NetBSD/OpenSolaris/... On OpenBSD you're likely not vulnerable as OpenBSD doesn't have any Firewire drivers at all, as far as I know ;-) As for FreeBSD, NetBSD, OpenSolaris, and other OSes I don't have any information. I might be able to test one or two of them in the nearer future, but please leave a comment if you have some information about whether they are vulnerable and/or how you can secure your system... Conclusion That's it for now. I hope you now have a good overview of these issues and how to protect. I can only urge you to take this problem seriously! Three or four minutes of leaving your laptop unattended are fully sufficient for an attacker to get a full forensic image of all your RAM contents for later analysis. This is at least as critical as the Cold Boot attacks, if not worse. I will follow-up with more articles about some more interesting details on these Firewire issues, how to use the above tools, and I'll report on some of the stuff I was able to find in RAM dumps gathered via Firewire...

• Websites that claim you'll never be able to get them taken down are quite easy to get taken down
• Legal threats are an excellent way of obtaining information
• The IP address used to subscribe me (and several others) to a vast number of mailing lists was 68.57.223.4. Which seems to belong to Ryan Farmer. "Fucking hero", my arse.

Edited to add:

07-08-2008 03:25:56 -!- AlmightyCthulhu!n=Almighty@c-68-57-223-4.hsd1.in.comcast.net has joined #ubuntu-kernel

$ host c-68-57-223-4.hsd1.in.comcast.net
c-68-57-223-4.hsd1.in.comcast.net has address 68.57.223.4

Just in case anyone thought I was being unfair here.

I occasionally like to fool around with statistics gathered from the Internet. Sometimes, to produce something like langpop.com, which, even if it's unscientific, I feel is useful, and more or less reflects my gut feeling about which way the wind is blowing. Other times, it's fun just to grab some numbers, add them up, and not worry too much if they really have any validity or meaning. In this case, I suspect there's something to them. What do you think? The technique: take different job search sites, like monster.com, craigslist, and so on, search them for various languages (with Yahoo's search API), count the hits, and then look at the ratios. For instance, Java jobs to Ruby jobs, with the idea that, painting with very broad brush strokes, the Java jobs are going to be more "enterprisy", and the Ruby jobs hipper, cooler, and maybe gone six months from now as the economy tanks and funding dries up. Rough techniques notwithstanding, there do seem to be two distinct groups of sites, one with lower rations, the other with higher ratios (more Ruby jobs compared to Java jobs). For fun, I also included Python and Erlang, although there are very few Erlang jobs out there. Of course, it's also true that the bigger sites, like dice.com, had more jobs total. Indeed, dice.com has more hits for Erlang than CrunchBoard does for Java!

Last month, the dutch-languaged C'T featured an article about retouching pictures with The Gimp, amongst others. Great timing, of course, since everyone goes on holiday during summer, right? Right. It's amazing what you can do with The Gimp and a bit of basic knowledge. Take this picture, for example, which I took at this year's FOSDEM: I kindof like it, but the colors are a bit blend. Too much of a red tint over everything. So, what we need to do, is change the curve: Pardon my, eh, Dutch. The trick is to remember that there are only so many colors which can be represented in a (JPEG) photo. The histogram on the background represents that; and by modifying the curve that is above the picture, you can effectively change the colorspace that is available for any peaks. If there's a lot of the darker blue in the picture, but less so in the lighter regions of the blue (as in this case), you just give the picture more space in the blue regions so that the blue that actually exists in the picture can be better represented. We do this by lowering the curve right before the peak, and by raising it afterwards, as the above screenshot shows. After doing so for red, green, and blue, this is the result: Looks much better, right? Way more contrast, and the colors look much more natural. And that only took me like 10 seconds to do. Do watch out, though -- it's easy to overshoot, resulting in an ugly picture; and you wouldn't necessarily want that. Also, if your camera has a RAW mode, you're going to be able to do a lot more of this kind of retouching with that than you would be using JPEG files. Then again, my camera has RAW too, but I don't use it most of the time...

Debian’s freeze sounds like a technical hack to address a social problem, and that disturbs me a bit. The social problem is: At some point, we need everybody in Debian to make only non-disruptive changes, so everything can converge very fast into a releasable state. The “solution” we are using is that we are blocking all packages from migrating to testing, and requiring manual review from someone on the release team. Consequences are:
- many people feel that you need to be very convincing to fix a small, not RC bug, even if fixing that bug definitely increases your package’s quality.
- the release team is completely overwhelmed by unblock requests during the freeze
- many people just stop trying to fix things during the freeze (which definitely doesn’t improve Debian’s quality), both because they think it’s hard to get a fix in, and because they don’t want to bother the release team I wonder if we really need such a strict policy. Are there other Free Software projects that use such a technical measure to prevent software from disrupting stable releases? I am the impression that most other projects rely on social pressure instead of technical measures for that, except maybe during the last few hours before the release. Couldn’t we act on the social level? We could default to allow everyone’s package to migrate to testing, and, when someone fucks up and uploads something that should not have been uploaded, block all his packages (switching to manual review mode) until the release. Of course, that require the release team to make decisions about _people_, which is harder than making decisions about _packages_. But if the rules are clearly stated, couldn’t this work?

For some time people have been telling me about the benefits of SpamAssassin (SA). I have installed it once for a client (at their demand and against my recommendation) but was not satisfied with the result (managing the spam folder was too complex for their users). The typical configuration of SA has it run after mail has been accepted by the server. Messages that it regards as spam are put into a spam folder. This means that when someone phones you about some important message you didn’t receive then you have to check that folder. Someone who sends mail to a user who has such a SA configuration can not expect that the message will either be received or rejected (thus giving them a bounce message). Even worse it seems to be quite common for technical users to train the Bayesian part of SA on messages from the spam folder - without reviewing them! Submitting a folder of spam that has been carefully reviewed for Bayesian training can increase the accuracy of classification (including taking account for locality and language differences in spam). Submitting a folder which is not reviewed means that when a false-positive gets into that folder (which will eventually happen) it is used as training for spam recognition thus increasing the incidence of false-positives! Spam has been becoming more of a problem for me recently, on a typical day between 20 and 40 spam messages would get past the array of DNSBL services I use and be re-sent to pass the grey-listing. Also I have been receiving complaints from people who want to send email to me about some of the DNSBL and RHSBL services I use (the rfc-ignorant.org service gets a lot of complaints - there are a huge number of ignorant and lazy people running mail servers). So now I have installed spamassassin-milter to have SA run during the SMTP protocol. Then if SA checks indicate that the message is SPAM my mail server can just reject the message with a 55x which will cause the sending mail server to generate a local bounce (if it’s a legitimate message) or to just be discard it in the case of a spam server. Here is how to set it up on Debian/Lenny and CentOS 5: Install the package yum install spamass-milter or apt-get install spamass-milter spamassassin spamc (spamassassin seems to be installed by default on CentOS). On a Debian system the milter will be setup and running. On CentOS you have to run the following commands:
useradd -m -c "Spamassassin Milter" -s /bin/false spamass-milter
mkdir /var/run/spamass-milter
chown spamass-milter /var/run/spamass-milter
chmod 711 /var/run/spamass-milter
echo SOCKET="/var/run/spamass-milter/spamass.sock" >> /etc/sysconfig/spamass-milter On CentOS edit /etc/init.d/spamass-milter and change the daemon start line to ‘runuser - spamass-milter -s /bin/bash -c "/usr/sbin/spamass-milter -p $SOCKET -f $EXTRA_FLAGS"‘ Then add the following lines below it:
chown postfix:postfix /var/run/spamass-milter/spamass.sock
chmod 660 /var/run/spamass-milter/spamass.sock The spamass-milter program talks to the SpamAssassin daemon spamd. On both Debian and CentOS run the command “useradd -c Spamassassin -m -s /bin/false spamassassin” to create an account for SA. The Debian bug #486914 [1] has a request to have SA not run as root by default. On CentOS it seems that SA wants to use a directory under the spamass-milter home directory, the following commands alllow this. It would be good to have it not do that, or maybe it would be better to have the one Unix account used for SA and the milter.
chmod 711 ~spamass-milter
mkdir ~spamassassin/.spamassassin
chown spamassassin ~spamassassin/.spamassassin On Debian edit the file /etc/default/spamassassin and add “-u spamassassin -g spamassassin” to the OPTIONS line. On CentOS edit the file /etc/sysconfig/spamassassin and add “-u spamassassin -g spamassassin” to the SPAMDOPTIONS line. To enable the daemons, on CentOS you need to run “chkconfig spamass-milter on ; chkconfig spamassassin on“, on Debian edit the file /etc/default/spamassassin and set ENABLED=1. Now start the daemons, on CentOS use the command “service spamassassin start ; service spamass-milter start“, on Debian use the command “/etc/init.d/spamassassin start“. Now you have to edit the mail server configuration, for Postfix on CentOS the command “postconf -e smtpd_milters=unix:/var/run/spamass-milter/spamass.sock” will do it, for Postfix on Debian the command “postconf -e smtpd_milters=unix:/var/spool/postfix/spamass/spamass.sock” will do it. Now restart Postfix and it should be working. For correct operation you need to ensure that the score needed for a bounce is specified as the same number in both the spamass-milter and SA configuration. If you have a lower number for the spamass-milter configuration (as is the default in Debian) then bounces can be generated - you should never generate a bounce for a spam. The config file /etc/default/spamass-milter allows you to specify the score for rejecting mail, I am currently using a score of 5. Any changes to the score need matching changes to /etc/mail/spamassassin/local.cf (which has a default required_score of 5 in Debian). You can grep for “spamd..result..Y” in your mail log to see entries for messages that were rejected. One problem that I have with this configuration on Debian (not on CentOS) is that spamd is logging messages such as “spamd: handle_user unable to find user: ‘russell’“. I don’t want it to look for ~russell when processing mail for russell@coker.com.au because I have a virtual domain set up and the delivery mailbox has a different name. Ideally I could configure it to know the mapping between users and mailboxes (maybe by parsing the /etc/postfix/virtual.db file). But having it simply not attempt to access per-user configuration would be good too. Any suggestions would be appreciated. Now that I have SpamAssassin running it seems that I am getting about 5 spams a day, the difference is significant. The next thing I will do is make some of the DNSBL checks that are prone to false-positives become SpamAssassin scores instead. When I started writing this post I was not planning to compare the sys-admin experiences of CentOS and Debian. But it does seem that there is less work involved in the task of installing Debian packages.

• [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=486914

Share This

Somehow (thankfully) I've managed to avoid much to do with backups in my career as a sysadmin. It's always been someone else's responsibility, that's been fine by me. For some reason, I've been more paranoid lately, and with all of the crazy shit I've been doing to daedalus, I wanted to have a backup, just in case. Maybe it's old age. So a while ago, I went and had a stab at setting up Amanda, since pretty much whenever I've been near a bunch of sysadmins and someone asks for a recommendation on what to use, "Amanda" is the answer. I jumped through all of the hoops to have caesar be the backup server, and daedalus be a client. I set up a virtual tape library on caesar. I think I then lost interest for a while, until I was going to upgrade daedalus to Etch, and I wanted to have a successful backup first. I had the holding disk and the virtual tape library on the same filesystem, and it was nice and big, yet I kept running into this error about insufficient space. "dump larger than available tape space" was the complaint. I tried making the filesystem larger. I tried adding more virtual tapes. Nothing would placate it. After another round of Googling, I discovered that the problem was because Amanda would only only span a maximum of four tapes, and the filesystem I was trying to backup was way bigger than that. So I told it to use up to 10 tapes in a run (runtapes 10) and then lo and behold, I got a successful backup out of the sucker. So that was mostly all good, but occasionally a backup would fail, or fail once an succeed on a retry, because tar exited non-zero, with "file changed as we read it". I did some more Googling tonight, and discovered this Amanda/GNU tar compatibility matrix. It turns out that the version of tar and the version of Amanda shipped in Debian Etch aren't compatible with each other. Suck! Oh wait, they're maintained by the same person! Argh! So aside from the occasional backup failing due to a file being written to at the time of the backup, it's working fairly well. My DSL link isn't the fastest, so a full backup of daedalus can take over 48 hours to complete, but I'm not paying for inbound bandwidth at home, so it's not a big deal. I initially thought that to do a restore, I'd have to run amrecover on daedalus, and then I'd have to stream the contents of multiple tapes back to it at the appalling uplink speed of my DSL connection, but fortunately amrecover has a sethost command, so I can restore just the files I need on another machine local to caesar at LAN speeds, and just transfer the files I restore to daedalus. So that's all pretty neat. I quite like the FTP-client-like interface of amrecover. It's cool how you can use setdate and just go back in time. I had the opportunity to do a test restore recently when one of my users nuked his Maildir directory by accident. Unfortunately I was unable to recover the needed files in that case, and I'm not entirely sure what the problem was, so there's some more testing to be done before I'll be fully confident that I can rely on it. There's a lot of configuration options available as well, and I think I got the config I'm currently using from a template or something, so there's probably a fair bit of tweaking I can do. Oh, and I'm constantly reminded of this quote from a previous life, where technical details bubbled too far up non-technical management:

"Who is Amanda, and what is her role in the backups?"

In a discussion which covered some of the differences between Linux and Windows, a Windows using friend asked me if I felt compassion for Windows users. I feel some compassion for people who have bad working environments. While using an operating system that has poor support for the business tasks does decrease the quality of the working environment, there are bigger issues. For example a while ago I was doing some sys-admin work for a financial organisation. I had to use Windows for running the SSH client to connect to Linux servers, this was annoying and decreased my productivity due to the inability to script connections etc. My productivity was also decreased because of my unfamiliarity with the Windows environment, it seems reasonable to assume that when you hire a Linux sys-admin they will have some experience of Linux on the desktop and be quite productive with a Linux desktop system - while the same can not be said for a Windows desktop. But what really made the working environment awful was the paperwork and the procedures. If a server doesn’t work properly and someone says “please fix it now” and I only have a VT100 terminal then I’ll be reasonably happy with that work environment (really - I wouldn’t mind a contract where the only thing on my desk was a VT100 connected to a Linux server). But when a server process hangs in the same way several times a week, when the cause of the problem is known and the fix (restarting the process) is known it really pains me to have to wait for a management discussion about the scope of the problem before restarting it. But I don’t have a great deal of sympathy for people who end up in bad working environments such as the one I was briefly in. Anyone who is capable of getting such a job is capable of getting a job with a better working environment while still earning significantly more than the median income. The people I feel sorry for are the ones who work on the minimum wage. I don’t think that the difference between Linux and Windows on the desktop would matter much if you were getting the minimum wage, and people who are on the minimum wage don’t have a lot of choice in regard to employment (I think that all options for them suck). I don’t have much sympathy for adults who use Windows at home. I have to admit that there are some benefits to running Windows at home, mainly that the hardware vendors support it better (few companies sell PCs with Linux pre-loaded) and there are some commercial games which are in some ways better than the free games (of course there are more than enough free Linux games to waste all your time - and some games are best suited to a console). Linux has significantly lower hardware requirements than Windows (my main machine which I am using to write this blog post is more than three years old and has less power than any other machine on sale today apart from some ultra-mobile PCs), so any long-term Windows user can install Linux on one of their machines which lacks the resources to run the latest version of Windows. The only Windows users for whom I have much sympathy are children. When I was young every PC came with a BASIC interpreter and everyone shared source code. Books were published which taught children how to program in BASIC which included fairly complete example programs. For the cases where proprietary software was needed the prices used to be quite low (admittedly the programs were much less complex - so pricing is probably in line with the effort or writing the code). Now it seems that computers are often being provided to children as closed systems that they can’t manipulate, the web browser has replaced the TV. I believe that Linux is the ideal OS for a child to use. There is a wide range of free educational programs (including kturtle - the traditional Logo turtle) and there are also a range of free powerful programs which can be used by any child. Few parents would buy Photoshop or Adobe Illustrator for a child to play with, but anyone can give a child a $100 PC with GIMP and Inkscape installed. They might as well give 3yo children access to the GIMP - it will be less messy than fingerpainting! I expect that some parents would not consider Linux for their children because they don’t know how to use it. Fortunately Linux is easy enough to use that a child can install it without effort. Some time ago the 11yo daughter of a friend who was visiting asked if she could play some computer games. I gave her a Fedora CD and one of the PCs from my test lab and told her that she had to install the OS first. Within a small amount of time she had Fedora installed and was playing games. While the games she played were not particularly educational, the fact that she could install the OS on a computer was a useful lesson. It seems to me that children who are raised as Windows users are less likely to learn how computers work or be able to control them properly. I expect that on average a child who is raised in such a manner will have fewer career options in today’s environment than one who properly understands and controls computers.

Share This

One of the things I've wanted to do since I discovered that Caltrain (and the VTA light rail) are well-equipped to take bikes, was go for an outing further afield on our bikes using one of these methods of public transport. So we finally got around to doing that yesterday, when we threw our bikes on the train, and BART, and went for a ride around Golden Gate Park. The grand plan was to bike to the Caltrain station, take Caltrain to Milbrae, BART to downtown, then bike from there to the park, and then catch an afternoon showing of The Dark Knight in IMAX at the Metreon, then bike back to the BART station, and reverse the whole thing. We had a minor hiccup in that Sarah had just gotten her bike back from a friend she'd lent it to, and as we were about to head to the Caltrain station, discovered that it a broken spoke. She managed to extract the spoke and we got to the station with a couple of minutes to spare. It turned out that Mike's Bikes were fairly close to the BART station, so we stopped in there on way to the park, and they were able to fix the spoke on the spot in 20 minutes for the princely sum of $21. Unfortunately the weather was pretty typical summer San Francisco weather, and it was quite cold and foggy. It didn't get any better as we got closer to the park (and the coast), so the ride around the park itself wasn't terribly exciting. The hill we had to get over (Hayes Street) was a bit of a slog. On the way back we saw the filming of a Japanese Nissan commercial (complete with right-hand drive car). I'd done some research into bike parking beforehand, and found that all of the parking garages are required to provide bicycle parking, and there's a pretty comprehensive list. So we parked our bikes in the Moscone Center garage, which was about a block from the Metreon, and went and caught the movie. The Dark Knight was really good. Very... dark. I really liked Ledger's portrayal of the Joker, much better than the original. I wonder if they'll just keep re-imagining the Batman movies over and over? There was pretty good continuity from Batman Begins. I thought the voice of Batman was a bit ridiculous though, at least initially. You kind of got used to it after a while. Anyway, we got back to the parking garage, and some lowlife had cut through the cable lock that we'd used to lock our bikes to each other and the rack, and stolen Sarah's bike! Annoyed does not begin to describe it. We trundled off the seven or eight blocks to the Hall of Justice to file a report. The SFPD weren't the least bit interested, really. The somewhat astounding thing was the bike racks were right behind the cashier's office, and there was video surveillance, and someone still managed to just walk right in, cut the cable, and walk out with the bike. We checked with the duty manager, and he reviewed the video footage and said they have footage of him walking in and then walking out with the bike. So hopefully the police will get hold that. We have to see if we can dredge up the purchase documentation for the bike and see if the serial number is recorded. I have no idea if we'll ever see the bike again. Our renter's insurance may cover it, but I suspect the deductible won't make it worth our while to make a claim. I have to say (and it's not like it's a recent discovery or anything) that the Bay Area's public transportation is a joke. It cost us $32 to go from Mountain View to Downtown San Francisco, via Caltrain and then BART. Our Prius takes about 11 gallons at worst, and with gas prices being at say $4.30 a gallon, that's about $47 a tank. We get about 400 miles out of a tank usually, and it's vaguely 40 miles from home to the guts of Golden Gate Park. So it's cheaper, and faster to drive to San Francisco than it is to use a bicycle and public transport. Particularly when you factor in the risk of theft of your bicycle. Our next bike lock will be one of those fancy Kevlar ones.

PTS, package descriptions (and more than a bit of UDD) After delivering my talk at DebConf, I went back to some (more) PTS hacking ... apparently I'm in the mood for it An apparently trivial feature request (showing package descriptions) provided for interesting thoughts. First of all the PTS is mostly source-based, but in Debian we have no source package descriptions (issue which probably deserves fixing by its own). Rough consensus has then been to add tooltips on binary package hyperlinks, to show short descriptions. Still, adding the import of Packages sounded like overkilling, not mentioning duplicated wrt packages.d.o. Hence I decided to be the first jumping on the UDD boat (wonderful GSOC project which I had the pleasure to co-mentor together with Lucas and HE) for routinely usage. Retrieving the most recent short description of each binary package at each PTS run is as easy as:

select distinct packages.package, packages.description
from packages,
  (select package, max(version) from packages group by package)
    as latest (package, version)
where packages.package = latest.package
  and packages.version = latest.version ;

To solve the more generic problem of the PTS missing a generic description (or, if you want, to mitigate the problem that in Debian we are missing a source package description), Lucas came up with the intriguing idea of showing the description of the most popular binary package, according to popocon. Piece of cake:

select distinct packages.package, popcon.insts, packages.description
from packages, popcon,
  (select package, max(version) from packages group by package)
    as latest (package, version)
where packages.package = latest.package
  and packages.version = latest.version 
  and packages.package = popcon.package ;

(same as before, but adding popularity: one has both all short descriptions, and a way to spot the "canonical" one) Long life to UDD. (BTW if you are interested in UDD, and you happen to be at DebConf8, do not miss tomorrow's QA BOF.)

One thing that I would like to get fixed for Lenny is the shared objects which can reduce the security of a system. Almost a year ago I blogged about the libsmpeg0 library which is listed as requiring an executable stack [1]. I submitted a two-line patch which fixes the problem while making no code changes (the patch gives the same result as running “execstack -c” on the resulting shared object). My previous post documents the results of the problem when running SE Linux (a process is not permitted to run and an AVC message is logged). Some people might incorrectly think that this is merely a SE Linux functionality issue. The program paxtest (which is in Debian but is i386 only) tests for a variety of kernel security features in terms of memory management. To demonstrate the problem that is caused by this issue I ran the commands “paxtest kiddie” and “LD_PRELOAD=/usr/lib/libsmpeg-0.4.so.0 paxtest kiddie“. The difference is that the test named “Executable stack” returns a result of Vulnerable when the object is loaded. This means for example that attacks which rely on an executable stack will be permitted if the libsmpeg-0.4.so.0 shared object is loaded. So for example a program that loads the library and which takes data from the Internet (EG FreeCiv in network mode) will become vulnerable to attacks which rely on an executable stack because of this bug! My Etch SE Linux repository has had a libsmpeg0 package which fixes this bug on i386 for almost a year [2] (the AMD64 packages are more recent). I have now added packages to fix this bug to my Lenny SE Linux repository [3]. I have also volunteered to NMU the package for Lenny. It seems that it would be rather embarrassing for everyone concerned systems were vulnerable to attack because of a two-line patch not being applied for almost a year. I expect that the Release Team will be very accepting of package updates for Lenny which have patches to address this issue. A patch that has one line per assembler file (in the worst-case) to mark the object code is very easy to review. The results of the patch can be tested easily, and failure to have such a patch opens potential security holes. Package maintainers who can’t fix the assembly code can always run “execstack -c” in the build scripts to give the same result. Lintian performs checks for executable stacks and the results are archived here [4]. There are currently 36 packages which contain binaries listed as needing executable stacks, I would be surprised if more than 6 of them actually contain shared objects that need an executable stack. If you use a package that is on that list then please test whether an executable stack is required by running “execstack -c” on the shared object and see if it still works. If a test of most of the high-level operations of the program in question can be completed successfully without an executable stack then it’s a strong indication that it’s not needed. Note that execstack is in the prelink package. I am happy to help with writing the patches to the packages and using my repositories to distribute the packages, but am not going to do so unless I can work with someone who uses the program in question and can test it’s functions. As an example of such testing I played a game of Frozen Bubble to test out the libsmpeg0 patch.

• [1] http://etbe.coker.com.au/2007/10/07/executable-stack-and-shared-objects/


• [2] http://etbe.coker.com.au/2007/10/19/my-se-linux-etch-repository/


• [3] http://doc.coker.com.au/computers/installing-se-linux-on-lenny/


• [4] http://lintian.debian.org/tags/shlib-with-executable-stack.html

Share This

Today is the day we start this years Assassins game, and I (together with the two other DebConf admins Mark and Stephen) have the role of the referee. As we all don’t want to do it by hand, and have seen bad examples in the past where the referee just wasn’t up to the task, we go another way. In the last two days, Stephen and me (mostly Stephen) developed a Pentabarf controller to do the work for us. It is ready for the game, so we start today. This controller basically tells the players their target, gets the “I killed em” notice and lets the target confirm it, then assigning a new target. And when we are done with the game - it even tells us the winner and also some information about the other players. This controller made us able to do another change to the known game rules: People can enter the game when it is already running. Which is useful, as now we only let people play that arrived at the venue. Late-comers get added to the players list whenever someone got killed. This ensures that no assassin has to wait for half the Conference time until their target finally arrives! (Ok, it might mean that Late-Comers have to wait a day or two before they finally get a target, as we only insert new players when someone dies). One more note if you are playing: Don’t tell anyone, not even your assassin, your target. Your assassin might not neccessarily get your target as his next (if we add people to the set), so keep it secret, all the time. The only people you should talk to about this game are us 3 admins, but only in case you have a problem. :) Comments: 0

Do you know code that compare versions of debian packages (for example, that knows if 2:23.2.3~rc1-1 is lower than 2:23.2.3-2?), besides dpkg –compare-versions? If yes, please write a comment to this blog post, preferably with a link to the code. Also, did someone already write a test suite for that? Who would be interested in such a test suite? I’m considering writing a function in PL/SQL to compare debian versions (for the Ultimate Debian Database project). If someone already wrote that, I’m interested as well.

more PTS hacking: mentors.d.n and gift/help bugs Some more PTS hacking at DebConf8. Among about 20 bugs triaged/fixed thus far, 2 are peculiar and integrate some more info in PTS package pages.

• Pending mentors.d.n uploads (bug #463050) in need of sponsoring, are now shown as TODO items. This way the maintainer has better chances to not overlook them.

A minor glitch is currently affecting this though: when pending uploads on mentors.d.n are not uploaded, but rather superseded by new versions in unstable, they are currently not removed automatically from mentors.d.n and will thus show up in the PTS. Apparently this is going to be fixed soon on mentors.d.n, so hopefully this bug won't have time to bother anyone.

• Highlight of gift/help bugs (bug #469627). We know the drill about bugs tagged help. They are now shown as a single item in the "problems" section of the PTS, in the hope to attract attention on them from people other than the package maintainer.

Bugs (user-)tagged gift on the other hand are a rather new (and thus far neglected) QA feature proposed by Lucas. Packages with gift bugs have now them shown as an extra line in the bug summary box. The idea is to point contributors to PTS pages to discover if some of the packages they are interested in have gifts for them. As an extra gift (pun, intended) I've written for the PTS a tiny Python script to retrieve bug summaries via SOAP. The lookup is via tag, supporting both official tags and usertags. In case it can be useful to someone else, I'm putting it up for download. The PTS is using tagged bugs.py it as follows:

$ tagged_bugs.py "gift" "debian-qa@lists.debian.org" > bugs.gift.txt
$ tagged_bugs.py "help" > bugs.gift.txt