13 Mar 2009
White Swan, Black Swan: nice vocals, I only caught a couple songs

The Donkeys
Decent band, fun to listen to. A good whale joke:
Two whales are sitting at a bar
one says to the other [long whale noises]
The other says, "man you are trashed."

Quick to get on stage and play

Phosphoressence
Good country rock band. Slow to start playing and some assholes decided to pop into the front of the crowd, they were tall so they blocked a bunch of peoples view. Then they gained an extra 3 people and started moving backwards. For once someone actually got inside my bubble.

7 Mar 2008 -- Red Devil Lounge
* "Three of" (good luck googling these guys) -- very good
* A, b, and C -- not good, the lead singers voice cracked a lot, they did look like they should be in the movie "That thing you do" (my first date was at this movie).
* Moonlight Sexy -- ok, Andy didn't like these guys
* Marcy Playground -- Great!

14 Mar 2008 -- Writers with drinks
S.G Browne with Breathers: A zombie's lament. The chapter he read was really funny. I think this was the best part of the night. The host was also really good, but the host could be described as slightly scripted talking out of her ass.

And there we have done it - new sections in the archive. Read more about it in my d-d-a post and the whole set of affected packages is also available - and updated slightly, as some small things slipped through. The whole process was less painful than feared. After some discussion, started a few weeks ago we finally went and coordinated the change today. There was no reliable and good way for changes that large, so we had to create our own way. (The last section changes had been done ages ago). Mike provided a little python script which was able to parse our input file with the section matches against our database, so we ended up with a list we could possibly feed back into the database. Except, not really. Thanks to various things the list generated needed some manual work (which I also managed to fuck up a little), but after that was done we were ready to go. And while Mark told the wanna-build admins about the new sections (for some reason our buildd network cares about sections) and prepared a patch for our policy, I prepared the mail for the d-d-a list. Then we all had a break, as a dinstall was still running, and we did not want to break that. The actual changes then took only a few seconds, sending the mail to d-d-a took longer, thankyou gnupg. :) I wonder how much afterwork this will generate. There are at least the policy changes, lintian, the packages control files to adjust. Also some websites, where Frank already changed packages.debian.org to know about the new names. (Before he fixed up me forgetting some PHP packages to move. This way all of the active ftpteam members did something in it :) ).

---

In other news, we are currently stress-testing our ftp-master host a little. Well, not really, it could take more, but right now we are scanning the full archive once, importing all filenames from all our Debian packages. The goal is to get projectb larger. Err, no, actually we do want to get rid of apt-ftparchive, as that is one of the worst tools in our dinstall runs. It is slow as hell in the mode it has to run for us, making up a large part of our runtime. Step 1 to get rid of it is to generate the content files on our own, and before we can do that we have to import them all Which means we have to unpack every binary package once, reading in all filenames to store them in our database. Thats a lot of data. Currently the associations table, which is nothing more than 4 integers per row, takes up 972MB, and import is still running. (The index on the binary pkg id in this table alone has 700MB). And this table does nothing more than linking the binary package to the filename and filepath. Which are two extra tables, with the filenames having only 87MB, the path entries only taking up 30MB. Yes, anyone using databases will tell me thats not much, and it really isn t that much. It s just that the largest table before this import had 120MB in it. And some of the others used more space for their indexes than for the actual data. :) Anyway, this is only step one on our way to get rid of the slowness apt-ftparchive is. When the contents all work as we want them, the next step will be to get rid of it generating Packages/Sources files. Which means we have some more data to store in projectb. And after that - we also intend to do a Contents file for our source. Each and every source file listed. Guess we really will end up with some more large tables. :) The final goal and main reason why we do this is to save us at least 60 minutes runtime per dinstall run. My own guess is that Contents alone will get us about 20 minutes down. No, before someone wants to yell at me - apt-ftparchive can t be changed as much to make this possible and keep using it. Not in the way the generate option works. It has the disadvantage that it scans the files after they are installed in their pool/ location and stores that in its own database . This approach has to negative points:

• first, it is using another database (BerkeleyDB). We do have one database already, projectb, thank you.
• second, and more important, the files only end up in pool/ on dinstall time. Which means the whole scanning is done during dinstall. Which eats time. The new way will surely use the same amount of cpu time in total - but it will spread it over the whole day. 4 runs each hour. Not 4 runs each day. And during dinstall it will boil down to one sql query per installed package moving data from a pending table into the real contents table. Which is a cheap operation.

---

The title says some more, so here is some more: We aren t only working on what I wrote above. There is also work on changing our database access. Historically dak used the pg module to do database work. While the module itself does work, the way its used isn t really the best one. Way too much groundwork everywhere. Not nearly as much of the nice things used people say python could have. So another largish work area is getting rid of pg. Migrating to psycopg2. And putting a proper OO class around it, which in the end hopefully hides those dark details from the application. And to achieve even more sanity, we (well, honestly, Mark) are looking into SQL Alchemy usage. (We also considered storm from Canonical, but that sucks when you take a look at its documentation. (Which documentation?). SQLA won there easily.) While I m ranting: WTF are people neglecting their docs that much? Ever tried looking at the documentation for psycopg2? Forget it. No, a db api compliance test suite is NOT documentation. If there would be any better python module to connect to postgres, I would veto the usage of psycopg2, but there doesn t seem to be one. Sucks.

No, you re not reading it wrong but what about 1.1.2? , you ask. Well, we made a small mistake with the soversion there (if mistakes regarding soversion are ever small, that is =/), so we had to quickly release a fixed up version before packages start showing up for distributions with an incorrect library soname. Apologies. With this release we have a shiny new website at http://webkitgtk.org/, hosted by Igalia and designed by Christian Dywan, who s one of the contributors to WebKitGTK+ s code. This new release sports many interesting features and fixes. You may remember me talking about the download support initially contributed by Collabora, which I was fixing up and pushing for inclusion. Well, it is now available, and already implemented in Epiphany 2.27.0. Christian Dywan tells me he won t let Midori stay behind Epiphany on this one, so expect support for downloading correctly integrated in Midori as well. Some more pet peeves got fixed with this release, such as scrolling with keys such as space bar, home, end, not behaving the way people expect. We also found more improvements which are in order for libsoup, and we ll be pushing them, even though we have added small workarounds to WebKitGTK+ itself for the time being. We have added proper logging support for our port, providing a WEBKIT_BUILD environment variable. Documentation on how to use this will follow soonish, but for now you can look at the LoggingGTK.cpp file to know what values you can add to that variable. Also notice that only debug builds use logging to its full extent! WebKitGTK+ 1.1.1 has now hit Debian experimental, by the way, and I plan to have 1.1.3 uploaded early this week. An epiphany-webkit package based on 2.27 is in the works, and should be in soon, too. There seems to be lots of interest in having a PPA for Ubuntu, as well, so we may see that happening soonish. We plan to get more testing this way, and be able to uncover and fix more bugs, so that we can get a rock solid release ready in time for GNOME 2.28.

I have just been asked for advice about secure filesystem and decided to blog my answers. The first issue is what is meant by secure filesystem, that could either mean the ability to restrict file access (EG by supporting SE Linux security contexts and using SE Linux for file access control) or the ability to encrypt data in case the machine is stolen. For access control I recommend SE Linux of course. For encryption on a local machine I mostly use dm-crypt which is configured with the cryptsetup utility. I encrypt at the LVM logical volume level as it is common that there are some LVs that don t need to be encrypted. For files that need extra encryption or files that are shared between machines I use GPG. A question was asked about kernel vs user-space filesystem encryption. AES is in the kernel so there is no lack in terms of strong encryption there. Also performance is pretty good (in most cases the CPU is fast enough that the hard drive is the bottleneck). For fine grained encryption (such as some of the experimental filesystems that encrypt data separately for each user) user-space is probably the only way to go. If you want servers to be high-security level and protected from hackers or unauthorised people then it s difficult to offer any advice that is smaller than a text book. I suggest that if you have such questions then you should do one of two things. If you are running a corporate IT department then hire an expert who can help with determine your specific requirements and meet them. If you want to learn about computer security and run your own systems in the best way possible then read as much from the experts as possible. If you are looking for a project to contribute to related to security then if you choose SE Linux I could offer some specific advice on things that need work. I suggest not deciding on whether to do kernel level or user level work up front, but decide first which area of security you want to work on and then select a project which fits - then you should be able to determine whether your skills are best suited to kernel or user space coding. As for whether developing a new filesystem is necessary, I will note that SE Linux works well on Ext3 and XFS, it has just become usable on JFFS2, and it will work on other newer filesystems in the near future. Adding SE Linux support to a filesystem is not a difficult task if the filesystem supports XATTRs. I believe that there is a lot of scope for other access control systems to be developed which use XATTRs for security labels. I can t advise on e-books. I generally don t read books, I read blogs and papers. Anything that I read which I consider to be worth recommending will probably have a link from my blog.

This weekend I mostly fiddled around migrating machines from Xen hosting to KVM hosting. Ultimately it was largely a waste of time, due to various other factors. Still with a bit of luck it will be possible to move the machiens next week. That aside I spent a while updating my blogspam detection site. As a brief recap this site offers a simple XML-RPC service which allows you to test whether incoming blog comments are spam or not. Originally this was put together to fight an invasion of comments submited to the Debian Administration website: The site currently shows:

Site	Spam	Non-Spam	% spam
debian-administration.org	238	372	60.98% spam

Depressing. But not as depressing as the real live stats which show since I last reset the counters 36,995 spam comments vs. 1,206 non-spam comments. (live updating counters here) Anyway I updated the service today to add two new plugins, both of which are a little reactionary. The first new plugin is called "multilink" and is based upon the observation that spammers rarely know the markup of the site they are submitting comments to. This means you can frequently see submitted comments like this:

 <a href="http://spam.com">buy viagra</a>
 [url=http://spam.com]buy viagra[/url]
 [link=http://spam.com]buy me[/link]

Here we have three different styles of links - "a href", "link=", and "url=". I figure this is a clear indicator of a confused mind, or more likely a spammer. The second new plugin is designed to stop people who enter "<strong>" words. It is a little coarse but actuall zero false positives in the real world so I'm going to leave it live to see how it works out. In happier news I'm just back from a trip to the beach. Sand rocks. Even if it wasn't windy enough for my kite .. ObFilm: Dracula ("Bram Stoker's Dracula" - 1992)

I m in the middle of migrating a mail server away from the Cyrus mail store [1]. Cyrus provides a POP and IMAP server, a local delivery agent (accepting mail via LMTP). It is widely believed that Cyrus will give better performance than other mail stores, but according to a review by linux-magazin.de Dovecot and Courier deliver comparable (and sometimes better) performance [2]. The biggest problem with Cyrus is that it is totally incompatible with the Unix way. This wouldn t be a problem if it would just work and if it would display reasonable error messages when it failed, but it doesn t. It often refuses to work as desired, gives no good explanation, and it s data structures can t be easily manipulated. Dovecot [3] and Courier [4] use the Maildir++ format [5] (as well as many other programs). I have set up a system with Courier Maildrop and Dovecot for the IMAP server [6] and it works well - it s good to have a choice! But also Maildir++ is reasonably well documented and is an extension to the well known Maildir format. This means that it s easy to manipulate things if necessary, I can use mv to rename folders and rm to remove them. Cyrus starts with a database (Berkeley DB file) of all folders in all mailboxes. Therefore it is not possible to move a user from one back-end server to another by merely copying all the files across and changing the LDAP (or whatever else contains the primary authentication data) to point to the new one. It also makes it impossible to add or remove folders by using maildirmake or rm -rf. The defined way of creating, deleting, and modifying mailboxes is through IMAP. One of the problems with this is that copying a mailbox from one server to another requires writing a program to open IMAP connections to both servers at once (tar piped through netcat is much faster and easier). Also if you need to rename a mailbox that contains many gigabytes of mail then it will be a time consuming process (as opposed to a fraction of a second for mv). Cyrus has a tendency to break while Dovecot is documented as being self-healing and Cyrus also seems to cope well in the fact of a corrupted mail store. Even manually repairing problems with Cyrus is a painful exercise. The Cyrus mail store is also badly designed - and it s design was worse for older filesystems (which were common when it was first released) than it is for modern ones. The top level of a Cyrus maildir contains all the messages in the INBOX stored one per file, as well as three files containing Cyrus indexes and sub-directories for each of the sub-folders. So if I want to discover what sub-folders a mailbox has then I can run ls and wait for it to stat every file in the directory or I can use an IMAP client (which takes more configuration time). As opposed to a Maildir++ store where every file that contains a message is stored in a folder subdirectory named new , cur , or tmp which means that I can run ls on the main directory of the mail store and get a short (and quick) result. Using tools such as ls to investigate the operation of a server is standard practice for a sysadmin, it should work well! A finall disadvantage of Cyrus seems to have many small and annoying bugs (such as the reconstruct program not correctly recursing the sub folders). I guess it s because not many people use Cyrus that such things don t get fixed. One trivial advantage of Cyrus is that by default it splits users into different sub-directories for the first letter of the account name. Dovecot supports using a hash of the user-name this is better than splitting by first-letter for performance (it gives a more equal distribution) but will make it slightly more difficult to manipulate the mail store by script. Ext3 can give decent performance without a two level directory structure for as many as 31,998 sub-directories (the maximum that it will support) due to directory indexing and Linux caching of dentries. There may be some other advantages of Cyrus, but I can t think of them at the moment. Here is a script I wrote to convert Cyrus mail boxes to Maildir++. To make this usable for a different site would require substituting a different domain name for example.com (or writing extra code to handle multiple domains) and inserting commands to modify a database or directory with the new server name. There is no chance of directly using this script on another system, but it should give some ideas for people performing similar tasks.

#!/bin/bash -e
# $1 is the user to move # BASEDIR is the directory for temporary files
BASEDIR=/mail/new
rm -rf $BASEDIR/$1 archive.tgz archive2.tgz ORIGSRV=$(echo "MYSQL_COMMAND_TO_GET_SERVER" mysql dbname)
echo "original server was $ORIGSRV" if [ "$ORIGSRV" = "" ]; then
echo "No server for account, strange error."
exit 1
fi ORIGPARENT="/mail/domain/e/example.com/$(echo $1 cut -c1)/user" # copy the data to a local archive
# USERDIR is for using ^ instead of . as Cyrus does.
mkdir -p $BASEDIR
cd $BASEDIR
USERDIR=$(echo $1 sed -e "s/\./^/g")
ssh cyrus@$ORIGSRV "(cd $ORIGPARENT ; tar czf - $USERDIR)" > archive.tgz
tar xzf archive.tgz
if [ "$USERDIR" != "$1" ]; then
mv $USERDIR $1
fi DEPTH=$(find . -mindepth 3 -type d wc -l)
if [ "$DEPTH" != "0" ]; then
echo "too deep"
exit 1
fi # remove cyrus index files
find $1 -name "cyrus.*" -exec rm \;
cd $1 # "mail" is a directory with sub-folders, every other name starting with
# [a-zA-Z] is a regular folder that needs to become a Maildir
for n in [a-zA-Z]* ; do
if [ "$n" != "mail" ]; then
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" ".$n"
fi
done # move the inbox files to Maildir storage
mkdir new cur tmp
mv [0-9]* new true
mv mail/[0-9]* cur true # move sub-folders of "mail/" and merge ones with duplicate names
cd mail
for n in * ; do
if [ -e "../.$n" ]; then
cd "$n"
mv [0-9]* "../../.$n/cur" true
cd .. ; rmdir "$n"
else
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" "../.$n"
fi
done
cd ..
rmdir mail # get a list of all names so we can recognise duplicates
find . -type d egrep -v "new cur tmp" cd $BASEDIR ssh root@localhost "mv /mail/new/$1 /mail/example.com ; chown -R mail:mail /mail/example.com/$1"
echo "MYSQL_COMMAND_TO_SET_SERVER" mysql dbname

• [1] http://cyrusimap.web.cmu.edu/


• [2] http://www.linux-magazin.de/heft_abo/ausgaben/2007/06/auf_der_teststrecke/


• [3] http://www.dovecot.org/


• [4] http://www.courier-mta.org/imap/


• [5] http://wiki.dovecot.org/MailboxFormat/Maildir


• [6] http://etbe.coker.com.au/2009/03/10/maildrop-imap-postfixadmin/

Excellent summary of where we find the newspaper business today:

Round and round this goes, with the people committed to saving newspapers demanding to know If the old model is broken, what will work in its place? To which the answer is: Nothing. Nothing will work. There is no general model for newspapers to replace the one the internet just broke.

Oops 3/16/09. Shirky, not Sharky.

For some time ago I ve been pondering about this question. As long as GSoC 2009 is about to start and people are looking for project ideas, I m posting here a very preliminar draft of my findings and an idea for a posible software project. It s pretty written on-the-fly while I ve managed to have some minutes between work and uni. Rigurous wording isn t P1. Debian s social contract 4th item states:

Our priorities are our users and free software We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities.

When we make decisions, either technical, legal and others regarding the operating system we deliver, this is one of the most referred argument. Altough that s seen as high priority often we don t have ways to properly know what our users want. While working on translations on the Spanish team we ve faced this situation. Do our users find them useful? they are able to understand them? they find that a much familiar wording would be bette than high technical details? We didn t know. Then, the question still exists. Altough Debian has ways for users to provide feedback (BTS, mailing lists, IRC, popcon, etc) none of them is designed to offer a way for the user to provide feedback on features, development roadmap and other non-bug aspects that developers and the project can tally and use for prioritize on releases. It s rather amusing to note that one of the most valuable assets free software community has is user involvement and contribution. But before we enter in details let s start on the basics. For using this information we first need to know who our users are? do we? From my findings and interaction with the community I ve identified two kinds of Debian users who can be clearly named. 1) The derivatives or pure blends, who use the Debian base and framework to build niche distributions. 2) The lead users (using E. Von Hippel s definition), who are the developers, contributors and a group of users. Debian is, if not the only, one of the projects who is best for fostering lead users. Most of them at some point involve themselves in the development process and some, later become developers. There is a more diverse group who are end-users but it s unclear to me how we can group them. Despite that they probably represent the biggest part of the pie. So, how we do please everyone? can we? In my opinion we can, at some degree. By implementing tools for first: gather user feedback, make statistical data, tally, we later can take informed decisions. In Debian decisions are voted and the set of people who votes are only developers with their own constraints. I think this would ease the constant that release-decisions-regarding-foo-tech-legal-issue represented on each release iteration, saving time, health and bits. It could be useful for knowing more about who are the ones we develop things for. I m thinking of some sort of Dell s IdeaStorm.com for software. What do you think? I m willing to co-mentor this if someone finds that is an interesting project for GSoC. Let me know!.

Since a recent update of one of my systems, amavisd-new keeps on spamming me. I get mails containing stuff like bayes: synced databases from journal in 0 seconds: 674 unique entries (742 total entries). The reasons seems to be a cronjob from amavisd-new (which can be deactivated via debconf):

grep amavisd-new-cron /etc/cron. d,daily /*
/etc/cron.d/amavisd-new:18 */3  * * *   amavis  test -e /usr/sbin/amavisd-new-cronjob && /usr/sbin/amavisd-new-cronjob sa-sync
/etc/cron.daily/amavisd-new:test -e /usr/sbin/amavisd-new-cronjob && exec /usr/sbin/amavisd-new-cronjob sa-clean

However, the called script contains:

egrep "sa-sync\) sa-clean\)" -A 2 /usr/sbin/amavisd-new-cronjob
        sa-sync)
                do_amavis_cmd "/usr/bin/sa-learn --sync >/dev/null"
                ;;
        sa-clean)
                do_amavis_cmd "/usr/bin/sa-learn --sync --force-expire >/dev/null"
                ;;

Note that output is redirected to /dev/null, so I should only mails if an error occurs, shouldn't I? I'm really baffled... To the best of my knowledge I shouldn't get these mails. Where am I wrong, and how to formulate that in a bug report?

For the day thats in it (a salute to all triskaidekaphobians out there ...) It appears that the project under Bush was shut down due to an accounting error. According to Physics World:

A $500m accounting error led to the Bush administration scrapping a carbon capture and storage demonstration plant last year according to a report by the US House of Representatives committee of science and technology. The project was scrapped because costs appeared to have ballooned from $1bn to $1.8bn when in fact it had only risen to $1.3bn. 

Oops. Tags oops, co2, friday13

With any luck, someone has seen this problem before and can fix it easily. I don t seem to be able to find the fix by searching, but the search terms feel like they re either too general or too specific. I was trying to install some perl software on a debian lenny system that was upgraded from etch (and previously from sarge). After installing the dependencies, I started getting errors like this:-

$ perl -MYAML::Syck -e print
Can't locate YAML/Syck.pm in @INC (@INC contains:
/usr/local/lib/perl5/5.10.0/i686-linux /usr/local/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl/5.10.0/i686-linux
/usr/local/lib/perl5/site_perl/5.10.0 .).
BEGIN failed--compilation aborted.

Update: Well done to Raphael who spotted the local that I d overlooked. D oh! Now to see if I can discover where that came from. Update 2: Also thanks to Florian who emailed in at about the same time (I don t read my email as much as my website dashboard, usually).

Then I discovered that the debian mirror being used (which is in the same city, and in a nearby network) is marked as bad on the health check. I switched mirrors to a known-good one that I ve used for other systems recently, but no joy. I think a key symptom is this bit near the end of the output of perl -V:

Compiled at Aug 4 2008 09:48:59
@INC:
/usr/local/lib/perl5/5.10.0/i686-linux
/usr/local/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl/5.10.0/i686-linux
/usr/local/lib/perl5/site_perl/5.10.0

Our good systems show a December compiled at date and a much longer @INC list. I ve tried reinstalling pretty much every package I think might be connected, including libc6, libperl5.10, perl, perl-base and perl-modules (those last four are all version 5.10.0-19). I really don t want to reinstall the whole system because it s behind an irritating firewall that will make it a 10-hour round-trip site visit. I m currently hunting and removing obsolete packages to try to make sure they re not causing problems, because I m running out of straws to clutch at. How would you recover a debian system from this?

If you code for long enough on x86-64, you'll eventually hit an error such as: Here's a little example that might help you figure out what you've done wrong. Consider the following code: In case it's not clear, that would look something like: Let's build that code, and see what it looks like We can see that the mov instruction has only allocated 4 bytes (00 00 00 00) for the linker to put in the address of foovar. If we check the relocations: The R_X86_64_32S relocation is indeed only a 32-bit relocation. Now we can tickle this error. Consider the following linker script, which puts the foo section about 5 gigabytes away from the code. This now means that we can not fit the address of foovar inside the space allocated by the relocation. When we try it: What this means is that the full 64-bit address of foovar, which now lives somewhere above 5 gigabytes, can't be represented within the 32-bit space allocated for it. For code optimisation purposes, the default immediate size to the mov instructions is a 32-bit value. This makes sense because, for the most part, programs can happily live within a 32-bit address space, and people don't do things like keep their data so far away from their code it requires more than a 32-bit address to represent it. Defaulting to using 32-bit immediates therefore cuts the code size considerably, because you don't have to make room for a possible 64-bit immediate for every mov. So, if you want to really move a full 64-bit immediate into a register, you want the movabs instruction. Try it out with the code above - with movabs you should get a R_X86_64_64 relocation and 64-bits worth of room to patch up the address, too. If you're seeing this and you're not hand-coding, you probably want to check out the -mmodel argument to gcc.