and now for something completely different... sometimes in a shell script you need to do stuff that modifies the positional parameters ($@, $1, $2, etc) from the cmdline, usually via the builtin shift, maybe from using getopt(1) or similar. for example, it's quite common to see the following in a shell script that parses cmdline flags with getopt

# parse cmdline options with getopt
TEMP= getopt -o h123 -n $0 -- "$@" 
if [ $? != 0 ] ; then usage >&2 ; exit 1 ; fi
# overwrites $@ with getopt-generated/sanitized arguments
eval set -- "$TEMP"
# destructively iterates across $@
while true; do
  case "$1" in
        -h)
          # handle option h
          shift
        ;;
        ...
        --)
          shift
          break
        ;;
      esac
...
done

but sometimes you want to be able to restore the original positional parameters later, or otherwise quote them in a manner where they can be passed along to another script/function, log them, embed them in a string/file, etc. you can't always just stash them in another variable to use that later, because "$@" has a special meaning to the shell that can't otherwise be reproduced with another variable. one way you could try to store/save them would be to save the original copy of $@ and use eval set -- later on to change it back to the saved variable. using a slightly simplified version of the above:

#!/bin/sh
# save the cmdline
saved="$@"
echo original command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done
echo mangled command line, $# arguments: "$@"
# restore the cmdline
eval set -- "$saved"
echo restored command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done

and at first glance, it seems to work:

rangda[/home/sean] ./bad.sh one two three
original command line, 3 arguments: one two three
arg: one
arg: two
arg: three
mangled command line, 0 arguments:
restored command line, 3 arguments: one two three
arg: one
arg: two
arg: three

but you may run into problems when some parameters have spaces, or quotes:

rangda[/home/sean] ./bad.sh "1: one" "2: two"
original command line, 2 arguments: 1: one 2: two
arg: 1: one
arg: 2: two
mangled command line, 0 arguments:
restored command line, 4 arguments: 1: one 2: two
arg: 1:
arg: one
arg: 2:
arg: two
rangda[/home/sean] ./bad.sh "one's one"
original command line, 1 arguments: one's one
arg: one's one
mangled command line, 0 arguments:
./bad.sh: eval: line 12: unexpected EOF while looking for matching  ''

or worse, shell meta characters:

rangda[/home/sean] ./bad.sh 'i wonder what  pwd  would do'
original command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do
mangled command line, 0 arguments:
restored command line, 6 arguments: i wonder what /home/sean would do
arg: i
arg: wonder
arg: what
arg: /home/sean
arg: would
arg: do

the solution is to build up the saved value a bit more manually, single-quoting each parameter (which should prevent any unintended side effects of space/quotes/metacharacters), and escaping existing single quotes inside of parameters.

# escape any single quotes in an argument
quote() 
  echo "$1"   sed -e "s,','\\\\'',g"
 
# save up a properly quoted/escaped version of "$@"
for arg in "$@"; do
  saved="$ saved:+$saved  '$(quote "$arg")'"
done
echo original command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done
echo mangled command line, $# arguments: "$@"
# restore the cmdline
eval set -- "$saved"
echo restored command line, $# arguments: "$@"
while test -n "$*"; do echo arg: "$1"; shift; done

let's see how it behaves:

rangda[/home/sean] ./good.sh "1 one" "2: two"
original command line, 2 arguments: 1 one 2: two
arg: 1 one
arg: 2: two
mangled command line, 0 arguments:
restored command line, 2 arguments: 1 one 2: two
arg: 1 one
arg: 2: two
rangda[/home/sean] ./good.sh "one's one"
original command line, 1 arguments: one's one
arg: one's one
mangled command line, 0 arguments:
restored command line, 1 arguments: one's one
arg: one's one
rangda[/home/sean] ./good.sh 'i wonder what  pwd  would do'
original command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do
mangled command line, 0 arguments:
restored command line, 1 arguments: i wonder what  pwd  would do
arg: i wonder what  pwd  would do

while not entirely pretty, it does to do the trick. i'd be interested to know if anyone has a better way of doing this (or if there are hidden problems in what i've posted above).

Stewart Smith has written about the removal of a blog from Planet Linux Australia [1] due to publishing a list of URLs that the Australian government wants to censor. The first point I want to make is that even if you had a list with thousands of entries that are not likely to offend anyone or incur any legal liability then it s still not suitable for syndication on most Planet feeds. The correct thing to do is to have a paragraph describing the list and why people would want to read it and then use the MORE feature of your blog so that the rest isn t in the RSS feed. If you use Wordpress which seems to have the MORE function broken then that would mean hosting the list somewhere else. In regard to the specific post, in a comment on Stewart s post Matt suggests that the Planet software somehow filter out certain blog posts. I am not aware of any way of doing that apart from through code changes, Matt could submit some patches to allow that sort of thing. One thing that would be really good would be to have an exclusion tag or category in a blog feed. So you for example you could have feed URLs such as /feed/lca which would be configured to list all posts without the tag not-lca. Another way for a blogger to do this would be to use Yahoo pipes [2]. The people who run a Planet should be prepared to take any feed URL. It would not be difficult for a blogger to create a pipe that excludes all items that have NSFW in the title (or any other possible way of listing them). A final option is to have multiple blogs. I have a blog for documents that I regularly update [3]. Many of those documents had been plain HTML files edited with vi for years before I started blogging. But Wordpress is a reasonable CMS and as I use it for blogging it made sense to use it for other documents too. Wordpress has no good option for managing two types of documents, ones that are date-based (regular blog posts with the date in the URL) and non-date based (which change periodically and have different date stamps). There are Wordpress pages, but the support for having moderate numbers of pages is not great. Also on my document blog I will often have articles appear new regularly as I change the date when updating them. Anyone is welcome to subscribe to the feed for my document blog if they are interested in seeing new versions of the documents, but I expect that most people don t want to. The Debian Wordpress package (as of last time I used it) and my fork of the Debian Wordpress package have great support for multiple blogs. There is Wordpress-MU for bulk blog hosting, but that is only designed for people who want to run something like LiveJournal or Blogger. If you just want a few blogs for friends and relatives then the regular Debian Wordpress package will do the job well. Some bloggers maintain two blogs, one for public things and another for close friends and relatives (people who ARE interested in what they ate for breakfast). Having one blog for the NSFW material would be a reasonable thing to do for certain bloggers. Finally while I doubt that someone who runs a Planet installation faces any legal liability, there is also the issue of a PR liability. From a PR perspective I think it s best for the reputation of Linux users in Australia for certain things to not appear on Planet Linux Australia. That said it would be good if there was a process for removing and reinstating blogs that was publicly documented. There will obviously be many differences of opinion as to what is too risky to allow on the Planet so we should expect that from time to time feeds will be temporarily removed. When that happens what does a blogger have to do to be done to be syndicated again? Update: A comment has revealed a way of filtering out RSS feeds via the feed URLs used by wordpress. A URL such as /feed/cat=-X will give a feed of all articles that don t contain category number X. Multiple categories can be specified when separated by commas. So this allows Wordpress users to exclude their NSFW category from Planet Linux Australia.

• [1] http://www.flamingspork.com/blog/2009/03/21/things-not-to-post-on-planet-la/


• [2] http://pipes.yahoo.com/pipes/


• [3] http://doc.coker.com.au/

As I already told you, I am using XMPP4R-Simple in an internal project. For this, I d have to add PubSub capabilities to it, and I did when I decided to get serious with git. Today I committed a version with integration of various patches by other forks I found on GitHub plus a lot of improvements on PubSub functions. With this version it s simple to use PubSub like this:

require 'xmpp4r-simple'
# Simple function just to parse an event
require 'time'
def parse_event(event)
  item = event.children[0]
  node = item.node
  time = nil; item.each_element("//published")    e  time = Time.parse(e.text)  
  text = nil; item.each_element("//body")    e  text = e.text  
  return   :item => item, :node => node, :time => time, :text => text  
end
# Create the clients
im1 = Jabber::Simple.new "im1@example.com", "password" 
im2 = Jabber::Simple.new "im2@example.com", "password" 
# im1 creates a node
im1.create_node("/some/node")
# im2 subscribe to that node
im2.pubsubscribe_to("/some/node")
# We'll start a simple thread to get the events coming from that node to im2
Thread.new   loop  
  sleep 1 while ! im2.received_events?
  im2.received_events    event 
    h = parse_event(event)
    puts ">>> Got an event from node # h[:node]  published at # h[:time]  with text # h[:text] " 
   
 
# Now im1 just publishes anything to that node.
im1.publish_atom_item("/some/node", "This is my node", "This is the content of my node")
# The thread should capture the event and run puts on the hash from parsing.

PubSub is great, isn t it?

Luke Peterson from UC Davis came to visit me in Reno and we spent the last weekend hacking on the Python Dynamics package that uses SymPy to calculate equations of motion for basically any rigid body system.

On Friday we did some preliminary work, mostly on the paper, Luke showed me his rolling torus demo that he did with the proprietary autolev package. We set ourselves a goal to get this implemented in SymPy by the time Luke leaves and then we went to the Atlantis casino together with my boss Pavel and other guys from the Desert Research Institute and I had my favourite meal here, a big burger, fries and a beer.

On Saturday we started to code and had couple lines of the autolev torus script working. Then we went on the bike ride from Reno to California. I took some pictures with Luke's iphone:


Those mountains are in California and we went roughly to the snow line level and back:

This is Nevada side:


That was fun. Then we worked hard and by the evening we had a dot product and a cross product working, so we went to an Irish pub to have couple beers and I had my burger as usual.

On Sunday we spent the whole day and evening coding and we got the equations of motion working. On Monday we worked very hard again:



and fixed some remaining nasty bugs. I taught Luke to use git, so our code is at: http://github.com/hazelnusse/pydy, for the time being we call it pydy and after we polish everything, we'll probably put it into sympy/physics/pydy.py. If you run rollingtorus.py, you get this plot of the trajectory of the torus in a plane:

It's basically if you throw a coin on the table, e.g. this model takes into account moments of inertia, yaw (heading), lean, spin and the x-y motion in the plane. Depending on the initial conditions, you can get many different trajectories, e.g for example:

or:


This is very exciting, as the code is very short, and most of the things that Luke needs are needed for all the other applications of sympy, e.g. a good printing of equations and vectors (both in the terminal and in latex), C code generation, fast handling of expressions, nice ipython terminal for experimentation, plotting, etc.

Together with the atomic physics package that we started to develop with Brian sympy will soon be able to cover some basic areas of physics. Other areas are general relativity (there is some preliminary code in examples/advanced/relativity.py) and quantum field theory and Feynman diagrams - for that we need someone enthusiastic that needs this for his/her research --- if you are interested, drop me an email, you can come to Reno (or work remotely) and we can get it done.

My vision is that sympy should be able to handle all areas of physics, e.g. it needs good assumptions (if you want to help out, please help us test Fabian's patches here), then faster core, we have a pretty good optional Cython core here, so we'll be merging it after the new assumptions are in place. Then sympy should have basic modules for most areas in physics so that one can get started really quickly. From our experience so far in sympy/physics, those modules will not be big, as most of the functionality is not module specific.

This morning I attempted LPI101 exam (I already passed LPI102 some time ago).

I did because on April 1st the objectives will be updated to newer (and more up-to-date) that they are now (they're still about ISA devices or so) and all the materials to study on would become old.

Pretty scared (due to weird question I couldn't remember) but still passed! Scored 720 points (500 was needed to pass the exam).

Now I own the full LPI-1 certification!

The book Expert Python Programming comes with subtitle Best practices for designing, coding, and distributing your Python software. Well this seems to be quite broad area to be covered by a single book, but the book really tries to cover all topics at least in some form of overview and pointing user to other documentation sources. In first chapter you can find information about installing Python on various platforms. As the book seems to be targeted to more experienced users, this chapter seems to be a bit useless, but author decided to give a chance also to users, who do not know the basics. Next chapters describe best practices for writing Python code, which was the most interesting part for me. I use Python several years, but I never really found time time to properly look at using decorators, generators and other "recently" introduced language features. Second chapter nicely covers these and I really appreciated that I got idea for what decorators can be used. Third chapter into class level and describes how to use metaclasses, attributes, constructors and other things you can face on class level. Fourth moves the topic to even more abstract way and in short summarises PEP 8 best coding practices. Here we leave solid ground of writing code and we move to more abstract areas of managing projects, testing, documenting and distributing software. It begins with description of distutils and setuptools in chapters five and six, where you will find basic information about distributing your software. Seventh chapter covers zc.buildout, which can help you with setting up and deploying applications in way Zope and Plone do. Next chapter is not really related to Python, it just describes advantages of version control. The most Python related part is using Mercurial VCS, which is written in Python, but you can of course use it for managing any project. Next tree chapters are devoted to project development in general, describing management of the project, generating documentation for project using Sphinx (what actually made me into starting documentation for python-gammu in it) and test-driven development practices. Now comes again two more interesting chapters describing optimization of Python code, giving the best advice not to optimize unless you are sure it is needed ;-). But it also goes more deeper and describes benchmarking procedures, how to debug memory usage and some basic tips for general things which might improve performance. Last chapter is some kind of extra in the book and to me it looks it was added just to use buzzwords design patters, but well it is there and it covers few design patterns which might be useful in Python like singleton or more advanced version of it called borg. In general this book covers quite wide range of topics, what can be seen as biggest advantage and disadvantage. You will get basic overview of creating Python project using modern methods (test driven development, design patters, best practices for Python code), but you can not expect it will go into much deep details. The book can extend your knowledge of development in Python, but you should carefully check examples with errata, because otherwise several of them will fail. I'm not that good at spotting problems in written English (as you can probably see in this post), but I spotted several errors in code and I think more careful editor work on the book would improve my feelings a bit. Also I personally would prefer much deeper look into some topics and skipping the others, but one book can not fit all ;-).

After reading the comments on my earlier post, Delayed allocation and the zero-length file problem as well as some of the comments on the Slashdot story as well as the Ubuntu bug, it s become very clear to me that there are a lot of myths and misplaced concerns about fsync() and how best to use it. I thought it would be appropriate to correct as many of these misunderstandings about fsync() in one comprehensive blog posting. As the Eat My Data presentation points out very clearly, the only safe way according that POSIX allows for requesting data written to a particular file descriptor be safely stored on stable storage is via the fsync() call. Linux s close(2) man page makes this point very clearly:

A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a file system to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored use fsync(2).

Why don t application programmers follow these sage words? These three reasons are most often given as excuses:

1. (Perceived) performance problems with fsync()
2. The application only needs atomicity, but not durability
3. The fsync() causing the hard drive to spin up unnecessarily in laptop_mode

Let s examine each of these excuses one at a time, to see how valid they really are. (Perceived) performance problems with fsync() Most of the bad publicity with fsync() originated with the now infamous problem with Firefox 3.0 that showed up about a year ago in May, 2008. What happened with Firefox 3.0 was that the primary user interface thread called the sqllite library each time the user clicked on a link to go to a new page. The sqllite library called fsync(), which in ext3 s data=ordered mode, caused a large, visible latency which was visible to the user if there was a large file copy happening by another process. Nearly all of the reported delays was a few seconds, which would be expected; normally there isn t that much dirty data that needs to be flushed out on a Linux system, even if it is even very busy. For example, consier the example of a laptop downloading an .iso image from a local file server; if the laptop has the exclusive link of a 100 megabit/second ethernet link, and the server has the .iso file in cache, or has a nice fast RAID array so it is not the bottleneck, then in the best case, the laptop will be able to download data at the rate of 10-12 MB/second. Assuming the default 5 second commit interval, that means that in the worst case, there will be at most 60 megabytes which must be written out before the commit can proceed. A reasonably modern 7200 rpm laptop drive can write between 60 and 70 MB/second. (The Seagate Momentus 7200.4 laptop drive is reported to be able to deliver 85-104 MB/second, but I can t find it for sale anywhere for love or money.) In this example, an fsync() will trigger a commit and might need to take a second while the download is going on; perhaps half a second if you have a really fast 7200 rpm drive, and maybe 2-3 seconds if you have a slow 5400 rpm drive. (Jump to Sidebar: What about those 30 second fsync reports?) Obviously, you can create workloads that aren t bottlenecked on the maximum ethernet download speed, or the speed of reading from a local disk drive; for example, dd if=/dev/zero of=big-zero-file will create a very large number of dirty pages that must be written to the hard drive at the next commit or fsync() call. It s important to remember though, fsync() doesn t create any extra I/O (although it may remove some optimization opportunities to avoid double writes); fsync() just pushes around when the I/O gets done, and whether it gets done synchronously or asynchronously. If you create a large number of pages that need to be flushed to disk, sooner or later it will have a significant and unfortunate effect on your system s performance. Fsync() might make things more visible, but if the fsync() is done off the main UI thread, the fact that fsync() triggers a commit won t actually disturb other processes doing normal I/O; in ext3 and ext4, we start a new transaction to take care of new file system operations while the committing transction completes. The final observation I ll make is that part of the problem is that Firefox as an application wants to make a huge number of updates to state files and was concerned about not losing that information even in the face of a crash. Every application writer should be asking themselves whether this sort of thing is really necessary. For example, doing some quick measurements using ext4, I determined that Firefox was responsible for 2.54 megabytes written to the disk for each web page visited by the user (and this doesn t include writes to the Firefox cache; I symlinked the cache directory to a tmpfs directory mounted on /tmp to reduce the write load to my SSD). So these 2.54 megabytes is just for Firefox s cookie cache and Places database to maintain its Awesome bar . Is that really worth it? If you visit 400 web pages in a day, that s 1GB of writes to your SSD, and if you write more than 20GB/day, the Intel SSD will enable its write endurance management feature which slows down the performance of the drive. In light of that, exactly how important is it to update those darned sqllite databases after every web click? What if Firefox saved a list of URL s that has been visited, and only updated every 30 or 60 minutes, instead? Is it really that every last web page that you browse be saved if the system crashes? An fsync() call every 15, 30, or 60 minutes, done by a thread which doesn t block the application s UI, would have never been noticed and would have not started the firestorm on Firefox s bugzilla #421482. Very often, after a little thinking, a small change in the application is all that s necessary for to really optimize the application s fsync() usage. (Skip over the sidebar if you ve already read it). Sidebar: What about those 30 second fsync reports? If you read through the Firefox s bugzilla entry, you ll find reports of fsync delays of 30 seconds or more. That tale has grown in the retelling, and I ve seen some hyperbolic claims of five minute delays. Where did that come from? Well, if you look that those claims, you ll find they were using a very read-heavy workload, and/or they were using the ionice command to set a real-time I/O priority. For example, something like ionice -c 1 -n 0 tar cvf /dev/null big-directory . This will cause some significant delays, first of all because ionice -c 1 causes the process to have a real-time I/O priority, such that any I/O requests issued by that process will be serviced before all others. Secondly, even without the real-time I/O priority, the I/O scheduler naturally prioritizes reads as higher priority than writes because normally processes are waiting for reads to complete, but writes are normally asynchronous. This is not at all realistic workload, and it is even more laughable that some people thought this might be an accurate representation of the I/O workload of a kernel compile. These folks had never tried the experiment, or measured how much I/O goes on during a kernel compile. If you try it, you ll find that a kernel compile sucks up a lot of CPU, and doesn t actually do that much I/O. (In fact, that s why an SSD only speeds up a kernel compile by about 20% or so, and that s in a completely cold cache case. If the commonly used include files are already in the system s page cache, the performance improvement of the SSD is much less.) Jump back to reading Performance problems with fsync. The atomicity not durability argument One argument that has commonly been made on the various comment streams is that when replacing a file by writing a new file and the renaming file.new to file , most applications don t need a guarantee that new contents of the file are committed to stable store at a certain point in time; only that either the new or the old contents of the file will be present on the disk. So the argument is essentially that the sequence:

• fd = open( foo.new , O_WRONLY);
• write(fd, buf, bufsize);
• fsync(fd);
• close(fd);
• rename( foo.new , foo );

is too expensive, since it provides atomicity and durability , when in fact all the application needed was atomicity (i.e., either the new or the old contents of foo should be present after a crash), but not durability (i.e., the application doesn t need to need the new version of foo now, but rather at some intermediate time in the future when it s convenient for the OS). This argument is flawed for two reasons. First of all, the squence above exactly provides desired atomicity without durability . It doesn t guarantee which version of the file will appear in the event of an unexpected crash; if the application needs a guarantee that the new version of the file will be present after a crash, it s necessary to fsync the containing directory. Secondly, as we discussed above, fsync() really isn t that expensive, even in the case of ext3 and data=ordered; remember, fsync() doesn t create extra I/O s, although it may introduce latency as the application waits for some of the pending I/O s to complete. If the application doesn t care about exactly when the new contents of the file will be committed to stable store, the simplest thing to do is to execute the above sequence (open-write-fsync-close-rename) in a separate, asynchronous thread. And if the complaint is that this is too complicated, it s not hard to put this in a library. For example, there is currently discussion on the gtk-devel-list on adding the fsync() call to g_file_set_contents(). Maybe if someone asks nicely, the glib developers will add an asynchronous version of this function which runs g_file_set_contents() in a separate thread. Voila! Avoiding hard drive spin-ups with laptop_mode Finally, as Nathaniel Smith said in Comment #111 of of my previous post:

The problem is that I don t, really, want to turn off fsync s, because I like my data. What I want to do is to spin up the drive as little as possible while maintaining data consistency. Really what I want is a knob that says I m willing to lose up to minutes of work, but no more . We even have that knob (laptop mode and all that), but it only works in simple cases.

This is a reasonable concern, and the way to fix this is to enhance laptop_mode in the Linux kernel. Bart Samwel, the author and maintainer of laptop_mode, actually discussed this idea with me last month at FOSDEM. Laptop_mode already adjusts /proc/sys/vm/dirty_expire_centisecs and /proc/sys/vm/dirty_writeback_centisecs based on the configuration parameter MAX_LOST_WORK_SECONDS, and it also adjusts the file system commit time (for ext3; it needs to be taught to do the same thing for ext4, which is a simple patch) to MAX_LOST_WORK_SECONDS as well. All that is necessary is a kernel patch to allow laptop_mode to disable fsync() calls, since the kernel knows that it is in laptop_mode, and it notices that the disk has spun up, it will sync out everything to disk, since once the energy has been spent to spin up the hard drive, we might as well write everything in memory that needs to be written out right away. Hence, a patch which allows fsync() calls to be disabled while in laptop_mode should do pretty much everything Nate has asked. I need to check to see if laptop_mode does this already, but if it doesn t force a file system commit when it detects that the hard drive has been spun up, it should obviously do this as well. (In addition to having a way to globally disable fsync() s, it may also be useful to have a way to selectively disable fsync() s on a per-process basis, or on the flip side, exempt some process from a global fsync-disable flag. This may be useful if there are some system daemons that really do want to wake up the hard drive and once the hard drive is spinning, naturally everything else that needs to pushed out to stable store should be immediately written.) With this relatively minor change to the kernel s support of laptop_mode, it should be possible to achieve the result that Nate desires, without needing force applications to worry about this issue; applications should be able to just simply use fsync() without fear. Summary As we ve seen, the reasons most people think fsync() should be avoided really don t hold water. The fsync() call really is your friend, and it s really not the villain that some have made it out to be. If used intelligently, it can provide your application with a portable way of assuring that your data has been safely written to stable store, without causing a user-visible latency in your application. The problem is getting people to not fear fsync(), understand fsync(), and then learning the techniques to use fsync() optimally. So just as there has been a Don t fear the penguin campaign, maybe we also need to have a Don t fear the fsync() campaign. All we need is a friendly mascot and logo for a Don t fear the fsync() campaign. Anybody want to propose an image? We can make some T-shirts, mugs, bumper stickers Related posts (automatically generated):

1. Delayed allocation and the zero-length file problem A recent Ubuntu bug has gotten slashdotted, and has started...

I m in the middle of migrating a mail server away from the Cyrus mail store [1]. Cyrus provides a POP and IMAP server, a local delivery agent (accepting mail via LMTP). It is widely believed that Cyrus will give better performance than other mail stores, but according to a review by linux-magazin.de Dovecot and Courier deliver comparable (and sometimes better) performance [2]. The biggest problem with Cyrus is that it is totally incompatible with the Unix way. This wouldn t be a problem if it would just work and if it would display reasonable error messages when it failed, but it doesn t. It often refuses to work as desired, gives no good explanation, and it s data structures can t be easily manipulated. Dovecot [3] and Courier [4] use the Maildir++ format [5] (as well as many other programs). I have set up a system with Courier Maildrop and Dovecot for the IMAP server [6] and it works well - it s good to have a choice! But also Maildir++ is reasonably well documented and is an extension to the well known Maildir format. This means that it s easy to manipulate things if necessary, I can use mv to rename folders and rm to remove them. Cyrus starts with a database (Berkeley DB file) of all folders in all mailboxes. Therefore it is not possible to move a user from one back-end server to another by merely copying all the files across and changing the LDAP (or whatever else contains the primary authentication data) to point to the new one. It also makes it impossible to add or remove folders by using maildirmake or rm -rf. The defined way of creating, deleting, and modifying mailboxes is through IMAP. One of the problems with this is that copying a mailbox from one server to another requires writing a program to open IMAP connections to both servers at once (tar piped through netcat is much faster and easier). Also if you need to rename a mailbox that contains many gigabytes of mail then it will be a time consuming process (as opposed to a fraction of a second for mv). Cyrus has a tendency to break while Dovecot is documented as being self-healing and Cyrus also seems to cope well in the fact of a corrupted mail store. Even manually repairing problems with Cyrus is a painful exercise. The Cyrus mail store is also badly designed - and it s design was worse for older filesystems (which were common when it was first released) than it is for modern ones. The top level of a Cyrus maildir contains all the messages in the INBOX stored one per file, as well as three files containing Cyrus indexes and sub-directories for each of the sub-folders. So if I want to discover what sub-folders a mailbox has then I can run ls and wait for it to stat every file in the directory or I can use an IMAP client (which takes more configuration time). As opposed to a Maildir++ store where every file that contains a message is stored in a folder subdirectory named new , cur , or tmp which means that I can run ls on the main directory of the mail store and get a short (and quick) result. Using tools such as ls to investigate the operation of a server is standard practice for a sysadmin, it should work well! A finall disadvantage of Cyrus seems to have many small and annoying bugs (such as the reconstruct program not correctly recursing the sub folders). I guess it s because not many people use Cyrus that such things don t get fixed. One trivial advantage of Cyrus is that by default it splits users into different sub-directories for the first letter of the account name. Dovecot supports using a hash of the user-name this is better than splitting by first-letter for performance (it gives a more equal distribution) but will make it slightly more difficult to manipulate the mail store by script. Ext3 can give decent performance without a two level directory structure for as many as 31,998 sub-directories (the maximum that it will support) due to directory indexing and Linux caching of dentries. There may be some other advantages of Cyrus, but I can t think of them at the moment. Here is a script I wrote to convert Cyrus mail boxes to Maildir++. To make this usable for a different site would require substituting a different domain name for example.com (or writing extra code to handle multiple domains) and inserting commands to modify a database or directory with the new server name. There is no chance of directly using this script on another system, but it should give some ideas for people performing similar tasks.

#!/bin/bash -e
# $1 is the user to move # BASEDIR is the directory for temporary files
BASEDIR=/mail/new
rm -rf $BASEDIR/$1 archive.tgz archive2.tgz ORIGSRV=$(echo "MYSQL_COMMAND_TO_GET_SERVER" mysql dbname)
echo "original server was $ORIGSRV" if [ "$ORIGSRV" = "" ]; then
echo "No server for account, strange error."
exit 1
fi ORIGPARENT="/mail/domain/e/example.com/$(echo $1 cut -c1)/user" # copy the data to a local archive
# USERDIR is for using ^ instead of . as Cyrus does.
mkdir -p $BASEDIR
cd $BASEDIR
USERDIR=$(echo $1 sed -e "s/\./^/g")
ssh cyrus@$ORIGSRV "(cd $ORIGPARENT ; tar czf - $USERDIR)" > archive.tgz
tar xzf archive.tgz
if [ "$USERDIR" != "$1" ]; then
mv $USERDIR $1
fi DEPTH=$(find . -mindepth 3 -type d wc -l)
if [ "$DEPTH" != "0" ]; then
echo "too deep"
exit 1
fi # remove cyrus index files
find $1 -name "cyrus.*" -exec rm \;
cd $1 # "mail" is a directory with sub-folders, every other name starting with
# [a-zA-Z] is a regular folder that needs to become a Maildir
for n in [a-zA-Z]* ; do
if [ "$n" != "mail" ]; then
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" ".$n"
fi
done # move the inbox files to Maildir storage
mkdir new cur tmp
mv [0-9]* new true
mv mail/[0-9]* cur true # move sub-folders of "mail/" and merge ones with duplicate names
cd mail
for n in * ; do
if [ -e "../.$n" ]; then
cd "$n"
mv [0-9]* "../../.$n/cur" true
cd .. ; rmdir "$n"
else
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" "../.$n"
fi
done
cd ..
rmdir mail # get a list of all names so we can recognise duplicates
find . -type d egrep -v "new cur tmp" cd $BASEDIR ssh root@localhost "mv /mail/new/$1 /mail/example.com ; chown -R mail:mail /mail/example.com/$1"
echo "MYSQL_COMMAND_TO_SET_SERVER" mysql dbname

• [1] http://cyrusimap.web.cmu.edu/


• [2] http://www.linux-magazin.de/heft_abo/ausgaben/2007/06/auf_der_teststrecke/


• [3] http://www.dovecot.org/


• [4] http://www.courier-mta.org/imap/


• [5] http://wiki.dovecot.org/MailboxFormat/Maildir


• [6] http://etbe.coker.com.au/2009/03/10/maildrop-imap-postfixadmin/

Earlier today I stumbled across a recording of a work I had not heard in a few years by the obscure Swiss-German composer Joachim Raff. The composition is an orchestral transcription of the Chaconne from Bach's Partita in D minor, without doubt the finest work ever written for solo violin. This piece has been transcribed countless times; the source material is clearly a masterpiece, but what makes the movement really attractive is its implicit nature - dispite the violin's limited facilities for counterpoint and polyphony in general, Bach succeeds in alluding to a multitude of voices within the same line. Larry Solomon provides a brief overview of what to look out for. Raff's transcription thus endeavours to render the implicit explicit; for example, where there is the implication of a held note, it can played for its implied duration. Indeed, Raff believed that Chaconne was in itself a "reduction" from an orchestrated original. Whilst I don't share that literal belief, it is certainly good working model for a transcription. And what a transcription! I've often thought of the romantic transcriptions such as these to be a guilty pleasures; they clearly violate any sensibilities I have about performing early music. However, what is most surprising (and guilt-relieving) about Raff's effort is the sheer amount of novel material involved. Here is an example starting at bar 216 - I have overlayed Bach's original line with a melody of Raff's own invention:

There are countless other examples which, following Bach's example, vary greatly in their subtlety. No prizes for guessing why I chose this excerpt though. However, dispite these additions I feel some crucial aspects are actually lost in translation. Firstly, we can muse over whether Bach would have approved of such a rendering - not wishing to dwell too deeply in well-tread arguments, many would point to Bach's re-use of his own material (as well as his transcriptions of other composers) to illustrate that he was not against the practice, but I find that argument difficult to apply to the Chaconne - the dualism between the implicit and the explicit would have greatly appealed to Bach, so to rob the work of it would seem to be doing him a serious discourtesy. Furthermore, the implications in Bach's work are certainly not followed to the same conclusions by all - a truly solo musician has the advantage of being able to make their own decisions about the music, perhaps even on the spur of the moment. In contrast, an ensemble is effectively forced to obey Hr. Raff's inferences. This lack of spontaneity is particularly damning; I am sure one would tire of differing interpretations of Raff's transcription quicker than one would of Bach's original. Lastly (and certainly most subjectively) there is something disarmingly solitary about the original work which is only amplified by being played solo - dispite my inability to provide a satisfactory rendition I would find performing this piece with (or even to) others rather discomforting. In conclusion, I would recommend listening to this work (and the rest of the CD). I have found it extremely illuminating, if only about Bach's original than of Raff's.

Working with multiple upstream kernel trees and keeping an eye on more my git repositories tend to end up with large numbers of remotes (this laptop has 22, for example). It s getting to the point where I need something like etckeeper to keep track of them all.

Eheh, I'm getting older... 22 today! I just wanted to put down some of the important things happened to me this year...

• sentimental life: epic fail... not really in the mood of talking about it;
• University life: I finally passed two exams which blocked me for all the previous year: yay! Now I'm being stuck with other things -- not too much enthusiasm in some courses...
• Debian life: I finally had my GPG key signed by TWO Debian Developers: I became DM, and had upload rights on non-NEW packages. Great! We had Lenny released. Uber-Great!
• misc life: I just tried getting more people-friendly, more active (meaning being less lazy), less nerd (I probably failed in this particular aspect), better, in a word.

Wishes for the 22-23 development cycle :

• finally becoming a DD! (I'm NM since Dec-2007..)
• trying to pass many exams as possible at University
• seriously fall in love (I think I've already started this with the girl I'm with now...)
• misc improvements (sorry, you can't send patches... or yes?)

Happy birthday to me!

I've been on Twitter for over a year and a half now. If I had to explain why I like it, and why it's so popular now, I would use an analogy:

Twitter : IRC :: Blogs : Usenet

(This applies equally to other "micro-blogging" services, but I am about to explain why I believe that's not the right metaphor. You may also substitute mailing lists for Usenet.) With the older media, you have a place -- a newsgroup, or a channel, that people went to, with a distinct culture, and that (mostly) weren't "owned" by anyone, but rather by the community. With the new ones, we are all sole proprietors of our own streams, and we "tune in" to the subset of people we find interesting, rather than topics we invest in. So, instead of bumping into the same person in a couple different groups, or never reading their words at all, you might find that your feeds overlap a bit more than they do with most people. This is how I find people to "follow" and blogs to read, in fact -- as my network expands, more people become loosely joined to it, and as I notice ones worth reading I add them. Is one model better than the other? Probably not. I could make an analogy to music. In theory, I'd rather read what my favorite critics have to say about a wide variety of new releases -- some of which I'd never know about otherwise -- than keep up with the discussion of bands and genres I really like, even if most people writing about them are terrible (remember, 90% of everything is crap). But I also lose that sense of community of being a "fan" of something; I no longer have a deep connection to what's going on in the fandom or the scene, which I also would never know about otherwise (some of it is just too obscure for my favorite writers to cover). In practice, it seems the new approach has been more popular, but maybe that's because more people are on the net now, and both kinds of communication are/were shaped by the technology available at the time (destinations make much more sense with limited/centralized computing resources, and aggregation makes much more sense with powerful clients and a wider, less specialized user base). Anyway. This post is not actually about social media theory or whatever you want to call it; it's about some software I have packaged. Because of all the above, I have always wished that I could use Twitter from something more like my IRC client. Like, say, my IRC client. One could abuse the concepts of IRC to make an "on the fly" channel of whoever happens to be in my feed. (I once read a blog comment somewhere complaining that Twitter could easily be implemented on any existing IRC server using one +m channel for everyone and some client-side direction of messages from all such channels to a single window. +1 for cleverness, but they did sort of miss the point of why normal people sign up for web sites rather than installing and configuring clients for obscure chat protocols.) So, I had grand plans to write a Twitter client which was an IRC server, with some clever mapping of IRC concepts and commands to their equivalents over there. I never got around to it, as I barely have enough time for anything I do now. But last month I noticed that someone else had implemented such a thing: tircd. I had seen Twitter/IRC services before, but like the official Twiter XMPP service, they were all implemented as bots, which I detest for this sort of application. Bitlbee, for example, translates various IM protocols to IRC, but only halfway -- for anything else you have to use a bot as a sort of poor man's command line. If I want a command line for Twitter, I already have several; IRC can do better. And tircd really does! It's great. You're not required to edit the config file, and there's no extra layer on top of IRC for things like logging in or adding people. I've finally packaged the latest release, which is waiting in NEW currently. I got a request for sneak preview packages, so if you want to install some unapproved .debs check this repository. I may still pick my own project back up, as Twitter itself, being a centralized service, feels like a stopgap solution on the way to to a more generalized 140-character equivalent of the, er, blogosphere as envisioned by open-source projects like Identi.ca. In the future, perhaps, when we use a certain micro-blogging "service" we might be randomly connected to one of any number of servers run by different individuals but all mirroring messages back and forth to each other. Which, now that I think of it, sounds vaguely like some obscure, obsolete chat and news-posting protocols I know.

If you code for long enough on x86-64, you'll eventually hit an error such as: Here's a little example that might help you figure out what you've done wrong. Consider the following code: In case it's not clear, that would look something like: Let's build that code, and see what it looks like We can see that the mov instruction has only allocated 4 bytes (00 00 00 00) for the linker to put in the address of foovar. If we check the relocations: The R_X86_64_32S relocation is indeed only a 32-bit relocation. Now we can tickle this error. Consider the following linker script, which puts the foo section about 5 gigabytes away from the code. This now means that we can not fit the address of foovar inside the space allocated by the relocation. When we try it: What this means is that the full 64-bit address of foovar, which now lives somewhere above 5 gigabytes, can't be represented within the 32-bit space allocated for it. For code optimisation purposes, the default immediate size to the mov instructions is a 32-bit value. This makes sense because, for the most part, programs can happily live within a 32-bit address space, and people don't do things like keep their data so far away from their code it requires more than a 32-bit address to represent it. Defaulting to using 32-bit immediates therefore cuts the code size considerably, because you don't have to make room for a possible 64-bit immediate for every mov. So, if you want to really move a full 64-bit immediate into a register, you want the movabs instruction. Try it out with the code above - with movabs you should get a R_X86_64_64 relocation and 64-bits worth of room to patch up the address, too. If you're seeing this and you're not hand-coding, you probably want to check out the -mmodel argument to gcc.

Nearly a month has elapsed since Debian released Lenny. The interesting thing for me about this release was that I failed to get to the point where running etch was causing me pain. This became even more evident when Katherine commented on my Lenny T-Shirt - I'd forgotten to mention our release to her! Previously there have been a bunch of features coming that I really wanted, sometimes to the point of using backports. It's probably something to do with the way I run Debian; I use stable for remote machines and while in the past this would have meant quite a few work boxes these days it's just a box at my parents' and my colo. Where I felt the pain during this release cycle was on the boxes where I run unstable (my home desktop) or testing (my work box and my laptops). Development on new shininess slows down significantly while we're preparing for release (not unsurprisingly). And there are things coming up I quite like the look of, largely aimed at the desktop arena. I've been running Network Manager 0.7 from experimental on my EEE, for example - I need the HSO 3G support to avoid unnecessary faff on a daily basis. I've got ATI cards in my work and home desktops, so I'm watching the Mesa developments with interest - roll on OpenGL 2.0 support hopefully. The Intel KMS support that doesn't seem to be quite there yet but RSN, no, really, will be nice for my laptops. Experimental VDRs are looking interesting and perhaps worth upgrading my PVR to.

For a long time I haven't really understood the point of view that Debian shouldn't release. Or rather, I haven't understood why people would think that was a serious option. The first Debian box I installed went into a colo facility shortly afterwards. This is the first release I've not been running Debian on a commercial colo box and while I treat my personal colo in a similar fashion it's not quite the same in terms of priorities. However I still think regular releases are hugely important to the project. Some pain for those of us more interested in new shiny is a small price to pay for producing a flexible, reliable distribution, be they server administrators (though I see a worrying number of people decide to install testing on their production servers and then rarely/never update it), derivers or appliance vendors.

Mind you, now we've released I'm going to have to do all those things I've been putting off. I should probably start with moving my Perl bits into pkg-perl. And maybe I should switch over some sources.list files to point at squeeze instead of lenny...

It looks like I'm definitely losing Monday release schedule, but it does not matter that much, new release is here anyway. I wanted to push some bigger changes into first testing release after 1.23, but instead of that I did bunch of bug fixes, documentation and test improvements. Full list of changes follows:

• Fixed decoding of Samsung memory entries.
• Properly handle corrupted messages from Nokia phones (bug #811).
• Fix parsing of calls from Alcatel C701 (bug #831).
• Fixed handling of error states with -len and -autolen parameters.
• Fixed possible crash in DBI engine.
• PostgreSQL engine properly uses sequences instead of reinventing.
• MySQL engine properly uses auto increments instead of reinventing.
• Fix escaping of field name in PostgreSQL engine.
• Better SMSD files logging.
• Added testing of SMSD files engine.
• Increase timeout for AT settings SIM card SMS storage.
• Fixed date formatting in PostgreSQL SMSD service.
• Add option to install without documentation.
• Add -textutf8 option for TEXT sms.
• Fix crash when some lines in config are too long.
• Make smsdrc example better looking by removing obsolete docs.
• Make dku2 connection types work on Linux (alias to fbususb).
• Better calculation of signal percent.
• Fixed getting of memory status for some Nokias (bug #836).

You can download from usual place: http://cihar.com/gammu/, Debian users will find packages in experimental soon, SUSE, Fedora and Mandriva users can check experimental repository in OpenSuse Build Service.

The recent problem of the Portuguese translation of GCompris shows how important a good translation is, especially as far as free software are concerned.
I have not been able to find any news in English relating this story, and think it is essential to spread it so that people better appreciate the role translators play in the free software development process. The Portuguese Magalh es Project lead to provide 300,000 netbooks to Portuguese children aged of 6 to 10, for a fee varying from 0 to 50 , depending on the earnings of the family. These netbooks have a dual boot, with Windows and Linux (a distribution based on Mandriva called Caixa M gica - Magic Box in English).

The problem has begun when some opponents to this project pointed out that the Portuguese translation of GCompris, a free educational software for children from 2 to 10, contained some grammar, syntax and spelling faults, and did not reach the level of quality expected to be distributed to Portuguese children. This story was even published by some major Portuguese newspaper, which is a very bad publicity not only for GCompris, but also for free software in general. Caixa M gica Software have also published a press release explaining the situation. The Portuguese Government have even published a document explaining how to uninstall GCompris, whereas work for correcting the translation was running, and automatic updates being set up on all Magalh es netbooks. The GNOME Portuguese translation team have made some hard work during the week-end to review and correct the existing translation, following to a call for help raised by GCompris lead developer. The new version released yesterday already include an updated Portuguese translation. I have noticed that this problem was already reported on launchpad 2 years ago, but people got lost as the translation was not made through Rosetta (a closed source platform for open source application translation). This should have been reported upstream or at least to the main translator! I take the occasion to thank all translators, l10n and i18n coordinators of the various free software projects I enjoy using in French!

I have recently configured my mail server to use IMAP. I started doing this when I was attending Linux.conf.au so that I could read urgent mail using my EeePC while at the conference and then be able to deal with the more complex stuff using my laptop later on. The next logical step is to have mail delivered to different folders in the IMAP account. While there are ways of doing this via the Subject and other header fields, my needs are not that great. All I need to do is to support user+extension@example.com going to a folder named extension in the user s mail store. While changing my mail server I decided to install Postfixadmin at the same time. My first attempt to use Maildrop was to put the following in the /etc/postfix/main.cf file:
mailbox_command = /usr/bin/maildrop -d mail -f $SENDER $DOMAIN $USER $EXTENSION That seems to only work when you have local accounts, so I ended up setting fallback_transport = maildrop and then putting the following in /etc/postfix/master.cf: maildrop unix - n n - - pipe flags=DRhu user=vmail argv=/usr/bin/maildrop -d vmail $ nexthop $ user $ extension Where vmail is a Unix account I created for storing mail. Then I added the following to /etc/postfix/main.cf. Some of these are probably redundant (such as the virtual_mailbox_base). The recipient limit is set to 1 because there are no command-line parameters for maildrop to support two recipients for the same message.
virtual_alias_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_mailbox_domains = mysql:/etc/postfix/mysql_virtual_domains_maps.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_gid_maps = static:2000
virtual_uid_maps = static:2000
virtual_mailbox_base = /mail
vmaildir_destination_recipient_limit = 1
virtual_transport = maildrop
maildrop_destination_recipient_limit = 1
The files /etc/postfix/mysql* all have fields user=, password=, hosts=, and dbname=. The queries in each of the files are as follows:
mysql_virtual_alias_maps.cf:query = SELECT goto FROM alias WHERE address='%s' AND active = 1
mysql_virtual_domains_maps.cf:query = SELECT domain FROM domain WHERE domain='%s'
mysql_virtual_mailbox_maps.cf:query = SELECT maildir FROM mailbox WHERE username='%s' AND active = 1 The /etc/courier/maildroprc file has the following contents: # log the maildrop actions
logfile "/var/log/maildrop.log" # parameters 1, 2, and 3 are the domain, user, and extension
DOMAIN=tolower("$1")
USER=tolower("$2")
EXTENSION=tolower("$3")
DEFAULT="/mail/$DOMAIN/$USER" # try making a backup (cc) copy of the mail but do not abort if it fails
exception
cc "$DEFAULT/.backup/"
# try delivering to the extension folder but do not abort if it fails
exception
if(length("$EXTENSION") != 0 && "$EXTENSION" ne "backup")

to "$DEFAULT/.$EXTENSION/"

# deliver to the main inbox if there is no folder matching the extension or if no extension is specified
to "$DEFAULT/"
Installing Postfixadmin [1] was another challenge entirely. One of the complications of this is that there is no Debian package for Lenny (it seems that there will be one in Squeeze - Lenny+1). I found David Goodwin s tutorial on installing Postfixadmin and lots of other things on Debian/Etch [2] to be a very useful resource. I look forward to seeing a Lenny version of that document. Please let me know if you can think of any way to improve this.

• [1] http://postfixadmin.sourceforge.net/


• [2] http://codepoets.co.uk/postfixadmin-postgresql-courier-squirrelmail-debian-etch-howto-tutorial