A couple of days ago, an email arrived in which I am kindly asked to provide my PayPal coordinates. And before you dear reader start to laugh, it was actually perfectly genuine and came from a very happy user of the Finance::YahooQuote Perl module. DJ Padzensky wrote this well over a decade ago, and I first introduced and then maintained it as a package in Debian for what is now almost twelve years. I have also been acting as upstream maintainer for the last eight or so years providing the occassional patch to keep up with changes at the Yahoo! server side. After a short reflection, I decided to keep it simple. I undusted a wishlist I had once created over at Amazon, yet never publicised. And amazingly, within minutes of minutes of passing the URL on, I received a cheerful email stating that three books are their merry way. Appropriately enough, two were from Springer's useR! series and another one was a brand new GPU programming title. And they all arrived yesterday. I am still stunned. So with that, my time to say thanks! This was a genuinely nice gesture, and I really appreciate it. I have been doing open source work for over fifteen years now, and I would not still be doing it if I did not enjoy it myself. I continue to learn a lot in the process, and have had a chance to meet lot interesting folks. Yet it is also nice to get a little Thanks every now and them.

These days, I've been used to a very specific $EDITOR usage. I use my (Xfce) workspaces as context switch. Basically, each time I need to start a new activity I open a new workspace, open some terminals, maybe a browser, an editor (gvim) window etc. Then when I'm done, I delete that workspace. I can have multiple activities at once, and switch from one to another depending on various stuff. On those workspaces, I'd like to be able to have only one editor window, even when I open new files using gvim foo.c . But I'd like to have one editor window per workspace, not one global editor, since I'd like the editor to be context-specific too. I recently heard about the gvim remote stuff, and I've came with the following gvim function (zsh syntax):

function gvim()
desktop=$(xprop -root -notype -f _NET_CURRENT_DESKTOP 0c '$0+\n' _NET_CURRENT_DESKTOP)
if [ "$#" -ge "1" ];
then
gvimargs="--remote-tab-silent"
fi
=gvim --servername $desktop $gvimargs $*
The xprop trick is a hackish way to stick one gvim window per workspace, it might need to be adjusted if one has a two monitors setup and want to be able to have one editor per monitor per workspace. Basically, that snippet checks if a gvim server already exists with the current workspace name. If yes, it opens the file(s) in it, if not it run a new gvim server and opens the file there. It works more or less fine, but it has one problem: it fails badly when one needs to give args to the gvim call. Those args are usually placed *before* filenames, so in my case it won't work. One way to fix this would be to loop on all the args, check if the arg is an existing file, in case add it to $files. If not, add it to $args, then call gvim with the correct order. Works, except when you want to edit a new file, where you're back to point one. All in all, it'd work, but it's not really a nice way to do it, imho. So I'm calling for help, in case anyone has an idea about that. Basically, is there a way to say to gvim always open files in a remote server, run it if it's not already running using the config file and not an argument (so I don't mess with the command line). I do need the servername to be desktop specific though, so it's still a difficulty. NOTE: I know that emacs has some kind of remote server habilities too, but I'm not sure how it works and if it'd be possible to do that in emacs, and I'm not really an emacs user and don't really intend to switch. If you have any idea, feel free to comment (by mail), I might do a later post when I have an enhanced solution.

Just say no. Filed under Debian since this is a Univention system which is based on Debian (still etch, though.) And what specifically annoyed me was today

• That the mailbox list and the actual on disk files can get out of sync.
• That this causes nice System I/O error: %m messages (the %m part is verbatim. Nice, isn't it?) in cyrquota, and similar messages in cyradm for sam commands.
• That, while this can be fixed with a cyrrecover on the mailbox (a recursive recover on the parent doesn't bring back the mailbox, so I need to call cyrrecover on each affected mailbox separately), the output of cyrquota displays the mailbox as domain!mailbox^name.folder while cyrrecover expects mailbox.name/folder@domain
• That the System I/O error causes cyrquota to stop, so I need to sit here and restart it countless times. (At least it seems to pick up where it stopped, so I don't have to wait that long until it comes to the next problem.)

I knew from other experiences that one shouldn't use cyrus, but it can't be said often enough... And since I usually don't use it, it amazes me anew every time I have to babysit an installation. While I don't have a similarly big installation to compare it, I've found Dovecot to be very nice. Admittedly it doesn't have that many features.

Since I couldn t seem to readily find any useful instructions for building firmware images for the Nexus S I thought I d publish some. In a clean directory this sequence of commands should result in a working set of application layer images. They assume you ve already got all the Android build dependencies installed:

repo init -u git://android.git.kernel.org/platform/manifest.git -b gingerbread
repo sync
MYDROID=$(pwd)
export MYDROID
. ./build/envsetup.sh
make PRODUCT-full_crespo-user

The resulting images will be in the directory out/target/product/crespo. You can get this running on your device by flashing the boot, recovery and system images:

out/host/linux-x86/bin/fastboot flash boot out/target/product/crespo/boot.img
out/host/linux-x86/bin/fastboot flash recovery out/target/product/crespo/recovery.img
out/host/linux-x86/bin/fastboot flash system out/target/product/crespo/system.img

Note that using these firmware images will void any warranty you have on the phone you ll need to unlock the phone using fastboot oem unlock, but note that this is irreversible. Similar instructions should work for Nexus One too, using passion instead of crespo in the make command above.

A year ago, I posted my reading list for 2010. I listed a few highlights, and a link to my Goodreads page, pointing out that this wasn t necessarily a goal, just a list of things that sounded interesting. I started off with Homer s Iliad, which I tremendously enjoyed and found parallels to modern life surprisingly common in that ancient tale. I enjoyed it so much, in fact, that I quickly jumped to a book that wasn t on my 2010 list: The Odyssey. I made a somewhat controversial post suggesting that the Old Testament of the Bible can be read similar to how we read The Odyssey. Homer turned out to be much more exciting than I d expected. Jordan s Fires of Heaven (WoT #5) was a good read, though it is one of those books that sometimes is action-packed and interesting, and other times slow-moving and almost depressing. I do plan to continue with the series but I m not enjoying it as much as I did at first. War and Peace is something I started late last year. I m about 400 pages into it, which means I ve not even read a third of it yet. It has some moving scenes, and is a fun read overall, but the work it takes to keep all the many characters straight can be a bit frustrating at times. Harvey Cox s The Future of Faith was one of the highlights of the year. A thought-provoking read by someone that embraces both science and religion, and shows a vision of religion that returns to its earlier roots, less concerned about what particular truths a person believes in than it is about more fundamental issues. Marcus Borg s Jesus: Uncovering the Life, Teachings, and Relevance of a Religious Revolutionary began with a surprisingly engaging history lesson on how agriculture caused the formation of domination societies. It also described in a lot of detail how historians analyze ancient texts their drafting, copying, etc. It paints a vivid portrait of Jewish society in the time that Jesus would have lived, and follows the same lines of thought as Cox regarding religion finally moving past the importance of intellectual assent to a set of statements. Among books that weren t on my 2010 list, I also read and here I m not listing all of them, just some highlights: The Cricket on the Hearth in something of a Christmastime tradition of reading one of the shorter Dickens works. I enjoyed it, but not as much as I enjoyed A Christmas Carol last year. Perhaps I made up for that by watching Patrick Stewart as Scrooge instead. How to Disappear Completely was a fun short humorous read, with a very well-developed first-person narrative. Paralleling my interest in amateur radio, I read and studied three books in order to prepare myself for the different exams. In something of a surprise, I laughed a lot at Sh*t My Dad Says, which was more interesting and funny than I expected it to be. All I can say is that Justin s got quite the dad and quite the interesting childhood. I even read two other recent releases: The Politician (about John Edwards) and Game Change (about the 2008 presidential race). Both were interesting, vibrant, and mostly unsourced so hard to know exactly how much to take from them. And finally, reflecting on and travel before my first trip to Europe, Travel as a Political Act, which encourages us to find the fun in my cultural furniture rearranged and my ethnocentric self-assuredness walloped. And that was fun. Now to make up the 2011 list

the release team has started to usertag [0] the remaining RC bugs yeah, another step towards the release!

here are my small contributions around RC bugs:

• #544310 xsidplay: "xsidplay: FTBFS on GNU/kFreeBSD (alsa is linux-specific)"
  add [linux-any] to libasound2-dev, upload to DELAYED/2, later integrated into maintainer upload
• #551543 muttprint: "cannot remove path when cwd is /tmp/muttprint*"
  add a patch for the second incarnation of the bug report
• #604234 emacspeak-ss: "emacspeak-ss: dpkg hangs while preconfiguring packages"
  update bug info
• #605240 tenshi: "tenshi: fails to upgrade from lenny"
  update bug info
• #605793 src:libclass-dbi-asform-perl: "libclass-dbi-asform-perl: FTBFS: failed tests"
  add patch and tighten build dependency (pkg-perl)
• #606000 libmail-spf-query-perl: "libmail-spf-query-perl: Incorrect query results with IPv6 addresses; should warn about missing IPv6 support and/or fail graciously"
  update bug info and clone to qpsmtpd (#607472), package removed later (#608335)
• #606777 sendfile: "sendfile: package fails to upgrade properly from lenny"
  update bug info, later upload chris' patch to DELAYED/2
• #606779 openswan: "openswan: package fails to upgrade properly from lenny"
  update bug info
• #606785 common-lisp-controller: "stumpwm: package fails to upgrade properly from lenny"
  reassign and update bug info, later sponsor maintainer upload
• #606802 bacula-director-pgsql: "bacula-director-pgsql: make_catalog_backup is broken"
  investigate further and make proposals, later upload to DELAYED/2
• #607472 qpsmtpd: "qpsmtpd: Switch from Mail::SPF::Query to Mail::SPF"
  propose patch, later closed by maintainer

[0] if I got it right:

• squeeze-will-remove
• squeeze-is-blocker
• squeeze-can-defer

update: all tags combined: http://tinyurl.com/squeeze-sort (thanks to the RT for the URL & to zack for the pointer)

I did some experiments with the small tool I wrote in previous post in order to gather some startup data about Firefox. It turns out it can t flush directories and other metadata from the page cache, making it unfortunately useless for what I m most interested in. So, I gathered various startup information about Firefox, showing how page cache (thus I/O) has a huge influence on startup. The data in this post are mean times and 95% confidence interval for 50 startups with an existing but fresh profile, in a mono-processor GNU/Linux x86-64 virtual machine (using kvm) with 1GB RAM and a 10GB raw hard drive partition over USB, running, except when said otherwise, on an i7 870 (up to 3.6GHz with Intel Turbo Boost). The Operating System itself is an up-to-date Debian Squeeze running the default GNOME environment. Firefox startup time is measured as the difference between the time in ms right before starting Firefox and time in ms as returned by javascript in a data:text/html page used as home page. Startup vs. page cache

	Average startup time (ms)
Entirely cold cache (drop_caches)	5887.62 0.88%
Cold cache after boot	3382.0 0.51%
Selectively cold cache (see below)	2911.46 0.48%
Hot cache (everything previously in memory)	250.74 0.18%

The selectively cold cache case makes use of the flush program from previous post and a systemtap script used to get the list of files read during startup. This script will be described in a separate post. As you can see, profiling startup after echo 3 > /proc/sys/vm/drop_caches takes significantly more time than in the normal conditions users would experience, because of all system libraries that would normally be in the page cache being flushed, hence biasing the view one can have of the actual startup performance. Mozilla build bots were running, until recently, a ts_cold startup test that, as I understand it, had this bias (which is part of why it was stopped). The Hot cache value is also interesting because it shows that the vast majority of cold startup time is due to hard disk I/O (and no, there is no page faults number difference). I/O vs CPU Interestingly, testing on a less beefy machine (Core 2 Duo 2.2GHz) with the same USB disk and kvm setup shows something not entirely intuitive:

	Average (ms)
Entirely cold cache	6748.42 1.01%
Cold cache after boot	3973.64 0.53%
Selectively cold cache	3445.7 0.43%
Hot cache	570.58 0.70%

I, for one, would have expected I/O bound startups to only be slower by around 320ms, which is roughly the hot cache startup difference, or, in other words, the CPU bound startup difference. But I figured I was forgetting an important factor. I/O vs. CPU scaling Modern processors do frequency scaling, which allows the processor to run slowly when underused, and faster when used, thus saving power. It was first used on laptop processors to reduce power drawn from the battery, allowing batteries to last longer, and is now used on desktop processors to reduce power consumption. It unfortunately has a drawback, in that it introduces some latency when the scaling kicks in. A not so nice side effect of frequency scaling is that when a process is waiting for I/O, the CPU is underused, making the CPU usually run at its slowest frequency. When the I/O ends and the process runs again, the CPU can go back to full speed. This means every I/O can induce, on top of latency because of, e.g. disk seeks, CPU scaling latency. And it actually has much more impact than I would have thought. Here are results on the same Core 2 Duo, with frequency scaling disabled, and the CPU forced to its top speed :

	Average (ms)
Entirely cold cache	5824.1 1.13%
Cold cache after boot	3441.8 0.43%
Selectively cold cache	3025.72 0.29%
Hot cache	576.88 0.98%

(I would have liked to do the same on the i7, but Intel Turbo Boost complicates things and I would have needed to get two new sets of data) Update: I actually found a way to force one core at its max frequency and run kvm processes on it, giving the following results:

	Average (ms)
Entirely cold cache	5395.94 0.83%
Cold cache after boot	3087.47 0.31%
Selectively cold cache	2673.64 0.21%
Hot cache	258.52 0.35%

I haven t gathered enough data to have accurate figures, but it also seems that forcing the CPU frequency to a fraction of the fastest supported frequency gives the intuitive results where the difference between all I/O bound startup times is the same as the difference for hot cache startup times. As such, I/O bound startup improvements would be best measured as an improvement in the difference between cold and hot cache startup times, i.e. (cold2 - hot2) - (cold1 - hot1), at a fixed CPU frequency. Startup vs. desktop environment We saw above that the amount of system libraries in page cache directly influences application startup times. And not all GNU/Linux systems are made equal. While the above times were obtained under a GNOME environment, some other desktop environments don t use the same base libraries, which can make Firefox require to load more of them at cold startup. The most used environment besides GNOME is KDE, and here is what cold startup looks like under KDE:

	Average startup time (ms)
GNOME cold cache	3382.0 0.51%
KDE cold cache	4031.9 0.48%

It s significantly slower, yet not as slow as the entirely cold cache case. This is due to KDE not using (thus not pre-loading) some of the GNOME core libraries, yet using libraries in common, like e.g. libc (obviously), or dbus.

I m working on a big site, importing a lot of files to a WordPress installation and started to get a bunch of 413 s. After a little bit of googling I found out what was happening: What needs to be done is on /etc/nginx/nginx.conf write:

http  
    client_max_body_size 20M;
    include       /etc/nginx/mime.types;
    A LOT MORE SETTINGS...

Yes, just add client_max_body_size and restart your webserver. This by default is 1M, so anything beyond this will be refused.

I have finally reached a decision regarding my involvement in the Debian Ruby packaging efforts. I have decided to stop. This has been a very hard decision to make. I have invested huge amounts of time in that work over the years. I still love the language, and will continue to use it on a daily basis for my own developments. I still hope that it will succeed. I know that some people will be disappointed by that decision (and that others will think your work was useless anyway, people should use RVM and rubygems ). But I also know that I won t be able to push for all the required changes alone. I just don t have the time, nor the motivation. For the record, here are the changes I would have liked to see in the Ruby community. The core Ruby development community should mature.
The core Ruby development community is still dominated by Japanese developers. While not a bad thing in itself, it is easily explained by the fact that the main development mailing list, where most of the important decisions are taken, is in japanese. ruby-dev@ should be closed, and all the technical discussions should happen on the english-speaking ruby-core@ list instead. The release management process should also improve. Currently, it looks like a total mess. The following Ruby development branches are actively maintained:
ruby_1_8 (106 commits over the last six months)
ruby_1_8_6 (4 commits over the last six months)
ruby_1_8_7 (35 commits over the last six months)
ruby_1_9_1 (4 commits over the last six months)
ruby_1_9_2 (227 commits over the last six months)
trunk (1543 commits over the last six months) While the state of the ruby_1_8_6 and ruby_1_9_1 branches is clear (very important bugfixes only), the state of all of the other branches is rather unclear.
What s the stable Ruby branch? 1.8 or 1.9? If it s 1.9, why are people still actively developing in the ruby_1_8 branch? How long will they continue to be maintained in parallel, dividing the manpower? Is a Ruby 1.8.8 release to be expected? Will it be ABI/API compatible with 1.8.7? Is the ruby_1_8_7 branch really bugfixes-only? How much testing of it has been done? If it s bugfixes-only and regression-free, I should push it to Debian squeeze, due to be released in a few weeks. But would you recommend that? Due to past breakages in the ruby_1_8_7 branch, it s unlikely that we will do it.
Is the ruby_1_9_2 a regression-free, bugfix-only branch? If yes, isn t 227 commits over 6 months a lot? What will be the version of the next release of trunk ? When is it expected? Will it be ABI-compatible with the current ruby_1_9_2 branch? API-compatible?
New releases in the 1.8.7 and 1.9.2 branches were done on december 25th. Why were they no betas or RCs allowing wider testing? How much testing has been done behind the scenes? Most of those questions have no clear answer. The Ruby development community should build a common understanding of the status of the various branches, and of their release expectations. Releasing on december 25th of each year sounds fun, but is releasing when everybody is on vacation really a good idea? It would be fantastic to have something similar to Python Enhancement Proposals in the Ruby community. But having open discussions in english about the major issues would already be great. Ruby is not just the interpreter.
The Ruby development community should clearly define what the Ruby platform is. There are some big players, like Rails, and newer interpreter releases should not be done before ensuring that those big players still work. Also, since we have alternative Ruby interpreters, like JRuby, Rubinius and MacRuby, we need a clear process on how they integrate with the rest of the ecosystem. For example, having each of them rely on their own outdated fork of the whole stdlib is ridiculous, since it s not where they compete. The Ruby community should acknowledge that RVM and Rubygems are not for everybody. People who say so should be laughed at. Of course, RVM and Rubygems are nice tools for some people. But it is completely wrong to believe that compiling from source using RVM should be the standard way of installing Ruby, or that all people interested in installing Redmine should know that Ruby has its own specific packaging system. The Ruby community should work with their target platforms to improve how Ruby is distributed instead of reinventing the wheel. That includes Debian, but also RedHat-based distros, for example. It is likely that it won t be possible to reach a one-size-fits-all situation. But that s real life. Some people in the Ruby community should stop behaving like assholes. As one of the Debian Ruby maintainers, I have been routinely accused of creating crippled packages on purpose (FTR, I don t think that the Debian packages are crippled, despite what the rumors says). Debian is not the only target of that. Just yesterday, someone called for abandonning YARV (the new Ruby VM in Ruby 1.9), calling it Yet Another Random Vailure. This kind of comments is really hurting the people who are investing their free time in Ruby, and is turning away people who consider getting involved. In Debian, we have had a lot of problems getting people to help with Ruby maintenance since they are getting shit from the community all the time. So, what s the future for Ruby in Debian?

• For the interpreter, the two other maintainers, Akira Yamada and Daigo Moriwaki, are of course free to continue their work, and I wish them good luck.
• For the pkg-ruby-extras team, which maintains most of the Ruby libraries and applications, the future is less clear. The team was already badly understaffed, and I feel that I should probably clean up its status by orphaning/removing the packages that are unmaintained otherwise. This won t affect all the packages that the team maintains: some packages (like redmine) are actively maintained, and will stay.
• For me, it just means I will have more time for other things (possibly not Free Software-related). If things improve dramatically, I might also come back to Ruby packaging at some point.

Update: there s also a number of interesting comments about this post on this site.
Update 2: First, thanks a lot for all the interesting comments. I will make some follow-up posts trying to summarize what was said. It seems that this post also triggered some reactions on ruby-core@, with Charles Olivier explaining the JRuby stdlib fork, and Yui Naruse clarifying that all questions are welcomed on ruby-core@. This is great, really.

This week:

• #607958 (apt): replied and tagged moreinfo ; jmm later downgraded it to normal
• #606951 (nsca): agreed with the submitter and reverted the change, uploaded straight to unstable
• #605784 (nagios-statd): thanks to the great debugging work of the submitter, uploaded a fix to DELAYED/2 (giving the maintainer time to make his own planned upload)
• #598588 (json-glib): cannot reproduce; reduced severity and emailed the submitter

The remaining bugs are either removal candidates or no longer low-hanging-fruit, so I don t expect to keep squashing very many more before Squeeze is released.

Assigning an IP address statically to a host with a given MAC address using ISC dhcpd is quite trivial, one host entry, a hardware ethernet entry and a fixed-address entry and you are up and running.
But what if you want to assign IP addresses from a pool to only a few hosts with specific MAC addresses?

Before you ask yourself why someone might want to do that, have a look at my (very real) use-case.
I am currently working on setting up an installation server for my employer, ANEXIA Internetdienstleistungs GmbH. The server itself uses PXE, TFTP and FAI for installing systems. To be able to do PXE booting one has to set up an DHCP server to provide configuration details, like the TFTP Server Address and the boot filename.

Now what one should consider is that this system is designed to provide automatic installations for internet-facing hosts, namely ones in public IP networks. Running a DHCP server in such a network is not a good idea. We neither want to dish out configurations to each and every hosts that asks for them, neither do not want to do a PXE boot each and every time one of our systems is restarted. Now the combination of FAI and pxelinux allows for default configurations which force local booting, but this still causes the (re-)boot time for those systems to increase and potentially also increases the load on the TFTP server. Also, let's not even consider thinking about whether this setup is "clean" or not. I personally believe that dishing out IP addresses in a public IP network is a bad thing(tm) and I guess a lot of people will be nodding when reading these lines.

What I was asking myself is how to get something like that set up in a cleaner way, and guess what, I found a solution.
The basic idea behind this is only providing IP configuration via DHCP to a specific set of hosts (with a specific set of MAC addresses) and not providing any information to all other hosts. The specific set of hosts are those that we want to do an install run on. This is a no-brainer and I guess the right way to do that, but implementing this approach is not as straight-forward as I initially thought.

Actually the implementation of that idea caused me a bit of a headache and cost me a few work-hours to get right, that's why I'd like to share the configuration details with you.



Let's have a look at how to get such a setup using ISC dhcpd. We are using the fact that ISC dhcpd allows you to not only configure a subnet, but rather also pools inside subnets, which can have allow and deny rules. Such rules can be in the form of "allow/deny member of <class>", where classes (and subclasses, keep on reading for details) can be defined inside the configuration file as well.</class>

What we first did was creating a subnet with a pool declaration, as follows:

subnet 10.0.0.0 netmask 255.255.255.0

option routers 10.0.0.254;

option broadcast-address 10.0.0.255;

filename "fai/pxelinux.0";

next-server 10.0.0.254;

server-name "10.0.0.254";

pool

allow members of "install";

range 10.0.0.10 10.0.0.230;

This one configures the subnet 10.0.0.0/24, with 10.0.0.254 being the network gateway, 10.0.0.254 being the TFTP server and "fai/pxelinux.0" being the TFTP filename. Additionally pool allows us to define a range of IP addresses we want to use, along with a line stating that only members of the "install" class should get a network configuration. If you do not have any other subnet defined in your config and a client that is not in this "install" class asks for an IP address you will see something like this in your syslog: "dhcpd: DHCPDISCOVER from 11:22:33:44:55:66 via eth1: network 10.0.0/24: no free leases". dhcpd will not even answer these requests and thus the client will not even know that there is a DHCP server running here. Exactly what we wanted.

I wrote about this giving me a headache, but so far things have been pretty straight-forward. Getting this far did not take very long, believe me.

Next thing we did was defining that "install" class as follows:

class "install" match hardware;

Again, not very hard to do. This tells dhcpd to look for subclasses of "install" with a matching hardware-address. So let's have a look at the subclass for, let's say the host with MAC address "11:22:33:44:55:66":

subclass "install" 1:11:22:33:44:55:66;

I intentionally highlighted the leading "1:" there. This means nothing more or less than "ethernet". Without that leading "1:" you won't get anywhere. Matching will fail, simple as that. It took me a while to find information about this in "man 5 dhcp-eval". Quoting parts of the interesting section:

The hardware operator returns a data string whose first element is

the type of network interface indicated in packet being considered,

and whose subsequent elements are client s link-layer address. [...] Hardware types include ethernet (1), token-ring (6), and fddi (8).

Now, with the combination of the subnet, pool, class and subclass directives we could get the setup we wanted: a DHCP server only providing IP configuration to a specific set of hosts and ignoring all other DHCP requests.

If you have any comments about this setup or ideas on how to get something similar set-up using another approach feel free to leave a comment.

Personal final note: accidentally typing 80 instead of 08 in a MAC address will cost you an additional two hours and will even have you re-compile ISC dhcpd with eval debugging turned on, believe me. :-)

It's new year (for almost 18 hours here) and it's time to balance a bit what has happened in year 2010. I'm not going to expose details from my personal life and anything related to work, so all what remains are open source projects where I am active. For phpMyAdmin it was year of quite big changes, most of them beeing agreed on Fosdem 2010. We have migrated to git from svn, what was great step forward and I really like the change. It was also decided to use gettext (or rather php-gettext) for translations in upcoming release. I can see this move (together with online translation service) has definitely attracted more contributors (hey, but there are still many languages not translated, you're welcome!). Also we've participated in Google Summer of Code 2010, what lead to several big projects being merged into our code base (see my summary for more details). All in all, we're heading to 3.4 release in first months of 2011, which will be quite major step and hopefully heading in right direction. In Gammu and Wammu, there were lot of code improvements, most of them probably going into Gammu SMS Daemon (the latest one being unified SQL backend removing lot of code duplication). Another major improvement is The Gammu Manual covering everything from Gammu command line, through SMSD up to python and C API. On the other side I've bought me a Symbian based phone, which does not work with Gammu and I use Series60-Remote for that, so my interest in Gammu has declined a bit. Even though I've started adopting some packages in Debian, so it looked like I will end up with much more than last year, but at the end I've decided I'm too overloaded with some of them (and don't use them anymore), so I've given some for adoption and got rid of nanoblogger (and nanoblogger-extra) and looking for somebody to pick up gpointing-device-settings. Maybe all mpd related packages will follow soon, because I currently do not use mpd, so it limits quite a bit my ability to test them. And of course: Happy New Year 2011!

Filed under: Debian Gammu Phpmyadmin English 0 comments Flattr this!

It would be really neat if there were some tool which recorded which dotfiles an application read, used, or created. As an example emacs uses .emacs, but won't create it. However firefox will create and fill ~/.mozilla if it isn't present, and links will create ~/.links2. What would we do with that data? I'm not sure off the top of my head, but I think it is interesting to collect regardless. Perhaps a simple tool such as apt-file to download the data and let you search:

who-creates ~/.covers
who-creates ~/.dia

Obviously the simple use is to purge user-data when matching packages are removed - e.g. dpkg-posttrigger hook. But that's a potentially dangerous thing to do. Anyway I'm just pondering - I expect that over time applications will start switching to using "centralised" settings such as ~/.gconf2 etc. In the menatime I've started cleaning up ~/ on my own machines - things like ~/.spectemurc, ~/.grip, etc. ObQuote: What a long sword. I like that in a man - Blood of the Samurai (Don't be tempted; awful film.)

Then this would be apt:-

Strange but very cool (via a friend of mine, hellocatfood)

Last night was the last time I came back from a pub with my clothes stinking due to tobacco smoke. The Spanish congress has finally approved a real anti-smoking law which will ban smoking on public areas, with no exceptions or ways to workaround the ban. Starting on January 2nd, the Spanish state will be a smoke-free region (or mostly, it seems it'll be permitted in open-air events like football stadiums or bullrings, and I don't think that will be a great problem for me, specially the latter). For years, my intolerance to smoke has been increasing and I'm really expectant to see the benefits of this law in my habits. After more than 30 years dealing with smoke around us, it's our turn now. I've been speaking to a few barmans. In general they seem worried this will affect their business, but I can't see how it will. Spain has a big culture for having mid-morning almuerzos in bars, and people are not going to give that up due to not being able to smoke. They will just do it after they get out, not during the coffee, and that's it. The barmans of the two bars I visit most are non-smokers, but have to breath the smoke of hundreds of cigarretes every day, and can't do anything about it. Until tomorrow, when this will end and everyone will have a right to breath better air. I hope this kind of legislation continues to be adopted throughout Europe, because the FOSDEM welcome party is probably the next smoke horror I'll have to face soonish. :)

Benchmark results before:

num_operations: 1000000
insert      : 191.090 s (  5238.3/s) CPU; 191.106 s (  5237.7/s) wall clock
lookup      : 30.430 s ( 33068.8/s) CPU; 30.428 s ( 33064.0/s) wall clock
lookup_range: 85.700 s ( 11694.5/s) CPU; 85.700 s ( 11693.6/s) wall clock
insert2     : 143.320 s (  6986.7/s) CPU; 143.315 s (  6986.6/s) wall clock
remove      : 246.010 s (  4068.0/s) CPU; 246.013 s (  4067.9/s) wall clock
remove_range: 225.210 s (  4444.0/s) CPU; 225.286 s (  4442.4/s) wall clock

That's insert, lookup, and remove operations (and range operations) on a B-tree with one million keys. insert2 is insertions where keys are sorted. All nodes are stored in memory, not on disk. After some micro-optimization (replacing my own flexible bsearch implementation with the Python standard library one):

num_operations: 1000000
insert      : 97.850 s ( 10237.5/s) CPU; 97.860 s ( 10237.1/s) wall clock
lookup      : 12.430 s ( 81566.1/s) CPU; 12.429 s ( 81609.8/s) wall clock
lookup_range: 52.670 s ( 19047.6/s) CPU; 52.669 s ( 19050.0/s) wall clock
insert2     : 72.600 s ( 13806.4/s) CPU; 72.595 s ( 13808.6/s) wall clock
remove      : 134.510 s (  7443.8/s) CPU; 134.512 s (  7444.0/s) wall clock
remove_range: 169.740 s (  5897.3/s) CPU; 169.795 s (  5895.6/s) wall clock

Not bad, for a train ride's work, with some additional improvements late at night in a hotel room. Code is clearer too, now, at least mostly. And there's less of it: duplicating standard library functionality always maks me feel icky. The numbers for removals look bad, really bad. Espcially range removal should be much better. If anyone wanted to get familiar with my btree code base, that would be an excellent place to start from. See the README for instructions.

It's midnight and I really want to be coding. I'm half way or so through adding a git-annex-shell to git-annex, which will enable some nice features with ssh access. I've done the research, and written the documentation[1], laid all the necessary refactoring groundwork, and written much of the code and just have to connect it all up and test it. This is a dangerous point to be at, because I have it all in my head, but I'm tired. If I start coding now, I could make a stupid mistake and find myself stuck in a blind alley at 5 am. I've been there before. Even if I avoid hard stuff, I'd surely look up and an hour or more would have gone by, it'd be past 1 am -- and I know that once I stop coding it can take me hours to spin my thoughts back down to the point where I can go to sleep. Also, I can feel the back of my mind still working on something. Some bits feel like they might not quite mesh up right in my mental model. I don't consciously know what the problem is. I need to sleep on it; it will probably be clearer later. Like yesterday when I lifted a bag of groceries into the car and paused, realizing code I'd written two months ago had a major bug, one I'd never seen, in an edge case that had never came up, but was surely there. I always assumed that when programmers got older and stopped coding late it was because they couldn't take the strain. Nah. We're just coding even as we sleep. :)

---

[1] Documentation is where I do my design. This is why git-annex has quantities of documentation that Linux Weekly News finds surprising.

After trying to compare Ogg Theora to the Digistan definition of a free and open standard, I concluded that this need to be done for more standards and started on a framework for doing this. As a start, I want to get the status for all the standards in the Norwegian reference directory, which include UTF-8, HTML, PDF, ODF, JPEG, PNG, SVG and others. But to be able to complete this in a reasonable time frame, I will need help. If you want to help out with this work, please visit the wiki pages I have set up for this, and let me know that you want to help out. The IRC channel #nuug on irc.freenode.net is a good place to coordinate this for now, as it is the IRC channel for the NUUG association where I have created the framework (I am the leader of the Norwegian Unix User Group). The framework is still forming, and a lot is left to do. Do not be scared by the sketchy form of the current pages. :)

... looking for an HOWTO? These days, possibly because it's holiday in several countries around the world, I'm receiving a higher number of "how can I help Debian?" mails than usual. This is good and makes me happy! Those mails show we have a community who cares about giving back to Free Software, in exchange of what they have received via Debian. Routinely, in my replies, I point people to the following list of documentation we have on how to contribute to Debian:

• "How can you help Debian", the official contribution page on www.d.o
• HelpDebian on the Debian wiki
• HelpDebian FAQ, companion FAQ of the above page
• Chapter "Contributing to the Debian Project" of the Debian FAQ

Am I missing something? At the same time, it happens that I reply to people who clearly went through (at least some of) the above resources, but which are still blocked at the "what I do next?" step. I guess this is because some of the day to day practices of Debian life are not described in the documentation we have. For instance, I find scarce references to "cultural" aspects of Debian technical life such as:

• a lot of coordination happens over IRC
• a typical way of starting collaborating on packaging is interacting with the package maintainers via the BTS, by doing triaging, submitting patches, etc.
• lots of packages are team-maintained (in fact, this is mentioned in a couple of places, but unfortunately our team list is not complete. Therefore by saying "choose from the list a team you're interested in and contact them", as I recurrently do, I know I'm leaving someone out)
• how to find a mentor
• ...

And I'm sure I'm missing quite some other cultural/folklore traits, which look obvious to longstanding members of the Debian community, but which are probably not so obvious to external people. So, dear lazyweb, have you started contributing to Debian and remember your experience? If so, I'd like to hear from you. In particular, I'd like to hear your suggestions on how we can improve our documentation on the matter (patches welcome, as usual!). Bonus point. Of the various pieces of documentation we have, it seems that a sufficiently detailed HOWTO-style document, covering both technical and cultural aspects, is missing. If you are into writing for Debian, that might be an experiment worth trying.

There s this great feature in modern operating systems called the page cache. Simply put, it keeps in memory what normally is stored on disk and helps both read and write performance. While it s all nice for a day to day use, it often gets in the way when one wants to track performance issues with cold cache (when the files you need to access are not in the page cache yet). A commonly used command to flush the Linux page cache is the following:

# echo 3 > /proc/sys/vm/drop_caches

Unfortunately, its effect is broad, and it flushes the whole page cache. When working on cold startup performance of an application like Firefox, what you really want is to have your page cache in a state close to what it was before you started the application. One way to get in a better position than flushing the entire page cache is to reboot: the page cache will be filled with system and desktop environment libraries during the boot process, making the application startup closer to what you want. But it takes time. A whole lot of it. In one of my what if moments, I wondered what happens to the page cache when using posix_fadvise with the POSIX_FADV_DONTNEED hint. Guess what? It actually reliably flushes the page cache for the given range in the given file. At least it does so with Debian Squeeze s Linux kernel. Provided you have a list of files your application loads that aren t already in the page cache, you can flush these files and only these from the page cache. The following source code compiles to a tool to which you give a list of files as arguments, and that flushes these files:

#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>

int main(int argc, char *argv[])  
  int i, fd;
  for (i = 1; i < argc; i++)  
    if ((fd = open(argv[i], O_RDONLY)) != -1)  
      struct stat st;
      fstat(fd, &st);
      posix_fadvise(fd, 0, st.st_size, POSIX_FADV_DONTNEED);
      close(fd);
     
   
  return 0;
 

It s actually a pretty scary feature, especially on multi-user environments, because any user can flush any file she can open, repeatedly, possibly hitting system performance. By the way, on systems using lxc (and maybe other containers, I don t know), running the echo 3 > /proc/sys/vm/drop_caches command from a root shell in a container does flush the host page cache, which could be dangerous for VPS hosting services using these solutions. Update: I have to revise my judgment, it appears posix_fadvise(,,,POSIX_FADV_DONTNEED) doesn t flush (parts of) files that are still in use by other processes, which still makes it very useful for my usecase, but also makes it less dangerous than I thought. The drop_cache problem is still real with lxc, though.