After a really crazy weekend in which it looked like I wasn't able to set up NAT[1], I went to work with the gear and tested the exact same thing I did at home and it worked without a glitch.

So, now I suspect that there might be something wrong with my ISP or the DSL modem, while ritter (my NSLU2) is fine. As a bonus I just installed Debian armel on it (installation report to arrive soon).


When I got home, after the laptop came back from hibernate[2] I saw these messages on all my terminals:

Message from syslogd@bounty at Mon Nov 12 23:57:04 2007 ...
bounty kernel: Uhhuh. NMI received for unknown reason b0.

Message from syslogd@bounty at Mon Nov 12 23:57:04 2007 ...
bounty kernel: You have some hardware problem, likely on the PCI bus.

Message from syslogd@bounty at Mon Nov 12 23:57:04 2007 ...
bounty kernel: Dazed and confused, but trying to continue

Is this a good time to panic?
I guess I'll have to dig into this, too.


In other news, svn-buildpackage 0.6.23 has been uploaded to unstable and it fixes yet another 7 bugs, which brings svn-bp's bug count down to 19 open valid bugs (2 more if you count wontfix bugs, too). This is the lowest bug count of svn-buildpackage since at least the end of April, according to the bug count graph.

Also, oolite 1.65-6 was also uploaded to unstable and fixes the breakage due to the gnustep build tools changes. Unfortunately, on arm is dep-waiting for libgnustep-base-dev which is broken on this arch.


Thanks to my sponsors, you know who you are ;-) .

[1] I have been trying to set up a simple NAT machine for more than 6 hours and, although everything seemed to be OK, checked and double checked it didn't work
[2] which works now only if I don't use fglrx which is broken from this PoV (bug 449095)

Divided Allegiance is the middle of the trilogy, the one that I hate reading. Not because Ms Moon s book is bad, which it is not, it is still as gripping as the others, and comes closer to the high fantasy of Tolkien it is just that I hate what happens to Paks in the book, and the fact that the books ends, leaving her in that state. I guess I am a wimp when it comes to some things that happen to characters I am identifying with. However, it has been so long since I read the series that I have begun to forget the details, so I went through and read it anyway. This is a transition book: the Deeds of Paksenarrion was about Paksenarrion the line warrior, and the final book is where she becomes the stuff of legends. I usually read the first and last here.

I invite Wouter to Switzerland when he says that the problem with using Trains as office space is “of course, is that train rides aren't free, and that office rent is much more affordable...”: Our General Abonement costs ca. CHF 3000 (EUR 1800) a year and includes virtually all public transport in Switzerland, excluding only some cablecars in the mountains and similar things (and even there you sometimes get a price reduction.)

Sometimes you hit the number :)
(that means: still 50k bugs left for bug #500000)
Noticed that thanks to Christian Perrier who blogged about bug #450000.

---

Installed-Size?

zack@aquarium:~$ dpkg -s python-django grep Installed-Size
Installed-Size: 7776
zack@aquarium:~$ du -sh /usr/share/python-support/python-django/
7,2M    /usr/share/python-support/python-django/
zack@aquarium:~$ du -shc /var/lib/python-support/python2.*/django/
4,8M    /var/lib/python-support/python2.4/django/
5,0M    /var/lib/python-support/python2.5/django/
9,7M    total
zack@aquarium:~$

Disclaimer: I'm fine with python-support and friends, they've smart architectures. However it's a fact that currently a huge deal of packages in the archive are size-wise not faithfully represented by the Installed-Size field (and the ratio can be quite impressive!). I wasn't here when that field was designed, but I bet an intended use case was to give admins a way of knowing which packages were wasting disk space the most. For sure dpigs relies on Installed-Size and don't have the necessary info to identify stuff like the above.

Best wishes to everyone in the Free Software community for the new year Vikari (VS 2064) Diwali/new year this time was kind of anticlimatic compared to last year when we were actually in India. It is cold and gloomy in NJ, and fireworks are hard to come by. (Though I did manage to scrounge up some so not a total loss in that department.) And we arrived late at the mandir for Annakut Darshan today so they were out of thor which sucks. The presents front has been the usual maddeningly practical things. The kids of course got better stuff. Shailu, for instance, got Hungry Hungry Hippos and we've been having fun playing that. So I shouldn't complain. Ma Lakshmi has blessed me tremendously over the past year and I just know this year will be even better.

I bought this book by Daniel Silva last week at SFO, faced with a long wait for the red eye back home, since I recalled hearing about it on NPR, and reading a review in Time magazine, or was it the New Yorker? Anyway, the review said he is his generation s finest writer of international intrigue, one of America s most gifted spy novelists ever. I guess Graham Greene and John le Carre belong to an older generation. Anyway, everything I read or heard about it was very positive. Daniel Silva is far less cynical than Le Carre, and his world does not gel quite as well, to my ears, as Smiley s circus did. The hero, Gabriel Allon, does have some super human traits, but, thank the lord, is not James bond. I was impressed by Silva s geo-politics, though - paragraphs from the book seem to appear almost verbatim in current event reports in the International Herald Tribune and BBC stories. I like this books (to the extent of ordering another 7 from this author from Amazon today), and appreciate the influx of new blood in the international espionage market. Lately, the genre has been treated by lack luster, mediocre knock offs of the Bourne Identity and the engaging pace of the original has never been successfully replicated in the sequels. And Silva s writing is better than Ludlum s.

I have just come back from a half week stay at the Hilton Indian Lakes resort (which is the second time in a month that I have stayed at a golf resort and club and proceeded to spend 9 hours a day in a windowless conference room). On Thursday night, an ex Chicago native wanted to show us the traditional Chicago pizza (which can be delivered, half cooked, and frozen, via Fed-Ex, anywhere in the lower 48). Google Maps to the rescue! One of the attendees had a car, and we piled in and drove to the nearest pizzeria. It was take out only. We headed to the next on the list, again to be met with disappointment; since making the pizza takes the best part of an hour, and we did not want to be standing out in a chilly parking lot while they made out pizza. So, I strongly advocated going to Tapas Valencia instead, since I have never had tapas before. Somewhat to our disappointment, they served tapas only as an appetizer, and had a limited selection; so we ended up ordering one tapas dish (I had beef kabobs with a garlic horseradish sauce and caramelized onions), and my very first paella (paella valencia), with shrimp, mussels, clams, chicken, and veggies. We ate well, and headed back to the hotel. As we parked, and started for the gate, I realized I no longer had my wallet with me so back to the restaurant we went. The waiter had not found the wallet. Nor had the busboy. The owner/hostess suggested perhaps it was in the parking lot? So we all went and combed the parking lot once, twice. At this point I am beginning to think about the consequences I can t get home, because I can t get into the airport, since I have no ID. I have no money, but Judy can t wire the money to me via western union because I have no ID. I need money to buy greyhound tickets to get home on a bus and then there is the cancelling credit cards, etc. Panic city. While I was on my fourth circuit of the parking lot, the owner went back and checked the laundry chute. I had apparently carelessly draped the napkin over my wallet when paying the tab, and walked away and the busboy just grabbed all the napkins, wallet and all, and dumped it down the chute. Judy suggests I carry an alternate form of ID and at least one credit card in a different location than my wallet for future trips. If that was not excitement enough, yesterday, I got on the plane home, uneventfully enough. We took off, and I was dozing comfortably, when there were two loud bags, and the plane juddered and lister to the port. There was a smell of burning rubber, and we stopped gaining altitude. After making a rough about turn with the left wing down, the pilot came on the intercom to say We just lost our left engine, and we are returning to O Hare. We should be in the ground in two minutes . Hearing the in , a guy up front started hyperventilating, and his wife was rubbing his back. My feelings were mostly of exasperation, I had just managed to get myself situated comfortably, and now lord only knows when we would get another aircraft. When we landed, the nervous dude reached over and kissed his wife like he had just escaped the jaws of death. And he asked if any of us knew statistics, and if we were fine now. (I was tempted to state that statistics are not really predictive, but hey). It was all very pretty, with six fire engines rushing over and spraying us with foam and all. When we got off the plane the nervous dude headed straight to some chairs in the terminal, and said his legs would not carry him further. He did make it to the replacement plane later, though. Turns out it was a bird flying into the engine that caused the flameout. Well, at least I have a story to tell, though it delayed getting home by about three hours.

I recently wrote some suggestions about appropriate swap space sizes [1], the main point being that the advice to use 2* RAM size never really applied to Linux and a system that actually has swap space equivalent to 2* RAM in use will usually be too slow to use (and you are better off having the kernel OOM handler kill some processes). There were a number of responses to that post, Ingo suggested that significantly larger amounts of swap can be used for compiling [2]. I have had similar experiences when compiling KDE long ago on a machine with 96M of RAM. Even in 1999 96M was not a lot of memory for a machine running KDE, and when compiling MICO (the CORBA system) a single G++ process could use more than 96M of memory. Obviously a lot of swap was used, but performance didn’t seem too bad. I avoided touching the keyboard or mouse (so most of the X server was paged out) and allowed all memory to be used by the build process. It seems to me that the build process is not as intensive as other processes in terms of swap use because it is a chain of processes that call each other. You may have several instances of make running (launching a new instance for each directory as you descend the build tree is common) and then a gcc process that launches some child processes. But unless you use the -j option to make at most times during the build there is only one process with any serious work to do. So the parent processes can be entirely paged out to make room for the process that is doing the work. A counter example is a shell server that has many interactive users, on such a machine if it runs slow then the users start opening more sessions so that they can perform other operations while waiting for a slow command to complete, opening more sessions uses more memory and makes the problem worse. Ingo notes that he uses several hard drives to give good performance. The priority of swap spaces is listed in /proc/swaps and can be set by pri=N in the options field in /etc/fstab. If you have multiple hard disks then the performance of swapping on a system that is being used intensively for multi-tasking will increase significantly at a rate that would hopefully be close to linear scaling with the number of disks. If a system only has one active process that is being paged (which might be reasonably common as part of a build process) then the performance benefits for reading will be significantly reduced.

• [1] http://etbe.coker.com.au/2007/09/28/swap-space/


• [2] http://blog.windfluechter.net/index.php?/archives/71-Swap-sizes.html

Share This

• Is It Time for 64-bit on the Desktop?
  As unbelievable as it may seem, Windows XP and Windows Vista do not support PAE, and the only way to use more than 3GB of RAM is to install the 64-bit version of Vista...
• Hollywords
  I like the English version of genre. The original is a totally boring word, at least for a native speaker.

As I stand in a near cattle-car packed South Station, waiting while no trains arrive and no trains depart (”signal difficulties”), I wonder if forces are conspiring to make MBTA service so poor that it enters a death spiral of poor reputation for reliability, increasing fares, lower ridership, less revenue, etc.. One or two long delays or canceled trains in a short time span can be ignored, but after a while it may become impossible for the T to recover the lost goodwill. There’s a saying in trademark law, “once you’ve lost a customer, they’re often gone forever.” Are signal difficulties for real, by the way? More on this topic: this excellent op-ed in the Globe about the importance of mass transit to the local economy and life sciences in particular; contrast with this news about what the T plans as its next big project (hint: “T TV”). Not much good news here, either. Is anyone listening? Technorati Tags: MBTA, Boston, Mass Transit

Share This

As I stand in a near cattle-car packed South Station, waiting while no trains arrive and no trains depart (”signal difficulties”), I wonder if forces are conspiring to make MBTA service so poor that it enters a death spiral of poor reputation for reliability, increasing fares, lower ridership, less revenue, etc.. One or two long delays or canceled trains in a short time span can be ignored, but after a while it may become impossible for the T to recover the lost goodwill. There’s a saying in trademark law, “once you’ve lost a customer, they’re often gone forever.” Are signal difficulties for real, by the way? More on this topic: this excellent op-ed in the Globe about the importance of mass transit to the local economy and life sciences in particular; contrast with this news about what the T plans as its next big project (hint: “T TV”). Not much good news here, either. Is anyone listening? Technorati Tags: MBTA, Boston, Mass Transit

Share This

The plan for tomorrow includes possibly breaking my computer by soldering the JTAG connector onto the CPU board. Then there will be much debugging of mISDN drivers, and hopefully some rejoicing when I find out what makes the kernel hang on loading the layer 1 driver. This will also be an excellent opportunity to clean up my desk to make space for the soldering iron, and to build another Internet router in case anything goes wrong.

Sheep Farmers Daughter is an old favourite, which I have read lord only knows how many times. Elizabeth Moon has written a gritty, enthralling story of the making of a Paladin. This is the first book of a trilogy, and introduces us to a new universe through the eyes of a young innocent (which is a great device to introduce us to a universe from the viewpoint of someone who is not seeing it through eyes jaundiced by experience). For me, books have always been an escape from the humdrum mundanity of everyday existence. Putting myself in the shoes of a character in the story is the whole point; and this story excels there: it is very believable. Not many people can tell a tale that comes alive, and Ms Moon is one of them. An ex-marine, much of the detail of the military life of Paks has been drawn from Moon s own military experience. More than just that, the world is richly drawn, and interesting. I read this book in a hotel room in Chicago, since, as usual, there was nothing really interesting on TV, and I don t get the whole bar scene.

You may have noticed a range of things mounted on a tmpfs file system. tmpfs is actually more pervasive than it might appear. It was created to build a filesystem interface ontop of shared pages of RAM. This is useful because there are a number of ways to access shared RAM; notably SysV IPC Shared Memory (shmget, shmat, etc) and mmap of anonymous memory (MAP_ANONYMOUS). In order to have a consistent infrastructure implementing all of this the kernel abstracts RAM as a filesystem and your mapping as a file. To this end the tmpfs filesystem was created to present this abstraction. It deals with issues such as caching, security and the ability to swap pages in a consistent manner (i.e. using the same infrastructure every other file system does). Overall, tmpfs is implemented in mm/shmem.c. In the initialisation function the file system is mounted internally by the kernel with vfs_kern_mount so it is always around. It gets fairly complicated because it has to keep track of swapped out pages and the general complexity that come with implementing a full filesystem. As described above, there are several distinct ways to get access to this file system.

• The SysV IPC shared memory layer (ipc/shm.c) is a wrapper layer which manages IPC keys and internally plugs itself into the tmpfs code. Essentially creating a key with shmget() ends up calling ipc/shmem.c:newseg() which calls mm/shmem.c:shmem_file_setup() which then creates a struct file for the shared memory. This new file reuses the file operations from the tmpfs filesystem, but is not actually put into the file system (i.e. linked to the superblock). When shat() is called to attach to this memory the SysV IPC layer will internally translate this into a mmap of this file. This is what is used by the Shared Global Area of Oracle and some Java implementations, amongst many other things (see /proc/sysvipc/shm for a comprehensive list).
• The second common thing to do is an anonymous mmap. By default MAP_ANONYMOUS ignores the file-descriptor, but an analagous methodology is to explicitly map /dev/zero. If you look at the implementation of /dev/zero in drivers/char/mem.c you can see code such as

  static int mmap_zero(struct file * file, struct vm_area_struct * vma)
   
          int err;
          if (vma->vm_flags & VM_SHARED)
                  return shmem_zero_setup(vma);
  

  which ends up backing shared areas with tmpfs.
• The third way is actually mounting tmpfs and using it from userspace as a normal file system. glibc implements the POSIX shared memory call shm_open by creating and mmaping files on a tmpsfs mount (by default looking in /dev/shm). Other utilities also use the tmpfs semantics and mount it at other points; the obvious one of course being mounting /tmp on it!

tmpfs: more than meets the eye! Update: hopefully a little clearer, thanks to Helge Bahmann for comments.

I'm thrilled to find out that Facebook has a group named The CTRL-X R K appreciation society; I'm member #4. C-x r k is one of my favorite Emacs commands.

Top posters in a few Debian-related Planets:

$ planet-scores.sh 
Planet Debian-FR :
     19 Rapha l Hertzog
      4 Roland Mas
      3 Jean-Christophe Dubacq
      2 Gr gory Colpart
      2 Alexis Sukrieh

Sometimes I think this should be renamed Planet Buxy.

Planet Debian-FR (utilisateurs) :
     10 Julien Candelier
      8 Emilien Macchi
      4 Guilhem Bonnefille
      3 Shams Fantar
      1 Rapha l Hertzog
      1 Olivier Berger (perso)
      1 Jean-Christophe Dubacq
      1 Jean-Baptiste H tier (djib)
      1 Eric Veiras Galisson

Newly added contributors to that planet have all their recent articles aggregated, not only the ones they wrote since they were added.

Planet Debian :
     40 Christian Perrier
      2 Russell Coker
      2 Raphael Geissert
      1 Wouter Verhelst
      1 Steve Kemp
      1 Romain Francoise
      1 NOKUBI Takatsugu
      1 Michal  iha 
      1 John Goerzen
      1 Joey Schulze
      1 Gerfried Fuchs
      1 Fathi Boudra
      1 Enrico Zini
      1 Emanuele Rocca
      1 Dirk Eddelbuettel
      1 David Welton
      1 Christine Spang
      1 Antti-Juhani Kaijanaho
      1 Adam Rosi-Kessel

Planet "Christian loves rugby".

debian-community.org :
      4 Holger Levsen
      3 Andrew Donnellan
      2 Evgeni Golov
      1 Wolfgang Lonien
      1 Rapha l Hertzog
      1 Martin Albisetti
      1 Marcos Marado
      1 Jean-Christophe Dubacq
      1 Cord Beermann
      1 Benjamin A'Lee
      1 Andreas Putzo
$

I know I have an encoding problem on some planets, but that script is a very basic curl+shell+sed+grep+recode+sort+uniq pipeline, and I only use it for the amusement value. Maybe I'll recode it with a proper RSS parser some day if I feel utterly bored.

Someone asked me today if I was 21. I’m not sure where that idea came from, but no, I’m not 21.

xen-tools has just got a new command:

xen-resize-guest --hostname=foo.my.flat --increase=5Gb

That will take the existing guest foo.my.flat and magically resize the main disk image to be 5Gb larger. This works for LVM & loopback images, but not yet for users of EVMS. Whilst doing this job manually isn't terribly difficult it can be troublesome to perform all the steps in order without screwing up. Hence the new command. In other news I've managed to fix my broken greylisting implementation - so all the mails which were previously being queued upon klecker.debian.org should now be delivered/bounced. I believe the blame here was 50/50 me and exim's back-off behaviour, but I'll know better in the future. Remember me when you come to choose your next anti-spam service. It copes beautifully with a sustained delivery rate of 300-700 messages a minute when queues suddenly restart delivery ;) TODO: Catchup on mail. Implement message tagging for mutt to better keep track of items which are pending/claimed by me. Until we get RT.

apt-xapian-index: search as you type I've recently posted:

• an introduction of apt-xapian-index;
• an example of how to query it;
• a way to add simple result filters to the query;
• a way to suggest keywords and tags to use to improve the query.
• a way to search for similar packages.
• a way to implement an adaptive cutoff on result quality.
• a smart way of querying tags

Note that I've rewritten all the old posts to only show the main code snippets: if you were put off by the large lumps of code, you may want to give it another go. Today I'll show how to implement a very attractive feature for a user interface: search as you type. The idea is that you don't need to press enter to fire up a query: instead, the results materialise in front of your eyes as you type them. The example I created uses curses, but the idea is good on any interactive user interface. The main thing to keep in mind with search as you type is that the last word is likely to be partially typed, unless maybe some timeout expired since the user's last keystroke. Xapian comes into help here, as it allows us to expand the partially typed word into an OR query with all the terms that start with it. This means that if we are typing, for example, "progr", we can turn the query into "program OR programmer OR programming OR programmed [...and so on...]". I won't show the UI code, except a simple input loop that triggers the query at every keystroke: The interesting part is in the update function. First we split the line in words and convert the words into a query: Then we expand the last word with all possible completions: Now we can build the query. Of course you can add all other sorts of things to the query, for example a boolean expression of tag filter like in axi-query-pkgtype.py; Xapian will cope. Finally the query. For bonus points you can do the adaptive cutoff trick to discard bad results. In my case, since I don't implement scrolling of results, I also limit them to what fits in the window: Finally, draw the results on screen: That's it, try it out. You can use the wsvn interface to get to the full source code and the module it uses. You can see a similar technique working in goplay, where it is also integrated with an interactive tag filter.