Haven't written anything for way too long, I tried to put something down again last week, and this is what I came up with. It's not influenced by anything special, it's not as well as I usual prefer, but still, I thought I should share.

life is strange
it starts off without knowledge
in the end you die love is strange
it starts off with bad heartaches
in the end - alone things are strange
still we try to survive it
go on day by day

From November 9 till November 15 I attended SAGE Days 6 in Bristol, UK. It was a conference and a coding sprint for the SAGE project, which wants to build something comparable to Mathematica, Maple, Matlab and Magma built only from open source components without reinventing the wheel. Sage also provides excellent novel implementations for many mathematical algorithms (not found in Mathematica/Maple etc.).

I was invited to give a talk about SymPy, which is a Python library for symbolic mathematics, that we are doing.

Here are my notes from each day:

Saturday: one, two
Sunday: one, two
Monday, Tuesday, Wednesday: one

Impressions

Very positive. I started SymPy two years ago, because I wanted to play with symbolic mathematics in Python, see for example my presentation I gave at this conference for details. SAGE at that time was just able to do some mathematics things, but it was very weak in calculus, which is the SymPy's main domain. This has changed last half a year, when SAGE people managed to wrap Maxima in Python (which I thought to be completely impossible), so I started to follow SAGE development more closely. After the SD6, I must say I became very excited about the project.

One problem with mailinglists, IRC and other online interaction is that it's very difficult for me to get an impression about the people and the project. Being able to meet the developers and discuss with them face to face gives me the impression very quickly and very accurately.

By far the biggest guarantee, why it is worthy for me to contribute to SAGE, is the project leader, William Stein. He is very rational and pragmatic (I like these two properties) and after many discussions with him, I came to realize that he has basically identical views on the important things as I do and were I on his place, I would do the same decisions as he did and does. That's very nice, because I can concentrate my energy on things that I like to improve and don't have to worry about other things, because I know he will do it right.

The other SAGE developers are experts and with similar attitude as William has. It's enthusiastic to be among people who make things happen. For example one of the authors of Cython, Robert, implemented during SD6 a very nice HTML output, that shows Cython code, with colors according to how many Python API calls are called on that particular line, and by clicking that line it shows the corresponding C code.

SAGE project has a high aim and it stricly goes for it, without looking too much to the right or left, and that's how it should be. And it does produce a lot of very useful and high quality stuff along the way, for example Cython (probably the best wrapper for C/C++ things now, only pypy could possibly beat it, but that's still more a research project) or the SAGE notebook, which looks like a Mathematica notebook, but better and in a browser (together with a revision history, sharing, SSL encryption, etc.).

The only little problem is that currently SAGE developers are all mathematicians and as is well-known, mathematicians looks at mathematic from a very different prospective than physicists. :) And so I need calculus, advanced calculus and only when this is working, and working well, I can build on it some more advanced features. SAGE currently goes a little the other way - it has a lot of advanced features, from number theory, modular forms, elliptic curves, etc., but the basic calculus still needs a lot of improvements. SAGE wraps Maxima, because Maxima is quite fast, very well tested, so it works well. It's difficult to extend and written in LISP and that's very bad. That's where SymPy could help - it's in Python, very easy to extend, but currently slower than Maxima (rewriting parts of SymPy using Cython, or even C directly, will make it faster, hopefully as fast as Maxima or faster).

It's not yet in Debian, but SAGE people are working on it. It's not an easy task unfortunately.

Conclusion

SAGE is a very promising young project and I think it will succeed to provide an open source alternative to Maple, Matlab, Mathematica and Magma.

Today I was reminded about a thought I was having earlier this year by a twitter from Garrick Van Buren about the new ultra-thin MacBooks. It seems altogether likely that the next laptop computer that I buy will have no moving parts. Currently the only moving parts in the MacBook that I have now are in the optical drive, the fan, and the hard drive. This is of course not counting the moving parts which I move myself - the buttons to actually interact with it and the lid. Apparently the optical drive in the laptop is going the way of the floppy drive in laptops, so there is only the fan for the CPU and the hard drive. Hard drives are also heading toward the realm of non-moving parts with solid state drives gaining acceptance and size. You can now get a 32GB solid state flash drive for pretty cheap, and they are sure to go in the direction all storage goes - faster, larger, and cheaper. That only leaves the fan which cools the CPU. It is not impossible to run a high-end computer without a fan nowadays, but unfortunately the heatsinks required in order to keep the most crucial part of the computer without burning up. The OLPC hardware is already in some ways the wave of the future - there are no moving parts at all. Unfortunately it is also completely underpowered and it's not possible to run a ton of programs on it. I'm not sure that a solid-state laptop for the general public will ever be possible with the general increase in computing power, but if it happens, I would bet it happens in the next 5 years.

People will be pointing out that the optical drive being missing is a new and novel concept and that Apple is pushing the boundaries of laptops, but they are hardly the first ones to ship a laptop without a optical drive. The world of sub notebooks have been taking out the optical drive in their smaller models for a while now. One model that I've seen around quite a bit is the Sony PictureBook which got quite a lot of press because it featured the Transmeta Crusoe chip. There are also a number of other sub notebooks which don't have a drive. However, I don't believe that the drives will be replaced by flash drives or network installs - there will always be a need for boot media for completely broken computers. The common solution in the sub notebook world is to just have a drive which attaches when it's needed, in the mode of the first drives. The solution which uses flash drives is not likely to happen anytime soon - software isn't getting any smaller, and the cost of flash media isn't falling quickly enough to catch up with the cost-effectiveness of pressing CDs.

You may have noticed a range of things mounted on a tmpfs file system. tmpfs is actually more pervasive than it might appear. It was created to build a filesystem interface ontop of shared pages of RAM. This is useful because there are a number of ways to access shared RAM; notably SysV IPC Shared Memory (shmget, shmat, etc) and mmap of anonymous memory (MAP_ANONYMOUS). In order to have a consistent infrastructure implementing all of this the kernel abstracts RAM as a filesystem and your mapping as a file. To this end the tmpfs filesystem was created to present this abstraction. It deals with issues such as caching, security and the ability to swap pages in a consistent manner (i.e. using the same infrastructure every other file system does). Overall, tmpfs is implemented in mm/shmem.c. In the initialisation function the file system is mounted internally by the kernel with vfs_kern_mount so it is always around. It gets fairly complicated because it has to keep track of swapped out pages and the general complexity that come with implementing a full filesystem. As described above, there are several distinct ways to get access to this file system.

• The SysV IPC shared memory layer (ipc/shm.c) is a wrapper layer which manages IPC keys and internally plugs itself into the tmpfs code. Essentially creating a key with shmget() ends up calling ipc/shmem.c:newseg() which calls mm/shmem.c:shmem_file_setup() which then creates a struct file for the shared memory. This new file reuses the file operations from the tmpfs filesystem, but is not actually put into the file system (i.e. linked to the superblock). When shat() is called to attach to this memory the SysV IPC layer will internally translate this into a mmap of this file. This is what is used by the Shared Global Area of Oracle and some Java implementations, amongst many other things (see /proc/sysvipc/shm for a comprehensive list).
• The second common thing to do is an anonymous mmap. By default MAP_ANONYMOUS ignores the file-descriptor, but an analagous methodology is to explicitly map /dev/zero. If you look at the implementation of /dev/zero in drivers/char/mem.c you can see code such as

  static int mmap_zero(struct file * file, struct vm_area_struct * vma)
   
          int err;
          if (vma->vm_flags & VM_SHARED)
                  return shmem_zero_setup(vma);
  

  which ends up backing shared areas with tmpfs.
• The third way is actually mounting tmpfs and using it from userspace as a normal file system. glibc implements the POSIX shared memory call shm_open by creating and mmaping files on a tmpsfs mount (by default looking in /dev/shm). Other utilities also use the tmpfs semantics and mount it at other points; the obvious one of course being mounting /tmp on it!

tmpfs: more than meets the eye! Update: hopefully a little clearer, thanks to Helge Bahmann for comments.

• Top Ten Worst Album covers of all time
  Some serious mustaches in there.
• CVE-2007-5795 (Emacs)
  Fixed in unstable.
• More CVSS corrections

I found some program today. It had a file, buttons.h which was autogenerated by:
uic -o buttons.h -embed buttons pics/attach.png pics/detach.png
Unfortunately, those pictures wasn’t included in the tarball - and as it is seen earlier, uic doesn’t generate endian-agnostic code. So what do smart man do now? Write upstream and wait? Could be done. Or figure out how to de-uic the header file. Qt and #kde4-devel to the rescue. #include "buttons.h"
#include "qimage.h" int main()

uic_findImage("attach.png").save("attach.png","PNG");
uic_findImage("detach.png").save("detach.png","PNG");

compiled with g++ -I /usr/include/qt3/ test.cpp -lqt-mt and now there is images that can be uic’ed.

Last year, as we were planning our move, I knew we would need to buy a new TV antenna. According to AntennaWeb (a very useful site), we are in the violet (or "fat chance") zone for several TV stations. Add to that the fact that we have a tin-covered building pretty much in the direct path to several of the transmitter. Metal buildings often cause "multipathing", where a signal bounces off buildings and arrives multiple times at the antenna. This causes ghosting on analog signals and can cause problems getting a signal lock on digital ones. We also have a fairly long run from the antenna to the receiver (a MythTV box) and are wanting to receive digital signals. Not only that, but the TV transmitters are in different areas about 40 degrees apart, and most high-gain antennas are also highly directional. So it's a difficult situation.

So in our situation, an outdoor antenna is a must. At our old place, we had used the best outdoor antenna Radio Shack sold. Despite having more friendly reception conditions, it didn't work well.

After a good deal of research, I bought a Terk TV38 outdoor antenna from Amazon. It comes with:

• The antenna, at a large 12.5 feet long, 9.25 feet wide, 2.6 feet high (though packaged in a much smaller carton)
• A 5-foot mast
• Chimney mount
• Wall mount
• Transformer (for direct connection to coax cabling)
• Various mounting hardware

It comes with the hardware you need to mount it in an attic or attach it to an exterior wall or chimney in most situations. However, we wanted to attach it to the roof directly, so we purchased a tripod mount from RadioShack for that purpose. I had also purchased some quad-shield RG6 coax from Cat5ECableGuy.Com.

I assembled the antenna on the ground. That took some time, but wasn't difficult. But this was in winter, a fairly wet winter, and I knew that weather wouldn't cooperate well enough for us to get the antenna up on our roof safely some Saturday for a few weeks. So I thought I'd just leave the antenna on the ground for a little while, hook it up, and see if we get anything. I put it on the ground, aimed it about in the middle of the TV transmitters, and went to set up the MythTV.

I was shocked to discover that, even with the antenna on the ground, I got a perfect digital picture on all channels but one. Once we got the antenna mounted on the roof, reception has been perfect. I've had no need for actuators (to rotate the antenna) or amplifiers. Just the antenna itself, even with the long coax run, has performed quite well.

For anyone that needs an outdoor antenna, I highly recommend the Terk TV38.

I request assistance with maintaining the dctrl-tools package. There are several tasks that could use more manpower (in no particular order):

• Writing test cases
  One could mine the BTS for past bug reports and create regression tests for them.
  One could use standard black-box and white-box testing techniques to generate general tests.
• Writing documentation
  The whole suite of tools could use a unified tutorial manual on how to best use it. The current documentation is reference material in the man pages.
• Internationalise the man pages
  Use po4a?
• Swatting the BTS wishlist entries
  I’ve kept the BTS clean of actual bugs pretty well, but there are a number of wishlist reports still outstanding.
• Take over maintaining the debian/ directory
  If you commit to maintaining it (and I trust your judgment), you’ll get last say in that part of the package (including deciding what helper to use).
• Whatever you wish
  Discuss on the dctrl-tools-devel mailing list first though.

Eventually I’d like to pass the package on to competent successors, but I have too much emotional attachment to the package to do that without a transitional period where I still retain a veto on what goes in the package. I also have some ideas for future tools that I’d like to be
able to concentrate on, and having co-maintainers might allow that. The package is now under Git in collab-maint. See the new README.Debian for information and a push-access code of conduct.

What a week. Most notably, I am still in a state of partial disbelief that my case with Blue Cross Blue Shield is now on a path toward resolution and I don t have do battle with them, or at least in the same vigorous, soul-crushing way I have been engaged in since May (they still have not paid all the bills but at least part of them are resolved). But the very same day that I received the good news about BCBS, I received not so good news from my dermatologist about the black-blue mole that lies right under my left eye. He wants it gone, removed. So my whole fight with BCBS concerned claims over pre-existing conditions related to my moles. My claim was that I never had a problem with them until I had BCBS insurance and that is indeed the case. Soon after I first started having problems, I went on mole patrol and found a suspicious mole on my sister, which turned out to be melanoma. Now whenever I see a dermatologist, they all say, given my family history, given I grew up in the tropics, and given the odd size and very dark color of my mole, that it has to go. In fact the last dermatologist said with the firmest of tones: there is no doubt in my mind that it must be removed. The words stung because, well, I am pretty attached to the mole. I have lived with it for my whole life and it is fundamentally, even if somewhat mysteriously, linked to my sense of self. If it goes, I would feel like a very important part of me would also vanish. On more prosaic grounds, I can t help think that part of the drive to remove such a spot is just part of somewhat unhealthy social trend toward measuring most everything in finer and finder gradations of risk. Given that this mole is on my face and I see my face at least 2x a day in the mirror, I would probably catch anything if it started to change but given our risk society it is unsurprising that every American dermatologist keeps saying it is time to bid adieu to it. Given my background in medical anthropology I can t help conceptualize all this advice within these much larger social trends and then I go back and forth endlessly in my head as to whether I should remove it because it is safe and right and sound or whether I am caving in to somewhat irrational trend toward the risk management of everything. Finally I fear my insurance company again. Though I have a good job and decent insurance, I may not have good enough insurance for what this procedure entails.
Because of the size and location of the mole (large and close to my eye), if removed it must be done by a plastic surgeon. And plastic surgery in the US and plastic surgery in NYC costs a very pretty penny. While I would plan on getting pre-approval for the surgery, I somehow have the feeling that they will not pay the full cost and I am going to be left with another bill. And I am trying to stop the unnecessary onslaught of bills. So should it stay or should it go? I bet that I will lean toward the latter but not without a lot more thoughts, internal cries of resistance, and sense that even if I decide it is the best thing, I will lose a small piece of myself once removed

A while back I posted about a couple of my irritations with GNU Screen. One of my irritations was the failure to reattach to sessions by name, if common prefixes were in use. For example with the following two (detached) sessions:

There are screens on:
        24419.abc       (Detached)
        24395.abcd      (Detached)
2 Sockets in /var/run/screen/S-skx.

The naive "screen -R abc" fails. Yesterday whilst looking over the screen bug list I came up with a patch. It isn't ideal as it introduces a new failure case, but I believe it is a step in the right direction and better than the current situation. See attachment to #361274 for the code. Also I patched screen so that #330036 is now fixed, and the blankerprg primitive works as expected. Finally I closed #317450 (with a version) as it has been fixed since Etch. Fun stuff. In the spirit of completeness I should say I had a stab at #447210 which is tilde (~) expansion in the chdir primitive, but gave up after a while as the code got too messy even for me. The trivial s/~/getenv("HOME")/ approach works fine for the simple case, but dealing with the expansion of strings such as ~foo/bar/ gets messy quickly. I can offer my patch if there is any interest though as a stop-gap measure. Now I'm almost tempted to look over another package's bugs, but I think I'd rather eat pie & drink beer... must. stop. talking. about. pies. Update: Patch for tilde expansion submitted to #447210 - tested and seemed to cover all cases. Now time for beer!

GNU Screen rocks, in general, but recently I've been using it a lot for custom applications and have discovered a pair of annoying bugs. If you're not familiar with it then please read this GNU Screen tutorial - it really is worth getting to know! Anyway onto the bugs:

• Session names may not start with numbers.
• Session names must be globally unique.

The two are related, but I'm not yet sure whether I should report bugs against the Debian package and the code is sufficiently cryptic that I cannot create a fix yet. Taking the bugs in reverse order please try this:

screen -S foo
[detach]
screen -S foo2
[detach]

At this point you should have two screen sessions "foo" and "foo2". You should be able to attach to them by running "screen -R foo" or "screen -R foo2". Lets try that:

skx@vain:~$ screen -R foo
There are several suitable screens on:
        22317.foo       (Detached)
        22342.foo2      (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.

Even though foo should be sufficient to identify a unique screen, the first one, it doesn't let you attach. Nasty. (Yes, you can attach to it if you use the number/number+name:

screen -R  22317.foo 

The second issue is related. Create a screen session with "screen -S 222". Now try to attach to it with "screen -R 222" - instead of attaching it gives you a brand new screen. Ugh.

Danny Angus writes about the potential threat posed by small storage devices with large capacity [1]. His post was prompted by a BBC article about Hitachi’s plans for new hard drives [2], they are aiming for 4TB of data on a single drive by 2011 and a 1TB laptop drive. One thing I noticed about the article is that they made the false claim that current drives are limited to 1TB, the storage capacity is determined by the total surface area which is proportional to the square of the radius and the height of the drive (AFAIK there are no practical limits to the number of platters apart from the height of the drive). So if a 5.25 inch hard drive was to be manufactured with today’s technology it should get a capacity equivalent to at least three times the capacity of the larger 3.5 inch drive. The reason that 5.25 inch drives are not manufactured is that for best performance you want multiple spindles so that multiple operations can be performed concurrently. Using 3.5 inch drives in servers allows the use of more disks for the same amount of space in the rack and the same amount of power. The latest trend is towards 2.5 inch (Small Form Factor AKA SFF) disks for servers to allow more drives for better performance. With 3.5 inch disks a 1U system was limited to 3 disks and a 2U system was often limited to 4 or 5 disks. But with 2.5 inch drives a 2U server can have 10 drives or more. I know of one hardware vendor that plans to entirely cease using 3.5 inch drives and claims that 2.5 inch disks will give better performance, capacity, and power use! In regard to Danny’s claim (which is entirely correct) about the threat posed by insiders. I don’t believe that a laptop with 1TB of capacity is the threat. In a server room people notice where laptops get connected and there are often strictly enforced policies about connecting machines that don’t belong to the company. I believe that the greatest threat is posed by USB flash devices. For example let’s consider a database with customer name (~20B), birth-date (10B), address (~80B), phone number (~12B), card type (1B), card number (16B), card expiry (5B), and card CVV code (3B). That’s ~155 bytes per record in CSV or TSV format. If you have data for a million customers that’s 155M uncompressed and probably about 50M when compressed with gzip or WinZip (depending on which platform is being ripped). No-one even sells a USB flash device that is smaller than 50M, I recently bought a 2G flash device that was physically very small and cheap (it was in the bargain bin). The next issue is, what data might be worth stealing that is large enough to not fit on a USB device? I guess that if you want to copy entire network file shares from a corporation then you would need more than the 16G that seems to be the maximum capacity of a USB device at the moment. Another theoretical possibility would be to copy the entire mail spool of a medium to large ISP. For the case of a corporate file server you could probably get the data at reasonable speed, 1TB of data would take 10,000 seconds or 2.8 hours to transfer at gigabit Ethernet speeds (if you max out a GigE link - it could be as much as five times that if the network is congested or if the server is slow). It’s doable, but it would be a rather tense three or more hours waiting by an illegally connected laptop. For the mail server of a large ISP there is often no chance of getting anywhere near line speed, it’s lots of small reads and seek performance is the bottleneck, such servers are usually running close to capacity (and trying to copy data fast would hurt performance and draw unwanted attention). Another possibility might be to copy the storage of an Intranet search device. If a company has a Google appliance or similar device indexing much of their secret data then copying the indexes would be very useful. It would allow offline searches of the corporate data to prepare a list of files to retrieve later. It would probably be more useful to get online access to the data from a remote site. I expect that an unethical person could sell remote access to someone who is out of range of extradition. All that would be required would be to intentionally leave a flaw in the security of the system. In most large corporations this could be done in a way that is impossible to prove. For example if management decrees that the Internet servers run some software that is known to be of low quality then a hostile insider could make configuration changes to increase the risk - it would look like an innocent mistake if the problem was ever discovered (the blame would entirely go to the buggy software and the person who recommended it). A large part of the solution to this problem is to hire good employees. The common checks performed grudgingly by financial companies are grossly inadequate for this. Checking whether a potential employee has a criminal record does not prevent hiring criminals, it merely prevents hiring unsuccessful criminals and people who have not yet been tempted enough! The best way to assess whether HR people are being smart about this is to ask them for an estimate of how many criminals are employed by the company. If you have a company that’s not incredibly small then it’s inevitable that some criminals will be employed. Anyone who thinks that it is possible to avoid hiring criminals simply isn’t thinking about the issues. I may write more about this issue in a future post. Another significant part of the solution to the problem is to grant minimum privileges to access data. Everyone should only be granted access to data that they need for their work so that the only people who can really compromise the company are senior managers and sys-admins, and for best security different departments or groups should have different sys-admin teams and separate server rooms. Of course this does increase the cost of doing business, and probably most managers would rather have it be cheap than secure.

• [1] http://blog.killerbees.co.uk/2007/10/all-your-data-are-belong-to-us.html


• [2] http://news.bbc.co.uk/1/hi/technology/7044606.stm

Share This

I don’t spend a lot of time customizing my “working” environment nowadays, like enhancing vim configuration, or tweaking the shell. But when I read MadCoder’s zsh git-enabled prompt, I though it was too convenient to not have something like that. Except I don’t work with git only (sadly, but that’s changing), and I don’t like colours in prompt (and a 2 lines prompt is too much). Anyways, since I have a bunch of directories in my $HOME that contain either svk, svn, mercurial, or git working trees, I thought it would be nice to have some information about all this on my prompt. After a few iterations, here are the sample results:

mh@namakemono:~/dd/packages$
mh@namakemono:(svn)~/dd/packages/iceape[trunk:39972]debian/patches$
mh@namakemono:(svk)~/dd/packages/libxml2[trunk:1308]include/libxml$
mh@namakemono:(hg)~/moz/cvs-trunk-mirror[default]uriloader/exthandler$
mh@namakemono:(git)~/git/webkit[debian]JavaScriptCore/wtf$


The script follows, with a bit of explanation intertwined.

_bold=$(tput bold)
_normal=$(tput sgr0)


tput is a tool I only dicovered recently, and avoids the need to know the escape codes. There are also options for cursor placement, colours, etc. It lies in the ncurses-bin package, if you want to play with it.

__vcs_dir()
  local vcs base_dir sub_dir ref
  sub_dir()
    local sub_dir
    sub_dir=$(readlink -f "$ PWD ")
    sub_dir=$ sub_dir#$1
    echo $ sub_dir#/
  


We declare as much as possible as local (even functions), so that we avoid cluttering the whole environment. sub_dir is going to be used in several places below, which is why we declare it as a function. It outputs the current directory, relative to the directory given as argument.

  git_dir()
    base_dir=$(git-rev-parse --show-cdup 2>/dev/null) return 1
    base_dir=$(readlink -f "$base_dir/..")
    sub_dir=$(git-rev-parse --show-prefix)
    sub_dir=$ sub_dir%/
    ref=$(git-symbolic-ref -q HEAD git-name-rev --name-only HEAD 2>/dev/null)
    ref=$ ref#refs/heads/
    vcs="git"
  


This is the first function to detect a working tree, for git this time. Each of these functions set the 4 variables we declared earlier: vcs, base_dir, sub_dir and ref. They are, respectively, the VCS type, the top-level directory of the working tree, the current directory, relative to base_dir, and the branch, revision or a reference in the repository, depending on the VCS in use. These functions return 1 if the current directory is not in a working tree of the currently considered VCS.
The base directory of a git working tree can be deduced from the result of git-rev-parse --show-cdup, which gives the way up to the top-level directory, relative to the current directory. readlink -f then gives the canonical top-level directory. The current directory, relative to the top-level, is simply given by git-rev-parse --show-prefix.
git-name-rev --name-only HEAD gives a nice reference for the current HEAD, especially if you’re on a detached head. But this can turn out to do a lot of work, introducing a slight lag when you cd for the first time in the git working tree, while most of the time, the HEAD is just a symbolic ref. This is why we first try git-symbolic-ref --name-only HEAD.

  svn_dir()
    [ -d ".svn" ] return 1
    base_dir="."
    while [ -d "$base_dir/../.svn" ]; do base_dir="$base_dir/.."; done
    base_dir=$(readlink -f "$base_dir")
    sub_dir=$(sub_dir "$ base_dir ")
    ref=$(svn info "$base_dir" awk '/^URL/ sub(".*/","",$0); r=$0 /^Revision/ sub("[^0-9]*","",$0); print r":"$0 ')
    vcs="svn"
  


Detecting an svn working tree is easier : it contains a .svn directory, be it top-level or sub directory. We look up the top-level directory by checking the last directory containing a .svn sub directory on the way up. This obviously doesn’t work if you checkout under another svn working tree, but I don’t do such things.
For the ref, I wanted something like the name of the directory that has been checked out at the top-level directory (usually “trunk” or a branch name), followed by the revision number.

  svk_dir()
    [ -f ~/.svk/config ] return 1
    base_dir=$(awk '/: *$/ sub(/^ */,"",$0); sub(/: *$/,"",$0); if (match("'$ PWD '", $0"(/ $)")) print $0; d=1; /depotpath/ && d == 1 sub(".*/","",$0); r=$0 /revision/ && d == 1 print r ":" $2; exit 1 ' ~/.svk/config) && return 1
    ref=$ base_dir##*

    base_dir=$ base_dir%%
*
    sub_dir=$(sub_dir "$ base_dir ")
    vcs="svk"
  


svk doesn’t have repository files in the working tree, so we would have to ask svk itself if the current directory is a working tree. Unfortunately, svk is quite slow at that (not that it takes several seconds, but that induces a noticeable delay to display the prompt), so we have to parse its config file by ourselves. We avoid running awk twice by outputing both the informations we are looking for, separated by a carriage return, and then do some tricks with bash variable expansion.

  hg_dir()
    base_dir="."
    while [ ! -d "$base_dir/.hg" ]; do base_dir="$base_dir/.."; [ $(readlink -f "$ base_dir ") = "/" ] && return 1; done
    base_dir=$(readlink -f "$base_dir")
    sub_dir=$(sub_dir "$ base_dir ")
    ref=$(< "$ base_dir /.hg/branch")
    vcs="hg"
  


I don’t use mercurial much, but I happen to have exactly one working tree (a clone of http://hg.mozilla.org/cvs-trunk-mirror/), so I got some basic information. There is no way we can ask mercurial itself for information, it is too slow for that (main culprit being the python interpreter startup), so we take the informations we can (and since I don’t know much about mercurial, that’s really basic). Note that if you’re deep in the VFS tree, but not in a mercurial working tree, the while loop may be slow. I didn’t bother much looking for a better solution.

  git_dir
  svn_dir
  svk_dir
  hg_dir
  base_dir="$PWD"


Here we just run all these functions one by one, stopping at the first that matches. Adding some more for other VCS would be easy.

  echo "$ vcs:+($vcs) $ _bold $ base_dir/$HOME/~ $ _normal $ vcs:+[$ref]$ _bold $ sub_dir $ _normal "

PS1='$ debian_chroot:+($debian_chroot) \u@\h:$(___vcs_dir)\$ '


Finally, we set up the prompt, so that it looks nice with all the gathered information. Update: made the last lines of the script a little better and factorized.

The good: I had fun building an enclosure for my Atari <-> PC serial adapter board out of legos, and I love the result. Been too long since I played with legos. I need to dig up the rest of my legos so i can make some enclosures for my ARM boards. The bad: I picked up a 1 gigabyte USB drive in the shape of an off-brand lego block today. The price was right (< $10), but the implementation is depressing -- the fake legos don't stick well and so the cap can't be mounted on the drive body when it's in use. And the size isn't lego standard, so it can't attach to other blocks. The ugly: In a better world, Lego would be an open standard. The patent expired in 1988 (according to Wikipedia. But the company still has lawyers, who seem busy doing what (corporate) lawyers do. Meh. Legos were more fun to play with when playing with them didn't involve worrying about patents and copyrights and the market's propensity to deter standardisation.

If you’re reading mails offline on Linux (or if you’re browsing from GMail,) and you have some contacts using a Mac who sends you emails with attachments from Mail.app, you may some of the time get attachments that may seem to be unreadable by the intended application (like, when some Mac guy sends you a mindmap file in FreeMind that you can read in Ubuntu.) I recently hit this snag at work, so I did a little looking around to see why this is so, and how to fix it. The first thing I did, after downloading the attachment, was to obviously pager it. What seems to be a single attachment is in fact a couple of MIME objects, one an application/applefile’ object, and an application/octet-stream’ object. Since both are Base64-encoded parts, I can run a simple filter (using Perl’s MIME::Base64::decode_base64 routine) on both to get the raw parts. It turns out that the first part is an AppleDouble-encoded resource fork that describes the metadata for the actual file (called the data fork in AppleDouble parlance.) One can read this on Linux via Perl using the Mac::AppleSingleDouble module to get Finder information, among other things. However, for non-Mac users, what they’re really interested in getting is the the data fork. This can be done by using Email::MIME on the original message to get the data fork subpart, then filtering it through MIME::Base64. (Or you can cheat by using the already-downloaded AppleDouble attachment and extract+decode the octet-stream part as mentioned earlier.)

Paul Russell writes about his 3-yearly laptop replacement at IBM [1]. It probably makes some sense to replace laptops periodically for a large company, but if you are buying for personal use then it makes sense to try and get a longer life out of an expensive machine. I think that aiming for 6 years is quite reasonable with today’s hardware - you should be able to buy a new machine now and have it last 6 years or buy a 3yo second-hand machine and hope to have it last 3 years (most second-hand laptops on sale in every place other than Ebay were trophies for managers and never had any serious use). If you are going to buy a second-hand laptop the first thing to consider is PAE support. If you get a laptop without PAE support (I think that means all Pentium-M CPUs) then you will not have proper Xen support (it seems that all distributions have abandoned Xen support for PAE for the moment). This may not be a big deal if you don’t want Xen, but if you are a programmer then you probably do want Xen (even if you don’t realise it yet). The next issue is support for the AMD64 instruction set. 32bit laptops are going cheap at the moment but if you buy one you will be significantly limited as to what software you can run at some future time (my 32bit laptop is doing well at the moment apart from the lack of PAE support). If you are buying a new laptop then the first thing to consider when planning a long life is the warranty. In my experience most computer gear does not need a long warranty, if it survives 3 months then it’ll probably last until it’s well obsolete. Laptops however periodically wear out if used seriously, I average one warranty replacement of a Thinkpad keyboard every two years and I have had a few motherboard replacements (the lighter Thinkpads flex and they eventually break inside if you use them on enough trains, trams, buses, planes, etc). On one of the Lenovo T series Thinkpads that I saw advertised (one that I would consider if I wanted a new laptop now) there was an offer to spend an extra $350AU to get extend the warrantee from 1 year to 5 years (according to my understanding of the confusing text on the web site) on a laptop that cost $3050. An increase in the purchase price of 12% for the extra warranty is a bargain (I know that for my use they would lose money on the deal). Repair of a laptop is generally very expensive, any serious damage to a laptop that is more than 18 months old will generally mean that the replacement cost is less than the repair cost. The next thing to consider is the screen resolution. After purchasing a laptop you can upgrade the RAM and the hard drive, the CPU power of all modern machines is great enough that for most typical use it’s difficult to imagine any need to upgrade. But screen resolution is something that can never be good enough and can never be improved after purchase. Lenovo is offering T series Thinkpads with 1920×1200 resolution for $4000AU and 1680×1050 resolution for $3050AU. That’s 31% more pixels for 31% more money and seems like a good deal to me. I believe that a larger display can significantly increase productivity [2] so it seems that the extra expense would be a good investment if you plan to earn money from work you do on your laptop. As a point of reference a desktop monitor from Dell (who seems to be the cheapest supplier for such gear) with resolution of 1920×1200 will cost at least $1000AU. The hard drive capacity should not be an issue, it seems that 100G is about the minimum size. The 60G drive in my current Thinkpad is adequate for my development work (including several Xen instances and some ISO files for a couple of distributions) so unless you plan to collect MPEG4 files of TV series and store them on your laptop I can’t imagine 100G being much of a limit. Also external storage is getting quite cheap, 2GB USB flash devices are now in the bargain bin of my local electronics store and USB attached hard drives with capacities of 40G or more are getting cheap. Also with a Thinkpad replacing a hard drive is really easy and does not risk damage to the drive or the rest of the laptop (I don’t know how well other brands rate in this regard). For RAM you can buy a model with a large memory module in socket 0 (or attached to the motherboard). Adding new RAM later is easy to do. Just try and avoid purchasing a memory capacity that involves having all sockets filled with modules that are not of the maximum size - it’s annoying if you have to try and sell modules on Ebay after you buy a memory upgrade. Finally one mistake I made in the past was to not get all the options for the motherboard. Make sure that every option for Ethernet ports and 802.11 type protocols is selected. It might sound like a good idea to save $100 or so on not getting one of those options, but if you end up repeatedly plugging a CardBus or USB device for many years you will regret it. Also external devices tend to break or get lost. Rusty documents his laptop replacement as a time for spring-cleaning. I use LVM for the root filesytem on my Thinkpad so that I can easily install a new distribution (or a new version of a distribution) at any time. I’ve been through that spring-cleaning a couple of times on my current Thinkpad without needing new hardware. From a quick view of the Lenovo site it seems that an ideal new Thinkpad that would last me 6 years would cost about $4500 while one that would last me 2 years would cost $1600 (and have a significantly lower screen resolution). A Thinkpad that would last 6 years and not be so great (but still better than the cheap option) would cost about $3500. Update: One significant issue is the life expectancy of laptop batteries. If you use a laptop for mobile use (as opposed to just moving between desks occasionally) then you are probably familiar with the problem of laptop batteries that discharge after 10 minutes. Last time I checked the warranty on Thinkpad batteries was 1 year or 300 charges (whichever comes first). My experience is that after 300 full cycles a Thinkpad battery will only last for a small fraction of the original charge time. When buying a laptop I suggest getting a spare battery at the time of purchase. The spare battery may last longer than the battery that is shipped with the laptop and two batteries means that you have twice the number of charge cycles before they are both useless. Batteries apparently don’t last long if completely discharged, so charge them up before storing them and periodically charge them if they have been left unused for any length of time (maybe every second or third month). With a Thinkpad it seems quite safe to change the battery while the machine is plugged in to mains power and running (I expect that Lenovo doesn’t recommend this though). You should probably plan to have a battery die every three years of use (or sooner if you do a lot of travelling). So one spare battery may last you 6 years of use but you will need two spare batteries if you travel a lot.

• [1] http://ozlabs.org/~rusty/index.cgi/2007/10/07#2007-10-07


• [2] http://etbe.coker.com.au/2007/01/04/monitors-for-developers/

Share This

Joey writes about the difficulty of dealing with the need of pristine tarballs and version control systems. He concludes that it's best to save the tarballs in the repository and has written a tool to store binary differences to save space to achieve his goal. This being another piece of software by Joey, it's probably worth checking out. But I wonder why we actually bother about pristine tarballs. Sure, our Debian archive scripts require a tarball, but why does that have to be the pristine upstream tarball? As Debian maintainer, I use git to keep track of the software that I package, and if possible, I track the upstream repository directly (Git makes it very easy to track repositories of the other, popular version control systems). When I decide to upload a new upstream version to the Debian archive, I tag the upstream tree, sign the tag, and generate a tarball from it:

git tag -s -m'preparing release of upstream 1.2.3' upstream/1.2.3
git archive --prefix=foo-1.2.3/ upstream/1.2.3   gzip -9 \
  > ../foo_1.2.3.orig.tar.gz

Then I merge the tag into my Debian branch and prepare a Debian release, which I also tag (debian/1.2.3-1) just before uploading the package to our archive. This is very flexible and allows me to release packages based on software snapshots, as I just did with mdadm. Now, a filename of mdadm_2.6.3+200709292116+4450e59.orig.tar.gz pretty clearly states that this is not mdadm-2.6.3.tar.gz, so people would be aware that this tarball is not "upstream-official". But what about official releases, like mdadm-2.6.3.tar.gz? When preparing the Debian package, it's only slightly more effort for me to download the official tarball and store it as ../mdadm_2.6.3.orig.tar.gz, but why do I need to keep it around after I uploaded Debian package 2.6.3-1? I know I need the orig tarball to prepare the diff.gz for 2.6.3-2, but for that purpose, I can really just git-archive the appropriate tag. The result won't produce the same hash (because the tar format stores timestamps and is thus not really appropriate for the cause), but it provides the same contents. In fact, what I (or someone else) should finally implement is a tool to create the diff.gz directly from the version control data or even better, we should finally get rid of source packages as I've previously suggested, long before Ubuntu came around to propose and blueprint the concept. I am aware of some people's preference to be able to "independently verify Debian's orig tarball using e.g. a detached GPG signature provided by upstream," but I dare to question the point of this verification. Those who care enough to verify that the Debian-provided orig tarball is pristine are hopefully building their own binary packages instead of trusting mine (another beef I've raised in the past), and then they might just as well download the source directly from upstream. I'd say most people (rightfully) don't do this and just use the Debian-provided content, which can be assumed to be pristine if you trust me and Debian: my GPG signature authenticates uploads to the Debian servers, and APT provides authenticity checks downstream up until the installation on an end-user system. and if you trust the version control system, and this actually raises an interesting point for me: how feasible would a man-in-the-middle attack between Neil's (the mdadm upstream) repository and mine be, given that I use the plain-text git transport protocol to fetch new commits. If you have any comments on that, please let me know. I'll leave this issue for another discussion or post. NP: The Beatles: Abbey Road Update: Matej Cepl pointed me to this thread on the fedora-devel mailing list, where they discuss switching from CVS to Git and talk a lot about their needs and how the end result is much the same as what I am proposing. Thanks!

From the ZFS FAQ:

Can devices be removed from a ZFS pool? You can remove a device from a mirrored ZFS configuration by using the zpool detach command. Removal of a top-level vdev, such as an entire RAID-Z group or a disk in an unmirrored configuration, is not currently supported. This feature is planned for a future release.

buzz, buzz, buzz…

I'm pretty sure someone in one of my syndicators was looking for software to send emails in the future from anywhere, but I'm damned if I can find the message. At the time, I was using a homebrew system involving shell scripts and mailx, so I wasn't much help. I'll upload a proper release soon, but I'm happy to upload a second hack of maildelayline here. It requires recent bash, sed, at and a sendmail-like executable. I've deliberately kept it simple to try to make it easier to debug and I dislike needing to install too much for such a simple task. Make a temporary directory, edit the top of the script, then install it on a pipe mail alias (I'd keep the address secret and improbable, like RUbAHy8961: " /path/to/maildelayline" and then save the address in my address book) and send it an email. At the start of the email body, write a pseudo-header with the time, like

"X-Send-At: now + 7 hours"

and follow it with the To:, Cc:, Bcc: and any other headers. Some headers (like from, message-id and threading ones) are kept from the original message, but the Date and Received headers are among those removed. Don't put the X-Send-At in the email header unless you know what you're doing - usually, put it in the body. The X-Send-At field can be any time that your at program understands. If you confuse it, or forget to give it a time, then you should get a bounce message that says something like "garbled time" among other errors. maildelayline is similar to Leave A Message or FutureMe but it sends mail to anyone, autonomously on your own server, from emails instead of a web form. I think it will resend MIME attachments as long as the pseudo-header is in a plain text/plain part and not quoted-printable or base64 or something. It is named after delay lines, which I believe are mostly used in audio effects to repeat a sound after a time delay. Please leave a comment if you find this useful, or if you have an improvement.

Sound Juicer "The Best Blue Is Through The Trees" 2.20.0 is out. Tarballs are available on burtonini.com, or from the GNOME FTP servers.

• Remove a work around now that GStreamer is fixed, and update requirement (#472650, Jaap Haitsma)

Also thanks to the tireless translation team: Djihed Afifi (ar), Ihar Hrachyshka (be), Alexander Shopov (bg), Runa Bhattacharjee (bn_IN), Jordi Mallach (ca), Ask Hjorth Larsen (da), Hendrik Richter (de), Tshewang Norbu (dz), Kostas Papadimas (el), David Lodge (en_GB), Jorge Gonz lez (es), Ivar Smolin (et), I aki Larra aga Murgoitio (eu), Ilkka Tuohela (fi), Christophe Benz (fr), Ignacio Casal Quinteiro (gl), Ankit Patel (gu), Eyal Mamo (he), Gabor Kelemen (hu), Francesco Marletta (it), Takeshi AIHANA (ja), Young-Ho Cha (ko), Erdal Ronahi (ku), ygimantas Beru ka (lt), Raivis Dejus (lv), Arangel Angov (mk), Kjartan Maraas (nb), Wouter Bolsterlee (nl), Tomasz Dominikowski (pl), Og Maciel (pt_BR), Duarte Loreto (pt), Mugurel Tudor (ro), Nickolay V. Shmyrev (ru), Danishka Navin (si), Matic gur (sl), Elian Myftiu (sq), (sr), Daniel Nylander (sv), Dr.T.Vasudevan (ta), Theppitak Karoonboonyanan (th), Baris Cicek (tr), Maxim Dziumanenko (uk), Clytie Siddall (vi), Funda Wang (zh_CN), Chao-Hsiung Liao (zh_HK, zh_TW).