I had been eyeing an inexpensive network-storage device -- for the non-geeks: think of a hard disk with an ethernet port and some controlling software -- for some time. I was aware of some of the hacking efforts around a few of these but somehow nothing really appealed. I was sort-of looking for nslu2-with-a-disk, and preferably not too expensive. Lo and behold, that's what I saw today in my techbargains feed: a Buffalo LinkStation Live which contains a 500gb SATA for $199 after rebates. Some quick googling lead to these wiki pages which looked promising: anything from enhancing the stock Linux setup by enabling a few more services to a custom Linux distro (similar to my wrt54 router running linux) to reportedly some work-in-progress for a native Debian installation. Nice! So off I went, ordered the thingie for local pickup for an additional 5% off and picked it up a little later at the local Circuit City (where visits are seemingly a recurring event these days). The documentation is very brief and insist that you install something on Windows -- just to find that the little box autoconfigures itself just fine. Presumably some network discovery is going to find the assigned dhcp address which is needed for the web interface. A few minutes later, a new (fixed) IP address was assigned, ntp was enabled and that was about that. After dinner, I quickly followed this tutorial to get the box a bit more Unixy without going too far (yet) from the default: start up telnet via a simple Java command-line tool, login, then enable ssh, set it up in /etc/init.d, add some extra binaries. All very quick and simple [ with the caveat that the addons.tar didn't want to get there via the Java tool, so manual scp once 'inside' did the trick ]. NFS, which I like for shuffling files around, appears to be little trickier for this ARM-based LinkStation Live. So at least for now I am content with simple rsync'ing of my backup directories on the few machines here. Much better than the current setup with mutual backups between workstations and semi-permanently being out of space. All in all a rather pleasant gadget and recommended at the price. The extra $100 in rebates are valid from today to the 12th.

I want to echo what Brice has said so well already. X is full of interesting problems and important things to work on. When you consider the free software that's out there, very little of it comes close to the importance of X in your day to day life. When we've done our job well, you don't even think about it because it just works, but when X fails it's like a minor catastrophe. Working on the kernel is largely considered to be something glorious and grand, but people tend to ignore X even though it plays very much in the same domain as the kernel. So working on X you not only get to play with interesting problems that are similar to the ones that the kernel folks get to play with, but you also get to work on software that's critically important and makes a significant difference in people's day to day lives.

Furthermore, X.org is a great organization to work in. The number of core contributers is very few (maybe 20 all told) so everyone knows who everyone else is and people are willing to help others out by answering questions about how things work. Reflecting this, the XSF has turned in to a fantastic team over the past few years, and it's one I couldn't be prouder to be a part of. Julien and Brice are doing insanely fantastic work, and we're constantly pushing to do a better job at just about everything, from collaborating with Ubuntu to using the best cutting edge tools out there for our work.

Despite all this though, as Brice said, we're overwhelmed by everything. We're still carrying hundreds of bugs in the Debian BTS alone, most of which we'll never have a chance to look at again. As mentioned elsewhere on planet upstream is totally swamped as well. There's just too many critical projects that need doing (new drivers, fixed drivers, adding capabilities to the server, and that's ignoring day to day needs like documentation and patch review) and too few people to do them effectively. We have a wonderful set of things to do and a great group of people to mentor motivated people to do them, and yet we're still lacking contributers to this critically important set of software.

So as Brice said, if you want to get involved in something that's essentially important to Linux and Free Software, if you want to get involved in something with great problems and really fantastic opportunities to make a difference, you can't do much better than getting involved in X. Brice's method of getting involved in processing bug reports is a perfect way to start. Another is to write much needed documentation. Another is to simply help the XSF maintain some small piece of the Xorg stack, be it a driver for some hardware that you own (the XSF only really runs two or three video drivers collectively so we badly need driver maintainers), or just start poking at something you're curious but mystified about like how mesa or the X server work. There's no magic to any of it, and while it can get complicated I promise that you will become a better coder or maintainer by working on something as challenging as X. You don't have to understand it all from the start, just picking a small thing to work on will be greatly appreciated. So drop us a note on the debian-x list or drop in to #debian-x on oftc and we'll help you get going.

I received a number of comments on my previous blog post, with people stating that my example code wasn't right, either. For reference, here's my code snippet again:

#include <stdio.h>
void* f();
int main()  
	long* a;
	a=(long*)f();
	printf("%Ld\n", *a);
	return 0;
 
void* f()  
	static long a=84;
	printf("%Ld\n", a);
	return &a;
 

First, let me make one thing perfectly clear: the "a" variable is not on the stack. It is a locally-scoped variable, but it is also declared as static, meaning that it's valid for the entire run of the process; if it were on the stack, then this variable would vanish when the function would end. Yes, it's wrong to return a pointer to a variable on the stack, since there's no guarantee that the stack value will still be valid; but I'm not doing that. Alternative implementations could've used a global variable, or could've malloc()'ed a pointer; but the compiled assembly code for my C code will only differ from a version with a global variable in the used label for the variable's location, and in the lack of an extra function call from a version that uses malloc(). Second, no, using A0 as a return value for pointers is not a bug in the compiler, and has not been since 1990, when this interesting book entitled "System V ABI Motorola 68000 Processor Family Supplement" was written. Allow me to quote a snippet from page 3-14 of that book:

%a0	Pointer return values appear in %a0. When calling a function that returns a structure or union, the caller allocates space for the return value and sets %a0 to its address. A function that returns a structure or union value places the same address in %a0 before it returns.

To this day, ELF-conforming implementations do it this way. Really. Go look it up, if you wish; the ISBN is 0-13-877663-6.

Chris, buildd hosts keep those time and space usage statistics in a database, which can be queried...

Script started on Sat Jan  5 19:54:15 2008
wouter@country:~$ ssh kiivi.cyber.ee
Linux kiivi 2.6.18-4-mac #1 Fri Mar 30 23:05:11 CEST 2007 m68k
No mail.
Last login: Sat Jan  5 20:41:16 2008 from d51532c45.access.telenet.be
wouter@kiivi:~$ avg-pkg-build-time -s -t   head -n 20
boost:			2015312k (2015312k lastest)
iceweasel:		1846628k (1846628k lastest)
iceape:			1674172k (1674416k lastest)
icedove:		1624848k (1624848k lastest)
xulrunner:		1611880k (1611880k lastest)
mesa:			1230432k (1230432k lastest)
koffice:		1208062k (2115156k lastest)
gcc-snapshot:		1191866k (1207164k lastest)
gcc-4.1:		1185620k (1185620k lastest)
gtk+2.0:		1148634k (1208684k lastest)
k3d:			1145580k (1145580k lastest)
ardour:			1077800k (1077800k lastest)
lyx:			924876k (924876k lastest)
gcc-4.2:		909488k (909488k lastest)
gcc-3.4:		892015k (571084k lastest)
glibc:			891408k (891408k lastest)
linux-2.6:		725961k (744700k lastest)
kdebase:		720618k (1106804k lastest)
kdelibs:		716376k (1150372k lastest)
kdepim:			711858k (784356k lastest)
wouter@kiivi:~$ avg-pkg-build-time -t   head -n 20
gcc-snapshot:		220:52:14 (2 entries, sigma 27:38:21)
axiom:			216:27:57 (1 entry, sigma 00:00:00)
gcc-4.1:		193:47:30 (1 entry, sigma 00:00:00)
k3d:			177:22:10 (1 entry, sigma 00:00:00)
gcc-3.4:		171:40:28 (4 entries, sigma 25:10:55)
boost:			141:13:38 (1 entry, sigma 00:00:00)
koffice:		140:40:23 (2 entries, sigma 53:48:58)
ace:			130:04:14 (1 entry, sigma 00:00:00)
linux-2.6:		129:36:19 (3 entries, sigma 10:12:24)
qt-x11-free:		105:00:32 (1 entry, sigma 00:00:00)
xulrunner:		97:03:38 (1 entry, sigma 00:00:00)
iceape:			96:23:13 (2 entries, sigma 03:12:58)
kdepim:			96:06:54 (2 entries, sigma 19:09:39)
icedove:		94:35:48 (1 entry, sigma 00:00:00)
iceweasel:		93:56:26 (1 entry, sigma 00:00:00)
gcc-4.2:		91:08:32 (1 entry, sigma 00:00:00)
glibc:			91:01:58 (1 entry, sigma 00:00:00)
kdevelop3:		84:27:21 (2 entries, sigma 02:47:35)
kdeedu:			82:09:55 (3 entries, sigma 10:59:18)
vtk:			81:44:20 (3 entries, sigma 00:46:32)
wouter@kiivi:~$ logout
Connection to kiivi.cyber.ee closed.
wouter@country:~$ ssh arrakis
Linux arrakis 2.6.12-1-amiga #1 Sat Aug 13 22:59:59 CEST 2005 m68k
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Please read "links2 http://nefud/info.txt"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
No mail.
Last login: Sat Jan  5 18:48:58 2008 from hahn.os.localnet on pts/5
wouter@arrakis:~$ avg-pkg-build-time -s -t   head -n 20
wxwidgets2.6:		1648020k (1648020k lastest)
qt4-x11:		1601086k (1753532k lastest)
gcj-4.1:		1135772k (1135772k lastest)
k3d:			1094345k (1159204k lastest)
kdebase:		1087375k (1087368k lastest)
python-kde3:		1018144k (1018144k lastest)
ghc6:			991524k (991524k lastest)
gcc-3.3:		988690k (990244k lastest)
glibc:			814536k (855720k lastest)
lyx:			804672k (804672k lastest)
bouml:			729888k (729888k lastest)
kdelibs:		685565k (1142304k lastest)
freedroidrpg:		673988k (673988k lastest)
mozilla:		663066k (636896k lastest)
ardour:			647580k (691128k lastest)
mozilla-thunderbird:	591384k (591384k lastest)
omniorb4:		576288k (576288k lastest)
subversion:		568764k (624768k lastest)
xorg-server:		527520k (527520k lastest)
mozilla-firebird:	498896k (502692k lastest)
wouter@arrakis:~$ avg-pkg-build-time -t   head -n 20
enlightenment:		198842:37:14 (3 entries, sigma 344403:17:35)
axiom:			164:43:54 (1 entry, sigma 00:00:00)
ghc6:			159:20:15 (1 entry, sigma 00:00:00)
k3d:			145:18:35 (3 entries, sigma 57:08:46)
python-kde3:		126:39:34 (1 entry, sigma 00:00:00)
qt4-x11:		121:26:11 (2 entries, sigma 03:07:35)
gcj-4.1:		116:05:34 (1 entry, sigma 00:00:00)
gcc-3.3:		108:45:19 (3 entries, sigma 02:01:50)
qt-x11-free:		99:12:17 (4 entries, sigma 14:12:32)
wxwindows2.4:		90:26:51 (1 entry, sigma 00:00:00)
openscenegraph:		74:37:21 (1 entry, sigma 00:00:00)
ardour:			63:44:55 (2 entries, sigma 07:23:05)
bouml:			62:01:16 (1 entry, sigma 00:00:00)
gcc-snapshot:		61:10:07 (1 entry, sigma 00:00:00)
mozilla-thunderbird:	60:47:29 (1 entry, sigma 00:00:00)
mico:			60:15:15 (1 entry, sigma 00:00:00)
kdebase:		58:41:04 (5 entries, sigma 08:00:17)
glibc:			58:04:26 (6 entries, sigma 40:01:12)
mozilla-firebird:	52:47:13 (2 entries, sigma 11:31:07)
zeroc-ice:		52:34:03 (2 entries, sigma 28:12:19)
wouter@arrakis:~$ logout
Connection to fremen-os.dyndns.org closed.
wouter@country:~$ exit
Script done on Sat Jan  5 22:28:46 2008

"avg-pkg-build-time" is a script that queries the statistics database. It's quite CPU-intensive, so I don't like to run it too often (certainly not from cron); but, hey, here you go. In case you didn't know, "arrakis" and "kiivi" are both official m68k/unstable buildd hosts, and especially arrakis has been for quite a long time, although it's been reinstalled once or twice since. Oh, and yes, it does appear as though some bits of the data are invalid, but most of the idea should be alright. (oh, and before you ask, no, it didn't take two and a half hours to run those commands -- I just needed to go while the last one was running, and had only come back several hours later)

The general idea on the 25th was to relax, watch some , and generally do not care too much about anything. Right, only after 13 episodes of ("Crest of the Stars") hunger set in, and it being late at night on a holiday, we scrounged the kitchen for things that might fit together. Idea 1: Let's make Spaghetti Alio e Olio. — Well, no Spaghetti in the house. So replace them with Farfalle. Idea 2: We could also add some onion. — Splendid. Idea 3: We could add sugar to the onion. — Yay, caramel. Unforeseen Circumstance 1: Whoops, not that much sugar. Idea 4: It being Christmas, and the food being awfully sweet now anyway, let's add apples and cinnamon. So, I present the ultimate Christmas dinner, featuring:

• as much garlic as you like
• an onion
• sugar
• an apple
• noodles

And it tasted quite good, actually. I shall remember to buy more food before the holidays next year though.

My next step in reducing the spam traffic in my life is removing my @gnu.org email address. This is one that has been good for me for 13 years but has largely been useful only as a spam trap. In fairness to the FSF, their spam filters have gotten really quite good in the last 5 years or so, but there's zero signal on that account so I may as well squish it.

I've set my alias in /com/mailer/aliases to :fail: with a 551, but the multi-level relay on monty-python seem to not respect that. I've filed a ticket, so that anyone trying that will get a proper message to say where to follow-on to.

The efforts seems to be making a difference. At the beginning of December, I had just over 11 thousand messages in my last-30-days spam folder. Right now it's 8925.

Sorted by build-time

1. openoffice.org - 7h02
2. gcc-4.2 - 5h58
3. gcc-snapshot - 4h17
4. gcc-4.1 - 3h30
5. ghc-cvs - 3h16
6. linux-2.6 - 2h34
7. gpc-4.1 - 2h25
8. gclcvs - 2h21
9. glibc - 2h15
10. python2.5 - 2h09
11. axiom - 2h07
12. gnat-4.1 - 2h06
13. gcj-4.1 - 1h52
14. gromacs - 1h51
15. mysql-dfsg-5.0 - 1h46
16. acl2 - 1h44
17. gcj-4.2 - 1h34
18. ghc6 - 1h33
19. python-kde3 - 1h32
20. aegis - 1h25

Sorted by build-space

1. openoffice.org - 7217MB
2. nel - 2773MB
3. gcc-snapshot - 2499MB
4. koffice - 2247MB
5. wxwidgets2.6 - 2151MB
6. linux-2.6 - 2150MB
7. webkit - 2059MB
8. glibc - 2028MB
9. xulrunner - 2023MB
10. iceape - 2007MB
11. boost - 1983MB
12. mesa - 1943MB
13. gcc-4.2 - 1943MB
14. qt4-x11 - 1915MB
15. python-kde3 - 1847MB
16. gcc-4.1 - 1828MB
17. gcj-4.1 - 1783MB
18. eclipse - 1763MB
19. iceweasel - 1727MB
20. gcj-4.2 - 1722MB

No real surprises here. The data was collected from the Buildd logs using the latest “maybe-successful” i386 or amd64 build. Factoring into that the varied nature of the buildd machine specifications make this is data somewhat unreliable… It would be nice if someone could obtain (and post) results from a more, umm, controlled environment.

A few weeks back, someone asked on debian-68k about an unexpected result of some piece of software when it was compiled on ARAnyM, an m68k emulator that can run Linux. His initial idea was that this could perhaps be a bug in ARAnyM, since it only occurred to him inside ARAnyM, and not on any of the other architectures he tried. In fact, it was a bug in his code. Let's have a look at what's going on here. The example program which Sergei provided and which exhibits the problematic behaviour is spread over two files. The first file contains this:

#include <stdio.h>
void *f();
main()  
	void *a;
	a = f();
	printf("%d\n", (long)a);
 

while the second contains this:

#include <stdio.h>
long f();
long f()  
	long a = 84;
	printf("%d\n", a);
	return a;
 

Many people familiar with the C programming language will intuitively expect that this program, when run, will output the following:

84
84

When in fact, when compiled and ran on an m68k machine, the output is as follows:

84
-1072577264

What's going on here? In order to understand that, we have to take a look at the assembly code. This is done by use of "objdump -d", and gives (amongst others) the following output:

800004c4 <main>:
800004c4:	4e56 0000	linkw %fp,#0
800004c8:	61ff 0000 001a	bsrl 800004e4 <f>
800004ce:	2f08		movel %a0,%sp@-
800004d0:	4879 8000 05aa	pea 800005aa <_IO_stdin_used+0x4>
800004d6:	61ff ffff feec	bsrl 800003c4 <printf@plt>
800004dc:	508f		addql #8,%sp
800004de:	4e5e		unlk %fp
800004e0:	4e75		rts
800004e2:	4e75		rts
800004e4 <f>:
800004e4:	4e56 0000	linkw %fp,#0
800004e8:	4878 0054	pea 54 <_init-0x800002f0>
800004ec:	4879 8000 05aa	pea 800005aa <_IO_stdin_used+0x4>
800004f2:	61ff ffff fed0	bsrl 800003c4 <printf@plt>
800004f8:	7054		moveq #84,%d0
800004fa:	4e5e		unlk %fp
800004fc:	4e75		rts
800004fe:	4e75		rts

So what's happening here?

• main acquires some stack space, and then (0x800004c8) immediately jumps to a subroutine 26 bytes ahead; our function f.
• f also acquires stack space (0x800004e4), then pushes two addresses on the stack (...4e8 and 4ec), and jumps to the printf function. The first of these two addresses is the constant 84 that is hardcoded in the binary; the second address is our format string.
• f then moves the constant value 84 to the D0 register (...4f8), frees its stack space (4fa), and returns (4fc; the second rts can be ignored, that's a harmless quirk in the compiler).
• main now copies whatever is in the A0 register to the stack (4ce), pushes the exact same format string to the stack (4d0), and again jumps to the printf function (4d6; the difference in hexadecimal values is due to the fact that the bsr opcode uses processor-indirect addressing).
• Finally, main clears up what it's been doing, and exits

The difference should be clear: the f function stores its result value in the D0 register, but main goes looking for it in A0. The reason for this discrepancy very simple. The m68k processors has three sets of registers: one set is for integer values (D0 through D7), one set is for address values (A0 through A7), and one is for floating-point values (FP0 through FP7). The m68k ABI specifies that integer return values should be stored in integer registers, that address return values should be stored in address registers, and that floating-point return values should be stored in floating-point registers. This makes sense, since register-indirect addressing modes require the address to be stored in an address register; and calculating values requires the value to be stored in either a floating-point or integer register. When main, then, tried to look for a return value in A0, it found something, but obviously not what it should have found... This issue would have been a non-issue had this function been written the way it should have, like so:

#include <stdio.h>
void* f();
int main()  
	long* a;
	a=(long*)f();
	printf("%Ld\n", *a);
	return 0;
 
void* f()  
	static long a=84;
	printf("%Ld\n", a);
	return &a;
 

To close up: there's a reason why compilers emit warnings if you do something strange or "clever". In this particular case, the warning was suppressed, since both files contained a declaration of an f() function, even if the declaration was different. This is a horrible hack that tries to work around those warnings, rendering them utterly useless. Please, pretty please, with sugar on top: don't do something like that. If you write C code, please compile it with -Wall -Werror, and make sure it compiles that way. If you want to access a function in a different file, create a common header file that both will #include, so that the compiler can notice differences between declaration and definition of a function; and don't expect that something will work because you fixed it for a common and well-known case, because there will often still be other places where your bug will still trigger. In this case, by declaring the function as one returning a long value, they made sure it could not break on 64-bit architectures. However, as shown above, that doesn't make the bug go away...) Writing clean and portable code is way more fun. Trust me.

Linked In always had the atmosphere of a more serious and professional social networking site, unlike MySpace and Facebook where people seem to collect friends like stickers when they were younger. Then I discovered TopLinked, a site dedicated to letting people grow their network massively to people they've never met. The top member has 37 thousand connections. I just don't understand this, what is the point of having so many connections when there is no value in the connections themselves? NP: Bricolage, Amon Tobin

I'm exhausted, but very relieved. The important news: the operation was a success, the surgeon, Dr D. Craig Miller, bless his cowboy boots, managed to spare the aortic valve (he'd given us a 99% probability of being able to do so beforehand), so we don't have to be concerned about pig/cow valves for now, and mechanical valves later. He was reluctant to say that her own valve will last her the rest of her life, but it should last a "long time, long enough to have a family". When I left the hospital, Sarah was in intensive care, extubated, conscious, and talking to me, but very very tired. She's got a daunting array of things coming out of her at the moment (a lot of which I couldn't see because she had blankets over her). It's been a long day, so I decided to head home, rather than potentially keeping her awake for the next two ICU visiting times tonight. Here's the time line that I recorded: 05:15 - Arrive at the hospital to go through the admissions process 06:35 - I leave Sarah shortly before they're going to wheel her off to the operating room (I'll get into my fun at the end). 09:30 - I ask for an update, and they tell me she's going on bypass 10:30 - It looks like they can save the aortic valve 12:25 - Still doing the valve stuff, making a new sinus of Valsalva 14:15 - I ask for another update, everything is going okay, someone will come out to talk to me 14:55 - Still going, at least another hour until they start to close up 15:15 - Closing up, surgeon will be out to talk to me 16:10 - Surgeon comes out to talk to me about how the operation went 16:50 - I got to see Sarah briefly in the ICU, she was still intubated, but conscious and seemed to respond to my voice ~18:15 - Extubated. Spoke with her briefly. She was very tired, and I decided to let her rest for the remaining visiting times and go home and get some rest myself So what the surgeon had to say for himself was very interesting indeed. He didn't end up doing exactly what he'd said previously, based on how things looked once he actually got in there. He said that the sinuses of Valsalva were paper thin. So thin, you could see the blood flowing through them. They were about 0.5mm thick, rather than the normal 1.5mm thick. Sounds a bit like a disaster waiting to happen. Apparently it's not possible to detect this with imaging, you have to get in there and see it for yourself. He didn't end up removing as much of the aorta as previously planned (specifically the part around the transcending arch) as whilst the diameter was a bit bigger, the tissue apparently looked healthy. He said the part of the aorta he did remove (preceding where the braciocephalic, left common carotid and left subclavian arteries branch off) was "cheesy" in consistency, so it's not like she just happened to have an enlarged aorta, no aneurysm, and way too thin sinuses of Valsalva. There was definitely some unhealthiness to the aortic tissue, just not as bad as thought from the imaging. So it's all good. I think she'll spend a day or two more in the ICU before getting moved to a normal ward, but it's too soon to say what's going on with all of that. There's some elevated risk of stroke due to blood clots for the next 24-48 hours, but she's on blood thinners, so it shouldn't be a problem. The day was pretty crazy for me. I'm not sure if it was stress, or the stomach bug that's been doing the rounds of Central Park Apartments, but I went to bed last night at 11pm, was up again at 1am, with shall we say, a gastrointestinal upset, and up again at 3am, throwing up. We got up at 4:15am to get to the hospital. I felt really nauseous all morning, and was an absolute wreck while I was in with Sarah when they were putting a peripheral line in her before they wheeled her off to the OR, so I bailed about 5 minutes early and just made it to the bathroom in time to throw up again. I haven't eaten all day, just tried to keep the fluids up. I'm debating having some plain boiled rice for dinner, or waiting until tomorrow to have something easy on the stomach. I definitely felt a lot better as the day progressed, but my stomach is still pretty tender. Sarah and I ate pretty much the same stuff yesterday, so I don't think it can be food poisoning. She tends to have a very sensitive stomach at the best of times, so I'd expect her to be showing problems before me if it was food poisoning. But enough about me, I'll live. I hope it's not a stomach bug, as I don't want Sarah to come down with it as well, now that she's on the road to recovery from this surgery. Many thanks to Shona, Briana, Laura, and Christina for spending time with me throughout the day to keep me occupied. ICU visiting hours are every even hour between 10am and 10pm, for thirty minutes at a time, so I plan on spending all day at the hospital tomorrow. Hopefully I can find a good spot to get some EVDO coverage, so I can update things as the day goes along, rather than at the end of the day when I get home, like I had to today. Hopefully I'll sleep better tonight than I did last night.

According to this test, sympycore is from 2.5x to 5x faster than Maxima. This is an absolutely fantastic result and also a perfect certificate for Python in scientific computing. Considering that we compare pure Python to LISP.

Ok, this made us excited, so we dugg deeper and ran more benchmarks. But first, let me say a few general remarks. I want a fast CAS (Computer Algebra System) in Python. General CAS, that people use, that is useful, that is easily extensible (!), that is not missing anything, that is comparable to Mathematica and Maple -- and most importantly -- I want it now and I don't care about 30 years horizons (I won't be able to do any serious programming in 30 years anyway). All right. How to do that? Well, many people tried... And failed. The only opensource CAS system, that has any chance of becoming the opensource CAS, in my own opinion, is Sage. You can read more about my impressions form Sage here. I am actually only interested in mathematical physics, so basically Sage.calculus. Currently Sage uses Maxima, because Maxima is old, proven, working system and it's reasonably fast and quite reliable, but written in LISP. Some people like LISP. I don't and I find it extremely difficult to extend Maxima. Also even though Maxima is in LISP, it uses it's own language for interacting with the user (well, that's not the way). I like python, so I want to use Python. Sage has written Python wrappers to Maxima, so Sage can do almost everything that Maxima can, plus many other things. Now. But the Sage.calculus has issues.

First, I don't know how to extend the wrappers with some new things, see my post in the sage-devel for details, it's almost 2 months old with no reaction, which shows that it's a difficult issue (or nonsense:)).

And second, it's slow. For some examples that Sage users have found out, even SymPy, as it is now, is 7x faster than Sage and sympycore 23x faster and with the recent speed improvements 40x faster than Sage.

So let's improve Sage.calculus. How? Well, no one knows for sure, but
I believe in my original idea of pure Python CAS (SymPy), possibly with some parts rewritten in C. Fortunately, quite a lot of us believe that this is the way.

What is this sympycore thing? In sympy, we wanted to have something now, instead of tomorrow, so we were adding a lot of features, not looking too much on speed. But then Pearu Peterson came and said, guys, we need speed too. So he rewrote the core (resulting in 10x to 100x speedup) and we moved to the new core. But first, the speed isn't sufficient, and second it destabilized SymPy a lot (there are still some problems with caching and assumptions half a year later). So with the next package of speed improvements, we decided to either port them to the current sympy, or wait until the new core stabilizes enough. So the new new core is called sympycore now, currently it only has the very basic arithmetics (and derivatives and simple integrals), but it's very fast. It's mainly done by Pearu. But for example the latest speed improvement using sexpressions was invented by Fredrik Johansson, another SymPy developer and the author of mpmath.

OK, let's go back to the benchmarks. First thing we realized is that Pearu was using CLISP 2.41 (2006-10-13) and compiled Maxima by hand in the above timings, but when I tried Maxima in Debian (which is compiled with GNU Common Lisp (GCL) GCL 2.6.8), I got different results, Maxima did beat sympycore.

SymPyCore:

In [5]: %time e=((x+y+z)**100).expand()
CPU times: user 0.57 s, sys: 0.00 s, total: 0.57 s
Wall time: 0.57

In [6]: %time e=((x+y+z)**20 * (y+x)**19).expand()
CPU times: user 0.25 s, sys: 0.00 s, total: 0.25 s
Wall time: 0.25

Maxima:

(%i7) t0:elapsed_real_time ()$ expand ((x+y+z)^100)$ elapsed_real_time ()-t0;
(%o9)                                0.41
(%i16) t0:elapsed_real_time ()$ expand ((x + y+z)^20*(x+z)^19)$ elapsed_real_time ()-t0;
(%o18)                         0.080000000000005

So when expanding, Maxima is comparable to sympycore (0.41 vs 0.57), but for general arithmetics, Maxima is 3.5x faster. We also compared GiNaC (resp. swiginac):

>>> %time e=((x+y+z)**20 * (y+x)**19).expand()
CPU times: user 0.03 s, sys: 0.00 s, total: 0.03 s
Wall time: 0.03

Then we compared just the (x+y+z)**200:

sympycore:
>>> %time e=((x+y+z)**200).expand()
CPU times: user 1.80 s, sys: 0.06 s, total: 1.86 s
Wall time: 1.92
swiginac:
>>> %time e=((x+y+z)**200).expand()
CPU times: user 0.52 s, sys: 0.02 s, total: 0.53 s
maxima:
(%i41) t0:elapsed_real_time ()$ expand ((x + y+z)^200)$ elapsed_real_time ()-t0;
(%o43)                         2.220000000000027

Where GiNaC still wins, but sympycore beats Maxima, but the timings really depend on the algorithm used, sympycore uses Millers algorithm which is the most efficient.

So then we tried a fair comparison: compare expanding x * y where x and y are expanded powers (to make more terms):

 sympycore:
 >>> from sympy import *
 >>> x,y,z=map(Symbol,'xyz')
 >>> xx=((x+y+z)**20).expand()
 >>> yy=((x+y+z)**21).expand()
 >>> %time e=(xx*yy).expand()
 CPU times: user 2.21 s, sys: 0.10 s, total: 2.32 s
 Wall time: 2.31
 swiginac:
 >>> xx=((x+y+z)**20).expand()
 >>> yy=((x+y+z)**21).expand()
 >>> %time e=(xx*yy).expand()
 CPU times: user 0.30 s, sys: 0.00 s, total: 0.30 s
 Wall time: 0.30
 maxima:
 (%i44) xx:expand((x+y+z)^20)$
 (%i45) yy:expand((x+y+z)^21)$
 (%i46) t0:elapsed_real_time ()$ expand (xx*yy)$ elapsed_real_time ()-t0;
 (%o48)                         0.57999999999993

So, sympycore is 7x slower than swiginac and 3x slower than maxima. We are still using pure Python, so that's very promising.

When using sexpr functions directly then 3*(a*x+..) is 4-5x faster than Maxima in Debian/Ubuntu. So, the headline of this post is justified. :)

Conclusion

Let's build the car. Sage has the most features and it is the most complete car. It has issues, some wheels need to be improved (Sage.calculus). Let's change them then. Maybe SymPy could be the new wheel, maybe not, we'll see. SymPy is quite a reasonable car for calculus (it has plotting, it has exports to latex, nice, simple but powerfull command line with ipython and all those bells and whistles and it can also be used as a regular python library). But it also has issues, one wheel should be improved. That's the sympycore project.

All those smaller and smaller wheels show, that this is indeed the way to go, but very important thing is to put them back in the car. I.e. sympycore back to sympy and sympy back to Sage and integrate them well. While also leaving them as separate modules, so that users, that only need one particular wheel, can use them.

I am very saddened by the state of the IT industry. It seems like nearly everyone who works in or around IT is, to put it bluntly, a cowboy. There appears to be very little desire for actually helping people; instead all I see is a very large number of people circling around looking for the next person they can make a quick buck off. In short, most of the IT industry is composed of people who are little better than shysters and conmen. As a case in point, I recently had contact with someone who is trying to get an online store running. They've been trying, unsuccessfully so far, for several months. They've been screwed over by basically every IT-related person they've had contact with so far. Now, after all of their suffering, they've finally gotten in contact with me. Most of their money is gone, and they're deeply behind schedule. So I'm left trying to pull a miracle job in little time and with little money. Yes, I could shoo them away, because they're asking for more than they can afford, but there's a little spark of helpfulness left in me, and I'll try to get them going as best they can with what they've got. If they'd found someone competent right at the beginning, they'd have been fine. They started with sufficient cash and time to produce a good end result. But chancing upon someone decent doesn't have good odds in IT; it's a bit like winning the lottery. This story isn't unique in my personal experience, by any stretch. There seems to be a near-endless procession of non-technical people who come out of their encounters with the IT industry battered and bruised, whether it be on the web, in their business infrastructure, or their personal computing needs. While it's easy to blame the customer for not managing themselves and their computing properly, I don't think it's fair to lay the blame on them -- to me, that's like blaming the patient for dying on the operating table, when the responsibility must lie, prima facie, with the surgeon. It's the surgeon, after all, who is the (presumably) trained and experienced professional in the relationship, and it is his responsibility, primarily, to ensure that everything goes smoothly[1]. Depressingly, I don't see much of a way around this problem. The few IT people who did care could try and group together, build some sort of a rating or "membership implies quality" system, educate the consumer, and try and drive the cowboys out, but guilds and trade associations are frowned upon these days, smacking (as they do) of restraint of trade and collusion. Government regulation never seems to produce anywhere near the desired result, while hoping that the consumer will either rise up and demand quality, or will gain sufficient knowledge to be able to pick a cowboy is a complete non-starter, for the same reason as teaching everyone basic surgical techniques isn't going to improve the quality of hospitals.

---

There's an ad that used to be on Australian TV for a certain bank, where a guy is at a party and he's asked by another of the partygoers what he does for a living. As he replies, "I work in banking", the entire party goes silent and stares at him in horror. It's only when he says, "It's OK, I'm with $BANK_BEING_ADVERTISED" that everyone looks happy again and continues on with their party. Although I've never had that particular experience, I'm almost getting to the point where I'm not happy telling people what I do for a living. This is because in any non-trivial collection of people, there's almost certainly going to be at least one person who has had such a poor experience with the IT industry that they're either going to take it out on me, ask me interminable questions about computers, or will just hate me on principle. It's unfortunate that brothels don't have pianos any more; if I were to say "I play piano in a whorehouse" I'm pretty sure the response would be "lawyer or sysadmin?". As Edsger Dijkstra said back in 2001, "The average customer of the computing industry has been served so poorly that he expects his system to crash all the time, and we witness a massive worldwide distribution of bug-ridden software for which we should be deeply ashamed." Are you?

---

1. That isn't to say that the patient (or client) has no responsibility in ensuring a positive outcome -- if you go bungee jumping the day after open heart surgery, or never send your web developer any content, you're very unlikely to have a positive outcome. However, it seems to me as though the only customer consultation done by most IT "professionals" on taking on a project is a Homeresque "I only have two questions. 'How much?' and 'give it to me'".

Yesterday after dinner we spontaneously ended up having a XO play night. I liked the interface and the things it could do, the complete difference with a normal notebook, the creeping into a child’s mind a bit. It was funny to notice that being older and experienced, things that probably are intuitive to children, are not/no longer intuitive to me anymore. This fact alone shows how “being used to” and experience are a large part of things one consider intuitive. I notice this also when people are trying to use GNOME when they are used to Windows: they find this difficult, but of course most of them are already “hard wired” to Windows. The reverse holds for me: due to not using Windows often at all, I keep losing the aptitude for it and it seems to be a hard system to use from my perspective. Fortunately for us all, “getting used to” is the specialty of the brain.

Tomorrow, all being well, I should receive a new desktop computer. I ordered it upon Boxing Day so the turnaround is pretty good. Right now I have two desktop machines:

yours.my.flat

This machine is occaisionally used by my partner, but primarily exists to remotely backup my various servers. It runs rsnapshot to do a full rsync of four machines every six hours.

vain.my.flat

This is my day-to-day desktop machine. It is going to be replaced, and this will be rotated into the backup machine. The machine yours will sit in a closet for a couple of months then be donated to somebody local.

As with all the machines I've bought recently the new one is coming from Novatech - (Another company whos website has a suprising hostname.) The interesting parts of the spec are :

• AMD X2 4200 processor
• 2 x 500Gb SATA drives
• 4Gb memory

The biggest difference to me will be the significantly increased disk space which means that I'll be able to remove the external 120Gb USB hard drive I currently house my MP3 collection upon. By retiring vain and moving my current desktop machine to be the "random guest machine" and backup host I'll also have a lot more space for storing backups upon. (Full database backups every six hours; I'm paranoid...) Having 4Gb of memory will also mean that I can run more than 2 or 3 Xen guests at the same time. My plan is to setup at least the following guests:

• sarge.security-build.xen
• etch32.security-build.xen
• etch64.security-build.xen
• etch32.desktop.xen
• etch64.desktop.xen

The desktop machines will run GDM / X11 and will be available for logins via xdmcp - so that I can have a fully seperate working environment which is pristine and easily regenerated. Giving 5 xen instances 256Mb of memory each will leave me with close to 3Gb of disk for the host and that is three times what I hav enow! All in all I'm looking forward to the new machine enormously a bargain at ~ £290.

A few days ago, I asked about how to properly embed spamtraps in web pages.Well, noone could tell me if using display: none is appropriate. I actually do not want Google to index the contents in that div. So as long as they don't punish me for using display: none at all, it's okay. And the page I placed the spamtrap on is a doorway-like page for others anyway; it's not part of an important site.It took the first spammer around 54 hours to send the first spam. Or try to send: all 10 retries with different zombies were rejected by my spam filter. Since then, I've been receiving another round of deliver attempts - around 5-15 per spamtrap address - almost every hour.Of the > 500 spam delivery attempts I've seen since, none made it through my initial spam filters (not to speak of the content filter behind that), but they were rejected at the SMTP level, even before the mail content was sent.I've now disabled some of my spam filters to allow the trap adresses to actually receive mail. After all, I want to use them to train my filter. :-)

My thesis is about data mining, clustering of correlated data in high dimensional vector spaces, to be a bit more precise.In detail, I'm working on methods to improve upon existing clustering algorithms such as 4C (Computing Clusters of Correlation Connected Objects) and ERiC (On Exploring Complex Relationships of Correlation Clusters), where you need to pick some parameters (e.g. k for a k nearest neighbour based approach) appropriately.My approach is twofold. On one hand, I'm improving upon the traditional covariance based correlation (which is quite sensitive to noise), so the parameters become easier to pick, on the other hand I'm working on an approach to automatically fine-tune the parameters to further improve stability.For testing my computations I needed a visualization of this data. I was considering using gnuplot (and in fact I'm using gnuplot a lot), but for some situation I needed animation capabilities, and thats where gnuplot becomes really messy.So I decided to dive into SVG and Javascript. Here's my first SVG project:Visualizing kNN correlation in SVG with Javascript(Internet Exploder is not supported. I don't have Windows, and for all I know it doesn't really support SVG. Use a Gecko-based browser such as Firefox, Opera and Safari (at least on Windows) also seem to work. I didn't get it to work on kHTML/Konqueror/Webkit. I'm just doing this for myself, so I have no need to support other browsers.)It's a 3D dataset, consisting of 300 points. 100 points are noise, 100 points are in a 2D cluster (green) and 100 points are on a 1D cluster embedded into this plane (I'm working on algorithms that support hierarchical clusters, so I needed a dataset with this property!).There are two buttons in the UI, one toggles rotation, the other one toggles the playback of "k". It will cycle k through a range of about 3-200. When offset hits 20 (so k would be 22 or 23), the main correlation vectors - the big blue lines - already point along the 1D cluster. At an offset of around 80 they have already diverged quite a bit from the 1D cluster - at this point, the correlation is seeing the 2D plane quite well already.I could also show you the behaviour for points in the 2D plane (but outside of the 1D cluster) and noise points.We're preparing a paper for SSDBM 2008.[Update: Safari works at least on Windows]

The second in the Vampire Earth series by E. E. Knight, this book has our protagonist moving to the next step in the genetically augmented hunter clans of the human resistance. Well written, though the next stage of the character development would require the people to actually get into some relationships. Apart from that, the series is moving well along, with other team members being brought into the fold.

Well I've been in Devon for the past week, and will remain here for another week. During that time I've averaged about ten minutes of online time a day! So far things are going well, but it will be wierd spending time with another family over Christmas. Still I've been vaguely productive. I've released a new version of chronicle - which has the CGI support for leaving comments upon my entries. (TODO: Reinstate some previous comments for Martin Krafft) One thing I'd not anticipated was the amount of spam I'd start recieving. The peak so far was 40+ comments a day, all with random URLs. I deleted them manually for the first day - but now I've written a shell script or two to classify comments via the spamfilter I use for handling email. I'm not 100% sure I should have done that. I suspect that over time I will find better results if I actually have a distinct "blog spam" and "blog ham" corpus - rather than risk confusion over "email" and "blog" texts to classify. Still I shall wait and see. The only thing that I think I want to do now is add a bit more fine control over the comment process, so the choice is not just between comments being globally on, or globally off. A nice middle-ground where I could say "Comments enabled for entries posted within the past two weeks or so". Anyway thats my quota for today. Now I have to walk the family dog ..