I just added all MySQL example databases to phpMyAdmin demo, so if you want to play with some more or less real data in phpMyAdmin, you now have great opportunity. Especially interesting is sakila database which exposes many new MySQL features. All databases are automatically periodically restored, so you don't have to be afraid to play with it :-).

Cory Doctorow wrote an interesting article about social networking [1]. One of his points is “Imagine how creepy it would be to wander into a co-worker’s cubicle and discover the wall covered with tiny photos of everyone in the office, ranked by “friend” and “foe,” with the top eight friends elevated to a small shrine decorated with Post-It roses and hearts“, another concerns the issue of forced “friends” where colleagues and casual acquaintances demand to be added to a friends list. He speculates that the reason for social networking systems to be a fad is that once too many people you don’t really like force themselves into your friends list then you will feel compelled to join a different service. I believe that the practice of ranking friends is simply a bad idea and wonder whether anyone who has completed high-school has ever used it seriously. If you publicly rank your friends then you will alienate other friends (particularly any who might have ranked you more highly than you ranked them). Everyone who has used social networking systems has discovered the pressure to avoid alienating people that you don’t actually like. It seems obvious that alienating people who you do like is even more of a problem. In my previous post about Better Social Networking [2] I suggested having multiple lists on your social networking server that are published to different people. That would allow segregating the lists as a way of dealing with some demands to be listed. People who are associated with work (colleagues, managers, and in an example Cory used students at a school where a teacher worked) would be on a work list. The work list would point to the work profiles of other people which would match whatever the standards are for the industry in question (which would still allow quite a range, the standards for sys-admins of ISPs differ significantly from those for primary school teachers). I’m sure that someone who worked for an ISP in Amsterdam would understand that a work-based friend request made to someone who teaches primary school in a part of the world that is religiously conservative would understand why that request would be declined (but the same person might add them to a personal friends list). Another way of alleviating such problems is to not require that listing be bi-directional. Current social networking systems involve one party making a friend request to another which is listed as pending in the GUI for both parties. The options are to either leave it in that state (which is an annoyance) or reject it (which may cause offence). With a uni-directional listing one party would add the other and hope for the best. If they aren’t obsessive about such things they may not even notice that the other party didn’t reciprocate. Also it would allow for famous people receiving links from many people to their public profile without any expectation of reciprocation. Of course a distributed social networking system such as I suggest would inherently have uni-directional links as there would be no central repository to force them to be all bi-directional.

• [1] http://informationweek.com/news/showArticle.jhtml?articleID=204203573


• [2] http://etbe.coker.com.au/2007/11/18/better-social-networking/

Share This

Now that I discovered this nice flag I checked a few of my packages. It works nice for the gnome-chemistry-utils application packages. Unfortunately libgcu0 does not benefit from the flag because of libtool bug #347650. However there is an appreciably difference in the dependencies for the plugin and application packages, where the amount of dependencies decreased. For example compare the old

Depends: iceweasel   iceape-browser   xulrunner, libart-2.0-2 (>= 2.3.18),
libatk1.0-0 (>= 1.20.0), libc6 (>= 2.6.1-1), libcairo2 (>= 1.4.0), libgcc1 (>= 1:4.2.1), libgconf2-4 (>= 2.13.5),
libgcu0 (<< 0.9), libgcu0 (>= 0.8), libgl1-mesa-glx   libgl1, libglade2-0 (>= 1:2.6.1), libglib2.0-0 (>= 2.14.0),
libglu1-mesa   libglu1, libgnomecanvas2-0 (>= 2.11.1), libgnomeprint2.2-0 (>= 2.17.0),
libgnomeprintui2.2-0 (>= 2.17.0), libgnomevfs2-0 (>= 1:2.17.90), libgoffice-0-4 (>= 0.4.2),
libgsf-1-114 (>= 1.14.7), libgtk2.0-0 (>= 2.12.0), libgtkglext1, libice6 (>= 1:1.0.0), libnspr4-0d (>= 1.8.0.10),
libopenbabel2, liborbit2 (>= 1:2.14.1), libpango1.0-0 (>= 1.18.3), libsm6, libstdc++6 (>= 4.2.1), libx11-6,
libxml2 (>= 2.6.27), libxmu6, libxt6, zlib1g (>= 1:1.2.3.3.dfsg-1)

and the new Depends field for gcu-plugin:

Depends: libc6 (>= 2.6.1-1), libgcc1 (>= 1:4.2.1), libgcu0 (<< 0.9),
libgcu0 (>= 0.8), libglib2.0-0 (>= 2.14.0), libgnomevfs2-0 (>= 1:2.17.90), libgtk2.0-0 (>= 2.12.0),
libnspr4-0d (>= 1.8.0.10), libstdc++6 (>= 4.2.1), libx11-6, libxml2 (>= 2.6.29), iceweasel   iceape-browser  
xulrunner

As another example bluefish now (the next upload) really only shows the necessary dependencies. Compare the current:

Depends: libart-2.0-2 (>= 2.3.18), libaspell15 (>= 0.60),
libatk1.0-0 (>= 1.20.0), libbonobo2-0 (>= 2.15.0), libbonoboui2-0 (>= 2.15.1), libc6 (>= 2.6.1-1),
libcairo2 (>= 1.4.0), libfontconfig1 (>= 2.4.0), libgconf2-4 (>= 2.13.5), libglib2.0-0 (>= 2.14.0),
libgnome2-0 (>= 2.17.3), libgnomecanvas2-0 (>= 2.11.1), libgnomeui-0 (>= 2.17.1),
libgnomevfs2-0 (>= 1:2.17.90), libgtk2.0-0 (>= 2.12.0), libice6 (>= 1:1.0.0), liborbit2 (>= 1:2.14.1),
libpango1.0-0 (>= 1.18.2), libpcre3 (>= 6.0), libpopt0 (>= 1.10), libsm6, libx11-6, libxcomposite1 (>= 1:0.3-1),
libxcursor1 (>> 1.1.2), libxdamage1 (>= 1:1.1), libxext6, libxfixes3 (>= 1:4.0.1), libxi6, libxinerama1,
libxrandr2 (>= 2:1.2.0), libxrender1

to the “upcoming” Depends field:

Depends: libaspell15 (>= 0.60), libc6 (>= 2.7-1), libglib2.0-0 (>= 2.14.0),
libgnomeui-0 (>= 2.17.1), libgnomevfs2-0 (>= 1:2.17.90), libgtk2.0-0 (>= 2.12.0), libpango1.0-0 (>= 1.18.3),
libpcre3 (>= 6.0)

Nice result. I should check (my) other packages too :) Maybe that’s also something for debichems TODO list.

We had a nice trip the rest of the way home from our Thanksgiving in New Jersey today. This morning we had breakfast in the Holiday Inn Express that we stayed in (very nice, new, clean, good pool and exercise room -- a good value for $109/night dual queens). We then spent the rest of our morning shopping at the Whole Foods Market. It's the closest Whole Foods to our house in Montreal -- hold it, no, it looks like the one in Portland (Maine) is closer. Who knew? We picked up a lot of hippy-dippy personal hygiene items that we can't seem to find in Montreal, and I also got a few bottles of wine to take across the border. I got a bottle of 2003 Andrew Will Cellars Champoux Vineyard -- an interesting Columbia Valley red blend, mostly Cabernet Sauvignon that I think will cellar nicely for a couple of years, but not too urgent. And I also picked up a 2004 Hanzell Vineyards Chardonnay, Sonoma Valley, which I also think will sit nicely for a year or so. We'll see -- I'm not really big on cellaring yet, but I'd like to give a few wines a try. We stopped in Amherst (Massachusetts) for lunch at the Amherst Brewing Company, which has good pub grub, generous portions, and really nice house brews available. We also dropped into a few bookstores. Books are a great cross-border purchase for Canadian residents, since their cover price is usually ridiculously skewed. For example: I got a Paul Auster hardback that was labeled at $22 US, $28 Canada. That's with the loonie trading at US$1.01! Our drive up to Montreal was pretty decent, although we had a ton of rain and snow that made the actual driving kind of sucky. We stopped in a cruddy gas station in Sharon, Vermont that didn't have a bathroom (doh!) and didn't stop much afterwards until Burlington. We got dinner at a great place in downtown Burlington, Burlington Hearth (AKA American Flatbread). Great oven-fired pizza, organic ingredients, nice atmosphere, more microbrew. Fabulous. We got across the border all right, although I needed some time to get my new work permit for 2008. Then a long snowy drive back home, where my friend Damien from Austin is staying with us for this week during MIGS. Should be a fun week -- we have lots to catch up on. tags: travel massachusetts wine vermont burlington amherst montreal

Now as anyone who knows me knows, I love Git. I find it gives a very productive and comfortable workflow. One of my favourite new(ish) features is git rebase --interactive, which along with cheap’n'fast branching gives a great way to hack and commit as you go and then later on refactor your work into sets of nice orthogonal patches. However, there I do still find a good few things wrong with git, so I’d like to discuss these problems and possible solutions:

1. Error Messages In general git has some terrible cryptic error messages. My favourite example is if you modify a file and then pull a change that modifies that file before you commit it: Updating beb8c37..b953e6f
test: needs update
fatal: Entry 'test' not uptodate. Cannot merge. WTF? what does uptodate mean? why isn’t it up to date? whats gone wrong? what should i do?
   Surely something like: Cancelling pull as 'test' has changes that would be overwritten by this pull. Please commit your changes or force an overwrite using the '--force' flag would be a lot better? I leave it as a reader excercise to repeat this process for most error messages…
2. The index Why oh why oh why is it called the index? It does no indexing as far as the user is concerned. Its name gives you no clue to what it is, you don’t know when or how you’re supposed to use it. git add and git rm modify it and it’s used all over the place by the obscure low-level git helpers that you stumble over. How about commit staging area, that being what it actually is. You could have nice commands like git stage and git unstage and git stage --interactive. That’d make more sense, right? Then git add and git rm would be a lot more meaningful, being only used when you actually want add or remove a file.
3. Remotes With git remote this is a lot better than it used to be - having to use cryptic refspecs (see git-pull manpage for the gory details). Unfortunately refspecs are still scattered all over the documentation just to confuse you… That should almost certainly be moved out to less user-visible documentation. The main issues I have now are:
   • Remote branches don’t always behave like local branches when they should. In particular git pull . <remote branch name> doesn’t work (and gives an obscure error) but git merge <remote branch name> does. remote branches should act just like local branches you can’t modify.
   • I’m not sure I like git branch -r to list remote branches, maybe git remote branches would be nicer and more obvious, or even git branch --remote
4. Lots of git-* commands that confuse people This is a common complaint, and I have to say that I agree. Why not put all the low-level commands in libexec? People who really use them often (which can’t be many!) can always add it to their path.
5. No bug tracker! For such a user-facing tool, a bug tracker would be very useful for users who lack the experience or time to get their hands dirty in the codebase. It certainly seems that just taking bugs to the mailinglist without a patch isn’t generally useful.

So, what do people think? What other bugbears do beginners and experienced git users see? Are these useful ideas? After some discussion, I’ll happily take these to the git mailing list and come up with some patches! Update:
Looks like people had some problems with my OpenID based login. Comments are now open.

In reply to a blog post by Russell Coker about air filters for server environments, I just confirm how important it is to take care of a dust free machine.
In most cases the users won't have air conditioning with filters, but are just using plain desktop machines that are collecting dust. In some cases one might argue that a computer with all of its fans is some sort of air filter by itself, but well... ;)

So, keep your machine and yourself happy and remove regularily the dust from the inside of your machines. For example, I just removed the dust from one of the Debian m68k buildds last week. It was running for about a whole year now without any problem or downtime, so removing the dust was necessary.

I removed the case, the expansion cards and other stuff to get most direct access to the machine. I bought some canned air (spray duster) and blew the dust out of the case. Take care to remove the dust from inside of the power supply as well (of course remove the power plug before opening the case anyway for your own security!). After dusting off with the spray cleaner, I'm always using an old tooth brush to clean the dust from the fans itself. All in all, it's some dirty work and maybe you should do it outside of your house whenever possible. ;)

But in the long run, cleaning your machines once or twice a year prolongs the life of your machines. Some of my machines are now running since 1991 in 24/7 mode. Alas, maybe the reason for some people not cleaning their machines is to raise a nice dust puppy. :-)


P.S.: If you can afford it, choose a compressor instead of compressed air in cans.

I am a member of the Greens. The main reason for joining them is that they have principles. The Greens Charter [1] guides everything, policy must comply with the charter and candidates agree to uphold the policies which have been ratified if they get elected. There are no “non-core promises“. The policies of the Greens are all positive. The major parties have some mean-spirited policies that aim to help Australians by making things worse for people in other countries. Unfortunately for them the world is very inter-connected at the moment, so it’s difficult to harm other countries without harming yourself in the process. As you might expect the Greens are very positive towards the environment. Anyone who expects to live for more than 30 years (or who has children) should be very concerned about this. Currently even the most cautious estimates of the scope of the climate change problem by reputable scientists suggest that there will be serious problems in the next few decades. If you are young or have children then you should vote for the Greens (that covers most people). Many religious groups have determined that God wants them to help the environment, for example the Australian Anglican Church General Synod 2007 resolved that “the Anglican Communion’s 5th mark of mission to safeguard the integrity of creation and sustain and renew the life of the earth; and recognises that human activity contributing to Climate change is one of the most pressing ethical issues of our time”. Anglicans and believers in other religions that have similar ideas should vote for the Greens. The Greens have policies that are very positive towards minority groups. If you are not a straight-white-Christian then this is a strong reason for voting for the Greens. If you are a straight-white-Christian but have compassion for others (as advocated in the Bible) then again voting for the Greens is the right thing to do. The Greens policies are humane towards people in unfortunate situations, including drug addicts, the mentally ill, and the unemployed. When considering who to vote for keep in mind that at some future time you or a close friend or relative may fall into one of those categories. Finally the Howard government’s industrial relations legislation is very bad for the majority of workers. If the Greens get the balance of power in the Senate then they will be able to get such laws significantly changed or removed. The Greens election web site is here [2]. As an aside, I never liked Paul Keating (our last Prime Minister) until I read his op-ed piece in The Age about John Howard [3]. Also here is an interesting article on preferences and why the major parties want people to misunderstand the Australian electoral system [4]. In summary if you vote for the Greens as your first preference and they don’t win, then your second preference gets counted, and if that party doesn’t win then the third preference counts, etc. So if you have the two major parties in last and second-last position then your second-last preference may make an impact on the result! Putting Greens in the #1 position does not “waste” your vote. Parties get government funding based on the number of #1 votes, vote for the Greens as your first preference and you effectively give them $2 for their next campaign. Finally when a party wins an election they immediately look at where the #1 votes went and often adjust their policies to please the people. Vote #1 for the Greens and if Labour wins they will have more of an incentive to adopt policies that are similar to those of the Greens.

• [1] http://www.vic.greens.org.au/about-the-greens/charter/english/


• [2] http://greens.org.au/Election


• [3] http://www.theage.com.au/articles/2007/11/21/1195321862029.html?page=fullpage


• [4] http://www.theage.com.au/news/opinion/your-voting-preferences-really-do-matter/2007/11/19/1195321692088.html?page=fullpage#contentSwap1

Share This

As mentioned in my previous post [1] the government is using our money to advertise its policies. I previously covered the “Internet as a threat to children” issue, the other big one is drugs. The first significant message in the “Talking with your kids about drugs” document concerns the criminal penalties for drug use. That could largely be removed if the penalties were dramatically decreased and if hard drugs were administered to registered addicts for a nominal fee. Forcing addicts to commit crimes or work as prostitutes to pay for their addiction just doesn’t work, economically or socially. The next message is that parents should be involved in their children’s lives and be trusted enough that their children will accept advice. A large part of this would be related to the amount of time spent with children which is determined to a large degree by the amount of time not spent working. As I mentioned in my previous post it seems that the actions of the Howard government over the last 10 years has made things significantly worse in this regard by forcing more parents to spend large amounts of their time working (including situations where both parents work full-time) just to pay for a home. One notable omission from the document was any mention of alcohol problems (apart from when mixed with illegal drugs). By many metrics alcohol causes more harm than all the illegal drugs combined. The US attempted to prohibit alcohol and failed dismally, and alcohol is legal in most countries. The government doesn’t want to adopt a harm-minimisation approach to drugs (which bears some similarities to the approach taken to alcohol) and therefore is stuck trying to claim that one somewhat addictive mind-altering substance is somehow inherently different to all other mind-altering substances. The document provided no information at all on harm minimisation. The idea seems to be that it’s better for some children to die of overdoses than for other children to potentially be less scared of the consequences of drug use. The problem is that children are very bad at assessing personal risk so they simply won’t be scared off. It’s best to provide equipment for testing drug purity etc to reduce the harm. The list of reasons for young people to try drugs starts with “availability and acceptability of the drug“, Parmesan cheese is more available and acceptable than any drug but I’ve been avoiding it at every opportunity since I was very young. :-# More serious reasons in the list include “rebellion“, “depression“, “as a way to relax or cope with stress, boredom or pain“, and “to feel OK, at least temporarily (self-medication)“. But predictably there was nothing in the document about why children might consider that their life sucks to badly that they want to rebel in that way, be depressed, stressed, bored, or in pain to a degree that drugs seem like the solution. It seems unreasonable to believe that the school system isn’t a significant part of the cause of teenage problems. Flag poles for flying the Australian Flag [2] seems to be John Howard’s idea of a solution to school problems. I believe that the solution to school problems (both in terms of education and protection of children) is smaller classes, better supervision, an active program to eradicate bullying, and better salaries for teachers. I’ve written about related issues in my school category. I believe that the best thing for individual families to do is to home-school their children from the age of 12 instead of sending them to a high-school (which is much more damaging than a primary school). Another significant issue is drug pushers. They are usually drug users who try to finance their own drug use by selling to others. As most drug users get the same idea there is a lot of competition for sales and therefore they try to encourage more people to use drugs to get more customers. In the Netherlands, Coffee Shops sell a range of the milder drugs at reasonable prices which removes the street market and therefore the pushers. When drugs are produced commercially the quality is standardised which dramatically reduces the risk of infection and overdoses. When drugs are sold by legal companies they are not sold to children (a typical pusher can be expected to sell to anyone). The document has 3.5 pages of information about some of the illegal drugs (out of 23 pages total). They include all the most severe symptoms that most users don’t encounter. If someone reads that information and then talks to a drug user they will be laughed at. They also group drugs in an unreasonable way, for example listing “marijuana” and “hashish” as synonyms and list problems that are caused indirectly. If alcohol was included in that list it would include beer and methylated spirits (ethanol mixed with methanol to supposedly make it undrinkable - rumoured to be consumed by homeless people) in the same section and imply that blindness and other bad results of drinking “metho” apply to beer. It would also say that alcohol kills many people through drunk-driving and is responsible for rapes, wife-beating, and any other crime which can statistically shown to be more likely to be committed by a drunk person. A final issue with the document is that it advises parents to be “informed, up-front and honest” when talking to their children about drugs. Unfortunately the document itself doesn’t demonstrate such qualities. By omitting all information on harm minimisation it is not being up-front at all, and by not mentioning alcohol it’s simply being dishonest. Anyone who tries to convince someone else to not use illegal drugs can expect to hear “but you drink alcohol” as the immediate response from the drug user, it’s necessary to have sensible answers to that issue if you want to be effective in discouraging drug use.

• [1] http://etbe.coker.com.au/2007/11/22/internet-and-an-election/


• [2] http://www.pm.gov.au/media/interview/2004/Interview925.cfm

Share This

Tollef Fog Heen asks:

Also, why does bloglines link to the completely wrong place on dburrow’s posts? It links to http://planet.debian.org/tag:blogger.com,1999:blog-$blah rather than the real URL.

I think I know why. I don’t know which Planet feed Tollef is reading, but a look at the Planet Debian RSS 2.0 feed is illuminating. Looking up Daniel’s entry, we can find the following sub-element:

<guid>tag:blogger.com,1999:blog-12729767.post-4989856010680867140</guid>

Looking at the RSS 2.0 specification, we can find the following note:

If the guid element has an attribute named “isPermaLink” with a value of true, the reader may assume that it is a permalink to the item, that is, a url that can be opened in a Web browser, that points to the full item described by the element. An example: <guid isPermaLink=”true”>http://inessential.com/2002/09/01.php#a2</guid> isPermaLink is optional, its default value is true. If its value is false, the guid may not be assumed to be a url, or a url to anything in particular.

Thus the Planet RSS 2.0 feed claims that tag:blogger.com,1999:blog-12729767.post-4989856010680867140 is a valid URL. There is no basis for this claim in Daniel’s Atom feed, as that string only occurs inside an id element, and RFC 4287 specifies that the string cannot be assumed to be dereferencable.

As correctly pointed out by cmot [1] my previous post about software RAID [2] made no mention of bus bandwidth. I have measured the bus bottlenecks of a couple of desktop machines running IDE disks with my ZCAV [3] benchmark (part of the Bonnie++ suite). The results show that two typical desktop machines had significant bottlenecks when running two disks for contiguous read operations [4]. Here is one of the graphs which shows that when two disks were active (on different IDE cables) the aggregate throughput was just under 80MB/s on a P4 1.5GHz while the disks were capable of delivering up to 120MB/s:
On a system such as the above P4 using software RAID will give a performance hit when compared to a hardware RAID device which is capable of driving both disks at full speed. I did not benchmark the relative speeds of read and write operations (writing is often slightly slower), but if for the sake of discussion we assume that read and write give the same performance then software RAID would only give 2/3 the performance of a theoretical perfect hardware RAID-1 implementation for large contiguous writes. On a RAID-5 array the bandwidth for large contiguous writes is the data size multiplied by N/(N-1) (where N is the number of disks), and on a RAID-6 array it is N/(N-2). For the case of a four disk RAID-6 array that would give the same overhead as writing to a RAID-1 and for the case of a minimal RAID-5 array it would be 50% more writes. So from the perspective of “I need X bandwidth, can my hardware deliver it” if I needed 40MB/s of bandwidth for contiguous writes then a 3 disk RAID-5 might work but a RAID-1 definitely would hit a bottleneck. Given that large contiguous writes to a RAID-1 is a corner case and that minimal sized RAID-5 and RAID-6 arrays are rare in most cases there should not be a significant overhead. As the number of seeks increases the actual amount of data transferred gets quite small. A few years ago I was running some mail servers which had a very intense IO load, four U320 SCSI disks in a hardware RAID-5 array was a system bottleneck - yet the IO was only 600KB/s of reads and 3MB/s of writes. In that case seeks were the bottleneck and write-back caching (which is another problem area for Linux software RAID) was necessary for good performance. For the example of my P4 system, it is quite obvious that with a four disk software RAID array consisting of disks that are reasonably new (anything slightly newer than the machine) there would be some bottlenecks. Another problem with Linux software RAID is that traditionally it has had to check the consistency of the entire RAID array in the case of an unexpected power failure. Such checks are the best way to get all disks in a RAID array fully utilised (Linux software RAID does not support reading from all disks in a mirror and checking that they are consistent for regular reads), so of course the issue of a bus bottleneck becomes an issue. Of course the solution to these problems is to use a server for server tasks and then you will not run out of bus bandwidth so easily. In the days before PCI-X and PCIe there were people running Linux software RAID-0 across multiple 3ware hardware controllers to get better bandwidth. A good server will have multiple PCI buses so getting an aggregate throughput greater than PCI bus bandwidth is possible. Reports of 400MB/s transfer rates using two 64bit PCI buses (each limited to ~266MB/s) were not uncommon. Of course then you run into the same problem, but instead of being limited to the performance of IDE controllers on the motherboard in a desktop system (as in my test machine) you would be limited to the number of PCI buses and the speed of each bus. If you were to install enough disks to even come close to the performance limits of PCIe then I expect that you would find that the CPU utilisation for the XOR operations is something that you want to off-load. But then on such a system you would probably want the other benefits of hardware RAID (dynamic growth, having one RAID that has a number of LUNs exported to different machines, redundant RAID controllers in the same RAID box, etc). I think that probably 12 disks is about the practical limit of Linux software RAID due to these issues and the RAID check speed. But it should be noted that the vast majority of RAID installations have significantly less than 12 disks. One thing that cmot mentioned was a RAID controller that runs on the system bus and takes data from other devices on that bus. Does anyone know of such a device?

• [1] http://fortytwo.ch/blog/archives/2007/11/#e2007-11-16T10_54_22.txt


• [2] http://etbe.coker.com.au/2007/11/16/software-vs-hardware-raid/


• [3] http://www.coker.com.au/bonnie++/zcav/


• [4] http://www.coker.com.au/bonnie++/zcav/results.html

Share This

Some years ago I worked on a project where software reliability should have been a priority (managing data that was sometimes needed by the police, the fire brigade, and the ambulance service). Unfortunately the project had been tainted by a large consulting company that was a subsidiary of an accounting firm (I would never have expected accountants to know anything about programming and several large accounting firms have confirmed my expectations). I was hired to help port the code from OS/2 1.2 to NT 4.0. The accounting firm had established a standard practice of never calling free() because “you might call free() on memory that was still being used”. This was a terribly bad idea at the best of times and on a 16 bit OS with memory being allocated in 64K chunks the problems were quite obvious to everyone who had any programming experience. The most amusing example of this was a function that allocated some memory and returned a pointer which was being called as if it returned a boolean, one function had a few dozen lines of code similar to if(allocate_some_memory()). I created a second function which called the first, free’d any memory which had been allocated and then returned a boolean. Another serious problem with that project was the use of copy and paste coding. A section of code would perform a certain task and someone would need it elsewhere. Instead of making it a function and calling it from multiple places the code would be copied. Then one copy would be debugged or have new features added and the other copy wouldn’t. One classic example of this was a section of code that displayed an array of data points where each row would be in a colour that indicated it’s status. However setting a row to red would change the colour of all it’s columns, setting a row to blue would change all except the last, and changing it to green would change all but the second-last. The code in question had been copied and pasted to different sections with the colours hard-coded. Naturally I wrote a function to change the colour of a row and made it take the colour as a parameter, the program worked correctly and was smaller too. The next programmer who worked on that section of code would only need to make one change - instead of changing code in multiple places and maybe missing one. Another example of the copy/paste coding was comparing time-stamps. Naturally using libc or OS routines for managing time stamps didn’t occur to them so they had a structure with fields for the year, month, day, hours, minutes, and seconds that was different from every other such structure that is in common use and had to write their own code to compare them, for further excitement some comparisons were only on date and some were on date and time. Many of these date comparisons were buggy and often there were two date comparisons in the same function which had different bugs. I created functions for comparing dates and the code suddenly became a lot easier to read, less buggy, and smaller. I have just read an interesting post by Theodore Ts’o on whether perfect code exists [1]. While I understand both Theodore’s and Bryan’s points of view in this discussion I think that a more relevant issue for most programmers is how to create islands of reasonably good code in the swamp that is a typical software development project. While it was impossible for any one person to turn around a badly broken software development project such as the one I describe, it is often possible to make some foundation code work well which gives other programmers a place to start when improving the code quality. Having the worst of the memory leaks fixed meant that memory use could be analysed to find other bugs and having good functions for comparing dates made the code more readable and thus programmers could understand what they were looking at. I don’t claim that my code was perfect, even given the limitations of the data structures that I was using there was certainly scope for improvement. But my code was solid, clean, commented, and accepted by all members of the team (so they would continue writing code in the same way). It might even have resulted in saving someone’s life as any system which provides data to the emergency services can potentially kill people if it malfunctions. Projects based on free software tend not to be as badly run, but there are still some nasty over-grown systems based on free software where no-one seems able to debug them. I believe that the plan of starting with some library code and making it reasonably good (great code may be impossible for many reasons) and then trying to expand the sections of good code is a reasonable approach to many broken systems. Of course the ideal situation would be to re-write such broken systems from scratch, but as that is often impossible rewriting a section at a time often gives reasonable results.

• [1] http://thunk.org/tytso/blog/2007/11/18/does-perfect-code-exist-abstractions-part-1/

Share This

Three days ago I released KCheckGMail v0.5.7.3: nothing but a one-bug fix release.

---

I know it's a little late, I mean, three days ago!

Anyways, better blog about it now than not bloging about it ever.

This release was prepared by the new dev, Luis Pereira. It fixes an authentication problem in newer versions of gmail.
This time I also uploaded the md5 checksums signed with my key, for those who are interested in that stuff.

As usually, the new version can be downloaded from [url]http://kcheckgmail.sourceforge.net/download.html
Currently there are no packages neither for sarge nor etch, I'll build them as soon as I fix my wannaBuild.
And for unstable, I've already sent the new package to Anibal so I hope he'll have some time soon to review and upload the package before bugs about the new gmail version are filed in the BTS :)

Now that I'm finally writing a blog post, I think I'll also write something about what's up with me lately:

Some days ago I submitted my NM application :yes:

But besides that, I've been very busy lately because of school (exams, home work, etc), university (preparation courses, admission exams, etc), Real Life (tm), and because of some Debian-related work.

And I don't expect this to get any better soon (not before the end of the year), I'll be taking some other courses after school exams so I might be even more absent from my other projects (only some QA work on Debian seems to be easy lately).

At this month’s meeting of the Linux Users of Victoria [1] I gave a talk titled Everyone Should Blog (unfortunately the title was missed in the meeting announcement). I’ve been meaning to write about this but was finally prodded into writing by a ProBlogger post about overcrowding in the blog market [2]. Darren (the ProBlogger) suggests that the blog space is overcrowded, blogs are being sold for unreasonably high prices, and a bust similar to the dot-com bust may occur. I agree that there is some over-commercialisation, and that for Darren’s core market (people who want to make money directly from blogging) this is a serious issue to consider. However I believe that there is a lot of scope for more blogs. Firstly all of the services in the social networking space seem to have some feature that resembles a blog in some way (the online diary if not the RSS feed part of the definition). I have previously blogged about some ideas for an open social networking system [3]. A reform of the social networking scene should have the possibility of adding even more users and having them all be bloggers. While not everyone is capable of writing a blog that is of interest to a large audience, everyone who is capable of writing is capable of writing (or contributing to) a blog that is of interest to some people. The next potential area for expansion is in sales/marketting blogs. An example of this is car sales, it seems that every web site run by a car manufacturer only has information on the current model and at best only has information on prices of second-hand cars. There is a real need for authoritative information on older cars, it helps customers and it helps the car companies. If Ford was to provide good information on the cars they manufactured a few years ago it would increase the chance that I would buy a second-hand Ford and if enough people made similar decisions then it would increase the prices of such cars (basic rules of supply and demand). Increasing the second-hand price increases the price that people are prepared to pay for new cars and also allows them to buy a new car sooner - both of which are benefits to the manufacturer. It wouldn’t be difficult to use a blog server for all the information on new cars, you could have a front page that lists the available marques and links to the posts announcing the latest models of those marques. With a small amount of work writing a theme you could have Wordpress giving an appearance that’s almost indistinguishable from any of the major car companies web sites - apart from the fact that every car’s technical data would have a permanent URL of the form www.carcompany.com.au/2007/MARQUE/CODE which would be obviously a permanent link (unlike the Java servlet and ASP URLs you currently see) and therefore would encourage deep links by enthusiasts. This would also give RSS feeds for free (I would be interested in adding RSS feeds for several of the Australian car companies to my Planet installation if possible). A final benefit is the issue of ethics. When car companies remove data about cars they sold the previous year you have to wonder whether they have unreasonable price increases or whether the new cars have some specs that are lesser than the current cars. The general trend is slow and steady increases in features and performance at each price point, the number of people who are more likely to buy from an ethical (trustworthy) company is probably greater than the number of people who will look for details to nit-pick (and such people will find out anyway). Another use for blogs is for public events. It’s not uncommon for people to want to get some background information on an event that they attended years ago. If all the information was archived in the blog of the organisation that hosted the event then it would be easy for everyone to access. This would give additional promotion opportunities for the hosting organisation at almost no cost. Anyone who is announcing a future event should have an RSS feed for new information related to it, then as information is released all the interested people can see it in their favourite feed reader. Many people refuse to subscribe to announcement lists due to fear of spam and the fact that it is often inconvenient or impossible to unsubscribe (sometimes a list server regards unsubscribe requests as spam). Corporations have many potential uses of blogs, Google announces news about all their products via blogs and Microsoft has a good recruiting blog. Smaller companies can still achieve some of the same benefits via blogging. Also it’s well known that a blog which provides useful information about products in a field (not just the products being sold) can really attract potential customers who then react to the adverts in the blog. The same technique can be used by job seekers. Blog well and mention that you are looking for work and you will get offers of interviews. There is also no shortage of political and lobbying blogs as well, they don’t tend to convince the opposition but they are good for converting people who are ambivalent about an issue. As the people who read blogs get older (and therefore richer and more influential) such blogs will become increasingly important. I think that I have demonstrated the potential for a significant number of new blogs, some of which would have large subscriber bases and a lot of influence (which means a lot of potential profit for the Problogger crowd). Have I missed any major areas for developing new blogs?

• [1] http://www.luv.asn.au/


• [2] http://www.problogger.net/archives/2007/11/18/leveraging-the-power-of-blogs-in-an-overcrowded-market/


• [3] http://etbe.coker.com.au/2007/11/18/better-social-networking/

Share This

When advogato.org was still cool I signed up to it. It was an interesting research project in skill metrics (determining the rating of people’s coding skills by the votes of others and weighting the votes by the rating of each person), and it was nice to be rated Master soon after I joined. I still use it on occasion for the blog syndication feature (when I find a good blog on Advogato I add it to my Planet installation). When orkut.com was really cool (when every time I had dinner with a group of people someone would ask if they could be an “orkut friend”) I signed up to it. It was interesting for a while but then most people got bored with it. Now there is Facebook and MySpace for social networking for social purposes and LinkedIn.com for business related social networking. I periodically get invited to join those services but have not been interested in any other than LinkedIn. I can’t join LinkedIn because their mail server is regarded as a SPAM source by my mail server but their web server refuses to provide any information on why this is (the rejection was apparently long enough ago that it’s rolled off my logs). The problem with all these services is that I am expected to sign up with each of them and go to a moderate amount of effort in writing up all the data in the various web pages. Writing it is a pain, keeping it up to date is more pain, and dealing with spam in “scrap-book” entries in Orkut is still an annoyance which I don’t want to multiply by four! So far the only step I’ve seen towards addressing this issue is the XFN - XHTML Friends Network [1] project. But that seems to be of fairly limited scope (just referring to the friendship status of people in a <A HREF link). I believe that the requirements for social networking are:

1. Personal data about each person, some of which may only be available to friends or friends of friends.


2. The user owns their own data, has full control over where it’s sent and the ability to request people who receive it to keep some parts of it secret.


3. Ability to send email to friends of friends (determined by the wishes of each friend and FOAF).


4. Ability to get a list of friends of friends.


5. Incorporation of a standard format for CVs (for business social networking).

I think that the only way to go is to have a standard XML format for storing all personal data (including financial, career, and CV data) that can be used on any web site. Then someone who wants to be involved in social networking could create an XML file for a static web server (or multiple files with different content and password protected access), or they could have a server-side script generate an XML file on the fly with a set of data that is appropriate for the reader. The resulting social network would be entirely distributed and anyone could write software to join in. This covers item 1 and part of item 2. For sending email to friends of friends it would be good to avoid spam as much as possible. One way of doing would be requesting that friends publish a forwarding address on their own mail server in a manner similar to SRS [2]. SRS include the ability for such addresses to expire after a certain period of time (which would be convenient for this). In fact publishing SRS versions of friends email addresses would be a good option if you already use SPF [3] and SRS in your mail server. This covers item 3. The XML format could include information on how far the recipient could transfer it. For example if my server sent an XML file to a recruiting agency with my CV it could state that they could distribute it without restriction (so that they can give it to hiring managers) with the possibility of some fields being restricted (EG not tell the hiring manager what I used to get paid). For my mobile phone number I could send it to my friends with a request that they not send it on. This covers part of item 2. The URL for the friends file would of course be in the main XML file, and therefore you could have different friends lists published from different versions of your profile (EG the profile you send to recruiting agencies wouldn’t include drinking buddies etc). This completes the coverage of item 2. Then to have a friends list you have a single XML file on a web server that has the public parts of the XML files from all your friends. This means that getting a list of friends of friends would involve getting a single XML file for each friend (if you have 100 friends and each friend has 50 unique friends on average then you do 100 HTTP operations instead of 5,000). Minimising the number of web transfer operations is essential for performance and for reliability in the face of unreliable web servers (there is no chance of having 5,000 random web servers for individuals all up and accessible at the same time). This covers item 4. Item 5 is merely a nice thing to have which allows more easily replacing some of the recruiting infrastructure. As any such XML format will have several sections for arbitrary plain text (or maybe HTML) for describing various things the CV could of course be in HTML, but it would be good to have the data in XML. I posted this in the “blog” category because blogs are the only commonly used systems where end users do anything related to XML files (the RSS and ATOM feeds are XML). A blog server could easily be extended to do these social networking systems. As with a blog users could run their own social networking server (publishing their XML files) or they could use a service that is similar in concept to blogger which does it all for them (for the less technical users). Then an analogy to Planet, Technorati, etc in the blog space would be public aggregation services that compare people based on the number of friends they have etc, and attempts to map paths between people based on friends. This could also include GPG [4] data such that signing someone’s GPG key would cause your server to automatically list them in some friend category. The XML format should also have a field for a GPG signature (one option would be to use a GPG sub-key to sign the files and have the sub-key owned by the server). I don’t have any serious time to spend on implementing this at the moment. But if someone else starts coding such a project then I would certainly help test it, debug it, and contribute towards the XML design.

• [1] http://gmpg.org/xfn/and/


• [2] http://www.openspf.org/SRS


• [3] http://www.openspf.org/


• [4] http://www.gnupg.org/

Share This

Bryan Cantrill recently wrote a blog entry, where among other things, he philosophized on the concept of “perfect code”. He compares software to math, arguing that Euclid’s greatest common denominator algorithm shows no sign of wearing out, and that when code achieves perfection (or gets close to perfection), “it sediments into the information infrastructure” and the abstractions defined by that code becomes “the bedrock that future generations may build upon”. Later, in the comments of his blogs, when pressed to give some examples of such perfection, he cites a clever algorithm coded by his mentor to divide a high resolution timestamp by a billion extremely efficiently, and Solaris’s “cyclic subsystem”, a timer dispatch function. Watching his talk at Google where his introduction and sidebar book review on Scott Rosenberg’s “On Dreaming in Code”, it’s clear that he very passionately believes that it is possible to write perfect code, and that one should strive for that at all times. Perhaps that’s because he mostly writes code for operating systems, where the requirements change slowly, and for one OS in particular, Solaris, which tries far harder than most software projects to keep published interfaces stable for as long as possible. In contrast, the OS I’ve spent a lot of time hacking around, Linux, quite proudly states that at least inside the kernel, interfaces can not and should not be stable. Greg Kroah-Hart’s “Stable API Nonsense” is perhaps one of the strongly and most passionate expositions of that philosophy. I can see both sides of the argument, and in their place, both have something to offer. To Bryan’s first point, it is absolutely true that interfaces can become “bedrock” upon which entire ecosystems are built. Perhaps one of the most enduring and impactful example would be the Unix programming interface, which has since become enshrined by POSIX.2 and successor standards. I would argue, though, that it is the interface that is important, and not the code which initially implemented it. If the interface is powerful enough, and if it appears at the right time, and the initial implementation is good enough (not perfect!), then it can establish itself by virtue of the software which uses it becoming large enough that it assumes an important all out of scale with its original intention. Of course, sometimes such an interface is not perfect. There is an apocryphal story that when S. Feldman at AT&T labs first wrote the ‘make’ utility, that he did so rather quickly, and then put it available for his fellow lab members to use, and then went home to sleep. In some versions of the story he had stayed up late and/or pulled an all-nighter to write it, so he slept a long time. When he came back to work, he had come up with a number of ways to improve the syntax of the Makefile. Unfortunately, (so goes the story) that too many teams were already using the ‘make’ utility, so he didn’t feel he could change the Makefile syntax. I have no evidence that this ever took place, and I suspect it is an urban myth that was invented to explain why Makefiles have a rather ugly and unfortunate syntax that many would call defects, including the use of syntactically significant tab characters which are indistinguishable from other forms of leading whitespace. Another example which is the bane of filesystem designers everywhere are the Unix readdir(2), telldir(2), and seekdir(2) interfaces. These interfaces fundamentally assume that directories are stored in linear linked lists, and filesystems that wish to use more sophisticated data structures, such as b-trees, have to go to extraordinary lengths in order support these interfaces. Very few programs use telldir(2) and seekdir(2), but some filesystems such as JFS maintain two b-trees instead of one just to cater to telldir/seekdir. And yet, it is absolutely true that interfaces can be the bedrock for an entire industry. Certainly no matter what its warts, the Unix/Posix interface has proven the test of time, and it has been responsible for the success of many a company and many billions of dollars of market capitalization. But is this the same as perfect code? No, but if billions of dollars of user applications are going to be depending on that code, it’s best if code which implement such an interface be high quality, and should attempt to achieve perfection. But what does it mean for code to be perfect? For a particular environment, if the requirements can be articulated clearly, I can accept that code can reach perfection, in that it becomes as fast as possible (for the given computer platform), and it handles all exception cases, etc., etc. Unfortunately, in the real world, the environment and the requirements inevitably change over time. For example, Bryan’s cyclic subsystem, which he proudly touts as being if not perfect, almost so, and which executes at least 100 times a second on every Solaris system in the world. I haven’t looked at the cyclic system in any detail, since I don’t want to get myself contaminated (the CDDL and GPLv2 licenses are intentionally incompatible, and given that companies — including Sun — have sued over IPR issues, well, one can’t be too careful), but waking up the CPU from its low-power state 100 times a second isn’t a good thing at all if you are worried about energy conservation in data centers — or in laptops. For example, on my laptop, it is possible to keep wakeups down to no more than 30-35 times a second and it would be possible to do more but for an abstraction limitation. Suppose for example an process wants to be sleep and then receive a wakeup 1 milliseconds later, and so requests this via usleep(). At the same time, another application wants to sleep until some file descriptor activity takes place, or after 1.2 milliseconds takes place. 0.3 milliseconds later, a third process requests a sleep, this time for 0.8 milliseconds. Now, it could be that in all of the above cases, the applications don’t actually need exact timing; if they all get their wakeups plus or minus some fraction of a millisecond, they would be quite cool with that. Unfortunately, the standard timer interfaces have no way of expressing this, and so the OS can’t combine the three wakeups at T+1.0, T+1.1, and T+1.2 milliseconds into one wakeup at T+1.1ms. So this is where Greg K-H’s “Stable API Nonsense” comes into play. We may not be able to solve this problem at the userspace level, but we darn well can solve this problem inside the kernel. Inside the kernel, we can change the timer abstraction to allow device drivers and kernel routines to provide a timer delta plus a notion of how much accuracy is required for a particular timer request. Doing so might change a timer structure that previously external device drivers had depended upon — but too bad, that’s why stable ABI/API’s are not supported for internal kernel interfaces. Is this that the interface could have been extended? Well, perhaps, and perhaps not; if an interface is well designed, it is possible it can be extended in an API and/or ABI compatible way. There usually is a performance cost to doing so, and sometimes it may make sense to pay that that cost, and sometimes it may not. I’ll talk more about that in a future essay. Yet note what happened to the timer implementation. We have a pretty sophisticated timer implementation inside Linux, that uses heap data structures and buckets of timers for efficiency. So while I might not call it perfect, it is pretty good. But, oops! Thanks to this new requirement of energy efficiency, it will likely need to get changed to support variable levels of accuracy and the ability to fire multiple timers that are (more or less) coming due in a single CPU wakeup cycle. Does that make it no longer perfect, or no longer pretty good? Well, it just had a new requirement impact the code, and if criteria for perfection is for the abstraction defined by the code to be “bedrock” and never-changing, and no need to make any changes in said code over multiple years, I would argue that little to no code, even OS code, can ever achieve perfection by that definition — unless that code is no longer being used. (This is the first of a multi-part series of essays on abstractions that I have planned. The next essay which I plan to write will be entitled “Layer surfing, or jumping between layers”, and will be coming soon to a blog near you….)

If you live in a hotel for an extended period of time (which can provide significant career benefits - click on this link for details [1]) the issue of food price and availability is going to concern you. If you are in a decent hotel you will have a fridge in your room that you can use for your own food. A recent trend downwards in hotel quality has been to use fridges that are stocked with over-priced drinks that have sensors and automatically bill you if you move any of the drinks. A good hotel will have a fridge that either has space for your own food/drink or which allows you to temporarily move their stuff out. If you are staying in a hotel for any period of time and the hotel is not run by robots then you should have the option to negotiate the removal of all the over-priced drinks to provide you space for your own food. If you have such fridge space then you have good options for making sandwiches - which are cheap and healthy. In UK hotels (which incidentally tend to not have a fridge in the room if they are affordable) the standard practice is to have breakfast included as part of the hotel fee. If you are flexible about your eating then you can eat a large breakfast and have a minimal lunch to reduce expenses. Finally you have to consider how much you earn as an hourly rate (after tax) and compare it to the cost of food. For example if dinner at a cheap restaurant costs $10 and you earn $30 per hour after tax then you only need to save 20 minutes of your time by eating at the restaurant (as opposed to making a meal and washing the dishes) to make it economically viable. I have previously written about the efficiency of work [2]. I think it’s reasonable to assume (in the absence of any formal studies on the topic) that when your efficiency of working decreases due to over-work your enjoyment of your leisure time is also reduced on a per-hour basis (in addition to having less leisure time). I know that some people enjoy cooking and consider it a leisure activity (my sister seems to be one of them [3]). But if cooking isn’t something you enjoy then you will probably feel that eating out is reducing the amount of “work” time and therefore increases the quality of your life and the quality of your work. Finally for the time spent living in a hotel while searching for work (if you travel to another country without arranging employment first) the main financial factor is not how much you can save money on a per-day basis, but how quickly you can find work. The ability to accept a job offer from any region has the potential to significantly reduce the amount of time taken to find work and thus put you in a better financial position in the long-term. This benefit of living in hotels should significantly outweigh the extra expenses of eating out etc.

• [1] http://etbe.coker.com.au/2007/11/14/more-about-living-in-hotels/


• [2] http://etbe.coker.com.au/2007/11/02/increasing-efficiency-through-less-work/


• [3] http://helen.coker.com.au/category/food/

Share This

From November 9 till November 15 I attended SAGE Days 6 in Bristol, UK. It was a conference and a coding sprint for the SAGE project, which wants to build something comparable to Mathematica, Maple, Matlab and Magma built only from open source components without reinventing the wheel. Sage also provides excellent novel implementations for many mathematical algorithms (not found in Mathematica/Maple etc.).

I was invited to give a talk about SymPy, which is a Python library for symbolic mathematics, that we are doing.

Here are my notes from each day:

Saturday: one, two
Sunday: one, two
Monday, Tuesday, Wednesday: one

Impressions

Very positive. I started SymPy two years ago, because I wanted to play with symbolic mathematics in Python, see for example my presentation I gave at this conference for details. SAGE at that time was just able to do some mathematics things, but it was very weak in calculus, which is the SymPy's main domain. This has changed last half a year, when SAGE people managed to wrap Maxima in Python (which I thought to be completely impossible), so I started to follow SAGE development more closely. After the SD6, I must say I became very excited about the project.

One problem with mailinglists, IRC and other online interaction is that it's very difficult for me to get an impression about the people and the project. Being able to meet the developers and discuss with them face to face gives me the impression very quickly and very accurately.

By far the biggest guarantee, why it is worthy for me to contribute to SAGE, is the project leader, William Stein. He is very rational and pragmatic (I like these two properties) and after many discussions with him, I came to realize that he has basically identical views on the important things as I do and were I on his place, I would do the same decisions as he did and does. That's very nice, because I can concentrate my energy on things that I like to improve and don't have to worry about other things, because I know he will do it right.

The other SAGE developers are experts and with similar attitude as William has. It's enthusiastic to be among people who make things happen. For example one of the authors of Cython, Robert, implemented during SD6 a very nice HTML output, that shows Cython code, with colors according to how many Python API calls are called on that particular line, and by clicking that line it shows the corresponding C code.

SAGE project has a high aim and it stricly goes for it, without looking too much to the right or left, and that's how it should be. And it does produce a lot of very useful and high quality stuff along the way, for example Cython (probably the best wrapper for C/C++ things now, only pypy could possibly beat it, but that's still more a research project) or the SAGE notebook, which looks like a Mathematica notebook, but better and in a browser (together with a revision history, sharing, SSL encryption, etc.).

The only little problem is that currently SAGE developers are all mathematicians and as is well-known, mathematicians looks at mathematic from a very different prospective than physicists. :) And so I need calculus, advanced calculus and only when this is working, and working well, I can build on it some more advanced features. SAGE currently goes a little the other way - it has a lot of advanced features, from number theory, modular forms, elliptic curves, etc., but the basic calculus still needs a lot of improvements. SAGE wraps Maxima, because Maxima is quite fast, very well tested, so it works well. It's difficult to extend and written in LISP and that's very bad. That's where SymPy could help - it's in Python, very easy to extend, but currently slower than Maxima (rewriting parts of SymPy using Cython, or even C directly, will make it faster, hopefully as fast as Maxima or faster).

It's not yet in Debian, but SAGE people are working on it. It's not an easy task unfortunately.

Conclusion

SAGE is a very promising young project and I think it will succeed to provide an open source alternative to Maple, Matlab, Mathematica and Magma.

Should you use software or hardware RAID? Many people claim that Hardware RAID is needed for performance (which can be true) but then claim that it’s because of the CPU use of the RAID calculations. Here is the data logged by the Linux kernel then the RAID-5 and RAID-6 drivers are loaded on a 1GHz Pentium-3 system: raid5: automatically using best checksumming function: pIII_sse
  pIII_sse  :  2044.000 MB/sec
raid5: using function: pIII_sse (2044.000 MB/sec)
raid6: int32×1    269 MB/s
raid6: int32×2    316 MB/s
raid6: int32×4    308 MB/s
raid6: int32×8    281 MB/s
raid6: mmxx1      914 MB/s
raid6: mmxx2    1000 MB/s
raid6: sse1×1    800 MB/s
raid6: sse1×2    1046 MB/s
raid6: using algorithm sse1×2 (1046 MB/s) There are few P3 systems that have enough IO capacity to support anywhere near 2000MB/s of disk IO and modern systems give even better CPU performance. The fastest disks available can sustain about 80MB/s when performing contiguous disk IO (which incidentally is a fairly rare operation). So if you had ten fast disks performing contiguous IO then you might be using 800MB/s of disk IO bandwidth, but that would hardly stretch your CPU performance. It’s obvious that CPU performance of the XOR calculations for RAID-5 (and the slightly more complex calculations for RAID-6) is not a bottleneck. Hardware RAID-5 often significantly outperforms software RAID-5 (in fact it should always outperform software RAID-5) even though in almost every case the RAID processor has significantly less CPU power than the main CPU. The benefit for hardware RAID-5 is in caching. A standard feature in a hardware RAID controller is a write-back disk cache in non-volatile RAM (RAM that has a battery backup and can typically keep it’s data for more than 24 hours without power). All RAID levels benefit from this but RAID-5 and RAID-6 gain particular benefits. In RAID-5 a small write (less than the stripe size) requires either that all the blocks other than the ones to be written are read or that the original content of the block to be written and the parity block are read - in either case writing less than a full stripe to a RAID-5 requires some reads. If the write-back cache can store the data for long enough that a second write is performed to the same stripe (EG to files being created in the same Inode block) then performance may double. There is one situation where software RAID will give better performance (often significantly better performance), that is for low-end hardware RAID devices. I suspect that some hardware RAID vendors deliberately cripple the performance of low-end RAID devices (by using an extremely under-powered CPU among other things) to drive sales of the more expensive devices. In 2001 I benchmarked one hardware RAID controller as being able to only sustain 10MB/s for contiguous read and write operations (software RAID on lesser hardware would deliver 100MB/s or more). But for random synchronous writes the performance was great and that’s what mattered for the application in question. Also there are reliability issues related to write-back caching. In a well designed system an update of an entire RAID-5 stripe (one block to each disk including the parity block) will first be performed to the cache and then the cache will be written back. If the power fails while the write is in progress then it will be attempted again when power is restored thus ensuring that all disks have the same data. With any RAID implementation without such a NVRAM cache a write to the entire stripe could be partially successful. This means that the parity block would not match the data! In such a situation the machine would probably work well (fsck would ensure that the filesystem state was consistent) until a disk failed. When the RAID-5 recovery procedure is used after a disk is failed it uses the parity block to re-generate the missing data, but if the parity doesn’t match then the re-generated data will be different. A disk failure may happen while the machine is online and this could potentially result in filesystem and/or database meta-data changing on a running system - this is a bad situation that most filesystems and databases will not handle well. A further benefit of a well designed NVRAM cache is that it can be used on multiple systems. For their servers HP makes some guarantees about which replacement machines will accept the NVRAM module. So if you have a HP server running RAID-5 with an NVRAM cache then you could have the entire motherboard die, have HP support provide a replacement server, then when the replacement machine is booted with the old hard drives and NVRAM module installed the data in the write-back cache will be written! This is a significant feature for improving reliability in bad corner cases. NB I’m not saying that HP is better than all other RAID vendors in this regard, merely that I know what HP equipment will do and don’t know about the rest. It would be good if there was a commodity standard for NVRAM on a PC motherboard. Perhaps a standard socket design that Intel could specify and that every motherboard manufacturer would eventually support. Then to implement such things on a typical PC all that would be required would be the NVRAM module, which while still being expensive would be significantly cheaper than current prices due to the increase in volume. If there was a significant quantity of PCs with such NVRAM (or which could be upgraded to it without excessive cost) then there would be an incentive for people to modify the Linux sotware RAID code to use it and thus give benefits for performance and reliability. Then it could be possible to install a NVRAM module and drives in a replacement server with Linux software RAID and have the data integrity preserved. But unless/until such things happen write-back caching that preserves the data integrity requires hardware RAID. Another limitation of Linux software RAID is expanding RAID groups. A HP server that I work on had two disks in a RAID-1 array, one of my colleagues added an extra disk and made it a RAID-5, the hardware RAID device moved the data around as appropriate while the machine was running and the disk space was expanded without any down-time. Some similar things can be done with Linux, for example here is documentation on converting RAID-1 to RAID-5 with Linux software RAID [1]. But that conversion operation requires some down-time and is not something that’s officially supported, while converting RAID-1 to RAID-5 with HP hardware RAID is a standard supported feature.

• [1] http://scott.wallace.sh/node/1521

Share This

Fixing bugs in code and committing it to VCS usually also means that you need to interact somehow with BTS you use to let it know that the bug has been fixed. This is manual work which could be done by some clever commit scripts, right? All what is needed is just spend some time to set it up. Most of my development currently goes to Gammu/Wammu which uses own Mantis bug tracker. It is already a bit prepared for integration with VCS, but I never found their documentation sufficient. Fortunately somebody wrote tutorial on integrating Mantis and Subversion, which made it quite easy to set up. All what was needed was to put little commit hook into SVN. Now the logical step was to make it also for Debian packages. A bit of Googling revealed discussion on debian-devel. Unfortunately no example for SVN, but ripping out needed things from Manoj's script for Arch was quite easy and I made my own commit hook. While looking at this, I also found wonderful tool called debcommit, which commits changes to Debian package using changelog entries as commit message. I still wonder how many useful tools are hidden to me. The only thing I don't know is why Google didn't show me yesterday svnlog, which seems to have support for Debian BTS out of the box. Maybe I just entered slightly different keywords than today when trying to find the mentioned thread on debian-devel.

I was viewing one of my most favorite speakers today, Lawrence Lessig, talk about copyright, giving a speech that is very similar to one that I have heard a few times online, but Lessig presents it so well that I end up watching it over and over again. This particular speech was given at TED. The nice thing about coming across a TED talk is that it is almost always enlightening in some way. Lessig's speech is enlightening enough, and it reminded me that I should watch something of his since he has switched to fighting corruption instead of copyright.

Another nice thing about the TED talks is that they are all sitting on the same site - so they are all interlinked with similar topics and they also include the new videos which are being posted fairly often. Sometimes I don't have time to watch more than one talk, but today I was blessed with a surfeit of time, so I clicked on another link which interested me. This talk was from Dan Gilbert and was coarsely about happiness, and more importantly synthetic happiness and how our brains work. It started with talk about the foundation of our evolved brains, of which the most important fact was considered the Impact Bias. Gilbert talked about the Impact bias as a primary result of the evolution of the mind from earlier man to modern man - it is the ability of our mind to predict the future without actually trying it. It is the reason why you have an aversion to a food that you haven't even eaten or don't think you will like a movie that isn't even released, or the opposite. I believe that the impact bias is very important to the psychology of human beings - while it may not actually be very important to the state of the world that you eat a pastrami sandwich today instead of a salad, it must mean something to you, or your life will end up seeming meaningless and insignificant. Gilbert focuses on the impact bias in the dual - that choices that we are given to make ourselves will eventually have an effect on our overall happiness. The experiments explained in the video show that, if a choice has been made, the user is more likely to be happy with that choice later, especially if the choice is an irreversible decision. In the talk he talks about simulated happiness and actual happiness, and that the mind is incredible at simulating happiness when placed in a situation where it did not come out for the better - some famous examples include people who are in jail for long times feeling that they are better for their incarceration after being released (or even while incarcerated). One important part which I think wasn't really touched upon by Gilbert was the fact that while the mind can produce synthetic happiness, in the long run the difference between synthetic and "real" happiness is actually rather small - over the long run all the happiness produced tends to even out, and the mind doesn't distinguish about really being happy or not. For a third point, it is a very interesting conclusion (and probably a significant jump) that most people strive to be happy overall, and most people succeed. This means that even though there are people in many different levels of happiness from an outside perspective, it is worthless to look at someone who is rich and happy and someone who is poor and happy and say that one must be happier than the other because they have more of the "real" and less of the synthetic happiness. Happiness is not a zero-sum game - everyone can be happy all of the time, and noone loses out. This may be an interesting ideal to reach for and may seem impossible to achieve considering all of the hate and rage which exists in the world today, but I believe that it is even possible yet - especially considering this "Happiness Bias" which seems to exist within everyone.