There is an interesting web site WorseThanFailure.com (with the slogan “Curious Perversions in Information Technology”) that documents amusingly failed projects. The name used to be TheDailyWTF.com but changed due to the idea that for some projects success (interpreted to mean limping along in production) is worse than failure (being scrapped and re-written). I’ve created a new category WTF [1] on my blog to document such projects, both ones that I have personally witnessed and ones that friends and colleagues have seen. In the 90’s I spent some time working on an OS/2 GUI program to be the front-end for a mainframe backed system used by call-center workers. The first thing that they did was to develop a file naming scheme, they decided that all source files should be in the same directory (not unreasonable), but that 8.3 should be the limit of file name lengths in case there was a need to port the system to Windows 95. The fact that porting a program which did little other than display a GUI and talk to a S/390 server to a different OS was difficult (given the large amount of platform specific GUI code) was lost on them. Then they decided that the first two letters would be “FE” in case the source was merged with another bank project (of course the bank had many “Front End” systems talking to mainframes - so even this didn’t guarantee that they would be free of name clashes). Characters 3 and 4 were to represent the section of the program, and as the 3 character extension was one of “.h” or “.cpp” that left exactly 4 characters to name the file - we may as well have numbered the source files. Eventually one of the guys started making jokes about the file names by trying to pronounce them as words and that convinced a majority of the developers that the files should have long human-readable names. As the NMAKE expert (they insisted on not using unsupported software such as GNU make) it was my job to fix all the file names. The bank had an internal users’ group for the C++ development environment, but the contractors were not invited to attend. You might think that it would make sense to have the most skillful C++ programmers attend the meetings and share their knowledge, but apparently attending such meetings was a perk of being a permanent employee. What do you do when you have a C++ development project running behind schedule and a permanent employee who has had no prior exposure to C++ and wants to learn? One manager believed that assigning the junior programmer to the project is a solution to both problems - it was good for teaching the programmer C++ in a hurry but not so good for the deadline. There was a lot of excitement related to the back-end development. The mainframe guys assured me that CICS could never handle bad data and it totally wasn’t their fault if my program sent bad data to them and the region crashed. The first CICS region crash occurred when the mainframe guys told me to make the unused space in text fields “empty” - to a C programmer this means filling them with the 0 character but it killed the CICS region (apparently “empty” means filled with spaces to CICS programmers). A later region crash came when they told me to change the field length for the account name everywhere that it occurred - they told me that they wanted the change immediately - so I changed a macro definition, ran “make”, and sent a transaction with the changed size. Apparently “immediately” really meant “after we have spent an hour of software development changing magic numbers throughout our system”. I’m assuming that COBOL has some facility that roughly compares to C macros for defining field lengths and that it was the programmers not the language that was at fault. When the project became seriously behind schedule they hired some expert programmers from another country to help with the development. The experts decided that we needed to redesign the system (not that it had been designed in the first place) to use a “model view controller” architecture. When the real business logic is all on a S/390 and the only logic in the front end is to combine the contents of some screen-scrapes from a virtual 3270 terminal into a single GUI dialogue (with the first screen occasionally getting data from the IVR system) the benefits of MVC seem rather small. The new MVC version of the system was dubbed “phase 2″ and there were a series of deadlines for changing over from “phase 1″ which were missed. So the original team of developers continued on their project without any help from the “experts”. The last report I heard (some time after leaving the project) was that Phase 1 had gone live in production while Phase 2 was still being developed (at that time Phase 2 had a year of development and Phase 1 had about 15 months). The lesson to be learned from this is that management should only hire people that they can control. If you hire a consulting company and then let them do whatever they want (as opposed to solving the problem that they were hired to solve) then at best they will do no good and they may just slow things down. One of the more memorable incidents was when the project manager was talking to his 2IC, he said “I think we’re really breaking ground, [pause] I’m breaking wind and we’re breaking ground”. My desk was a few meters from where the wind (and ground) were being broken, I unfortunately was able to verify that wind was being broken but was not as certain about the ground. I did learn some useful things during this project, one was the fact that sample code from library vendors should not be trusted and ideally should be read from a print-out not pasted into the project source tree. The CTI library vendor had buffer overflows in their sample code. Another useful thing was the coin-tossing method of dispute resolution. If two good programmers have a disagreement about how to implement something tossing a coin is a quick way of resolving the dispute - this only works with two good programmers (in which case either idea will work). The project in question was messed up in many ways. I may write another post in the WTF category about it.

• [1] http://etbe.coker.com.au/category/wtf/

Share This

I just discovered the recently-opened Velouria Espresso in Jamaica Plain via this subscribers-only article in the Atlantic. Velouria marks the latest arrival in the single-origin coffee movement pioneered by George Howell. In a nutshell, the idea of single-origin coffee beans is that if beans are identified with a particular region (or even particular grower), the farmers will have more incentive to distinguish themselves. Ultimately, coffee beans will be known like fine wines, and the best growers will earn premium prices (thus bringing even more money to underdeveloped areas than coffee sold as fair-trade). Flatblack Coffee Company, near my office, operates under similar principles. More details in this episode of On Point with George Howell. Velouria distinguishes itself with its focus on brewed coffee rather than espresso (although they have plenty of that as well). The shop features the “Clover” coffee machine (”Bring out the subtle nuances of all of your coffees through complete, independent control of all of the important brew parameters: grind size, dose, water temperature, and contact time.”) I got a cappucino and a single-origin Kenyan coffee made with the Clover Machine. (Actually, two distinct single-origin Kenyan coffees, which Steve surreptitiously swapped on me while I wasn’t looking). They were all excellent. My cappucino started with a little foam heart in the crema: Amazingly, even when it was finished, the heart was still faintly visible: Steve enjoyed his espresso as well: Velouria has no website that I could find. it’s at 389 Centre Street, Jamaica Plain, MA 02131. Beware the Yelp page as it has inaccurate hours information — just call them at (617) 522-2400 if you’re wondering if they’re open. Now if we could only get a place like this in one of the vacant storefronts in Roslindale… Technorati Tags: Coffee, espresso, flatblack, single origin, velouria espresso, Boston

Share This

So as it turns out, two years ago today, was when I arrived in the US with the intent of staying here for a bit. Aside from recent developments, it's been a good couple of years. We're both pretty settled, have made plenty of wonderful friends, and seen a bit (not as much as we'd like) of the country. Sarah would have liked to have had a better time with employment, but it's not like she's been sitting around at home climbing the walls in her unemployment, either. We keep talking about "when we move back", but I'm personally in no huge rush. I also don't have any strong desire to do anything relating to putting down roots over here. Buying a place just feels extremely unfeasible, and we can't really stay forever on the visa type I'm on, and I can't really see myself ever qualifying for a H-1B.

At this month’s meeting of the Linux Users of Victoria [1] I gave a talk titled Everyone Should Blog (unfortunately the title was missed in the meeting announcement). I’ve been meaning to write about this but was finally prodded into writing by a ProBlogger post about overcrowding in the blog market [2]. Darren (the ProBlogger) suggests that the blog space is overcrowded, blogs are being sold for unreasonably high prices, and a bust similar to the dot-com bust may occur. I agree that there is some over-commercialisation, and that for Darren’s core market (people who want to make money directly from blogging) this is a serious issue to consider. However I believe that there is a lot of scope for more blogs. Firstly all of the services in the social networking space seem to have some feature that resembles a blog in some way (the online diary if not the RSS feed part of the definition). I have previously blogged about some ideas for an open social networking system [3]. A reform of the social networking scene should have the possibility of adding even more users and having them all be bloggers. While not everyone is capable of writing a blog that is of interest to a large audience, everyone who is capable of writing is capable of writing (or contributing to) a blog that is of interest to some people. The next potential area for expansion is in sales/marketting blogs. An example of this is car sales, it seems that every web site run by a car manufacturer only has information on the current model and at best only has information on prices of second-hand cars. There is a real need for authoritative information on older cars, it helps customers and it helps the car companies. If Ford was to provide good information on the cars they manufactured a few years ago it would increase the chance that I would buy a second-hand Ford and if enough people made similar decisions then it would increase the prices of such cars (basic rules of supply and demand). Increasing the second-hand price increases the price that people are prepared to pay for new cars and also allows them to buy a new car sooner - both of which are benefits to the manufacturer. It wouldn’t be difficult to use a blog server for all the information on new cars, you could have a front page that lists the available marques and links to the posts announcing the latest models of those marques. With a small amount of work writing a theme you could have Wordpress giving an appearance that’s almost indistinguishable from any of the major car companies web sites - apart from the fact that every car’s technical data would have a permanent URL of the form www.carcompany.com.au/2007/MARQUE/CODE which would be obviously a permanent link (unlike the Java servlet and ASP URLs you currently see) and therefore would encourage deep links by enthusiasts. This would also give RSS feeds for free (I would be interested in adding RSS feeds for several of the Australian car companies to my Planet installation if possible). A final benefit is the issue of ethics. When car companies remove data about cars they sold the previous year you have to wonder whether they have unreasonable price increases or whether the new cars have some specs that are lesser than the current cars. The general trend is slow and steady increases in features and performance at each price point, the number of people who are more likely to buy from an ethical (trustworthy) company is probably greater than the number of people who will look for details to nit-pick (and such people will find out anyway). Another use for blogs is for public events. It’s not uncommon for people to want to get some background information on an event that they attended years ago. If all the information was archived in the blog of the organisation that hosted the event then it would be easy for everyone to access. This would give additional promotion opportunities for the hosting organisation at almost no cost. Anyone who is announcing a future event should have an RSS feed for new information related to it, then as information is released all the interested people can see it in their favourite feed reader. Many people refuse to subscribe to announcement lists due to fear of spam and the fact that it is often inconvenient or impossible to unsubscribe (sometimes a list server regards unsubscribe requests as spam). Corporations have many potential uses of blogs, Google announces news about all their products via blogs and Microsoft has a good recruiting blog. Smaller companies can still achieve some of the same benefits via blogging. Also it’s well known that a blog which provides useful information about products in a field (not just the products being sold) can really attract potential customers who then react to the adverts in the blog. The same technique can be used by job seekers. Blog well and mention that you are looking for work and you will get offers of interviews. There is also no shortage of political and lobbying blogs as well, they don’t tend to convince the opposition but they are good for converting people who are ambivalent about an issue. As the people who read blogs get older (and therefore richer and more influential) such blogs will become increasingly important. I think that I have demonstrated the potential for a significant number of new blogs, some of which would have large subscriber bases and a lot of influence (which means a lot of potential profit for the Problogger crowd). Have I missed any major areas for developing new blogs?

• [1] http://www.luv.asn.au/


• [2] http://www.problogger.net/archives/2007/11/18/leveraging-the-power-of-blogs-in-an-overcrowded-market/


• [3] http://etbe.coker.com.au/2007/11/18/better-social-networking/

Share This

So it looks like I will have to use the other computer, and hope that Dell sorts things out quickly. I'm pretty disappointed, as I just purchased the machine (an Inspiron 1505 with Ubuntu) in July, and haven't really traveled with it much.

[440525.460000] ata2.00: configured for UDMA/33
[440525.460000] ata2: EH complete
[440942.908000] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
[440942.908000] ata2.00: cmd a0/01:00:00:00:00/00:00:00:00:00/a0 tag 0 cdb 0x43 data 12 in
[440942.908000]          res 50/00:01:00:00:00/00:00:00:00:00/a0 Emask 0x2 (HSM violation)
[440942.908000] ata2: soft resetting port

It required fsck'ing, and that was pretty much all she wrote. The disk itself, according to smartctl:

Device: ATA      ST980813AS       Version: 3.AD
Serial number:             5NH0301G

It doesn't support SMART though, so I don't know if the recent fuss about Ubuntu and Laptop HD's has anything to do with it. I never ran it much without power in any case, so I doubt it. It's especially frustrating, because I was making good progress on the Android port for Hecl, and will in the best of cases have to get a new computer "broken in", even though I do have good backups. My fingers are crossed that the money I spent for extra coverage will be worth something and that Dell Austria will be able to resolve the problem in a hurry, without requiring that I send the machine to the US or anything else that takes a lot of time. Sigh:-(

Bryan Cantrill recently wrote a blog entry, where among other things, he philosophized on the concept of “perfect code”. He compares software to math, arguing that Euclid’s greatest common denominator algorithm shows no sign of wearing out, and that when code achieves perfection (or gets close to perfection), “it sediments into the information infrastructure” and the abstractions defined by that code becomes “the bedrock that future generations may build upon”. Later, in the comments of his blogs, when pressed to give some examples of such perfection, he cites a clever algorithm coded by his mentor to divide a high resolution timestamp by a billion extremely efficiently, and Solaris’s “cyclic subsystem”, a timer dispatch function. Watching his talk at Google where his introduction and sidebar book review on Scott Rosenberg’s “On Dreaming in Code”, it’s clear that he very passionately believes that it is possible to write perfect code, and that one should strive for that at all times. Perhaps that’s because he mostly writes code for operating systems, where the requirements change slowly, and for one OS in particular, Solaris, which tries far harder than most software projects to keep published interfaces stable for as long as possible. In contrast, the OS I’ve spent a lot of time hacking around, Linux, quite proudly states that at least inside the kernel, interfaces can not and should not be stable. Greg Kroah-Hart’s “Stable API Nonsense” is perhaps one of the strongly and most passionate expositions of that philosophy. I can see both sides of the argument, and in their place, both have something to offer. To Bryan’s first point, it is absolutely true that interfaces can become “bedrock” upon which entire ecosystems are built. Perhaps one of the most enduring and impactful example would be the Unix programming interface, which has since become enshrined by POSIX.2 and successor standards. I would argue, though, that it is the interface that is important, and not the code which initially implemented it. If the interface is powerful enough, and if it appears at the right time, and the initial implementation is good enough (not perfect!), then it can establish itself by virtue of the software which uses it becoming large enough that it assumes an important all out of scale with its original intention. Of course, sometimes such an interface is not perfect. There is an apocryphal story that when S. Feldman at AT&T labs first wrote the ‘make’ utility, that he did so rather quickly, and then put it available for his fellow lab members to use, and then went home to sleep. In some versions of the story he had stayed up late and/or pulled an all-nighter to write it, so he slept a long time. When he came back to work, he had come up with a number of ways to improve the syntax of the Makefile. Unfortunately, (so goes the story) that too many teams were already using the ‘make’ utility, so he didn’t feel he could change the Makefile syntax. I have no evidence that this ever took place, and I suspect it is an urban myth that was invented to explain why Makefiles have a rather ugly and unfortunate syntax that many would call defects, including the use of syntactically significant tab characters which are indistinguishable from other forms of leading whitespace. Another example which is the bane of filesystem designers everywhere are the Unix readdir(2), telldir(2), and seekdir(2) interfaces. These interfaces fundamentally assume that directories are stored in linear linked lists, and filesystems that wish to use more sophisticated data structures, such as b-trees, have to go to extraordinary lengths in order support these interfaces. Very few programs use telldir(2) and seekdir(2), but some filesystems such as JFS maintain two b-trees instead of one just to cater to telldir/seekdir. And yet, it is absolutely true that interfaces can be the bedrock for an entire industry. Certainly no matter what its warts, the Unix/Posix interface has proven the test of time, and it has been responsible for the success of many a company and many billions of dollars of market capitalization. But is this the same as perfect code? No, but if billions of dollars of user applications are going to be depending on that code, it’s best if code which implement such an interface be high quality, and should attempt to achieve perfection. But what does it mean for code to be perfect? For a particular environment, if the requirements can be articulated clearly, I can accept that code can reach perfection, in that it becomes as fast as possible (for the given computer platform), and it handles all exception cases, etc., etc. Unfortunately, in the real world, the environment and the requirements inevitably change over time. For example, Bryan’s cyclic subsystem, which he proudly touts as being if not perfect, almost so, and which executes at least 100 times a second on every Solaris system in the world. I haven’t looked at the cyclic system in any detail, since I don’t want to get myself contaminated (the CDDL and GPLv2 licenses are intentionally incompatible, and given that companies — including Sun — have sued over IPR issues, well, one can’t be too careful), but waking up the CPU from its low-power state 100 times a second isn’t a good thing at all if you are worried about energy conservation in data centers — or in laptops. For example, on my laptop, it is possible to keep wakeups down to no more than 30-35 times a second and it would be possible to do more but for an abstraction limitation. Suppose for example an process wants to be sleep and then receive a wakeup 1 milliseconds later, and so requests this via usleep(). At the same time, another application wants to sleep until some file descriptor activity takes place, or after 1.2 milliseconds takes place. 0.3 milliseconds later, a third process requests a sleep, this time for 0.8 milliseconds. Now, it could be that in all of the above cases, the applications don’t actually need exact timing; if they all get their wakeups plus or minus some fraction of a millisecond, they would be quite cool with that. Unfortunately, the standard timer interfaces have no way of expressing this, and so the OS can’t combine the three wakeups at T+1.0, T+1.1, and T+1.2 milliseconds into one wakeup at T+1.1ms. So this is where Greg K-H’s “Stable API Nonsense” comes into play. We may not be able to solve this problem at the userspace level, but we darn well can solve this problem inside the kernel. Inside the kernel, we can change the timer abstraction to allow device drivers and kernel routines to provide a timer delta plus a notion of how much accuracy is required for a particular timer request. Doing so might change a timer structure that previously external device drivers had depended upon — but too bad, that’s why stable ABI/API’s are not supported for internal kernel interfaces. Is this that the interface could have been extended? Well, perhaps, and perhaps not; if an interface is well designed, it is possible it can be extended in an API and/or ABI compatible way. There usually is a performance cost to doing so, and sometimes it may make sense to pay that that cost, and sometimes it may not. I’ll talk more about that in a future essay. Yet note what happened to the timer implementation. We have a pretty sophisticated timer implementation inside Linux, that uses heap data structures and buckets of timers for efficiency. So while I might not call it perfect, it is pretty good. But, oops! Thanks to this new requirement of energy efficiency, it will likely need to get changed to support variable levels of accuracy and the ability to fire multiple timers that are (more or less) coming due in a single CPU wakeup cycle. Does that make it no longer perfect, or no longer pretty good? Well, it just had a new requirement impact the code, and if criteria for perfection is for the abstraction defined by the code to be “bedrock” and never-changing, and no need to make any changes in said code over multiple years, I would argue that little to no code, even OS code, can ever achieve perfection by that definition — unless that code is no longer being used. (This is the first of a multi-part series of essays on abstractions that I have planned. The next essay which I plan to write will be entitled “Layer surfing, or jumping between layers”, and will be coming soon to a blog near you….)

So, I've been off blogging for almost three months now -- from about mid-August to now. It's been kind of guilt-ridden, and it's also been kind of a relief. Being a B-list blogebrity is a direful burden, and having some time away was restful and relaxing and energizing. But now I'm ready to come back! One of the reasons that I haven't been blogging is that once you fall behind, it's really hard to catch up. So I've decided to not do the entire task at once. Instead, I'm just going to start a page on my 2007 Hiatus and fill that up as time permits. For those of you who care greatly about exactly what I did each and every day for the last three months, you've got a spot to read it. And, truthfully, I dropped out of the B-list ranks on Technorati during my time away. I'm back to my natural position as a C-list blogger, and baby, it feels good. Get ready for some photos of my cats, coming up soon! tags: blogging blogs me c-list hiatus

Feelings of loss One of the other main reasons I haven't been blogging in the last few months is that I've been losing weight. A lot of weight. I've dropped from a high body mass index (BMI) of about 43 -- dangerously high -- to my current sane but still high 33.6. I'd like to get down to a more reasonable BMI of 30 ("overweight") by the end of the year, and be somewhere around 25 ("normal") by next summer. I can lose weight pretty easily when I put my mind to it. Well, actually, it's still insanely hard, but I'm so damn stubborn and obsessive about things, when I actually sink my teeth into them, that it seems easy to the casual observer. My recipe is simple: reduce the amount of food I eat, do low-intensity exercise about an hour a day, and measure everything as well as possible. I eat 3 good meals a day -- about 50% fruits and vegetables, 25% complex carbs like rice or bread, and 25% protein like beans, tofu, fish or cheese. I steer clear of sweets and I drink alcohol only on occasion, and seldom more than a glass or two of wine. For exercise, I've been running around Parc Lafontaine or on a treadmill three days in a row, then on the fourth day lifting weights at a local gym on rue Gilford. I still have a ways to go, but the results have been good so far, and I'm feeling like I'm getting close to the neighbourhood of what I "should" weigh. I've been getting a lot of compliments from friends and family, which is a great motivator. Most of all, I feel good running, jumping, and playing with my daughter, which is the biggest reward of all. All that said, the exercise is taking an extra 1-2 hours out of my day, and blogging has been the main thing cut out to make room. But I think I can work around that now. tags: weight loss BMI diet exercise

Road trip We're leaving Tuesday for a trip to New Jersey to visit my dad's family for American Thanksgiving. (Yeah, they're my family, too.) We're all looking forward to seeing aunts and uncles and cousins and other extended family, but we're also looking forward to spending our powerful Canadian loonies in the US. The Canadian dollar's gone up and up this last few months, and though it's no longer trading at its ridiculous high of US$1.10 (!), it's still above parity, which means our money stretches pretty well. Coupled with this is the fact that most products here in Canada are still priced as if the loonie were still worth US$0.62. It's not atypical to see a book priced for $7.99 US, $10.99 Canadian. So we're going to do some shopping for clothes, books, music and other goods while we're stateside. I plan on bringing an empty suitcase with me and bringing it back full. (The fact that I've lost a lot of weight makes shopping for clothes particularly enticing.) All within our legal limits for customs returning to Canada, of course! Not to worry, random customs official who's reading this blog for who knows what reason. I'm also looking forward to this trip because we haven't done a driving trip together in a while. Going to Thanksgiving in 2005 was Amita June's second long trip (our overnighter to Ottawa for our anniversary in Oct 2005 was her first), and I think she's going to enjoy it again this time. tags: road trip thanksgiving loonie parity shopping

Painful upgrade I just upgraded my main server, Antipater, from Debian "sarge" to Ubuntu "gutsy". I was getting pretty frustrated by being 3+ versions back from a lot of the important software running on the system. I wanted to get on Ubuntu for my server platform to have more frequent upgrades (about every 6 months). The upgrade went well, mostly. I updated /etc/apt/sources.list to use the Ubuntu archives, then got all the packages. The installation was pretty quick, even for something like 700 packages. However, when I rebooted to use the new 2.6.22 kernel (up from 2.6.8 or so), the server's responsiveness went way down. My guess is that somewhere in the 2.6.x series, one of the new schedulers is causing my particular configuration some grief. I've trimmed my cron jobs and background processes down to a minimum, but the server is still running a little too loaded. The great folks at iWeb have been really helpful, but they don't support Ubuntu, so I've got to brainstorm for myself. I'm going to have to go in over the next few days to reboot the server and experiment with other kernels -- not fun. tags: iweb ubuntu antipater

One-liners Some other odds and ends:

• It's getting cold! We had our first snow flurries a few days ago, and it's been regularly under freezing around here. Time to get winter coats out of the closet -- or, better, buy new ones in the US.
• My friends at Kokoromi are organizing an excellent games+art event called GAMMA 256 on November 28 at the SAT. Should be a lot of fun.
• One such friend, Heather Kelley is going to be talking at the Montreal Geek Girl Dinner on Tuesday. Good for her!

tags: gamma kokoromi weather heatherkelley montrealgirlgeekdinner

In a nutshell: If your system is kind of older than sarge (as in installation date, updates done in the mean time don’t matter), beware of 2.6.23.x or update your grub boot sector, which Debian doesn’t do automatically on package installation. After 2.6.23.1 lasting for more than a month, I decided on wednesday to finally roll 2.6.23.1 out to most systems. Only to be overrun by seven releases in a row (which even kernel.org infrastructure not properly handled). Ok, so it’s 2.6.23.8. Installation of kernel on my test and semi-productive systems went just fine, so I took the opportunity of being in the datacenter for some unrelated matter on a saturday to update two productive boxes. Just to find them not coming up again. The first box, not connected to a KVM switch, but only to a serial console, looked like it simply died at the place where grub usually says “Loading Linux...”, while the second box, on a “real” KVM switch, was a little more helpful, saying “No setup signature found” (or something along this wording). Since there was no internet on site with the boxes down (one being the main router, the second one being the full service DNS resolver), I quickly chose the fall-back kernel and got back in business. Googling revealed that this is an issue connected with old grub and kernel 2.6.23, and indeed, running grub-setup brought both systems back online. Had I done this - as usual - from remote, this would have been an unexpected downtime of at least half an hour. Since all systems in question have the same version of grub (the one from etch) installed, I proceeded to investigate what has been going wrong here. Finding that grub does not have any maintainer scripts at all, it would have been my responsiblity to update the grub code in the boot sectors to the new version after updating to etch. The package didn’t bother doing this, and this is also not documented in README.Debian or NEWS.Debian. As this is a major pitfall for systems that have been installed pre-sarge, and since updating the Distribution does not solve the issue, I filed Bug #451701. I hope that this blog entry saves other people from unexpected downtimes. Ah, yes, and barfing the error message on an unused console is a bad foul of grub, see #451710

due to the latest changes in apt (which installs packages marked as Recommends: by default since 0.7.7) aptitude — even if "Install recommended packages automatically" is not checked in aptitude's options dialog — pulls in recommended packages just like apt-get, whether called from the command line or interactively; cf. also #448561.

the fix to keep both apt-get & aptitude from installing recommended packages is something like:

$ cat /etc/apt/apt.conf.d/15recommends
APT  
        Install-Recommends "false";
 ;

(also very useful for (p cow)builder where installing recommended packages in the chroot is just wrong & takes a long time; cf. also #448562.)

Ok, this is the first time of the year also for me - H0lger was the first to remind me that, as proud inhabitant of the not-precisely-warm tropic, once again this year I won't get a chance to get my first snowfall yet (although the volcanos around the city do get their share of snow, and as a child I've been there to play with it... Imagine a city of then-almost-20 million people rushing to the three nearby volcanos just to get a taste of what snow feels like - Yes, heavy traffic is the foremost sensation I remember)...Sigh... But, sadly, that's not the main concern right now. My life feels quite upside-down, inside-out, wrongside-right. I hope to be out of this situation soon. It is just not sustainable.

Recently a user has been asking about SE Linux support in MEPIS [1]. He seems to expect that as the distribution is based on Debian it should have the same SE Linux support as is in Debian. The problem with derived distributions (which potentially applies to all variants of Debian, Fedora, and RHEL) is that the compilation options used may differ from what is required for SE Linux support. If an application works in Debian then you can expect that it will work in all derived distributions. But SE Linux is not an application, it is a security extension to the OS which includes code in the kernel, login, cron, pam, sshd, logrotate, and others. For any one of these packages a maintainer of a derived distribution might decide to turn off features to save disk space or memory, or because they want to use features which don’t work well with them (due to functional differences or bugs). The maintainer of a derived distribution might even decide that they just don’t like a feature and disable it for that reason alone! I believe that it is possible to use APT with multiple repositories and specify preferences for each repository. So it should be possible to use a source such as MEPIS for most packages but Debian (or my private repository of SE Linux back-ports [2]) for the packages which need SE Linux support. That said, I am not sure why someone would want to use MEPIS with SE Linux. Currently the benefits of SE Linux are of most use for a server and MEPIS is a desktop focussed distribution. Debian works reasonably well for a desktop (it has worked well for me for most of the past 11 years), so it seems that Debian for a SE Linux desktop machine is a good choice and Debian is a better choice than MEPIS for a server.

• [1] http://www.mepis.org/


• [2] http://etbe.coker.com.au/2007/10/19/my-se-linux-etch-repository/

Share This

Is it possible to secure Internet banking with SMS? As secure tokens are too expensive ($10 or more in bulk) and considered to be too difficult to use by many (most?) customers banks have sought out other options. One option that has been implemented by the National Australia Bank and will soon be available from the Commonwealth Bank is SMS authentication of transfers. The idea is that when you issue an online banking request you receive an SMS with a password and then have to enter that password to authenticate it. If you receive an unexpected password then you know you have been attacked. I wonder how much information is in the SMS, does it include the amount and where the money is to be transferred (in the case of a funds transfer - the operation most likely to be used by attackers)? If the full details are not included then an attacker could hijack an active session, get the user to enter the password, and then act as if the user entered the password incorrectly. The user would then request a new SMS and complete their desired transfer without realising that they just authorised a transfer to Russia… If the full details are recorded will the user look at them? Online banking fraud often involves transferring the funds to an idiot in the same country as the victim. Then the idiot sends the money to the attacker in some other manner which is more difficult to track. I wonder whether an attacker could divert the funds transfer to one of the idiots in question and have the victim not realise that the wrong account number was used. Another issue is that of SMS interception. Anyone who can hack the network of a phone company could steal money from any bank account in the country! For wealthy people there is also the possibility of stealing their mobile phone and making funds transfers before they report the theft. Another possibility is to register for a new phone company. Last time I changed phone companies it took about an hour for the new company to have the phone number and I don’t recall the phone company doing anything to verify that I owned the number in question. If an attacker had a credit card with the same name as the victim (names are not unique so this is not impossible or even inherently illegal) they could open a new phone service and steal the number. Someone who’s mobile phone stops working probably wouldn’t assume that it was part of a bank fraud scheme and act accordingly, in fact if they don’t use their mobile phone later it might be several days before someone contacts them in some other manner and mentions that they weren’t answering their mobile phone. A final possibility is the situation where a mobile phone is connected to a computer. Devices that combine mobile phone and PDA functionality are becoming common. A trojan horse program that offered to do something useful when a mobile phone was connected to the PC via a USB cable might fool some users. All that would be required is a few minutes of the phone being connected if the attacker already has the password for online banking. Maybe they could even make it appear that the bank was demanding that the phone be connected to the PC - that should fool users who don’t understand how SMS authentication works. It seems to me that SMS authentication is an improvement (it adds an external device which usually can’t be directly manipulated by the attacker) but is far from perfect security. I previously wrote about the bad idea that you can bank with an infected computer [1]. SMS authentication is a good step towards making things more difficult for attackers (which is always a good idea) but doesn’t really secure the system. Also it costs 5 cents for each SMS, I expect that the banks will want their customers to pay for this - I would rather pay for a $10 token up-front.

• [1] http://etbe.coker.com.au/2007/10/19/banking-with-an-infected-computer/

Share This

Should you use software or hardware RAID? Many people claim that Hardware RAID is needed for performance (which can be true) but then claim that it’s because of the CPU use of the RAID calculations. Here is the data logged by the Linux kernel then the RAID-5 and RAID-6 drivers are loaded on a 1GHz Pentium-3 system: raid5: automatically using best checksumming function: pIII_sse
  pIII_sse  :  2044.000 MB/sec
raid5: using function: pIII_sse (2044.000 MB/sec)
raid6: int32×1    269 MB/s
raid6: int32×2    316 MB/s
raid6: int32×4    308 MB/s
raid6: int32×8    281 MB/s
raid6: mmxx1      914 MB/s
raid6: mmxx2    1000 MB/s
raid6: sse1×1    800 MB/s
raid6: sse1×2    1046 MB/s
raid6: using algorithm sse1×2 (1046 MB/s) There are few P3 systems that have enough IO capacity to support anywhere near 2000MB/s of disk IO and modern systems give even better CPU performance. The fastest disks available can sustain about 80MB/s when performing contiguous disk IO (which incidentally is a fairly rare operation). So if you had ten fast disks performing contiguous IO then you might be using 800MB/s of disk IO bandwidth, but that would hardly stretch your CPU performance. It’s obvious that CPU performance of the XOR calculations for RAID-5 (and the slightly more complex calculations for RAID-6) is not a bottleneck. Hardware RAID-5 often significantly outperforms software RAID-5 (in fact it should always outperform software RAID-5) even though in almost every case the RAID processor has significantly less CPU power than the main CPU. The benefit for hardware RAID-5 is in caching. A standard feature in a hardware RAID controller is a write-back disk cache in non-volatile RAM (RAM that has a battery backup and can typically keep it’s data for more than 24 hours without power). All RAID levels benefit from this but RAID-5 and RAID-6 gain particular benefits. In RAID-5 a small write (less than the stripe size) requires either that all the blocks other than the ones to be written are read or that the original content of the block to be written and the parity block are read - in either case writing less than a full stripe to a RAID-5 requires some reads. If the write-back cache can store the data for long enough that a second write is performed to the same stripe (EG to files being created in the same Inode block) then performance may double. There is one situation where software RAID will give better performance (often significantly better performance), that is for low-end hardware RAID devices. I suspect that some hardware RAID vendors deliberately cripple the performance of low-end RAID devices (by using an extremely under-powered CPU among other things) to drive sales of the more expensive devices. In 2001 I benchmarked one hardware RAID controller as being able to only sustain 10MB/s for contiguous read and write operations (software RAID on lesser hardware would deliver 100MB/s or more). But for random synchronous writes the performance was great and that’s what mattered for the application in question. Also there are reliability issues related to write-back caching. In a well designed system an update of an entire RAID-5 stripe (one block to each disk including the parity block) will first be performed to the cache and then the cache will be written back. If the power fails while the write is in progress then it will be attempted again when power is restored thus ensuring that all disks have the same data. With any RAID implementation without such a NVRAM cache a write to the entire stripe could be partially successful. This means that the parity block would not match the data! In such a situation the machine would probably work well (fsck would ensure that the filesystem state was consistent) until a disk failed. When the RAID-5 recovery procedure is used after a disk is failed it uses the parity block to re-generate the missing data, but if the parity doesn’t match then the re-generated data will be different. A disk failure may happen while the machine is online and this could potentially result in filesystem and/or database meta-data changing on a running system - this is a bad situation that most filesystems and databases will not handle well. A further benefit of a well designed NVRAM cache is that it can be used on multiple systems. For their servers HP makes some guarantees about which replacement machines will accept the NVRAM module. So if you have a HP server running RAID-5 with an NVRAM cache then you could have the entire motherboard die, have HP support provide a replacement server, then when the replacement machine is booted with the old hard drives and NVRAM module installed the data in the write-back cache will be written! This is a significant feature for improving reliability in bad corner cases. NB I’m not saying that HP is better than all other RAID vendors in this regard, merely that I know what HP equipment will do and don’t know about the rest. It would be good if there was a commodity standard for NVRAM on a PC motherboard. Perhaps a standard socket design that Intel could specify and that every motherboard manufacturer would eventually support. Then to implement such things on a typical PC all that would be required would be the NVRAM module, which while still being expensive would be significantly cheaper than current prices due to the increase in volume. If there was a significant quantity of PCs with such NVRAM (or which could be upgraded to it without excessive cost) then there would be an incentive for people to modify the Linux sotware RAID code to use it and thus give benefits for performance and reliability. Then it could be possible to install a NVRAM module and drives in a replacement server with Linux software RAID and have the data integrity preserved. But unless/until such things happen write-back caching that preserves the data integrity requires hardware RAID. Another limitation of Linux software RAID is expanding RAID groups. A HP server that I work on had two disks in a RAID-1 array, one of my colleagues added an extra disk and made it a RAID-5, the hardware RAID device moved the data around as appropriate while the machine was running and the disk space was expanded without any down-time. Some similar things can be done with Linux, for example here is documentation on converting RAID-1 to RAID-5 with Linux software RAID [1]. But that conversion operation requires some down-time and is not something that’s officially supported, while converting RAID-1 to RAID-5 with HP hardware RAID is a standard supported feature.

• [1] http://scott.wallace.sh/node/1521

Share This

Many thanks to everyone who sent in their thoughts on my post about vi and CTRL-[. Here's a quick summary of the comments I received on my blog and by IRC/email:

• The vast majority of the people who commented use the Esc key.
• Very few people use CTRL-[ (4 out of ~70).
• Many people use keyboard layouts where CTRL-[ is difficult to type (French, German, Finnish, Swedish, Spanish, ...)
• A few people have moved Esc elsewhere, such as the Caps Lock key (and here's how to do it, thanks Mark).
• A better alternative is to use CTRL-C, which works on all keyboard layouts. It doesn't do the exact same thing as Esc/CTRL-[ because it doesn't check for abbreviations; dato notes in a comment that this can be remedied in Vim with inoremap <C-c> <Esc><Esc>.
• Some people use the Esc key because it's hard to miss, being physically apart from other keys. Some people, on the other hand, complain that Esc is error-prone on laptops because it's too close to F1, or just awkwardly placed.
• Everybody helix! (not to be confused with the helix wind turbine)

As a closing note, a followup question from reader Laurent C.: :wq, :x or ZZ? :)

What I want for christmas:

So, our country is again in one of its recurrent strike madness. Yesterday (Wednesday), unions of the public transportation companies (mostly SNCF for trains all over the country, RATP for Paris and suburbs metros/bus system and maybe a few transport companies in other cities) called for an unlimited strike. The point is protesting against the project of harmonizing the retirement systems of these companies with the system used in private companies (AND many public ones such as Onera who I work for). The French retirement financing system is facing the challenge of all participative systems (where the current workers pay for the current retired people, contrary to cpitalization systems where one spares one's own retirement money during one's work life....I hope this is clear with my broken English). As there are less and less workers and more and more retired peoplen the load on workers is becoming harder and harder while the retirement 'salaries' (what's the word for 'pension'?) can only be lower. As a consequence, in the standard retirment system in France, ppl now have to work for 42 years to get a full pension when they retire (theoretically we can at 60....but practically, this happens at 65). In the same time, workers at SNCF, RATP and a few others still have a special retirement system with only 37 years to work and the possibility to retire when reaching 55 for some of them (good ol' days of steam trains justify the work being "hard"). All attempts to change this have failed up to now because of the power of unions at SNCF/RATP/others and the power they have to screw out the entire country. Attempts in 1999 lead to huge strike with, at that time, a strong support by the general public (who feared for their own retirement system.....the strike also being a good way to protest against the general policitcs of the government at that time...as usual). Things changed now and the majority of ppl no longer support so-called privileged people to try hanging to their special retirement system....but of course, unions still have to show their power and, anyway, very few strikers may break things a lot.....and here we are with huge strikes. Yesterday, I managed to do it well with a good train coverage on my line (1 all-stops train per 30minutes) and running at both ends (good for health!). Today was a little bit jerkier with the scheduled train being cancelled and the next one 30 minutes later...and well packed..:-) Hopefully, for tomorrow, the situation should improve as negotiations between government and unions are planned (they were already but I guess that unions had to show their strength....and check whether the average people support them like it happened in 1999....which is actually a complete failure as the general feeling over French population is "enough!"). That's good as I'm not completely sure I can sustain running 12km every day.:)

Update: problem fixed, thanks for the comments; it was MTU related issues.

Note: This is a long post, but I expect it to bring in some questions for many Linux people; I advise you to read this when you have enough time.


After the last events related to connectivity at home which have been lasting since Thursday evening and the weird fact that it seems that NAT doesn't work at home, but does at work for the same setup, I called the ISP's support and wasted 15 minutes trying to convince them at least to send a technical team with another modem just to test if that is the reason for the breakage.

Of course, talking to the ISP support and trying to convince them that there might be a problem on their side since NAT works with another provider but doesn't with them, was fruitless.

Tonight I was stuck and tried another approach, convinced I would confirm my suspicion that there is something wrong on ISP's side. Still I am not yet sure what to conclude from what happened.


So, to get a better view of what is going on, I'll describe the setup I have and what are its limitations and characteristics. People in a hurry can skip to the paragraph starting with "I tested" and stare in wonder at will.


So the connection I have is done through a DSL modem which gets the MAC of the network card connected to it and exposes that as its own to the ISP's network. This MAC seems to be quite persistent and special measures must be taken in order to be able to use another NIC to connect. The modem (or probably some machine in ISP's; the IP is 10.10.0.1) offers a DHCP address and everything should work fine.

Because of this "your MAC is my MAC" issue, when I connected the first time, I used an USB NIC since a broken router or some temporary failure would have allowed me to use the Internet connection directly from my laptop. I can say this decision has proven in time to be wise.

The router I used until now is a NSLU2 with Debian installed on it. The built in network interface always faced the internal network.

The router (which I call ritter) served as a NAT router for two machines inside the network, my laptop and my apartment mate's laptop. All until last Thursday, after which it never got back properly.



I tested (doing NAT on my laptop; the laptop shouldn't have been affected by the power problems):

• with ritter behind the laptop NAT, at home
  
• with two different virtual machines as NAT "clients", at home
  
• with ritter behind the laptop NAT, at work
• with a virtual machine as NAT client, at home
• directly from the laptop
• NAT made through a SNAT rule
• NAT made through a MASQUERADING rule
• with TTL mangled (increased by one, although is was never in the ballpark of a low TTL)

Of course, I have n-checked that /proc/sys/net/ipv4/ip_forward was set to 1, the tables had policy ACCEPT and there were no extra rules, except the basic NAT-ting stuff, the routes were correctly set on both the clients and the machine doing the NAT.


All I could see is that:

1. the machine doing the NAT was always working fine
2. at work all NAT clients worked fine
   
3. at home any of the NAT "clients" were
1. able to resolve addresses while the DNS server was in ISP territory and another LAN
2. ping the outside world (if ping was available - in D-I is not)
3. hanging when trying to get a http page
4. telnet-ting directly to the port 80 was ok (but I didn't try to "GET / HTTP/1.0")

So after all of this, I was thinking of trying to see if a new client (the laptop of my apartment mate, a Windows XP machine) would work using the connection at home. It didn't work.

Then I thought of trying to do "Internet Connection Sharing" as is called in Windows. Of course, there was some pain to find Windows XP drivers for the ASIX AX88172 network card (remember, the modem needed to see the MAC of that NIC), but I managed to find the proper one.


And, almost sure the NAT wouldn't work for this case either, I configured the new connection as a shared one. I didn't even disabled the firewall, as I was thinking I could take those down gradually.

I wasn't expecting this, not even by pure chance, but my laptop which was a NAT "client" now was able to browse, ping and do whatever was normal through NAT, while the Windows machine was doing the "Connection sharing".

I was utterly flabbergasted. And that was just the beginning.

I was expecting that the problem coincidentally went away, but after a minute I was proven otherwise. It still didn't work with Linux as the NAT-ting machine. I connected back the USB NIC to the Windows machine and I saw the same thing. NAT was just working.


At that point I was to observe an even more shocking fact: the IP that the Windows machine got was different from the one the Linux machine received, in spite of the fact that the network card was the same, so it would have made sense to get the same one. More than that, the IP that the Windows machine got was from an entirely different network, although it was a valid IP belonging to my ISP.

I was thinking that one reason why it works with Windows might be that there could be some TCP protocol twist that is differently implemented in Windows and the equipment from my ISP gets along better with the Windows network stack.


As a way to test that, I am thinking of forcing somehow the IP on the Linux machine to see if anything changes. But before doing that, I felt the urge to post these, maybe some kind soul will shed some light on this issue for me or drop a hint.

Another reason might be different DHCP servers answering, but I don't know how I can see in Windows who offered the lease.


If anyone has any clue why these weird things happen, please drop a line. I would greatly appreciate it. TIA.

My relearned lesson for the day: when Linux networking weirdness occurs in a NAT environment, remember to try MTU clamping.

Thanks to the comments by Justin and Sesse, I was fast-tracked to the core of the problems I have been experiencing since Thursday, MTU issues. What's worse (from my pov) is that I have encountered this issue before with the provider I had in Timi oara, but, since that ISP was using PPPoE and my current ISP in Bucharest doesn't, I never really made the connection. I even had a commented out iptables rule for MTU clamping in my firewall script.

The rule I am talking about looks like this:

iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o $EXT_IF -j TCPMSS --clamp-mss-to-pmtu

or like the one I have been using (seems more logical to me):

iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu


Note that this is not a fix, but a workaround and the real problem is over-zealous admins or weird setups[1] which think that banning TCP fragmentation (or the entire ICMP traffic) is a way to secure networks.


Once again, thanks to everybody who read and/or commented about my issue.

[1] Sesse told me that in his case there was a transparent proxy involved when he exeprienced MTU weirdness.

For a while I’ve been trying to find some easy way to keep a few machines I admin behaving the way I want them too with minimal effort. They don’t really need much maintenance – but I would like something to help keep them all in sync. Basically, something like FAI, but much, much simpler – ideally something that takes five minutes to understand, and another five minutes to deploy; and leave the more complicated and powerful tools for when they’re actually needed. I figured what I really want was just a simple way to make a meta-package – one that doesn’t really provide any functionality, just tells apt/dpkg what I want installed (via Depends), and what I don’t want installed (via Conflicts) and adds any extra configuration stuff or local scripts that I decide I want. But doing that with a real Debian package is harder than I’m really comfortable with – I don’t want to have to worry about potential lintian errors, or rules files and debhelper commands, or writing a Makefile to get my files installed or whatever, I want something more trivial than that. Looking for meta-package creators, the only one I spotted that I thought looked likely was cdd-dev, described as “Custom Debian Distributions common files for developing meta packages”. Unfortunately it seems to just provide templates, which makes things quicker, but no less complex. Fortunately equivs (“Circumvent Debian package dependencies”) is actually used for metapackages these days, according to its maintainer on IRC and its long description:

This package provides a tool to create Debian packages that only contain dependency information. One use for this is to create a metapackage: a package whose sole purpose is to declare dependencies and conflicts on other packages so that these will be automatically installed, upgraded, or removed. Another use is to circumvent dependency checking. […]

That turned out to work much better than I remembered (from whenever I last tried it – back in ‘99 I guess?), with the only drawback being that I couldn’t add files easily. But that’s just a matter of creating a patch to equivs, which I then won’t have to worry about again. So having done that, I can now create a metapackage to do whatever I want by creating a file like:

Suite: client
Section: misc
Priority: standard
Package: ajs-client-stuff
Version: 20071114.1
Maintainer: Anthony Towns <aj@erisian.com.au>
Description: Metapackage for aj's client computers
 Depends on necessary packages, etc.
File: /etc/apt/sources.list.d/client.list
 deb http://mirror.localnet/debian etch main contrib non-free
 deb http://mirror.localnet/debian etch-proposed-updates main contrib non-free
 .
 deb http://security.debian.org/ etch/updates main contrib non-free
File: postinst
 #!/bin/sh -e
 .
 apt-key add - <<EOF
 [output of gpg --armour --export $KEY]
 EOF
 .
 ##DEBHELPER##

debhelper kindly takes care of getting the permissions right for me, and equivs will generate a full source package if I tell it to, which I can just upload to mini-dinstall and have a regular Debian repository just by writing a text file and running equivs-build. And my metapackage can add dependencies, conflicts, apt sources, cronjobs, scripts, configuration files, documentation, or whatever I happen to want – which means I can make it automatically update itself, and thus install any dependencies or remove any conflicts, which then means that modifying the config on all the machines is just a matter of updating the metapackage. And new installs is (hopefully) just a matter of doing a standard install and then adding the metapackage. Perfect. …even if it is really little more than a reinvention of rpm’s .spec files. :)