A couple of hours of today went on dealing with the effects of a big spam attack on a managed server. One user had installed the Joomla ExtCal component on their web site, while another reported that email was running slowly. The attacker was exploiting com_extcalendar/admin_events.php to send spam and trying to send 9000 emails meant they hit the maximum permitted sending rate quite quickly. As soon as I spotted it, I shut down outgoing mail, killed the sending process, disabled the exploited site and component, cleaned the spam from the queue and restarted mail. I'm searching the rest of the server for any other com_extcalendar installations and then I should contact the other exploited servers I've seen during this attack. It looks like there's an entire class of evil "Mambots" out there, using exploitable Mambo and Joomla sites to crawl in search of others. I don't think the exploited site's owner will be billed for the clean-up, because I can't find a security advisory for this. Neither the mambo site (whose site search doesn't work for me) nor the joomla site seems clear about security alerts. I didn't find a third-party advisory that was clearly about this (but there were a few maybe-relevant ones). The homepage at Mambo last released in 2005 and doesn't mention this in its release notes, while the homepage listed by Joomla 404s (and I don't enjoy using the Joomla site because, as it says,

"we are aware, and regret, that our website does not comply with many WCAG/508 requirements"

). If you are using com_extcalendar on your site, check whether it's safe, or replace it with a better calendar.

A few days ago I wrote my first DMCA take-down notice, I followed the instructions on the Wikipedia page. The reason for this was that someone was mirroring my blog and putting google adverts on the copy. Before I started putting Google adverts on my web sites I wouldn’t have been bothered about this. But now that I’m making a small amount of money from Google advertising I don’t want someone else just mirroring my content and taking the money away from me. The person who managed the site in question took a surprisingly large amount of time to comply with the request (a discussion of several messages plus a couple of reminders over the course of a few days). The most recent news about DMCA abuse is the case of trying to prevent the distribution of a code used for decrypting DVD-HD. It is widely believed that copyright was used to prevent the distribution. Strangely many people who otherwise have a good understanding of technology have been saying “you can’t copyright a number”. What precisely is a program binary if not a long series of numbers (or a single large number depending on how you look at it)? For that matter a JPEG file or the ASCII representation of a book is also either a very large number or a series of small numbers. Also apparently it’s not protected under copyright but under the anti-circumvention clause of the DMCA. If it was a matter of copyright it would not be an issue of whether a number can be copyrighted, but what defines such a number. One criteria for copyright is that it has to be on something non-trivial (EG I couldn’t copyright the use of “a few days ago” as an introduction) so length is a criteria. Another is that it has to be a creative expression (so an encryption key can’t be copyright). However in many jurisdictions there are separate laws regarding distributing passwords without permission, such laws are designed for preventing people from granting unauthorised access to computers but I believe that they can be used more generally (I have been advised that such laws exist in the state of Pennsylvania in the US - I’m not sure what the law is in other regions but expect that something so useful would be copied). Another breaking story is that the RIAA has created an organisation with a US government mandate to collect royalties on ALL music that is played over Internet radio. This includes music for which the copyright owner is not an RIAA member and does not consent to have the royalties applied. You can create your own music, grant free access to everyone out of philanthropy, and then have the RIAA tax the music! It’s unfortunate that only the down-side of this dramatic change in copyright law has been discussed. Compulsory licenses have a lot of potential in other areas of copyright material. Recently people have been complaining that government sponsored scientific research is often only published in journals that cost large amounts of money. Why not have a compulsory license for journals at a fair price that everyone can afford? Software is often unreasonably expensive (Windows Vista with the latest version of MS Office can cost up to twice as much as a new PC), let’s have compulsory licenses for software at a reasonable fee! Software vendors often cease selling old versions of software to force customers to upgrade, a compulsory license scheme would permit us to buy MS-DOS 3.30 at a reasonable price regardless of whether MS wants to sell it. Finally there is at least one evil cult that claims it’s “religious” texts are copyright as a way of preventing the public from seeing what a drug-addled second-rate sci-fi author produces. Let’s have a compulsory license for them so everyone can read them! The only thing that’s wrong with the RIAA scheme is that there is no option for copyright owners to directly license their material to the users (including granting a free license if they so desire). The up-side of this is that it proves beyond all doubt that the RIAA is not representing copyright owners. Update: I initially accepted the claims about the DMCA take-down notices being based on copyright rather than anti-circumvention. Since learning of my mistake I modified this post to reflect the fact that it was not a copyright issue.

I already had the idea to get LPI certified - expensive fun. Thanks to Joerg and co. for organizing the cheap exams during Debconf, I already have signed-up for LPI101 in the morning and LPI102 in the afternoon and wrote our $$$-director of the comany a mail that I’d like to get 95 more at the end of this month. Finally a little reference to the blog of Martin F. Krafft - and especially “Why? I am a Debian developer”. I think this is just bullshit, sorry. LPI is dedicated for system administrators, BOFHs, techies, however. Not that stuff what a Debian developer is doing. I mean, a DD should have knowledge about Debian including knowledge about software development, documentation, how to build packages, write some scripts… But does he need to know how to manage iptables or know where you need to set the fontserver? I think no… LPI asks many questions about old stuff, like the XFS, ISA Cards, Analog modems usw. The normal user today doesn’t need to care about this things, even a Debian developer mustn’t. I think many DDs never have touched an ISA card. So, I will participate in the exams, please don’t blame me if I fail.

That mad duck writes about the LPI exams during DebConf and isnt sure if he should do them or not. Well, first a reply (in a blog, yeah, shudder) to one of his points - yes, Klaus had something to do with this LPI offer. He was my initial contact for LPI, getting the UK LPI people in the boat, and then also made the contact to the “Area Operation Manager” who was the one that offered the special pack in response to one of my requests. So yes, he had a lot to do with this. Second, the “What do I need LPI for?”. Good question. It really depends. Do you have a good job? Do you have a good track record with your work and is that enough for your (possible future) boss? Or is it likely you get one of those that don’t look at what you are actually able to do but instead look at some “certification” from $whoever? Or the company you work for does support for $some-other-big-company and those ask for certifications. Those are the cases when it (can) help to do those exams. If one just wants to do it “for my own fun/private stuff”, then I dont think its the right motivation. I personally don’t have any certification and dont think I need one. I only thought about doing such an exam for a minute as I had “the luck” to win 2 LPI vouchers during Cebit, so it would be free for me.

As already announced, we’ve arranged for interested people to take LPI exams during DebConf7. As the announcement says, you need to register for the exams in two places. First go to the LPI system to get an LPI-ID, and then register for this event, so the LPI people know how many attendees to expect, and for which of the 6 possible tests. (Ignore the MySQL ID, no MySQL certifications will be carried out). You will see that there are currently two timeslots to choose from on 19 June, one at 10:00 and one at 14:00. This should be enough for the number of people we expect to show up, but if not we will arrange more slots with the LPI team. As also stated in the announcement, there is a special pack available. If you registered for DebConf7 and also reconfirmed in time, you can take two LPI exams for only 95 EUR, which is less than one exam normally costs. If you haven’t managed to register and reconfirm in time, don’t worry: the LPI exam will still be available at half the normal price. Oh, and someone just asked: you can register for the LPI exams right up to the day they’re held at DebConf. Only the special pack is limited to those who registered and reconfirmed for DebConf in time…

A couple of months ago I was sitting in a north Sydney pub cradling a beer with the ineluctable jdub, the decidedly unparsimonious rusty and (if I’m accurately remembering who’d left and who hadn’t by this point) the irremediable mrd. As you might imagine, these sort of conditions are a perfect breeding ground for a particular type of discussion, and with Jeff’s departure from Canonical not that long beforehand accompanied by the then ongoing DPL elections, talk turned to the future of Debian, and in particular Jeff’s views on that future. Obviously more than a few weeks have passed since then, so please imagine this as a dream sequence with a stylish TV fade-in accompanied by appropriate fine print caveats about paraphrasing and the perfidy of memory. I think it’s fair to sum Jeff’s main point about the future of distros as one of “disintermediation” – that is removing, or at least minimising, the barriers between the authors and users of a piece of software, and in general getting closer and quicker collaboration between everyone involved and interested in developing and maintaining that software. Which means things like trying to remove the line between the Debian bug tracker and upstream bug trackers entirely, so that Debian users end up talking directly with the upstream maintainers, rather than having a Debian maintainer relaying information between the two. It also means making the path for updated packages to get to users quicker and smoother – so that as soon as a developer commits a new version of the software to version control, the next time users of that software “apt-get update” they see it, download it, and start using it. Or at least, they do if they’ve chosen to follow the bleeding edge. Jeff had a bunch more ideas in that vein – like not bothering with a central archive, but having users collect the packages they’re interested in from upstream sites all over the place, standardising, reducing or removing the “control” information for packages so that creating the correct packages for 80% of free software (that uses ./configure in the usual way, eg) was a complete no-brainer, and perhaps viewing distributions not so much as the gatekeepers and central players of the open source world, but perhaps more as systems of providing resources, assistance and opportunities to software authors who deal with their users directly – and in so doing have more of an opportunity for their users to actively participate in the project’s development, and have more feedback (and control) of how their project is perceived by its users. I’m not sure if I’m doing it justice – it’s a pretty radical notion, and as an upstream-type hacker, Rusty was pretty enthralled by it. As a distro-type hacker, on the other hand, I’d have to say I was daunted: getting rid of the middleman here means getting rid of Debian, which I must admit I have something of an attachment for… and yet, in principle at least, I can’t see anything but good sense in the idea – getting users and developers interacting more closely is important to improving the free software community, and while distros have historically been a force for making free software more accessible to people, it’s entirely possible we can now make it even more accessible now by, in some sense, doing away with distros. And if that’s the case, then we definitely should, because it’ll benefit both users and developers of free software, which is what we’re all about. On the other hand, distributions (at least currently) aren’t just about collecting and compiling software which could be automated; it’s also about integrating it, standardising interactions between software, and in some cases acting as a buffer between users who want to use a bit of software and developers who for one reason or another don’t actually want to deal with those users (perhaps due to lack of a shared language or political or philosophical disagreements). I think it’s also pretty interesting to view Gentoo through this light; that is, not so much as a distro that makes you compile everything from source, but rather as a distro that gives its users a system based on exactly what the developers produce, with the addition of only a comparably tiny amount of Gentoo-derived content. Maybe it’s just me, but Gentoo sounds a lot more interesting to me in that light. So is that a foretaste of the future of binary free software distributions too? There’s a lot of things that would stand in the way of Debian managing that, but so far all I can think of is a thousand problems that should probably be solved anyway, and not even one that can’t. I think, btw, that jdub had been reading Accelerando (also available as a Creative Commons licensed ebook) somewhat concurrently with thinking about this stuff, so perhaps it should be called “Distributions 2.0” or something. Accelerando’s a crazy-awesome near-future/singularity book by Charles Stross, who happens to live in the city that will host the forthcoming DebConf 7…

I just looked on server statistics and I was shocked by amount of spam and viruses rejected during last three days. Is there some new virus for Windows which tries random email addresses on all servers it can reach? I didn't analyse those messages, as they are rejected at RCPT time, because sender tries random addresses which do not work on my server. For now I try to reduce them by blacklisting in iptables, but this doesn't seem to help much as IP addresses change too often. I think that graph shows best what big change has happened: Update: It seems to be new Sober variant causing this mess.

I love email newsletters that move slickly from ranting to advertising. Here's one from Gossypium:

"We believe that something is critically wrong with all those head in the sand neoclassical giant companies, that have been built on economical models that we would now consider antique and even dangerous for the human evolution. New business models like Gossypium have managed to rebalance the relationships between the economy, society and the environment. It is now accepted by all that we will need to change and that there is no turning back. We as a brand are trying to make this easier by offering positive alternatives. So here is our great T-shirt collection"

Source I just wish I could do that in such an amusing way. But I can't, so I'm currently shuffling a few things around to make my Business feed a bit more work-relevant. If you are subscribed to my Business-only feed for the Media items, please switch to the Hacks feed. Meanwhile, Webmastering articles will change from the Hacks feed to the Business one. And there's some more changes which I'm still working out. Watch this space.

Recovering a corrupted FAT partition A friend has seen his FAT partition in his USB key become unmountable. Here's how I recovered it. Analysis:

1. Make a dump: dd if=/dev/sdb1 of=partition.img
2. Have a look. I like mc for this, as it has a nice hex viewer and editor (F3 to view, then F4 for hex view; F2 would then allow hex edit, if needed).
3. Turns out that the initial part of the partition has been completely overwritten by 0xff bytes. After some point (around block 224), the 0xff bytes end and there seems to be a FAT structure (easy to identify because it contains sequences of increasing 16bit values).

It seems that the damage is only limited to filesystem structures. I should be able to recover most data by regenerating the filesystem data and hoping that at least the second FAT is still intact. Solution:

# Make a copy of the partition
cp partition.img partition1.img
# Reformat it: that gives us the bits of filesystem structures we need
mkdosfs partition1.img
# Graft together the two parts
dd if=partition1.img of=partition2.img count=224
dd if=partition.img of=partition2.img seek=224 skip=224
# Tidy up the result, telling dosfsck to use the 2nd FAT.
dosfsck -rw partition2.img
# Mount and check what's available
mount -o loop,ro partition2.img /mnt

Yo! Everything was there again. When I teach about standard Unix commands, it takes a while for people to realise all the powerful things you can easily do with them.

In a comment on my blog post “a Heartbeat developer comments on my blog post” Alan Robertson writes:

---

I got in a hurry on my math because of the emergency. So, there are even more assumptions (errors?) than I documented. In particular, the probability model I gave was for a particular node to fail. So the probability of either of two failing would be double that, and either of three failing would be triple that. Note that the probability of multiple simultaneous failures goes up as a power, but the probability of either of only goes up linearly. I really need to sit down and do the math carefully - but the idea of the simultaneous failures going up as a power is true. And the “any of” probability goes up linearly. That’s also true. This is why people can actually use larger HA clusters ;-). The 5 years figure is the industry standard quoted figure for an average Intel-based server to fail. The four hours to repair is a common high-quality of service response time from a hardware vendor. I admit that’s not the same as actual repair time, but if some “repairs” are just reboots, then it’s not a horrible number to start with - if your vendor has cached some spares nearby. I suppose I should sit down and do the math right, and make a spreadsheet of it. (I wonder if I remember that much math?) I assume disk failures are taken care of by hot swap disks, RAID, etc. and so in effect they “never fail” (at least not totally) so that these failures don’t have to be accounted for by the overall availability model. Here’s an intuitive way of thinking about it “from your gut”… If I took your whole data center and made a cluster out of it, what’s the chance that at least half of your servers would fail at once? Pretty darn small, is the short answer ;-). If it’s not pretty darn small, you need to buy better servers, and IBM has just the servers for you ;-). Or maybe they need to hire a better SysAdmin ;-) If you ask yourself “when is the last time at least half my machines in my data center couldn’t communicate with the other half”, then hopefully that’s also a “pretty darn small” chance too. If not, there are well-known methods for making networks highly reliable too. [I’m still ignoring “catastrophes” that you haven’t accounted for in your HA architecture]. I’m not saying this is free, and it can be pricey. One of my other favorite sayings is “Paranoia is an expensive hobby”. How much do you want to spend? You tell me how much you want to spend, and you can figure out how to spend it. I’ll make a separate comment on quorum models later. It’s getting late here.

---

My only comment in response to that is to say that I still believe the calculations of probability to be correct in my original post and I am interested to see someone prove otherwise. Another comment by Alan:

---

Data corruption, no doubt, is almost always much worse than loss of availability. And some kinds of data corruption are worse than others. For example, mounting a non-clustered shared disk filesystem twice simultaneously is usually much worse, than updating two replicas of the data simultaneously. In the first case, you have to restore to your previous backups and lose all data since then. In the second case, you only lose updates that were made to one of the sides, and you instantly have a working copy of the data which is nearly always much newer than your last backup (with the possibility of recovering them by significant effort). Typically you would only lose a few minutes of updates at worst - and depending on the kind of networking failure, you might not lose anything. Heartbeats certainly aren’t enough. You need to monitor the health of your servers and the health of your applications. Heartbeat monitors applications and can easily be informed of and act on the health of your servers (with release 2 style Linux-HA Heartbeat configurations).

---

Followed by:

---

Since data corruption is so serious, this is why cluster designers worry so much about split-brain, which is managed using the ideas of quorum and it’s sibling fencing. This is all about keeping bad things from happening. This post is really about quorum, since Russ had expressed interest in it. Quorum is the idea that you can uniquely choose a subcluster to represent the whole cluster in those cases where communication failure has caused the cluster to split into separate sub-clusters which cannot properly communicate with each other. In this way, only one of the subclusters continues on, and the others will sit on their hands and do nothing waiting for a person to fix things. Some of the kinds of quorum mentioned below are better than others. But, most importantly, they can be used in combination as described later. The most common kind of quorum is that Russ mentioned in his earlier post - the majority quorum. In this method, for a cluster of n nodes, you grant quorum to a sub-cluster which has more than INT(n/2) members. This means that if you have a 3-node cluster, you have to have two nodes to continue. If you have 4 nodes, you have to have 3 nodes to continue. For 5 nodes, you have to have 3 nodes, and so on. Other basic methods include disk reserve, so that you have reserve a disk to have quorum. In this case, if only one node survives and it can reserve the disk, it continues to run. However, the disk becomes a single point of failure. This may not be a problem if this single disk is required to run any of the cluster services, since they would fail without it anyway. [Heartbeat does not support this method]. An analagous method is to implement a software resource which grants quorum to one subcluster in a fashion analagous to the disk reserve method. This has the advantage of not requiring disk reserves, or a shared disk, but it has the same SPOF disadvantage as the disk reserve method. Heartbeat does support this method using the quorum daemon. It’s incredibly useful for those cases (like split-site clusters) where you cannot use fencing. Another method is to grant quorum to any subcluster which can ping a certain set of nodes, and not grant it to any which can’t access those nodes. This isn’t a wonderful method, and has obvious disadvantages with respect to uniqueness, and single points of failure. (Heartbeat doesn’t yet implement this one). Another method is to grant quorum to any node which is a member of a 2-node cluster. This is better than losing quorum and stopping when one node stops, but obviously completely ignores the uniqueness requirement of quorum. Another method is to ask a human being if you have quorum. This is hardly an ideal circumstance, but useful in some contexts as described below. (Heartbeat doesn’t yet implement this one). Perhaps you say, really the only one of these that’s really good is the first one - the majority vote method. And, I would generally agree with you. But, Heartbeat has the ability to use these in combination which makes some of those methods that seem flaky to be much more reasonable. Heartbeat has the ability to have multiple quorum modules declared, and they’re used in this way: Any module can return HAVEQUORUM, NOQUORUM, or TIE. If they return HAVEQUORUM or NOQUORUM, then no further quorum modules are consulted. However, if they return TIE, then the next quorum module is consulted for its opinion. If the last quorum module returns TIE, it is treated the same as NOQUORUM. This enables you to use one quorum module to break the tie declared by a previous quorum module. You could then use the quorumd to break the tie created by a voting module. Or you could use the quorumd instead of the “two-node” module. Or you could use the “pingable” module instead of the “two-node” module. Or you could at the end always tack on a “human” module, in case all else returns TIE. This is kind of cool, actually. My favorites for next implementation are the pingable and consult human modules. And, of course, if your cluster loses quorum due to real server failures failures, there are always ways to work around it, with a little human intervention. One method is to tell Heartbeat to ignore quorum. Another is to tell Heartbeat to remove certain nodes from the cluster, after you verify that they’re really dead. And, I’m sure that in a pinch, some new methods will be invented. And some of them might actually work ;-).

---

Regarding the quorumd, it seems that this is an extra server that will generally run on another machine separate from the rest of the cluster. So if we had a two-node cluster with a quorumd then it would effectively be a three-node cluster where one node is not configured to run any resources. It seems that the simpler approach in many cases would be to merely have a three-node cluster with resources not configured to run on one of the nodes. For example if I was running a mail server cluster for an ISP I might configure a three node cluster of the two mail server back-end machines and one other machine that is lightly loaded (EG a DNS server) and have it configured not to run MTA resoures on the DNS machine.

Turbogears form quirk I had a great idea:

@validate(model_form)
@error_handler()
@expose(template='kid:myproject.templates.new')
def new(self, id, tg_errors=None, **kw):
    """Create new records in model"""
    if tg_errors:
        # Ask until there is still something missing
        return dict(record = defaults, form = model_form)
    else:
        # We have everything: save it
        i = Item(**kw)
        flash("Item was successfully created.")
        raise redirect("../show/%d" % i.id)

It was perfect: one simple method, simple error handling, nice helpful messages all around. Except, check boxes and select fields would not get the default values while all other fields would. After two hours searching and cursing and tracing things into widget code, I found this bit in InputWidget.adjust_value:

# there are some input fields that when nothing is checked/selected
# instead of sending a nice name="" are totally missing from
# input_values, this little workaround let's us manage them nicely
# without interfering with other types of fields, we need this to
# keep track of their empty status otherwise if the form is going to be
# redisplayed for some errors they end up to use their defaults values
# instead of being empty since FE doesn't validate a failing Schema.
# posterity note: this is also why we need if_missing=None in
# validators.Schema, see ticket #696.

So, what is happening here is that since check boxes and option fields don't have a nice behaviour when unselected, turbogears has to work around it. So in order to detect the difference between "I selected 'None'" and "I didn't select anything", it reasons that if the input has been validated, then the user has made some selections, so it defaults to "The user selected 'None'". If the input has not been validated, then we're showing the form for the first time, then a missing value means "Use the default provided". Since I was doing the validation all the time, this meant that Checkboxes and Select fields would never use the default values. Hence, if you use those fields then you necessarily need two different controller methods, one to present the form and one to save it:

@expose(template='kid:myproject.templates.new')
def new(self, id, **kw):
    """Create new records in model"""
    return dict(record = defaults(), form = model_form)
@validate(model_form)
@error_handler(new)
@expose()
def savenew(self, id, **kw):
    """Create new records in model"""
    i = Item(**kw)
    flash("Item was successfully created.")
    raise redirect("../show/%d"%i.id)

If someone else stumbles on the same problem, I hope they'll find this post and they won't have to spend another two awful hours tracking it down again.

GREAT!

This is the word that can describe what I felt today at the moto exhibition that takes place here, in Bucharest (until the 1st of May).

Hondas, Yamahas, Aprilias, Harleys, Suzukis, BMWs... all kinds of bikes, speeders, choppers, cruisers, naked...

Of course, I couldn't resist and I touched, sat on, admired and analysed them all that I could (the Honda and the Ducatti people were the most uptight about this subject).

I was with a friend and he made pictures, but I will have them only on Monday...


And all of these after I met my father this morning before he left for a ride up to Vama Veche (a known rebels' meeting point at the Black Sea). I met him and gave him a pair of chaps for the ride with the beautiful Virago he bought a month ago!


I just need to pass over the formalities, take the driving license exam for A category (motorcycles are in that category, here in Romania; I already have a B category, for cars) and get on with riding myself... on the driver's seat...


Today I felt alive...



P.S.: I would be really cool to go to DebConf7 with the bike and meet some people, but I don't think that is legal for me to drive outside my country in the first year after taking the driving license (although I don't know yet if that would apply to me, having already a B driving license). More than that, the long ride alone I would have to make until I meet with others seems a little risky (you never know what the trip has to offer).

Here are a number of links to various postings and other information related to my research about release management in free software projects:

• Bruce Byfield wrote an article about my research for Linux.com.
• I recently gave a presentation about my research at Google as part of their Tech Talks series. A video of this talk is now available.
• MORITA Hajime kindly translated my blog postings about my research to Japanese.
• Matt Asay briefly discussed my work in his blog and relates it to commercial and proprietary software development. Roberto Galoppini relates my findings to his work and concludes that hierarchy is a must. Murray Cumming summarizes what he believes are the key findings of my research.

I doubt I am the first one to wonder how a huge internet based company like amazon.(de at) can manage to be unable to send un-broken e-mails, however I just cannot resist. The regular plain text e-mails about orders (Bestellung erhalten/Bestellung versandt) are broken in a simple way, they are declared as Content-Type: text/plain; charset=ascii while actually containing Windows-1252 characters (Umlauts, Euro-character). I usually fix these in mutt with <Ctrl>-e. This week I returned an incorrectly labeled product back to Amazon and in return received a surprisingly broken mail, header and footer as UTF-8 while the core of the mail is Windows-1252 again. Looks like this in mutt

Guten Tag!
Wir grüßen Sie ganz herzlich und informieren  ine Erstattung zu der
Betreff genannten Bestellung.
Unsere Zahlungsabteilung hat Ihnen EUR 12,29 \374berwiesen.

etc...

As mentioned previously in this space, I've been thinking about what exactly to work on, and one of the conclusions that I've come to is that Stuff To Do is a very handy tool, but there are already too many similar applications out there. Even if they might not have quite the mix of features that makes me like Stuff To Do and keep me using it, it's not the best space for me to be competing in. So, with that in mind, I dropped the price to 6 USD a month, and opened it up so that anyone can use the group enabled version for free as long as they don't mind a few adsense ads. Paying the money gets you an ad free version and a warm fuzzy feeling. We'll see how things go with those parameters, and of course I reserve the right to change things in the future. Depending on how things go, I suppose I might even consider open sourcing it in the future, although since it does what I need, and I don't want to be responsible for trying to manage it as an open source project, I'm not sure, yet. Now... on to the next thing...

Back in mid-2000, I set up my first Internet bank account. I have been using the same bank ever since. The whole notion of Internet banks has changed a lot since then, since virtually every bank offers varying degrees of online access these days. Lately I checked on banks to see if I was still using the best one. Here are my reviews of the two banks.

Internet Banks

Let's first talk about what makes a bank an "Internet bank" or "online bank". So many banks let you manage your account online now, that the distinction is blurry. But generally, I would say that unique features of Internet banks are:

• Significantly higher interest rates than traditional banks
• Ability to open an account online or by mail, and perform all banking functions without visiting the bank in person
• Free bill payment services
• Free self-initiated electronic transfers from your other bank accounts
• Immediate, real-time posting of transactions
• Your own bank never imposes a fee for using an ATM card at another bank's ATM
• Web access to account info, history, canceled check images, and statements
• Toll-free number for account basics (balance, transfer between your accounts at the bank, etc) over the phone
• Great telephone customer support
• An absence of outlandish fees

There are a few to highlight.

Real-time transaction posting is a great thing to have. If you go to the website and transfer money from your savings to your checking account, you can go to the ATM *now* and withdraw it, and when you get back to your desk, the ATM transaction shows up. There are none of these confusing "business day" rules with cutoff times that traditional banks seem to love. And none of this annoying "process all withdrawals first" business that large traditional banks (hello, Bank of America) love to do in order to screw with customers.

High interest rates. Don't even bother with an online bank if the savings interest rate is less than 3% (or checking less than 1%, in my opinion). It is almost impossible to find a traditional bank with these rates.

Note: if you're reading this after April, 2007, these guidelines may no longer apply due to interest rate changes.

Electronic transfers from other banks. Another great feature. A number of banks let you set up, and then initiate, electronic transfers to and from your accounts at other banks. This is usually free if you are using the feature to transfer money into your account, and with a flat per-transaction fee (usually about the same as a foreign ATM fee) for transferring money out.

Ability to do all business without visiting a branch. Not only should this be possible, but routine business (such as making deposits) should be easy to do without visiting a branch. You should be able to order new checks online.

Now let's look at the individual banks.

First Internet Bank of Indiana

FirstIB (they provide nationwide service, despite the name) is the bank I signed up with in 2000. I had found a bank in Dallas that I really liked, but when I moved to Indianapolis, I tried two and liked neither. They all seemed to have really poor interest rates, offer few useful services, and fees for so many conditions that it was hard to avoid them. (More on that in the traditional banks section below)

I looked at the online banking scene at the time, and wound up opening a checking and savings account with FirstIB. FirstIB started as an online-only bank, but recently acquired a bricks-and-mortar bank as well.

I found FirstIB's customer service at the time to be top-notch; I would dial their toll-free number, select *one* option from a menu, and a rep would answer *immediately*. (Take that, Bank of America, with your 45-minute hold times...) Since FirstIB was based in Indianapolis, I actually visited their offices a couple of times. They had a locked lobby. You picked up a phone, pressed 0, and went into the same queue that callers to the 800 number did, then just asked them to come to the lobby and transact your business. (They would not give out cash there, but could take new account paperwork and deposits and the like)

Their interest rates have consistently outperformed traditional banks, and they offer all the online banking amenities listed above. They did not offer online statements at first, but these days online statements are the norm (there is a fee if you want paper statements). Check ordering is accomplished using a linking with the Deluxe check printers. The web interface uses Digital Insight, and the ACH transfer interface uses CashEdge.

FirstIB will actually reimburse you, at the end of the month, for ATM fees that other banks charge you, up to $6/mo. They also provide an unlimited supply of free postage-paid deposit envelopes, and have a PDF of deposit slips that you can print out should you run out.

FirstIB's customer service has slipped in recent times. I know of some people that have been impacted by some errors that FirstIB has made (simply not processing paperwork that they should have processed). Wait times have gone up for the phone line, and a lot of the great "small bank" feel (where they actually knew their customers, and you might actually know the reps) is gone.

Presidential Bank, FSB

Presidential Bank is the bank I'm looking at now. I was initially drawn to them by their extraordinarly high interest rates (5.25% for savings!), but some other things have convinced me to use them.

Initially, I was suspicious. Presidential is a bricks-and-mortar operation that has been around for awhile, and was one of the very first Internet banks. Their website says that they started offering Internet accounts in 1995, and very much looks like it hasn't been updated since. (Thankfully, the account management interface is more modern). I almost thought the site was a fraud up front, a "too good to be true" deal with the high interest rates and bad-looking website. But after doing some research, realized that it is quite real.

Before setting up accounts with them, I had a few questions (such as whether or not transactions post in real time). Presidential has a simple email address for people that want to send in questions (unlike FirstIB's obvious queue system). I sent an email, and got a response an hour or two later, from a real person with a first and last name, and a personal e-mail address I could use to follow up. Nice. I also had occasion to talk to people on the phone. They were polite, speedy, and helpful.

So I set up my accounts with them. I activated the online access, bill payment, and money transfer features. All worked as expected. I did have a couple more occasions to call them after the account was set up (a more crucial test than the pre-account calls), and they remained helpful. Strangely, this bank, which seems to be a much larger operation than FirstIB, had a more small bank feel. Several times, as I started talking to a rep, he or she would say something like "Oh, I remember processing the setup paperwork on this account last week" or "I think I talked to you about this before?"

Customer service is one of those things you rarely need after the first month or so of an account. But with your bank, it is vital that the reps know what they are doing and are willing to help you when you call -- which Presidential is doing better than FirstIB these days.

Presidential does not rebate ATM fees like FirstIB does. They do provide a few deposit envelopes with your initial account setup packet, but they are not postage paid or unlimited. However, because Presidential's interest rates are so much better than FirstIB's, it doesn't take much in your account to more than balance out that difference. Plus, with so many retail stores offering "cash back" when you use an ATM or debit card, it isn't too bad to get cash for free with Presidential. (And, most employers offer direct deposit, so we rarely use deposit-by-mail) One other trick is to find some local bank and open a no-minimum free checking account with them. You can deposit checks there and withdraw cash from their ATM, and use Presidential's free incoming ACH to electronically transfer the funds to Presidential when you get too much in the local account. (Chances are that these local accounts will not pay any interest)

Presidential offers a few features FirstIB does not. First, they scan all your deposits (not just your canceled checks) and offer to let you view them online. Secondly, they automatically send you an email when they've received a bank-by-mail deposit (you can opt out of that, of course). Third, they let you view *all* of your old statements online (FirstIB only lets you view the last 6 months worth, though both have the complete history available in the web interface).

Like FirstIB, Presidential uses DigitalInsight for the web interface and CashEdge for the ACH feature.

Internet Bank Comparison Both banks offer two primary checking and two primary savings accounts. This table compares all four accounts.

Feature	Checking				Savings
	FirstIB		Presidential		FirstIB		Presidential
Account	Interest	Free	Internet Plus	Internet	Money Mkt	Reg	Premier	Internet
APY	1.26%	0%	4.50%	1.25%	3.90%	2.75%	5.25%	1.50-2.50%
Min Balance	$500	$0	$1000	$500	$4000	$1000	$0	$100
Min Method	Avg Daily		Daily		Avg Daily		Daily
Min to Open	$100	$25	$1500	$500	$100	$100	$5000	$100
Online Bill Pay	$0	$4.95/mo	$0	$5.95/mo (up to 10 payments)	Not available
ATM surchage rebate	$6	$0	$0		$0
Online Canceled Check Images	Free				N/A
Online Deposit Images	No		Free		No		Free
Email Deposit Notifications	No		Free		No		Free
Email Balance Alerts	Free
ACH policy	Free when initiated at other end; free for deposits when initiated at the bank's website; small fee for withdrawals when initiated at bank's website

Online options at traditional banks

As I mentioned, Presidential is a traditional bank that has offered Internet accounts for some time now. Some other traditional banks do that as well, but look carefully at them, especially if they're a big nationwide bank.

I looked at options from Bank of America, Citibank, HSBC USA, ETrade Bank, and some other large national banks. In general, these were just regular bank accounts with a web interface (HSBC and ETrade being somewhat of exceptions). Read the fine print and you'll find poor interest rates and traditional banking hassles (lack of immediacy in transactions, predatory posting practices, fees for just about everything).

Plus I can tell from personal experience that Bank of America customer service is downright awful. One example: they used to pressure me to open a credit card with them every time I visited a branch. I declined every time, and had to be forceful about it sometimes. Then one day, not long after a visit with a particularly annoying employee, I got a Bank of America credit card -- which I had specifically refused -- in the mail. Hmmmmmm.

They also, unlike any other bank I've ever used, charged a fee to receive wire transfers. Which they had not disclosed. Then, of course, they apply withdrawals before deposits, and caused an overdraft from the undisclosed fee, even though the wire transfer was a *deposit* that would have very easily covered it. That was the last straw for me. After nearly an hour on the phone with them, I got them to agree to reverse both the fee and the overdraft charge. THEN I said I wanted to close my account.

When traditional banks are better

There are a few situations where Internet banks don't work out so well:

Living paycheck to paycheck. If you are constantly running out of money, and it is vital that any deposits get posted *NOW*, you probably don't want an Internet bank (and wouldn't benefit from their higher interest anyway). When you mail in a check, it can take some time before you have access to the money; the postal service has to deliver it, and then most of your checks will be subject to the "non-local" availability policy and held for about a week (though interest will start to accrue immediately).

Not being able to meet minimum balances. Internet banks do tend to offer a variety of accounts, but if you aren't able to regularly maintain $500 to $1000 in your checking account or $2000 to $5000 in your savings, you will likely fall into a category with lower interest and fewer free benefits. These accounts may not have many benefits over traditional banks, and may in fact be worse in some ways. The interest rate gap won't make that much of a difference either. There are several online banks that offer savings accounts only, with high interest rates and low minimums, that you may want to investigate -- though you will have availability into your checking account measured in days, not seconds. Presidential's savings has no minimum, but requires $5000 to open. (They do have lower-interest savings accounts without that requirement) Watch out for FirstIB's accounts; you can open an account with less than the monthly minimum balance, and will run into fees unless you deposit enough to meet the minimums.