Daniel Stone: fd.o affected by portland state uni power failure
As ajax quite
elegantly summed up, due to a series of catastrophic power failures
at PSU, where fd.o is hosted, we were down for a good chunk of
yesterday. Despite the machines having redundant power supplies, being
connected to separate power rails in the rack, which were hooked up to
independent, UPS-backed, power supplies, we still (like a good chunk of
Portland, and certainly everyone in the PSU machine room) lost our
power.
As far as we can tell, when annarchy.fd.o (websites, people.fd.o, cgit, anongit, et al) came back up, power was again interrupted while the ext3 journal was being replayed. When it came up the n'th time, fsck dumped almost the entire filesystem in lost+found, then started saying increasingly unhappy things about the state of the filesystem on its second pass. In the end, we just went with mkfs, and now we have a brand new and shiny filesystem.
It's worth pointing out that even if this was another filesystem, such as /srv, which hosts all project data, we would've been fine, as they're all backed up. But, unfortunately for some, we made a decision a while ago to not back /home up, and didn't advertise that as widely as we should have. So, if you had stuff in annarchy:/home, it's now gone, and I hope you have backups.
Sorry about that. On the upside, I got to see PSU's new and really very nice machine room this morning, thanks to XDC being about 250m away from the PSU machine room, and fd.o is otherwise running fine. We've been talking this week about replacing our ageing hardware, which would also allow for more redundancy as well as better performance from those machines. But we still have no plans to back up /home, so if you put stuff there, please, please keep your own backups (or make sure the Wayback Machine knows about it).
Due to some things falling into place, I had an opportunity to attend the
first two days of last week's
The German online magazine
Some time ago I wrote about the
For some time ago I ve been pondering about this question. As long as GSoC 2009 is about to start and people are looking for project ideas, I m posting here a very preliminar draft of my findings and an idea for a posible software project. It s pretty written on-the-fly while I ve managed to have some minutes between work and uni. Rigurous wording isn t P1.