I'm so annoyed with myself.
So today was the day I was going to upgrade the disks in
daedalus.
I'd finally managed to convince Dell to sell me two new 140Gb disks to
replace to two 70Gb disks that were fast approaching capacity. I'd organised
for access to the datacentre from 10am. I arrived early.
I didn't manage to get into the datacentre until about 11am, because they
were having problems remotely unlocking the door, and I had to wait until an
air conditioning repair guy turned up, who was fixing some air conditioning
problem.
I hadn't had a lot of time to plan how I was going to do the upgrade, but I
figured it was going to be pretty straightforward. Each disk is partitioned
into three partitions, all of which are mirrored using software RAID. The
first partition is the root filesystem, which is just an ext3 filesystem,
mirrored. The second partition is swap, also mirrored, and the third
partition is an LVM physical volume. The first two partitions are only a
couple of gigabytes.
So the basic plan was to remove the second disk (disk 1, aka
/dev/sdb), put in the new bigger disk, load the partition table
from the original disk 1, resync the three mirrors, and then boot off the
new disk 1, and rinse and repeat with disk 0.
This was all going swimmingly, when right at the end (in fact after the
third partition finished syncing according to the kernel),
/dev/sda
decided it had a few unreadable sectors. This made the software mirroring
have a dummy-spit, and decide it needed to restart the syncing from scratch.
I'd seen this before with a bad disk.
So I wishfully hoped it was something to do with the new disk being bigger
than the old disk, so tried to roll back to the original disk 1. As was
pretty much expected, disk 0 exhibited the same problems at the same point.
So the problem was disk 0, not the new disk 1.
So it seemed like I just managed to pick the wrong half of the mirror to
work from.
I thought I'd try manually dd'ing the third partition from
/dev/sda
to
/dev/sdb, ignoring the errors, but that didn't result in a
usable half of a mirror either.
Interestingly, these bad sectors hadn't made themselves known until this
point. The RAID array had been otherwise healthy, and SMART hadn't uttered a
peep.
I think at this point, after spending about 3 hours fighting with it, I
decided to move up a layer. I could successfully sync
/dev/md0 (the
root filesystem) and
/dev/md1 (the swap device), and the only
reason I could think of these bad sectors not giving me grief already, was
because they were right at the end of the disk, and I hadn't quite used all
the free physical extents in LVM yet. Lucky I hadn't decided to grow things
any further.
So I decided that rather than trying to convince this existing
/dev/md2 to sync with the new
/dev/sdb3, I'd just build a
new degraded RAID-1 on
/dev/sdb3 and move all the physical extents
from the degraded RAID-1 on
/dev/sda3 to the new degraded RAID-1 on
/dev/sdb3. I tried the logical volume with
/tmp on it
first, since it was fairly sacrificial. It worked fine. So I tried the
largest logical volume,
/srv, which was 30Gb. That went fine as
well, so I moved the rest. All of them went without complaint. So then I
removed the now unused
/dev/md2 from the volume group, and finally
managed to boot with just the new disk 1. Then I was able to put in the new
disk 0, and just do a standard RAID-1 rebuild onto it. Hooray for LVM saving
the day yet again.
And this is where I should have stopped.
At this point it was about 3pm. I'd been aiming to leave by 4pm to pick up a
rental car that I had to get by 5pm. I'd wanted to also upgrade to Etch
while I was physically in front of the box, in case there were any nasty
kernel/udev issues that made the box unbootable. The catch was, I needed to
grow
/var, which was quite full.
The intention had been that once I'd got the two new disks in and all synced
up, I'd create a fourth mirrored partition, and add this to LVM as a new
physical volume, then I could just keep merrily growing things. It felt a
little bit dirty having to make a new partition to get at the extra space,
and I'd reading about
mdadm's "grow" mode, whilst I was sitting
around waiting for things to sync.
So I thought to myself, I'd just delete the third partition, and recreate it
using the remaining additional cylinders I now had at my disposal, then use
this grow feature of
mdadm to tell Linux the array was now bigger,
then resize the physical volume, and go from there.
This is where things went a bit pear-shaped.
I deleted and recreated (larger) the third partition on each disk, and
rebooted.
I was then messing around with
mdadm trying to update
/etc/mdadm/mdadm.conf, since the UUID of
/dev/md2 had
changed, when I discovered that
/dev/md2 had completely
disappeared.
It seems that recreating that partition made Linux completely fail to see
/dev/sda3 and
/dev/sdb3 as RAID-1 members, and so because
of the fairly transparent nature of Linux's RAID-1, LVM had seen the
underlying physical volumes, and just decided to run with
/dev/sda3
as the physical volume, and everything had continued to work. I just had no
redundancy any more.
So it was about 3:20pm at this stage, and I was starting to panic, thinking
I'd never get this sorted out before I had to leave, and I'd end up
returning to the US with a non-redundant mess on my hands. Then I remembered
that I had the backups of the partition tables of the old disks, from before
I removed them. So I thought I'd go back to that state, and just have the
additional cylinders unallocated. I hadn't yet attempted to grow the RAID-1
(partly because it had vanished), so in my haste, I didn't see this being a
problem.
The problem was, I'd booted into multi-user mode, and things had merrily
been operating on just half the mirror. I restored the partitions, and
rebooted, and lo and behold,
/dev/md2 magically reappeared, already
100% consistent according to
/proc/mdstat. I was somewhat surprised
by that, but I booted back into multi-user mode again, only to discover that
the
/var filesystem was panicking and remounting itself into
read-only mode. This was pretty weird, given that during boot, it was
considered clean by
e2fsck.
So I was really getting bothered by this stage, so I rebooted into emergency
mode (single-user mode in Debian starts way too much), and manually started
up LVM, and forcibly fscked everything. There was a bit of filesystem
damage to
/var,
/home and
/srv.
Thinking I'd now fixed the problem, I rebooted into multi-user mode again,
but again
/var, which was considered clean during the
fscking/mounting phase of booting, panicked shortly after mounting, and
remounted itself read-only.
So I rebooted again, broke the mirror, and forcibly fscked the filesystems
again (only operating on what was on
/dev/sda3. Then I rebooted
without incident into multi-user mode and resynced the mirror.
At this point I ran away.
So I'm really annoyed with myself. This should have been a blog post about
how crap Linux's software RAID is at error recovery, and how LVM saved the
day yet again, but it was overshadowed by me overreaching in my haste, and
managing to cause silent data corruption.
The morals of this story:
- If things go off the rails, and you get to a reasonable milestone, and
you're approaching the end of your maintenance window, quit while you're
ahead.
- Don't do stuff in multi-user mode, just because you can and you want to
minimise the length and magnitude of an outage
- Don't rush to try and do something (growing a RAID-1) without
researching it properly
- Linux's software RAID-1 sucks when the underlying devices are having
issues (I already knew this)
- Linux's software RAID-1 can look perfectly healthy when the underlying
disk has issues, as long as the bad bits aren't getting any exercise (I
think newer versions of mdadmand/or Linux address this by doing a
full array check on a regular basis)
- Linux's software RAID-1 can get out of sync with reality and report that
it's in-sync with itself when it isn't really, which causes silent data
corruption
- e2fsck can report a filesystem is clean when it's really got issues
I think that is all. I'm just so pissed off that what could have been a
perfect (enough) upgrade with zero data-loss, even considering I had a bum
disk, wasn't.
Thursday evening, I joined the inhabitants of the twin town of Valga-Valka, sitting smack on the Estonian-Latvian border, to celebrate their entry into the Schengen treaty. Ever since I first crossed the border there on a roadtrip from Tallinn to Fallingbostel, in year 2001, I knew I would have to return and, sure enough, I briefly passed through during last summer with an Estonian friend, on our way to an acquaintance's birthday party.
Still, that told me nothing of the town's life and left me wanting for more.
Upon hearing that year 2004 EU accession countries would join Schengen in December 2007, I immediately promised myself to show up and join the crowd. As it turns out, I missed Aleks Tapin of the All About Latvia blog by very little, having I caught his last-minute e-mail the next afternoon. Aleks blogged a great article depicting the Latvian side of life and providing some background info on the town, if you're curious.
I arrived late-evening on Thursday via the Tallinn-Viljandi-Valga bus and checked into my hotel, then proceeded to checkpoint Valga II around 23:30, with the intention of grabbing a drink on the Latvian side and returning just before midnight for the celebrations. Hardly anyone was in sight, except for Latvian officer abana who was completing the inspection of a Russian car with three noisy passengers. Upon presenting her with my passport, officer abana cheerfully lead me to the office and slid my passport through a slot to officer Bukss, who was visibly surprised to have any tourist show up on the last day of his job to get their passport stamped. Upon explaining to him the reasons for my presence, he pointed me to a nearby bar where I could have a drink, visibly amused by the situation.
I spent the next few minutes in a local casino in the company of three hilarious Latvian truck drivers, Zintars and Aivars, two Latvians with limited English language skills and Yuri, an ethnic Russian living in Ireland who was born stateless on the Latvian side of the town and who later acquired Estonian citizenship by claiming ancestral land on the Estonian side.
Upon returning to the border, just minutes before midnight, I found myself in the middle of a huge crowd of villagers, police officers, border guards and politicians from both countries - barely getting noticed by anyone. I handed my passport to an Estonian border guard who emotionally commented to a civilian friend of his nearby that, "Wow! That was the last one!", handing me my passport back just as the midnight bells rang and the mayor of the Estonian side started his speech amidst pyrotechnics lining the road.
I then had a chance to chat with the Estonian mayor, who promptly handed me a bilingual certificate, signed by the mayors of both towns, attesting that I was the first to cross the open border, while introducing me to his two young daughters with whom he was about to take a stroll on the Latvian side.
The next morning, an even bigger and more symbolic event took place: the demolition of the fence that had been cutting the S prus/Rai a street in half. You can see a picture of what the street looked like before on Aleks' blog article [edit: or on Jens-Olaf's blog article]. After the fence was removed, it instead looked like this:
The crowd then proceeded to checkpoint Valga III, where a stage had been setup for the politicians to make their speeches. An interesting fact is that, because of the way the border was drawn, along a creek leading to Pedeli river, an Estonian main road was technically on the Latvian border and thus aptly named "Raja" (border) street. The Estonian mayor commented in his speech that the the brand new supermarket standing behind us was also technically on borderland and could not have possibly been built earlier, weren't it for the Schengen treaty matter-of-factedly eliminating borders between participating EU countries. The speeches were closed by a youth group whose choreography featured six break-dancing boys, dressed as Estonian and Latvian border guards and as border posts. The choreography ended with the teenage border guards carrying away the human posts on their shoulders, just as a choir of young girls replaced them with songs in either languages.
Before catching my bus back to Tallinn, I dropped by the Valga tourist info, only to face a nervous-looking Kapo officer in plain clothes. Upon mumbling something about the tourist info and wanting to grab maps, I noticed that a press conference was taking place behind. As he finally let me pass to the tourist info side of the building, a familiar voice said to me, "Hey! Weren't you at our place last summer with Martin Ranna?" Yup, the wife of the acquaintance whose birthday party I had attended last summer is working there and greeted with me refreshments and munchies she had cooked in prevision for the presidential visit! As was my luck, she had one of the commemorative plaques that had been given to the politicians at Valga III on hand:
I was glad to see the town becoming one again, even though it is ethnically divided. There are already signs of people on both sides learning one another's languages and shopping on either side of town, not to mention plans of merging the municipal bus operations of the Estonian and Latvian halves of the town, so I'm sure that they'll get there in due course.
For me, Schengen brought a much more practical and quite welcome change: the end of messy border crossing stamps that were rapidly filling my almost new passport, every time I visited the head office of our company in Tallinn. While Estonian border guards mostly stamp passports in an orderly fashion, being careful to fit exactly 8 stamps in a single page and to put entry and exit stamps side-by-side, Finnish border guards apparently are incapable of doing so, instead systematically wasting pages by either stamping sloppily in a way that makes it impossible for more than 4 stamps to appear on a given page or by flipping to a brand new page, every time I had to cross the border.
In case anyone ever wondered why I'm currently representing Estonian interests abroad, despite living in Finland, this example is one of the many reasons: Finnish authorities routinely display arrogant carelessness towards the population and doubly so towards immigrants. It amounts to an accumulation of idiocy that has costly consequences on people's lives. For some, it's about being denied the public institutions' support when they need it the most and their life forever going downhill thereafter. For me, it has been about countless missed opportunities (jobs, love affairs, travel plans), plus very costly passport and residence permit renewals. Given this, I simply don't see myself ever representing Finnish interests ever again. Faith no more. Besides, the Estonians are fun and easy-going people. Siski ma m tlen, kas ma peaksen n d Eestisse kolima v i?
After a very long and stressing day which seems to never end I can't find a better way to feel better than blogging about it.
Today I was cold-called by someone announcing themselves as "UK
Bank Claim", a service of "the leading company", asking me questions
about my bank charges.
Now, that put my back up right away. I co-own
most of my banking service providers (aren't
cooperatives and mutuals great?), and
you can't get much more confusing service and
company names if you tried.
Then they claimed to be registered with the
Ministry of Justice, but I can find no mention of this registration
service on
Following Lucas[
This kind of smoothing actually eliminates high frequencies, hence we see what a usual behavior each step has. What is clear to me is that FD situation is clearly fixed for quite a long time. FD was definitely an issue one year ago, but it has been really really fixed: for more than a year, even on the 6-months smoothing graph that has quite a long memory, the average queue is around 3 people, I can't see how it could even be reasonably improved.
Of course, given that the Keyring step is just after the DAM, and that people at those steps works in batches, the data is a bit biased, though it gives a pretty good idea of the bottlenecks of the whole process.
Today I went for the second time[1] to 
… or something like that… I planned to write a short note about how to start with using git-svn so I can provide a pointer to some of my colleagues. It turned out that git has too many nice features that you should be aware of. 
Hopefully my notes (now being a reference for myself as well, thx to 
Before takedown request:
As I wrote 
A curiously interesting thread has sprung up on the Xorg mailing list recently regarding the lack of bandwidth the developers might have for accepting patches. Bernardo Innocenti, who hacks on OLPC stuff amongst other things, wants to get some patches reviewed for acceptance, but there aren't enough people around with the time and inclination to review and accept his work.
He did a quick and dirty analysis of Xorg LoC vs Linux LoC, to give a "X codebase is roughly 1/2 Linux codebase" (which surprised me, actually, I'd always thought it was larger) and then a similarly rough analysis that suggests that Xorg has roughly 1/20th of the developer base.
To some extent Bernardo's metrics are arguable, but the basic facts are that Linux gets the lions share of the developers. Is this what we should expect? Is this a healthy way for the free and open source software community to support the development of what I personally would like to see as the future desktop base?
Linux does a fantastic job as an OS, right now. It's been doing it for years, really, but we still need to get behind some of the differential further up the stack. And all credit to those stalwarts of X development who are shouldering more than their fair share of the burden.
So if you were looking for an open source software project where you can be appreciated, don't overlook X.