A week or two ago, one of the hard disks in
rock utterly
broke down.
In itself, this wasn't a problem, since the disk was part of a 3x40G
RAID5 set. Moreover, I knew it was dying, and was preparing to buy me a
new disk anyway; the fact that it broke down only sped up matters. No
worries.
However, there's one slight problem with rock: it's a pretty old SMP
box, and because of some bug somewhere (which I still need to hunt
down), it can't run anything later than 2.6.12 in SMP mode.
The new disk was a 80G disk, to replace a 40G disk. I wasn't very
happy about the RAID5 array, but since I needed more disk space than
what a RAID1 or RAID10 array on three 40G partitions would give me, I
didn't have much choice. One of the disks in the RAID5 set was a 80G,
too, however, with a 40G partition dedicated to the RAID5 set; so rather
than doing ugly things with two disks only halfway committed to the
RAID1, I decided to convert the RAID5 set to a RAID1 one. When you use
LVM-on-RAID, as I do, that's pretty easy: you just create the new array
(in degraded mode, on the new disk); add it to the volume group; use
pvmove to move all data to the new array; use
vgreduce
to remove the old array from the volume group; and finally, repurpose
the freed-up space to make sure the RAID1 set is no longer degraded.
Which is all nice,
except that on 2.6.12, when
pvmove locks the entire volume group, you can't write to it
anymore, which is quite painful if things like
/var and
/ are on that volume group; gives you a nice deadlock.
So I compiled me a 2.6.19.2 kernel with SMP support explicitly
disabled, in order to be able to do the
pvmove dance
without much issues. That worked. I only forgot to do one rather
important thing: to regenerate the initrd after all data had been moved;
mainly because it took quite a while for the RAID1 resync to finish, and
I had other things to do by then.
Yesterday, we had a power failure, and obviously
rock failed
to come up again.
By doing some utterly ugly things with the sarge d-i and forcing the
yaird-generated initrd into a debug shell, I was able to boot
the box properly again, and had it running 2.6.12. Which took all of
three hours, or so. By then, it was kinda late, so I went to bed.
This morning, my brother woke me to say he couldn't reach the
Internet. Since
rock is my gateway, I got suspicious. Running
dmesg showed me a number of warnings from the kernel, where it
said that it didn't properly understand some of the hashing stuff in
some directory inodes. As a result, several files had been
corrupted—including
/var/lib/dpkg/status and
/var/lib/dpkg/status-old. The file with 'extra' information
that
aptitude maintains, however,
/var/lib/aptitude/pkgstates, still existed and was readable. So
I downloaded me a one-week-old Packages file from snapshot.debian.net,
and wrote this perl script:
#!/usr/bin/perl
open(APTITUDE,"</var/lib/aptitude/pkgstates");
open(STATUS,">status");
open(PKGS,"<Packages");
my $pkgent;
my $aptent;
my %aptitude;
my $i;
my $key;
my $val;
while(<APTITUDE>)
if(/^$/)
$aptitude "Package:" . $aptent-> "Package" . "\n" =$aptent;
$aptent = ;
else
chomp;
($key,$val) = split /:/;
$aptent-> $key = $val;
close APTITUDE;
while(<PKGS>)
if(/^$/)
$aptent = $aptitude $pkgent->[0] ;
if($aptent-> "State" == " 1")
print STATUS $pkgent->[0];
print STATUS "Status: install ok installed\n";
for($i=1;$i<$# $pkgent ;$i++)
if(!($pkgent->[$i] =~ /^(Filename Size MD5sum)/))
print STATUS $pkgent->[$i];
print STATUS "\n";
$pkgent = [];
else
push @ $pkgent , $_;
close PKGS;
close STATUS;
... which generated a somewhat valid status file, that I copied to
its right place. Yeah, that's an ugly hack, but at least I don't have to
reinstall
rock now. Phew.
NP: Kino / Picture