This morning, I found myself in
Gunnar's position and was looking
at an empty
/home directory because of the following
mount failure:
piper:~# mount /dev/mapper/vg0-home
mount: Unknown error 990
Starting XFS recovery on filesystem: dm-0 (logdev: internal)
Filesystem "dm-0": xfs_inode_recover: Bad inode magic number, dino ptr =
0xffff810028a6bd00, dino bp = 0xffff81002851d140, ino = 33629
Filesystem "dm-0": XFS internal error xlog_recover_do_inode_trans(1) at
line 2352 of file fs/xfs/xfs_log_recover.c. Caller 0xffffffff88307729
I am
used to XFS giving me hard times, but this time at
least I didn't have much to worry as it happened at home, I have plenty other
machines I can use instead, and all data was backed up.
Since this is on a Debian
unstable machine, I had the latest
xfs_repair available, and told it to do its thing, but it refused:
piper:~# xfs_repair /dev/mapper/vg0-home
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
Great.
Catch-22.
I decided to have a go at fixing it, but as much as I pretended to grok
xfs_db, I did not manage to get the filesystem into a consistent state. I
thus fired off an email to the mailing list (which has not arrived there 4
hours later) and started
pv to dump the raw data to another machine for
later use or debugging.
Watching the progress meter of
pv from time to time, it occured to me that
my
/home is in fact on
LVM. I put it on
LVM way back when
LVM
was new and cool and I wanted to gather experience with it. I've never
actually done anything with it, but today it saved my little bits of remaining
sanity.
Why? Because
LVM can take snapshots of volumes. It took me a while to put
the pieces together, but in the end, a snapshot would enable me at least to
run a test-round of
xfs_repair -L and assess the damage, and if it wasn't
all that bad, I could just run the repair process on the actual volume again.
With the encouragement and knowledge of Christian Aichinger and Daniel J.
Priem on IRC, I hence proceeded to give it a try. First I had to extend the
volume group to make room for the snapshot data, and since I had no disks
handy, I just used a loop device based on a sparse file:
dd if=/dev/zero of=/srv/loop bs=1 count=1 seek=5368709120
losetup -f /srv/loop
pvcreate /dev/loop0
vgextend vg0 /dev/loop0
lvcreate -s -n home-snapshot /dev/vg0/home -L 5G
(Thanks to Ian Nicholls for spotting the error: I was using
pvcreate
instead of that last
lvcreate previously).
Now /dev/vg0/home-snapshot was a copy-on-write snapshot of the original volume,
and I could do whatever I wanted with it, as long as I did not exceed the 5G
size allotted for the copied data blocks that I modified. Using
mount and
xfs_repair, I could verify that the problem also shows up with the
snapshot.
Thus, I ran
xfs_repair -L on the snapshot and finally was able to mount
the snapshot to inspect the
/lost+found directory. Fortunately, it wasn't
all that full, nor was there anything important in there, so I sighed, removed
the snapshot (
lvremove) and unleashed
xfs_repair -L on the original
volume, which is now again mounted under
/home.
Finally, remove the loop device from the volume group (
vgreduce). If you
are like me and forgot to do so before rebooting and now find your LVM volumes
not starting anymore, look into the
--removemissing option.
I like this method. I knew the filesystem was corrupted, but I did not know to
what extend. Had corruption been worse, I could have waited for the
XFS
guys to help me out. But seeing the low damage, I was able to decide to just
purge the journal and accept the damage.
If this had happened on a more important machine (i.e. without recent
backups), I would have proceeded a bit differently: once the snapshot was
mounted, I'd have run
xfs_dump on it to save the data. I would also have
used more space for the snapshot and not removed it until the parent volume
was repaired and mounted. It's unlikely that
xfs_repair will produce
different results when run on a volume and its snapshot, but better be safe
than sorry.
All this said, I just want to point out that the above method is filesystem
independent and hence would work for any filesystem out there. I've never been
a big fan of
LVM, especially without
RAID to build on (striping across
physical media multiplies your chance of data loss), but this method is a good
reason for using
LVM regardless: use a volume group instead of your
physical disk, and logical volumes instead of partitions, but don't span disks
(unless you have to). Then, if stuff goes bad, you can hook in another disk,
extend the volume group with it, and work on snapshots.