Martin F. Krafft: Using LVM snapshots for filesystem recovery
This morning, I found myself in Gunnar's position and was looking
at an empty /home directory because of the following mount failure:
piper:~# mount /dev/mapper/vg0-home mount: Unknown error 990 Starting XFS recovery on filesystem: dm-0 (logdev: internal) Filesystem "dm-0": xfs_inode_recover: Bad inode magic number, dino ptr = 0xffff810028a6bd00, dino bp = 0xffff81002851d140, ino = 33629 Filesystem "dm-0": XFS internal error xlog_recover_do_inode_trans(1) at line 2352 of file fs/xfs/xfs_log_recover.c. Caller 0xffffffff88307729I am used to XFS giving me hard times, but this time at least I didn't have much to worry as it happened at home, I have plenty other machines I can use instead, and all data was backed up. Since this is on a Debian unstable machine, I had the latest xfs_repair available, and told it to do its thing, but it refused:
piper:~# xfs_repair /dev/mapper/vg0-home
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
Great. Catch-22.
I decided to have a go at fixing it, but as much as I pretended to grok
xfs_db, I did not manage to get the filesystem into a consistent state. I
thus fired off an email to the mailing list (which has not arrived there 4
hours later) and started pv to dump the raw data to another machine for
later use or debugging.
Watching the progress meter of pv from time to time, it occured to me that
my /home is in fact on LVM. I put it on LVM way back when LVM
was new and cool and I wanted to gather experience with it. I've never
actually done anything with it, but today it saved my little bits of remaining
sanity.
Why? Because LVM can take snapshots of volumes. It took me a while to put
the pieces together, but in the end, a snapshot would enable me at least to
run a test-round of xfs_repair -L and assess the damage, and if it wasn't
all that bad, I could just run the repair process on the actual volume again.
With the encouragement and knowledge of Christian Aichinger and Daniel J.
Priem on IRC, I hence proceeded to give it a try. First I had to extend the
volume group to make room for the snapshot data, and since I had no disks
handy, I just used a loop device based on a sparse file:
dd if=/dev/zero of=/srv/loop bs=1 count=1 seek=5368709120 losetup -f /srv/loop pvcreate /dev/loop0 vgextend vg0 /dev/loop0 lvcreate -s -n home-snapshot /dev/vg0/home -L 5G(Thanks to Ian Nicholls for spotting the error: I was using pvcreate instead of that last lvcreate previously). Now /dev/vg0/home-snapshot was a copy-on-write snapshot of the original volume, and I could do whatever I wanted with it, as long as I did not exceed the 5G size allotted for the copied data blocks that I modified. Using mount and xfs_repair, I could verify that the problem also shows up with the snapshot. Thus, I ran xfs_repair -L on the snapshot and finally was able to mount the snapshot to inspect the /lost+found directory. Fortunately, it wasn't all that full, nor was there anything important in there, so I sighed, removed the snapshot (lvremove) and unleashed xfs_repair -L on the original volume, which is now again mounted under /home. Finally, remove the loop device from the volume group (vgreduce). If you are like me and forgot to do so before rebooting and now find your LVM volumes not starting anymore, look into the --removemissing option. I like this method. I knew the filesystem was corrupted, but I did not know to what extend. Had corruption been worse, I could have waited for the XFS guys to help me out. But seeing the low damage, I was able to decide to just purge the journal and accept the damage. If this had happened on a more important machine (i.e. without recent backups), I would have proceeded a bit differently: once the snapshot was mounted, I'd have run xfs_dump on it to save the data. I would also have used more space for the snapshot and not removed it until the parent volume was repaired and mounted. It's unlikely that xfs_repair will produce different results when run on a volume and its snapshot, but better be safe than sorry. All this said, I just want to point out that the above method is filesystem independent and hence would work for any filesystem out there. I've never been a big fan of LVM, especially without RAID to build on (striping across physical media multiplies your chance of data loss), but this method is a good reason for using LVM regardless: use a volume group instead of your physical disk, and logical volumes instead of partitions, but don't span disks (unless you have to). Then, if stuff goes bad, you can hook in another disk, extend the volume group with it, and work on snapshots.
other
icons for different user types). I also probably should not hotlink to the .gifs on the livejournal servers.