
So. I m trying to get familiar with libvirt and friends. To this end, I ve set up a Lucid virtual machine booting from PXE into an initrd environment which does a pivot_root to an AoE block device.
The #virt channel on irc.oftc.net told me that in order to have libvirt provide PXE capability, I would have to install a recent version of libvirt. I built version 0.7.5-3 from sid on my karmic laptop and it seems to be working okay.
I decided to set up the pxe root directoy in /var/lib/tftproot just because that s what the
example code had in it.
I had to manually configure a virtual network. Here is the XML config file:
$ sudo virsh net-dumpxml netboot
<network>
<name>netboot</name>
<uuid>81ff0d90-c91e-6742-64da-4a736edb9a9b</uuid>
<forward mode='nat'/>
<bridge name='virbr1' stp='off' delay='1' />
<domain name='example.com'/>
<ip address='192.168.123.1' netmask='255.255.255.0'>
<tftp root='/var/lib/tftproot' />
<dhcp>
<range start='192.168.123.2' end='192.168.123.254' />
<bootp file='pxelinux.0' />
</dhcp>
</ip>
</network>
This, of course, depends on the pxelinux.0 file. Luckily, this is packaged up in syslinux and can be installed with a simple
$ sudo apt-get install syslinux
$ sudo mkdir /var/lib/tftproot
$ sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftproot
I had to create a pxelinux config file for the virtual machine (indexed by mac address). Note that I put a console=ttyS0,115200 argument on the kernel command line so that I can attach to the serial port from the host system for copy/paste debugging. Also of importance is the root=/dev/etherd/e0.1p1 argument, specifying which block device we ll be doing the pivot_root to eventually.
$ mkdir /var/lib/tftproot/pxelinux.cfg/
$ cat /var/lib/tftproot/pxelinux.cfg/01-52-54-00-44-34-67
DEFAULT linux
LABEL linux
SAY Now booting the kernel from PXELINUX...
KERNEL vmlinuz-lucid0
APPEND ro root=/dev/etherd/e0.1p1 console=ttyS0,115200 initrd=initrd.img-lucid0
I decided to use the karmic kernel for lucid initially. I ll eventually switch over to the lucid kernel ;)
$ sudo cp /boot/vmlinuz-2.6.31-17-generic /var/lib/tftproot/vmlinuz-lucid0
I copied /etc/initramfs-tools to ~/tmp/lucid so that I didn t mess up the system initrd scripts:
$ mkdir -p ~/tmp/lucid && cp -r /etc/initramfs-tools ~/tmp/lucid/
Since mkinitramfs doesn t currently have a system for AoE root, I had to do a bit of fiddling. I copied the NFS root boot script and made a couple of modifications.
$ diff -u /usr/share/initramfs-tools/scripts/nfs ~/tmp/lucid/initramfs-tools/scripts/aoe
--- /usr/share/initramfs-tools/scripts/nfs 2008-06-23 23:10:21.000000000 -0700
+++ /home/cjac/tmp/lucid/initramfs-tools/scripts/aoe 2010-01-15 14:56:28.098298027 -0800
@@ -5,59 +5,25 @@
retry_nr=0
# parse nfs bootargs and mount nfs
-do_nfsmount()
+do_aoemount()
-
configure_networking
- # get nfs root from dhcp
- if [ "x$ NFSROOT " = "xauto" ]; then
- # check if server ip is part of dhcp root-path
- if [ "$ ROOTPATH#*: " = "$ ROOTPATH " ]; then
- NFSROOT=$ ROOTSERVER :$ ROOTPATH
- else
- NFSROOT=$ ROOTPATH
- fi
-
- # nfsroot=[<server-ip>:]<root-dir>[,<nfs-options>]
- elif [ -n "$ NFSROOT " ]; then
- # nfs options are an optional arg
- if [ "$ NFSROOT#*, " != "$ NFSROOT " ]; then
- NFSOPTS="-o $ NFSROOT#*, "
- fi
- NFSROOT=$ NFSROOT%%,*
- if [ "$ NFSROOT#*: " = "$NFSROOT" ]; then
- NFSROOT=$ ROOTSERVER :$ NFSROOT
- fi
- fi
+ ip link set up dev eth0
- if [ -z "$ NFSOPTS " ]; then
- NFSOPTS="-o retrans=10"
- fi
+ ls /dev/etherd/
- [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-premount"
- run_scripts /scripts/nfs-premount
- [ "$quiet" != "y" ] && log_end_msg
+ echo > /dev/etherd/discover
- if [ $ readonly = y ]; then
- roflag="-o ro"
- else
- roflag="-o rw"
- fi
+ ls /dev/etherd/
- nfsmount -o nolock $ roflag $ NFSOPTS $ NFSROOT $ rootmnt
+ mount $ ROOT $ rootmnt
-# NFS root mounting
+# AoE root mounting
mountroot()
- [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-top"
- run_scripts /scripts/nfs-top
- [ "$quiet" != "y" ] && log_end_msg
-
- modprobe nfs
- # For DHCP
- modprobe af_packet
+ modprobe aoe
# Default delay is around 180s
# FIXME: add usplash_write info
@@ -67,17 +33,13 @@
delay=$ ROOTDELAY
fi
- # loop until nfsmount succeds
+ # loop until aoemount succeds
while [ $ retry_nr -lt $ delay ] && [ ! -e $ rootmnt $ init ]; do
[ $ retry_nr -gt 0 ] && \
- [ "$quiet" != "y" ] && log_begin_msg "Retrying nfs mount"
- do_nfsmount
+ [ "$quiet" != "y" ] && log_begin_msg "Retrying AoE mount"
+ do_aoemount
retry_nr=$(( $ retry_nr + 1 ))
[ ! -e $ rootmnt $ init ] && /bin/sleep 1
[ $ retry_nr -gt 0 ] && [ "$quiet" != "y" ] && log_end_msg
done
-
- [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-bottom"
- run_scripts /scripts/nfs-bottom
- [ "$quiet" != "y" ] && log_end_msg
(below is the full file in case udiff is less convenient)
$ cat ~/tmp/lucid/initramfs-tools/scripts/aoe
# NFS filesystem mounting -*- shell-script -*-
# FIXME This needs error checking
retry_nr=0
# parse nfs bootargs and mount nfs
do_aoemount()
configure_networking
ip link set up dev eth0
ls /dev/etherd/
echo > /dev/etherd/discover
ls /dev/etherd/
mount $ ROOT $ rootmnt
# AoE root mounting
mountroot()
modprobe aoe
# Default delay is around 180s
# FIXME: add usplash_write info
if [ -z "$ ROOTDELAY " ]; then
delay=180
else
delay=$ ROOTDELAY
fi
# loop until aoemount succeds
while [ $ retry_nr -lt $ delay ] && [ ! -e $ rootmnt $ init ]; do
[ $ retry_nr -gt 0 ] && \
[ "$quiet" != "y" ] && log_begin_msg "Retrying AoE mount"
do_aoemount
retry_nr=$(( $ retry_nr + 1 ))
[ ! -e $ rootmnt $ init ] && /bin/sleep 1
[ $ retry_nr -gt 0 ] && [ "$quiet" != "y" ] && log_end_msg
done
There was also a small modification to the initramfs.conf file:
$ diff -u /etc/initramfs-tools/initramfs.conf ~/tmp/lucid/initramfs-tools/initramfs.conf
--- /etc/initramfs-tools/initramfs.conf 2008-07-08 18:37:42.000000000 -0700
+++ /home/cjac/tmp/lucid/initramfs-tools/initramfs.conf 2010-01-15 14:33:38.088295207 -0800
@@ -47,14 +47,16 @@
#
#
-# BOOT: [ local nfs ]
+# BOOT: [ local nfs aoe]
#
# local - Boot off of local media (harddrive, USB stick).
#
# nfs - Boot using an NFS drive as the root of the drive.
#
+# aoe - Boot using an AoE drive as the root of the drive.
+#
-BOOT=local
+BOOT=aoe
#
# DEVICE: ...
I also needed to add aoe to the list of modules included in the initramfs:
$ echo aoe >> ~/tmp/lucid/initramfs-tools/modules
In order to generate the initrd.img file from this new config, I ran the following:
$ sudo mkinitramfs -d ~/tmp/lucid/initramfs-tools/ -o /var/lib/tftproot/initrd.img-lucid0
I created a lucid VM by installing from the desktop install disk. You can grab the ISO here:
http://cdimage.ubuntu.com/daily-live/current/
I ll leave the creation of the virtual machine and installation as an exercise for the reader. I put the filesystem on an lvm volume group called vg0 in a logical volume called lucid0 (ie, /dev/vg0/lucid0).
At this point, I created a new virtual machine called lucid0. Here is the xml for the domain:
$ sudo virsh dumpxml lucid0
<domain type='kvm' id='1'>
<name>lucid0</name>
<uuid>96fbad21-4f25-5700-ddd8-1a565c7170ee</uuid>
<memory>524288</memory>
<currentMemory>524288</currentMemory>
<vcpu>1</vcpu>
<os>
<type arch='x86_64' machine='pc-0.11'>hvm</type>
<boot dev='network'/>
</os>
<features>
<pae/>
</features>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<interface type='network'>
<mac address='52:54:00:44:34:67'/>
<source network='netboot'/>
<target dev='vnet0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/4'/>
<target port='0'/>
</serial>
<console type='pty' tty='/dev/pts/4'>
<source path='/dev/pts/4'/>
<target port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1' keymap='en-us'/>
<sound model='es1370'/>
<video>
<model type='cirrus' vram='9216' heads='1'/>
</video>
</devices>
</domain>
Now we re ready to start the AoE target and launch the virtual machine. If you don t have vblade installed, do so now:
$ sudo apt-get install vblade
Start the target up with the following command:
$ sudo vbladed 0 1 virbr1 /dev/vg0/lucid0
Now, if all goes well, you should be able to watch the virtual machine boot up and do its thing like so:
$ sudo virsh start lucid0 && sudo screen -S lucid0 sudo virsh ttyconsole lucid0 115200
If you get errors about /dev/etherd/e0.1p1 not existing (these might look like this):
Begin: Retrying AoE mount ...
err discover interfaces revalidate flush
err discover interfaces revalidate flush
mount: mounting /dev/etherd/e0.1p1 on /root failed: No such file or directory
Done.
then you might want to try restarting vbladed like this:
$ sudo kill -9 ps auwx grep vblade grep -v grep awk ' print $2 ' && sudo vbladed 0 1 virbr1 /dev/vg0/lucid0
So. Now you should have a lucid gdm in your virt-manager console. Any questions? #virt on irc.oftc.net
<script>//<script language="javascript" src="http://reddit.com/button.js?t=1"></script>