Dual Boot Citrix Xenserver 5

I'm very proud of this hack. So proud that it's actually a quad boot, but two of the four are Windows, which we don't care about, so we'll stick with the title of dual boot. Here's the scenario:

A couple weeks ago, I did the VCP-310 cert, with the anticipation of rolling that into the VCP-410 next month. Until then, I'm going to do the Citrix CCA cert. This means we need a Citrix Xenserver... Unfortunately, the only machine I have that is not doing something is my oldest kid's desktop. Now, this is not as bad as it sounds, as he's off doing a summer session at some university somewhere, or something.

Since he's gone, we'll wipe the drive and load Citrix. Not so fast! He's got Windows on that thing and uses it as a game station when he's in town. I'd already partitioned it and loaded it with boot to Fedora as well as Windows, so Citrix should be a non-issue. Except, Citrix Xenserver thinks it should be the only thing on the system... Just like ESX.

So, here we go:
1. Disconnect the hard drive and load a second hard drive, pinned out as master.
2. Load Citrix Xenserver in the default configuration. Reboot and test.
3. Disconnect second hard drive and pin out as slave. Cable it and the original drive to the primary controller.
4. Boot to BIOS and ensure both are visible. This ensures that both are pinned correctly.
5. Boot off the previously existing Linux partition to Single. Use fdisk -l to validate that the second drive is visible.

(This is where it starts to get hairy.)

6. The master appears as /dev/sda and the secondary as /dev/sdb. Mount the second drive:

mount /dev/sdb1 /mnt
ls -l /mnt

Notice anything unusual? No boot partition. Citrix is loading the MBR with a LILO like loader.
7. Inspect /boot:

chain.c32
config-2.6.18-128.1.6.el5.xs5.5.0.496.1012kdump
config-2.6.18-128.1.6.el5.xs5.5.0.496.1012xen
extlinux.conf
extlinux.sys
initrd-2.6.18-128.1.6.el5.xs5.5.0.496.1012kdump.img
initrd-2.6.18-128.1.6.el5.xs5.5.0.496.1012xen.img
initrd-2.6-xen.img
mboot.c32
menu.c32
System.map-2.6.18-128.1.6.el5.xs5.5.0.496.1012kdump
System.map-2.6.18-128.1.6.el5.xs5.5.0.496.1012xen
vmlinuz-2.6.18-128.1.6.el5.xs5.5.0.496.1012kdump
vmlinuz-2.6.18-128.1.6.el5.xs5.5.0.496.1012xen
vmlinuz-2.6-xen
vmlinuz-kdump
xen-3.3.1.gz
xen-3.3.1.map
xen-3.3.gz
xen-3.gz
xen.gz

Some of this looks familiar. We just need to arrange it into our existing grub.conf.
8. I looked at an old FC6 Xen platform's grub.conf:

title Fedora Core (2.6.20-1.3002.fc6xen)
    root (hd0,0)
    kernel /xen.gz-2.6.20-1.3002.fc6
    module /vmlinuz-2.6.20-1.3002.fc6xen ro root=/dev/hda3
    module /initrd-2.6.20-1.3002.fc6xen.img

(Why in the world am I still running Fedora Core 6? Because, it is most similar to RHEL5.)
9. A closer look at /boot will confirm that many of those files are actually symlinks. So lets add the following to grub.conf:

title Citrix Xenserver 5.5.0
    root (hd1,0)
    kernel /xen-3.3.1.gz
    module /vmlinuz-2.6-xen ro root=/dev/sdb1
    module /initrd-2.6-xen.img

Notice the root line-- hd1,0 not hd0,0-- we're on the second drive. Notice the vmlinuz line-- we are using /dev/sdb1-- that's what we mounted.
10. Reboot... and watch it fail!

I'll save you the agony of troubleshooting. First, in the Fedora example, boot is partition 1, so we prefix the files with a slash (/). Citrix did not use a boot partition: root is partition 1. Therefore, we need to prefix the files with the directory: /boot/

Second, we mounted /dev/sdb1, which was the Xenserver's root partition. Yet, when Xenserver boots, it sees the drive as /dev/hdb1. This is because I happen to have Fedora 8 loaded, and effective F8, all drives are sd, even though the drives are actually IDE rather than SCSI. Since Xenserver is based on el5 (look at the vmlinuz links) it sees the drive as hd.

Here's what actually worked:

title Citrix Xenserver 5.5.0
    root (hd1,0)
    kernel /boot/xen-3.3.1.gz
    module /boot/vmlinuz-2.6-xen ro root=/dev/hdb1
    module /boot/initrd-2.6-xen.img