Citrix Xenserver: Apply Multiple Updates

As a result of reorganizing the servers in my lab, I had to reinstall Citrix Xenserver.  I should have downloaded 6.1, but decided to keep it at 6.0 and apply the updates that I already had on the NAS.  All went well with the install, I moved all my VMs and templates to this machine, and retasked the other machine.

When I went to load the updates, a funny thing happened... It refused to load more than one, and expected to reboot after each.  After a moment of thought, I realized that I had probably never tried to load two at a time before.  It seems like something that should be simple, but the procedure is not obvious.

Here's how:

1. Highlight a server, click the "General" tab, and expand the "Updates" pane.
2. In the "Updates" pane, notice which updates have already been applied.
3. On the menu bar, click "Tools", "Install Software Update", and "Next".
4. Click the "Add" button, select the lowest numbered update, and click "Open".
5. At this point, its tempting to add another update, but don't: yet.
6. Click "Next", select one or more servers, and click "Next".
7. A check will be run against the update.

If the check succeeds, click "Previous", "Previous", and repeat from step 4.

If the check fails, then two things.  First, click "Cancel" and start the entire procedure over again, but don't add the update that failed the test.  Second, don't blame me-- I didn't create the interface.

Once you've added all the relevant updates, click "Next".  You'll have the choice of performing post install steps automatically or manually.  What this really means is reboot now or later.  If you select manually (reboot later,) it is possible that some of the updates will fail, but that's actually okay.  When an update succeeds, it appears in the "Updates" pane as Applied.  If it fails, it appears as Not applied.

To get activate the not applied updates, repeat steps 1, 2, and 3, but instead of step 4, highlight the not applied update.  Continue through the rest of the steps, making sure to do automatic, as recommended.

VM Autostart on XenServer

Its a good thing I don't need to run VMware, because they are so dependent on a pre-existing Microsoft infrastructure, that I couldn't run it, even if I tried. And I've tried. Of course, I should just go ahead and invest the $3,000 in Microsoft software... just so I can invest a $1,000 in VMware software. Or, I could use Citrix XenServer.

Unfortunately, Citrix is doing everything in their power to ruin their entry level product, based on the philosophy that if they strip enough useful features from their product, eventually people will have no choice but to buy it. I don't know... If I'm going to throw two grand at them, I might as well up the ante and buy VMware.

Or just hack their product. I mean, come on guys... are you even trying?

So, I've got a cluster of XenServers, and I want to start a VM with the host boots. Ah! Upgrade XenServer to enable! Or add the following to the /etc/rc.d/rc.local for all hosts:

xe vm-start name-lable=YourVmName

If the host is the first to boot, it starts the VM. If the host is not the first, it attempts the command, gets a failure message (becuase the VM is in the wrong power state... on), and boots as normal.

And now we give thanks to the command line Gods.

Linux Rescue for VM on XenServer

I finally figured out how to rescue a Linux VM running on a Citrix XenServer host, an believe it or not, it is completely unintuitive! First, power off the VM, though if you're going to rescue mode... you're probably "down" already. Second, mount the rescue media (CD/DVD). Third, make sure the VM is highlighted on the left pane of XenCenter.

Here's the trick: Across the top of the XenCenter menu, select VM, and Start/Shut down, and Start in Recovery Mode. The machine should boot from the rescue media. Proceed per rescue SOP.

Be forewarned... For some reason, booting is incredibly slow.

Unlocking Citrix Xensever Memory

Wow... Has it been that long since I posted? Yeah, its been a wild few weeks, what with hurricanes, earthquakes, hail storms, trying to sell a piece of underwater real-estate at a 30% loss. Whew! But hey, here's a Xenserver hack for you:

I loaded Xenserver 6 Beta on a cluster of servers, and was disappointed to find that they had moved memory management out of the "free" product and into one of the "pay" tiers. This means that if you want to change an VMs memory reallocation, you have to pay an extra licensing fee. Silly. Especially since it was so easy to bypass.

Create a template from a VM. Log into the Xenserver console via SSH as root. Using the template name that appears in XenCenter:

# xe template-list name-label=a-Windows_Vista_x86-x2
uuid ( RO) : 4c<snip>28
name-label ( RW): a-Windows_Vista_x86-x2
name-description ( RW): SP2, Registered

What we need is the UID. View the template parameters:

# xe template-param-list uuid=4c<snip>28

Here, look for the min/max lines:

memory-static-max ( RW): 1073741824
memory-dynamic-max ( RW): 1073741824
memory-dynamic-min ( RW): 1073741824
memory-static-min ( RW): 1073741824

I this case, the template will create a VM can only be 1G... never more... never less.

Lets change the bottom value:

# xe template-param-set uuid=4c<snip>28 memory-static-min=1
# xe template-param-list uuid=4c<snip>28 | grep " mem.*-m"
memory-static-max ( RW): 1073741824
memory-dynamic-max ( RW): 1073741824
memory-dynamic-min ( RW): 1073741824
memory-static-min ( RW): 1

I haven't figured out the other three values, as any attempt to change them throws an error saying they must all be equal. They do form the top limit, but once you strip out all that Aeroglass crap and disable half the services, Vista runs just fine at less than 512Mb. But just for the record, I wouldn't suggest you try to run Vista on 1 byte of memory.

MSI 880GM MainBoard... Oops: E41

I bought a box of computer guts from Newegg, to upgrade my Linux virtualization host. This box has not been doing anything recently, because it was still running Fedora 8. I know F8 is soooo old, but the machine is only a PIII with 512Mb. Since it didn't have VMX or SVM, old school Xen was all it could run.

That board was an Asus, and it had served me well, as have several other Asus boards. My Citrix XenServer machine is running on a Foxconn mainboard and my VMware ESX is running on HP board (because VMware will not lower themselves to run on anything but name brand hardware.) This time around I selected an MSI 880GM-E41.

My plan was to run RHEL 6 for a few weeks to look at the changes to their virtualization stack, then move to either SL60, Citrix XenServer, or VMware. To determine my options, I attempted an install of each. Of course VMware threw up all over the box, but I expected that.

What happened next scared me: Citrix XenServer refused to load, since the machine had no network port. No need to panic. I tried Red Hat 6, which installed, but didn't recognize the onboard port.

Uh oh... Bad board? That means pulling out all the guts and spending more money to send the thing in for a replacement. Maybe I should check the Google Interwebs to see if this is a known issue.

And it is. The 880GM-E41 has an Atheros AR8131M, which is not "fully" supported by Linux. If I had gone $5 more to the 880GM-E43, I would have gotten a Realtek 8111DL, but I didn't check the drivers first. The really fun part is that if I'd gone $5 less to the 760GM-E51, I would have only lost the ability to overclock the memory. For some reason Newegg's site only displays the 760GM-E51 when you specifically search for MSI boards. (It must be a special order.)
http://www.blogger.com/img/blank.gif
So what do we do? Turns out, Red Hat's Anaconda installer does not recognize the Atheros NIC, but the Red Hat kernel does. Once the install was finished, I was able to manually configure the interface. At the first boot, udev had seen the card, and installed the driver, but was not able to active it without an ifcfg-eth0 file.

I've got a couple hours left, tonight. I'll either try to hack Citrix Xenserver to recognize the NIC, or mess around on I Can Has Cheeseburger.

XenServer System Alerts From the Future

I got the system alert on my XenServer this weekend. Its telling me that on the 16th there was a set of updates released. Except, its the 6th. So this system alert hasn't happened yet. Or mamybe they are going to release the updates on the 16th. Nope, the updates are there.

I'm soooooo confused.

More XenServer Paravirt Adventures

While working with someone else about running Linux under Citrix Xenserver 5, I ran across an interesting paradox. Daniel points out that the ability to run Linux as a DomU has been moved into the kernel, no longer requiring the kernel-xen RPM.

Yet, when I tried to install F11, on XenServer as a paravirt VM using the "CentOS 5.3 x64" template, it failed. When I clicked the Logs tab in XenCenter, I got:

Unable to access a required file in the the specified repository:
file:///xxxyyyzzz/images/xen/vmlinuz

The template is expecting the standard install image format.

Tried a few other templates with the same result. The only way I was able to get it to run was by installing using "Other install media", which meant it was running HVM.

The paradoxical part of this, is that Fedora implies that it can be done, yet when attempted on a Fedora Xen Dom0, an F11 paravirt VM fails:

libvir: Xen error : Domain not found:
    xenUnifiedDomainLookupByUUID
Invalid URL location given: [Errno 14] HTTP Error 404

In a way, its the message XenServer threw: I can't find the specially compiled paravirt kernel.

My guess at this point is that there has to be a new installer that understands how to use the newly compile options. Looks like there is some more research to be done. Stay tuned.

Citrix XenDesktop Express

Citrix finally released the evaluation product for XenDesktop 4, with a free Express edition for ten users. Before downloading, watch the introductory video, then come back here, to find out that for 87.3% of you the product will be unusable.

Forty seconds into the video, the narrator announces that the download "includes everything you need for a basic environment". BRAAAAAM! Wrong: You need at least one Windows Server 2003 licenses, and a functioning Microsoft Active Directory Controller. (Which is a second 2003 license... but who's counting.)

A significant disappointment, as I had hoped this could be a tool used by a small mobile work force. Instead, the Express product is completely useless. If an organization has MS Server and AD, they need that full version, not the express.

I guess that's four hours wasted.

P2V Happiness

It worked. I was able to use the Citrix Xenconvert to P2V a Windows XP Home system to a VM on a XenServer 5. There were a few lessons learned that should have been obvious... or maybe documented.

Citrix XenConvert works differently than some other P2V utilities in the respect that most of its operations are offline. It creates the image on the local hard drive, and does not contact the virtual infrastructure until the image is finished and ready for import. This is good because it prevents network issues from preventing image creation. The downside is that it requires the source machine to have 60% free space. (Preferably on a second disk.)

On the server side, you have to have a default landing zone (Storage Repository) for the imported VMs. I forgot to define the default SR, so my import failed. The good news is that the image was retained on the source machine, so I was able to transfer the image, and import directly.

It took about an hour to create on the source side, but about four hours to import the image on the server side. This is because the server is "governed" to prevent impact on production VMs. (In the real world, I'd have a staging server just for imports, deployments, and installs.)

Last detail is that once on the server, I had to strip the hardware specific software. The drivers were already loaded.

Windows P2V Failures

First, the bad news: I have made several attempts at doing a Windows P2V, and they have all failed. Last night's attempt with the Citrix XenConvert software was no better... Except it lead me to realize why my attempts were all dismal failures.

Before we get to the the why, let me point out that Citrix XenConvert is one seriously cool tool. More on the later.

Now, the good news: All my P2V attempts have grabbed the drive from the Windows system (all have been XP), and written the content to the VMs drive. I can mount the drive and see the NTFS partition. When I boot the VM, they uniformly refuse to load.

And the problem is... All my Windows systems are dual boot. This means that when the P2V software packages the physical machine, it grabs the MBR and the NTFS, but misses the /boot partition. Without /boot, it can't get the grub.conf to realize that it doesn't need /boot.

The solution should be fdisk /mbr, but in order for that to work, you needed the Administrator password to the physical system. (Notice we are talking about the Administrative account, not an account with admin access.) Unfortunately, Compaq and HP obscure the account on their pre-installed systems. Oops.

In order to test that the problem is grub related, I plan to recover one of the systems to factory defaults, try the converter, then reinstall Linux.

Fedora On Citrix XenServer 5

I've been having problems getting Fedora to run on Citrix XenServer 5.5.0 and have been disappointed with the resources available to complete the task. But, once again, persistence has rewarded us with victory. The problem is "an issue" with the paravirt console driver.

On the Citrix XenServer, the console is meant to redirect to /dev/xvc0 rather than tty0. With Red Hat (and Fedora's) implementation of xen, the paravirt driver intercepts tty0 and seamlessly handles the reroute. No so with Citrix, meaning we need to help the VM find the location of the console. Three changes to the VM are needed:

vi /boot/grub/grub.conf
add a kernel argument:
...xen ro console=xvc0 root=...

vi /etc/inittab
change the tty1 line:
.../sbin/mingetty xvc0

to allow root login at console:
echo xvc0 >> /etc/securetty

What might make this difficult is the fact that without these changes, you can't get to the console. Unfortunately, without the console, you can't make the changes.

The interesting thing is that you can boot to single user mode, and avoid the problem. To do this:
  1. Using XenCenter, right click the newly created VM and select Properties.
  2. Click Startup Options.
  3. For OS Boot Parameters add single.
  4. Click OK and reboot the VM.
  5. Make the changes listed above.
  6. Use the same steps to remove single from the boot params.
  7. Reboot.
This should allow you to modify the VMs configuration.

Citrix XenServer P2V

An interesting feature of the Citrix XenServer install CD is the option to execute a physical to virtual (P2V) migration. Having significant experience with VMware Converter, I thought I'd use this to create a VM from a Windows XP instance sitting on a drive that I never use. From my point of view, it would be a legal version of XP, though I'm sure Microsoft's lawyers would argue otherwise.

The first shot at P2V failed with a message specifying a partition error with /dev/dm-5. My immediate concern was that the USB multi-card reader was causing trouble. I disconnected it, and tried again with the same results.

A little research found a completely unexpected result. Turn out Citrix P2V only works on Linux, and not Windows. Looks like the old NTFS patent issue. Further, it only works on "older" Linux, somewhere in the Fedora 8 and before range. The problem doesn't seem to have anything to do with USB, but instead the way the migration tool interprets the device mapper.

I decided to hold off on further use of Citrix P2V, for the time being, but am continuing research on doing a manual P2V of that Windows instance.

Dual Boot Citrix Xenserver 5

I'm very proud of this hack. So proud that it's actually a quad boot, but two of the four are Windows, which we don't care about, so we'll stick with the title of dual boot. Here's the scenario:

A couple weeks ago, I did the VCP-310 cert, with the anticipation of rolling that into the VCP-410 next month. Until then, I'm going to do the Citrix CCA cert. This means we need a Citrix Xenserver... Unfortunately, the only machine I have that is not doing something is my oldest kid's desktop. Now, this is not as bad as it sounds, as he's off doing a summer session at some university somewhere, or something.

Since he's gone, we'll wipe the drive and load Citrix. Not so fast! He's got Windows on that thing and uses it as a game station when he's in town. I'd already partitioned it and loaded it with boot to Fedora as well as Windows, so Citrix should be a non-issue. Except, Citrix Xenserver thinks it should be the only thing on the system... Just like ESX.

So, here we go:
1. Disconnect the hard drive and load a second hard drive, pinned out as master.
2. Load Citrix Xenserver in the default configuration. Reboot and test.
3. Disconnect second hard drive and pin out as slave. Cable it and the original drive to the primary controller.
4. Boot to BIOS and ensure both are visible. This ensures that both are pinned correctly.
5. Boot off the previously existing Linux partition to Single. Use fdisk -l to validate that the second drive is visible.

(This is where it starts to get hairy.)

6. The master appears as /dev/sda and the secondary as /dev/sdb. Mount the second drive:

mount /dev/sdb1 /mnt
ls -l /mnt

Notice anything unusual? No boot partition. Citrix is loading the MBR with a LILO like loader.
7. Inspect /boot:

chain.c32
config-2.6.18-128.1.6.el5.xs5.5.0.496.1012kdump
config-2.6.18-128.1.6.el5.xs5.5.0.496.1012xen
extlinux.conf
extlinux.sys
initrd-2.6.18-128.1.6.el5.xs5.5.0.496.1012kdump.img
initrd-2.6.18-128.1.6.el5.xs5.5.0.496.1012xen.img
initrd-2.6-xen.img
mboot.c32
menu.c32
System.map-2.6.18-128.1.6.el5.xs5.5.0.496.1012kdump
System.map-2.6.18-128.1.6.el5.xs5.5.0.496.1012xen
vmlinuz-2.6.18-128.1.6.el5.xs5.5.0.496.1012kdump
vmlinuz-2.6.18-128.1.6.el5.xs5.5.0.496.1012xen
vmlinuz-2.6-xen
vmlinuz-kdump
xen-3.3.1.gz
xen-3.3.1.map
xen-3.3.gz
xen-3.gz
xen.gz

Some of this looks familiar. We just need to arrange it into our existing grub.conf.
8. I looked at an old FC6 Xen platform's grub.conf:

title Fedora Core (2.6.20-1.3002.fc6xen)
    root (hd0,0)
    kernel /xen.gz-2.6.20-1.3002.fc6
    module /vmlinuz-2.6.20-1.3002.fc6xen ro root=/dev/hda3
    module /initrd-2.6.20-1.3002.fc6xen.img

(Why in the world am I still running Fedora Core 6? Because, it is most similar to RHEL5.)
9. A closer look at /boot will confirm that many of those files are actually symlinks. So lets add the following to grub.conf:

title Citrix Xenserver 5.5.0
    root (hd1,0)
    kernel /xen-3.3.1.gz
    module /vmlinuz-2.6-xen ro root=/dev/sdb1
    module /initrd-2.6-xen.img

Notice the root line-- hd1,0 not hd0,0-- we're on the second drive. Notice the vmlinuz line-- we are using /dev/sdb1-- that's what we mounted.
10. Reboot... and watch it fail!

I'll save you the agony of troubleshooting. First, in the Fedora example, boot is partition 1, so we prefix the files with a slash (/). Citrix did not use a boot partition: root is partition 1. Therefore, we need to prefix the files with the directory: /boot/

Second, we mounted /dev/sdb1, which was the Xenserver's root partition. Yet, when Xenserver boots, it sees the drive as /dev/hdb1. This is because I happen to have Fedora 8 loaded, and effective F8, all drives are sd, even though the drives are actually IDE rather than SCSI. Since Xenserver is based on el5 (look at the vmlinuz links) it sees the drive as hd.

Here's what actually worked:

title Citrix Xenserver 5.5.0
    root (hd1,0)
    kernel /boot/xen-3.3.1.gz
    module /boot/vmlinuz-2.6-xen ro root=/dev/hdb1
    module /boot/initrd-2.6-xen.img