Showing posts with label xen. Show all posts
Showing posts with label xen. Show all posts

Saturday, April 03, 2010

OpenNebula- Red Hat Xen Node

For the record, if your are running Red Hat, consider making the move to KVM-- That is where they will be focusing their attentions. This is not to say that Xen is dead, since it is re-included in Fedora 12. Having said that, my test cluster is old hardware that won't support KVM, so I'm having to do my OpenNebula development on Xen.

To test Red Hat functionality, I needed to build a Xen node. I could have used CrapOS... I'm sorry that was a type... I meant to say: I could have used CentOS for this test, but we all know how stupid that would be.

What follows is a little black magic used only for testing. These are the steps needed to take an @base install on 5.5 and get Xen running. Since yum sometimes gets confused on this process, it is best done directly off the CD or a mounted image:
rpm -Uvh Server/kernel-xen-[0-9]*.rpm
rpm -Uvh Server/bridge-utils-[0-9]*.rpm
rpm -Uvh Server/xen-libs-[0-9]*.rpm
rpm -Uvh --nodeps Server/SDL-[0-9]*.rpm
rpm -Uvh VT/libvirt-[0-9]*.rpm
rpm -Uvh VT/libvirt-python-[0-9]*.rpm
rpm -Uvh VT/python-virtinst-[0-9]*.rpm
rpm -Uvh VT/xen-[0-9]*.rpm
sed -i "s/default=1/default=0/" /boot/grub/grub.conf
reboot
Note: the nodeps argument is to avoid complaints about sound drivers. Since we are building a cloud node, we don't care 'bout no stinkin' sound drivers.

Now, the dependencies for OneNebula:
yum install -y ruby xmlrpc
Unfortunately, there is one dependancy missing from the Red Hat disto, so we have to grab it from Fedora. Due to some glibc versioning issues, we go back to F8:
wget http://archive.fedoraproject.org/pub/archive/fedora/linux/updates/8/i386.newkey/xmlrpc-c-1.06.31-2.fc8.i386.rpm
rpm -Uvh --nodeps xmlrpc-c-1.06.31-2.fc8.i386.rpm
And lets try Open Nebula:
rpm -Uvh one-1.4.0-1.i386.rpm
Preparing...   ########## [100%]
  1:one           ########## [100%]

Tuesday, March 30, 2010

OpenNebula Cluster

My OpenNebula cluster is up an running, and all seems to be stable. There were a few fits along the way, but must was probably due to the crap hardware I'm running this thing on. Understand, of course, that this is not a production cluster, so the fact that it runs on four Pentium III's can be forgiven.

The thing to understand about OpenNebula, is a cloud environment, not a virtualization platform. This means that we need to choose an OS first. Because of the hardware, I'm using Fedora 8 with Xen. (I prefer Fedora Core 6, since it is more like RHEL 5.2, but it was not stable with OpenNebula. Lesson learned: use F8.)

I did a base kickstart on the nodes, to ensure a slim footprint. I loaded the Xen kernel and libraries, but left off virt-manager to avoid the overhead of an X server. In my cluster, three of the four nodes are identical, but the fourth is more powerful. That will be our head node.

When I tried to load the one-1.4.0 rpm on the head node, I ran into dependencies. (Ah, yes: the download says its for F11, and I'm using F8.) Needed packages:
yum install -y xmlrpc xmlrpc-c
yum install -y ruby
This extracted to the /srv/cloud/one directory.

The RPM created accounts:
/etc/passwd: oneadmin:x:512:903
/etc/group: cloud:x:903
The users's home directory is set to the containerized directory created above. A little simple sysad magic to clean up the account and assign keys to allow oneadmin the SSH to localhost without a password.

And here is where it gets beautiful: We now use NFS to export the directory to the nodes. Log in to the nodes, mount the NFS share, and replicate the user accounts. (Note to self: add user and group to NIS.) With the mount in place, return to the head node, become the oneadmin user, and SSH to the node. Since the user's home is the share, and keys are in the share, we get right in.

The infrastructure is in place, now its on to VMs.

Tuesday, December 29, 2009

More XenServer Paravirt Adventures

While working with someone else about running Linux under Citrix Xenserver 5, I ran across an interesting paradox. Daniel points out that the ability to run Linux as a DomU has been moved into the kernel, no longer requiring the kernel-xen RPM.

Yet, when I tried to install F11, on XenServer as a paravirt VM using the "CentOS 5.3 x64" template, it failed. When I clicked the Logs tab in XenCenter, I got:
Unable to access a required file in the the specified repository:
file:///xxxyyyzzz/images/xen/vmlinuz
The template is expecting the standard install image format.

Tried a few other templates with the same result. The only way I was able to get it to run was by installing using "Other install media", which meant it was running HVM.

The paradoxical part of this, is that Fedora implies that it can be done, yet when attempted on a Fedora Xen Dom0, an F11 paravirt VM fails:
libvir: Xen error : Domain not found:
    xenUnifiedDomainLookupByUUID
Invalid URL location given: [Errno 14] HTTP Error 404
In a way, its the message XenServer threw: I can't find the specially compiled paravirt kernel.

My guess at this point is that there has to be a new installer that understands how to use the newly compile options. Looks like there is some more research to be done. Stay tuned.

Monday, November 30, 2009

Windows P2V Failures

First, the bad news: I have made several attempts at doing a Windows P2V, and they have all failed. Last night's attempt with the Citrix XenConvert software was no better... Except it lead me to realize why my attempts were all dismal failures.

Before we get to the the why, let me point out that Citrix XenConvert is one seriously cool tool. More on the later.

Now, the good news: All my P2V attempts have grabbed the drive from the Windows system (all have been XP), and written the content to the VMs drive. I can mount the drive and see the NTFS partition. When I boot the VM, they uniformly refuse to load.

And the problem is... All my Windows systems are dual boot. This means that when the P2V software packages the physical machine, it grabs the MBR and the NTFS, but misses the /boot partition. Without /boot, it can't get the grub.conf to realize that it doesn't need /boot.

The solution should be fdisk /mbr, but in order for that to work, you needed the Administrator password to the physical system. (Notice we are talking about the Administrative account, not an account with admin access.) Unfortunately, Compaq and HP obscure the account on their pre-installed systems. Oops.

In order to test that the problem is grub related, I plan to recover one of the systems to factory defaults, try the converter, then reinstall Linux.

Wednesday, August 26, 2009

Console is not yet active for guest

I was barking up the wrong tree on this one. I had a VM that would not start, displaying the message:
Console is not yet active for guest
My confusion was caused by the fact that if I shutdown a different VM, this one worked fine. Furthermore, this was effecting multiple VMs, besides just the two. I even went so far as to call tech support on this one. Gasp!

Needless to say, I solved this before they did. The problem is a quirk in the RHEL5 virt-clone command. This command is suppose to copy a VM image, while making the needed changes in the descriptor file in the process. It does not, however, change the VNC port number. This has the effect of forcing two machines to use the same video card. In the directory with the VM descriptor files, try:
cd /etc/xen
grep "^vncdisplay" * | sed 's/"//g' | sort -n -k3
Any duplicate values define a conflict-- whoever boots "powers on" first, wins. The looser gets power, but can't initialize their BIOS.

To resolve, edit the descriptor file, and assign a unique value: But watchout! Make sure it is unique across the cluster, otherwise a migration could cause a collision.

Now I guess I got to tell Red Hat.

Saturday, August 01, 2009

Xen error "could not find any free loop device"

On one of my Xen systems, I added a disk to a VM, but when I attempted to start the machine, it threw up an error:
could not find any free loop device
With a little help fron Google, I found a solution.

Normally, I run m VMs in LVM because they are faster to create, easier to resize, and perform better. In this case, the VM had to be an image file for easy conversion from Xen to ESX. There were four VMs with two disks each. That's 8 disk total. (Yes, I realize you can count, but that number is important.

Turns out that each image file uses a loop device, as if mounting an ISO file. By default there are only 8 /dev/loop's. (See, I told you it was important.) Here's how we add more:
ls -l /dev/loop* | wc -l
          8
echo "options loop max_loop=64" >> /etc/modprobe.conf
rmmod loop
modprobe loop
ls -l /dev/loop* | wc -l
          64

Friday, July 24, 2009

Converting a Xen Paravirt VM to Fullvirt

We had a VM that would occasionally try to load the wrong kernel at boot time. The machine has to load the para-virt ELxen kernel to successfully start, but it might try to load a PAE instead. As a result the VM would fail to boot.

It might seem that the solution would be to fix why it was loading the wrong kernel, but, for reasons we will not discuss at the moment, we needed to accept that it would load the wrong kernel occasionally. So, our mission became to move the VM from para-virt to full-virt. As it turns out, it was just a case of modifying the VM config file.

Comparing the two configs, we see some differences:
diff vm-para vm-full
< bootloader='/usr/bin/pygrub'
< vfb=['type=vnc,vncunused=1,keymap=en-us']
---
> builder = "hvm"
> kernel = "/usr/lib/xen/boot/hvmloader"
> boot = "c"
> pae = 1
> acpi = 1
> apic = 1
> localtime = 0
> device_model = "/usr/lib64/xen/bin/qemu-dm"
> sdl = 0
> vnc = 1
> vncunused = 1
> keymap = "en-us"
This tells us that the para-virt VM uses a different Grub and that it needs a set of variables to describe its connections to the outside world. The full-virt VM, has a much wider set of variables, and uses a different boot mechanism.

So, how do we convert?
1. With VM running, install a standard kernel, using rpm -ivh
    or yum install kernel.
2. Power off VM.
3. Edit config file, removing the two line specified by "<" above.
4. Add the lines specified by ">" above. Make sure to remove the leading greater-than (>) symbols!
5. Start VM.

Thursday, July 02, 2009

FC-6 VM On F11 Platform

I recently upgraded one of my virtualization platforms from F10 to F11. It is significantly more stable and refined. An odd thing happened, however. The target system is an AMD-64x2 with 5G. Its was hosting:
FC6 application server
F8 VNC server
WXP Pro with RDC (need it for my current contract)
F10 on 4 node cluster for clustering R&D
F8 on 2 node cluster for web development
F8 application server
Under F10, the box gave great VM performance while at an idle.

Under F11, the FC6 VM pegged the virt-manager's performance graph at 45% of the physical CPU cycles. Yet, within the VM, top saw nothing. Effectively an entire core spinning away on nothing! I re-imaged (via kickstart) the VM as FC8, and now that same machine sits at half a percentage point. I'm not sure I have an explanation other than a different kernel or virt drivers.

Friday, June 19, 2009

Xen error: xc_dom_find_loader

One of my Xen virtualization platforms crashed yesterday, throwing the error:
Error: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')
Had the hardest time figuring out the cause.

Then, for whatever reason, I did a df -h and found my the /var/lib/xen mount was at 100%. I wiped out the contents of the save subdir, and moved all the boot_ files to /tmp. Suddenly, all my VMs could start.

Friday, March 06, 2009

Updating XenServer

I'm a little disappointed that the XenCenter management platform for XenServer only runs under Windows, but then VMware VIC only runs under Windows, so I shouldn't really expect much from them. What's more odd is the way the XenServer is updated. Since the box is a bastardized version of RHEL 5, I expect Yum to be the update mechanism. Here's how it works, instead:

1. XenCenter polls Citrix for updates and provides a list.
2. Items are downloaded via an IE download manager plug-in to the Windows station.
    (Select "Save", not "Install"!!!)
3. Navigate to the download folder, double click the xsupdate file.
4. XenCenter will launch with a system update wizard.
5. Select the servers to be updated.
6. Sit back and watch all the pretty colors flash by.

Just don't like having to use Windows to update Linux.

Tuesday, March 03, 2009

XenServer Licensing

Since I have nothing better to do with my time, I've been playing with the Citrix XenServer and XenCenter. I imagine this will change after the March 25th update, but I noticed that XenCenter only applied the license to the first server you brought online. My assumption was that it was replicated to all others as they were added, but that does not seem to be the case.

To license a new server, highlight the machine in the inventory. On the menu bar, click Server, and select Install License Key. Should be point an click from there.

Wednesday, February 11, 2009

More virsh Documentation Oversites

Tried to add a disk to a VM, the documentation was lacking. Here's the virsh command:
virsh attach-disk vmname /dev/disk sdb
In this example, it will appear as /dev/sdb to the VM. Unfortunately, partprobe did not work, we had to reboot.

In this case, I've added it to two VMs with the anticipation of granting multiple write access via GFS. Stay tuned.

Thursday, January 15, 2009

No VMX on RHEL4

And here's another mind blower: If you have a machine with a VMX enabled CPU, you boot it to RHEL5, and cat /proc/cpuinfo, everything works as expected. If you boot that same machine to RHEL4, it masks the VMX flag.

Actually, what happens is that /proc is created at boot time. The cpuinfo file is created and formatted, based up on the kernels interpretation. Sine VMX wans't around in RHEL4 days, it doesn't know how to translate the bits and bytes of the flag.

VMX Disabled in BIOS

I was trying to build out a full-virt VM, and was disappointed to find get an error:
Can't do --hvm on this system: HVM guest is not supported by your CPU or enabled in your BIOS
I knew that wasn't right: vmx showed up in the cpu flags.

Then I learned a new trick. When we cat /proc/cpuinfo, we are looking straight into the CPU. But... That doesn't mean a feature isn't disabled either in the BIOS or on the motherboard. To see what the system's capabilities, we have to go to /sys. In this case we need to:
$ cat /sys/hypervisor/properties/capabilities
xen-3.0-x86_32p
If vmx was working, we'd see a list of hvm capabilites. (Remember: vmx and svm are proprietary, hvm is open.)

On the HP I was working on, virtualization was disabled in the BIOS. That seemed strange, until a co-worker found the setting in the BIOS security settings. Oh, that makes sense-- VM's are a security risk. Either that or it cuts into the number of physical machines HP sells.

Now that it is enabled in BIOS, we get:
$ cat /sys/hypervisor/properties/capabilities
xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
Molto bueno.

In all fairness, it's not only an HP issue. I found that Compaq is disabling this also.

What's that you say? They are the same company? Nooooo.

Friday, November 07, 2008

F10-beta Kickstart Memory Reservation

Got another glitch for you. If you'll recall, one of my saddest moments in my work with F10-beta KVM was when I realized that we had lost the ability to dynamically control guest memory. It bit me again.

When attempting a kickstart, I was allocating 256 Meg of memory. The process would reach package number 10 (I think it was gcc... but who needs that) and freeze. Eventually, I found that by allocating 512, it continued the install. This man have nothing to do with KVM at all, but my have been an issue on a physical machine of less than half a gig of RAM.

There may be a sweet spot between 256 and 512, but I did not investigate any further.

Bottom line: Install at 256, then move down from there.

Thursday, November 06, 2008

F10-beta Error Kickstarting VM

Here's a tremendously perplexing bug. When attempting to kickstart a VM install from the command line, the kickstart dies with any error that it can not read from /dev/sdb.

First oddity is a warning that "The partition table on device sdb was unreadable" and needed to be initialized. This message has always been a anomoly solved by simply responding "Yes". Unfortunately, this time it is for a non-existent /dev/sdb.

After a too long delay, the second warning states "Input/output error during write on /dev/sdb" with the options of Retry / Ignore / Cancel. When I ignore, Anaconda throws exception 11.3.0.50 for KeyError: 'sdb'.

For some reason, virt-install was reserving 3876 MB of disk space for an sdb partition, but not creating the sdb1.

A dozen attempts at a kickstart %pre work around later, broke for dinner, and watched a free HD OnDemand Bond movie (Dalton). When I walk back to the system, the kicktart had worked. It turns out, this time, the kickstart worked. This one, however, was executed via the gui wizard. I examined the log files, and found the difference.

Doesn't work:
virt-install -n vinci -r 256 -f /dev/img/vm02f8
-l /vinf/ISO/Fedora-8-x86_64-DVD.iso -b virbr1
-x "ks=ftp://192.168.169.13/ks/vinci.cfg"

Works:
virt-install -n vinci -r 256 -f /dev/img/vm02f8
-l ftp://192.168.169.13/f8-64 -b virbr1
-x "ks=ftp://192.168.169.13/ks/vinci.cfg"

Can you see the difference? Look at -l. The first (bad) is attempting a local install from an ISO. The second (good) is using an extracted image via FTP.

Now the sad part: I can't report it as a bug, since I can't solve the problem. I do however have a work around, which moves me forward. It doesn't help the developer's with their code problems, but then I'm not a programmer.

Wednesday, November 05, 2008

F10-beta Network Bridge

With a little direction from a fellow Fedora'r, I've finally got the last piece of the F10-beta virtualization suite working.

By default, KVM (I hate that name... a KVM is a type of switch) creates a bridge process between the physical NICs and the virtual machines. The bridge process handles DHCP on an internal subnet and NATs the VMs to the physical world. In other words, virbr0 emulated a Linksys or D-Link home router. This works fine for outbound communications, but does not lend itself to connecting to virtual servers.

What is needed is the ability to route from an external address to a VM. This means slaving an ethernet to a bridge, assigning them to the same network, and establishing a route between the physical card and the bridge. To accomplish this, I created the following script:
[root@adama network-scripts]# pwd
/etc/sysconfig/network-scripts
[root@adama network-scripts]# cat ifcfg-virbr1
#!/bin/sh

IF="eth1"
NET="192.168.69"
HOST="11"
BR="virbr1"

brctl addbr $BR
ifconfig $IF 0.0.0.0
brctl addif $BR $IF
ifconfig $BR $NET.$HOST netmask 255.255.255.0 up
route add -net $NET.0 netmask 255.255.255.0 $BR
route add default gw $NET.1 $BR

exit 0
First the script location. By placing it in network-scripts, it is sourced at boot and service network restart. The name is not relevant except for the minor detail that the ifcfg's are executed alphabetically, so the filename has to start with letter FOX or higher.

Second, the script uses variables for portability. Notice that $NET.$HOST form an IP address. This will end up being the primary interface on the system, as the last command will switch the server's default route to the bridged interface.

Fianlly, I would suspect there is a way to embed this into the virtlibd subsystem. At this time that is not a priority and I think this may provide more flexibility. I say that because adding this script to a kickstart is simply the addition of a cat >> x << EOFcommand in post.

Wednesday, October 15, 2008

F10-beta Virtualization: native -vs- hosted

More evidence that the Fedora 10 Beta virtualization technology, KVM, is a hosted solution rather than the native solution provided by Xen in versions 5 through 8. In FC6, restarting the virtualization library deamon (libvirtd), was a standard troubleshooting technique. In F10, it crashs the VM, since the VM is running inside the deamon.

FC6 using Xen:
[root@baltar ~]# xm list
Name           ID Mem VCPUs State
Domain-0     0   141         1       r-----
valhalla         5     95         1       -b----
[root@baltar ~]# service libvirtd restart
Stopping libvirtd daemon:       [ OK ]
Starting libvirtd daemon:       [ OK ]
[root@baltar ~]# xm list
Name           ID Mem VCPUs State
Domain-0     0   141         1       r-----
valhalla         5     95         1       -b----
[root@baltar ~]#
The VM survives.

F10-Beta running KVM:
[root@adama ~]# virsh list
Id Name             State
----------------------------------
14 vinci             running

[root@adama ~]# service libvirtd restart
Stopping libvirtd daemon:       [ OK ]
Starting libvirtd daemon:       [ OK ]
[root@adama ~]# virsh list
Id Name             State
----------------------------------

[root@adama ~]#
The VM dies.

The moral: don't bounce libvirtd.

Thursday, October 09, 2008

F10-beta VM Memory Management

Sadness. Gone is the ability to dynamically change the memory allocation for a VM. Previously, we could change the RAM on the fly:
# xm mem-set 7 256
This was a nice feature, especially for loading new software packages. Since the xm command is gone, the new format is:
# virsh setmem vinci 256000
libvir: QEMU error :
this function is not supported by the hypervisor:
cannot set memory of an active domain

The first thing to notice is the memory must be stated in kilobytes rather than megabytes (how very 80's.) Second, the big nasty error message. In passing, the xm command was issued on an FC6 system, and used the domain number rather than name. In F7 and F8, you would use the name, as in F10.

Will there be a significant impact? Yes. This will impact the density of VMs on a host, but that's assuming the sysad is a perfectionist, like myself. In production, its probably not a big deal. After all, Vmware does not allow dynamic RAM allocation... Until 2009.

Wednesday, October 08, 2008

First Impressions: F10-beta Virtualization

My first impression of Fedora 10 Beta's virtualization suite is:
            WOW!

Somebody did something right.

1. We no longer must build a VM at 256Mb of RAM. I don't know what the low limit is, but that's nice.
2. Via GUI, ability to autostart a VM at boot.
3. VM's that Pixie boot. Mind boggeling!
4. Via GUI, connect to a CD in the host or a shared ISO.
5. VM sound support.
6. The ability to define infrastructure storage.
7. Via GUI, create new LVM's for VM images.
8. Ability to run TLS encrypted console windows.
9. Flat text config files under /etc/libvirt/qemu. Oh, how I hated having to export the XML from F7.
10. An F8 VM up in under 10 minutes.
11. Via GUI, take a VM screen shot... NOT a snap shot!
12. Ability to close virt-manager, and keep VM console connections.

Installs and works like a dream.