PVM, aka Beowulf Cluster

I stumbled upon a little project this weekend and found myself without a Beowulf cluster to help out.  It had been several years since I'd built a computational cluster, so I noticed a few "new" gothchas.  But... before we get to the fun stuff, let's review:

• No, Beowulf is not "dead technology"
• No, Hadoop is not the perfect tool for every job

That which was once called a Beowulf cluster is actually the use of a Message Passing Interface (MPI) that when deployed across nodes, create a Parallel Virtual Machine (PVM).  Ready for this-- the current generation of Hadoop (et al) clusters, are actually MPIs, and thus PVMs.  Given network attached storage, a map-reduce cluster is, in theory, Beowulf compliant.

To set up the absolute simplest PVM, we need two nodes, with an NFS share, and a user account.  The user needs an SSH key pair distributed to all nodes such that the user can login to any machine, from any machine.  Each node's hostname must be able to resolve via DNS or /etc/hosts.  Each node's hostname and address must be statically configured, and cannot be "localhost".

The first step is to install the base package from an EPEL repo.  (I'm using Scientific Linux 6.)  The package is delivered as source and must be compiled with a minimal set of options:

yum install -y pvm --nogpgcheck
rpm -ql pvm | grep -m1 pvm3
/usr/share/pvm3

This shows us where the RPM installed the source.  The issue with this incarnation is that it is still configured for RSH rather than SSH:

export PVM_ROOT=/usr/share/pvm3/
cd $PVM_ROOT
find . -type f -exec sed -i "s~bin/rsh~bin/ssh~g" {} \;
make; make install

Unfortunately, there are still hard-coded references to RSH in some of the binary libraries, so we spoof the references with a symlink:

ln -s /usr/bin/ssh /usr/bin/rsh

Repeat these steps on all (both) nodes.

On only one of the nodes (it doesn't matter which one) validate that PVM is not running, configure the PVM_ROOT variable, and start the first instance as the non-root user:

ps -ef | awk '!/awk/ && /pvm/'
echo "export PVM_ROOT=/usr/share/pvm3" >> ~/.bashrc
echo id | pvm
pvm> id
t40001
Console: exit handler called
pvmd still running.
ps -ef | awk '!/awk/ && /pvm/'
compute <snip> /usr/share/pvm3/lib/LINUXX86_64/pvmd3

Notice that the PVM deamon launched and remained resident.  Individual commands can be piped to PVM, or an interactive console can be used.  From the same node, remotely configure the next node:

ssh pvm2 'echo "export PVM_ROOT=/usr/share/pvm3" \
                >> ~/.bashrc'
# should not prompt for a password
ssh pvm2 'echo $PVM_ROOT'
/usr/share/pvm3
ssh pvm2 'rm -f /tmp/pvm*'

The last line is the very, very, important.  From the first node, remotely start the second node:

pvm
pvmd already running.
pvm> conf
conf
1 host, 1 data format
   HOST     DTID     ARCH   SPEED       DSIG
   pvm1    40000 LINUXX86_64    1000 0x00408c41
pvm> add pvm2
add pvm2
1 successful
   HOST     DTID
   pvm2    80000
pvm> conf
conf
2 hosts, 1 data format
   HOST     DTID     ARCH   SPEED       DSIG
   pvm1    40000 LINUXX86_64    1000 0x00408c41
   pvm2    80000 LINUXX86_64    1000 0x00408c41

In this sequence, we have accessed the console on pvm1 to view the clusters configuration (conf).  Next, we started the second node.  It is now displayed in the cluster's conf.

Just for fun, let's throw it the simplest of compute jobs:

pvm> spawn -4 -> /bin/hostname
4 successful
t8000b
t8000c
t4000c
t4000d
pvm>
[3:t4000d] pvm1
[3:t4000c] pvm1
[3:t4000c] EOF
[3:t4000d] EOF
[3:t8000b] pvm2
[3:t8000b] EOF
[3:t8000c] pvm2
[3:t8000c] EOF
[3] finished

There are a few things to notice about the output:

1. The command asked the cluster to spawn the command "/bin/hostname" four times.
2. The "->" option indicates we wanted the output returned to the console, which is completely abnormal... we only do this for testing.
3. The prompt returned before the output.  The assumption is that our compute jobs will take an extended period of time.
4. The responses were not displayed correctly.  They were displayed as they returned, because all this magic is happening asynchronously.
5. Each job's responses, from each node, could be grep'ed from the output using a unique serial number, automatically assigned to the job.

To leave the console and return to the command prompt issue the quit command.  All started nodes will continue to run.  To shutdown the compute cluster, execute:

echo halt | pvm

Finally, remember this one last thing: The cluster is a peer-to-peer grid.  Any node can manage any other, any node can schedule jobs, and any node can issue a halt.

Removing Old Linux Kernels

Today, I had trouble removing an obsolete kernel from my workstation. It should have been simple enough, but I tried to use yum erase rather than rpm -e, and kept running into errors. That is obviously the bad news, so let's make sure to report the good news: YUM is such an improvement over RPM alone, that it is smart enough to know which kernels are obsolete. For instance:

# rpm -qa kernel
kernel-2.6.32-279.el6.x86_64
kernel-2.6.32-279.19.1.el6.x86_64
kernel-2.6.32-279.22.1.el6.x86_64
# uname -r
2.6.32-279.22.1.el6.x86_64
# yum erase kernel
<snip>
Removing:
kernel   x86_64   2.6.32-279.el6
kernel   x86_64   2.6.32-279.19.1.el6
Is this ok [y/N]:

First, we determine the machine has three kernels. Second, we see that that it is running the most recent version, dot-22. Finally, YUM demonstrates that it is smart enough to erase the two old kernels, but not the current kernel.

One small problem: I don't want to remove dot-19 because I have a driver problem with dot-22. I only want to remove dot-null. Here's the trick:

# yum list kernel
Loaded plugins: refresh-packagekit, security
Installed Packages
kernel.x86_64   2.6.32-279.el6
kernel.x86_64   2.6.32-279.19.1.el6
kernel.x86_64   2.6.32-279.22.1.el6
# yum erase kernel-2.6.32-279.el6

The critical success factors are to drop the arch and t0 add a dash(-) between the package name and the version number.

RHEL6 Udev Rules

I recently moved my home workstation from Fedora to Scientific Linux 6, on the grounds that Fedora has diverged too far from the current RedHat distribution.  Sure, bleeding edge is cool, but as a self professed Linux mercenary, I need to be in sync with what the real world is doing... not what it might be doing.

After the move, I've found myself annoyed by the way the Gnome desktop handles removable media, in particular media cards such as Flash and Secure Digital (SD).  One trick I learned a while back, was to make sure to assign an e2label to cards formatted with an ext filesystem.  This way, when Gnome automounts the media and places an icon on the desktop, the name is the e2label.  Without an e2label, the icon's text is the device size.  This is also true of FAT devices.

The real problem, however, is the fact that the device is owned by root.  Since the desktop is running as an unprivileged user (because we never login the GUI as root... right?) we are faced with an icon for an device that we can't drop-and-drag to.  Doh! Here's how I used Udev to trick the system into allowing my GUI account to use these devices.

First, insert the device, allow it to automount, and appear on the desktop.  (We won't worry with how the kernel, udev, fuse, and the desktop is accomplishing this.)  Assuming an ext device, it was probably mounted to a dynamic mountpoint under /media; in this case, we ended up with:

# mount | grep media
/dev/sdb1 on /media/Flash_1GB type ext3
   (rw,nosuid,nodev,uhelper=udisks)
# ls -ld /media/Flash_1GB
drwxr-xr-x. 4 root root 4096 Feb  9 15:55 /media/Flash_1GB/

The goal is to modify a few mount options and change the ownership of the device.  To accomplish this, we need to tell Udev to watch for a given device and respond in a specific manner.  This requires isolating a unique aspect of the device that can be used a s trigger.  The command to manage Udev changed with RHEL6:

# udevadm info --query=all --attribute-walk --name=/dev/sdb
<snip>
looking at device '/devices/<snip>/6:0:0:0/block/sdb':
    KERNEL=="sdb"
    SUBSYSTEM=="block"
    DRIVER==""
    ATTR{range}=="16"
    ATTR{ext_range}=="256"
    ATTR{removable}=="1"
    ATTR{ro}=="0"
    ATTR{size}=="2001888"
    <snip>
  looking at parent device '/devices/<snip>/6:0:0:0':
    KERNELS=="6:0:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS=="sd"
    <snip>
    ATTRS{vendor}=="Generic-"
    ATTRS{model}=="Compact Flash   "
    <snip>

There are a few things to notice about this output.  On the command line, the name is the disk, not the mounted partition.  The top most block is the device, blocks that follow are upstream devices.  We are most interested in the ATTR fields.  Don't be seduced by the first directive, "KERNEL=="sdb"... we all know that Linux is notorious for changing device letters on reboot.

Second, Udev rules are created as code snippets in the /etc/udev/rules.d dir.  For simplicity sake, create a file called 99-local.rules and add all machine specific rules to this one file.  Each rule is one line.  There are many sophisticated and elegant things that can be done by Udev, but my example is a simple sledgehammer:

SUBSYSTEMS=="scsi", 
ATTRS{model}=="Compact Flash   ",
ATTR{size}=="2001888",
RUN+="/bin/sh -c 'mount /media/Flash_1GB
   -o remount,noatime,nodiratime,async;
   chown doug:doug /media/Flash_1GB' "

The first directive tells the machine that we're dealing with a disk (we could have used "block".)  The second directive is an attribute that was listed for the device (notice the spaces: it has to exactly match the output from udevadm.)  The third attribute is the device size, so this rule applies just to this card, or atleast to cards with exactly this number of sectors.  The last part of the rule is the RUN command, which executes a set of bash commands.  In this case, I'm changing the default mount options, then I'm changing the mount point ownership.  Using the RUN feature provides infinite flexibility.

dracut: FATAL: initial SELinux policy load failed

Here's an obnoxious install failure: Using the RHEL/Scientific Linux 6.3 DVD, it is possible for an install to crash on first boot with an SELinux error.  The problem is a bug in the post of the target policy RPM.  The bug is immediately fixed by running yum update... assuming you can figure out how to get the machine booted.  Luckily, the error is nice enough to tell you how move forward.

Reboot, and at the GRUB menu, append "selinux=0" to boot into Permissive mode.  From the root prompt:

ls /etc/selinux/targeted/policy/policy.24
ls: cannot access /etc/selinux/targeted/policy/policy.24:
No such file or directory

If possible, issue: yum update selinux-policy-*

If the machine is not network connected, the problem can be resolved by restoring the file policy.24 from install media.  And it it were that simple, you wouldn't need me.  You will have to force install two RPMs:

rpm -ivh --force selinux-policy-3.7.19-*
rpm -ivh --force selinux-policy-targeted-3.7.19-*

The second force install will take several minutes to complete.

Regardless of how the issue is resolved, it is best to relabel the filesystem:

touch /.autorelabel

RHEL Cluster Anti-Affinity Configuration

I'm often amused by how vendors define "High Availability", aka HA.  Customers always talk about Five Nines, but "off the shelf" HA solutions seldom achieve 99% availability.  In the case of RedHat's HA cluster service, the default configuration might provide unattended failover within 60 seconds.  Given that Five Nines only allows 25.9 seconds per month, a single failure can blow away a service level agreement.

To counter the real-world lag of application failover, an system must be load balanced across a cluster.  A real HA environment would be at least three nodes, running two instances of the service.  If one node fails, the load balancer will redirect everything to the second instance, while the service is recovered on the third node.

There is an HA problem that VMware has addressed in their HA solution, that RedHat has not, known as the anti-affinity rule.  Affinity is when two "processes" favor the same resource.  An example would be when running a web and database instance on the same machine improve performance.  In the case of redundant services, running them on the same machine is pointless, if the machine fails.  To prevent this, we need an anti-affinity rule that requires the two processes to never be on the same machine.

RedHat cluster suite provides affinity in the form of child services.  If the cluster moves the web service to another node, the database has to follow.  What they don't provide is an anti-affinity rule to prevent the load balanced services from trying to run on a single node.  As a matter of fact, by default, all services will start on the same cluster node.  (It will be the node with the lowest number.)

I found I could implement anti-affinity from with in the service,s init.d script.  First, we add an /etc/sysconfig/ file for the process, with the following variables:

CLUST_ENABLED="true"
CLUST_MYSERVICE="service:bark"
CLUST_COLLISION="service:meow service:moo"

A collision is when the presence of a service prevents this service from starting on this node.  The names should be listed exactly as they appear in clustat.  Make sure the script sources the config file:

# source sysconfig file
[ -f /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

Next, add a new subroutine to the existing init.d script:

cluster(){
  # look for other services on this host
  K=$(for J in $CLUST_COLLISION; do \
          clustat | grep "$J.*$HOSTNAME.*started" \
          >/dev/null; \
          [ $? == 0 ] && echo "$J "; \
          done)
  if [ $K ]; then
    # show service names
    echo -n "Cluster, collision $prog: $K"
    # fail, but with a success return code
    failure; echo; exit 0
  fi
  # look for this service running on other nodes
  K=$(clustat | grep "$CLUST_MYSERVICE.*started" | \
          awk '{print $2}')
  if [ $K ]; then
    # show hostname of other instance
    echo -n "Cluster, $prog exists: `echo $K | cut -d. -f1`"
    # fail but with a success return code
    failure; echo; exit 0
  fi
}

Finally, add a reference to the cluster sub in the start sub:

start(){
  if [ $(ps -C cluster-$prog.sh | grep -c $prog) == 0 ]; then
    # only check cluster status if enabled
    [ "$CLUST_ENABLED" == "true" ] && cluster
    echo -n "Starting $prog"

Here's what happens in the case of a collision:

• rgmanager issues a start
• the cluster sub recognizes the collision, but tells rgmanage that it started successfully (exit 0)
• rgmanager shows the service as running
• 30 seconds pass
• rgmanager issues a status against the service, which fails, since the init.d script lied about the service running
• the cluster orders a relocation of the service
• rgmanager issues a start... on a different node
• there is no collision this time, so the init.d runs as expected

RHEL 6 Virtualization, Paravirtualization

Paravirtualization is a big deal. It is avoided by VMware, is alchemy in Citrix Xen, and is cryptically alluded to in RHEL 6. Yet, for those of us that are almost exclusively Linux, the performance and density advantages are huge. Even Windows XP performance is noticeably improved running "para-virt". (As for Vista and Win7... they're both hogs, no matter what.)

In ten words or less, paravirtualization improves performance by loading a version of the operating system optimized for the host's hypervisor.

In the RHEL 6 Virtualization guide Chapter 8, they state that para-virt does not work with KVM. This would imply that there is no way to optimize a RHEL 6 VM on the RHEL 6 platform. Given that, why not just run VMware?

Yet in Chapter 11, they mention that para-virt drivers are automatically loaded and installed for RHEL 6 VMs and Linux VMs based on the 2.6.27 or newer kernel.

So... Which is it... Para-virt yes or para-virt no?

Survey says: Kernel no, drivers yes. But, of course, there's a catch.

Once the VM is installed and running, execute an lsmod | grep virtio. Look at the last line. The items at the right of the numbers will indicate which para-virt drivers are used. You want four of them, but may only have three. Depending on how the VM accesses the outside network, the virtio_net driver may be missing.

To enable virtio_net, a specific sequence of events must be followed:

1. Power off the VM
2. From virt-manager, Open the VM
3. Select View and Details
4. Select NIC
5. Change Device model to "vrtio"
6. Apply, exit, and Run the VM

Upon boot, the virtio_net driver should be listed.

Call it a bug, but if you try this with the VM powered on, it will claim to work, but will not.

As for performance without a para-virt kernel, I am still a little skeptical.

RHEL 6 Virtualization - Bridged Interface

With version 6, Red Hat has finally fixed the long standing problem of not being able to use the GUI to configure a Virtual Machine shared connection. Lets first review the types of VM network connections:

* RH=y  CX=y  VW=y Internal
* RH=y  CX=y  VW=y Dedicated or Slaved NIC
* RH=y  CX=y  VW=y Routed or VLAN'd
* RH=y  CX=y  VM=y Shared or Bridged
* RH=y  CX=n  VW=n Network Address Translation (NAT)
(Bold entries are the "default" config.)

An internal connection does not route traffic off the host system. A dedicated (also called slaved) connection, requires a separate NIC for each VM, which is very inefficient. A connection that is routed or VLAN'd requires the network be aware of the the specialized configuration. A shared or bridged connection (what we're after) extends the real world subnets in to the virtual machines. The last type, NAT, allows the VM's to communicate out, but does not permit inbound request, thus rendering it useless to servers.

Unfortunately, Red Hat uses NAT, by default, and their virtualization technology is principally used for server consolidation. As if this is a good idea. To make matters worse, their documentation still suggests manually configuring a shared connection, and does not explain that once you're done, you won't be able to see the connection in the GUI.

Here's how to do it right:

From virt-manager, connect to the host, click Edit, and select Host Details. On the Network Interfaces tab, click the plus sign (+) at the lower left. In the pop-up, select "Bridge" and Forward. Assign a name-- I recommend br followed by the eth number of the card you are sharing. In other words, if you are sharing eth1, name it br1.

On the same screen, set the start mode to "onboot", check "Activate now", and check the target NIC that you want the VMs to access. Take a deep breath, hold it, and click Finish. Scary things will happen, but after about a minute, the window should respond.

Notice that the eth item has disappeared from th elist and been replaced by the newly defined bridge. Now click the Virtual Networks tab, and notice nothing has changed. Why doesn't the new connection appear in the list? More evidence that Red Hat's interface is the least intuitive of all the vendors. This tab is a list of virtual networks, and a bridged connection is an extension of the physical network. (Yeah, while technically correct, it doesn't make sense to me either.)

When provisioning VMs, make sure to select expand Advanced options and choose the br device.

*** NOTE ***
An excellent discussion of the underlying technology is available on Dale Bewley's blog. You'll need this for kickstarting hosts.

RHEL 6 Virtualization As Non-root User

Red Hat has always been overconfident about the use of the root account over SSH. By default they allow direct logon to the root account via SSH, because the first "S" stands for Secure. But that's not the point... You should never let anyone logon directly as root. Always access as a user, and escalate privileges.

With RHEL 6 virtualization, I ran into a problem with virt-manager, in that it refused access from a non-root account. If I logged in as myself, the app would start, but not allow connections, complaining:

Unable to open a connection to the libvirt management daemon.
Verify that:
- The 'libvirtd' daemon has been started

Further digging (clicking "Details") shows a couple Permission denied messages. Using sudo doesn't help, switching to root doesn't help, but SSH'ing in as root works. So, what's the fix?

Turns out there is security group that allows the identification of trusted users. Edit the /etc/libvirt/libvirtd.conf file and uncomment three lines:

unix_sock_group = "libvirt"
unix_sock_rw_perms = "0770"
auth_unix_rw = "none"

The third line bypasses the "polkit" security mechanism... since it doesn't work, anyway.

The natural reaction would now be to issue a service libvirtd reload command, but not so fast... that won't work. (But feel free to try it if you like-- I'll wait.)

List the /var/run/libvirt dir. You should see a libvirt-sock file owned by root.root with permissions of 700. Use service libvirtd stop, and file will disappear. Issue service libvirtd start and it should fail to start, claiming it can not load the configuration file, which is not true. The problem is that we told it to allow access to members of group "libvirt", which does not exist. Create it, add yourself:

groupadd -g 170 libvirt; usermod -a -G libvirt doug

Logoff, login, and verify with the id command.

Issue the start command again, and the service should start. List the /var/run/libvirt dir. This time, the file is owned by root.libvirt with permissions of 770. We can now access and control the server as a non-root user.

RHEL 6 - Network Gotcha

Earlier, I had reported problems with my Atheros integrated NIC on my new mainboard. I finished the build on my RHEL 6 cloud node by adding an Intel Pro100 I had lying around as eth1. As expected, Anaconda recognized the card on install. To my surprise, the card was not active on first boot. But then the real surprise... I concurrently installed a RHEL 6 VM, and its virtual 8139 was also not energized on boot.

Something wicked is afoot, and it smells putridously similar to:

NetworkManager

When we cat ifcfg-eth0, by default, the file includes DEVICE, HWADDR, ONBOOT equal "no" (the symptom), and NM_CONTROLLED equal "yes" (the disease). I've hated NetworkManger ever since it showed up on RHEL 5. (It uses that ridiculous mixed cap, no space, spelling because the Red Hat developer who "innovated" this solution wishes he worked at Apple instead, and that's the way Apple processes are named.) My traditional solution was to chkconfig NetworkManger off, hardcode the network params, and life was good.

There is a perplexing part to this story, however, that forces more review. On RHEL 6, if you select @Base, @Minimal, or @Server-Platform, NetworkManager is not installed. That's fine with me, but the cards are tagged as being controlled by a service that doesn't exist. So, I approached this from the point of view that just as we are going to have to learn to use postfix instead of sendmail, know how to configure static networking with NetworkManager. Preferably, without the GUI.

So, I installed a slew of RPMs from DVD to get the NetworkManager service running. I read man pages, researched in the web, played with the tools, and found the solution. The solution is to hard code the static IP in a script under /etc/NetworkManager/dispatcher.d. But wait a second... If we have to hard code the data in a script, why not put it in /etc/sysconfig/network-scripts as we have been for over ten years. Good idea: Let's uninstall all this piece of crap.

The workarounds:
* Kickstart the server with a static address
* Set the address using firstboot
* Manually edit the interface files
* Use system-config-network-tui to set the address. (Don't try to use the TUI to set it to DHCP, it won't work)

Scientific Linux (SL-6.0) Pt 2

For the record, SL-60 will not boot with 128Mb of memory, it reports "Out of memory". Works with 196Mb. I generally run my VMs on the ragged edge, so I may try to chew that a little lower.

*** Update ***
Nope, 196Mb is as low as I'm willing to try to go.

Scientific Linux

With the introduction of Red Hat Enterprise Linux 6, I've had to renew my certification. A couple Friday's ago, I got my RHCSA, and in a couple more weeks I do the RHCE. In preparing for the test, I've come across a distribution called Scientific Linux

Scientific Linux is a recompiled version of RHEL, similar in concept to CentOS, with one very important difference: CentOS sucks and is for posers. Scientific Linux, or SL, however, is the output of Fermilab and CERN. Where as CentOS is a disorganized, bickering bunch of dot.com wannabee's, who have created a product for people that are too cheap to run Red Hat, but too scared to run Fedora, SL is put together by people who need raw computing horsepower, and lots of it.

I've played with it, and thus far, am impressed with what they've done. Here are some observations:
* I was a little surprised when I used the boot.iso to launch a installed only to find that it automatically does a minimal install without asking.
* It defaults to postfix instead of sendmail, but I guess its about time to let sendmail die.
* At install time, you can select several well respected third party repos.
* They've got a lean, mean, minimal install.
* The GUI installer would not run in 640Mb of RAM, but would at 768Mb. (But that might be a RH thing.) Once it was installed, I dropped the memory down to 256Mb and everything run spiffily.
* They've got the install DVD's laid out wrong-- You can't do a base install using only DVD1. For me, that meant I had to actually extract the DVD's to create an intall repo rather than mount just DVD1.

I'm running this in a VM under Citrix XenServer and am having some problems with the vmtools utility, but that does not surprise me. Once I've got the base VM tweaked, my first project will be an OpenLDAP server, followed by a Kerberos.

OpenNebula- Red Hat Xen Node

For the record, if your are running Red Hat, consider making the move to KVM-- That is where they will be focusing their attentions. This is not to say that Xen is dead, since it is re-included in Fedora 12. Having said that, my test cluster is old hardware that won't support KVM, so I'm having to do my OpenNebula development on Xen.

To test Red Hat functionality, I needed to build a Xen node. I could have used CrapOS... I'm sorry that was a type... I meant to say: I could have used CentOS for this test, but we all know how stupid that would be.

What follows is a little black magic used only for testing. These are the steps needed to take an @base install on 5.5 and get Xen running. Since yum sometimes gets confused on this process, it is best done directly off the CD or a mounted image:

rpm -Uvh Server/kernel-xen-[0-9]*.rpm
rpm -Uvh Server/bridge-utils-[0-9]*.rpm
rpm -Uvh Server/xen-libs-[0-9]*.rpm
rpm -Uvh --nodeps Server/SDL-[0-9]*.rpm
rpm -Uvh VT/libvirt-[0-9]*.rpm
rpm -Uvh VT/libvirt-python-[0-9]*.rpm
rpm -Uvh VT/python-virtinst-[0-9]*.rpm
rpm -Uvh VT/xen-[0-9]*.rpm
sed -i "s/default=1/default=0/" /boot/grub/grub.conf
reboot

Note: the nodeps argument is to avoid complaints about sound drivers. Since we are building a cloud node, we don't care 'bout no stinkin' sound drivers.

Now, the dependencies for OneNebula:

yum install -y ruby xmlrpc

Unfortunately, there is one dependancy missing from the Red Hat disto, so we have to grab it from Fedora. Due to some glibc versioning issues, we go back to F8:

wget http://archive.fedoraproject.org/pub/archive/fedora/linux/updates/8/i386.newkey/xmlrpc-c-1.06.31-2.fc8.i386.rpm
rpm -Uvh --nodeps xmlrpc-c-1.06.31-2.fc8.i386.rpm

And lets try Open Nebula:

rpm -Uvh one-1.4.0-1.i386.rpm
Preparing...   ########## [100%]
  1:one           ########## [100%]

No VMX on RHEL4

And here's another mind blower: If you have a machine with a VMX enabled CPU, you boot it to RHEL5, and cat /proc/cpuinfo, everything works as expected. If you boot that same machine to RHEL4, it masks the VMX flag.

Actually, what happens is that /proc is created at boot time. The cpuinfo file is created and formatted, based up on the kernels interpretation. Sine VMX wans't around in RHEL4 days, it doesn't know how to translate the bits and bytes of the flag.