Showing posts with label kvm. Show all posts
Showing posts with label kvm. Show all posts

Saturday, July 05, 2014

IPtables Blocking KVM Bridge

Recently, I've been having problems with VM networking on RHEL KVM hosts. The initial symptom is the VM cannot get a DHCP address from the physical network, through a bridged virtual NIC. I've determined the problem is the with the FORWARD chain of IPtables.

Assuming a bridged ethernet called br0, I've added the following rules:
iptables -I FORWARD -i br0 -o br0 \
  -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD -i br0 -o br0 \
  -m state --state NEW -j ACCEPT
This will allow UDP, TCP, and ICMP initiated from the physical network to be routed to VMs attached to the br0 bridge.

This configuration is that it route all traffic, which some might consider bad form. If the KVM host only ran a fixed set of VMs, it might be wise to lock down specific ports. In a dynamic environment (like my lab) the level of effort to support IPtables on the host exceeds the risk, as the VMs are all running IPtables themselves.

Thursday, March 13, 2014

KVM Network Bridge

Finally, a simple way to configure a bridged network for a KVM server:
virsh iface-bridge eth2 br0

Thursday, November 08, 2012

RHEL 6 Clustering, VM Fencing

I recently retasked one of my lab machines as a RedHat virtualization server, which RedHat calls RHEV, but is really KVM.  One of this machine's tasks is to support a test cluster of VMs.  Under normal circumstances, clustering would require a remote management interface such as an ILO, DRAC, or RMM.

As usual, I was disappointed with how difficult this was.  To make matters more difficult, for you, I won't be covering clustering in the article.  This document's scope will be limited setting up to RHEV VM fencing.

On the host machine, we need to install the fence daemon.  Considering this is very lightweight, the I'm going to do a shotgun install:
yum install fence-virtd-*
On my machine, this loaded four packages: the daemon, the interface between the daemon and the hypervisor, and two "plugins".  (The serial plugin is probably not needed.)

The base RPM will provide the /etc/fence_virt.conf file.  Modify it to look like this:
listeners {
  multicast {
    family = "ipv4";
    interface = "virbr0";
    address = "225.0.0.12";
    port = "1229";
    key_file = "/etc/cluster/fence_xvm.key";
  }
}
fence_virtd {
  module_path = "/usr/lib64/fence-virt";
  backend = "libvirt";
  listener = "multicast";
}
backends {
  libvirt {
    uri = "qemu:///system";
  }
}
Two things to notice about the config file.  The key_file option is little more than a password in a text file, which is going to have to be duplicated on all the VMs in the cluster.  The "theory" is that only a device with the password will be able to fence other nodes.  This brings us to the second point, the multicast option.  If a cluster node issues a fence command, the symmetric authentication key will be multicast on the network in the clear.  Thus, the reality is that the key_file provides no real security.

Which brings us to a second issue with the multicast.  Per RedHat, cross host fencing is not supported.  As such, all cluster nodes have to exist on the same physical machine, rending real world VM clustering pretty much worthless.  Here's the reality of cross host fencing: It is not supported because of the security concerns of multicasting the clear text fencing password and the fact that RedHat cannot guarantee the multicast configuration of the switch infrastructure.  Given properly configured switches, a dedicated host NIC and virtual bridge in each host, cross host fencing works.  In this lab configuration, however, it is not a concern.

After creating a key_file, open the fenced port in IPtables:
-A INPUT -s 192.168.122.0/24 -m tcp -p tcp --dport 1229 -j ACCEPT
Copy the key_file to each clustered VM (they don't need the config file) and add the opposite IPtables rule:
-A OUTPUT -d 225.0.0.12 -m tcp -p tcp --dport 1229 -j ACCEPT
On the host, chkconfig and start fence_virtd.  Running netstat should show the host listening on 1229.  What it is listening "for" is the name of a VM to "destroy" (power off.)  This means the names of the cluster nodes and VMs recognized by KVM/QEMU have to match.  On the host, display the status of the VMs using: 
watch virsh list
Given a two node cluster, on node1 issue:
fence_node node2
On the host, the status of node2 should change from running to inactive, and a moment later, back to running.  For testing purposes, the fence_node command can be installed on the host, without the host being part of the cluster.  If you try this using yum, you'll get the entire cluster suite.  Instead, force these three RPMs:
rpm -ivh clusterlib-*.x86_64.rpm  --nodeps
rpm -ivh corosynclib-*.x86_64.rpm  --nodeps
rpm -ivh cman-*.x86_64.rpm  --nodeps

 Truthfully, the better choice is to build a VM to manage the cluster using Luci.

Sunday, May 01, 2011

RHEL 6 Virtualization, Memory

There are two memory settings presented in virt-manager: Allocation and Maximum. The Allocation setting is what will appear in /proc/meminfo and top. The Maximum is what can be used to boot the VM. Interestingly, the kernel will refuse to boot with out enough RAM, but once booted, will run with significantly less.

The Maximum value needs to be more than 348MB. Any lower, and boot time is noticeably slower due to swap activity.

The Allocation value needs to be more than 148MB. The VM won't crash until the allocation is about 115MB, but there are several factors that could effect that number. Obviously, 148MB may not meet a every VMs needs, but it seems to be the lowest reasonable limit.

Keep in mind, the Maximum is always allocated to the VM at boot time, and them lowers to the Allocations, so don't set it too high. The Allocation can be dynamically changed for a running VM, but cannot exceed the Maximum. Any changes to the Maximum require a reboot.

RHEL 6 Virtualization, Paravirtualization

Paravirtualization is a big deal. It is avoided by VMware, is alchemy in Citrix Xen, and is cryptically alluded to in RHEL 6. Yet, for those of us that are almost exclusively Linux, the performance and density advantages are huge. Even Windows XP performance is noticeably improved running "para-virt". (As for Vista and Win7... they're both hogs, no matter what.)

In ten words or less, paravirtualization improves performance by loading a version of the operating system optimized for the host's hypervisor.

In the RHEL 6 Virtualization guide Chapter 8, they state that para-virt does not work with KVM. This would imply that there is no way to optimize a RHEL 6 VM on the RHEL 6 platform. Given that, why not just run VMware?

Yet in Chapter 11, they mention that para-virt drivers are automatically loaded and installed for RHEL 6 VMs and Linux VMs based on the 2.6.27 or newer kernel.

So... Which is it... Para-virt yes or para-virt no?

Survey says: Kernel no, drivers yes. But, of course, there's a catch.

Once the VM is installed and running, execute an lsmod | grep virtio. Look at the last line. The items at the right of the numbers will indicate which para-virt drivers are used. You want four of them, but may only have three. Depending on how the VM accesses the outside network, the virtio_net driver may be missing.

To enable virtio_net, a specific sequence of events must be followed:
1. Power off the VM
2. From virt-manager, Open the VM
3. Select View and Details
4. Select NIC
5. Change Device model to "vrtio"
6. Apply, exit, and Run the VM
Upon boot, the virtio_net driver should be listed.

Call it a bug, but if you try this with the VM powered on, it will claim to work, but will not.

As for performance without a para-virt kernel, I am still a little skeptical.

Tuesday, April 26, 2011

RHEL 6 Virtualization - Bridged Interface

With version 6, Red Hat has finally fixed the long standing problem of not being able to use the GUI to configure a Virtual Machine shared connection. Lets first review the types of VM network connections:
* RH=y  CX=y  VW=y Internal
* RH=y  CX=y  VW=y Dedicated or Slaved NIC
* RH=y  CX=y  VW=y Routed or VLAN'd
* RH=y  CX=y  VM=y Shared or Bridged
* RH=y  CX=n  VW=n Network Address Translation (NAT)
(Bold entries are the "default" config.)
An internal connection does not route traffic off the host system. A dedicated (also called slaved) connection, requires a separate NIC for each VM, which is very inefficient. A connection that is routed or VLAN'd requires the network be aware of the the specialized configuration. A shared or bridged connection (what we're after) extends the real world subnets in to the virtual machines. The last type, NAT, allows the VM's to communicate out, but does not permit inbound request, thus rendering it useless to servers.

Unfortunately, Red Hat uses NAT, by default, and their virtualization technology is principally used for server consolidation. As if this is a good idea. To make matters worse, their documentation still suggests manually configuring a shared connection, and does not explain that once you're done, you won't be able to see the connection in the GUI.

Here's how to do it right:

From virt-manager, connect to the host, click Edit, and select Host Details. On the Network Interfaces tab, click the plus sign (+) at the lower left. In the pop-up, select "Bridge" and Forward. Assign a name-- I recommend br followed by the eth number of the card you are sharing. In other words, if you are sharing eth1, name it br1.

On the same screen, set the start mode to "onboot", check "Activate now", and check the target NIC that you want the VMs to access. Take a deep breath, hold it, and click Finish. Scary things will happen, but after about a minute, the window should respond.

Notice that the eth item has disappeared from th elist and been replaced by the newly defined bridge. Now click the Virtual Networks tab, and notice nothing has changed. Why doesn't the new connection appear in the list? More evidence that Red Hat's interface is the least intuitive of all the vendors. This tab is a list of virtual networks, and a bridged connection is an extension of the physical network. (Yeah, while technically correct, it doesn't make sense to me either.)

When provisioning VMs, make sure to select expand Advanced options and choose the br device.

*** NOTE ***
An excellent discussion of the underlying technology is available on Dale Bewley's blog. You'll need this for kickstarting hosts.

Sunday, April 24, 2011

RHEL 6 Virtualization As Non-root User

Red Hat has always been overconfident about the use of the root account over SSH. By default they allow direct logon to the root account via SSH, because the first "S" stands for Secure. But that's not the point... You should never let anyone logon directly as root. Always access as a user, and escalate privileges.

With RHEL 6 virtualization, I ran into a problem with virt-manager, in that it refused access from a non-root account. If I logged in as myself, the app would start, but not allow connections, complaining:
Unable to open a connection to the libvirt management daemon.
Verify that:
- The 'libvirtd' daemon has been started
Further digging (clicking "Details") shows a couple Permission denied messages. Using sudo doesn't help, switching to root doesn't help, but SSH'ing in as root works. So, what's the fix?

Turns out there is security group that allows the identification of trusted users. Edit the /etc/libvirt/libvirtd.conf file and uncomment three lines:
unix_sock_group = "libvirt"
unix_sock_rw_perms = "0770"
auth_unix_rw = "none"
The third line bypasses the "polkit" security mechanism... since it doesn't work, anyway.

The natural reaction would now be to issue a service libvirtd reload command, but not so fast... that won't work. (But feel free to try it if you like-- I'll wait.)

List the /var/run/libvirt dir. You should see a libvirt-sock file owned by root.root with permissions of 700. Use service libvirtd stop, and file will disappear. Issue service libvirtd start and it should fail to start, claiming it can not load the configuration file, which is not true. The problem is that we told it to allow access to members of group "libvirt", which does not exist. Create it, add yourself:
groupadd -g 170 libvirt; usermod -a -G libvirt doug
Logoff, login, and verify with the id command.

Issue the start command again, and the service should start. List the /var/run/libvirt dir. This time, the file is owned by root.libvirt with permissions of 770. We can now access and control the server as a non-root user.

Saturday, August 01, 2009

F11: service libvirtd restart

Fedora Core 4 introduced the ability to run Xen virtualization, but it required a kernel recompile. When 5 came out, it was a question of significant config, but no recompile. Fedora Core 6 allowed for single click Xen virtualization during install. It was easy, stable, and effectively the same model deployed in Red Hat Enterprise Linux 5.

From a Xen standpoint Fedora 7 and 8 were incremental releases, with 9 being a breakthrough release. I say that, because Xen was pulled as a feature of 9-- if you wanted virtualization, you were back to compile and manual module load. The reason was a decision by Red Hat to move away from Xen (as Xensource had been purchased by Citrix) and to adapt Qemu-KVM.

With Fedora 10, Qemu-KVM was the native virtualization engine, but there was a major problem. The virtualization layer was running as service. This meant that if the service was restarted, all the VM's reboot. Mucho bado.

The good news is that in F11, it looks like the kernel virtualization modules (KVM) have been integrated with the kernel such that a restart of the service only effects the Qemu management layer. So, go ahead:
service libvirtd restart

Wednesday, November 19, 2008

KVM "TCP/IP error: VNC"

I got an error when attempting to use virt-manager to connect to the console of a virtual machine. TCP/IP error: VNC connection to hypervisor host got refused or disconnected! This seem to be from spawning a new console too fast after closing a console to the same machine. Here's a work-around:
[root@adama qemu]# virsh vncdisplay valk-b
:3
[root@adama qemu]# vncviewer 127.0.0.1:3
This assumes you have X Window support.

Monday, November 17, 2008

KVM virt-clone

One of the things I love about Vmware is the ability to deploy a VM from a template. Linux has never had the same capacity, but there is a clone feature. Just as with Vmware, the original VM must be powered off. This is because we can't copy and image file if the VM is making live changs to the image.

Unlike Vmware, we have the advantage of being able to use Logical Volumes in Linux, rather than image files. An LVM image provides better performance that a flat file because we are "closer to the bare metal" of the drive, rather than inside a file system. We can resize an LMV image using native Linux commands, rather than using a third party tool. Unfortunately, this mean cloning requires one extra step: we have to preallocate the LogVol. That should take about 10 seconds.
[root@adama ~]# lvcreate -L 5G img -n vm06f8
  Logical volume "vm06f8" created
[root@adama ~]# virt-clone -o vcl02 -n vcl03 \
  -f /dev/img/vm06f8 --force

Cloning from /dev/img/vm05f8 to /dev/img/vm06f8
Cloning domain... |         5.0 GB     02:55
In this example, we created the image LogVol, /dev/img/vm6f8, and replicated Dom vcl02 as vcl03. Since we were using a preallocated volume, we needed the --force. (No, it is not smart enough to make the the LogVol. Yes, I tried.)

Our big problem now is that both VMs have the same hostname and IP address. Since vcl02 is down, we can bring vcl03 up, and make the changes with out an conflicts. If... You had used LVM inside vcl02, and you used a different VG name than on Dom0, then you could use kpartx to mount the root filesystem, and change the config files directly.

Friday, November 07, 2008

F10-beta Kickstart Memory Reservation

Got another glitch for you. If you'll recall, one of my saddest moments in my work with F10-beta KVM was when I realized that we had lost the ability to dynamically control guest memory. It bit me again.

When attempting a kickstart, I was allocating 256 Meg of memory. The process would reach package number 10 (I think it was gcc... but who needs that) and freeze. Eventually, I found that by allocating 512, it continued the install. This man have nothing to do with KVM at all, but my have been an issue on a physical machine of less than half a gig of RAM.

There may be a sweet spot between 256 and 512, but I did not investigate any further.

Bottom line: Install at 256, then move down from there.

Thursday, November 06, 2008

F10-beta Error Kickstarting VM

Here's a tremendously perplexing bug. When attempting to kickstart a VM install from the command line, the kickstart dies with any error that it can not read from /dev/sdb.

First oddity is a warning that "The partition table on device sdb was unreadable" and needed to be initialized. This message has always been a anomoly solved by simply responding "Yes". Unfortunately, this time it is for a non-existent /dev/sdb.

After a too long delay, the second warning states "Input/output error during write on /dev/sdb" with the options of Retry / Ignore / Cancel. When I ignore, Anaconda throws exception 11.3.0.50 for KeyError: 'sdb'.

For some reason, virt-install was reserving 3876 MB of disk space for an sdb partition, but not creating the sdb1.

A dozen attempts at a kickstart %pre work around later, broke for dinner, and watched a free HD OnDemand Bond movie (Dalton). When I walk back to the system, the kicktart had worked. It turns out, this time, the kickstart worked. This one, however, was executed via the gui wizard. I examined the log files, and found the difference.

Doesn't work:
virt-install -n vinci -r 256 -f /dev/img/vm02f8
-l /vinf/ISO/Fedora-8-x86_64-DVD.iso -b virbr1
-x "ks=ftp://192.168.169.13/ks/vinci.cfg"

Works:
virt-install -n vinci -r 256 -f /dev/img/vm02f8
-l ftp://192.168.169.13/f8-64 -b virbr1
-x "ks=ftp://192.168.169.13/ks/vinci.cfg"

Can you see the difference? Look at -l. The first (bad) is attempting a local install from an ISO. The second (good) is using an extracted image via FTP.

Now the sad part: I can't report it as a bug, since I can't solve the problem. I do however have a work around, which moves me forward. It doesn't help the developer's with their code problems, but then I'm not a programmer.

Wednesday, November 05, 2008

F10-beta Network Bridge

With a little direction from a fellow Fedora'r, I've finally got the last piece of the F10-beta virtualization suite working.

By default, KVM (I hate that name... a KVM is a type of switch) creates a bridge process between the physical NICs and the virtual machines. The bridge process handles DHCP on an internal subnet and NATs the VMs to the physical world. In other words, virbr0 emulated a Linksys or D-Link home router. This works fine for outbound communications, but does not lend itself to connecting to virtual servers.

What is needed is the ability to route from an external address to a VM. This means slaving an ethernet to a bridge, assigning them to the same network, and establishing a route between the physical card and the bridge. To accomplish this, I created the following script:
[root@adama network-scripts]# pwd
/etc/sysconfig/network-scripts
[root@adama network-scripts]# cat ifcfg-virbr1
#!/bin/sh

IF="eth1"
NET="192.168.69"
HOST="11"
BR="virbr1"

brctl addbr $BR
ifconfig $IF 0.0.0.0
brctl addif $BR $IF
ifconfig $BR $NET.$HOST netmask 255.255.255.0 up
route add -net $NET.0 netmask 255.255.255.0 $BR
route add default gw $NET.1 $BR

exit 0
First the script location. By placing it in network-scripts, it is sourced at boot and service network restart. The name is not relevant except for the minor detail that the ifcfg's are executed alphabetically, so the filename has to start with letter FOX or higher.

Second, the script uses variables for portability. Notice that $NET.$HOST form an IP address. This will end up being the primary interface on the system, as the last command will switch the server's default route to the bridged interface.

Fianlly, I would suspect there is a way to embed this into the virtlibd subsystem. At this time that is not a priority and I think this may provide more flexibility. I say that because adding this script to a kickstart is simply the addition of a cat >> x << EOFcommand in post.