Saturday, August 29, 2009

HowTo stunnel, Pt 2

In my example config, I specified:
First, the entry syslog=yes writes to the /var/log/secure, which renders stunnel.log to be redundant.

Second, the debug of 7 is more than we need for production. Under normal load, does not log anything, including a restart of the service. Debug 5 gets the starts and stops, but logs three lines per transaction. It's a thin line to tread, but looks like I'm going to use 5.

Next problem is the fact that, as presented, the server will accept connections from anyone. The client's PEM was unused. We need to tell the server force clients authenticate. We'll do this with key exchange. Add two lines to the server's config file:
We won't really have a CA, but it will be close enough. Copy the server.pem to, which we modify in a few minutes.

On the client:
echo "-----`hostname`-----" > `hostname -s`.crt
sed -ne '/BEGIN CERT/,/END CERT/p' \
    <server.pem   >>`hostname -s`.crt
Ship this off to the server.

On the server, we will add the CRT we just transferred from the client to the CA we created a few moments ago:
cat clientfile.crt   >>
Restart stunnel on both ends. Unless a client's CRT has been added to the server's CA, they can't get in.

Friday, August 28, 2009

Chinatown Arch: Almost Done

They unveiled the DC Chinatown Arch. Still have some of the scaffold left. Newly refurbished, nice and pretty. I'm covering for a coworker on the morning shift, so the sun is at the wrong angle. I'll have to get a better picture next week.

Wednesday, August 26, 2009

Console is not yet active for guest

I was barking up the wrong tree on this one. I had a VM that would not start, displaying the message:
Console is not yet active for guest
My confusion was caused by the fact that if I shutdown a different VM, this one worked fine. Furthermore, this was effecting multiple VMs, besides just the two. I even went so far as to call tech support on this one. Gasp!

Needless to say, I solved this before they did. The problem is a quirk in the RHEL5 virt-clone command. This command is suppose to copy a VM image, while making the needed changes in the descriptor file in the process. It does not, however, change the VNC port number. This has the effect of forcing two machines to use the same video card. In the directory with the VM descriptor files, try:
cd /etc/xen
grep "^vncdisplay" * | sed 's/"//g' | sort -n -k3
Any duplicate values define a conflict-- whoever boots "powers on" first, wins. The looser gets power, but can't initialize their BIOS.

To resolve, edit the descriptor file, and assign a unique value: But watchout! Make sure it is unique across the cluster, otherwise a migration could cause a collision.

Now I guess I got to tell Red Hat.

Sunday, August 23, 2009

Limited Edition Star Trek Waffles

I bought all they had. I've cornered the market! Just imagine: in a couple years, there no telling how much these will be worth... After all-- they're limited edition. Watch out E-bay!

Saturday, August 22, 2009

HowTo stunnel

I ran into a problem that I thought stunnel might be the solution, but found Fedora's stunnel RPM to be in complete. First it has no init script and second it's documentation is lacking. Further the same problems exist with the code (which I couldn't get to compile!) I'm sure there is more than two problems, but...

Over at, there is a working init script. Grab it and save it to /etc/init.d. Next we need an stunnel.conf (also not included, though there is an example... which won't work.) I found a good set of example files over at Edna Narrabilis' Blog. These may need a few tweaks, but since I'm working on an remote mySql issue, these are good.

Notice in Edna's configs, there are really only two differences between the client and the server. First, not the client=yes directive: just to be safe, add client=no to the server side. Second, notice the client's connect line includes the server's name. Drop her configs on each sides /etc/stunnel directory.

Now that we've got the Fedora RPM, Gaztronics init.d, and Edna's configs, we need an SSL certificate. When you generate the certs on both sides, it is important that for Common Name, you use the fully qualified, DNS resolvable, name for each machine.

As I mentioned, I made a few tweaks. For instance, I moved the PID out of /tmp. I also forced a few other options. Just for good measure, do this first:
mkdir /opt/stunnel
chown nobody /opt/stunnel
chmod 700 /opt/stunnel


One thing to notice: we are flipping the port as they go across the network from 3306 to 6639. This way we can use iptables to block 3306 as adefense against automated SQL injection attacks. I found that If I tried to do the client accept at 3306, it didn't work, but I think that's a mySql thing.

Wednesday, August 12, 2009

Virtual PC 2007 VM Component Files

I'm finally satisfied with my Virtual PC 2007 install on my Vista laptop. Just for reference, the VM is made up of several files in a dedicated directory... just like VMware. In the table below, the first item is the file extension, followed by the extension's name, followed by a description of what the file does.
vmc = Virtual Machine Code, VM descriptor file
vhd = Virtual Hard Disk, the actual image (may be more than one)
vud = Virtual Undo Disk, changes to disks are discarded when
            the VM is powered off (great for testing things)
vsv = Virtual SaVe state, allows the VM to be paused

Citrix XenServer P2V

An interesting feature of the Citrix XenServer install CD is the option to execute a physical to virtual (P2V) migration. Having significant experience with VMware Converter, I thought I'd use this to create a VM from a Windows XP instance sitting on a drive that I never use. From my point of view, it would be a legal version of XP, though I'm sure Microsoft's lawyers would argue otherwise.

The first shot at P2V failed with a message specifying a partition error with /dev/dm-5. My immediate concern was that the USB multi-card reader was causing trouble. I disconnected it, and tried again with the same results.

A little research found a completely unexpected result. Turn out Citrix P2V only works on Linux, and not Windows. Looks like the old NTFS patent issue. Further, it only works on "older" Linux, somewhere in the Fedora 8 and before range. The problem doesn't seem to have anything to do with USB, but instead the way the migration tool interprets the device mapper.

I decided to hold off on further use of Citrix P2V, for the time being, but am continuing research on doing a manual P2V of that Windows instance.

Monday, August 10, 2009

Trouble Makers

Metro has an advertising campaign meant to convince people to be nice to handicapped people. Then I noticed something about the posters: Its the same two people causing all the trouble. So I figure these are wanted posters, not advertisements. If you see them, grab'em!

Sunday, August 09, 2009

Vista VM Disk Size < 12Gb

I recently installed Vista in a VM on my Linux virtualization platform, running Qemu/KVM. Today, I installed a Vinat on a VM on my Vista laptopn, running Microsoft Virtual PC 2007. As part of the experiment, I wanted to how small I could make the virtual disk. On my Linux platform, I gave Vista 20 gigabytes, and was disappointed to find that it took about 12GB of space.

This time, I gave it 10GB, expecting to get an error, and found that it installed on about 7GB, This seems to imply that Vista, which no long prompts for "what do you want to install" makes these decisions for you based upon the space it sees. By giving it less space, it put less crap on the drive. I was able to clean about another 500MB by deleting Games and few other silly things.

Unfortunately, the VM would not install SP1 from disk, as it needed 4GB to extract. By adding a second virtual drive of 5GB as a "D Drive", SP1 was able to install. I actually removed the second drive once loaded. Thus, it looks like it could be possible to run Vista on 8GB, but 12GB seems to be the base.

*** Update 8/12/2009 ***
Using a second disk was too unstable. After installing SP1 and a dozen security patches, the 8GB disk reached capacity. and the VM would lock up. There was no easy way to shift components from the C: drive to D:. I ended up rebuilding at 12GB, and after updates have 3GB left-- acceptable for small testing tasks.

Tuesday, August 04, 2009

Dual Boot Citrix Xenserver 5

I'm very proud of this hack. So proud that it's actually a quad boot, but two of the four are Windows, which we don't care about, so we'll stick with the title of dual boot. Here's the scenario:

A couple weeks ago, I did the VCP-310 cert, with the anticipation of rolling that into the VCP-410 next month. Until then, I'm going to do the Citrix CCA cert. This means we need a Citrix Xenserver... Unfortunately, the only machine I have that is not doing something is my oldest kid's desktop. Now, this is not as bad as it sounds, as he's off doing a summer session at some university somewhere, or something.

Since he's gone, we'll wipe the drive and load Citrix. Not so fast! He's got Windows on that thing and uses it as a game station when he's in town. I'd already partitioned it and loaded it with boot to Fedora as well as Windows, so Citrix should be a non-issue. Except, Citrix Xenserver thinks it should be the only thing on the system... Just like ESX.

So, here we go:
1. Disconnect the hard drive and load a second hard drive, pinned out as master.
2. Load Citrix Xenserver in the default configuration. Reboot and test.
3. Disconnect second hard drive and pin out as slave. Cable it and the original drive to the primary controller.
4. Boot to BIOS and ensure both are visible. This ensures that both are pinned correctly.
5. Boot off the previously existing Linux partition to Single. Use fdisk -l to validate that the second drive is visible.

(This is where it starts to get hairy.)

6. The master appears as /dev/sda and the secondary as /dev/sdb. Mount the second drive:
mount /dev/sdb1 /mnt
ls -l /mnt
Notice anything unusual? No boot partition. Citrix is loading the MBR with a LILO like loader.
7. Inspect /boot:
Some of this looks familiar. We just need to arrange it into our existing grub.conf.
8. I looked at an old FC6 Xen platform's grub.conf:
title Fedora Core (2.6.20-1.3002.fc6xen)
    root (hd0,0)
    kernel /xen.gz-2.6.20-1.3002.fc6
    module /vmlinuz-2.6.20-1.3002.fc6xen ro root=/dev/hda3
    module /initrd-2.6.20-1.3002.fc6xen.img
(Why in the world am I still running Fedora Core 6? Because, it is most similar to RHEL5.)
9. A closer look at /boot will confirm that many of those files are actually symlinks. So lets add the following to grub.conf:
title Citrix Xenserver 5.5.0
    root (hd1,0)
    kernel /xen-3.3.1.gz
    module /vmlinuz-2.6-xen ro root=/dev/sdb1
    module /initrd-2.6-xen.img
Notice the root line-- hd1,0 not hd0,0-- we're on the second drive. Notice the vmlinuz line-- we are using /dev/sdb1-- that's what we mounted.
10. Reboot... and watch it fail!

I'll save you the agony of troubleshooting. First, in the Fedora example, boot is partition 1, so we prefix the files with a slash (/). Citrix did not use a boot partition: root is partition 1. Therefore, we need to prefix the files with the directory: /boot/

Second, we mounted /dev/sdb1, which was the Xenserver's root partition. Yet, when Xenserver boots, it sees the drive as /dev/hdb1. This is because I happen to have Fedora 8 loaded, and effective F8, all drives are sd, even though the drives are actually IDE rather than SCSI. Since Xenserver is based on el5 (look at the vmlinuz links) it sees the drive as hd.

Here's what actually worked:
title Citrix Xenserver 5.5.0
    root (hd1,0)
    kernel /boot/xen-3.3.1.gz
    module /boot/vmlinuz-2.6-xen ro root=/dev/hdb1
    module /boot/initrd-2.6-xen.img

Saturday, August 01, 2009

Xen error "could not find any free loop device"

On one of my Xen systems, I added a disk to a VM, but when I attempted to start the machine, it threw up an error:
could not find any free loop device
With a little help fron Google, I found a solution.

Normally, I run m VMs in LVM because they are faster to create, easier to resize, and perform better. In this case, the VM had to be an image file for easy conversion from Xen to ESX. There were four VMs with two disks each. That's 8 disk total. (Yes, I realize you can count, but that number is important.

Turns out that each image file uses a loop device, as if mounting an ISO file. By default there are only 8 /dev/loop's. (See, I told you it was important.) Here's how we add more:
ls -l /dev/loop* | wc -l
echo "options loop max_loop=64" >> /etc/modprobe.conf
rmmod loop
modprobe loop
ls -l /dev/loop* | wc -l

F11: service libvirtd restart

Fedora Core 4 introduced the ability to run Xen virtualization, but it required a kernel recompile. When 5 came out, it was a question of significant config, but no recompile. Fedora Core 6 allowed for single click Xen virtualization during install. It was easy, stable, and effectively the same model deployed in Red Hat Enterprise Linux 5.

From a Xen standpoint Fedora 7 and 8 were incremental releases, with 9 being a breakthrough release. I say that, because Xen was pulled as a feature of 9-- if you wanted virtualization, you were back to compile and manual module load. The reason was a decision by Red Hat to move away from Xen (as Xensource had been purchased by Citrix) and to adapt Qemu-KVM.

With Fedora 10, Qemu-KVM was the native virtualization engine, but there was a major problem. The virtualization layer was running as service. This meant that if the service was restarted, all the VM's reboot. Mucho bado.

The good news is that in F11, it looks like the kernel virtualization modules (KVM) have been integrated with the kernel such that a restart of the service only effects the Qemu management layer. So, go ahead:
service libvirtd restart