No Reserved Words in XML

Or so goes the mantra, but if that's true then why can't I use this syntax:

<?xml version="1.0"?>
<parse>
    <rule id="1">
      <type>pattern</type>
      <description>might be time value</description>
    </rule>
</parse>

It passes all my tests for well formed XML.  Why can't I use it?

Well, obviously, I can... so here's the story.  I've got some Perl code that I've used dozens of times before, but all of a sudden didn't work with this data.  After scoring the code and the the internet for a reason that this data structure wouldn't work, I finally changed the data to read , and everything worked fine.  This would seem to be a happy ending except for on small detail:  Its not my data.

To better understand what's going on, let me explain "doesn't work."  There is a Perl module called XML::Simple.  It combines dozens of steps into three lines:

use XML::Simple;
my $xml = new XML::Simple;
my $data = $xml->XMLin("file.xml");

These lines open the file, read the lines, parse out the tags, and assign the values into a dynamically allocated hash.  By adding a call to a Data::Dumper, we can look at the hash structure of the XML data:

$VAR1 = {
  'rule' => [
    {
      'id' => '1',
      'type' => 'pattern',
      'description' => 'might be time value',
    },

Or that's how it should break out.  Instead, it breaks out like this:

$VAR1 = {
  'rule' => [
       '1' => {
           'type' => 'pattern',
           'description' => 'might be time value',
        },

Which broke my normal subroutines.  Yet, if I change "id" to "item", everything works as expected.  So if its true that there are no reserved words in XML, why doesn't it work?

It turns out, for some bizarre, undocumented reason, the Perl XML::Simple module has decided that there are reserved words in XML.  And those words are name, key, and id.  If those words are found as tags in an XML structure, they are promoted to elements. Though CPAN does not explain why this is the case, they do provide a solution:

my $data = $xml->XMLin("file.xml",KeyAttr=>[]);

By setting the option KeyAttr to "none", the parser behaves as it should.

dracut: FATAL: initial SELinux policy load failed

Here's an obnoxious install failure: Using the RHEL/Scientific Linux 6.3 DVD, it is possible for an install to crash on first boot with an SELinux error.  The problem is a bug in the post of the target policy RPM.  The bug is immediately fixed by running yum update... assuming you can figure out how to get the machine booted.  Luckily, the error is nice enough to tell you how move forward.

Reboot, and at the GRUB menu, append "selinux=0" to boot into Permissive mode.  From the root prompt:

ls /etc/selinux/targeted/policy/policy.24
ls: cannot access /etc/selinux/targeted/policy/policy.24:
No such file or directory

If possible, issue: yum update selinux-policy-*

If the machine is not network connected, the problem can be resolved by restoring the file policy.24 from install media.  And it it were that simple, you wouldn't need me.  You will have to force install two RPMs:

rpm -ivh --force selinux-policy-3.7.19-*
rpm -ivh --force selinux-policy-targeted-3.7.19-*

The second force install will take several minutes to complete.

Regardless of how the issue is resolved, it is best to relabel the filesystem:

touch /.autorelabel

RHEL Cluster Anti-Affinity Configuration

I'm often amused by how vendors define "High Availability", aka HA.  Customers always talk about Five Nines, but "off the shelf" HA solutions seldom achieve 99% availability.  In the case of RedHat's HA cluster service, the default configuration might provide unattended failover within 60 seconds.  Given that Five Nines only allows 25.9 seconds per month, a single failure can blow away a service level agreement.

To counter the real-world lag of application failover, an system must be load balanced across a cluster.  A real HA environment would be at least three nodes, running two instances of the service.  If one node fails, the load balancer will redirect everything to the second instance, while the service is recovered on the third node.

There is an HA problem that VMware has addressed in their HA solution, that RedHat has not, known as the anti-affinity rule.  Affinity is when two "processes" favor the same resource.  An example would be when running a web and database instance on the same machine improve performance.  In the case of redundant services, running them on the same machine is pointless, if the machine fails.  To prevent this, we need an anti-affinity rule that requires the two processes to never be on the same machine.

RedHat cluster suite provides affinity in the form of child services.  If the cluster moves the web service to another node, the database has to follow.  What they don't provide is an anti-affinity rule to prevent the load balanced services from trying to run on a single node.  As a matter of fact, by default, all services will start on the same cluster node.  (It will be the node with the lowest number.)

I found I could implement anti-affinity from with in the service,s init.d script.  First, we add an /etc/sysconfig/ file for the process, with the following variables:

CLUST_ENABLED="true"
CLUST_MYSERVICE="service:bark"
CLUST_COLLISION="service:meow service:moo"

A collision is when the presence of a service prevents this service from starting on this node.  The names should be listed exactly as they appear in clustat.  Make sure the script sources the config file:

# source sysconfig file
[ -f /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

Next, add a new subroutine to the existing init.d script:

cluster(){
  # look for other services on this host
  K=$(for J in $CLUST_COLLISION; do \
          clustat | grep "$J.*$HOSTNAME.*started" \
          >/dev/null; \
          [ $? == 0 ] && echo "$J "; \
          done)
  if [ $K ]; then
    # show service names
    echo -n "Cluster, collision $prog: $K"
    # fail, but with a success return code
    failure; echo; exit 0
  fi
  # look for this service running on other nodes
  K=$(clustat | grep "$CLUST_MYSERVICE.*started" | \
          awk '{print $2}')
  if [ $K ]; then
    # show hostname of other instance
    echo -n "Cluster, $prog exists: `echo $K | cut -d. -f1`"
    # fail but with a success return code
    failure; echo; exit 0
  fi
}

Finally, add a reference to the cluster sub in the start sub:

start(){
  if [ $(ps -C cluster-$prog.sh | grep -c $prog) == 0 ]; then
    # only check cluster status if enabled
    [ "$CLUST_ENABLED" == "true" ] && cluster
    echo -n "Starting $prog"

Here's what happens in the case of a collision:

• rgmanager issues a start
• the cluster sub recognizes the collision, but tells rgmanage that it started successfully (exit 0)
• rgmanager shows the service as running
• 30 seconds pass
• rgmanager issues a status against the service, which fails, since the init.d script lied about the service running
• the cluster orders a relocation of the service
• rgmanager issues a start... on a different node
• there is no collision this time, so the init.d runs as expected

RHEL 6 Clustering, VM Fencing

I recently retasked one of my lab machines as a RedHat virtualization server, which RedHat calls RHEV, but is really KVM.  One of this machine's tasks is to support a test cluster of VMs.  Under normal circumstances, clustering would require a remote management interface such as an ILO, DRAC, or RMM.

As usual, I was disappointed with how difficult this was.  To make matters more difficult, for you, I won't be covering clustering in the article.  This document's scope will be limited setting up to RHEV VM fencing.

On the host machine, we need to install the fence daemon.  Considering this is very lightweight, the I'm going to do a shotgun install:

yum install fence-virtd-*

On my machine, this loaded four packages: the daemon, the interface between the daemon and the hypervisor, and two "plugins".  (The serial plugin is probably not needed.)

The base RPM will provide the /etc/fence_virt.conf file.  Modify it to look like this:

listeners {
  multicast {
    family = "ipv4";
    interface = "virbr0";
    address = "225.0.0.12";
    port = "1229";
    key_file = "/etc/cluster/fence_xvm.key";
  }
}
fence_virtd {
  module_path = "/usr/lib64/fence-virt";
  backend = "libvirt";
  listener = "multicast";
}
backends {
  libvirt {
    uri = "qemu:///system";
  }
}

Two things to notice about the config file.  The key_file option is little more than a password in a text file, which is going to have to be duplicated on all the VMs in the cluster.  The "theory" is that only a device with the password will be able to fence other nodes.  This brings us to the second point, the multicast option.  If a cluster node issues a fence command, the symmetric authentication key will be multicast on the network in the clear.  Thus, the reality is that the key_file provides no real security.

Which brings us to a second issue with the multicast.  Per RedHat, cross host fencing is not supported.  As such, all cluster nodes have to exist on the same physical machine, rending real world VM clustering pretty much worthless.  Here's the reality of cross host fencing: It is not supported because of the security concerns of multicasting the clear text fencing password and the fact that RedHat cannot guarantee the multicast configuration of the switch infrastructure.  Given properly configured switches, a dedicated host NIC and virtual bridge in each host, cross host fencing works.  In this lab configuration, however, it is not a concern.

After creating a key_file, open the fenced port in IPtables:

-A INPUT -s 192.168.122.0/24 -m tcp -p tcp --dport 1229 -j ACCEPT

Copy the key_file to each clustered VM (they don't need the config file) and add the opposite IPtables rule:

-A OUTPUT -d 225.0.0.12 -m tcp -p tcp --dport 1229 -j ACCEPT

On the host, chkconfig and start fence_virtd.  Running netstat should show the host listening on 1229.  What it is listening "for" is the name of a VM to "destroy" (power off.)  This means the names of the cluster nodes and VMs recognized by KVM/QEMU have to match.  On the host, display the status of the VMs using: 

watch virsh list

Given a two node cluster, on node1 issue:

fence_node node2

On the host, the status of node2 should change from running to inactive, and a moment later, back to running.  For testing purposes, the fence_node command can be installed on the host, without the host being part of the cluster.  If you try this using yum, you'll get the entire cluster suite.  Instead, force these three RPMs:

rpm -ivh clusterlib-*.x86_64.rpm  --nodeps
rpm -ivh corosynclib-*.x86_64.rpm  --nodeps
rpm -ivh cman-*.x86_64.rpm  --nodeps

 Truthfully, the better choice is to build a VM to manage the cluster using Luci.

Kickstart from Hard Drive ISO

I'm building a machine that may need to be remotely re-imaged, without the benefit a kickstart server.  I've always heard that you can kick a machine from itself, but had never tried it.  Truthfully, it's probably more trouble than its worth.  The best option would be to install a DVD drive with media, but configure the BIOS such that DVD is after the main drive.  Since I didn't want an optical drive on this machine, here's how to kickstart a machine from a hard drive.

Let's do this backwards.  Get the machine to image off an HTTP server and then change the ks.cfg:

#url --url http://1.2.3.4/rhel6
harddrive --partition=/dev/sdb1 --dir=/

Notice that I'm telling the machine that its image is on sdb, not sda.  Providing two drives is safer that trying to image off the boot/root drive, but it could be a partition on the same drive.  Besides, I've got dozens of little drives laying around doing nothing.  Further down in the file, I also indicated:

#clearpart --all
clearpart --drives=sda

Next, we mount the second drive and copy the ISO into the root of the drive.  To clarify:

mount /dev/sdb1 /mnt
cp rhel-6-x86_64-dvd.iso /mnt/

When kickstarting from a local drive, we use the ISO file itself... not an extracted or loop mounted filesystem.  Looking back at the first change we made to the ks.cfg, we indicated --dir=/, so we are telling the installer that the ISO is on the top level of the drive.

As a matter of convenience, mount the ISO, because we need a few files from it:

mkdir /mnt/rhel6
mount rhel-6-x86_64-dvd.iso /mnt/rhel6 -o loop

Copy three files:

mkdir /mnt/images
cp /mnt/rhel6/images/install.img /mnt/images
cp /mnt/rhel6/isolinux/vmlinuz /mnt/images
cp /mnt/rhel6/isolinux/initrd.img /mnt/images

If you were going to allow the machine to rebuild different versions (or distros) you would want to add a version number to each file.

To initiate the rebuild, we will use the tried and true Grub rebuild hack:

cd /boot
cp /mnt/images/vmlinuz /boot
cp /mnt/images/initrd.img /boot
cat >> /boot/grub/grub.conf << EOF
title Rebuild
  root (hd0,0)
  kernel /vmlinuz ramdisk_size=8192 ks=hd:sdb1/ks/ks.cfg
  initrd /initrd.img
EOF

Many docs indicate that the second drive has to be made bootable.  In this case, we are still booting off the primary drive, but only long enough to read the ks.cfg from sdb and switch to install.img.  Once install.img has control, it will read the clearpart command, wipe sda, and reinstall from sdb.

There is a "gotcha" I've not quite worked out yet.  As we all know, Linux is notorious for renaming drive letters at boot time.  It is possible that  the machine might confuse sda and sdb.  This could be disastrous if the machine crashed, and while trying to rebuild, it wiped the image drive!  The good news is that the installer reports that it can't find the image and kickstart file, and fails.  Just reboot until it finds the correct drive.

* It would seem that either a UUID or LABEL could be used in both Grub and the kickstart file.  I'll add checking those possibilities to my ToDo list.  Or you could figure that part out and let me know which works.  Its only fair: I've already done the hard part for you.

Citrix Xenserver: Apply Multiple Updates

As a result of reorganizing the servers in my lab, I had to reinstall Citrix Xenserver.  I should have downloaded 6.1, but decided to keep it at 6.0 and apply the updates that I already had on the NAS.  All went well with the install, I moved all my VMs and templates to this machine, and retasked the other machine.

When I went to load the updates, a funny thing happened... It refused to load more than one, and expected to reboot after each.  After a moment of thought, I realized that I had probably never tried to load two at a time before.  It seems like something that should be simple, but the procedure is not obvious.

Here's how:

1. Highlight a server, click the "General" tab, and expand the "Updates" pane.
2. In the "Updates" pane, notice which updates have already been applied.
3. On the menu bar, click "Tools", "Install Software Update", and "Next".
4. Click the "Add" button, select the lowest numbered update, and click "Open".
5. At this point, its tempting to add another update, but don't: yet.
6. Click "Next", select one or more servers, and click "Next".
7. A check will be run against the update.

If the check succeeds, click "Previous", "Previous", and repeat from step 4.

If the check fails, then two things.  First, click "Cancel" and start the entire procedure over again, but don't add the update that failed the test.  Second, don't blame me-- I didn't create the interface.

Once you've added all the relevant updates, click "Next".  You'll have the choice of performing post install steps automatically or manually.  What this really means is reboot now or later.  If you select manually (reboot later,) it is possible that some of the updates will fail, but that's actually okay.  When an update succeeds, it appears in the "Updates" pane as Applied.  If it fails, it appears as Not applied.

To get activate the not applied updates, repeat steps 1, 2, and 3, but instead of step 4, highlight the not applied update.  Continue through the rest of the steps, making sure to do automatic, as recommended.

Linux KVM Disk Drivers

I was having a problem with storage device names on virtual machines running on a RedHat KVM host.  Occasionally, I'd build a VM and the storage device would be named /dev/sda and other times /dev/xvda.  I quickly found that if I created the VM using virt-install, I got a xvda device, and if I used virt-manager (the GUI app), I got the sda device. After some investigation, I've discovered where things went wrong.

First, the syntax of the virt-install command changed in RHEL 6, and I was still using the RHEL 5 command.  Rather than complaining, RHEL 6 would guess what it thought I meant.  Here's the wrong command:

virt-install -n server -r 512 -w bridge=virbr0 \
-f /var/lib/libvirt/images/server.img -s 10

The -f/-s options says to create an image file the is 10GB.
Here's what was implemented:

virt-install -n server -r 512 -w bridge=virbr0 \
-disk path=/var/lib/libvirt/images/server.img,\
bus=ide,size=10

Rather than complaining that the -f/-s options were deprecated, it invoked the new syntax and assumed I wanted an IDE drive, which on RHEL 6, is named as if it were an SCSI device.  We can force the paravirt driver by using the correct command:

virt-install -n server -r 512 -w bridge=virbr0 \
-disk path=/var/lib/libvirt/images/server.img,\
bus=virtio,size=10

Second, the GUI does not allow a VM's disk type to be selected from the install wizard-- it always defaults to the paravirt driver.  To force a specific driver, on the last screen of the wizard, check the box for "Customize configuration before install".

This will open a new window listing the VM's hardware.  Select "Add Hardware", select "storage", and configure a second disk the same size as the first.  At this point, there is a pull down menu that will specify the driver.  Once the second disk is in place, remove the first.  Removing the first disk before the adding the replacement disk can cause problems.

Hint: Once you've modified the hardware, there is not "Apply" option.  Just close the window and the VM will launch.

Kitchen is Getting Close

Got the cabinets-- waiting on the counter top.

Turns out installation was not included with the vent hood, like that makes sense. What's that you say? "Nickel and dime"? Yep.

Certified Ethical Hacker

I recently took the Certified Ethical Hacker (CEH) class and certification exam.  First, I passed.  Second, I was a little disappointed with the class.

Let's take a look at the first item: I passed the test.  How can anyone reasonably complain about passing a certification test?  Let me contrast the certification test with three other tests. 

RedHat certification tests are all hands on:  Here's a broken computer, fix it.  Generally speaking, if you have at least one year of experience with Linux, take a RedHat class, understand the hands-on labs, you can pass the test.  The ITIL advanced certifications are almost the opposite.  Unless you have several years of workplace experience making IT management decisions, the class is of little help with the certification exams.  In the case of ITIL's advanced certifications, the proctored, paper exams test your ability to apply their methods to your real-world experience.  And then there is VMware, whose certification is a multiple choice, computer based test.  A quick breeze through the VMware docs, and just about anyone can pass the test.  As such, VMware requires you to take their class before you can get the certification, which makes the VCP little more than an indicator of class attendance.

The CEH is a multiple choice, computer based, exam, like VMware's.  The difference, however, is that (having taken the class) I'm not certain I could have passed the test based only on what I learned in the class.  Even though the class is structured like a RedHat class, with lecture and hands-on labs, I feel the exam required some real world experience.

Don't get me wrong... I'm not saying that "simply" taking the class should be enough.  I do agree that a candidate should have some experience in the area of study, but I feel that the purpose of the class and labs should be to solidify what they've seen, fill the gaps in what they haven't, and help them identify where they are weak.

And this brings me to the class.

Every class starts the same way-- introduce yourself and say what you hope to get out of this class.  Most people say something like "I'm John, and I want to pass the test."  This time, I said:

I'm Doug, and for the last ten years customers have insisted that I implement obscure security protocols, but I've never seen someone demonstrate that they can successfully breach a properly configured system.  I'm hoping this class will provide some validation that there really is a threat more sophisticated than scripts looking for default passwords.

What did I learn from the class?  Three things:  Windows sucks, Linux is invincible, and once a month 10% of users should all be shot.

At this point, let me interject that the CEH course material was the highest quality training material I have ever seen.  They had color graphics, high quality artwork, no diagrams stolen from vendor brochures, the books event had spines... like a real book that you'd buy at a book store.  We got six disks worth of tools.  And goodies like a backpack and a T-shirt!  The first half hour of class was like being six years old on Christmas day.  The rest of the class was like the feeling you have after you've opened all the Christmas presents, and you realize that the its all over.

Some of the labs were interesting, but there are only so many times you can demonstrate that Microsoft has sacrificed security for usability.  After a couple days, the fact that insiders and stupid users are allowing access to the network was well worn.  There really was no need for more than one lab demonstrating that organizations expose too much information to Google and Netcraft.

I'm going to end with one last thought, and this really doesn't have anything to do with CEH.  Human beings can learn anything from books, but we like to be taught by other human beings.  An instructor provides three basic services in a class:

1. Focus the student's attention on what is really important in the book, and identify the fluff and filler.
2. When a student indicates they do not understand the book, they offer more detail or alternate examples.
3. Provide value-add in the form of real world examples or relevant material outside of the book.

If you ever find yourself as a technical instructor, pay heed to what I'm about to say next:  If you can't fulfil at least one of the services above, simply being [ cool | fun | entertaining ] isn't enough.

Tracking SSH Tunnels

Native to Secure Shell (SSH) is the ability to create point-to-point, encrypted, tunnels.  The function was designed to provide legacy protocols, such as mail (SMTP/POP) with encryption.  A user could login to an SSH server in their company's DMZ, open a tunnel from their laptop to the server, and redirect their mail client through the tunnel.  On the surface, this sounds like a good idea: it protects the exchange of company data from the "protected" corporate intranet to users "in the field".

But, as with all good things, there is room for abuse.  Consider the opposite scenario:  What if a user inside the corporate intranet SSH'ed to the DMZ server and built a tunnel to allow them to surf the web, thus bypassing the content filters?

Granted, content filters are just a way for the man to oppress middle class workers.  By censoring free thought, the 1% is able to keep the 47% running on the hamster wheel of consumerism.  Hear me, my brothers!  There will come a day when the proletariat will raise up and declare their freedom from the jack-booted thugs of Wall Street and their Illuminati masters.

But I digress...  Where was I?  Oh yes, SSH tunnels. So the question is this:

How can we monitor the SSH tunnels defined on the server to ensure they are not being abused?

Much to my surprise, the answer is:  You can't.

There does not seem to be any mechanism for determining what tunnels exist, and here's why.  The tunnel is defined on the client end, where the SSH application is always listening.  When the client receives a packet that matches a tunnel, the packet is shipped to the server with handling instructions.  When the server gets the packet, it opens the needed socket, fires it off, then closes the socket.  In other words, the connection from the server to the destination is not persistent... it behaves more like UDP than TCP.

Since a socket is opened, it is possible to capture it with lsof -i, but since the socket is transient, trying to catch it in a while/do loop is a matter of pure luck.

This means we have two choices, one of which shouldn't count.

In order to catch someone using a tunnel to surf out of the DMZ, we need an IPtables rule to catch the outbound packets.  As it turns out, any packet originating from a tunnel will use the server's IP address as the source address.  We only need to log the initial connect, so we only need to log the SYN flag.  To further complicate things, our abusive user has to be using a proxy, so we can't restrict our checks on port 80 and 443.

iptables -A OUTPUT -s 192.168.0.1
  -o eth0 -p tcp --syn
  -j LOG --log-prefix "out-syn "

Here, we are looking for OUTPUT, since we are assuming that this DMZ machine is supposed to be building tunnels.  The (-s) address is the address of the DMZ machine.  In this case (-o) eth0 is the internet side of the machine and eth1 would be the intranet side of the machine.  Notice that no port number is assigned to the (-p) TCP statement.  Lastly, we are going to log this message.  (The trailing space in the quotes is significant.)

This rule will catch bad tunnels, but ignore good tunnels, on the grounds that good tunnels will use (-o) eth1 to get to the intranet resources.

If you'll recall, I said there were two choices.  The second is this:

iptables -A OUTPUT -s 192.168.0.1
-o eth0 -p tcp --syn
-j DROP

In this case, we are refusing all outbound TCP traffic from the DMZ machine.  (Since DNS is UDP, we can still resolve the addresses of the inbound SSH connections.)  As stated above, we are allowing the good tunnels, since they use (-o) eth1.

So which of the two rules shouldn't matter?  The first:  We shouldn't have to "catch" abusive users, we should just stop them.  Of course, we could use both lines to first log them and, second, prevent the connection.  This allows us to know who the abusers are, and bitch slap them for their feeble attempt-- for they are probably using Windows workstations, and deserve to be degraded.

What's that you say Mr. Boss?  You want me to prove abuse exists before locking down the DMZ.  Okay, we implement rule number 1, log the abuse, and then later lock down with rule number 2.

What's that you say Mr. Boss?  Prove the abuse exists without implementing rule number 1.  Ah...  No can do.

Oh well, if you want me, I'll be in my cube.  Listening to Slacker internet radio, via an SSH tunnel, through the DMZ.